OrrinInfinity SDK

01 // Introduction

The canvas comes alive
from your process.

OrrinInfinity SDK is the Python connection between your code and the infinite board - a thin contract for compute actions, typed responses, and interactive surfaces that land as first-class objects on the canvas.

It is a subset of OrrinSDK: register handlers, return structured results, open a live window onto the machine running your process. The board places what you emit - text, components, images, or a living surface driven by WebRTC.

One package. One session entrypoint. Your imagination, on an infinite landscape.

Python 3.10+ InfinityBoardComputeAPI Interactive surfaces WebRTC screen stream Part of Stellr

02 // Install

Install with pip

One package gives you the board compute API, response builders, screenshot helpers, and interactive surface controls.

Terminal

$ pip install orrin-sdk

Requires Python 3.10+ · Depends on websockets, Pillow, aiortc, pyautogui

03 // Architecture

How a compute session works

Your process connects to the Infinity central unit over WebSocket, registers named compute actions, and serves them for the lifetime of the session. Results are placed on the board as native objects.

Instantiate the API

Create an InfinityBoardComputeAPI instance. Optionally set a max WebSocket message size (default 10 MB).

Register compute actions

Decorate handler functions with @api.compute_action(name, supported_components). Each must return an InfinityComputeResponse.

Start the session

Call asyncio.run(api.start()). The SDK connects, sends a handler handshake, registers your command list, and receives a compute ID.

Serve requests

The central unit dispatches use_compute_command, surface stream, interaction, and keystroke requests. Your handlers run; responses are sent back as JSON.

Board placement

The board interprets result_type and payload fields - raw text, text/image components, or interactive-surface metadata - and mounts them as spatial objects.

04 // Core types

Three types define the contract.

The response object, the builder that constructs it (including the nested WrappedScreenshot), and the API that registers actions and drives the session.

Response

InfinityComputeResponse

A dict subclass stamped with a result_type and any payload fields the board understands. It is the only valid return type from a compute action - anything else is rejected as an invalid response.

Internally the constructor calls super().__init__(result_type=rt, **kwargs), so the object serializes cleanly over the WebSocket as JSON.

Parameter	Type	Description
rt	str	Result type string - one of raw_text, component, image, or interactive_surface.
**kwargs	any	Payload fields: raw_text, component, or information depending on the result type.

InfinityComputeResponse(rt, **kwargs) Construct with a result type and optional payload. Prefer the builder over calling this directly.

Builder

InfinityComputeResponseBuilder

Fluent helper for constructing sealed responses. Set a result type, then call the matching terminal method. Each terminal method validates that the result type matches before emitting an InfinityComputeResponse.

RAW_TEXT COMPONENT IMAGE INTERACTIVE_SURFACE

result_type(rt) Set the result type string. Returns self for chaining. Defaults to RAW_TEXT.

raw_text(text) Emit a raw-text response. Requires result type RAW_TEXT; raises ValueError otherwise. Payload field: raw_text.

text_component(title, contents) Emit a text board component. Requires COMPONENT. Builds {"type": "text", "title": …, "contents": …} under the component key.

image_component(image_bytes, width, height) Emit an image component. Accepts base64 string or raw bytes plus dimensions. Payload: {"type": "image", "image": …, "width": …, "height": …}.

interactive_surface_request(ss_info) Request an interactive surface from a WrappedScreenshot. Sends dimensions as presupposed_width / presupposed_height and sets send_screen_stream_req: True. Deletes the screenshot file after use.

Validation

Calling raw_text() when the result type is not RAW_TEXT, or text_component() when it is not COMPONENT, raises a ValueError. Always chain result_type(…) first.

Nested type · Builder

WrappedScreenshot

Nested under InfinityComputeResponseBuilder. Holds a filesystem path to a captured screenshot plus its pixel dimensions. This is the object returned by capture_screen() and accepted by surface interaction helpers.

The screenshot file is temporary. Interaction methods and interactive_surface_request() unlink it after use. If capture fails, error is set to True and width/height are zero - always check before interacting.

Attribute	Type	Description
ss_path	Path	Filesystem path to the PNG screenshot.
width	int	Image width in pixels (0 on failure).
height	int	Image height in pixels (0 on failure).
error	bool	True if capture failed. Do not pass an errored instance to interaction methods.

raw_image() Read the file, base64-encode it, delete the path, and return an IMAGE response via the builder. Raises if error is set.

WrappedScreenshot · construction

# Returned by capture_screen() - you rarely construct this yourself
ss = api.capture_screen("shot.png")

if ss.error:
    raise RuntimeError("screenshot failed")

# Dimensions for coordinate scaling
print(ss.width, ss.height)   # e.g. 1920 1080

# Turn the capture into a board IMAGE component
image_response = ss.raw_image()

API

InfinityBoardComputeAPI

The central object for registering compute actions, capturing the desktop, driving interactive surfaces, and running the WebSocket session. Instantiate once, decorate handlers, then call asyncio.run(api.start()).

InfinityComponents.ALL TEXT CONTAINER

Registration & session

__init__(mls=10*1024*1024) Create the API. mls is the max WebSocket message size in bytes (default 10 MB).

@compute_action(name, supported_components) Decorator that registers a handler under name. supported_components is a list of component scopes (e.g. [InfinityComponents.ALL]). The wrapped function must return InfinityComputeResponse; otherwise the SDK returns a failure payload.

asyncio.run(api.start()) Connect to the central unit, handshake as a handler, register commands, receive a compute ID, then enter the request loop. Blocks for the session lifetime.

Screen capture

capture_screen(path?) Take a desktop screenshot. On macOS uses screencapture -x; on Windows/Linux uses pyautogui. Returns a WrappedScreenshot with path, width, height (or error=True on failure). Default path: screenshot.png.

remove_capture(ss_path) Delete a screenshot file at the given path. Convenience wrapper around Path.unlink(missing_ok=True).

Interactive surfaces

create_interactive_surface_req() Capture the screen and return an INTERACTIVE_SURFACE response with send_screen_stream_req: True and the screenshot dimensions. Use this from a compute action to open a live surface on the board.

perform_surface_interaction(on, x, y) Click at board-relative coordinates (x, y), scaled against the dimensions of a prior WrappedScreenshot (on). Deletes the old screenshot, then returns a new WrappedScreenshot of the post-click desktop.

perform_surface_keystrokes(cc, keys, enter?) Type keys into the desktop (supports <Backspace> segments). If enter=True, presses Enter after typing. Takes a prior WrappedScreenshot as cc, deletes it, and returns a new capture.

Coordinate scaling

perform_surface_interaction scales board coordinates against the screenshot’s width/height versus the real screen size reported by pyautogui. Always pass the WrappedScreenshot that corresponds to the frame the user clicked on - using a stale or mismatched capture will click the wrong place.

05 // Interactive Surfaces

A living window on the desktop

Interactive surfaces place a live view of the machine running your handler onto the board.

Clicks and keystrokes from the canvas are forwarded as surface_interaction and surface_interaction_keystrokes requests. The SDK scales coordinates, applies them via pyautogui, and can return a refreshed surface response so the board stays in sync.

From a developer’s compute action you typically only need create_interactive_surface_req() - the stream lifecycle (offer, answer, stop) is handled by the InfinityBoardComputeAPI class.

Surface lifecycle (handled by the SDK)

surface_start_stream

Central unit requests a stream. SDK creates an RTCPeerConnection, attaches a DesktopVideoTrack, builds an SDP offer, and sends it.

webrtc_answer

Central unit returns an SDP answer. SDK sets the remote description and the stream goes live.

surface_interaction / keystrokes

Board clicks and typed input arrive as requests. SDK clicks or types on the real desktop and may return a new surface payload.

surface_stop_stream

Central unit asks to tear down. SDK stops the track, closes the peer connection, and cleans up.

06 // Capture & interact

Screenshots drive interaction

Every click and keystroke is anchored to a prior screenshot. You capture, pass the WrappedScreenshot into an interaction method, and receive a fresh capture of the post-action desktop.

capture_screen

Takes an optional path (default screenshot.png). Creates parent directories if needed. On Darwin runs screencapture -x; on Windows and Linux uses pyautogui. Opens the result with Pillow to read dimensions.

capture_screen

ss = api.capture_screen()          # → WrappedScreenshot
ss = api.capture_screen("tmp/a.png")

if ss.error:
    # width/height are 0; do not interact
    ...

# Use dimensions or pass to interaction helpers
w, h = ss.width, ss.height

perform_surface_interaction

Signature: perform_surface_interaction(on, x, y). on is the WrappedScreenshot whose frame the coordinates refer to. The method scales (x, y) from screenshot space into real screen space, deletes the old file, and returns a new WrappedScreenshot.

click flow

# 1. Capture the frame the user is looking at
frame = api.capture_screen()

if frame.error:
    raise RuntimeError("capture failed")

# 2. Click at board coordinates (e.g. from a canvas event)
#    `on` = the screenshot those coords are relative to
after = api.perform_surface_interaction(
    on=frame,
    x=420,
    y=310,
)

# 3. `after` is a new WrappedScreenshot of the post-click desktop
#    Use it for the next interaction or turn it into an IMAGE:
response = after.raw_image()

perform_surface_keystrokes

Signature: perform_surface_keystrokes(cc, keystrokes, enter=False). cc is the prior capture (consumed and deleted). Keystrokes may include <Backspace> markers for segmented typing. If enter=True, Enter is pressed after the text.

keystroke flow

frame = api.capture_screen()

# Type into the focused window, then press Enter
after = api.perform_surface_keystrokes(
    cc=frame,
    keystrokes="hello world",
    enter=True,
)

# Segmented typing with backspaces
frame2 = api.capture_screen()
after2 = api.perform_surface_keystrokes(
    cc=frame2,
    keystrokes="hel<Backspace>lo",
)

Ownership of the screenshot

Once you pass a WrappedScreenshot into perform_surface_interaction, perform_surface_keystrokes, or interactive_surface_request, the file is deleted. Do not reuse the same instance. Always capture again (or use the returned capture) for the next step.

07 // Examples

Register actions and run

Define compute actions on an InfinityBoardComputeAPI instance, return builder responses, then start the session.

Basic handlers

handler.py

from orrinsdk.sdk.InfinitySDK import (
    InfinityBoardComputeAPI,
    InfinityComputeResponse,
    InfinityComputeResponseBuilder,
)
import os
import subprocess
import asyncio

IBC_API = InfinityBoardComputeAPI()

@IBC_API.compute_action(
    name='Compile C',
    supported_components=[InfinityBoardComputeAPI.InfinityComponents.ALL],
)
def run_gcc(code: str) -> InfinityComputeResponse:
    if code == '':
        return InfinityComputeResponseBuilder().result_type(
            InfinityComputeResponseBuilder.ResultTypes.RAW_TEXT
        ).raw_text('No code provided')

    with open('main.c', 'w') as file:
        file.write(code)

    os.system('gcc main.c -o main.o')
    result = subprocess.run(
        './main.o',
        shell=True,
        capture_output=True,
        text=True,
    )
    os.system('rm -rf main.c main.o')

    return InfinityComputeResponseBuilder().result_type(
        InfinityComputeResponseBuilder.ResultTypes.COMPONENT
    ).text_component(
        title="Output",
        contents=result.stdout if result.stdout else result.stderr,
    )

@IBC_API.compute_action(
    name='Create Interactive Surface',
    supported_components=[InfinityBoardComputeAPI.InfinityComponents.ALL],
)
def cis(d):
    return IBC_API.create_interactive_surface_req()

asyncio.run(IBC_API.start())

Capture → click → type → image

A full interaction loop inside a compute action: screenshot, click a button, type into a field, then return the final desktop as an image component.

interaction_loop.py

from orrinsdk.sdk.InfinitySDK import (
    InfinityBoardComputeAPI,
    InfinityComputeResponse,
    InfinityComputeResponseBuilder,
)
import asyncio

api = InfinityBoardComputeAPI()

@api.compute_action(
    name='Fill form',
    supported_components=[api.InfinityComponents.ALL],
)
def fill_form(payload: dict) -> InfinityComputeResponse:
    # payload may contain board-relative click targets and text
    click_x = payload.get("x", 200)
    click_y = payload.get("y", 400)
    text = payload.get("text", "")

    # 1. Capture current desktop
    frame = api.capture_screen()
    if frame.error:
        return InfinityComputeResponseBuilder().result_type(
            InfinityComputeResponseBuilder.ResultTypes.RAW_TEXT
        ).raw_text("Screen capture failed")

    # 2. Click the input field (coords relative to this frame)
    frame = api.perform_surface_interaction(
        on=frame, x=click_x, y=click_y
    )
    if frame.error:
        return InfinityComputeResponseBuilder().result_type(
            InfinityComputeResponseBuilder.ResultTypes.RAW_TEXT
        ).raw_text("Post-click capture failed")

    # 3. Type into the focused field and press Enter
    frame = api.perform_surface_keystrokes(
        cc=frame, keystrokes=text, enter=True
    )
    if frame.error:
        return InfinityComputeResponseBuilder().result_type(
            InfinityComputeResponseBuilder.ResultTypes.RAW_TEXT
        ).raw_text("Post-keystroke capture failed")

    # 4. Return the final desktop as an IMAGE component
    return frame.raw_image()

asyncio.run(api.start())

Open a live surface from a handler

The shortest path to putting a live desktop on the board - one line inside the action.

open_surface.py

@api.compute_action(
    name='Show Desktop',
    supported_components=[api.InfinityComponents.ALL],
)
def show_desktop(_):
    # Captures screen, packages dimensions, requests stream
    return api.create_interactive_surface_req()

08 // Abilities

What you can wire up

// Named actions

Decorate functions, declare supported components, return structured responses the board can place.

// Typed results

Raw text, text components, images, and interactive surfaces - all as InfinityComputeResponse.

// Surfaces

Open a live interactive surface from a compute action and drive clicks and keystrokes from the board.

// Desktop capture

Screenshot helpers return WrappedScreenshot with dimensions so interaction stays aligned with the screen.

// Fluent builders

Chain result_type with raw_text, text_component, image_component, or interactive_surface_request.

// One entrypoint

Start

Extend Infinity to your imagination.

Install the SDK, register a compute action, and open a live connection to the board - your process becomes another surface on the infinite canvas.

pip install orrin-sdk

An SDK for the infinite canvas. Your code, on the board.

The canvas comes alive from your process.

Install with pip

How a compute session works

Instantiate the API

Register compute actions

Start the session

Serve requests

Board placement

Three types define the contract.

InfinityComputeResponse

InfinityComputeResponseBuilder

WrappedScreenshot

InfinityBoardComputeAPI

Registration & session

Screen capture

Interactive surfaces

A living window on the desktop

Surface lifecycle (handled by the SDK)

surface_start_stream

webrtc_answer

surface_interaction / keystrokes

surface_stop_stream

Screenshots drive interaction

capture_screen

perform_surface_interaction

perform_surface_keystrokes

Register actions and run

Basic handlers

Capture → click → type → image

Open a live surface from a handler

What you can wire up

// Named actions

// Typed results

// Surfaces

// Desktop capture

// Fluent builders

// One entrypoint

Extend Infinity to your imagination.

An SDK for the
infinite canvas.
Your code, on the board.

The canvas comes alive
from your process.