Skip to content

A desktop coding agent with free access to frontier models.

TypeScript
16.4 KB
Raw Blame

Computer Use

Computer Use operates one explicitly approved macOS app window through a short-lived session. Its native runtime needs no HandsFree service, voice loop, private background-activation API, model SDK, API key, HTTP listener, or legacy whole-desktop tool.

Start

In OmniRush.ai, open Library → Computer Use and choose Enable Computer Use for the current workspace. If macOS access is missing, open permission setup from that page and grant Accessibility and Screen Recording to the app macOS identifies. Return to OmniRush.ai and wait for Ready. macOS permissions alone do not enable the workspace connection. Mention an app in a message. OmniRush.ai can launch an installed app, then asks you to choose its window and Allow and start in the main app. Ordinary work shows no control dashboard. Your input interrupts the agent; it must wait and obtain a fresh observation before continuing. Stop in the native preview ends access. Hide leaves the grant intact; OmniRush.ai can show the preview again. The small floating preview shows the last approved observation and the last action location, not a continuous desktop recording. The setup window can stop all sessions.

Mode Access Foreground behavior
observe Window text and screenshots No input
assist Observation plus advertised accessibility press/value operations Does not activate an app or move the pointer
control Observation, accessibility, and targeted mouse/keyboard input Requires the selected window in front; clicks, typing, scrolling, dragging or switching apps pause control

The app and mode are fixed for the session. Changing either requires a new approval. A session lasts at most 15 minutes, pauses after two minutes without an operation, and permits at most 200 action attempts. System or window failures can still require an explicit Continue prompt in OmniRush.ai. In control mode, it brings the approved window forward and requires a fresh observation before any further input. The main app shows approval and exceptional recovery prompts. There is no resume, scope-upgrade, clipboard, URL-open, shell, whole-screen, or permission-grant tool.

The default is a new app approval for every session. There are no persistent "always allow" grants in this version. This intentionally keeps the first migration's authority understandable and revocable.

Runtime and protocol

Contract v2 (desktop): ComputerUse worker is the native layer of the shared computer-use host (apps/server/src/computer-use/shared/portable-host.mjs), started by apps/desktop/electron/computer-use-host.mjs. The shared host owns the tool schemas, validation, results, approvals, events and media spool; the worker lists windows, captures one window (password fields and recognized credential text are reported as boxes to paint), hit-tests target points, posts window-targeted input and reports person input. Its stdio protocol is described in native/Sources/ComputerUse/NativeWorker.swift. The mcp / mcp-hosted / relay modes below are the earlier v1 surface.

OmniRush.ai engine → stdio relay → desktop broker → native MCPServer
OmniRush.ai main window → desktop IPC → broker's private UI channel
                                      └─ native session authority
                                          ├─ MacAccessibility: AX + window-only capture
                                          ├─ MacInput: exact-process input and focus checks
                                          └─ SessionControls: approval + small preview
Standalone MCP client → ComputerUse mcp → native approval and controls

OmniRush.ai launches a filtered stdio-to-Unix-socket relay for MCP and owns the native helper process. Only the main window's desktop IPC can approve or recover a blocked session; private UI messages are removed from the engine transport. The socket is inside a user-only directory. Existing enabled bundled commands migrate when a local workspace loads. Custom commands and disabled entries are preserved. Standalone ComputerUse mcp retains native approval and controls.

The executable owns authority. The agent's MCP client cannot loosen its mode, supply another PID for an action, or act after an interruption without waiting and refreshing state. A kernel-held lock permits one session at a time across helper processes and releases on process exit. Each stdio connection owns its own unpredictable session ID. Grants, element references, screenshots, and action receipts stay in process memory and are discarded on close. The native session lock is an empty file under Application Support; desktop coordination also uses a private temporary Unix socket.

App identity includes bundle identifier, process ID, executable URL and launch time. Window identity includes a ScreenCaptureKit window ID and the exact AX window object; ambiguous matches are rejected. Accessibility reads have depth, node, record, text, and elapsed-time budgets. Standard protected fields are omitted from the semantic tree and masked in the returned PNG.

All action arguments are validated at the native boundary. Coordinates refer to the actual returned PNG dimensions and are mapped to the selected window's current bounds, including negative display origins. A visual action also requires a foreground-window check, a fresh matching image, and target hit testing. Input is posted to the pinned process; no global HID fallback exists.

An observation is valid for 15 seconds and one action attempt. Before input, the runtime verifies the app, window, geometry, semantic content and exact element identity. It consumes the observation before dispatch and caches the receipt under request_id, including failures after dispatch begins. An identical retry returns the same receipt. Reusing the ID for different input is an error. The client must observe after every attempt.

status: "dispatched" means an OS operation was accepted or posted; it does not verify the intended outcome. Outcome verification belongs in the next observation and the caller's task-specific assertion. Input failures can have partial effects, so their receipts explicitly require observation before a retry. Cancellation, sleep, takeover and Stop invalidate observations.

Tool Purpose
computer_discover App identities, permission status, modes, keys and limits; no window text
computer_open_session Launch if needed, then person approval of app/window/mode
computer_observe Semantic state plus optional window PNG
computer_act One validated action with session, observation and request IDs
computer_session_status State, scope, action count and expiry
computer_close_session Revoke the grant and release control

Tool failures use MCP isError: true with machine-readable code and next fields. The server negotiates supported MCP protocol versions. The stdio reader stays off the AppKit thread, and approval uses an asynchronous native sheet so cancellation and the UI remain responsive.

Model and orchestration strategy

Use dedicated integrations for structured data and the built-in browser for websites. For native apps, use assist when the accessibility tree exposes the needed controls. Choose control only when visual interaction is needed. The runtime has no provider lock or second hidden model loop; OmniRush.ai's selected tool-capable model invokes the same MCP contract. A model must accept images to make visual decisions. Text-only models can request include_image: false and work with accessible controls.

For persistent JavaScript orchestration, src/session.mjs exports openComputerSession({callTool, appId, purpose, mode, signal}). Its handle offers observe(), act(), status() and close() / Symbol.asyncDispose. act() consumes the current observation, dispatches exactly once, and returns a new observation plus the dispatch receipt. It never retries an uncertain action. If post-action capture fails, the error preserves the receipt and asks for another observation. Use a dedicated MCP connection per task and dispose the handle when the task stops.

Keep model reasoning, planning and provider credentials in the caller. Bound the caller's turn count, duration and cost independently. Preserve text and image tool results and their request IDs across continuation. Do not infer task success from a tool receipt, a model's prose, or a screenshot alone. For an API-native computer-tool adapter, route every action through this same session boundary and stop for pending safety checks; never automatically acknowledge them. No such adapter is enabled by this package.

Pointer event compatibility

The pointer backend constructs AppKit events with the approved window number, then posts them only to the pinned process. On recent macOS, these events also need a window-local coordinate in addition to the public screen coordinate. MacInput isolates an availability-checked lookup of the private CGEventSetWindowLocation symbol for this purpose. This is an OS compatibility dependency, not a public-API stability guarantee. If unavailable, pointer input fails explicitly before a press; accessible app controls remain usable. Foreground, hit-test, consent and per-event session checks still apply. There is no background-activation or global-input fallback.

Boundaries and known limits

  • macOS 14+ is the only native backend. Windows needs a separately tested UI Automation/input backend and active-desktop/UIPI handling; Linux needs its own accessibility and compositor/portal permission design. They are not emulated through shell scripts or silently routed to the old helper.
  • This is an app/window dispatch boundary, not an operating-system sandbox. A permitted app can itself save files, send messages, navigate to a site, or perform other consequential work. The calling host remains responsible for action authorization and prompt-injection handling. Treat screen/AX content as untrusted data. Use an isolated desktop for adversarial workloads.
  • Known terminal, permission, credential and host-control apps are excluded. This blocklist is defense in depth, not a claim that every executable or command-capable app can be identified. Custom widgets can mislabel or omit protected fields; never assume screenshots cannot contain sensitive data.
  • Exact visual freshness deliberately rejects animation, a changed caret or any other changed pixels. Prefer accessible actions; observe again when a visual target changes. There is no speculative click or stale-image retry.
  • Screenshot and AX calls are not an atomic OS transaction. Identity, bounds, content and per-event checks reduce races but cannot make a third-party application's behavior atomic. The native input path does not claim background mouse/keyboard operation, secure-desktop automation or locked use.
  • Focusable text fields, public AX actions and a limited key vocabulary are supported. Context menus, arbitrary shortcuts, file dialogs outside the approved window, and inaccessible windows may require person takeover.
  • Screenshots and typed text are not logged by the runtime, but returned tool content enters the calling host's conversation and its configured retention and provider policies. Runtime in-memory retention is not a promise about the host's transcripts.

Handoff behavior

In the hosted desktop flow, person input releases held pointer input and invalidates the current observation. Passive mouse movement does not interrupt. The agent receives user_interacting during a one-second quiet period, then requery_required for actions until it requests a new observation. Time passing alone never resumes input. A new observation after quiet brings only the approved window forward in control mode, checks focus, and refreshes state. Further person input interrupts that recovery too. Interrupted typing or dragging is never replayed. The one-second duration and event selection are OmniRush.ai choices, not verified constants from another product.

Sleep, desktop unavailability, idle expiry, and failed focus recovery still need an explicit Continue prompt in OmniRush.ai. Normal work and recoverable person input do not show a permanent controller. Hide hides only the native preview; Show preview in OmniRush.ai restores it. Stop revokes the grant, and neither observation nor further actions can restart that session. The access countdown continues while interrupted.

Standalone ComputerUse mcp preserves its native approval and text controls: Take over, Continue, Stop, Hide panel, and the OW menu-bar restore action. Unlike hosted recovery, standalone person input still requires manual Continue after quiet, then a fresh observation.

Session status includes the purpose, approved window, preview visibility, and phase: person_interacting, requery_required, ready_to_continue, refreshing, or working. Desktop approval/preview coordination is global, not bound to an OmniRush.ai task ID. The preview displays the latest requested image, not a continuous capture stream. There is no combined start-and-get-state tool. Stop is enforced for the existing grant, not a native assistant-turn ID: caller instructions end the current turn, but another open-session call can request new approval. These are remaining gaps, not claims of full Codex/Sky parity.

When the window changes during observation, the runtime makes at most three read-only capture attempts, 75 ms apart. Every attempt checks the same session generation, window identity, geometry, and semantic state. Continuous changes still return stale_observation; a failed refresh invalidates the old token. This does not relax the exact-image checks for visual actions, retry any input, or recover during active person input.

Build, migration and verification

pnpm --filter @omnirush/computer-use build:native
swift test --package-path packages/computer-use/native
pnpm evals:e2e computer-use-window-scope

The desktop build stages this package as OmniRush.ai Computer Use.app while preserving the existing bundle identifier and executable location for TCC and installed MCP configurations. The helper's --check includes protocolVersion: "omnirush.computer-use/1"; setup identifies an old helper and asks for a rebuild/reinstall. Reconnect Computer Use in each existing workspace to replace an old package command and refresh tool schemas. There is no runtime fallback to @omnirush/handsfree; its historical sources remain separate and are not part of this implementation.

The native journey owns two disposable windows and drives the real helper over stdio. It verifies native consent, isolated connection grants, accessible actions, replay protection, read scope, protected-field omission, stale geometry, person takeover during typing, pointer release during interrupted drags, Continue and Stop. Its person-input fixture uses the real approval UI, never a production auto-approve flag. It requires macOS plus existing person- granted Accessibility and Screen Recording; it never changes TCC permissions. If the selected placement is Linux/Daytona, it skips explicitly and the verdict is Incomplete. A skip is not native proof. Platform coverage, visual-input journeys, provider task-success benchmarking and packaged permission-upgrade verification remain separate verification work.

Design sources (reviewed September 4, 2026)