Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Python agentagent-osaiai-agentsai-partnerchatopsclaudeclaude-codecodexdevtoolsdiscord-botlark-botllmlocal-firstlong-term-memoryopencodeslack-botvibe-codingvibeywechat
docs(computer-use): design JavaScript SDK (#2622)
Max54a708b704ed
diff --git a/docs/plans/desktop-computer-use-js-sdk-design.md b/docs/plans/desktop-computer-use-js-sdk-design.mdnew file mode 100644index 000000000..35a6af77d--- /dev/null+++ b/docs/plans/desktop-computer-use-js-sdk-design.md@@ -0,0 +1,180 @@+# Avibe Computer Use JavaScript SDK++**Status:** Owner-facing design record+**Date:** 2026-10-14+**Scope:** macOS desktop Computer Use, v1++## Decision++Agents will drive the desktop by writing ordinary JavaScript scripts that+import the Avibe SDK. They will not import or call cua's SDK. A proven flow+becomes a reusable Skill: `SKILL.md`, one script, declared parameters, and+declared side effects, stored in the owner's personal Skill directory and+available across projects. An agent may propose saving a flow; the owner+approves before it is saved.++The SDK is the single user-facing abstraction over desktop actions. The+existing `vibe computer` CLI is re-based on the same implementation: a single+step remains convenient on the CLI, while scripts provide sequencing,+branching, waits, recovery boundaries, and a JSON return value.++## Why Avibe owns the SDK++Cua's SDK would bypass Controller admission and its allowlist, exact-window+checks, observe-first rule, desktop lease, and foreground consent. Its+in-process driver would also attribute macOS TCC to the wrong identity, and+its N-API runtime is MPL-2.0. The Avibe SDK therefore sends every action+through the same Controller path as `vibe computer`; there is one safety rule+set and one owner-visible policy.++The current CLI already establishes this boundary. It reads only the injected+`AVIBE_SESSION_ID`, calls the Controller's `/internal/computer-use` endpoint,+and never accepts a client-provided Turn identifier. The Controller checks that+the Session has an active Turn before admitting the request, then applies the+Session-keyed driver state and desktop lease. On POSIX the transport is HTTP+over an owner-only Unix `dispatch.sock`; on Windows the existing control IPC+uses authenticated loopback TCP with a bearer token and instance binding. The+Node SDK must reuse that discovery, authentication, and Session binding path,+not invent a second endpoint. See [`vibe/cli.py`](../../vibe/cli.py),+[`vibe/internal_client.py`](../../vibe/internal_client.py), and+[`core/internal_server.py`](../../core/internal_server.py).++## Object model and selectors++The object model is:++```text+computer -> app -> window -> element+```++An element is located declaratively with `role`, `label`, `labelContains`,+`value`, `within`, and `index`. The SDK hides driver element tokens. It+re-observes a window when a token is stale, while the Controller still owns+observe-first admission. The first use of a window observes it automatically.+Zero matches and multiple matches are errors; the SDK never guesses.++Every action returns the existing effect vocabulary, including `confirmed` and+`unverifiable`. Scripts use `waitFor(selector, timeout)` for eventual+availability and `expect(selector).toHaveValue(...)` and related assertions+for post-action state. These are robustness primitives, not permission+overrides: a timeout or failed expectation stops the script.++Representative script shape:++```js+import { computer, avibe } from "@avibe/computer";++export default async ({ to, subject, body }) => {+ const mail = await computer.app({ bundleId: "com.apple.mail" });+ const window = await mail.window({ role: "AXWindow", index: 0 });+ await window.element({ role: "button", label: "Compose" }).click();+ const draft = await mail.window({+ role: "AXWindow",+ labelContains: "New Message",+ });+ await draft.element({ role: "textbox", label: "To" }).fill(to);+ await draft.element({ role: "textbox", label: "Subject" }).fill(subject);+ await draft.element({ role: "textbox", label: "Message" }).fill(body);+ const verifyDraft = async () => {+ await draft.expect({ role: "textbox", label: "To" }).toHaveValue(to);+ await draft.expect({ role: "textbox", label: "Subject" }).toHaveValue(subject);+ await draft.expect({ role: "textbox", label: "Message" }).toHaveValue(body);+ };+ await verifyDraft();+ await avibe.confirm(+ `Send email with To=${to}, Subject=${subject}, Message=${body}?`,+ );+ await verifyDraft();+ const effect = await draft.element({ role: "button", label: "Send" }).click();+ await avibe.notify(`Mail send effect: ${effect.status}`);+ return { to, subject, effect };+};+```++The exact export convention is implementation-owned; the important contract is+that the program is a normal Node module and its returned value becomes the+JSON result.++## Errors and failure recovery++The SDK exposes typed errors: `NotFound`, `Ambiguous`, `Timeout`, `DesktopBusy`,+`ForegroundConsentRequired`, `Unavailable`, and the Controller's stable error+codes. A script failure stops execution and returns the failing step plus a+screenshot path when one is available. This lets an agent fall back to+step-by-step work without claiming that later steps ran.++An action with an ambiguous transport outcome is not blindly retried. The+script or agent observes the resulting state first, matching the existing+Computer Use guidance in [`skills/computer-use/SKILL.md`](../../skills/computer-use/SKILL.md).+GUI text, accessibility labels, clipboard contents, and application output+remain untrusted data.++## Avibe primitives++`avibe.notify(message)` reuses existing message delivery. `avibe.confirm(message)`+asks the owner on the current Web or IM surface and waits for an answer.+Irreversible operations such as send, submit, and pay call `confirm` by+default. Irreversible steps follow verify → confirm → re-verify → act: the script+verifies the exact form state it will show, asks with that state in the message,+re-verifies the same state immediately after confirm returns, and acts only if it+still matches; otherwise it stops without acting. A Skill may declare a flow+confirmation-free only with explicit owner approval. Unattended runs may execute only scripts that require neither+confirmation nor foreground control; all others are refused.++Avibe already has `QuestionUIHandler`, which renders questions through an IM+client and waits on a per-Session event until a callback or reply supplies the+answer. It is not currently a generic script-facing, Web/IM-neutral primitive.+The implementation should reuse that existing delivery and wait lifecycle,+extracting only the smallest Controller-owned `ask-and-wait` boundary needed+by `confirm`. This is a continuation primitive, not a new approval system.+Whether the current Web surface can satisfy the same contract is **unverified**+on `origin/master`; the implementation PR must verify it before claiming Web+parity. See [`modules/agents/question_ui.py`](../../modules/agents/question_ui.py).++## Execution and runtime++The command is:++```text+vibe computer run <file.mjs> --arg k=v+```++`@file` values are supported. The command runs on the Node runtime Avibe+already ships, is bound to the current Avibe Session, keeps one persistent+Controller connection per script, and enforces one total script timeout.+Scripts have no privilege beyond the agent's existing shell; only desktop+actions pass through Controller rules.++Desktop packages currently carry private Node `v22.23.3` for macOS and+Windows, with the launcher injecting its absolute path as+`VIBE_SHOW_RUNTIME_NODE_BIN`; the packaged tree also carries npm. The desktop+package engine accepts `^20.19.0 || >=22.12.0`. See+[`desktop/runtime-sources.json`](../../desktop/runtime-sources.json),+[`vibe/desktop_runtime.py`](../../vibe/desktop_runtime.py), and+[`desktop/package.json`](../../desktop/package.json).++The repository does not currently contain `@avibe/computer`, an SDK module+resolver, or `vibe computer run`; those are implementation work. On a+non-desktop install, Node is currently an optional system dependency for the+Show Pages runtime and is not a private Avibe runtime. The exact non-desktop+packaging and the injected module-path mechanism for resolving+`@avibe/computer` are therefore **unverified**. The intended boundary is an+Avibe-injected absolute SDK module path selected by the runner; the exact+environment name and Node loader mechanism belong to the implementation PR.+The SDK must not resolve from the agent's working directory or global npm.+v1 must fail as `Unavailable` when the Avibe SDK runtime is absent. Any new npm+dependency must be recorded in the repository's dependency files in the+implementation change.++## Reuse and boundaries++Skills should prefer app scripting interfaces and service APIs when they exist;+GUI scripting is the fallback because fixed selectors break when UIs change.+The CLI and SDK must share one rule implementation, including effects, typed+errors, observation, lease, foreground consent, and scheduled-run admission.++This record does not design Route B internals, browser access, Windows+Computer Use, or a persistent JS REPL (`js`/`js_reset`). Those remain separate+implementation or design work. The implementation PR owns exact module loading,+script export mechanics, argument parsing details, screenshot artifact+placement, and the Web continuation adapter, subject to the decisions above.