Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Desktop computer use via embedded Cua Driver
Status
- Owner: Avibe core
- Date: 2026-10-01 (Phase 0 findings through 2026-10-03)
- Stage: Slice 1 implementation in PR #2442; Route A is already on master
- Upstream: trycua/cua
libs/cua-driver, releasecua-driver-rs-v0.31.0(2026-09-30), MIT - Companion docs:
desktop-tray-lifecycle.md(native toggle pattern),desktop-auto-update.md(signing),desktop-product-gaps.md
Amendment (2026-10-29, #3050)
Foreground input and window activation are ordinary Computer Use operations.
The Controller forwards delivery_mode: "foreground" and focus-changing
chords, and the managed surface adds the pinned driver's bring_to_front
(vibe computer activate), for 29 tools. None needs per-action consent.
Readiness is observable window state rather than a fixed wait, and IM
require_bind is not a Computer Use control. The focus-shortcut rejection
and the activation exclusion recorded below are superseded; see
2026-10-29-computer-use-foreground-and-readiness.md.
Current Slice 1 contract (2026-10-11)
Route A is shipped on master at merge commit
61e315abef588657be7cca19940a289c588ba2c5; the exact reviewed Route A head
was d8be44c74bf45b18e55ec35ffcd98edb97ef272f. Its embedded Cua Driver,
raw click_mode, native toggle and lifecycle, macOS permission flow,
Workbench status line, lease, observe-first, exact-window, and same-user
safety boundaries remain the foundation for this slice.
Slice 1 delivers the user-facing path that replaced the earlier backend MCP proposal:
vibe computercovers the allowlisted observe, zoom, input, clipboard, launch, and quit operations. Each request enters Controller IPC with only the caller's injected Avibe Session. The Controller checks its own live active-Turn registry before admission; no Turn identifier is accepted as a client-presented capability. The existingComputerUseServerchecks still run with the Session-keyed driver state and per-call desktop lease. Driver state is reused across Turns and retired when the Session is explicitly ended or archived.- A built-in managed
computer-useSkill is loaded on demand. It contains the CLI contract, untrusted-screen guidance, observe-before-input and observe-before-retry rules, and the required unavailable-state handling. - Claude, Codex, and OpenCode receive no
avibe_computerMCP configuration or always-on Computer Use prompt. OpenCode removes stale released configuration under that reserved name while preserving unrelated user MCP entries. - The Workbench status line describes Controller-owned CLI availability and is valid for Vibey and every backend, including backends with no MCP client.
The shipped surface has no compatibility layer: Computer Use was shipped only
in 3.1.4rc4, so released backend MCP injection is removed rather than
maintained. Route B, live GUI/TCC verification, and self-drawn-app,
QQ, and WeChat acceptance are explicit non-goals for this slice. Those checks
belong to a later owner-gated Slice 2 from the mergeable Slice 1 head.
F6 is not part of this slice. The macOS shell already unlinks the driver socket during endpoint reclamation and daemon stop; the embedded driver source is an external pinned asset rather than repository-owned code. Adding driver-side parent-liveness cleanup would therefore expand ownership and lifecycle scope without closing a demonstrated shell-owned gap.
The earlier MCP injection sections below remain as implementation history for Route A and its safety review. They do not describe the current Slice 1 backend contract. There is no compatibility layer for released backend MCP configuration: the CLI plus on-demand Skill is the only shipped backend path.
Slice 1 validation and gate
The implementation must be validated through the real CLI → local IPC →
Controller → ComputerUseServer path, with focused coverage for:
- caller environment projection, server-side active-Turn admission, Session-keyed driver reuse, explicit-end/archive retirement, and orphan-lease release;
- the no-ID
desktop_busyresponse, screenshot output handling, observation rendering, and post-connect transport failures that may have completed a mutation; - on-demand Skill loading, Claude hook identity, removal of
avibe_computerconfiguration and prompt injection for Claude, Codex, and OpenCode, and the Workbench status line for Vibey and non-MCP backends.
Route B, live GUI/TCC operation, and self-drawn-app, QQ, or WeChat verification are explicit non-goals. Those are the owner-gated Slice 2 manual gate and must start from a mergeable Slice 1 head; this plan does not authorize launching apps, touching TCC, using credentials, restarting Runtime, or performing live acceptance.
2026-10-11 P2 follow-up contract
The invariant is that the shipped Session-keyed Computer Use path stays truthful at every existing boundary: retiring an Avibe Session releases only a lease still owned by that Session without starting or reviving a driver session; CLI and Workbench status copy describe the structured state that can actually be used; screenshot errors remain errors before any output write; and every packaged Skill example is accepted by the CLI and the pinned driver schema.
The changed boundaries are limited to:
ComputerUseServerretirement and the existing desktop lease manager;- localized CLI rendering of
computer_use_unavailablestatus and reason; - the Workbench status API and component copy when Controller IPC is unhealthy;
- the packaged
computer-useSkill examples and their parser/schema contract.
Owning tests cover an inactive-proxy lease through the real retirement path, preservation of a replacement holder, localized human and unchanged JSON error output, API-to-component Controller-unavailable rendering, structured errors before both screenshot output paths, and every command and flag shown by the packaged Skill against the pinned allowed driver schema.
Preserved behavior includes same-user local trust, Session-keyed state and archive admission fencing, exact transport ownership, lease TTL and heartbeat, ambiguous forwarded-action handling, observe-first and exact-window checks, focus and foreground restrictions, native toggle/lifecycle ownership, all backend CLI parity, no managed backend MCP, no always-on prompt, and the already-corrected no-Turn-capability documentation. This follow-up adds no authority mechanism, per-Turn teardown, credential, Route B behavior, GUI/TCC operation, service restart, installation, release, or publication.
Route A: raw single-click selector (2026-10-10)
Status: implementation lane, based on remote master
0438a99916e82c9fe0387321dd65fd87b4412587, branch
fix/desktop-cua-raw-single-click-20261010.
The bounded research probe showed that the existing routed window-pointer
CGEvent path can deliver a hit-tested click to a test-owned AppKit target in
the background without changing the frontmost PID or hardware cursor
coordinates.
The failure in self-drawn apps was the click tool's AX-first optimization:
AXPress returned success while the target did not receive mouseDown.
Upstream investigation found no stable, maintainer-accepted selector contract
that bypasses this hollow AX success while preserving the current default
behavior. The pinned source is cua-driver-rs-v0.31.0 at commit
5272e492d61b96caf08e3bf434d91126c1f3dccc, with source archive SHA-256
8134337f2beb1854918d8bbc5aea22d7ae00de9838ce6426f275d7ba6734d191.
Relevant upstream work and the upgrade plan are recorded in
desktop/cua-driver/PATCHES.md and
desktop/cua-driver/sources.json.
The smallest accepted contract is click_mode:
- missing or
"auto"keeps today's AX-firstcount=1click behavior; "raw"is opt-in and requires an exactpid+window_id, pixelx/y, background left-click delivery,actionpressorclick,count=1, and no modifiers, element token, or portable target;- raw delivery defaults to background when
delivery_modeis omitted; when supplied, the field must be exactly"background"rather than null, foreground, or an unknown future value; - raw mode bypasses only the macOS click tool's AX hit-test/
AXPressshortcut and uses the existing routed CGEvent path; - unknown values and invalid combinations are structured
invalid_arguments.
Avibe admits and forwards click_mode only for the click tool, after the
existing observe-first and exact-window admission. A request observed before
the target window is admitted is never forwarded. The tool snapshot hash is
updated in the core and Runtime host, and the package workflow now builds the
patched source and emits schema-2 provenance containing source, patch, snapshot,
prepared-binary, packaged-binary, and cdhash evidence.
The package verifier keeps two generated schema surfaces distinct. The
driver's dump-docs --type mcp output uses input_schema, while the managed
MCP snapshot uses inputSchema. Packaging validates the former against the
built binary; core/computer_server.py consumes the latter. Owning tests reject
the snapshot shape at the dump-docs boundary so a guessed key cannot silently
replace the producer's actual contract again.
Driver provenance also keeps the unsigned and signed binary surfaces distinct.
Immediately before Tauri packaging, the workflow hashes the prepared sidecar
at desktop/src-tauri/binaries/cua-driver-<target> and requires it to match
prepared_binary_sha256. Release recording separately hashes the signed helper
inside the app as packaged_sha256; signing may make the two hashes differ.
Owning mutation tests reject both an arbitrary valid-looking digest and the
signed helper's digest when either is substituted for the prepared sidecar
digest.
The test-owned AppKit fixture makes AXPress a hollow success and counts
native mouseDown/mouseUp. Its ignored live harness is compiled in this
lane but is not run here because live GUI/TCC actions are outside the current
authorization. The owning contract tests cover snapshot/manifest locking,
source-subtree extraction, rejection of non-regular archive members, Avibe
admission/forwarding, and release provenance verification.
Route B, a bounded foreground mode for applications that reject both accessibility actions and background synthetic events, is explicitly deferred. It must be designed and accepted from new real evidence; it cannot become a silent fallback for Route A. QQ/WeChat verification likewise requires a later owner handoff after the patched artifact exists.
Background
Avibe agents can run shell commands and take a desktop screenshot
(vibe screenshot), but they cannot operate GUI apps. Codex's own
features.computer_use is deliberately disabled
(modules/agents/codex/transport.py), and Claude Code and OpenCode have no
equivalent tool. The result is that one GUI-only step blocks a whole task.
Cua Driver is a Rust runtime that observes and operates native apps on macOS, Windows, and Linux. It exposes an MCP stdio server: screenshots, window and accessibility state, click, type, hotkeys, scroll, drag, clipboard, and sessions. It delivers input in the background where the platform allows it, so it does not move the user's pointer and normally leaves focus alone (see Focus boundary). Its embedded mode exists for agent harnesses: the host app requests the macOS grants once, and the driver runs inside the host's TCC responsibility chain.
Other parts of cua (Fleets, Lume, Bench, CUA-S1) are out of scope.
Goal
When a user enables computer use in the Avibe desktop app, every Avibe Agent Turn can use the same Controller-owned Computer Use CLI, whichever backend runs it (Claude Code, Codex, OpenCode) and wherever the request came from (IM, Web, Harness, scheduled run). Users grant OS permissions to Avibe once.
Owner decision: IM-triggered computer use is allowed
Remote IM requests may drive the desktop. This adds no new trust tier: a bound
IM user's agent already runs with full shell access on this machine
(bypassPermissions / danger-full-access). Whoever may start an Agent Turn
under Avibe's general message-ingress rules may therefore use computer use,
with these Computer Use controls:
- Explicit enablement. Computer use is off until the user turns it on in the desktop app. While it is off, the driver does not run.
- A visible actor. The agent-cursor overlay stays on, so on-screen actions are visibly the agent's.
- A native stop. The same tray/app-menu toggle stops the driver at once. That ends every session's access, not only one session's.
- Untrusted screen content. The injected prompt tells agents to treat on-screen text as data, never as instructions.
A locked Mac cannot be driven (Phase 0 finding). Remote use therefore needs an unlocked login session. User docs must say so, and must also say that window screenshots still capture app contents while the screen is locked.
There is no per-action approval prompt. Unattended IM use is the point of the feature, and per-call prompts would not reach a user who is away from the machine.
Architecture
Avibe.app (Tauri shell; owns TCC grants, toggle, lifecycle; writes state file D)
└─ cua-driver serve --embedded --socket <S> ← executes tools, checks TCC
Avibe Runtime (adopted or shell-started; reads D, never spawns the daemon)
└─ Controller internal IPC
└─ vibe computer → ComputerUseServer
└─ cua-driver mcp --embedded --socket <S> ← per daemon generation
Only the shell process may spawn the daemon. The Runtime runs detached, may
be adopted from a terminal-started vibe, and keeps running after the window
closes. A daemon it spawned would take the TCC identity of whatever started the
Runtime, not Avibe.app. Cua's docs name this exact gateway wiring as wrong. The
MCP proxy never executes tools and needs no grant, so any process may spawn it.
Two things change at different rates, and the design keeps them apart:
- Configuration: whether sessions carry the
avibe_computerserver at all. It follows the user's toggle and changes only when the user flips it. - Availability: whether a daemon answers right now. It changes with daemon restarts, crashes, grants, shell restarts, and the lock screen.
Availability never changes backend configuration. A stable Avibe-owned server absorbs it at call time.
As a consequence, computer use is available only while the desktop shell is running. The tray keeps the shell alive after the window closes.
Shell (desktop/src-tauri, desktop/runtime-host)
Packaging. On macOS, ship the raw, plist-less driver at
Contents/Helpers/cua-driverthrough Tauri'sbundle.macOS.files; do not useexternalBin, and ship no stale copy inContents/MacOS. A raw helper inContents/MacOSresolves the outer Avibe bundle before the pinned driver's AppKit startup can set its accessory policy. In acceptance this registered every daemon under Avibe's regular-app identity, made the daemon frontmost, and fed a driver-exit → host-reactivation spawn loop.Contents/Helperskeeps the executable bundle-less while remaining a standard nested-code location. Launch it directly withCommand, never through LaunchServices. Pin the version and SHA-256 in a sources manifest besideruntime-sources.json. Fetch only the plain driver source, never thecua-perceptionextension, which is AGPL. Apply only the explicitly recorded patch variant, add the upstream MIT notice, and record the source archive and patch hashes. Desktop packages are per architecture (aarch64-apple-darwin,x86_64-apple-darwin,x86_64-pc-windows-msvc), so build one target-specific macOS helper per package. Record the prepared binary digest before packaging, separately from the packaged digest and cdhash after signing. Newly produced desktop source metadata uses schema 2: every macOS record must include this complete driver provenance block. Schema-1 metadata remains accepted only as a legacy published input. Sign nested code explicitly before the outer app;--deepis verification-only. Windows shipscua-driver.exe; whether the default-offcua-driver-uia.exeworker is needed is a Windows-run question. Spawn it withCREATE_NO_WINDOW(it is a console program), asruntime-hostalready does for the Runtime.- Asset path trust. Release builds resolve the driver, managed policy,
snapshot, and state directory from the signed bundle and app-data paths.
AVIBE_COMPUTER_USE_DRIVER_PATH,AVIBE_COMPUTER_USE_POLICY_PATH,AVIBE_COMPUTER_USE_SNAPSHOT_PATH, andAVIBE_COMPUTER_USE_STATE_DIRare ignored in production. Hermetic acceptance may enable the explicitcomputer-use-test-overridesbuild feature; the release package workflow does not enable it and does not inject these variables. This prevents a launch environment from replacing the binary, policy, or state contract inherited by the shell's TCC responsibility.
- Asset path trust. Release builds resolve the driver, managed policy,
snapshot, and state directory from the signed bundle and app-data paths.
Toggle. A native checkable "Computer Use" item in the tray and the app menu, following the Start at Login pattern. It adds no webview command. Its only durable record is
D.enabled. The shell keeps no second copy in its native preferences: it readsDat launch and writesDon every flip, so the next launch and the live Runtime can never disagree.Runtime compatibility. The shell adopts a running Runtime as-is, so after a desktop update that Runtime can be older than the shell.
GET /readystays byte-for-byte unchanged, because released shells parse it with an exact key set (runtime-host/src/health.rs). A newGET /api/desktop/capabilities(under/api/, where an unknown route is a real 404; elsewhere the UI's SPA fallback answers 200 withindex.html) returnscomputer_use_schema, the newestDschema_versionthe Runtime can read. The UI process obtains it from the Controller over the existing internal IPC socket and does not derive it from its own package version. The Controller is what readsD, and/readymay pair a newer UI with an older Controller. Only a definitive answer means unsupported: a 404, or a schema below the one the shell writes. A Controller too old to know the capability operation answers the IPC call with its unknown-operation error. The UI maps that tocomputer_use_schema: 0, which is definitive. The answer also carries the Controller'scontroller_id, which is random per Controller process. The cached answer is keyed by that id, and it is cleared whenever the/readymonitor re-runs adoption. Only a cached answer from the same Controller is ever reused. A transport error, such as an IPC timeout or a reset, keeps that same-Controller answer. A transient failure therefore never stops a healthy daemon. Losing the Runtime itself is already handled by the/readymonitor. Any other non-definitive answer, such as a 500 or a malformed body, also keeps the last known answer. Before the first definitive answer, the shell spawns nothing and retries on the next/readytick.- Turning the toggle on requires that value to cover the schema the shell writes. Otherwise the shell refuses to turn it on, and the menu text says the Avibe service must restart to support computer use.
- Turning it off always works. A toggle that was already on stays on as
needs_runtime, with no daemon, until a capabilities probe sees support. The shell re-probes on each successful/readycheck. The checked item stays interactive in every state, so turning it off is always possible. - The shell never restarts an adopted Runtime itself. A pre-feature Runtime lacks the field and therefore reads as unsupported.
Permission requests (macOS). The shell requests Accessibility with
AXIsProcessTrustedWithOptionsand a prompt. It requests Screen Recording withCGRequestScreenCaptureAccess, falling back to opening the Settings pane.- It prompts only on the user's toggle-on action. Every other check is silent, because each prompting call while a grant is missing queues another system dialog.
- On macOS 26 an app appears in the Screen Recording pane only after a real
capture attempt. So the request also makes one attempt from the shell
process itself. It enumerates ordinary
SCShareableContent, selects a display, and performs a filter-based ScreenCaptureKit capture whosesourceRect, output dimensions, and destination are all explicitly bounded to 1 x 1. Pixels are discarded, and the shell neither logs nor retains the enumerated windows and applications. The shell dispatches both request calls to the AppKit main thread, gives enumeration and capture one shared 5 s deadline, and records whether either stage returned an image, anNSError, an empty result, or no callback inbootstrap.log. A late enumeration callback cannot start capture after that deadline. TCC attributes the attempt to Avibe.app, and it needs no daemon, which matters because the daemon starts only after both grants exist. - Phase 1 acceptance requires the toggle-on request to register the fixed
installed app in the Screen Recording pane before anyone uses the pane's
manual
+control. Manually adding the app is a recovery workaround, not evidence that the product request path succeeded. - Owner decision (2026-10-07). On macOS 26 ad-hoc builds, the guided
manual
+/drag recovery is accepted as the Phase 1 product path when the automatic registration attempt leaves the app absent. The shell must open the Screen & System Audio Recording pane and explain, through the localized shell/UI catalog, that the user should click+or drag Avibe into the list and then enable it. This accepts the workaround for Phase 1 while preserving the distinction that automatic registration did not succeed. Revisit the automatic path separately when Developer ID signing exists; Developer ID remains outside this Phase 1 signing scope.
Lifecycle. The shell runs one state machine whose states are the
Dstates. Every transition writesDfirst, except thatreadyis written only after the health check passes.- Failure budget. One rule, shared by every failure row. Each failed
start and each failure from
readycounts once. When the count reaches 3 within 5 minutes, the state becomeserrorwith the reason. Otherwise it becomesstartingand respawns after backoff. A daemon that passes startup and then crashes therefore cannot restart forever. Raw driver health details are folded at this lifecycle boundary into the stable user-facing reason set:bundle_identity,ax_capability,capture_failed, ordriver_health_failed. Process, endpoint, asset, and state failures retain their existing stable reasons. The Workbench maps every emitted reason to localized remediation instead of displaying raw driver codes. - Failed writes. A failed
Dwrite never blocks a stop. Toggle-off, quit, and revocation always stop the daemon. If their write failed, the menu says the setting was not saved, and the shell retries the write every 5 s while it runs. A failed write never starts anything: a transition intostartingthat cannot writeDgoes toerrorwith reasonstate_unwritable. A failed post-healthreadywrite does the same: it stops and reaps the healthy daemon, then holdserror/state_unwritablein memory and shows it in the menu. - Optional initialization. Computer Use initialization is best effort. A lock race, unwritable state directory, or relaunch/updater overlap logs a warning and leaves a disabled native control with localized initialization failure text; it never aborts Tauri startup or unrelated Runtime, notification, and updater setup.
- Responsiveness bound. The native permission request uses one shared five-second deadline for its main-thread checks and ScreenCaptureKit probe. A startup health wait has a five-second deadline. The actor consumes a queued toggle-off after the current bounded operation completes, invalidates pending retry events, and then stops the daemon. Daemon reaping waits at most three seconds before termination. These are explicit upper bounds, not a retry or debounce policy; tests cover the shared permission deadline, child cancellation, and retry invalidation.
- Shell exit boundary. Cmd+Q, Dock and logout/restart termination, updater restart, tray quit, and the explicit exit helper share one idempotent Computer Use shutdown claim. Shutdown invalidates delayed startup retries, stops and reaps the daemon, removes its socket, and releases the shell lock before acknowledging the native shutdown or starting a replacement process. A stuck startup attempt remains bounded by the health and reap deadlines and cannot start another daemon after shutdown has been requested.
The first matching row wins:
From Event To Action any toggle off offstop the daemon and any permission-registration child if running offtoggle on, Runtime lacks a covering computer_use_schemaoffrefuse; the menu says the Avibe service must restart offtoggle on, both grants held startingspawn offtoggle on, Screen Recording missing needs_permissionprompt, run the shell's one-pixel probe, and start one bounded child capture-registration attempt offtoggle on, only Accessibility missing needs_permissionprompt and run the shell's one-pixel probe; do not start the child capture shell launch Dunreadable, malformed, or of a newerschema_versionunchanged spawn nothing and do not rewrite D; the menu warns. Only an explicit toggle action overwrites itshell launch enabledand recordedstateiserrorerrornone; no spawn shell launch Dmissing or notenabledoffif Dexists, writeoffwith this shell'sinstance_idshell launch enabled, Runtime lacks a coveringcomputer_use_schemaneeds_runtimenone; no spawn shell launch enabled, both grants heldstartingspawn shell launch enabled, a grant missingneeds_permissionnone; silent needs_runtimecapabilities now cover the schema, both grants held startingspawn needs_runtimecapabilities now cover the schema, a grant missing needs_permissionnone; silent any enabledstate excepterrora successful /readyis followed by an unsupported capabilities answerneeds_runtimestop the daemon and any permission-registration child if running any enabledstate excepterroradoption found a new controller_idwith no definitive supported answer yetneeds_runtimestop the daemon if running needs_permissiongrant check changes from missing to all granted startingspawn once and consume the edge needs_permissionapp activation or 5 s check, grant check still fails needs_permissionnone; never start a child needs_permissiondaemon previously reported a missing grant and shell preflight still reports all granted needs_permissionnone; wait for a shell-observed missing → granted edge or explicit off/on startingsocket accepts and the health check returns passreadynone startingthe health check returns missing_grantneeds_permissionstop the daemon; no prompt startingspawn or socket fails, or the health check returns unhealthyper the failure budget stop the daemon; apply the failure budget readythe health check returns missing_grantneeds_permissionstop the daemon readydaemon exits unexpectedly, or its socket refuses or the health check returns unhealthyon 2 consecutive ticksper the failure budget stop the daemon if alive; apply the failure budget any enabledstate excepterrorquit stoppedstop the daemon and permission-registration child errorquit errornone; erroris kept across relaunchGrant check. Silent, and it runs only while
enabled. It fires on app activation and every 5 s. Inneeds_permission, it uses the shell's ownAXIsProcessTrustedandCGPreflightScreenCaptureAccess. A false result leaves the state unchanged and cannot authorize a daemon or registration child. A proven missing→granted transition entersstartingimmediately and consumes that edge. If daemon startup or a ready heartbeat reports a missing grant, automatic eligibility is disarmed: repeated shell all-granted observations stay inneeds_permission. The shell must first observe a missing grant and then all granted, or the user must explicitly turn Computer Use off and on, before another daemon can start. Inready, the 5 s tick runs the daemon health check below instead. The same tick also probes the daemon's socket. A wedged daemon that is alive but not accepting is treated like one that exited.Grant recovery. macOS can cache TCC answers per process. If the silent check remains false after the user grants access, returning to Avibe alone is not a retry mechanism. The user explicitly turns Computer Use off and on. If that fixed-byte process still has a stale answer, the user quits and reopens the same installed app, then turns the toggle on. The acceptance harness must restore its isolated launch environment and verify the Runtime origin after such an OS relaunch. Neither action rebuilds or re-signs the app, and neither adds a background permission retry.
Capture registration child. Only an explicit toggle-on while Screen Recording is missing starts it, after the shell's main-thread AX, CGRequest, and one-pixel ScreenCaptureKit probe. The shell directly executes the pinned helper as
cua-driver mcp --direct --embedded --no-overlay, initializes MCP over stdio, and callsget_desktop_stateonce with a one-pixel output cap. This is the ordinary child-owned ScreenCaptureKit path that made the Phase 0 app grantable. The returned pixels are discarded and never logged or persisted. The child gets private HOME/XDG/state directories, the managed policy, telemetry and update checks disabled, one 10 s deadline, and no retry. Toggle-off, shell quit, stdin EOF, and timeout all bound and reap it. Its completion never changesD: missing permission remainsneeds_permissionuntil a later silent check proves the grant.Activation feedback boundary.
NSApplicationDidBecomeActiveis an observation, not user authorization. The injected sequenceauthorized spawn → host deactivates → child exits → host activatesmust produce exactly one child spawn. This prevents driver lifecycle focus changes from recursively authorizing more driver processes.Support invariant. Every row that can spawn or prompt sits behind a supported capabilities answer. So any enabled state other than
needs_runtimeimplies that the latest probe said supported. An unsupported answer always leads toneeds_runtimewith no daemon.No indirection. Every row names its target state and action.
errorleaves only through toggle off (the first row), and a later toggle on starts fromoff.Health check. One composite operation with one 5 s deadline in total. It runs in two modes. The full mode runs on every start (
starting) and includes step 4. The light mode is thereadyheartbeat every 5 s and skips step 4, so an idlereadydaemon never captures the desktop. A capture failure later surfaces in the agent's own tool results. It returns exactly one ofpass,missing_grant, orunhealthy, and the lifecycle rows refer only to those three outcomes. A check added later therefore needs no new row. On macOS, the check runs these steps:- Call
health_reportand read itschecksarray. - If
tcc_accessibilityortcc_screen_recordingfails, returnmissing_grant. - Require
bundle_identityto pass withidentity_source: parent_application, which is the host-attribution evidence of Phase 0 Q1. Also requireax_capabilityto pass. - (Full mode only.) Call
get_desktop_statewithmax_image_dimension: 64, and require an image.health_reportis read-only and skipsscreen_capture_capability(Phase 0 output), so capture is proven this way.
Any other failure, an error, or the deadline expiring returns
unhealthy, with the failing step as the reason.source.attributionbelongs tocheck_permissions, which the policy excludes, so it is never read. Windows emits different checks (ax_capability,screen_capture_capability), so the Windows Q6 run sets the steps there. The menu shows the localized reason forneeds_permissionanderror.- Call
Orderly quit. Write
stopped, then stop the daemon.Dkeepsenabled, and the effective-status table readsstoppedasshell_not_running, the same as a released shell lock after a crash.
- Failure budget. One rule, shared by every failure row. Each failed
start and each failure from
Daemon. Spawn directly with
Command, never throughopen/LaunchServices. Mirror upstreamEmbeddedCuaDriverHost(incua-driver-sdk/src/embedded.rs) at the pinned tag. Both the daemon and the per-generation MCP proxy resolve the sameContents/Helpers/cua-driverexecutable.- Arguments:
serve --embedded --parent-liveness-stdio --no-permissions-gate --socket <S> --host-bundle-id <bundle id> --permission-mode standard --glide-ms 200.--glide-msfixes the virtual cursor's glide at 200 ms (see Latency). - Environment: start from an empty environment, then set
CUA_DRIVER_EMBEDDED=1,CUA_DRIVER_EMBEDDED_HOST_PID=<shell pid>,CUA_DRIVER_MANAGED_POLICY_FILE=<bundled tool policy>,CUA_DRIVER_RS_TELEMETRY_ENABLED=0,CUA_DRIVER_RS_UPDATE_CHECK=0, andCUA_DRIVER_WINDOW_CHANGE_TIMEOUT_MS=500, which bounds the post-action watch and the focus-steal lease that comes with it at 500 ms (see Latency). - Avibe does not depend on that crate. It is unpublished and pulls in the whole platform stack, while the protocol needs only these few lines.
- Liveness. The shell keeps the child's stdin pipe open for the daemon's
lifetime and never writes to it. When the shell dies, the OS closes the pipe
and the daemon exits, so a crash leaves no daemon holding Avibe's grants. An
orderly stop closes the pipe and waits up to 3 s for exit. It then
terminates the process (
SIGKILL, orTerminateProcesson Windows) and reaps it. So a daemon that ignores EOF can never hold a stop open or block the next spawn. The spike host did not pass--parent-liveness-stdio; Phase 1 tests this with a killed shell. - Endpoint. The socket lives in the shell's app data directory (see
D). On Windows it is a private pipe name. A dead daemon leaves its socket file behind, and a new daemon refuses an existing endpoint. So before each spawn the shell probes the socket. If it still accepts, the shell waits up to 2 s for an orphan to see its closed pipe. It then removes the file, or goes toerrorwith reasonendpoint_busyif the socket still accepts.
- Arguments:
State file
D.computer-use.jsonin the shell's per-user app data directory is the only shell-to-Runtime contract. On macOS that directory is~/Library/Application Support/bot.avibe.desktop/, wherebootstrap.logalready lives. On Windows it is%APPDATA%\bot.avibe.desktop\. The shell writes it atomically on every change. It is also the only cross-process record: the Runtime, the computer MCP server, and the separately running Workbench API process all read the same file.- Every write carries
schema_version,enabled(the toggle's only durable record and the configuration input),state,reason(a stable code, or null),shell_pid,instance_id(random per shell process),generation,driver_version, andtool_snapshot(the absolute path and SHA-256 of the tool snapshot bundled beside the driver).stateis one of these:off: the toggle is off.needs_permission: the toggle is on and a grant is missing.starting: the toggle is on, both grants are held, and the daemon is not yet accepting connections.ready: the socket accepts connections.error: start or health failed for a reason other than a missing grant.stopped: the toggle is on and the shell is quitting or has quit.needs_runtime: the toggle is on, but the adopted Runtime cannot read this schema. No daemon runs.
readyalso carriessocket_path,proxy_executable(absolute path to the bundled binary) andhost_bundle_id.generationincreases with each daemon spawn within oneinstance_id. The pair identifies a daemon across shell restarts.- The shell writes a non-ready state before stopping the daemon (
offfor toggle-off,stoppedfor quit). It never deletes the file, soenabledsurvives quits and restarts. A missing file means the user never enabled the feature. - Shell liveness is a lock, not a PID. Each shell holds an exclusive
lock on
computer-use.lock, next toD, for its whole lifetime:flockon macOS,LockFileExon Windows. The OS releases it when the shell dies, even on power loss. A reader that can take the lock knows no shell is running, so a reused PID cannot fake liveness.shell_pidstays inDfor diagnostics only. - The location belongs to the desktop shell, of which each OS user has
one. It does not belong to the Avibe home. So every Runtime finds
Dwithout seeing the shell's launch environment, whatever home it resolves:AVIBE_HOME, the default~/.avibe, or a legacy-only~/.vibe_remote.core/computer_use.pyresolves the same per-platform path, and tests redirect it.
- Every write carries
Tool policy. Ship a YAML allow-list as the driver's managed policy (Phase 0 finding). It is pinned with the driver version, because the list must be reviewed against each new tool surface. The list has exactly 29 tools (28 in v1, plus
bring_to_frontfrom #3050), and the bundled tool snapshot must equal it:- Observation:
list_apps,list_windows,get_window_state,verify_state,get_accessibility_tree,get_screen_size,get_desktop_state,get_cursor_position,zoom. - Apps and windows:
launch_app,kill_app,bring_to_front,set_window_frame. - Input:
click,double_click,right_click,drag,scroll,type_text,press_key,hotkey,set_value. - Clipboard:
clipboard_read,clipboard_write. - Sessions:
start_session,end_session,get_session,list_sessions.- Diagnostics:
health_report.
- Diagnostics:
check_permissionsis excluded. Itsprompt: trueraises system dialogs, and only the native toggle-on action may prompt. The policy applies to every caller, the shell included, so the shell's health check useshealth_reportinstead.move_cursorandinvoke_menuare excluded: the first moves the user's real pointer outside any window action, andinvoke_menuactivates the target and restores the previous app only on a best-effort basis.bring_to_frontwas excluded in v1 and is admitted by #3050 as explicit, exact-window activation.- Focus boundary (macOS). Background delivery is the default and asks
the driver not to take focus, although an app can still activate itself
in response to any event. Since #3050,
delivery_mode: "foreground", the focus-intent shortcuts named by the driver guide, andbring_to_frontare forwarded like other window operations; input must still address one exact window (below). The driver serializes foreground and activation calls on its process-wide desktop lane. - Window-only input (macOS). The computer MCP server admits an input
call only when it addresses one window: a
pidwith awindow_id, or an element token. It rejects any other targeting, including a desktop target (target.kind: "desktop"orscope: "desktop"),delivery_mode: "foreground", and anything it does not recognise. This is an allow rule, so a targeting option added in a later driver is refused until it is reviewed. The Windows Q6 run decides the Windows policy, where some surfaces work only in the foreground.
It omits config, update, extension, recording/replay, cursor-theme, legacy
page, the typed browser tools, and visual parsing. Typed browser tools are deferred, not rejected: they need their own runtime and origin scope.- Observation:
Runtime (core/, modules/agents/)
One owner. A new
core/computer_use.pyowns both reads ofD.Configuration. It returns the single managed stdio MCP spec (name
avibe_computer, launching Avibe's computer MCP server with Avibe's Python) only whenenabledis true andD.tool_snapshotexists with a matching SHA-256. Otherwise it returns nothing. A failed snapshot check logs one warning per distinct hash, and the status readsunavailable/snapshot_invalid. An optional feature with a broken bundle is never injected, and the rest of the agent keeps working. Backends only translate this spec.Effective status. One total derivation, where the first matching row wins. The server and the Workbench both use this one reader.
DStatus Reason missing offnever_enabledunreadable, malformed, or unknown schema_versionunavailableinvalid_state_fileenabledis falseofftoggle_offtool_snapshotmissing or hash mismatchunavailablesnapshot_invalidshell lock not held, or stateisstoppedunavailableshell_not_runningstateother thanreadythat state its recorded reasonready, socket refuses a connectionunavailabledaemon_unreachableready, socket acceptsreadynull Configuration reads only
enabledandtool_snapshot. A missing or invalid file counts as not enabled. Atomic writes keep a half-written file from appearing. Runtime adoption uses two stable reasons:runtime_unavailablemeans the shell has not yet confirmed the adopted Runtime or it is restarting;runtime_too_oldis reserved for a definitive capability answer below the shell's schema. The native menu and Workbench keep those cases distinct, so a restarting Runtime is never described as too old.
Computer MCP server. A small Avibe-owned stdio server is the one stable endpoint for every backend.
- It starts without a daemon. It serves the tool list from the snapshot
that
D.tool_snapshotnames, and checks the hash. The shell bundles that snapshot with the driver and the policy, so the tools always match the driver the shell runs, whatever version the Runtime is. - On each call it reads the effective status. If it is
ready, the server forwards the call through an upstreamcua-driver mcp --embedded --socket <S>child. It respawns that child when the (instance_id,generation) pair changes or the child has exited. The child getsCUA_DRIVER_EMBEDDED=1,CUA_DRIVER_RS_TELEMETRY_ENABLED=0, andCUA_DRIVER_RS_UPDATE_CHECK=0; without the last, the proxy runs its own update check and prints a banner. - Otherwise it returns an error result that names the state and reason (off, needs permission, starting, error, unavailable). The agent can then tell the user what to do.
- After a respawn, element tokens from the old daemon are invalid. Its error tells the agent to observe again.
- It moves
structuredContentinto a trailing text block and dropsoutputSchemafor every backend. Codex needs this (Phase 0, Q4), and it costs the other backends nothing. - Concurrent sessions. Claude spawns one server per session. Codex
and OpenCode share one server across their conversations, so a caller
cannot be identified by process. Isolation uses the Avibe session id
that every session prompt already carries
(
core/prompts/session-start.md), and the server enforces it:- The server adds a required
sessionstring to every advertised tool schema, so schema-driven clients always send it. The injected prompt says to fill it with the session id. - The server calls
start_sessionfor a session's name before forwarding in four cases: its first call, its first call after anend_session, its first call after the daemon key (instance_id,generation) changes, and its first call after the server replaced its upstream proxy child. Closing that transport ends its sessions in Cua. That call creates or revives the session. Agents never manage the Cua session lifecycle. - Calls from one session run one at a time, and the server queues the
rest. So a session never has two calls in flight. An
end_sessionreleases the lease only after the calls queued before it have finished. - The server forwards the id as the named Cua session. Element tokens and session state are therefore per Avibe session, even on one shared upstream connection.
- The server adds a required
- Desktop lease. There is one desktop, so only one session acts at a
time.
- Acquisition, refresh, and release run under an exclusive OS lock on
computer-lease.lock, next toD(flockon macOS,LockFileExon Windows). Inside that lock the server reads and writes the lease record: holder session id, last refresh time, a leaseepochthat increases with each new holder, and the daemon key (instance_id,generation) it was taken under. Acquisition also re-reads the effective ready daemon key while holding this same lock. A caller that read staleDtherefore cannot overwrite a holder admitted for the current generation. - Acquisition reports whether this call newly claimed the holder or merely
renewed the same session. An ordinary pre-forward failure releases only
a newly claimed lease; a renewed lease survives guards such as
observe_firstand a transient same-key readiness failure. If the daemon generation really changed, the next acquisition invalidates the old-key lease while holding the lease lock; cleanup never needs to release a renewed holder speculatively. - Two first calls from different processes therefore serialize. One
wins, and the other gets
desktop_busynaming the holder. - The holder starts an ownership-aware heartbeat immediately after
acquisition and refreshes every 10 s through proxy setup, session
revival, and forwarding. A long setup or call, such as a 120 s
get_accessibility_tree, therefore keeps the lease. If refresh proves that another holder owns the lease, setup is cancelled before forwarding. Loss after an input may already have reached Cua returns an error and discards the proxy; it never claims that the delivered action was undone. - The lease lapses 60 s after the last refresh. A holder that crashed,
even mid-call, stops refreshing and frees the desktop within a minute,
without any PID check. Calling
end_sessionreleases it at once. - A lease whose daemon key differs from
D's current one is void. A native stop, a toggle off and on, or a respawn therefore clears every lease without waiting out the 60 s. - After creating or reusing an upstream proxy, the server re-reads the effective status before session revival or forwarding. A daemon-key mismatch closes only that exact proxy and may retry admission once, before any tool request has been forwarded. EOF, timeout, or another failure after forwarding is never replayed, because an input result may be unknown.
- Proxy liveness includes the JSON-RPC reader task, not only the child process. Request registration checks reader completion on both sides of publishing its future. One 180 s transport deadline covers waiting for the shared write lock, flushing the request, and waiting for the response; this remains above Cua's documented 120 s accessibility-call bound. EOF and timeout discard only the failed transport; a late failure from an old proxy cannot clear its replacement or the replacement's observations.
- Each session remembers the epoch it last held. Another session may have
held the desktop in between, so a session can reacquire the lease under a
newer epoch. Before forwarding that call, the server runs
end_sessionand thenstart_sessionfor its name. A tool-level error from the internalend_sessionaborts revival before any user tool is forwarded. A successful handoff makes Cua invalidate every element token the session held, so a stale token fails with Cua's own error. Coordinates from an old screenshot are not covered by that, so the server tracksobserve_firstper target window. After an epoch change, the server refuses input to a window (pid,window_id) withobserve_firstuntil that session has calledget_window_stateorzoomon that same window. Observing window A therefore never clears input to window B, and calls such asget_desktop_stateorget_cursor_positionclear nothing. A server process that has no record of a session, for example after the backend restarted it, starts that session with every window requiring observation. Nothing has to persist across server processes.
- Acquisition, refresh, and release run under an exclusive OS lock on
- Same-user trust boundary. The Python server's target locks,
observe_first, and window-target checks are cooperative controls for Avibe-owned callers. A same-user local process that discoversD.socket_pathorD.proxy_executablecan connect to the daemon or invoke the proxy directly and bypass those Python-layer checks; they cannot constrain arbitrary local processes under the same OS account. This is an accepted Phase 1 trust boundary and is stated in the user documentation. - Tool restriction stays in the driver's managed policy, not in this server.
- The proxy exits with its daemon and never reconnects (Phase 0, Q3), so this server is what keeps one endpoint alive per backend lifetime.
- It starts without a daemon. It serves the tool list from the snapshot
that
Claude Code. Pass
mcp_servers={"avibe_computer": ...}in the session options built incore/handlers/session_handler.py. SDK 0.2.93 supportsMcpStdioServerConfig. Keepstrict_mcp_configunset so user MCP config is preserved. An agent file that setstoolsmust listmcp__avibe_computer__*to keep access.Codex. Append
-c mcp_servers.avibe_computer.*overrides to the app-server launch, after Avibe's fixed overrides.features.computer_use=falsestays: Avibe supplies one cross-backend tool instead of Codex's own. Also setmcp_servers.avibe_computer.default_tools_approval_mode="approve"(Phase 0, Q4). Avibe runs Codex withapprovalPolicy: never, and Codex cancels every non-read-only MCP call under that policy.OpenCode. Add
mcp.avibe_computer(type: "local") through the existing config overlay /PATCH /global/configpath.Why the server follows the toggle. The allowed tool schemas measured about 70 KB, roughly 17.6k tokens per session (on the 32-tool draft, which also included
check_permissions). Every session of a user who never enabled the feature would pay that, so the server is not injected unconditionally. Only availability is absorbed at call time.Configuration reconciliation.
core/computer_use.pyowns the one change that alters backend configuration. A controller background task readsenabled, thetool_snapshothash, and whether the snapshot file currently verifies, every 2 s. This is the same polling model asRuntimeCommandWatcher. The hash is recomputed only when the file's size or mtime changes. The task compares this triple with the triple each live consumer was built from. A deleted, corrupted, or restored snapshot therefore reconciles even when the expected hash inDis unchanged. A desktop update that ships a new driver therefore reaches running sessions the same way a toggle flip does. On a mismatch it reconciles every consumer that caches MCP configuration:- Codex and OpenCode: the same
AgentAuthService.renew_backend_runtimehandler that restart markers call. - Cached Claude clients: marked stale, then retired and recreated at their next turn boundary, never mid-turn.
The comparison is against the final value, so a flip and flip-back between polls correctly needs nothing. Availability changes never reach this path. The stop guarantee does not depend on it either: turning the toggle off stops the daemon at once, so calls fail with
offbefore reconciliation removes the tools.- Codex and OpenCode: the same
Prompt. Add a short section in
core/system_prompt_injection.py, only when the server is configured. It says the tools can report a state such asofforneeds_permission, and then the agent tells the user instead of retrying. It says to fill the requiredsessionfield with the session id. It saysdesktop_busymeans another session holds the desktop, so the agent waits or tells the user, and to callend_sessionwhen a GUI task is done. It also says to prefer CLI/API routes, use GUI tools for GUI-only steps, prefer accessibility element-token actions over pixel input, treat screen content as untrusted, and observe state before retrying any action, since an error result does not prove the action failed. If every window comes back AX-unresolved and the desktop shows no apps, the screen is likely locked: stop and tell the user. Background delivery is the default; since #3050 the agent may repeat an action that had no observable effect with foreground delivery, or activate the exact window, without asking first. On Windows, the Q6 run decides foreground use.
Workbench
A read-only status line, derived from the effective status: ready, off, needs
permission, needs a newer Runtime (with "restart the Avibe service"),
starting, error with reason-specific remediation, or unavailable with
reason-specific remediation. Unknown future reasons use a localized fallback
and never expose an internal reason code. The line appears only in a
macOS Runtime whose interpreter or launcher marker proves desktop-managed
provenance; a normal terminal-installed macOS Runtime does not advertise a
native control it cannot own. Validation covers one rendering per status and
every emitted error or unavailable reason. The Workbench API reads state
through the same core/computer_use.py reader, not through Runtime memory,
because it runs in a separate process. Copy goes through
ui/src/i18n/en.json and zh.json. The control itself stays native in v1.
Installed-build prerequisite
The app must also be a registered bundle in a normal location
(/Applications or ~/Applications). An unregistered bundle run from /tmp
never appears in the Privacy panes, so the user cannot grant it.
TCC binds a grant to the app's code requirement. Ad-hoc acceptance builds match by cdhash, so every new build loses the grant. In the spike, toggling the stale row off and on in System Settings did not repair it: tccd kept checking the old cdhash requirement until the app was granted again.
Owner amendment, 2026-10-07: the Phase 1 acceptance gate is an ad-hoc app
installed in /Applications or ~/Applications and held byte-for-byte fixed
between the grant and the test. Developer ID signing and notarization are
separate release infrastructure and are not a Phase 1 gate. When an update
changes the ad-hoc cdhash and orphans the old grant, an enabled computer-use
toggle must settle at needs_permission with a clear re-grant path. It must
not become stuck in error or start a permission retry loop.
Upstream mechanism risk
On macOS, background pixel input goes through private SkyLight SPI
(SLEventPostToPid, plus an authentication message for keyboard events to
Chromium targets on macOS 14+). The symbols are resolved with dlsym, and the
driver falls back to CGEventPostToPid if they are missing. Accessibility
actions (AXUIElementPerformAction) and ScreenCaptureKit are public API. A
macOS update can therefore degrade background pixel input without notice.
Pinning the driver version and adding a release smoke check are the mitigation.
This also rules out Mac App Store distribution, which Avibe does not use.
Phase 0 spike: questions that change the design
Run on a local ad-hoc build. Grant permissions manually.
- Embedded attribution: does a Tauri-spawned daemon report
attribution: host, and doesbundle_identitypass? - End to end: can Claude Code, through the proxy, background-click Calculator
to
42with Terminal frontmost, and receive screenshots as images? - Proxy without a daemon: what happens at startup, and when the daemon restarts mid-session (new generation)? This decides inject-on-availability vs. always-inject (see Availability changes).
- Codex and OpenCode: do MCP image results reach the model?
- Standard mode over MCP: what happens to residual authorizations (for
example, an existing logged-in Chromium profile) when no
DriverAuthorizationHostis installed? v1 accepts a refusal. - Windows: is a release asset available, and does the same spawn flow work (UIPI: elevated windows refuse)?
- Size: the macOS universal binary tarball is about 44 MB. Confirm the installed size delta.
Phase 0 findings (2026-10-01 to 2026-10-03, macOS 26.2, driver 0.31.0)
Direct-mode runs used cua-driver mcp --direct from a scratch directory with
HOME redirected. Embedded runs used a minimal ad-hoc signed Swift host
(adapted from upstream examples/embedded-host-macos) that spawned
serve --embedded directly and talked to it through mcp --embedded.
Q1 embedded attribution: yes.
check_permissionsreportsattribution: hostandbundle_identitypasses withidentity_source: parent_application. tccd logs the daemon's Screen Recording requests with the host as the responsible process. Granting Accessibility and Screen Recording to the host alone was enough for a background Finder AX tree plus window screenshot, and the driver raised no prompt of its own. Grants are cached per process, so the host restarts the daemon after a grant changes, as planned.Q2 end to end with Claude Code (macOS): works, with reliability caveats. A headless Claude Code 2.1 session got the server through
--mcp-config(mcp --embedded --socket), saw all 58 tools, and received screenshot image blocks (it described window details that were only in the image). Calculator was never frontmost in a 0.2 s frontmost-app sample across both runs while the user kept working. Run 1 (6×7) used AX element tokens: 9 calls, 54 s. Run 2 (8×9) started with background pixel clicks, which were unreliable on Calculator: one was refused, one landed late, and later digits registered twice. It recovered by switching to AX presses. Those returned-25204yet took effect (upstream trycua/cua#3836 tracks this). Run 2 took 34 calls and 7 minutes. Consequence: the injected prompt must say to prefer element-token actions, to use pixel input only as a fallback, and to observe before retrying any action, because an error result does not prove the action failed.Q3 proxy without a daemon: it exits. With no daemon on the socket, the proxy exits before the MCP handshake. When the daemon stops, the connected proxy exits too, and it does not come back after the daemon restarts. A fresh proxy works against the restarted daemon. So no backend may hold the raw proxy as its MCP server. The Avibe computer MCP server holds it and respawns it per daemon generation.
Q4 Codex (codex-cli 0.145.0): images are dropped, and writes are cancelled. Both have workarounds. Every cua image tool (
get_window_state,get_desktop_state,zoom) returnsstructuredContentnext to its image. Codex then forwards only the structured part. An A/B check with a one-tool server showed this: withoutstructuredContentthe model read72from the image; with it, the model saw no image (same result on two models). Upstream openai/codex#10334 tracks this and is open. A stdio filter that foldsstructuredContentinto a text block fixed it end to end: Codex read72from a cuaget_window_statescreenshot. Separately, underapproval_policy=neverCodex cancelled every MCP tool withoutreadOnlyHint: true, reportinguser cancelled MCP tool call. cua marksclick,type_text,press_key,launch_app, andkill_appthat way. Settingdefault_tools_approval_mode="approve"on the server let a destructive-annotated tool run;autostill cancelled it.Q4 OpenCode (1.18.16): images arrive and need no workaround. A sandboxed
opencode run(all XDG dirs redirected, provider passed throughOPENCODE_CONFIG_CONTENT) returned the image as an attachment, with and withoutstructuredContent. Through the real cua proxy, the model read72from aget_window_statescreenshot. A destructive-annotated tool ran under the default permissions. So only the approval override is Codex-specific. FoldingstructuredContentis harmless here, so the shared server does it for every backend.Artifact. The universal tarball's SHA-256 matches
checksums.txt. The binary is Developer ID signed by Cua AI (YCK386LBJ7) but not notarized, and is 71 MB unpacked. Avibe re-signs it as a nested executable. From 0.33.0 each asset also carries a sigstore bundle; pin the hash either way.Telemetry is on by default. Even
list-toolscreates a telemetry id.CUA_DRIVER_RS_TELEMETRY_ENABLED=0disables it and nothing is written. The shell must set this together withCUA_DRIVER_RS_UPDATE_CHECK=0; Avibe owns the driver version.Tool surface: restrict with a managed tool policy, not a manifest. The server exposes 58 tools, including
install_extension,set_config,check_for_update, andreplay_trajectory. Agents must not change driver config, install extensions (cua-perceptionis AGPL), or update the driver. A capability manifest does not fit general computer use: it is deny by default for resources too, so every app would have to be listed by bundle id. The driver's YAML tool policy only names tools. Set asCUA_DRIVER_MANAGED_POLICY_FILEon the embedded daemon, a 30-tool draft allow-list shranktools/listto 30 through the proxy. The final v1 list (see Tool policy) addshealth_reportand removesbring_to_front,move_cursor, andinvoke_menu, giving 28. A 32-tool draft that also allowedcheck_permissionslisted exactly its 32 tools under the policy. Omitted tools returnedpermission_denied, and allowed ones worked. Agent-side environment on the proxy could not widen it: a widening user policy, a widening managed policy, andunrestrictedmode variables each left the daemon's surface unchanged.kill_appstays allowed, because standard mode only terminates processes this runtime launched.End to end.
launch_appstarted Calculator in the background (self_activation_suppressed: true).get_window_statereturned a PNG plus a 154-element AX tree in about 1.9 s. Five AXclicks byelement_tokengave6×7 = 42, read back from the AX tree.Latency. After each action the driver watches for window changes for up to 1000 ms by default, and the daemon reads that bound from
CUA_DRIVER_WINDOW_CHANGE_TIMEOUT_MS(0 to 10000). An A/B on embedded background Calculator AX clicks ran 2 rounds of six clicks per setting. 1000 ms gave about 2.3 s per click, 300 ms about 1.6 s, and 0 ms about 1.35 s. Every run computed12×3 = 36correctly with no errors. A no-op call through the proxy takes 1 to 2 ms, so the remaining 1.3 s floor sits inside Calculator's AXPress (the slow-25204path, trycua/cua#3836). The watch also holds the driver's focus-steal lease, which reverts another app's activation while it runs. Upstream warns that shortening it weakens that protection. Upstream measured 78 ms at0, so the floor depends on how fast the target app answers. Click results reporteffect: unverifiableeither way, so agents observe after acting.- v1 (until 2026-10-21): 1000 ms. v1 kept the driver default and gave up the measured 0.7 s per action, because focus protection outweighed latency for unattended use and latency tuning waited for a measured need.
- Since 2026-10-21: 500 ms, chosen by the owner. The need arrived.
Profiling on the owner's Mac (rc13, driver at
5272e49) showed that when an action changes no window, the driver waits out the whole watch. That is about 85 % of a click. Halving the watch saves up to 0.5 s per action. The owner chose 500 ms rather than 300 ms to keep half a second of focus protection after each action. - Trade-off. A pop-up or focus steal that appears between 500 ms and 1000 ms after the action was caught before and now is not: the lease does not revert it, and the action result does not report it. At either bound, an app that activates itself after the watch can stay frontmost. The idle-desktop validation measures this, and the user docs state it.
- Cursor glide: 200 ms. The virtual cursor's default glide is
speed-based and adds 0.45 to 1.5 s whenever the click position changes.
The daemon starts with
--glide-ms 200, the pinned driver'sserveflag for a fixed glide duration. It takes 200 ms at any distance, and the cursor still visibly moves to each target.--cursor-reduced-motion onis not the knob: at the pinned driver it only stills the idle animation and leaves the glide unchanged.
Element tokens are session-scoped (
s00000001:5). Observe and act on one persistent MCP connection. A one-shot CLI call cannot reuse a token.A locked screen blocks all input; window capture still works. With the session locked (
CGSSessionScreenIsLocked), everyget_window_statereturned zero AX elements. The cause wasax_window_unresolved(no AXWindow under the pid), and the driver refused every background input route. Calculator also hit 7 s AX timeouts.get_desktop_statereturned only the lock-screen wallpaper, but window capture still returned the real Calculator contents.health_reportstill said all checks were ok, and no result names the lock. Consequence: the injected prompt must say that when every window comes back AX-unresolved and the desktop shows no apps, the screen is likely locked, so the agent stops and tells the user instead of retrying. Ask upstream to report the lock explicitly. Avibe will not try to unlock or keep the screen awake.Q5 residual authorization: unreachable under the v1 tool policy. In standard mode the residual boundary that needs a grant or a
DriverAuthorizationHostis attaching to an existing logged-in Chromium profile. Only the typed browser tools andpagereach it, and the managed policy hides all of them, so v1 needs no authorization host. Two other standard-mode refusals held over MCP:kill_appon a process the runtime did not launch returned "may terminate only a process proven to have been launched by this Cua runtime", and the process survived.check_permissionswithprompt: trueraised nothing once the host held both grants. Attaching to a real Chrome profile without the policy was not exercised, on purpose.Q6 Windows: assets exist; the spawn flow is unverified. Releases ship
windows-x86_64andwindows-arm64zips.cua-driver.exe(33 MB) andcua-driver-uia.exe(21 MB) are Authenticode signed by Cua AI, Inc., and both are console programs. Upstream says embedding works on Windows with no permission step, provided the daemon runs in the interactive logon session (Session 0 is refused). The daemon is a child of the desktop shell, which runs in that session, so it qualifies. Upstream also lists three Windows limits that matter for the prompt and docs:- Elevated targets need a high-integrity daemon. v1 accepts the refusal.
- Chromium page content, terminals, and GTK or custom-drawn surfaces
cannot be driven in the background. They need
delivery_mode: "foreground", which briefly takes focus. - The window-change wait does not exist on Windows.
Still open, and only answerable on a Windows machine:
- Does
serve --embedded --socketwork there? - Can the agent's proxy reach the daemon?
- Is the
cua-driver-uia.exeworker needed?
Windows stays behind macOS (see Todo).
Q7 size. Today's macOS app is 17 MB installed. The thin driver adds 33 MB (arm64) or 35 MB (x86_64) installed, and about 12 MB to a zlib-compressed DMG. On Windows,
cua-driver.exeadds 33 MB installed and about 8 MB to the LZMA-compressed NSIS installer. With the UIA worker it is 54 MB and about 12 MB. This is acceptable for a feature that is off by default. A lazy download was considered and rejected, because it adds a second supply-chain and update path for about 12 MB.Focus/cursor evidence was confounded. The user was using the machine during the run, so frontmost-app and cursor changes cannot be attributed. Redo this on an idle desktop.
Todo
- Phase 0 spike on macOS; answers recorded above
- Phase 0 Q6 on a Windows machine: embedded spawn, proxy reach, UIA worker
- Shell: sidecar packaging, pinned manifest, tool policy and tool snapshot, signing order, notices
- Shell: toggle, permission flow, daemon lifecycle, state file
D, health - Core:
core/computer_use.py, the computer MCP server, configuration reconciliation, Claude/Codex/OpenCode translation, prompt - Workbench status line + i18n
- User docs: enabling, permissions, stop
- Installed byte-fixed ad-hoc build, idle desktop: the agent cursor overlay stays visible, and representative AX and pixel actions neither take focus nor move the user's pointer (redoes the confounded Phase 0 check)
- Windows parity
Validation
Each case lives in the suite of the component that owns the behavior.
- Rust, shell lifecycle (
desktop/), with grant checks, spawn, socket, health,/ready, and capabilities faked. These cases cover:- One case per lifecycle-table row. Each asserts the resulting
Dand that no prompt is raised outside the toggle-on row. - Activation feedback: inject an explicit authorized spawn, host
deactivation, missing-grant child exit, and host reactivation. Assert
exactly one spawn. Repeated activations while the grant remains missing
never spawn or queue a child; a later silent check that proves the grant
enters
startingimmediately. Explicit toggle-off/on remains the retry path. - Permission disagreement: after either startup health or a ready heartbeat
reports
missing_grant, feed repeated all-granted activation and tick observations. Assert no daemon respawn, stableneeds_permission, and no failure-budget entries. A subsequent shell-observed missing → granted edge starts once, and explicit off/on also remains an authorized retry. Restore and capability adoption deliberately arm only after observing a missing grant. - Permission registration: one explicit toggle-on with Screen Recording missing starts exactly one direct child capture; an Accessibility-only failure does not. A fake child verifies the private HOME/XDG/state paths, managed policy, disabled telemetry/update checks, direct MCP arguments, one tool call, timeout, cancellation, and reap. Activation and background ticks never start this child.
- Health check: the full mode runs at start and includes the capture
probe. The
readyheartbeat never captures. A step that hangs or fails returnsunhealthyin both modes. - Start failures, on a first start or a respawn, retry with backoff.
They reach
erroronly on the third failure within 5 minutes. A daemon that passes startup health and then crashes fromreadyeach time also reacheserroron its third failure. - Capabilities: a transport error or a 500 keeps the last answer, but
only for the same
controller_id. Only a 404 or a lower schema means unsupported. A newcontroller_idwith no definitive answer stops a running daemon. A Runtime without a coveringcomputer_use_schemanever gets a daemon. Turning the toggle on is refused there, while turning it off still works. - Process: a daemon that ignores EOF is killed after 3 s. A killed shell
takes its daemon down and releases its lock, and the next launch
reclaims the endpoint.
generationbumps on each daemon spawn, and nothing is spawned through LaunchServices. Both daemon and proxy resolveContents/Helpers/cua-driver. - State file: a failed
Dwrite still stops the daemon, and the toggle persists only inD.
- One case per lifecycle-table row. Each asserts the resulting
- Release check (shell build), covering:
- The prepared archive, extracted universal binary, and thinned upstream slice have separate recorded digests. The final packaged digest and cdhash are recorded after signing.
- The helper exists only at
Contents/Helpers/cua-driver; noContents/MacOS/cua-driverships. - Nested code is signed before the outer app. Driver and outer signatures
verify with
codesign --verify --deep --strict, and identity-signed builds use the same authority and team.CodeResourcesrecords the helper as nested code. - The daemon environment names the bundled managed policy.
- The pinned driver's
tools/list, under that policy, equals the bundled tool snapshot. - The driver guide's no-foreground section is re-read for the new driver version.
- Python, UI process (
vibe/), covering these cases:/readyis byte-for-byte unchanged for released shells./api/desktop/capabilitiesanswers from the Controller over IPC.- An old Controller's unknown-operation error reads as schema
0.
- Python, Runtime configuration (
core/computer_use.py), covering these cases:- One case per row of the effective-status table, asserting both status and reason.
- A Runtime with
AVIBE_HOMEset reads the sameD. - The spec and the prompt section exist exactly when
enabledis true and the snapshot verifies. A missing or mismatched snapshot yields neither, plussnapshot_invalid. - Each backend translation is checked; Codex also carries the approval override.
- Reconciliation brings every live consumer (Codex, OpenCode, cached
Claude clients) to the final (
enabled, snapshot hash, snapshot valid) triple. Claude clients are recreated only between turns. - Availability changes, and a triple that returns to its old value between polls, cause no reconciliation. Deleting and then restoring the snapshot file reconciles both ways.
- Python, computer MCP server, with the upstream command faked. These
cases cover:
- Upstream: a call while not ready returns the named state and never
spawns a child. An (
instance_id,generation) change or a child exit respawns exactly once. Folding keeps image blocks. - Sessions: every advertised schema requires
session. Two sessions get separate Cua sessions on one upstream connection. A session is revived before its next call afterend_session, after a daemon respawn, and after a child replacement. A session's calls run one at a time, andend_sessionwaits for the calls queued before it. - Lease: simultaneous first calls from two processes yield exactly one
holder, and the other gets
desktop_busyuntilend_sessionor 60 s idle. Admission revalidates the daemon key under the lease lock, a renewed holder survivesobserve_firstand a same-key readiness blip, and a newly claimed setup failure releases immediately. Proxy creation is followed by generation revalidation; admission retries exactly once. A 120 s call keeps its lease. A stop and re-enable voids every lease. - Transport: reader EOF before request registration fails promptly; a hung response, blocked drain, or caller queued behind the write lock reaches the same 180 s bound; and a late failure from an old proxy leaves the current replacement and its observations intact.
- Stale input: after an epoch change, old element tokens are rejected,
and input to a window gets
observe_firstuntil that same window is observed. Observing window A does not clear window B. A new server process requires an observation before a session's first input. - macOS input: a desktop-targeted or unrecognised input target is rejected; foreground delivery and focus-intent shortcuts are forwarded (#3050).
- Upstream: a call while not ready returns the named state and never
spawns a child. An (
- Manual, on the one final installed byte-fixed ad-hoc build.
- The final candidate must be built from the guidance-fix exact source head under a new bundle identifier. Candidate 5 remains fixed and ungranted; its retained evidence is pre-guidance evidence only.
- Toggle Computer Use once and follow the product guidance: use the opened
Screen & System Audio Recording pane, click
+or drag the exact Avibe app into the list, then enable it. The source inspects the list before any manual action, and the owner performs exactly this one grant sequence. - Record the final candidate's missing → granted recovery. Returning to
Avibe should let the existing edge enter
ready; if the fixed app still reports a stale grant, document the exact isolated Quit & Reopen recovery instead of claiming a relaunch requirement in advance. - During the permission attempt and the post-grant idle window, record the
actual helper CHECKIN identity, require no outer bundle id and no helper
frontmost assertion, and require no respawn loop. Keep the
needs_permissionidle count across multiple five-second ticks. - After the owner grant, validate the persistent serve path, overlay and visible cursor, idle focus and pointer behavior, native stop, parent-liveness cleanup, normal Runtime MCP injection, and the Claude product-prompt E2E. The direct no-overlay registration child cannot substitute for these checks.
All automated cases are hermetic: the D path and the upstream command are
redirected to test-owned fakes.
Owner decision ledger
- 2026-10-11 — Slice 1 deferred follow-up. The owner deferred the
orphaned lease retirement behavior, human status/reason rendering,
Workbench Controller health status, and remaining Computer Use documentation
polish to a follow-up. The false Turn-capability statement in
docs/COMMANDS.mdis corrected in this delivery; the remaining P2 work does not expand the Slice 1 contract. - 2026-10-08 — Workbench support ownership follow-up. The owner accepts
terminal-started Runtimes advertising the desktop shell's shared Computer Use
status as a Phase 1 limitation. Track
a per-Runtime shell-adoption signal for Workbench supportas a follow-up rather than changing the support predicate in this delivery. - 2026-10-08 — acquire-to-forward lease ownership. The earlier deferral of pre-forward lease protection is superseded: a 180 s proxy/session request can outlive the 60 s lease TTL. One ownership-aware heartbeat now covers acquisition through setup, revival, and forwarding, while preserving the single server, lock, state file, bounded admission retry, and no-replay contracts. Retained follow-ups are in-process proxy contention, the narrow stale-admission/newer-proxy interleaving, retry after an already-removed old socket, and the separate shell-lock startup retry.
- 2026-10-08 — f043 review scope. Keep
ComputerUseServeras the single owner and close lease generation, proxy reader, admitted Codex prompt, Workbench provenance/remediation, workflow dependency, and release metadata contracts in one exact head. Do not add another lock, state file, or server type. - 2026-10-08 — shell-lock retry follow-up. A Python shell-lock probe can race a desktop relaunch. Preserve one-shell ownership in Phase 1 and track a bounded startup retry as a separate follow-up instead of weakening the lock in this review head.
- 2026-10-07 19:22 UTC+8 — accepted Phase 1 workaround. The owner accepts
the localized Screen Recording
+/drag and enable flow when macOS 26 ad-hoc signing leaves the automatic row absent. Automatic registration remains a known limitation and is deferred for a separate Developer ID investigation. The accepted path is product guidance, not documentation alone. - 2026-10-07 18:56 UTC+8 — Candidate 5 evidence boundary. Candidate 5 established the Helpers child identity and TCC responsibility chain, with no helper CHECKIN under the outer bundle identity and no recorded helper frontmost assertion. It did not establish generic idle focus, pointer, or cursor acceptance, and it remains ungranted.
2026-10-09 follow-up contract
This follow-up is based on the post-merge acceptance ledger at
/tmp/avibe-cua-final-acceptance-ses9qeb46suvv-84fcff21/acceptance-ledger.md
and the frozen evidence roots
/tmp/avibe-cua-final-sd and
/tmp/avibe-cua-phase1-acceptance-ses9qeb46suvv-be8e909/home.
The implementation branch starts from exact base
84fcff2101223a65ddc6f5b878d7ba61805ffb96. The sensitive
live-step9-consented-retry-outcome.json is excluded from implementation and
publication.
The five findings and their owning contracts are:
- F1, Darwin socket and diagnostics. The native shell derives a stable
per-user socket directory under
/tmpfrom the current state-directory flavor, keeps that directory owned by the current user with mode0700, and records the absolute socket path in the existing state record. A preflight rejects any Darwin socket path over 103 bytes with the stable reasonsocket_path_too_long; it must not be reported asdaemon_exited. The managed daemon's stderr is drained into a bounded tail and appended to the existing shell bootstrap diagnostics. Owning tests cover path construction, flavor isolation, permissions, over-limit classification, and stderr propagation. - F2, Screen Recording guidance. English and Chinese native and Workbench guidance must state that macOS may quit and reopen Avibe after the Screen Recording grant. The two locales keep the same recovery steps and consuming catalog/UI tests protect the parity.
- F3, Workbench support on first install. Workbench support is derived
from the live native shell's lock ownership. A live shell therefore keeps
Computer Use visible when the state record is missing or invalid; the
status reader remains responsible for returning
off/never_enabledorunavailable/invalid_state_file. This preserves the Phase 1 accepted limitation that a terminal-started Runtime on the same machine may advertise the shell's native Computer Use. A per-Runtime shell-adoption signal remains future work. - F4, Runtime bootout cleanup. The service shutdown boundary reaps the
exact UI process belonging to its desktop Runtime before releasing the
service lock. Ownership is checked through the existing Runtime id and the
existing UI process identity; no pattern kill is introduced. A managed
service shutdown reaps the UI only when no managed shutdown intent was
consumed, which covers launchd bootout and external termination. Every
managed stop or restart caller retains its existing UI lifecycle ownership,
regardless of its reason; this includes Workbench service stop,
service-only and full restart, and explicit unscoped
vibe stop. The lifecycle test covers unmanaged signal-driven service exit and exact UI ownership, while the desktop Runtime stop tests retain the real process scan and rescan contract. - F5, Runtime re-adoption. When enablement persists in
needs_runtime/runtime_unavailable, the shell retains the last monitored origin and normal ticks probe the Runtime capability endpoint again. A supported response transitions through the existing lifecycle contract and launches one new driver generation, preserving the current lease, ownership, and no-replay behavior. An explicit user Runtime stop uses a separate stopped event that clears the retained origin, so the shell does not re-adopt a Runtime the user has stopped. Transient and unsupported responses do not start a driver.
Validation for this follow-up must include the owning Python, UI, and Rust tests; Ruff on changed Python; the UI build and relevant UI tests; Rust formatting, tests, and required Clippy/package/workflow checks; repository CI; an exact-head Codex review; and zero unresolved review threads. The PR is non-draft and remains unmerged until the owner authorizes merge.
Known-by-design limits remain explicit: Step 9b, one normal Workbench Claude
tool-driven desktop action, is deferred to real-machine acceptance; no Claude
run, login, credential access, GUI action, TCC mutation, release, candidate
Runtime restart, or owner Runtime restart is part of implementation. The
accepted ad-hoc macOS Screen Recording +/drag recovery remains the Phase 1
path. The optional resource-governance hermeticity and server-level Claude
allowlist follow-ups are recorded for later triage and do not expand this
change unless they remain strictly local and necessary.