Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

feat(host): run bot computer use on the bundled cua-driver

leepokai0c781596423e
diff --git a/.github/workflows/host.yml b/.github/workflows/host.yml
index c9fe879e..da1e0ac0 100644
--- a/.github/workflows/host.yml
+++ b/.github/workflows/host.yml
@@ -76,7 +76,11 @@ jobs:
NAME=codync-host-${{ matrix.name }}
mkdir $NAME
cp host/target/${{ matrix.target }}/release/codync-host $NAME/
- if [ "$RUNNER_OS" = Linux ]; then cp apps/screen-linux/target/release/codync-screen $NAME/; fi
+ if [ "$RUNNER_OS" = Linux ]; then
+ cp apps/screen-linux/target/release/codync-screen $NAME/
+ # The computer-use driver the host starts from beside itself (pinned, checked).
+ node packaging/cua-driver/fetch.mjs ${{ matrix.name }} $NAME/
+ fi
cp LICENSE $NAME/
tar czf $NAME.tar.gz $NAME
shasum -a 256 $NAME.tar.gz > $NAME.tar.gz.sha256
diff --git a/CLAUDE.md b/CLAUDE.md
index e03d6358..3f9d0404 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -32,7 +32,7 @@ Bot-based remote for coding agents: persistent named bots on your computer, mess
- **Sync**: every mutation stamps a global `rev`. Clients call `GET /events?since=<rev>` (catch-up in rev order, then live). Emission happens under `Hub::emit_lock` so events leave in rev order. Clients upsert by id; never skip undecodable events (the iOS app rewinds to rev 0).
- API: `POST /api/<method>` + SSE with the bearer token (`~/.codync/token`) is **loopback only** (desktop app, SSH tunnel, local helpers). Phones and other remote clients use the E2E channel (`/channel` direct, or the Cloudflare relay). Default port **19222**.
- Remote access (Cloudflare relay primary, direct LAN/Tailscale alternative, accounts, SSH, public routine webhooks queued in the relay §7.8): [docs/reference/remote-relay.md](docs/reference/remote-relay.md).
-- Remote screen (`host/src/screen/` is the reference): phones view/control the computer over WebRTC (hardware H.264, non-trickle SDP relayed by `screenOffer`, input on data channels `input` / `input-fast`); bots get the built-in `computer` MCP server (`codync-host mcp computer`, `host/src/mcp.rs`) when their `computer` flag is on. Capture/input live in a helper on `~/.codync/screen.sock`: `apps/screen-macos` (macOS, launchd agent the desktop app registers via `SMAppService`, owns the TCC grants) or `apps/screen-linux` (`codync-screen`: portals + GStreamer, started by the host). On by default (Linux: only with a graphical session); `setScreenEnabled` is accepted only from loopback. An interactive phone takes over (bots may only look).
+- Remote screen (`host/src/screen/` is the reference): phones view/control the computer over WebRTC (hardware H.264, non-trickle SDP relayed by `screenOffer`, input on data channels `input` / `input-fast`); bots get the built-in `computer` MCP server (`codync-host mcp computer`, `host/src/mcp.rs`) when their `computer` flag is on; its tools run on the bundled cua-driver, which Codync Screen starts on macOS so its grants apply ([computer use](docs/features/computer-use.md)). Capture/input live in a helper on `~/.codync/screen.sock`: `apps/screen-macos` (macOS, launchd agent the desktop app registers via `SMAppService`, owns the TCC grants) or `apps/screen-linux` (`codync-screen`: portals + GStreamer, started by the host). On by default (Linux: only with a graphical session); `setScreenEnabled` is accepted only from loopback. An interactive phone takes over (bots may only look).
- Multiple computers per account are future work, not a current priority: the product targets one computer. The multi-computer structure (`AccountStore`, `BotReference`) stays because relay/accounts/SSH build on it; don't extend or polish multi-computer features unless asked.
- Environments: **dev** (`dev-api.codync.dev`, Clerk development instance, Debug builds) and **main** (`api.codync.dev`, Clerk production, Release builds); `apps/shared/Config/<env>.plist` becomes the iOS `AccountConfig.plist` and the desktop `resources/account-config.json` (`tools/account-config.mjs`). Details: [spec §14.0](docs/reference/remote-relay.md).
- Push: iOS registers its APNs token with `relay/` → gets an AES-GCM ticket → gives it (plus its X25519 push key) to the host; the host seals title/body to that key and a Notification Service Extension opens it, so `relay/` sees only generic text. Alert kinds: *needs you*, *done*, and *failed*, suppressed while the iOS app is connected. Delivery and lifecycle: [notification design](docs/design/push-and-live-activity.md).
diff --git a/apps/Codync.xcodeproj/project.pbxproj b/apps/Codync.xcodeproj/project.pbxproj
index 86bd9cdd..53df6014 100644
--- a/apps/Codync.xcodeproj/project.pbxproj
+++ b/apps/Codync.xcodeproj/project.pbxproj
@@ -64,6 +64,7 @@
E78E3B5FAFEB79F31AAC104D /* CodyncBot-teardrop-Tint.ttf in Resources */ = {isa = PBXBuildFile; fileRef = C8F46BD8301E7E2C83C35111 /* CodyncBot-teardrop-Tint.ttf */; };
EBB0417A5FB2CE6D514D21BE /* ClerkKit in Frameworks */ = {isa = PBXBuildFile; productRef = D9AB2F442A5F243653EAE936 /* ClerkKit */; };
F171F16563D1FF6308697173 /* CodyncUI in Frameworks */ = {isa = PBXBuildFile; productRef = 06C9F37C6502131CAF0F145E /* CodyncUI */; };
+ F6A380E8A026CA849620F2F7 /* Driver.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8D16AD9979C77D976BC832D1 /* Driver.swift */; };
F842AE1C5ED96AF9A7F5D16C /* Widgets.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = 6D734BEE7CD3F84677B0218B /* Widgets.appex */; settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; };
F97C489DDB806224CEFCEB57 /* ScreenHelper.swift in Sources */ = {isa = PBXBuildFile; fileRef = 28B2CB561A43ED9F2F1A4391 /* ScreenHelper.swift */; };
FCE6E509AEFC88C3D05D8E3D /* CodyncBot-tablet-Tint.ttf in Resources */ = {isa = PBXBuildFile; fileRef = F7CAD5C4F330637102B40288 /* CodyncBot-tablet-Tint.ttf */; };
@@ -133,6 +134,7 @@
89A0D530BEF30AF36F9DBEA8 /* CodyncBot-hex-Ink.ttf */ = {isa = PBXFileReference; lastKnownFileType = file; path = "CodyncBot-hex-Ink.ttf"; sourceTree = "<group>"; };
8BA06CEE98A7CDED62678522 /* Input.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Input.swift; sourceTree = "<group>"; };
8CC3E47D480A5F15CA95C504 /* HostLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = HostLink.swift; sourceTree = "<group>"; };
+ 8D16AD9979C77D976BC832D1 /* Driver.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Driver.swift; sourceTree = "<group>"; };
8E783FFF5EC2697C16FA2E1B /* CodyncOrb-Ghost.ttf */ = {isa = PBXFileReference; lastKnownFileType = file; path = "CodyncOrb-Ghost.ttf"; sourceTree = "<group>"; };
9AD0BD16672E128A4F26272B /* SignInButtons.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SignInButtons.swift; sourceTree = "<group>"; };
A65FE0504B741D0B44DB2516 /* CodyncApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CodyncApp.swift; sourceTree = "<group>"; };
@@ -321,6 +323,7 @@
children = (
D4D13418ABB5FD7CFC087677 /* AXTree.swift */,
CE63C84233B14A2D63640EAA /* Capture.swift */,
+ 8D16AD9979C77D976BC832D1 /* Driver.swift */,
8CC3E47D480A5F15CA95C504 /* HostLink.swift */,
8BA06CEE98A7CDED62678522 /* Input.swift */,
28B2CB561A43ED9F2F1A4391 /* ScreenHelper.swift */,
@@ -379,6 +382,7 @@
buildPhases = (
34C567D8C7B2C02F5D48CA29 /* Sources */,
9BF4C7DD5C26BAD44BD453BF /* Frameworks */,
+ 558AFCA61717F426992F99B8 /* Embed cua-driver */,
);
buildRules = (
);
@@ -550,6 +554,25 @@
/* End PBXResourcesBuildPhase section */
/* Begin PBXShellScriptBuildPhase section */
+ 558AFCA61717F426992F99B8 /* Embed cua-driver */ = {
+ isa = PBXShellScriptBuildPhase;
+ alwaysOutOfDate = 1;
+ buildActionMask = 2147483647;
+ files = (
+ );
+ inputFileListPaths = (
+ );
+ inputPaths = (
+ );
+ name = "Embed cua-driver";
+ outputFileListPaths = (
+ );
+ outputPaths = (
+ );
+ runOnlyForDeploymentPostprocessing = 0;
+ shellPath = /bin/sh;
+ shellScript = "export PATH=\"/opt/homebrew/bin:/usr/local/bin:$PATH\"\nnode \"$SRCROOT/../packaging/cua-driver/fetch.mjs\" darwin \"$TARGET_BUILD_DIR/$CONTENTS_FOLDER_PATH/Helpers\" \"$TARGET_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH\"\n";
+ };
9900BE8858E71975C4052787 /* Codync environment */ = {
isa = PBXShellScriptBuildPhase;
alwaysOutOfDate = 1;
@@ -589,6 +612,7 @@
files = (
54CEEA0734C2940E2D9FAC7D /* AXTree.swift in Sources */,
75C4B7F344DF1E52A1EADEFC /* Capture.swift in Sources */,
+ F6A380E8A026CA849620F2F7 /* Driver.swift in Sources */,
9638BAA981354C4CE72884A0 /* HostLink.swift in Sources */,
974A5D8D89758A61E7D8D1B6 /* Input.swift in Sources */,
F97C489DDB806224CEFCEB57 /* ScreenHelper.swift in Sources */,
diff --git a/apps/desktop/electron-builder.yml b/apps/desktop/electron-builder.yml
index c6f2f52f..6bbab8e3 100644
--- a/apps/desktop/electron-builder.yml
+++ b/apps/desktop/electron-builder.yml
@@ -3,7 +3,8 @@
# build/native/codync-host the host for this platform (macOS: universal)
# build/native/codync-host.exe Windows: the host, and codync-hostw.exe that runs it at sign-in
# build/native/codync-speech macOS on-device speech helper
-# build/native/CodyncScreen.app macOS screen helper (Xcode `Screen` target)
+# build/native/CodyncScreen.app macOS screen helper (Xcode `Screen` target, with cua-driver inside)
+# build/native/cua-driver.exe Windows: the computer-use driver and its UI Automation helper
appId: com.pokai.Codync
productName: Codync
copyright: ""
@@ -39,6 +40,9 @@ mac:
# The zip (update feed) takes this name; the DMG sets its own.
artifactName: codync-macos.${ext}
hardenedRuntime: true
+ # The computer-use driver keeps its maker's Developer ID signature (packaging/cua-driver).
+ signIgnore:
+ - CodyncScreen\.app/Contents/Helpers/cua-driver$
entitlements: resources/entitlements.mac.plist
entitlementsInherit: resources/entitlements.mac.plist
extendInfo:
@@ -94,6 +98,13 @@ win:
to: codync-host.exe
- from: build/native/codync-hostw.exe
to: codync-hostw.exe
+ # The computer-use driver bots act through; the host starts it from beside itself.
+ - from: build/native/cua-driver.exe
+ to: cua-driver.exe
+ - from: build/native/cua-driver-uia.exe
+ to: cua-driver-uia.exe
+ - from: build/native/cua-driver.LICENSE
+ to: cua-driver.LICENSE
# The tray icon (template glyphs are for macOS).
- from: resources/icon.png
to: icon.png
diff --git a/apps/desktop/src/renderer/views/bots/BotEditorView.tsx b/apps/desktop/src/renderer/views/bots/BotEditorView.tsx
index 9437b8a7..1c4d23c4 100644
--- a/apps/desktop/src/renderer/views/bots/BotEditorView.tsx
+++ b/apps/desktop/src/renderer/views/bots/BotEditorView.tsx
@@ -204,7 +204,9 @@ function BotSettingsForm({ draft, onChange, error, plugins }: {
detail={
store.screen.enabled
? 'Let this bot see the screen and use the mouse and keyboard. You can watch and take over from your phone.'
- : "Let this bot see the screen and use the mouse and keyboard. Turn on Remote screen in Codync's menu on the computer first."
+ : store.screen.computerUse
+ ? 'Let this bot see the screen and use the mouse and keyboard.'
+ : "Let this bot see the screen and use the mouse and keyboard. Turn on Remote screen in Codync's menu on the computer first."
}
on={draft.computer ?? false}
onChange={(on) => set('computer', on)}
diff --git a/apps/desktop/src/shared/models.ts b/apps/desktop/src/shared/models.ts
index 797a3d67..63f1c31e 100644
--- a/apps/desktop/src/shared/models.ts
+++ b/apps/desktop/src/shared/models.ts
@@ -237,6 +237,8 @@ export interface ScreenState {
displays: ScreenDisplay[]
userControl: boolean
agentBot?: string | null
+ /** Bots with computer use can act here (Windows: always; elsewhere Remote screen is on). */
+ computerUse: boolean
viewers: number
}
@@ -251,6 +253,7 @@ export function normalizeScreen(raw: Partial<ScreenState> | undefined | null): S
displays: raw.displays ?? [],
userControl: raw.userControl ?? false,
agentBot: raw.agentBot ?? null,
+ computerUse: raw.computerUse ?? raw.enabled ?? false,
viewers: raw.viewers ?? 0,
}
}
diff --git a/apps/desktop/tools/native-host.mjs b/apps/desktop/tools/native-host.mjs
index 30226cfc..ba58c7a5 100644
--- a/apps/desktop/tools/native-host.mjs
+++ b/apps/desktop/tools/native-host.mjs
@@ -1,8 +1,9 @@
// Builds the host the app bundles for the same environment as the app: node tools/native-host.mjs
// dev|main compiles it in as CODYNC_ENV. Built on every package so a host made for another
// environment is never shipped. macOS: build/native/codync-host (universal). Windows:
-// build/native/codync-host.exe and codync-hostw.exe (runs it at sign-in). Linux apps use the
-// installed host, so there's nothing to do there.
+// build/native/codync-host.exe and codync-hostw.exe (runs it at sign-in), plus the computer-use
+// driver the host starts from beside itself. Linux apps use the installed host, so there's
+// nothing to do there.
import { copyFileSync, mkdirSync } from 'node:fs'
import { execFileSync } from 'node:child_process'
import { dirname, join } from 'node:path'
@@ -25,6 +26,7 @@ if (process.platform === 'win32') {
const target = 'x86_64-pc-windows-msvc'
build(target)
for (const exe of ['codync-host.exe', 'codync-hostw.exe']) copyFileSync(join(host, 'target', target, 'release', exe), join(out, exe))
+ run(process.execPath, [join(root, '../../packaging/cua-driver/fetch.mjs'), 'windows-x86_64', out])
} else {
const targets = ['aarch64-apple-darwin', 'x86_64-apple-darwin']
for (const target of targets) build(target)
diff --git a/apps/desktop/tools/native-screen.mjs b/apps/desktop/tools/native-screen.mjs
index 2365435d..35f9a66c 100644
--- a/apps/desktop/tools/native-screen.mjs
+++ b/apps/desktop/tools/native-screen.mjs
@@ -1,6 +1,7 @@
// Builds Codync Screen (Remote screen capture and input) into build/native/CodyncScreen.app for the
-// Mac app to bundle, unsigned (electron-builder signs it). Without it the app registers a launch
-// agent pointing at a missing helper and Remote screen never starts.
+// Mac app to bundle, unsigned (electron-builder signs it; the cua-driver the Xcode target embeds
+// keeps its own signature). Without it the app registers a launch agent pointing at a missing
+// helper and Remote screen never starts.
import { execFileSync } from 'node:child_process'
import { cpSync, mkdirSync, rmSync } from 'node:fs'
import { dirname, join } from 'node:path'
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/Models.swift b/apps/ios/Kit/Sources/CodyncKit/Models/Models.swift
index 942dc05e..7ecbd075 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Models/Models.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/Models.swift
@@ -351,6 +351,8 @@ public struct ScreenState: Codable, Equatable, Sendable {
public var userControl = false
/// The bot using the computer right now.
public var agentBot: String?
+ /// Bots with computer use can act here (Windows: always; elsewhere Remote screen is on).
+ public var computerUse = false
public var viewers = 0
public init() {}
@@ -368,6 +370,7 @@ public struct ScreenState: Codable, Equatable, Sendable {
displays = try c.decodeIfPresent([ScreenDisplay].self, forKey: .displays) ?? []
userControl = try c.decodeIfPresent(Bool.self, forKey: .userControl) ?? false
agentBot = try c.decodeIfPresent(String.self, forKey: .agentBot)
+ computerUse = try c.decodeIfPresent(Bool.self, forKey: .computerUse) ?? enabled
viewers = try c.decodeIfPresent(Int.self, forKey: .viewers) ?? 0
}
}
diff --git a/apps/ios/Kit/Sources/CodyncUI/Bots/BotEditorView.swift b/apps/ios/Kit/Sources/CodyncUI/Bots/BotEditorView.swift
index 22fd36d3..e1443755 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Bots/BotEditorView.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Bots/BotEditorView.swift
@@ -170,6 +170,8 @@ struct BotSettingsForm: View {
if model.screen != nil {
SwitchRow("Use the computer", detail: model.screen?.enabled == true
? "Let this bot see the screen and use the mouse and keyboard. You can watch and take over from your phone."
+ : model.screen?.computerUse == true
+ ? "Let this bot see the screen and use the mouse and keyboard."
: "Let this bot see the screen and use the mouse and keyboard. Turn on Remote screen in Codync's menu on the computer first.",
isOn: Binding(get: { draft.computer ?? false }, set: { draft.computer = $0 }))
}
diff --git a/apps/ios/WhatsNew.md b/apps/ios/WhatsNew.md
index 34fe0fc6..644b6e9e 100644
--- a/apps/ios/WhatsNew.md
+++ b/apps/ios/WhatsNew.md
@@ -5,16 +5,14 @@ listing what iPhone users notice. tools/asc-submit.py reads the released version
a version without one gets a generic "Bug fixes and improvements" line. Delete old sections.
-->
-## 2.9.1
+## 2.9.2
### zh-Hant
-- 修正 iPad 外接鍵盤用注音等輸入法打字時,選好的字從輸入框消失、之後無法再輸入的問題。
-- 在遠端畫面用 iPad 外接鍵盤打注音等輸入法時,現在看得到正在組字的文字和選字框。
-- 鎖定畫面和動態島縮成小圓點時,工作中的 bot 也會動起來;鎖定畫面卡片的文字不再白底白字看不清楚。
+- 開啟「使用電腦」的 bot 現在盡量在背景操作 app:直接按 app 裡的按鈕和欄位,不會搶走電腦上的滑鼠或把視窗拉到最前面。
+- Windows 電腦上的 bot 現在也能使用電腦。
### en-US
-- Fixed text typed with Zhuyin or another input method on an iPad hardware keyboard vanishing from the message box and blocking further input.
-- On the remote screen, text you're composing with Zhuyin or another input method on an iPad hardware keyboard now shows, along with its candidates.
-- While a bot works, it now moves on the Lock Screen and in the smallest Dynamic Island too, and the Lock Screen card's text no longer disappears white on white.
+- Bots with Use the computer on now work in apps in the background where they can: they press the app's own buttons and fields without taking the computer's pointer or bringing windows to the front.
+- Bots on Windows computers can now use the computer too.
diff --git a/apps/project.yml b/apps/project.yml
index bcf8099d..1bd0c00f 100644
--- a/apps/project.yml
+++ b/apps/project.yml
@@ -55,6 +55,14 @@ targets:
ENABLE_HARDENED_RUNTIME: YES
ENABLE_APP_SANDBOX: NO
SKIP_INSTALL: YES
+ postBuildScripts:
+ # The computer-use driver bots act through (host/src/screen/cua.rs), pinned and checked
+ # by packaging/cua-driver. Its own Developer ID signature stays: it is never re-signed.
+ - name: Embed cua-driver
+ basedOnDependencyAnalysis: false
+ script: |
+ export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
+ node "$SRCROOT/../packaging/cua-driver/fetch.mjs" darwin "$TARGET_BUILD_DIR/$CONTENTS_FOLDER_PATH/Helpers" "$TARGET_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH"
iOS:
type: application
platform: iOS
diff --git a/apps/screen-macos/AXTree.swift b/apps/screen-macos/AXTree.swift
index a8d83791..20bf2665 100644
--- a/apps/screen-macos/AXTree.swift
+++ b/apps/screen-macos/AXTree.swift
@@ -1,45 +1,23 @@
import AppKit
import ApplicationServices
-/// The frontmost app's accessibility tree, compact enough to hand to an agent.
-/// Frames are `[x, y, w, h]` in points relative to the display's top-left corner.
+/// What `type_login` needs to know about the field it would type into.
@MainActor
enum AXTree {
- private static let maxNodes = 600
- private static let maxDepth = 30
- /// Wrappers with nothing to say and a single child are skipped.
- private static let passThrough: Set<String> = ["AXGroup", "AXSplitGroup", "AXScrollArea", "AXLayoutArea", "AXUnknown"]
-
- static func frontmost(display: CGDirectDisplayID) throws -> [String: Any] {
- guard AXIsProcessTrusted() else {
- throw HelperError("Codync Screen isn't allowed to read the screen's controls yet. Allow it under Privacy & Security → Accessibility.")
- }
- guard let app = NSWorkspace.shared.frontmostApplication else { throw HelperError("No app is in front.") }
- let root = AXUIElementCreateApplication(app.processIdentifier)
- AXUIElementSetMessagingTimeout(root, 1)
- var budget = maxNodes
- let origin = CGDisplayBounds(display).origin
- let windows: [AXUIElement] = attribute(root, kAXWindowsAttribute) ?? []
- let tree: [String: Any] = [
- "role": "AXApplication",
- "title": app.localizedName ?? "",
- "children": windows.compactMap { node($0, depth: 0, budget: &budget, origin: origin) },
- ]
- return ["app": app.localizedName ?? "", "tree": tree, "truncated": budget <= 0]
- }
-
- /// The focused control of the frontmost app: whether it's a password field, and the
- /// URL of the web page it sits in (if any), so the host types a login only where it belongs.
- static func focusedField() throws -> [String: Any] {
+ /// The focused control of app `pid` (the frontmost app by default): whether it's a password
+ /// field, and the URL of the web page it sits in (if any), so the host types a login only
+ /// where it belongs. An app keeps its focused control while in the background.
+ static func focusedField(pid: pid_t?) throws -> [String: Any] {
guard AXIsProcessTrusted() else {
throw HelperError("Codync Screen isn't allowed to read the screen's controls yet. Allow it under Privacy & Security → Accessibility.")
}
- guard let app = NSWorkspace.shared.frontmostApplication else { throw HelperError("No app is in front.") }
+ let app = pid.flatMap { NSRunningApplication(processIdentifier: $0) } ?? NSWorkspace.shared.frontmostApplication
+ guard let app else { throw HelperError(pid == nil ? "No app is in front." : "No app is running with pid \(pid ?? 0).") }
let root = AXUIElementCreateApplication(app.processIdentifier)
AXUIElementSetMessagingTimeout(root, 1)
// Chromium only exposes web content to assistive apps that ask for it.
AXUIElementSetAttributeValue(root, "AXManualAccessibility" as CFString, kCFBooleanTrue)
- var out: [String: Any] = ["app": app.localizedName ?? "", "secure": false]
+ var out: [String: Any] = ["app": app.localizedName ?? "", "pid": Int(app.processIdentifier), "secure": false]
guard let focused: AXUIElement = attribute(root, kAXFocusedUIElementAttribute) else { return out }
let role: String? = attribute(focused, kAXRoleAttribute)
let subrole: String? = attribute(focused, kAXSubroleAttribute)
@@ -56,39 +34,6 @@ enum AXTree {
return out
}
- private static func node(_ el: AXUIElement, depth: Int, budget: inout Int, origin: CGPoint) -> [String: Any]? {
- guard budget > 0, depth < maxDepth else { return nil }
- budget -= 1
- var n: [String: Any] = [:]
- let role: String = attribute(el, kAXRoleAttribute) ?? "AXUnknown"
- n["role"] = role
- for (key, attr) in [("title", kAXTitleAttribute), ("description", kAXDescriptionAttribute), ("help", kAXHelpAttribute), ("id", kAXIdentifierAttribute)] {
- if let s: String = attribute(el, attr), !s.isEmpty { n[key] = String(s.prefix(200)) }
- }
- if let v: Any = attribute(el, kAXValueAttribute) {
- if let s = v as? String, !s.isEmpty { n["value"] = String(s.prefix(200)) } else if let num = v as? NSNumber { n["value"] = num }
- }
- if let enabled: Bool = attribute(el, kAXEnabledAttribute), !enabled { n["enabled"] = false }
- if let focused: Bool = attribute(el, kAXFocusedAttribute), focused { n["focused"] = true }
- if let frame = frame(el) {
- n["frame"] = [frame.minX - origin.x, frame.minY - origin.y, frame.width, frame.height].map { ($0 * 10).rounded() / 10 }
- }
- let kids: [AXUIElement] = attribute(el, kAXChildrenAttribute) ?? []
- let children = kids.compactMap { node($0, depth: depth + 1, budget: &budget, origin: origin) }
- if passThrough.contains(role), children.count == 1, n["title"] == nil, n["description"] == nil, n["value"] == nil {
- return children[0]
- }
- if !children.isEmpty { n["children"] = children }
- return n
- }
-
- private static func frame(_ el: AXUIElement) -> CGRect? {
- guard let pos: AXValue = attribute(el, kAXPositionAttribute), let size: AXValue = attribute(el, kAXSizeAttribute) else { return nil }
- var p = CGPoint.zero, s = CGSize.zero
- guard AXValueGetValue(pos, .cgPoint, &p), AXValueGetValue(size, .cgSize, &s) else { return nil }
- return CGRect(origin: p, size: s)
- }
-
private static func attribute<T>(_ el: AXUIElement, _ name: String) -> T? {
var value: CFTypeRef?
guard AXUIElementCopyAttributeValue(el, name as CFString, &value) == .success else { return nil }
diff --git a/apps/screen-macos/Capture.swift b/apps/screen-macos/Capture.swift
index a17cbbe9..92177b62 100644
--- a/apps/screen-macos/Capture.swift
+++ b/apps/screen-macos/Capture.swift
@@ -1,8 +1,6 @@
import CoreMedia
import Foundation
-import ImageIO
@preconcurrency import ScreenCaptureKit
-import UniformTypeIdentifiers
@preconcurrency import WebRTC
/// Hands ScreenCaptureKit frames to WebRTC. Lives on its own queue: frames never touch the main thread.
@@ -135,20 +133,4 @@ final class Capture {
let own = content.applications.filter { $0.processID == getpid() }
return SCContentFilter(display: d, excludingApplications: own, exceptingWindows: [])
}
-
- /// A still for agents: exactly `width`×`height`, JPEG, base64.
- static func screenshot(display: CGDirectDisplayID, width: Int, height: Int) async throws -> String {
- let cfg = SCStreamConfiguration()
- cfg.width = width
- cfg.height = height
- cfg.showsCursor = true
- let image = try await SCScreenshotManager.captureImage(contentFilter: try await filter(display), configuration: cfg)
- let data = NSMutableData()
- guard let dest = CGImageDestinationCreateWithData(data, UTType.jpeg.identifier as CFString, 1, nil) else {
- throw HelperError("Couldn't encode the screenshot.")
- }
- CGImageDestinationAddImage(dest, image, [kCGImageDestinationLossyCompressionQuality: 0.8] as CFDictionary)
- guard CGImageDestinationFinalize(dest) else { throw HelperError("Couldn't encode the screenshot.") }
- return (data as Data).base64EncodedString()
- }
}
diff --git a/apps/screen-macos/Driver.swift b/apps/screen-macos/Driver.swift
new file mode 100644
index 00000000..cb17f08c
--- /dev/null
+++ b/apps/screen-macos/Driver.swift
@@ -0,0 +1,76 @@
+import Foundation
+
+/// The computer-use driver bots act through (`cua-driver`, shipped in Contents/Helpers; see
+/// `host/src/screen/cua.rs`). We spawn it ourselves, never through `open` or LaunchServices, so
+/// macOS treats it as part of Codync Screen: it uses our Accessibility and Screen Recording
+/// grants and never asks for its own. It exits with us (its stdin closes).
+@MainActor
+final class Driver {
+ private var process: Process?
+ /// Never written: the driver stops when it closes, which happens when we end, however we end.
+ private var liveness: Pipe?
+
+ /// Starts the driver if it isn't running and returns its socket.
+ func socket() async throws -> String {
+ let path = Self.socketPath
+ if process?.isRunning == true { return path }
+ let exe = Bundle.main.bundleURL.appending(path: "Contents/Helpers/cua-driver")
+ guard FileManager.default.isExecutableFile(atPath: exe.path) else {
+ throw HelperError("The computer-use driver is missing from Codync Screen. Reinstall Codync.")
+ }
+ let dir = (path as NSString).deletingLastPathComponent
+ try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true, attributes: [.posixPermissions: 0o700])
+ try? FileManager.default.removeItem(atPath: path)
+
+ let p = Process()
+ p.executableURL = exe
+ p.arguments = ["serve", "--socket", path]
+ var env = ProcessInfo.processInfo.environment
+ env["CUA_DRIVER_EMBEDDED"] = "1"
+ env["CUA_DRIVER_PARENT_LIVENESS_STDIN"] = "1"
+ env["CUA_DRIVER_HOST_BUNDLE_ID"] = Bundle.main.bundleIdentifier ?? ""
+ env["CUA_DRIVER_RS_TELEMETRY_ENABLED"] = "0"
+ env["CUA_DRIVER_RS_UPDATE_CHECK"] = "0"
+ p.environment = env
+ let stdin = Pipe()
+ p.standardInput = stdin
+ p.standardOutput = FileHandle.nullDevice
+ p.standardError = FileHandle.nullDevice
+ p.terminationHandler = { [weak self] ended in
+ Task { @MainActor in if self?.process === ended { self?.process = nil } }
+ }
+ try p.run()
+ process = p
+ liveness = stdin
+ for _ in 0..<100 where p.isRunning {
+ if Self.listening(path) { return path }
+ try await Task.sleep(for: .milliseconds(150))
+ }
+ p.terminate()
+ throw HelperError("The computer-use driver didn't start.")
+ }
+
+ /// In the per-user temporary directory (private, and short enough for a socket path).
+ private static var socketPath: String {
+ NSTemporaryDirectory() + "codync-cua/cua.sock"
+ }
+
+ private static func listening(_ path: String) -> Bool {
+ let fd = Darwin.socket(AF_UNIX, SOCK_STREAM, 0)
+ guard fd >= 0 else { return false }
+ defer { close(fd) }
+ var addr = sockaddr_un()
+ addr.sun_family = sa_family_t(AF_UNIX)
+ let capacity = MemoryLayout.size(ofValue: addr.sun_path)
+ guard path.utf8.count < capacity else { return false }
+ withUnsafeMutableBytes(of: &addr.sun_path) { buf in
+ buf.copyBytes(from: path.utf8)
+ buf[path.utf8.count] = 0
+ }
+ return withUnsafePointer(to: &addr) {
+ $0.withMemoryRebound(to: sockaddr.self, capacity: 1) {
+ connect(fd, $0, socklen_t(MemoryLayout<sockaddr_un>.size)) == 0
+ }
+ }
+ }
+}
diff --git a/apps/screen-macos/ScreenHelper.swift b/apps/screen-macos/ScreenHelper.swift
index f6395175..c891062a 100644
--- a/apps/screen-macos/ScreenHelper.swift
+++ b/apps/screen-macos/ScreenHelper.swift
@@ -8,6 +8,7 @@ import CoreGraphics
final class ScreenHelper {
private let link = HostLink()
let input = InputInjector()
+ private let driver = Driver()
private var sessions: [String: Session] = [:]
private let badge = ViewingBadge()
private var lastStatus: [String: AnyHashable] = [:]
@@ -79,21 +80,10 @@ final class ScreenHelper {
case "closeAll":
closeAll()
return [String: Any]()
- case "screenshot":
- guard let w = InputInjector.number(p["width"]), let h = InputInjector.number(p["height"]) else { throw HelperError("width and height are required") }
- return ["data": try await Capture.screenshot(display: display(p["display"]), width: Int(w), height: Int(h))]
- case "input":
- guard let event = p["event"] as? [String: Any] else { throw HelperError("event is required") }
- try await input.perform(event, display: display(p["display"]))
- return [String: Any]()
- case "uiTree":
- return try AXTree.frontmost(display: display(p["display"]))
case "focusedField":
- return try AXTree.focusedField()
- case "openApp":
- guard let name = p["name"] as? String, !name.isEmpty else { throw HelperError("name is required") }
- try await openApp(name)
- return [String: Any]()
+ return try AXTree.focusedField(pid: (p["pid"] as? NSNumber).map { pid_t($0.int32Value) })
+ case "driver":
+ return ["socket": try await driver.socket()]
default:
throw HelperError("unknown method \(method)")
}
@@ -131,17 +121,6 @@ final class ScreenHelper {
(v as? NSNumber)?.uint32Value ?? CGMainDisplayID()
}
- private func openApp(_ name: String) async throws {
- let p = Process()
- p.executableURL = URL(filePath: "/usr/bin/open")
- p.arguments = ["-a", name]
- let err = Pipe()
- p.standardError = err
- try p.run()
- await withCheckedContinuation { c in p.terminationHandler = { _ in c.resume() } }
- guard p.terminationStatus == 0 else { throw HelperError("Couldn't find an app named \(name).") }
- }
-
// MARK: status
private func sendStatus() {
diff --git a/docs/architecture/file-structure.md b/docs/architecture/file-structure.md
index 89bee31c..e9306c23 100644
--- a/docs/architecture/file-structure.md
+++ b/docs/architecture/file-structure.md
@@ -12,7 +12,7 @@ Codync/
│ │ ├── hub.rs, store/ # Shared state + event fan-out / SQLite persistence (schema, model, bots, entries, devices)
│ │ ├── service.rs # Data dir, launchd/systemd install, keep-awake
│ │ ├── usage.rs, mcp.rs
-│ │ ├── screen/ # Remote screen: helper protocol, helper socket, phone viewers, bot computer use
+│ │ ├── screen/ # Remote screen: helper protocol, helper socket, phone viewers, bot computer use (cua-driver)
│ │ ├── api/ # Dispatch (mod.rs), HTTP/SSE routes (http.rs, events.rs), method groups (bots.rs, host.rs, marketplace.rs, remote.rs), caller permissions (devices.rs)
│ │ ├── agent/ # ACP client, harness discovery, registry, sign-in, setup PTYs, bot actor
│ │ ├── chat/ # Groups, bot-to-bot requests, prompt snapshots, memory
@@ -69,7 +69,7 @@ Codync/
- `apps/desktop/` — desktop app for macOS, Linux and Windows (Electron; `npm run dev`, packaged by `.github/workflows/release-desktop.yml`)
- `iOS` (`apps/ios/`) — iOS app
-- `Screen` (`apps/screen-macos/`) — Codync Screen: capture, input and WebRTC for Remote screen, embedded in the macOS desktop app (`Contents/Library/LoginItems`)
+- `Screen` (`apps/screen-macos/`) — Codync Screen: capture, input and WebRTC for Remote screen, embedded in the macOS desktop app (`Contents/Library/LoginItems`); starts the computer-use driver it carries in `Contents/Helpers`
- `apps/screen-linux/` — `codync-screen` (Rust, GStreamer + xdg portals), the Linux Remote screen helper
- `Widgets` (`apps/ios/Widgets/`) — usage widget + bot Live Activity (bundle id `com.pokai.Codync.ios.LiveActivity`)
- `CodyncKit` (`apps/ios/Kit/`) — the iPhone app's Swift package: `CodyncKit` + `CodyncUI` libraries (tests: `xcodebuild test -scheme CodyncKit-Package` on an iOS simulator)
diff --git a/docs/features/computer-use.md b/docs/features/computer-use.md
new file mode 100644
index 00000000..6d8b76e4
--- /dev/null
+++ b/docs/features/computer-use.md
@@ -0,0 +1,49 @@
+# Computer use
+
+Bots with **Use the computer** on get the built-in `computer` MCP server (`codync-host mcp computer`, `host/src/mcp.rs`). Every agent sees the same server; the host lists its tools and runs every call (`computerTools`, `computerCall`, loopback only), so Codync's rules apply whatever the harness:
+
+- One bot at a time. While a phone has taken over Remote screen, bots may only use tools that look.
+- `type_login` types a saved login only into its own site or app, and a password only into a password field. The agent never sees the value.
+- The harness's own approval rules still apply to each call.
+
+## The driver
+
+The tools come from [cua-driver](https://github.com/trycua/cua) (MIT), which works on apps in the background: by accessibility element where it can (`element_token` from `get_window_state`), by pixels posted to the app otherwise, and in the foreground only when a call asks for `delivery_mode: "foreground"`. The user's cursor and front app stay theirs.
+
+`host/src/screen/cua.rs` owns it. One daemon serves every bot, started on first use and stopped with whoever started it (`CUA_DRIVER_PARENT_LIVENESS_STDIN`: its stdin closes when the parent ends, crash included). Requests use its line protocol on a private socket (Windows: a random named pipe), with the session `codync-<bot>`. Telemetry and update checks are off.
+
+| | Who starts the driver | Permissions |
+| --- | --- | --- |
+| macOS | Codync Screen (`apps/screen-macos/Driver.swift`, `driver` request), spawned directly with `CUA_DRIVER_EMBEDDED=1` | Codync Screen's Accessibility and Screen Recording grants: macOS attributes the child to it, so the user grants nothing new and the driver never prompts |
+| Windows | The host, from beside `codync-host.exe` | None; windows running as administrator are out of reach |
+| Linux X11, Sway | The host, from beside `codync-host` (`DISPLAY` filled in when the service has none; Sway gets `CUA_DRIVER_RS_ENABLE_WAYLAND=1`) | None |
+| Linux GNOME, KDE, other Wayland | Not used: bots get Codync Screen's portal tools instead (`helper_tools.rs`: screenshot, click, type… on screenshot pixels) | The Remote screen portal grant |
+
+On macOS, windows on another Space have no accessibility tree, so background actions there are refused; the driver says so and suggests `delivery_mode: "foreground"`.
+
+Computer use needs Remote screen on where Codync Screen takes part (macOS, Linux); on Windows it is always available. The host's screen state says which (`computerUse`), and the bot editor's toggle explains it.
+
+## Tools
+
+The host offers 16 driver tools (`packaging/cua-driver/tools.mjs` lists them): `list_apps`, `list_windows`, `get_window_state`, `get_desktop_state`, `zoom`, `launch_app`, `click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`, `invoke_menu`, plus `type_login` (not on Windows yet). Left out: `bring_to_front` and `kill_app` (they take over or end the user's apps), `clipboard_read`, the `browser_*` tools (each browser asks for Automation separately), and the driver's configuration, recording and session tools.
+
+Descriptions come from the running driver, so they match the platform; when it can't start, the host lists the copy in `host/src/screen/cua_tools.json` and each call explains why it failed. Results pass through as MCP content. When the text is only a summary, the structured data follows as JSON; window snapshots add how to spell element tokens (`<snapshot_id>:<index>`).
+
+`type_login {login, field, pid?}`: Codync Screen reads the focused control of that app (`focusedField`: app, page URL, secure field), the host checks it against the login, then types through the driver into that app in the background (macOS) or into the checked active window (Linux). The driver's reply is not passed on.
+
+## Packaging
+
+`packaging/cua-driver/driver.json` pins the release and the SHA-256 of each archive; `fetch.mjs <target> <dir>` downloads, checks and copies the executables with `cua-driver.LICENSE`:
+
+- macOS: the `Screen` target's *Embed cua-driver* build phase puts it in `CodyncScreen.app/Contents/Helpers/`. electron-builder leaves its Developer ID signature alone (`mac.signIgnore`).
+- Windows: `tools/native-host.mjs` puts `cua-driver.exe` and `cua-driver-uia.exe` next to the host in `resources/`.
+- Linux: the host release archive carries `cua-driver` (needs libX11, libXi and libxkbcommon); `install.sh`, Homebrew and `codync-host update` put it next to the host.
+
+Updating the driver: change `driver.json` (version, tag, assets and sums from the release's `SHA256SUMS`), run `node packaging/cua-driver/tools.mjs <path to the new cua-driver>`, check the tool list diff, and run the checks below.
+
+## Verification
+
+- Host: `cargo test` covers the tool list, result shaping, the line protocol and takeover.
+- macOS: with a fresh build, ask a computer-use bot to read a background app's window and click a control in it. The front app must not change and macOS must not ask for any new permission. Take over from the phone mid-task: actions stop, looking still works.
+- Windows and Linux X11: the same task; Linux GNOME still gets the portal tools.
+- `type_login` into a test site's password field, and a refusal for the wrong site or a non-password field.
diff --git a/docs/features/remote-screen.md b/docs/features/remote-screen.md
index 204e21da..5c1e484f 100644
--- a/docs/features/remote-screen.md
+++ b/docs/features/remote-screen.md
@@ -30,11 +30,11 @@ computer display without choosing the phone's orientation.
## Boundaries
- `host/src/screen/` coordinates access to the local helper and owns viewer sessions. Another device cannot renegotiate or close a session it does not own.
-- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions.
+- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions, including those of the computer-use driver it starts.
- `apps/screen-linux/` implements the Linux helper using desktop portals and GStreamer, including ICE URL conversion and TURN transport configuration. The host starts it from beside its own executable (then `PATH`). Linux host releases ship `codync-screen` in the same archive (built on Ubuntu 24.04: glibc 2.39+ and GStreamer 1.22+ with the base, good and bad plugins, PipeWire and xdg-desktop-portal at run time); `install.sh`, Homebrew and `codync-host update` put it next to the host.
- Helpers communicate locally through `~/.codync/screen.sock`. SDP is non-trickle; input uses the `input` and `input-fast` data channels.
- Screen access is on by default on macOS and on Linux while a graphical session (Wayland or X11) is running; headless servers stay off. Turning it on or off is remembered. Enabling through `setScreenEnabled` requires a loopback caller. Interactive OS permission prompts must be completed on the computer.
-- Bots with their computer capability enabled receive the built-in `computer` MCP tools (`host/src/mcp.rs`). Their permission policy still applies. An interactive phone can take over; bots may still look.
+- Bots with their computer capability enabled receive the built-in `computer` MCP tools, run by the computer-use driver that Codync Screen starts on macOS ([computer use](computer-use.md)). Their permission policy still applies. An interactive phone can take over; bots may still look.
- Each device may hold four sessions; the host allows 32 total, including pending sessions. Credential issuance is limited to 12 requests per minute per account by `TURN_LIMITER` (Cloudflare's per-location rate limiter).
- Closing a session stops its media; issued TURN credentials remain usable until their one-hour TTL. Issuance limits and encoder limits do not constitute a hard monthly spending cap. Track Realtime egress by the credential's custom identifier.
diff --git a/host/src/api/devices.rs b/host/src/api/devices.rs
index cccdd263..62812281 100644
--- a/host/src/api/devices.rs
+++ b/host/src/api/devices.rs
@@ -42,6 +42,7 @@ const LOCAL_ONLY: &[&str] = &[
"setVoiceKey",
"pairing",
"computerCall",
+ "computerTools",
"chatCall",
"teamCall",
"memoryCall",
diff --git a/host/src/api/host.rs b/host/src/api/host.rs
index 36e71a40..e1100428 100644
--- a/host/src/api/host.rs
+++ b/host/src/api/host.rs
@@ -180,10 +180,10 @@ pub(super) async fn call(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value
hub.screen.set_enabled(&hub.store, b["enabled"].as_bool().unwrap_or(false)).await?;
hub.screen.state()
}
+ "computerTools" => crate::screen::computer_tools(hub).await,
"computerCall" => {
- let tool: crate::screen::ComputerTool =
- serde_json::from_value(b.clone()).context("invalid computer tool call")?;
- json!({"content": crate::screen::computer(hub, str_arg(&b, "botId")?, tool).await?})
+ let args = b.get("arguments").filter(|a| a.is_object()).cloned().unwrap_or_else(|| json!({}));
+ crate::screen::computer(hub, str_arg(&b, "botId")?, str_arg(&b, "name")?, args).await?
}
_ => return Ok(ControlFlow::Continue(b)),
}))
diff --git a/host/src/api/mod.rs b/host/src/api/mod.rs
index eb9e29af..4144f2de 100644
--- a/host/src/api/mod.rs
+++ b/host/src/api/mod.rs
@@ -70,7 +70,8 @@ async fn route(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value) -> Resul
|| method.starts_with("composio")
|| method == "setComposioKey"
|| method.starts_with("credential")
- || method == "computerCall"
+ // Only signing in reads the vault; other computer tools work without a keyring.
+ || (method == "computerCall" && b["name"] == "type_login")
|| method.starts_with("voice")
|| method == "setVoiceKey"
|| matches!(method, "agentAuth" | "agentAuthenticate" | "setAgentEnv")
diff --git a/host/src/mcp.rs b/host/src/mcp.rs
index 7c1e3fbd..b6f33584 100644
--- a/host/src/mcp.rs
+++ b/host/src/mcp.rs
@@ -20,99 +20,11 @@ pub enum Server {
Composio,
}
-const INSTRUCTIONS: &str = "Operate this computer's desktop like a person would. Start with `screenshot` \
-(or `ui_tree` to find controls precisely), then act; every action returns a fresh screenshot. \
-Coordinates are pixels in the latest screenshot. Prefer keyboard shortcuts and `open_app` over hunting with the mouse. \
-Never type a password yourself: use type_login with a saved login (request_login asks the user for one). Don't approve payments: ask the user to do that from their phone.";
-
-fn tools() -> Value {
- let display = json!({"type": "integer", "description": "Display id; defaults to the main display."});
- let xy = |what: &str| json!({"type": "number", "description": format!("{what} in screenshot pixels.")});
- json!([
- {
- "name": "screenshot",
- "description": "Capture the screen. Returns a JPEG whose pixels are the coordinate space for every other tool.",
- "inputSchema": {"type": "object", "properties": {"display": display}},
- "annotations": {"readOnlyHint": true},
- },
- {
- "name": "ui_tree",
- "description": "Accessibility tree of the frontmost app: roles, titles, values and frames in screenshot pixels. Cheaper and more precise than reading pixels when you need to find a control.",
- "inputSchema": {"type": "object", "properties": {"display": display}},
- "annotations": {"readOnlyHint": true},
- },
- {
- "name": "click",
- "description": "Click at a point. Returns a screenshot afterwards.",
- "inputSchema": {
- "type": "object",
- "properties": {
- "x": xy("X"), "y": xy("Y"),
- "button": {"type": "string", "enum": ["left", "right", "middle"], "default": "left"},
- "count": {"type": "integer", "minimum": 1, "maximum": 3, "default": 1, "description": "2 = double-click, 3 = triple-click."},
- "modifiers": {"type": "array", "items": {"type": "string", "enum": ["cmd", "option", "ctrl", "shift"]}, "description": "Keys held during the click."},
- "display": display,
- },
- "required": ["x", "y"],
- },
- },
- {
- "name": "move",
- "description": "Move the pointer (for hover menus and tooltips). Returns a screenshot afterwards.",
- "inputSchema": {"type": "object", "properties": {"x": xy("X"), "y": xy("Y"), "display": display}, "required": ["x", "y"]},
- },
- {
- "name": "drag",
- "description": "Press at (x, y), drag to (to_x, to_y), release. Returns a screenshot afterwards.",
- "inputSchema": {
- "type": "object",
- "properties": {"x": xy("Start x"), "y": xy("Start y"), "to_x": xy("End x"), "to_y": xy("End y"), "display": display},
- "required": ["x", "y", "to_x", "to_y"],
- },
- },
- {
- "name": "scroll",
- "description": "Scroll with the pointer over (x, y). Returns a screenshot afterwards.",
- "inputSchema": {
- "type": "object",
- "properties": {
- "x": xy("X"), "y": xy("Y"),
- "dx": {"type": "number", "description": "Lines to scroll right (negative = left)."},
- "dy": {"type": "number", "description": "Lines to scroll down (negative = up)."},
- "display": display,
- },
- "required": ["x", "y"],
- },
- },
- {
- "name": "type",
- "description": "Type text into the focused field (any Unicode). Returns a screenshot afterwards.",
- "inputSchema": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]},
- },
- {
- "name": "key",
- "description": "Press a key or shortcut, e.g. `return`, `escape`, `tab`, `cmd+t`, `cmd+shift+4`, `ctrl+c`. On Linux `cmd` is Super. Returns a screenshot afterwards.",
- "inputSchema": {"type": "object", "properties": {"keys": {"type": "string"}}, "required": ["keys"]},
- },
- {
- "name": "type_login",
- "description": "Type a saved login (see list_logins / request_login) into the focused field. Works only while the login's website or app is in front; `password` only into a password field. You never see the value. Returns a screenshot afterwards.",
- "inputSchema": {
- "type": "object",
- "properties": {
- "login": {"type": "string", "description": "The login's id."},
- "field": {"type": "string", "enum": ["username", "password"]},
- },
- "required": ["login", "field"],
- },
- },
- {
- "name": "open_app",
- "description": "Open or bring an application to the front by name (e.g. `Safari`, `Simulator`, `firefox`). Returns a screenshot afterwards.",
- "inputSchema": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]},
- },
- ])
-}
+const INSTRUCTIONS: &str = "Operate this computer's apps like a person would, in the background where you can: \
+find the window (list_windows, list_apps, launch_app), look at it (get_window_state: its controls and a screenshot), \
+act on a control by its element_token, then look again. Use pixel coordinates only for what has no control. \
+Never type a password yourself: use type_login with a saved login (request_login asks the user for one). \
+Don't approve payments: ask the user to do that from their phone.";
/// Serves MCP on stdio until the agent closes it.
pub async fn serve(bot: String, port: u16, server: Server) -> Result<()> {
@@ -124,7 +36,8 @@ pub async fn serve(bot: String, port: u16, server: Server) -> Result<()> {
Server::Connectors => {
("codync-connectors", crate::market::requests::INSTRUCTIONS, crate::market::requests::tools())
}
- Server::Computer => ("codync-computer", INSTRUCTIONS, tools()),
+ // Listed by the host: they depend on this computer's driver.
+ Server::Computer => ("codync-computer", INSTRUCTIONS, Value::Null),
Server::Routines => ("codync-routines", crate::routines::INSTRUCTIONS, crate::routines::tools()),
Server::Chat => ("codync-chat", crate::chat::outbox::INSTRUCTIONS, crate::chat::outbox::tools()),
Server::Team => ("codync-team", crate::chat::team::INSTRUCTIONS, crate::chat::team::tools()),
@@ -147,7 +60,10 @@ pub async fn serve(bot: String, port: u16, server: Server) -> Result<()> {
"instructions": instructions,
})),
"ping" => Ok(json!({})),
- "tools/list" => Ok(json!({"tools": available_tools})),
+ "tools/list" => Ok(match server {
+ Server::Computer => computer_tools(port, &token).await,
+ _ => json!({"tools": available_tools}),
+ }),
"tools/call" => Ok(call(port, &token, &bot, params, server).await),
method => Err(json!({"code": -32601, "message": format!("unknown method {method}")})),
};
@@ -163,6 +79,21 @@ pub async fn serve(bot: String, port: u16, server: Server) -> Result<()> {
Ok(())
}
+/// The `computer` tools the host offers; none when it can't be reached.
+async fn computer_tools(port: u16, token: &str) -> Value {
+ let res = crate::http()
+ .post(format!("http://127.0.0.1:{port}/api/computerTools"))
+ .bearer_auth(token)
+ .json(&json!({}))
+ .timeout(Duration::from_secs(30))
+ .send()
+ .await;
+ match res {
+ Ok(r) if r.status().is_success() => r.json::<Value>().await.unwrap_or_else(|_| json!({"tools": []})),
+ _ => json!({"tools": []}),
+ }
+}
+
/// One tool call through the host. Failures are tool errors the agent can read, not protocol errors.
async fn call(port: u16, token: &str, bot: &str, params: &Value, server: Server) -> Value {
let body = json!({
@@ -192,6 +123,7 @@ async fn call(port: u16, token: &str, bot: &str, params: &Value, server: Server)
match r.json::<Value>().await {
Ok(v) if ok => {
return match server {
+ Server::Computer if v["isError"] == true => json!({"content": v["content"], "isError": true}),
Server::Computer => json!({"content": v["content"]}),
Server::Chat | Server::Team | Server::Memory | Server::Routines | Server::Connectors => {
json!({"content": [{"type": "text", "text": v.to_string()}]})
@@ -531,25 +463,4 @@ mod tests {
assert_eq!(sse_data("data:{\"a\":\ndata: 2}\n\n").unwrap()["a"], 2);
assert!(sse_data(": ping\n\n").is_none());
}
-
- #[test]
- fn every_tool_parses_as_a_computer_tool() {
- for t in tools().as_array().unwrap() {
- let name = t["name"].as_str().unwrap();
- let required = t["inputSchema"]["required"].as_array().cloned().unwrap_or_default();
- let mut args = json!({});
- for r in required {
- let k = r.as_str().unwrap();
- let prop = &t["inputSchema"]["properties"][k];
- args[k] = match prop["enum"].get(0) {
- Some(first) => first.clone(),
- None if prop["type"] == "string" => json!("x"),
- None => json!(1),
- };
- }
- let parsed =
- serde_json::from_value::<crate::screen::ComputerTool>(json!({"name": name, "arguments": args}));
- assert!(parsed.is_ok(), "{name}: {parsed:?}");
- }
- }
}
diff --git a/host/src/remote/channel.rs b/host/src/remote/channel.rs
index 4259290d..dcbea32a 100644
--- a/host/src/remote/channel.rs
+++ b/host/src/remote/channel.rs
@@ -760,6 +760,7 @@ mod tests {
"revokeDevice",
"claimSign",
"computerCall",
+ "computerTools",
] {
assert_eq!(p.call(2, m, json!({})).await["err"]["status"], 403, "{m}");
}
diff --git a/host/src/screen/computer.rs b/host/src/screen/computer.rs
index 88e6f1a8..ef435730 100644
--- a/host/src/screen/computer.rs
+++ b/host/src/screen/computer.rs
@@ -1,88 +1,44 @@
-//! Bots operating the computer: the `computer` MCP tools and who is in control.
+//! Bots operating the computer: the `computer` MCP tools, who is in control, and signing in.
+//! Tools run on the computer-use driver (`cua.rs`); on Linux desktops it doesn't support, on
+//! Codync Screen's portal input (`helper_tools.rs`).
-use super::helper::Link;
-use super::protocol::{Button, Display, InputEvent, parse_keys, parse_modifiers};
-use super::{AGENT_HOLD, SETTLE, Screen};
+#[cfg(target_os = "linux")]
+use super::helper_tools;
+use super::{AGENT_HOLD, Screen, cua};
use crate::LockExt;
use crate::hub::Hub;
use anyhow::{Result, anyhow, bail};
-use serde::Deserialize;
use serde_json::{Value, json};
use std::sync::Arc;
-use std::time::{Duration, Instant};
+use std::time::Instant;
-/// A `computer` MCP tool call, as the MCP server forwards it (`{name, arguments}`).
-#[derive(Debug, Deserialize)]
-#[serde(tag = "name", content = "arguments", rename_all = "snake_case")]
-pub enum ComputerTool {
- Screenshot {
- display: Option<u32>,
- },
- Click {
- x: f64,
- y: f64,
- #[serde(default)]
- button: Button,
- count: Option<u8>,
- #[serde(default)]
- modifiers: Vec<String>,
- display: Option<u32>,
- },
- Move {
- x: f64,
- y: f64,
- display: Option<u32>,
- },
- Drag {
- x: f64,
- y: f64,
- to_x: f64,
- to_y: f64,
- display: Option<u32>,
- },
- Scroll {
- x: f64,
- y: f64,
- #[serde(default)]
- dx: f64,
- #[serde(default)]
- dy: f64,
- display: Option<u32>,
- },
- Type {
- text: String,
- },
- Key {
- keys: String,
- },
- UiTree {
- display: Option<u32>,
- },
- OpenApp {
- name: String,
- },
- TypeLogin {
- login: String,
- field: LoginField,
- },
+const TYPE_LOGIN: &str = "type_login";
+
+/// What runs the tools on this computer.
+#[derive(Clone, Copy)]
+enum Backend {
+ Driver,
+ #[cfg(target_os = "linux")]
+ Helper,
}
-/// Which half of a saved login `type_login` types.
-#[derive(Clone, Copy, Debug, PartialEq, Eq, Deserialize)]
-#[serde(rename_all = "lowercase")]
-pub enum LoginField {
- Username,
- Password,
+#[cfg(not(target_os = "linux"))]
+fn backend() -> Backend {
+ Backend::Driver
}
-impl ComputerTool {
- /// Tools that only look: allowed while the user has control.
- fn read_only(&self) -> bool {
- matches!(self, Self::Screenshot { .. } | Self::UiTree { .. })
- }
+#[cfg(target_os = "linux")]
+fn backend() -> Backend {
+ if cua::linux_session() && cua::executable().is_some() { Backend::Driver } else { Backend::Helper }
}
impl Screen {
+ /// Whether bots may use the computer here: Windows needs nothing else; macOS and Linux run
+ /// their part through Codync Screen, which runs while Remote screen is on.
+ pub(super) fn computer_use(&self) -> bool {
+ cfg!(windows) || self.enabled()
+ }
+
/// Marks `bot` as the one using the computer. Returns whether that's news.
fn claim(&self, bot: &str, read_only: bool) -> Result<bool> {
let mut c = self.control.locked();
@@ -102,15 +58,39 @@ impl Screen {
}
}
-/// Runs one `computer` tool for `bot`: MCP `CallToolResult` content.
-pub async fn computer(hub: &Arc<Hub>, bot: &str, tool: ComputerTool) -> Result<Value> {
+/// The `computer` MCP tools on this computer.
+pub async fn computer_tools(hub: &Arc<Hub>) -> Value {
+ let screen = &hub.screen;
+ let mut tools = match backend() {
+ Backend::Driver => cua::tools(screen).await,
+ #[cfg(target_os = "linux")]
+ Backend::Helper => helper_tools::tools(),
+ };
+ if cfg!(not(windows)) {
+ tools.push(type_login_tool());
+ }
+ json!({"tools": tools})
+}
+
+/// Runs one `computer` tool for `bot`: `{content, isError?}` for the agent.
+pub async fn computer(hub: &Arc<Hub>, bot: &str, name: &str, args: Value) -> Result<Value> {
let row = hub.store.bot(bot)?.filter(|r| !r.deleted).ok_or_else(|| anyhow!("unknown bot"))?;
if !row.config.computer {
bail!("Computer use is turned off for this bot.");
}
let screen = &hub.screen;
- let link = screen.link()?;
- if screen.claim(bot, tool.read_only())? {
+ if !screen.computer_use() {
+ bail!("Remote screen is turned off on this computer. Turn it on in Codync's menu there.");
+ }
+ let backend = backend();
+ let read_only = match backend {
+ _ if name == TYPE_LOGIN => false,
+ Backend::Driver if !cua::offered(name) => bail!("unknown computer tool `{name}`"),
+ Backend::Driver => cua::read_only(name),
+ #[cfg(target_os = "linux")]
+ Backend::Helper => helper_tools::read_only(name),
+ };
+ if screen.claim(bot, read_only)? {
screen.emit();
let screen = screen.clone();
tokio::spawn(async move {
@@ -124,119 +104,96 @@ pub async fn computer(hub: &Arc<Hub>, bot: &str, tool: ComputerTool) -> Result<V
}
});
}
- let display = |id| screen.display(id);
- let act = |id: Option<u32>, event: InputEvent| {
- let link = link.clone();
- async move {
- let d = display(id)?;
- link.request("input", json!({"display": d.id, "event": event})).await?;
- tokio::time::sleep(SETTLE).await;
- shot(&link, &d).await
- }
- };
- match tool {
- ComputerTool::Screenshot { display: id } => shot(&link, &display(id)?).await,
- ComputerTool::Click { x, y, button, count, modifiers, display: id } => {
- let d = display(id)?;
- let (x, y) = d.to_points(x, y)?;
- let modifiers = parse_modifiers(&modifiers)?;
- let count = count.unwrap_or(1).clamp(1, 3);
- act(Some(d.id), InputEvent::Click { x, y, button, count, modifiers }).await
- }
- ComputerTool::Move { x, y, display: id } => {
- let d = display(id)?;
- let (x, y) = d.to_points(x, y)?;
- act(Some(d.id), InputEvent::Move { x, y }).await
- }
- ComputerTool::Drag { x, y, to_x, to_y, display: id } => {
- let d = display(id)?;
- let (x, y) = d.to_points(x, y)?;
- let (to_x, to_y) = d.to_points(to_x, to_y)?;
- act(Some(d.id), InputEvent::Drag { x, y, to_x, to_y }).await
- }
- ComputerTool::Scroll { x, y, dx, dy, display: id } => {
- let d = display(id)?;
- let (x, y) = d.to_points(x, y)?;
- act(Some(d.id), InputEvent::Scroll { x, y, dx, dy }).await
- }
- ComputerTool::Type { text } => act(None, InputEvent::Text { text }).await,
- ComputerTool::Key { keys } => {
- let (key, modifiers) = parse_keys(&keys)?;
- act(None, InputEvent::Key { key, modifiers }).await
- }
- ComputerTool::UiTree { display: id } => {
- let d = display(id)?;
- let mut res = link.request("uiTree", json!({"display": d.id})).await?;
- scale_frames(&mut res["tree"], d.shot_scale());
- let text = format!(
- "Accessibility tree of {} (frames are [x, y, width, height] in screenshot pixels):\n{}",
- res["app"].as_str().unwrap_or("the frontmost app"),
- res["tree"]
- );
- Ok(json!([{"type": "text", "text": text}]))
- }
- ComputerTool::TypeLogin { login, field } => {
- let login = crate::market::logins::get(&hub.store, &login)?;
- // Type only into the login's own site, and a password only into a password field,
- // so a misled bot can't paste it into a page or chat that would show it.
- let focus = link.request("focusedField", json!({})).await?;
- let app = focus["app"].as_str().unwrap_or_default();
- let url = focus["url"].as_str();
- if !crate::market::logins::matches(&login.site, url, app) {
- bail!(
- "The focused window is {}, not {}. Open {} and focus its sign-in field first.",
- url.unwrap_or(app),
- login.site,
- login.site
- );
- }
- let text = match field {
- LoginField::Username => login.username,
- LoginField::Password => {
- if focus["secure"] != true {
- bail!(
- "Click into the password field first; type_login types a password only into a password field."
- );
- }
- crate::market::passwords::resolve(&hub.store, &login.password).await?
- }
- };
- act(None, InputEvent::Text { text }).await
- }
- ComputerTool::OpenApp { name } => {
- link.request("openApp", json!({"name": name})).await?;
- tokio::time::sleep(Duration::from_secs(1)).await;
- shot(&link, &display(None)?).await
- }
+ if name == TYPE_LOGIN {
+ return type_login(hub, bot, backend, args).await;
+ }
+ match backend {
+ Backend::Driver => cua::call(screen, bot, name, args).await,
+ #[cfg(target_os = "linux")]
+ Backend::Helper => helper_tools::run(screen, name, args).await,
}
}
-async fn shot(link: &Link, d: &Display) -> Result<Value> {
- let (width, height) = d.shot_size();
- let res = link.request("screenshot", json!({"display": d.id, "width": width, "height": height})).await?;
- let data = res["data"].as_str().ok_or_else(|| anyhow!("the screen helper sent no image"))?;
- Ok(json!([
- {"type": "image", "data": data, "mimeType": "image/jpeg"},
- {"type": "text", "text": format!(
- "Display {} \"{}\", {width}×{height}. Coordinates for click/move/drag/scroll are pixels in this image.",
- d.id, d.name
- )},
- ]))
+#[derive(serde::Deserialize)]
+#[serde(rename_all = "lowercase")]
+enum LoginField {
+ Username,
+ Password,
}
-/// Display points → screenshot pixels, for every `frame` in an accessibility tree.
-fn scale_frames(node: &mut Value, s: f64) {
- if let Some(frame) = node.get_mut("frame").and_then(Value::as_array_mut) {
- for v in frame.iter_mut() {
- if let Some(f) = v.as_f64() {
- *v = json!((f * s).round());
+#[derive(serde::Deserialize)]
+struct LoginArgs {
+ login: String,
+ field: LoginField,
+ pid: Option<i64>,
+}
+
+fn type_login_tool() -> Value {
+ json!({
+ "name": TYPE_LOGIN,
+ "description": "Type a saved login (see list_logins / request_login) into an app's focused sign-in field: click the field first, then pass that app's `pid`. Works only when the field belongs to the login's website or app; `password` only into a password field. You never see the value.",
+ "inputSchema": {
+ "type": "object",
+ "properties": {
+ "login": {"type": "string", "description": "The login's id."},
+ "field": {"type": "string", "enum": ["username", "password"]},
+ "pid": {"type": "integer", "description": "The app with the sign-in field (from list_windows); the frontmost app when left out."},
+ },
+ "required": ["login", "field"],
+ },
+ })
+}
+
+/// Types a saved login only into its own site or app, and a password only into a password
+/// field, so a misled bot can't paste it into a page or chat that would show it. Codync Screen
+/// checks the focused field (accessibility): which app, which web page, whether it's secure.
+/// Windows has no Codync Screen to check it yet.
+async fn type_login(hub: &Arc<Hub>, bot: &str, backend: Backend, args: Value) -> Result<Value> {
+ if cfg!(windows) {
+ bail!("type_login isn't available on Windows yet. Ask the user to sign in themselves.");
+ }
+ let a: LoginArgs = serde_json::from_value(args).map_err(|e| anyhow!("invalid type_login call: {e}"))?;
+ let login = crate::market::logins::get(&hub.store, &a.login)?;
+ let screen = &hub.screen;
+ let focus = screen.link()?.request("focusedField", json!({"pid": a.pid})).await?;
+ let app = focus["app"].as_str().unwrap_or_default();
+ let url = focus["url"].as_str();
+ if !crate::market::logins::matches(&login.site, url, app) {
+ bail!(
+ "The focused field is in {}, not {}. Open {} and click its sign-in field first.",
+ url.unwrap_or(app),
+ login.site,
+ login.site
+ );
+ }
+ let text = match a.field {
+ LoginField::Username => login.username,
+ LoginField::Password => {
+ if focus["secure"] != true {
+ bail!("Click into the password field first; type_login types a password only into a password field.");
}
+ crate::market::passwords::resolve(&hub.store, &login.password).await?
}
- }
- if let Some(children) = node.get_mut("children").and_then(Value::as_array_mut) {
- for c in children {
- scale_frames(c, s);
+ };
+ match backend {
+ Backend::Driver => {
+ // macOS: into the checked app, in the background; Linux: the active window it checked.
+ let args = if cfg!(target_os = "macos") {
+ json!({"pid": focus["pid"], "text": text})
+ } else {
+ json!({"scope": "desktop", "text": text})
+ };
+ // The driver's answer may quote what it typed: the agent gets only the outcome.
+ let res = cua::call(screen, bot, "type_text", args).await?;
+ if res["isError"] == true {
+ bail!("Couldn't type into {app}'s field. Click it again and retry.");
+ }
+ Ok(
+ json!({"content": [{"type": "text", "text": format!("Typed the saved value into {app}. Take a fresh look to check it landed.")}]}),
+ )
}
+ #[cfg(target_os = "linux")]
+ Backend::Helper => helper_tools::type_text(screen, text).await,
}
}
@@ -245,15 +202,6 @@ mod tests {
use super::*;
use tokio::sync::broadcast;
- #[test]
- fn tools_parse_from_mcp_calls() {
- let t: ComputerTool =
- serde_json::from_value(json!({"name": "click", "arguments": {"x": 1, "y": 2, "button": "right"}})).unwrap();
- assert!(matches!(t, ComputerTool::Click { button: Button::Right, count: None, .. }));
- let t: ComputerTool = serde_json::from_value(json!({"name": "screenshot", "arguments": {}})).unwrap();
- assert!(t.read_only());
- }
-
#[test]
fn user_takeover_blocks_bot_actions_but_not_looking() {
let (tx, _) = broadcast::channel(8);
@@ -267,10 +215,9 @@ mod tests {
}
#[test]
- fn frames_scale_recursively() {
- let mut t = json!({"frame": [100, 50, 200, 20], "children": [{"frame": [10.0, 10.0, 5.0, 5.0]}]});
- scale_frames(&mut t, 0.5);
- assert_eq!(t["frame"], json!([50.0, 25.0, 100.0, 10.0]));
- assert_eq!(t["children"][0]["frame"], json!([5.0, 5.0, 3.0, 3.0]));
+ fn type_login_names_its_target() {
+ let t = type_login_tool();
+ assert_eq!(t["inputSchema"]["required"], json!(["login", "field"]));
+ assert!(t["inputSchema"]["properties"]["pid"].is_object());
}
}
diff --git a/host/src/screen/cua.rs b/host/src/screen/cua.rs
new file mode 100644
index 00000000..b81116b8
--- /dev/null
+++ b/host/src/screen/cua.rs
@@ -0,0 +1,297 @@
+//! The computer-use driver bots act through: cua-driver (MIT, <https://github.com/trycua/cua>),
+//! pinned in `packaging/cua-driver/driver.json` and shipped beside the host (macOS: inside
+//! Codync Screen). It works on apps in the background, by accessibility element where it can.
+//!
+//! One daemon serves every bot, started on first use and stopped with whoever started it. On
+//! macOS Codync Screen starts it (`driver` request), so macOS counts its Accessibility and
+//! Screen Recording use as Codync Screen's and the grants the user gave stay the only ones; on
+//! Windows and Linux the host starts it. Requests use the daemon's line protocol: one JSON object
+//! per line, `{method: "call", name, args, session_id, client_kind}` → `{ok, result | error}`.
+
+use super::Screen;
+use anyhow::{Context, Result, anyhow, bail};
+use serde_json::{Value, json};
+use std::sync::LazyLock;
+use std::time::Duration;
+use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWrite, AsyncWriteExt, BufReader};
+
+/// The tools bots get (`packaging/cua-driver/tools.mjs`), listed as-is when the driver can't start.
+static TOOLS: LazyLock<Vec<Value>> =
+ LazyLock::new(|| serde_json::from_str(include_str!("cua_tools.json")).expect("cua_tools.json is valid JSON"));
+/// The longest one call may take: the MCP bridge gives up at 60 s.
+const CALL_TIMEOUT: Duration = Duration::from_secs(55);
+/// Content text shorter than this is a summary; the data is then only in `structuredContent`.
+const SUMMARY_LEN: usize = 400;
+
+pub(super) fn offered(name: &str) -> bool {
+ TOOLS.iter().any(|t| t["name"] == name)
+}
+
+pub(super) fn read_only(name: &str) -> bool {
+ TOOLS.iter().any(|t| t["name"] == name && t["annotations"]["readOnlyHint"] == true)
+}
+
+/// The daemon's own descriptions for this platform, for the tools Codync offers.
+pub(super) async fn tools(screen: &Screen) -> Vec<Value> {
+ let live = async {
+ let list =
+ request(&screen.driver.endpoint(screen).await?, json!({"method": "list", "client_kind": "cli"})).await?;
+ let tools = list["tools"].as_array().context("the driver listed no tools")?;
+ anyhow::Ok(
+ tools
+ .iter()
+ .filter(|t| t["name"].as_str().is_some_and(offered))
+ .map(|t| {
+ json!({
+ "name": t["name"],
+ "description": t["description"],
+ "inputSchema": t["input_schema"],
+ "annotations": {
+ "readOnlyHint": t["read_only"],
+ "destructiveHint": t["destructive"],
+ "idempotentHint": t["idempotent"],
+ "openWorldHint": t["open_world"],
+ },
+ })
+ })
+ .collect::<Vec<_>>(),
+ )
+ };
+ match live.await {
+ Ok(tools) if !tools.is_empty() => tools,
+ Ok(_) => TOOLS.clone(),
+ Err(e) => {
+ tracing::debug!(error = format!("{e:#}"), "listing the stored driver tools");
+ TOOLS.clone()
+ }
+ }
+}
+
+/// Runs one driver tool for `bot`: `{content, isError?}` for the agent.
+pub(super) async fn call(screen: &Screen, bot: &str, name: &str, args: Value) -> Result<Value> {
+ let body = json!({"method": "call", "name": name, "args": args, "session_id": format!("codync-{bot}"), "client_kind": "cli"});
+ let mut result = request(&screen.driver.endpoint(screen).await?, body).await?;
+ let mut out = json!({"content": agent_content(&mut result)});
+ if result["isError"] == true {
+ out["isError"] = true.into();
+ }
+ Ok(out)
+}
+
+/// The tool result as the agent reads it. Most tools put their data in `structuredContent` and
+/// only a summary in `content`, which some agents never look past, so a short summary gets the
+/// data as text. Window snapshots spell out their element tokens (`<snapshot_id>:<index>`).
+fn agent_content(result: &mut Value) -> Value {
+ let mut content = match result["content"].take() {
+ Value::Array(c) => c,
+ _ => vec![],
+ };
+ let structured = &result["structuredContent"];
+ let text_len: usize = content.iter().filter_map(|c| c["text"].as_str()).map(str::len).sum();
+ if let Some(id) = structured["snapshot_id"].as_str() {
+ let line = format!("element_token for element [N] in this snapshot: {id}:N");
+ content.push(json!({"type": "text", "text": line}));
+ } else if text_len < SUMMARY_LEN && structured.as_object().is_some_and(|s| !s.is_empty()) {
+ content.push(json!({"type": "text", "text": structured.to_string()}));
+ }
+ Value::Array(content)
+}
+
+/// One request on the daemon's line protocol.
+async fn request(endpoint: &str, body: Value) -> Result<Value> {
+ let exchange = async {
+ #[cfg(unix)]
+ let stream = tokio::net::UnixStream::connect(endpoint).await?;
+ #[cfg(windows)]
+ let stream = tokio::net::windows::named_pipe::ClientOptions::new().open(endpoint)?;
+ exchange(stream, &body).await
+ };
+ let reply = tokio::time::timeout(CALL_TIMEOUT, exchange)
+ .await
+ .map_err(|_| anyhow!("the computer-use driver didn't answer in time"))?
+ .context("can't reach the computer-use driver")?;
+ if reply["ok"] == true {
+ Ok(reply["result"].clone())
+ } else {
+ bail!("{}", reply["error"].as_str().unwrap_or("the computer-use driver refused the request"))
+ }
+}
+
+async fn exchange(stream: impl AsyncRead + AsyncWrite + Unpin, body: &Value) -> Result<Value> {
+ let mut stream = BufReader::new(stream);
+ let mut line = serde_json::to_vec(body)?;
+ line.push(b'\n');
+ stream.get_mut().write_all(&line).await?;
+ let mut reply = String::new();
+ stream.read_line(&mut reply).await?;
+ serde_json::from_str(&reply).context("the computer-use driver sent no JSON")
+}
+
+/// Starts the daemon when needed and says where it listens.
+#[derive(Default)]
+pub(super) struct Driver {
+ #[cfg(not(target_os = "macos"))]
+ daemon: tokio::sync::Mutex<Option<(tokio::process::Child, String)>>,
+}
+
+impl Driver {
+ /// macOS: Codync Screen runs the daemon, so its grants are the ones that count.
+ #[cfg(target_os = "macos")]
+ async fn endpoint(&self, screen: &Screen) -> Result<String> {
+ let res = screen.link()?.request("driver", json!({})).await?;
+ res["socket"].as_str().map(str::to_owned).context("Codync Screen didn't start the computer-use driver")
+ }
+
+ #[cfg(not(target_os = "macos"))]
+ async fn endpoint(&self, _screen: &Screen) -> Result<String> {
+ let mut daemon = self.daemon.lock().await;
+ if let Some((child, endpoint)) = daemon.as_mut()
+ && matches!(child.try_wait(), Ok(None))
+ {
+ return Ok(endpoint.clone());
+ }
+ let started = start().await?;
+ let endpoint = started.1.clone();
+ *daemon = Some(started);
+ Ok(endpoint)
+ }
+}
+
+/// The driver installed beside the host (`packaging/cua-driver/fetch.mjs`).
+#[cfg(not(target_os = "macos"))]
+pub(super) fn executable() -> Option<std::path::PathBuf> {
+ let name = if cfg!(windows) { "cua-driver.exe" } else { "cua-driver" };
+ std::env::current_exe().ok()?.parent().map(|dir| dir.join(name)).filter(|p| p.is_file())
+}
+
+/// Linux sessions the driver works on in the background: X11 and Sway. Elsewhere (GNOME, KDE
+/// and other Wayland desktops) bots use Codync Screen's portal input instead.
+#[cfg(target_os = "linux")]
+pub(super) fn linux_session() -> bool {
+ let wayland = std::env::var_os("WAYLAND_DISPLAY").is_some()
+ || std::env::var("XDG_SESSION_TYPE").is_ok_and(|t| t == "wayland");
+ std::env::var_os("SWAYSOCK").is_some() || (!wayland && x_display().is_some())
+}
+
+/// `$DISPLAY`, or the first X server's when a service started without one.
+#[cfg(target_os = "linux")]
+fn x_display() -> Option<String> {
+ std::env::var("DISPLAY").ok().filter(|d| !d.is_empty()).or_else(|| {
+ let mut servers: Vec<String> = std::fs::read_dir("/tmp/.X11-unix")
+ .ok()?
+ .flatten()
+ .filter_map(|e| e.file_name().to_str()?.strip_prefix('X').map(|n| format!(":{n}")))
+ .collect();
+ servers.sort();
+ servers.into_iter().next()
+ })
+}
+
+/// Spawns `cua-driver serve` and waits until it listens. It exits when the host does: its stdin
+/// closes then (`CUA_DRIVER_PARENT_LIVENESS_STDIN`), crash or not.
+#[cfg(not(target_os = "macos"))]
+async fn start() -> Result<(tokio::process::Child, String)> {
+ let exe = executable().context("The computer-use driver isn't installed next to codync-host. Reinstall Codync.")?;
+ #[cfg(windows)]
+ let endpoint = endpoint_name();
+ #[cfg(target_os = "linux")]
+ let endpoint = endpoint_name()?;
+ let log = std::fs::File::create(crate::service::data_dir().join("cua-driver.log"))?;
+ let mut cmd = tokio::process::Command::new(&exe);
+ cmd.args(["serve", "--socket", &endpoint])
+ .env("CUA_DRIVER_EMBEDDED", "1")
+ .env("CUA_DRIVER_PARENT_LIVENESS_STDIN", "1")
+ .env("CUA_DRIVER_RS_TELEMETRY_ENABLED", "0")
+ .env("CUA_DRIVER_RS_UPDATE_CHECK", "0")
+ .stdin(std::process::Stdio::piped())
+ .stdout(log.try_clone()?)
+ .stderr(log)
+ .kill_on_drop(true);
+ #[cfg(target_os = "linux")]
+ {
+ if std::env::var_os("SWAYSOCK").is_some() {
+ cmd.env("CUA_DRIVER_RS_ENABLE_WAYLAND", "1");
+ } else if let Some(display) = x_display() {
+ cmd.env("DISPLAY", display);
+ }
+ }
+ #[cfg(windows)]
+ {
+ const CREATE_NO_WINDOW: u32 = 0x0800_0000;
+ cmd.creation_flags(CREATE_NO_WINDOW);
+ }
+ let mut child = cmd.spawn().with_context(|| format!("can't start {}", exe.display()))?;
+ for _ in 0..100 {
+ if let Some(status) = child.try_wait()? {
+ bail!("the computer-use driver exited ({status}); see cua-driver.log in Codync's data folder");
+ }
+ if request(&endpoint, json!({"method": "list", "client_kind": "cli"})).await.is_ok() {
+ tracing::info!(endpoint, "computer-use driver started");
+ return Ok((child, endpoint));
+ }
+ tokio::time::sleep(Duration::from_millis(150)).await;
+ }
+ bail!("the computer-use driver didn't start listening")
+}
+
+/// A socket in a directory only this user can open (whoever reaches it can drive the desktop),
+/// short enough for `sun_path`: under the login session's runtime dir, else Codync's data dir,
+/// both private to this user already.
+#[cfg(target_os = "linux")]
+fn endpoint_name() -> Result<String> {
+ use std::os::unix::fs::PermissionsExt;
+ let parent = std::env::var_os("XDG_RUNTIME_DIR").map_or_else(crate::service::data_dir, std::path::PathBuf::from);
+ let dir = parent.join("codync-cua");
+ std::fs::create_dir_all(&dir)?;
+ std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700))?;
+ let socket = dir.join("cua.sock");
+ let _ = std::fs::remove_file(&socket);
+ Ok(socket.to_string_lossy().into_owned())
+}
+
+/// A random pipe name: Windows lets another process add an instance to a name it can guess.
+#[cfg(windows)]
+fn endpoint_name() -> String {
+ format!(r"\\.\pipe\codync-cua-{}", uuid::Uuid::new_v4())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn stored_tools_say_which_only_look() {
+ assert!(offered("click") && !read_only("click"));
+ assert!(read_only("get_window_state") && read_only("list_windows"));
+ assert!(!offered("bring_to_front") && !offered("kill_app") && !offered("clipboard_read"));
+ }
+
+ #[test]
+ fn summaries_get_their_data_and_snapshots_their_tokens() {
+ let mut r = json!({"content": [{"type": "text", "text": "Found 2 window(s)."}], "structuredContent": {"windows": [1, 2]}});
+ assert_eq!(agent_content(&mut r)[1]["text"], r#"{"windows":[1,2]}"#);
+ let mut r = json!({"content": [{"type": "image", "data": "x"}, {"type": "text", "text": "- [0] AXWindow"}], "structuredContent": {"snapshot_id": "s00000002", "elements": []}});
+ let c = agent_content(&mut r);
+ assert_eq!(c.as_array().unwrap().len(), 3);
+ assert!(c[2]["text"].as_str().unwrap().ends_with("s00000002:N"));
+ let long = "x".repeat(SUMMARY_LEN);
+ let mut r = json!({"content": [{"type": "text", "text": long}], "structuredContent": {"apps": []}});
+ assert_eq!(agent_content(&mut r).as_array().unwrap().len(), 1, "the text already has the data");
+ }
+
+ #[tokio::test]
+ #[cfg(unix)]
+ async fn line_protocol_roundtrip() {
+ let (a, b) = tokio::net::UnixStream::pair().unwrap();
+ tokio::spawn(async move {
+ let mut b = BufReader::new(b);
+ let mut line = String::new();
+ b.read_line(&mut line).await.unwrap();
+ let req: Value = serde_json::from_str(&line).unwrap();
+ let reply = json!({"ok": true, "result": {"echo": req["name"]}});
+ b.get_mut().write_all(format!("{reply}\n").as_bytes()).await.unwrap();
+ });
+ let reply = exchange(a, &json!({"method": "call", "name": "list_windows"})).await.unwrap();
+ assert_eq!(reply["result"]["echo"], "list_windows");
+ }
+}
diff --git a/host/src/screen/cua_tools.json b/host/src/screen/cua_tools.json
new file mode 100644
index 00000000..114911ea
--- /dev/null
+++ b/host/src/screen/cua_tools.json
@@ -0,0 +1,1262 @@
+[
+ {
+ "name": "list_apps",
+ "description": "List macOS apps — both currently running and installed-but-not-running — with per-app state flags:\n\n- running: is a process for this app live? (pid is 0 when false)\n- active: is it the system-frontmost app? (implies running)\n- launch_path: filesystem path to the `.app` bundle, when known. Pass this to `launch_app` to start the app cold.\n- kind: `\"desktop\"` for `.app` bundles on macOS.\n- last_used: RFC3339 timestamp from the bundle's filesystem mtime, when readable; otherwise null.\n\nStandalone running entries include only apps with NSApplicationActivationPolicyRegular — background helpers and system UI agents are filtered out. Installed apps resolve their running/pid state against all live processes by bundle identifier, so an installed app whose process runs as an accessory (LSUIElement / menu-bar apps, e.g. Cua Driver itself) still reports its live pid. Installed apps come from scanning /Applications, /Applications/Utilities, ~/Applications, /System/Applications, and /System/Applications/Utilities.\n\nUse this for \"is X installed?\" as well as \"is X running?\". For per-window state — on-screen, on-current-Space, minimized, window titles — call list_windows instead. For just opening an app — running or not — call launch_app({bundle_id: ...}) directly; list_apps is not a prerequisite.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": true,
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": false
+ }
+ },
+ {
+ "name": "list_windows",
+ "description": "List all layer-0 top-level windows currently known to WindowServer. Includes off-screen windows (minimized, on another Space, hidden-launched). Use this to find a window_id before calling get_window_state. AppKit-internal helper windows (off screen, untitled, in no Space, and absent from the app's AXWindows) are omitted: no tool can read, move, or focus them.\n\nPer-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one), is_on_screen, space_ids, current_space_id (the active Space on that window's display), and on_current_space. The top-level current_space_id is WindowServer's main/global active Space and can differ from a record's current_space_id when displays use independent Spaces. To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "on_screen_only": {
+ "description": "When true, drop windows not on the current Space. Default false.",
+ "type": "boolean"
+ },
+ "pid": {
+ "description": "Optional pid filter. When set, only this pid's windows are returned.",
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": true,
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": false
+ }
+ },
+ {
+ "name": "get_window_state",
+ "description": "Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array; pass each element's `element_token` to click, type_text, press_key, etc.\n\nINVARIANT: call get_window_state once per turn per (pid, window_id) before any element action. The next snapshot of the window replaces this one, stales its element tokens, and lists the replaced ids in `invalidated_snapshot_ids`.\n\nPREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value` (the element's text/AXValue when present — use it to verify what a field holds), `actions` (names of AX actions exposed by the element, omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_image_dimension` overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy `max_dimension` remains a tighter cap for compatibility.\n\nThe snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with `delivery_mode:\"foreground\"`. When that pid is an app still launching (its window exists before it answers accessibility), the walk first waits up to `timeout_ms` for it; if it never answers, the tree comes back EMPTY with `degraded_reason: ax_app_launching`, `truncated: true` and `truncation_reason: app_lookup_timeout`. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.\n\nOptional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.\n\nOptional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "capture_mode": {
+ "description": "DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old \"som\"/\"screenshot\" aliases) is accepted but has no effect.",
+ "enum": [
+ "ax",
+ "vision"
+ ],
+ "type": "string"
+ },
+ "include_accessibility_tree": {
+ "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).",
+ "type": "boolean"
+ },
+ "include_screenshot": {
+ "description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk.",
+ "type": "boolean"
+ },
+ "max_depth": {
+ "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees.",
+ "minimum": 1,
+ "type": "integer"
+ },
+ "max_dimension": {
+ "description": "Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default.",
+ "minimum": 1,
+ "type": "integer"
+ },
+ "max_elements": {
+ "description": "Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows.",
+ "minimum": 1,
+ "type": "integer"
+ },
+ "max_image_dimension": {
+ "description": "Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior.",
+ "minimum": 0,
+ "type": "integer"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "query": {
+ "description": "Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values.",
+ "type": "string"
+ },
+ "screenshot_out_file": {
+ "description": "When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead.",
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "timeout_ms": {
+ "default": 1000,
+ "description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.",
+ "maximum": 120000,
+ "minimum": 100,
+ "type": "integer"
+ },
+ "window_id": {
+ "description": "Target window ID from list_windows.",
+ "type": "integer"
+ }
+ },
+ "required": [
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": true,
+ "destructiveHint": false,
+ "idempotentHint": false,
+ "openWorldHint": false
+ }
+ },
+ {
+ "name": "get_desktop_state",
+ "description": "Capture the full display in true screen pixels, full size unless `max_image_dimension` caps it. Use its PNG as the coordinate source for actions whose target is {kind:\"desktop\",display_id:\"primary\"}. Returns the true screen size and backing scale factor. Vision-only: no AX tree walk.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "max_image_dimension": {
+ "description": "Optional long-edge cap for the returned PNG, in pixels (aspect ratio preserved). Omitted or 0 returns the full-size capture. When the cap downsizes the image, the response reports `screenshot_original_width/height`, and x/y read off it for this session's later scope:\"desktop\" actions (or passed with its `capture_id`) are mapped back to the full-size frame automatically.",
+ "minimum": 0,
+ "type": "integer"
+ },
+ "screenshot_out_file": {
+ "description": "Write PNG here instead of base64.",
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": true,
+ "destructiveHint": false,
+ "idempotentHint": false,
+ "openWorldHint": false
+ }
+ },
+ {
+ "name": "zoom",
+ "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide.\n\nAfter a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "pid": {
+ "description": "Optional target pid. When omitted, the driver resolves the unique current snapshot for this session and window.",
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "CGWindowID from list_windows.",
+ "type": "integer"
+ },
+ "x1": {
+ "description": "Left edge of region in screenshot pixels.",
+ "type": "number"
+ },
+ "x2": {
+ "description": "Right edge of region in screenshot pixels.",
+ "type": "number"
+ },
+ "y1": {
+ "description": "Top edge of region in screenshot pixels.",
+ "type": "number"
+ },
+ "y2": {
+ "description": "Bottom edge of region in screenshot pixels.",
+ "type": "number"
+ }
+ },
+ "required": [
+ "window_id",
+ "x1",
+ "y1",
+ "x2",
+ "y2"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": true,
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": false
+ }
+ },
+ {
+ "name": "launch_app",
+ "description": "Launch a macOS app in the background — the target does NOT come to the foreground.\n\nProvide either `bundle_id` (preferred — unambiguous, e.g. `com.apple.calculator`) or `name` (e.g. \"Calculator\"). If both are given, bundle_id wins.\n\nOptional `urls` are handed to the app as open targets — for Finder, pass a folder path to open a backgrounded Finder window there.\n\nBrowser DevTools setup belongs to `browser_prepare`, which can prove that a separate isolated profile is driver-owned before enabling CDP.\n\nOptional `webkit_inspector_port`: opens a WebKit inspector server on the specified port (sets WEBKIT_INSPECTOR_SERVER=127.0.0.1:N + TAURI_WEBVIEW_AUTOMATION=1). Use this for Tauri/WebKit-based apps.\n\nOptional `creates_new_application_instance`: when true, forces a new app instance even if one is already running (passes -n to open). Reach for this when another agent or session may drive the SAME app concurrently — it returns a fresh pid + window so each session acts on its own isolated window instead of clobbering one shared instance. Without it, single-instance apps (Calculator, many utilities) hand every caller the same window, so two sessions fight over it.\n\nOptional `additional_arguments`: extra argv strings appended after --args.\n\nReturns the launched app's pid, bundle_id, name, and a `windows` array (same shape as `list_windows`) so callers can skip an extra round-trip before `get_window_state(pid, window_id)`. `launch_state` distinguishes whether the request was sent, the process is running, and a window is ready. When the focus-steal belt-and-braces demotion check ran (target pid ≠ prior frontmost), the response also includes `self_activation_suppressed: bool` — true if focus stayed with the prior frontmost, false if the launched app held focus despite the re-demote attempt.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "additional_arguments": {
+ "description": "Extra arguments appended after --args when launching.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "bundle_id": {
+ "description": "App bundle identifier, e.g. com.apple.calculator. Preferred over name.",
+ "type": "string"
+ },
+ "creates_new_application_instance": {
+ "description": "When true, force a new app instance even if already running (open -n). Use for concurrent multi-agent/multi-session work so each session gets an isolated instance + window instead of sharing one — on single-instance apps (e.g. Calculator) every caller otherwise gets the same window and the sessions clobber each other.",
+ "type": "boolean"
+ },
+ "name": {
+ "description": "App display name. Used only when bundle_id is absent.",
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "urls": {
+ "description": "Optional file paths or URLs to open with the app (e.g. a folder path for Finder).",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "webkit_inspector_port": {
+ "description": "Open a WebKit inspector server on this port (sets WEBKIT_INSPECTOR_SERVER env var).",
+ "type": "integer"
+ }
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "click",
+ "description": "Click against a target pid. **Prefer `element_token` over pixel coordinates** — the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree.\n\nTwo addressing modes:\n\n- element_token (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking.\n\n- x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion.\n\nbutton: \"left\" (default), \"right\", or \"middle\". Defaults to left so the field is fully back-compat — omit it and you get the legacy left-click behaviour. Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. AX path: \"right\" maps to AXShowMenu (same surface as the dedicated `right_click` tool); \"middle\" has no AX equivalent and falls back to a pixel middle-click at the element's center.\naction: press (default), show_menu, pick, confirm, cancel, open.\nfrom_zoom: set true after a zoom call to auto-translate zoom-image pixel coordinates to full-window space.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "action": {
+ "description": "AX action: press, show_menu, pick, confirm, cancel, open.",
+ "type": "string"
+ },
+ "button": {
+ "description": "Mouse button. Default: \"left\" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps \"right\" to AXShowMenu and falls back to a pixel middle-click at the element's center for \"middle\".",
+ "enum": [
+ "left",
+ "right",
+ "middle"
+ ],
+ "type": "string"
+ },
+ "capture_id": {
+ "description": "Optional immutable source capture ID returned by get_window_state or get_desktop_state. With x,y, Driver atomically admits and consumes that exact capture before dispatch; stale, mismatched, or out-of-bounds captures are refused without fallback.",
+ "type": "string"
+ },
+ "count": {
+ "description": "Click count (pixel path only). Default 1.",
+ "type": "integer"
+ },
+ "debug_image_out": {
+ "description": "Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom.",
+ "type": "string"
+ },
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": perform the AX action or post the CGEvent without fronting. \"foreground\": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require \"foreground\" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_token) → snapshot → background pixel (x/y) → snapshot → delivery_mode:\"foreground\".",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "from_zoom": {
+ "description": "When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates.",
+ "type": "boolean"
+ },
+ "modifier": {
+ "description": "Modifier keys: cmd, shift, option/alt, ctrl.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "scope": {
+ "description": "Coordinate frame for a windowless screen-absolute click (default \"window\"). Pass \"desktop\" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope=\"desktop\"). Per-call; not a setting.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "target": {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "window_id": {
+ "description": "Target window ID. Omit when element_token is supplied (the token carries it).",
+ "type": "integer"
+ },
+ "x": {
+ "description": "X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Y in screenshot pixels from the image selected by target.",
+ "type": "number"
+ }
+ },
+ "required": [],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "double_click",
+ "description": "Double-click at (x, y) or on an AX element identified by element_token.\n\nAX path (element_token provided): performs `AXOpen` when the element advertises it (Finder items, openable list rows/cells); otherwise resolves the element's on-screen center and falls back to a pixel double-click there.\n\nPixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordinates.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "type": "integer"
+ },
+ "x": {
+ "description": "Screen X coordinate (pixel path).",
+ "type": "number"
+ },
+ "y": {
+ "description": "Screen Y coordinate (pixel path).",
+ "type": "number"
+ }
+ },
+ "required": [
+ "pid"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "right_click",
+ "description": "Right-click against a target pid. Two addressing modes:\n\n- `element_token` (from the last `get_window_state` snapshot) — performs `AXShowMenu` on the cached element. Pure AX RPC, works on backgrounded / hidden windows, no cursor move or focus steal. Requires a prior `get_window_state(pid, window_id)` in this turn.\n\n- `x`, `y` — synthesizes `rightMouseDown` / `rightMouseUp` CGEvent pair posted to the pid. Driver converts image-pixel → screen-point internally. `modifier` forces the CGEvent path (AX actions don't propagate modifier keys).\n\nExactly one of `element_token` or (`x` AND `y`) must be provided. `pid` always required.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "modifier": {
+ "description": "Modifier keys held during the right-click: cmd/shift/option/ctrl. Pixel path only.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "type": "integer"
+ },
+ "x": {
+ "description": "X in window-local screenshot pixels. Must be provided together with y.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Y in window-local screenshot pixels. Must be provided together with x.",
+ "type": "number"
+ }
+ },
+ "required": [
+ "pid"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "drag",
+ "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window.\n\nUse for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel.\n\n`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures.\n\n`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture.\n\nWhen `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching.\n\nmacOS has no background drag: a window-scoped drag needs delivery_mode:\"foreground\" and window_id. It briefly fronts the exact window, moves the physical pointer along the path, then restores the prior frontmost app. Without delivery_mode:\"foreground\" the call is refused with background_unavailable and nothing is sent.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "button": {
+ "description": "Mouse button used for the drag. Default: left.",
+ "enum": [
+ "left",
+ "right",
+ "middle"
+ ],
+ "type": "string"
+ },
+ "delivery_mode": {
+ "description": "Window-scoped drag on macOS supports only \"foreground\": briefly front the exact window (window_id required), drive the physical pointer through the gesture, then restore the prior frontmost app. The default \"background\" is refused with background_unavailable because macOS has no background drag route. Desktop scope ignores this field.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "duration_ms": {
+ "description": "Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500.",
+ "maximum": 10000,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "from_x": {
+ "description": "Drag-start X in window-local screenshot pixels. Top-left origin.",
+ "type": "number"
+ },
+ "from_y": {
+ "description": "Drag-start Y in window-local screenshot pixels. Top-left origin.",
+ "type": "number"
+ },
+ "from_zoom": {
+ "description": "When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates.",
+ "type": "boolean"
+ },
+ "modifier": {
+ "description": "Modifier keys held across the entire gesture: cmd/shift/option/ctrl.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "scope": {
+ "default": "window",
+ "description": "Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "steps": {
+ "description": "Number of intermediate mouseDragged events linearly interpolated along the path. Default: 20.",
+ "maximum": 200,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "target": {
+ "anyOf": [
+ {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
+ },
+ "to_x": {
+ "description": "Drag-end X in window-local screenshot pixels.",
+ "type": "number"
+ },
+ "to_y": {
+ "description": "Drag-end Y in window-local screenshot pixels.",
+ "type": "number"
+ },
+ "window_id": {
+ "description": "CGWindowID for the window the pixel coordinates were measured against. Optional only when pid owns exactly one eligible top-level window; otherwise the action refuses with ambiguous_window_target.",
+ "type": "integer"
+ }
+ },
+ "required": [
+ "from_x",
+ "from_y",
+ "to_x",
+ "to_y"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "scroll",
+ "description": "Scroll the target pid. Two paths, picked by how you address the scroll:\n\n• **Targeted wheel path** — when you pass a target, either `element_token` (preferred) or window-local `x, y` pixels: the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, at that screen point. The renderer hit-tests the wheel at the cursor, so the scroll lands on whatever element is under the point — exactly like physically rolling the wheel over it. This is the ONLY way to scroll a nested `overflow:auto` region (e.g. a scrollable <div> with no tabindex): such regions never take keyboard focus, so the keystroke path below no-ops on them. Use this for inner/nested scrollers in web views.\n\n• **Keystroke path (focused region)** — when you pass NO target (just pid + direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows (by='line'); horizontal uses Left/Right arrows. Drives the focused / page scroller only.\n\nMapping: by='page' → larger step; by='line' → smaller step; amount = number of wheel notches (targeted path) or keystroke repetitions (keystroke path).",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "amount": {
+ "description": "Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Larger requests are clamped to the maximum. Default: 3.",
+ "maximum": 50,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "by": {
+ "description": "Scroll granularity. Default: line.",
+ "enum": [
+ "line",
+ "page"
+ ],
+ "type": "string"
+ },
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "direction": {
+ "description": "Scroll direction.",
+ "enum": [
+ "up",
+ "down",
+ "left",
+ "right"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "pid": {
+ "description": "Target process ID. Required unless scope is \"desktop\".",
+ "type": "integer"
+ },
+ "scope": {
+ "default": "window",
+ "description": "Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "target": {
+ "anyOf": [
+ {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
+ },
+ "window_id": {
+ "description": "CGWindowID of the target window. Required with x/y; optional with element_token (the token carries it).",
+ "type": "integer"
+ },
+ "x": {
+ "description": "Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Window-local screenshot Y. See `x`.",
+ "type": "number"
+ }
+ },
+ "required": [
+ "direction"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": false,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "type_text",
+ "description": "Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa text fields and text views. No keystrokes are synthesized — special keys (Return / Escape / arrows) go through `press_key` / `hotkey`. For Chromium / Electron inputs that don't implement `kAXSelectedText`, the tool falls back to CGEvent character synthesis automatically when the estimated route stays within the daemon transport budget. Longer synthesized routes are refused before character events and return a safe chunk size; one-call AX insertion remains uncapped.\n\nOptional `element_token` (from the last `get_window_state` snapshot) directs the write to a specific field. Without `element_token`, the write goes to the pid's currently focused element.\n\nWEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's compose box): AXValue is not independent proof that the renderer/DOM observed an AX write or synthesized keystrokes. The driver detects this at the element level (an AXWebArea ancestor) and refuses to trust AXValue-only read-back there. Electron AX targets that are web content or cannot be proven native refuse background delivery before mutation because the AX route cannot establish exact renderer focus; use the px form or explicit foreground delivery. Other web-content paths return effect:\"unverifiable\" + escalation, never a false \"confirmed\" (a browser's own native address bar/toolbar stays trusted). For a browser TAB the reliable path is the `page` tool (drives the DOM via CDP); for an embedded web view use this tool's px form: pass x,y (no element_token) to pixel-click the field then type, in one call. NOTE: a px focus-click won't reliably open+focus a CLOSED control; AX-press to open/activate it first (works in the background), then px-type. Always confirm via the screenshot; if px-background still drops, escalate to delivery_mode:\"foreground\".",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "delay_ms": {
+ "description": "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds.",
+ "maximum": 200,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:\"unverifiable\". \"foreground\": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with \"foreground\" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "scope": {
+ "default": "window",
+ "description": "Use desktop with no pid/window_id to type into the frontmost application.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "target": {
+ "anyOf": [
+ {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
+ },
+ "text": {
+ "description": "Text to insert at the target's cursor.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "type": "integer"
+ },
+ "x": {
+ "description": "Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Screenshot-pixel Y of the field (see x).",
+ "type": "number"
+ }
+ },
+ "required": [
+ "text"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "press_key",
+ "description": "Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:\n• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_token it focuses that AX element first. `window_id` only targets; it does not raise.\n• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id.\n\nA key press is confirmed only when a bounded native AX value/selection read-back changes on the same control. Otherwise a successfully attempted post remains effect:\"unverifiable\" without implying delivery failure or recommending foreground. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "key": {
+ "description": "Key name: return, tab, escape, up, down, etc.",
+ "type": "string"
+ },
+ "modifiers": {
+ "description": "Modifier keys: cmd, shift, option/alt, ctrl, fn.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "scope": {
+ "default": "window",
+ "description": "Use desktop with no pid/window_id to send the key to the frontmost application.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "target": {
+ "anyOf": [
+ {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
+ },
+ "window_id": {
+ "description": "Target window. Required for delivery_mode:\"foreground\". Does NOT itself raise the window — raising is gated on delivery_mode.",
+ "type": "integer"
+ },
+ "x": {
+ "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Screenshot-pixel Y (see x).",
+ "type": "number"
+ }
+ },
+ "required": [
+ "key"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "hotkey",
+ "description": "Press a key combination — e.g. `[\"cmd\", \"c\"]` for Copy, `[\"cmd\", \"shift\", \"4\"]` for screenshot selection. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default:\n• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. `window_id` here only targets the combo; it does not raise.\n• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.\n\nA combo is never driver-verifiable (no read-back) → effect:\"unverifiable\"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field — to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then `type_text`. If an app only accepts paste, call `clipboard_write`, then `clipboard_read` and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V.\n\nRecognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modifier keys use the same vocabulary as `press_key`. Order: modifiers first, one non-modifier last.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "delivery_mode": {
+ "description": "Best-effort-background ladder rung (default \"background\"). \"background\": inject without fronting or raising the target — no focus steal. \"foreground\": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with \"foreground\" only for the action that needs it.",
+ "enum": [
+ "background",
+ "foreground"
+ ],
+ "type": "string"
+ },
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "keys": {
+ "description": "Modifier(s) and one non-modifier key, e.g. [\"cmd\", \"c\"].",
+ "items": {
+ "type": "string"
+ },
+ "minItems": 2,
+ "type": "array"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "scope": {
+ "default": "window",
+ "description": "Use desktop with no pid/window_id to send the chord to the frontmost application.",
+ "enum": [
+ "window",
+ "desktop"
+ ],
+ "type": "string"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "target": {
+ "anyOf": [
+ {
+ "description": "Exact capture/input target selected independently for each action.\n\n`display_id=\"primary\"` is the portable desktop target in this release.\nPlatforms that cannot address another display reject it explicitly rather\nthan silently changing coordinate spaces.",
+ "oneOf": [
+ {
+ "additionalProperties": true,
+ "properties": {
+ "kind": {
+ "const": "window",
+ "type": "string"
+ },
+ "pid": {
+ "minimum": 0,
+ "type": "integer"
+ },
+ "window_id": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "kind",
+ "pid",
+ "window_id"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": true,
+ "properties": {
+ "display_id": {
+ "type": "string"
+ },
+ "kind": {
+ "const": "desktop",
+ "type": "string"
+ }
+ },
+ "required": [
+ "kind",
+ "display_id"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`)."
+ },
+ "window_id": {
+ "description": "Target window. Required for delivery_mode:\"foreground\" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode.",
+ "type": "integer"
+ },
+ "x": {
+ "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Cmd+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach.",
+ "type": "number"
+ },
+ "y": {
+ "description": "Screenshot-pixel Y (see x).",
+ "type": "number"
+ }
+ },
+ "required": [
+ "keys"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "set_value",
+ "description": "Set a value on a UI element. Two modes depending on element role:\n\n- **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly — the native macOS popup menu is never opened, so focus is never stolen. Use this for HTML <select> elements in Safari or any native NSPopUpButton.\n\n- **All other elements**: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose settable AXValue).\n\n- **A file's name as Finder lists it** (a text field carrying AXFilename and a file URL, not being edited) or **as Finder's Get Info window shows it** (Finder's text field with AXIdentifier `Name`, not being edited): refused with `file_name_needs_rename`, because the write changes only what Finder shows, never the file. The refusal names the keyboard route that renames it.\n\nFor free-form text entry into web inputs, prefer `type_text_chars` which synthesises key events — AXValue writes are ignored by WebKit.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "properties": {
+ "element_token": {
+ "description": "Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. Returns an explicit stale error naming the current snapshots once a newer read supersedes it.",
+ "pattern": "^s[0-9a-f]{8}:[0-9]+$",
+ "type": "string"
+ },
+ "pid": {
+ "description": "Target process ID.",
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "value": {
+ "description": "New value. AX will coerce to the element's native type.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "CGWindowID. Omit when element_token is supplied (the token carries it).",
+ "type": "integer"
+ }
+ },
+ "required": [
+ "pid",
+ "value"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": true,
+ "openWorldHint": true
+ }
+ },
+ {
+ "name": "invoke_menu",
+ "description": "Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels.",
+ "inputSchema": {
+ "additionalProperties": false,
+ "description": "Exact, immediate-child application menu path to resolve and invoke through\nthe operating system's accessibility API. Path labels are matched after\ntrimming surrounding whitespace and otherwise remain case-sensitive. On\nmacOS, three periods in a label also match the ellipsis character that\nnative menu titles use (`Save As...` finds `Save As…`).",
+ "properties": {
+ "path": {
+ "description": "Menu labels from the top-level menu to the item, e.g. `[\"File\", \"Save As...\"]`\n(1 to 16 labels).",
+ "items": {
+ "maxLength": 200,
+ "minLength": 1,
+ "type": "string"
+ },
+ "maxItems": 16,
+ "minItems": 1,
+ "type": "array"
+ },
+ "pid": {
+ "description": "Process ID of the application that owns the menu.",
+ "minimum": 1,
+ "type": "integer"
+ },
+ "session": {
+ "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
+ "type": "string"
+ },
+ "window_id": {
+ "description": "Window ID from list_windows whose menu is invoked.",
+ "minimum": 1,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "pid",
+ "window_id",
+ "path"
+ ],
+ "type": "object"
+ },
+ "annotations": {
+ "readOnlyHint": false,
+ "destructiveHint": true,
+ "idempotentHint": false,
+ "openWorldHint": true
+ }
+ }
+]
diff --git a/host/src/screen/helper_tools.rs b/host/src/screen/helper_tools.rs
new file mode 100644
index 00000000..c4248697
--- /dev/null
+++ b/host/src/screen/helper_tools.rs
@@ -0,0 +1,457 @@
+//! Linux desktops the computer-use driver doesn't support (GNOME, KDE and other Wayland
+//! sessions): bots act through Codync Screen's portal input on screenshot pixels instead.
+
+use super::helper::Link;
+use super::protocol::Display;
+use super::{SETTLE, Screen};
+use anyhow::{Result, anyhow, bail};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::time::Duration;
+
+/// Long edge of screenshots handed to agents: sharp enough to read UI text,
+/// small enough to keep every turn cheap.
+const SHOT_MAX_EDGE: f64 = 1280.0;
+
+/// A tool call, as the MCP server forwards it (`{name, arguments}`).
+#[derive(Debug, Deserialize)]
+#[serde(tag = "name", content = "arguments", rename_all = "snake_case")]
+enum Tool {
+ Screenshot {
+ display: Option<u32>,
+ },
+ Click {
+ x: f64,
+ y: f64,
+ #[serde(default)]
+ button: Button,
+ count: Option<u8>,
+ #[serde(default)]
+ modifiers: Vec<String>,
+ display: Option<u32>,
+ },
+ Move {
+ x: f64,
+ y: f64,
+ display: Option<u32>,
+ },
+ Drag {
+ x: f64,
+ y: f64,
+ to_x: f64,
+ to_y: f64,
+ display: Option<u32>,
+ },
+ Scroll {
+ x: f64,
+ y: f64,
+ #[serde(default)]
+ dx: f64,
+ #[serde(default)]
+ dy: f64,
+ display: Option<u32>,
+ },
+ Type {
+ text: String,
+ },
+ Key {
+ keys: String,
+ },
+ UiTree {
+ display: Option<u32>,
+ },
+ OpenApp {
+ name: String,
+ },
+}
+
+pub(super) fn read_only(name: &str) -> bool {
+ matches!(name, "screenshot" | "ui_tree")
+}
+
+pub(super) fn tools() -> Vec<Value> {
+ let display = json!({"type": "integer", "description": "Display id; defaults to the main display."});
+ let xy = |what: &str| json!({"type": "number", "description": format!("{what} in screenshot pixels.")});
+ let tools = json!([
+ {
+ "name": "screenshot",
+ "description": "Capture the screen. Returns a JPEG whose pixels are the coordinate space for every other tool.",
+ "inputSchema": {"type": "object", "properties": {"display": display}},
+ "annotations": {"readOnlyHint": true},
+ },
+ {
+ "name": "ui_tree",
+ "description": "Accessibility tree of the frontmost app: roles, titles, values and frames in screenshot pixels. Cheaper and more precise than reading pixels when you need to find a control.",
+ "inputSchema": {"type": "object", "properties": {"display": display}},
+ "annotations": {"readOnlyHint": true},
+ },
+ {
+ "name": "click",
+ "description": "Click at (x, y) in screenshot pixels. Returns a screenshot afterwards.",
+ "inputSchema": {
+ "type": "object",
+ "properties": {
+ "x": xy("X"), "y": xy("Y"),
+ "button": {"type": "string", "enum": ["left", "right", "middle"]},
+ "count": {"type": "integer", "description": "2 for a double click."},
+ "modifiers": {"type": "array", "items": {"type": "string", "enum": ["cmd", "option", "ctrl", "shift"]}},
+ "display": display,
+ },
+ "required": ["x", "y"],
+ },
+ },
+ {
+ "name": "move",
+ "description": "Move the pointer (for hover menus and tooltips). Returns a screenshot afterwards.",
+ "inputSchema": {"type": "object", "properties": {"x": xy("X"), "y": xy("Y"), "display": display}, "required": ["x", "y"]},
+ },
+ {
+ "name": "drag",
+ "description": "Press at (x, y), drag to (to_x, to_y), release. Returns a screenshot afterwards.",
+ "inputSchema": {
+ "type": "object",
+ "properties": {"x": xy("Start x"), "y": xy("Start y"), "to_x": xy("End x"), "to_y": xy("End y"), "display": display},
+ "required": ["x", "y", "to_x", "to_y"],
+ },
+ },
+ {
+ "name": "scroll",
+ "description": "Scroll with the pointer over (x, y). Returns a screenshot afterwards.",
+ "inputSchema": {
+ "type": "object",
+ "properties": {
+ "x": xy("X"), "y": xy("Y"),
+ "dx": {"type": "number", "description": "Lines to scroll right (negative = left)."},
+ "dy": {"type": "number", "description": "Lines to scroll down (negative = up)."},
+ "display": display,
+ },
+ "required": ["x", "y"],
+ },
+ },
+ {
+ "name": "type",
+ "description": "Type text into the focused field (any Unicode). Returns a screenshot afterwards.",
+ "inputSchema": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]},
+ },
+ {
+ "name": "key",
+ "description": "Press a key or shortcut, e.g. `return`, `escape`, `tab`, `ctrl+t`, `ctrl+shift+t`. `cmd` is Super. Returns a screenshot afterwards.",
+ "inputSchema": {"type": "object", "properties": {"keys": {"type": "string"}}, "required": ["keys"]},
+ },
+ {
+ "name": "open_app",
+ "description": "Open or bring an application to the front by name (e.g. `firefox`). Returns a screenshot afterwards.",
+ "inputSchema": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]},
+ },
+ ]);
+ match tools {
+ Value::Array(t) => t,
+ _ => unreachable!("a JSON array literal"),
+ }
+}
+
+/// Runs one tool: `{content}` for the agent.
+pub(super) async fn run(screen: &Screen, name: &str, args: Value) -> Result<Value> {
+ let tool: Tool = serde_json::from_value(json!({"name": name, "arguments": args}))
+ .map_err(|e| anyhow!("invalid {name} call: {e}"))?;
+ let link = screen.link()?;
+ let display = |id| screen.display(id);
+ let content = match tool {
+ Tool::Screenshot { display: id } => shot(&link, &display(id)?).await?,
+ Tool::Click { x, y, button, count, modifiers, display: id } => {
+ let d = display(id)?;
+ let (x, y) = d.to_points(x, y)?;
+ let modifiers = parse_modifiers(&modifiers)?;
+ let count = count.unwrap_or(1).clamp(1, 3);
+ act(&link, &d, InputEvent::Click { x, y, button, count, modifiers }).await?
+ }
+ Tool::Move { x, y, display: id } => {
+ let d = display(id)?;
+ let (x, y) = d.to_points(x, y)?;
+ act(&link, &d, InputEvent::Move { x, y }).await?
+ }
+ Tool::Drag { x, y, to_x, to_y, display: id } => {
+ let d = display(id)?;
+ let (x, y) = d.to_points(x, y)?;
+ let (to_x, to_y) = d.to_points(to_x, to_y)?;
+ act(&link, &d, InputEvent::Drag { x, y, to_x, to_y }).await?
+ }
+ Tool::Scroll { x, y, dx, dy, display: id } => {
+ let d = display(id)?;
+ let (x, y) = d.to_points(x, y)?;
+ act(&link, &d, InputEvent::Scroll { x, y, dx, dy }).await?
+ }
+ Tool::Type { text } => act(&link, &display(None)?, InputEvent::Text { text }).await?,
+ Tool::Key { keys } => {
+ let (key, modifiers) = parse_keys(&keys)?;
+ act(&link, &display(None)?, InputEvent::Key { key, modifiers }).await?
+ }
+ Tool::UiTree { display: id } => {
+ let d = display(id)?;
+ let mut res = link.request("uiTree", json!({"display": d.id})).await?;
+ scale_frames(&mut res["tree"], d.shot_scale());
+ let text = format!(
+ "Accessibility tree of {} (frames are [x, y, width, height] in screenshot pixels):\n{}",
+ res["app"].as_str().unwrap_or("the frontmost app"),
+ res["tree"]
+ );
+ json!([{"type": "text", "text": text}])
+ }
+ Tool::OpenApp { name } => {
+ link.request("openApp", json!({"name": name})).await?;
+ tokio::time::sleep(Duration::from_secs(1)).await;
+ shot(&link, &display(None)?).await?
+ }
+ };
+ Ok(json!({"content": content}))
+}
+
+/// Types into the focused field (`type_login`): `{content}` with a screenshot.
+pub(super) async fn type_text(screen: &Screen, text: String) -> Result<Value> {
+ let link = screen.link()?;
+ Ok(json!({"content": act(&link, &screen.display(None)?, InputEvent::Text { text }).await?}))
+}
+
+async fn act(link: &Link, d: &Display, event: InputEvent) -> Result<Value> {
+ link.request("input", json!({"display": d.id, "event": event})).await?;
+ tokio::time::sleep(SETTLE).await;
+ shot(link, d).await
+}
+
+async fn shot(link: &Link, d: &Display) -> Result<Value> {
+ let (width, height) = d.shot_size();
+ let res = link.request("screenshot", json!({"display": d.id, "width": width, "height": height})).await?;
+ let data = res["data"].as_str().ok_or_else(|| anyhow!("the screen helper sent no image"))?;
+ Ok(json!([
+ {"type": "image", "data": data, "mimeType": "image/jpeg"},
+ {"type": "text", "text": format!(
+ "Display {} \"{}\", {width}×{height}. Coordinates for click/move/drag/scroll are pixels in this image.",
+ d.id, d.name
+ )},
+ ]))
+}
+
+impl Display {
+ /// Screenshot scale: points → agent image pixels.
+ fn shot_scale(&self) -> f64 {
+ (SHOT_MAX_EDGE / self.width.max(self.height)).min(1.0)
+ }
+
+ /// Agent image size for this display.
+ fn shot_size(&self) -> (u32, u32) {
+ let s = self.shot_scale();
+ // Display sizes are a few thousand points: always in range.
+ #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
+ ((self.width * s).round() as u32, (self.height * s).round() as u32)
+ }
+
+ /// Agent image pixels → display points, rejecting points off the image.
+ fn to_points(&self, x: f64, y: f64) -> Result<(f64, f64)> {
+ let (width, height) = self.shot_size();
+ if !(0.0..=f64::from(width)).contains(&x) || !(0.0..=f64::from(height)).contains(&y) {
+ bail!("({x}, {y}) is outside the {width}×{height} screenshot");
+ }
+ // Per axis: the image size is rounded, so its aspect differs slightly from the display's.
+ Ok((x * self.width / f64::from(width), y * self.height / f64::from(height)))
+ }
+}
+
+/// Display points → screenshot pixels, for every `frame` in an accessibility tree.
+fn scale_frames(node: &mut Value, s: f64) {
+ if let Some(frame) = node.get_mut("frame").and_then(Value::as_array_mut) {
+ for v in frame.iter_mut() {
+ if let Some(f) = v.as_f64() {
+ *v = json!((f * s).round());
+ }
+ }
+ }
+ if let Some(children) = node.get_mut("children").and_then(Value::as_array_mut) {
+ for c in children {
+ scale_frames(c, s);
+ }
+ }
+}
+
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+enum Button {
+ #[default]
+ Left,
+ Right,
+ Middle,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+enum Modifier {
+ /// Super.
+ Cmd,
+ Option,
+ Ctrl,
+ Shift,
+}
+
+/// One input action for the helper, in display points.
+#[derive(Clone, Debug, PartialEq, Serialize)]
+#[serde(tag = "type", rename_all = "camelCase", rename_all_fields = "camelCase")]
+enum InputEvent {
+ Move {
+ x: f64,
+ y: f64,
+ },
+ Click {
+ x: f64,
+ y: f64,
+ button: Button,
+ count: u8,
+ modifiers: Vec<Modifier>,
+ },
+ Drag {
+ x: f64,
+ y: f64,
+ to_x: f64,
+ to_y: f64,
+ },
+ /// Scroll by `dx`/`dy` lines (positive = right/down) with the pointer at `x`,`y`.
+ Scroll {
+ x: f64,
+ y: f64,
+ dx: f64,
+ dy: f64,
+ },
+ Text {
+ text: String,
+ },
+ Key {
+ key: String,
+ modifiers: Vec<Modifier>,
+ },
+}
+
+/// Named keys the helper understands; any other key is a single character.
+const NAMED_KEYS: &[&str] = &[
+ "return",
+ "tab",
+ "space",
+ "escape",
+ "delete",
+ "forwardDelete",
+ "left",
+ "right",
+ "up",
+ "down",
+ "home",
+ "end",
+ "pageUp",
+ "pageDown",
+ "f1",
+ "f2",
+ "f3",
+ "f4",
+ "f5",
+ "f6",
+ "f7",
+ "f8",
+ "f9",
+ "f10",
+ "f11",
+ "f12",
+];
+
+/// Modifier names with common aliases, deduplicated.
+fn parse_modifiers(names: &[impl AsRef<str>]) -> Result<Vec<Modifier>> {
+ let mut out = vec![];
+ for m in names {
+ let m = match m.as_ref().to_lowercase().as_str() {
+ "cmd" | "command" | "meta" | "super" | "win" => Modifier::Cmd,
+ "option" | "opt" | "alt" => Modifier::Option,
+ "ctrl" | "control" => Modifier::Ctrl,
+ "shift" => Modifier::Shift,
+ other => bail!("unknown modifier `{other}` (use cmd, option, ctrl, shift)"),
+ };
+ if !out.contains(&m) {
+ out.push(m);
+ }
+ }
+ Ok(out)
+}
+
+/// `"ctrl+shift+t"` → key `t` with ⌃⇧. Accepts common aliases (`ctrl`, `alt`, `enter`, `esc`, …).
+fn parse_keys(combo: &str) -> Result<(String, Vec<Modifier>)> {
+ let parts: Vec<&str> = combo.split('+').map(str::trim).collect();
+ let (key, mods) =
+ parts.split_last().filter(|(k, _)| !k.is_empty()).ok_or_else(|| anyhow!("no key in `{combo}`"))?;
+ let modifiers = parse_modifiers(mods)?;
+ let lower = key.to_lowercase();
+ let key = match lower.as_str() {
+ "enter" => "return".to_owned(),
+ "esc" => "escape".to_owned(),
+ "backspace" => "delete".to_owned(),
+ "del" | "forwarddelete" => "forwardDelete".to_owned(),
+ "pageup" | "pgup" => "pageUp".to_owned(),
+ "pagedown" | "pgdn" => "pageDown".to_owned(),
+ "arrowleft" => "left".to_owned(),
+ "arrowright" => "right".to_owned(),
+ "arrowup" => "up".to_owned(),
+ "arrowdown" => "down".to_owned(),
+ _ if NAMED_KEYS.contains(&lower.as_str()) => lower,
+ _ if key.chars().count() == 1 => lower,
+ _ => bail!("unknown key `{key}`"),
+ };
+ Ok((key, modifiers))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::screen::tests::retina;
+
+ #[test]
+ fn tools_parse_from_mcp_calls() {
+ let t: Tool =
+ serde_json::from_value(json!({"name": "click", "arguments": {"x": 1, "y": 2, "button": "right"}})).unwrap();
+ assert!(matches!(t, Tool::Click { button: Button::Right, count: None, .. }));
+ assert!(read_only("screenshot") && !read_only("click"));
+ assert!(tools().iter().all(|t| serde_json::from_value::<Tool>(json!({"name": t["name"], "arguments": {"x": 1, "y": 1, "to_x": 1, "to_y": 1, "text": "", "keys": "a", "name": "x"}})).is_ok()));
+ }
+
+ #[test]
+ fn screenshot_coordinates_map_to_points() {
+ let d = retina();
+ assert_eq!(d.shot_size(), (1280, 831));
+ assert_eq!(d.to_points(1280.0, 831.0).unwrap(), (1512.0, 982.0));
+ let (x, y) = d.to_points(640.0, 415.5).unwrap();
+ assert!((x - 756.0).abs() < 0.01 && (y - 491.0).abs() < 0.01);
+ assert!(d.to_points(1281.0, 10.0).is_err());
+ // Small displays aren't upscaled.
+ let small = Display { width: 800.0, height: 600.0, ..retina() };
+ assert_eq!(small.shot_size(), (800, 600));
+ }
+
+ #[test]
+ fn key_combos_parse() {
+ assert_eq!(parse_keys("cmd+shift+T").unwrap(), ("t".into(), vec![Modifier::Cmd, Modifier::Shift]));
+ assert_eq!(parse_keys("Enter").unwrap(), ("return".into(), vec![]));
+ assert_eq!(parse_keys("ctrl+alt+Delete").unwrap(), ("delete".into(), vec![Modifier::Ctrl, Modifier::Option]));
+ assert_eq!(parse_keys("cmd++").unwrap_err().to_string(), "no key in `cmd++`");
+ assert!(parse_keys("hyper+x").is_err());
+ assert!(parse_keys("cmd+banana").is_err());
+ }
+
+ #[test]
+ fn input_events_serialize_for_helpers() {
+ let e = InputEvent::Drag { x: 1.0, y: 2.0, to_x: 3.0, to_y: 4.0 };
+ assert_eq!(
+ serde_json::to_value(e).unwrap(),
+ json!({"type": "drag", "x": 1.0, "y": 2.0, "toX": 3.0, "toY": 4.0})
+ );
+ }
+
+ #[test]
+ fn frames_scale_recursively() {
+ let mut t = json!({"frame": [100, 50, 200, 20], "children": [{"frame": [10.0, 10.0, 5.0, 5.0]}]});
+ scale_frames(&mut t, 0.5);
+ assert_eq!(t["frame"], json!([50.0, 25.0, 100.0, 10.0]));
+ assert_eq!(t["children"][0]["frame"], json!([5.0, 5.0, 3.0, 3.0]));
+ }
+}
diff --git a/host/src/screen/mod.rs b/host/src/screen/mod.rs
index b32defc8..6447ac76 100644
--- a/host/src/screen/mod.rs
+++ b/host/src/screen/mod.rs
@@ -1,13 +1,14 @@
//! Remote screen: the phone views and controls this computer over WebRTC, and
-//! bots operate it through the built-in `computer` MCP server (`mcp.rs`).
+//! bots operate it through the built-in `computer` MCP server (`mcp.rs`, `computer.rs`).
//!
-//! Capture, encoding and input injection live in a separate *screen helper*
+//! Capture, encoding and the phone's input live in a separate *screen helper*
//! that connects to `~/.codync/screen.sock` — on macOS `CodyncScreen.app`
//! (a signed bundle, so Screen Recording / Accessibility grants stick), on
//! Linux `codync-screen` (xdg portals + `GStreamer`). The host stays the only
//! network-facing process: it relays WebRTC signaling, gates access (on by
//! default where there is a desktop, toggled only from this computer) and arbitrates control between
-//! the user and bots.
+//! the user and bots. Bots act through the computer-use driver (`cua.rs`), which Codync Screen
+//! starts on macOS and the host starts on Windows and Linux.
//!
//! Helper protocol: newline-delimited JSON-RPC 2.0 over the socket.
//! - helper → host notifications
@@ -17,24 +18,27 @@
//! - `answer {session, sdp, display?, iceServers, maxBitrateBps, maxFramerate}` → `{sdp}`:
//! non-trickle ICE, with short-lived STUN/TURN credentials for remote connections.
//! - `close {session}`, `closeAll {}`
-//! - `screenshot {display, width, height}` → `{data}` (base64 JPEG, exactly
-//! that size)
-//! - `input {display, event}` → `{}`; `event` is an [`InputEvent`](protocol::InputEvent) in display points
-//! - `uiTree {display}` → `{app, tree}`: accessibility tree of the frontmost
-//! app; nodes are `{role, title?, value?, frame: [x, y, w, h], children?}`
-//! with frames in display points
-//! - `openApp {name}` → `{}`
+//! - `focusedField {pid?}` → `{app, pid, url?, secure}`: the focused control of that app (the
+//! frontmost one by default), for `type_login`
+//! - macOS: `driver {}` → `{socket}`: starts the computer-use driver if it isn't running
+//! - Linux, for bots on desktops the driver doesn't support (`helper_tools.rs`):
+//! `screenshot {display, width, height}` → `{data}` (base64 JPEG, exactly that size);
+//! `input {display, event}` → `{}` (display points); `uiTree {display}` → `{app, tree}`;
+//! `openApp {name}` → `{}`
//!
//! The phone's input travels straight to the helper over WebRTC data channels
-//! (`input-fast` unordered for moves, `input` reliable) using the same
-//! [`InputEvent`](protocol::InputEvent) JSON plus `down`/`up` for touch drags and `clipboard {text}`.
+//! (`input-fast` unordered for moves, `input` reliable): `move`, `click`, `drag`, `scroll`,
+//! `text`, `key` in display points, plus `down`/`up` for touch drags and `clipboard {text}`.
mod computer;
+mod cua;
mod helper;
+#[cfg(target_os = "linux")]
+mod helper_tools;
mod protocol;
mod viewer;
-pub use computer::{ComputerTool, computer};
+pub use computer::{computer, computer_tools};
pub use helper::serve_helpers;
#[cfg(target_os = "linux")]
pub use helper::supervise_linux_helper;
@@ -43,7 +47,7 @@ pub use viewer::{IceConfig, watch_viewer};
use crate::LockExt;
use anyhow::{Result, anyhow, bail};
use helper::{Link, graphical_session};
-use protocol::{Display, HelperStatus};
+use protocol::HelperStatus;
use serde_json::{Value, json};
use std::collections::HashMap;
use std::sync::Arc;
@@ -55,6 +59,7 @@ use tokio::sync::broadcast;
/// How long after its last computer call a bot still counts as "using the computer".
const AGENT_HOLD: Duration = Duration::from_secs(20);
/// Lets the UI react to an action before the follow-up screenshot.
+#[cfg(target_os = "linux")]
const SETTLE: Duration = Duration::from_millis(400);
const HELPER_TIMEOUT: Duration = Duration::from_secs(15);
const KV_ENABLED: &str = "screen_enabled";
@@ -88,6 +93,7 @@ pub struct Screen {
status: Mutex<HelperStatus>,
control: Mutex<Control>,
sessions: Mutex<HashMap<String, Viewer>>,
+ driver: cua::Driver,
#[cfg_attr(windows, expect(dead_code, reason = "Windows has no screen helper yet"))]
next_link: AtomicU64,
}
@@ -101,6 +107,7 @@ impl Screen {
status: Mutex::default(),
control: Mutex::default(),
sessions: Mutex::default(),
+ driver: cua::Driver::default(),
next_link: AtomicU64::new(1),
}
}
@@ -135,6 +142,7 @@ impl Screen {
"displays": st.displays,
"userControl": c.user,
"agentBot": c.active_agent(),
+ "computerUse": self.computer_use(),
"viewers": viewers,
})
}
@@ -150,7 +158,8 @@ impl Screen {
self.link.locked().clone().ok_or_else(|| anyhow!("The screen helper isn't running on this computer."))
}
- fn display(&self, id: Option<u32>) -> Result<Display> {
+ #[cfg(target_os = "linux")]
+ fn display(&self, id: Option<u32>) -> Result<protocol::Display> {
let st = self.status.locked();
let found = match id {
Some(id) => st.displays.iter().find(|d| d.id == id),
@@ -160,11 +169,11 @@ impl Screen {
}
}
-#[cfg(test)]
+#[cfg(all(test, unix))]
mod tests {
use super::*;
- pub(super) fn retina() -> Display {
- Display { id: 1, name: "Built-in".into(), width: 1512.0, height: 982.0, main: true }
+ pub(super) fn retina() -> protocol::Display {
+ protocol::Display { id: 1, name: "Built-in".into(), width: 1512.0, height: 982.0, main: true }
}
}
diff --git a/host/src/screen/protocol.rs b/host/src/screen/protocol.rs
index 64c79dd7..55b56771 100644
--- a/host/src/screen/protocol.rs
+++ b/host/src/screen/protocol.rs
@@ -1,50 +1,20 @@
-//! Helper protocol types: displays, helper status and input events, plus key-combo parsing.
+//! Helper protocol types: displays and the helper's status.
-use anyhow::{Result, anyhow, bail};
use serde::{Deserialize, Serialize};
-/// Long edge of screenshots handed to agents: sharp enough to read UI text,
-/// small enough to keep every turn cheap.
-pub const SHOT_MAX_EDGE: f64 = 1280.0;
-
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "camelCase")]
pub struct Display {
pub id: u32,
#[serde(default)]
pub name: String,
- /// Size in points (the coordinate space of [`InputEvent`]).
+ /// Size in points.
pub width: f64,
pub height: f64,
#[serde(default)]
pub main: bool,
}
-impl Display {
- /// Screenshot scale: points → agent image pixels.
- pub(super) fn shot_scale(&self) -> f64 {
- (SHOT_MAX_EDGE / self.width.max(self.height)).min(1.0)
- }
-
- /// Agent image size for this display.
- pub(super) fn shot_size(&self) -> (u32, u32) {
- let s = self.shot_scale();
- // Display sizes are a few thousand points: always in range.
- #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
- ((self.width * s).round() as u32, (self.height * s).round() as u32)
- }
-
- /// Agent image pixels → display points, rejecting points off the image.
- pub(super) fn to_points(&self, x: f64, y: f64) -> Result<(f64, f64)> {
- let (width, height) = self.shot_size();
- if !(0.0..=f64::from(width)).contains(&x) || !(0.0..=f64::from(height)).contains(&y) {
- bail!("({x}, {y}) is outside the {width}×{height} screenshot");
- }
- // Per axis: the image size is rounded, so its aspect differs slightly from the display's.
- Ok((x * self.width / f64::from(width), y * self.height / f64::from(height)))
- }
-}
-
/// What the helper reports about itself.
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
@@ -62,171 +32,3 @@ pub struct HelperStatus {
#[serde(default)]
pub input: bool,
}
-
-#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-pub enum Button {
- #[default]
- Left,
- Right,
- Middle,
-}
-
-#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(rename_all = "lowercase")]
-pub enum Modifier {
- /// ⌘ on macOS, Super on Linux.
- Cmd,
- Option,
- Ctrl,
- Shift,
-}
-
-/// One input action for the helper, in display points.
-#[derive(Clone, Debug, PartialEq, Serialize)]
-#[serde(tag = "type", rename_all = "camelCase", rename_all_fields = "camelCase")]
-pub enum InputEvent {
- Move {
- x: f64,
- y: f64,
- },
- Click {
- x: f64,
- y: f64,
- button: Button,
- count: u8,
- modifiers: Vec<Modifier>,
- },
- Drag {
- x: f64,
- y: f64,
- to_x: f64,
- to_y: f64,
- },
- /// Scroll by `dx`/`dy` lines (positive = right/down) with the pointer at `x`,`y`.
- Scroll {
- x: f64,
- y: f64,
- dx: f64,
- dy: f64,
- },
- Text {
- text: String,
- },
- Key {
- key: String,
- modifiers: Vec<Modifier>,
- },
-}
-
-/// Named keys the helpers understand; any other key is a single character.
-const NAMED_KEYS: &[&str] = &[
- "return",
- "tab",
- "space",
- "escape",
- "delete",
- "forwardDelete",
- "left",
- "right",
- "up",
- "down",
- "home",
- "end",
- "pageUp",
- "pageDown",
- "f1",
- "f2",
- "f3",
- "f4",
- "f5",
- "f6",
- "f7",
- "f8",
- "f9",
- "f10",
- "f11",
- "f12",
-];
-
-/// Modifier names with common aliases, deduplicated.
-pub(super) fn parse_modifiers(names: &[impl AsRef<str>]) -> Result<Vec<Modifier>> {
- let mut out = vec![];
- for m in names {
- let m = match m.as_ref().to_lowercase().as_str() {
- "cmd" | "command" | "meta" | "super" | "win" => Modifier::Cmd,
- "option" | "opt" | "alt" => Modifier::Option,
- "ctrl" | "control" => Modifier::Ctrl,
- "shift" => Modifier::Shift,
- other => bail!("unknown modifier `{other}` (use cmd, option, ctrl, shift)"),
- };
- if !out.contains(&m) {
- out.push(m);
- }
- }
- Ok(out)
-}
-
-/// `"cmd+shift+t"` → key `t` with ⌘⇧. Accepts common aliases (`ctrl`, `alt`, `enter`, `esc`, …).
-pub fn parse_keys(combo: &str) -> Result<(String, Vec<Modifier>)> {
- let parts: Vec<&str> = combo.split('+').map(str::trim).collect();
- let (key, mods) =
- parts.split_last().filter(|(k, _)| !k.is_empty()).ok_or_else(|| anyhow!("no key in `{combo}`"))?;
- let modifiers = parse_modifiers(mods)?;
- let lower = key.to_lowercase();
- let key = match lower.as_str() {
- "enter" => "return".to_owned(),
- "esc" => "escape".to_owned(),
- "backspace" => "delete".to_owned(),
- "del" | "forwarddelete" => "forwardDelete".to_owned(),
- "pageup" | "pgup" => "pageUp".to_owned(),
- "pagedown" | "pgdn" => "pageDown".to_owned(),
- "arrowleft" => "left".to_owned(),
- "arrowright" => "right".to_owned(),
- "arrowup" => "up".to_owned(),
- "arrowdown" => "down".to_owned(),
- _ if NAMED_KEYS.contains(&lower.as_str()) => lower,
- _ if key.chars().count() == 1 => lower,
- _ => bail!("unknown key `{key}`"),
- };
- Ok((key, modifiers))
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::screen::tests::retina;
- use serde_json::json;
-
- #[test]
- fn screenshot_coordinates_map_to_points() {
- let d = retina();
- assert_eq!(d.shot_size(), (1280, 831));
- assert_eq!(d.to_points(1280.0, 831.0).unwrap(), (1512.0, 982.0));
- let (x, y) = d.to_points(640.0, 415.5).unwrap();
- assert!((x - 756.0).abs() < 0.01 && (y - 491.0).abs() < 0.01);
- assert!(d.to_points(1281.0, 10.0).is_err());
- // Small displays aren't upscaled.
- let small = Display { width: 800.0, height: 600.0, ..retina() };
- assert_eq!(small.shot_size(), (800, 600));
- }
-
- #[test]
- fn key_combos_parse() {
- assert_eq!(parse_keys("cmd+shift+T").unwrap(), ("t".into(), vec![Modifier::Cmd, Modifier::Shift]));
- assert_eq!(parse_keys("Enter").unwrap(), ("return".into(), vec![]));
- assert_eq!(parse_keys("ctrl+alt+Delete").unwrap(), ("delete".into(), vec![Modifier::Ctrl, Modifier::Option]));
- assert_eq!(parse_keys("cmd++").unwrap_err().to_string(), "no key in `cmd++`");
- assert!(parse_keys("hyper+x").is_err());
- assert!(parse_keys("cmd+banana").is_err());
- }
-
- #[test]
- fn input_events_serialize_for_helpers() {
- let e = InputEvent::Drag { x: 1.0, y: 2.0, to_x: 3.0, to_y: 4.0 };
- assert_eq!(
- serde_json::to_value(e).unwrap(),
- json!({"type": "drag", "x": 1.0, "y": 2.0, "toX": 3.0, "toY": 4.0})
- );
- }
-}
diff --git a/host/src/update/install.rs b/host/src/update/install.rs
index a91a51cb..8a58e52d 100644
--- a/host/src/update/install.rs
+++ b/host/src/update/install.rs
@@ -182,10 +182,15 @@ pub async fn apply(
}
let bytes = release::archive(release).await?;
let stage = target.with_file_name(format!(".codync-host-update-{}", uuid::Uuid::new_v4()));
- let helper = Helper::new(target);
+ let helpers: Vec<Helper> = HELPERS.iter().map(|name| Helper::new(target, name)).collect();
let result = async {
release::extract(&bytes, &release.platform, &stage)?;
- let has_helper = release::extract_file(&bytes, &release.platform, HELPER, &helper.stage)?;
+ let mut shipped = vec![];
+ for helper in &helpers {
+ if release::extract_file(&bytes, &release.platform, helper.name, &helper.stage)? {
+ shipped.push(helper);
+ }
+ }
let output = tokio::process::Command::new(&stage).arg("--version").kill_on_drop(true).output();
let output = tokio::time::timeout(Duration::from_secs(10), output).await??;
ensure!(
@@ -216,10 +221,12 @@ pub async fn apply(
if managed {
tokio::task::spawn_blocking(service::stop).await??;
}
- // Swapped while the host is stopped, so the new host never starts the old helper.
+ // Swapped while the host is stopped, so the new host never starts an old helper.
// A failed swap keeps the old helper rather than leaving the host stopped.
- if has_helper && let Err(error) = helper.swap() {
- tracing::warn!(error = format!("{error:#}"), "screen helper not updated");
+ for helper in &shipped {
+ if let Err(error) = helper.swap() {
+ tracing::warn!(helper = helper.name, error = format!("{error:#}"), "helper not updated");
+ }
}
Ok(())
},
@@ -231,7 +238,9 @@ pub async fn apply(
Ok(())
},
|| async {
- helper.restore();
+ for helper in &helpers {
+ helper.restore();
+ }
if managed {
tokio::task::spawn_blocking(service::stop).await??;
tokio::task::spawn_blocking(service::start).await??;
@@ -245,16 +254,19 @@ pub async fn apply(
}
.await;
let _ = std::fs::remove_file(stage);
- helper.finish();
+ for helper in &helpers {
+ helper.finish();
+ }
result
}
-/// The Linux Remote screen helper ships in the host's archive; the host starts it from beside
-/// itself, so an update replaces it there too.
-const HELPER: &str = "codync-screen";
+/// Linux helpers in the host's archive: the Remote screen helper and the computer-use driver.
+/// The host starts them from beside itself, so an update replaces them there too.
+const HELPERS: &[&str] = &["codync-screen", "cua-driver"];
-/// Swaps the helper beside the host for the staged one, keeping a copy to roll back to.
+/// Swaps a helper beside the host for the staged one, keeping a copy to roll back to.
struct Helper {
+ name: &'static str,
path: std::path::PathBuf,
stage: std::path::PathBuf,
backup: std::path::PathBuf,
@@ -262,21 +274,22 @@ struct Helper {
}
impl Helper {
- fn new(target: &Path) -> Self {
+ fn new(target: &Path, name: &'static str) -> Self {
let id = uuid::Uuid::new_v4();
Self {
- path: target.with_file_name(HELPER),
- stage: target.with_file_name(format!(".{HELPER}-update-{id}")),
- backup: target.with_file_name(format!(".{HELPER}-backup-{id}")),
+ name,
+ path: target.with_file_name(name),
+ stage: target.with_file_name(format!(".{name}-update-{id}")),
+ backup: target.with_file_name(format!(".{name}-backup-{id}")),
swapped: std::sync::atomic::AtomicBool::new(false),
}
}
fn swap(&self) -> Result<()> {
if self.path.exists() {
- std::fs::hard_link(&self.path, &self.backup).context("saving the screen helper")?;
+ std::fs::hard_link(&self.path, &self.backup).with_context(|| format!("saving {}", self.name))?;
}
- std::fs::rename(&self.stage, &self.path).context("replacing the screen helper")?;
+ std::fs::rename(&self.stage, &self.path).with_context(|| format!("replacing {}", self.name))?;
self.swapped.store(true, std::sync::atomic::Ordering::Relaxed);
Ok(())
}
diff --git a/packaging/cua-driver/LICENSE b/packaging/cua-driver/LICENSE
new file mode 100644
index 00000000..b8b198ce
--- /dev/null
+++ b/packaging/cua-driver/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Cua AI, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/packaging/cua-driver/driver.json b/packaging/cua-driver/driver.json
new file mode 100644
index 00000000..8adb7638
--- /dev/null
+++ b/packaging/cua-driver/driver.json
@@ -0,0 +1,26 @@
+{
+ "version": "0.34.0",
+ "tag": "cua-driver-rs-v0.34.0",
+ "targets": {
+ "darwin": {
+ "asset": "cua-driver-rs-0.34.0-darwin-universal-binary.tar.gz",
+ "sha256": "940dc008e0f7c5d217d14c0f247d1ebab91b1bac965f4a649d19e8c789bdfd81",
+ "files": ["cua-driver"]
+ },
+ "linux-x86_64": {
+ "asset": "cua-driver-rs-0.34.0-linux-x86_64-binary.tar.gz",
+ "sha256": "629ac96eff829d4dfd5cf221f3f2165c2d813aed91e5efb7b20777a741cd70a7",
+ "files": ["cua-driver"]
+ },
+ "linux-arm64": {
+ "asset": "cua-driver-rs-0.34.0-linux-arm64-binary.tar.gz",
+ "sha256": "9db8b9084add57eb97be8164367b24b6be54ed4f3dc01213e64b72d7fc09fddb",
+ "files": ["cua-driver"]
+ },
+ "windows-x86_64": {
+ "asset": "cua-driver-rs-0.34.0-windows-x86_64-binary.zip",
+ "sha256": "bcc520e50861c7092cf775846fec76ae386d7dcd6b5b408608b0ea4423a8b888",
+ "files": ["cua-driver.exe", "cua-driver-uia.exe"]
+ }
+ }
+}
diff --git a/packaging/cua-driver/fetch.mjs b/packaging/cua-driver/fetch.mjs
new file mode 100644
index 00000000..0b3872c4
--- /dev/null
+++ b/packaging/cua-driver/fetch.mjs
@@ -0,0 +1,48 @@
+// Puts the pinned computer-use driver (cua-driver, MIT, https://github.com/trycua/cua) for one
+// target into a directory: node packaging/cua-driver/fetch.mjs <target> <dir> [<license dir>],
+// where <target> is a key of driver.json. The archive is checked against the pinned SHA-256 and
+// cached under build/; only the listed executables are copied, and the license goes beside them
+// unless another directory is given (a Mac bundle's code folders hold only code). Updating the
+// driver: change driver.json (version, tag, assets and their sums from the release's SHA256SUMS).
+import { execFileSync } from 'node:child_process'
+import { createHash } from 'node:crypto'
+import { chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { dirname, join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+const here = dirname(fileURLToPath(import.meta.url))
+const pin = JSON.parse(readFileSync(join(here, 'driver.json'), 'utf8'))
+const [target, out, licenseDir = out] = process.argv.slice(2)
+const spec = pin.targets[target]
+if (!spec || !out) throw new Error(`usage: fetch.mjs <${Object.keys(pin.targets).join('|')}> <dir> [<license dir>]`)
+
+const sha256 = (data) => createHash('sha256').update(data).digest('hex')
+const cache = join(here, '../../build/cua-driver')
+const archive = join(cache, spec.asset)
+if (!existsSync(archive) || sha256(readFileSync(archive)) !== spec.sha256) {
+ const url = `https://github.com/trycua/cua/releases/download/${pin.tag}/${spec.asset}`
+ const res = await fetch(url)
+ if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`)
+ const data = Buffer.from(await res.arrayBuffer())
+ if (sha256(data) !== spec.sha256) throw new Error(`${spec.asset} doesn't match its pinned SHA-256`)
+ mkdirSync(cache, { recursive: true })
+ writeFileSync(archive, data)
+}
+
+const unpacked = mkdtempSync(join(tmpdir(), 'cua-driver-'))
+try {
+ // Windows' own tar (bsdtar) reads zip; a Git Bash tar earlier on PATH doesn't.
+ const tar = process.platform === 'win32' ? join(process.env.SystemRoot ?? 'C:\\Windows', 'System32', 'tar.exe') : 'tar'
+ execFileSync(tar, ['-xf', archive, '-C', unpacked], { stdio: 'inherit' })
+ mkdirSync(out, { recursive: true })
+ for (const file of spec.files) {
+ copyFileSync(join(unpacked, file), join(out, file))
+ chmodSync(join(out, file), 0o755)
+ }
+ mkdirSync(licenseDir, { recursive: true })
+ copyFileSync(join(here, 'LICENSE'), join(licenseDir, 'cua-driver.LICENSE'))
+} finally {
+ rmSync(unpacked, { recursive: true, force: true })
+}
+console.log(`cua-driver ${pin.version} (${target}) → ${out}`)
diff --git a/packaging/cua-driver/tools.mjs b/packaging/cua-driver/tools.mjs
new file mode 100644
index 00000000..f96cf14d
--- /dev/null
+++ b/packaging/cua-driver/tools.mjs
@@ -0,0 +1,72 @@
+// Writes host/src/screen/cua_tools.json: the driver tools bots get, as MCP tools. The host offers
+// exactly these names, reads which ones only look from them, and lists them when the driver can't
+// start. Run after changing driver.json or TOOLS: node packaging/cua-driver/tools.mjs <cua-driver>
+import { spawn } from 'node:child_process'
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
+import { connect } from 'node:net'
+import { tmpdir } from 'node:os'
+import { dirname, join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+// Background-safe GUI work. Left out: bring_to_front and kill_app (take over or end the user's
+// apps), clipboard_read (whatever the user copied), browser_* (each browser asks for Automation),
+// and the driver's own configuration, recording, session and update tools.
+const TOOLS = [
+ 'list_apps', 'list_windows', 'get_window_state', 'get_desktop_state', 'zoom', 'launch_app',
+ 'click', 'double_click', 'right_click', 'drag', 'scroll', 'type_text', 'press_key', 'hotkey',
+ 'set_value', 'invoke_menu',
+]
+
+const [driver] = process.argv.slice(2)
+if (!driver) throw new Error('usage: tools.mjs <path to cua-driver>')
+const dir = mkdtempSync(join(tmpdir(), 'cua-'))
+const socket = join(dir, 'c.sock')
+const daemon = spawn(driver, ['serve', '--socket', socket, '--no-overlay'], {
+ env: { ...process.env, CUA_DRIVER_EMBEDDED: '1', CUA_DRIVER_RS_TELEMETRY_ENABLED: '0', CUA_DRIVER_RS_UPDATE_CHECK: '0' },
+ stdio: 'ignore',
+})
+try {
+ const list = await request(socket, { method: 'list', client_kind: 'cli' })
+ const byName = new Map(list.tools.map((t) => [t.name, t]))
+ const tools = TOOLS.map((name) => {
+ const t = byName.get(name)
+ if (!t) throw new Error(`the driver has no ${name} tool`)
+ return {
+ name,
+ description: t.description,
+ inputSchema: t.input_schema,
+ annotations: { readOnlyHint: t.read_only, destructiveHint: t.destructive, idempotentHint: t.idempotent, openWorldHint: t.open_world },
+ }
+ })
+ const out = join(dirname(fileURLToPath(import.meta.url)), '../../host/src/screen/cua_tools.json')
+ writeFileSync(out, `${JSON.stringify(tools, null, 1)}\n`)
+ console.log(`${tools.length} tools → ${out}`)
+} finally {
+ daemon.kill()
+ rmSync(dir, { recursive: true, force: true })
+}
+
+/** One request on the daemon's line protocol, retried until it listens. */
+async function request(path, body) {
+ for (let i = 0; ; i++) {
+ try {
+ return await new Promise((resolve, reject) => {
+ const s = connect(path)
+ let buf = ''
+ s.on('connect', () => s.write(`${JSON.stringify(body)}\n`))
+ s.on('data', (d) => {
+ buf += d
+ const end = buf.indexOf('\n')
+ if (end < 0) return
+ s.destroy()
+ const reply = JSON.parse(buf.slice(0, end))
+ reply.ok ? resolve(reply.result) : reject(new Error(reply.error))
+ })
+ s.on('error', reject)
+ })
+ } catch (e) {
+ if (i > 50) throw e
+ await new Promise((r) => setTimeout(r, 200))
+ }
+ }
+}
diff --git a/packaging/homebrew/codync-host.rb b/packaging/homebrew/codync-host.rb
index 76c30208..bf5f38eb 100644
--- a/packaging/homebrew/codync-host.rb
+++ b/packaging/homebrew/codync-host.rb
@@ -35,6 +35,8 @@ class CodyncHost < Formula
bin.install "codync-host"
# Linux: the Remote screen helper (needs the system's GStreamer and xdg-desktop-portal).
bin.install "codync-screen" if OS.linux?
+ # Linux: the computer-use driver bots act through (MIT, cua-driver.LICENSE).
+ bin.install "cua-driver" if OS.linux?
end
def caveats
diff --git a/packaging/install.sh b/packaging/install.sh
index a1ebc79b..69174281 100755
--- a/packaging/install.sh
+++ b/packaging/install.sh
@@ -67,6 +67,8 @@ install_host() {
place "$TMP/$name/codync-host" "$BIN_DIR/codync-host"
# Linux: the Remote screen helper, which the host starts from beside itself.
if [ -f "$TMP/$name/codync-screen" ]; then place "$TMP/$name/codync-screen" "$BIN_DIR/codync-screen"; fi
+ # Linux: the computer-use driver bots act through, also started from beside the host.
+ if [ -f "$TMP/$name/cua-driver" ]; then place "$TMP/$name/cua-driver" "$BIN_DIR/cua-driver"; fi
say "Installed $("$BIN_DIR/codync-host" --version) to $BIN_DIR"
restart_service "$BIN_DIR/codync-host"
}