Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

feat(macos): add query, a targeted accessibility lookup (#73)

ATG8fdda42639c9
diff --git a/README.md b/README.md
index 45a5582..d8b764a 100644
--- a/README.md
+++ b/README.md
@@ -181,7 +181,7 @@ evaluation. `ocu repl` keeps them for the current terminal session until
`.exit`, Ctrl-D, or termination; `.reset` clears its bindings. On macOS, the
separate hidden `Open Computer Use.app` permission agent may remain resident so
later commands reuse the same permission identity. `ocu mcp` continues to expose
-the native nine-tool compatibility surface.
+the nine core native tools, plus `query` on macOS.
The npm launcher itself currently requires Node.js 18 or newer. Its help always lists
`js` / `repl`, while `ocu capabilities --json` reports whether Node, the REPL
@@ -190,6 +190,10 @@ the current Node executable instead of looking up another `node` on PATH.
`ocu doctor --json` (macOS only) returns `platform`, `accessibilityTrusted`, `screenCaptureGranted`, `allGranted`, and `missingPermissions`. Missing permissions still exit with code 0; hosts should check `allGranted` and query again after authorization. The snapshot uses existing system API/TCC diagnostics and does not prove an operation succeeded; some permission changes may still require a process restart.
+### Targeted query (macOS)
+
+Find controls by text or role without taking a screenshot. See [query usage](docs/targeted-query.md).
+
## Cursor Motion
Cursor Motion is an open-source cursor motion system for macOS, based on public information shared by members of the Software.Inc team. You can download the app from the [Releases page](https://github.com/iFurySt/open-codex-computer-use/releases).
diff --git a/README.zh-CN.md b/README.zh-CN.md
index d40193c..b2d60a7 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -165,6 +165,10 @@ Cursor Motion 是一个面向 macOS 的开源光标运动系统,基于 Softwar
[MIT](./LICENSE)。
+### 定向查找(macOS)
+
+知道控件文案或角色时,可以用 `app.query()` 查找,不用先截图和读取整棵 AX 树。详见[查询说明](docs/targeted-query.zh-CN.md)。
+
### 持久化配置
使用 `ocu config` 查看生效配置,用 `ocu config set image.format jpg` 保存 macOS 截图偏好。默认文件是 `~/.config/ocu/config.json`,ENV 优先于文件。详见[中文配置说明](docs/configuration.zh-CN.md)([English](docs/configuration.md))。
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 4fef697..65603a4 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -61,15 +61,15 @@
### 3. Tool Service 层
-- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
-- Codex plugin 的 `.mcp.json` 默认不再把这 9 个 tools 直接暴露给模型,而是启动 `scripts/node-repl/open-computer-use-repl.mjs`。adapter 以 child MCP client 连接同一个 native runtime,并只对外列出 `js` / `js_reset`;JavaScript 通过异步 app-bound API(`cua.getApp(...)`、`app.click(...)`、`app.getAXState()` 等)调用 native tools。
+- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 native tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。
+- Codex plugin 的 `.mcp.json` 默认不再把 native tools 直接暴露给模型,而是启动 `scripts/node-repl/open-computer-use-repl.mjs`。adapter 以 child MCP client 连接同一个 native runtime,并只对外列出 `js` / `js_reset`;JavaScript 通过异步 app-bound API(`cua.getApp(...)`、`app.click(...)`、`app.getAXState()` 等)调用 native tools。
- REPL 使用 Node.js 自带的 evaluator,因此保留 top-level `await` 和跨调用 lexical bindings;真正执行代码的 kernel 位于 Worker 中,超时会终止 Worker 并重建干净 session,避免 `while (true)` 挂死 MCP transport。
- npm launcher 还直接提供 `ocu js` 和 `ocu repl`:前者为一次性 Worker/native MCP session,后者在当前终端内保留同一个 session 和 binding;两者退出后都会关闭 Worker 与 native MCP child。`ocu capabilities [--json]` 在不启动 native MCP 的情况下报告 Node、adapter、kernel 和当前平台 native artifact 是否齐全。help 始终显示 code-first 命令并标记 availability,不按环境动态隐藏接口。
- npm bin 当前本身通过 `#!/usr/bin/env node` 启动,因此完全没有 Node 的 shell 无法进入 help/capability 检测;launcher 启动后会复用 `process.execPath`,不会再次从 PATH 解析 Node。若未来要求零 Node 前置,应把最外层入口替换成 native bootstrap 或随包分发 Node。
- 这是增量迁移:`open-computer-use mcp` 仍是原生 9-tool compatibility surface,已有 CLI、其他 MCP host 和 smoke 不需要切换。
- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
- `get_app_state` 优先走真实 AX / 窗口截图;真实 app 必须同时有未最小化的 `AXWindow` 和可匹配的 on-screen `CGWindow`。如果目标 app 只是隐藏或暂时没有 on-screen window,会先 best-effort unhide / activate / `open -b` / `AXRaise` 并短暂重试,以贴近官方 `computer-use` 会把 Lark / Electron 窗口拉回再采集的行为;恢复后仍无法匹配时返回官方风格的 `Apple event error -10005: cgWindowNotFound`,不再把 application 根节点或无截图窗口伪装成可操作状态。当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 默认在 macOS、Linux、Windows 上最多渲染 1200 个节点、64 层深度;显式 `get_app_state` / `snapshot` 可通过 `max_tree_nodes` / `max_tree_depth` 覆盖预算,action tools 的刷新结果仍使用默认预算。snapshot 文本默认截断到 500 字符;显式 `get_app_state` / `snapshot` 可通过 `text_limit` 正整数或 `"max"` 覆盖,action tools 的刷新结果仍使用 500 字符默认值。对 Electron/WebView 这类深层 UI 会压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算;但通用节点中的 `AXPress` / `AXConfirm` / `AXOpen` 子节点会形成文本摘要边界,避免多个可点击选项被合并成一个 container。这类动作节点如果 frame 有效、尺寸紧凑且不包含带 URL 的 `AXLink` 后代,会保留为带窗口相对 `Frame` 的 `button`,并用短文本后代作为按钮摘要,让 icon-only 和文字 Web 控件都能获得可区分的 `element_index`;包含带 URL 的 `AXLink` 后代时保留通用 wrapper 和链接子节点,避免导航链接被摘要吞掉。对原生 open panel / Finder column view 这类把内容放在 `AXContents` / `AXVisibleChildren` 里的控件,也会把可见文件项纳入元素树。
-- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
+- MCP 的 9 个核心 tools 的 description / input schema 按官方 `computer-use` 文案和参数面收敛,macOS 额外提供 `query`,尽量减少 host 侧提示词和 tool surface 偏差。
- JS REPL 的 API 和当前官方 code-first 形态对齐到异步 app binding,而不是把离散 tools 简单包成同步函数。snapshot 的 `windowPlacement` 以及 keyboard action 的 `keyMethod` 会转换成 native tool 的 snake_case 参数。实现不复制或运行 proprietary `@oai/*` package;协议和行为记录在 `docs/references/js-repl.md`。
- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。
- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。
@@ -175,6 +175,13 @@
macOS screenshot 在每轮捕获读取用户 JSON 配置,优先级为 ENV > 文件 > 默认值。npm launcher 提供 `ocu config` 查看、set、reset 和 path;支持 PNG/JPG/WebP、JPEG 质量、最大长边/超限缩小策略、最小总像素数丢弃与 SCK 超时,保留 hardware capture 优先路径。app-agent 按请求转发解析后的配置路径并清除未提供的 image 环境,防止常驻值污染。见[中文配置说明](configuration.zh-CN.md)与[English](configuration.md)。
截图像素丢弃门槛按 width×height 计算,低于(不含等于)门槛不返回 image,但保留 AX 和原因;配置省略截图后拒绝坐标输入。WebP 无损编码通过固定版本 libwebp 静态链接,随 app/npm 制品带上许可文本。
+
+## 定向 AX 查询
+
+macOS 在原有 9 个工具之外新增 `query`,按文字或角色查找目标窗口的 AX 元素,不截图、不渲染完整树。Windows/Linux 保持原来的 9 个工具。查询只解析已运行的应用,使用有界 BFS(默认 500 个节点、最多 5000 个,2 秒协作式遍历截止时间),返回匹配项和完整性元数据。字符串和子节点分页同样有界,AX 读取错误会标记结果不完整。
+
+查询索引属于当前 native 会话,最多保留 5000 个、120 秒过期,结束会话时清理且永不复用。元素操作前核对进程身份、窗口归属、查询条件和最新几何;任何验证失败都要求重新查询,不使用旧坐标。JS `cua.getApp(name, { initialState: false })` 可跳过初始快照,`app.query()` 返回结构化结果。详见[查询说明](targeted-query.zh-CN.md)。
+
## 结构化权限查询
macOS `doctor --json` 在 CLI 和 app-agent proxy 路径复用 `PermissionDiagnostics`,只输出一份 JSON 快照,不打开 onboarding。字段为 `platform: "macos"`、`accessibilityTrusted`、`screenCaptureGranted`、`allGranted` 和 `missingPermissions`(`accessibility` / `screenRecording`)。缺权限仍退出 0,宿主应读取 `allGranted` 并重新查询授权变化。此状态延续系统 API/TCC 合并诊断语义,并非操作成功证明;Linux/Windows 明确拒绝该权限查询模式。普通 `doctor` 保留文本输出与权限引导。
diff --git a/docs/exec-plans/completed/20261010-targeted-ax-query-hardening.md b/docs/exec-plans/completed/20261010-targeted-ax-query-hardening.md
new file mode 100644
index 0000000..01710f4
--- /dev/null
+++ b/docs/exec-plans/completed/20261010-targeted-ax-query-hardening.md
@@ -0,0 +1,27 @@
+# Harden PR #73 targeted AX query
+
+## Scope
+Update the contributor’s existing PR on current main, retaining author history. Integrate persistent screenshot config, hardware/SCK capture, background-window and agent-display behavior. Keep query macOS-only and add its JS entry points.
+
+## Decisions
+- Query resolves running applications only. Exact means exact, with no automatic substring retry.
+- Query indexes are session-local, bound to the process/window, bounded and never reused. Reject wrong-app, stale-window, stale-element and expired indexes before input; refresh geometry without cached-coordinate fallback.
+- Return matches with truncation/stopping metadata. Bound traversal nodes, child fetches/queue, strings and AX call/query time. Use a single bounded traversal for consistent role/text/visibility matching instead of the opaque native predicate path.
+- Add app.query and a lightweight getApp option that does not fetch initial state. Preserve existing getApp behavior by default; document the persistent-session requirement for query then action.
+
+## Progress
+- [x] Merge current main without discarding screenshot/background changes.
+- [x] Harden registry, arguments, window/element validation and traversal.
+- [x] Add JS API and focused regression tests.
+- [x] Update docs/history and complete checks. Follow-up commits are ready for the original #73 branch.
+
+## Validation
+Swift unit tests with injected registry/search inputs; Node mock-native and persistent-session tests; actual invalid window_id subprocess regression; docs/whitespace checks. No app installation, foreground interaction or unrelated PR changes. Live AX latency remains a separate opt-in check; cooperative deadline is not a hard interrupt of an in-flight system AX call.
+
+## Results
+- `swift test`: 207 tests, 7 skipped, zero failures.
+- `node --test scripts/node-repl/*.test.mjs`: 33 tests passed, including persistent query/action, malformed options and response metadata.
+- `make check-docs` and `git diff --check`: passed.
+- Direct native CLI regression with negative window_id: structured tool error and exit 1, no process trap; app-agent proxy disabled for the check.
+- Fresh origin/main and contributor source fetched; current main is integrated and the original contributor head is an ancestor. No force push is needed.
+- Real desktop latency and stale-window behavior have not been exercised interactively. Registry/walk/argument logic is covered with injected tests; the cooperative traversal deadline cannot interrupt an in-flight AX system call.
diff --git a/docs/histories/2026-09/20260915-2100-targeted-ax-query.md b/docs/histories/2026-09/20260915-2100-targeted-ax-query.md
new file mode 100644
index 0000000..1f0ab73
--- /dev/null
+++ b/docs/histories/2026-09/20260915-2100-targeted-ax-query.md
@@ -0,0 +1,57 @@
+## [2026-09-15 21:00] | Task: macOS 新增定向 AX 查找 tool `query`
+
+### 🤖 Execution Context
+* **Agent ID**: `claude-code`
+* **Base Model**: `Claude Opus 5`
+* **Runtime**: `Claude Code CLI / macOS`
+
+### 📥 User Query
+> 把 fork 里已经验证过的定向 AX 查找整理成上游 PR。
+
+### 🛠 Changes Overview
+**Scope:** `packages/OpenComputerUseKit`、`skills/open-computer-use`
+
+**Key Actions:**
+- **[TargetedAX]**: 新增 `TargetedAX`(`Criteria` / `WindowContext` / `SearchResult`)与 `SnapshotBuilder.resolveTargetWindow`、`targetedSearch`、`currentGeometry`。窗口解析是只读的:不激活、不抬升、不截图。
+- **[查找策略]**: 优先调用应用自带的 `AXUIElementsForSearchPredicate`;应用不支持时退回广度优先遍历——边遍历边匹配、凑够 `limit` 立即停止、按窗口自身 AX frame 做几何裁剪、对 table/outline/list 这类大容器只走 `AXVisibleChildren`,每个节点的属性用一次 `AXUIElementCopyMultipleAttributeValues` 批量读取。
+- **[index 注册表]**: `query` 命中的控件登记在 `ComputerUseService` 的定向注册表里,index 从 `1_000_000` 起单调递增、永不复用,因此后续快照不会把同一个 index 重新指到别的控件;上限 5000 条,超出后按先进先出淘汰。
+- **[action 复用]**: `click`、`set_value`、`scroll`、`perform_secondary_action` 改为经 `snapshotForAction` 取快照——遇到定向 index 时用该控件自己的窗口上下文构造 lite snapshot(无树、无截图),其余情况仍走原来的当前快照。
+- **[精确匹配兜底]**: `exact` 未命中且遍历没有被节点上限截断时,自动以子串重试一次,命中的记录标记 `match: "contains"`。
+- **[支撑改动]**: `ElementRecord` 增加带默认值的 `title` / `value`;`WindowCapture` 增加按 `exactWindowID` 的精确绑定与 `capture: false` 的纯几何读取;`SkyLightSPI` 绑定 `_AXUIElementGetWindow`。
+- **[测试]**: 新增 `TargetedAXTests`(纯匹配逻辑,无需真实桌面);tool 数量断言 9 → 10。
+
+### 🧠 Design Intent (Why)
+点一个按钮不该先付一棵完整 AX 树加一张截图的代价。已经知道控件文案时,定向查找只返回匹配项,成本和窗口复杂度基本无关。
+
+只读是这里的硬约束:查找绝不能把前台抢过去,否则“看一眼”本身就改变了被观察的状态。
+
+`index` 单调且不复用,是因为复用是这类注册表最危险的失效方式——一个后台快照把旧 index 指到新控件上,action 会安静地点错东西。
+
+**对上游 tool surface 的影响**:这是在官方 `computer-use` 9 个 tool 之外新增的第 10 个 tool,与 `docs/ARCHITECTURE.md` 里“尽量减少 tool surface 偏差”的取向有冲突,已在该文档中明确记下这处偏离及其理由。Windows / Linux 运行时不受影响,仍是对齐的 9 个。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/SkyLightSPI.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedAXTests.swift`
+- `docs/ARCHITECTURE.md`
+- `skills/open-computer-use/SKILL.md`
+- `skills/open-computer-use/references/usage.md`
+
+## 2026-10-10:PR #73 评审后的补充提交
+
+用户要求按当前主线统一处理评审问题,补充 commits 到原 PR。保留作者原提交,合入 main 的截图配置、hardware/SCK 捕获、后台输入和 agent-display 行为。
+
+- 查询改为单一有界 BFS,限制节点、队列、子节点分页、字符串和 AX 等待时间。返回结构化完整性元数据;精确匹配不再自动改成子串。长字段保留额外边界字符,避免截断制造假精确匹配。
+- 严格验证 query 参数,非法 window_id 返回工具错误,避免浮点转 UInt32 崩溃。只解析已运行应用。
+- 索引绑定进程身份和窗口,增加 120 秒有效期和会话结束清理。操作前验证控件归属、查询条件和最新位置,拒绝失效目标与旧坐标。
+- JS 增加 `app.query()` 与 `getApp(..., { initialState: false })`,默认绑定行为不变。补齐中英文查询文档、README 入口、skill 使用说明及架构和发布记录。
+- 新增 registry、预算、循环、错误、超时、参数校验和持久 JS 查询后操作回归。Swift 207 项测试通过(7 项跳过);Node 测试及文档检查见完成计划。没有安装或重启 app,也未执行真实桌面交互;实际 AX 延迟仍需在真实应用中验证。
+
+主要新增文件为 `TargetedQuery.swift`、`TargetedQuerySafetyTests.swift`、`docs/targeted-query.md` 和 `docs/targeted-query.zh-CN.md`。原记录中的 native predicate 优先路径和 exact 自动兜底描述为初始实现,已由本节方案替代。
+
+## 2026-10-12:同步 main,解决文档冲突
+
+用户反馈 PR #73 再次出现冲突。原因是主线新增 `doctor --json`,与定向查询在 README 和架构文档相邻位置发生冲突。合入主线提交,保留结构化权限查询与定向 AX 查询两份说明;query 实现不变。代码与测试自动合并,重新验证:Swift 208 项测试(7 项跳过)、Node 33 项测试、`make check-docs` 和 whitespace 检查通过。提交推送到原贡献者分支。
diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md
index 4cc6485..7bc0518 100644
--- a/docs/releases/feature-release-notes.md
+++ b/docs/releases/feature-release-notes.md
@@ -4,6 +4,7 @@
| 日期 | 功能域 | 用户价值 | 变更摘要 |
| --- | --- | --- | --- |
+| 2026-10-10 | macOS 定向 AX 查询 | 知道控件文案时,可以直接查询并操作,不必先获取截图和完整 AX tree。 | 新增 native `query` 和 JS `app.query()`,按文字、角色和窗口查找元素,返回完整性元数据;查询索引仅在同一会话内有效,操作前重新验证目标和位置。 |
| 2026-10-10 | macOS 截图配置 | 普通用户可用 `ocu config` 持久化截图偏好,无需维护 shell ENV。 | 新增用户 JSON、ENV 覆盖、PNG/JPG/WebP 格式、明确的超大缩小/丢弃开关、最小总像素数过滤、质量/SCK 超时;取消字节预算缩小,支持关闭长边上限,省略截图时说明原因,保留 #65 捕获路径与坐标映射。 |
| 2026-10-10 | 结构化权限查询 | 宿主可以读取和轮询 macOS 授权状态,无需解析文本或重复弹引导。 | 新增 `ocu doctor --json`,输出权限布尔值、`allGranted` 和缺失权限列表;普通 `doctor` 保留权限引导,Linux/Windows 明确返回不支持。 |
| 2026-10-10 | 1.0.0 正式发布 | 用户可通过 npm 或直接下载使用跨平台桌面自动化,并以持久 JavaScript 会话编排操作。 | 基于当前 main 发布 `1.0.0`:统一所有版本源,交付 macOS 后台键盘、遮挡/其他 Space 窗口读取及显式 agent display,包含 Linux 快捷键与 Windows 无效截图修复;Linux / Windows runtime 继续标为实验性。 |
diff --git a/docs/targeted-query.md b/docs/targeted-query.md
new file mode 100644
index 0000000..5dcd602
--- /dev/null
+++ b/docs/targeted-query.md
@@ -0,0 +1,48 @@
+# Targeted query (macOS)
+
+Find controls by text or role without a screenshot or a full AX tree. The app must already be running. Query does not launch, activate or raise it.
+
+Use a persistent MCP session or `ocu repl`:
+
+```js
+var safari = await cua.getApp("Safari", { initialState: false });
+var result = await safari.query({ text: "Compose", role: "button", exact: true });
+nodeRepl.write(result);
+```
+
+After checking the result, act on its index in the same session:
+
+```js
+if (!result.truncated && result.matches.length === 1) {
+ await safari.click(result.matches[0].index);
+}
+```
+
+`getApp()` still reads initial state by default. Set `initialState: false` to skip it.
+
+For inspection from the CLI:
+
+```sh
+ocu call query --args '{"app":"Safari","text":"Compose","role":"button","exact":true}'
+```
+
+Separate CLI calls cannot share indexes. Handles expire after 120 seconds, with at most 5000 retained in one native session. Query again after reset or session end. Actions revalidate the app process, window, control and current geometry; failed validation never falls back to cached coordinates.
+
+## Options and results
+
+Provide at least one of `text` and `role`. When both are present, both must match. Text matches title, description or value without case sensitivity. Roles accept either `button` or `AXButton`.
+
+| JS option | Native option | Default | Meaning |
+| --- | --- | --- | --- |
+| `text` | `text` | None | Control text, at most 1000 UTF-16 code units |
+| `role` | `role` | None | AX role, at most 1000 UTF-16 code units |
+| `exact` | `exact` | `false` | Match the whole text; otherwise use a substring. An exact miss never retries as a substring |
+| `limit` | `limit` | `20` | Positive result limit, capped at 100 |
+| `maxNodes` | `max_nodes` | `500` | Positive traversal/queue budget, capped at 5000 |
+| `windowId` | `window_id` | Current window | An app window ID, from 1 through 4294967295 |
+
+The result contains `matches`, `truncated`, `stop_reason`, `visited_nodes`, `window_id` and the effective `limit` / `max_nodes`. Each match includes `index` and the available role, title, value, identifier, window-relative bounds and actions.
+
+`truncated: true` means the search is incomplete. An empty `matches` array then does not establish that the control is absent. Reasons are `limit`, `max_nodes`, `timeout`, `ax_error` and `text_limit`; a complete search reports `complete`. Long text is searched only within a bounded prefix, and output text is capped at 1000 characters. Reaching the result limit conservatively marks the search incomplete.
+
+Traversal has a two-second cooperative deadline and per-call AX messaging timeouts. The deadline cannot forcibly interrupt an in-flight system call. Results depend on the app's exposed AX controls; unrendered rows and controls outside the window may be absent. Windows and Linux do not support query yet.
diff --git a/docs/targeted-query.zh-CN.md b/docs/targeted-query.zh-CN.md
new file mode 100644
index 0000000..f30fb30
--- /dev/null
+++ b/docs/targeted-query.zh-CN.md
@@ -0,0 +1,48 @@
+# 定向查找(macOS)
+
+知道控件的文案或角色时,可以直接查找,不用先截图或读取整棵 AX 树。应用需要已经运行,查询不会启动、激活或抬升窗口。
+
+在持久 MCP 会话或 `ocu repl` 中:
+
+```js
+var safari = await cua.getApp("Safari", { initialState: false });
+var result = await safari.query({ text: "Compose", role: "button", exact: true });
+nodeRepl.write(result);
+```
+
+确认结果后,可以在同一会话里操作索引:
+
+```js
+if (!result.truncated && result.matches.length === 1) {
+ await safari.click(result.matches[0].index);
+}
+```
+
+`getApp()` 默认仍会读取初始状态。加上 `initialState: false` 才会跳过。
+
+CLI 也可以单独查看查询结果:
+
+```sh
+ocu call query --args '{"app":"Safari","text":"Compose","role":"button","exact":true}'
+```
+
+不同 CLI 调用不能共享索引。索引只在当前 native 会话有效,120 秒后过期,最多保留 5000 个;重置或结束会话后需要重新查询。操作前会重新检查应用进程、窗口、控件和当前位置,验证失败时不会使用旧坐标。
+
+## 参数与结果
+
+`text` 和 `role` 至少填一个。同时填写时,两者都要匹配。文字不区分大小写,匹配 title、description 或 value;角色可以写 `button` 或 `AXButton`。
+
+| JS 参数 | native 参数 | 默认值 | 含义 |
+| --- | --- | --- | --- |
+| `text` | `text` | 无 | 查找文案,最多 1000 个 UTF-16 编码单元 |
+| `role` | `role` | 无 | AX 角色,最多 1000 个 UTF-16 编码单元 |
+| `exact` | `exact` | `false` | 完整文字匹配;不开则按子串匹配,精确匹配失败不会自动放宽 |
+| `limit` | `limit` | `20` | 最多返回多少项,正整数,上限 100 |
+| `maxNodes` | `max_nodes` | `500` | 最多遍历多少个节点,正整数,上限 5000 |
+| `windowId` | `window_id` | 当前窗口 | 指定应用的窗口 ID,范围 1–4294967295 |
+
+返回对象包含 `matches`、`truncated`、`stop_reason`、`visited_nodes`、`window_id` 和生效的 `limit` / `max_nodes`。每项包含 `index` 及可读取的角色、文案、值、标识、窗口内坐标、动作。
+
+`truncated: true` 表示查询不完整,即使 `matches` 是空数组,也不能认定控件不存在。`stop_reason` 为 `limit`、`max_nodes`、`timeout`、`ax_error` 或 `text_limit`;完整搜索为 `complete`。长文字只搜索开头的有界片段,输出文案最多 1000 字符。达到结果上限时,也会保守标记不完整。
+
+遍历采用 2 秒协作式截止时间,并限制每次 AX 调用的等待时间;正在执行的系统调用不能被这个截止时间强行中断。查询依赖应用暴露的 AX 控件,不保证找到未渲染的列表行或窗口外的控件。Windows / Linux 暂不支持。
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 6c236d7..ae2213c 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -10,6 +10,8 @@ final class ElementRecord {
let element: AXUIElement?
let localFrame: CGRect?
let role: String?
+ let title: String?
+ let value: String?
let rawActions: [String]
let prettyActions: [String]
let isSyntheticText: Bool
@@ -20,6 +22,8 @@ final class ElementRecord {
element: AXUIElement?,
localFrame: CGRect?,
role: String? = nil,
+ title: String? = nil,
+ value: String? = nil,
rawActions: [String],
prettyActions: [String],
isSyntheticText: Bool = false
@@ -29,6 +33,8 @@ final class ElementRecord {
self.element = element
self.localFrame = localFrame
self.role = role
+ self.title = title
+ self.value = value
self.rawActions = rawActions
self.prettyActions = prettyActions
self.isSyntheticText = isSyntheticText
@@ -506,7 +512,7 @@ private struct WindowCapture {
var imageConfig = ImageCaptureConfig.defaults
/// Exact binding: the window the AX tree was walked from.
- static func resolve(for pid: pid_t, exactWindowID: CGWindowID?) -> WindowCapture? {
+ static func resolve(for pid: pid_t, exactWindowID: CGWindowID?, capture: Bool = true) -> WindowCapture? {
guard let exactWindowID,
let infoList = CGWindowListCopyWindowInfo([.optionAll], kCGNullWindowID) as? [[String: Any]]
else {
@@ -522,21 +528,21 @@ private struct WindowCapture {
else {
continue
}
- let config = ImageCaptureConfig.current
- return WindowCapture(windowID: exactWindowID, layer: layer, bounds: bounds, image: captureImage(windowID: exactWindowID, bounds: bounds, config: config), imageConfig: config)
+ let config = capture ? ImageCaptureConfig.current : .defaults
+ return WindowCapture(windowID: exactWindowID, layer: layer, bounds: bounds, image: capture ? captureImage(windowID: exactWindowID, bounds: bounds, config: config) : nil, imageConfig: config)
}
return nil
}
- static func resolve(for pid: pid_t, titleHint: String?) -> WindowCapture? {
+ static func resolve(for pid: pid_t, titleHint: String?, capture: Bool = true) -> WindowCapture? {
// On-screen windows first (front-to-back order is meaningful there);
// fall back to every window of the pid so a window on another Space
// still resolves instead of triggering the activate-and-raise recovery.
- resolve(for: pid, titleHint: titleHint, options: [.optionOnScreenOnly])
- ?? resolve(for: pid, titleHint: titleHint, options: [.optionAll])
+ resolve(for: pid, titleHint: titleHint, options: [.optionOnScreenOnly], capture: capture)
+ ?? resolve(for: pid, titleHint: titleHint, options: [.optionAll], capture: capture)
}
- private static func resolve(for pid: pid_t, titleHint: String?, options: CGWindowListOption) -> WindowCapture? {
+ private static func resolve(for pid: pid_t, titleHint: String?, options: CGWindowListOption, capture: Bool) -> WindowCapture? {
guard let infoList = CGWindowListCopyWindowInfo(options, kCGNullWindowID) as? [[String: Any]] else {
return nil
}
@@ -569,8 +575,8 @@ private struct WindowCapture {
return nil
}
- let config = ImageCaptureConfig.current
- let image = captureImage(windowID: best.windowID, bounds: best.bounds, config: config)
+ let config = capture ? ImageCaptureConfig.current : .defaults
+ let image = capture ? captureImage(windowID: best.windowID, bounds: best.bounds, config: config) : nil
return WindowCapture(windowID: best.windowID, layer: best.layer, bounds: best.bounds, image: image, imageConfig: config)
}
@@ -2275,3 +2281,312 @@ private extension CGRect {
"x=\(Int(origin.x)), y=\(Int(origin.y)), w=\(Int(width)), h=\(Int(height))"
}
}
+
+enum TargetedAX {
+ struct Criteria {
+ var text: String?
+ var exact: Bool
+ var role: String?
+ var limit: Int
+ var maxNodes: Int
+ }
+
+ struct WindowContext {
+ let app: RunningAppDescriptor
+ let appElement: AXUIElement
+ let windowElement: AXUIElement
+ let windowID: CGWindowID?
+ var windowLayer: Int?
+ var windowBounds: CGRect?
+ let focusedElement: AXUIElement?
+ }
+
+ struct SearchResult {
+ let records: [ElementRecord]
+ let truncated: Bool
+ let stopReason: String?
+ let visitedNodes: Int
+ }
+
+}
+
+extension SnapshotBuilder {
+ /// Resolve a target window read-only: no activation, no raise, no screenshot.
+ /// Mirrors the window-finding prologue of `build` without the tree walk or
+ /// capture, so it never redirects input or steals focus. `windowID` selects a
+ /// specific window; otherwise the app's current focused window is used.
+ static func resolveTargetWindow(for app: RunningAppDescriptor, windowID: CGWindowID? = nil) throws -> TargetedAX.WindowContext {
+ if app.name == FixtureBridge.appName {
+ throw ComputerUseError.message("targeted lookup is not supported for fixture apps")
+ }
+
+ guard PermissionDiagnostics.current().accessibilityTrusted else {
+ throw ComputerUseError.permissionDenied("Accessibility permission is required. Run `open-computer-use doctor` and grant access to Open Computer Use.")
+ }
+
+ let appElement = AXUIElementCreateApplication(app.pid)
+ AXUIElementSetMessagingTimeout(appElement, 0.1)
+ _ = enableBestEffortAccessibilityModes(appElement)
+ let systemWide = AXUIElementCreateSystemWide()
+ AXUIElementSetMessagingTimeout(systemWide, 0.1)
+ let focusedApplication = copyElement(systemWide, attribute: kAXFocusedApplicationAttribute)
+
+ let rootWindow: AXUIElement
+ if let windowID, let named = windowElement(for: windowID, appElement: appElement) {
+ rootWindow = named
+ } else if let windowID {
+ throw ComputerUseError.stateUnavailable("window_id \(windowID) is not a current window of \(app.bundleIdentifier ?? app.name).")
+ } else if let focused = preferredFocusedWindow(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide) {
+ rootWindow = focused
+ } else {
+ throw ComputerUseError.stateUnavailable(computerUseNoWindowFoundMessage)
+ }
+
+ AXUIElementSetMessagingTimeout(rootWindow, 0.1)
+ guard let axWindowID = SkyLightSPI.shared.windowID(for: rootWindow),
+ let meta = WindowCapture.resolve(for: app.pid, exactWindowID: axWindowID, capture: false) else {
+ throw ComputerUseError.stateUnavailable("query could not bind this AX window to its current window geometry")
+ }
+ let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
+
+ return TargetedAX.WindowContext(
+ app: app,
+ appElement: appElement,
+ windowElement: rootWindow,
+ windowID: meta.windowID,
+ windowLayer: meta.layer,
+ windowBounds: meta.bounds,
+ focusedElement: focusedElement
+ )
+ }
+
+ /// Current window bounds and element frame for a queried control, so an action
+ /// lands correctly even if the window moved after the query.
+ static func currentGeometry(of record: ElementRecord, in context: TargetedAX.WindowContext,
+ criteria: TargetedAX.Criteria) throws -> (TargetedAX.WindowContext, ElementRecord) {
+ guard !context.app.runningApplication.isTerminated, let element = record.element else {
+ throw ComputerUseError.stateUnavailable("Query target is no longer available; run query again")
+ }
+ let freshContext = try resolveTargetWindow(for: context.app, windowID: context.windowID)
+ AXUIElementSetMessagingTimeout(element, 0.1)
+ guard belongsToWindow(element, window: freshContext.windowElement),
+ let bounds = freshContext.windowBounds else {
+ throw ComputerUseError.stateUnavailable("Query target no longer belongs to its window; run query again")
+ }
+ let scan = batchScan(element)
+ guard scan.role == record.role,
+ displayIdentifier(scan.identifier) == record.identifier,
+ targetedRecordMatches(criteria, role: scan.role, title: scan.title, description: scan.description, value: scan.value),
+ let frame = scan.globalFrame, !frame.isEmpty, frame.intersects(bounds) else {
+ throw ComputerUseError.stateUnavailable("Query target changed or its current position is unavailable; run query again")
+ }
+ let fresh = makeRecord(element, scan: scan, windowBounds: bounds, index: record.index)
+ return (freshContext, fresh)
+ }
+
+ private static func belongsToWindow(_ element: AXUIElement, window: AXUIElement) -> Bool {
+ if CFEqual(element, window) { return true }
+ if let owner = copyElement(element, attribute: kAXWindowAttribute) { return CFEqual(owner, window) }
+ var current = element
+ let deadline = ProcessInfo.processInfo.systemUptime + 1
+ for _ in 0..<64 {
+ guard ProcessInfo.processInfo.systemUptime < deadline else { return false }
+ AXUIElementSetMessagingTimeout(current, 0.05)
+ guard let parent = copyElement(current, attribute: kAXParentAttribute), !CFEqual(parent, current) else { return false }
+ if CFEqual(parent, window) { return true }
+ current = parent
+ }
+ return false
+ }
+
+ private static func windowElement(for windowID: CGWindowID, appElement: AXUIElement) -> AXUIElement? {
+ copyArray(appElement, attribute: kAXWindowsAttribute)?.first {
+ SkyLightSPI.shared.windowID(for: $0) == windowID
+ }
+ }
+
+ // Attributes read for every visited node, in one batched IPC call. Position
+ // and size ride along so we can (a) skip descending into off-screen subtrees
+ // and (b) fill the record's frame without another round trip.
+ private static let searchScanAttributes: [String] = [
+ kAXRoleAttribute as String,
+ kAXTitleAttribute as String,
+ kAXDescriptionAttribute as String,
+ kAXValueAttribute as String,
+ kAXIdentifierAttribute as String,
+ kAXPositionAttribute as String,
+ kAXSizeAttribute as String,
+ ]
+
+ /// Use one bounded traversal so every app has the same visibility and match rules.
+ static func targetedSearch(_ criteria: TargetedAX.Criteria, in context: TargetedAX.WindowContext) throws -> TargetedAX.SearchResult {
+ let deadline = ProcessInfo.processInfo.systemUptime + 2
+ let expired = { ProcessInfo.processInfo.systemUptime >= deadline }
+ var scans: [AXSearchNode: NodeScan] = [:]
+ var records: [AXSearchNode: ElementRecord] = [:]
+ var textTruncated = false
+ var clip: CGRect?
+ let root = AXSearchNode(element: context.windowElement)
+ let result = try boundedTargetedWalk(root: root, maxNodes: criteria.maxNodes, limit: criteria.limit,
+ expired: expired, inspect: { node in
+ AXUIElementSetMessagingTimeout(node.element, 0.05)
+ let scan = batchScan(node.element)
+ guard scan.role != nil else { throw ComputerUseError.stateUnavailable("AX node could not be read") }
+ scans[node] = scan
+ if node == root { clip = scan.globalFrame }
+ let visible = scan.globalFrame.map { frame in frame.isEmpty || clip.map { frame.intersects($0) } ?? true } ?? true
+ if visible, criteria.text != nil, [scan.title, scan.description, scan.value].contains(where: { ($0?.count ?? 0) > 1000 }) { textTruncated = true }
+ let matches = visible && targetedRecordMatches(criteria, role: scan.role, title: scan.title, description: scan.description, value: scan.value)
+ if matches { records[node] = makeRecord(node.element, scan: scan, windowBounds: context.windowBounds) }
+ return (matches, visible)
+ }, children: { node, remaining in
+ guard !expired() else { return ([], true) }
+ let role = scans[node]?.role
+ let attributes = role.map { largeContainerRoles.contains($0) } == true
+ ? [axVisibleChildrenAttribute, kAXChildrenAttribute as String]
+ : [kAXChildrenAttribute as String, "AXContents"]
+ for attribute in attributes {
+ var count: CFIndex = 0
+ let error = AXUIElementGetAttributeValueCount(node.element, attribute as CFString, &count)
+ if error == .attributeUnsupported || error == .noValue { continue }
+ guard error == .success else { throw ComputerUseError.stateUnavailable("AX child count could not be read") }
+ guard count > 0, remaining > 0 else { return ([], count > 0) }
+ var values: CFArray?
+ guard AXUIElementCopyAttributeValues(node.element, attribute as CFString, 0, min(count, remaining), &values) == .success,
+ let elements = values as? [AXUIElement] else {
+ throw ComputerUseError.stateUnavailable("AX children could not be read")
+ }
+ return (elements.map { AXSearchNode(element: $0) }, count > remaining)
+ }
+ return ([], false)
+ })
+ let reason = result.stopReason ?? (textTruncated ? "text_limit" : nil)
+ return .init(records: result.matches.compactMap { records[$0] }, truncated: reason != nil, stopReason: reason, visitedNodes: result.visitedNodes)
+ }
+
+ private struct AXSearchNode: Hashable {
+ let element: AXUIElement
+ static func == (lhs: Self, rhs: Self) -> Bool { CFEqual(lhs.element, rhs.element) }
+ func hash(into hasher: inout Hasher) { hasher.combine(CFHash(element)) }
+ }
+
+ private static let largeContainerRoles: Set<String> = [
+ kAXTableRole as String, kAXOutlineRole as String, kAXListRole as String,
+ "AXBrowser", "AXCollection", "AXGrid",
+ ]
+
+ private struct NodeScan {
+ let role: String?
+ let title: String?
+ let description: String?
+ let value: String?
+ let identifier: String?
+ let globalFrame: CGRect?
+ }
+
+ /// Read the scan attributes in a single `AXUIElementCopyMultipleAttributeValues`
+ /// IPC call (falls back to individual reads if the batch call is unsupported).
+ private static func batchScan(_ element: AXUIElement) -> NodeScan {
+ var values: CFArray?
+ let error = AXUIElementCopyMultipleAttributeValues(
+ element,
+ searchScanAttributes as CFArray,
+ AXCopyMultipleAttributeOptions(),
+ &values
+ )
+ let raw: [AnyObject]
+ if error == .success, let array = values as? [AnyObject], array.count == searchScanAttributes.count {
+ raw = array
+ } else {
+ raw = searchScanAttributes.map { attribute -> AnyObject in
+ var v: CFTypeRef?
+ _ = AXUIElementCopyAttributeValue(element, attribute as CFString, &v)
+ return v ?? (NSNull() as AnyObject)
+ }
+ }
+ func str(_ i: Int) -> String? { (raw[i] as? String).map { String($0.prefix(1001)) } }
+ return NodeScan(
+ role: str(0), title: str(1), description: str(2), value: str(3), identifier: str(4),
+ globalFrame: axFrame(position: raw[5], size: raw[6])
+ )
+ }
+
+ private static func axFrame(position: AnyObject?, size: AnyObject?) -> CGRect? {
+ guard let position, let size,
+ CFGetTypeID(position as CFTypeRef) == AXValueGetTypeID(),
+ CFGetTypeID(size as CFTypeRef) == AXValueGetTypeID() else {
+ return nil
+ }
+ var point = CGPoint.zero
+ var extent = CGSize.zero
+ guard AXValueGetValue(position as! AXValue, .cgPoint, &point),
+ AXValueGetValue(size as! AXValue, .cgSize, &extent),
+ point.x.isFinite, point.y.isFinite, extent.width.isFinite, extent.height.isFinite,
+ extent.width >= 0, extent.height >= 0 else {
+ return nil
+ }
+ return CGRect(origin: point, size: extent)
+ }
+
+ private static func makeRecord(_ element: AXUIElement, scan: NodeScan, windowBounds: CGRect?, index: Int = 0) -> ElementRecord {
+ let rawActions = copyActions(element) ?? []
+ // Reuse the frame from the batched scan; window-relative when we have bounds.
+ let localFrame: CGRect?
+ if let frame = scan.globalFrame {
+ localFrame = windowBounds.map { windowRelativeFrame(elementFrame: frame, windowBounds: $0) } ?? frame
+ } else {
+ localFrame = resolveLocalFrame(of: element, windowBounds: windowBounds)
+ }
+ return ElementRecord(
+ index: index, // reassigned by the caller when registered
+ identifier: displayIdentifier(scan.identifier),
+ element: element,
+ localFrame: localFrame,
+ role: scan.role,
+ title: (scan.title ?? scan.description).map { String($0.prefix(1000)) },
+ value: scan.value.map { String($0.prefix(1000)) },
+ rawActions: rawActions,
+ prettyActions: scan.role.map { meaningfulActions(rawActions, role: $0) } ?? rawActions
+ )
+ }
+
+
+}
+
+/// Pure record filter, factored out so it is unit-testable without a live
+/// accessibility tree. `text` matches title/description/value (substring, or a
+/// full match when `exact`); `role` matches the AX role (the AX prefix optional).
+func targetedRecordMatches(
+ _ criteria: TargetedAX.Criteria,
+ role: String?,
+ title: String?,
+ description: String?,
+ value: String?
+) -> Bool {
+ if let wantedRole = criteria.role, !targetedRoleEquals(role, wantedRole) {
+ return false
+ }
+ if let wantedText = criteria.text, !wantedText.isEmpty {
+ let fields = [title, description, value]
+ let hit = fields.contains { field in
+ guard let field else { return false }
+ if criteria.exact {
+ return field.caseInsensitiveCompare(wantedText) == .orderedSame
+ }
+ return field.range(of: wantedText, options: .caseInsensitive) != nil
+ }
+ if !hit { return false }
+ }
+ return true
+}
+
+/// Role match tolerant of the `AX` prefix ("button" matches "AXButton").
+func targetedRoleEquals(_ actual: String?, _ wanted: String) -> Bool {
+ guard let actual else { return false }
+ func normalize(_ s: String) -> String {
+ var r = s.trimmingCharacters(in: .whitespaces).lowercased()
+ if r.hasPrefix("ax") { r.removeFirst(2) }
+ return r
+ }
+ return normalize(actual) == normalize(wanted)
+}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
index 88a533a..ba52bc9 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
@@ -141,6 +141,17 @@ enum AppDiscovery {
}
}
+ static func resolveRunning(_ query: String, applications: [RunningAppDescriptor]? = nil) throws -> RunningAppDescriptor {
+ let query = query.trimmingCharacters(in: .whitespacesAndNewlines)
+ if let bundleIdentifier = blockedBundleIdentifier(forQuery: query) {
+ throw AppSafetyPolicy.permissionDenied(bundleIdentifier: bundleIdentifier)
+ }
+ guard let app = resolvedRunningApp(in: applications ?? runningApps(), matching: query) else {
+ throw ComputerUseError.appNotFound(query)
+ }
+ return app
+ }
+
static func resolve(_ query: String) throws -> RunningAppDescriptor {
let normalizedQuery = query.trimmingCharacters(in: .whitespacesAndNewlines)
let running = runningApps()
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index b0779cd..f6aca66 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -490,6 +490,20 @@ public final class ComputerUseService {
// for the duration of each request.
private var actionReadBack: Bool { actionReadBackEnabled(environment: ProcessInfo.processInfo.environment) }
+ // Targeted-lookup registry. `query` stores each match here keyed by a high,
+ // monotonic element index; the existing element actions resolve that index to
+ // the queried control and act without any snapshot. Indexes sit above the
+ // snapshot tree range and are never reissued, so a later snapshot cannot
+ // reassign one to a different control.
+ private struct TargetedElement {
+ let record: ElementRecord
+ let context: TargetedAX.WindowContext
+ let criteria: TargetedAX.Criteria
+ }
+ private let targetedElements = TargetedElementRegistry<TargetedElement>()
+
+ public func clearQueryIndexes() { targetedElements.clear() }
+
public init() {}
/// An action's result reads the window back, so the app gets a beat to redraw
@@ -570,7 +584,7 @@ public final class ComputerUseService {
clickCount: clickCount
)
- let snapshot = try currentSnapshot(for: query)
+ let snapshot = try snapshotForAction(app: query, elementIndex: elementIndex)
let button = MouseButtonKind(rawValue: mouseButton.lowercased()) ?? .left
if snapshot.mode == .fixture {
guard clickMethod == .auto else {
@@ -732,7 +746,7 @@ public final class ComputerUseService {
}
public func performSecondaryAction(app query: String, elementIndex: String, action: String) throws -> ToolCallResult {
- let snapshot = try currentSnapshot(for: query)
+ let snapshot = try snapshotForAction(app: query, elementIndex: elementIndex)
let record = try lookupElement(snapshot: snapshot, index: elementIndex)
if snapshot.mode == .fixture {
@@ -769,7 +783,7 @@ public final class ComputerUseService {
throw ComputerUseError.message("pages must be > 0")
}
- let snapshot = try currentSnapshot(for: query)
+ let snapshot = try snapshotForAction(app: query, elementIndex: elementIndex)
let record = try lookupElement(snapshot: snapshot, index: elementIndex)
if snapshot.mode == .fixture {
@@ -897,7 +911,7 @@ public final class ComputerUseService {
}
public func setValue(app query: String, elementIndex: String, value: String) throws -> ToolCallResult {
- let snapshot = try currentSnapshot(for: query)
+ let snapshot = try snapshotForAction(app: query, elementIndex: elementIndex)
let record = try lookupElement(snapshot: snapshot, index: elementIndex)
if snapshot.mode == .fixture {
@@ -940,6 +954,101 @@ public final class ComputerUseService {
return try actionResult(for: query)
}
+ /// Find controls in the app's current window by text and/or role, using the
+ /// app's own accessibility search where it offers one. No tree render, no
+ /// screenshot. Each result carries an `index` the existing element actions
+ /// (click, set_value, scroll, perform_secondary_action) accept.
+ public func query(
+ app query: String,
+ text: String? = nil,
+ role: String? = nil,
+ exact: Bool = false,
+ limit: Int = 20,
+ maxNodes: Int = 500,
+ windowID: CGWindowID? = nil
+ ) throws -> [String: Any] {
+ guard (text.map { !$0.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty } ?? false) || (role.map { !$0.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty } ?? false) else {
+ throw ComputerUseError.invalidArguments("query requires at least one of text or role")
+ }
+
+ guard text.map({ $0.utf16.count <= 1000 }) ?? true, role.map({ $0.utf16.count <= 1000 }) ?? true else {
+ throw ComputerUseError.invalidArguments("query text and role must be at most 1000 UTF-16 code units")
+ }
+ let app = try AppDiscovery.resolveRunning(query)
+ let context = try SnapshotBuilder.resolveTargetWindow(for: app, windowID: windowID)
+ let criteria = TargetedAX.Criteria(text: text, exact: exact, role: role,
+ limit: min(100, max(1, limit)), maxNodes: min(5000, max(1, maxNodes)))
+ let result = try SnapshotBuilder.targetedSearch(criteria, in: context)
+ let matches = result.records.map { registerTargetedElement(record: $0, context: context, criteria: criteria) }
+ return ["matches": matches, "truncated": result.truncated,
+ "stop_reason": result.stopReason ?? "complete", "visited_nodes": result.visitedNodes,
+ "window_id": context.windowID.map { Int($0) } ?? 0,
+ "max_nodes": criteria.maxNodes, "limit": criteria.limit]
+ }
+
+ /// A snapshot carrying real window context but no rendered tree or screenshot,
+ /// so the existing action internals run unchanged against a queried element.
+ private func liteSnapshot(context: TargetedAX.WindowContext, elements: [Int: ElementRecord]) -> AppSnapshot {
+ AppSnapshot(
+ app: context.app,
+ windowTitle: nil,
+ windowBounds: context.windowBounds,
+ targetWindowID: context.windowID,
+ targetWindowLayer: context.windowLayer,
+ screenshotData: nil,
+ mode: .accessibility,
+ treeLines: [],
+ focusedSummary: nil,
+ focusedElement: context.focusedElement,
+ selectedText: nil,
+ windowElement: context.windowElement,
+ elements: elements
+ )
+ }
+
+ /// The snapshot an element action runs against. A queried index resolves to
+ /// that control's own window context with no snapshot; any other index uses
+ /// the normal current snapshot.
+ private func snapshotForAction(app query: String, elementIndex: String?) throws -> AppSnapshot {
+ if let elementIndex, let index = Int(elementIndex), index > TargetedElementRegistry<TargetedElement>.indexBase {
+ let app = try AppDiscovery.resolveRunning(query)
+ let target = try targetedElements.resolve(index, app: TargetedAppIdentity(app: app))
+ do {
+ let (context, record) = try SnapshotBuilder.currentGeometry(of: target.record, in: target.context, criteria: target.criteria)
+ return liteSnapshot(context: context, elements: [index: record])
+ } catch {
+ targetedElements.remove(index)
+ throw error
+ }
+ }
+ return try currentSnapshot(for: query)
+ }
+
+ private func registerTargetedElement(record source: ElementRecord, context: TargetedAX.WindowContext,
+ criteria: TargetedAX.Criteria) -> [String: Any] {
+ var record: ElementRecord!
+ _ = targetedElements.insert(app: TargetedAppIdentity(app: context.app)) { index in
+ record = ElementRecord(index: index, identifier: source.identifier, element: source.element,
+ localFrame: source.localFrame, role: source.role, title: source.title, value: source.value,
+ rawActions: source.rawActions, prettyActions: source.prettyActions, isSyntheticText: source.isSyntheticText)
+ return TargetedElement(record: record, context: context, criteria: criteria)
+ }
+ return compactRecordDictionary(record)
+ }
+
+ private func compactRecordDictionary(_ record: ElementRecord) -> [String: Any] {
+ var dict: [String: Any] = ["index": record.index]
+ if let role = record.role, !role.isEmpty { dict["role"] = role }
+ if let title = record.title, !title.isEmpty { dict["title"] = title }
+ if let value = record.value, !value.isEmpty { dict["value"] = value }
+ if let identifier = record.identifier, !identifier.isEmpty { dict["identifier"] = identifier }
+ if let frame = record.localFrame {
+ dict["bounds"] = ["x": frame.origin.x, "y": frame.origin.y, "w": frame.size.width, "h": frame.size.height]
+ }
+ if !record.prettyActions.isEmpty { dict["actions"] = record.prettyActions }
+ return dict
+ }
+
private func currentSnapshot(for query: String) throws -> AppSnapshot {
if let snapshot = snapshotsByApp[query.lowercased()] {
return snapshot
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift
index 23a6b12..6afbc8a 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift
@@ -1,3 +1,4 @@
+import CoreGraphics
import Foundation
func normalizedElementIndexArgument(_ value: Any?) -> String? {
@@ -47,6 +48,17 @@ public final class ComputerUseToolDispatcher {
switch name {
case "list_apps":
return service.listApps()
+ case "query":
+ let matches = try service.query(
+ app: requireString("app", in: arguments),
+ text: try validatedQueryString(arguments["text"], key: "text"),
+ role: try validatedQueryString(arguments["role"], key: "role"),
+ exact: try validatedQueryExact(arguments["exact"]),
+ limit: try optionalPositiveInt("limit", in: arguments) ?? 20,
+ maxNodes: try optionalPositiveInt("max_nodes", in: arguments) ?? 500,
+ windowID: try validatedQueryWindowID(arguments["window_id"])
+ )
+ return .text(try renderedQueryMatches(matches))
case "get_app_state":
return try service.getAppState(
app: requireString("app", in: arguments),
@@ -133,6 +145,24 @@ public final class ComputerUseToolDispatcher {
return value
}
+ private func optionalBool(_ key: String, in arguments: [String: Any]) -> Bool? {
+ switch arguments[key] {
+ case let value as Bool: return value
+ case let value as NSNumber: return value.boolValue
+ case let value as String: return Bool(value.lowercased())
+ default: return nil
+ }
+ }
+
+ /// Machine-readable matches with explicit completeness metadata.
+ private func renderedQueryMatches(_ matches: [String: Any]) throws -> String {
+ let data = try JSONSerialization.data(withJSONObject: matches, options: [.sortedKeys])
+ guard let text = String(data: data, encoding: .utf8) else {
+ throw ComputerUseError.stateUnavailable("Query response could not be encoded")
+ }
+ return text
+ }
+
private func optionalString(_ key: String, in arguments: [String: Any]) -> String? {
arguments[key] as? String
}
@@ -407,3 +437,29 @@ private func decodeOpenComputerUseJSONObject(_ source: String) throws -> Any {
)
}
}
+
+func validatedQueryWindowID(_ value: Any?) throws -> CGWindowID? {
+ guard let value else { return nil }
+ guard let number = value as? NSNumber, CFGetTypeID(number) != CFBooleanGetTypeID(),
+ number.doubleValue.isFinite, number.doubleValue >= 1,
+ number.doubleValue <= Double(UInt32.max), number.doubleValue.rounded(.down) == number.doubleValue else {
+ throw ComputerUseError.invalidArguments("window_id must be an integer from 1 to \(UInt32.max)")
+ }
+ return CGWindowID(number.doubleValue)
+}
+
+func validatedQueryString(_ value: Any?, key: String) throws -> String? {
+ guard let value else { return nil }
+ guard let string = value as? String, string.utf16.count <= 1000 else {
+ throw ComputerUseError.invalidArguments("\(key) must be a string of at most 1000 UTF-16 code units")
+ }
+ return string
+}
+
+func validatedQueryExact(_ value: Any?) throws -> Bool {
+ guard let value else { return false }
+ guard let number = value as? NSNumber, CFGetTypeID(number) == CFBooleanGetTypeID() else {
+ throw ComputerUseError.invalidArguments("exact must be true or false")
+ }
+ return number.boolValue
+}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
index dc7c873..e592e41 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
@@ -18,6 +18,7 @@ Ask the user before taking destructive or externally visible actions such as sen
"""
public final class StdioMCPServer {
+ private let service: ComputerUseService
private let dispatcher: ComputerUseToolDispatcher
private let backgroundStateReset: () -> Void
@@ -25,6 +26,7 @@ public final class StdioMCPServer {
service: ComputerUseService = ComputerUseService(),
backgroundStateReset: @escaping () -> Void = resetOpenComputerUseBackgroundWindowState
) {
+ self.service = service
self.dispatcher = ComputerUseToolDispatcher(service: service)
self.backgroundStateReset = backgroundStateReset
}
@@ -127,6 +129,7 @@ public final class StdioMCPServer {
/// connections call this explicitly because they use `handle(line:)`
/// instead of the stdio `run()` loop.
public func endSession() {
+ service.clearQueryIndexes()
VisualCursorSupport.performOnMain {
SoftwareCursorOverlay.reset()
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TargetedQuery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TargetedQuery.swift
new file mode 100644
index 0000000..b58381d
--- /dev/null
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/TargetedQuery.swift
@@ -0,0 +1,108 @@
+import Foundation
+
+struct TargetedAppIdentity: Equatable {
+ let pid: Int32
+ let bundleIdentifier: String?
+ let launchedAt: Date?
+
+ init(app: RunningAppDescriptor) {
+ pid = app.pid
+ bundleIdentifier = app.bundleIdentifier
+ launchedAt = app.runningApplication.launchDate
+ }
+
+ init(pid: Int32, bundleIdentifier: String?, launchedAt: Date?) {
+ self.pid = pid
+ self.bundleIdentifier = bundleIdentifier
+ self.launchedAt = launchedAt
+ }
+}
+
+/// Session-local handles. A cleared/evicted handle is never assigned again.
+final class TargetedElementRegistry<Value> {
+ static var indexBase: Int { 1_000_000 }
+ private var nextIndex = indexBase
+ private let capacity: Int
+ private let lifetime: TimeInterval
+ private let now: () -> Date
+ private struct Entry {
+ let app: TargetedAppIdentity
+ let value: Value
+ let expires: Date
+ }
+ private var entries: [Int: Entry] = [:]
+ private var order: [Int] = []
+
+ init(capacity: Int = 5000, lifetime: TimeInterval = 120, now: @escaping () -> Date = Date.init) {
+ self.capacity = max(1, capacity)
+ self.lifetime = lifetime
+ self.now = now
+ }
+
+ func insert(app: TargetedAppIdentity, value: (Int) -> Value) -> Int {
+ nextIndex += 1
+ let index = nextIndex
+ entries[index] = Entry(app: app, value: value(index), expires: now().addingTimeInterval(lifetime))
+ order.append(index)
+ if order.count > capacity { entries.removeValue(forKey: order.removeFirst()) }
+ return index
+ }
+
+ func resolve(_ index: Int, app: TargetedAppIdentity) throws -> Value {
+ guard let entry = entries[index], now() < entry.expires else {
+ entries.removeValue(forKey: index)
+ throw ComputerUseError.stateUnavailable("Query index \(index) has expired or is no longer available; run query again in this session")
+ }
+ guard entry.app == app else {
+ throw ComputerUseError.invalidArguments("Query index \(index) belongs to a different application process; run query for the requested app")
+ }
+ return entry.value
+ }
+
+ func remove(_ index: Int) { entries.removeValue(forKey: index) }
+ func clear() { entries.removeAll(); order.removeAll() }
+}
+
+struct TargetedWalkResult<Node> {
+ var matches: [Node] = []
+ var visitedNodes = 0
+ var stopReason: String?
+ var truncated: Bool { stopReason != nil }
+}
+
+/// A bounded queue and child page keep wide AX containers within the node budget.
+/// Dependencies are injectable so budget/timeout/cycle behavior can be tested without a desktop.
+func boundedTargetedWalk<Node: Hashable>(
+ root: Node, maxNodes: Int, limit: Int,
+ expired: () -> Bool,
+ inspect: (Node) throws -> (matches: Bool, descend: Bool),
+ children: (Node, Int) throws -> (nodes: [Node], truncated: Bool)
+) throws -> TargetedWalkResult<Node> {
+ var result = TargetedWalkResult<Node>()
+ var queue = [root]
+ var seen: Set<Node> = [root]
+ var head = 0
+ while head < queue.count {
+ if expired() { result.stopReason = "timeout"; break }
+ if result.visitedNodes >= maxNodes { result.stopReason = "max_nodes"; break }
+ let node = queue[head]
+ head += 1
+ result.visitedNodes += 1
+ let scan: (matches: Bool, descend: Bool)
+ do { scan = try inspect(node) } catch { result.stopReason = "ax_error"; break }
+ if expired() { result.stopReason = "timeout"; break }
+ if scan.matches {
+ result.matches.append(node)
+ if result.matches.count >= limit { result.stopReason = "limit"; break }
+ }
+ if scan.descend {
+ let remaining = max(0, maxNodes - seen.count)
+ let page: (nodes: [Node], truncated: Bool)
+ do { page = try children(node, remaining) } catch { result.stopReason = "ax_error"; break }
+ if expired() { result.stopReason = "timeout"; break }
+ if page.truncated || page.nodes.count > remaining { result.stopReason = "max_nodes" }
+ for child in page.nodes.prefix(remaining) where seen.insert(child).inserted { queue.append(child) }
+ }
+ }
+ return result
+}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
index 11a8f97..e8d0c51 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
@@ -68,6 +68,26 @@ public enum ToolDefinitions {
required: ["app", "from_x", "from_y", "to_x", "to_y"]
)
),
+ ToolDefinition(
+ name: "query",
+ description: "Targeted accessibility lookup: find controls in the app's current window using bounded AX traversal, with no full snapshot and no screenshot. Returns matches with truncated, stop_reason, visited_nodes and window_id metadata. Each match has a session-local `index` usable by the element actions (click, set_value, scroll, perform_secondary_action). Requires text and/or role. Only queries already-running apps; indexes expire after 120 seconds. This tool is part of plugin `Computer Use`.",
+ annotations: readOnlyAnnotations(),
+ inputSchema: objectSchema(
+ properties: [
+ "app": stringProperty(description: "App name or bundle identifier"),
+ "text": stringProperty(description: "Match text in the control's title, description or value (up to 1000 UTF-16 code units). Case-insensitive substring unless exact is true. Long fields are bounded and reported as truncated."),
+ "role": stringProperty(description: "Match this accessibility role (the AX prefix is optional, e.g. button or AXButton)"),
+ "exact": [
+ "type": "boolean",
+ "description": "Require a complete text match instead of a substring. Defaults to false. No substring retry.",
+ ],
+ "limit": positiveIntegerProperty(description: "Maximum matches to return. Defaults to 20; capped at 100."),
+ "max_nodes": positiveIntegerProperty(description: "Node/queue cap for AX traversal. Defaults to 500; capped at 5000. Search has a 2-second cooperative deadline."),
+ "window_id": positiveIntegerProperty(description: "Search this specific window (CGWindowID, 1–4294967295) instead of the app's current window."),
+ ],
+ required: ["app"]
+ )
+ ),
ToolDefinition(
name: "get_app_state",
description: "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 8388001..60cdfb7 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -279,7 +279,7 @@ final class OpenComputerUseKitTests: XCTestCase {
}
func testToolDefinitionCount() {
- XCTAssertEqual(ToolDefinitions.all.count, 9)
+ XCTAssertEqual(ToolDefinitions.all.count, 10)
}
func testReadToolArgumentsAcceptsJSONObject() throws {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedAXTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedAXTests.swift
new file mode 100644
index 0000000..09c5214
--- /dev/null
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedAXTests.swift
@@ -0,0 +1,50 @@
+import XCTest
+@testable import OpenComputerUseKit
+
+/// Pure record-filter tests for the targeted lookup. The native AX search and
+/// window resolution need separate live desktop validation; the
+/// criteria matching is pure string logic and verified here.
+final class TargetedAXTests: XCTestCase {
+ private func criteria(
+ text: String? = nil,
+ exact: Bool = false,
+ role: String? = nil
+ ) -> TargetedAX.Criteria {
+ TargetedAX.Criteria(text: text, exact: exact, role: role, limit: 20, maxNodes: 500)
+ }
+
+ func testRoleEqualsToleratesAXPrefix() {
+ XCTAssertTrue(targetedRoleEquals("AXButton", "button"))
+ XCTAssertTrue(targetedRoleEquals("AXButton", "AXButton"))
+ XCTAssertTrue(targetedRoleEquals("button", "AXButton"))
+ XCTAssertFalse(targetedRoleEquals("AXButton", "textfield"))
+ XCTAssertFalse(targetedRoleEquals(nil, "button"))
+ }
+
+ func testTextMatchesTitleDescriptionOrValueSubstring() {
+ let c = criteria(text: "compose")
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXButton", title: "Compose", description: nil, value: nil))
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXButton", title: nil, description: "Compose a message", value: nil))
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXTextField", title: nil, description: nil, value: "re: compose"))
+ XCTAssertFalse(targetedRecordMatches(c, role: "AXButton", title: "Send", description: nil, value: "hi"))
+ }
+
+ func testExactRequiresFullMatch() {
+ let c = criteria(text: "Compose", exact: true)
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXButton", title: "compose", description: nil, value: nil))
+ XCTAssertFalse(targetedRecordMatches(c, role: "AXButton", title: "Compose mail", description: nil, value: nil))
+ }
+
+ func testRoleFilterCombinesWithText() {
+ let c = criteria(text: "send", role: "button")
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXButton", title: "Send", description: nil, value: nil))
+ // right text, wrong role
+ XCTAssertFalse(targetedRecordMatches(c, role: "AXStaticText", title: "Send", description: nil, value: nil))
+ }
+
+ func testRoleOnlyMatchesAnyText() {
+ let c = criteria(role: "AXButton")
+ XCTAssertTrue(targetedRecordMatches(c, role: "AXButton", title: "anything", description: nil, value: nil))
+ XCTAssertFalse(targetedRecordMatches(c, role: "AXTextField", title: "anything", description: nil, value: nil))
+ }
+}
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedQuerySafetyTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedQuerySafetyTests.swift
new file mode 100644
index 0000000..63dddb3
--- /dev/null
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/TargetedQuerySafetyTests.swift
@@ -0,0 +1,118 @@
+import AppKit
+import XCTest
+@testable import OpenComputerUseKit
+
+final class TargetedQuerySafetyTests: XCTestCase {
+ func testWindowIDValidationRejectsCrashInputsAndWrongTypes() throws {
+ XCTAssertNil(try validatedQueryWindowID(nil))
+ XCTAssertEqual(try validatedQueryWindowID(1), 1)
+ XCTAssertEqual(try validatedQueryWindowID(UInt32.max), UInt32.max)
+ for value: Any in [-1, 0, 1.5, Double.infinity, Double.nan, Double(UInt32.max) + 1, true, "1", NSNull()] {
+ XCTAssertThrowsError(try validatedQueryWindowID(value), "\(value)")
+ }
+ let dispatcher = ComputerUseToolDispatcher()
+ let result = dispatcher.callToolAsResult(name: "query", arguments: ["app": "NoSuchReviewApp", "text": "Send", "window_id": -1])
+ XCTAssertTrue(result.isError)
+ XCTAssertTrue(result.primaryText?.contains("window_id") == true)
+ }
+
+ func testQueryStringsAndExactRejectSilentCoercion() throws {
+ XCTAssertEqual(try validatedQueryString("Send", key: "text"), "Send")
+ XCTAssertThrowsError(try validatedQueryString(3, key: "text"))
+ XCTAssertThrowsError(try validatedQueryString(String(repeating: "x", count: 1001), key: "text"))
+ XCTAssertTrue(try validatedQueryExact(true))
+ XCTAssertFalse(try validatedQueryExact(nil))
+ XCTAssertThrowsError(try validatedQueryExact(1))
+ XCTAssertThrowsError(try validatedQueryExact("true"))
+ }
+
+ func testRegistryRejectsWrongAppAndRestartedProcess() throws {
+ let registry = TargetedElementRegistry<String>()
+ let original = TargetedAppIdentity(pid: 1, bundleIdentifier: "com.example.A", launchedAt: Date(timeIntervalSince1970: 1))
+ let index = registry.insert(app: original) { _ in "control" }
+ XCTAssertEqual(try registry.resolve(index, app: original), "control")
+ for identity in [TargetedAppIdentity(pid: 2, bundleIdentifier: "com.example.A", launchedAt: original.launchedAt),
+ TargetedAppIdentity(pid: 1, bundleIdentifier: "com.example.B", launchedAt: original.launchedAt),
+ TargetedAppIdentity(pid: 1, bundleIdentifier: "com.example.A", launchedAt: Date(timeIntervalSince1970: 2))] {
+ XCTAssertThrowsError(try registry.resolve(index, app: identity))
+ }
+ XCTAssertEqual(try registry.resolve(index, app: original), "control")
+ }
+
+ func testRegistryEvictsExpiresAndDoesNotReuseClearedIndexes() throws {
+ var now = Date(timeIntervalSince1970: 0)
+ let app = TargetedAppIdentity(pid: 1, bundleIdentifier: nil, launchedAt: nil)
+ let registry = TargetedElementRegistry<Int>(capacity: 2, lifetime: 10, now: { now })
+ let first = registry.insert(app: app) { $0 }
+ let second = registry.insert(app: app) { $0 }
+ let third = registry.insert(app: app) { $0 }
+ XCTAssertThrowsError(try registry.resolve(first, app: app))
+ XCTAssertEqual(try registry.resolve(second, app: app), second)
+ registry.remove(second)
+ XCTAssertThrowsError(try registry.resolve(second, app: app))
+ now = now.addingTimeInterval(10)
+ XCTAssertThrowsError(try registry.resolve(third, app: app))
+ registry.clear()
+ let fourth = registry.insert(app: app) { $0 }
+ XCTAssertGreaterThan(fourth, third)
+ XCTAssertThrowsError(try registry.resolve(third, app: app))
+ }
+
+ func testReadOnlyResolutionNeverLaunchesAbsentApp() {
+ XCTAssertThrowsError(try AppDiscovery.resolveRunning("NoSuchReviewApp", applications: []))
+ XCTAssertThrowsError(try AppDiscovery.resolveRunning("com.1password.1password", applications: []))
+ }
+
+ func testWideTreeBoundsChildFetchAndReportsPartialMatches() throws {
+ var fetched = 0
+ let result = try boundedTargetedWalk(root: 0, maxNodes: 3, limit: 10, expired: { false }, inspect: { node in
+ (node > 0, node == 0)
+ }, children: { _, remaining in
+ fetched = remaining
+ return (Array(1...remaining), true)
+ })
+ XCTAssertEqual(fetched, 2)
+ XCTAssertEqual(result.visitedNodes, 3)
+ XCTAssertEqual(result.matches, [1, 2])
+ XCTAssertTrue(result.truncated)
+ XCTAssertEqual(result.stopReason, "max_nodes")
+ }
+
+ func testTraversalDeduplicatesCyclesAndStopsAtResultLimit() throws {
+ let cycle = try boundedTargetedWalk(root: 0, maxNodes: 10, limit: 10, expired: { false }, inspect: { _ in (true, true) }, children: { node, _ in
+ ([node, (node + 1) % 2], false)
+ })
+ XCTAssertEqual(cycle.matches, [0, 1])
+ XCTAssertFalse(cycle.truncated)
+ let limited = try boundedTargetedWalk(root: 0, maxNodes: 10, limit: 1, expired: { false }, inspect: { _ in (true, true) }, children: { _, _ in XCTFail("should not fetch children"); return ([], false) })
+ XCTAssertEqual(limited.stopReason, "limit")
+ }
+
+ func testSlowChildFetchReportsTimeoutRatherThanNodeLimit() throws {
+ var expired = false
+ let result = try boundedTargetedWalk(root: 0, maxNodes: 10, limit: 10, expired: { expired },
+ inspect: { _ in (true, true) }, children: { _, _ in
+ expired = true
+ return ([], true)
+ })
+ XCTAssertEqual(result.matches, [0])
+ XCTAssertEqual(result.stopReason, "timeout")
+ }
+
+ func testExactDoesNotMatchTruncatedLongLabel() {
+ let criteria = TargetedAX.Criteria(text: String(repeating: "x", count: 1000), exact: true, role: nil, limit: 20, maxNodes: 500)
+ XCTAssertFalse(targetedRecordMatches(criteria, role: "AXButton", title: String(repeating: "x", count: 1001), description: nil, value: nil))
+ }
+
+ func testTraversalReportsTimeoutAndAXErrors() throws {
+ let timedOut = try boundedTargetedWalk(root: 0, maxNodes: 10, limit: 10, expired: { true }, inspect: { _ in XCTFail("should not inspect"); return (false, true) }, children: { _, _ in ([], false) })
+ XCTAssertEqual(timedOut.visitedNodes, 0)
+ XCTAssertEqual(timedOut.stopReason, "timeout")
+ let failed = try boundedTargetedWalk(root: 0, maxNodes: 10, limit: 10, expired: { false }, inspect: { node in
+ if node == 1 { throw ComputerUseError.stateUnavailable("unreadable") }
+ return (true, true)
+ }, children: { _, _ in ([1], false) })
+ XCTAssertEqual(failed.matches, [0])
+ XCTAssertEqual(failed.stopReason, "ax_error")
+ }
+}
diff --git a/scripts/node-repl/open-computer-use-repl.mjs b/scripts/node-repl/open-computer-use-repl.mjs
index 2bbb122..7d101d8 100755
--- a/scripts/node-repl/open-computer-use-repl.mjs
+++ b/scripts/node-repl/open-computer-use-repl.mjs
@@ -24,7 +24,7 @@ const HERE = path.dirname(fileURLToPath(import.meta.url));
const SERVER_INSTRUCTIONS = `UI automation through a persistent JavaScript REPL using the initialized cua API.
-On the first call after startup or js_reset, bind the requested app with await cua.getApp("Example App"), or call await cua.getState() only when an app inventory is actually needed. App bindings persist across js calls. Batch deterministic actions and the resulting getAXState() in one js call, then verify the returned UI state. Use nodeRepl.write(value) for additional text and await nodeRepl.emitImage(image) for additional images. Ask the user before destructive or externally visible actions such as sending, deleting, or purchasing.`;
+On the first call after startup or js_reset, bind the requested app with await cua.getApp("Example App"), or call await cua.getState() only when an app inventory is actually needed. On macOS, use getApp(name, { initialState: false }) followed by app.query({ text, role }) for targeted lookup without an initial snapshot. App bindings persist across js calls. Batch deterministic actions and the resulting getAXState() in one js call, then verify the returned UI state. Use nodeRepl.write(value) for additional text and await nodeRepl.emitImage(image) for additional images. Ask the user before destructive or externally visible actions such as sending, deleting, or purchasing.`;
const JS_DESCRIPTION = `Run JavaScript in a persistent Node.js REPL with top-level await and an initialized asynchronous cua API for Open Computer Use. Use this for all desktop interactions. Bind an app with let app = await cua.getApp("Example App"); then call await app.click(...), await app.typeText(...), and await app.getAXState(). Top-level bindings persist until js_reset, so prefer top-level var for names that may be assigned again. Batch deterministic actions and the resulting state read in one call to reduce round trips. Use nodeRepl.write(value) for extra text and await nodeRepl.emitImage(image) for extra images. If timeout_ms is omitted, execution times out after 30000 ms.`;
@@ -36,7 +36,8 @@ The runtime exposes an asynchronous app-bound API:
- \`await cua.getState({ emit? })\`: list current apps.
- \`await cua.listApps({ emit? })\`: list current apps.
-- \`await cua.getApp(nameOrBundleID)\`: bind an app and emit its initial accessibility state.
+- \`await cua.getApp(nameOrBundleID, { initialState? })\`: bind an app; initialState defaults to true. Use false for a lightweight binding before query.
+- \`await app.query({ text?, role?, exact?, limit?, maxNodes?, windowId? })\`: macOS-only AX lookup; returns matches and truncation metadata, with indexes usable in this native session.
- \`await app.getAXState({ emit?, textLimit?, maxTreeNodes?, maxTreeDepth?, windowPlacement? })\`
- \`await app.getScreenshot({ emit? })\`
- \`await app.getAXStateAndScreenshot(options?)\`
@@ -260,6 +261,25 @@ export function createCuaApi(native, activeOutput) {
}
function appBinding(app) {
return Object.freeze({
+ async query(options = {}) {
+ if (!isPlainObject(options)) throw new Error("app.query expects an options object");
+ for (const key of Object.keys(options)) {
+ if (!["text", "role", "exact", "limit", "maxNodes", "windowId"].includes(key)) throw new Error(`Unknown query option: ${key}`);
+ }
+ for (const key of ["text", "role"]) {
+ if (options[key] !== undefined && (typeof options[key] !== "string" || options[key].length > 1000)) throw new Error(`${key} must be a string of at most 1000 UTF-16 code units`);
+ }
+ if (!options.text?.trim() && !options.role?.trim()) throw new Error("query requires text and/or role");
+ if (options.exact !== undefined && typeof options.exact !== "boolean") throw new Error("exact must be true or false");
+ for (const key of ["limit", "maxNodes", "windowId"]) {
+ if (options[key] !== undefined && (!Number.isSafeInteger(options[key]) || options[key] < 1 || (key === "windowId" && options[key] > 0xffffffff))) throw new Error(`${key} must be a positive integer${key === "windowId" ? " up to 4294967295" : ""}`);
+ }
+ const result = await call("query", { app, text: options.text, role: options.role, exact: options.exact,
+ limit: options.limit, max_nodes: options.maxNodes, window_id: options.windowId });
+ const response = JSON.parse(toolResultText(result));
+ if (!isPlainObject(response) || !Array.isArray(response.matches) || typeof response.truncated !== "boolean" || typeof response.stop_reason !== "string") throw new Error("Invalid query response");
+ return response;
+ },
async getAXState(options = {}) {
const result = await call("get_app_state", { app, ...optionsToSnapshotArgs(options) });
const text = toolResultText(result);
@@ -318,10 +338,14 @@ export function createCuaApi(native, activeOutput) {
if (options.emit !== false) activeOutput().write(state.apps, "cua.state");
return state.apps;
},
- async getApp(app) {
+ async getApp(app, options = {}) {
+ if (typeof app !== "string" || !app.trim()) throw new Error("getApp requires an app name or bundle ID");
+ if (!isPlainObject(options) || (options.initialState !== undefined && typeof options.initialState !== "boolean")) throw new Error("initialState must be true or false");
await emitDocs();
- const result = await call("get_app_state", { app, text_limit: "max" });
- await emitText(toolResultText(result));
+ if (options.initialState !== false) {
+ const result = await call("get_app_state", { app, text_limit: "max" });
+ await emitText(toolResultText(result));
+ }
return appBinding(app);
},
async rewriteDocumentation() { activeOutput().write(COMPUTER_USE_GUIDANCE, "cua.core"); },
diff --git a/scripts/node-repl/open-computer-use-repl.test.mjs b/scripts/node-repl/open-computer-use-repl.test.mjs
index fd1d4b6..62ad87d 100644
--- a/scripts/node-repl/open-computer-use-repl.test.mjs
+++ b/scripts/node-repl/open-computer-use-repl.test.mjs
@@ -245,3 +245,48 @@ test("worker session serializes concurrent callers", async () => {
assert.equal((await second).content.at(-1).text, "first,second");
await session.close();
});
+
+test("lightweight app binding queries and acts across persistent JS calls without a snapshot", async () => {
+ const native = mockNative();
+ const request = native.request;
+ native.request = async (method, params) => {
+ if (params.name !== "query") return request(method, params);
+ native.calls.push(params);
+ return textResult(JSON.stringify({ matches: [{ index: 1000001, title: "Send" }], truncated: false, stop_reason: "complete", visited_nodes: 3, window_id: 12 }));
+ };
+ const session = new PersistentJavaScriptSession({ native });
+ let result = await session.run(`var queriedApp = await cua.getApp("Text", { initialState: false }); var found = await queriedApp.query({ text: "Send", role: "button", exact: true, maxNodes: 50, windowId: 12 }); nodeRepl.write(found);`);
+ assert.equal(result.isError, false, result.content.at(-1).text);
+ assert.deepEqual(native.calls.map(c => c.name), ["query"]);
+ assert.deepEqual(native.calls[0].arguments, { app: "Text", text: "Send", role: "button", exact: true, limit: undefined, max_nodes: 50, window_id: 12 });
+ result = await session.run(`await queriedApp.click(found.matches[0].index);`);
+ assert.equal(result.isError, false);
+ assert.equal(native.calls.at(-1).arguments.element_index, 1000001);
+ assert.equal(native.calls.some(c => c.name === "get_app_state"), false);
+});
+
+test("query rejects invalid options and preserves native errors", async () => {
+ const native = mockNative(), session = new PersistentJavaScriptSession({ native });
+ await session.run(`var queriedApp = await cua.getApp("Text", { initialState: false });`);
+ for (const options of [`{}`, `{text: "Send", windowId: -1}`, `{text: "Send", windowId: 4294967296}`, `{text: "Send", exact: 1}`, `{text: 3}`, `{role: "button", maxNodes: 1.5}`, `{text: "Send", typo: true}`]) {
+ const result = await session.run(`await queriedApp.query(${options});`);
+ assert.equal(result.isError, true, options);
+ }
+ assert.equal(native.calls.length, 0);
+ native.request = async () => textResult("query is unavailable on this platform", true);
+ const result = await session.run(`await queriedApp.query({text: "Send"});`);
+ assert.equal(result.isError, true);
+ assert.match(result.content.at(-1).text, /unavailable on this platform/);
+});
+
+test("query returns truncation metadata and rejects malformed native responses", async () => {
+ const native = mockNative(), session = new PersistentJavaScriptSession({ native });
+ native.request = async () => textResult(JSON.stringify({ matches: [], truncated: true, stop_reason: "timeout" }));
+ const result = await session.run(`var queriedApp = await cua.getApp("Text", { initialState: false }); nodeRepl.write(JSON.stringify(await queriedApp.query({role: "button"})));`);
+ assert.equal(result.isError, false);
+ assert.equal(JSON.parse(result.content.at(-1).text).stop_reason, "timeout");
+ native.request = async () => textResult("[]");
+ const bad = await session.run(`await queriedApp.query({role: "button"});`);
+ assert.equal(bad.isError, true);
+ assert.match(bad.content.at(-1).text, /Invalid query response/);
+});
diff --git a/skills/open-computer-use/SKILL.md b/skills/open-computer-use/SKILL.md
index ace070b..b4f3952 100644
--- a/skills/open-computer-use/SKILL.md
+++ b/skills/open-computer-use/SKILL.md
@@ -15,6 +15,9 @@ It supports the same core tool surface across macOS, Linux, and Windows:
`list_apps`, `get_app_state`, `click`, `perform_secondary_action`, `scroll`,
`drag`, `type_text`, `press_key`, and `set_value`.
+macOS additionally exposes `query`: a targeted accessibility lookup that finds
+controls by text and/or role without rendering a tree or taking a screenshot.
+
## Core Workflow
1. On macOS, run `sw_vers -productVersion` before invoking the CLI and require macOS 14.0 or later. On older versions, explain that the runtime cannot launch; do not recommend `doctor` or permission changes as a fix for binary incompatibility.
diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md
index 2815f59..40a896e 100644
--- a/skills/open-computer-use/references/usage.md
+++ b/skills/open-computer-use/references/usage.md
@@ -41,7 +41,7 @@ press_key
set_value
```
-This remains the native compatibility surface. The bundled Codex plugin uses a
+macOS also exposes `query` for targeted AX lookup. These nine core tools remain the cross-platform compatibility surface. The bundled Codex plugin uses a
code-first adapter instead and advertises only `js` and `js_reset`. In that
surface, bind an app and compose actions with asynchronous JavaScript:
@@ -78,7 +78,7 @@ ocu repl
session and top-level bindings for the current terminal until `.exit`, Ctrl-D,
or termination; `.editor` / `.end` accepts multiline input and `.reset`
discards JavaScript bindings. Use `nodeRepl.write(value)` for explicit output.
-The native `ocu mcp` surface remains the nine tools listed above. Run
+The native `ocu mcp` surface exposes the nine core tools above, plus `query` on macOS. Run
`ocu capabilities --json` to inspect component availability instead of
inferring support from a command being present in help.
@@ -145,6 +145,15 @@ open-computer-use snapshot --max-tree-nodes 3000 --max-tree-depth 96 "Google Chr
- Run `get_app_state` immediately before element-targeted actions.
- Re-run `get_app_state` after navigation, modal changes, page reloads, or failed actions.
- Use coordinate actions only when the rendered tree does not expose the target as an element.
+- On macOS, use `app.query` when you know the control's text or role. Bind with
+ `await cua.getApp("Safari", { initialState: false })` to skip the initial snapshot.
+ Query returns `{ matches, truncated, stop_reason, visited_nodes, window_id, limit, max_nodes }`.
+ An empty result only means no matching exposed control was found when `truncated` is false.
+ `exact: true` never retries substring matching. Query does not launch or activate the app.
+- Query indexes last up to 120 seconds in the same native session (5000 retained handles).
+ Use persistent MCP or `ocu repl` for query then action; separate CLI calls cannot share indexes.
+ Actions revalidate the process/window/control and current geometry. On failure, query again.
+ See [targeted query](../../../docs/targeted-query.md) for options and examples.
## Choosing a Click Method