Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

Align official computer-use behavior and bump to 0.1.2

Leo07bee810f954
diff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
index 1bc9c80..665de19 100644
--- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
+++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
@@ -34,7 +34,7 @@ final class MCPClient {
_ = try request(method: "initialize", params: [
"clientInfo": [
"name": "OpenComputerUseSmokeSuite",
- "version": "0.1.1",
+ "version": "0.1.2",
],
"capabilities": [:],
"protocolVersion": "2025-03-26",
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 8fd1dca..a445274 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -48,9 +48,10 @@
### 3. Tool Service 层
- `ComputerUseService` 负责把 MCP tool 请求映射到本地能力。
-- `list_apps` 通过 `NSWorkspace` 枚举运行中的 app。
+- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。
- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。
- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。
+- 对真实 app 的 `get_app_state` / action tool 入口,当前新增了一层官方风格的高风险 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app,尽量贴近官方对终端、密码管理器、Chrome 与少量系统敏感组件的防护行为。
- 普通 app 的 element frame 当前按“窗口左上角为原点”的 window-relative 坐标输出,便于后续把 `element_index` 和截图坐标统一到同一套参考系。
- `click` 在执行真实动作前后,会额外驱动一层透明 `SoftwareCursorOverlay` window:移动阶段走曲线动画,点击阶段做 pulse,动作结束后只保留一小段停驻与轻微 sway,随后自动淡出。
- overlay 的 visual style 优先在运行时从本机官方 `Codex Computer Use.app` 的 `Package_ComputerUse.bundle` / `Package_SlimCore.bundle` 读取 `SoftwareCursor` 资产并做一次本地处理;如果本机没有这份 bundle,则回退到仓库内的矢量样式。
diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.md
index c6708f1..7b7a1c0 100644
--- a/docs/QUALITY_SCORE.md
+++ b/docs/QUALITY_SCORE.md
@@ -11,8 +11,8 @@
| 区域 | 评分 | 原因 | 下一步 |
| --- | --- | --- | --- |
-| 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及 9 个 tools 的可运行实现。 | 继续收敛权限 UI 细节、app allowlist 和更清晰的用户错误提示。 |
+| 产品面 | B | 已经有 Swift 本地 `computer-use` MCP server、默认 app 模式权限引导,以及一轮按官方 surface / result 行为收敛过的 9 个 tools。 | 继续收敛复杂 AX 场景下的 state rendering 细节、权限 UI 和更清晰的用户错误提示。 |
| 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 |
| 测试 | B | `swift test` + smoke suite 已覆盖 9 个 tools 的回归,并新增了针对“前台焦点是否被抢占”的手工对比样本沉淀。 | 增加更多普通 app 的录制回归,减少只依赖 fixture 和一次性手工检查。 |
| 可观测性 | C | 已有 `doctor`、`snapshot`、smoke 输出,以及一组仓库内留档的官方 `computer-use` / 本仓库实现对比样本。 | 补统一日志级别、失败上下文和 release artifact 里的诊断信息,把一次性样本收敛成可重复采集流程。 |
-| 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域。 | 增加 app allowlist / session approval,避免对任意 app 默认开放。 |
+| 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域,并补了一层高风险 app denylist。 | 增加用户可配置 allowlist / session approval,避免策略长期硬编码在仓库里。 |
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
index 2530aaa..9e042ad 100644
--- a/docs/SECURITY.md
+++ b/docs/SECURITY.md
@@ -14,14 +14,18 @@
## 授权与最小权限
-- 当前还没有官方闭源实现里的 session approval / app allowlist。
+- 当前已经补上一层官方风格的高风险 bundle denylist / bundle-id gate:
+ - 会阻止对终端类 app、密码管理器、Chrome 和少量系统敏感组件做直接 `get_app_state` / action 调用。
+ - 对 bundle identifier 直传时返回 safety denial;对 app name 查询时默认不把这些高风险 app 暴露成可解析目标。
+- 但当前仍然没有官方闭源实现里的 session approval / 动态 app allowlist。
- 这意味着开源版当前的安全边界主要由:
- 明确的 tool 调用参数
+ - 内置高风险 denylist
- 宿主进程权限
- 本地使用场景
共同提供。
- 下一阶段应优先补:
- - app allowlist
+ - 用户可配置的 app allowlist / policy
- session 级审批
- 更清楚的高风险 app / 系统设置防护策略
diff --git a/docs/exec-plans/active/20260417-official-tool-alignment.md b/docs/exec-plans/active/20260417-official-tool-alignment.md
new file mode 100644
index 0000000..983835e
--- /dev/null
+++ b/docs/exec-plans/active/20260417-official-tool-alignment.md
@@ -0,0 +1,79 @@
+# Official Tool Alignment
+
+## 目标
+
+把仓库内 `open-computer-use` 暴露给 MCP host 的 9 个 tools,按当前官方 `computer-use` 的真实 `tools/list` 和 `tools/call` 返回做一轮收口:尽量对齐 description、schema、annotations、错误语义、`list_apps` 列表形态,以及 `get_app_state` / action tool 的文本输出风格。
+
+## 范围
+
+- 包含:
+ - 用 `scripts/computer-use-cli` 实测官方 `tools/list` 与代表性 `tools/call`。
+ - 收敛 9 个 tools 的 description、input schema 和 annotations。
+ - 让 `list_apps` 输出更接近官方的“运行中 + 近 14 天使用过 app”视图。
+ - 调整 `get_app_state` / action tool 文本渲染,减少内部实现细节,靠近官方树形输出。
+ - 同步架构文档、history 和必要测试。
+- 不包含:
+ - 当前阶段不承诺 100% 复刻官方闭源安全策略、allowlist、overlay UI 或私有 host 集成。
+ - 当前阶段不把一次性本机样本抽成完整自动化 diff 平台。
+
+## 背景
+
+- 相关文档:
+ - `docs/ARCHITECTURE.md`
+ - `docs/references/codex-computer-use-cli.md`
+ - `docs/references/codex-local-runtime-logs.md`
+ - `docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md`
+- 相关代码路径:
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift`
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift`
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+ - `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`
+- 已知约束:
+ - 官方 `computer-use` 只能稳定通过 `codex app-server` 间接调用,不能假设普通 stdio client 可直连。
+ - 官方 `list_apps` 的 `uses` 数值最终确认来自系统 Spotlight metadata(`kMDItemUseCount` / `kMDItemLastUsedDate_Ranking`),而不是此前猜测的 `Knowledge/knowledgeC.db`。
+ - Accessibility 树和索引分配高度依赖宿主 app 与 AX 层结构,完全一致需要做裁剪和抽象,而不是原样暴露本地 AX 全量细节。
+
+## 风险
+
+- 风险:只对齐 schema,不对齐结果文本,host 仍会看到明显不同的树结构和错误语义。
+ - 缓解方式:同一轮内连带调整 `list_apps`、`get_app_state` 和 tool error payload。
+- 风险:过度裁剪 AX 树导致现有 smoke path 或 action tool 命中能力退化。
+ - 缓解方式:保留底层 element map,只收口用户可见文本;行为命中继续依赖内部全量 snapshot。
+- 风险:`list_apps` 引入新的系统数据源后,在某些机器上查不到 usage 数据。
+ - 缓解方式:做只读、可失败的回退;查不到时仍返回运行中 app。
+
+## 里程碑
+
+1. 官方 surface 与差异点实测确认。
+2. schema、错误语义、`list_apps`、state rendering 收口。
+3. 双链路复测、文档同步和归档。
+
+## 验证方式
+
+- 命令:
+ - `swift test`
+ - `go run . list-tools --transport app-server`
+ - `go run . list-tools --transport direct --server-bin ../../.build/debug/OpenComputerUse`
+ - `go run . call list_apps --transport app-server`
+ - `go run . call list_apps --transport direct --server-bin ../../.build/debug/OpenComputerUse`
+- 手工检查:
+ - 官方与开源 `tools/list` 的 9 个 tool surface 能一眼对齐。
+ - `get_app_state` 文本不再暴露 `_NS:` 内部 identifier、frame 噪音和过量 cell 递归。
+ - `appNotFound`、安全拒绝等错误形态与官方一致地回到 `content` + `isError: true`。
+- 观测检查:
+ - `list_apps` 输出包含 `running`、`last-used`、`uses`,且顺序接近官方。
+ - 代表性 action tool 返回继续附带状态文本和截图。
+
+## 进度记录
+
+- [x] 里程碑 1
+- [x] 里程碑 2
+- [x] 里程碑 3
+
+## 决策记录
+
+- 2026-04-17:本轮对齐以 `computer-use-cli` 实测官方返回为准,而不是继续基于先前推测维护“近似文案”。
+- 2026-04-17:`list_apps` 改为优先使用 Spotlight metadata query,按官方同源的 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 排序和筛选 app;运行态 app 继续由 `NSWorkspace` 合并补齐。
+- 2026-04-17:对 bundle-id 直传的高风险 app 增加官方风格 safety denial,并让名称匹配路径默认不解析到这些 app,复刻官方 `appNotFound("iTerm2")` / `not allowed to use the app 'com.googlecode.iterm2'` 的边界行为。
+- 2026-04-17:直接 MCP tool `content[0].text` 的当前官方基线应从 `App=<bundle-id> (pid ...)` 起头,不再把 `Computer Use state (CUA App Version: 750)` / `<app_state>` 这层旧包裹当作响应文本的一部分。
diff --git a/docs/histories/2026-04/20260417-1816-align-computer-use-tool-results.md b/docs/histories/2026-04/20260417-1816-align-computer-use-tool-results.md
index 07bcc24..169883c 100644
--- a/docs/histories/2026-04/20260417-1816-align-computer-use-tool-results.md
+++ b/docs/histories/2026-04/20260417-1816-align-computer-use-tool-results.md
@@ -37,3 +37,32 @@
- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift`
- `docs/ARCHITECTURE.md`
- `docs/SECURITY.md`
+
+### 🔁 Follow-up | 2026-04-17 19:01
+
+**Additional Actions:**
+- **[Official source discovery]**: 继续通过 `computer-use-cli` 和本机二进制字符串检查,确认官方 `list_apps` 的 `uses`/排序来自 Spotlight metadata 的 `kMDItemUseCount` 与 `kMDItemLastUsedDate_Ranking`,而不是 `Knowledge/knowledgeC.db`。
+- **[App list exact match]**: 重写 `AppDiscovery` 的 app catalog 逻辑,按标准 application scope 做 metadata query,再和运行中 app 合并;本机实测 `list_apps` 与官方当前输出做到无 diff。
+- **[Safety parity]**: 新增一层官方风格的高风险 bundle denylist,让 `iTerm2` 这类 name query 返回 `appNotFound(...)`、bundle-id query 返回 safety denial,和官方边界行为对齐。
+- **[State rendering tightening]**: 继续收口 `get_app_state` 的 AX 渲染顺序、traits/value 文案、toolbar/group 展示与 secondary action 过滤;复杂 outline 场景已明显更接近官方,但 Activity Monitor 这类样本仍有少量尾部细节差异。
+- **[Docs sync]**: 同步更新架构、安全、质量和 execution plan 文档,避免仓库继续保留已失效的 `Knowledge` 推测和“尚无安全策略”的旧说法。
+
+**Extra Files Modified:**
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `docs/QUALITY_SCORE.md`
+- `docs/exec-plans/active/20260417-official-tool-alignment.md`
+
+### 🔁 Follow-up | 2026-04-17 20:30
+
+**Additional Actions:**
+- **[State text prefix tightening]**: 把 `get_app_state` / action tool 的文本头部进一步收口到直接从 `App=<bundle-id> (pid ...)` 开始,不再输出 `Computer Use state (CUA App Version: 750)` 或 `<app_state>` 包裹。
+- **[Selected text formatting parity]**: 把选中文本从我们自己的 code fence + explanatory note 改成官方当前更接近的单行 `Selected text: [...]` 形式。
+- **[Canonical app identifier in errors]**: 对依赖 window bounds 的错误提示优先输出 bundle identifier,减少后续 tool 调用又漂回应用名。
+- **[Regression coverage]**: 新增单测锁定 state 文本起始格式和 `Selected text` 渲染,避免后续重构把这些差异带回来。
+- **[Reference correction]**: 更新仓库内 reverse-engineering 样本文档,明确当前应以直接 MCP `content[0].text` 的 `App=...` 起始格式作为官方基线。
+
+**Extra Files Modified:**
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md`
diff --git a/docs/histories/2026-04/20260417-2043-bump-open-computer-use-to-0.1.2.md b/docs/histories/2026-04/20260417-2043-bump-open-computer-use-to-0.1.2.md
new file mode 100644
index 0000000..02b415d
--- /dev/null
+++ b/docs/histories/2026-04/20260417-2043-bump-open-computer-use-to-0.1.2.md
@@ -0,0 +1,33 @@
+## [2026-04-17 20:43] | Task: 升级 open-computer-use 到 0.1.2 并刷新 Codex 插件安装
+
+### 🤖 Execution Context
+* **Agent ID**: `primary`
+* **Base Model**: `gpt-5`
+* **Runtime**: `Codex CLI + SwiftPM`
+
+### 📥 User Query
+> 改到 `0.1.2` 然后安装到 Codex plugin。
+
+### 🛠 Changes Overview
+**Scope:** `plugins/open-computer-use`, `packages/OpenComputerUseKit`, `apps/OpenComputerUseSmokeSuite`, `scripts`, `docs`
+
+**Key Actions:**
+- **[Version bump]**: 将插件 manifest、MCP server 自报版本、smoke client 版本、CLI 版本与 app bundle 版本统一提升到 `0.1.2`。
+- **[Bundle metadata sync]**: 将打包脚本中的 `CFBundleShortVersionString` 提升到 `0.1.2`,并把 `CFBundleVersion` 递增到 `3`,避免本机缓存里的 app 元信息落后于源码版本。
+- **[Docs sync]**: 同步更新 `computer-use-cli` 相关文档里的本地插件缓存路径示例,避免继续引用旧的 `0.1.1` 目录。
+- **[Codex install refresh]**: 执行 `./scripts/install-codex-plugin.sh --rebuild`,把本地插件缓存刷新到 `~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.2`,并确认 `~/.codex/config.toml` 仍启用 `open-computer-use@open-computer-use-local`。
+- **[Verification]**: 运行 `swift test` 通过,并校验缓存中的 `.codex-plugin/plugin.json` 已显示 `version = 0.1.2`。
+
+### 🧠 Design Intent (Why)
+这次改动的目标仍然是保持“源码版本、打包产物版本、插件缓存版本、MCP 握手版本”四者一致,避免 Codex 实际加载的插件缓存和当前仓库源码不一致。把安装动作放在同一轮里完成,能确保后续通过 Codex 调用到的就是这次刚升级后的 `0.1.2`。
+
+### 📁 Files Modified
+- `plugins/open-computer-use/.codex-plugin/plugin.json`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`
+- `scripts/build-open-computer-use-app.sh`
+- `scripts/computer-use-cli/main.go`
+- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/references/codex-computer-use-cli.md`
+- `scripts/computer-use-cli/README.md`
+- `docs/histories/2026-04/20260417-2043-bump-open-computer-use-to-0.1.2.md`
diff --git a/docs/references/codex-computer-use-cli.md b/docs/references/codex-computer-use-cli.md
index 0e790e8..89091b7 100644
--- a/docs/references/codex-computer-use-cli.md
+++ b/docs/references/codex-computer-use-cli.md
@@ -76,7 +76,7 @@ go run . call list_apps --transport app-server
cd scripts/computer-use-cli
go run . call list_apps \
--transport direct \
- --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.1/scripts/launch-open-computer-use.sh
+ --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.2/scripts/launch-open-computer-use.sh
```
## 什么时候不要再重试 direct
diff --git a/docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md b/docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md
index f412295..df061c5 100644
--- a/docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md
+++ b/docs/references/codex-computer-use-reverse-engineering/tool-call-samples-2026-04-17.md
@@ -10,6 +10,7 @@
- 截图策略:`get_app_state`、`click`、`scroll`、`drag` 等调用都会在工具 UI 中附带截图;文档里只保留文本响应,不嵌入截图。
- 测试 app:`Finder`、`Activity Monitor`、`System Settings`。
- 安全边界:只选择了目录选择、搜索框输入、滚动、分隔条拖拽这类低风险动作,没有切换系统权限开关,也没有双击打开文件。
+- 2026-04-17 晚些时候再次核对后确认:直接看 MCP tool `content[0].text` 时,官方文本是从 `App=...` 起头的;这里早先保留下来的 `Computer Use state (CUA App Version: 750)` / `<app_state>` 包裹不应再视为当前官方基线。
## `list_apps`
@@ -94,8 +95,6 @@ Request
Response excerpt
```text
-Computer Use state (CUA App Version: 750)
-<app_state>
App=com.apple.finder (pid 1106)
Window: "open-codex-computer-use", App: Finder.
0 standard window open-codex-computer-use, ID: FinderWindow, Secondary Actions: Raise
@@ -110,7 +109,6 @@ Window: "open-codex-computer-use", App: Finder.
119 row (selectable, collapsed) Secondary Actions: Expand
130 row (selected)
...
-</app_state>
```
### Sample 2
@@ -126,8 +124,6 @@ Request
Response excerpt
```text
-Computer Use state (CUA App Version: 750)
-<app_state>
App=com.apple.ActivityMonitor (pid 988)
Window: "Activity Monitor", App: Activity Monitor.
0 standard window Activity Monitor – All Processes, Secondary Actions: Raise
@@ -144,7 +140,6 @@ Window: "Activity Monitor", App: Activity Monitor.
48 button search
...
The focused UI element is 2 outline.
-</app_state>
```
### Sample 3
@@ -160,8 +155,6 @@ Request
Response excerpt
```text
-Computer Use state (CUA App Version: 750)
-<app_state>
App=com.apple.systempreferences (pid 73431)
Window: "Screen & System Audio Recording", App: System Settings.
0 standard window Screen & System Audio Recording, ID: main, Secondary Actions: Raise
@@ -177,7 +170,6 @@ Window: "Screen & System Audio Recording", App: System Settings.
63 row (selectable) Codex Computer Use
64 switch Value: on, ID: Codex Computer Use_Toggle
...
-</app_state>
```
额外边界样本:
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index bcebb24..8b89fbf 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -36,6 +36,7 @@ public struct AppSnapshot {
let mode: SnapshotMode
let treeLines: [String]
let focusedSummary: String?
+ let selectedText: String?
let elements: [Int: ElementRecord]
@@ -45,24 +46,21 @@ public struct AppSnapshot {
public func renderedText(style: SnapshotTextStyle) -> String {
var lines: [String] = []
+ let displayTitle = displayWindowTitle(windowTitle, appName: app.name)
+ let appReference = app.bundleIdentifier ?? app.name
- if style == .fullState {
- lines.append("Computer Use state (Open Computer Use 0.1.0)")
- lines.append("<app_state>")
- }
-
- lines.append("App=\(app.bundleIdentifier ?? app.name) (pid \(app.pid))")
- lines.append("Window: \(quoted(windowTitle ?? "")), App: \(app.name).")
+ lines.append("App=\(appReference) (pid \(app.pid))")
+ lines.append("Window: \(quoted(displayTitle)), App: \(app.name).")
lines.append(contentsOf: treeLines)
- if let focusedSummary {
+ if let selectedText, !selectedText.isEmpty {
+ lines.append("")
+ lines.append("Selected text: [\(selectedText)]")
+ } else if let focusedSummary {
+ lines.append("")
lines.append("The focused UI element is \(focusedSummary).")
}
- if style == .fullState {
- lines.append("</app_state>")
- }
-
return lines.joined(separator: "\n")
}
}
@@ -94,6 +92,7 @@ enum SnapshotBuilder {
let windowBounds = windowCapture?.bounds
let screenshotPNGData = windowCapture?.pngDataIfAvailable()
let focusedElement = preferredFocusedElement(appElement: appElement, appPID: app.pid, focusedApplication: focusedApplication, systemWide: systemWide)
+ let selectedText = focusedElement.flatMap(copySelectedText(_:))
let context = RenderContext(windowBounds: windowBounds, focusedElement: focusedElement)
var renderer = TreeRenderer(context: context)
@@ -109,6 +108,7 @@ enum SnapshotBuilder {
mode: .accessibility,
treeLines: renderer.lines,
focusedSummary: renderer.focusedSummary,
+ selectedText: selectedText,
elements: renderer.records
)
}
@@ -182,6 +182,7 @@ enum SnapshotBuilder {
mode: .fixture,
treeLines: lines,
focusedSummary: focusedSummary,
+ selectedText: nil,
elements: records
)
}
@@ -280,31 +281,77 @@ private struct TreeRenderer {
}
let index = nextIndex
- nextIndex += 1
let role = stringValue(of: root, attribute: kAXRoleAttribute) ?? "AXUnknown"
let subrole = stringValue(of: root, attribute: kAXSubroleAttribute)
- let title = stringValue(of: root, attribute: kAXTitleAttribute)
+ let baseRoleText = roleDescription(of: root, role: role, subrole: subrole)
let label = stringValue(of: root, attribute: kAXDescriptionAttribute)
+ let help = stringValue(of: root, attribute: kAXHelpAttribute)
let value = sanitizedValue(of: root)
- let axIdentifier = stringValue(of: root, attribute: kAXIdentifierAttribute)
+ let axIdentifier = displayIdentifier(stringValue(of: root, attribute: kAXIdentifierAttribute))
let traits = summarizeTraits(of: root)
let actions = copyActions(root) ?? []
- let prettyActions = actions
- .filter { $0 != kAXPressAction as String }
- .map(prettyActionName(_:))
+ let prettyActions = meaningfulActions(actions, role: role)
let localFrame = resolveLocalFrame(of: root, windowBounds: context.windowBounds)
+ let rowTexts = role == kAXRowRole as String ? flattenedRowTexts(of: root) : []
+ let childElements = children(of: root)
+ let title = preferredDisplayTitle(
+ for: root,
+ role: role,
+ label: label,
+ identifier: axIdentifier,
+ explicitValue: value,
+ rowTexts: rowTexts
+ )
+ let inlineRowSummary = outlineRowSummary(for: root, role: role)
+ let hidesChildren = shouldSuppressChildren(
+ role: role,
+ title: title,
+ label: label,
+ help: help,
+ value: value,
+ identifier: axIdentifier,
+ traits: traits,
+ actions: prettyActions,
+ children: childElements
+ )
+ let roleText = displayRoleText(
+ baseRoleText: baseRoleText,
+ role: role,
+ title: title,
+ label: label,
+ suppressChildren: hidesChildren
+ )
+
+ if shouldElideNode(role: role, title: title, label: label, value: value, identifier: axIdentifier, traits: traits, actions: prettyActions, childCount: childElements.count) {
+ for child in childElements {
+ render(child, depth: depth)
+ }
+ return
+ }
+
+ nextIndex += 1
- let roleText = humanizeRole(role: role, subrole: subrole)
- let titleSegment = title.map { " \($0)" } ?? ""
let traitsSegment = traits.isEmpty ? "" : " (\(traits.joined(separator: ", ")))"
- let labelSegment = label.map { " Description: \($0)" } ?? ""
- let identifierSegment = axIdentifier.map { " ID: \($0)" } ?? ""
- let valueSegment = value.map { " Value: \($0)" } ?? ""
- let actionsSegment = prettyActions.isEmpty ? "" : " Secondary Actions: \(prettyActions.joined(separator: ", "))"
- let frameSegment = localFrame.map { " Frame: \($0.renderedLocalFrame)" } ?? ""
+ let titleSegment = title.map { " \($0)" } ?? ""
+ let rowSummarySegment = inlineRowSummary.map { " \($0)" } ?? ""
+ let labelSegment = label != nil && label != title ? " Description: \(label!)" : ""
+ let helpSegment = {
+ guard let help else {
+ return ""
+ }
+ if help == title || help == label {
+ return ""
+ }
+ return ", Help: \(help)"
+ }()
+ let identifierSegment = displayIdentifierSegment(for: root, role: role, identifier: axIdentifier, title: title)
+ let valueSegment = formattedValueSegment(for: root, roleText: roleText, title: title, value: value)
+ let actionsPrefix = (title != nil || inlineRowSummary != nil) ? ", Secondary Actions: " : " Secondary Actions: "
+ let actionsSegment = prettyActions.isEmpty ? "" : "\(actionsPrefix)\(prettyActions.joined(separator: ", "))"
+ let linePrefix = roleText.isEmpty ? "\(index)" : "\(index) \(roleText)"
- lines.append("\(String(repeating: " ", count: depth))\(index) \(roleText)\(titleSegment)\(traitsSegment)\(labelSegment)\(identifierSegment)\(valueSegment)\(actionsSegment)\(frameSegment)")
+ lines.append("\(String(repeating: "\t", count: depth + 1))\(linePrefix)\(traitsSegment)\(titleSegment)\(rowSummarySegment)\(labelSegment)\(helpSegment)\(identifierSegment)\(valueSegment)\(actionsSegment)")
let record = ElementRecord(
index: index,
@@ -321,10 +368,21 @@ private struct TreeRenderer {
}
if let focusedElement = context.focusedElement, CFEqual(focusedElement, root) {
- focusedSummary = "\(index) \(roleText)"
+ focusedSummary = roleText.isEmpty ? "\(index)" : "\(index) \(roleText)"
}
- for child in children(of: root) {
+ if role == kAXRowRole as String, boolValue(of: root, attribute: kAXSelectedAttribute) != true {
+ for text in Array(rowTexts.dropFirst()) {
+ lines.append(text)
+ }
+ return
+ }
+
+ if hidesChildren {
+ return
+ }
+
+ for child in childElements {
render(child, depth: depth + 1)
}
}
@@ -339,10 +397,21 @@ private struct TreeRenderer {
var seen: Set<String> = []
for attribute in attributes {
- guard let values = copyArray(element, attribute: attribute) else {
+ guard let sourceValues = copyArray(element, attribute: attribute) else {
continue
}
+ if attribute == kAXChildrenAttribute,
+ let rows = copyArray(element, attribute: kAXRowsAttribute),
+ !rows.isEmpty
+ {
+ continue
+ }
+
+ let values = attribute == kAXRowsAttribute
+ ? visibleRows(in: sourceValues, parent: element)
+ : sourceValues
+
for child in values {
let id = opaqueIdentifier(for: child)
if seen.insert(id).inserted {
@@ -358,29 +427,49 @@ private struct TreeRenderer {
private func summarizeTraits(of element: AXUIElement) -> [String] {
var values: [String] = []
- if boolValue(of: element, attribute: kAXFocusedAttribute) == true {
- values.append("focused")
+ if let selected = boolValue(of: element, attribute: kAXSelectedAttribute) {
+ values.append(selected ? "selected" : "selectable")
}
- if boolValue(of: element, attribute: kAXSelectedAttribute) == true {
- values.append("selected")
+ if let expanded = boolValue(of: element, attribute: kAXExpandedAttribute) {
+ values.append(expanded ? "expanded" : "collapsed")
}
- if boolValue(of: element, attribute: kAXExpandedAttribute) == true {
- values.append("expanded")
+ if boolValue(of: element, attribute: kAXEnabledAttribute) == false {
+ values.append("disabled")
}
if isSettable(of: element, attribute: kAXValueAttribute) {
values.append("settable")
}
- if let role = stringValue(of: element, attribute: kAXRoleAttribute), role == kAXTextFieldRole as String {
- values.append("string")
+ if let valueType = valueTypeTrait(of: element) {
+ values.append(valueType)
}
return values
}
+private func valueTypeTrait(of element: AXUIElement) -> String? {
+ guard isSettable(of: element, attribute: kAXValueAttribute) else {
+ return nil
+ }
+
+ guard let value = attributeValue(of: element, attribute: kAXValueAttribute) else {
+ return nil
+ }
+
+ if CFGetTypeID(value) == CFStringGetTypeID() {
+ return "string"
+ }
+
+ if value is NSNumber {
+ return "float"
+ }
+
+ return nil
+}
+
private func copyElement(_ element: AXUIElement, attribute: String) -> AXUIElement? {
var value: CFTypeRef?
let error = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)
@@ -411,10 +500,18 @@ private func copyActions(_ element: AXUIElement) -> [String]? {
return actions as? [String]
}
-private func stringValue(of element: AXUIElement, attribute: String) -> String? {
+private func attributeValue(of element: AXUIElement, attribute: String) -> CFTypeRef? {
var value: CFTypeRef?
let error = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)
- guard error == .success, let value else {
+ guard error == .success else {
+ return nil
+ }
+
+ return value
+}
+
+private func stringValue(of element: AXUIElement, attribute: String) -> String? {
+ guard let value = attributeValue(of: element, attribute: attribute) else {
return nil
}
@@ -425,10 +522,17 @@ private func stringValue(of element: AXUIElement, attribute: String) -> String?
return nil
}
+private func copySelectedText(_ element: AXUIElement) -> String? {
+ guard let value = stringValue(of: element, attribute: kAXSelectedTextAttribute) else {
+ return nil
+ }
+
+ let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines)
+ return trimmed.isEmpty ? nil : trimmed
+}
+
private func boolValue(of element: AXUIElement, attribute: String) -> Bool? {
- var value: CFTypeRef?
- let error = AXUIElementCopyAttributeValue(element, attribute as CFString, &value)
- guard error == .success, let value else {
+ guard let value = attributeValue(of: element, attribute: attribute) else {
return nil
}
@@ -452,9 +556,7 @@ private func sanitizedValue(of element: AXUIElement) -> String? {
return sanitizeText(string)
}
- var value: CFTypeRef?
- let error = AXUIElementCopyAttributeValue(element, kAXValueAttribute as CFString, &value)
- guard error == .success, let value else {
+ guard let value = attributeValue(of: element, attribute: kAXValueAttribute) else {
return nil
}
@@ -465,6 +567,86 @@ private func sanitizedValue(of element: AXUIElement) -> String? {
return nil
}
+private func preferredDisplayTitle(
+ for element: AXUIElement,
+ role: String,
+ label: String?,
+ identifier: String?,
+ explicitValue: String?,
+ rowTexts: [String]
+) -> String? {
+ if let title = stringValue(of: element, attribute: kAXTitleAttribute), !title.isEmpty {
+ return sanitizeText(title)
+ }
+
+ if role == kAXRowRole as String {
+ return rowTexts.first
+ }
+
+ if (role == kAXOutlineRole as String || role == kAXListRole as String), let identifier {
+ return identifier
+ }
+
+ if (role == kAXButtonRole as String || role == kAXPopUpButtonRole as String), let label, !label.isEmpty {
+ return sanitizeText(label)
+ }
+
+ guard roleDescription(of: element, role: role, subrole: stringValue(of: element, attribute: kAXSubroleAttribute)) == "search text field" else {
+ return nil
+ }
+
+ return explicitValue
+}
+
+private func outlineRowSummary(for element: AXUIElement, role: String) -> String? {
+ guard role == kAXOutlineRole as String || role == kAXListRole as String else {
+ return nil
+ }
+
+ guard let allRows = copyArray(element, attribute: kAXRowsAttribute), !allRows.isEmpty else {
+ return nil
+ }
+
+ let visibleRows = visibleRows(in: allRows, parent: element)
+ guard !visibleRows.isEmpty, visibleRows.count < allRows.count else {
+ return nil
+ }
+
+ return "(showing 0-\(visibleRows.count - 1) of \(allRows.count) items)"
+}
+
+private func formattedValueSegment(for element: AXUIElement, roleText: String, title: String?, value: String?) -> String {
+ guard let value, !value.isEmpty else {
+ return ""
+ }
+
+ if roleText == "search text field", title == value {
+ return ""
+ }
+
+ if title == nil, let role = stringValue(of: element, attribute: kAXRoleAttribute), role == kAXStaticTextRole as String {
+ return " \(value)"
+ }
+
+ if ["scroll bar", "value indicator"].contains(roleText) {
+ return " \(value)"
+ }
+
+ return " Value: \(value)"
+}
+
+private func displayIdentifierSegment(for element: AXUIElement, role: String, identifier: String?, title: String?) -> String {
+ guard let identifier else {
+ return ""
+ }
+
+ if (role == kAXOutlineRole as String || role == kAXListRole as String), title == identifier {
+ return ""
+ }
+
+ return " ID: \(identifier)"
+}
+
private func resolveLocalFrame(of element: AXUIElement, windowBounds: CGRect?) -> CGRect? {
var positionValue: CFTypeRef?
var sizeValue: CFTypeRef?
@@ -496,6 +678,84 @@ private func resolveLocalFrame(of element: AXUIElement, windowBounds: CGRect?) -
return windowRelativeFrame(elementFrame: frame, windowBounds: windowBounds)
}
+private func shouldElideNode(
+ role: String,
+ title: String?,
+ label: String?,
+ value: String?,
+ identifier: String?,
+ traits: [String],
+ actions: [String],
+ childCount: Int
+) -> Bool {
+ guard childCount == 1 else {
+ return false
+ }
+
+ let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]
+ guard genericRoles.contains(role) else {
+ return false
+ }
+
+ return title == nil
+ && label == nil
+ && value == nil
+ && identifier == nil
+ && traits.isEmpty
+ && actions.isEmpty
+}
+
+private func shouldSuppressChildren(
+ role: String,
+ title: String?,
+ label: String?,
+ help: String?,
+ value: String?,
+ identifier: String?,
+ traits: [String],
+ actions: [String],
+ children: [AXUIElement]
+) -> Bool {
+ guard role == kAXGroupRole as String else {
+ return false
+ }
+
+ guard
+ title == nil,
+ label == nil,
+ help == nil,
+ value == nil,
+ identifier == nil,
+ traits.isEmpty,
+ actions.isEmpty,
+ !children.isEmpty
+ else {
+ return false
+ }
+
+ return children.allSatisfy { child in
+ stringValue(of: child, attribute: kAXRoleAttribute) == kAXStaticTextRole as String
+ }
+}
+
+private func displayRoleText(
+ baseRoleText: String,
+ role: String,
+ title: String?,
+ label: String?,
+ suppressChildren: Bool
+) -> String {
+ if suppressChildren {
+ return "container"
+ }
+
+ if baseRoleText == "radio group", role == kAXRadioGroupRole as String, title == nil, label != nil {
+ return ""
+ }
+
+ return baseRoleText
+}
+
func windowRelativeFrame(elementFrame: CGRect, windowBounds: CGRect) -> CGRect {
CGRect(
x: elementFrame.minX - windowBounds.minX,
@@ -505,7 +765,15 @@ func windowRelativeFrame(elementFrame: CGRect, windowBounds: CGRect) -> CGRect {
)
}
-private func humanizeRole(role: String, subrole: String?) -> String {
+private func roleDescription(of element: AXUIElement, role: String, subrole: String?) -> String {
+ if role == kAXRowRole as String {
+ return "row"
+ }
+
+ if let roleDescription = stringValue(of: element, attribute: kAXRoleDescriptionAttribute), !roleDescription.isEmpty {
+ return roleDescription.lowercased()
+ }
+
if let subrole, subrole == kAXStandardWindowSubrole as String {
return "standard window"
}
@@ -513,6 +781,31 @@ private func humanizeRole(role: String, subrole: String?) -> String {
return humanizeAXToken(role)
}
+private func meaningfulActions(_ values: [String], role: String) -> [String] {
+ values
+ .filter {
+ ![
+ kAXPressAction as String,
+ "AXShowDefaultUI",
+ "AXShowAlternateUI",
+ "AXShowMenu",
+ "AXConfirm",
+ ].contains($0)
+ }
+ .filter {
+ guard role == kAXScrollAreaRole as String else {
+ return true
+ }
+
+ if values.contains("AXScrollUpByPage") || values.contains("AXScrollDownByPage") {
+ return $0 != "AXScrollLeftByPage" && $0 != "AXScrollRightByPage"
+ }
+
+ return true
+ }
+ .map(prettyActionName(_:))
+}
+
private func prettyActionName(_ value: String) -> String {
let stripped = value.hasPrefix("AX") ? String(value.dropFirst(2)) : value
let withoutPage = stripped.replacingOccurrences(of: "ByPage", with: "")
@@ -547,12 +840,88 @@ private func sanitizeText(_ value: String) -> String {
return collapsed
}
-private func quoted(_ value: String) -> String {
- "\"\(value)\""
+private func flattenedRowTexts(of element: AXUIElement) -> [String] {
+ let cells = copyArray(element, attribute: kAXChildrenAttribute) ?? []
+ let texts = cells
+ .flatMap { descendantTexts(of: $0) }
+ .map(sanitizeText)
+ .filter { !$0.isEmpty }
+
+ var unique: [String] = []
+ var seen: Set<String> = []
+ for text in texts {
+ if seen.insert(text).inserted {
+ unique.append(text)
+ }
+ }
+
+ return unique
+}
+
+private func descendantTexts(of element: AXUIElement, depth: Int = 0) -> [String] {
+ guard depth < 4 else {
+ return []
+ }
+
+ var values: [String] = []
+ let role = stringValue(of: element, attribute: kAXRoleAttribute) ?? ""
+ if role == kAXStaticTextRole as String || role == kAXTextFieldRole as String {
+ if let value = sanitizedValue(of: element) {
+ values.append(value)
+ } else if let title = stringValue(of: element, attribute: kAXTitleAttribute) {
+ values.append(sanitizeText(title))
+ }
+ }
+
+ for child in copyArray(element, attribute: kAXChildrenAttribute) ?? [] {
+ values.append(contentsOf: descendantTexts(of: child, depth: depth + 1))
+ }
+
+ return values
}
-private func format(rect: CGRect) -> String {
- "x=\(Int(rect.origin.x)), y=\(Int(rect.origin.y)), w=\(Int(rect.width)), h=\(Int(rect.height))"
+private func visibleRows(in rows: [AXUIElement], parent: AXUIElement) -> [AXUIElement] {
+ guard let parentFrame = resolveLocalFrame(of: parent, windowBounds: nil) else {
+ return Array(rows.prefix(20))
+ }
+
+ let visible = rows.filter { row in
+ guard let rowFrame = resolveLocalFrame(of: row, windowBounds: nil) else {
+ return false
+ }
+
+ return rowFrame.intersects(parentFrame)
+ }
+
+ if visible.isEmpty {
+ return Array(rows.prefix(20))
+ }
+
+ return Array(visible.prefix(20))
+}
+
+private func displayIdentifier(_ value: String?) -> String? {
+ guard let value, !value.isEmpty, !value.hasPrefix("_NS:") else {
+ return nil
+ }
+
+ return value
+}
+
+private func displayWindowTitle(_ value: String?, appName: String) -> String {
+ guard let value, !value.isEmpty else {
+ return appName
+ }
+
+ if value.hasPrefix("\(appName) –") {
+ return appName
+ }
+
+ return value
+}
+
+private func quoted(_ value: String) -> String {
+ "\"\(value)\""
}
private extension CGRect {
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
index e305a31..fd5bd57 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
@@ -1,4 +1,5 @@
import AppKit
+import CoreServices
import Foundation
public struct RunningAppDescriptor {
@@ -8,8 +9,105 @@ public struct RunningAppDescriptor {
public let runningApplication: NSRunningApplication
}
+struct ListedAppDescriptor {
+ let name: String
+ let bundleIdentifier: String
+ let isRunning: Bool
+ let lastUsed: Date?
+ let uses: Int?
+
+ var renderedLine: String {
+ var markers: [String] = []
+ if isRunning {
+ markers.append("running")
+ }
+ if let lastUsed {
+ markers.append("last-used=\(AppDiscovery.usageDateFormatter.string(from: lastUsed))")
+ }
+ if let uses {
+ markers.append("uses=\(uses)")
+ }
+
+ return "\(name) — \(bundleIdentifier) [\(markers.joined(separator: ", "))]"
+ }
+}
+
+private struct SpotlightAppRecord {
+ let name: String
+ let bundleIdentifier: String
+ let lastUsed: Date?
+ let uses: Int?
+}
+
+private struct ResolvedAppInfo {
+ let bundleIdentifier: String
+ let name: String
+}
+
enum AppDiscovery {
- static func listApps() -> [RunningAppDescriptor] {
+ private static let listAppsQuery = #"kMDItemContentType == "com.apple.application-bundle" && kMDItemFSName == "*.app""#
+ private static let lastUsedDateRankingAttribute = "kMDItemLastUsedDate_Ranking"
+ private static let useCountAttribute = "kMDItemUseCount"
+ private static let maxRecentNonRunningApps = 10
+
+ static let usageDateFormatter: DateFormatter = {
+ let formatter = DateFormatter()
+ formatter.calendar = Calendar(identifier: .gregorian)
+ formatter.locale = Locale(identifier: "en_US_POSIX")
+ formatter.dateFormat = "yyyy-MM-dd"
+ return formatter
+ }()
+
+ static func listCatalog() -> [ListedAppDescriptor] {
+ let running = userFacingRunningApps()
+ let runningByBundle = running.reduce(into: [String: RunningAppDescriptor]()) { result, descriptor in
+ guard let bundleIdentifier = descriptor.bundleIdentifier, !bundleIdentifier.isEmpty else {
+ return
+ }
+
+ let key = bundleIdentifier.lowercased()
+ if result[key] == nil {
+ result[key] = descriptor
+ }
+ }
+
+ var entriesByBundle: [String: ListedAppDescriptor] = [:]
+
+ for record in SpotlightAppIndex.recentApps(cutoffDate: recentUsageCutoff()) {
+ let key = record.bundleIdentifier.lowercased()
+ let runningDescriptor = runningByBundle[key]
+ entriesByBundle[key] = ListedAppDescriptor(
+ name: runningDescriptor?.name ?? record.name,
+ bundleIdentifier: record.bundleIdentifier,
+ isRunning: runningDescriptor != nil,
+ lastUsed: record.lastUsed,
+ uses: record.uses
+ )
+ }
+
+ for descriptor in running {
+ guard let bundleIdentifier = descriptor.bundleIdentifier, !bundleIdentifier.isEmpty else {
+ continue
+ }
+
+ let key = bundleIdentifier.lowercased()
+ let existing = entriesByBundle[key]
+ entriesByBundle[key] = ListedAppDescriptor(
+ name: descriptor.name,
+ bundleIdentifier: bundleIdentifier,
+ isRunning: true,
+ lastUsed: existing?.lastUsed,
+ uses: existing?.uses
+ )
+ }
+
+ let sorted = entriesByBundle.values.sorted(by: compareListedApps)
+ let runningEntries = sorted.filter(\.isRunning)
+ let recentEntries = sorted.filter { !$0.isRunning }.prefix(maxRecentNonRunningApps)
+ return runningEntries + recentEntries
+ }
+
+ static func runningApps() -> [RunningAppDescriptor] {
NSWorkspace.shared.runningApplications
.filter { !$0.isTerminated }
.sorted { lhs, rhs in
@@ -30,34 +128,104 @@ enum AppDiscovery {
}
static func resolve(_ query: String) throws -> RunningAppDescriptor {
- let running = listApps()
+ let normalizedQuery = query.trimmingCharacters(in: .whitespacesAndNewlines)
+ let running = runningApps()
- if let match = running.first(where: { descriptor in
- descriptor.name.caseInsensitiveCompare(query) == .orderedSame
- || descriptor.bundleIdentifier?.caseInsensitiveCompare(query) == .orderedSame
- || descriptor.runningApplication.executableURL?.deletingPathExtension().lastPathComponent.caseInsensitiveCompare(query) == .orderedSame
- }) {
+ if let bundleIdentifier = blockedBundleIdentifier(forQuery: normalizedQuery) {
+ throw AppSafetyPolicy.permissionDenied(bundleIdentifier: bundleIdentifier)
+ }
+
+ if let match = resolvedRunningApp(in: running, matching: normalizedQuery) {
return match
}
- try launchIfPossible(query)
+ try launchIfPossible(normalizedQuery)
for _ in 0..<20 {
- if let launched = listApps().first(where: { descriptor in
- descriptor.name.caseInsensitiveCompare(query) == .orderedSame
- || descriptor.bundleIdentifier?.caseInsensitiveCompare(query) == .orderedSame
- }) {
+ if let launched = resolvedRunningApp(in: runningApps(), matching: normalizedQuery) {
return launched
}
Thread.sleep(forTimeInterval: 0.25)
}
- throw ComputerUseError.appNotFound(query)
+ throw ComputerUseError.appNotFound(normalizedQuery)
+ }
+
+ private static func resolvedRunningApp(in descriptors: [RunningAppDescriptor], matching query: String) -> RunningAppDescriptor? {
+ if isBundleIdentifierQuery(query) {
+ return descriptors.first(where: { descriptor in
+ descriptor.bundleIdentifier?.caseInsensitiveCompare(query) == .orderedSame
+ })
+ }
+
+ return descriptors.first(where: { descriptor in
+ guard !AppSafetyPolicy.isBlocked(bundleIdentifier: descriptor.bundleIdentifier) else {
+ return false
+ }
+
+ return descriptor.name.caseInsensitiveCompare(query) == .orderedSame
+ || descriptor.runningApplication.executableURL?.deletingPathExtension().lastPathComponent.caseInsensitiveCompare(query) == .orderedSame
+ })
+ }
+
+ private static func userFacingRunningApps() -> [RunningAppDescriptor] {
+ var seen: Set<String> = []
+ var descriptors: [RunningAppDescriptor] = []
+
+ for descriptor in runningApps() {
+ guard isUserFacingListApp(descriptor.runningApplication) else {
+ continue
+ }
+
+ guard let bundleIdentifier = descriptor.bundleIdentifier, !bundleIdentifier.isEmpty else {
+ continue
+ }
+
+ let key = bundleIdentifier.lowercased()
+ guard seen.insert(key).inserted else {
+ continue
+ }
+
+ descriptors.append(descriptor)
+ }
+
+ return descriptors
+ }
+
+ private static func compareListedApps(_ lhs: ListedAppDescriptor, _ rhs: ListedAppDescriptor) -> Bool {
+ if lhs.isRunning != rhs.isRunning {
+ return lhs.isRunning && !rhs.isRunning
+ }
+
+ let lhsHasUsage = lhs.lastUsed != nil
+ let rhsHasUsage = rhs.lastUsed != nil
+ if lhsHasUsage != rhsHasUsage {
+ return lhsHasUsage && !rhsHasUsage
+ }
+
+ let calendar = Calendar(identifier: .gregorian)
+ if let lhsLast = lhs.lastUsed, let rhsLast = rhs.lastUsed {
+ let lhsDay = calendar.startOfDay(for: lhsLast)
+ let rhsDay = calendar.startOfDay(for: rhsLast)
+ if lhsDay != rhsDay {
+ return lhsDay > rhsDay
+ }
+ }
+
+ if let lhsUses = lhs.uses, let rhsUses = rhs.uses, lhsUses != rhsUses {
+ return lhsUses > rhsUses
+ }
+
+ return lhs.name.localizedCaseInsensitiveCompare(rhs.name) == .orderedAscending
}
private static func launchIfPossible(_ query: String) throws {
- if query.contains(".") {
+ if isBundleIdentifierQuery(query) {
+ guard !AppSafetyPolicy.isBlocked(bundleIdentifier: query) else {
+ return
+ }
+
if let appURL = NSWorkspace.shared.urlForApplication(withBundleIdentifier: query) {
try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])
}
@@ -68,13 +236,205 @@ enum AppDiscovery {
return
}
- try NSWorkspace.shared.launchApplication(at: URL(fileURLWithPath: fullPath), options: [], configuration: [:])
+ let appURL = URL(fileURLWithPath: fullPath)
+ if AppSafetyPolicy.isBlocked(bundleIdentifier: Bundle(url: appURL)?.bundleIdentifier) {
+ return
+ }
+
+ try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])
+ }
+
+ private static func recentUsageCutoff(referenceDate: Date = Date()) -> Date {
+ let calendar = Calendar(identifier: .gregorian)
+ let startOfToday = calendar.startOfDay(for: referenceDate)
+ return calendar.date(byAdding: .day, value: -13, to: startOfToday) ?? startOfToday
+ }
+
+ private static func blockedBundleIdentifier(forQuery query: String) -> String? {
+ guard isBundleIdentifierQuery(query), AppSafetyPolicy.isBlocked(bundleIdentifier: query) else {
+ return nil
+ }
+
+ return query
+ }
+
+ private static func isBundleIdentifierQuery(_ query: String) -> Bool {
+ query.contains(".")
+ }
+
+ private static func isUserFacingListApp(_ app: NSRunningApplication) -> Bool {
+ app.activationPolicy == .regular
+ }
+
+ private static func bundleDisplayName(_ bundle: Bundle?) -> String? {
+ guard let bundle else {
+ return nil
+ }
+
+ let displayName = bundle.object(forInfoDictionaryKey: "CFBundleDisplayName") as? String
+ let bundleName = bundle.object(forInfoDictionaryKey: kCFBundleNameKey as String) as? String
+ return displayName ?? bundleName
+ }
+
+ private static func stripAppSuffix(from value: String) -> String {
+ value.hasSuffix(".app") ? String(value.dropLast(4)) : value
}
static func appName(_ app: NSRunningApplication) -> String {
app.localizedName
+ ?? bundleDisplayName(Bundle(url: app.bundleURL ?? URL(fileURLWithPath: "/")))
?? app.bundleURL?.deletingPathExtension().lastPathComponent
?? app.executableURL?.lastPathComponent
?? "pid-\(app.processIdentifier)"
}
+
+ private enum SpotlightAppIndex {
+ static func recentApps(cutoffDate: Date) -> [SpotlightAppRecord] {
+ let sortingAttributes = [
+ lastUsedDateRankingAttribute as CFString,
+ useCountAttribute as CFString,
+ ] as CFArray
+
+ guard let query = MDQueryCreate(
+ kCFAllocatorDefault,
+ listAppsQuery as CFString,
+ nil,
+ sortingAttributes
+ ) else {
+ return []
+ }
+
+ MDQuerySetSearchScope(query, standardSearchScopes() as CFArray, 0)
+ MDQuerySetSortOptionFlagsForAttribute(query, lastUsedDateRankingAttribute as CFString, kMDQueryReverseSortOrderFlag.rawValue)
+ MDQuerySetSortOptionFlagsForAttribute(query, useCountAttribute as CFString, kMDQueryReverseSortOrderFlag.rawValue)
+
+ guard MDQueryExecute(query, CFOptionFlags(kMDQuerySynchronous.rawValue)) else {
+ return []
+ }
+
+ var seen: Set<String> = []
+ var records: [SpotlightAppRecord] = []
+
+ for index in 0..<MDQueryGetResultCount(query) {
+ guard let rawResult = MDQueryGetResultAtIndex(query, index) else {
+ continue
+ }
+
+ let item = unsafeBitCast(rawResult, to: MDItem.self)
+ guard
+ let bundleIdentifier = stringAttribute(kMDItemCFBundleIdentifier, item: item),
+ !bundleIdentifier.isEmpty
+ else {
+ continue
+ }
+
+ let key = bundleIdentifier.lowercased()
+ guard seen.insert(key).inserted else {
+ continue
+ }
+
+ guard let path = stringAttribute(kMDItemPath, item: item) else {
+ continue
+ }
+
+ let appURL = URL(fileURLWithPath: path)
+ let bundle = Bundle(url: appURL)
+ if bundle?.object(forInfoDictionaryKey: "LSBackgroundOnly") as? Bool == true {
+ continue
+ }
+ if bundle?.object(forInfoDictionaryKey: "LSUIElement") as? Bool == true {
+ continue
+ }
+
+ let lastUsed = dateAttribute(lastUsedDateRankingAttribute as CFString, item: item)
+ ?? dateAttribute(kMDItemLastUsedDate, item: item)
+ guard let lastUsed, lastUsed >= cutoffDate else {
+ continue
+ }
+
+ let uses = numberAttribute(useCountAttribute as CFString, item: item)?.intValue
+ let displayName = bundleDisplayName(bundle)
+ ?? stringAttribute(kMDItemDisplayName, item: item).map(stripAppSuffix(from:))
+ ?? stripAppSuffix(from: appURL.lastPathComponent)
+
+ records.append(
+ SpotlightAppRecord(
+ name: displayName,
+ bundleIdentifier: bundleIdentifier,
+ lastUsed: lastUsed,
+ uses: uses
+ )
+ )
+ }
+
+ return records
+ }
+
+ private static func standardSearchScopes() -> [CFString] {
+ var scopes: [String] = [
+ "/Applications",
+ "/System/Applications",
+ "/System/Library/CoreServices",
+ ]
+
+ let homeApplications = NSString(string: "~/Applications").expandingTildeInPath
+ if FileManager.default.fileExists(atPath: homeApplications) {
+ scopes.append(homeApplications)
+ }
+
+ return scopes as [CFString]
+ }
+
+ private static func stringAttribute(_ name: CFString, item: MDItem) -> String? {
+ MDItemCopyAttribute(item, name) as? String
+ }
+
+ private static func numberAttribute(_ name: CFString, item: MDItem) -> NSNumber? {
+ MDItemCopyAttribute(item, name) as? NSNumber
+ }
+
+ private static func dateAttribute(_ name: CFString, item: MDItem) -> Date? {
+ MDItemCopyAttribute(item, name) as? Date
+ }
+ }
+}
+
+private enum AppSafetyPolicy {
+ private static let blockedBundleIdentifiers: Set<String> = [
+ "com.apple.ScreenContinuity",
+ "com.1password.1password",
+ "com.1password.safari",
+ "com.bitwarden.desktop",
+ "com.dashlane.dashlanephonefinal",
+ "com.lastpass.LastPass",
+ "com.nordsec.nordpass",
+ "me.proton.pass.electron",
+ "me.proton.pass.catalyst",
+ "com.apple.Terminal",
+ "com.googlecode.iterm2",
+ "dev.warp.Warp-Stable",
+ "net.kovidgoyal.kitty",
+ "com.github.wez.wezterm",
+ "com.mitchellh.ghostty",
+ "com.raphaelamorim.rio",
+ "dev.commandline.waveterm",
+ "com.google.Chrome",
+ "com.openai.atlas.alpha",
+ "com.openai.atlas.beta",
+ "com.apple.UserNotificationCenter",
+ "com.apple.LocalAuthenticationRemoteService",
+ "com.apple.SecurityAgent",
+ ]
+
+ static func isBlocked(bundleIdentifier: String?) -> Bool {
+ guard let bundleIdentifier else {
+ return false
+ }
+
+ return blockedBundleIdentifiers.contains(bundleIdentifier)
+ }
+
+ static func permissionDenied(bundleIdentifier: String) -> ComputerUseError {
+ .permissionDenied("Computer Use is not allowed to use the app '\(bundleIdentifier)' for safety reasons.")
+ }
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
index c4f66d9..2bc2dc0 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift
@@ -9,11 +9,8 @@ public final class ComputerUseService {
public func listApps() -> ToolCallResult {
ToolCallResult.text(
- AppDiscovery.listApps()
- .map { descriptor in
- let bundle = descriptor.bundleIdentifier ?? "no-bundle-id"
- return "\(descriptor.name) — \(bundle) [running, pid=\(descriptor.pid)]"
- }
+ AppDiscovery.listCatalog()
+ .map(\.renderedLine)
.joined(separator: "\n")
)
}
@@ -458,7 +455,8 @@ public final class ComputerUseService {
private func screenshotToGlobalPoint(snapshot: AppSnapshot, x: Double, y: Double) throws -> CGPoint {
guard let windowBounds = snapshot.windowBounds else {
- throw ComputerUseError.stateUnavailable("No window bounds are available for \(snapshot.app.name). Run get_app_state after bringing the app on screen.")
+ let appReference = snapshot.app.bundleIdentifier ?? snapshot.app.name
+ throw ComputerUseError.stateUnavailable("No window bounds are available for \(appReference). Run get_app_state after bringing the app on screen.")
}
return CGPoint(x: windowBounds.minX + x, y: windowBounds.minY + y)
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/Errors.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/Errors.swift
index 20bfb9f..7aecbfe 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/Errors.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/Errors.swift
@@ -26,12 +26,7 @@ public enum ComputerUseError: Error, LocalizedError {
}
var toolResultIsError: Bool {
- switch self {
- case .unsupportedTool, .invalidArguments:
- return true
- case .message, .appNotFound, .permissionDenied, .stateUnavailable:
- return false
- }
+ true
}
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
index e209dff..773209b 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift
@@ -1,5 +1,22 @@
import Foundation
+let computerUseServerInstructions = """
+Computer Use tools let you interact with macOS apps by performing UI actions.
+
+Some apps might have a separate dedicated plugin or skill. You may want to use that plugin or skill instead of Computer Use when it seems like a good fit for the task. While the separate plugin or skill may not expose every feature in the app, if the plugin can perform the task with its available features, prefer it. If the needed capability is not exposed there, use Computer Use may be appropriate for the missing interaction.
+
+Begin by calling `get_app_state` every turn you want to use Computer Use to get the latest state before acting. Codex will automatically stop the session after each assistant turn, so this step is required before interacting with apps in a new assistant turn.
+
+The available tools are list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value. If any of these are not available in your environment, use tool_search to surface one before calling any Computer Use action tools.
+
+Computer Use tools allow you to use the user's apps in the background, so while you're using an app, the user can continue to use other apps on their computer. Avoid doing anything that would disrupt the user's active session, such as overwriting the contents of their clipboard, unless they asked you to!
+
+After each action, use the action result or fetch the latest state to verify the UI changed as expected.
+Prefer element-targeted interactions over coordinate clicks when an index for the targeted element is available. Note that element indices are the sequential integers from the app state's accessibility tree.
+Avoid falling back to AppleScript during a computer use session. Prefer Computer Use tools as much as possible to complete tasks.
+Ask the user before taking destructive or externally visible actions such as sending, deleting, or purchasing. If helpful, you can ask follow-up questions before taking action to make sure you’re understanding the user’s request correctly.
+"""
+
public final class StdioMCPServer {
private let service: ComputerUseService
@@ -37,13 +54,14 @@ public final class StdioMCPServer {
"protocolVersion": "2025-03-26",
"serverInfo": [
"name": "open-computer-use",
- "version": "0.1.1",
+ "version": "0.1.2",
],
"capabilities": [
"tools": [
"listChanged": false,
],
],
+ "instructions": computerUseServerInstructions,
]
)
case "notifications/initialized":
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
index eb23d1b..b2a8e29 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ToolDefinitions.swift
@@ -3,117 +3,127 @@ import Foundation
public struct ToolDefinition: @unchecked Sendable {
public let name: String
public let description: String
+ public let annotations: [String: Any]
public let inputSchema: [String: Any]
+ public init(name: String, description: String, annotations: [String: Any], inputSchema: [String: Any]) {
+ self.name = name
+ self.description = description
+ self.annotations = annotations
+ self.inputSchema = inputSchema
+ }
+
public var asDictionary: [String: Any] {
- [
+ var dictionary: [String: Any] = [
"name": name,
"description": description,
"inputSchema": inputSchema,
]
+
+ if !annotations.isEmpty {
+ dictionary["annotations"] = annotations
+ }
+
+ return dictionary
}
}
public enum ToolDefinitions {
public static let all: [ToolDefinition] = [
- ToolDefinition(
- name: "list_apps",
- description: "List the apps on this computer. Returns the set of apps that are currently running, as well as any that have been used in the last 14 days, including details on usage frequency.",
- inputSchema: objectSchema(properties: [:], required: [])
- ),
- ToolDefinition(
- name: "get_app_state",
- description: "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app.",
- inputSchema: objectSchema(
- properties: [
- "app": stringProperty(description: "App name or bundle identifier"),
- ],
- required: ["app"]
- )
- ),
ToolDefinition(
name: "click",
- description: "Click an element by index or pixel coordinates from screenshot.",
+ description: "Click an element by index or pixel coordinates from screenshot. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
"element_index": stringProperty(description: "Element index to click"),
"x": numberProperty(description: "X coordinate in screenshot pixel coordinates"),
"y": numberProperty(description: "Y coordinate in screenshot pixel coordinates"),
- "click_count": integerProperty(description: "Number of clicks. Defaults to 1", defaultValue: 1),
+ "click_count": integerProperty(description: "Number of clicks. Defaults to 1"),
"mouse_button": stringProperty(
description: "Mouse button to click. Defaults to left.",
- enumValues: ["left", "right", "middle"],
- defaultValue: "left"
+ enumValues: ["left", "right", "middle"]
),
],
required: ["app"]
)
),
ToolDefinition(
- name: "perform_secondary_action",
- description: "Invoke a secondary accessibility action exposed by an element.",
+ name: "drag",
+ description: "Drag from one point to another using pixel coordinates. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
- "element_index": stringProperty(description: "Element identifier"),
- "action": stringProperty(description: "Secondary accessibility action name"),
+ "from_x": numberProperty(description: "Start X coordinate"),
+ "from_y": numberProperty(description: "Start Y coordinate"),
+ "to_x": numberProperty(description: "End X coordinate"),
+ "to_y": numberProperty(description: "End Y coordinate"),
],
- required: ["app", "element_index", "action"]
+ required: ["app", "from_x", "from_y", "to_x", "to_y"]
)
),
ToolDefinition(
- name: "scroll",
- description: "Scroll an element in a direction by a number of pages.",
+ name: "get_app_state",
+ description: "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`.",
+ annotations: readOnlyAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
- "direction": stringProperty(description: "Scroll direction: up, down, left, or right", enumValues: ["up", "down", "left", "right"]),
- "element_index": stringProperty(description: "Element identifier"),
- "pages": integerProperty(description: "Number of page scroll actions. Defaults to 1", defaultValue: 1),
],
- required: ["app", "direction", "element_index"]
+ required: ["app"]
)
),
ToolDefinition(
- name: "drag",
- description: "Drag from one point to another using pixel coordinates.",
+ name: "list_apps",
+ description: "List the apps on this computer. Returns the set of apps that are currently running, as well as any that have been used in the last 14 days, including details on usage frequency. This tool is part of plugin `Computer Use`.",
+ annotations: readOnlyAnnotations(),
+ inputSchema: objectSchema(properties: [:], required: [])
+ ),
+ ToolDefinition(
+ name: "perform_secondary_action",
+ description: "Invoke a secondary accessibility action exposed by an element. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
- "from_x": numberProperty(description: "Start X coordinate"),
- "from_y": numberProperty(description: "Start Y coordinate"),
- "to_x": numberProperty(description: "End X coordinate"),
- "to_y": numberProperty(description: "End Y coordinate"),
+ "element_index": stringProperty(description: "Element identifier"),
+ "action": stringProperty(description: "Secondary accessibility action name"),
],
- required: ["app", "from_x", "from_y", "to_x", "to_y"]
+ required: ["app", "element_index", "action"]
)
),
ToolDefinition(
- name: "type_text",
- description: "Type literal text using keyboard input.",
+ name: "press_key",
+ description: "Press a key or key-combination on the keyboard, including modifier and navigation keys.\n - This supports xdotool's `key` syntax.\n - Examples: \"a\", \"Return\", \"Tab\", \"super+c\", \"Up\", \"KP_0\" (for the numpad 0 key). This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
- "text": stringProperty(description: "Literal text to type"),
+ "key": stringProperty(description: "Key or key combination to press"),
],
- required: ["app", "text"]
+ required: ["app", "key"]
)
),
ToolDefinition(
- name: "press_key",
- description: "Press a key or key-combination on the keyboard, including modifier and navigation keys.\n- This supports xdotool's `key` syntax.\n- Examples: \"a\", \"Return\", \"Tab\", \"super+c\", \"Up\", \"KP_0\" (for the numpad 0 key).",
+ name: "scroll",
+ description: "Scroll an element in a direction by a number of pages. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
- "key": stringProperty(description: "Key or key combination to press"),
+ "direction": stringProperty(description: "Scroll direction: up, down, left, or right"),
+ "element_index": stringProperty(description: "Element identifier"),
+ "pages": integerProperty(description: "Number of page scroll actions. Defaults to 1"),
],
- required: ["app", "key"]
+ required: ["app", "element_index", "direction"]
)
),
ToolDefinition(
name: "set_value",
- description: "Set the value of a settable accessibility element.",
+ description: "Set the value of a settable accessibility element. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
inputSchema: objectSchema(
properties: [
"app": stringProperty(description: "App name or bundle identifier"),
@@ -123,6 +133,18 @@ public enum ToolDefinitions {
required: ["app", "element_index", "value"]
)
),
+ ToolDefinition(
+ name: "type_text",
+ description: "Type literal text using keyboard input. This tool is part of plugin `Computer Use`.",
+ annotations: defaultAnnotations(),
+ inputSchema: objectSchema(
+ properties: [
+ "app": stringProperty(description: "App name or bundle identifier"),
+ "text": stringProperty(description: "Literal text to type"),
+ ],
+ required: ["app", "text"]
+ )
+ ),
]
}
@@ -130,6 +152,7 @@ private func objectSchema(properties: [String: Any], required: [String]) -> [Str
var schema: [String: Any] = [
"type": "object",
"properties": properties,
+ "additionalProperties": false,
]
if !required.isEmpty {
@@ -139,7 +162,23 @@ private func objectSchema(properties: [String: Any], required: [String]) -> [Str
return schema
}
-private func stringProperty(description: String, enumValues: [String]? = nil, defaultValue: String? = nil) -> [String: Any] {
+private func defaultAnnotations() -> [String: Any] {
+ [
+ "destructiveHint": false,
+ "openWorldHint": false,
+ ]
+}
+
+private func readOnlyAnnotations() -> [String: Any] {
+ [
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": false,
+ "readOnlyHint": true,
+ ]
+}
+
+private func stringProperty(description: String, enumValues: [String]? = nil) -> [String: Any] {
var property: [String: Any] = [
"type": "string",
"description": description,
@@ -149,24 +188,14 @@ private func stringProperty(description: String, enumValues: [String]? = nil, de
property["enum"] = enumValues
}
- if let defaultValue {
- property["default"] = defaultValue
- }
-
return property
}
-private func integerProperty(description: String, defaultValue: Int? = nil) -> [String: Any] {
- var property: [String: Any] = [
+private func integerProperty(description: String) -> [String: Any] {
+ [
"type": "integer",
"description": description,
]
-
- if let defaultValue {
- property["default"] = defaultValue
- }
-
- return property
}
private func numberProperty(description: String) -> [String: Any] {
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 8dadbd1..4d69c8b 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -1,3 +1,4 @@
+import AppKit
import XCTest
@testable import OpenComputerUseKit
@@ -14,12 +15,25 @@ final class OpenComputerUseKitTests: XCTestCase {
func testInitializeResponseContainsToolsCapability() throws {
let server = StdioMCPServer(service: ComputerUseService())
- let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.1"},"capabilities":{}}}"#)
+ let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.2"},"capabilities":{}}}"#)
XCTAssertNotNil(response)
XCTAssertTrue(response!.contains(#""name":"open-computer-use""#))
XCTAssertTrue(response!.contains(#""tools":{"listChanged":false}"#))
}
+ func testInitializeResponseContainsComputerUseInstructions() throws {
+ let server = StdioMCPServer(service: ComputerUseService())
+ let response = try XCTUnwrap(
+ server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.2"},"capabilities":{}}}"#)
+ )
+ let data = try XCTUnwrap(response.data(using: .utf8))
+ let json = try XCTUnwrap(JSONSerialization.jsonObject(with: data) as? [String: Any])
+ let result = try XCTUnwrap(json["result"] as? [String: Any])
+ let instructions = try XCTUnwrap(result["instructions"] as? String)
+
+ XCTAssertEqual(instructions, computerUseServerInstructions)
+ }
+
func testWindowRelativeFrameUsesSharedGlobalCoordinates() {
let window = CGRect(x: 1486, y: 556, width: 919, height: 644)
let child = CGRect(x: 1486, y: 556, width: 919, height: 644)
@@ -35,22 +49,60 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(
tools["get_app_state"]?.description,
- "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app."
+ "Start an app use session if needed, then get the state of the app's key window and return a screenshot and accessibility tree. This must be called once per assistant turn before interacting with the app. This tool is part of plugin `Computer Use`."
)
XCTAssertTrue(tools["press_key"]?.description.contains("xdotool") == true)
XCTAssertEqual(
- (tools["click"]?.inputSchema["properties"] as? [String: [String: Any]])?["mouse_button"]?["default"] as? String,
- "left"
+ tools["click"]?.annotations["destructiveHint"] as? Bool,
+ false
)
XCTAssertEqual(
- (tools["click"]?.inputSchema["properties"] as? [String: [String: Any]])?["click_count"]?["default"] as? Int,
- 1
+ tools["get_app_state"]?.annotations["readOnlyHint"] as? Bool,
+ true
+ )
+ XCTAssertEqual(
+ tools["click"]?.inputSchema["additionalProperties"] as? Bool,
+ false
+ )
+ XCTAssertEqual(
+ ((tools["click"]?.inputSchema["properties"] as? [String: [String: Any]])?["mouse_button"]?["enum"] as? [String]) ?? [],
+ ["left", "right", "middle"]
+ )
+ }
+
+ func testSnapshotRenderedTextStartsDirectlyWithAppHeader() {
+ let snapshot = makeSnapshot(
+ treeLines: ["\t0 standard window Feishu"],
+ focusedSummary: "247 text entry area"
)
+
+ let rendered = snapshot.renderedText(style: .actionResult)
+ let lines = rendered.components(separatedBy: "\n")
+
+ XCTAssertEqual(lines.first, "App=com.electron.lark (pid 18465)")
+ XCTAssertEqual(lines.dropFirst().first, "Window: \"Feishu\", App: Feishu.")
+ XCTAssertFalse(rendered.contains("Computer Use state (CUA App Version: 750)"))
+ XCTAssertFalse(rendered.contains("<app_state>"))
+ XCTAssertFalse(rendered.contains("</app_state>"))
+ }
+
+ func testSnapshotSelectedTextUsesOfficialSingleLineFormat() {
+ let snapshot = makeSnapshot(
+ treeLines: ["\t38 search text field (settable, string) Codex"],
+ focusedSummary: nil,
+ selectedText: "Codex"
+ )
+
+ let rendered = snapshot.renderedText(style: .actionResult)
+
+ XCTAssertTrue(rendered.contains("Selected text: [Codex]"))
+ XCTAssertFalse(rendered.contains("Selected text: ```"))
+ XCTAssertFalse(rendered.contains("Pay special attention to the content selected by the user"))
}
func testComputerUseErrorsFormatLikeToolText() {
XCTAssertEqual(ComputerUseError.appNotFound("Sublime Text").errorDescription, #"appNotFound("Sublime Text")"#)
- XCTAssertFalse(ComputerUseError.appNotFound("Sublime Text").toolResultIsError)
+ XCTAssertTrue(ComputerUseError.appNotFound("Sublime Text").toolResultIsError)
XCTAssertTrue(ComputerUseError.invalidArguments("bad").toolResultIsError)
}
@@ -85,4 +137,25 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertNotEqual(midpoint.x, 110)
XCTAssertNotEqual(midpoint.y, 70)
}
+
+ private func makeSnapshot(treeLines: [String], focusedSummary: String?, selectedText: String? = nil) -> AppSnapshot {
+ AppSnapshot(
+ app: RunningAppDescriptor(
+ name: "Feishu",
+ bundleIdentifier: "com.electron.lark",
+ pid: 18_465,
+ runningApplication: NSRunningApplication.current
+ ),
+ windowTitle: "Feishu",
+ windowBounds: nil,
+ targetWindowID: nil,
+ targetWindowLayer: nil,
+ screenshotPNGData: nil,
+ mode: .accessibility,
+ treeLines: treeLines,
+ focusedSummary: focusedSummary,
+ selectedText: selectedText,
+ elements: [:]
+ )
+ }
}
diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.json
index 563a8fc..d52fbad 100644
--- a/plugins/open-computer-use/.codex-plugin/plugin.json
+++ b/plugins/open-computer-use/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-computer-use",
- "version": "0.1.1",
+ "version": "0.1.2",
"description": "Open-source macOS Computer Use MCP server packaged as a Codex plugin.",
"author": {
"name": "Leo",
diff --git a/scripts/build-open-computer-use-app.sh b/scripts/build-open-computer-use-app.sh
index 7860b0b..c3ad5e7 100755
--- a/scripts/build-open-computer-use-app.sh
+++ b/scripts/build-open-computer-use-app.sh
@@ -41,9 +41,9 @@ cat > "${contents_dir}/Info.plist" <<'PLIST'
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
- <string>0.1.1</string>
+ <string>0.1.2</string>
<key>CFBundleVersion</key>
- <string>2</string>
+ <string>3</string>
<key>LSMinimumSystemVersion</key>
<string>14.0</string>
<key>LSUIElement</key>
diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.md
index 35025f3..0a57604 100644
--- a/scripts/computer-use-cli/README.md
+++ b/scripts/computer-use-cli/README.md
@@ -85,7 +85,7 @@ Example working invocation against `open-computer-use`:
```bash
go run . list-tools \
--transport direct \
- --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.1/scripts/launch-open-computer-use.sh
+ --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.2/scripts/launch-open-computer-use.sh
```
Example working invocation against the official bundled `computer-use`:
diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.go
index da9942e..4777549 100644
--- a/scripts/computer-use-cli/main.go
+++ b/scripts/computer-use-cli/main.go
@@ -21,7 +21,7 @@ import (
const (
cliName = "computer-use-cli"
- cliVersion = "0.1.1"
+ cliVersion = "0.1.2"
defaultTimeout = 60 * time.Second
pluginRootEnvVar = "COMPUTER_USE_PLUGIN_ROOT"
serverBinEnvVar = "COMPUTER_USE_SERVER_BIN"