👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
Fix Feishu Electron snapshots
Leoba3ceec55a0b
diff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swiftindex 889f77c..686cf1e 100644--- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift+++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift@@ -187,7 +187,7 @@ enum OpenComputerUseSmokeSuite {}private static func runFullSmoke(serverURL: URL, appName: String) throws {- let client = try MCPClient(executableURL: serverURL, arguments: ["mcp"])+ let client = try MCPClient(executableURL: serverURL, arguments: ["mcp"], environment: smokeServerEnvironment())defer {client.terminate()}@@ -302,7 +302,7 @@ enum OpenComputerUseSmokeSuite {print("1. cursor idle observation setup")let observationURL = cursorObservationFileURL()try? FileManager.default.removeItem(at: observationURL)- var environment = ProcessInfo.processInfo.environment+ var environment = smokeServerEnvironment()environment["OPEN_COMPUTER_USE_VISUAL_CURSOR"] = "1"environment["OPEN_COMPUTER_USE_VISUAL_CURSOR_OBSERVATION_FILE"] = observationURL.path@@ -344,6 +344,12 @@ enum OpenComputerUseSmokeSuite {print("Cursor idle smoke completed.")}+ private static func smokeServerEnvironment() -> [String: String] {+ var environment = ProcessInfo.processInfo.environment+ environment["OPEN_COMPUTER_USE_DISABLE_APP_AGENT_PROXY"] = "1"+ return environment+ }+private static func locateProductsDirectory() throws -> URL {let executableURL = URL(fileURLWithPath: CommandLine.arguments[0]).resolvingSymlinksInPath()return executableURL.deletingLastPathComponent()diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex be85ac2..011fc43 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -60,7 +60,7 @@- `ComputerUseService` 负责把 Computer Use tool 请求映射到本地能力,`ComputerUseToolDispatcher` 则把 9 个 tool 的参数解析与 service 方法分发收敛成 MCP server 和 `open-computer-use call` 共用的一层。- `list_apps` 通过 Spotlight metadata query 拉取标准 application 目录里的 app bundle,并读取 `kMDItemUseCount` / `kMDItemLastUsedDate_Ranking` 这类系统元数据;再与 `NSWorkspace` 的运行态 app 合并,输出“当前运行中 + 近 14 天用过”的视图。-- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。+- `get_app_state` 优先走真实 AX / 窗口截图,但不再为了读状态而显式 `activate` 目标 app;当目标是仓库内 fixture app 时,回退到 fixture 导出的合成状态。真实 AX tree 对 Electron/WebView 这类深层 UI 会放宽遍历深度,同时压缩空 `AXGroup` / `AXUnknown` wrapper、过滤 `AXScrollToVisible` 噪音和空字符串属性,避免 action-critical 的输入框被无语义容器挤出节点预算。- MCP `tools/list` 的 description / input schema 当前按官方 `computer-use` 的 9 个 tools 文案和参数面收敛,尽量减少 host 侧提示词和 tool surface 偏差。- `open-computer-use call <tool> --args '{...}'` 会直接输出 MCP-style JSON result;`open-computer-use call --calls '[...]'` / `--calls-file <path>` 会在同一进程里顺序执行 JSON 数组里的 tool calls,并复用同一个 `ComputerUseService` 内存态,因此 `get_app_state` 之后的 action tool 可以继续使用同一轮 snapshot 的 `element_index`。序列执行默认会在成功的相邻操作之间 sleep 1 秒,也可以用 `--sleep <seconds>` 覆盖;遇到 `isError=true` 的 tool result 后停止。- 对真实 app 的 `get_app_state` / action tool 入口,当前只保留一层密码管理器 bundle denylist:bundle-id 直传时直接返回 safety denial;名称匹配时默认不解析到这些 app。终端、Chrome / Atlas 和系统组件不再属于内置阻止目标。@@ -125,7 +125,7 @@- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。-- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。+- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。## 主要验证路径diff --git a/docs/RELIABILITY.md b/docs/RELIABILITY.mdindex 1c85291..a25df21 100644--- a/docs/RELIABILITY.md+++ b/docs/RELIABILITY.md@@ -14,7 +14,7 @@- macOS 上必须给 `Open Computer Use.app` 授权 `Accessibility` 与 `Screen Recording`;终端本身不应该再是必需授权对象。- smoke suite 依赖本地 GUI session,不能把它当成无头环境命令。-- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异。+- 普通 app 的 `get_app_state` 结果依赖 AX tree 和窗口截图,复杂 app 上输出会有差异;Electron/WebView app 的 AX tree 通常很深,当前会压缩空 wrapper 并放宽遍历深度,以优先保留可操作文本、按钮和输入框。- Linux runtime 依赖已登录桌面用户 session;缺少 `XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS` 或 display 环境时,会尝试从 `/run/user/<uid>` 和常见桌面进程自动发现当前用户的 session env。纯 SSH tty 如果找不到桌面 session 仍不能直接访问 AT-SPI GUI tree。- GNOME Wayland 截图可能被 compositor 限制,当前 Linux bridge 会把黑图视为无效截图并省略 image block。@@ -29,7 +29,7 @@## 后续补强方向- 增加结构化日志和失败原因分类。-- 对 screenshot capture / AX traversal 增加更明确的 timeout 和 fallback。+- 继续补充 screenshot capture / AX traversal 的失败上下文和普通 app 回归样本。- 增加普通 app 回归样本,而不是只覆盖 fixture。CI/CD 流程结构和 release 自动化的默认方案,统一写在 `docs/CICD.md`。diff --git a/docs/exec-plans/completed/20260507-feishu-electron-snapshot-stability.md b/docs/exec-plans/completed/20260507-feishu-electron-snapshot-stability.mdnew file mode 100644index 0000000..36b4f81--- /dev/null+++ b/docs/exec-plans/completed/20260507-feishu-electron-snapshot-stability.md@@ -0,0 +1,45 @@+# Feishu Electron Snapshot Stability++## 目标++对比官方 `computer-use` 与本仓库 `open-computer-use` 操作 Feishu 的真实表现,修复本仓库在 Electron/WebView 深层 UI 上 `get_app_state` 树渲染不完整的问题,并用可重复命令验证修复有效。++## 范围++- 包含:macOS `get_app_state` 的 AX tree traversal、Feishu/Electron 深层 UI 的手工对比验证、必要的单元测试与文档/history。+- 不包含:重写输入模拟策略、引入新 MCP tool、处理 Windows/Linux runtime 的 Electron 兼容性。++## 背景++- 相关文档:`docs/ARCHITECTURE.md`、`docs/RELIABILITY.md`、`docs/QUALITY_SCORE.md`+- 相关代码路径:`packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- 已知约束:普通第三方 app 的 AX tree 和截图依赖 macOS Accessibility 与 ScreenCaptureKit;Electron app 的 WebView 层级比原生 AppKit 控件更深。++## 风险++- 风险:放宽遍历深度可能让某些复杂 app 输出过长或遍历耗时增加。+- 缓解方式:保留节点总数上限,并增加测试覆盖深层树可达性和节点预算。++## 里程碑++1. 对比官方 `computer-use` 与 `open-computer-use` 在 Feishu 上的 `list_apps` / `get_app_state` 行为。+2. 收敛代码层原因并实现最小修复。+3. 运行单元测试、smoke 和 Feishu 实机回归,记录结果。++## 验证方式++- 命令:`swift test`+- 命令:`./scripts/run-tool-smoke-tests.sh`+- 手工检查:分别用 `computer-use` 和 `open-computer-use` 对 `com.electron.lark` 执行 `get_app_state`,确认本仓库输出能覆盖聊天消息与输入框,并返回截图。++## 进度记录++- [x] 确认两个 MCP 都能发现运行中的 `Feishu — com.electron.lark`。+- [x] 复现差异:官方 `get_app_state` 能展开到消息与 entry area,本仓库截图正常但 AX 树停在浅层 WebView 容器。+- [x] 完成最小修复与测试。+- [x] 完成修复后验证并记录 history。++## 决策记录++- 2026-05-07:优先修 AX tree traversal 深度,不改变动作型 tool 的输入策略;本轮复现的失败点是树过浅,截图链路在现场样本中已经返回有效 PNG。+- 2026-05-07:仅加深 traversal 仍会被 Electron 空 wrapper 消耗 500 节点预算,因此同时过滤空字符串、`AXScrollToVisible` 噪音和无语义 generic wrapper;修复后 Feishu state 能在 500 节点内返回消息、entry area 和 PNG 截图。diff --git a/docs/histories/2026-05/20260507-1638-feishu-electron-snapshot-stability.md b/docs/histories/2026-05/20260507-1638-feishu-electron-snapshot-stability.mdnew file mode 100644index 0000000..70ec692--- /dev/null+++ b/docs/histories/2026-05/20260507-1638-feishu-electron-snapshot-stability.md@@ -0,0 +1,30 @@+## [2026-05-07 16:38] | Task: Stabilize Feishu Electron snapshots++### Execution Context+* **Agent ID**: `Codex`+* **Base Model**: `GPT-5`+* **Runtime**: `macOS local CLI/MCP`++### User Query+> 对比 `computer-use` 与 `open-computer-use` 操作 Feishu 是否正常,重点验证给 xuyusong 发消息、排查 `open-computer-use` UI tree / screenshot 偶发不稳定,并先验证再修复、修复后再验证。++### Changes Overview+**Scope:** macOS snapshot rendering in `OpenComputerUseKit`++**Key Actions:**+- **[Electron AX tree]**: Increased the AX traversal depth budget and compressed empty generic wrappers so Feishu/Electron WebView content can expose chat messages and the entry area within the existing 500 node cap.+- **[AX noise filtering]**: Stopped treating empty strings, `selected=false`, `expanded=false`, and `AXScrollToVisible` as meaningful output, reducing tree noise and making action-critical nodes visible sooner.+- **[Screenshot timeout]**: Added a ScreenCaptureKit capture timeout so screenshot stalls omit the image block instead of blocking the full app state call.+- **[Smoke reliability]**: Made the smoke suite disable the app-agent proxy for `.build/debug/OpenComputerUse mcp`, so local smoke verifies the just-built server and visual cursor observation file.+- **[Verification]**: Compared official `computer-use` and fixed local `open-computer-use` against `com.electron.lark`; both sent test messages to xuyusong after the fix.++### Design Intent (Why)+Feishu is an Electron app, and its WebView exposes a much deeper and noisier AX tree than native AppKit controls. The old renderer stopped at depth 16 and preserved empty wrapper nodes, so the screenshot could be present while the useful UI tree missed the chat input. The fix keeps the global 500-node safety cap but spends that budget on semantic nodes.++### Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift`+- `docs/ARCHITECTURE.md`+- `docs/RELIABILITY.md`+- `docs/exec-plans/completed/20260507-feishu-electron-snapshot-stability.md`diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 68591c9..4849be2 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -27,6 +27,10 @@ enum SnapshotMode {case fixture}+let accessibilityTreeMaxNodeCount = 500+let accessibilityTreeMaxDepth = 64+let screenshotCaptureTimeout: TimeInterval = 5+public struct AppSnapshot {public let app: RunningAppDescriptorpublic let windowTitle: String?@@ -239,7 +243,7 @@ private struct WindowCapture {}private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? {- try? BlockingAsyncBridge.run {+ try? BlockingAsyncBridge.run(timeout: screenshotCaptureTimeout) {let shareableContent = try await SCShareableContent.currentguard let window = shareableContent.windows.first(where: { $0.windowID == windowID }) else {return nil@@ -279,12 +283,12 @@ private final class AsyncResultBox<T>: @unchecked Sendable {var result: Result<T, Error>?}-private enum BlockingAsyncBridge {- static func run<T>(_ operation: @escaping @Sendable () async throws -> T) throws -> T {+enum BlockingAsyncBridge {+ static func run<T>(timeout: TimeInterval? = nil, _ operation: @escaping @Sendable () async throws -> T) throws -> T {let semaphore = DispatchSemaphore(value: 0)let resultBox = AsyncResultBox<T>()- Task.detached {+ let task = Task.detached {do {resultBox.result = .success(try await operation())} catch {@@ -294,21 +298,36 @@ private enum BlockingAsyncBridge {semaphore.signal()}- waitForSignal(semaphore)+ guard waitForSignal(semaphore, timeout: timeout) else {+ task.cancel()+ throw ComputerUseError.message("ScreenCaptureKit screenshot task timed out after \(timeout ?? 0) seconds.")+ }+return try resultBox.result?.get() ?? {throw ComputerUseError.message("ScreenCaptureKit screenshot task finished without producing a result.")}()}- private static func waitForSignal(_ semaphore: DispatchSemaphore) {+ private static func waitForSignal(_ semaphore: DispatchSemaphore, timeout: TimeInterval?) -> Bool {+ let deadline = timeout.map { Date(timeIntervalSinceNow: $0) }+if Thread.isMainThread {while semaphore.wait(timeout: .now()) == .timedOut {+ if let deadline, Date() >= deadline {+ return false+ }+RunLoop.current.run(mode: .default, before: Date(timeIntervalSinceNow: 0.01))}- return+ return true+ }++ if let timeout {+ return semaphore.wait(timeout: .now() + timeout) == .success}semaphore.wait()+ return true}}@@ -331,7 +350,7 @@ private struct TreeRenderer {}mutating func render(_ root: AXUIElement, depth: Int = 0) {- guard nextIndex < 500, depth < 16 else {+ guard shouldContinueRendering(nextIndex: nextIndex, depth: depth) else {return}@@ -484,15 +503,19 @@ private struct TreeRenderer {}}+func shouldContinueRendering(nextIndex: Int, depth: Int) -> Bool {+ nextIndex < accessibilityTreeMaxNodeCount && depth < accessibilityTreeMaxDepth+}+private func summarizeTraits(of element: AXUIElement) -> [String] {var values: [String] = []- if let selected = boolValue(of: element, attribute: kAXSelectedAttribute) {- values.append(selected ? "selected" : "selectable")+ if boolValue(of: element, attribute: kAXSelectedAttribute) == true {+ values.append("selected")}- if let expanded = boolValue(of: element, attribute: kAXExpandedAttribute) {- values.append(expanded ? "expanded" : "collapsed")+ if boolValue(of: element, attribute: kAXExpandedAttribute) == true {+ values.append("expanded")}if boolValue(of: element, attribute: kAXEnabledAttribute) == false {@@ -576,7 +599,11 @@ private func stringValue(of element: AXUIElement, attribute: String) -> String?}if CFGetTypeID(value) == CFStringGetTypeID() {- return value as? String+ guard let string = value as? String else {+ return nil+ }++ return string.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty ? nil : string}return nil@@ -613,7 +640,8 @@ private func isSettable(of element: AXUIElement, attribute: String) -> Bool {private func sanitizedValue(of element: AXUIElement) -> String? {if let string = stringValue(of: element, attribute: kAXValueAttribute) {- return sanitizeText(string)+ let sanitized = sanitizeText(string)+ return sanitized.isEmpty ? nil : sanitized}guard let value = attributeValue(of: element, attribute: kAXValueAttribute) else {@@ -738,7 +766,7 @@ private func resolveLocalFrame(of element: AXUIElement, windowBounds: CGRect?) -return windowRelativeFrame(elementFrame: frame, windowBounds: windowBounds)}-private func shouldElideNode(+func shouldElideNode(role: String,title: String?,label: String?,@@ -748,10 +776,6 @@ private func shouldElideNode(actions: [String],childCount: Int) -> Bool {- guard childCount == 1 else {- return false- }-let genericRoles = [kAXGroupRole as String, kAXUnknownRole as String]guard genericRoles.contains(role) else {return false@@ -763,6 +787,7 @@ private func shouldElideNode(&& identifier == nil&& traits.isEmpty&& actions.isEmpty+ && childCount > 0}private func shouldSuppressChildren(@@ -841,7 +866,7 @@ private func roleDescription(of element: AXUIElement, role: String, subrole: Strreturn humanizeAXToken(role)}-private func meaningfulActions(_ values: [String], role: String) -> [String] {+func meaningfulActions(_ values: [String], role: String) -> [String] {values.filter {![@@ -850,6 +875,7 @@ private func meaningfulActions(_ values: [String], role: String) -> [String] {"AXShowAlternateUI","AXShowMenu","AXConfirm",+ "AXScrollToVisible",].contains($0)}.filter {diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex cdbb7ae..f3117de 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -539,6 +539,61 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertFalse(rendered.contains("Pay special attention to the content selected by the user"))}+ func testAccessibilityTreeBudgetAllowsDeepElectronWebViews() {+ XCTAssertEqual(accessibilityTreeMaxNodeCount, 500)+ XCTAssertEqual(accessibilityTreeMaxDepth, 64)+ XCTAssertTrue(shouldContinueRendering(nextIndex: 120, depth: 16))+ XCTAssertTrue(shouldContinueRendering(nextIndex: 499, depth: 63))+ XCTAssertFalse(shouldContinueRendering(nextIndex: 500, depth: 20))+ XCTAssertFalse(shouldContinueRendering(nextIndex: 120, depth: 64))+ }++ func testAccessibilityRendererElidesEmptyGenericElectronWrappers() {+ XCTAssertTrue(shouldElideNode(+ role: kAXGroupRole as String,+ title: nil,+ label: nil,+ value: nil,+ identifier: nil,+ traits: [],+ actions: [],+ childCount: 3+ ))+ XCTAssertFalse(shouldElideNode(+ role: kAXGroupRole as String,+ title: "Send",+ label: nil,+ value: nil,+ identifier: nil,+ traits: [],+ actions: [],+ childCount: 3+ ))+ }++ func testAccessibilityRendererFiltersScrollToVisibleNoise() {+ XCTAssertEqual(+ meaningfulActions(+ [kAXPressAction as String, "AXScrollToVisible", "AXShowMenu", "AXRaise"],+ role: kAXButtonRole as String+ ),+ ["Raise"]+ )+ }++ func testBlockingAsyncBridgeTimesOutScreenshotWork() {+ XCTAssertThrowsError(+ try BlockingAsyncBridge.run(timeout: 0.01) {+ try await Task.sleep(nanoseconds: 200_000_000)+ return "late"+ }+ ) { error in+ XCTAssertTrue(+ (error as? ComputerUseError)?.errorDescription?.contains("timed out") == true+ )+ }+ }+func testComputerUseErrorsFormatLikeToolText() {XCTAssertEqual(ComputerUseError.appNotFound("Sublime Text").errorDescription, #"appNotFound("Sublime Text")"#)XCTAssertTrue(ComputerUseError.appNotFound("Sublime Text").toolResultIsError)