👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
release: bump version to 0.1.48
Leof02d35f48409
diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.goindex eeafc33..464561d 100644--- a/apps/OpenComputerUseLinux/main.go+++ b/apps/OpenComputerUseLinux/main.go@@ -19,7 +19,7 @@ import ("time")-var version = "0.1.47"+var version = "0.1.48"//go:embed runtime.pyvar linuxRuntimeScript stringdiff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swiftindex 67e8e21..0d4d947 100644--- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift+++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift@@ -35,7 +35,7 @@ final class MCPClient {_ = try request(method: "initialize", params: ["clientInfo": ["name": "OpenComputerUseSmokeSuite",- "version": "0.1.47",+ "version": "0.1.48",],"capabilities": [:],"protocolVersion": "2025-03-26",diff --git a/apps/OpenComputerUseWindows/main.go b/apps/OpenComputerUseWindows/main.goindex 9d8979b..173f912 100644--- a/apps/OpenComputerUseWindows/main.go+++ b/apps/OpenComputerUseWindows/main.go@@ -16,7 +16,7 @@ import ("time")-var version = "0.1.47"+var version = "0.1.48"//go:embed runtime.ps1var windowsRuntimeScript stringdiff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex b01a964..683b968 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -125,7 +125,7 @@- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。-- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。+- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;编码前会按最大尺寸和目标字节数自适应缩小,避免复杂页面的大 PNG 触发 host 侧 MCP result 降级,同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。## 主要验证路径diff --git a/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.md b/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.mdnew file mode 100644index 0000000..e747fc3--- /dev/null+++ b/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.md@@ -0,0 +1,22 @@+# Bound screenshot result size++## 用户诉求++用户发现 `open-computer-use` 的部分 tool call 在 Codex 历史记录里变成 `text` 下的 JSON 字符串,怀疑最近几次 commit 引入了额外包装,并提供了历史 session id 辅助定位。++## 本次改动++- **[Root cause]**: 确认 MCP server 原始 `tools/call` 响应是标准 `content: [text, image]`,嵌套只出现在 Codex 为了持久化超大 MCP result 而做的事件日志降级路径里。+- **[Screenshot bound]**: macOS `ScreenCaptureKit` 窗口截图在编码为 PNG 前按最大尺寸和目标字节数自适应缩小,减少复杂页面触发 host 侧大结果降级的概率。+- **[Regression tests]**: 增加截图压缩边界测试,覆盖大图会缩小、小图保持原尺寸。+- **[Docs]**: 更新架构文档,说明截图仍以 MCP image block 返回,但会做尺寸/字节上限控制,coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射。++## 设计动机++问题不是协议层重复包了一层,而是截图 PNG 过大时,Codex 会把整个 `CallToolResult` 序列化成一个 text preview 来保护 rollout storage。直接改 MCP shape 会破坏兼容性;更稳妥的修复是控制截图结果大小,让正常 `get_app_state` / action tool 返回保持扁平的 `text + image` 结构。++## 影响文件++- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`+- `docs/ARCHITECTURE.md`diff --git a/docs/histories/2026-05/20260508-1518-release-0.1.48.md b/docs/histories/2026-05/20260508-1518-release-0.1.48.mdnew file mode 100644index 0000000..7478a80--- /dev/null+++ b/docs/histories/2026-05/20260508-1518-release-0.1.48.md@@ -0,0 +1,39 @@+# Release 0.1.48++## 用户诉求++用户要求提交本轮相关改动并推送发版,同时确认 Codex TOML 配置改回连接线上 `open-computer-use`,不要指向 dev 构建。++## 本次改动++- **[Version bump]**: 将 macOS / Linux / Windows runtime、插件 manifest、smoke suite、测试和 reverse-engineering helper 版本统一 bump 到 `0.1.48`。+- **[Release notes]**: 更新用户可见 release notes,说明本版解决复杂窗口截图过大导致 Codex 历史事件降级成 JSON text preview 的问题。+- **[Config check]**: 确认本机 Codex 配置中的 `open-computer-use` MCP server 使用线上命令 `open-computer-use mcp`,没有指向本地 dev binary。++## 设计动机++本轮修复已经影响公开 npm 包里的 macOS runtime 行为,且用户明确要求发版,因此按 release guide 走 patch release。版本源继续以插件 manifest 为准,并同步所有辅助工具与测试里的版本字符串,避免 tag 与 npm staging 包版本不一致。++## 验证++- `swift test`+- `./scripts/run-tool-smoke-tests.sh`+- `node ./scripts/npm/build-packages.mjs --out-dir dist/release/npm-staging-check`+- `./scripts/build-cursor-motion-dmg.sh --configuration release --arch universal --version 0.1.48`+- `node -p "require('./dist/release/npm-staging-check/open-computer-use/package.json').version"` -> `0.1.48`+- `test -x dist/release/npm-staging-check/open-computer-use/dist/linux/arm64/open-computer-use`+- `test -f dist/release/npm-staging-check/open-computer-use/dist/windows/arm64/open-computer-use.exe`+- `node -e "if (require('./dist/release/npm-staging-check/open-computer-use/package.json').optionalDependencies) process.exit(1)"`+- `ls dist/release/cursor-motion/CursorMotion-0.1.48.dmg`+- `git diff --check`++## 影响文件++- `plugins/open-computer-use/.codex-plugin/plugin.json`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift`+- `apps/OpenComputerUseLinux/main.go`+- `apps/OpenComputerUseWindows/main.go`+- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift`+- `scripts/computer-use-cli/main.go`+- `scripts/computer-use-cli/README.md`+- `docs/releases/feature-release-notes.md`diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.mdindex db670b3..d03ac47 100644--- a/docs/releases/feature-release-notes.md+++ b/docs/releases/feature-release-notes.md@@ -4,6 +4,7 @@| 日期 | 功能域 | 用户价值 | 变更摘要 || --- | --- | --- | --- |+| 2026-05-08 | MCP 结果大小控制 | Chrome / Electron 等复杂窗口返回截图时更不容易触发 Codex 的大结果降级,历史记录和上层工具消费里会继续保持正常的 `text + image` 结构,而不是把整个 MCP result 塞进 text 字符串。 | 发布 `0.1.48`,macOS `ScreenCaptureKit` 截图在编码为 MCP image block 前会按最大尺寸和目标字节数自适应缩小,并补测试锁住大图压缩、小图保真路径。 || 2026-05-08 | Smoke 测试静默运行 | 开发者运行 `./scripts/run-tool-smoke-tests.sh` 时不再被内部 fixture 的橙色测试窗口打断,回归验证更安静,也不会让用户误以为桌面上弹出了额外应用。 | 发布 `0.1.47`,smoke suite 默认以 headless 模式启动内部 fixture,并保留 fixture 在 `list_apps` smoke 路径里的可发现性。 || 2026-05-07 | 窗口不可用错误文案 | Lark / Chrome 等 no-window 场景下,开源版 `get_app_state` 的返回文本与官方 `computer-use` 当前输出一致,减少上层 prompt、测试和用户排查时的差异噪音。 | 发布 `0.1.46`,将 missing AX window 和 missing visible CGWindow 两条路径统一为 `Apple event error -10005: cgWindowNotFound`,并补单元测试锁住该官方风格文本。 || 2026-05-07 | 可见窗口匹配 | `get_app_state` 在 app 没有可见窗口、窗口已最小化,或 AX 窗口无法匹配到 on-screen CGWindow 时会更接近官方 `computer-use` 的 `cgWindowNotFound` 语义,避免返回没有截图坐标基础的半残状态。 | 发布 `0.1.45`,基于官方 `1.0.770` 逆向字符串中的 `cgWindowNotFound`、`noWindowsAvailable`、`matchingWindowNotFound` 和 `AXWindowMiniaturized` 线索,要求真实 app snapshot 同时满足未最小化 `AXWindow` 与可见 `CGWindow`。 |diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 1fdfc37..ab22860 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -30,6 +30,9 @@ enum SnapshotMode {let accessibilityTreeMaxNodeCount = 1200let accessibilityTreeMaxDepth = 64let screenshotCaptureTimeout: TimeInterval = 5+let screenshotResultMaxPNGBytes = 900_000+let screenshotResultMaxDimension: CGFloat = 1280+let screenshotResultMinScale: CGFloat = 0.25private let windowVisibilityRecoveryDelay: TimeInterval = 0.7private let axWebAreaRole = "AXWebArea"@@ -412,11 +415,75 @@ private struct WindowCapture {return nil}- let bitmap = NSBitmapImageRep(cgImage: image)- return bitmap.representation(using: .png, properties: [:])+ return boundedScreenshotPNGData(for: image)}}+func boundedScreenshotPNGData(+ for image: CGImage,+ maxBytes: Int = screenshotResultMaxPNGBytes,+ maxDimension: CGFloat = screenshotResultMaxDimension,+ minScale: CGFloat = screenshotResultMinScale+) -> Data? {+ guard image.width > 0, image.height > 0, maxBytes > 0 else {+ return nil+ }++ let original = pngData(for: image)+ let largestDimension = CGFloat(max(image.width, image.height))+ var scale = min(1, maxDimension / largestDimension)++ if scale >= 1, let original, original.count <= maxBytes {+ return original+ }++ var best = original+ while scale >= minScale {+ guard let resized = resizedCGImage(image, scale: scale),+ let data = pngData(for: resized)+ else {+ break+ }++ best = data+ if data.count <= maxBytes {+ return data+ }++ scale *= 0.85+ }++ return best+}++private func pngData(for image: CGImage) -> Data? {+ let bitmap = NSBitmapImageRep(cgImage: image)+ return bitmap.representation(using: .png, properties: [:])+}++private func resizedCGImage(_ image: CGImage, scale: CGFloat) -> CGImage? {+ let width = max(1, Int((CGFloat(image.width) * scale).rounded()))+ let height = max(1, Int((CGFloat(image.height) * scale).rounded()))+ let colorSpace = CGColorSpaceCreateDeviceRGB()+ let bitmapInfo = CGImageAlphaInfo.premultipliedLast.rawValue++ guard let context = CGContext(+ data: nil,+ width: width,+ height: height,+ bitsPerComponent: 8,+ bytesPerRow: 0,+ space: colorSpace,+ bitmapInfo: bitmapInfo+ ) else {+ return nil+ }++ context.interpolationQuality = .medium+ context.draw(image, in: CGRect(x: 0, y: 0, width: width, height: height))+ return context.makeImage()+}+private final class AsyncResultBox<T>: @unchecked Sendable {var result: Result<T, Error>?}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swiftindex 4337a2e..19ad9ca 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift@@ -1,6 +1,6 @@import Foundation-public let openComputerUseVersion = "0.1.47"+public let openComputerUseVersion = "0.1.48"public func resolvedOpenComputerUseVersion(bundle: Bundle = .main) -> String {if let version = bundle.object(forInfoDictionaryKey: "CFBundleShortVersionString") as? String,diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swiftindex 217399a..c39307f 100644--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift@@ -1,4 +1,5 @@import AppKit+import ImageIOimport XCTest@testable import OpenComputerUseKit@@ -136,6 +137,29 @@ final class OpenComputerUseKitTests: XCTestCase {XCTAssertEqual(resolvedOpenComputerUseVersion(bundle: Bundle(for: Self.self)), openComputerUseVersion)}+ func testBoundedScreenshotPNGDataShrinksLargeScreenshots() throws {+ let image = try makeNoisyTestImage(width: 800, height: 600)+ let data = try XCTUnwrap(boundedScreenshotPNGData(+ for: image,+ maxBytes: 50_000,+ maxDimension: 320,+ minScale: 0.05+ ))+ let size = try imageSize(in: data)++ XCTAssertLessThanOrEqual(data.count, 50_000)+ XCTAssertLessThanOrEqual(max(size.width, size.height), 320)+ }++ func testBoundedScreenshotPNGDataKeepsSmallScreenshotsAtOriginalSize() throws {+ let image = try makeSolidTestImage(width: 32, height: 24)+ let data = try XCTUnwrap(boundedScreenshotPNGData(for: image, maxBytes: 1_000_000, maxDimension: 320))+ let size = try imageSize(in: data)++ XCTAssertEqual(size.width, 32)+ XCTAssertEqual(size.height, 24)+ }+func testToolDefinitionCount() {XCTAssertEqual(ToolDefinitions.all.count, 9)}@@ -436,7 +460,7 @@ final class OpenComputerUseKitTests: XCTestCase {func testInitializeResponseContainsToolsCapability() throws {let server = StdioMCPServer(service: ComputerUseService())- let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.47"},"capabilities":{}}}"#)+ let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.48"},"capabilities":{}}}"#)XCTAssertNotNil(response)XCTAssertTrue(response!.contains(#""name":"open-computer-use""#))XCTAssertTrue(response!.contains(#""tools":{"listChanged":false}"#))@@ -445,7 +469,7 @@ final class OpenComputerUseKitTests: XCTestCase {func testInitializeResponseContainsComputerUseInstructions() throws {let server = StdioMCPServer(service: ComputerUseService())let response = try XCTUnwrap(- server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.47"},"capabilities":{}}}"#)+ server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.48"},"capabilities":{}}}"#))let data = try XCTUnwrap(response.data(using: .utf8))let json = try XCTUnwrap(JSONSerialization.jsonObject(with: data) as? [String: Any])@@ -1420,4 +1444,59 @@ final class OpenComputerUseKitTests: XCTestCase {let length = max(hypot(dx, dy), 0.001)return CGVector(dx: dx / length, dy: dy / length)}++ private func makeNoisyTestImage(width: Int, height: Int) throws -> CGImage {+ var pixels = [UInt8](repeating: 0, count: width * height * 4)+ for y in 0..<height {+ for x in 0..<width {+ let offset = (y * width + x) * 4+ pixels[offset] = UInt8((x * 37 + y * 17) & 0xFF)+ pixels[offset + 1] = UInt8((x * 11 + y * 43) & 0xFF)+ pixels[offset + 2] = UInt8((x * 71 + y * 5) & 0xFF)+ pixels[offset + 3] = 255+ }+ }++ return try makeTestImage(width: width, height: height, pixels: pixels)+ }++ private func makeSolidTestImage(width: Int, height: Int) throws -> CGImage {+ var pixels = [UInt8](repeating: 0, count: width * height * 4)+ for index in stride(from: 0, to: pixels.count, by: 4) {+ pixels[index] = 80+ pixels[index + 1] = 140+ pixels[index + 2] = 220+ pixels[index + 3] = 255+ }++ return try makeTestImage(width: width, height: height, pixels: pixels)+ }++ private func makeTestImage(width: Int, height: Int, pixels: [UInt8]) throws -> CGImage {+ let data = Data(pixels)+ let provider = try XCTUnwrap(CGDataProvider(data: data as CFData))+ let image = CGImage(+ width: width,+ height: height,+ bitsPerComponent: 8,+ bitsPerPixel: 32,+ bytesPerRow: width * 4,+ space: CGColorSpaceCreateDeviceRGB(),+ bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue),+ provider: provider,+ decode: nil,+ shouldInterpolate: false,+ intent: .defaultIntent+ )++ return try XCTUnwrap(image)+ }++ private func imageSize(in data: Data) throws -> (width: Int, height: Int) {+ let source = try XCTUnwrap(CGImageSourceCreateWithData(data as CFData, nil))+ let properties = try XCTUnwrap(CGImageSourceCopyPropertiesAtIndex(source, 0, nil) as? [CFString: Any])+ let width = try XCTUnwrap(properties[kCGImagePropertyPixelWidth] as? Int)+ let height = try XCTUnwrap(properties[kCGImagePropertyPixelHeight] as? Int)+ return (width, height)+ }}diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.jsonindex 85c2bd0..279affb 100644--- a/plugins/open-computer-use/.codex-plugin/plugin.json+++ b/plugins/open-computer-use/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-computer-use",- "version": "0.1.47",+ "version": "0.1.48","description": "Open-source cross-platform Computer Use MCP server packaged as a Codex plugin.","author": {"name": "Leo",diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.mdindex c803b1f..da78974 100644--- a/scripts/computer-use-cli/README.md+++ b/scripts/computer-use-cli/README.md@@ -131,7 +131,7 @@ Example working invocation against `open-computer-use`:```bashgo run . list-tools \--transport direct \- --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.47/scripts/launch-open-computer-use.sh+ --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.48/scripts/launch-open-computer-use.sh```Example working comparison flow against a local repo build:diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.goindex 32cb0d8..532c56e 100644--- a/scripts/computer-use-cli/main.go+++ b/scripts/computer-use-cli/main.go@@ -21,7 +21,7 @@ import (const (cliName = "computer-use-cli"- cliVersion = "0.1.47"+ cliVersion = "0.1.48"defaultTimeout = 60 * time.SecondpluginRootEnvVar = "COMPUTER_USE_PLUGIN_ROOT"pluginVersionEnvVar = "COMPUTER_USE_PLUGIN_VERSION"