Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

release: bump version to 0.1.48

Leof02d35f48409
diff --git a/apps/OpenComputerUseLinux/main.go b/apps/OpenComputerUseLinux/main.go
index eeafc33..464561d 100644
--- a/apps/OpenComputerUseLinux/main.go
+++ b/apps/OpenComputerUseLinux/main.go
@@ -19,7 +19,7 @@ import (
"time"
)
-var version = "0.1.47"
+var version = "0.1.48"
//go:embed runtime.py
var linuxRuntimeScript string
diff --git a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
index 67e8e21..0d4d947 100644
--- a/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
+++ b/apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift
@@ -35,7 +35,7 @@ final class MCPClient {
_ = try request(method: "initialize", params: [
"clientInfo": [
"name": "OpenComputerUseSmokeSuite",
- "version": "0.1.47",
+ "version": "0.1.48",
],
"capabilities": [:],
"protocolVersion": "2025-03-26",
diff --git a/apps/OpenComputerUseWindows/main.go b/apps/OpenComputerUseWindows/main.go
index 9d8979b..173f912 100644
--- a/apps/OpenComputerUseWindows/main.go
+++ b/apps/OpenComputerUseWindows/main.go
@@ -16,7 +16,7 @@ import (
"time"
)
-var version = "0.1.47"
+var version = "0.1.48"
//go:embed runtime.ps1
var windowsRuntimeScript string
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index b01a964..683b968 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -125,7 +125,7 @@
- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。
- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。
- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。
-- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。
+- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录;编码前会按最大尺寸和目标字节数自适应缩小,避免复杂页面的大 PNG 触发 host 侧 MCP result 降级,同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。
- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。
## 主要验证路径
diff --git a/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.md b/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.md
new file mode 100644
index 0000000..e747fc3
--- /dev/null
+++ b/docs/histories/2026-05/20260508-1508-bound-screenshot-result-size.md
@@ -0,0 +1,22 @@
+# Bound screenshot result size
+
+## 用户诉求
+
+用户发现 `open-computer-use` 的部分 tool call 在 Codex 历史记录里变成 `text` 下的 JSON 字符串,怀疑最近几次 commit 引入了额外包装,并提供了历史 session id 辅助定位。
+
+## 本次改动
+
+- **[Root cause]**: 确认 MCP server 原始 `tools/call` 响应是标准 `content: [text, image]`,嵌套只出现在 Codex 为了持久化超大 MCP result 而做的事件日志降级路径里。
+- **[Screenshot bound]**: macOS `ScreenCaptureKit` 窗口截图在编码为 PNG 前按最大尺寸和目标字节数自适应缩小,减少复杂页面触发 host 侧大结果降级的概率。
+- **[Regression tests]**: 增加截图压缩边界测试,覆盖大图会缩小、小图保持原尺寸。
+- **[Docs]**: 更新架构文档,说明截图仍以 MCP image block 返回,但会做尺寸/字节上限控制,coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射。
+
+## 设计动机
+
+问题不是协议层重复包了一层,而是截图 PNG 过大时,Codex 会把整个 `CallToolResult` 序列化成一个 text preview 来保护 rollout storage。直接改 MCP shape 会破坏兼容性;更稳妥的修复是控制截图结果大小,让正常 `get_app_state` / action tool 返回保持扁平的 `text + image` 结构。
+
+## 影响文件
+
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift`
+- `docs/ARCHITECTURE.md`
diff --git a/docs/histories/2026-05/20260508-1518-release-0.1.48.md b/docs/histories/2026-05/20260508-1518-release-0.1.48.md
new file mode 100644
index 0000000..7478a80
--- /dev/null
+++ b/docs/histories/2026-05/20260508-1518-release-0.1.48.md
@@ -0,0 +1,39 @@
+# Release 0.1.48
+
+## 用户诉求
+
+用户要求提交本轮相关改动并推送发版,同时确认 Codex TOML 配置改回连接线上 `open-computer-use`,不要指向 dev 构建。
+
+## 本次改动
+
+- **[Version bump]**: 将 macOS / Linux / Windows runtime、插件 manifest、smoke suite、测试和 reverse-engineering helper 版本统一 bump 到 `0.1.48`。
+- **[Release notes]**: 更新用户可见 release notes,说明本版解决复杂窗口截图过大导致 Codex 历史事件降级成 JSON text preview 的问题。
+- **[Config check]**: 确认本机 Codex 配置中的 `open-computer-use` MCP server 使用线上命令 `open-computer-use mcp`,没有指向本地 dev binary。
+
+## 设计动机
+
+本轮修复已经影响公开 npm 包里的 macOS runtime 行为,且用户明确要求发版,因此按 release guide 走 patch release。版本源继续以插件 manifest 为准,并同步所有辅助工具与测试里的版本字符串,避免 tag 与 npm staging 包版本不一致。
+
+## 验证
+
+- `swift test`
+- `./scripts/run-tool-smoke-tests.sh`
+- `node ./scripts/npm/build-packages.mjs --out-dir dist/release/npm-staging-check`
+- `./scripts/build-cursor-motion-dmg.sh --configuration release --arch universal --version 0.1.48`
+- `node -p "require('./dist/release/npm-staging-check/open-computer-use/package.json').version"` -> `0.1.48`
+- `test -x dist/release/npm-staging-check/open-computer-use/dist/linux/arm64/open-computer-use`
+- `test -f dist/release/npm-staging-check/open-computer-use/dist/windows/arm64/open-computer-use.exe`
+- `node -e "if (require('./dist/release/npm-staging-check/open-computer-use/package.json').optionalDependencies) process.exit(1)"`
+- `ls dist/release/cursor-motion/CursorMotion-0.1.48.dmg`
+- `git diff --check`
+
+## 影响文件
+
+- `plugins/open-computer-use/.codex-plugin/plugin.json`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift`
+- `apps/OpenComputerUseLinux/main.go`
+- `apps/OpenComputerUseWindows/main.go`
+- `apps/OpenComputerUseSmokeSuite/Sources/OpenComputerUseSmokeSuite/main.swift`
+- `scripts/computer-use-cli/main.go`
+- `scripts/computer-use-cli/README.md`
+- `docs/releases/feature-release-notes.md`
diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md
index db670b3..d03ac47 100644
--- a/docs/releases/feature-release-notes.md
+++ b/docs/releases/feature-release-notes.md
@@ -4,6 +4,7 @@
| 日期 | 功能域 | 用户价值 | 变更摘要 |
| --- | --- | --- | --- |
+| 2026-05-08 | MCP 结果大小控制 | Chrome / Electron 等复杂窗口返回截图时更不容易触发 Codex 的大结果降级,历史记录和上层工具消费里会继续保持正常的 `text + image` 结构,而不是把整个 MCP result 塞进 text 字符串。 | 发布 `0.1.48`,macOS `ScreenCaptureKit` 截图在编码为 MCP image block 前会按最大尺寸和目标字节数自适应缩小,并补测试锁住大图压缩、小图保真路径。 |
| 2026-05-08 | Smoke 测试静默运行 | 开发者运行 `./scripts/run-tool-smoke-tests.sh` 时不再被内部 fixture 的橙色测试窗口打断,回归验证更安静,也不会让用户误以为桌面上弹出了额外应用。 | 发布 `0.1.47`,smoke suite 默认以 headless 模式启动内部 fixture,并保留 fixture 在 `list_apps` smoke 路径里的可发现性。 |
| 2026-05-07 | 窗口不可用错误文案 | Lark / Chrome 等 no-window 场景下,开源版 `get_app_state` 的返回文本与官方 `computer-use` 当前输出一致,减少上层 prompt、测试和用户排查时的差异噪音。 | 发布 `0.1.46`,将 missing AX window 和 missing visible CGWindow 两条路径统一为 `Apple event error -10005: cgWindowNotFound`,并补单元测试锁住该官方风格文本。 |
| 2026-05-07 | 可见窗口匹配 | `get_app_state` 在 app 没有可见窗口、窗口已最小化,或 AX 窗口无法匹配到 on-screen CGWindow 时会更接近官方 `computer-use` 的 `cgWindowNotFound` 语义,避免返回没有截图坐标基础的半残状态。 | 发布 `0.1.45`,基于官方 `1.0.770` 逆向字符串中的 `cgWindowNotFound`、`noWindowsAvailable`、`matchingWindowNotFound` 和 `AXWindowMiniaturized` 线索,要求真实 app snapshot 同时满足未最小化 `AXWindow` 与可见 `CGWindow`。 |
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 1fdfc37..ab22860 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -30,6 +30,9 @@ enum SnapshotMode {
let accessibilityTreeMaxNodeCount = 1200
let accessibilityTreeMaxDepth = 64
let screenshotCaptureTimeout: TimeInterval = 5
+let screenshotResultMaxPNGBytes = 900_000
+let screenshotResultMaxDimension: CGFloat = 1280
+let screenshotResultMinScale: CGFloat = 0.25
private let windowVisibilityRecoveryDelay: TimeInterval = 0.7
private let axWebAreaRole = "AXWebArea"
@@ -412,11 +415,75 @@ private struct WindowCapture {
return nil
}
- let bitmap = NSBitmapImageRep(cgImage: image)
- return bitmap.representation(using: .png, properties: [:])
+ return boundedScreenshotPNGData(for: image)
}
}
+func boundedScreenshotPNGData(
+ for image: CGImage,
+ maxBytes: Int = screenshotResultMaxPNGBytes,
+ maxDimension: CGFloat = screenshotResultMaxDimension,
+ minScale: CGFloat = screenshotResultMinScale
+) -> Data? {
+ guard image.width > 0, image.height > 0, maxBytes > 0 else {
+ return nil
+ }
+
+ let original = pngData(for: image)
+ let largestDimension = CGFloat(max(image.width, image.height))
+ var scale = min(1, maxDimension / largestDimension)
+
+ if scale >= 1, let original, original.count <= maxBytes {
+ return original
+ }
+
+ var best = original
+ while scale >= minScale {
+ guard let resized = resizedCGImage(image, scale: scale),
+ let data = pngData(for: resized)
+ else {
+ break
+ }
+
+ best = data
+ if data.count <= maxBytes {
+ return data
+ }
+
+ scale *= 0.85
+ }
+
+ return best
+}
+
+private func pngData(for image: CGImage) -> Data? {
+ let bitmap = NSBitmapImageRep(cgImage: image)
+ return bitmap.representation(using: .png, properties: [:])
+}
+
+private func resizedCGImage(_ image: CGImage, scale: CGFloat) -> CGImage? {
+ let width = max(1, Int((CGFloat(image.width) * scale).rounded()))
+ let height = max(1, Int((CGFloat(image.height) * scale).rounded()))
+ let colorSpace = CGColorSpaceCreateDeviceRGB()
+ let bitmapInfo = CGImageAlphaInfo.premultipliedLast.rawValue
+
+ guard let context = CGContext(
+ data: nil,
+ width: width,
+ height: height,
+ bitsPerComponent: 8,
+ bytesPerRow: 0,
+ space: colorSpace,
+ bitmapInfo: bitmapInfo
+ ) else {
+ return nil
+ }
+
+ context.interpolationQuality = .medium
+ context.draw(image, in: CGRect(x: 0, y: 0, width: width, height: height))
+ return context.makeImage()
+}
+
private final class AsyncResultBox<T>: @unchecked Sendable {
var result: Result<T, Error>?
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift
index 4337a2e..19ad9ca 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift
@@ -1,6 +1,6 @@
import Foundation
-public let openComputerUseVersion = "0.1.47"
+public let openComputerUseVersion = "0.1.48"
public func resolvedOpenComputerUseVersion(bundle: Bundle = .main) -> String {
if let version = bundle.object(forInfoDictionaryKey: "CFBundleShortVersionString") as? String,
diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
index 217399a..c39307f 100644
--- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
+++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift
@@ -1,4 +1,5 @@
import AppKit
+import ImageIO
import XCTest
@testable import OpenComputerUseKit
@@ -136,6 +137,29 @@ final class OpenComputerUseKitTests: XCTestCase {
XCTAssertEqual(resolvedOpenComputerUseVersion(bundle: Bundle(for: Self.self)), openComputerUseVersion)
}
+ func testBoundedScreenshotPNGDataShrinksLargeScreenshots() throws {
+ let image = try makeNoisyTestImage(width: 800, height: 600)
+ let data = try XCTUnwrap(boundedScreenshotPNGData(
+ for: image,
+ maxBytes: 50_000,
+ maxDimension: 320,
+ minScale: 0.05
+ ))
+ let size = try imageSize(in: data)
+
+ XCTAssertLessThanOrEqual(data.count, 50_000)
+ XCTAssertLessThanOrEqual(max(size.width, size.height), 320)
+ }
+
+ func testBoundedScreenshotPNGDataKeepsSmallScreenshotsAtOriginalSize() throws {
+ let image = try makeSolidTestImage(width: 32, height: 24)
+ let data = try XCTUnwrap(boundedScreenshotPNGData(for: image, maxBytes: 1_000_000, maxDimension: 320))
+ let size = try imageSize(in: data)
+
+ XCTAssertEqual(size.width, 32)
+ XCTAssertEqual(size.height, 24)
+ }
+
func testToolDefinitionCount() {
XCTAssertEqual(ToolDefinitions.all.count, 9)
}
@@ -436,7 +460,7 @@ final class OpenComputerUseKitTests: XCTestCase {
func testInitializeResponseContainsToolsCapability() throws {
let server = StdioMCPServer(service: ComputerUseService())
- let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.47"},"capabilities":{}}}"#)
+ let response = server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.48"},"capabilities":{}}}"#)
XCTAssertNotNil(response)
XCTAssertTrue(response!.contains(#""name":"open-computer-use""#))
XCTAssertTrue(response!.contains(#""tools":{"listChanged":false}"#))
@@ -445,7 +469,7 @@ final class OpenComputerUseKitTests: XCTestCase {
func testInitializeResponseContainsComputerUseInstructions() throws {
let server = StdioMCPServer(service: ComputerUseService())
let response = try XCTUnwrap(
- server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.47"},"capabilities":{}}}"#)
+ server.handle(line: #"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"test","version":"0.1.48"},"capabilities":{}}}"#)
)
let data = try XCTUnwrap(response.data(using: .utf8))
let json = try XCTUnwrap(JSONSerialization.jsonObject(with: data) as? [String: Any])
@@ -1420,4 +1444,59 @@ final class OpenComputerUseKitTests: XCTestCase {
let length = max(hypot(dx, dy), 0.001)
return CGVector(dx: dx / length, dy: dy / length)
}
+
+ private func makeNoisyTestImage(width: Int, height: Int) throws -> CGImage {
+ var pixels = [UInt8](repeating: 0, count: width * height * 4)
+ for y in 0..<height {
+ for x in 0..<width {
+ let offset = (y * width + x) * 4
+ pixels[offset] = UInt8((x * 37 + y * 17) & 0xFF)
+ pixels[offset + 1] = UInt8((x * 11 + y * 43) & 0xFF)
+ pixels[offset + 2] = UInt8((x * 71 + y * 5) & 0xFF)
+ pixels[offset + 3] = 255
+ }
+ }
+
+ return try makeTestImage(width: width, height: height, pixels: pixels)
+ }
+
+ private func makeSolidTestImage(width: Int, height: Int) throws -> CGImage {
+ var pixels = [UInt8](repeating: 0, count: width * height * 4)
+ for index in stride(from: 0, to: pixels.count, by: 4) {
+ pixels[index] = 80
+ pixels[index + 1] = 140
+ pixels[index + 2] = 220
+ pixels[index + 3] = 255
+ }
+
+ return try makeTestImage(width: width, height: height, pixels: pixels)
+ }
+
+ private func makeTestImage(width: Int, height: Int, pixels: [UInt8]) throws -> CGImage {
+ let data = Data(pixels)
+ let provider = try XCTUnwrap(CGDataProvider(data: data as CFData))
+ let image = CGImage(
+ width: width,
+ height: height,
+ bitsPerComponent: 8,
+ bitsPerPixel: 32,
+ bytesPerRow: width * 4,
+ space: CGColorSpaceCreateDeviceRGB(),
+ bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue),
+ provider: provider,
+ decode: nil,
+ shouldInterpolate: false,
+ intent: .defaultIntent
+ )
+
+ return try XCTUnwrap(image)
+ }
+
+ private func imageSize(in data: Data) throws -> (width: Int, height: Int) {
+ let source = try XCTUnwrap(CGImageSourceCreateWithData(data as CFData, nil))
+ let properties = try XCTUnwrap(CGImageSourceCopyPropertiesAtIndex(source, 0, nil) as? [CFString: Any])
+ let width = try XCTUnwrap(properties[kCGImagePropertyPixelWidth] as? Int)
+ let height = try XCTUnwrap(properties[kCGImagePropertyPixelHeight] as? Int)
+ return (width, height)
+ }
}
diff --git a/plugins/open-computer-use/.codex-plugin/plugin.json b/plugins/open-computer-use/.codex-plugin/plugin.json
index 85c2bd0..279affb 100644
--- a/plugins/open-computer-use/.codex-plugin/plugin.json
+++ b/plugins/open-computer-use/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-computer-use",
- "version": "0.1.47",
+ "version": "0.1.48",
"description": "Open-source cross-platform Computer Use MCP server packaged as a Codex plugin.",
"author": {
"name": "Leo",
diff --git a/scripts/computer-use-cli/README.md b/scripts/computer-use-cli/README.md
index c803b1f..da78974 100644
--- a/scripts/computer-use-cli/README.md
+++ b/scripts/computer-use-cli/README.md
@@ -131,7 +131,7 @@ Example working invocation against `open-computer-use`:
```bash
go run . list-tools \
--transport direct \
- --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.47/scripts/launch-open-computer-use.sh
+ --server-bin ~/.codex/plugins/cache/open-computer-use-local/open-computer-use/0.1.48/scripts/launch-open-computer-use.sh
```
Example working comparison flow against a local repo build:
diff --git a/scripts/computer-use-cli/main.go b/scripts/computer-use-cli/main.go
index 32cb0d8..532c56e 100644
--- a/scripts/computer-use-cli/main.go
+++ b/scripts/computer-use-cli/main.go
@@ -21,7 +21,7 @@ import (
const (
cliName = "computer-use-cli"
- cliVersion = "0.1.47"
+ cliVersion = "0.1.48"
defaultTimeout = 60 * time.Second
pluginRootEnvVar = "COMPUTER_USE_PLUGIN_ROOT"
pluginVersionEnvVar = "COMPUTER_USE_PLUGIN_VERSION"