Skip to content

👾 Open Computer Use – Open-Source Alternative to Codex Computer Use

fix: clean up app build warnings

usoon.xucd0b30dc2297
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index f414486..04c612b 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -111,7 +111,7 @@
- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。
- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。
- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。
-- screenshot 当前使用系统窗口截图 API,但默认直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。
+- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。
- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。
## 主要验证路径
diff --git a/docs/histories/2026-04/20260422-1705-build-warning-cleanup.md b/docs/histories/2026-04/20260422-1705-build-warning-cleanup.md
new file mode 100644
index 0000000..d27f4ce
--- /dev/null
+++ b/docs/histories/2026-04/20260422-1705-build-warning-cleanup.md
@@ -0,0 +1,28 @@
+## [2026-04-22 17:05] | Task: 清理 app 打包构建告警
+
+### 🤖 Execution Context
+* **Agent ID**: `codex-main`
+* **Base Model**: `GPT-5`
+* **Runtime**: `Codex CLI`
+
+### 📥 User Query
+> `./scripts/build-open-computer-use-app.sh debug` 运行时有很多 warning。
+
+### 🛠 Changes Overview
+**Scope:** `OpenComputerUseKit`, `docs`
+
+**Key Actions:**
+- **[Modernize app launch path]**: 去掉 `AppDiscovery` 里已废弃的 `NSWorkspace.launchApplication(...)` 和 `fullPath(forApplication:)`,改成标准应用目录解析 + 现代 `openApplication(at:configuration:)`。
+- **[Modernize window capture path]**: 去掉 `AccessibilitySnapshot` 里已废弃的 `CGWindowListCreateImage`,改成基于 `ScreenCaptureKit` 的单窗口截图。
+- **[Trim test-only warning]**: 去掉 `CursorMotion` 实验 target 里已无效果的 `activateIgnoringOtherApps` 选项,避免 `swift test` 额外报废弃告警。
+- **[Sync docs]**: 更新架构文档里的截图实现说明。
+
+### 🧠 Design Intent (Why)
+这次任务的目标不是压掉 warning,而是把已经有替代方案的系统 API 真正迁移掉,确保 `./scripts/build-open-computer-use-app.sh debug` 在当前 Xcode / Swift 组合下保持干净,并减少后续 SDK 升级带来的噪音。
+
+### 📁 Files Modified
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift`
+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
+- `experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift`
+- `docs/ARCHITECTURE.md`
+- `docs/histories/2026-04/20260422-1705-build-warning-cleanup.md`
diff --git a/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift b/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift
index c00426a..89e0ff9 100644
--- a/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift
+++ b/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift
@@ -36,7 +36,7 @@ final class CursorMotionAppDelegate: NSObject, NSApplicationDelegate {
window.orderFrontRegardless()
window.makeKeyAndOrderFront(nil)
window.level = .normal
- NSRunningApplication.current.activate(options: [.activateAllWindows, .activateIgnoringOtherApps])
+ NSRunningApplication.current.activate(options: [.activateAllWindows])
NSApp.activate(ignoringOtherApps: true)
}
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
index 8b89fbf..e591f3c 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift
@@ -2,6 +2,7 @@ import AppKit
import ApplicationServices
import CoreGraphics
import Foundation
+import ScreenCaptureKit
final class ElementRecord {
let index: Int
@@ -232,16 +233,38 @@ private struct WindowCapture {
return nil
}
- let image = CGWindowListCreateImage(
- best.2,
- .optionIncludingWindow,
- best.0,
- [.bestResolution, .boundsIgnoreFraming]
- )
+ let image = captureImage(windowID: best.0, bounds: best.2)
return WindowCapture(windowID: best.0, layer: best.1, bounds: best.2, image: image)
}
+ private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? {
+ try? BlockingAsyncBridge.run {
+ let shareableContent = try await SCShareableContent.current
+ guard let window = shareableContent.windows.first(where: { $0.windowID == windowID }) else {
+ return nil
+ }
+
+ let configuration = SCStreamConfiguration()
+ let scaleFactor = bestEffortScaleFactor(for: bounds)
+ let captureSize = window.frame.isEmpty ? bounds.size : window.frame.size
+ configuration.width = max(1, Int(ceil(captureSize.width * scaleFactor)))
+ configuration.height = max(1, Int(ceil(captureSize.height * scaleFactor)))
+ configuration.showsCursor = false
+ configuration.scalesToFit = false
+ configuration.ignoreShadowsSingleWindow = true
+
+ let filter = SCContentFilter(desktopIndependentWindow: window)
+ return try await SCScreenshotManager.captureImage(contentFilter: filter, configuration: configuration)
+ }
+ }
+
+ private static func bestEffortScaleFactor(for bounds: CGRect) -> CGFloat {
+ NSScreen.screens.first(where: { $0.frame.intersects(bounds) })?.backingScaleFactor
+ ?? NSScreen.main?.backingScaleFactor
+ ?? 1
+ }
+
func pngDataIfAvailable() -> Data? {
guard let image else {
return nil
@@ -252,6 +275,43 @@ private struct WindowCapture {
}
}
+private final class AsyncResultBox<T>: @unchecked Sendable {
+ var result: Result<T, Error>?
+}
+
+private enum BlockingAsyncBridge {
+ static func run<T>(_ operation: @escaping @Sendable () async throws -> T) throws -> T {
+ let semaphore = DispatchSemaphore(value: 0)
+ let resultBox = AsyncResultBox<T>()
+
+ Task.detached {
+ do {
+ resultBox.result = .success(try await operation())
+ } catch {
+ resultBox.result = .failure(error)
+ }
+
+ semaphore.signal()
+ }
+
+ waitForSignal(semaphore)
+ return try resultBox.result?.get() ?? {
+ throw ComputerUseError.message("ScreenCaptureKit screenshot task finished without producing a result.")
+ }()
+ }
+
+ private static func waitForSignal(_ semaphore: DispatchSemaphore) {
+ if Thread.isMainThread {
+ while semaphore.wait(timeout: .now()) == .timedOut {
+ RunLoop.current.run(mode: .default, before: Date(timeIntervalSinceNow: 0.01))
+ }
+ return
+ }
+
+ semaphore.wait()
+ }
+}
+
private struct RenderContext {
let windowBounds: CGRect?
let focusedElement: AXUIElement?
diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
index a5135fe..2bbbe08 100644
--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift
@@ -50,6 +50,12 @@ enum AppDiscovery {
private static let useCountAttribute = "kMDItemUseCount"
private static let maxRecentNonRunningApps = 10
private static let fixtureListBundleIdentifier = "dev.opencodex.opencomputeruse.fixture"
+ private static let standardApplicationSearchRoots: [URL] = [
+ URL(fileURLWithPath: "/Applications", isDirectory: true),
+ URL(fileURLWithPath: "/System/Applications", isDirectory: true),
+ URL(fileURLWithPath: "/System/Library/CoreServices", isDirectory: true),
+ FileManager.default.homeDirectoryForCurrentUser.appendingPathComponent("Applications", isDirectory: true),
+ ]
static let usageDateFormatter: DateFormatter = {
let formatter = DateFormatter()
@@ -240,21 +246,91 @@ enum AppDiscovery {
}
if let appURL = NSWorkspace.shared.urlForApplication(withBundleIdentifier: query) {
- try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])
+ try openApplication(at: appURL)
}
return
}
- guard let fullPath = NSWorkspace.shared.fullPath(forApplication: query) else {
+ guard let appURL = applicationURL(named: query) else {
return
}
- let appURL = URL(fileURLWithPath: fullPath)
if AppSafetyPolicy.isBlocked(bundleIdentifier: Bundle(url: appURL)?.bundleIdentifier) {
return
}
- try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])
+ try openApplication(at: appURL)
+ }
+
+ private static func applicationURL(named query: String) -> URL? {
+ let targetName = stripAppSuffix(from: query).trimmingCharacters(in: .whitespacesAndNewlines)
+ guard !targetName.isEmpty else {
+ return nil
+ }
+
+ let fileManager = FileManager.default
+ let resourceKeys: [URLResourceKey] = [.isApplicationKey, .isDirectoryKey, .nameKey]
+ var visitedPaths: Set<String> = []
+
+ for root in standardApplicationSearchRoots where fileManager.fileExists(atPath: root.path) {
+ guard let enumerator = fileManager.enumerator(
+ at: root,
+ includingPropertiesForKeys: resourceKeys,
+ options: [.skipsHiddenFiles, .skipsPackageDescendants]
+ ) else {
+ continue
+ }
+
+ for case let candidateURL as URL in enumerator {
+ guard candidateURL.pathExtension.caseInsensitiveCompare("app") == .orderedSame else {
+ continue
+ }
+
+ let normalizedPath = candidateURL.standardizedFileURL.path.lowercased()
+ guard visitedPaths.insert(normalizedPath).inserted else {
+ continue
+ }
+
+ let candidateName = stripAppSuffix(from: candidateURL.lastPathComponent)
+ if candidateName.caseInsensitiveCompare(targetName) == .orderedSame {
+ return candidateURL
+ }
+ }
+ }
+
+ return nil
+ }
+
+ private static func openApplication(at appURL: URL) throws {
+ let configuration = NSWorkspace.OpenConfiguration()
+ let semaphore = DispatchSemaphore(value: 0)
+ let errorBox = LaunchErrorBox()
+
+ NSWorkspace.shared.openApplication(at: appURL, configuration: configuration) { _, error in
+ errorBox.error = error
+ semaphore.signal()
+ }
+
+ waitForSignal(semaphore)
+
+ if let launchError = errorBox.error {
+ throw launchError
+ }
+ }
+
+ private static func waitForSignal(_ semaphore: DispatchSemaphore) {
+ if Thread.isMainThread {
+ while semaphore.wait(timeout: .now()) == .timedOut {
+ RunLoop.current.run(mode: .default, before: Date(timeIntervalSinceNow: 0.01))
+ }
+ return
+ }
+
+ semaphore.wait()
+ }
+
+ private final class LaunchErrorBox: @unchecked Sendable {
+ var error: Error?
}
private static func recentUsageCutoff(referenceDate: Date = Date()) -> Date {