👾 Open Computer Use – Open-Source Alternative to Codex Computer Use
Swift accessibilityai-agentai-agentsclaude-codecodexcodex-computer-usecomputer-usecomputer-use-agentcross-platformcuadesktop-automationgemini-cligui-automationmacosmcpmodel-context-protocolopen-computer-useopen-computer-use-mcpopenai-codexopencode
fix: clean up app build warnings
usoon.xucd0b30dc2297
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex f414486..04c612b 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -111,7 +111,7 @@- 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints,普通 stdio MCP client 在本机上可能被系统直接杀掉;如果要探测官方 bundled `computer-use`,`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪,外部 raw helper 即使走已签名 Codex binary,也可能返回 `Sender process is not authenticated`;需要真实使用官方工具时应走正常 Codex agent/tool 调用链,开源版则继续提供可直连的 `open-computer-use` MCP server。- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。-- screenshot 当前使用系统窗口截图 API,但默认直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。+- screenshot 当前通过 `ScreenCaptureKit` 捕获目标窗口,并直接以 MCP `image` content block 的 base64 PNG 返回,不再把普通 app 截图落盘到仓库或临时目录。- 会话状态现在是进程内内存态,保存每个 app 最近一次 snapshot 和 element index 映射。## 主要验证路径diff --git a/docs/histories/2026-04/20260422-1705-build-warning-cleanup.md b/docs/histories/2026-04/20260422-1705-build-warning-cleanup.mdnew file mode 100644index 0000000..d27f4ce--- /dev/null+++ b/docs/histories/2026-04/20260422-1705-build-warning-cleanup.md@@ -0,0 +1,28 @@+## [2026-04-22 17:05] | Task: 清理 app 打包构建告警++### 🤖 Execution Context+* **Agent ID**: `codex-main`+* **Base Model**: `GPT-5`+* **Runtime**: `Codex CLI`++### 📥 User Query+> `./scripts/build-open-computer-use-app.sh debug` 运行时有很多 warning。++### 🛠 Changes Overview+**Scope:** `OpenComputerUseKit`, `docs`++**Key Actions:**+- **[Modernize app launch path]**: 去掉 `AppDiscovery` 里已废弃的 `NSWorkspace.launchApplication(...)` 和 `fullPath(forApplication:)`,改成标准应用目录解析 + 现代 `openApplication(at:configuration:)`。+- **[Modernize window capture path]**: 去掉 `AccessibilitySnapshot` 里已废弃的 `CGWindowListCreateImage`,改成基于 `ScreenCaptureKit` 的单窗口截图。+- **[Trim test-only warning]**: 去掉 `CursorMotion` 实验 target 里已无效果的 `activateIgnoringOtherApps` 选项,避免 `swift test` 额外报废弃告警。+- **[Sync docs]**: 更新架构文档里的截图实现说明。++### 🧠 Design Intent (Why)+这次任务的目标不是压掉 warning,而是把已经有替代方案的系统 API 真正迁移掉,确保 `./scripts/build-open-computer-use-app.sh debug` 在当前 Xcode / Swift 组合下保持干净,并减少后续 SDK 升级带来的噪音。++### 📁 Files Modified+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift`+- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`+- `experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift`+- `docs/ARCHITECTURE.md`+- `docs/histories/2026-04/20260422-1705-build-warning-cleanup.md`diff --git a/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift b/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swiftindex c00426a..89e0ff9 100644--- a/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift+++ b/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift@@ -36,7 +36,7 @@ final class CursorMotionAppDelegate: NSObject, NSApplicationDelegate {window.orderFrontRegardless()window.makeKeyAndOrderFront(nil)window.level = .normal- NSRunningApplication.current.activate(options: [.activateAllWindows, .activateIgnoringOtherApps])+ NSRunningApplication.current.activate(options: [.activateAllWindows])NSApp.activate(ignoringOtherApps: true)}diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swiftindex 8b89fbf..e591f3c 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift@@ -2,6 +2,7 @@ import AppKitimport ApplicationServicesimport CoreGraphicsimport Foundation+import ScreenCaptureKitfinal class ElementRecord {let index: Int@@ -232,16 +233,38 @@ private struct WindowCapture {return nil}- let image = CGWindowListCreateImage(- best.2,- .optionIncludingWindow,- best.0,- [.bestResolution, .boundsIgnoreFraming]- )+ let image = captureImage(windowID: best.0, bounds: best.2)return WindowCapture(windowID: best.0, layer: best.1, bounds: best.2, image: image)}+ private static func captureImage(windowID: CGWindowID, bounds: CGRect) -> CGImage? {+ try? BlockingAsyncBridge.run {+ let shareableContent = try await SCShareableContent.current+ guard let window = shareableContent.windows.first(where: { $0.windowID == windowID }) else {+ return nil+ }++ let configuration = SCStreamConfiguration()+ let scaleFactor = bestEffortScaleFactor(for: bounds)+ let captureSize = window.frame.isEmpty ? bounds.size : window.frame.size+ configuration.width = max(1, Int(ceil(captureSize.width * scaleFactor)))+ configuration.height = max(1, Int(ceil(captureSize.height * scaleFactor)))+ configuration.showsCursor = false+ configuration.scalesToFit = false+ configuration.ignoreShadowsSingleWindow = true++ let filter = SCContentFilter(desktopIndependentWindow: window)+ return try await SCScreenshotManager.captureImage(contentFilter: filter, configuration: configuration)+ }+ }++ private static func bestEffortScaleFactor(for bounds: CGRect) -> CGFloat {+ NSScreen.screens.first(where: { $0.frame.intersects(bounds) })?.backingScaleFactor+ ?? NSScreen.main?.backingScaleFactor+ ?? 1+ }+func pngDataIfAvailable() -> Data? {guard let image else {return nil@@ -252,6 +275,43 @@ private struct WindowCapture {}}+private final class AsyncResultBox<T>: @unchecked Sendable {+ var result: Result<T, Error>?+}++private enum BlockingAsyncBridge {+ static func run<T>(_ operation: @escaping @Sendable () async throws -> T) throws -> T {+ let semaphore = DispatchSemaphore(value: 0)+ let resultBox = AsyncResultBox<T>()++ Task.detached {+ do {+ resultBox.result = .success(try await operation())+ } catch {+ resultBox.result = .failure(error)+ }++ semaphore.signal()+ }++ waitForSignal(semaphore)+ return try resultBox.result?.get() ?? {+ throw ComputerUseError.message("ScreenCaptureKit screenshot task finished without producing a result.")+ }()+ }++ private static func waitForSignal(_ semaphore: DispatchSemaphore) {+ if Thread.isMainThread {+ while semaphore.wait(timeout: .now()) == .timedOut {+ RunLoop.current.run(mode: .default, before: Date(timeIntervalSinceNow: 0.01))+ }+ return+ }++ semaphore.wait()+ }+}+private struct RenderContext {let windowBounds: CGRect?let focusedElement: AXUIElement?diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swiftindex a5135fe..2bbbe08 100644--- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift+++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift@@ -50,6 +50,12 @@ enum AppDiscovery {private static let useCountAttribute = "kMDItemUseCount"private static let maxRecentNonRunningApps = 10private static let fixtureListBundleIdentifier = "dev.opencodex.opencomputeruse.fixture"+ private static let standardApplicationSearchRoots: [URL] = [+ URL(fileURLWithPath: "/Applications", isDirectory: true),+ URL(fileURLWithPath: "/System/Applications", isDirectory: true),+ URL(fileURLWithPath: "/System/Library/CoreServices", isDirectory: true),+ FileManager.default.homeDirectoryForCurrentUser.appendingPathComponent("Applications", isDirectory: true),+ ]static let usageDateFormatter: DateFormatter = {let formatter = DateFormatter()@@ -240,21 +246,91 @@ enum AppDiscovery {}if let appURL = NSWorkspace.shared.urlForApplication(withBundleIdentifier: query) {- try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])+ try openApplication(at: appURL)}return}- guard let fullPath = NSWorkspace.shared.fullPath(forApplication: query) else {+ guard let appURL = applicationURL(named: query) else {return}- let appURL = URL(fileURLWithPath: fullPath)if AppSafetyPolicy.isBlocked(bundleIdentifier: Bundle(url: appURL)?.bundleIdentifier) {return}- try NSWorkspace.shared.launchApplication(at: appURL, options: [], configuration: [:])+ try openApplication(at: appURL)+ }++ private static func applicationURL(named query: String) -> URL? {+ let targetName = stripAppSuffix(from: query).trimmingCharacters(in: .whitespacesAndNewlines)+ guard !targetName.isEmpty else {+ return nil+ }++ let fileManager = FileManager.default+ let resourceKeys: [URLResourceKey] = [.isApplicationKey, .isDirectoryKey, .nameKey]+ var visitedPaths: Set<String> = []++ for root in standardApplicationSearchRoots where fileManager.fileExists(atPath: root.path) {+ guard let enumerator = fileManager.enumerator(+ at: root,+ includingPropertiesForKeys: resourceKeys,+ options: [.skipsHiddenFiles, .skipsPackageDescendants]+ ) else {+ continue+ }++ for case let candidateURL as URL in enumerator {+ guard candidateURL.pathExtension.caseInsensitiveCompare("app") == .orderedSame else {+ continue+ }++ let normalizedPath = candidateURL.standardizedFileURL.path.lowercased()+ guard visitedPaths.insert(normalizedPath).inserted else {+ continue+ }++ let candidateName = stripAppSuffix(from: candidateURL.lastPathComponent)+ if candidateName.caseInsensitiveCompare(targetName) == .orderedSame {+ return candidateURL+ }+ }+ }++ return nil+ }++ private static func openApplication(at appURL: URL) throws {+ let configuration = NSWorkspace.OpenConfiguration()+ let semaphore = DispatchSemaphore(value: 0)+ let errorBox = LaunchErrorBox()++ NSWorkspace.shared.openApplication(at: appURL, configuration: configuration) { _, error in+ errorBox.error = error+ semaphore.signal()+ }++ waitForSignal(semaphore)++ if let launchError = errorBox.error {+ throw launchError+ }+ }++ private static func waitForSignal(_ semaphore: DispatchSemaphore) {+ if Thread.isMainThread {+ while semaphore.wait(timeout: .now()) == .timedOut {+ RunLoop.current.run(mode: .default, before: Date(timeIntervalSinceNow: 0.01))+ }+ return+ }++ semaphore.wait()+ }++ private final class LaunchErrorBox: @unchecked Sendable {+ var error: Error?}private static func recentUsageCutoff(referenceDate: Date = Date()) -> Date {