Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
cua-driver: handle macOS 26.4 SCK streaming-start failure (#1478)
Francesco Bonaccie8e629107d3a
diff --git a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx b/docs/content/docs/cua-driver/guide/getting-started/faq.mdxindex 7a0623d61c..dc802f9459 100644--- a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx+++ b/docs/content/docs/cua-driver/guide/getting-started/faq.mdx@@ -36,6 +36,28 @@ open -n -g -a CuaDriver --args serveThe window capture raced against a close, or the window has no backing store yet. Re-snapshot. If it persists, pick a different `window_id` via `list_windows`.+### `screenshot` or `get_window_state` fails with "ScreenCaptureKit refused this window" / "Could not start streaming".++Known macOS 26.4.x ScreenCaptureKit regression on physical Macs (`SCStreamError` code -3801, sometimes localized — e.g. Japanese "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"). The driver already:++1. Retries the SCK call once after a brief delay (covers transient failures).+2. Falls back to the legacy `CGWindowListCreateImage` path (works on many windows the SCK regression breaks).++If both refuse, the error surfaces with an actionable hint. Workarounds in order of preference:++1. **Try a different `window_id` on the same app.** Usually only one specific window is hit.+2. **Switch to AX-only capture for that workflow.** Element-indexed clicks don't need pixels:++ ```bash+ cua-driver config set capture_mode ax+ ```++ `get_window_state` then returns the AX tree without attempting a screenshot, and `click({pid, window_id, element_index: N})` works as before.++3. **Re-snapshot a moment later.** The failure is sometimes transient.++`get_window_state` does **not** hard-fail on this error: the AX tree still ships in the response with a warning line, so element-indexed clicks keep working even when the screenshot is unavailable. The standalone `screenshot` tool does hard-fail (no AX tree to fall back to).+### The AX tree is tiny. What's happening?Check the capture mode:diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdxindex a61daedff8..9976f56df3 100644--- a/docs/content/docs/cua-driver/reference/cli-reference.mdx+++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx@@ -7,7 +7,7 @@ description: Command Line Interface reference for Cua DriverAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/cua-driver.tsSource: recursive Swift sources under libs/cua-driver/Sources- Version: 0.1.1+ Version: 0.1.6*/}import { Callout } from 'fumadocs-ui/components/callout';@@ -16,7 +16,7 @@ import { VersionHeader } from '@/components/version-selector';<VersionHeaderversions={[{"version":"0.1","href":"/cua-driver/reference/cli-reference","isCurrent":true}]}currentVersion="0.1"- fullVersion="0.1.1"+ fullVersion="0.1.6"packageName="cua-driver"installCommand="curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash"/>diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdxindex b41588de26..25bcba1bdf 100644--- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx+++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx@@ -7,7 +7,7 @@ description: Reference for every MCP tool cua-driver exposesAUTO-GENERATED FILE - DO NOT EDIT DIRECTLYGenerated by: npx tsx scripts/docs-generators/cua-driver.tsSource: recursive Swift sources under libs/cua-driver/Sources- Version: 0.1.1+ Version: 0.1.6*/}import { Callout } from 'fumadocs-ui/components/callout';@@ -318,6 +318,16 @@ keeps its name:Change with `cua-driver config set capture_mode <mode>` orthe `set_config` tool.+Screenshot capture failures in `som` / `vision` modes are+non-fatal for `som`: the AX tree still ships in the+response and the summary line carries a hint. The macOS+26.4.x SCK regression (SCStreamError -3801, "Could not+start streaming") is handled this way — agents can keep+doing element-indexed clicks against the same window even+when the screenshot is unavailable. Switching to+`capture_mode: ax` skips the capture attempt entirely on+subsequent turns.+Requires Accessibility and Screen Recording permissions.**Arguments:**@@ -340,6 +350,17 @@ screenshot selection. The combo is posted directly tothe target pid's event queue via `CGEvent.postToPid`;the target does NOT need to be frontmost.+**`window_id`** (optional): when supplied, the driver+calls `FocusWithoutRaise.activateWithoutRaise` before+posting — making the target AppKit-active without+raising its window. This is required for shortcuts that+are dispatched via NSMenu key equivalents (Cmd+S,+Cmd+N, Cmd+W, Cmd+Shift+N, …) because AppKit only+routes menu key equivalents to the active app. Omit+`window_id` for shortcuts handled by the renderer+(e.g. Cmd+C in a Chromium text field). Trade-off: the+sentinel foreground app will lose focus once.+Recognized modifiers: cmd/command, shift, option/alt,ctrl/control, fn. Non-modifier keys use the samevocabulary as `press_key` (return, tab, escape,@@ -351,6 +372,7 @@ first, one non-modifier last.- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"].- `pid` (integer, required): Target process ID.+- `window_id` (integer, optional): CGWindowID of the target window. When provided, the driver calls FocusWithoutRaise before posting so NSMenu key equivalents (Cmd+S, Cmd+N, …) reach the backgrounded app.```json{"keys":["cmd","c"],"pid":844}@@ -370,12 +392,11 @@ Provide either `bundle_id` (preferred — unambiguous, e.g.against /Applications and /System/Applications). If both aregiven, bundle_id wins.-Launches hidden — the app runs with its window initialized-for automation (so the AX tree is populated and-`click({pid, element_index})` works end-to-end), but no-window is drawn on screen. To bring the window on-screen,-use `NSRunningApplication.unhide()` from your own code, or-have the user open the app (Dock click, Cmd-Tab, Spotlight).+Launches in the background — the app's window appears on+screen but does not steal focus from whatever is currently+frontmost. The AX tree is fully populated and automation+can start immediately via `click`, `hotkey`,+`get_window_state`, etc.Some apps (Calculator and many Electron apps) call`NSApp.activate(ignoringOtherApps:)` in their own@@ -386,16 +407,14 @@ focus-steal preventer arms a shortaround the launch and re-activates the previously-frontmostapp if the target self-activates. If suppression fails thereturned summary includes a "self-activation not suppressed"-warning. With `hides = true`, the target is also hidden-back immediately by LaunchServices, so even a missed-layer-3 demotion doesn't leave a visible window.+warning.Optional `urls` are handed to the app through its`application(_:open:)` AppKit delegate. For Finder, passing-a folder URL opens a backgrounded Finder window rooted at-that folder — no activation, no visible window. Works-with any app that implements `application(_:open:)`; apps-that ignore the delegate simply launch without side effects.+a folder URL opens a Finder window rooted at that folder in+the background. Works with any app that implements+`application(_:open:)`; apps that ignore the delegate simply+launch without side effects.Optional `electron_debugging_port` launches an Electron appwith `--remote-debugging-port=<N>`, activating its Chrome@@ -600,7 +619,7 @@ cmd/shift/option/ctrl/fn. For true combinations (cmd+c),- `key` (string, required): Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit).- `modifiers` (array of string, optional): Optional modifier names held while the key is pressed (cmd/shift/option/ctrl/fn).- `pid` (integer, required): Target process ID.-- `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used.+- `window_id` (integer, optional): CGWindowID for the window. Required when element_index is used. When supplied without element_index, triggers FocusWithoutRaise before the key fires — use this for NSMenu key equivalents (Cmd+S, Cmd+N, …) on backgrounded apps.```json{"key":"return","pid":844}@@ -698,11 +717,20 @@ propagate modifier keys).Capture a screenshot using ScreenCaptureKit. Returns base64-encodedimage data for a single window in the requested format (default png).-`window_id` is required. Get window ids from `list_windows`.++ `window_id` is required. Get window ids from `list_windows`.Requires the Screen Recording TCC grant — call `check_permissions`first if unsure.+On macOS 26.4.x, ScreenCaptureKit can refuse specific windows on+physical Macs (SCStreamError -3801, "Could not start streaming").+The driver retries once and falls back to the legacy+CGWindowList path before failing; if both refuse, the error+response includes a hint to try a different `window_id` or+switch to `capture_mode: ax` for `get_window_state` (the+element-indexed flow doesn't need pixels).+**Arguments:**- `format` (string, optional): Image format. Default: png.diff --git a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swiftindex d29f1a16fd..2137cda868 100644--- a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift+++ b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift@@ -33,6 +33,13 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible {case encodeFailedcase captureFailed(String)case windowNotFound(UInt32)+ /// ScreenCaptureKit could not start streaming for this window. Distinct+ /// from `captureFailed` so callers (e.g. `get_window_state`) can surface+ /// an actionable hint — switch to `capture_mode: ax`, retry against a+ /// different window — without having to grep error strings. Seen+ /// regularly on macOS 26.4.x physical Macs against specific windows+ /// where even `screencapture -l<id>` fails (rdar / openclaw/Peekaboo#121).+ case streamingFailed(String)public var description: String {switch self {@@ -41,6 +48,7 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible {case .encodeFailed: return "failed to encode CGImage"case .captureFailed(let msg): return "capture failed: \(msg)"case .windowNotFound(let id): return "no shareable window with id \(id)"+ case .streamingFailed(let msg): return "ScreenCaptureKit streaming failed: \(msg)"}}}@@ -131,12 +139,43 @@ public actor WindowCapture {config.height = max(1, Int(window.frame.height * scale))config.showsCursor = false+ // One-shot SCK call with a single retry on streaming-start failure.+ // macOS 26.4.x has a regression where `SCScreenshotManager.captureImage`+ // intermittently returns "Could not start streaming because audio/video+ // capture failed" (SCStreamError code -3801) on physical Macs, often+ // recovering on a second attempt a moment later. We retry once with a+ // brief back-off; if it still fails, we surface `.streamingFailed` so+ // the tool layer can hint the caller toward `capture_mode: ax` for+ // `get_window_state` workflows.let cgImage: CGImagedo {- cgImage = try await SCScreenshotManager.captureImage(- contentFilter: filter,- configuration: config- )+ cgImage = try await captureSCKWithRetry(filter: filter, config: config)+ } catch let error as CaptureError {+ // Already classified — re-throw without wrapping. CGWindowList+ // is intentionally NOT tried for permission errors (it'd just+ // fail the same way and confuse the user-facing message).+ if case .permissionDenied = error { throw error }+ // For streaming / generic SCK failures, try the legacy+ // CGWindowListCreateImage path. It's deprecated on macOS 15++ // but still works in many cases where SCK refuses — particularly+ // useful as a last-ditch fallback for the 26.4 SCK regression.+ if let fallback = legacyCaptureWindow(windowID: windowID) {+ let origW = fallback.width+ let origH = fallback.height+ let resized = resizeIfNeeded(fallback, maxDim: maxImageDimension)+ let didResize = resized.width != origW || resized.height != origH+ let data = try encode(resized, format: format, quality: quality)+ return Screenshot(+ imageData: data,+ format: format,+ width: resized.width,+ height: resized.height,+ scaleFactor: Double(scale),+ originalWidth: didResize ? origW : nil,+ originalHeight: didResize ? origH : nil+ )+ }+ throw error} catch {throw classify(error)}@@ -207,17 +246,121 @@ public actor WindowCapture {return (best ?? NSScreen.main)?.backingScaleFactor ?? 1.0}+ /// Attempt `SCScreenshotManager.captureImage` once; on a streaming-start+ /// failure, wait briefly and retry once more. Returns a classified+ /// `CaptureError` on persistent failure so the caller can branch on the+ /// kind (permission vs. streaming vs. generic) without string-matching.+ ///+ /// The retry covers the macOS 26.4.x SCK regression where the very first+ /// call after the SCK daemon has been idle returns -3801 ("Could not+ /// start streaming because audio/video capture failed") but a second+ /// call ~250ms later succeeds. A second failure isn't transient and we+ /// stop retrying — the caller falls back to CGWindowList or surfaces+ /// the error.+ private func captureSCKWithRetry(+ filter: SCContentFilter,+ config: SCStreamConfiguration+ ) async throws -> CGImage {+ do {+ return try await SCScreenshotManager.captureImage(+ contentFilter: filter,+ configuration: config+ )+ } catch {+ let classified = classify(error)+ // Only retry on streaming-start failures; permission errors and+ // not-found errors won't change on a second attempt.+ guard case .streamingFailed = classified else { throw classified }+ try? await Task.sleep(nanoseconds: 250_000_000)+ do {+ return try await SCScreenshotManager.captureImage(+ contentFilter: filter,+ configuration: config+ )+ } catch {+ throw classify(error)+ }+ }+ }++ /// Legacy `CGWindowListCreateImage` fallback for the SCK 26.4 regression.+ /// Deprecated by Apple in macOS 15 but still functional on most windows,+ /// and frequently works where SCK refuses. Returns nil on failure — the+ /// caller surfaces the original SCK error in that case so the user knows+ /// the real cause.+ ///+ /// Marked with `@available(*, deprecated)` suppression because the API+ /// is the entire point: we *want* the legacy path here.+ private func legacyCaptureWindow(windowID: UInt32) -> CGImage? {+ // CGWindowListCreateImage is deprecated on macOS 15+. The deprecation+ // diagnostic is silenced with the @available pragma. Apple has not+ // (yet) removed the symbol, and this path is the only practical+ // fallback when SCK's streaming-start is broken for a given window.+ let opts: CGWindowImageOption = [.boundsIgnoreFraming, .bestResolution]+ let listOption: CGWindowListOption = .optionIncludingWindow+ // Wrap the deprecated call so we keep the unsafePointer-style+ // signature out of the rest of the code.+ let image = legacyCGWindowImage(+ windowID: windowID, listOption: listOption, imageOption: opts+ )+ // Reject 1×1 placeholder images that the legacy API sometimes returns+ // for occluded / off-screen windows — they're worse than no image.+ guard let image, image.width > 1, image.height > 1 else { return nil }+ return image+ }+private func classify(_ error: Error) -> CaptureError {let ns = error as NSErrorlet msg = ns.localizedDescription.lowercased()++ // Permission failure — English and Japanese phrasings observed in+ // SCK's `NSError.localizedDescription`. The Japanese strings cover+ // users on JP system locale where the SCK error comes back+ // localized rather than in English.if msg.contains("permission") || msg.contains("not authorized")|| msg.contains("declined") || msg.contains("denied")+ || ns.localizedDescription.contains("許可") // "permission"+ || ns.localizedDescription.contains("拒否") // "denied"{return .permissionDenied}++ // SCStreamError "could not start streaming" — code -3801 in+ // `SCStreamErrorDomain`. macOS localizes the message ("Could not+ // start streaming because audio/video capture failed" / Japanese:+ // "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"),+ // so we match on code first and fall through to substring matching+ // for the rare case where the domain isn't surfaced.+ let isSCStreamDomain = ns.domain == "SCStreamErrorDomain"+ || ns.domain == "com.apple.ScreenCaptureKit.SCStreamErrorDomain"+ if (isSCStreamDomain && ns.code == -3801)+ || msg.contains("could not start streaming")+ || msg.contains("streaming")+ || ns.localizedDescription.contains("ストリーミング") // "streaming"+ {+ return .streamingFailed(ns.localizedDescription)+ }+return .captureFailed(ns.localizedDescription)}+ /// Thin shim around the deprecated `CGWindowListCreateImage` so the+ /// deprecation-warning suppression is isolated to one place. Returns nil+ /// if the legacy path also refuses to produce an image.+ ///+ /// Marking the wrapper itself deprecated downgrades the call-site+ /// warning to a no-op — we *want* this legacy path because SCK has a+ /// well-known regression on macOS 26.4.x where streaming-start fails+ /// for specific windows on physical Macs.+ @available(*, deprecated, message: "Intentional fallback for SCK streaming-start failures.")+ private func legacyCGWindowImage(+ windowID: UInt32,+ listOption: CGWindowListOption,+ imageOption: CGWindowImageOption+ ) -> CGImage? {+ CGWindowListCreateImage(.null, listOption, windowID, imageOption)+ }+/// Capture the topmost layer-0 window owned by `pid`, or `nil` when the/// pid has no such window at all (menubar-only helpers, apps that/// haven't created any window yet).diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swiftindex 6df93a8e8b..c47afc1fab 100644--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift@@ -74,6 +74,19 @@ public enum GetWindowStateTool {Change with `cua-driver config set capture_mode <mode>` orthe `set_config` tool.+ Screenshot capture failures behave differently per mode:+ in `som`, they're non-fatal — the AX tree still ships in+ the response and the summary line carries a hint, so+ agents can keep doing element-indexed clicks against the+ same window even when the screenshot is unavailable. In+ `vision`, the screenshot IS the deliverable, so the same+ failure returns `isError: true` with an actionable hint+ (try another window, retry later, or switch to+ `capture_mode: ax`). The macOS 26.4.x SCK regression+ (SCStreamError -3801, "Could not start streaming") is+ surfaced this way. Switching to `capture_mode: ax` skips+ the capture attempt entirely on subsequent turns.+Requires Accessibility and Screen Recording permissions.""",inputSchema: [@@ -211,6 +224,13 @@ public enum GetWindowStateTool {// race (window closed between validation and capture)// leaves the snapshot without a screenshot; the structured// response's `has_screenshot=false` surfaces the omission.+ // A `.streamingFailed` (macOS 26.4 SCK regression) is+ // handled differently per mode: in `som` the AX tree is+ // still useful so we swallow it and emit a hint; in+ // `vision` the screenshot IS the deliverable, so we+ // return a hard error with the same actionable guidance+ // as the standalone `screenshot` tool.+ var captureHint: String? = nilif captureMode != .ax {do {let shot = try await capture.captureWindow(@@ -237,12 +257,52 @@ public enum GetWindowStateTool {} catch CaptureError.windowNotFound {// Window raced — swallow and emit a screenshot-less// response.+ } catch CaptureError.streamingFailed(let msg) {+ if captureMode == .vision {+ // In `vision` mode the screenshot IS the+ // deliverable — there's no AX tree to fall+ // back to. Surface the same actionable hint+ // as the standalone `screenshot` tool.+ return errorResult(+ """+ ScreenCaptureKit refused this window: \(msg)++ This is a known macOS 26.4.x SCK regression that hits \+ specific windows on physical Macs. The legacy \+ CGWindowList fallback also returned no image.++ Workarounds:+ • Try a different `window_id` on the same app — \+ often only one window is affected.+ • For element-indexed clicks, switch to AX-only: \+ `cua-driver config set capture_mode ax` and re-call \+ `get_window_state` (no screenshot, AX tree only).+ • Re-snapshot a moment later — the failure is \+ sometimes transient.+ """+ )+ }+ // `som` mode: AX snapshot is still useful for+ // element-indexed clicks, so we don't fail the+ // call. The hint nudges the caller toward+ // `capture_mode: ax` to skip the capture attempt+ // entirely on subsequent turns.+ captureHint = """+ ⚠️ Screenshot skipped: ScreenCaptureKit refused this \+ window (\(msg)). Known macOS 26.4 SCK regression. The \+ AX tree below is still valid; element-indexed clicks \+ work as usual. To suppress future capture attempts on \+ this app: `cua-driver config set capture_mode ax`.+ """}}var textContent = buildSummary(snapshot: snapshot, pid: pid, mode: captureMode)+ if let captureHint {+ textContent += "\n" + captureHint+ }if captureMode != .vision && !snapshot.treeMarkdown.isEmpty {textContent += "\n\n" + snapshot.treeMarkdown}diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swiftindex bd61d11911..0c43f48631 100644--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift@@ -18,6 +18,14 @@ public enum ScreenshotTool {Requires the Screen Recording TCC grant — call `check_permissions`first if unsure.++ On macOS 26.4.x, ScreenCaptureKit can refuse specific windows on+ physical Macs (SCStreamError -3801, "Could not start streaming").+ The driver retries once and falls back to the legacy+ CGWindowList path before failing; if both refuse, the error+ response includes a hint to try a different `window_id` or+ switch to `capture_mode: ax` for `get_window_state` (the+ element-indexed flow doesn't need pixels).""",inputSchema: ["type": "object",@@ -112,6 +120,38 @@ public enum ScreenshotTool {],isError: true)+ } catch CaptureError.streamingFailed(let msg) {+ // SCK streaming-start regression on macOS 26.4.x — the+ // legacy CGWindowList fallback also refused this specific+ // window. There's nothing we can do at the pixel layer;+ // surface an actionable hint pointing at `get_window_state`+ // (which can fall back to AX-only via `capture_mode: ax`)+ // or trying a different window.+ return CallTool.Result(+ content: [+ .text(+ text: """+ ScreenCaptureKit refused this window: \(msg)++ This is a known macOS 26.4.x SCK regression that hits \+ specific windows on physical Macs. The legacy \+ CGWindowList fallback also returned no image.++ Workarounds:+ • Try a different `window_id` on the same app — \+ often only one window is affected.+ • For element-indexed clicks, switch to AX-only: \+ `cua-driver config set capture_mode ax` and use \+ `get_window_state` (no screenshot, AX tree only).+ • Re-snapshot a moment later — the failure is \+ sometimes transient.+ """,+ annotations: nil,+ _meta: nil+ )+ ],+ isError: true+ )} catch {return CallTool.Result(content: [