Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cua-driver): add screenshot_out_file param; CLI no longer dumps base64 by default (#1414)
Francesco Bonacci9d623c03c204
diff --git a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdxindex 6b657edcde..daabda628b 100644--- a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx+++ b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx@@ -90,7 +90,43 @@ Tools appear prefixed as `mcp_cua-driver_*`.cua-driver mcp-config --client opencode```-Paste the output into your `opencode.json`.+Paste the output into `~/.config/opencode/config.json` (global) or `opencode.json` at the project root.++<Callout type="warn">+**Always configure cua-driver as an MCP server — never rely on the CLI fallback.** If MCP is not wired up, OpenCode calls `cua-driver` as a shell subprocess. The `get_window_state` response no longer includes base64 by default, but the screenshot image block is silently dropped — the model receives only the AX tree with no visual context. Use `--screenshot-out-file` or the `screenshot_out_file` param to preserve the image when using the CLI path.+</Callout>++### Local vision models (Ollama)++If you are using a vision-capable model via Ollama, you must also declare its input modalities in `config.json` — otherwise OpenCode strips images before they reach the model:++```json+{+ "mcp": {+ "cua-driver": {+ "type": "local",+ "command": ["/Users/you/.local/bin/cua-driver", "mcp"],+ "enabled": true+ }+ },+ "provider": {+ "ollama": {+ "npm": "@ai-sdk/openai-compatible",+ "options": { "baseURL": "http://localhost:11434/v1" },+ "models": {+ "gemma4:26b": {+ "modalities": {+ "input": ["text", "image"],+ "output": ["text"]+ }+ }+ }+ }+ }+}+```++The `modalities` field is required because OpenCode's `@ai-sdk/openai-compatible` provider defaults to text-only when no capabilities are declared. Without it, screenshots are replaced with an error string and never reach the model.## Hermes (NousResearch)diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.mdindex 3858f74922..69d8177ccd 100644--- a/libs/cua-driver/Skills/cua-driver/SKILL.md+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md@@ -432,33 +432,26 @@ In `som` mode the response carries:~1600 elements, ~190 KB); when it exceeds token limits the MCPharness saves it to a file and returns the path. Use `Bash` +`jq -r '.tree_markdown'` + `grep` to pull the section you need.-- `screenshot_png_b64` + `screenshot_width` / `_height` /- `_scale_factor` — the window screenshot (actually JPEG-85 despite- the `_png_` field name, hard-coded in- `WindowCapture.captureFrontmostWindow`). Present in `som` mode- (spliced into the structured JSON alongside the tree). In `vision`- mode the image arrives as a native MCP image content block with no- structured wrapper. Omitted when the target has no on-screen- window.-- `has_screenshot: bool` — **gate on this before piping the PNG**.- Otherwise `jq -r '.screenshot_png_b64'` emits the literal- `"null"`, base64-decodes into 3 bytes of garbage, and downstream- vision APIs reject it with an opaque "Could not process image"- error.--```-# canonical, works in every capture mode — writes the image bytes-# wherever you point, stdout stays readable (tree in som, summary-# in vision). stderr warns (exit 0) if the response had no image.-cua-driver get_window_state '{"pid":N,"window_id":W}' --image-out /tmp/shot.png--# som-only legacy path: pull the spliced base64 out of structuredContent.-# Prefer --image-out above — it's one flag vs a probe + pipe.-if [ "$(cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.has_screenshot')" = "true" ]; then- cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.screenshot_png_b64' | base64 -d > shot.png-fi+- `screenshot_file_path` — absolute path to the saved screenshot when+ `screenshot_out_file` was passed. Absent otherwise.+- `screenshot_width` / `_height` / `_scale_factor` — dimensions of the+ captured image. Present whenever a screenshot was taken.+**Getting the screenshot as a file (CLI and context-constrained agents):**++```bash+# write to file — stdout stays readable (AX tree / summary only, no base64)+cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'++# CLI --screenshot-out-file flag is equivalent and works for all capture modes+cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg```+Pass `screenshot_out_file` when using `get_window_state` via CLI or from an+agent whose context window can't absorb ~31 KB of inline base64 (e.g.+OpenCode with a local Ollama model). The MCP image content block is omitted+from the response when this param is set — the model receives only the AX+tree and `screenshot_file_path`, then reads the image from disk.+**Reason over both the tree AND the screenshot — they'recomplementary, not redundant.** In `som` mode everyturn's `get_window_state` gives you both halves and you should pull@@ -556,7 +549,7 @@ below against the full-resolution file in that case.1. `get_window_state({pid, window_id})` returns an image cappedat 1568 long-side (default) plus its dimensions(`screenshot_width` / `screenshot_height`). Write the bytes to- disk with `--image-out <path>` in any capture mode — works+ disk with `--screenshot-out-file <path>` in any capture mode — worksidentically in `vision` (where it's the only way) and `som`(where it sidesteps the jq + base64 dance on the spliced`screenshot_png_b64` field).diff --git a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swiftindex 1f09722339..46488d59b1 100644--- a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift+++ b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift@@ -83,13 +83,13 @@ struct CallCommand: AsyncParsableCommand {file path. `vision` and `screenshot` capture modes return aPNG as a native MCP image block with no accompanyingstructuredContent, so the CLI's text formatter would otherwise- drop the bytes silently. With `--image-out /tmp/shot.png` the- raw PNG lands on disk and downstream tooling (PIL, sips,+ drop the bytes silently. With `--screenshot-out-file /tmp/shot.jpg`+ the raw image lands on disk and downstream tooling (PIL, sips,ffprobe) can read it directly. Silently warns (no error exit)when the response carries no image.""")- var imageOut: String?+ var screenshotOutFile: String?@OptionGroup var output: JSONOutputOptions@OptionGroup var daemon: DaemonForwardingOptions@@ -113,7 +113,7 @@ struct CallCommand: AsyncParsableCommand {arguments: arguments,socketPath: daemon.resolvedSocketPath,raw: raw,- imageOut: imageOut,+ imageOut: screenshotOutFile,output: output)return@@ -184,7 +184,7 @@ struct CallCommand: AsyncParsableCommand {throw ExitCode(Exit.software)}- if let path = imageOut {+ if let path = screenshotOutFile {writeFirstImageContent(result.content, to: path)}@@ -251,10 +251,7 @@ struct CallCommand: AsyncParsableCommand {let encoder = JSONEncoder()encoder.outputFormatting = output.encoderOutputFormattinglet encoded = try encoder.encode(structured)- let merged = mergeImageContentIntoJSON(- encoded, content: result.content, output: output- )- printDataLine(merged)+ printDataLine(encoded)return}@@ -352,7 +349,7 @@ func forwardCallToDaemon(arguments: [String: Value]?,socketPath: String,raw: Bool,- imageOut: String?,+ screenshotOutFile: String?,output: JSONOutputOptions) async throws {let request = DaemonRequest(method: "call", name: toolName, args: arguments)@@ -381,7 +378,7 @@ func forwardCallToDaemon(throw ExitCode(Exit.software)}- if let path = imageOut {+ if let path = screenshotOutFile {writeFirstImageContent(result.content, to: path)}@@ -426,10 +423,7 @@ private func emitUnwrappedResultForDaemon(let encoder = JSONEncoder()encoder.outputFormatting = output.encoderOutputFormattinglet encoded = try encoder.encode(structured)- let merged = mergeImageContentIntoJSON(- encoded, content: result.content, output: output- )- FileHandle.standardOutput.write(merged)+ FileHandle.standardOutput.write(encoded)FileHandle.standardOutput.write(Data("\n".utf8))return}@@ -487,70 +481,23 @@ enum DaemonCLIError: Error {case protocolMismatch}-/// MCP delivers screenshot bytes as a native `.image()` content block-/// separate from `structuredContent`, so shell consumers of-/// `cua-driver <tool>` only see the metadata half ("has_screenshot",-/// dimensions) and the base64 pixels silently vanish. This reunites them-/// at CLI emit time: if any image block is present in `content`, splice-/// its base64 into the outgoing JSON as `screenshot_png_b64` (plus-/// `screenshot_mime_type`) alongside the existing fields. The MCP wire-/// contract is unchanged — only the CLI's pretty-printed output gains-/// the pixels that downstream `jq -r '.screenshot_png_b64' | base64 -d`-/// pipelines expect.-///-/// Returns the input `encoded` unchanged when there's no image block to-/// splice OR when the structured content didn't decode as a JSON object-/// (e.g. a bare array / scalar from some other tool). Never throws —-/// merge failures fall through silently so a malformed structured-/// response still emits its original bytes.-private func mergeImageContentIntoJSON(- _ encoded: Data,- content: [Tool.Content],- output: JSONOutputOptions-) -> Data {- var imageBase64: String? = nil- var imageMime: String? = nil- for item in content {- if case let .image(data, mime, _, _) = item {- imageBase64 = data- imageMime = mime- break- }- }- guard let imageBase64, let imageMime else { return encoded }-- guard- var object = (try? JSONSerialization.jsonObject(with: encoded))- as? [String: Any]- else { return encoded }- object["screenshot_png_b64"] = imageBase64- object["screenshot_mime_type"] = imageMime-- var writingOptions: JSONSerialization.WritingOptions = [- .sortedKeys, .withoutEscapingSlashes,- ]- if !output.compact { writingOptions.insert(.prettyPrinted) }- return (try? JSONSerialization.data(- withJSONObject: object, options: writingOptions- )) ?? encoded-}/// Write the first `.image(...)` content block from a tool result to-/// `path`, decoded from base64. Used by the `--image-out` flag so+/// `path`, decoded from base64. Used by the `--screenshot-out-file` flag so/// vision-mode screenshots land on disk without callers needing to/// speak the raw daemon protocol or the MCP bridge.////// Never throws — an empty/malformed image, a write failure, or a/// response that carries no image at all each emit a stderr warning/// and return. The exit code is NOT affected: callers have already-/// gotten the textual result on stdout, and `--image-out` is+/// gotten the textual result on stdout, and `--screenshot-out-file` is/// advisory ("if the tool returned a PNG, put it here"), not a hard/// contract.func writeFirstImageContent(_ content: [Tool.Content], to path: String) {for item in content {if case let .image(base64, mime, _, _) = item {guard let bytes = Data(base64Encoded: base64) else {- printToStderr("--image-out: base64 decode failed for \(mime) block")+ printToStderr("--screenshot-out-file: base64 decode failed for \(mime) block")return}let url = URL(fileURLWithPath: (path as NSString).expandingTildeInPath)@@ -558,7 +505,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {try bytes.write(to: url)} catch {printToStderr(- "--image-out: failed to write \(url.path): \(error.localizedDescription)"+ "--screenshot-out-file: failed to write \(url.path): \(error.localizedDescription)")}return@@ -568,7 +515,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {// tools legitimately return nothing (hidden/minimized windows,// `ax` capture mode, list operations). Warn so the user notices// the file they expected didn't get written.- printToStderr("--image-out: no image content in tool response; file not written")+ printToStderr("--screenshot-out-file: no image content in tool response; file not written")}private func printUnknownTool(_ name: String, registry: ToolRegistry) {diff --git a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swiftindex a463005c86..65b4cbfd71 100644--- a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift+++ b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift@@ -54,6 +54,9 @@ public struct AppStateSnapshot: Sendable, Codable {public let screenshotOriginalWidth: Int?/// Original height before maxImageDimension resize. nil = no resize.public let screenshotOriginalHeight: Int?+ /// File-system path to the saved screenshot (JPEG) when the caller+ /// passed `screenshot_out_file`. Absent when no path was written.+ public let screenshotFilePath: String?public init(pid: Int32,@@ -67,7 +70,8 @@ public struct AppStateSnapshot: Sendable, Codable {screenshotHeight: Int? = nil,screenshotScaleFactor: Double? = nil,screenshotOriginalWidth: Int? = nil,- screenshotOriginalHeight: Int? = nil+ screenshotOriginalHeight: Int? = nil,+ screenshotFilePath: String? = nil) {self.pid = pidself.bundleId = bundleId@@ -81,6 +85,7 @@ public struct AppStateSnapshot: Sendable, Codable {self.screenshotScaleFactor = screenshotScaleFactorself.screenshotOriginalWidth = screenshotOriginalWidthself.screenshotOriginalHeight = screenshotOriginalHeight+ self.screenshotFilePath = screenshotFilePath}private enum CodingKeys: String, CodingKey {@@ -95,6 +100,7 @@ public struct AppStateSnapshot: Sendable, Codable {case screenshotScaleFactor = "screenshot_scale_factor"case screenshotOriginalWidth = "screenshot_original_width"case screenshotOriginalHeight = "screenshot_original_height"+ case screenshotFilePath = "screenshot_file_path"}/// Encodes the struct with ``screenshot_*`` keys omitted when nil.@@ -116,6 +122,7 @@ public struct AppStateSnapshot: Sendable, Codable {try container.encodeIfPresent(screenshotScaleFactor, forKey: .screenshotScaleFactor)try container.encodeIfPresent(screenshotOriginalWidth, forKey: .screenshotOriginalWidth)try container.encodeIfPresent(screenshotOriginalHeight, forKey: .screenshotOriginalHeight)+ try container.encodeIfPresent(screenshotFilePath, forKey: .screenshotFilePath)}public init(from decoder: Decoder) throws {@@ -132,6 +139,7 @@ public struct AppStateSnapshot: Sendable, Codable {screenshotScaleFactor = try container.decodeIfPresent(Double.self, forKey: .screenshotScaleFactor)screenshotOriginalWidth = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalWidth)screenshotOriginalHeight = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalHeight)+ screenshotFilePath = try container.decodeIfPresent(String.self, forKey: .screenshotFilePath)}}diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swiftindex 18e0ee4d29..6df93a8e8b 100644--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift@@ -107,6 +107,19 @@ public enum GetWindowStateTool {For mutations or side effects use the `page` tool instead.""",],+ "screenshot_out_file": [+ "type": "string",+ "description": """+ Optional absolute path to write the screenshot to (e.g. \+ "/tmp/shot.jpg"). When set, the screenshot bytes are written \+ to this file and the MCP image content block is omitted from \+ the response — `screenshot_file_path` is returned instead of \+ `screenshot_png_b64`. Useful for CLI callers and agents that \+ cannot consume inline base64 without saturating their context \+ window (e.g. OpenCode with a local Ollama model). The directory \+ must already exist; the file is created or overwritten.+ """,+ ],],"additionalProperties": false,],@@ -137,6 +150,7 @@ public enum GetWindowStateTool {}let query = arguments?["query"]?.stringValuelet javascript = arguments?["javascript"]?.stringValue+ let screenshotOutFile = arguments?["screenshot_out_file"]?.stringValue// Validate that the window belongs to this pid. The driver// never guesses which window to snapshot — the caller names@@ -254,16 +268,37 @@ public enum GetWindowStateTool {}}+ // When the caller supplied screenshot_out_file, write the bytes+ // to disk and omit the MCP image content block entirely — the+ // path is surfaced via structuredContent.screenshot_file_path+ // so the caller knows where to find the image without paying+ // the base64-in-context token cost.+ var resolvedScreenshotFilePath: String? = nil+ if let outPath = screenshotOutFile, let b64 = snapshot.screenshotPngBase64 {+ let expandedPath = (outPath as NSString).expandingTildeInPath+ if let bytes = Data(base64Encoded: b64) {+ let url = URL(fileURLWithPath: expandedPath)+ do {+ try bytes.write(to: url)+ resolvedScreenshotFilePath = expandedPath+ } catch {+ // Write failed — fall through so the inline image+ // content block is still emitted rather than silently+ // dropping the screenshot entirely.+ }+ }+ }+var content: [Tool.Content] = []- if let b64 = snapshot.screenshotPngBase64 {+ if resolvedScreenshotFilePath == nil, let b64 = snapshot.screenshotPngBase64 {content.append(.image(data: b64, mimeType: "image/jpeg", annotations: nil, _meta: nil))}content.append(.text(text: textContent, annotations: nil, _meta: nil))// Strip the b64 bytes from the structured snapshot — the image- // is already the first content block and tests just need the- // metadata fields (screenshot_scale_factor, dimensions, etc.).+ // is already the first content block (or on disk) and tests+ // just need the metadata fields.let structuredSnapshot = AppStateSnapshot(pid: snapshot.pid,bundleId: snapshot.bundleId,@@ -276,7 +311,8 @@ public enum GetWindowStateTool {screenshotHeight: snapshot.screenshotHeight,screenshotScaleFactor: snapshot.screenshotScaleFactor,screenshotOriginalWidth: snapshot.screenshotOriginalWidth,- screenshotOriginalHeight: snapshot.screenshotOriginalHeight+ screenshotOriginalHeight: snapshot.screenshotOriginalHeight,+ screenshotFilePath: resolvedScreenshotFilePath)if let result = try? CallTool.Result(content: content,