Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cua-driver): add screenshot_out_file param; CLI no longer dumps base64 by default (#1414)

Francesco Bonacci9d623c03c204
diff --git a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
index 6b657edcde..daabda628b 100644
--- a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
+++ b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
@@ -90,7 +90,43 @@ Tools appear prefixed as `mcp_cua-driver_*`.
cua-driver mcp-config --client opencode
```
-Paste the output into your `opencode.json`.
+Paste the output into `~/.config/opencode/config.json` (global) or `opencode.json` at the project root.
+
+<Callout type="warn">
+**Always configure cua-driver as an MCP server — never rely on the CLI fallback.** If MCP is not wired up, OpenCode calls `cua-driver` as a shell subprocess. The `get_window_state` response no longer includes base64 by default, but the screenshot image block is silently dropped — the model receives only the AX tree with no visual context. Use `--screenshot-out-file` or the `screenshot_out_file` param to preserve the image when using the CLI path.
+</Callout>
+
+### Local vision models (Ollama)
+
+If you are using a vision-capable model via Ollama, you must also declare its input modalities in `config.json` — otherwise OpenCode strips images before they reach the model:
+
+```json
+{
+ "mcp": {
+ "cua-driver": {
+ "type": "local",
+ "command": ["/Users/you/.local/bin/cua-driver", "mcp"],
+ "enabled": true
+ }
+ },
+ "provider": {
+ "ollama": {
+ "npm": "@ai-sdk/openai-compatible",
+ "options": { "baseURL": "http://localhost:11434/v1" },
+ "models": {
+ "gemma4:26b": {
+ "modalities": {
+ "input": ["text", "image"],
+ "output": ["text"]
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+The `modalities` field is required because OpenCode's `@ai-sdk/openai-compatible` provider defaults to text-only when no capabilities are declared. Without it, screenshots are replaced with an error string and never reach the model.
## Hermes (NousResearch)
diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.md
index 3858f74922..69d8177ccd 100644
--- a/libs/cua-driver/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md
@@ -432,33 +432,26 @@ In `som` mode the response carries:
~1600 elements, ~190 KB); when it exceeds token limits the MCP
harness saves it to a file and returns the path. Use `Bash` +
`jq -r '.tree_markdown'` + `grep` to pull the section you need.
-- `screenshot_png_b64` + `screenshot_width` / `_height` /
- `_scale_factor` — the window screenshot (actually JPEG-85 despite
- the `_png_` field name, hard-coded in
- `WindowCapture.captureFrontmostWindow`). Present in `som` mode
- (spliced into the structured JSON alongside the tree). In `vision`
- mode the image arrives as a native MCP image content block with no
- structured wrapper. Omitted when the target has no on-screen
- window.
-- `has_screenshot: bool` — **gate on this before piping the PNG**.
- Otherwise `jq -r '.screenshot_png_b64'` emits the literal
- `"null"`, base64-decodes into 3 bytes of garbage, and downstream
- vision APIs reject it with an opaque "Could not process image"
- error.
-
-```
-# canonical, works in every capture mode — writes the image bytes
-# wherever you point, stdout stays readable (tree in som, summary
-# in vision). stderr warns (exit 0) if the response had no image.
-cua-driver get_window_state '{"pid":N,"window_id":W}' --image-out /tmp/shot.png
-
-# som-only legacy path: pull the spliced base64 out of structuredContent.
-# Prefer --image-out above — it's one flag vs a probe + pipe.
-if [ "$(cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.has_screenshot')" = "true" ]; then
- cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.screenshot_png_b64' | base64 -d > shot.png
-fi
+- `screenshot_file_path` — absolute path to the saved screenshot when
+ `screenshot_out_file` was passed. Absent otherwise.
+- `screenshot_width` / `_height` / `_scale_factor` — dimensions of the
+ captured image. Present whenever a screenshot was taken.
+**Getting the screenshot as a file (CLI and context-constrained agents):**
+
+```bash
+# write to file — stdout stays readable (AX tree / summary only, no base64)
+cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'
+
+# CLI --screenshot-out-file flag is equivalent and works for all capture modes
+cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg
```
+Pass `screenshot_out_file` when using `get_window_state` via CLI or from an
+agent whose context window can't absorb ~31 KB of inline base64 (e.g.
+OpenCode with a local Ollama model). The MCP image content block is omitted
+from the response when this param is set — the model receives only the AX
+tree and `screenshot_file_path`, then reads the image from disk.
+
**Reason over both the tree AND the screenshot — they're
complementary, not redundant.** In `som` mode every
turn's `get_window_state` gives you both halves and you should pull
@@ -556,7 +549,7 @@ below against the full-resolution file in that case.
1. `get_window_state({pid, window_id})` returns an image capped
at 1568 long-side (default) plus its dimensions
(`screenshot_width` / `screenshot_height`). Write the bytes to
- disk with `--image-out <path>` in any capture mode — works
+ disk with `--screenshot-out-file <path>` in any capture mode — works
identically in `vision` (where it's the only way) and `som`
(where it sidesteps the jq + base64 dance on the spliced
`screenshot_png_b64` field).
diff --git a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
index 1f09722339..46488d59b1 100644
--- a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
+++ b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
@@ -83,13 +83,13 @@ struct CallCommand: AsyncParsableCommand {
file path. `vision` and `screenshot` capture modes return a
PNG as a native MCP image block with no accompanying
structuredContent, so the CLI's text formatter would otherwise
- drop the bytes silently. With `--image-out /tmp/shot.png` the
- raw PNG lands on disk and downstream tooling (PIL, sips,
+ drop the bytes silently. With `--screenshot-out-file /tmp/shot.jpg`
+ the raw image lands on disk and downstream tooling (PIL, sips,
ffprobe) can read it directly. Silently warns (no error exit)
when the response carries no image.
"""
)
- var imageOut: String?
+ var screenshotOutFile: String?
@OptionGroup var output: JSONOutputOptions
@OptionGroup var daemon: DaemonForwardingOptions
@@ -113,7 +113,7 @@ struct CallCommand: AsyncParsableCommand {
arguments: arguments,
socketPath: daemon.resolvedSocketPath,
raw: raw,
- imageOut: imageOut,
+ imageOut: screenshotOutFile,
output: output
)
return
@@ -184,7 +184,7 @@ struct CallCommand: AsyncParsableCommand {
throw ExitCode(Exit.software)
}
- if let path = imageOut {
+ if let path = screenshotOutFile {
writeFirstImageContent(result.content, to: path)
}
@@ -251,10 +251,7 @@ struct CallCommand: AsyncParsableCommand {
let encoder = JSONEncoder()
encoder.outputFormatting = output.encoderOutputFormatting
let encoded = try encoder.encode(structured)
- let merged = mergeImageContentIntoJSON(
- encoded, content: result.content, output: output
- )
- printDataLine(merged)
+ printDataLine(encoded)
return
}
@@ -352,7 +349,7 @@ func forwardCallToDaemon(
arguments: [String: Value]?,
socketPath: String,
raw: Bool,
- imageOut: String?,
+ screenshotOutFile: String?,
output: JSONOutputOptions
) async throws {
let request = DaemonRequest(method: "call", name: toolName, args: arguments)
@@ -381,7 +378,7 @@ func forwardCallToDaemon(
throw ExitCode(Exit.software)
}
- if let path = imageOut {
+ if let path = screenshotOutFile {
writeFirstImageContent(result.content, to: path)
}
@@ -426,10 +423,7 @@ private func emitUnwrappedResultForDaemon(
let encoder = JSONEncoder()
encoder.outputFormatting = output.encoderOutputFormatting
let encoded = try encoder.encode(structured)
- let merged = mergeImageContentIntoJSON(
- encoded, content: result.content, output: output
- )
- FileHandle.standardOutput.write(merged)
+ FileHandle.standardOutput.write(encoded)
FileHandle.standardOutput.write(Data("\n".utf8))
return
}
@@ -487,70 +481,23 @@ enum DaemonCLIError: Error {
case protocolMismatch
}
-/// MCP delivers screenshot bytes as a native `.image()` content block
-/// separate from `structuredContent`, so shell consumers of
-/// `cua-driver <tool>` only see the metadata half ("has_screenshot",
-/// dimensions) and the base64 pixels silently vanish. This reunites them
-/// at CLI emit time: if any image block is present in `content`, splice
-/// its base64 into the outgoing JSON as `screenshot_png_b64` (plus
-/// `screenshot_mime_type`) alongside the existing fields. The MCP wire
-/// contract is unchanged — only the CLI's pretty-printed output gains
-/// the pixels that downstream `jq -r '.screenshot_png_b64' | base64 -d`
-/// pipelines expect.
-///
-/// Returns the input `encoded` unchanged when there's no image block to
-/// splice OR when the structured content didn't decode as a JSON object
-/// (e.g. a bare array / scalar from some other tool). Never throws —
-/// merge failures fall through silently so a malformed structured
-/// response still emits its original bytes.
-private func mergeImageContentIntoJSON(
- _ encoded: Data,
- content: [Tool.Content],
- output: JSONOutputOptions
-) -> Data {
- var imageBase64: String? = nil
- var imageMime: String? = nil
- for item in content {
- if case let .image(data, mime, _, _) = item {
- imageBase64 = data
- imageMime = mime
- break
- }
- }
- guard let imageBase64, let imageMime else { return encoded }
-
- guard
- var object = (try? JSONSerialization.jsonObject(with: encoded))
- as? [String: Any]
- else { return encoded }
- object["screenshot_png_b64"] = imageBase64
- object["screenshot_mime_type"] = imageMime
-
- var writingOptions: JSONSerialization.WritingOptions = [
- .sortedKeys, .withoutEscapingSlashes,
- ]
- if !output.compact { writingOptions.insert(.prettyPrinted) }
- return (try? JSONSerialization.data(
- withJSONObject: object, options: writingOptions
- )) ?? encoded
-}
/// Write the first `.image(...)` content block from a tool result to
-/// `path`, decoded from base64. Used by the `--image-out` flag so
+/// `path`, decoded from base64. Used by the `--screenshot-out-file` flag so
/// vision-mode screenshots land on disk without callers needing to
/// speak the raw daemon protocol or the MCP bridge.
///
/// Never throws — an empty/malformed image, a write failure, or a
/// response that carries no image at all each emit a stderr warning
/// and return. The exit code is NOT affected: callers have already
-/// gotten the textual result on stdout, and `--image-out` is
+/// gotten the textual result on stdout, and `--screenshot-out-file` is
/// advisory ("if the tool returned a PNG, put it here"), not a hard
/// contract.
func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
for item in content {
if case let .image(base64, mime, _, _) = item {
guard let bytes = Data(base64Encoded: base64) else {
- printToStderr("--image-out: base64 decode failed for \(mime) block")
+ printToStderr("--screenshot-out-file: base64 decode failed for \(mime) block")
return
}
let url = URL(fileURLWithPath: (path as NSString).expandingTildeInPath)
@@ -558,7 +505,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
try bytes.write(to: url)
} catch {
printToStderr(
- "--image-out: failed to write \(url.path): \(error.localizedDescription)"
+ "--screenshot-out-file: failed to write \(url.path): \(error.localizedDescription)"
)
}
return
@@ -568,7 +515,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
// tools legitimately return nothing (hidden/minimized windows,
// `ax` capture mode, list operations). Warn so the user notices
// the file they expected didn't get written.
- printToStderr("--image-out: no image content in tool response; file not written")
+ printToStderr("--screenshot-out-file: no image content in tool response; file not written")
}
private func printUnknownTool(_ name: String, registry: ToolRegistry) {
diff --git a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
index a463005c86..65b4cbfd71 100644
--- a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
+++ b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
@@ -54,6 +54,9 @@ public struct AppStateSnapshot: Sendable, Codable {
public let screenshotOriginalWidth: Int?
/// Original height before maxImageDimension resize. nil = no resize.
public let screenshotOriginalHeight: Int?
+ /// File-system path to the saved screenshot (JPEG) when the caller
+ /// passed `screenshot_out_file`. Absent when no path was written.
+ public let screenshotFilePath: String?
public init(
pid: Int32,
@@ -67,7 +70,8 @@ public struct AppStateSnapshot: Sendable, Codable {
screenshotHeight: Int? = nil,
screenshotScaleFactor: Double? = nil,
screenshotOriginalWidth: Int? = nil,
- screenshotOriginalHeight: Int? = nil
+ screenshotOriginalHeight: Int? = nil,
+ screenshotFilePath: String? = nil
) {
self.pid = pid
self.bundleId = bundleId
@@ -81,6 +85,7 @@ public struct AppStateSnapshot: Sendable, Codable {
self.screenshotScaleFactor = screenshotScaleFactor
self.screenshotOriginalWidth = screenshotOriginalWidth
self.screenshotOriginalHeight = screenshotOriginalHeight
+ self.screenshotFilePath = screenshotFilePath
}
private enum CodingKeys: String, CodingKey {
@@ -95,6 +100,7 @@ public struct AppStateSnapshot: Sendable, Codable {
case screenshotScaleFactor = "screenshot_scale_factor"
case screenshotOriginalWidth = "screenshot_original_width"
case screenshotOriginalHeight = "screenshot_original_height"
+ case screenshotFilePath = "screenshot_file_path"
}
/// Encodes the struct with ``screenshot_*`` keys omitted when nil.
@@ -116,6 +122,7 @@ public struct AppStateSnapshot: Sendable, Codable {
try container.encodeIfPresent(screenshotScaleFactor, forKey: .screenshotScaleFactor)
try container.encodeIfPresent(screenshotOriginalWidth, forKey: .screenshotOriginalWidth)
try container.encodeIfPresent(screenshotOriginalHeight, forKey: .screenshotOriginalHeight)
+ try container.encodeIfPresent(screenshotFilePath, forKey: .screenshotFilePath)
}
public init(from decoder: Decoder) throws {
@@ -132,6 +139,7 @@ public struct AppStateSnapshot: Sendable, Codable {
screenshotScaleFactor = try container.decodeIfPresent(Double.self, forKey: .screenshotScaleFactor)
screenshotOriginalWidth = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalWidth)
screenshotOriginalHeight = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalHeight)
+ screenshotFilePath = try container.decodeIfPresent(String.self, forKey: .screenshotFilePath)
}
}
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
index 18e0ee4d29..6df93a8e8b 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
@@ -107,6 +107,19 @@ public enum GetWindowStateTool {
For mutations or side effects use the `page` tool instead.
""",
],
+ "screenshot_out_file": [
+ "type": "string",
+ "description": """
+ Optional absolute path to write the screenshot to (e.g. \
+ "/tmp/shot.jpg"). When set, the screenshot bytes are written \
+ to this file and the MCP image content block is omitted from \
+ the response — `screenshot_file_path` is returned instead of \
+ `screenshot_png_b64`. Useful for CLI callers and agents that \
+ cannot consume inline base64 without saturating their context \
+ window (e.g. OpenCode with a local Ollama model). The directory \
+ must already exist; the file is created or overwritten.
+ """,
+ ],
],
"additionalProperties": false,
],
@@ -137,6 +150,7 @@ public enum GetWindowStateTool {
}
let query = arguments?["query"]?.stringValue
let javascript = arguments?["javascript"]?.stringValue
+ let screenshotOutFile = arguments?["screenshot_out_file"]?.stringValue
// Validate that the window belongs to this pid. The driver
// never guesses which window to snapshot — the caller names
@@ -254,16 +268,37 @@ public enum GetWindowStateTool {
}
}
+ // When the caller supplied screenshot_out_file, write the bytes
+ // to disk and omit the MCP image content block entirely — the
+ // path is surfaced via structuredContent.screenshot_file_path
+ // so the caller knows where to find the image without paying
+ // the base64-in-context token cost.
+ var resolvedScreenshotFilePath: String? = nil
+ if let outPath = screenshotOutFile, let b64 = snapshot.screenshotPngBase64 {
+ let expandedPath = (outPath as NSString).expandingTildeInPath
+ if let bytes = Data(base64Encoded: b64) {
+ let url = URL(fileURLWithPath: expandedPath)
+ do {
+ try bytes.write(to: url)
+ resolvedScreenshotFilePath = expandedPath
+ } catch {
+ // Write failed — fall through so the inline image
+ // content block is still emitted rather than silently
+ // dropping the screenshot entirely.
+ }
+ }
+ }
+
var content: [Tool.Content] = []
- if let b64 = snapshot.screenshotPngBase64 {
+ if resolvedScreenshotFilePath == nil, let b64 = snapshot.screenshotPngBase64 {
content.append(
.image(data: b64, mimeType: "image/jpeg", annotations: nil, _meta: nil)
)
}
content.append(.text(text: textContent, annotations: nil, _meta: nil))
// Strip the b64 bytes from the structured snapshot — the image
- // is already the first content block and tests just need the
- // metadata fields (screenshot_scale_factor, dimensions, etc.).
+ // is already the first content block (or on disk) and tests
+ // just need the metadata fields.
let structuredSnapshot = AppStateSnapshot(
pid: snapshot.pid,
bundleId: snapshot.bundleId,
@@ -276,7 +311,8 @@ public enum GetWindowStateTool {
screenshotHeight: snapshot.screenshotHeight,
screenshotScaleFactor: snapshot.screenshotScaleFactor,
screenshotOriginalWidth: snapshot.screenshotOriginalWidth,
- screenshotOriginalHeight: snapshot.screenshotOriginalHeight
+ screenshotOriginalHeight: snapshot.screenshotOriginalHeight,
+ screenshotFilePath: resolvedScreenshotFilePath
)
if let result = try? CallTool.Result(
content: content,