Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
CuaSpacesTranscript
Terminal scrollback in, structured UI out.
AgentTypes.swift carries output_tail: the Spaces harness publishes what the
agent's terminal looks like, not what the agent did. Every proposal for
agent.events is blocked on that, and
examples/openkoalabot-example-swift/FRICTION.md §19 and
§20 predicted the consequence: one undifferentiated text stream with no turn
boundary, so every app writes its own parser and they all guess differently.
This target is that parser, written once, with golden data recorded from the real CLI so it is checkable rather than plausible.
Four pieces
| Piece | Type | What it does |
|---|---|---|
| Emulator | TerminalEmulator → ScreenBuffer |
VT100/xterm subset. Claude Code repaints; concatenating the byte stream gives a wall of glyphs no human saw. |
| Playback | CastPlayer → RenderedFrame |
The rendered screen at time T in an asciinema v2 recording. |
| Parser | ClaudeCodeParser → ParsedFrame |
A frame becomes a typed model of what is on screen. |
| Encoding | ParsedFrame.jsonUIDocument() |
That model becomes cua.transcript.jsonui/1, a document with no Swift in it. |
Stages three and four are deliberately separate: a change to the Swift model that does not change the document is not a wire change, and a non-Swift client gets the same structure a Swift one does.
Without a recording
The output_tail case needs no cast:
let frame = RenderedFrame.render(stream: snapshot.outputTail ?? "",
columns: 100, rows: 34,
cli: "claude", cliVersion: knownVersion)
let document = ClaudeCodeParser().parse(frame: frame).jsonUIDocument()
Terminal size is the caller's to supply, because the tail does not carry it and guessing it changes where every line wraps.
The honesty rule
The rule is binding: the SDK must never mint a structured element the agent
did not actually offer. Every element carries a Provenance, and so does the
JSON:
observed: the text is on the screen and this element is a direct reading of it.⏺ Update(calc.py)givestoolCall(name: "Update")because those characters were drawn.inferred: the parser concluded it from layout, colour or adjacency and the agent never said it. A tool call with no result row under it isstatus: running, statusProvenance: inferred, because "running" is our reading of an absence. A diff classified from a background colour rather than a drawn+isinferredtoo.unrecognised: the parser could not classify the region and is handing back the text unchanged. This is the required behaviour when the CLI's layout moves, not a failure.
ToolCall and Subagent split statusProvenance out from the element's own
provenance on purpose: the name can be read off the screen while the
status is concluded, and one flag for both would overstate one or understate
the other.
A renderer that draws a confident card for an inferred element is
misrepresenting the agent. ParsedFrame.observed and .inferred exist so that
is a choice rather than an accident.
Versions
Claude Code's rendering changes between releases. Every cast records the CLI's
own --version output, and it travels into ParsedFrame.cliVersion and into
the JSON's source.cliVersion. layoutProfile names the layout that matched:
claude-code/2.x, or null when nothing did and the whole frame degraded to
text.
Goldens
Tests/CuaSpacesTranscriptTests/Goldens holds real recordings and the expected
JSON-UI at named timestamps. swift test --filter GoldenTests replays, parses
and compares. A parser change that alters output must change a golden, and the
diff is visible in review.
To re-bless after an intentional change:
CUA_TRANSCRIPT_RECORD=1 swift test --filter GoldenTests
Read every changed file before committing it. Re-blessing without reading the diff defeats the mechanism.
To re-record against a newer CLI, see scripts/cli-cast/.