Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

Delete JOURNAL_VIDEO.md

Francesco Bonacci02fdd98ad59a
diff --git a/JOURNAL_VIDEO.md b/JOURNAL_VIDEO.md
deleted file mode 100644
index fdc5ce2d6f..0000000000
--- a/JOURNAL_VIDEO.md
+++ /dev/null
@@ -1,363 +0,0 @@
-# Recording rename + cross-platform video — JOURNAL
-
-**Started:** 2026-05-26 ~13:15
-**Branch:** `cua-driver-rs-recording-rename-video` (local only, no PR per instructions)
-**Base:** `origin/main` @ `02f1f033` (MSAA fallback PR merged earlier today)
-**Time budget:** ~2 hours, working autonomously
-**Goal:**
-1. Rename `set_recording` MCP tool → `start_recording` + `stop_recording` (two tools)
-2. Promote `video_experimental` flag → default-on video recording (drop the
- `_experimental` suffix; video records by default)
-3. Implement cross-platform video capture (macOS / Windows / Linux). The user
- notes the Swift implementation worked; the Rust port hardcodes
- `video.present: false` on all platforms.
-
-**Out of scope per user:** no PR, no remote pushes. Local commit on the
-working branch only.
-
----
-
-## Survey results (15 min in)
-
-**Swift reference** (`libs/cua-driver/swift/Sources/CuaDriverCore/Recording/VideoRecorder.swift`):
-- ScreenCaptureKit (SCStream) → AVAssetWriter
-- MP4 / H.264 High profile, 30 fps, main-display-only
-- Synchronized via NSLock; clean teardown on permission revocation
-
-**Rust port** (`libs/cua-driver/rust/crates/mcp-server/src/recording.rs` +
-`recording_tools.rs`):
-- `RecordingSession` writes turn folders + `session.json` (hardcoded
- `"video": { "present": false }` — line 103)
-- No video capture infrastructure at all on any platform
-- The `video_experimental` flag is accepted by the tool schema but only
- emits a text note "(not yet implemented on this platform)" — that note is
- also being stripped by the MCP `structured_content` shape so the warning
- is invisible to callers
-
-**Tool registry** (`tool.rs:79-84`):
-- `register_recording_tools()` registers `SetRecordingTool`,
- `GetRecordingStateTool`, `ReplayTrajectoryTool`. CLI dispatch under
- `cua-driver/src/cli.rs` knows the name `"set_recording"`.
-
-**ffmpeg on this host:** NOT installed. `where ffmpeg` returns nothing.
-
----
-
-## Strategy decision
-
-Cross-platform video capture options I considered:
-
-| Strategy | Pros | Cons | Verdict |
-|---|---|---|---|
-| Native APIs per platform (ScreenCaptureKit / Windows.Graphics.Capture / PipeWire) | Zero runtime deps, matches Swift impl | Weeks of work × 3 platforms; D3D11/DXGI boilerplate on Windows alone is ~500 lines | ❌ not 2-hour-feasible |
-| `scap` Rust crate (cross-platform capture, returns BGRA frames) | Cross-platform, native APIs underneath | Still need encoder → mp4. `mp4` crate exists but no built-in H.264 encoder; would need to vendor x264 or another encoder. Bigger than the 2-hour budget. | ❌ |
-| `ffmpeg-next` (libffmpeg bindings) | Full encoder | Requires linking against system ffmpeg libs; build complexity, cross-compile woes | ❌ |
-| **ffmpeg subprocess** | Cross-platform single binary, simple stdout/stdin handling, robust, fast to ship | External dep (ffmpeg must be on PATH); user has to install it | ✅ |
-
-**Going with ffmpeg subprocess.** The trade — "user installs ffmpeg" — is the
-right one for this budget. The implementation becomes "spawn ffmpeg with the
-right `-f <device> -i <source>` flags per platform, kill on stop, surface a
-clear error when ffmpeg isn't on PATH." Platform-specific input device:
-
-- **Windows:** `ffmpeg -f gdigrab -framerate 30 -i desktop -c:v libx264 -preset ultrafast -pix_fmt yuv420p <output>.mp4`
-- **macOS:** `ffmpeg -f avfoundation -framerate 30 -i "1:" -c:v libx264 -preset ultrafast -pix_fmt yuv420p <output>.mp4` ("1" = main display, ":" = no audio)
-- **Linux (X11):** `ffmpeg -f x11grab -framerate 30 -i :0.0 -c:v libx264 -preset ultrafast -pix_fmt yuv420p <output>.mp4`
-- **Linux (Wayland):** harder — needs PipeWire / wf-recorder. Skip for now, ship X11 path, document.
-
-Encoder choice: `libx264` with `-preset ultrafast` keeps CPU low; `-pix_fmt yuv420p` ensures broad player compatibility (some platforms default to yuv444p which QuickTime / Windows Media Player won't decode).
-
-## Plan
-
-1. **Tool rename** — split `SetRecordingTool` into `StartRecordingTool` (takes
- `output_dir`, optional `record_video`) and `StopRecordingTool` (no args).
- Update `tool.rs` registration + CLI dispatch + tests + skill docs.
-2. **Video runner abstraction** — new `VideoRecorder` trait + per-platform
- impl. Each platform spawns ffmpeg with the right flags. Trait methods:
- `start(path: &Path) -> Result<()>`, `stop() -> Result<FinalMetadata>`.
-3. **Wire into RecordingSession** — `start_recording` constructs a
- `VideoRecorder` per host, calls `start()`. `stop_recording` calls
- `stop()`, records metadata in `session.json`.
-4. **Promote out of experimental** — drop `_experimental` suffix. Default
- `record_video: true` (caller can opt out with `false`).
-5. **ffmpeg-not-found error** — surface a structured error pointing at the
- relevant package manager (winget install Gyan.FFmpeg / brew install ffmpeg
- / apt install ffmpeg).
-6. **Test** — install ffmpeg, run the 5+7 calc demo with the new API, verify
- the MP4 plays.
-
-Will journal each step as I land it.
-
----
-
-## Phase 2: tool rename (done ~50 min in)
-
-`SetRecordingTool` → `StartRecordingTool` + `StopRecordingTool`.
-
-**Schema changes:**
-- `start_recording`: required `output_dir`, optional `record_video` (default
- **true**), nothing else. No more `enabled` boolean — verb-first matches CLI.
-- `stop_recording`: no args. Idempotent.
-
-**Files touched in this phase:**
-- `mcp-server/src/recording_tools.rs` — rewrote the tool defs
-- `mcp-server/src/tool.rs` — registry update + recording exclusion list
- updated (`start_recording` / `stop_recording` get excluded so the
- recorded turn stream stays the user-action sequence, not the meta
- start/stop frames)
-- `mcp-server/src/recording.rs` — `RecordingSession` grew `start()`/`stop()`
- methods; old `configure()` kept as a thin shim so the bridge layer
- doesn't break mid-refactor
-- `cua-driver/src/cli.rs` — `cua-driver recording start|stop` subcommand now
- dispatches to the two new tool names (the CLI subcommand verbiage was
- already verb-first; this just realigns the underlying tool name)
-- `platform-windows/examples/list_tools_parity.rs` — registry list updated
-- `platform-windows/examples/recording_parity.rs` — rewrote against the new
- surface (dropped the Swift-parity framing since the rename intentionally
- breaks parity with the Swift CLI)
-- `cua-driver/tests/mcp_protocol_test.rs` — all `set_recording` references
- converted; `test_set_recording_video_experimental_accepted{_windows}` →
- `test_start_recording_record_video_flag_accepted{_windows}`
-- `Skills/cua-driver/SKILL.md` + `RECORDING.md` — doc updates; the
- RECORDING doc now headlines "Video on by default" near the top
-
-The `configure()` shim kept the migration safe — built incrementally and
-no callers broke between commits.
-
-## Phase 3: video promotion (done in the same rename pass)
-
-Dropped `video_experimental` entirely. New API: `record_video: true`
-default. When `record_video: false` is passed, behaves identically to
-the old non-video code path. When omitted or `true`, the recording
-session spawns ffmpeg.
-
-The `_experimental` suffix was a code smell — flags named like that
-either get promoted or never used. Promoting it now and committing to
-the contract is the right call given the user said "should be out of
-experimental and be enabled by default."
-
-## Phase 4: cross-platform video capture (done ~90 min in)
-
-New module: `mcp-server/src/video.rs`.
-
-**`VideoRecorder` lifecycle:**
-- `VideoRecorder::start(path)` — spawns ffmpeg with platform-appropriate
- flags writing to `path`. Returns the recorder handle.
-- `VideoRecorder::stop()` — sends `q\n` on ffmpeg's stdin (ffmpeg's
- clean-shutdown trigger that finalizes the mp4's moov atom), polls for
- exit up to 3 s, falls back to `kill()`. Returns `VideoMetadata` with
- `duration_ms` + `finalized` so the caller can decide what to do with a
- forcibly-terminated file.
-
-**Platform-specific ffmpeg input args:**
-- Windows: `-f gdigrab -framerate 30 -draw_mouse 1 -i desktop`
-- macOS: `-f avfoundation -framerate 30 -pix_fmt uyvy422 -i 1:`
-- Linux: `-f x11grab -framerate 30 -i $DISPLAY`
-
-**Cross-platform encoder flags:** `libx264 -preset ultrafast -pix_fmt
-yuv420p -movflags +faststart -g 30 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2"`.
-The padding filter was a learning — yuv420p needs even dimensions and
-the host's 949 px screen height made the first run produce a 0-byte mp4
-(`height not divisible by 2 (1512x949)`). Padding by 1 px on the
-bottom/right beats cropping (keeps the full display in frame).
-
-**ffmpeg discovery:** `find_ffmpeg()` first probes `ffmpeg` on PATH,
-then falls back to well-known package-manager install locations
-(winget Gyan.FFmpeg / Homebrew / apt). Lets a fresh winget install
-work without a shell restart — caught a real bug since the cua-driver
-process inherits PATH from the parent shell which doesn't see winget's
-freshly-added entries until the shell is restarted.
-
-**Stderr drain:** Spawned a thread that drains ffmpeg's stderr into a
-4 KB ring buffer. Two reasons: (1) ffmpeg's stderr pipe can fill up
-and block the encoder, (2) when the process exits non-zero we want to
-log the tail of stderr at `warn` level so the failure mode isn't
-silent — that's how I caught the height-divisible-by-2 issue inside
-2 min.
-
-## Phase 5: end-to-end test (done ~110 min in)
-
-`flash-repro/test_video_recording.py` — minimal: start, sleep 5s, stop,
-verify the mp4 plays. Result:
-- mp4: 1.94 MB, H.264, 1512×950 (after padding), yuv420p, ~7 s duration,
- `finalized: true`. ffprobe accepts it cleanly.
-
-`flash-repro/test_video_calc_demo.py` — full agent flow:
-- `start_recording(output_dir=…)` (video default on)
-- `launch_app(Calculator)` → recorded as turn-00001
-- 4 clicks (5, +, 7, =) → turn-00002 through turn-00005
-- `get_window_state(query: Display)` → not recorded (read-only)
-- One final get_window_state recorded as turn-00006 because it's not
- read-only-tagged (TODO: this is actually a small bug; get_window_state
- should be marked read-only)
-- `stop_recording()` → mp4 finalizes
-- Final: 6.6 MB mp4, 6 turn folders, `Display is 12`, session.json
- carries full video metadata
-
-A regression I caught with the existing test suite: turn-00001 was
-being filled with `start_recording` itself before I added the names to
-the exclusion list in `tool.rs::ToolRegistry::invoke()`. Test
-`test_recording_session_windows` failed loud and fast — fixed and
-green.
-
-## Status at handoff
-
-**Done:**
-- `set_recording` → `start_recording` + `stop_recording` rename
-- `video_experimental` flag dropped; new `record_video: true` default
-- ffmpeg-subprocess video capture wired up; works end-to-end on Windows
- (verified with calc 5+7 demo, 6.6 MB mp4, finalized:true)
-- Cross-platform input args coded for macOS (avfoundation) and Linux
- (x11grab) — **not validated** since I'm on Windows
-- Updated tests + examples + skill docs
-
-**Not done / known limitations:**
-- macOS and Linux ffmpeg branches are coded but unverified — I expect
- they work but the avfoundation device-index "1" might need to be
- resolved programmatically per-host (the Swift impl did via
- SCShareableContent). Worth a smoke test from a Mac.
-- Wayland Linux path is missing — x11grab only. PipeWire / wf-recorder
- is the right backend there.
-- The `get_window_state` tool isn't tagged `read_only` despite reading
- the UIA tree, so it's being recorded as a turn. Minor, not a blocker
- for the recording feature itself.
-- Audio capture: not done. Recording is video-only.
-- Per-window video: not done. Currently records the main display.
- Trivially could be a future flag.
-- This change breaks Swift-CLI parity intentionally. The `recording_parity.rs`
- example was rewritten to validate the new surface instead.
-
-**One concrete TODO I'd file as a separate ticket:** `cua-driver doctor`
-should grow an `ffmpeg` check that surfaces the same install-hint message
-the start_recording tool surfaces in `last_error`. Right now the only
-way to learn "video doesn't work because ffmpeg isn't installed" is to
-actually call start_recording and inspect `last_error`.
-
-**Committed locally on branch `cua-driver-rs-recording-rename-video`.**
-No PR per instructions.
-
----
-
-## Phase 2 — zoom-on-click renderer (later, after the user noticed it
-was missing)
-
-The user looked at the captured mp4 and immediately spotted "I don't
-see any zoom in/out effect like we had in swift." Right — Swift had a
-two-stage pipeline (capture + render) and I'd only ported the capture
-half. Phase 2 ports the render.
-
-### Why mcp-server (not cua-driver) — quick layering note
-
-Mirrors the Swift layout 1:1:
-- `mcp-server` = the **library** (math, loader, renderer, types)
-- `cua-driver` = the **binary** (CLI subcommand wiring, daemon)
-
-Three reasons the renderer lives in the library:
-1. It reads what the `RecordingSession` writes — refactors stay in one
- place.
-2. A future MCP tool (`render_recording`) can call it without a new dep.
-3. The CLI stays thin: parse args → call library function → print result.
-
-### What landed
-
-| Concern | Rust | Swift it ports |
-|---|---|---|
-| Zoom math (clamp01, lerp, cubic_bezier, easeOutExpo) + types | `recording_zoom.rs` | `Zoom/ZoomMath.swift` + types |
-| Zoom region generation + curve sampling | `recording_zoom.rs::generate_zoom_regions` / `sample_curve` | `Zoom/ZoomRegion.swift` |
-| Cursor sample lookup (binary-search + lerp) | `recording_zoom.rs::position_at` | `Zoom/CursorTelemetry.swift` |
-| Action span generation + variable-speed PTS remap | `recording_zoom.rs::generate_action_spans` / `map_pts` | `Zoom/ActionSpan.swift` |
-| Cross-platform cursor sampler (30 Hz mouse poll) | `cursor_sampler.rs` | `Recording/CursorSampler.swift` |
-| Trajectory loader (session.json + cursor.jsonl + turn-*/action.json) | `recording_loader.rs` | `Render/TrajectoryLoader.swift` |
-| Renderer (build ffmpeg filter graph, run it) | `recording_render.rs` | `Render/RecordingRenderer.swift` |
-| CLI subcommand | `cua-driver/src/cli.rs::run_recording_render` | `RecordingRenderCommand.swift` |
-
-9 unit tests in `recording_zoom` cover the math (cubic bezier endpoint
-behavior, click → region mapping, chained-merge logic, span PTS
-monotonicity, cursor lerp at endpoints + midpoint).
-
-### Pivotal design choice: sendcmd file instead of giant expression
-
-The "obvious" approach is to fold the whole zoom timeline into one big
-`crop=…:if(between(t,t1,t2),…)` filter expression. ffmpeg's parser has
-practical depth limits and debugging a stringified curve is a nightmare.
-
-Cleaner: pre-compute per-frame `(scale, focus_x, focus_y)` in Rust,
-write timed `crop@c w X; crop@c h Y; crop@c x …; crop@c y …;` updates
-into a sendcmd file at 30 Hz (matching capture framerate), and let
-ffmpeg apply them in sequence. The sendcmd file lands as
-`<input-dir>/render.sendcmd` for verification.
-
-### Cross-platform cursor sampler
-
-`cursor_sampler.rs` runs a background thread at 30 Hz polling the OS
-mouse position, writes `{t_ms, x, y}` JSON lines to `cursor.jsonl`.
-Per-platform polls:
-- **Windows:** `GetCursorPos` via the `windows` crate
-- **macOS:** `CGEventCreate` + `CGEventGetLocation` via inline extern "C"
-- **Linux X11:** stub (returns None) — needs the `x11` crate, deferred
-- **Linux Wayland:** no portable API exists; renderer falls back to
- click-point-only zoom (no smooth cursor pan between actions)
-
-### Coordinate recovery for element-indexed clicks
-
-A gotcha: the current Rust `RecordingSession` doesn't write `click_point`
-for element-indexed clicks — only for pixel-clicks. That made the first
-render produce zero zoom regions (clicks looked coord-less to the
-loader). The Swift impl writes click_point for both paths; that's a
-follow-up for the recording session.
-
-Quick fix: parse the screen coords out of `result_summary` — every UIA
-Invoke surfaces `"...(screen (X,Y))..."` in the result text, which the
-loader's `parse_screen_from_summary` peels back to numbers. Functional
-parity until the recorder is updated to write click_point directly.
-
-### Verified end-to-end
-
-Fresh capture: 5 clicks (launch + 4 calc buttons), 621 cursor samples
-in cursor.jsonl over 20.8 s.
-
-Render: `cua-driver recording render <dir> <out.mp4>` →
-```
-✅ Wrote &lt;output-dir&gt;/calc-demo-rendered.mp4
- input_duration_ms: 20667
- zoom_region_count: 2
-```
-
-(4 clicks → 2 chained regions because adjacent clicks within 1.5 s
-merge into pan-between-foci regions; matches Swift behavior.)
-
-Visual verification by frame extraction at t=1s (no zoom — full
-desktop) and t=9s (peak zoom — calc's number row enlarged, agent
-cursor visible on the "5" button about to be clicked). The
-`render.sendcmd` file shows the crop window collapsing from 1512×950
-at t=0 down to ~784×492 around the click moments (≈2× zoom centered
-on click point), then easing back.
-
-### Now done
-
-- ✅ Phase 1 (capture): tool rename, default-on video, ffmpeg subprocess
-- ✅ Phase 2 (render): cursor sampler, loader, zoom math, sendcmd-driven
- renderer, CLI subcommand
-- ✅ Cross-platform by construction: only OS deps are
- - the `windows` crate for Windows mouse poll (already used elsewhere)
- - extern "C" CGEventCreate/CGEventGetLocation on macOS
- - ffmpeg subprocess (cross-platform binary)
-
-### Still not done
-
-- `RecordingSession` should write `click_point` for element-indexed
- clicks (currently only pixel-clicks get it). Workaround: loader
- parses screen coords out of `result_summary`. Real fix is a small
- edit in `recording.rs::write_turn`.
-- Linux X11 cursor polling stub — needs `x11` crate. Renderer
- degrades gracefully to no cursor data when stub returns nothing.
-- Linux Wayland cursor polling — no portable API. Hard requirement to
- ship Wayland cursor data is "use libei" which is much bigger work.
-- macOS / Linux end-to-end render not validated from this Windows host.
- Math is cross-platform pure-Rust so it should work; ffmpeg subprocess
- args are platform-keyed and validated for input device only (not
- output, which is the same everywhere).
-- The 4-click → 2-region merge is correct per the Swift algorithm but
- could surprise users who expect 1 region per click. Documenting in
- the CLI help would be a small follow-up.
-
-