Skip to content

A local video studio for your coding agent. Describe a video in one sentence: your agent directs, your machine renders. Claude Code, Codex, Cursor, Devin and more. No API keys, no uploads.

8.5 KB
Raw Blame

Round 4: showtime 0.3.0

Estimate (from round 3's own numbers, API-equivalent; the runs use the subscription token like round 3). 13 new agent runs: about $37 if 0.3.0 costs what 0.2.0 did per task (round-3 cells of record), about $20 if it meets its cost gate (median at or under plain Opus, ~$1.3). Judging: ranking judges about $10 (24 judgments, $0.125 per candidate shown), fact checks about $2.40 (new cells only). Total $32-49. Wall time: the runs take about 166 min one after another, about 65-75 min at -j 3; scoring adds about 60-90 min (automatic metrics with local speech recognition on every cell, fact checks, 24 ranking judgments three at a time, 8 blind boards). About 2.5-3 hours end to end, then the owner's vote. round.py plan prints the same estimate from the reuse source on the box.

Manifest: benchmarks/rounds/r4.json. Driver: benchmarks/harness/round.py. Report additions: benchmarks/scoring/round_report.py.

What is compared

Cell What Tasks Made how
showtime showtime 0.3.0 (the merged release candidate) t1 t2 t3 t4 t6 t8 t9 t10 new runs
showtime-rep2 a second 0.3.0 run, to see the spread t1 t2 t8 new runs; ranked by the judges, kept off the owner's boards
showtime-020 showtime 0.2.0 t1 t2 t3 t4 t6 t8 reused: round 3's cells of record (r3-latest, the ones the owner voted on)
baseline plain Claude Code, same model and settings all eight reused from round 3 (made in round 1; the prompts are unchanged, checked); new runs on t9 and t10
skill-b the launch task's competitor (the single-file skill that won t1 three times) t1 reused (round 1 output, also in round 3)

Round 3's other competitor cells (one per task, made in round 1) could be added to the manifest at judge cost only; they are left out so each board stays at 3-4 versions and the owner's vote stays short.

New tasks (same format as t1-t8; each file says what the judges look for):

  • t9 repo explainer (tasks/t9-repo-explainer.md): "Make a 60-90 second video that explains what this repo does and how it works, for developers seeing it for the first time." Fixture: fixtures/keelson/, a small working CLI written for the benchmark (README, CHANGELOG, four modules, tests, example env files).
  • t10 make it like this reference (tasks/t10-like-reference.md): "Make a 20-second video announcing our spring plant swap (details in swap.md) in the same style as reference.mp4." The reference is drawn by fixtures/make_reference.py with ffmpeg only (flat colour, one OFL typeface, synthesised beat): original, no third-party footage. The judges get stills of the reference in their packet (reference/) and the owner sees it on the board as "Reference: the style to match (not a candidate)".

Isolation (the round-3 lesson)

Round 3 ran t2 and t3 on another branch's code: the plugin arm's runtime home was a link to the shared ~/.showtime, whose skill-path another worker's launcher had rewritten. For round 4:

  • arms.py setup --arms baseline,showtime --force from the checkout being measured. The plugin arm gets a frozen snapshot of the checkout (minus benchmarks/) and its own runtime-home overlay: every entry of the shared home linked in except bin/ and skill-path, which point at the snapshot (a version tie keeps the recorded skill unless an installed one runs, so a same-version checkout elsewhere cannot win).
  • round.py plan refuses to start when the plugin arm's home is not such an overlay or its skill-path points outside the snapshot, and warns when the snapshot differs from the checkout's skills/.
  • Each new cell: fresh workspace under /srv/vbench-w (no .claude/ or CLAUDE.md above it), fresh CLAUDE_CONFIG_DIR, per-arm HOME.
  • After the runs, round_report.py lists every showtime folder a run's commands or file reads touched outside its own snapshot and overlay (the check that would have caught round 3's cat <operator home>/w/docs/skills/showtime/...). Any hit fails the isolation line of the report.

Launch (coordinator, on the box, after the final v0.3.0 is merged)

# a checkout of the merged v0.3.0 on the box (the arm snapshots it; benchmarks/ comes from it too)
git clone -q ~/hub/showtime.git ~/w/r4 && git -C ~/w/r4 checkout -q v0.3.0
~/bench-r4.sh ~/w/r4          # checks, arm setup + audit, plan, then round.py all in tmux "r4"
tail -f ~/logs/bench-r4.log   # done when ~/vbench-r4/.r4-done exists

~/bench-r4.sh (on the box, outside the repository) does, with round 3's environment (token file, PATH dirs, workspace root, Claude binary) and its own bench home ~/vbench-r4:

python3 $B/fixtures/fetch_fixtures.py                       # interview.mp4 + the drawn style reference
python3 $B/harness/arms.py setup --arms baseline,showtime --force
python3 $B/harness/arms.py audit --arms showtime            # baseline loads no video skill; showtime loads its own
python3 $B/harness/round.py plan $B/rounds/r4.json          # stops on any problem
python3 $B/harness/round.py all  $B/rounds/r4.json          # reuse, 13 runs at -j 3, score, report

Outputs in ~/vbench-r4/runs/r4/: report.md (with the release gates and the stream metrics appended), results.json, stream_metrics.json, gates.json, rank.jsonl; boards in ~/vbench-r4/human/r4/*.html with the letter key in key.json (never send it).

The owner's blind vote

Each board is one HTML page (8 boards; t1 has 4 versions, t9 has 2, t10 has 2 plus the reference, the others 3). Publish each as a private artifact and send the links. At the top of every board: How to vote (watch, rate 0-5 and answer the A-or-B questions, then press Copy my votes and paste the text in the chat); the same button sits in the bar at the bottom of the phone screen. Letters are shuffled per task; nothing names a tool (the build fails if an arm id, arm label or output path appears in the page or its media).

Import each pasted digest and recompute the gates:

python3 $B/scoring/human_board.py import --run r4 --task t1 --file vote-t1.txt   # one per task
python3 $B/harness/round.py report $B/rounds/r4.json                            # tally, aggregate, gates

Release gates (from work/v030/PLAN.md), as the report prints them

Each is PASS, FAIL or PENDING (PENDING = waiting for the vote or for data).

  1. cost: median cost per video of showtime <= median of baseline on the same eight tasks.
  2. images: images read into the main context, median <= 12 per job (showtime cells).
  3. renders: full renders per job, median 1 (from each job's receipt.json, else job.json, else an estimate from the stream's render commands, marked *).
  4. quality: on at least 5 of the 6 round-3 tasks, the ranking judge ranks showtime at or above showtime-020 AND the owner's blind vote picks it (or ties) in their A/B question (their ratings when the pair was not asked).
  5. qa: showtime qa FAIL findings = 0 on every new 0.3.0 run (both reps).
  6. launch: on t1, showtime is not last, neither by the judges' mean rank nor by the owner's rating (its own second run excluded from the field).
  7. receipts: every new 0.3.0 run left a receipt.json (and how many have a cost).
  8. isolation: no run touched another showtime folder.

Stream metrics per cell (every cell, reused ones with the numbers of their own round): cost, wall, turns, model calls, tokens in and out, images in the main context and in sub-agents, largest main context, tool text characters, full renders, receipt; plus a spread table for the three tasks run twice.

Known limits to state with the results

  • One run per cell (two on t1, t2, t8). Treat single-task differences as anecdotes.
  • Reused cells were made on earlier days under other machine load; wall times are not comparable across rounds.
  • Reused cells are re-scored with this checkout's showtime qa; their fact checks are kept from round 3 (same judge protocol with proof of looking). Their deliverable is the one their round picked; new cells use one extra tie-break (a video as long as the task asked beats a newer partial re-render), found in the paid smoke, where an 18 s final-2.mp4 span render would otherwise have been judged instead of the 82 s video.
  • The owner has seen earlier rounds' videos, so the vote is partly blind; letters are shuffled anyway.
  • t10's reference and t9's repo were written for this benchmark; showtime 0.3.0 has a reference and a repo explainer workflow that plain Opus does not, which is the point of the tasks, not a flaw in them.