A local video studio for your coding agent. Describe a video in one sentence: your agent directs, your machine renders. Claude Code, Codex, Cursor, Devin and more. No API keys, no uploads.
Round 4: showtime 0.3.0
Estimate (from round 3's own numbers, API-equivalent; the runs use the subscription token like round 3).
13 new agent runs: about $37 if 0.3.0 costs what 0.2.0 did per task (round-3 cells of record), about
$20 if it meets its cost gate (median at or under plain Opus, ~$1.3). Judging: ranking judges about
$10 (24 judgments, $0.125 per candidate shown), fact checks about $2.40 (new cells only). Total
$32-49. Wall time: the runs take about 166 min one after another, about 65-75 min at -j 3; scoring
adds about 60-90 min (automatic metrics with local speech recognition on every cell, fact checks, 24 ranking
judgments three at a time, 8 blind boards). About 2.5-3 hours end to end, then the owner's vote.
round.py plan prints the same estimate from the reuse source on the box.
Manifest: benchmarks/rounds/r4.json. Driver: benchmarks/harness/round.py. Report additions:
benchmarks/scoring/round_report.py.
What is compared
| Cell | What | Tasks | Made how |
|---|---|---|---|
showtime |
showtime 0.3.0 (the merged release candidate) | t1 t2 t3 t4 t6 t8 t9 t10 | new runs |
showtime-rep2 |
a second 0.3.0 run, to see the spread | t1 t2 t8 | new runs; ranked by the judges, kept off the owner's boards |
showtime-020 |
showtime 0.2.0 | t1 t2 t3 t4 t6 t8 | reused: round 3's cells of record (r3-latest, the ones the owner voted on) |
baseline |
plain Claude Code, same model and settings | all eight | reused from round 3 (made in round 1; the prompts are unchanged, checked); new runs on t9 and t10 |
skill-b |
the launch task's competitor (the single-file skill that won t1 three times) | t1 | reused (round 1 output, also in round 3) |
Round 3's other competitor cells (one per task, made in round 1) could be added to the manifest at judge cost only; they are left out so each board stays at 3-4 versions and the owner's vote stays short.
New tasks (same format as t1-t8; each file says what the judges look for):
- t9 repo explainer (
tasks/t9-repo-explainer.md): "Make a 60-90 second video that explains what this repo does and how it works, for developers seeing it for the first time." Fixture:fixtures/keelson/, a small working CLI written for the benchmark (README, CHANGELOG, four modules, tests, example env files). - t10 make it like this reference (
tasks/t10-like-reference.md): "Make a 20-second video announcing our spring plant swap (details in swap.md) in the same style as reference.mp4." The reference is drawn byfixtures/make_reference.pywith ffmpeg only (flat colour, one OFL typeface, synthesised beat): original, no third-party footage. The judges get stills of the reference in their packet (reference/) and the owner sees it on the board as "Reference: the style to match (not a candidate)".
Isolation (the round-3 lesson)
Round 3 ran t2 and t3 on another branch's code: the plugin arm's runtime home was a link to the shared
~/.showtime, whose skill-path another worker's launcher had rewritten. For round 4:
arms.py setup --arms baseline,showtime --forcefrom the checkout being measured. The plugin arm gets a frozen snapshot of the checkout (minusbenchmarks/) and its own runtime-home overlay: every entry of the shared home linked in exceptbin/andskill-path, which point at the snapshot (a version tie keeps the recorded skill unless an installed one runs, so a same-version checkout elsewhere cannot win).round.py planrefuses to start when the plugin arm's home is not such an overlay or itsskill-pathpoints outside the snapshot, and warns when the snapshot differs from the checkout'sskills/.- Each new cell: fresh workspace under
/srv/vbench-w(no.claude/orCLAUDE.mdabove it), freshCLAUDE_CONFIG_DIR, per-armHOME. - After the runs,
round_report.pylists every showtime folder a run's commands or file reads touched outside its own snapshot and overlay (the check that would have caught round 3'scat <operator home>/w/docs/skills/showtime/...). Any hit fails the isolation line of the report.
Launch (coordinator, on the box, after the final v0.3.0 is merged)
# a checkout of the merged v0.3.0 on the box (the arm snapshots it; benchmarks/ comes from it too)
git clone -q ~/hub/showtime.git ~/w/r4 && git -C ~/w/r4 checkout -q v0.3.0
~/bench-r4.sh ~/w/r4 # checks, arm setup + audit, plan, then round.py all in tmux "r4"
tail -f ~/logs/bench-r4.log # done when ~/vbench-r4/.r4-done exists
~/bench-r4.sh (on the box, outside the repository) does, with round 3's environment (token file, PATH dirs,
workspace root, Claude binary) and its own bench home ~/vbench-r4:
python3 $B/fixtures/fetch_fixtures.py # interview.mp4 + the drawn style reference
python3 $B/harness/arms.py setup --arms baseline,showtime --force
python3 $B/harness/arms.py audit --arms showtime # baseline loads no video skill; showtime loads its own
python3 $B/harness/round.py plan $B/rounds/r4.json # stops on any problem
python3 $B/harness/round.py all $B/rounds/r4.json # reuse, 13 runs at -j 3, score, report
Outputs in ~/vbench-r4/runs/r4/: report.md (with the release gates and the stream metrics appended),
results.json, stream_metrics.json, gates.json, rank.jsonl; boards in ~/vbench-r4/human/r4/*.html
with the letter key in key.json (never send it).
The owner's blind vote
Each board is one HTML page (8 boards; t1 has 4 versions, t9 has 2, t10 has 2 plus the reference, the others 3). Publish each as a private artifact and send the links. At the top of every board: How to vote (watch, rate 0-5 and answer the A-or-B questions, then press Copy my votes and paste the text in the chat); the same button sits in the bar at the bottom of the phone screen. Letters are shuffled per task; nothing names a tool (the build fails if an arm id, arm label or output path appears in the page or its media).
Import each pasted digest and recompute the gates:
python3 $B/scoring/human_board.py import --run r4 --task t1 --file vote-t1.txt # one per task
python3 $B/harness/round.py report $B/rounds/r4.json # tally, aggregate, gates
Release gates (from work/v030/PLAN.md), as the report prints them
Each is PASS, FAIL or PENDING (PENDING = waiting for the vote or for data).
- cost: median cost per video of
showtime<= median ofbaselineon the same eight tasks. - images: images read into the main context, median <= 12 per job (
showtimecells). - renders: full renders per job, median 1 (from each job's
receipt.json, elsejob.json, else an estimate from the stream's render commands, marked*). - quality: on at least 5 of the 6 round-3 tasks, the ranking judge ranks
showtimeat or aboveshowtime-020AND the owner's blind vote picks it (or ties) in their A/B question (their ratings when the pair was not asked). - qa:
showtime qaFAIL findings = 0 on every new 0.3.0 run (both reps). - launch: on t1,
showtimeis not last, neither by the judges' mean rank nor by the owner's rating (its own second run excluded from the field). - receipts: every new 0.3.0 run left a
receipt.json(and how many have a cost). - isolation: no run touched another showtime folder.
Stream metrics per cell (every cell, reused ones with the numbers of their own round): cost, wall, turns, model calls, tokens in and out, images in the main context and in sub-agents, largest main context, tool text characters, full renders, receipt; plus a spread table for the three tasks run twice.
Known limits to state with the results
- One run per cell (two on t1, t2, t8). Treat single-task differences as anecdotes.
- Reused cells were made on earlier days under other machine load; wall times are not comparable across rounds.
- Reused cells are re-scored with this checkout's
showtime qa; their fact checks are kept from round 3 (same judge protocol with proof of looking). Their deliverable is the one their round picked; new cells use one extra tie-break (a video as long as the task asked beats a newer partial re-render), found in the paid smoke, where an 18 sfinal-2.mp4span render would otherwise have been judged instead of the 82 s video. - The owner has seen earlier rounds' videos, so the vote is partly blind; letters are shuffled anyway.
- t10's reference and t9's repo were written for this benchmark; showtime 0.3.0 has a reference and a repo explainer workflow that plain Opus does not, which is the point of the tasks, not a flaw in them.