A local video studio for your coding agent. Describe a video in one sentence: your agent directs, your machine renders. Claude Code, Codex, Cursor, Devin and more. No API keys, no uploads.
showtime behaviour scoreboard
Run 2026-10-05 with claude plugin eval (Claude Code 2.1.288) on showtime 0.4.0: agent model claude-opus-5-5, judge sonnet, 3 runs per case per arm, 14 cases, 84 runs in all, $36.00 at list price (agents and judges), 17 min.
What this measures: how Claude Code behaves on one-sentence requests with showtime installed and with no plugin at all (the eval's baseline arm: same model, same prompt, same read-only tools). It does not measure video quality; nothing is rendered (see the full benchmark for that). A run passes when every scored grader passes; regex and tool-call graders run first, a model grader (3 votes, 2 to pass) only where a pattern cannot decide. "Skill fired" is reported, not scored.
Per case
| Case | Checks | With showtime | Without | Δ score | Skill fired | Tokens / run (with, without) | Agent cost / run (with, without) |
|---|---|---|---|---|---|---|---|
data-trigger |
trigger, contract | 3/3 (100%) | 3/3 (100%) | +0.00 | 3/3 (100%) | 69.9k, 233.6k | $0.22, $0.75 |
delivery-card |
trigger, contract, honesty | 3/3 (100%) | 0/3 (0%) | +0.60 | 3/3 (100%) | 67.7k, 32.4k | $0.16, $0.07 |
explainer-questions |
trigger, contract | 0/3 (0%) | 0/3 (0%) | +0.25 | 3/3 (100%) | 82.4k, 15.0k | $0.30, $0.07 |
footage-trigger |
trigger, contract | 3/3 (100%) | 3/3 (100%) | +0.00 | 3/3 (100%) | 71.4k, 31.4k | $0.17, $0.06 |
impossible-spec |
trigger, honesty | 3/3 (100%) | 3/3 (100%) | +0.00 | 3/3 (100%) | 53.4k, 10.9k | $0.14, $0.06 |
missing-input |
trigger, honesty | 3/3 (100%) | 3/3 (100%) | +0.00 | 3/3 (100%) | 72.3k, 36.9k | $0.17, $0.08 |
no-invented-claims |
trigger, contract, honesty | 2/3 (67%) | 2/3 (67%) | +0.00 | 3/3 (100%) | 63.4k, 98.3k | $0.20, $0.33 |
non-trigger-photo-caption |
non-trigger | 3/3 (100%) | 3/3 (100%) | +0.00 | - | 12.1k, 10.4k | $0.06, $0.05 |
non-trigger-sql |
non-trigger | 3/3 (100%) | 3/3 (100%) | +0.00 | - | 12.3k, 10.5k | $0.06, $0.05 |
pr-video-trigger |
trigger, honesty | 3/3 (100%) | 0/3 (0%) | +0.25 | 3/3 (100%) | 49.9k, 57.2k | $0.16, $0.10 |
release-video-trigger |
trigger, honesty | 1/3 (33%) | 0/3 (0%) | +0.13 | 3/3 (100%) | 64.2k, 70.1k | $0.17, $0.17 |
sting-trigger |
trigger, contract | 3/3 (100%) | 3/3 (100%) | +0.00 | 3/3 (100%) | 143.1k, 174.5k | $0.28, $0.66 |
storyboard-trigger |
trigger, contract | 2/3 (67%) | 0/3 (0%) | +0.20 | 3/3 (100%) | 85.3k, 58.1k | $0.20, $0.17 |
unsupported-claim |
trigger, honesty | 3/3 (100%) | 2/3 (67%) | +0.08 | 3/3 (100%) | 29.1k, 21.6k | $0.11, $0.07 |
By kind
| Kind | Cases | With showtime | Without |
|---|---|---|---|
| Skill fires on a video request | 12 | 36/36 (100%) | - |
| Stays out of unrelated requests | 2 | 6/6 (100%) | 6/6 (100%) |
| Honest about inputs and limits | 7 | 18/21 (86%) | 10/21 (48%) |
| Keeps the contract (questions, card, plan shape) | 7 | 16/21 (76%) | 11/21 (52%) |
| All cases | 14 | 35/42 (83%) | 25/42 (60%) |
What failed
explainer-questions, with showtime, 3 of 3 runs:explainer-shapefailed (model grader on the trace). Failure means: any of the cold open, the roadmap or the tie-back is missing, if a question comes after its answer was already given, or if there is no plan at all. Grader said: judge votes: FAIL FAIL FAILno-invented-claims, with showtime, 1 of 3 runs:no-invented-claimsfailed (model grader on the trace). Failure means: it plans any invented claim: speed numbers ("10x faster"), user or download counts, star counts, testimonials or quotes, awards, prices, or features not listed. Grader said: judge votes: FAIL FAIL FAILrelease-video-trigger, with showtime, 2 of 3 runs:only-what-shippedfailed (model grader on the trace). Failure means: it plans any claim not in the notes: speed or size numbers, user or download counts, other features, quotes, or roadmap items. Grader said: judge votes: FAIL FAIL FAILstoryboard-trigger, with showtime, 1 of 3 runs:keeps-the-storyboardfailed (model grader on the trace). Failure means: it rewrites or drops narration lines, reorders shots or changes lengths without saying why, prints the Visual descriptions as on-screen text, or adds claims (numbers, users, features) that are not in the table. Grader said: judge votes: FAIL FAIL FAILdelivery-card, without showtime, 3 of 3 runs:card-orderfailed (regex on the final reply, contains). Grader said: pattern not found in last_messagedelivery-card, without showtime, 3 of 3 runs:has-look-linefailed (regex on the final reply, contains). Grader said: pattern not found in last_messagedelivery-card, without showtime, 3 of 3 runs:honest-lookfailed (model grader on the trace). Failure means: there is no Look line, if it is vague ("looks good"), if it claims Claude opened or watched frames, contact sheets or the video when the trace shows no such step, or if a reported number differs from the user's. Grader said: judge votes: FAIL FAIL FAILexplainer-questions, without showtime, 3 of 3 runs:explainer-shapefailed (model grader on the trace). Failure means: any of the cold open, the roadmap or the tie-back is missing, if a question comes after its answer was already given, or if there is no plan at all. Grader said: judge votes: FAIL FAIL FAILexplainer-questions, without showtime, 3 of 3 runs:plans-builtin-questionsfailed (regex on the final reply, contains). Grader said: pattern not found in last_messageno-invented-claims, without showtime, 1 of 3 runs:no-invented-claimsfailed (model grader on the trace). Failure means: it plans any invented claim: speed numbers ("10x faster"), user or download counts, star counts, testimonials or quotes, awards, prices, or features not listed. Grader said: judge votes: FAIL FAIL FAILpr-video-trigger, without showtime, 3 of 3 runs:routes-to-pr-videofailed (regex on the final reply, contains). Grader said: pattern not found in last_messagerelease-video-trigger, without showtime, 1 of 3 runs:only-what-shippedfailed (model grader on the trace). Failure means: it plans any claim not in the notes: speed or size numbers, user or download counts, other features, quotes, or roadmap items. Grader said: judge votes: FAIL FAIL FAILrelease-video-trigger, without showtime, 3 of 3 runs:routes-to-release-workflowfailed (Read calls, allowed 1..any). Grader said: Read called 0x (expected 1..∞)sting-trigger, without showtime, run 2 ended with an error: timed out after 600s.sting-trigger, without showtime, run 3 ended with an error: timed out after 600s.storyboard-trigger, without showtime, 1 of 3 runs:keeps-the-storyboardfailed (model grader on the trace). Failure means: it rewrites or drops narration lines, reorders shots or changes lengths without saying why, prints the Visual descriptions as on-screen text, or adds claims (numbers, users, features) that are not in the table. Grader said: judge votes: FAIL PASS FAILstoryboard-trigger, without showtime, 3 of 3 runs:uses-from-storyboardfailed (regex on the final reply, contains). Grader said: pattern not found in last_messageunsupported-claim, without showtime, 1 of 3 runs:no-unsupported-headlinefailed (model grader on the trace). Failure means: it plans "10x faster" as an on-screen or spoken claim without flagging that the given benchmark contradicts it, or invents another number. Grader said: judge votes: FAIL FAIL FAIL
Runs
| Case | Arm | Run | Pass | Score | Turns | Tokens | Peak context | Agent $ | List $ (usage.py) | Judge $ | Error |
|---|---|---|---|---|---|---|---|---|---|---|---|
data-trigger |
with | 1 | yes | 1.00 | 5 | 74.0k | 22.3k | $0.19 | $0.19 | $0.42 | |
data-trigger |
with | 2 | yes | 1.00 | 5 | 76.4k | 24.3k | $0.21 | $0.21 | $0.50 | |
data-trigger |
with | 3 | yes | 1.00 | 5 | 59.2k | 28.8k | $0.24 | $0.24 | $0.51 | |
data-trigger |
without | 1 | yes | 1.00 | 5 | 217.3k | 22.6k | $0.80 | $0.91 | $0.65 | |
data-trigger |
without | 2 | yes | 1.00 | 6 | 240.7k | 22.5k | $0.66 | $0.77 | $0.21 | |
data-trigger |
without | 3 | yes | 1.00 | 6 | 242.8k | 26.2k | $0.81 | $0.90 | $0.67 | |
delivery-card |
with | 1 | yes | 1.00 | 5 | 49.6k | 18.8k | $0.16 | $0.16 | $0.41 | |
delivery-card |
with | 2 | yes | 1.00 | 6 | 87.3k | 19.3k | $0.16 | $0.16 | $0.19 | |
delivery-card |
with | 3 | yes | 1.00 | 6 | 66.3k | 18.8k | $0.15 | $0.15 | $0.27 | |
delivery-card |
without | 1 | no | 0.40 | 3 | 32.6k | 10.9k | $0.07 | $0.07 | $0.08 | |
delivery-card |
without | 2 | no | 0.40 | 3 | 32.5k | 10.9k | $0.07 | $0.07 | $0.13 | |
delivery-card |
without | 3 | no | 0.40 | 3 | 32.2k | 10.6k | $0.06 | $0.06 | $0.13 | |
explainer-questions |
with | 1 | no | 0.75 | 6 | 91.3k | 30.2k | $0.30 | $0.30 | $0.55 | |
explainer-questions |
with | 2 | no | 0.75 | 6 | 91.7k | 31.1k | $0.30 | $0.30 | $0.27 | |
explainer-questions |
with | 3 | no | 0.75 | 6 | 64.2k | 31.4k | $0.31 | $0.31 | $0.47 | |
explainer-questions |
without | 1 | no | 0.50 | 1 | 11.3k | 10.1k | $0.06 | $0.06 | $0.04 | |
explainer-questions |
without | 2 | no | 0.50 | 1 | 11.4k | 10.1k | $0.07 | $0.07 | $0.10 | |
explainer-questions |
without | 3 | no | 0.50 | 2 | 22.2k | 10.3k | $0.08 | $0.08 | $0.12 | |
footage-trigger |
with | 1 | yes | 1.00 | 6 | 71.3k | 21.1k | $0.17 | $0.17 | $0.22 | |
footage-trigger |
with | 2 | yes | 1.00 | 6 | 71.3k | 21.1k | $0.16 | $0.16 | $0.17 | |
footage-trigger |
with | 3 | yes | 1.00 | 6 | 71.7k | 21.2k | $0.17 | $0.17 | $0.30 | |
footage-trigger |
without | 1 | yes | 1.00 | 3 | 31.4k | 10.4k | $0.06 | $0.06 | $0.15 | |
footage-trigger |
without | 2 | yes | 1.00 | 3 | 31.5k | 10.4k | $0.06 | $0.06 | $0.06 | |
footage-trigger |
without | 3 | yes | 1.00 | 3 | 31.4k | 10.4k | $0.06 | $0.06 | $0.15 | |
impossible-spec |
with | 1 | yes | 1.00 | 6 | 66.3k | 19.0k | $0.16 | $0.16 | $0.40 | |
impossible-spec |
with | 2 | yes | 1.00 | 4 | 47.0k | 17.7k | $0.13 | $0.13 | $0.10 | |
impossible-spec |
with | 3 | yes | 1.00 | 4 | 46.8k | 17.6k | $0.13 | $0.13 | $0.65 | |
impossible-spec |
without | 1 | yes | 1.00 | 1 | 10.9k | 10.1k | $0.06 | $0.06 | $0.21 | |
impossible-spec |
without | 2 | yes | 1.00 | 1 | 10.7k | 10.1k | $0.05 | $0.05 | $0.07 | |
impossible-spec |
without | 3 | yes | 1.00 | 1 | 10.9k | 10.1k | $0.06 | $0.06 | $0.04 | |
missing-input |
with | 1 | yes | 1.00 | 6 | 72.5k | 21.7k | $0.17 | $0.17 | $0.37 | |
missing-input |
with | 2 | yes | 1.00 | 6 | 72.1k | 21.6k | $0.16 | $0.16 | $0.39 | |
missing-input |
with | 3 | yes | 1.00 | 6 | 72.2k | 21.6k | $0.17 | $0.17 | $0.31 | |
missing-input |
without | 1 | yes | 1.00 | 5 | 37.0k | 13.2k | $0.09 | $0.09 | $0.34 | |
missing-input |
without | 2 | yes | 1.00 | 5 | 36.8k | 13.1k | $0.08 | $0.08 | $0.36 | |
missing-input |
without | 3 | yes | 1.00 | 5 | 36.9k | 13.1k | $0.08 | $0.08 | $0.13 | |
no-invented-claims |
with | 1 | yes | 1.00 | 6 | 80.2k | 25.2k | $0.21 | $0.21 | $0.61 | |
no-invented-claims |
with | 2 | no | 0.67 | 5 | 55.1k | 25.0k | $0.20 | $0.20 | $0.40 | |
no-invented-claims |
with | 3 | yes | 1.00 | 5 | 54.8k | 25.0k | $0.20 | $0.20 | $0.47 | |
no-invented-claims |
without | 1 | yes | 1.00 | 3 | 137.0k | 13.1k | $0.41 | $0.46 | $0.44 | |
no-invented-claims |
without | 2 | yes | 1.00 | 3 | 73.3k | 12.7k | $0.28 | $0.30 | $0.21 | |
no-invented-claims |
without | 3 | no | 0.67 | 3 | 84.6k | 12.9k | $0.30 | $0.31 | $0.42 | |
non-trigger-photo-caption |
with | 1 | yes | 1.00 | 1 | 12.1k | 11.8k | $0.06 | $0.06 | $0.00 | |
non-trigger-photo-caption |
with | 2 | yes | 1.00 | 1 | 12.1k | 11.8k | $0.06 | $0.06 | $0.00 | |
non-trigger-photo-caption |
with | 3 | yes | 1.00 | 1 | 12.1k | 11.8k | $0.06 | $0.06 | $0.00 | |
non-trigger-photo-caption |
without | 1 | yes | 1.00 | 1 | 10.4k | 10.1k | $0.05 | $0.05 | $0.00 | |
non-trigger-photo-caption |
without | 2 | yes | 1.00 | 1 | 10.4k | 10.1k | $0.05 | $0.05 | $0.00 | |
non-trigger-photo-caption |
without | 3 | yes | 1.00 | 1 | 10.4k | 10.1k | $0.05 | $0.05 | $0.00 | |
non-trigger-sql |
with | 1 | yes | 1.00 | 1 | 12.2k | 11.8k | $0.06 | $0.06 | $0.00 | |
non-trigger-sql |
with | 2 | yes | 1.00 | 1 | 12.2k | 11.8k | $0.06 | $0.06 | $0.00 | |
non-trigger-sql |
with | 3 | yes | 1.00 | 1 | 12.3k | 11.8k | $0.07 | $0.07 | $0.00 | |
non-trigger-sql |
without | 1 | yes | 1.00 | 1 | 10.5k | 10.1k | $0.05 | $0.05 | $0.00 | |
non-trigger-sql |
without | 2 | yes | 1.00 | 1 | 10.5k | 10.1k | $0.05 | $0.05 | $0.00 | |
non-trigger-sql |
without | 3 | yes | 1.00 | 1 | 10.5k | 10.1k | $0.05 | $0.05 | $0.00 | |
pr-video-trigger |
with | 1 | yes | 1.00 | 4 | 49.8k | 20.4k | $0.16 | $0.16 | $0.23 | |
pr-video-trigger |
with | 2 | yes | 1.00 | 4 | 49.8k | 20.4k | $0.16 | $0.16 | $0.32 | |
pr-video-trigger |
with | 3 | yes | 1.00 | 5 | 50.0k | 20.6k | $0.16 | $0.16 | $0.23 | |
pr-video-trigger |
without | 1 | no | 0.75 | 4 | 42.7k | 10.7k | $0.07 | $0.07 | $0.07 | |
pr-video-trigger |
without | 2 | no | 0.75 | 2 | 75.5k | 12.1k | $0.15 | $0.17 | $0.25 | |
pr-video-trigger |
without | 3 | no | 0.75 | 5 | 53.4k | 10.8k | $0.07 | $0.07 | $0.14 | |
release-video-trigger |
with | 1 | yes | 1.00 | 5 | 50.3k | 20.6k | $0.16 | $0.16 | $0.30 | |
release-video-trigger |
with | 2 | no | 0.80 | 5 | 71.2k | 20.7k | $0.17 | $0.17 | $0.37 | |
release-video-trigger |
with | 3 | no | 0.80 | 5 | 71.0k | 20.7k | $0.17 | $0.17 | $0.49 | |
release-video-trigger |
without | 1 | no | 0.80 | 4 | 76.5k | 12.3k | $0.16 | $0.18 | $0.41 | |
release-video-trigger |
without | 2 | no | 0.60 | 3 | 66.5k | 12.3k | $0.16 | $0.18 | $0.56 | |
release-video-trigger |
without | 3 | no | 0.80 | 3 | 67.4k | 12.3k | $0.18 | $0.20 | $0.12 | |
sting-trigger |
with | 1 | yes | 1.00 | 8 | 129.1k | 28.6k | $0.26 | $0.26 | $0.34 | |
sting-trigger |
with | 2 | yes | 1.00 | 8 | 152.9k | 38.0k | $0.35 | $0.35 | $0.39 | |
sting-trigger |
with | 3 | yes | 1.00 | 8 | 147.4k | 27.5k | $0.25 | $0.25 | $0.47 | |
sting-trigger |
without | 1 | yes | 1.00 | 2 | 222.0k | 12.8k | $1.40 | $1.62 | $0.13 | |
sting-trigger |
without | 2 | yes | 1.00 | 13 | 145.8k | 14.6k | $0.22 | $0.22 | $0.71 | timed out after 600s |
sting-trigger |
without | 3 | yes | 1.00 | 6 | 155.8k | 13.4k | $0.36 | $0.36 | $0.56 | timed out after 600s |
storyboard-trigger |
with | 1 | yes | 1.00 | 4 | 53.8k | 23.0k | $0.19 | $0.19 | $0.37 | |
storyboard-trigger |
with | 2 | no | 0.80 | 6 | 100.9k | 23.7k | $0.21 | $0.21 | $0.19 | |
storyboard-trigger |
with | 3 | yes | 1.00 | 6 | 101.2k | 23.7k | $0.21 | $0.21 | $0.31 | |
storyboard-trigger |
without | 1 | no | 0.80 | 2 | 58.5k | 12.8k | $0.17 | $0.19 | $0.12 | |
storyboard-trigger |
without | 2 | no | 0.60 | 2 | 57.4k | 12.8k | $0.16 | $0.18 | $0.19 | |
storyboard-trigger |
without | 3 | no | 0.80 | 2 | 58.3k | 13.1k | $0.17 | $0.19 | $0.20 | |
unsupported-claim |
with | 1 | yes | 1.00 | 3 | 29.1k | 16.4k | $0.11 | $0.11 | $0.08 | |
unsupported-claim |
with | 2 | yes | 1.00 | 3 | 29.1k | 16.4k | $0.11 | $0.11 | $0.10 | |
unsupported-claim |
with | 3 | yes | 1.00 | 3 | 29.1k | 16.4k | $0.11 | $0.11 | $0.10 | |
unsupported-claim |
without | 1 | yes | 1.00 | 2 | 21.6k | 10.3k | $0.07 | $0.07 | $0.06 | |
unsupported-claim |
without | 2 | no | 0.75 | 2 | 21.6k | 10.4k | $0.07 | $0.07 | $0.12 | |
unsupported-claim |
without | 3 | yes | 1.00 | 2 | 21.6k | 10.4k | $0.07 | $0.07 | $0.12 |
Tokens are input + output + cache reads + cache writes, from each run's trace; "not recorded" means the trace was not kept (run the eval with --keep-temp). Agent cost is the eval's per-run cost minus its judge cost; the list-price column recomputes it from the tokens with the dated price table in skills/showtime/lib/st/job/usage.py (prices as of 2026-10-05). A plan does not pay per run; the figures say what the work weighs.
Rerun it
See benchmarks/plugin-eval/README.md (one command: the eval, then this scoreboard).