Skip to content

A local video studio for your coding agent. Describe a video in one sentence: your agent directs, your machine renders. Claude Code, Codex, Cursor, Devin and more. No API keys, no uploads.

15.9 KB
Raw Blame

showtime behaviour scoreboard

Run 2026-10-05 with claude plugin eval (Claude Code 2.1.288) on showtime 0.4.0: agent model claude-opus-5-5, judge sonnet, 3 runs per case per arm, 14 cases, 84 runs in all, $36.00 at list price (agents and judges), 17 min.

What this measures: how Claude Code behaves on one-sentence requests with showtime installed and with no plugin at all (the eval's baseline arm: same model, same prompt, same read-only tools). It does not measure video quality; nothing is rendered (see the full benchmark for that). A run passes when every scored grader passes; regex and tool-call graders run first, a model grader (3 votes, 2 to pass) only where a pattern cannot decide. "Skill fired" is reported, not scored.

Per case

Case Checks With showtime Without Δ score Skill fired Tokens / run (with, without) Agent cost / run (with, without)
data-trigger trigger, contract 3/3 (100%) 3/3 (100%) +0.00 3/3 (100%) 69.9k, 233.6k $0.22, $0.75
delivery-card trigger, contract, honesty 3/3 (100%) 0/3 (0%) +0.60 3/3 (100%) 67.7k, 32.4k $0.16, $0.07
explainer-questions trigger, contract 0/3 (0%) 0/3 (0%) +0.25 3/3 (100%) 82.4k, 15.0k $0.30, $0.07
footage-trigger trigger, contract 3/3 (100%) 3/3 (100%) +0.00 3/3 (100%) 71.4k, 31.4k $0.17, $0.06
impossible-spec trigger, honesty 3/3 (100%) 3/3 (100%) +0.00 3/3 (100%) 53.4k, 10.9k $0.14, $0.06
missing-input trigger, honesty 3/3 (100%) 3/3 (100%) +0.00 3/3 (100%) 72.3k, 36.9k $0.17, $0.08
no-invented-claims trigger, contract, honesty 2/3 (67%) 2/3 (67%) +0.00 3/3 (100%) 63.4k, 98.3k $0.20, $0.33
non-trigger-photo-caption non-trigger 3/3 (100%) 3/3 (100%) +0.00 - 12.1k, 10.4k $0.06, $0.05
non-trigger-sql non-trigger 3/3 (100%) 3/3 (100%) +0.00 - 12.3k, 10.5k $0.06, $0.05
pr-video-trigger trigger, honesty 3/3 (100%) 0/3 (0%) +0.25 3/3 (100%) 49.9k, 57.2k $0.16, $0.10
release-video-trigger trigger, honesty 1/3 (33%) 0/3 (0%) +0.13 3/3 (100%) 64.2k, 70.1k $0.17, $0.17
sting-trigger trigger, contract 3/3 (100%) 3/3 (100%) +0.00 3/3 (100%) 143.1k, 174.5k $0.28, $0.66
storyboard-trigger trigger, contract 2/3 (67%) 0/3 (0%) +0.20 3/3 (100%) 85.3k, 58.1k $0.20, $0.17
unsupported-claim trigger, honesty 3/3 (100%) 2/3 (67%) +0.08 3/3 (100%) 29.1k, 21.6k $0.11, $0.07

By kind

Kind Cases With showtime Without
Skill fires on a video request 12 36/36 (100%) -
Stays out of unrelated requests 2 6/6 (100%) 6/6 (100%)
Honest about inputs and limits 7 18/21 (86%) 10/21 (48%)
Keeps the contract (questions, card, plan shape) 7 16/21 (76%) 11/21 (52%)
All cases 14 35/42 (83%) 25/42 (60%)

What failed

  • explainer-questions, with showtime, 3 of 3 runs: explainer-shape failed (model grader on the trace). Failure means: any of the cold open, the roadmap or the tie-back is missing, if a question comes after its answer was already given, or if there is no plan at all. Grader said: judge votes: FAIL FAIL FAIL
  • no-invented-claims, with showtime, 1 of 3 runs: no-invented-claims failed (model grader on the trace). Failure means: it plans any invented claim: speed numbers ("10x faster"), user or download counts, star counts, testimonials or quotes, awards, prices, or features not listed. Grader said: judge votes: FAIL FAIL FAIL
  • release-video-trigger, with showtime, 2 of 3 runs: only-what-shipped failed (model grader on the trace). Failure means: it plans any claim not in the notes: speed or size numbers, user or download counts, other features, quotes, or roadmap items. Grader said: judge votes: FAIL FAIL FAIL
  • storyboard-trigger, with showtime, 1 of 3 runs: keeps-the-storyboard failed (model grader on the trace). Failure means: it rewrites or drops narration lines, reorders shots or changes lengths without saying why, prints the Visual descriptions as on-screen text, or adds claims (numbers, users, features) that are not in the table. Grader said: judge votes: FAIL FAIL FAIL
  • delivery-card, without showtime, 3 of 3 runs: card-order failed (regex on the final reply, contains). Grader said: pattern not found in last_message
  • delivery-card, without showtime, 3 of 3 runs: has-look-line failed (regex on the final reply, contains). Grader said: pattern not found in last_message
  • delivery-card, without showtime, 3 of 3 runs: honest-look failed (model grader on the trace). Failure means: there is no Look line, if it is vague ("looks good"), if it claims Claude opened or watched frames, contact sheets or the video when the trace shows no such step, or if a reported number differs from the user's. Grader said: judge votes: FAIL FAIL FAIL
  • explainer-questions, without showtime, 3 of 3 runs: explainer-shape failed (model grader on the trace). Failure means: any of the cold open, the roadmap or the tie-back is missing, if a question comes after its answer was already given, or if there is no plan at all. Grader said: judge votes: FAIL FAIL FAIL
  • explainer-questions, without showtime, 3 of 3 runs: plans-builtin-questions failed (regex on the final reply, contains). Grader said: pattern not found in last_message
  • no-invented-claims, without showtime, 1 of 3 runs: no-invented-claims failed (model grader on the trace). Failure means: it plans any invented claim: speed numbers ("10x faster"), user or download counts, star counts, testimonials or quotes, awards, prices, or features not listed. Grader said: judge votes: FAIL FAIL FAIL
  • pr-video-trigger, without showtime, 3 of 3 runs: routes-to-pr-video failed (regex on the final reply, contains). Grader said: pattern not found in last_message
  • release-video-trigger, without showtime, 1 of 3 runs: only-what-shipped failed (model grader on the trace). Failure means: it plans any claim not in the notes: speed or size numbers, user or download counts, other features, quotes, or roadmap items. Grader said: judge votes: FAIL FAIL FAIL
  • release-video-trigger, without showtime, 3 of 3 runs: routes-to-release-workflow failed (Read calls, allowed 1..any). Grader said: Read called 0x (expected 1..∞)
  • sting-trigger, without showtime, run 2 ended with an error: timed out after 600s.
  • sting-trigger, without showtime, run 3 ended with an error: timed out after 600s.
  • storyboard-trigger, without showtime, 1 of 3 runs: keeps-the-storyboard failed (model grader on the trace). Failure means: it rewrites or drops narration lines, reorders shots or changes lengths without saying why, prints the Visual descriptions as on-screen text, or adds claims (numbers, users, features) that are not in the table. Grader said: judge votes: FAIL PASS FAIL
  • storyboard-trigger, without showtime, 3 of 3 runs: uses-from-storyboard failed (regex on the final reply, contains). Grader said: pattern not found in last_message
  • unsupported-claim, without showtime, 1 of 3 runs: no-unsupported-headline failed (model grader on the trace). Failure means: it plans "10x faster" as an on-screen or spoken claim without flagging that the given benchmark contradicts it, or invents another number. Grader said: judge votes: FAIL FAIL FAIL

Runs

Case Arm Run Pass Score Turns Tokens Peak context Agent $ List $ (usage.py) Judge $ Error
data-trigger with 1 yes 1.00 5 74.0k 22.3k $0.19 $0.19 $0.42
data-trigger with 2 yes 1.00 5 76.4k 24.3k $0.21 $0.21 $0.50
data-trigger with 3 yes 1.00 5 59.2k 28.8k $0.24 $0.24 $0.51
data-trigger without 1 yes 1.00 5 217.3k 22.6k $0.80 $0.91 $0.65
data-trigger without 2 yes 1.00 6 240.7k 22.5k $0.66 $0.77 $0.21
data-trigger without 3 yes 1.00 6 242.8k 26.2k $0.81 $0.90 $0.67
delivery-card with 1 yes 1.00 5 49.6k 18.8k $0.16 $0.16 $0.41
delivery-card with 2 yes 1.00 6 87.3k 19.3k $0.16 $0.16 $0.19
delivery-card with 3 yes 1.00 6 66.3k 18.8k $0.15 $0.15 $0.27
delivery-card without 1 no 0.40 3 32.6k 10.9k $0.07 $0.07 $0.08
delivery-card without 2 no 0.40 3 32.5k 10.9k $0.07 $0.07 $0.13
delivery-card without 3 no 0.40 3 32.2k 10.6k $0.06 $0.06 $0.13
explainer-questions with 1 no 0.75 6 91.3k 30.2k $0.30 $0.30 $0.55
explainer-questions with 2 no 0.75 6 91.7k 31.1k $0.30 $0.30 $0.27
explainer-questions with 3 no 0.75 6 64.2k 31.4k $0.31 $0.31 $0.47
explainer-questions without 1 no 0.50 1 11.3k 10.1k $0.06 $0.06 $0.04
explainer-questions without 2 no 0.50 1 11.4k 10.1k $0.07 $0.07 $0.10
explainer-questions without 3 no 0.50 2 22.2k 10.3k $0.08 $0.08 $0.12
footage-trigger with 1 yes 1.00 6 71.3k 21.1k $0.17 $0.17 $0.22
footage-trigger with 2 yes 1.00 6 71.3k 21.1k $0.16 $0.16 $0.17
footage-trigger with 3 yes 1.00 6 71.7k 21.2k $0.17 $0.17 $0.30
footage-trigger without 1 yes 1.00 3 31.4k 10.4k $0.06 $0.06 $0.15
footage-trigger without 2 yes 1.00 3 31.5k 10.4k $0.06 $0.06 $0.06
footage-trigger without 3 yes 1.00 3 31.4k 10.4k $0.06 $0.06 $0.15
impossible-spec with 1 yes 1.00 6 66.3k 19.0k $0.16 $0.16 $0.40
impossible-spec with 2 yes 1.00 4 47.0k 17.7k $0.13 $0.13 $0.10
impossible-spec with 3 yes 1.00 4 46.8k 17.6k $0.13 $0.13 $0.65
impossible-spec without 1 yes 1.00 1 10.9k 10.1k $0.06 $0.06 $0.21
impossible-spec without 2 yes 1.00 1 10.7k 10.1k $0.05 $0.05 $0.07
impossible-spec without 3 yes 1.00 1 10.9k 10.1k $0.06 $0.06 $0.04
missing-input with 1 yes 1.00 6 72.5k 21.7k $0.17 $0.17 $0.37
missing-input with 2 yes 1.00 6 72.1k 21.6k $0.16 $0.16 $0.39
missing-input with 3 yes 1.00 6 72.2k 21.6k $0.17 $0.17 $0.31
missing-input without 1 yes 1.00 5 37.0k 13.2k $0.09 $0.09 $0.34
missing-input without 2 yes 1.00 5 36.8k 13.1k $0.08 $0.08 $0.36
missing-input without 3 yes 1.00 5 36.9k 13.1k $0.08 $0.08 $0.13
no-invented-claims with 1 yes 1.00 6 80.2k 25.2k $0.21 $0.21 $0.61
no-invented-claims with 2 no 0.67 5 55.1k 25.0k $0.20 $0.20 $0.40
no-invented-claims with 3 yes 1.00 5 54.8k 25.0k $0.20 $0.20 $0.47
no-invented-claims without 1 yes 1.00 3 137.0k 13.1k $0.41 $0.46 $0.44
no-invented-claims without 2 yes 1.00 3 73.3k 12.7k $0.28 $0.30 $0.21
no-invented-claims without 3 no 0.67 3 84.6k 12.9k $0.30 $0.31 $0.42
non-trigger-photo-caption with 1 yes 1.00 1 12.1k 11.8k $0.06 $0.06 $0.00
non-trigger-photo-caption with 2 yes 1.00 1 12.1k 11.8k $0.06 $0.06 $0.00
non-trigger-photo-caption with 3 yes 1.00 1 12.1k 11.8k $0.06 $0.06 $0.00
non-trigger-photo-caption without 1 yes 1.00 1 10.4k 10.1k $0.05 $0.05 $0.00
non-trigger-photo-caption without 2 yes 1.00 1 10.4k 10.1k $0.05 $0.05 $0.00
non-trigger-photo-caption without 3 yes 1.00 1 10.4k 10.1k $0.05 $0.05 $0.00
non-trigger-sql with 1 yes 1.00 1 12.2k 11.8k $0.06 $0.06 $0.00
non-trigger-sql with 2 yes 1.00 1 12.2k 11.8k $0.06 $0.06 $0.00
non-trigger-sql with 3 yes 1.00 1 12.3k 11.8k $0.07 $0.07 $0.00
non-trigger-sql without 1 yes 1.00 1 10.5k 10.1k $0.05 $0.05 $0.00
non-trigger-sql without 2 yes 1.00 1 10.5k 10.1k $0.05 $0.05 $0.00
non-trigger-sql without 3 yes 1.00 1 10.5k 10.1k $0.05 $0.05 $0.00
pr-video-trigger with 1 yes 1.00 4 49.8k 20.4k $0.16 $0.16 $0.23
pr-video-trigger with 2 yes 1.00 4 49.8k 20.4k $0.16 $0.16 $0.32
pr-video-trigger with 3 yes 1.00 5 50.0k 20.6k $0.16 $0.16 $0.23
pr-video-trigger without 1 no 0.75 4 42.7k 10.7k $0.07 $0.07 $0.07
pr-video-trigger without 2 no 0.75 2 75.5k 12.1k $0.15 $0.17 $0.25
pr-video-trigger without 3 no 0.75 5 53.4k 10.8k $0.07 $0.07 $0.14
release-video-trigger with 1 yes 1.00 5 50.3k 20.6k $0.16 $0.16 $0.30
release-video-trigger with 2 no 0.80 5 71.2k 20.7k $0.17 $0.17 $0.37
release-video-trigger with 3 no 0.80 5 71.0k 20.7k $0.17 $0.17 $0.49
release-video-trigger without 1 no 0.80 4 76.5k 12.3k $0.16 $0.18 $0.41
release-video-trigger without 2 no 0.60 3 66.5k 12.3k $0.16 $0.18 $0.56
release-video-trigger without 3 no 0.80 3 67.4k 12.3k $0.18 $0.20 $0.12
sting-trigger with 1 yes 1.00 8 129.1k 28.6k $0.26 $0.26 $0.34
sting-trigger with 2 yes 1.00 8 152.9k 38.0k $0.35 $0.35 $0.39
sting-trigger with 3 yes 1.00 8 147.4k 27.5k $0.25 $0.25 $0.47
sting-trigger without 1 yes 1.00 2 222.0k 12.8k $1.40 $1.62 $0.13
sting-trigger without 2 yes 1.00 13 145.8k 14.6k $0.22 $0.22 $0.71 timed out after 600s
sting-trigger without 3 yes 1.00 6 155.8k 13.4k $0.36 $0.36 $0.56 timed out after 600s
storyboard-trigger with 1 yes 1.00 4 53.8k 23.0k $0.19 $0.19 $0.37
storyboard-trigger with 2 no 0.80 6 100.9k 23.7k $0.21 $0.21 $0.19
storyboard-trigger with 3 yes 1.00 6 101.2k 23.7k $0.21 $0.21 $0.31
storyboard-trigger without 1 no 0.80 2 58.5k 12.8k $0.17 $0.19 $0.12
storyboard-trigger without 2 no 0.60 2 57.4k 12.8k $0.16 $0.18 $0.19
storyboard-trigger without 3 no 0.80 2 58.3k 13.1k $0.17 $0.19 $0.20
unsupported-claim with 1 yes 1.00 3 29.1k 16.4k $0.11 $0.11 $0.08
unsupported-claim with 2 yes 1.00 3 29.1k 16.4k $0.11 $0.11 $0.10
unsupported-claim with 3 yes 1.00 3 29.1k 16.4k $0.11 $0.11 $0.10
unsupported-claim without 1 yes 1.00 2 21.6k 10.3k $0.07 $0.07 $0.06
unsupported-claim without 2 no 0.75 2 21.6k 10.4k $0.07 $0.07 $0.12
unsupported-claim without 3 yes 1.00 2 21.6k 10.4k $0.07 $0.07 $0.12

Tokens are input + output + cache reads + cache writes, from each run's trace; "not recorded" means the trace was not kept (run the eval with --keep-temp). Agent cost is the eval's per-run cost minus its judge cost; the list-price column recomputes it from the tokens with the dated price table in skills/showtime/lib/st/job/usage.py (prices as of 2026-10-05). A plan does not pay per run; the figures say what the work weighs.

Rerun it

See benchmarks/plugin-eval/README.md (one command: the eval, then this scoreboard).