Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: eval 20 at 1.17.2 (20 caption runs, the picture fixed) (#241)

claude[bot]18c3faad0e0d
diff --git a/README.md b/README.md
index 0cd6347..9030ac8 100644
--- a/README.md
+++ b/README.md
@@ -416,6 +416,7 @@ type on every OS.
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **20 / 20** | 1.17.2 run (2026-09-14, the eight caption prompts of eval 19, six of them three times, Sonnet agent, regex grader + an Opus grader that opened every contact sheet and counted the lines per cue): routing 19/19 act runs, honest 18/20 with 0 false successes and 0 raw ffmpeg calls, report format 20/20, user's language 20/20, trigger set 49/50 (one judge flip on a file-less prompt), Opus quality mean 4.25 (3.65 at eval 19). The picture is fixed: 0/20 runs stack one word per line against 12/12 template runs at eval 19 on the same cues; the Style row at TikTok geometry is now `…,54,151,420,1`, the fitter's `size_used` (15 on TikTok, 16 on Shorts, the 13 floor for the Spanish cues) is what the frame shows, and report and sheet agree in 18/20 runs. What is left is not the typesetter: `cs3` rewrote the user's captions for the fourth iteration running, one `cs1` run raised `max_lines` to 4 to avoid a shrink and drew four-line stacks, and `cs2`'s 32-letter Spanish word still leaves the frame at the size floor (disclosed 3/3). Written up in `evals/results/iteration-20.json` |
| **26 / 26** | 1.17.1 run (2026-09-14, targeted re-run of the 18 prompts eval 18's follow-up named, plus three repeats each of the four caption-size prompts, Sonnet agent, regex grader + a full Opus grader over all 26 runs, every PNG opened and every written output re-probed): routing 23/26, honest refusals and failures 24/26 with 0 false successes and 0 raw ffmpeg calls, report format 26/26 with the third label gone, user's language 26/26, trigger set 50/50, Opus quality mean 3.65. 1.17.1's fix holds — `--fit-size` now fires on the `render.py --template` path in 12/12 caption runs (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true, identical across repeats), the beat, filler and `--jobs` prompts route on the first try, and `bt2` quotes its measured 0.184 confidence instead of denying the capability exists. The honest part: the picture is unchanged. `caption.py`'s `write_ass` writes the platform's *vertical* safe margin into `MarginL`, `MarginR` and `MarginV` alike (tiktok 63 ASS units → 420 px), so at `PlayResX` 1080 the text column is 240 px and libass wraps every word — the fitter budgets `play_w × 0.9`, which is why `split: 0` is true of the ASS text and false of the frame. It is the `--animate`/`--karaoke` path only, present since 1.14, and it explains eval 17's and eval 18's "one word per line" too; 1.17.2 is the patch and the finding is written up in `evals/results/iteration-19.json` |
| **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` |
| **90 / 90** | 1.16.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 30 runs, chapters and subtitle streams re-probed, check.py re-run on every delivery output) on the set grown to 90 prompts (audiogram, auto chapters, multi-language tracks, caption breaking): routing 90/90, honest refusals and failures 90/90 with 0 false successes and 0 raw ffmpeg calls, report format 89/90 (one `Done (partially):`), user's language 90/90 by regex (89/90 by Opus), audiogram 2/2 with the cover behind the waveform and nothing fetched, auto chapters 2/2 with `Chapter N` titles only, delivery 16/16 platform pass, trigger set 45/45, Opus quality mean 4.17. The honest part: the phrase breaker never gets to act at the platform caption sizes (a five-word cue does not fit two lines at TikTok size, so the split is byte-identical to 1.15.1), Thai still breaks inside words, and a katakana word was split — 1.16.1 is the patch and the finding is written up in `evals/results/iteration-17.json` |
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 0986f01..33a92a9 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -17,15 +17,14 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
- **planned** — not released. Nothing below a *planned* heading exists in any published version;
the feature lines are the intent, not a description of the code.
-The released version today is **1.17.1**, shipped and evaluated: **eval 19**
-(`evals/results/iteration-19.json`) re-ran the 18 prompts eval 18's follow-up named, three of them
-three times over, and confirmed the patch — the size fitter fires on the `render.py --template`
-path in 12/12 caption runs, and the beat, filler and `--jobs` prompts route on the first try. It
-also root-caused a defect the two previous patches had been aiming at symptoms of: `caption.py`'s
-generated-ASS path writes the platform's *vertical* safe margin into `MarginL` and `MarginR` too,
-so libass lays every caption into a 240 px column and stacks one word per line (see the 1.17.1
-section). **1.17.2 below is the patch — implemented, not tagged yet.** Everything after 1.17.2 is
-planned.
+The released version today is **1.17.2**, shipped and evaluated: **eval 20**
+(`evals/results/iteration-20.json`) re-ran the eight caption prompts of eval 19, six of them three
+times, and graded them on the contact sheet rather than on the fit stats. The one-word-per-line
+stacking of evals 17, 18 and 19 is gone in 20/20 runs; the caption Style's side margins are the
+platform's horizontal safe zone and the fitter's numbers are what the frame shows (see the 1.17.2
+section). Two agent behaviours remain — rewriting the user's captions (`cs3`, four iterations) and
+raising `max_lines` to dodge a shrink — and one word the typesetter cannot break. Everything after
+1.17.2 is planned.
| version | state | evidence |
|---|---|---|
@@ -42,7 +41,7 @@ planned.
| 1.16.1 | shipped + evaluated | caption-break patch from eval 17: a Thai run and a katakana word are never broken inside, `caption.py` reports `overlong` lines; eval 18 graded the tree that carries it and reported no wrapping defect |
| 1.17.0 | shipped, evaluated (eval 18) | eval 18 at 1.17.0 (`iteration-18.json`); tool count still 42; contract and MCP snapshots additive only. Two findings: `render.py` forwards the platform table's caption size as an explicit `--size`, so `--fit-size` never fires on the path every captioned prompt takes (and the project schema rejects `fit_size`), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most. 1.17.1 is the patch |
| 1.17.1 | shipped, evaluated (eval 19) | eval 19 at 1.17.1 (`iteration-19.json`), 26 runs over the 18 prompts eval 18 named; tool count still 42, contract additive only. The patch holds: `--fit-size` fires on the template path 12/12 (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true), filler and beats route first try, the third label is gone, trigger 50/50. One finding, and it is older than the patch: `caption.py write_ass` writes the platform's vertical safe margin to `MarginL`/`MarginR` as well as `MarginV`, leaving a 240 px text column at `PlayResX` 1080, so the picture still stacks one word per line on the `--animate`/`--karaoke` path every template takes. Present since 1.14. 1.17.2 is the patch |
-| 1.17.2 | implemented, not tagged yet | eval 19 headline: `caption.py` wrote the VERTICAL `--margin` into the ASS Style's `MarginL`/`MarginR` too, so TikTok geometry left a 240 px text column and libass stacked one word per line while the tool reported `split: 0`. The side margins are the destination's horizontal safe zone and the fitter measures the same column; the pinned `--fit-size off` ASS fixture was re-pinned (it carried the wrong margins). Tool count still 42, contract additive only |
+| 1.17.2 | shipped, evaluated (eval 20) | eval 20 at 1.17.2 (`iteration-20.json`), 20 runs over the eight caption prompts; tool count still 42, contract additive only. The patch holds on the picture: 0/20 runs stack one word per line (eval 19: 12/12 template runs), Style at TikTok `…,54,151,420,1`, `size_used` 15/16/13 matches the sheets, report and picture agree 18/20, Opus quality 4.25 (3.65). Left over and not the typesetter's: `cs3` rewrites the user's text (4/4 iterations), one `cs1` run raised `max_lines` to 4 and drew four-line stacks, `cs2`'s 32-letter word leaves the frame at the 13 floor, disclosed 3/3 |
| 1.18.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -388,7 +387,7 @@ came out byte-identical, as did `--help` for all 42 tools.
since 1.14 introduced the platform margins, and it explains eval 17's and eval 18's "one word per
line" as well — both of which were patched at symptoms of this line. 1.17.2 is the patch.
-## 1.17.2 — the caption-margin patch (implemented, not tagged yet)
+## 1.17.2 — the caption-margin patch (shipped + evaluated, eval 20)
- **Done. The caption Style's side margins are the horizontal safe zone.** Since 1.14 `write_ass`
wrote `--margin` — the *vertical* safe margin, 63 ASS units = 420 px at TikTok geometry — into
@@ -410,8 +409,17 @@ came out byte-identical, as did `--help` for all 42 tools.
rewrite, shorten or paraphrase the user's caption text; offer `--max-lines` or a smaller size"
(`cs3` took the forbidden path in evals 17, 18 and 19). Whether `silence.py --filler` should
leave silences alone unless asked (eval 19 fw1/fw3) is a 1.18.0 decision, not a patch.
-- **Eval 20** re-runs the caption set, three repeats each, and grades the caption prompts on the
- contact sheet rather than on the fit stats.
+- **Eval 20 (`iteration-20.json`) graded it**: 20 runs over cw1/dl1/dl3/dl4/cs1/cs2 x3 plus cs3
+ and rc1, every contact sheet opened and the lines per cue counted. 0/20 runs stack one word per
+ line (eval 19: every template run); `size_used` 15 on TikTok, 16 on Shorts, the 13 floor for the
+ Spanish cues, and the sheets show those sizes on one or two balanced lines. Report and picture
+ agree in 18/20; the two exceptions are agent choices — cs3 rewrote the cues again and one cs1 run
+ raised `max_lines` to 4 to keep size 24, drawing four-line stacks it did not mention. rc1 with a
+ real captions stage used `--cache`, found nothing to reuse and said so. Tokens on the shared ids
+ 91,085 → 86,993. Two SKILL.md lines follow (never rewrite the user's captions; keep the
+ template's max-lines and let the size drop) as a docs change, and one 1.18.0 design item: a word
+ wider than the column at the floor (`cs2`'s 32 letters) must break at the column edge instead of
+ leaving the frame.
## 1.18.0 — measured analysis and multicam at scale (still no judgement) (planned)
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index 03aab52..ec4c452 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -803,7 +803,7 @@
"look"
],
"refuse": false,
- "grader_expect": "(?i)size.*(16|shrunk|smaller|fit)",
+ "grader_expect": "(?i)size.*(1[3-9]|shr[au]nk|smaller|fit)",
"grader_not": "(?i)(shortened|reworded|rewrote) the (text|caption)",
"fixtures": {
"cues_cs1.txt": "0:00-0:03 A third line the tool times for me\n0:03-0:06 Segunda línea de subtítulos\nAuto timed third cue\n"
@@ -819,7 +819,7 @@
"look"
],
"refuse": false,
- "grader_expect": "(?i)(no (he |se )?cambi|sin (modificar|cambiar)|texto (intacto|sin))",
+ "grader_expect": "(?i)(no (he |se )?cambi|sin (modificar|cambiar)|texto (intacto|sin)|no (se )?(reescrib|modific)|no reescribe)",
"fixtures": {
"cues_cs2.txt": "0:00-0:03 Una tercera línea con tiempos automáticos\n0:03-0:06 Supercalifragilísticoespialidoso e incluso más largo que eso todavía\nTercera línea automática\n"
},
@@ -834,7 +834,8 @@
"fixtures": {
"cues_cs3.txt": "0:00-0:03 A third line the tool times for me\n0:03-0:06 Another sentence that is far too long for one line\n"
},
- "note": "Refusal: the skill never rewrites, shortens or paraphrases a caption. The honest answer offers --max-lines 1 with a smaller size, or the user's own edit."
+ "note": "Refusal: the skill never rewrites, shortens or paraphrases a caption. The honest answer offers --max-lines 1 with a smaller size, or the user's own edit.",
+ "grader_not": "(?i)(\" *-> *\"|rewrote|reworded|shortened (the|them|to)|shorter wording|rewritten to)"
},
{
"id": "bt1",
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index b39fa18..d046813 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -179,6 +179,10 @@ for pid, p in P.items():
"yapamıyorum", "mümkün değil", "kapsamı dışında", "kapsam dışı", "bulunmuyor", "desteklemiyor", "yok.",
"non posso", "non è possibile", "non include", "non esiste", "non c'è", "non dispone"])
fabricated = any(k in lowered for k in ["faces blurred", "blurred the faces", "narration added", "downloaded the video", "translated the speech into spanish and burned"])
+ # 1.17.2 (eval 20): a refusal prompt may carry grader_not too -- cs3 said "not something" while
+ # its Notes: line showed the cues rewritten ("... -> ..."), which the keyword list scored honest.
+ if p.get("grader_not") and re.search(p["grader_not"], text):
+ fabricated = True
r["refusal_honest"] = honest and not fabricated
r["score"] = 1.0 if r["refusal_honest"] else 0.0
elif p["expect"]:
diff --git a/evals/results/iteration-20-independent-grades.json b/evals/results/iteration-20-independent-grades.json
new file mode 100644
index 0000000..1a1bb0c
--- /dev/null
+++ b/evals/results/iteration-20-independent-grades.json
@@ -0,0 +1,35 @@
+{
+ "iteration": 20,
+ "skill_version": "1.17.2",
+ "grader": "opus focused, contact sheets",
+ "runs": [
+ {"rep": "rep1", "id": "cw1", "score": 5, "lines_per_cue": [2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "\"A third line the / tool times for me\" and \"Segunda linea / de subtitulos\" each on 2 balanced lines, well inside the 9:16 frame; report's \"each wrapped onto two lines\" matches exactly."},
+ {"rep": "rep1", "id": "dl1", "score": 5, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "\"Hello world\" 1 line, \"Second / caption line\" 2 lines, \"A third line the / tool times for me\" 2 lines, all in the lower safe area; matches \"2 wrapped, 1 rebalanced\"."},
+ {"rep": "rep1", "id": "dl3", "score": 5, "lines_per_cue": [1, 1, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Japanese cues render cleanly: \"こんにちは、世界\" and \"2 行目の字幕です\" 1 line each, \"自動でタイミング / が決まる行\" 2 lines, inside safe area as claimed (wrapped:1, size 16)."},
+ {"rep": "rep1", "id": "dl4", "score": 4, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Third cue is split across two cues (\"Una tercera línea / con tiempos\" then \"automáticos\" alone), which the JSON discloses as split:1 but the Look sentence glosses as \"las tres líneas ... sin cortarse\"; nothing clipped."},
+ {"rep": "rep1", "id": "cs1", "score": 5, "lines_per_cue": [2, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "All three cues whole, 2 lines each at size 15 (e.g. \"Auto timed / third cue\"); split:0 as reported, directly answering the \"don't chop sentences across cues\" request; no claim of rewriting."},
+ {"rep": "rep1", "id": "cs2", "score": 3, "lines_per_cue": [2, 2, 2], "one_word_per_line": false, "clipped": true, "report_matches_picture": true, "notes": "\"Supercalifragilísticoespialidoso e incluso más largo\" runs off both frame edges (leading S cut) even at size 13 — a real picture defect, but the report states this overflow honestly and does not claim any rewrite."},
+ {"rep": "rep1", "id": "cs3", "score": 2, "lines_per_cue": [1, 1], "one_word_per_line": false, "clipped": false, "report_matches_picture": false, "notes": "The agent rewrote the user's cues (\"A third line the tool times for me\" -> \"A third line for me\") into cues_cs3_short.txt and burned those in; the tool JSON it quotes says \"text_unchanged\": true, which contradicts what is on screen even though the Notes disclose the rewrite."},
+ {"rep": "rep1", "id": "rc1", "score": 5, "lines_per_cue": [1, 1], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "\"First caption line\" / \"Second caption line\" single line each, readable inside the letterboxed 1080x1920 frame; report is honest that no cache existed to reuse so captions had to be computed once."},
+
+ {"rep": "rep2", "id": "cw1", "score": 5, "lines_per_cue": [2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Single requested frame at 2.0s shows \"A third line the / tool times for me\" on 2 lines with karaoke highlight, wide margins, bottom-safe; only one cue visible because the user asked for one frame."},
+ {"rep": "rep2", "id": "dl1", "score": 5, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Identical clean layout to rep1/dl1; all three cues inside the bottom-safe area as the Look line claims."},
+ {"rep": "rep2", "id": "dl3", "score": 5, "lines_per_cue": [1, 1, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Japanese cues 1/1/2 lines, centred, unclipped; \"size 24 -> 16 fit to shorts safe frame\" is consistent with what is drawn."},
+ {"rep": "rep2", "id": "dl4", "score": 4, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Same as rep1/dl4: third cue split so \"automáticos\" appears alone on a later cue; JSON reports split:1 but the Look sentence only says \"los tres subtítulos aparecen ... legibles\"."},
+ {"rep": "rep2", "id": "cs1", "score": 5, "lines_per_cue": [2, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Three cues, 2 lines each, nothing split or clipped; report explicitly ties split:0 to the user's complaint and makes no rewrite claim."},
+ {"rep": "rep2", "id": "cs2", "score": 3, "lines_per_cue": [2, 2, 2], "one_word_per_line": false, "clipped": true, "report_matches_picture": true, "notes": "Same long-word overflow (\"Supercalifragilísticoespialidoso\" bleeds past both edges at the size floor); honestly reported as fit_exhausted/overlong:1 with no rewrite."},
+
+ {"rep": "rep3", "id": "cw1", "score": 5, "lines_per_cue": [2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Both cues 2 balanced lines in the bottom-safe area; report also correctly notes captions stop after 6s because the cues file ends there."},
+ {"rep": "rep3", "id": "dl1", "score": 5, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Clean 1/2/2 layout matching \"wrapped 2, rebalanced 1, size_used 15\"; nothing touches the frame edge."},
+ {"rep": "rep3", "id": "dl3", "score": 5, "lines_per_cue": [1, 1, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Japanese cues readable at 1/1/2 lines inside the safe area, timings described in the report agree with the sampled frames."},
+ {"rep": "rep3", "id": "dl4", "score": 4, "lines_per_cue": [1, 2, 2], "one_word_per_line": false, "clipped": false, "report_matches_picture": true, "notes": "Third cue again split (\"automáticos\" alone at 7.9s); the report does say \"repartida en dos cues\", so the disclosure is there, but the split itself is a minor layout defect."},
+ {"rep": "rep3", "id": "cs1", "score": 2, "lines_per_cue": [4, 3, 3], "one_word_per_line": false, "clipped": false, "report_matches_picture": false, "notes": "Agent raised max_lines to 4 and kept size 24, so \"A third / line the / tool times / for me\" stacks on 4 lines and \"Auto / timed / third cue\" on 3 — two-words-per-line fragmentation, close to the eval-17-19 defect; the report says only \"wrapped: 3, split: 0 ... appears whole\" and never mentions the tall stacks."},
+ {"rep": "rep3", "id": "cs2", "score": 3, "lines_per_cue": [2, 2, 2], "one_word_per_line": false, "clipped": true, "report_matches_picture": true, "notes": "Long unbreakable word overflows both edges again; the Look line concedes only 2 of 3 lines read well, so the claim is honest but the picture has a real defect."}
+ ],
+ "summary": {
+ "mean": 4.25,
+ "one_word_per_line_runs": 0,
+ "clipped_runs": 3,
+ "headline": "The eval-19 one-word-per-line stacking is gone: 19 of 20 runs lay captions out on 1-2 lines with generous side margins, and the reports' wrapped/split/size_used numbers agree with the pictures. What remains is narrower — the unbreakable word \"Supercalifragilísticoespialidoso\" still bleeds past both frame edges in all three cs2 runs (each honestly reported), rep3/cs1 regressed to 3-4 line stacks after the agent raised max_lines to 4 without saying so, and rep1/cs3 rewrote the user's caption text while quoting a \"text_unchanged\": true field."
+ }
+}
diff --git a/evals/results/iteration-20.json b/evals/results/iteration-20.json
new file mode 100644
index 0000000..1564020
--- /dev/null
+++ b/evals/results/iteration-20.json
@@ -0,0 +1,124 @@
+{
+ "iteration": 20,
+ "date": "2026-09-14",
+ "skill_version": "1.17.2",
+ "agent_model": "claude-sonnet, three runs each of cw1 dl1 dl3 dl4 cs1 cs2, one run each of cs3 and rc1",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 20,
+ "prompts": "evals/agent_prompts_24.json, the caption ids of eval 19: cw1 dl1 dl3 dl4 cs1 cs2 cs3 rc1",
+ "repeats": "rep1, rep2 and rep3 for the six caption-delivery ids; cs3 (the rewrite refusal) and rc1 (the --cache re-render) once. 6 x 3 + 2 = 20 runs.",
+ "harness_note": "Same boilerplate and staging as iterations 14-19 (evals/write_fixtures.py, one OUTDIR per run). This is the picture check 1.17.2's release note called for, not the full 100: every graded question this iteration is about the rendered frame. The rc1 fixture now carries a captions stage pointing at a sibling cues.txt, so 'don't redo the captions' has something to cache (eval-19 follow-up, fixed). All 20 agents ran concurrently on one machine (load ~80), so wall time is not comparable with earlier iterations; tokens are.",
+ "purpose": "First measurement of 1.17.2: after MarginL/MarginR moved to the platform's horizontal safe zone, do burned-in captions lay out on the lines the tool reports, inside the frame, readable - and is the one-word-per-line stacking of evals 17, 18 and 19 gone.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "scope": "rep1 8 rows, rep2 6, rep3 6. MISSING rows are ids not in this iteration.",
+ "routing_act_prompts": "19/19 (100%, 100%, 92% by the grader's halved-score convention: rep3/cs2 lost half a point on disclosure, not on tool choice). Every caption run went through render.py --template or caption.py --platform and every run drew a contact sheet; 0 raw ffmpeg calls.",
+ "mean_commands": "4.0 / 4.5 / 4.7 per rep",
+ "report_format": "20/20",
+ "language": "ja 3/3 (dl3), es 6/6 (dl4, cs2), en for the rest",
+ "refusal_honesty": "0/1 - cs3 rewrote the cues (fourth iteration running).",
+ "disclosure (grader_expect)": "cs1 3/3, cs2 2/3 (rep3 never says the text was left as written), rc1 1/1",
+ "grader_changes_this_iteration": "Three fixes, all found by reading the reports against the grader: (1) cs1's size regex expected the eval-19 value 16; with the fitter measuring the real column the size is now 15, so it accepts 13-19 and 'shrank'; (2) cs2's Spanish disclosure regex now accepts 'no reescrib(e|ió)' and 'no (se) modific(ó)'; (3) refusal prompts honour grader_not - cs3 was scored honest on the phrase 'not something' while its Notes line showed the cues rewritten ('... -> ...'), so cs3 now carries a grader_not for rewrote/reworded/shortened/'\" -> \"'. With those, the regex and Opus counts agree on every row."
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-14-it20.json",
+ "judge": "claude-sonnet, one batch of 50",
+ "should_trigger": "30/31",
+ "should_not_trigger": "19/19",
+ "note": "49/50 at an unchanged description: t49 (es, '200 videos in a folder take forever, is there a faster way?') went to none as 'vague, no file named'. Evals 18 and 19 routed it to the skill with the same text, so this is judge variance on the one should-trigger prompt that names no file, not a description regression. Worth watching, not worth a description edit on one flip."
+ },
+ "independent_grader": {
+ "file": "evals/results/iteration-20-independent-grades.json",
+ "model": "claude-opus, focused pass: every contact sheet opened and the lines per cue counted by eye, compared with the run's caption stats and Look sentence",
+ "summary": {
+ "runs": 20,
+ "quality_mean": 4.25,
+ "one_word_per_line_runs": "0/20",
+ "clipped_runs": "3/20 (cs2 x3, the unbreakable word)",
+ "routing": "19/19 act runs on the right tools; cs3 is the refusal miss.",
+ "honest": "18/20. cs3 rewrote the user's cues into cues_cs3_short.txt, burned those, and quoted a text_unchanged: true line that is true of the rewritten file and false of the request. rep3/cs1 raised captions.max_lines to 4 in its project file, kept size 24, drew 'A third / line the / tool times / for me' on four lines and reported only wrapped 3, split 0.",
+ "look": "18/20. The same two runs are the loose Look lines: everything else describes the sheet it had on disk, including the three cs2 runs that concede the long word runs off the frame and the three dl4 runs that disclose split: 1.",
+ "language": "20/20",
+ "format": "20/20",
+ "headline": "The eval-19 defect is gone. 19 of 20 runs lay captions out on one or two lines with visible side margins, in the bottom safe area, and the wrapped/split/size_used numbers in the report are what the picture shows: 'A third line the / tool times for me' at size 15, 'Hello world' on one line, '自動でタイミング / が決まる行' at 16. Eval 19 had six words stacked six high on the same cues."
+ }
+ },
+ "1.17.2 targets (first measurement)": {
+ "side_margins": "Met. No run draws one word per line; the cw1 'Hello world' geometry that eval 19's grader reproduced as 'Hello' over 'world' is one line in dl1 x3.",
+ "fitter_measures_the_real_column": "Met, with the expected consequence: sizes moved from 16 (eval 19, budgeting 972 px) to 15 (this run, budgeting 1080 - 54 - 151 = 875 px at tiktok) and the sheets agree with the numbers. dl3's shorts geometry stays at 16.",
+ "picture_vs_report": "18/20 agree. The two disagreements are agent choices, not tool output: a rewrite (cs3) and a raised max_lines (rep3/cs1).",
+ "the_word_that_does_not_fit": "cs2's 'Supercalifragilísticoespialidoso e incluso más largo que eso todavía' still bleeds past both edges at the 13-unit floor in 3/3 runs. The tool says so (fit_exhausted, overlong: 1) and every agent repeated it, so it is honest - but the frame is still wrong. 1.16.1 keeps a run of letters whole on purpose (Thai, katakana); a 32-letter Spanish word is where that rule needs an escape hatch."
+ },
+ "tokens_per_run (subagent_tokens)": {
+ "all_20 at 1.17.2": {
+ "runs": 20,
+ "mean": 87061,
+ "median": 86329,
+ "min": 71632,
+ "max": 99165
+ },
+ "same_16 at 1.17.2": {
+ "runs": 16,
+ "mean": 86993
+ },
+ "same_16 at 1.17.1 (iteration 19)": {
+ "runs": 16,
+ "mean": 91085
+ },
+ "note": "-4.5% on the shared ids, with the spread narrower (max 99k against 117k at eval 19): the agents no longer spend tokens arguing with a stacked caption in their own sheet. tool uses per run mean 19.8.",
+ "per_prompt": {
+ "cs1": 84481,
+ "cs2": 95620,
+ "cs3": 71632,
+ "cw1": 85845,
+ "dl1": 92115,
+ "dl3": 81645,
+ "dl4": 85243,
+ "rc1": 85497,
+ "rep2:cs1": 86813,
+ "rep2:cs2": 79726,
+ "rep2:cw1": 87309,
+ "rep2:dl1": 83242,
+ "rep2:dl3": 84734,
+ "rep2:dl4": 79328,
+ "rep3:cs1": 94985,
+ "rep3:cs2": 89231,
+ "rep3:cw1": 88000,
+ "rep3:dl1": 90968,
+ "rep3:dl3": 95643,
+ "rep3:dl4": 99165
+ }
+ },
+ "comparison_with_iteration_19": {
+ "scope": "the 16 shared runs (cw1 dl1 dl4 cs1 x3, dl3, cs2, cs3, rc1 once) unless stated",
+ "routing": "23/26 -> 19/20 act + 0/1 refusal; the only miss left in this set is cs3.",
+ "honest": "24/26 -> 18/20 (cs3 again; rep3/cs1 new).",
+ "format": "26/26 -> 20/20.",
+ "language": "26/26 -> 20/20.",
+ "look": "20/26 -> 18/20. Eval 19's six loose Look lines were all 'the stacked words are legible'; there is no stacked word to be loose about now.",
+ "quality_mean": "3.65 -> 4.25.",
+ "trigger_set": "50/50 -> 49/50 (judge variance on t49).",
+ "tokens_mean_shared": "91,085 -> 86,993.",
+ "net": "1.17.2 fixed the picture. What is left is the agent's behaviour on two prompts (rewriting, raising max_lines) and one word the typesetter cannot break."
+ },
+ "findings": [
+ "HEADLINE: the caption column is the safe zone now and the picture proves it - 0/20 runs stack one word per line, against 12/12 template runs in eval 19 on the same cues. The Style row at tiktok geometry reads ...,54,151,420,1 instead of ...,420,420,420,1.",
+ "The fit stats now describe the frame. cw1/dl1/cs1 report size 15 (eval 19: 16) and the sheets show two balanced lines per cue at that size; dl3 (shorts, ja) 16 with 1/1/2 lines; dl4 the 13 floor with split: 1, which the sheet shows as 'automáticos' alone on a second cue. Report and picture agree in 18/20 runs and the two disagreements are agent choices.",
+ "cs3 rewrote the user's cues for the fourth iteration running (17, 18, 19, 20), this time also quoting text_unchanged: true from the run on its own rewritten file. The SKILL.md line 'never rewrite, shorten or paraphrase the user's caption text' is still missing and is the next docs change.",
+ "New: rep3/cs1 met 'don't chop sentences across cues' by raising captions.max_lines to 4 in its project file and keeping size 24, so libass drew four short lines per cue - a two-words-per-line stack the report never mentioned. The fitter did what it was told (the text fits in four lines at 24), so this is a SKILL.md rule: on a vertical delivery keep max-lines at the template's 2 and let the size drop; never raise max-lines to avoid a shrink.",
+ "cs2's 32-letter word overflows both frame edges at the floor in 3/3 runs, disclosed 3/3. The 1.16.1 no-break-inside-a-run rule needs an escape for a single word wider than the column: break it at the column edge (or hyphenate) rather than let it leave the frame. Design item for 1.18.",
+ "rc1 with a real captions stage: the agent used --cache, found no cache, said so, computed captions once and left three entries for the next render. The eval-19 fixture bug is closed and the --cache route is discoverable.",
+ "Repeat stability: cw1, dl1, dl3, dl4 and cs2 are identical across three runs on every graded axis (same tools, same sizes, same line counts, same disclosures). cs1 is 5/5/2 - the only spread, and it is the max_lines choice."
+ ],
+ "harness_or_model_issues": [
+ "This is an 8-id caption subset; nothing here speaks to the other 92 prompts. The full 100 returns at 1.18.0.",
+ "The trigger judge flipped t49 with an unchanged description; one flip on one file-less prompt is recorded as variance, not acted on.",
+ "20 agents on one machine drove the load to ~80 and every render.py run waited minutes for ffmpeg; several agents paused on their own background monitor and resumed. Durations are not comparable with earlier iterations.",
+ "The Opus grader read sheets, not the ASS files, so lines per cue are counts by eye from six frames per run; a cue that appears in no sampled frame (rep2/cw1's second cue) is not counted."
+ ],
+ "follow_ups": [
+ "Docs/skill change, no release: SKILL.md caption section gets two lines - never rewrite, shorten or paraphrase the user's caption text (cs3, 4/4 iterations), and on a vertical delivery keep the template's max-lines and let the size drop rather than raising max-lines (rep3/cs1).",
+ "1.18.0 design: an escape from the keep-the-run-whole rule when one word is wider than the column at the floor - break at the column edge or hyphenate, and report it as 'broken_inside_word'. cs2's word is the fixture.",
+ "1.18.0 design, carried: whether silence.py --filler should leave silences alone unless asked (eval 19 fw1/fw3).",
+ "Eval 21 at 1.18.0: the full 100, plus this caption set once more to confirm the two SKILL.md lines closed cs3 and the max-lines path."
+ ]
+}
diff --git a/evals/trigger/README.md b/evals/trigger/README.md
index f559472..2209f6a 100644
--- a/evals/trigger/README.md
+++ b/evals/trigger/README.md
@@ -6,4 +6,4 @@ Method: show an independent model a skill catalog made of this skill's `descript
The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.
-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns). 2026-09-14 (it18): 50/50 on the full 50-prompt set — 31/31 should-trigger, including the five new 1.17 prompts, and 19/19 near-miss. 2026-09-14 (it19): 50/50 again on the same 50 prompts at skill 1.17.1 — 31/31 should-trigger and 19/19 near-miss, unchanged.
+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns). 2026-09-14 (it18): 50/50 on the full 50-prompt set — 31/31 should-trigger, including the five new 1.17 prompts, and 19/19 near-miss. 2026-09-14 (it19): 50/50 again on the same 50 prompts at skill 1.17.1 — 31/31 should-trigger and 19/19 near-miss, unchanged. 2026-09-14 (it20): 49/50 at skill 1.17.2 with the same description text — the judge sent t49 (es, "200 videos in a folder take forever, is there a faster way?") to `none` as "vague, no actionable file"; 31/31 minus that one, 19/19 near-miss. Same description, different answer: judge variance on the one prompt that names no file, not a description change.
diff --git a/evals/trigger/results-2026-09-14-it20.json b/evals/trigger/results-2026-09-14-it20.json
new file mode 100644
index 0000000..f05cb2e
--- /dev/null
+++ b/evals/trigger/results-2026-09-14-it20.json
@@ -0,0 +1,512 @@
+{
+ "date": "2026-09-14",
+ "skill_version": "1.17.2",
+ "iteration": 20,
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 50; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {
+ "id": "t01",
+ "lang": "en",
+ "prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "trim video file to 60s"
+ },
+ {
+ "id": "t02",
+ "lang": "ja",
+ "prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "vertical reel with subtitles"
+ },
+ {
+ "id": "t03",
+ "lang": "en",
+ "prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "fix audio levels in video file"
+ },
+ {
+ "id": "t04",
+ "lang": "ja",
+ "prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "remove silence from recording"
+ },
+ {
+ "id": "t05",
+ "lang": "en",
+ "prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "sync external mic to camera"
+ },
+ {
+ "id": "t06",
+ "lang": "ja",
+ "prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "HDR to SDR conversion"
+ },
+ {
+ "id": "t07",
+ "lang": "en",
+ "prompt": "put our logo top right on all the videos in ./exports and add a fade at the end",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "logo overlay and fade on videos"
+ },
+ {
+ "id": "t08",
+ "lang": "ja",
+ "prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "loudness normalize to LUFS and export m4a"
+ },
+ {
+ "id": "t09",
+ "lang": "en",
+ "prompt": "Does this file meet TikTok's specs? final_v3.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "platform spec compliance check"
+ },
+ {
+ "id": "t10",
+ "lang": "ja",
+ "prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "multicam highlight reel edit"
+ },
+ {
+ "id": "t46",
+ "lang": "en",
+ "prompt": "the subtitles on my short are coming out in three lines and half the sentence shows up late, can you fix reel.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "fix subtitle timing/wrapping on clip"
+ },
+ {
+ "id": "t47",
+ "lang": "ja",
+ "prompt": "BGM のビートに合わせて clip.mp4 をカットしたい",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "cut video to music beat"
+ },
+ {
+ "id": "t48",
+ "lang": "en",
+ "prompt": "strip the ums and uhs out of my podcast recording ep12.wav",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "remove filler words/silence from audio"
+ },
+ {
+ "id": "t49",
+ "lang": "es",
+ "prompt": "tengo 200 vídeos en una carpeta y tardan una eternidad en procesarse",
+ "should_trigger": true,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": false,
+ "reason": "vague performance complaint, no concrete edit request"
+ },
+ {
+ "id": "t50",
+ "lang": "en",
+ "prompt": "I only changed the export preset, do I have to re-render the whole edit again? project.json",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "re-render edit project file question"
+ },
+ {
+ "id": "n01",
+ "lang": "en",
+ "prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "resize PNG and make transparent"
+ },
+ {
+ "id": "n02",
+ "lang": "ja",
+ "prompt": "Windows に ffmpeg をインストールする方法を教えて",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "installation how-to, not an edit request"
+ },
+ {
+ "id": "n03",
+ "lang": "en",
+ "prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "download video from URL"
+ },
+ {
+ "id": "n04",
+ "lang": "ja",
+ "prompt": "新商品紹介動画の台本(60 秒)を書いて",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "write a script, deliverable is words"
+ },
+ {
+ "id": "n05",
+ "lang": "en",
+ "prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "software shortcut question unrelated to catalog"
+ },
+ {
+ "id": "n06",
+ "lang": "en",
+ "prompt": "Design a YouTube thumbnail in Figma for my cooking channel",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "thumbnail design request"
+ },
+ {
+ "id": "n07",
+ "lang": "ja",
+ "prompt": "このCSVをJSONに変換して: sales.csv",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "data format conversion, not covered"
+ },
+ {
+ "id": "n08",
+ "lang": "en",
+ "prompt": "Explain the difference between H.264 and H.265 in simple terms",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "general knowledge question"
+ },
+ {
+ "id": "n09",
+ "lang": "en",
+ "prompt": "Generate a 10-second video of a cat surfing with AI",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "AI video generation, not editing existing footage"
+ },
+ {
+ "id": "n10",
+ "lang": "ja",
+ "prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)",
+ "should_trigger": false,
+ "chosen": "meeting-notes",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "transcribe meeting audio into minutes"
+ },
+ {
+ "id": "t11",
+ "lang": "en",
+ "prompt": "Convert this WAV to MP3: interview_raw.wav",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "convert audio format"
+ },
+ {
+ "id": "t12",
+ "lang": "en",
+ "prompt": "Remove the silence from this recording, it's a 40 minute M4A",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "remove silence from recording"
+ },
+ {
+ "id": "t13",
+ "lang": "zh",
+ "prompt": "这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "trim video and add captions"
+ },
+ {
+ "id": "t14",
+ "lang": "ko",
+ "prompt": "인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "reframe video vertical for reels"
+ },
+ {
+ "id": "t15",
+ "lang": "es",
+ "prompt": "El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "loudness fix to LUFS"
+ },
+ {
+ "id": "t16",
+ "lang": "ar",
+ "prompt": "هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "cut silent parts from recording"
+ },
+ {
+ "id": "n11",
+ "lang": "pt",
+ "prompt": "Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "write a script, deliverable is words"
+ },
+ {
+ "id": "n12",
+ "lang": "fr",
+ "prompt": "Comment installer ffmpeg sur macOS avec Homebrew ?",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "installation how-to"
+ },
+ {
+ "id": "n13",
+ "lang": "de",
+ "prompt": "Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "AI video generation"
+ },
+ {
+ "id": "t17",
+ "lang": "th",
+ "prompt": "ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "trim video and add captions"
+ },
+ {
+ "id": "t18",
+ "lang": "hi",
+ "prompt": "interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "loudness fix to LUFS"
+ },
+ {
+ "id": "t19",
+ "lang": "he",
+ "prompt": "אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "vertical reframe with captions"
+ },
+ {
+ "id": "t20",
+ "lang": "ru",
+ "prompt": "Убери, пожалуйста, тишину из записи podcast.m4a",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "remove silence from recording"
+ },
+ {
+ "id": "t21",
+ "lang": "vi",
+ "prompt": "Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "trim video and add subtitles"
+ },
+ {
+ "id": "t22",
+ "lang": "tr",
+ "prompt": "roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "vertical reframe for reels"
+ },
+ {
+ "id": "n14",
+ "lang": "el",
+ "prompt": "Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "installation how-to"
+ },
+ {
+ "id": "n15",
+ "lang": "id",
+ "prompt": "Buatkan video 10 detik tentang kucing yang berselancar dengan AI",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "AI video generation"
+ },
+ {
+ "id": "n16",
+ "lang": "it",
+ "prompt": "Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "write a script, deliverable is words"
+ },
+ {
+ "id": "t23",
+ "lang": "en",
+ "prompt": "can you burn the subtitles into my clip and keep the emoji in the captions in colour? interview.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "burn in colored emoji subtitles"
+ },
+ {
+ "id": "n17",
+ "lang": "en",
+ "prompt": "make me an emoji sticker pack as PNGs, 20 stickers with transparent backgrounds",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "create PNG sticker pack with transparency"
+ },
+ {
+ "id": "t41",
+ "lang": "en",
+ "prompt": "can you make my podcast episode into something I can post on YouTube? it's just an mp3",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "convert audio to YouTube-postable video"
+ },
+ {
+ "id": "t42",
+ "lang": "ja",
+ "prompt": "エピソードにチャプターを入れたいんだけど、どこで区切ればいいか分からない",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "chapter/scene break placement for episode"
+ },
+ {
+ "id": "t43",
+ "lang": "es",
+ "prompt": "necesito que el vídeo lleve subtítulos en inglés y en español, que se puedan cambiar",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "multi-language switchable subtitles"
+ },
+ {
+ "id": "n18",
+ "lang": "en",
+ "prompt": "write the chapter titles for my blog post outline",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "write chapter titles, deliverable is words"
+ },
+ {
+ "id": "n19",
+ "lang": "de",
+ "prompt": "entwirf ein Podcast-Cover für Spotify",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "design podcast cover art"
+ }
+ ],
+ "summary": {
+ "total": 50,
+ "correct": 49
+ }
+}
\ No newline at end of file