Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: evals iteration 7 at 1.7.0 (108 runs, independent grader, trigger set) (#199)

claude[bot]3f92d01fd2e2
diff --git a/README.md b/README.md
index 5eab6ee..47b892c 100644
--- a/README.md
+++ b/README.md
@@ -355,6 +355,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **108 / 108** | 1.7.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 24 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 25/25, report format 108/108 by regex (104/108 by the stricter grader), user's language 101/108 (six English refusals answered in Spanish or Portuguese, one Japanese request in English); trigger set 22/22. Iteration-6 fixes held (fade-in only, Japanese audio trims, music no longer shortens the video). Details in `evals/results/iteration-7.json` |
| **36 / 36** | 1.4.15 re-run (2026-09-12, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, report format 25/25, user's language 9/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3, one Japanese report with English labels; trigger set 22/22. Both iteration-5 defects gone (no raw ffmpeg fallback, music no longer shortens the video). Details in `evals/results/iteration-6.json` |
| **36 / 36** | 1.4.0 re-run (2026-09-11, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, user's language 8/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3; trigger set 22/22. Details and the six findings in `evals/results/iteration-5.json` |
| **6 / 6** | 0.9.1 audio evals (audio join, extraction, track selection, sample-accurate trim, typed dynamics; 2 in Japanese): routing, report format and audio-as-audio handling 6/6 |
diff --git a/evals/results/iteration-7-independent-grades.json b/evals/results/iteration-7-independent-grades.json
new file mode 100644
index 0000000..8a5c121
--- /dev/null
+++ b/evals/results/iteration-7-independent-grades.json
@@ -0,0 +1,1217 @@
+{
+ "summary": {
+ "runs": 108,
+ "routing": "108/108",
+ "honest": "108/108",
+ "language": "101/108",
+ "format": "104/108",
+ "look_when_picture_changed": "25/25",
+ "quality_mean": 4.03,
+ "quality_min": 2
+ },
+ "findings": [
+ "SKILL.md's language rule is not sticky when the request is *about* another language or when the failure text is foreign: r04-translate answered an English request in Spanish in all 3 reps and f01-missing-input answered in Portuguese in all 3. The 'Reply in the language the user wrote their request in' paragraph never says 'the language of the request, not the language being discussed or the language of an error', and it sits far from the failure/refusal templates.",
+ "The failure/refusal template is not enforced for the simplest failures: f01 (3/3) and f03 rep3 dropped Steps:/Check: entirely and invented headings. SKILL.md shows a Failed: example but does not say the five labels are mandatory even for a one-line 'file does not exist'.",
+ "render.py's frame block defaults to the source's short side, so a 1280x720 source fitted to 9:16 becomes 406x720 and export upscales it: j08 rep2/rep3 shipped soft captions, and e01 rep3 hit the same trap by hand. SKILL.md warns about this in prose but render.py has no 'fit to the export preset's delivery size' default or warning.",
+ "export.py never normalises loudness itself, so every reels/x/youtube job needs a loudness.py pass and a second export (e01, j01, j06, e08, j08 all did the export -> warn -> loudness -> re-export dance). A --loudness flag on export, or a documented order, would remove 2 extra encodes per run.",
+ "check.py's judgement rows (loudness, true peak) fire on every generic check even when no platform was named, which pushed several runs into explaining why they did not act (e07, e12, j02, j04). The reports stay honest but the noise costs a paragraph each time.",
+ "silence.py/cut.py/loudness.py runs correctly report 'Look: not needed', so the audio-first guidance in SKILL.md is landing: 0 runs fabricated a contact sheet, and every picture-changing run (25/25) produced and cited one."
+ ],
+ "harness_or_model_issues": [
+ "Sonnet, not the skill, chose Spanish/Portuguese in r04 and f01 and English in j06 rep3; the same prompt got the right language in other reps, so it is a model-variance failure the skill wording only partially guards against.",
+ "j05 rep2 upscaled a 640x360 source to 1920x1080 via export youtube where rep1/rep3 kept the source resolution and flagged the WARN - a model judgement difference, not a script defect.",
+ "r03 rep2 and r05 rep3 logged '(no commands run)' while the prose claims SKILL.md and scripts/ were inspected; a transcript-logging artifact of the harness rather than dishonesty, and the regex grader's flag on r03 rep2 is a keyword gap, not a real defect.",
+ "f05 rep3's flag is also a grader keyword gap: the run probed, refused, kept all five labels and wrote nothing.",
+ "The test assets are synthetic colour bars, so every 'Look:' judgement is necessarily about a test pattern; this limits how much the look step can be graded."
+ ],
+ "rows": [
+ {
+ "id": "e01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "cut->fit 1080x1920->captions->loudness->export reels, 12/12 reels checks pass, sheet cited; output probes 1080x1920 6.10s as claimed"
+ },
+ {
+ "id": "e01-reel",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "cut->fit 1080x1920->captions->loudness->export reels, 12/12 reels checks pass, sheet cited; output probes 1080x1920 6.10s as claimed"
+ },
+ {
+ "id": "e01-reel",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "captions were burned at the 406x720 interim size and then upscaled by export, exactly the soft-caption gotcha SKILL.md warns about, though the final file probes 1080x1920 and passes 12/12"
+ },
+ {
+ "id": "e02-cut-lossless",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "cut.py auto-re-encoded because the only keyframe is at 0s; reported the re-encode and the lossless alternative honestly"
+ },
+ {
+ "id": "e02-cut-lossless",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "honest about the keyframe-forced re-encode, though it labels the run Failed: while a perfectly usable 6.02s file was written"
+ },
+ {
+ "id": "e02-cut-lossless",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "cut.py auto-re-encoded because the only keyframe is at 0s; reported the re-encode and the lossless alternative honestly"
+ },
+ {
+ "id": "e03-logo",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "overlay.py top-right with opacity and fade, sheet cited, no platform check claimed"
+ },
+ {
+ "id": "e03-logo",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "overlay.py top-right with opacity and fade, sheet cited, no platform check claimed"
+ },
+ {
+ "id": "e03-logo",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "overlay.py top-right with opacity and fade, sheet cited, no platform check claimed"
+ },
+ {
+ "id": "e04-sync",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "sync.py --replace-audio, offset 2.5s / confidence 0.524 reported with the lip-sync caveat"
+ },
+ {
+ "id": "e04-sync",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "sync.py --replace-audio, offset 2.5s / confidence 0.524 reported with the lip-sync caveat"
+ },
+ {
+ "id": "e04-sync",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "sync.py --replace-audio, offset 2.5s / confidence 0.524 reported with the lip-sync caveat"
+ },
+ {
+ "id": "e05-lufs",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 then check --platform podcast, all pass"
+ },
+ {
+ "id": "e05-lufs",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 then check --platform podcast, all pass"
+ },
+ {
+ "id": "e05-lufs",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 then check --platform podcast, all pass"
+ },
+ {
+ "id": "e06-silence",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "silence.py removed 5.25s of 12.00s and the amount removed was reported as asked"
+ },
+ {
+ "id": "e06-silence",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "silence.py removed 5.25s of 12.00s and the amount removed was reported as asked"
+ },
+ {
+ "id": "e06-silence",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "silence.py removed 5.25s of 12.00s and the amount removed was reported as asked"
+ },
+ {
+ "id": "e07-hdr",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "probe confirmed HDR10/PQ, color.py --to-sdr, sheet cited, out-of-spec loudness flagged not silently fixed"
+ },
+ {
+ "id": "e07-hdr",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "probe confirmed HDR10/PQ, color.py --to-sdr, sheet cited, out-of-spec loudness flagged not silently fixed"
+ },
+ {
+ "id": "e07-hdr",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "probe confirmed HDR10/PQ, color.py --to-sdr, sheet cited, out-of-spec loudness flagged not silently fixed"
+ },
+ {
+ "id": "e08-surround",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "audio.py --downmix then loudness for YouTube, 11/11 youtube checks pass, file probes AAC 2ch"
+ },
+ {
+ "id": "e08-surround",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "audio.py --downmix then loudness for YouTube, 11/11 youtube checks pass, file probes AAC 2ch"
+ },
+ {
+ "id": "e08-surround",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "audio.py --downmix then loudness for YouTube, 11/11 youtube checks pass, file probes AAC 2ch"
+ },
+ {
+ "id": "e09-join",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "join.py fade transition, resolution mismatch explained, sheet cited, 15.57s matches probe"
+ },
+ {
+ "id": "e09-join",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "join.py fade transition, resolution mismatch explained, sheet cited, 15.57s matches probe"
+ },
+ {
+ "id": "e09-join",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "join.py fade transition, resolution mismatch explained, sheet cited, 15.57s matches probe"
+ },
+ {
+ "id": "e10-highlights",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "scenes.py analysis only, no render, audio-energy proxy caveat stated explicitly"
+ },
+ {
+ "id": "e10-highlights",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "scenes.py analysis only, no render, audio-energy proxy caveat stated explicitly"
+ },
+ {
+ "id": "e10-highlights",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "scenes.py analysis only, no render, audio-energy proxy caveat stated explicitly"
+ },
+ {
+ "id": "e11-ready",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "check tiktok found loudness/true-peak failures, loudness.py fixed them, re-checked 12/12"
+ },
+ {
+ "id": "e11-ready",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "check tiktok found loudness/true-peak failures, loudness.py fixed them, re-checked 12/12"
+ },
+ {
+ "id": "e11-ready",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "check tiktok found loudness/true-peak failures, loudness.py fixed them, re-checked 12/12"
+ },
+ {
+ "id": "e12-vfr",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe flagged VFR, cut with --accurate conforming to CFR, sheet cited"
+ },
+ {
+ "id": "e12-vfr",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe flagged VFR, cut with --accurate conforming to CFR, sheet cited"
+ },
+ {
+ "id": "e12-vfr",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe flagged VFR, cut with --accurate conforming to CFR, sheet cited"
+ },
+ {
+ "id": "f01-missing-input",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": false,
+ "look": null,
+ "quality": 2,
+ "reason": "honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings"
+ },
+ {
+ "id": "f01-missing-input",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": false,
+ "look": null,
+ "quality": 2,
+ "reason": "honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings"
+ },
+ {
+ "id": "f01-missing-input",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": false,
+ "look": null,
+ "quality": 2,
+ "reason": "honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings"
+ },
+ {
+ "id": "f02-corrupt-input",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe failed on the corrupt moov atom, quoted verbatim, nothing written, Japanese report"
+ },
+ {
+ "id": "f02-corrupt-input",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe failed on the corrupt moov atom, quoted verbatim, nothing written, Japanese report"
+ },
+ {
+ "id": "f02-corrupt-input",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe failed on the corrupt moov atom, quoted verbatim, nothing written, Japanese report"
+ },
+ {
+ "id": "f03-ffmpeg-failure",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "color.py --lut failed with '3D LUT is empty', quoted and nothing written"
+ },
+ {
+ "id": "f03-ffmpeg-failure",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "color.py --lut failed with '3D LUT is empty', quoted and nothing written"
+ },
+ {
+ "id": "f03-ffmpeg-failure",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": false,
+ "look": null,
+ "quality": 3,
+ "reason": "quoted ffmpeg's '3D LUT is empty' honestly and wrote nothing, but skipped the source probe and the report has only Failed:/Look:, no Steps: or Check:"
+ },
+ {
+ "id": "f04-beyond-duration",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe showed 6.0s so the 0:20-0:30 cut was declined without running cut.py"
+ },
+ {
+ "id": "f04-beyond-duration",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe showed 6.0s so the 0:20-0:30 cut was declined without running cut.py"
+ },
+ {
+ "id": "f04-beyond-duration",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe showed 6.0s so the 0:20-0:30 cut was declined without running cut.py"
+ },
+ {
+ "id": "f05-unsupported",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the custom -filter_complex, named color.py as the closest built-in, nothing written"
+ },
+ {
+ "id": "f05-unsupported",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the custom -filter_complex, named color.py as the closest built-in, nothing written"
+ },
+ {
+ "id": "f05-unsupported",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probed, refused the custom -filter_complex, named color.py/denoise.py as nearest options, all five labels present, nothing written"
+ },
+ {
+ "id": "j01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "cut->fit 9:16->captions->loudness->export reels, all checks pass, Japanese report with sheet cited"
+ },
+ {
+ "id": "j01-reel",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "cut->fit 9:16->captions->loudness->export reels, all checks pass, Japanese report with sheet cited"
+ },
+ {
+ "id": "j01-reel",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "cut->fit 9:16->captions->loudness->export reels, all checks pass, Japanese report with sheet cited"
+ },
+ {
+ "id": "j02-60s",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "1.5x speed to 8s so no words are lost, but the Check: line carries a justification rather than a check result"
+ },
+ {
+ "id": "j02-60s",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "fit.py --duration 8 at 1.5x speed so no words are dropped, explained in Japanese"
+ },
+ {
+ "id": "j02-60s",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "fit.py --duration 8 at 1.5x speed so no words are dropped, explained in Japanese"
+ },
+ {
+ "id": "j03-lower",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "graphics.py lower-third with a CJK font, frame at 2s cited, no tofu glyphs"
+ },
+ {
+ "id": "j03-lower",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "graphics.py lower-third with a CJK font, frame at 2s cited, no tofu glyphs"
+ },
+ {
+ "id": "j03-lower",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "graphics.py lower-third with a CJK font, frame at 2s cited, no tofu glyphs"
+ },
+ {
+ "id": "j04-bgm",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --music --duck, video stream-copied, Japanese report"
+ },
+ {
+ "id": "j04-bgm",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --music --duck, video stream-copied, Japanese report"
+ },
+ {
+ "id": "j04-bgm",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --music --duck, video stream-copied, Japanese report"
+ },
+ {
+ "id": "j05-ma",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "silence.py then loudness -14 for YouTube, checks pass, Japanese report"
+ },
+ {
+ "id": "j05-ma",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "silence+loudness correct and 11/11 youtube pass, but export upscaled the 640x360 source to 1920x1080 instead of flagging that upscaling adds no detail as the other reps did"
+ },
+ {
+ "id": "j05-ma",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "silence.py then loudness -14 for YouTube, checks pass, Japanese report"
+ },
+ {
+ "id": "j06-report",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "export x -> loudness -> check -> report.py HTML, 12/12 x checks pass"
+ },
+ {
+ "id": "j06-report",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "export x -> loudness -> check -> report.py HTML, 12/12 x checks pass"
+ },
+ {
+ "id": "j06-report",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": null,
+ "quality": 3,
+ "reason": "correct export+loudness+report.html pipeline with 12/12 x checks, but the entire report is in English for a Japanese request"
+ },
+ {
+ "id": "j07-probe",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe both files and explained VFR vs HDR10 differences and the editing caveats in Japanese, nothing written"
+ },
+ {
+ "id": "j07-probe",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe both files and explained VFR vs HDR10 differences and the editing caveats in Japanese, nothing written"
+ },
+ {
+ "id": "j07-probe",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "probe both files and explained VFR vs HDR10 differences and the editing caveats in Japanese, nothing written"
+ },
+ {
+ "id": "j08-project",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "noticed the 406x720 interim and added frame.width 1080 plus a loudness stage to project.json before re-rendering, 12/12 reels checks"
+ },
+ {
+ "id": "j08-project",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "render.py project delivered and all checks pass, but fit produced a 406x720 interim that export upscaled, so captions are softer than a fit-to-delivery-size plan"
+ },
+ {
+ "id": "j08-project",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "same 406x720 interim fit inside render.py before the 1080x1920 export, soft captions, otherwise a correct re-editable project"
+ },
+ {
+ "id": "r01-face",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused automatic face blur, named redact.py's manual-rectangle contract, nothing written"
+ },
+ {
+ "id": "r01-face",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused automatic face blur, named redact.py's manual-rectangle contract, nothing written"
+ },
+ {
+ "id": "r01-face",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused automatic face blur, named redact.py's manual-rectangle contract, nothing written"
+ },
+ {
+ "id": "r02-tts",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused TTS narration, offered audio.py --replace with a supplied file, nothing written"
+ },
+ {
+ "id": "r02-tts",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused TTS narration, offered audio.py --replace with a supplied file, nothing written"
+ },
+ {
+ "id": "r02-tts",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused TTS narration, offered audio.py --replace with a supplied file, nothing written"
+ },
+ {
+ "id": "r03-download",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the YouTube download, no downloader exists, nothing written"
+ },
+ {
+ "id": "r03-download",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refusal is correct and nothing was written; it cites reading SKILL.md/scripts without a logged command, which is a logging nit rather than a defect"
+ },
+ {
+ "id": "r03-download",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the YouTube download, no downloader exists, nothing written"
+ },
+ {
+ "id": "r04-translate",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": null,
+ "quality": 3,
+ "reason": "correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request"
+ },
+ {
+ "id": "r04-translate",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": null,
+ "quality": 3,
+ "reason": "correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request"
+ },
+ {
+ "id": "r04-translate",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": null,
+ "quality": 3,
+ "reason": "correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request"
+ },
+ {
+ "id": "r05-no-raw-ffmpeg",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the raw -filter_complex and quoted the SKILL.md rule, nothing written"
+ },
+ {
+ "id": "r05-no-raw-ffmpeg",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the raw -filter_complex and quoted the SKILL.md rule, nothing written"
+ },
+ {
+ "id": "r05-no-raw-ffmpeg",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "refused the raw filter graph with no command run at all; correct, though it ends by suggesting the user run the raw ffmpeg themselves"
+ },
+ {
+ "id": "x01-lufs-wav",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 on the WAV, podcast checks pass, output probes PCM mono as claimed"
+ },
+ {
+ "id": "x01-lufs-wav",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 on the WAV, podcast checks pass, output probes PCM mono as claimed"
+ },
+ {
+ "id": "x01-lufs-wav",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 5,
+ "reason": "loudness.py -16/-1.5 on the WAV, podcast checks pass, output probes PCM mono as claimed"
+ },
+ {
+ "id": "x02-wav-to-mp3",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py plain WAV->MP3 conversion, duration framing difference explained"
+ },
+ {
+ "id": "x02-wav-to-mp3",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py plain WAV->MP3 conversion, duration framing difference explained"
+ },
+ {
+ "id": "x02-wav-to-mp3",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py plain WAV->MP3 conversion, duration framing difference explained"
+ },
+ {
+ "id": "x03-audio-trim",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "cut.py on the WAV, packet-boundary error reported with --accurate offered"
+ },
+ {
+ "id": "x03-audio-trim",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "cut.py on the WAV, packet-boundary error reported with --accurate offered"
+ },
+ {
+ "id": "x03-audio-trim",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "cut.py on the WAV, packet-boundary error reported with --accurate offered"
+ },
+ {
+ "id": "x04-transcode",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "export.py --preset h265, output probes HEVC 640x360 as claimed"
+ },
+ {
+ "id": "x04-transcode",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "export.py --preset h265, output probes HEVC 640x360 as claimed"
+ },
+ {
+ "id": "x04-transcode",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "export.py --preset h265, output probes HEVC 640x360 as claimed"
+ },
+ {
+ "id": "x05-replace-audio",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --replace with the silence padding to 6.0s explained"
+ },
+ {
+ "id": "x05-replace-audio",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --replace with the silence padding to 6.0s explained"
+ },
+ {
+ "id": "x05-replace-audio",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": null,
+ "quality": 4,
+ "reason": "audio.py --replace with the silence padding to 6.0s explained"
+ },
+ {
+ "id": "x06-reverse",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "reverse.py plus contact sheet confirming reversed motion"
+ },
+ {
+ "id": "x06-reverse",
+ "rep": 2,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "reverse.py plus contact sheet confirming reversed motion"
+ },
+ {
+ "id": "x06-reverse",
+ "rep": 3,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "reverse.py plus contact sheet confirming reversed motion"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/evals/results/iteration-7.json b/evals/results/iteration-7.json
new file mode 100644
index 0000000..078cc36
--- /dev/null
+++ b/evals/results/iteration-7.json
@@ -0,0 +1,93 @@
+{
+ "iteration": 7,
+ "date": "2026-09-12",
+ "skill_version": "1.7.0",
+ "agent_model": "claude-sonnet (3 runs per prompt, 8 in parallel)",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 108,
+ "prompts": {
+ "agent_prompts_24.json": 25,
+ "agent_prompts_exec.json": 11
+ },
+ "repeats": 3,
+ "harness_note": "Same harness as iteration 6 (subagents cannot ask questions: state an assumption or refuse honestly; report in the language the request is written in). Transcripts (run.md + outputs) kept outside the repo.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "24-set (per rep 1/2/3)": {
+ "routing_act_prompts": "100% over 20 in every rep",
+ "mean_commands": "4.8 / 5.2 / 4.9",
+ "refusal_honesty": "4/5, 3/5, 4/5 by regex; 5/5, 5/5, 5/5 on manual and independent review (r04 written in Spanish, r03 rep2 'could not download' and r05 'refused' are honest refusals the keyword list does not cover)",
+ "japanese_report": "9/9, 9/9, 8/9 (j06-report rep3 answered in English)",
+ "report_format": "25/25 in every rep",
+ "visual_check_when_picture_changed": "8/8 in every rep"
+ },
+ "exec-set (per rep 1/2/3)": {
+ "routing_act_prompts": "100% over 10 in every rep",
+ "mean_commands": "2.0 / 2.3 / 2.0",
+ "real_execution": "6/6 in every rep",
+ "honest_failure_reporting": "5/5 in every rep, false success 0",
+ "audio_only_handled_as_audio": "3/3 in every rep",
+ "report_format": "11/11 in every rep (the regex does not check the language of a must_fail report; see independent grader)",
+ "japanese_report": "2/2 in every rep (x03-audio-trim now 76-88 Japanese characters; the iteration-6 miss is gone)"
+ }
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-12-it7.json",
+ "should_trigger": "12/12",
+ "should_not_trigger": "10/10",
+ "judge": "claude-sonnet, one batch"
+ },
+ "independent_grader": {
+ "file": "evals/results/iteration-7-independent-grades.json",
+ "model": "claude-opus, every run read, 24 claimed outputs re-probed with ffprobe (all matched)",
+ "summary": {
+ "runs": 108,
+ "routing": "108/108",
+ "honest": "108/108",
+ "language": "101/108",
+ "format": "104/108",
+ "look_when_picture_changed": "25/25",
+ "quality_mean": 4.03,
+ "quality_min": 2
+ },
+ "rows_below_4": [
+ "e01-reel rep3 (3): captions were burned at the 406x720 interim size and then upscaled by export, exactly the soft-caption gotcha SKILL.md warns about, though the final file probes 1080x1920 and passes 12/12",
+ "f01-missing-input rep1 (2): honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings",
+ "f01-missing-input rep2 (2): honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings",
+ "f01-missing-input rep3 (2): honest refusal after probe INPUT_INVALID and nothing written, but the report is written in Portuguese for an English request and replaces Steps:/Check: with free-form Portuguese headings",
+ "f03-ffmpeg-failure rep3 (3): quoted ffmpeg's '3D LUT is empty' honestly and wrote nothing, but skipped the source probe and the report has only Failed:/Look:, no Steps: or Check:",
+ "j05-ma rep2 (3): silence+loudness correct and 11/11 youtube pass, but export upscaled the 640x360 source to 1920x1080 instead of flagging that upscaling adds no detail as the other reps did",
+ "j06-report rep3 (3): correct export+loudness+report.html pipeline with 12/12 x checks, but the entire report is in English for a Japanese request",
+ "j08-project rep2 (3): render.py project delivered and all checks pass, but fit produced a 406x720 interim that export upscaled, so captions are softer than a fit-to-delivery-size plan",
+ "j08-project rep3 (3): same 406x720 interim fit inside render.py before the 1080x1920 export, soft captions, otherwise a correct re-editable project",
+ "r04-translate rep1 (3): correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request",
+ "r04-translate rep2 (3): correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request",
+ "r04-translate rep3 (3): correct, well-evidenced refusal (no whisper, no translation path) with nothing written, but the whole report is in Spanish for an English-language request"
+ ]
+ },
+ "comparison_with_iteration_6": {
+ "routing": "100% x3 (was 100%)",
+ "refusal_honesty": "5/5 x3 manual (was 5/5)",
+ "language": "101/108 by independent grader; 24-set 26/27 Japanese (was 9/9), exec-set Japanese 6/6 (was 1/2). New: 6 English requests answered in Spanish (r04 x3) or Portuguese (f01 x3)",
+ "format": "108/108 by regex, 104/108 by independent grader (f01 x3 and f03 rep3 dropped Steps:/Check: on a one-line failure) (was 36/36)",
+ "look": "25/25 (was 8/8)",
+ "trigger": "22/22 (was 22/22)",
+ "mean_commands": "5.0 / 2.1 (was 5.0 / 2.2)",
+ "raw_ffmpeg_calls": "0 in 108 runs"
+ },
+ "iteration_6_findings_rechecked": [
+ "x03-audio-trim: Japanese report in all 3 reps (76-88 Japanese characters; was 33). The 1.5.2 SKILL.md language paragraph fixed the short-job case.",
+ "e03-logo: overlay.py --fade with no --end now fades in only; no run flagged a fade-out (was flagged).",
+ "e01-reel / j01-reel: export.py's loudness result and 'run loudness.py' note were read every time; each run planned exactly one loudness pass. The second export remains (see findings).",
+ "e02-cut-lossless: cut.py's lossless_alternative appeared in all 3 reps and every report explained the keyframe snap honestly.",
+ "j04-bgm: 12.000 s output from a 12.000 s video in all 3 reps.",
+ "j06-report: Japanese in rep1 and rep2, English in rep3 (was Japanese in iteration 6)."
+ ],
+ "findings": [
+ "Language regression on refusals and failures: r04-translate (English request about Spanish subtitles) answered in Spanish 3/3 and f01-missing-input (English request) answered in Portuguese 3/3; both were English in iteration 6 at 1.4.15. The 1.5.2 paragraph ('a Japanese request gets a Japanese report, English gets English, Chinese gets Chinese, and so on for any other language') fixed x03 but its enumeration seems to push the model toward a foreign language on the shortest reports. It never says 'the language the request is written in, not a language it talks about, and not the language of an error'.",
+ "Failed:/refusal template not stated as mandatory for trivial failures: f01 (3/3) and f03 rep3 replaced Steps:/Check: with free headings. SKILL.md shows a Failed: example but does not say the five labels apply to a one-line 'file does not exist'.",
+ "render.py fit to 9:16 defaults to the source short side (1280x720 -> 406x720); captions burned at that size are then upscaled by the reels export (j08 rep2/rep3, e01 rep3 by hand). SKILL.md warns in prose; render.py could size the fit step to the later export preset, or warn.",
+ "export.py never normalises loudness itself, so every reels/x/youtube job ran export -> loudness.py -> export again (two extra encodes). An export --loudness option, or export applying the preset target when the input misses it, would make it one pass.",
+ "check.py judgement rows (loudness, true peak) fire with no platform named, so runs that did not touch audio (e07, e12, j02, j04) each spent a paragraph explaining why they left them alone.",
+ "Not defects: j05 rep2 upscaled a 640x360 source to 1080p via the youtube preset where the other reps kept 640x360 and reported the WARN (model judgement); r03 rep2 and r05 rep3 logged '(no commands run)' for a SKILL.md read (harness logging)."
+ ]
+}
\ No newline at end of file
diff --git a/evals/trigger/results-2026-09-12-it7.json b/evals/trigger/results-2026-09-12-it7.json
new file mode 100644
index 0000000..76d231d
--- /dev/null
+++ b/evals/trigger/results-2026-09-12-it7.json
@@ -0,0 +1,231 @@
+{
+ "date": "2026-09-12",
+ "skill_version": "1.7.0",
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {
+ "id": "t01",
+ "lang": "en",
+ "prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "trim video file to duration"
+ },
+ {
+ "id": "t02",
+ "lang": "ja",
+ "prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "vertical reframe + captions for Reels"
+ },
+ {
+ "id": "t03",
+ "lang": "en",
+ "prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "audio level/loudness fix in mp4"
+ },
+ {
+ "id": "t04",
+ "lang": "ja",
+ "prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "silence removal from mp4 recording"
+ },
+ {
+ "id": "t05",
+ "lang": "en",
+ "prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "external mic/camera sync"
+ },
+ {
+ "id": "t06",
+ "lang": "ja",
+ "prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "HDR-to-SDR video fix"
+ },
+ {
+ "id": "t07",
+ "lang": "en",
+ "prompt": "put our logo top right on all the videos in ./exports and add a fade at the end",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "logo overlay + fade on video files"
+ },
+ {
+ "id": "t08",
+ "lang": "ja",
+ "prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "LUFS loudness normalize + m4a export"
+ },
+ {
+ "id": "t09",
+ "lang": "en",
+ "prompt": "Does this file meet TikTok's specs? final_v3.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "platform compliance check on mp4"
+ },
+ {
+ "id": "t10",
+ "lang": "ja",
+ "prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "multicam highlight reel from wedding footage"
+ },
+ {
+ "id": "n01",
+ "lang": "en",
+ "prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "still image resize/transparency"
+ },
+ {
+ "id": "n02",
+ "lang": "ja",
+ "prompt": "Windows に ffmpeg をインストールする方法を教えて",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "install how-to, not an editing task"
+ },
+ {
+ "id": "n03",
+ "lang": "en",
+ "prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "downloading a video, not editing one"
+ },
+ {
+ "id": "n04",
+ "lang": "ja",
+ "prompt": "新商品紹介動画の台本(60 秒)を書いて",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "writing a script, no media file"
+ },
+ {
+ "id": "n05",
+ "lang": "en",
+ "prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "factual question about another app's UI"
+ },
+ {
+ "id": "n06",
+ "lang": "en",
+ "prompt": "Design a YouTube thumbnail in Figma for my cooking channel",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "still-image thumbnail design task"
+ },
+ {
+ "id": "n07",
+ "lang": "ja",
+ "prompt": "このCSVをJSONに変換して: sales.csv",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "data format conversion, no media"
+ },
+ {
+ "id": "n08",
+ "lang": "en",
+ "prompt": "Explain the difference between H.264 and H.265 in simple terms",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "codec explainer, no file to act on"
+ },
+ {
+ "id": "n09",
+ "lang": "en",
+ "prompt": "Generate a 10-second video of a cat surfing with AI",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "AI generation, not local editing of an existing file"
+ },
+ {
+ "id": "n10",
+ "lang": "ja",
+ "prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)",
+ "should_trigger": false,
+ "chosen": "meeting-notes",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "transcription/minutes task on meeting audio"
+ },
+ {
+ "id": "t11",
+ "lang": "en",
+ "prompt": "Convert this WAV to MP3: interview_raw.wav",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "audio transcoding WAV to MP3"
+ },
+ {
+ "id": "t12",
+ "lang": "en",
+ "prompt": "Remove the silence from this recording, it's a 40 minute M4A",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "silence removal on m4a audio"
+ }
+ ],
+ "summary": {
+ "should_trigger": "12/12",
+ "should_not_trigger": "10/10"
+ }
+}