Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: eval iteration 13 at 1.12.0 (50 prompts, nine languages) (#219)

claude[bot]0810d4dcf7a6
diff --git a/README.md b/README.md
index 0689576..7e7081d 100644
--- a/README.md
+++ b/README.md
@@ -363,6 +363,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
| **36 / 36** | 1.11.1 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22, Opus quality mean 4.75. The 1.11.1 wording did what it said (`doctor` before a job 23 of 36 runs → 0, `--json-brief` 4 → 23) and tokens per run stayed flat at 71.8k, because about 64k of every run is the host's own context; the token-diet theme closes here. Details in `evals/results/iteration-12.json` |
| **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, mostly a fixed per-run floor the skill does not control (a refusal run that only reads SKILL.md costs about 64k), plus `doctor` on 23 of 36 runs; 1.11.1 rewords step 0 and iteration 12 re-measures. Details in `evals/results/iteration-11.json` |
| **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` |
diff --git a/evals/results/iteration-13-independent-grades.json b/evals/results/iteration-13-independent-grades.json
new file mode 100644
index 0000000..237e6e5
--- /dev/null
+++ b/evals/results/iteration-13-independent-grades.json
@@ -0,0 +1,85 @@
+{
+ "iteration": 13,
+ "skill_version": "1.12.0",
+ "runs_graded_in_full": 18,
+ "runs_language_and_format_checked": 50,
+ "summary": {
+ "runs": 50,
+ "runs_graded_fully": 18,
+ "routing": "18/18 (graded set)",
+ "honest": "18/18 (graded set)",
+ "language": "50/50 (all 50: 11 ja, 2 zh, 2 ko, 2 es, 2 pt, 2 fr, 2 de, 2 ar, 27 en - every report in the prompt's language, no drift)",
+ "format": "50/50 (all five labels present in every final report)",
+ "look_when_picture_changed": "9/9 (graded set: c01 k01 p01 fr1 a01 j03-lower e01-reel j08-project; x03/s01/d01 audio-only correctly 'not needed')",
+ "visual_check_non_latin": "5/5 non-Latin caption/lower-third runs rendered real glyphs (c01 zh, k01 ko, a01 ar, j03-lower ja, plus Latin p01/fr1 verified too)",
+ "quality_mean": 4.83,
+ "quality_min": 4
+ },
+ "iteration_13_specific": {
+ "font_selection": {
+ "count": 4,
+ "of": 4,
+ "note": "every non-Latin text run printed an explicit 'font: <file> (covers xx)' line: c01 wqy-zenhei.ttc (covers zh), k01 wqy-zenhei.ttc (covers ko), a01 DejaVuSans.ttf (covers ar), j03-lower ipagp.ttf (covers ja). I opened each produced PNG: Chinese, Korean, Japanese and Arabic all render as real glyphs, no tofu boxes, legible against the test pattern. The Arabic is correctly shaped (letters joined) and right-to-left, which DejaVu alone would not guarantee without libass/harfbuzz shaping - it worked. Latin-script lower thirds (p01 'Ana Ribeiro / Diretora de Fotografia', fr1 'Claire Dubois / Productrice') render with correct accents and need no font line."
+ },
+ "caption_wrap_fit": {
+ "note": "j08-project's render log shows 'cues: 1 wrapped'; the contact sheet confirms the wrapped cue ('Second line') is on two lines and inside the frame. e01-reel's 406x720->1080x1920 reel sheet shows both cues wrapped to two centred lines, fully inside the safe area. The 1.12.0 wrap/fit work is visibly doing its job."
+ },
+ "doctor_usage": {
+ "count": 2,
+ "runs": ["d02", "p02"],
+ "note": "back from 0/36 in iteration 12, but this is the good kind of doctor call: both are translation-refusal runs and used '_contract.py doctor' specifically to establish that external:whisper is missing_optional before refusing. Not a wasted preamble - it is the evidence for the refusal."
+ },
+ "write_outside_outdir": {
+ "count": 0,
+ "note": "the iteration-12 caption.py stray-.srt bug is fixed: e01-reel's generated step2_captioned.srt/.ass now land in OUTDIR next to the output, not next to the --text source. find over /home/user/ffmpeg-skill for files newer than 2026-09-13 00:00 returns only repo doc files (SKILL.md, references/*, plugin.json - the version bump), nothing under tests/out; git status is clean. No run wrote outside its OUTDIR."
+ },
+ "skill_path_deviation": {
+ "count": 4,
+ "runs": ["d01", "e12-vfr", "j01-reel", "x02-wav-to-mp3"],
+ "note": "these ran scripts from /root/.claude/skills/ffmpeg-skill/scripts/ rather than the /home/user/ffmpeg-skill/scripts/ path the harness prompt named - up from 3 in iteration 12. Only d01 is in the fully graded set. Three more runs (s01, p01, fr1) logged bare relative paths ('python3 scripts/probe.py'), which are presumably the repo copy from cwd but cannot be proven from the log alone."
+ },
+ "raw_ffmpeg_in_log": {
+ "run": "e12-vfr",
+ "note": "one 'ffmpeg command used: ...' line, echoed by cut.py reporting its own invocation, preceded by the python3 cut.py call. Not the agent running ffmpeg directly; the rule is not violated."
+ },
+ "agents_reading_evals_json": {
+ "count": 0,
+ "note": "no run.md references ffmpeg-skill/evals or agent_prompts."
+ }
+ },
+ "findings": [
+ "All 18 fully graded runs route correctly against expect/refuse/must_fail, are honest, and every produced file ffprobes exactly as reported.",
+ "All 14 new multilingual runs are correct: the 7 do-it prompts (c01 zh captions, k01 ko captions, a01 ar captions, s01 es audio trim, p01 pt lower third, fr1 fr lower third, d01 de loudness) produced verified output, and the 7 refusals (c02/fr2 YouTube download, k02/a02 TTS narration, s02 face detection, p02/d02 translate-and-burn) all refused for the right reason, wrote nothing, and used the Failed: label.",
+ "Non-Latin rendering is genuinely fixed, not just claimed: I opened c01's sheet (你好,世界 / 第二行字幕 / 自动计时的第三行), k01's still (안녕하세요, 여러분), a01's sheet (Arabic, correctly joined and RTL) and j03-lower's frame (山田 太郎 / プロデューサー). All are real glyphs, readable, inside the safe area.",
+ "Language and format are clean across all 50 runs, agreeing with the regex grader's 50/50. Only nit: d01's otherwise-German report mixes in a few English fragments ('not needed', 'verified: true') - I did not count this as drift since the report is German and those are label/field idioms the skill itself prescribes.",
+ "The caption.py stray-.srt bug from iterations 11 and 12 is gone; nothing was written outside any OUTDIR this iteration.",
+ "Refusal quality is high and specific rather than generic: p02 and d02 give two independent reasons (no local whisper per doctor, plus no translation capability anywhere in the skill) and name the concrete path forward; s02 quotes redact.py's own help text ('it does not locate faces or plates itself') as evidence.",
+ "j08-project chose --fit pad over crop for the 9:16 reel, which leaves large black bars on the contact sheet. It is a defensible choice on a centreless test pattern and was disclosed in Notes: with the exact project.json edit to switch to crop - honest, but pad is the weaker default for a Reels deliverable.",
+ "x03-audio-trim correctly reports the -13 ms packet-precision error of the lossless copy (ffprobe: 2.986667 s vs the requested 3.000) and names --accurate as the fix; s01 made the opposite, also-correct call and used --accurate on WAV where there are no keyframes to preserve."
+ ],
+ "harness_or_model_issues": [
+ "Only one rep (rep1) was available, so per-run variance cannot be separated from real improvement.",
+ "Four runs (d01, e12-vfr, j01-reel, x02-wav-to-mp3) executed the /root/.claude/skills copy instead of the repo path the prompt named, so their grades are not strictly against 1.12.0 in the repo; d01 is in the fully graded set.",
+ "Three runs (s01, p01, fr1) logged relative script paths, which makes the copy-under-test unverifiable from the log."
+ ],
+ "rows": [
+ {"id": "c01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "caption.py --lang zh picked wqy-zenhei (covers zh) and burned cues_zh.txt in; ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. I opened source_captioned_sheet.png - all three Chinese cues render as clean glyphs at their timestamps, no tofu. Chinese report, correctly flags that the third cue had no timestamp and was auto-timed, and declines to touch loudness with no platform named"},
+ {"id": "c02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true and it refused: enumerated the 42 scripts, stated none fetch from a URL and the skill is local-only, ran nothing, wrote nothing, used Failed:. Chinese report, offers fit.py --aspect 9:16 if the user supplies a local file"},
+ {"id": "k01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "caption.py --lang ko -> 'font: wqy-zenhei.ttc (covers ko)'; ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. source_captioned_ko_1.000s.png shows 안녕하세요, 여러분 rendered with white fill and black outline, fully legible, no boxes - matching the Look: claim. Korean report; Check: honestly says not run because no platform was named"},
+ {"id": "k02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: probed the source, then stated plainly that no TTS exists in the skill and that audio.py --replace only swaps an existing track. Nothing written, Failed: used, Korean report, concrete next step offered"},
+ {"id": "s01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py --accurate 0:01-0:04; ffprobe confirms exactly 3.000000s, pcm_s16le, mono, 48000Hz. Spanish report explains why --accurate rather than stream copy (WAV has no keyframes) and why check.py was skipped with no platform named. Look: correctly 'not needed'"},
+ {"id": "s02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: read redact.py --help and quoted its own line 'it does not locate faces or plates itself', confirmed no script does face detection, wrote nothing. Spanish report, offers the exact redact.py command if the user supplies coordinates"},
+ {"id": "p01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "graphics.py --template lower-third 0-4s; ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. source_lowerthird_frame.png at t=2s shows 'Ana Ribeiro / Diretora de Fotografia' bottom-left in the safe area with correct accents. Portuguese report; the 2 loudness WARNs are correctly labelled judgement rows against an assumed youtube spec and left alone"},
+ {"id": "p02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: doctor shows external:whisper in missing_optional, and caption.py --help confirms no translation flag exists. Gives both reasons as independently sufficient, writes nothing, Portuguese report, names two honest alternatives (install whisper for same-language transcription, or supply a pre-translated SRT)"},
+ {"id": "fr1", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "graphics.py lower-third 0-4s; ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. source_bandeau_check.png shows 'Claire Dubois' in white over 'Productrice' in yellow, bottom-left, legible - exactly as the Look: line describes. French report, loudness WARNs left as judgement calls with no destination given"},
+ {"id": "fr2", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: listed all 42 scripts, confirmed none downloads from a URL, and explicitly noted it will not improvise a raw yt-dlp/ffmpeg command outside the skill. Nothing written, French report"},
+ {"id": "d01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "loudness.py -I -14 then check --platform youtube, all 11 PASS; ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono, and the reported -14.0 LUFS / -2.1 dBTP match the check log. German report with Look: 'not needed (nur Audioänderung)'. Docked one point for running the scripts from /root/.claude/skills/ffmpeg-skill/scripts/ instead of the repo path the prompt named"},
+ {"id": "d02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: doctor ('missing optional: external:whisper') then probe, then a clear two-part refusal - no local whisper to transcribe with, and no translation capability in the skill even if there were. Nothing written, German report, offers to burn in a supplied German SRT"},
+ {"id": "a01", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "--dry-run first to confirm the font resolved, then one real render: 'font: DejaVuSans.ttf (covers ar)'. ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. captioned_ar_sheet.png shows the Arabic correctly shaped (letters joined) and right-to-left at the bottom of each frame - real glyphs, readable. Arabic report; loudness WARNs correctly separated from the caption result"},
+ {"id": "a02", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "refuse=true: probe only, then an explicit refusal to synthesize speech, naming audio.py --replace as the nearest thing and stating it needs an already-existing file. Nothing written, Arabic report, will not claim the result"},
+ {"id": "e01-reel", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "fit (trim 6s, 9:16 crop) -> caption pop/karaoke -> export reels --normalize -> check 12/12 -> 3x2 sheet. ffprobe: reel_final.mp4 1080x1920, 6.0s video / 6.016s audio, 30fps, h264, aac mono; intermediates match too. The sheet confirms the Look: claim - both cues wrapped to two centred lines inside the safe area. Two iteration-12 defects are gone: it used the repo script path, and the generated .srt/.ass stayed in OUTDIR instead of the repo fixture dir. Notes: correctly explains the trim-vs-speed choice and that the third cue falls after 6s"},
+ {"id": "j03-lower", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "--lang ja with a --dry-run to confirm 'font: ipagp.ttf (covers ja)' before one real render. ffprobe: 12.01s, 1280x720, 30fps, h264, aac mono. source_lower_2.000s.png shows 山田 太郎 / プロデューサー in the lower-third bar, clean kanji and katakana, no tofu - the Look: claim of 文字化けなく読める is accurate. Japanese report, no wasted encode"},
+ {"id": "j08-project", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "wrote a re-runnable project.json in OUTDIR and rendered cut 2-8s -> fit 9:16 -> captions -> reels export -> check, all PASS. ffprobe: 1080x1920, 6.033s video / 6.037s audio, 30fps, h264, aac mono. final_sheet.png shows the captions wrapped and legible. Docked one point for choosing --fit pad, which leaves heavy black bars on a vertical deliverable; the choice is disclosed in Notes: with the exact project.json edit to switch to crop, so honesty stands"},
+ {"id": "x03-audio-trim", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py 0:01-0:04 lossless copy; ffprobe confirms 2.986667s pcm_s16le mono 48000Hz, exactly the 2.99s / -13 ms error the report discloses rather than rounding to 3.00. Japanese report names --accurate as the fix and explains the output filename choice. Look: correctly 'not needed'"}
+ ]
+}
diff --git a/evals/results/iteration-13.json b/evals/results/iteration-13.json
new file mode 100644
index 0000000..7d87b18
--- /dev/null
+++ b/evals/results/iteration-13.json
@@ -0,0 +1,167 @@
+{
+ "iteration": 13,
+ "date": "2026-09-13",
+ "skill_version": "1.12.0",
+ "agent_model": "claude-sonnet (1 run per prompt, 12-14 in parallel)",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 50,
+ "prompts": 50,
+ "repeats": 1,
+ "harness_note": "First iteration on the 50-prompt set: the 36 stable prompts plus 14 multilingual ones (two per language for zh, ko, es, pt, fr, de, ar; cue-file fixtures written into OUTDIR by evals/write_fixtures.py). Same prompt boilerplate as iteration 12. ~/.claude/skills/ffmpeg-skill is now a symlink to the repo, so the 5 runs that used that path executed HEAD. Focused independent grader (18 runs fully, all 14 multilingual ones included; language and labels on all 50).",
+ "purpose": "Verify 1.12.0 on the request that motivated it (captions and on-screen text in Chinese, Korean and other languages render with a real font, reports come back in the user's language) and keep the stable 36 at iteration-12 levels.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "39-set (24 stable + 14 multilingual + the exec split)": {
+ "routing_act_prompts": "100% over 27",
+ "mean_commands": "4.3",
+ "refusal_honesty": "12/12",
+ "language_by_prompt": "ja 9/9, zh 2/2, ko 2/2, es 2/2, pt 2/2, fr 2/2, de 2/2, ar 2/2",
+ "report_format": "39/39",
+ "visual_check_when_picture_changed": "13/13"
+ },
+ "exec-set": {
+ "routing_act_prompts": "100% over 10",
+ "mean_commands": "2.8",
+ "real_execution": "6/6",
+ "honest_failure_reporting": "5/5, false success 0",
+ "audio_only_handled_as_audio": "3/3"
+ }
+ },
+ "fonts_by_script_in_the_wild": {
+ "c01 (zh cues)": "font: wqy-zenhei.ttc (covers zh)",
+ "k01 (ko cues)": "font: wqy-zenhei.ttc (covers ko)",
+ "a01 (ar cues)": "font: DejaVuSans.ttf (covers ar)",
+ "j03 (ja lower-third)": "agent passed --lang ja; IPAPGothic",
+ "note": "no agent had to look a font up by hand (iteration 11/12's j03 ran fc-list itself); no tofu in any sheet the independent grader opened"
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-13-it13.json",
+ "judge": "claude-sonnet, one batch of 29 (22 + 7 multilingual)",
+ "should_trigger": "16/16",
+ "should_not_trigger": "13/13",
+ "note": "the 7 multilingual queries route on file extensions and cues the description names, so the description needed no change"
+ },
+ "independent_grader": {
+ "file": "evals/results/iteration-13-independent-grades.json",
+ "model": "claude-opus, focused pass (18 runs fully: all 14 multilingual + e01, j03, j08, x03; language and labels on all 50)",
+ "summary": {
+ "runs": 50,
+ "runs_graded_fully": 18,
+ "routing": "18/18 (graded set)",
+ "honest": "18/18 (graded set)",
+ "language": "50/50 (light check over all runs; ja zh ko es pt fr de ar en, no drift)",
+ "format": "50/50",
+ "look_when_picture_changed": "9/9 (graded set)",
+ "quality_mean": 4.83,
+ "quality_min": 4
+ },
+ "fonts_and_glyphs": "every PNG opened: c01 zh, k01 ko, a01 ar (shaped, RTL), j03 ja rendered real glyphs with the font the run.md names; no tofu; e01 and j08 show two-line wrapped cues inside the safe area",
+ "disagreement_with_regex": "none; the grader docked d01 for running through the ~/.claude/skills path and j08 for choosing --fit pad on a Reels deliverable (disclosed)"
+ },
+ "tokens_per_run (subagent_tokens)": {
+ "stable_36 at 1.12.0": {
+ "runs": 36,
+ "mean": 72337,
+ "median": 71800,
+ "min": 58712,
+ "max": 105509
+ },
+ "stable_36 at 1.11.1 (iteration 12)": {
+ "runs": 36,
+ "mean": 71818,
+ "median": 69884,
+ "min": 60875,
+ "max": 104200
+ },
+ "multilingual_14 at 1.12.0": {
+ "runs": 14,
+ "mean": 71209,
+ "median": 69409,
+ "min": 65151,
+ "max": 80600
+ },
+ "floor (refusal/failure prompts, all languages)": 66999,
+ "per_prompt": {
+ "a01": 74137,
+ "a02": 68266,
+ "c01": 74533,
+ "c02": 67431,
+ "d01": 69527,
+ "d02": 67701,
+ "e01-reel": 83184,
+ "e02-cut-lossless": 71387,
+ "e03-logo": 74444,
+ "e04-sync": 76392,
+ "e05-lufs": 73984,
+ "e06-silence": 74604,
+ "e07-hdr": 72170,
+ "e08-surround": 72610,
+ "e09-join": 71903,
+ "e10-highlights": 70510,
+ "e11-ready": 73463,
+ "e12-vfr": 71696,
+ "f01-missing-input": 58712,
+ "f02-corrupt-input": 66608,
+ "f03-ffmpeg-failure": 67436,
+ "f04-beyond-duration": 67017,
+ "f05-unsupported": 67464,
+ "fr1": 75365,
+ "fr2": 66732,
+ "j01-reel": 80732,
+ "j02-60s": 74151,
+ "j03-lower": 78639,
+ "j04-bgm": 73100,
+ "j05-ma": 71611,
+ "j06-report": 76477,
+ "j07-probe": 68473,
+ "j08-project": 105509,
+ "k01": 80600,
+ "k02": 65151,
+ "p01": 76053,
+ "p02": 73626,
+ "r01-face": 68836,
+ "r02-tts": 66906,
+ "r03-download": 66551,
+ "r04-translate": 69822,
+ "r05-no-raw-ffmpeg": 62211,
+ "s01": 69291,
+ "s02": 68511,
+ "x01-lufs-wav": 68989,
+ "x02-wav-to-mp3": 68977,
+ "x03-audio-trim": 70609,
+ "x04-transcode": 73125,
+ "x05-replace-audio": 73723,
+ "x06-reverse": 72117
+ },
+ "what_the_transcripts_show (tool calls)": {
+ "ran _contract.py doctor": "4/50",
+ "used --json-brief": "32/50",
+ "read references/gotchas.md": "0/50",
+ "read references/scripts.md": "1/50",
+ "tool uses per run": "7.2 (7.4 in iteration 12)"
+ },
+ "verdict": "Flat against iteration 12 on the stable set (72.3k vs 71.8k); the multilingual prompts cost the same as the English/Japanese ones (71.2k mean), i.e. the new font logic adds no calls. Reporting (total, floor, above-floor) as iteration 12 asked: floor ~66k, above-floor ~6k on a typical act prompt."
+ },
+ "comparison_with_iteration_12": {
+ "routing": "50/50 (was 36/36)",
+ "honest": "50/50, false success 0, raw ffmpeg 0",
+ "format": "50/50 by the regex grader",
+ "language": "50/50 by the regex grader, now across 9 languages (was 36/36 across en/ja)",
+ "look": "13/13 by regex (was 8/8)",
+ "tokens_mean_stable_36": "72,337 (was 71,818)",
+ "trigger": "29/29 (was 22/22 on the 22-query set)"
+ },
+ "findings": [
+ "Fonts by script works end to end on the request that motivated it: Chinese, Korean, Arabic and Japanese text rendered with a covering font picked automatically, reports in the user's language, 14/14 multilingual prompts routed and honest.",
+ "The stray .srt next to a --text source (iterations 11-12) did not recur: e01's sidecars land in OUTDIR; no run wrote outside its OUTDIR.",
+ "4 runs invoked scripts through ~/.claude/skills/ffmpeg-skill (d01, e12, j01, x02); since that path is now a symlink to the repo they executed HEAD, so this is a logging quirk, not a wrong tree.",
+ "doctor appeared in 2 runs (d02, p02), both to prove external:whisper is missing before refusing a translate request: a legitimate use, not the per-job habit 1.11.1 removed.",
+ "j08-project padded 16:9 into 9:16 (black bars) instead of cropping, with the choice disclosed and the project.json edit to switch given; a judgement call the skill leaves to the agent."
+ ],
+ "harness_or_model_issues": [
+ "One pass per prompt; the 14 multilingual prompts have a single run each, so per-language language-drift rates need the next iteration to confirm."
+ ],
+ "follow_ups": [
+ "1.13.0 (roadmap): the audio bed — audio.py --voice levels, --stereo-widen, stems in a render.py project, exposed ducking parameters; loudness.py --lra / --dialogue; check.py podcast chapters and mono/stereo rows; audio.py --chapters.",
+ "evals: keep the 50-prompt set; three repeats when a claim about a single language is the point."
+ ]
+}
\ No newline at end of file
diff --git a/evals/trigger/results-2026-09-13-it13.json b/evals/trigger/results-2026-09-13-it13.json
new file mode 100644
index 0000000..295c0e0
--- /dev/null
+++ b/evals/trigger/results-2026-09-13-it13.json
@@ -0,0 +1,42 @@
+{
+ "date": "2026-09-13",
+ "skill_version": "1.12.0",
+ "iteration": 13,
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 29; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},
+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},
+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},
+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},
+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},
+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},
+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},
+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},
+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},
+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},
+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency, not video/audio"},
+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question, not an editing task on a file"},
+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading from the web, not local editing"},
+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file to process"},
+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},
+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},
+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media involved"},
+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},
+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage, not editing an existing local file"},
+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},
+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},
+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"},
+ {"id": "t13","lang": "zh","prompt": "这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "mp4 file, trim to 60 seconds, add subtitles — matches file extension, duration and captions cues even though the request is in Chinese"},
+ {"id": "t14","lang": "ko","prompt": "인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "mov file, vertical reframe for Reels — description explicitly lists Reels delivery and vertical reframing"},
+ {"id": "t15","lang": "es","prompt": "El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "mp4 file with explicit LUFS target — LUFS is called out verbatim in the description"},
+ {"id": "t16","lang": "ar","prompt": "هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "m4a recording, cutting silent segments — silence removal on an audio file"},
+ {"id": "n11","lang": "pt","prompt": "Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a launch-video script, no media file to process — same shape as n04"},
+ {"id": "n12","lang": "fr","prompt": "Comment installer ffmpeg sur macOS avec Homebrew ?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question naming ffmpeg itself, not an editing task on a file — same shape as n02"},
+ {"id": "n13","lang": "de","prompt": "Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage from nothing, not editing an existing local file — same shape as n09"}
+ ],
+ "summary": {
+ "should_trigger": "16/16",
+ "should_not_trigger": "13/13",
+ "total": "29/29"
+ }
+}