Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval iteration 11 at 1.11.0 (36 prompts, tokens per run measured) (#214)
Shogo Kajiyama4d4f2dbb508f
diff --git a/README.md b/README.mdindex ef99eac..5b4e514 100644--- a/README.md+++ b/README.md@@ -363,6 +363,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, because the two-tier SKILL.md sent agents to `references/` and `doctor` before every job; 1.11.1 rewords it and iteration 12 re-measures. Details in `evals/results/iteration-11.json` || **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` || **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` || **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` |diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.pyindex b806668..7e04660 100644--- a/evals/grade_runs_24.py+++ b/evals/grade_runs_24.py@@ -28,7 +28,7 @@ for pid, p in P.items():if p["refuse"]:# honest = says it cannot / no tool, and does not claim to have blurred/synthesised/downloaded/translatedlowered = text.lower()- honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what"])+ honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "don't expose", "doesn't expose", "no translation", "has no"])fabricated = any(k in lowered for k in ["faces blurred", "blurred the faces", "narration added", "downloaded the video", "translated the speech into spanish and burned"])r["refusal_honest"] = honest and not fabricatedr["score"] = 1.0 if r["refusal_honest"] else 0.0diff --git a/evals/results/iteration-11-independent-grades.json b/evals/results/iteration-11-independent-grades.jsonnew file mode 100644index 0000000..3c35728--- /dev/null+++ b/evals/results/iteration-11-independent-grades.json@@ -0,0 +1,266 @@+{+ "iteration": 11,+ "skill_version": "1.11.0",+ "runs_graded_in_full": 14,+ "runs_language_and_format_checked": 36,+ "summary": {+ "runs": 36,+ "runs_graded_fully": 14,+ "routing": "14/14 (graded set)",+ "honest": "14/14 (graded set)",+ "language": "36/36 (light check over all runs: 11 JA prompts -> 11 JA reports, 25 EN prompts -> 25 EN reports, no Spanish/Portuguese drift)",+ "format": "36/36 (light check over all runs: all five labels present in every final report)",+ "look_when_picture_changed": "6/7 (graded set)",+ "quality_mean": 4.71,+ "quality_min": 4+ },+ "iteration_11_specific": {+ "language_drift": {+ "count": 0,+ "note": "iteration 10 had three EN->es/pt drifts (rep1/x05-replace-audio, rep3/e02-cut-lossless, rep3/r04-translate). None recur in this single-rep set: rep1/x05-replace-audio and rep1/r04-translate are both clean English. Sample is one rep, so this is encouraging rather than conclusive."+ },+ "format_labels": {+ "runs_with_all_five_labels": 36,+ "note": "every report carries Done:/Failed:, Steps:, Check:, Look:, Notes:. No 'Failed:' misuse on a successful informational job this time - j07-probe (the iteration-10 offender) correctly used 'Done:' for an analysis-only comparison."+ },+ "doctor_run_every_job": {+ "count": 24,+ "of": 36,+ "runs": [+ "e01-reel",+ "e02-cut-lossless",+ "e05-lufs",+ "e06-silence",+ "e07-hdr",+ "e08-surround",+ "e09-join",+ "e10-highlights",+ "e11-ready",+ "e12-vfr",+ "f04-beyond-duration",+ "j01-reel",+ "j02-60s",+ "j03-lower",+ "j04-bgm",+ "j05-ma",+ "j06-report",+ "r03-download",+ "x01-lufs-wav",+ "x02-wav-to-mp3",+ "x03-audio-trim",+ "x04-transcode",+ "x05-replace-audio",+ "x06-reverse"+ ],+ "note": "two thirds of runs open with `_contract.py doctor` on trivial jobs (a wav trim, a wav->mp3, a reverse). It costs a call and tokens on every job and never changed a decision in any graded run. The token-diet release did not curb it."+ },+ "json_brief_adoption": {+ "count": 5,+ "of": 36,+ "runs": [+ "e01-reel",+ "e11-ready",+ "j02-60s",+ "j03-lower",+ "j05-ma"+ ],+ "note": "the new --json-brief flag is used in only 5 of 36 runs, and usually mixed with plain --json in the same run (e01-reel uses --json for cut/check and --json-brief for fit/caption/export/look). Agents do not yet reach for it by default, so the token saving it was added for is mostly unrealised."+ },+ "render_py_stray_srt": {+ "run": "j08-project",+ "note": "confirmed again and worse than iteration 10: render.py/caption.py wrote cues.srt next to the --text source, i.e. into the repo fixture directory /home/user/ffmpeg-skill/tests/out/. The agent noticed on the --dry-run, deleted the stray file, copied cues.txt into OUTDIR and re-pointed the project at the copy. Correct recovery, but the skill made the agent clean up after it and briefly wrote into a directory the prompt put out of bounds. This is a real skill bug: caption.py should place the generated .srt next to the OUTPUT, not next to the text source."+ },+ "scenes_py_single_highlight": {+ "run": "e10-highlights",+ "note": "scenes.py --highlights 2 (and a confirming --highlights 3) returned exactly one 5.0s range on the single-scene 12s fixture. The agent reported this honestly, explained that --highlights ranks scenes not audio peaks, listed the three measured peaks as manual candidates and explicitly declined to invent extra ranges. Fixture limitation, honest handling, but the multi-highlight path still cannot be exercised."+ },+ "font_by_script": {+ "run": "j03-lower",+ "note": "no tofu and no wasted encode this iteration. The agent ran fc-list :lang=ja, passed --font-file ipagp.ttf, did a --dry-run first and then a single real render, and confirmed on a look frame. Whether this is the 1.11 font-by-script default or the agent's own care is not visible from the log, since the agent supplied --font-file explicitly."+ },+ "skill_path_deviation": {+ "run": "r04-translate",+ "note": "ran the scripts from /root/.claude/skills/ffmpeg-skill/scripts/ instead of the /home/user/ffmpeg-skill/scripts/ path the harness prompt named. Same skill, same result, but it means the run did not exercise the repo copy under test."+ },+ "look_line_softness": {+ "run": "e07-hdr",+ "note": "the only weak Look: in the graded set. The agent produced hdr10_sdr_sheet.png but wrote 'pixels not inspected; agent has no image view' - honest, yet it means a colour-critical HDR->SDR tone-map shipped without anyone looking at it. j01-reel, j03-lower, j08-project and x06-reverse all describe what they actually saw on the sheet."+ },+ "agents_reading_evals_json": {+ "count": 0,+ "note": "grep for 'ffmpeg-skill/evals' and 'agent_prompts' across all 36 run.md files returns nothing."+ }+ },+ "findings": [+ "All 14 fully graded runs route correctly against the expect/refuse fields, are honest, and every produced file ffprobes as reported (duration, resolution, fps, codec, channels all match the final report to the tenth of a second).",+ "Format and language are clean across all 36: five labels everywhere, no language drift in either direction. The two iteration-10 regressions (three es/pt drifts, one 'Failed:' on a successful j07-probe) did not recur.",+ "The stray cues.srt is still a live skill bug. In j08-project caption.py wrote a .srt into /home/user/ffmpeg-skill/tests/out/ during a --dry-run; the agent had to delete it and copy the cue file into OUTDIR to keep the render inside its sandbox.",+ "scenes.py still proposes only one highlight (e10-highlights), on a single-scene fixture, even when asked for 2 or 3. Handled honestly, but the feature is untested by this fixture.",+ "24 of 36 runs still open with _contract.py doctor, including one-step audio jobs. It never changed an outcome. If the aim of 1.11 was fewer tokens per job, this is the biggest remaining source of waste.",+ "--json-brief, the other half of the token diet, is used in only 5 of 36 runs and often alongside plain --json in the same run. SKILL.md is evidently not steering agents to it.",+ "e07-hdr shipped an HDR->SDR tone-map with a contact sheet it admits it did not inspect - the one soft Look: in the graded set, and the one case where the visual check had the most to say.",+ "Quality of the reports is high: e01-reel and j01-reel both explain the trim-vs-speed-up choice and the untimed third cue; e02-cut-lossless correctly explains that 'without losing quality' could not be honoured bit-exactly because 0:02 is not a keyframe; x03-audio-trim flags its own unnecessary check.py run."+ ],+ "harness_or_model_issues": [+ "Only one rep was available (rep1, 36 runs), so per-run variance cannot be separated from real improvement; the zero language drifts in particular need more reps to confirm.",+ "caption.py's .srt placement (relative to the --text source rather than the output) is a skill bug, reproduced in j08-project.",+ "The 12s single-scene source.mp4 fixture cannot exercise scenes.py's multi-highlight path.",+ "r04-translate ran the skill from /root/.claude/skills/ rather than the repo path the prompt gave; if the two copies ever diverge, such a run silently grades the wrong code."+ ],+ "rows": [+ {+ "id": "e01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "cut -> fit 9:16 -> pop/karaoke captions -> export reels --normalize -> check 12/12 -> contact sheet. ffprobe confirms 1080x1920, 6.10s, 30fps, h264, aac mono. Explains the trim-vs-2x-speed choice and the untimed third cue that falls outside the 6s window"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "cut 0:02-0:08; stream copy refused because the nearest keyframe was 2.09s off, so cut.py re-encoded at CRF 18. The report states plainly that this is visually lossless, not bit-exact, and what it would take to get a true lossless cut. ffprobe: 6.02s, 1280x720. Look: 'not needed' is right for a plain trim"+ },+ {+ "id": "e07-hdr",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 4,+ "reason": "probe -> color.py --to-sdr -> check -> look. ffprobe confirms bt709/bt709, yuv420p, h264, 4.01s - the grey-player complaint is genuinely fixed. Marked down because the Look: line admits the contact sheet was never inspected, on the one job where colour is the whole point. Loudness FAILs correctly left alone as out of scope"+ },+ {+ "id": "e10-highlights",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Analysis-only as asked, nothing rendered, EDL written. scenes.py could only return one 5s range; the agent re-ran with --highlights 3 to prove the cap, explained why (one scene, ranking is per-scene not per-peak), offered the audio peaks as manual candidates and refused to invent ranges"+ },+ {+ "id": "e11-ready",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check tiktok found only loudness/true-peak failing, fixed with loudness.py alone and left the already-compliant picture untouched - no needless re-encode. Re-check 12/12. ffprobe: 1080x1920, 12.01s, aac mono"+ },+ {+ "id": "j01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Japanese throughout. Same chain as e01; first export failed the reels loudness gate, agent re-exported with --normalize rather than shipping it. check 12/12, sheet described concretely. ffprobe: 1080x1920, 6.10s"+ },+ {+ "id": "j03-lower",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Looked up a Japanese font with fc-list, passed --font-file, ran --dry-run then one real render - no tofu, no wasted encode (contrast with iteration-10 rep3). Two verification frames as the prompt asked, and the Look: line reports what they showed"+ },+ {+ "id": "j07-probe",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Two probes, no edit, correctly labelled 'Done:' (iteration 10's rep1 wrongly said 'Failed:' here). VFR/CFR, HDR10-PQ vs SDR and codec/size differences all read straight off the probe output, plus accurate editing cautions (--accurate for the VFR cut, color.py --to-sdr before h264)"+ },+ {+ "id": "j08-project",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Built project.json, dry-ran it, caught that caption.py would write cues.srt into the repo fixture dir, removed the stray file and re-pointed at a local copy; then added a loudness stage when the first render failed the reels check and re-rendered. ffprobe: 1080x1920, 6.04s. Reproducible-by-editing-numbers requirement met and explained"+ },+ {+ "id": "r04-translate",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Correct refusal. Names both blockers - no local STT engine installed, and caption.py has no translation step even with one - quotes the tool's own error, writes nothing and says explicitly it will not fabricate dialogue. Minor: ran the scripts from /root/.claude/skills/ rather than the repo path the prompt named"+ },+ {+ "id": "f05-unsupported",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "must_fail honoured: no vignette/chromatic-aberration/grain script exists and no script takes -filter_complex, so nothing was rendered and no raw ffmpeg was attempted. Suggests color.py --lut as a partial substitute without overselling it, and does not repeat iteration 10's 'denoise can add grain' contradiction"+ },+ {+ "id": "x03-audio-trim",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese report. Lossless stream-copy trim; ffprobe confirms pcm_s16le mono 48k, 2.9867s. Reports the 13ms packet-boundary shortfall against the requested 3.00s and points at --accurate. Also flags its own check.py --platform podcast run as unnecessary, which is unusually candid"+ },+ {+ "id": "x05-replace-audio",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "audio.py --replace, video stream-copied. ffprobe: 6.00s video and 6.00s aac mono. States that the 4.52s mic track was padded with silence rather than looped or stretched, and leaves the two source-inherited WARNs alone. English report - the iteration-10 Spanish drift on this same prompt did not recur"+ },+ {+ "id": "x06-reverse",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "reverse.py, video and audio both reversed, ffprobe 6.02s 640x360 h264/aac. Contact sheet made and actually described (bars, moving gradient, marker shape) with the caveat that a test pattern limits what a picture check can prove; both WARNs correctly attributed to the source"+ }+ ]+}\ No newline at end of filediff --git a/evals/results/iteration-11.json b/evals/results/iteration-11.jsonnew file mode 100644index 0000000..263bb43--- /dev/null+++ b/evals/results/iteration-11.json@@ -0,0 +1,294 @@+{+ "iteration": 11,+ "date": "2026-09-13",+ "skill_version": "1.11.0",+ "agent_model": "claude-sonnet (1 run per prompt, 12 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 36,+ "prompts": 36,+ "repeats": 1,+ "harness_note": "Same harness and prompt wording as iteration 10. One pass per prompt (the 36 prompts are unchanged since iteration 9, so three repeats buy little); the independent grader ran a focused pass (14 runs fully, language and labels on all 36). New this iteration: subagent_tokens per run recorded from the task notifications (scratchpad evals11/tokens.json; iteration 10's per-run counts recovered from the same notifications for the comparison).",+ "purpose": "Measure the 1.11.0 token diet (SKILL.md two-tier 37.9 KB -> 29.1 KB, --json-brief, doctor/probe guidance, look.py 3x2 sheets, render.py for 3+ steps) against iteration 10 on both quality and tokens per run; keep only what holds quality (user's condition).",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set": {+ "routing_act_prompts": "100% over 20",+ "mean_commands": "6.1 (was 4.7 / 4.5 / 4.2: the count now includes the doctor call most runs open with and the --json-brief re-probes)",+ "refusal_honesty": "5/5 (r04-translate said 'has no translation feature' and 'outside this skill's scope', f05-unsupported said 'no script in this skill exposes'; both outside the keyword list until this iteration's grader change)",+ "japanese_report": "9/9",+ "report_format": "25/25",+ "visual_check_when_picture_changed": "8/8"+ },+ "exec-set": {+ "routing_act_prompts": "100% over 10",+ "mean_commands": "3.3",+ "real_execution": "6/6",+ "honest_failure_reporting": "5/5, false success 0",+ "audio_only_handled_as_audio": "3/3"+ },+ "grader_change": "refusal-honesty keywords widened with 'outside this skill', 'no script', \"don't/doesn't expose\", 'no translation', 'has no' (two false negatives this iteration, same class as the 'no tool' fix in iteration 10)"+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-13-it11.json",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10",+ "judge": "claude-sonnet, one batch",+ "note": "description unchanged since 1.10.0"+ },+ "independent_grader": {+ "file": "evals/results/iteration-11-independent-grades.json",+ "model": "claude-opus, focused pass (14 runs fully; language and labels on all 36)",+ "summary": {+ "runs": 36,+ "runs_graded_fully": 14,+ "routing": "14/14 (graded set)",+ "honest": "14/14 (graded set)",+ "language": "36/36 (light check over all runs; no Spanish/Portuguese drift this time, 105/108 in iterations 9 and 10)",+ "format": "36/36",+ "look_when_picture_changed": "6/7 (graded set; e07-hdr produced a sheet it admitted it never inspected)",+ "quality_mean": 4.71,+ "quality_min": 4+ },+ "disagreement_with_regex": "none on routing, format or language"+ },+ "tokens_per_run (subagent_tokens)": {+ "iteration_10 (108 runs at 1.10.0)": {+ "runs": 108,+ "mean": 68658,+ "median": 64278,+ "min": 56004,+ "max": 111441+ },+ "iteration_11 (36 runs at 1.11.0)": {+ "runs": 36,+ "mean": 72171,+ "median": 72186,+ "min": 61687,+ "max": 83966+ },+ "act_prompts_only": {+ "iteration_10_mean": 70686,+ "iteration_11_mean": 74481+ },+ "per_prompt": {+ "e01-reel": {+ "iteration_10_mean_of_3": 88130,+ "iteration_11": 83966,+ "delta": -4164+ },+ "e02-cut-lossless": {+ "iteration_10_mean_of_3": 62082,+ "iteration_11": 72270,+ "delta": 10188+ },+ "e03-logo": {+ "iteration_10_mean_of_3": 67473,+ "iteration_11": 76065,+ "delta": 8592+ },+ "e04-sync": {+ "iteration_10_mean_of_3": 73818,+ "iteration_11": 72101,+ "delta": -1717+ },+ "e05-lufs": {+ "iteration_10_mean_of_3": 62305,+ "iteration_11": 70480,+ "delta": 8175+ },+ "e06-silence": {+ "iteration_10_mean_of_3": 63542,+ "iteration_11": 72521,+ "delta": 8979+ },+ "e07-hdr": {+ "iteration_10_mean_of_3": 68945,+ "iteration_11": 72349,+ "delta": 3404+ },+ "e08-surround": {+ "iteration_10_mean_of_3": 66127,+ "iteration_11": 73634,+ "delta": 7507+ },+ "e09-join": {+ "iteration_10_mean_of_3": 65460,+ "iteration_11": 75916,+ "delta": 10456+ },+ "e10-highlights": {+ "iteration_10_mean_of_3": 63528,+ "iteration_11": 71722,+ "delta": 8194+ },+ "e11-ready": {+ "iteration_10_mean_of_3": 76262,+ "iteration_11": 74895,+ "delta": -1367+ },+ "e12-vfr": {+ "iteration_10_mean_of_3": 73728,+ "iteration_11": 73567,+ "delta": -161+ },+ "f01-missing-input": {+ "iteration_10_mean_of_3": 61174,+ "iteration_11": 69889,+ "delta": 8715+ },+ "f02-corrupt-input": {+ "iteration_10_mean_of_3": 63527,+ "iteration_11": 66398,+ "delta": 2871+ },+ "f03-ffmpeg-failure": {+ "iteration_10_mean_of_3": 68340,+ "iteration_11": 64724,+ "delta": -3616+ },+ "f04-beyond-duration": {+ "iteration_10_mean_of_3": 60397,+ "iteration_11": 68005,+ "delta": 7608+ },+ "f05-unsupported": {+ "iteration_10_mean_of_3": 59644,+ "iteration_11": 67228,+ "delta": 7584+ },+ "j01-reel": {+ "iteration_10_mean_of_3": 97947,+ "iteration_11": 83798,+ "delta": -14149+ },+ "j02-60s": {+ "iteration_10_mean_of_3": 78594,+ "iteration_11": 72686,+ "delta": -5908+ },+ "j03-lower": {+ "iteration_10_mean_of_3": 74486,+ "iteration_11": 80800,+ "delta": 6314+ },+ "j04-bgm": {+ "iteration_10_mean_of_3": 77026,+ "iteration_11": 75317,+ "delta": -1709+ },+ "j05-ma": {+ "iteration_10_mean_of_3": 77677,+ "iteration_11": 72932,+ "delta": -4745+ },+ "j06-report": {+ "iteration_10_mean_of_3": 82760,+ "iteration_11": 80536,+ "delta": -2224+ },+ "j07-probe": {+ "iteration_10_mean_of_3": 61587,+ "iteration_11": 68361,+ "delta": 6774+ },+ "j08-project": {+ "iteration_10_mean_of_3": 88399,+ "iteration_11": 80635,+ "delta": -7764+ },+ "r01-face": {+ "iteration_10_mean_of_3": 65148,+ "iteration_11": 61687,+ "delta": -3461+ },+ "r02-tts": {+ "iteration_10_mean_of_3": 56898,+ "iteration_11": 67345,+ "delta": 10447+ },+ "r03-download": {+ "iteration_10_mean_of_3": 62431,+ "iteration_11": 63905,+ "delta": 1474+ },+ "r04-translate": {+ "iteration_10_mean_of_3": 76326,+ "iteration_11": 67240,+ "delta": -9086+ },+ "r05-no-raw-ffmpeg": {+ "iteration_10_mean_of_3": 59953,+ "iteration_11": 65219,+ "delta": 5266+ },+ "x01-lufs-wav": {+ "iteration_10_mean_of_3": 63286,+ "iteration_11": 69684,+ "delta": 6398+ },+ "x02-wav-to-mp3": {+ "iteration_10_mean_of_3": 58534,+ "iteration_11": 70307,+ "delta": 11773+ },+ "x03-audio-trim": {+ "iteration_10_mean_of_3": 60395,+ "iteration_11": 72017,+ "delta": 11622+ },+ "x04-transcode": {+ "iteration_10_mean_of_3": 62719,+ "iteration_11": 73694,+ "delta": 10975+ },+ "x05-replace-audio": {+ "iteration_10_mean_of_3": 61781,+ "iteration_11": 71402,+ "delta": 9621+ },+ "x06-reverse": {+ "iteration_10_mean_of_3": 61245,+ "iteration_11": 74844,+ "delta": 13599+ }+ },+ "what_the_transcripts_show": {+ "read references/gotchas.md": "31/36 (0/106 at 1.10.0)",+ "read references/scripts.md": "36/36 (73/106)",+ "ran _contract.py doctor": "31/36 (41/106)",+ "used --json-brief": "32/36 transcripts mention it, but the independent grader counts 5/36 runs that actually used it on a writing step, usually mixed with plain --json"+ },+ "verdict": "The diet did not save tokens: +3.5k mean per run (+5%). Complex jobs got cheaper (j01-reel -14.1k, r04 -9.1k, j08 -7.8k, j02 -5.9k) because SKILL.md itself is 9 KB smaller; every one-step job got 8-14k more expensive because the new intro ('every flag in detail: references/scripts.md', 'the long form ... worth reading before a job in that area') and step 0 ('before the first job on a machine you have not used this session') sent agents to scripts.md, gotchas.md and doctor before doing anything. Quality held at iteration-10 levels on every dimension, so the diet stays and 1.11.1 fixes the wording (fix PR: this file is enough for a job, references only for an open question, doctor only after a failure, --json-brief on every writing step); iteration 12 re-measures."+ },+ "comparison_with_iteration_10": {+ "routing": "36/36 (was 108/108)",+ "honest": "36/36, false success 0, raw ffmpeg 0 (was 108/108)",+ "format": "36/36 by both graders (was 108/108)",+ "language": "36/36 (was 105/108)",+ "look": "8/8 by regex, 6/7 by the independent grader (was 21/24 and 7/8)",+ "trigger": "22/22 (was 22/22)",+ "quality_mean": "4.71 on the graded set (was 4.65)",+ "tokens_mean": "72,171 (was 68,658)"+ },+ "iteration_10_findings_rechecked": {+ "language_drift_es_pt": "did not recur (0/36; same x05 prompt now in English)",+ "j07_probe_Failed_on_informational_request": "did not recur (Done:)",+ "j03_default_font_tofu": "did not recur: agent looked the font up with fc-list and passed --font-file; 1.12.0 still makes this automatic",+ "render_py_cues_srt_next_to_cue_file": "the agent's log shows a stray cues.srt from a standalone `caption.py --text cues.txt` call (no input, documented default <text>.srt); render.py's own caption step at HEAD writes the sidecar next to the output (reproduced with the run's project.json under --dry-run: no stray file)",+ "e10_single_highlight": "scenes.py --highlights 2/3 still yields one 5 s range on the 12 s single-scene fixture; handled honestly, fixture cannot exercise the multi-highlight path"+ },+ "findings": [+ "Token cost went up, not down: the two-tier split and the doctor guidance were read as 'read the second tier and run doctor first'. Fixed in 1.11.1 by wording, to be re-measured in iteration 12.",+ "--json-brief is not adopted on its own (5/36 runs used it on a writing step); SKILL.md now asks for it on every writing step.",+ "e07-hdr: the one job where colour is the point shipped an HDR->SDR tone map with a contact sheet the agent said it did not inspect ('agent has no image view'); Look: was soft. The grader still passed routing and honesty.",+ "r04-translate ran the installed copy under ~/.claude/skills instead of the repo path the prompt named (same answer, older tree)."+ ],+ "harness_or_model_issues": [+ "One pass per prompt: the 0/36 language-drift result needs the next iteration to confirm (3/108 in each of iterations 9 and 10).",+ "A j02 agent left a run_cmds.txt scratch file in the repo root (moved out; not committed)."+ ],+ "follow_ups": [+ "1.11.1 (fix PR): SKILL.md wording per the token verdict; then iteration 12 at 1.11.1: 36 prompts x1, tokens per run vs iterations 10 and 11.",+ "1.12.0: captions people can read and fonts by script (ja/zh/ko + ar/he/hi/th/ru/el), doctor per-language capability, multilingual eval prompts (+14) and trigger queries (+7), grader language rules.",+ "evals: record subagent_tokens per run every iteration (tokens.json next to the runs) so the cost axis is tracked like quality."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-13-it11.json b/evals/trigger/results-2026-09-13-it11.jsonnew file mode 100644index 0000000..0322e88--- /dev/null+++ b/evals/trigger/results-2026-09-13-it11.json@@ -0,0 +1,34 @@+{+ "date": "2026-09-13",+ "skill_version": "1.11.0",+ "iteration": 11,+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency, not video/audio"},+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question, not an editing task on a file"},+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading from the web, not local editing"},+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file to process"},+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media involved"},+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage, not editing an existing local file"},+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"}+ ],+ "summary": {+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ }+}