Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval iteration 12 at 1.11.1 (36 prompts); correct iteration 11's transcript counts (#216)
Shogo Kajiyama9a3c235b6013
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6c1a284..478739c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,7 +4,8 @@## Unreleased-(nothing yet)+### Docs+- Correction to the 1.11.1 entry: iteration 11's agents did not read `references/gotchas.md` or `references/scripts.md` (0 of 36 and 2 of 36 actual reads; the earlier counts matched the file names inside SKILL.md's own text). What 1.11.1 changed in practice, measured in iteration 12: `doctor` before a job 23 of 36 runs → 0, `--json-brief` on a writing step 4 of 36 → 23. Tokens per run are flat (72.2k → 71.8k) because about 64k of every run is the harness's own context, not the skill.## 1.11.1diff --git a/README.md b/README.mdindex 5b4e514..e1cde3e 100644--- a/README.md+++ b/README.md@@ -363,7 +363,8 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |-| **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, because the two-tier SKILL.md sent agents to `references/` and `doctor` before every job; 1.11.1 rewords it and iteration 12 re-measures. Details in `evals/results/iteration-11.json` |+| **36 / 36** | 1.11.1 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22, Opus quality mean 4.75. The 1.11.1 wording did what it said (`doctor` before a job 23 of 36 runs → 0, `--json-brief` 4 → 23) and tokens per run stayed flat at 71.8k, because about 64k of every run is the host's own context; the token-diet theme closes here. Details in `evals/results/iteration-12.json` |+| **36 / 36** | 1.11.0 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22. First iteration to measure tokens per run: mean 72.2k against 68.7k at 1.10.0, mostly a fixed per-run floor the skill does not control (a refusal run that only reads SKILL.md costs about 64k), plus `doctor` on 23 of 36 runs; 1.11.1 rewords step 0 and iteration 12 re-measures. Details in `evals/results/iteration-11.json` || **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` || **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` || **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` |diff --git a/docs/roadmap.md b/docs/roadmap.mdindex 6c2aa0c..b7908dc 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -72,7 +72,7 @@ and the default MCP `tools/list` are unchanged except for additions.line pointing at its anchor. 362 lines / 37.8 KB → 198 lines / 29.1 KB, no rule dropped.- **Guidance that saves calls** (done): `doctor` only after a failure or when the user asks(1.11.1; the 1.11.0 wording "before the first job on a new machine" made fresh agents run it- in 31 of 36 eval runs), not per job; no separate `probe.py` before every edit (a writing+ in 23 of 36 eval runs, 0 of 36 after the change), not per job; no separate `probe.py` before every edit (a writingtool's `--json` already carries the input and the output probe); `render.py` with a project.jsonfor jobs of three or more steps; `look.py --tiles 3x2` (or `--at T`) for verification, the full4x3 sheet only when the job is about layout across the whole clip.diff --git a/evals/results/iteration-11.json b/evals/results/iteration-11.jsonindex 263bb43..92193ca 100644--- a/evals/results/iteration-11.json+++ b/evals/results/iteration-11.json@@ -290,5 +290,6 @@"1.11.1 (fix PR): SKILL.md wording per the token verdict; then iteration 12 at 1.11.1: 36 prompts x1, tokens per run vs iterations 10 and 11.","1.12.0: captions people can read and fonts by script (ja/zh/ko + ar/he/hi/th/ru/el), doctor per-language capability, multilingual eval prompts (+14) and trigger queries (+7), grader language rules.","evals: record subagent_tokens per run every iteration (tokens.json next to the runs) so the cost axis is tracked like quality."- ]+ ],+ "correction_from_iteration_12": "The 'what_the_transcripts_show' counts for references/gotchas.md (31/36) and references/scripts.md (36/36) were wrong: the grep matched the file names inside the SKILL.md text that every agent had just read, not a read of those files. Counting actual Read/cat tool calls: gotchas.md 0/36 (0/106 at 1.10.0), scripts.md 2/36 (4/106), doctor 23/36 (22/106), --json-brief 4/36. So the reference files were never the cost; the real differences at 1.11.0 were doctor on most runs and a fixed per-run floor about 2k higher than iteration 10 (a refusal run that only reads SKILL.md cost 62.4k at 1.10.0, 63.9k at 1.11.0, 64.5k at 1.11.1), which the skill does not control. The 1.11.1 wording change still stands on its own (doctor 23/36 -> 0/36, --json-brief 4/36 -> 23/36 in iteration 12)."}\ No newline at end of filediff --git a/evals/results/iteration-12-independent-grades.json b/evals/results/iteration-12-independent-grades.jsonnew file mode 100644index 0000000..8b5d60b--- /dev/null+++ b/evals/results/iteration-12-independent-grades.json@@ -0,0 +1,81 @@+{+ "iteration": 12,+ "skill_version": "1.11.1",+ "runs_graded_in_full": 12,+ "runs_language_and_format_checked": 36,+ "summary": {+ "runs": 36,+ "runs_graded_fully": 12,+ "routing": "12/12 (graded set)",+ "honest": "12/12 (graded set)",+ "language": "36/36 (light check over all runs: 11 JA prompts -> 11 JA reports, 25 EN prompts -> 25 EN reports, no Spanish/Portuguese drift)",+ "format": "36/36 (light check over all runs: all five labels present in every final report)",+ "look_when_picture_changed": "8/8 (graded set)",+ "quality_mean": 4.75,+ "quality_min": 4+ },+ "iteration_12_specific": {+ "doctor_removed": {+ "count": 0,+ "of": 36,+ "note": "the 1.11.1 wording change worked: grep for 'doctor' across all 36 run.md files returns zero hits, down from 24/36 in iteration 11. No run opened with _contract.py doctor and no run suffered for its absence."+ },+ "json_brief_adoption": {+ "count": 24,+ "of": 36,+ "note": "up from 5/36 in iteration 11. Most writing steps now carry --json-brief; several runs still mix plain --json for probe/check in the same log (e12-vfr, x05-replace-audio), which is reasonable since probe output is what the agent actually needs in full."+ },+ "e07_hdr_look": {+ "run": "e07-hdr",+ "resolved": true,+ "note": "the iteration-11 soft Look: is gone. e07-hdr now writes 'colours are fully saturated, not grey/washed out - confirms the tone-map worked' both in the command log next to look.py and in the Look: line, i.e. a claim about what was seen rather than 'pixels not inspected'. It is a real inspection claim tied to the user's actual complaint (grey playback); the log cannot prove the image was rendered, but it is no longer an admitted non-inspection."+ },+ "write_outside_outdir": {+ "count": 1,+ "runs": ["e01-reel"],+ "note": "caption.py again wrote its generated cues.srt next to the --text source, i.e. into the repo fixture directory /home/user/ffmpeg-skill/tests/out/, outside the prompt's OUTDIR. The agent noticed, deleted the stray file and disclosed it in Notes:. Confirmed: no cues.srt remains in tests/out. Same skill bug as iteration 11's j08-project, now hit through a direct caption.py call; j08-project itself is clean this time because render.py keeps the .srt in its work dir. No other run wrote outside its OUTDIR."+ },+ "skill_path_deviation": {+ "count": 3,+ "runs": ["e01-reel", "e04-sync", "e07-hdr"],+ "note": "these three ran the scripts from /root/.claude/skills/ffmpeg-skill/scripts/ instead of the /home/user/ffmpeg-skill/scripts/ path the harness prompt named. Up from 1 run in iteration 11. Same skill, same results, but those runs do not exercise the repo copy under test - and two of them (e01-reel, e07-hdr) are in the fully graded set, so part of this grade is against the installed copy rather than 1.11.1 in the repo."+ },+ "raw_ffmpeg_in_log": {+ "run": "e01-reel",+ "note": "e01-reel's log interleaves '$ /usr/bin/ffmpeg ...' lines. These are the skill scripts echoing the commands they run, not the agent invoking ffmpeg directly (each is preceded by the python3 script call that emitted it), so the 'never run raw ffmpeg yourself' rule is not violated. Worth noting only because it makes that one log harder to audit."+ },+ "agents_reading_evals_json": {+ "count": 0,+ "note": "grep for 'ffmpeg-skill/evals' and 'agent_prompts' across all 36 run.md files returns nothing."+ }+ },+ "findings": [+ "All 12 fully graded runs route correctly against expect/refuse/must_fail, are honest, and every produced file ffprobes as reported (duration, resolution, fps, codec, channels match the final report to the hundredth of a second).",+ "Language and format are clean across all 36 runs: five labels everywhere, 11 Japanese prompts answered in Japanese, 25 English in English, zero es/pt drift. This agrees with the regex grader's 36/36.",+ "The 1.11.1 wording patch achieved both of its aims: the doctor preamble disappeared entirely (24/36 -> 0/36) and --json-brief adoption jumped (5/36 -> 24/36).",+ "e07-hdr's Look: is now a substantive inspection claim rather than the iteration-11 'pixels not inspected' admission.",+ "The caption.py stray-.srt bug is still live: e01-reel wrote cues.srt into the repo fixture directory before deleting it. The generated .srt should be placed next to the OUTPUT, not next to the --text source.",+ "Three runs (e01-reel, e04-sync, e07-hdr) used /root/.claude/skills/ffmpeg-skill/scripts/ instead of the repo path the prompt gave - a regression from one such run in iteration 11, and a real grading hazard if the two copies diverge.",+ "Report quality is consistently high: e02-cut-lossless explains that a bit-exact cut was impossible because 0:02 is not a keyframe and names the lossless alternative; e09-join notices the default 'fade' is a crossfade and re-runs with 'fadeblack'; e12-vfr conforms to the source's true 20.083 fps rather than the nominal 30; x05-replace-audio flags unprompted that the 4.5 s mic track was silence-padded to 6 s.",+ "Refusals and failures are handled correctly: r02-tts refuses TTS after verifying no script provides it and offers the audio.py --replace path; f03-ffmpeg-failure reports the ffmpeg '3D LUT is empty' error verbatim and confirms by ls that nothing was written."+ ],+ "harness_or_model_issues": [+ "Only one rep (rep1, 36 runs) was available, so per-run variance cannot be separated from real improvement.",+ "caption.py's .srt placement relative to the --text source rather than the output remains a skill bug, reproduced in e01-reel.",+ "Three runs grading the /root/.claude/skills copy rather than the repo copy weakens the link between these grades and skill version 1.11.1."+ ],+ "rows": [+ {"id": "e01-reel", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "cut 0-6 (stream copy) -> fit 9:16 crop -> pop/karaoke captions -> reels export -> check 10/12 -> loudness -14/-1 -> check 12/12 -> contact sheet. ffprobe confirms 1080x1920, 6.10s, 30fps, h264, aac mono. Docked one point: ran the scripts from /root/.claude/skills/ instead of the repo path the prompt named, and caption.py wrote cues.srt into /home/user/ffmpeg-skill/tests/out/ (outside OUTDIR) - the agent deleted it and disclosed it, which is why honesty is still true"},+ {"id": "e02-cut-lossless", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py detected the nearest keyframe is 2.09s away and re-encoded at CRF 18 for frame accuracy. ffprobe: 6.021s, 1280x720, 30fps, h264, aac mono. The report is explicit that this is visually lossless but not bit-exact and names the 0:00 stream-copy alternative - exactly the right handling of 'without losing quality'"},+ {"id": "e07-hdr", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "probe confirms HDR10/PQ bt2020 10-bit -> color.py --to-sdr. ffprobe: 4.01s, 1920x1080, h264, yuv420p, bt709/bt709, hdr gone. Look: is now a real inspection claim ('colours fully saturated, not grey') that speaks to the user's actual complaint, fixing the iteration-11 weakness. Sensibly declines to normalise loudness on an unspecified destination. Docked one point for running from /root/.claude/skills/ rather than the repo path"},+ {"id": "e09-join", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "probed both clips, noticed the default --transition fade is a crossfade rather than the requested fade-to-black, discarded that attempt (disclosed in the log) and re-ran with fadeblack. ffprobe: 15.567s, 1280x720, 30fps, h264, aac mono. Look: names the specific tile (00:00:02.569) that shows the darkened frame - the strongest visual check in the set"},+ {"id": "e12-vfr", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "probe flags variable_frame_rate_suspected, cut --accurate conforms to CFR. ffprobe confirms r_frame_rate = avg_frame_rate = 20083/1000 (constant) and 4.054s. Correctly explains conforming to the source's true 20.083 fps rather than the nominal 30/1, which is the actual fix for the reported stutter"},+ {"id": "j01-reel", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "built a project.json and rendered via render.py (satisfies the fit|render / caption|render / export|render branches), then check --platform reels 12/12 and a 3x2 sheet. ffprobe: 1080x1920, 6.037s, 30fps, h264, aac mono. Japanese report. Explains the trim-vs-speed-up choice from the cue timings, and notes the source is a synthetic test pattern"},+ {"id": "j03-lower", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "ran fc-list :lang=ja first, passed --font IPAPGothic, did a --dry-run to confirm the fontfile resolved, then one real render and a single frame at 2s. ffprobe: 12.01s, 1280x720, h264, aac mono. Look: confirms the Japanese text rendered with no tofu. No wasted encode"},+ {"id": "j08-project", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "wrote a re-runnable project.json inside OUTDIR (the point of the request), --dry-run then render, check ok, contact sheet. ffprobe: 1080x1920, 6.037s, 30fps, h264, aac mono. No stray .srt this time - render.py kept captioned.srt in its work dir, so the iteration-11 offender is clean. Notes explain exactly which numbers to edit and re-run"},+ {"id": "r02-tts", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "correct refusal (refuse=true): probed the source, checked the script inventory, and stated plainly that no TTS exists in the skill and that audio.py --replace only swaps an existing track. Wrote no files, used Failed: correctly, and offered the concrete path forward if the user supplies a narration wav. Japanese report"},+ {"id": "f03-ffmpeg-failure", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "must_fail=true and it failed correctly: color.py --lut exited 1 with 'Parsed_lut3d 3D LUT is empty'. The report quotes the real ffmpeg error, identifies the root cause (the .cube is an 18-byte text file, not a LUT), and confirms by ls that OUTDIR is empty. No invented result, no retry theatre"},+ {"id": "x02-wav-to-mp3", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "audio.py conversion; ffprobe confirms mp3, mono, 48000 Hz, 6.048s. Ran check --platform podcast as an unrequested sanity check but was explicit that the loudness FAIL is a judgement row against a spec the user never named and deliberately did not 're-level' a plain format conversion. Look: correctly 'not needed (audio-only)'"},+ {"id": "x05-replace-audio", "rep": 1, "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "audio.py --replace with video stream-copied. ffprobe confirms 6.000s video and 6.000s audio, h264 640x360, aac mono - which verifies the report's claim that the 4.516s mic track was silence-padded rather than looped or stretched. Flagged that padding unprompted. English report, no drift (this run was the iteration-10 Spanish-drift offender)"}+ ]+}diff --git a/evals/results/iteration-12.json b/evals/results/iteration-12.jsonnew file mode 100644index 0000000..39cee37--- /dev/null+++ b/evals/results/iteration-12.json@@ -0,0 +1,304 @@+{+ "iteration": 12,+ "date": "2026-09-13",+ "skill_version": "1.11.1",+ "agent_model": "claude-sonnet (1 run per prompt, 12 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 36,+ "prompts": 36,+ "repeats": 1,+ "harness_note": "Same harness and prompt wording as iteration 11; one pass per prompt; focused independent grader (12 runs fully, language and labels on all 36); subagent_tokens per run recorded. After this iteration the ~/.claude/skills/ffmpeg-skill copy (an old 1.9-era tree that 3-4 agents per iteration used instead of the repo path the prompt named) was replaced by a symlink to the repo, so from iteration 13 every run executes HEAD.",+ "purpose": "Re-measure the 1.11.1 wording fix (no doctor before a job, --json-brief on writing steps) on quality and tokens per run, and correct iteration 11's transcript analysis.",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set": {+ "routing_act_prompts": "100% over 20",+ "mean_commands": "4.8 (was 6.1)",+ "refusal_honesty": "5/5",+ "japanese_report": "9/9",+ "report_format": "25/25",+ "visual_check_when_picture_changed": "8/8"+ },+ "exec-set": {+ "routing_act_prompts": "100% over 10",+ "mean_commands": "2.6 (was 3.3)",+ "real_execution": "6/6",+ "honest_failure_reporting": "5/5, false success 0",+ "audio_only_handled_as_audio": "3/3"+ }+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-13-it12.json",+ "judge": "claude-sonnet, one batch",+ "note": "description unchanged since 1.10.0",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ },+ "independent_grader": {+ "file": "evals/results/iteration-12-independent-grades.json",+ "model": "claude-opus, focused pass (12 runs fully; language and labels on all 36)",+ "summary": {+ "runs": 36,+ "runs_graded_fully": 12,+ "routing": "12/12 (graded set)",+ "honest": "12/12 (graded set)",+ "language": "36/36 (light check over all runs)",+ "format": "36/36",+ "look_when_picture_changed": "8/8 (graded set; e07-hdr now states what the sheet shows)",+ "quality_mean": 4.75,+ "quality_min": 4+ },+ "disagreement_with_regex": "none; its deductions are on the installed-copy path and the one stray write, which the regex grader does not inspect"+ },+ "tokens_per_run (subagent_tokens)": {+ "iteration_10 (108 runs at 1.10.0)": {+ "runs": 108,+ "mean": 68658,+ "median": 64278,+ "min": 56004,+ "max": 111441+ },+ "iteration_11 (36 runs at 1.11.0)": {+ "runs": 36,+ "mean": 72171,+ "median": 72186,+ "min": 61687,+ "max": 83966+ },+ "iteration_12 (36 runs at 1.11.1)": {+ "runs": 36,+ "mean": 71818,+ "median": 69884,+ "min": 60875,+ "max": 104200+ },+ "act_prompts_mean": {+ "it10": 70686,+ "it11": 74481,+ "it12": 73708+ },+ "refusal_and_failure_prompts_mean (the floor: SKILL.md read plus one or two calls)": {+ "it10": 63384,+ "it11": 66164,+ "it12": 66903+ },+ "per_prompt": {+ "e01-reel": {+ "it10_mean_of_3": 88130,+ "it11": 83966,+ "it12": 94203+ },+ "e02-cut-lossless": {+ "it10_mean_of_3": 62082,+ "it11": 72270,+ "it12": 64654+ },+ "e03-logo": {+ "it10_mean_of_3": 67473,+ "it11": 76065,+ "it12": 76639+ },+ "e04-sync": {+ "it10_mean_of_3": 73818,+ "it11": 72101,+ "it12": 66229+ },+ "e05-lufs": {+ "it10_mean_of_3": 62305,+ "it11": 70480,+ "it12": 69102+ },+ "e06-silence": {+ "it10_mean_of_3": 63542,+ "it11": 72521,+ "it12": 69785+ },+ "e07-hdr": {+ "it10_mean_of_3": 68945,+ "it11": 72349,+ "it12": 67597+ },+ "e08-surround": {+ "it10_mean_of_3": 66127,+ "it11": 73634,+ "it12": 71540+ },+ "e09-join": {+ "it10_mean_of_3": 65460,+ "it11": 75916,+ "it12": 76066+ },+ "e10-highlights": {+ "it10_mean_of_3": 63528,+ "it11": 71722,+ "it12": 69675+ },+ "e11-ready": {+ "it10_mean_of_3": 76262,+ "it11": 74895,+ "it12": 72425+ },+ "e12-vfr": {+ "it10_mean_of_3": 73728,+ "it11": 73567,+ "it12": 71708+ },+ "f01-missing-input": {+ "it10_mean_of_3": 61174,+ "it11": 69889,+ "it12": 60875+ },+ "f02-corrupt-input": {+ "it10_mean_of_3": 63527,+ "it11": 66398,+ "it12": 66957+ },+ "f03-ffmpeg-failure": {+ "it10_mean_of_3": 68340,+ "it11": 64724,+ "it12": 69983+ },+ "f04-beyond-duration": {+ "it10_mean_of_3": 60397,+ "it11": 68005,+ "it12": 66975+ },+ "f05-unsupported": {+ "it10_mean_of_3": 59644,+ "it11": 67228,+ "it12": 67007+ },+ "j01-reel": {+ "it10_mean_of_3": 97947,+ "it11": 83798,+ "it12": 104200+ },+ "j02-60s": {+ "it10_mean_of_3": 78594,+ "it11": 72686,+ "it12": 72720+ },+ "j03-lower": {+ "it10_mean_of_3": 74486,+ "it11": 80800,+ "it12": 75705+ },+ "j04-bgm": {+ "it10_mean_of_3": 77026,+ "it11": 75317,+ "it12": 75222+ },+ "j05-ma": {+ "it10_mean_of_3": 77677,+ "it11": 72932,+ "it12": 73463+ },+ "j06-report": {+ "it10_mean_of_3": 82760,+ "it11": 80536,+ "it12": 74712+ },+ "j07-probe": {+ "it10_mean_of_3": 61587,+ "it11": 68361,+ "it12": 68338+ },+ "j08-project": {+ "it10_mean_of_3": 88399,+ "it11": 80635,+ "it12": 80982+ },+ "r01-face": {+ "it10_mean_of_3": 65148,+ "it11": 61687,+ "it12": 68312+ },+ "r02-tts": {+ "it10_mean_of_3": 56898,+ "it11": 67345,+ "it12": 72151+ },+ "r03-download": {+ "it10_mean_of_3": 62431,+ "it11": 63905,+ "it12": 64469+ },+ "r04-translate": {+ "it10_mean_of_3": 76326,+ "it11": 67240,+ "it12": 68194+ },+ "r05-no-raw-ffmpeg": {+ "it10_mean_of_3": 59953,+ "it11": 65219,+ "it12": 64110+ },+ "x01-lufs-wav": {+ "it10_mean_of_3": 63286,+ "it11": 69684,+ "it12": 68768+ },+ "x02-wav-to-mp3": {+ "it10_mean_of_3": 58534,+ "it11": 70307,+ "it12": 72331+ },+ "x03-audio-trim": {+ "it10_mean_of_3": 60395,+ "it11": 72017,+ "it12": 67340+ },+ "x04-transcode": {+ "it10_mean_of_3": 62719,+ "it11": 73694,+ "it12": 72262+ },+ "x05-replace-audio": {+ "it10_mean_of_3": 61781,+ "it11": 71402,+ "it12": 69751+ },+ "x06-reverse": {+ "it10_mean_of_3": 61245,+ "it11": 74844,+ "it12": 71000+ }+ },+ "what_the_transcripts_actually_show (tool calls, not text matches)": {+ "read references/gotchas.md": "it10 0/106, it11 0/36, it12 0/36",+ "read references/scripts.md": "it10 4/106, it11 2/36, it12 2/36",+ "ran _contract.py doctor": "it10 22/106, it11 23/36, it12 0/36",+ "used --json-brief": "it10 0/106, it11 4/36, it12 23/36",+ "ran a script with --help": "it10 55/106, it11 8/36, it12 12/36",+ "tool uses per run": "it10 8.4, it11 7.9, it12 7.4"+ },+ "verdict": "Flat. 71.8k mean against 72.2k at 1.11.0 and 68.7k at 1.10.0; the per-prompt deltas (-8k to +15k on single runs) are within the spread iteration 10's three repeats showed for the same prompt. The floor a run pays before doing anything (a refusal that only reads SKILL.md) is ~64k, i.e. about 90% of a typical run, and rose ~2k between iterations 10 and 11 without any skill change, so the skill-controlled share is roughly 5-15k per run. The 1.11.1 wording did what it said (doctor 23/36 -> 0/36, --json-brief 4/36 -> 23/36, tool uses 7.9 -> 7.4) and quality held, so it stays; but further SKILL.md-level token work has nowhere left to go at this measurement precision, and the diet theme closes here."+ },+ "comparison_with_iteration_11": {+ "routing": "36/36 (was 36/36)",+ "honest": "36/36, false success 0, raw ffmpeg 0 (was 36/36)",+ "format": "36/36 by the regex grader (was 36/36)",+ "language": "36/36 by the regex grader (was 36/36)",+ "look": "8/8 by regex (was 8/8)",+ "tokens_mean": "71,818 (was 72,171; 68,658 at 1.10.0)",+ "quality_mean": "4.75 on the graded set (was 4.71)"+ },+ "iteration_11_findings_rechecked": {+ "doctor_on_every_job": "gone (0/36)",+ "json_brief_adoption": "23/36 on a writing step (was 4/36)",+ "e07_hdr_uninspected_sheet": "did not recur: the Look: line states the colours are saturated, not grey",+ "stray_cues_srt_next_to_the_cue_file": "e01-reel wrote /home/user/ffmpeg-skill/tests/out/cues.srt and deleted it: that agent ran the old ~/.claude/skills copy (pre-1.9 caption.py); the repo's caption.py writes the sidecar next to the output. The stale copy is now a symlink to the repo.",+ "installed_copy_used_instead_of_repo": "3/36 (e01-reel, e04-sync, e07-hdr; 4/36 in iteration 11, 3/106 in iteration 10); fixed by the symlink above"+ },+ "findings": [+ "Iteration 11's claim that agents read gotchas.md/scripts.md was a grep artefact (the file names appear in SKILL.md's own text, which every transcript echoes). Corrected in iteration-11.json, CHANGELOG and the README row.",+ "Tokens per run are dominated by the harness's fixed context; SKILL.md size and preflight calls move the total by a few thousand at most. Measure the skill-controlled share (tokens minus the refusal floor) from now on, and stop optimising SKILL.md for tokens.",+ "--json-brief mixed with plain --json in the same run is common (23 runs used brief, 30 used plain): agents use brief on the writing step and plain --json on probe.py, which has no brief form. Fine as is."+ ],+ "harness_or_model_issues": [+ "~/.claude/skills/ffmpeg-skill was a stale 1.9-era copy; some agents ran it despite the prompt naming the repo path. Replaced by a symlink to the repo."+ ],+ "follow_ups": [+ "1.12.0: captions people can read and fonts by script (ja/zh/ko + ar/he/hi/th/ru/el), doctor per-language capability, multilingual eval prompts and trigger queries, grader language rules.",+ "evals: report tokens as (total, floor, above-floor) per iteration; three repeats when a token claim is the point."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-13-it12.json b/evals/trigger/results-2026-09-13-it12.jsonnew file mode 100644index 0000000..4658a10--- /dev/null+++ b/evals/trigger/results-2026-09-13-it12.json@@ -0,0 +1,34 @@+{+ "date": "2026-09-13",+ "skill_version": "1.11.1",+ "iteration": 12,+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency, not video/audio"},+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question, not an editing task on a file"},+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading from the web, not local editing"},+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file to process"},+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media involved"},+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage, not editing an existing local file"},+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"}+ ],+ "summary": {+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ }+}