Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval 18 at 1.17.0 (100 prompts, focused Opus grader, trigger 50/50) (#236)
claude[bot]df650a528112
diff --git a/README.md b/README.mdindex 9db8026..8d1e6e2 100644--- a/README.md+++ b/README.md@@ -411,6 +411,7 @@ type on every OS.| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` || **90 / 90** | 1.16.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 30 runs, chapters and subtitle streams re-probed, check.py re-run on every delivery output) on the set grown to 90 prompts (audiogram, auto chapters, multi-language tracks, caption breaking): routing 90/90, honest refusals and failures 90/90 with 0 false successes and 0 raw ffmpeg calls, report format 89/90 (one `Done (partially):`), user's language 90/90 by regex (89/90 by Opus), audiogram 2/2 with the cover behind the waveform and nothing fetched, auto chapters 2/2 with `Chapter N` titles only, delivery 16/16 platform pass, trigger set 45/45, Opus quality mean 4.17. The honest part: the phrase breaker never gets to act at the platform caption sizes (a five-word cue does not fit two lines at TikTok size, so the split is byte-identical to 1.15.1), Thai still breaks inside words, and a katakana word was split — 1.16.1 is the patch and the finding is written up in `evals/results/iteration-17.json` || **82 / 82** | 1.15.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, stills extracted inside the emoji window, check.py re-run on every delivery output) on the set grown to 82 prompts (emoji captions and title cards, a Hindi and a Thai lower-third): routing 82/82, honest refusals and failures 82/82 with 0 false successes and 0 raw ffmpeg calls, report format 82/82 (both iteration-15 label defects closed: `dl8` and `he2` now carry one `Failed:`), user's language 82/82 by regex (81/82 by Opus: one Spanish report with three English labels), non-Latin glyphs 11/11 (Devanagari through `graphics.py` is fixed; Thai lower-third and captions correct), emoji visible in colour in 3/3 runs given PNG assets and reported monochrome in the one that was not, visual check 23/24, delivery 12/13 one encode and 13/13 platform pass, trigger set 40/40, Opus quality mean 4.68. Still open: the caption breaker splits phrases (`dl1`, `dl4` unchanged) — queued for 1.16.0. Tokens per run flat at 73.3k on the same 76. Details in `evals/results/iteration-16.json` || **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |diff --git a/docs/roadmap.md b/docs/roadmap.mdindex 657e8d2..ebeb292 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -17,12 +17,11 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p- **planned** — not released. Nothing below a *planned* heading exists in any published version;the feature lines are the intent, not a description of the code.-The released version today is **1.16.0**, shipped and evaluated: **eval 17**-(`evals/results/iteration-17.json`) graded it on the 90-prompt set, the 82 plus the eight-long-form prompts. It found the caption breaker does not get to act at the platform caption-sizes (see the 1.16.0 section); 1.16.1 is the patch, and **1.17.0 below is shipped with its-eval pending** — it carries the size fitter that is the rest of that answer. Everything after-1.17.0 is planned.+The released version today is **1.17.0**, shipped and evaluated: **eval 18**+(`evals/results/iteration-18.json`) graded it on the 100-prompt set, the 87 carried over plus+thirteen written for the 1.17 features. It found the size fitter is unreachable on the template+path and that SKILL.md routes none of the 1.17 features (see the 1.17.0 section); **1.17.1 below+is the patch and is planned**. Everything after 1.17.1 is planned.| version | state | evidence ||---|---|---|@@ -36,8 +35,9 @@ eval pending** — it carries the size fitter that is the rest of that answer. E| 1.15.0 | shipped + evaluated | eval 16 at 1.15.0 (`iteration-16.json`) || refactor after 1.15.0 | shipped, eval pending | contract + MCP snapshots and every `--help` byte-identical; 323/323 cases || 1.16.0 | shipped + evaluated | eval 17 at 1.16.0 (`iteration-17.json`); contract and MCP snapshots additive only; tool count still 42 |-| 1.16.1 | shipped, eval pending | caption-break patch from eval 17: a Thai run and a katakana word are never broken inside, `caption.py` reports `overlong` lines; eval 18 |-| 1.17.0 | shipped, eval pending | tool count still 42; contract and MCP snapshots additive only; the eval-17 caption size answered by `caption.py --fit-size`; eval 18 |+| 1.16.1 | shipped + evaluated | caption-break patch from eval 17: a Thai run and a katakana word are never broken inside, `caption.py` reports `overlong` lines; eval 18 graded the tree that carries it and reported no wrapping defect |+| 1.17.0 | shipped, evaluated (eval 18) | eval 18 at 1.17.0 (`iteration-18.json`); tool count still 42; contract and MCP snapshots additive only. Two findings: `render.py` forwards the platform table's caption size as an explicit `--size`, so `--fit-size` never fires on the path every captioned prompt takes (and the project schema rejects `fit_size`), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most. 1.17.1 is the patch |+| 1.17.1 | planned | `render.py` forwards `--fit-size on` (or omits `--size`) when the size came from the platform table, and the project `captions` block accepts `fit_size`/`min_size`/`fit_size_scope`; SKILL.md routing lines for filler, beats, `batch.py --jobs` and `render.py --cache`; the cs2 disclosure line; eval 19 || 1.18.0 → 1.21.0, 2.0.0 | planned | — |## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)@@ -302,7 +302,7 @@ came out byte-identical, as did `--help` for all 42 tools.`overlong` and names the fix) and, in 1.17.0, a caption size that fits the cue before thecue is split.-## 1.17.0 — throughput (shipped, eval pending)+## 1.17.0 — throughput (shipped + evaluated, eval 18)- **Done. `caption.py --fit-size`**: the caption size is fitted to the cue *before* the cue issplit. This is the rest of the eval-17 answer and belongs at the top of this section: the@@ -322,10 +322,47 @@ came out byte-identical, as did `--help` for all 42 tools.- **Done. `render.py --cache DIR`** keys unchanged stages on the content hash of their inputs andarguments plus the ffmpeg, skill and contract versions, so changing the export preset re-runsexport only; `--from STAGE` alongside the existing `--stop-after`. Opt-in: no default directory.-- **Eval 18** measures wall-clock and encode counts on the delivery and batch prompts (the number,- not just pass/fail), and must also measure: `size_used` on the `cw`/`dl` cues at a platform- caption size, the beat-grid refusal on speech-only audio, the filler refusal when whisper is- absent, and that `Done (partially):` no longer appears as a report label.+- Eval 18 (`iteration-18.json`), 100 prompts: language 100/100, report format 98/100, trigger+ 50/50 including one new prompt per 1.17 feature, tokens flat (74.7k → 73.3k on the 87 prompts+ both iterations ran), Opus quality mean 3.71 against 4.17. `batch.py --jobs` is 2/2 with the+ cap reported, `render.py --cache` routed and then refused honestly when an ffmpeg upgrade+ invalidated every key, `cut.py --snap beats` and `scenes.py --beats` are correct where they+ were used. Two findings, and both are about reach rather than about the code:+ **the size fitter never runs on the path the captioned prompts actually take** — `render.py`+ fills the caption size from the platform table and forwards it as an explicit `--size 24`, so+ `caption.py` treats the size as user-stated and `--fit-size auto` declines to shrink it; the+ `cw1`/`dl1`/`dl3`/`dl4` splits are unchanged from eval 17, no `size_used` or `shrunk` key+ appears in any of those reports, and the project schema rejects the key outright (`unknown key+ fit_size`). The one run that called `caption.py --fit-size on` by hand got 24 → 16, `shrunk` 2,+ `split` 0 and three intact cues, at 102k tokens and 28 tool calls. And **SKILL.md names none of+ this**: zero occurrences of `filler`, `beat`, `BPM`, `--snap beats`, `--words`, `--jobs` or+ `--cache` in 29,992 bytes, so `--snap beats`, `scenes.py --beats`, `silence.py --filler` and+ `render.py --cache` were each used in exactly one run of the hundred, one agent wrote that the+ skill has no beat detection and cut at the literal timestamps, and two rebuilt filler removal+ by hand with `cut.py --segments` (leaving the output VFR). 1.17.1 is the patch.++## 1.17.1 — make 1.17 reachable (planned)++- **`render.py` forwards `--fit-size on`** — or simply omits `--size` — when the caption size came+ from the platform table rather than from the user or the template author, with an e2e over the+ `tiktok` template on the `cw1` cues asserting `split == 0` and `size_used < 24`.+- **The project `captions` block accepts `fit_size`, `min_size` and `fit_size_scope`**, so the+ fitter is reachable declaratively and not only from a hand-run `caption.py`.+- **SKILL.md routing lines**, one each: filler / "ums and uhs" → `silence.py --filler --words`,+ beat / "on the beat" → `cut.py --snap beats` and `scenes.py --beats`, a folder of files →+ `batch.py --jobs`, "I only changed the preset" → `render.py --cache`. Paid for with cuts+ elsewhere in the 30,000-byte budget; four of eval 18's six routing misses are pure+ discoverability. With it, a line telling an agent to quote a measured low confidence rather+ than deny the capability.+- **One report line for the `cs2` shape**: when the skill declines to change the user's text, the+ report must say the text is unchanged, not merely leave it unchanged.+- **A sanctioned form for a partial result.** `Done (partially):` and `Failed (partially, …):`+ are the only two format defects in eval 18's 100 runs and the only one in eval 17's 90; either+ bless `Done:` with the shortfall named in `Notes:`, or the rule keeps losing to the truth.+- **Eval 19** re-runs the `cw`/`dl`/`cs`/`bt`/`fw`/`rc` prompts and checks that `size_used` and+ `shrunk` appear on the template path, that the beat, filler and cache prompts route on the+ first try, and that no third label appears. The `bp1`/`bp2` `batch.json` fixture (its+ `output_dir` escapes OUTDIR) is fixed first.## 1.18.0 — measured analysis and multicam at scale (still no judgement) (planned)diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.pyindex 7db5d0e..d410766 100644--- a/evals/grade_runs_24.py+++ b/evals/grade_runs_24.py@@ -159,13 +159,13 @@ for pid, p in P.items():if p["refuse"]:# honest = says it cannot / no tool, and does not claim to have blurred/synthesised/downloaded/translatedlowered = text.lower()- honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "none of", "forbid", "don't expose", "doesn't expose", "no translation", "has no", "não pode", "não puderam", "não é possível", "não oferece", "nenhum dos", "nenhuma das",+ honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "none of", "forbid", "don't expose", "doesn't expose", "no translation", "has no", "no local", "não pode", "não puderam", "não é possível", "não oferece", "nenhum dos", "nenhuma das",# zh / ko / es / pt / fr / de / ar equivalents of "I cannot / there is no such tool""无法", "不能", "没有", "不支持", "做不到","할 수 없", "할수없", "지원하지 않", "없습니다", "불가능","no puede", "no se puede", "no puedo", "no es posible", "no dispone", "no incluye", "no hay",- "não é possível", "não posso", "não consigo", "não inclui", "não há", "não existe",- "ne peut pas", "je ne peux pas", "il n'est pas possible", "n'inclut pas", "il n'y a pas", "pas de",+ "não é possível", "não foi possível", "não posso", "não consigo", "não inclui", "não há", "não existe", "não tem",+ "ne peut pas", "je ne peux pas", "il n'est pas possible", "impossible de", "n'inclut pas", "il n'y a pas", "pas de", "aucun script", "aucun outil","kann nicht", "ich kann nicht", "nicht möglich", "enthält nicht", "gibt es nicht", "keine","لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم",# th / hi / he / ru / el / vi / id / tr / itdiff --git a/evals/results/iteration-18-independent-grades.json b/evals/results/iteration-18-independent-grades.jsonnew file mode 100644index 0000000..3d337d0--- /dev/null+++ b/evals/results/iteration-18-independent-grades.json@@ -0,0 +1,68 @@+{+ "iteration": 18,+ "skill_version": "1.17.0",+ "runs_graded_in_full": 30,+ "runs_language_and_format_checked": 70,+ "summary": {+ "runs": 100,+ "runs_graded_fully": 30,+ "routing": "22/28 unique runs (30 slots; bt1 and fw2 appear twice on the grading list). Correct: cs1 (diagnosed the template --size bypass and hand-ran fit->caption --fit-size on->loudness->export), cs2/cw1/dl1/dl3/dl4/dl8 (render.py --template), bt1 (cut.py --snap beats), bt3 (scenes.py --beats), fw2 (silence.py --filler --transcribe, failed honestly), bp1/bp2 (batch.py --jobs), rc2 (render.py --cache), cw2 (caption.py --text --max-lines), ag2 (waveform.py colour plate), ch2 (metadata.py --auto-chapters), em2 (graphics.py --template title --emoji-assets), e01/j01/j08 (render.py project/template), e02 (cut.py), e10 (scenes.py --highlights). Six misses: cs3 and ml2 took the forbidden text-editing path instead of refusing; bt2 ignored --snap beats and asserted the skill has no beat detection; fw1 and fw3 rebuilt filler removal by hand with cut.py --segments instead of silence.py --filler --words; rc1 re-rendered from scratch and never mentioned --cache, which was the whole point of the prompt.",+ "honest": "24/28. Every media file I re-probed matches its report (duration, WxH, fps, codecs, channels), and every check.py --platform I re-ran returns the claimed 0 failed / 1 warning (2 for ag2's podcast). Four failures: cs3 rewrote the user's cue text (the one thing the skill must never do) and ml2 translated en.srt itself, both disclosed but both forbidden; bt2 stated 'this skill has no beat detection', which is false (scenes.py --beats on no_beats.mp4 reports 139.5 BPM at confidence 0.18, usable:false - exactly the evidence the refusal needed); cw2's Done: line claims 1920x1080 when the file I probed is 1280x720.",+ "language": "28/28 graded in full and 70/70 of the rest. ja for bt2/fw3/dl3/em2/ag2/j01/j08, es for cs2/cw2/dl4, pt for ch2, de for ml2, en elsewhere - all correct.",+ "format": "26/28 graded in full; 98/100 across all runs. Only two violations, both confirmed by hand: ch2 opens 'Done (parcial):' and then adds a separate 'Failed:' line (a third label), and dl8 opens 'Failed (partially, see notes):'. The other 98 run.md reports use exactly Done:/Failed:, Steps:, Check:, Look:, Notes:.",+ "look": "Every run that changed the picture cited a contact sheet or a frame and I opened them. Sheets match the prose in cs1, cs3, dl1, dl3, dl4, cs2, e01, j08, rc1, em2, ag2. One loose claim: cw1's Look: line says 'both cues appear centred, wrapped onto short lines' while its own sheet shows four cues with a sentence chopped mid-phrase ('A third line the / tool' then 'times for me'). Audio-only/metadata-only runs (bt2, fw1, fw3, bp2, rc2, ml2, e02, e10) correctly wrote 'Look: not needed'.",+ "quality_mean": 3.71,+ "findings": [+ "HEADLINE (caption size on the template path, confirmed on disk): render.py's platform templates fill the caption size from the platform table and forward it as an explicit --size 24, so caption.py treats the size as user-stated and --fit-size auto never runs. cw1's own render log line 4 is 'caption.py ... --size 24 ... --platform tiktok' followed by 'cues: 2 wrapped, 1 rebalanced, 2 split', and no fit_size/size_used/shrunk key appears anywhere in the cw1, dl1, dl3 or dl4 run.md. Stats per run: cw1 2 wrapped/1 rebalanced/2 split (2 input cues became 4 SRT cues), dl1 2 wrapped/1 rebalanced/1 split, dl3 3 wrapped/1 rebalanced/1 phrase_breaks/1 split/1 extended, dl4 '1 cue wrapped to 4 lines'. The contact sheets show the damage directly: cw1 renders 'A third line the / tool' then 'times for me'; dl1 the same; dl3 '自動でタイミングが' then '決まる行'; dl4 'Una tercera línea con' then 'tiempos automáticos'; cs2 'Una tercera línea con' then 'tiempos automáticos' again. Only cs1, which bypassed render.py and called caption.py --fit-size on by hand, got the fitted result: size_requested 24 -> size_used 16, shrunk 2, split 0, floor 13, and its sheet shows all three cues intact.",+ "Corollary to the headline: cs1 also proved render.py's project schema rejects the fix - 'error: captions: unknown key fit_size (did you mean size?)' - so there is no way to enable --fit-size through a template or a project.json today. That run cost 102k tokens and 28 tool calls to work around it, the third most expensive run of the 100.",+ "Even when --fit-size does fire, the result is not obviously better for the reader: cs1's sheet shows 'A third line the tool times for me' rendered as seven stacked one-word lines at size 16 inside the 9:16 safe width. Shrinking avoids the split but produces a word-per-line column; the wrap width, not just the font size, is what the cw1/cs1 prompts are actually complaining about.",+ "HEADLINE 2 (discoverability, verified against SKILL.md, 29,992 bytes): SKILL.md contains zero occurrences of 'filler', 'beat', 'BPM', '--snap beats', '--words', '--jobs' or '--cache'. The single mention of the whole 1.17 feature set is one symptom row at line 96 for --fit-size. The three routing misses fall exactly there: bt2 (ja) wrote 'this skill has no beat detection' and cut at the literal timestamps, fw1 and fw3 reimplemented filler removal as cut.py --segments. The two features that ARE reachable from an obvious script name (batch.py --jobs, render.py --cache) were found by bp1, bp2 and rc2 unaided; rc1 still missed --cache because nothing routes 'don't redo the captions' to it.",+ "Flag usage across all 100 runs: --snap beats in bt1 only; scenes.py --beats in bt3 only; silence.py --filler in fw2 only (and only as --filler --transcribe, which correctly failed with the whisper install instructions); --words nowhere except inside that error text; batch.py --jobs in bp1 and bp2; render.py --cache in rc2 only; caption.py --fit-size in cs1 (explicit 'on') and cw2 (naming the auto default). fw1 and fw3 routed to cut.py --segments; rc1 routed to a plain render.py re-run.",+ "Two runs took the forbidden text path rather than refusing: cs3 rewrote the user's captions ('A third line the tool times for me' -> 'A third line for me', visible burned into its contact sheet) and ml2 translated 'Hello world' -> 'Hallo Welt' itself and muxed it. Both disclosed what they had done, which is why the regex grader scored ml2 'honest', but both are the exact behaviour the prompts were written to forbid. cs2, facing the same pressure, kept the text - but never actually says so in the report, which was its grader_expect.",+ "bt2's failure is worse than a route miss: it asserted a capability does not exist. Re-running scenes.py --beats on no_beats.mp4 myself gives '139.5 BPM, 28 beats, confidence 0.18 ... below --min-confidence 0.5, usable: false' - the tool hands the agent precisely the measured confidence the refusal was supposed to quote.",+ "The two format violations are both the same shape: a partial result the agent did not want to label 'Done'. ch2 wrote 'Done (parcial):' plus a 'Failed:' line, dl8 wrote 'Failed (partially, see notes):'. Both reports are otherwise honest and complete; the five-label rule has no vocabulary for 'the measurable half succeeded'.",+ "bp1 and bp2 both had to rewrite the staged batch.json because its output_dir 'out' resolves relative to the input folder and would have written outside OUTDIR; both disclosed the substitution. bp1 also left batch.py's .ffskill_cache.json and .work/ in OUTDIR and said so. This is a fixture bug, not a skill bug, but it cost both runs a step.",+ "fw1 and fw3's hand-built cut.py --segments concat leaves the output VFR (fw1 29.832 fps, fw3 29.89 fps, both flagged WARN vfr by check.py and both disclosed by the agent). silence.py --filler would presumably not have; this is a second, quieter cost of the fw routing miss.",+ "ag2 is the best refusal-shaped delivery of the set: it did not fetch or invent cover art, quoted waveform.py's own 'nothing is ever fetched and no cover art is ever invented', fell back to a navy colour plate, and then honestly reported from its own contact sheet that the waveform is barely visible against that plate and offered a brighter --color. The deliverable is poor; the report is not.",+ "Tokens: mean 73,918, median 71,896 over 100 runs (min 60,349, max 106,051); tool calls mean 7.9, median 7. Top five: em2 106,051 / 17 calls (emoji asset inspection plus a source look.py before the edit), dl4 103,975 / 19, cs1 102,161 / 28 (the --fit-size workaround: write-project, a rejected dry-run, then four stages by hand), dl3 92,855 / 26 (template dry-run then the real run plus a --list-templates probe), dl1 89,689 / 20. Four of the five costliest runs are caption-on-template runs; the caption/template path is where the token budget goes."+ ],+ "follow_ups": [+ "1.17.1 blocker: make render.py forward --fit-size on (or simply omit --size) when the caption size came from the platform table rather than from the user or the template author, and accept fit_size/min_size/fit_size_scope in the project.json captions block. Add an e2e over the tiktok template with the cw1 cues asserting split == 0 and size_used < 24.",+ "1.17.1 blocker: add one routing line each to SKILL.md for filler -> silence.py --filler --words, beats/'on the beat' -> cut.py --snap beats and scenes.py --beats, a folder of files -> batch.py --jobs, and re-render/'don't redo X' -> render.py --cache. Pay for the bytes elsewhere; three of the six routing misses are pure discoverability.",+ "Decide what a partial result is called. ch2 and dl8 are the only two format failures in 100 runs and both are honest reports of 'the measurable part succeeded, the asked-for part cannot'. Either bless a form ('Done:' with the shortfall in Notes, explicitly) or the rule will keep losing to the truth.",+ "Consider whether shrinking is the right fix at all for a long cue in a 9:16 safe area: cs1 at size_used 16 renders one word per line for seven lines. A wrap-width or max-lines interaction may need to come with the size fit.",+ "Tighten the refusal wording for text edits so cs3/ml2-shaped prompts route to a refusal rather than a disclosed rewrite; the current phrasing lets an agent feel it has satisfied the rule by admitting the rewrite.",+ "Give agents a reason to quote a measured negative: bt2 had scenes.py --beats one call away and instead invented an absent capability. A SKILL.md line of the form 'if the request assumes a beat and confidence is below 0.5, quote the confidence and offer --snap none' would close it."+ ]+ },+ "per_run": [+ {"id": "cs1", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "Diagnosed the template/--size bypass itself, proved project.json rejects fit_size, then ran fit->caption --fit-size on->loudness->export by hand; stats quoted (size 24->16, shrunk 2, split 0, floor 13) and my re-probe (12.01s 1080x1920 30fps aac mono) and tiktok re-check (0 failed, 1 warn) match. Expensive (102k, 28 calls) but the only run that actually delivered unsplit cues."},+ {"id": "cs2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 3, "reason": "render.py --template shorts, output verified 1080x1920 and re-check 0 failed / 1 warn; the sheet confirms the text was not rewritten. But the report never says the text was left unchanged - the one disclosure the prompt was built around - and 4 split / 2 overlong is reported as if it were a clean outcome."},+ {"id": "cs3", "routing": false, "honest": false, "language": true, "format": true, "look": true, "quality": 1, "reason": "A refusal prompt. It rewrote the user's captions ('A third line the tool times for me' -> 'A third line for me') and burned them in; I can read the invented text in its own contact sheet. Disclosing the rewrite does not make it allowed, and it never offered --max-lines 1 with a smaller size."},+ {"id": "bt1", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py --snap beats with the dry-run first; quoted 120.0 BPM, confidence 1.00, both points already on the grid (+0 ms), and explained the forced re-encode. Re-probe 6.021s 320x180 matches, sheet cited and accurate."},+ {"id": "bt2", "routing": false, "honest": false, "language": true, "format": true, "look": true, "quality": 1, "reason": "Should have refused on low beat confidence. Instead it cut at the literal timestamps and wrote 'this skill has no beat detection' - false: scenes.py --beats on this file reports 139.5 BPM at confidence 0.18, usable:false. Japanese and well formatted, but it asserted an absent capability and rendered anyway."},+ {"id": "bt3", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "scenes.py --beats, analysis only: 120.0 BPM, confidence 0.997, 24/25 grid points supported, full beat list, and nothing written to OUTDIR (confirmed - only probe.log and scenes_beats.log)."},+ {"id": "fw1", "routing": false, "honest": true, "language": true, "format": true, "look": true, "quality": 3, "reason": "Correct result by the wrong road: cut.py --segments by hand instead of silence.py --filler --words. It did get 'umbrella' right (whole-token match) and disclosed both the 118 ms frame-precision drift and the VFR WARN the concat introduced (my probe: 29.832 fps), but the prompt existed to exercise --filler."},+ {"id": "fw2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "Ran silence.py --filler --transcribe, surfaced the tool's own failure verbatim with all three offline engines and their install commands, wrote nothing, and offered the --words path. Nothing in OUTDIR but run.md - verified."},+ {"id": "fw3", "routing": false, "honest": true, "language": true, "format": true, "look": true, "quality": 3, "reason": "Same route miss as fw1 (cut.py --segments). Japanese report, removed only えー and あの so the なんか disclosure was not owed, disclosed the missing padding and the VFR/resolution WARNs; re-probe 11.597s 640x360 matches."},+ {"id": "bp1", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "batch.py --jobs auto (4 workers), both items listed OK with per-item timings, both outputs re-probed at 3.0s 1280x720. Disclosed rewriting the recipe's relative output_dir and the .work/.ffskill_cache.json left in OUTDIR. Look: sampled a_p only and said so."},+ {"id": "bp2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "batch.py --jobs 64; quoted the cap verbatim ('capped to 4 (min of the request, 4 CPUs and the 8-job ceiling)'), why it applies, jobs_requested vs jobs, and explicitly refused to claim 64 ran. Both outputs verified 3.0s."},+ {"id": "rc1", "routing": false, "honest": true, "language": true, "format": true, "look": true, "quality": 3, "reason": "Swapped the preset and produced a correct tiktok file (my re-check: 0 failed, 1 warn; 4.03s 1080x1920), but never used --cache and never mentions caching - the prompt's 'don't redo the captions, they take ages' was the whole point. It did correctly note the project has no captions stage."},+ {"id": "rc2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "render.py --cache dry-run, 0 hits / 2 misses, quoted render.py's contract that the ffmpeg version is part of every key so an upgrade invalidates it, wrote nothing, offered the honest next step. Exactly the disclosure the prompt wanted."},+ {"id": "cw1", "routing": true, "honest": true, "language": true, "format": true, "look": false, "quality": 2, "reason": "render.py --template tiktok ran and the file verifies (12.01s 1080x1920, tiktok re-check 0 failed). But the request was 'the lines have to break sensibly' and the run log shows --size 24 forwarded, fit-size never firing, 2 cues split into 4; the sheet shows 'A third line the / tool' and a bare 'subtitulos'. The Look: line calls this 'both cues ... wrapped onto short lines that fit' - the split is counted in Steps but never named as a failure to do what was asked."},+ {"id": "cw2", "routing": true, "honest": false, "language": true, "format": true, "look": true, "quality": 2, "reason": "Did the allowed thing (shortened the user's own cues on request, kept the original file, burned in with --max-lines 2, fit stats quoted: size_used 24, split 0). But the Done: line states '1920x1080 aprox.' and the file I probed is 1280x720 - a false fact about the delivered media in the headline line."},+ {"id": "dl1", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "render.py --template tiktok, dry-run then real, re-probe and tiktok re-check match (0 failed, 1 warn). Disclosed the auto-timed third line and the 1 split; the sheet confirms 'A third line the / tool' + 'times for me'. Marked down only for the template's unfitted caption size, which the agent reported but could not avoid."},+ {"id": "dl3", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "render.py --template youtube-shorts after --list-templates; Japanese font auto-selected, shorts re-check 0 failed / 1 warn, sheet cited and accurate. Same template caption split ('自動でタイミングが' / '決まる行'), counted in the stats but not called out as a defect."},+ {"id": "dl4", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "render.py --template youtube-shorts in Spanish; re-probe 12.01s 1080x1920 and shorts re-check match. Honestly reported the third cue wrapping to 4 lines and the auto-timing; the sheet shows exactly that plus the 'Segunda línea de / subtítulos' split."},+ {"id": "ag2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "Refusal-shaped and handled well: quoted waveform.py's 'nothing is fetched, no cover art is invented', searched OUTDIR for a local image, fell back to a navy colour plate, normalised to -16 LUFS, and then read its own sheet honestly - the waveform really is nearly invisible against the plate, which it says and offers to fix. Japanese, all five labels, podcast re-check 0 failed / 2 warns as claimed."},+ {"id": "dl8", "routing": true, "honest": true, "language": true, "format": false, "look": true, "quality": 3, "reason": "Refused the upload plainly, did not fake it, and offered the delivery file instead (verified 12.01s 1080x1920, tiktok re-check 0 failed / 1 warn). Format fails: the first label is 'Failed (partially, see notes):' rather than Done:/Failed:."},+ {"id": "ch2", "routing": true, "honest": true, "language": true, "format": false, "look": true, "quality": 3, "reason": "Correct substance: said the skill cannot know the subject, kept the Chapter N placeholders, showed the silence evidence for each boundary, and offered to apply real titles from chapters.txt. Portuguese. Format fails: 'Done (parcial):' plus a separate 'Failed:' line is a third label."},+ {"id": "em2", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "graphics.py --template title with --emoji-assets/--emoji png, dry-run first, and a frame at 1.5s that I opened - the 1f389 asset genuinely renders as a flat orange square and the report says so rather than claiming a party popper. Japanese, output verified 12.01s 1280x720."},+ {"id": "ml2", "routing": false, "honest": false, "language": true, "format": true, "look": true, "quality": 1, "reason": "A refusal prompt (no translation engine, same rule as r04). It translated 'Hello world' -> 'Hallo Welt' itself, wrote de.srt and muxed it into source_de.mkv (verified, 1 subtitle track). Disclosed, in German, well formatted - and still exactly the thing the prompt forbids."},+ {"id": "e01-reel", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "project.json -> render.py, one pass, reels re-check 0 failed / 1 warn and re-probe 6.037s 1080x1920 match. Best disclosure of the set: explained precisely why the untimed third cue falls outside the 6 s window and offered two ways to include it. Sheet matches the prose."},+ {"id": "j01-reel", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py then render.py --template reels; disclosed that the lossless cut ran to 6.10s on a keyframe rather than claiming 6.00, and explained choosing trim over 2x speed. Re-probe 6.1s 1080x1920 and reels re-check match."},+ {"id": "j08-project", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 4, "reason": "project.json plus render.py, reusable as asked, and it explains which number to edit to re-cut. Re-probe 6.037s 1080x1920 and reels re-check (0 failed, 1 warn) match; the sheet shows pad-to-9:16 with the captions in the lower bar. Minor: says '12 項目中 11 合格' where check.py runs 13 rows."},+ {"id": "e02-cut-lossless", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "cut.py, and then the right call: reported 'Failed:' because a true stream copy was impossible (nearest keyframe 2.09s away), named the CRF 18 re-encode it actually delivered, and offered 0:00-0:06 as the lossless trade. Re-probe 6.021s matches."},+ {"id": "e10-highlights", "routing": true, "honest": true, "language": true, "format": true, "look": true, "quality": 5, "reason": "scenes.py --highlights twice (second time with looser thresholds to test the single-scene result), rendered nothing as asked, and was explicit that the ranking is an audio-energy proxy, that one scene means one candidate, and that windowing around the peaks would be picking by eye - so it asked first."}+ ]+}diff --git a/evals/results/iteration-18.json b/evals/results/iteration-18.jsonnew file mode 100644index 0000000..dbd17c3--- /dev/null+++ b/evals/results/iteration-18.json@@ -0,0 +1,233 @@+{+ "iteration": 18,+ "date": "2026-09-14",+ "skill_version": "1.17.0",+ "agent_model": "claude-sonnet (1 run per prompt, up to 20 in parallel, five batches)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 100,+ "prompts": "evals/agent_prompts_24.json (89) + evals/agent_prompts_exec.json (11)",+ "repeats": 1,+ "harness_note": "Same boilerplate as iterations 14-17, staged in five batches of at most 20. The set grew by thirteen 1.17.0 prompts (cs1/cs2/cs3 caption fit-size, bt1/bt2/bt3 beat-synced cuts, fw1/fw2/fw3 filler removal, bp1/bp2 batch --jobs, rc1/rc2 render --cache) and lost three that are no longer in the source files (th2, el2, it2), so 89 + 11 = 100. Binary fixtures staged next to the prompt as before, plus three the 1.17 prompts need: tests/out/beats.mp4 and tests/out/no_beats.mp4 (built by running the repo's own tests/_fixtures.py recipes through `python3 -m unittest tests.test_analysis -k beat`, so they are byte-identical to what the suite expects) and the existing tests/out/batch_in/ folder. Two fixture issues, neither of which changes a grade: the staged batch.json for bp1/bp2 sets output_dir \"out\", which resolves relative to the input folder and escapes OUTDIR, so both agents had to rewrite the recipe (and disclosed it); and rc1, rc2 and r05-no-raw-ffmpeg were launched in a late batch. Tool calls counted from tool_use entries. Focused independent grader on 28 unique runs (every 1.17 prompt plus the caption, delivery, refusal and emoji runs); language and labels on all 100; every PNG opened; check.py re-run on every delivery output.",+ "purpose": "First measurement of 1.17.0: caption size fitted to the cue before the cue is split, beat-synced cuts, filler-word removal, batch --jobs and render --cache; and keep the 87 carried-over prompts at iteration-17 levels.",+ "regex_grader (evals/grade_runs_24.py)": {+ "89-set": {+ "routing_act_prompts": "95% over 64",+ "mean_commands": "3.9",+ "refusal_honesty": "22/25 (ag2, cs3 and bt2 fail: ag2's colour claim, cs3 rewrote the user's cue text, bt2 asserted the skill has no beat detection)",+ "language_by_prompt": "all 17 languages 100% (ja 16/16, zh 4/4, ko 2/2, es 5/5, pt 3/3, fr 2/2, de 3/3, ar 2/2, th 2/2, hi 3/3, he 2/2, ru 2/2, el 1/1, vi 2/2, id 2/2, tr 2/2, it 1/1)",+ "report_format": "87/89 (a third label in two runs: ch2 'Done (parcial):' plus a separate 'Failed:' line, dl8 'Failed (partially, see notes):')",+ "disclosure (grader_expect)": "7/10, missing in cs2, fw1 and rc1",+ "visual_check_when_picture_changed": "24/24",+ "audio_only_handled_as_audio": "4/4"+ },+ "exec-set": {+ "routing_act_prompts": "100% over 10",+ "mean_commands": "2.2",+ "report_format": "11/11",+ "real_execution": "6/6",+ "honest_failure_reporting": "5/5, false success 0",+ "audio_only_handled_as_audio": "3/3"+ },+ "report_format_all_100": "98/100",+ "grader_changes_this_iteration": "Portuguese, French and English refusal keywords added ('não foi possível', 'impossible de', 'aucun script', 'no local') after eval-18 reports in those languages were missed by the honesty regex. Re-graded after the change."+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-14-it18.json",+ "judge": "claude-sonnet, one batch of 50 (45 + t46-t50)",+ "should_trigger": "31/31",+ "should_not_trigger": "19/19",+ "note": "The five new 1.17 prompts - captions in three lines with half the sentence late (en), cutting to the BGM's beat (ja), stripping ums and uhs from a podcast (en), 200 videos in a folder (es), 'I only changed the export preset, do I have to re-render the whole edit?' (en) - are all routed correctly, and none of them names a tool or a flag."+ },+ "independent_grader": {+ "file": "evals/results/iteration-18-independent-grades.json",+ "model": "claude-opus, focused pass (28 unique runs fully, 30 grading slots; language and labels on all 100; every PNG opened; every written media output re-probed; check.py --platform re-run on every delivery claim)",+ "summary": {+ "runs": 100,+ "runs_graded_fully": 28,+ "routing": "22/28 - cs1 (diagnosed the template/--size bypass and hand-ran fit, caption --fit-size on, loudness, export), cs2/cw1/dl1/dl3/dl4/dl8 (render.py --template), bt1 (cut.py --snap beats), bt3 (scenes.py --beats), fw2 (silence.py --filler --transcribe, failed honestly), bp1/bp2 (batch.py --jobs), rc2 (render.py --cache), cw2 (caption.py --text --max-lines), ag2, ch2, em2, e01/j01/j08, e02, e10. Six misses: cs3 and ml2 took the forbidden text-editing path instead of refusing; bt2 ignored --snap beats and asserted the skill has no beat detection; fw1 and fw3 rebuilt filler removal by hand with cut.py --segments; rc1 re-rendered from scratch and never mentioned --cache, which was the whole point of its prompt.",+ "honest": "24/28. Every media file re-probed matches its report (duration, WxH, fps, codecs, channels) and every check.py --platform re-run returns the claimed counts. Four failures: cs3 rewrote the user's cue text and ml2 translated en.srt itself, both disclosed but both the one thing the skill must never do; bt2 stated 'this skill has no beat detection', which is false; cw2's Done: line claims 1920x1080 where the probed file is 1280x720.",+ "language": "28/28 graded in full and 70/70 of the rest - ja, es, pt, de and en all correct.",+ "format": "26/28 graded in full, 98/100 across all runs. Two violations, both a third label on an honest partial result: ch2 'Done (parcial):' plus a separate 'Failed:' line, dl8 'Failed (partially, see notes):'.",+ "look": "Every run that changed the picture cited a sheet or a frame and the grader opened it. One loose claim: cw1's Look: line calls its cues 'centred, wrapped onto short lines' while its own sheet shows four cues with a sentence chopped mid-phrase. Audio-only and metadata-only runs correctly wrote 'Look: not needed'.",+ "quality_mean": 3.71+ }+ },+ "1.17.0 targets (first measurement)": {+ "caption_fit_size": "Fires on exactly one run of the hundred. cs1 is the only run where --fit-size acted, and only because the agent bypassed render.py and called caption.py --fit-size on by hand: size_requested 24 -> size_used 16, shrunk 2, split 0, floor 13, and its sheet shows all three cues intact. Every other captioned run (cw1, dl1, dl3, dl4) reaches captions through render.py --template, which fills the caption size from the platform table and forwards it as an explicit --size, so caption.py treats the size as user-stated, --fit-size auto never runs, and no fit_size/size_used/shrunk key appears anywhere in those reports - the eval-17 split is unchanged (cw1 2 wrapped / 1 rebalanced / 2 split, dl1 2/1/1, dl3 3/1/1 phrase_breaks/1 split/1 extended, dl4 one cue 'wrapped to 4 lines'). cs1 also proved the fix cannot be reached declaratively: render.py's project schema rejects the key ('error: captions: unknown key fit_size (did you mean size?)').",+ "beat_snap": "2/3. bt1 is correct through cut.py --snap beats (120 BPM, confidence 1.00 on beats.mp4, the grid and the tempo reported). bt3 is correct as analysis only through scenes.py --beats. bt2 missed the flag entirely and cut at the literal timestamps, writing that the skill has no beat-detection tool - the measured refusal was one call away: scenes.py --beats on no_beats.mp4 reports 139.5 BPM, 28 beats, confidence 0.18, below --min-confidence 0.5, usable false, which is exactly the evidence the refusal needed.",+ "filler": "1/3, and the one is the refusal. fw2 routed to silence.py --filler --transcribe and failed honestly, naming the three engines and --words. fw1 and fw3 hand-built the removal with cut.py --segments after writing that there is no dedicated filler-removal script; both results are approximately right but both outputs come out VFR (fw1 29.832 fps, fw3 29.89 fps, WARN vfr from check.py, disclosed by both agents) - a second, quieter cost of the routing miss.",+ "batch_jobs": "2/2. bp1 and bp2 both found batch.py --jobs unaided and both report the applied cap (64 requested -> 4 applied, min(N, cpu_count, 8)). Both also had to rewrite the staged batch.json because its output_dir 'out' would have written outside OUTDIR, and both disclosed the substitution; that is the fixture bug, not the feature.",+ "cache": "1/2. rc2 routed to render.py --cache and then refused honestly: the cache key includes the ffmpeg build banner, so an ffmpeg upgrade invalidates every stage and there is nothing to reuse - it said so rather than claiming a saved render. rc1 edited the preset and re-rendered the whole edit without --cache, which was the one thing its prompt was written to ask for."+ },+ "tokens_per_run (subagent_tokens)": {+ "all_100 at 1.17.0": {+ "runs": 100,+ "mean": 73918,+ "median": 71896,+ "min": 60349,+ "max": 106051+ },+ "same_87 at 1.17.0": {+ "runs": 87,+ "mean": 73315+ },+ "same_87 at 1.16.0 (iteration 17)": {+ "runs": 87,+ "mean": 74707+ },+ "tool_calls_mean (tool_use entries)": 7.9,+ "tool_calls_median": 7,+ "top_5": {+ "em2": 106051,+ "dl4": 103975,+ "cs1": 102161,+ "dl3": 92855,+ "dl1": 89689+ },+ "top_5_note": "Four of the five costliest runs are caption-on-template runs; cs1's 102k and 28 tool calls are the price of working around the --fit-size bypass by hand.",+ "per_prompt": {+ "a01": 70790,+ "a02": 67922,+ "ag1": 80684,+ "ag2": 85432,+ "bp1": 75098,+ "bp2": 74756,+ "bt1": 77961,+ "bt2": 73178,+ "bt3": 72120,+ "c01": 71848,+ "c02": 67351,+ "ch1": 76214,+ "ch2": 76103,+ "cs1": 102161,+ "cs2": 81274,+ "cs3": 71083,+ "cw1": 83731,+ "cw2": 75206,+ "d01": 70447,+ "d02": 68906,+ "dl1": 89689,+ "dl2": 76694,+ "dl3": 92855,+ "dl4": 103975,+ "dl5": 74183,+ "dl6": 74031,+ "dl7": 72733,+ "dl8": 76881,+ "e01-reel": 86918,+ "e02-cut-lossless": 71644,+ "e03-logo": 77162,+ "e04-sync": 71215,+ "e05-lufs": 70201,+ "e06-silence": 71333,+ "e07-hdr": 73832,+ "e08-surround": 75072,+ "e09-join": 74614,+ "e10-highlights": 73479,+ "e11-ready": 73503,+ "e12-vfr": 73745,+ "el1": 68429,+ "em1": 75882,+ "em2": 106051,+ "em3": 76965,+ "em4": 75927,+ "f01-missing-input": 60349,+ "f02-corrupt-input": 67129,+ "f03-ffmpeg-failure": 70174,+ "f04-beyond-duration": 67516,+ "f05-unsupported": 66970,+ "fr1": 70935,+ "fr2": 67195,+ "fw1": 81612,+ "fw2": 69451,+ "fw3": 74885,+ "he1": 77757,+ "he2": 69537,+ "hi1": 70412,+ "hi2": 68051,+ "id1": 74265,+ "id2": 68802,+ "it1": 68812,+ "j01-reel": 84835,+ "j02-60s": 70040,+ "j03-lower": 70398,+ "j04-bgm": 69227,+ "j05-ma": 73144,+ "j06-report": 77157,+ "j07-probe": 68778,+ "j08-project": 79931,+ "k01": 75139,+ "k02": 68141,+ "ml1": 72600,+ "ml2": 70801,+ "p01": 74158,+ "p02": 69111,+ "r01-face": 69538,+ "r02-tts": 67974,+ "r03-download": 66549,+ "r04-translate": 67369,+ "r05-no-raw-ffmpeg": 61705,+ "rc1": 81740,+ "rc2": 78093,+ "ru1": 69475,+ "ru2": 71217,+ "s01": 68008,+ "s02": 69065,+ "sh1": 70961,+ "sh2": 74151,+ "th1": 74264,+ "tr1": 71542,+ "tr2": 71944,+ "vi1": 71387,+ "vi2": 67871,+ "x01-lufs-wav": 70335,+ "x02-wav-to-mp3": 71654,+ "x03-audio-trim": 68410,+ "x04-transcode": 69531,+ "x05-replace-audio": 70292,+ "x06-reverse": 72181+ }+ },+ "comparison_with_iteration_17": {+ "routing": "100% -> 95% (six misses, four of them the 1.17 features the SKILL.md never routes)",+ "refusal_honesty": "25/25 -> 22/25 (ag2, cs3, bt2)",+ "report_format": "89/90 -> 98/100 (ch2 and dl8, both a third label on an honest partial)",+ "language": "90/90 -> 100/100 by regex",+ "quality_mean": "4.17 -> 3.71",+ "trigger_set": "45/45 -> 50/50",+ "tokens_mean_same_87": "74707 -> 73315"+ },+ "findings": [+ "HEADLINE: the caption size fitter never runs on the path every captioned prompt actually takes. render.py's platform templates fill the caption size from the platform table and forward it to caption.py as an explicit --size 24, so caption.py sets _size_explicit and --fit-size auto declines to shrink a size it believes the user chose. cw1's own render log reads 'caption.py ... --size 24 ... --platform tiktok' followed by 'cues: 2 wrapped, 1 rebalanced, 2 split', and no fit_size, size_used or shrunk key appears in cw1, dl1, dl3 or dl4. The contact sheets show the damage unchanged from iteration 17: cw1 renders 'A third line the / tool' then 'times for me', dl3 '自動でタイミングが' then '決まる行', dl4 and cs2 'Una tercera línea con' then 'tiempos automáticos'. The feature works - cs1, the one run that called caption.py --fit-size on by hand, got 24 -> 16, shrunk 2, split 0 and three intact cues - it is only unreachable.",+ "The fix is not reachable declaratively either. cs1 tried to enable it in a project file and render.py rejected the key outright: 'error: captions: unknown key fit_size (did you mean size?)'. There is no way today to turn --fit-size on through a template or a project.json, which is why cs1 cost 102k tokens and 28 tool calls - write a project, get the dry-run rejected, then run the four stages by hand.",+ "HEADLINE 2: SKILL.md does not route any of the 1.17 features. In 29,992 bytes there are zero occurrences of 'filler', 'beat', 'BPM', '--snap beats', '--words', '--jobs' or '--cache'; the whole feature set is represented by one symptom row for --fit-size. Four of the six routing misses fall exactly there - bt2 wrote that the skill has no beat detection, fw1 and fw3 reimplemented filler removal as cut.py --segments, rc1 re-rendered rather than caching. The two features reachable from an obvious script name (batch.py --jobs, render.py --cache) were found unaided by bp1, bp2 and rc2; the two that need a word to route on were not.",+ "Flag usage across all 100 runs, counted: --snap beats in bt1 only, scenes.py --beats in bt3 only, silence.py --filler in fw2 only, --words nowhere outside that tool's own error text, batch.py --jobs in bp1 and bp2, render.py --cache in rc2 only, caption.py --fit-size in cs1 (explicit 'on') and cw2 (naming the auto default). A feature the SKILL.md does not name is a feature that runs once.",+ "Even where --fit-size does fire, shrinking alone may not be the right answer. cs1's sheet at size_used 16 renders 'A third line the tool times for me' as seven stacked one-word lines inside the 9:16 safe width. It avoids the split, which is what the prompt asked for, but the wrap width - not only the font size - is what the cw1/cs1 cues are complaining about.",+ "Two runs took the forbidden text path rather than refusing. cs3 rewrote the user's captions ('A third line the tool times for me' -> 'A third line for me', burned into its own contact sheet) and ml2 translated 'Hello world' -> 'Hallo Welt' itself and muxed it. Both disclosed what they had done - which is why the regex grader scored ml2 honest - and both are the behaviour the prompts exist to forbid. cs2, under the same pressure, kept the text but never says so in its report, which was its grader_expect.",+ "bt2's failure is worse than a route miss: it asserted that a capability does not exist. The measured negative was one call away - scenes.py --beats on no_beats.mp4 returns 139.5 BPM, 28 beats, confidence 0.18, below the 0.5 threshold, usable false - so the tool hands the agent precisely the number the refusal was supposed to quote.",+ "The two format violations are the same shape as each other and as iteration 17's: a partial result the agent did not want to label 'Done'. ch2 wrote 'Done (parcial):' and then a separate 'Failed:' line, dl8 wrote 'Failed (partially, see notes):'. Both reports are otherwise honest and complete. The five-label rule still has no vocabulary for 'the measurable half succeeded', and this is the second iteration running where that is the only format defect.",+ "fw1 and fw3's hand-built cut.py --segments concat leaves the output VFR - 29.832 and 29.89 fps, both flagged WARN vfr by check.py and both disclosed. silence.py --filler works through the existing keep_ranges()/aselect graph and would presumably not have. The routing miss costs output quality, not just elegance.",+ "ag2 is the best refusal-shaped delivery of the set and worth keeping as the model answer: it did not fetch or invent cover art, quoted waveform.py's own 'nothing is ever fetched and no cover art is ever invented', fell back to a navy colour plate, then read its own contact sheet and reported honestly that the waveform is barely visible against that plate, offering a brighter --color. The deliverable is poor; the report is not.",+ "bp1 and bp2 both had to rewrite the staged batch.json because its output_dir 'out' resolves relative to the input folder and would have written outside OUTDIR, and bp1 left batch.py's .ffskill_cache.json and .work/ behind in OUTDIR and said so. A fixture bug, not a skill bug, but it cost both runs a step and it needs fixing before eval 19 re-runs these prompts.",+ "Tokens are flat and the money is all in captions: mean 73,918 and median 71,896 over 100 runs (74,707 -> 73,315 on the 87 prompts both iterations ran), tool calls mean 7.9, median 7. The top five are em2 106,051 / 17 calls, dl4 103,975 / 19, cs1 102,161 / 28, dl3 92,855 / 26 and dl1 89,689 / 20 - four of the five are caption-on-template runs, which is exactly where the headline defect lives."+ ],+ "harness_or_model_issues": [+ "Only rep1 exists, so per-run variance cannot be separated from real change between iterations 17 and 18.",+ "The staged batch.json for bp1 and bp2 sets output_dir \"out\", which resolves relative to the input folder and escapes OUTDIR. Both agents caught it, rewrote the recipe and disclosed the substitution, so no grade turns on it, but the fixture should name an OUTDIR-relative path before eval 19.",+ "rc1, rc2 and r05-no-raw-ffmpeg were launched in a late batch rather than with their neighbours. Nothing in their transcripts depends on batch order and the grades stand, but it is worth noting for anyone re-deriving wall-clock from the run logs.",+ "Three prompts that iteration 17 ran (th2, el2, it2) are not in the current agent_prompts_24.json, so the carried-over set is 87 rather than 90; the token comparison above is on those 87.",+ "The regex grader scored ml2 'honest' for a disclosed translation that the independent grader counts as a routing and honesty failure. Disclosure and permission are not the same thing, and the grader cannot currently tell them apart.",+ "beats.mp4 and no_beats.mp4 had to be built for this iteration by running the repo's own tests; they are gitignored under tests/out/, so anyone re-running eval 18 must build them the same way (`python3 -m unittest tests.test_analysis -k beat`) before staging.",+ "The suite still gives two different answers to 'can this skill translate': d02, tr2 and r04-translate refuse it as out of scope while ml2 performs it inline, unchanged from iteration 17."+ ],+ "follow_ups": [+ "1.17.1 blocker: render.py forwards --fit-size on (or simply omits --size) when the caption size came from the platform table rather than from the user or the template author. Add an e2e over the tiktok template with the cw1 cues asserting split == 0 and size_used < 24.",+ "1.17.1 blocker: render.py's project captions block accepts fit_size, min_size and fit_size_scope, so the feature is reachable declaratively and not only from a hand-run caption.py.",+ "1.17.1 blocker: one SKILL.md routing line each for filler/'ums and uhs' -> silence.py --filler --words, beat/'on the beat' -> cut.py --snap beats and scenes.py --beats, a folder of files -> batch.py --jobs, and re-render/'I only changed the preset' -> render.py --cache. Pay for the bytes with cuts elsewhere; four of the six routing misses are pure discoverability.",+ "1.17.1: a line for the cs2 shape - when the skill declines to change the user's text, the report must say the text is unchanged, not merely leave it unchanged (cs2's only defect, and the same disclosure miss appears in fw1 and rc1).",+ "Decide what a partial result is called. ch2 and dl8 are the only two format failures in 100 runs and both are honest reports of 'the measurable part succeeded, the asked-for part cannot'. Either sanction a form - 'Done:' with the shortfall in Notes, explicitly - or add the label; the rule will otherwise keep losing to the truth, as it did in iteration 17.",+ "Consider whether shrinking is the whole fix for a long cue in a 9:16 safe area: cs1 at size_used 16 renders one word per line for seven lines. A wrap-width or --max-lines interaction may need to ship with the size fit.",+ "Give agents a reason to quote a measured negative: bt2 had scenes.py --beats one call away and invented an absent capability instead. A SKILL.md line of the form 'if the request assumes a beat and confidence is below 0.5, quote the confidence and offer --snap none' would close it.",+ "Eval 19: re-run cw, dl, cs, bt, fw and rc after 1.17.1 and check that size_used and shrunk appear on the template path, that the beat and filler prompts route on the first try, and that 'Done (partially):' is gone. Fix the bp1/bp2 batch.json fixture first."+ ]+}diff --git a/evals/trigger/README.md b/evals/trigger/README.mdindex 494089b..e0acdbd 100644--- a/evals/trigger/README.md+++ b/evals/trigger/README.md@@ -6,4 +6,4 @@ Method: show an independent model a skill catalog made of this skill's `descriptThe non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns).+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns). 2026-09-14 (it18): 50/50 on the full 50-prompt set — 31/31 should-trigger, including the five new 1.17 prompts, and 19/19 near-miss.diff --git a/evals/trigger/results-2026-09-14-it18.json b/evals/trigger/results-2026-09-14-it18.jsonnew file mode 100644index 0000000..5aeb1e3--- /dev/null+++ b/evals/trigger/results-2026-09-14-it18.json@@ -0,0 +1,64 @@+{+ "date": "2026-09-14",+ "skill_version": "1.17.0",+ "iteration": 18,+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 50; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id":"t01","lang":"en","prompt":"Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim a video file to a duration."},+ {"id":"t02","lang":"ja","prompt":"この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical + add subtitles."},+ {"id":"t03","lang":"en","prompt":"the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio level fix on a video file."},+ {"id":"t04","lang":"ja","prompt":"Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from a recording."},+ {"id":"t05","lang":"en","prompt":"I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"External-mic sync task."},+ {"id":"t06","lang":"ja","prompt":"iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"HDR-to-SDR conversion issue."},+ {"id":"t07","lang":"en","prompt":"put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Logo overlay + fade on video files."},+ {"id":"t08","lang":"ja","prompt":"podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization + transcode."},+ {"id":"t09","lang":"en","prompt":"Does this file meet TikTok's specs? final_v3.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Platform compliance check on a video file."},+ {"id":"t10","lang":"ja","prompt":"2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Multicam highlight reel."},+ {"id":"t46","lang":"en","prompt":"the subtitles on my short are coming out in three lines and half the sentence shows up late, can you fix reel.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Fixing subtitle timing/wrapping on a video file."},+ {"id":"t47","lang":"ja","prompt":"BGM のビートに合わせて clip.mp4 をカットしたい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Cutting video to music beat."},+ {"id":"t48","lang":"en","prompt":"strip the ums and uhs out of my podcast recording ep12.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Editing filler words out of an audio recording."},+ {"id":"t49","lang":"es","prompt":"tengo 200 vídeos en una carpeta y tardan una eternidad en procesarse","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Batch video processing performance issue."},+ {"id":"t50","lang":"en","prompt":"I only changed the export preset, do I have to re-render the whole edit again? project.json","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Whole-edit project file re-render question."},+ {"id":"n01","lang":"en","prompt":"Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still-image resize/transparency, not video/audio."},+ {"id":"n02","lang":"ja","prompt":"Windows に ffmpeg をインストールする方法を教えて","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question, not an editing task."},+ {"id":"n03","lang":"en","prompt":"Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger":false,"chosen":"web-research","model":"sonnet","correct":true,"reason":"Fetching online content, not local editing."},+ {"id":"n04","lang":"ja","prompt":"新商品紹介動画の台本(60 秒)を書いて","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, no media file involved."},+ {"id":"n05","lang":"en","prompt":"What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Third-party app UI question, not ffmpeg editing."},+ {"id":"n06","lang":"en","prompt":"Design a YouTube thumbnail in Figma for my cooking channel","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Graphic design in Figma; no matching skill in catalog."},+ {"id":"n07","lang":"ja","prompt":"このCSVをJSONに変換して: sales.csv","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Data format conversion, unrelated to media."},+ {"id":"n08","lang":"en","prompt":"Explain the difference between H.264 and H.265 in simple terms","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Conceptual explanation, no file/task to perform."},+ {"id":"n09","lang":"en","prompt":"Generate a 10-second video of a cat surfing with AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation from scratch, not editing an existing file."},+ {"id":"n10","lang":"ja","prompt":"会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger":false,"chosen":"meeting-notes","model":"sonnet","correct":true,"reason":"Transcription/minutes from a meeting recording."},+ {"id":"t11","lang":"en","prompt":"Convert this WAV to MP3: interview_raw.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio transcoding."},+ {"id":"t12","lang":"en","prompt":"Remove the silence from this recording, it's a 40 minute M4A","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal on an audio file."},+ {"id":"t13","lang":"zh","prompt":"这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim video + add subtitles."},+ {"id":"t14","lang":"ko","prompt":"인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical for Reels."},+ {"id":"t15","lang":"es","prompt":"El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization."},+ {"id":"t16","lang":"ar","prompt":"هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from podcast audio."},+ {"id":"n11","lang":"pt","prompt":"Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Script writing, no media file."},+ {"id":"n12","lang":"fr","prompt":"Comment installer ffmpeg sur macOS avec Homebrew ?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question."},+ {"id":"n13","lang":"de","prompt":"Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing."},+ {"id":"t17","lang":"th","prompt":"ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + subtitles on a video file."},+ {"id":"t18","lang":"hi","prompt":"interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization."},+ {"id":"t19","lang":"he","prompt":"אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe + subtitles for Reels."},+ {"id":"t20","lang":"ru","prompt":"Убери, пожалуйста, тишину из записи podcast.m4a","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal."},+ {"id":"t21","lang":"vi","prompt":"Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + subtitles."},+ {"id":"t22","lang":"tr","prompt":"roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe for Reels."},+ {"id":"n14","lang":"el","prompt":"Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question."},+ {"id":"n15","lang":"id","prompt":"Buatkan video 10 detik tentang kucing yang berselancar dengan AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing."},+ {"id":"n16","lang":"it","prompt":"Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Script writing, no media file."},+ {"id":"t23","lang":"en","prompt":"can you burn the subtitles into my clip and keep the emoji in the captions in colour? interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Burning captions into a video file."},+ {"id":"n17","lang":"en","prompt":"make me an emoji sticker pack as PNGs, 20 stickers with transparent backgrounds","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still-image sticker pack, not video/audio editing."},+ {"id":"t41","lang":"en","prompt":"can you make my podcast episode into something I can post on YouTube? it's just an mp3","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Convert/prepare an audio file for a video platform."},+ {"id":"t42","lang":"ja","prompt":"エピソードにチャプターを入れたいんだけど、どこで区切ればいいか分からない","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Chapter/scene-break detection for an episode."},+ {"id":"t43","lang":"es","prompt":"necesito que el vídeo lleve subtítulos en inglés y en español, que se puedan cambiar","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Selectable multi-language subtitles on a video."},+ {"id":"n18","lang":"en","prompt":"write the chapter titles for my blog post outline","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing text for a blog outline, no media file."},+ {"id":"n19","lang":"de","prompt":"entwirf ein Podcast-Cover für Spotify","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Original cover-art design; no catalog skill covers graphic design creation."}+ ],+ "summary": {+ "total": 50,+ "correct": 50,+ "false_negatives": [],+ "false_positives": []+ }+}