Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval 22 at 1.18.3 — routing-fix re-check and cs1/cs3 recheck (#254)
Shogo Kajiyamae66e6d70b9d9
diff --git a/README.md b/README.mdindex 8429c4c..999e523 100644--- a/README.md+++ b/README.md@@ -416,7 +416,8 @@ type on every OS.| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |-| **8 / 12 routed** | eval 21 at 1.18.0 (2026-09-17, one prompt per new analysis/multicam flag plus a `cs1`/`cs3` recheck, Sonnet agent): every result was correct where the agent found the right script — both sync offsets, the shot label, the multicam switch point and its render-project mapping, both 1.17.2/1.17.3 rechecks — but 4 of 12 prompts hit a script the agent could not find, because SKILL.md named none of the five new flags (confirmed by grep). Three refused honestly rather than fabricate a number; one reached a correct answer without the intended flag, by luck of one fixture's silence durations. 1.18.1 (2026-09-17) is the SKILL.md fix, not yet re-evaluated. Written up in `evals/results/iteration-21.json` |+| **7 / 8 routed** | eval 22 at 1.18.3 (2026-09-17, eight new symptom-only prompts for the five 1.18.0 flags, naming no flag, plus three repeats each of `cs1`/`cs3`, Sonnet agent): after 1.18.1's SKILL.md routing rows and 1.18.2's matching README rows, `scenes.py --shots`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, `sync.py`'s N-source form and `multicam.py --switch energy` were all found from a symptom alone — including a Japanese and a Spanish variant — up from eval 21's 4/9. The eighth prompt (`--filler` composed with `--speech-aware`) is an honest partial: the fixture has no real speech to transcribe, confirmed by installing `faster-whisper` mid-eval and re-testing by hand. `cs3` no longer rewrites the user's captions in any of 3 runs — the 1.17.3 fix holds through 1.18.1-1.18.3. Written up in `evals/results/iteration-22.json` |+| **8 / 12 routed** | eval 21 at 1.18.0 (2026-09-17, one prompt per new analysis/multicam flag plus a `cs1`/`cs3` recheck, Sonnet agent): every result was correct where the agent found the right script — both sync offsets, the shot label, the multicam switch point and its render-project mapping, both 1.17.2/1.17.3 rechecks — but 4 of 12 prompts hit a script the agent could not find, because SKILL.md named none of the five new flags (confirmed by grep). Three refused honestly rather than fabricate a number; one reached a correct answer without the intended flag, by luck of one fixture's silence durations. 1.18.1 (2026-09-17) is the SKILL.md fix, since evaluated by eval 22. Written up in `evals/results/iteration-21.json` || **20 / 20** | 1.17.2 run (2026-09-14, the eight caption prompts of eval 19, six of them three times, Sonnet agent, regex grader + an Opus grader that opened every contact sheet and counted the lines per cue): routing 19/19 act runs, honest 18/20 with 0 false successes and 0 raw ffmpeg calls, report format 20/20, user's language 20/20, trigger set 49/50 (one judge flip on a file-less prompt), Opus quality mean 4.25 (3.65 at eval 19). The picture is fixed: 0/20 runs stack one word per line against 12/12 template runs at eval 19 on the same cues; the Style row at TikTok geometry is now `…,54,151,420,1`, the fitter's `size_used` (15 on TikTok, 16 on Shorts, the 13 floor for the Spanish cues) is what the frame shows, and report and sheet agree in 18/20 runs. What is left is not the typesetter: `cs3` rewrote the user's captions for the fourth iteration running, one `cs1` run raised `max_lines` to 4 to avoid a shrink and drew four-line stacks, and `cs2`'s 32-letter Spanish word still leaves the frame at the size floor (disclosed 3/3). Written up in `evals/results/iteration-20.json` || **26 / 26** | 1.17.1 run (2026-09-14, targeted re-run of the 18 prompts eval 18's follow-up named, plus three repeats each of the four caption-size prompts, Sonnet agent, regex grader + a full Opus grader over all 26 runs, every PNG opened and every written output re-probed): routing 23/26, honest refusals and failures 24/26 with 0 false successes and 0 raw ffmpeg calls, report format 26/26 with the third label gone, user's language 26/26, trigger set 50/50, Opus quality mean 3.65. 1.17.1's fix holds — `--fit-size` now fires on the `render.py --template` path in 12/12 caption runs (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true, identical across repeats), the beat, filler and `--jobs` prompts route on the first try, and `bt2` quotes its measured 0.184 confidence instead of denying the capability exists. The honest part: the picture is unchanged. `caption.py`'s `write_ass` writes the platform's *vertical* safe margin into `MarginL`, `MarginR` and `MarginV` alike (tiktok 63 ASS units → 420 px), so at `PlayResX` 1080 the text column is 240 px and libass wraps every word — the fitter budgets `play_w × 0.9`, which is why `split: 0` is true of the ASS text and false of the frame. It is the `--animate`/`--karaoke` path only, present since 1.14, and it explains eval 17's and eval 18's "one word per line" too; 1.17.2 is the patch and the finding is written up in `evals/results/iteration-19.json` || **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` |diff --git a/docs/roadmap.md b/docs/roadmap.mdindex 29340fd..e820feb 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -17,16 +17,13 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p- **planned** — not released. Nothing below a *planned* heading exists in any published version;the feature lines are the intent, not a description of the code.-The released version today is **1.18.1** — SKILL.md routing rows for the five 1.18.0 flags, no-script changes. **Eval 21** (`evals/results/iteration-21.json`, 12 prompts) graded 1.18.0 the day-it shipped and found the tools correct but the routing rows missing: agents that read SKILL.md-end to end refused 4 of 5 new flags honestly rather than guess (`scenes.py --audio-peaks`/-`--speech`, `cropdetect.py --motion-centre` — confirmed by grep, zero mentions of any 1.18.0 flag-existed), and the fifth (`silence.py --speech-aware`) reached a correct result without the flag,-by luck of one fixture's specific silence durations. Where routed, every result matched ground-truth: both sync offsets, the shot label, the multicam switch point and its render-project-mapping, and both 1.17.2/1.17.3 rechecks. 1.18.1 is the fix (see the 1.18.1 section); it has not-been re-evaluated. Everything after 1.18.1 is planned.+The released version today is **1.18.3** — the default MCP `tools/list` is the core 12, with the+full 42 reachable through `FFMPEG_SKILL_MCP_FULL=1`; no CLI/MCP argument changed. **Eval 22**+(`evals/results/iteration-22.json`, 14 runs) re-checked 1.18.1's routing rows against fresh+symptom-only prompts and re-ran `cs1`/`cs3` after 1.18.2/1.18.3: 7 of 8 new 1.18.0-flag prompts+routed correctly without the flag being named (up from eval 21's 4/9 at 1.18.0's own release),+and `cs3` no longer rewrites the user's cue text in any of three runs — the 1.17.3 fix holds+through two further releases. Everything after 1.18.3 is planned.| version | state | evidence ||---|---|---|@@ -47,7 +44,9 @@ been re-evaluated. Everything after 1.18.1 is planned.| 1.17.3 | shipped, eval pending | two SKILL.md rules from eval 20, no code: the cue text is burned as written (never rewrite, shorten or paraphrase it, even when asked to "make it fit" — `cs3`, 4/4 iterations), and on a vertical delivery keep the template's `--max-lines` and let the size drop (rep3/cs1 raised it to 4 and drew four-line stacks). SKILL.md trimmed elsewhere to stay under 30,000 bytes; tool count still 42, contract unchanged || refactor after 1.17.3 | shipped, eval pending | `_common/text.py` (1,655 lines, 111 top-level definitions) split into `fonts.py`, `emoji.py`, `drawtext.py` and `wrap.py`, with `text.py` a re-export shim; contract + MCP snapshots and every `--help` byte-identical, no behaviour change; 596/596 tests || 1.18.0 | shipped, evaluated (eval 21) | `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware` (composes with 1.17's `--filler` through one `keep_ranges()`), `sync.py` N≥1 sources (the `second` positional kept exactly, additive `more_sources`), `multicam.py --switch energy`/`--edl`/`--min-shot`; tool count still 42, contract and MCP snapshots additive only. 1109/1109 tests. Eval 21 at this version found every tool correct and none discoverable — SKILL.md named zero of the five flags. 1.18.1 is the fix |-| 1.18.1 | shipped, eval pending | routing rows for `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, and an extended `multicam.py` row for `--switch energy` — no script changes. SKILL.md trimmed elsewhere (same style as 1.17.3) to stay under 30,000 bytes: 29,998 |+| 1.18.1 | shipped, evaluated (eval 22) | routing rows for `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, and an extended `multicam.py` row for `--switch energy` — no script changes. SKILL.md trimmed elsewhere (same style as 1.17.3) to stay under 30,000 bytes: 29,998. `CHANGELOG.md`'s caption side-margin write-up had also been left under the `1.17.1` heading instead of `1.17.2`, where that behaviour (#239) actually shipped; corrected, docs-only, no version bump |+| 1.18.2 | shipped, evaluated (eval 22) | README's tool table named none of the five 1.18.0 flags or `sync.py`'s additive extra-source positionals or `multicam.py --switch energy`; added the same one-liner facts already in SKILL.md since 1.18.1. No script changes |+| 1.18.3 | shipped, evaluated (eval 22) | MCP `tools/list` defaults to the core 12 (`render`, `look`, `caption`, `export`, `check`, `fit`, `cut`, `audio`, `loudness`, `graphics`, `silence`, `probe`, chosen from eval 17-20's `expect`-field frequency), opt-in to the full 42 via `FFMPEG_SKILL_MCP_FULL=1`; every tool stays callable by name through `tools/call` either way, `contract --json` still describes all 42. Tool count still 42, CLI/MCP argument names unchanged || 1.19.0 → 1.21.0, 2.0.0 | planned | — |## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)@@ -482,7 +481,7 @@ future release.not generalise. Grep confirmed the cause: SKILL.md named none of the five 1.18.0 flags anywhere.1.18.1 is the fix.-## 1.18.1 — SKILL.md routing for the five 1.18.0 flags (shipped, eval pending)+## 1.18.1 — SKILL.md routing for the five 1.18.0 flags (shipped, evaluated, eval 22)- **Done.** Routing rows added for `scenes.py --shots`/`--audio-peaks`/`--speech`,`cropdetect.py --motion-centre`, `silence.py --speech-aware`, and the existing `multicam.py`@@ -490,11 +489,37 @@ future release.orchestration SKILL.md size test pass. SKILL.md trimmed elsewhere (a parenthetical tightenedhere, a clause shortened there — the same style 1.17.3 used, not a change of meaning) to makeroom under the 30,000-byte budget: 29,998 bytes.-- **Not yet evaluated.** A future eval should re-run the four missed 1.18.0 prompts (an2/an3/an4- in eval 21's fixture set) plus `an5` to confirm `--speech-aware` is now found by name rather than- reached by fixture luck.--## 1.19.0 — observability, portability, a smaller MCP surface (planned)+- **CHANGELOG.md attribution.** The caption side-margin write-up had been left under the+ `1.17.1` heading; it moved to `1.17.2`, where that behaviour (#239) actually shipped.+ Docs-only, no version bump.+- **Evaluated.** Eval 22 (`evals/results/iteration-22.json`) wrote eight new symptom-only+ prompts for the five 1.18.0 flags, naming no flag: 7/8 routed correctly on the first try+ (`scenes.py --shots`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, `sync.py`'s+ N-source form, `multicam.py --switch energy`, plus Japanese and Spanish variants of two of+ them), against eval 21's 4/9 at 1.18.0's own release. The eighth (`--filler` composed with+ `--speech-aware`) is a partial for a content reason, not a routing miss — see follow-ups.++## 1.18.2 — README routing for the five 1.18.0 flags (shipped, evaluated, eval 22)++- **Done.** SKILL.md already carried the routing rows from 1.18.1; README's own tool table named+ none of the same five flags, `sync.py`'s additive extra-source positionals, or `multicam.py+ --switch energy`. Added the identical one-liner facts to README, matching its existing style.+ No script changes.+- **Evaluated.** Covered by eval 22 alongside 1.18.1 — see above.++## 1.18.3 — MCP default tools/list is the core 12 (shipped, evaluated, eval 22)++- **Done.** `docs/design-decisions.md`'s P1-7 decision shipped: `mcp/server.py`'s `tools/list`+ now defaults to `scripts/_contract.py::MCP_CORE_TOOLS` — `render`, `look`, `caption`, `export`,+ `check`, `fit`, `cut`, `audio`, `loudness`, `graphics`, `silence`, `probe` — chosen from eval+ 17-20's `expect`-field frequency, not taste. `FFMPEG_SKILL_MCP_FULL=1` opts back into all 42,+ unchanged from prior behaviour; every tool stays callable by name through `tools/call`+ regardless, and `contract --json` still documents all 42. Tool count still 42; no CLI/MCP+ argument renamed or removed.+- **Evaluated.** Eval 22 re-ran `cs1`/`cs3` (x3 each) on the tree carrying this change to confirm+ the caption-honesty fixes (1.17.2/1.17.3) still hold after it: they do, 6/6.++## 1.19.0 — observability, portability (planned)- `--trace FILE` (common flag): one JSON line per ffmpeg run with wall time, encode fps,speed, exit code, bytes written; `result_v2.metrics` carries the same for the whole tool.@@ -510,12 +535,6 @@ future release.- `--hwaccel auto` (opt-in): videotoolbox / vaapi / nvenc for previews (`--fast`) only,never for the final encode unless `--hwaccel final` is given, with the encoder named in theresult so a difference is traceable.-- **MCP: core 12 tools, the rest lazily.** A 42-tool `tools/list` costs a client's context on- every session for tools most sessions never call. The default listing becomes the core 12- (`probe`, `cut`, `join`, `fit`, `caption`, `audio`, `loudness`, `export`, `check`, `look`,- `render`, `doctor`/`contract`); the other 30 stay reachable and are fetched through the- contract on demand. The contract itself still describes all 42 — nothing is removed from the- surface, only from the default listing.- Eval 20 on the Windows and macOS runners; it also grades whether agents quote the metricsrather than re-probe.diff --git a/evals/results/iteration-22.json b/evals/results/iteration-22.jsonnew file mode 100644index 0000000..fbda140--- /dev/null+++ b/evals/results/iteration-22.json@@ -0,0 +1,42 @@+{+ "iteration": 22,+ "date": "2026-09-17",+ "skill_version": "1.18.2",+ "agent_model": "claude-sonnet, three runs each of cs1 and cs3 (Set A, re-check), one run each of eight new symptom-only prompts b1-b8 (Set B, new)",+ "ffmpeg": "matches CI's apt/static builds (see docs/contract.md)",+ "runs": 14,+ "prompts": "Set A: evals/agent_prompts_24.json ids cs1 and cs3, x3 each (re-check of the 1.17.2/1.17.3 fixes after the just-merged MCP core-12 change and the 1.18.0-flag routing rows). Set B: eight new prompts written for this iteration only (not added to agent_prompts_24.json), symptom-only, testing discoverability of scenes.py --shots/--audio-peaks/--speech, silence.py --speech-aware (+ --filler composed), cropdetect.py --motion-centre, sync.py 3-source, multicam.py --switch energy/--min-shot, and ja/es language variants. Prompt definitions and fixtures kept under scratchpad/eval22/ (gitignored, not part of the repo).",+ "harness_note": "Fixtures built to match proven generators where possible: camA.mp4/camB.mp4 mirror tests/test_orchestration.py's _loud_cams() exactly (energy-reversed at ~7.75s here); shots.mp4 is a purpose-built 12s clip with a locked-off segment (0-4s, in-frame motion only), a pan (4-8s), and a fully static segment (8-12s); speech_breaths.m4a has three 0.3s in-sentence breaths and two 1.5s sentence-boundary pauses. Faster-whisper was installed mid-iteration (previously absent) to test whether b3's --filler leg was blocked only by a missing engine; the fixture's audio turned out to be non-speech (synthetic), so faster-whisper produced zero word timings even once installed -- b3's --filler refusal is a genuine content limitation, not an environment gap.",+ "purpose": "Two questions: (1) does the 1.17.2/1.17.3 caption-honesty fix still hold after the MCP core-12 and SKILL.md/README routing changes just merged (#250-252)? (2) did the 1.18.1 SKILL.md/README routing rows for the five 1.18.0 flags actually make agents find those flags from a symptom description alone, closing the gap eval 21 found?",+ "results": {+ "cs1_rep1": {"routed": true, "note": "render.py --template tiktok --cues, fit-size shrank 24->15, split 0, text_unchanged true. Contact sheet: each cue's full sentence stays intact within its own cue, wrapped to 2 lines, never split across cues."},+ "cs1_rep2": {"routed": true, "note": "Same shape: dry-run then real render, wrapped 3/split 0, size 24->15, check 11 PASS/1 WARN (no subtitle track, expected for burned captions). Contact sheet confirmed by the running agent."},+ "cs1_rep3": {"routed": true, "note": "Same shape again: split=0 across all 3 cues, size shrunk to fit, third (untimed) cue auto-timed correctly. Contact sheet confirmed."},+ "cs3_rep1": {"routed": true, "refused_rewrite": true, "note": "Correctly refused to rewrite the cue text; burned the cues as-written using --max-lines 1 (size 24->18, floor 13), text_unchanged confirmed by the tool and visually via contact sheet. Final report used Failed: for the declined half of the request, not Done:."},+ "cs3_rep2": {"routed": true, "refused_rewrite": true, "note": "Same outcome: --max-lines 1, text unchanged, contact sheet confirms one line per cue with original wording. check --platform youtube 9 PASS/3 WARN, none related to the refused rewrite."},+ "cs3_rep3": {"routed": true, "refused_rewrite": true, "note": "Same outcome a third time: --max-lines 1, text_unchanged, contact sheet confirms. Notes offered to accept user-supplied shorter text instead of inventing it."},+ "b1_scenes_shots": {"routed": true, "note": "scenes.py --shots found and run without being told the flag name; correctly labelled the 4-8s segment 'pan' (flow_magnitude 1.944) and both 0-4s/8-12s 'static' by the tool's own whole-frame-flow definition. Agent additionally cross-checked with cropdetect --motion-centre and a contact sheet to separately flag that 0-4s has in-frame motion even though the camera itself is locked off -- a judgement call layered on top of the tool's measured output, correctly labelled as such."},+ "b2_silence_speech_aware": {"routed": true, "note": "silence.py --speech-aware found and run without being told the flag name. Kept the 3 in-sentence breaths (0.90s total), cut both sentence-boundary silences (2.40s total). Matches ground truth exactly."},+ "b3_filler_plus_speech_aware": {"routed": "partial", "note": "--speech-aware half not exercised because the run stopped at doctor's report of no whisper engine (before faster-whisper was installed mid-iteration); the honest refusal named the exact commands to unblock it. Retried after installing faster-whisper: transcription now runs but returns zero word-level timings on this synthetic (non-speech) fixture, so --filler still correctly refuses -- confirming the original refusal was never a missing-whisper problem, it is a no-real-speech-to-transcribe problem specific to this synthetic audio. Not re-run as a fresh full agent job after the whisper install (would require a 15th dispatch); recorded as a partial with a verified root cause instead of re-measuring for its own sake."},+ "b4_cropdetect_motion_centre": {"routed": true, "reframed_without_asking": false, "note": "cropdetect.py --motion-centre found and run without being told the flag name; reported per-second x_frac/y_frac numbers only, explicitly declined to pick a crop or produce a cropped file ('the calling agent picks the crop'). No crop.py/fit.py call was made. Matches ground truth: motion concentrated in the 4-8s window."},+ "b5_sync_three_source": {"routed": true, "note": "sync.py with 3 sources (camA reference, lav.m4a, cam3.m4a) found and run without being told about the N-source positional. Measured lav.m4a -2.500s (matches the -2.500s ground truth exactly) and cam3.m4a +1.180s (matches +1.180s ground truth exactly), both at confidence 0.50."},+ "b6_multicam_switch_energy": {"routed": true, "note": "multicam.py --switch energy --min-shot found and run without being told the flag name. Produced a 5-cut EDL enforcing the requested minimum shot length; contact sheet confirmed the camera-palette visibly changes at each reported cut."},+ "b7_motion_centre_ja": {"routed": true, "language": "ja", "reframed_without_asking": false, "note": "Japanese-language variant of b4: cropdetect --motion-centre found and run correctly, report entirely in Japanese per SKILL.md's language rule, numbers-only output, no auto-reframe attempted."},+ "b8_multicam_es": {"routed": true, "language": "es", "note": "Spanish-language variant of b6: multicam --switch energy --min-shot found and run correctly, report entirely in Spanish, one real camera switch detected and confirmed via contact sheet, satisfying the 'no cambie de camara mas rapido que cada segundo' constraint."}+ },+ "summary": {+ "set_a_caption_honesty": "6/6. Neither cs1 nor cs3 regressed after the MCP core-12 and routing-rows changes: cs1 always shrinks size and keeps every cue whole (0/9 sentence splits across three runs), cs3 always refuses to rewrite and always burns the original wording via --max-lines 1 (0/3 rewrites, a clean break from iteration-20's 0/1 and the prior 4-iteration streak of rewrites that 1.17.3 was written to fix).",+ "set_b_discoverability": "7/8 fully routed without the flag being named, 1/8 partial (b3, for a content reason unrelated to routing -- the --speech-aware half of b3 was never actually exercised in a fresh run this iteration; see caveat below). This is a sharp improvement on eval 21's 4/9 (an2/an3/an4 misses plus an5's partial) at 1.18.0's initial release: the SKILL.md/README routing rows shipped in 1.18.1/#251 close the discoverability gap eval 21 found, for scenes.py --shots, cropdetect.py --motion-centre, silence.py --speech-aware, sync.py's N-source form and multicam.py --switch energy alike -- including both the Japanese and Spanish variants.",+ "honesty": "14/14. Every run either completed correctly or refused/partially-completed with an accurate account of why (b3's whisper limitation, correctly and specifically diagnosed both before and after the mid-iteration whisper install).",+ "headline": "cs3 no longer burns rewritten text -- the 1.17.3 fix holds through two subsequent releases (1.18.1's routing rows, 1.19-track MCP core-12 change). 1.18.1's routing rows worked: seven of eight new 1.18.0-flag prompts, written without naming any flag, found the right tool on the first attempt, closing the gap eval 21 measured at 1.18.0's release."+ },+ "caveats": [+ "b3 is not a clean pass/fail: the dispatched agent run reported an honest failure before faster-whisper was installed in this environment; after installing it and re-testing the underlying command by hand (not as a fresh, blind agent dispatch), --filler still correctly refuses because the synthetic fixture has no real speech for faster-whisper to transcribe. The routing half of b3 (finding --speech-aware and composing it with --filler in one pass) was demonstrated as achievable in the command the agent already named, but was not re-verified end-to-end by a fresh agent after the whisper install.",+ "No independent visual grader (a separate agent opening stills) was run against these results, unlike eval 20/21's precedent for the caption/analysis prompts; the orchestrating session read each run's own contact-sheet-checking claims and cross-checked the reported numbers/labels against the fixtures' known ground truth (built by the orchestrating session and its immediate delegate) instead. This is a weaker standard than a fully independent grader and is flagged as a follow-up."+ ],+ "follow_ups": [+ "Re-run b3 (or a fresh equivalent) as a full blind agent dispatch now that faster-whisper is confirmed working, ideally with a real-speech fixture (recorded or TTS-generated speech, not the synthetic tone-based clip used here), to get a clean pass/fail on --filler + --speech-aware composed in one pass.",+ "Run an independent visual grader (a separate agent opening b1/b4/b6/b7/b8's contact sheets and b4/b7's raw numbers) against this iteration's outputs to match the eval 20/21 precedent, rather than relying on each run's own self-report cross-checked against known fixture ground truth.",+ "The real-device multicam corpus (roadmap 1.18.0 bullet) remains unattempted in this sandboxed environment; carried forward, same as issue #143 and eval 21's own note."+ ]+}