Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: eval 21 at 1.18.0 (12 prompts, discoverability gap found) and 1.18.1 shipped (#249)

Shogo Kajiyamaf4049d2382fc
diff --git a/README.md b/README.md
index c6e4723..7e6c60a 100644
--- a/README.md
+++ b/README.md
@@ -416,6 +416,7 @@ type on every OS.
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **8 / 12 routed** | eval 21 at 1.18.0 (2026-09-17, one prompt per new analysis/multicam flag plus a `cs1`/`cs3` recheck, Sonnet agent): every result was correct where the agent found the right script — both sync offsets, the shot label, the multicam switch point and its render-project mapping, both 1.17.2/1.17.3 rechecks — but 4 of 12 prompts hit a script the agent could not find, because SKILL.md named none of the five new flags (confirmed by grep). Three refused honestly rather than fabricate a number; one reached a correct answer without the intended flag, by luck of one fixture's silence durations. 1.18.1 (2026-09-17) is the SKILL.md fix, not yet re-evaluated. Written up in `evals/results/iteration-21.json` |
| **20 / 20** | 1.17.2 run (2026-09-14, the eight caption prompts of eval 19, six of them three times, Sonnet agent, regex grader + an Opus grader that opened every contact sheet and counted the lines per cue): routing 19/19 act runs, honest 18/20 with 0 false successes and 0 raw ffmpeg calls, report format 20/20, user's language 20/20, trigger set 49/50 (one judge flip on a file-less prompt), Opus quality mean 4.25 (3.65 at eval 19). The picture is fixed: 0/20 runs stack one word per line against 12/12 template runs at eval 19 on the same cues; the Style row at TikTok geometry is now `…,54,151,420,1`, the fitter's `size_used` (15 on TikTok, 16 on Shorts, the 13 floor for the Spanish cues) is what the frame shows, and report and sheet agree in 18/20 runs. What is left is not the typesetter: `cs3` rewrote the user's captions for the fourth iteration running, one `cs1` run raised `max_lines` to 4 to avoid a shrink and drew four-line stacks, and `cs2`'s 32-letter Spanish word still leaves the frame at the size floor (disclosed 3/3). Written up in `evals/results/iteration-20.json` |
| **26 / 26** | 1.17.1 run (2026-09-14, targeted re-run of the 18 prompts eval 18's follow-up named, plus three repeats each of the four caption-size prompts, Sonnet agent, regex grader + a full Opus grader over all 26 runs, every PNG opened and every written output re-probed): routing 23/26, honest refusals and failures 24/26 with 0 false successes and 0 raw ffmpeg calls, report format 26/26 with the third label gone, user's language 26/26, trigger set 50/50, Opus quality mean 3.65. 1.17.1's fix holds — `--fit-size` now fires on the `render.py --template` path in 12/12 caption runs (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true, identical across repeats), the beat, filler and `--jobs` prompts route on the first try, and `bt2` quotes its measured 0.184 confidence instead of denying the capability exists. The honest part: the picture is unchanged. `caption.py`'s `write_ass` writes the platform's *vertical* safe margin into `MarginL`, `MarginR` and `MarginV` alike (tiktok 63 ASS units → 420 px), so at `PlayResX` 1080 the text column is 240 px and libass wraps every word — the fitter budgets `play_w × 0.9`, which is why `split: 0` is true of the ASS text and false of the frame. It is the `--animate`/`--karaoke` path only, present since 1.14, and it explains eval 17's and eval 18's "one word per line" too; 1.17.2 is the patch and the finding is written up in `evals/results/iteration-19.json` |
| **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` |
diff --git a/docs/roadmap.md b/docs/roadmap.md
index e8bb15e..29340fd 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -17,17 +17,16 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
- **planned** — not released. Nothing below a *planned* heading exists in any published version;
the feature lines are the intent, not a description of the code.
-The released version today is **1.18.0** — measured analysis and multicam at scale, shipped and
-not yet evaluated (eval 21 covers the analysis/multicam prompts and re-runs `cs1`/`cs3` for the
-1.17.3 SKILL.md lines). **1.17.2** is the last evaluated version: **eval 20**
-(`evals/results/iteration-20.json`) re-ran the eight caption prompts of eval 19, six of them three
-times, and graded them on the contact sheet rather than on the fit stats. The one-word-per-line
-stacking of evals 17, 18 and 19 is gone in 20/20 runs; the caption Style's side margins are the
-platform's horizontal safe zone and the fitter's numbers are what the frame shows (see the 1.17.2
-section). Two agent behaviours remained — rewriting the user's captions (`cs3`, four iterations)
-and raising `max_lines` to dodge a shrink — closed in 1.17.3's SKILL.md lines; the word a
-typesetter cannot break inside is a 1.18.0+ design item (see below). Everything after 1.18.0 is
-planned.
+The released version today is **1.18.1** — SKILL.md routing rows for the five 1.18.0 flags, no
+script changes. **Eval 21** (`evals/results/iteration-21.json`, 12 prompts) graded 1.18.0 the day
+it shipped and found the tools correct but the routing rows missing: agents that read SKILL.md
+end to end refused 4 of 5 new flags honestly rather than guess (`scenes.py --audio-peaks`/
+`--speech`, `cropdetect.py --motion-centre` — confirmed by grep, zero mentions of any 1.18.0 flag
+existed), and the fifth (`silence.py --speech-aware`) reached a correct result without the flag,
+by luck of one fixture's specific silence durations. Where routed, every result matched ground
+truth: both sync offsets, the shot label, the multicam switch point and its render-project
+mapping, and both 1.17.2/1.17.3 rechecks. 1.18.1 is the fix (see the 1.18.1 section); it has not
+been re-evaluated. Everything after 1.18.1 is planned.
| version | state | evidence |
|---|---|---|
@@ -47,7 +46,8 @@ planned.
| 1.17.2 | shipped, evaluated (eval 20) | eval 20 at 1.17.2 (`iteration-20.json`), 20 runs over the eight caption prompts; tool count still 42, contract additive only. The patch holds on the picture: 0/20 runs stack one word per line (eval 19: 12/12 template runs), Style at TikTok `…,54,151,420,1`, `size_used` 15/16/13 matches the sheets, report and picture agree 18/20, Opus quality 4.25 (3.65). Left over and not the typesetter's: `cs3` rewrites the user's text (4/4 iterations), one `cs1` run raised `max_lines` to 4 and drew four-line stacks, `cs2`'s 32-letter word leaves the frame at the 13 floor, disclosed 3/3 |
| 1.17.3 | shipped, eval pending | two SKILL.md rules from eval 20, no code: the cue text is burned as written (never rewrite, shorten or paraphrase it, even when asked to "make it fit" — `cs3`, 4/4 iterations), and on a vertical delivery keep the template's `--max-lines` and let the size drop (rep3/cs1 raised it to 4 and drew four-line stacks). SKILL.md trimmed elsewhere to stay under 30,000 bytes; tool count still 42, contract unchanged |
| refactor after 1.17.3 | shipped, eval pending | `_common/text.py` (1,655 lines, 111 top-level definitions) split into `fonts.py`, `emoji.py`, `drawtext.py` and `wrap.py`, with `text.py` a re-export shim; contract + MCP snapshots and every `--help` byte-identical, no behaviour change; 596/596 tests |
-| 1.18.0 | shipped, eval pending | `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware` (composes with 1.17's `--filler` through one `keep_ranges()`), `sync.py` N≥1 sources (the `second` positional kept exactly, additive `more_sources`), `multicam.py --switch energy`/`--edl`/`--min-shot`; tool count still 42, contract and MCP snapshots additive only, SKILL.md unchanged (29,976 bytes) — every reported number is measured, none is a judgement. 1109/1109 tests |
+| 1.18.0 | shipped, evaluated (eval 21) | `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware` (composes with 1.17's `--filler` through one `keep_ranges()`), `sync.py` N≥1 sources (the `second` positional kept exactly, additive `more_sources`), `multicam.py --switch energy`/`--edl`/`--min-shot`; tool count still 42, contract and MCP snapshots additive only. 1109/1109 tests. Eval 21 at this version found every tool correct and none discoverable — SKILL.md named zero of the five flags. 1.18.1 is the fix |
+| 1.18.1 | shipped, eval pending | routing rows for `scenes.py --shots`/`--audio-peaks`/`--speech`, `cropdetect.py --motion-centre`, `silence.py --speech-aware`, and an extended `multicam.py` row for `--switch energy` — no script changes. SKILL.md trimmed elsewhere (same style as 1.17.3) to stay under 30,000 bytes: 29,998 |
| 1.19.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -439,7 +439,7 @@ came out byte-identical, as did `--help` for all 42 tools.
automatic) and a few parentheticals to stay under the 30,000-byte budget. Eval 21 at 1.18.0
re-runs cs1 and cs3 to check both lines land.
-## 1.18.0 — measured analysis and multicam at scale (still no judgement) (shipped, eval pending)
+## 1.18.0 — measured analysis and multicam at scale (still no judgement) (shipped, evaluated, eval 21)
- `scenes.py --shots` labels each shot static / pan / motion by measured optical flow;
`--audio-peaks` and `--speech` (speech-vs-music energy ratio) as separate lists.
@@ -463,10 +463,36 @@ dests directly into the MCP `inputSchema`, so a rename would have broken the CLI
guarantee) — a new optional `more_sources` positional carries the extra cameras/recorders instead,
and a 1-source run keeps the original 2-source JSON shape with the same numbers additively
available under `sources`. A multicam `--switch energy` timeline needed no new `render.py` project
-stage — it maps onto the existing `clips[]` array. Eval 21 (this bullet's last line) and the
-real-device corpus have not run yet. The caption word-too-wide-for-the-frame item from 1.17.2's
-eval (`cs2`, a break-at-the-column-edge escape for the 1.16.1 keep-the-run-whole rule) is still
-unaddressed and is not part of this section — it stays an open design item for a future release.
+stage — it maps onto the existing `clips[]` array. The caption word-too-wide-for-the-frame item
+from 1.17.2's eval (`cs2`, a break-at-the-column-edge escape for the 1.16.1 keep-the-run-whole
+rule) is still unaddressed and is not part of this section — it stays an open design item for a
+future release.
+
+- **Eval 21 (`iteration-21.json`) graded it the day it shipped**: 12 synthetic prompts (no real
+ cameras or footage available in this environment; the roadmap's real-device multicam corpus has
+ not run — carried forward, same as issue #143), one per new flag plus a re-check of `cs1`/`cs3`.
+ The tools measured correctly every time they were reached: both sync offsets (2.5 s and 1.18 s,
+ both within 0.15 s), the shot label on a panning clip, the multicam switch point and its
+ render-project mapping (verified by dry-running a hand-built project.json against multicam's own
+ cuts), and both 1.17.2/1.17.3 rechecks. What was not reached: `scenes.py --audio-peaks` and
+ `--speech`, `cropdetect.py --motion-centre` were each refused by an agent that read SKILL.md end
+ to end and, correctly finding no routing row, declined to guess rather than fabricate a number —
+ the honest failure mode, but a failure mode all the same. `silence.py --speech-aware` reached a
+ correct result without the flag, by luck of one fixture's specific silence durations, which would
+ not generalise. Grep confirmed the cause: SKILL.md named none of the five 1.18.0 flags anywhere.
+ 1.18.1 is the fix.
+
+## 1.18.1 — SKILL.md routing for the five 1.18.0 flags (shipped, eval pending)
+
+- **Done.** Routing rows added for `scenes.py --shots`/`--audio-peaks`/`--speech`,
+ `cropdetect.py --motion-centre`, `silence.py --speech-aware`, and the existing `multicam.py`
+ row extended for `--switch energy`. No script changes; `tests.test_contract -k skill` and the
+ orchestration SKILL.md size test pass. SKILL.md trimmed elsewhere (a parenthetical tightened
+ here, a clause shortened there — the same style 1.17.3 used, not a change of meaning) to make
+ room under the 30,000-byte budget: 29,998 bytes.
+- **Not yet evaluated.** A future eval should re-run the four missed 1.18.0 prompts (an2/an3/an4
+ in eval 21's fixture set) plus `an5` to confirm `--speech-aware` is now found by name rather than
+ reached by fixture luck.
## 1.19.0 — observability, portability, a smaller MCP surface (planned)
diff --git a/evals/results/iteration-21.json b/evals/results/iteration-21.json
new file mode 100644
index 0000000..6f81603
--- /dev/null
+++ b/evals/results/iteration-21.json
@@ -0,0 +1,69 @@
+{
+ "iteration": 21,
+ "date": "2026-09-17",
+ "skill_version": "1.18.0",
+ "agent_model": "claude-sonnet, one run per prompt",
+ "prompts": "12 synthetic prompts covering every 1.18.0 flag once, plus a re-check of the two 1.17.3 SKILL.md caption rules",
+ "harness_note": "Synthetic fixtures built for this iteration (no real cameras available in this environment): camA.mp4/camB.mp4 (two views of one event, energy-reversed at 6s, matches tests/test_orchestration.py's proven _loud_cams() pattern), lav.m4a (camA's audio delayed 2.5s), cam3.m4a (camA's audio, +1.18s), panning.mp4 (a moving test pattern for --shots/--motion-centre), static.mp4, speech_breaths.m4a (two short breaths + one real pause). The roadmap's real-device multicam corpus (phone + camera + lav) was not attempted: this environment has no cameras and issue #143 (real-device corpus, general) has no acceptance criteria and is carried as an open item, same precedent.",
+ "results": {
+ "an1_scenes_shots": {
+ "routed": true,
+ "note": "scenes.py --shots on the panning clip: 1 shot, label motion, flow_magnitude 1.111 -- matches the ground truth (the clip has no hard cuts, is a single continuous pan)."
+ },
+ "an2_scenes_audio_peaks": {
+ "routed": false,
+ "note": "MISS. Asked for a per-second volume-peak list; the agent ran probe.py and loudness.py --measure-only (whole-file only) and refused, reporting no script gives per-second data. scenes.py --audio-peaks does exactly this (new in 1.18.0) and was never tried."
+ },
+ "an3_scenes_speech": {
+ "routed": false,
+ "note": "MISS. Asked for a speech-vs-music ratio; the agent read SKILL.md, found no routing row, and refused honestly rather than fabricate -- the refusal itself was correct behaviour, but scenes.py --speech (new in 1.18.0) exists and was never discovered."
+ },
+ "an4_cropdetect_motion_centre": {
+ "routed": false,
+ "note": "MISS. Asked for a per-second motion-centre reading (explicitly to crop later, not asking the skill to crop); the agent used look.py's contact sheet and refused with 'picking a subject is outside this skill's scope' -- correct reasoning for a different question (this one asked for a measured point, not a subject) but it never tried cropdetect.py --motion-centre (new in 1.18.0), which is exactly report-only and exists for this."
+ },
+ "an5_silence_speech_aware": {
+ "routed": "partial",
+ "note": "The agent solved the task (kept both breaths, cut only the real pause) but never used --speech-aware -- it ran silence.py at DEFAULT settings and the fixture's specific breath/pause durations happened to fall on either side of the 0.6s default floor. Correct result by fixture luck, not the intended flag; would not generalise to breaths >=0.6s."
+ },
+ "an6_silence_speech_aware_plus_filler": {
+ "routed": true,
+ "note": "--speech-aware routed and worked correctly (kept two breaths, cut the one sentence pause, verified: true). --filler honestly refused (no local whisper, no --words transcript) -- expected in this environment, same as prior evals' fw2."
+ },
+ "sy1_sync_two_source": {
+ "routed": true,
+ "note": "sync.py camA.mp4 lav.m4a: offset -2.500s, confidence 0.447. Matches ground truth (2.5s, magnitude) exactly."
+ },
+ "sy2_sync_three_source": {
+ "routed": true,
+ "note": "sync.py camA.mp4 lav.m4a cam3.m4a: lav.m4a -2.500s, cam3.m4a +1.181s. Matches ground truth for both sources."
+ },
+ "mc1_multicam_switch_energy": {
+ "routed": true,
+ "note": "multicam.py --switch energy --min-shot 1: correct camera-switch cuts around the 6s energy-reversal point, --edl written, verified visually via contact sheet."
+ },
+ "mc2_multicam_to_render_project": {
+ "routed": true,
+ "note": "Correctly explained and DEMONSTRATED that a multicam --switch energy timeline maps onto a render.py project.json (built one by hand from multicam's own cuts/offsets JSON, dry-ran it, reproduced the identical edit) and correctly distinguished that from re-running --switch energy (which would redo the decision). No push-button converter exists -- the agent had to hand-translate cuts[] into clips[]; this is consistent with the 1.18.0 design decision ('no new project stage needed') but confirms there is no one-command path yet."
+ },
+ "cs1r_caption_recheck": {
+ "routed": true,
+ "note": "1.17.2's fix holds: split 0, text_unchanged true, size 24->15, matches the eval-20 result on the same fixture."
+ },
+ "cs3r_refusal_recheck": {
+ "routed": true,
+ "note": "1.17.3's SKILL.md rule holds: the agent refused to rewrite cue text and named --max-lines 1 as the honest alternative, without even attempting caption.py."
+ }
+ },
+ "summary": {
+ "routing": "8/10 analysis+multicam act prompts correct, 1 partial (fixture-lucky, not flag-routed), 4 clear misses (an2, an3, an4, an5-as-flag) all tracing to one cause: SKILL.md added zero routing rows for the five 1.18.0 flags, on the stated assumption that existing rows already covered these requests. That assumption did not hold for scenes.py --audio-peaks/--speech or cropdetect.py --motion-centre.",
+ "honesty": "12/12. Every miss was an honest refusal, not a fabrication -- an2/an3/an4 correctly declined to invent numbers when the agent (wrongly) believed no tool existed; an6's filler refusal was a real missing-whisper case.",
+ "correctness_where_routed": "8/8 correct against ground truth: shot label, both sync offsets (2.5s and 1.18s, both within 0.15s tolerance), multicam switch point and EDL, the project-mapping claim (verified by dry-run), and both 1.17.2/1.17.3 rechecks.",
+ "headline": "1.18.0's tools are correct; 1.18.0's SKILL.md is not discoverable. Zero routing rows were added for any of the five new flags (verified by grep: SKILL.md contains no mention of --shots, --audio-peaks, --speech, --motion-centre or --speech-aware). Four of five were missed by an agent that read SKILL.md end to end and refused rather than guess -- the honest failure mode, but a failure mode all the same; the fifth (an5, --speech-aware) reached a correct result without the flag, by luck of the fixture's specific silence durations. 1.18.1 is the routing patch: rows for all five."
+ },
+ "follow_ups": [
+ "1.18.1: SKILL.md routing rows for scenes.py --shots/--audio-peaks/--speech, cropdetect.py --motion-centre, and silence.py --speech-aware -- none exist today (grep confirms). Byte budget is the constraint (SKILL.md was 29,976/30,000 after 1.17.3); expect another trim pass.",
+ "The real-device multicam corpus (roadmap 1.18.0 bullet, phone + camera + lav) has not run in this environment; carried forward, same as issue #143.",
+ "A push-button multicam-cuts-to-render-project helper does not exist; mc2 needed a hand-built project.json. Consider a `multicam.py --write-project FILE` flag mirroring caption.py's pattern, if this recurs in real use."
+ ]
+}