Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval 19 at 1.17.1 (26 runs, the margin root cause, 1.17.2 queued) (#238)
claude[bot]1c70b50dc84e
diff --git a/README.md b/README.mdindex 41a5bc7..e881dc6 100644--- a/README.md+++ b/README.md@@ -416,6 +416,7 @@ type on every OS.| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **26 / 26** | 1.17.1 run (2026-09-14, targeted re-run of the 18 prompts eval 18's follow-up named, plus three repeats each of the four caption-size prompts, Sonnet agent, regex grader + a full Opus grader over all 26 runs, every PNG opened and every written output re-probed): routing 23/26, honest refusals and failures 24/26 with 0 false successes and 0 raw ffmpeg calls, report format 26/26 with the third label gone, user's language 26/26, trigger set 50/50, Opus quality mean 3.65. 1.17.1's fix holds — `--fit-size` now fires on the `render.py --template` path in 12/12 caption runs (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true, identical across repeats), the beat, filler and `--jobs` prompts route on the first try, and `bt2` quotes its measured 0.184 confidence instead of denying the capability exists. The honest part: the picture is unchanged. `caption.py`'s `write_ass` writes the platform's *vertical* safe margin into `MarginL`, `MarginR` and `MarginV` alike (tiktok 63 ASS units → 420 px), so at `PlayResX` 1080 the text column is 240 px and libass wraps every word — the fitter budgets `play_w × 0.9`, which is why `split: 0` is true of the ASS text and false of the frame. It is the `--animate`/`--karaoke` path only, present since 1.14, and it explains eval 17's and eval 18's "one word per line" too; 1.17.2 is the patch and the finding is written up in `evals/results/iteration-19.json` || **100 / 100** | 1.17.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, every written output re-probed, `check.py` re-run on every delivery output) on the set grown to 100 prompts (caption size fitting, beat-synced cuts, filler removal, batch `--jobs`, render `--cache`): routing 95% over the 64 act prompts, honest refusals and failures 22/25 with 0 false successes and 0 raw ffmpeg calls, report format 98/100 (two runs label an honest partial result with a third label), user's language 100/100 across seventeen languages, visual check 24/24, real execution 6/6 with honest failure 5/5, trigger set 50/50 including all five new 1.17 prompts, Opus quality mean 3.71. The honest part: `--fit-size` is unreachable on the template path (`render.py` forwards the platform table's caption size as an explicit `--size`, so the fitter declines to shrink a size it thinks the user chose, and the project schema rejects `fit_size` outright — only the one run that called `caption.py` by hand got 24 → 16, `split` 0), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most — 1.17.1 is the patch and the finding is written up in `evals/results/iteration-18.json` || **90 / 90** | 1.16.0 run (2026-09-14, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 30 runs, chapters and subtitle streams re-probed, check.py re-run on every delivery output) on the set grown to 90 prompts (audiogram, auto chapters, multi-language tracks, caption breaking): routing 90/90, honest refusals and failures 90/90 with 0 false successes and 0 raw ffmpeg calls, report format 89/90 (one `Done (partially):`), user's language 90/90 by regex (89/90 by Opus), audiogram 2/2 with the cover behind the waveform and nothing fetched, auto chapters 2/2 with `Chapter N` titles only, delivery 16/16 platform pass, trigger set 45/45, Opus quality mean 4.17. The honest part: the phrase breaker never gets to act at the platform caption sizes (a five-word cue does not fit two lines at TikTok size, so the split is byte-identical to 1.15.1), Thai still breaks inside words, and a katakana word was split — 1.16.1 is the patch and the finding is written up in `evals/results/iteration-17.json` || **82 / 82** | 1.15.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, stills extracted inside the emoji window, check.py re-run on every delivery output) on the set grown to 82 prompts (emoji captions and title cards, a Hindi and a Thai lower-third): routing 82/82, honest refusals and failures 82/82 with 0 false successes and 0 raw ffmpeg calls, report format 82/82 (both iteration-15 label defects closed: `dl8` and `he2` now carry one `Failed:`), user's language 82/82 by regex (81/82 by Opus: one Spanish report with three English labels), non-Latin glyphs 11/11 (Devanagari through `graphics.py` is fixed; Thai lower-third and captions correct), emoji visible in colour in 3/3 runs given PNG assets and reported monochrome in the one that was not, visual check 23/24, delivery 12/13 one encode and 13/13 platform pass, trigger set 40/40, Opus quality mean 4.68. Still open: the caption breaker splits phrases (`dl1`, `dl4` unchanged) — queued for 1.16.0. Tokens per run flat at 73.3k on the same 76. Details in `evals/results/iteration-16.json` |diff --git a/docs/roadmap.md b/docs/roadmap.mdindex 85945d5..0a1485e 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -17,11 +17,15 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p- **planned** — not released. Nothing below a *planned* heading exists in any published version;the feature lines are the intent, not a description of the code.-The released version today is **1.17.0**, shipped and evaluated: **eval 18**-(`evals/results/iteration-18.json`) graded it on the 100-prompt set, the 87 carried over plus-thirteen written for the 1.17 features. It found the size fitter is unreachable on the template-path and that SKILL.md routes none of the 1.17 features (see the 1.17.0 section); **1.17.1 below-is the patch and is planned for release** — implemented, not tagged yet. Everything after 1.17.1 is planned.+The released version today is **1.17.1**, shipped and evaluated: **eval 19**+(`evals/results/iteration-19.json`) re-ran the 18 prompts eval 18's follow-up named, three of them+three times over, and confirmed the patch — the size fitter fires on the `render.py --template`+path in 12/12 caption runs, and the beat, filler and `--jobs` prompts route on the first try. It+also root-caused a defect the two previous patches had been aiming at symptoms of: `caption.py`'s+generated-ASS path writes the platform's *vertical* safe margin into `MarginL` and `MarginR` too,+so libass lays every caption into a 240 px column and stacks one word per line (see the 1.17.1+section). **1.17.2 below is the patch and is planned for release.** Everything after 1.17.2 is+planned.| version | state | evidence ||---|---|---|@@ -37,7 +41,8 @@ is the patch and is planned for release** — implemented, not tagged yet. Every| 1.16.0 | shipped + evaluated | eval 17 at 1.16.0 (`iteration-17.json`); contract and MCP snapshots additive only; tool count still 42 || 1.16.1 | shipped + evaluated | caption-break patch from eval 17: a Thai run and a katakana word are never broken inside, `caption.py` reports `overlong` lines; eval 18 graded the tree that carries it and reported no wrapping defect || 1.17.0 | shipped, evaluated (eval 18) | eval 18 at 1.17.0 (`iteration-18.json`); tool count still 42; contract and MCP snapshots additive only. Two findings: `render.py` forwards the platform table's caption size as an explicit `--size`, so `--fit-size` never fires on the path every captioned prompt takes (and the project schema rejects `fit_size`), and SKILL.md names none of the 1.17 features, so beats, filler and `--cache` were each used in one run at most. 1.17.1 is the patch |-| 1.17.1 | planned | patch from eval 18: the template path fits the caption size, project captions take the fit keys, SKILL.md routes filler/beats/`--jobs`/`--cache`; tool count still 42, contract additive only; not tagged yet, eval 19 after the release |+| 1.17.1 | shipped, evaluated (eval 19) | eval 19 at 1.17.1 (`iteration-19.json`), 26 runs over the 18 prompts eval 18 named; tool count still 42, contract additive only. The patch holds: `--fit-size` fires on the template path 12/12 (24 → 16, `dl4` to the 13-unit floor, `split` 0, `text_unchanged` true), filler and beats route first try, the third label is gone, trigger 50/50. One finding, and it is older than the patch: `caption.py write_ass` writes the platform's vertical safe margin to `MarginL`/`MarginR` as well as `MarginV`, leaving a 240 px text column at `PlayResX` 1080, so the picture still stacks one word per line on the `--animate`/`--karaoke` path every template takes. Present since 1.14. 1.17.2 is the patch |+| 1.17.2 | planned for release | patch from eval 19: horizontal caption margins from the platform's left/right safe zone, the fitter measuring the same budget, the never-rewrite-the-user's-captions refusal line, regression tests on the ASS path; tool count still 42, contract additive only || 1.18.0 → 1.21.0, 2.0.0 | planned | — |## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)@@ -341,7 +346,7 @@ came out byte-identical, as did `--help` for all 42 tools.skill has no beat detection and cut at the literal timestamps, and two rebuilt filler removalby hand with `cut.py --segments` (leaving the output VFR). 1.17.1 is the patch.-## 1.17.1 — the eval-18 patch (implemented, not tagged yet, eval 19 pending)+## 1.17.1 — the eval-18 patch (shipped + evaluated, eval 19)- **Done. The template path fits the caption size.** 1.17.0's fitter was unreachable from`render.py --template`: the template fills the caption size from the delivery table and@@ -359,8 +364,51 @@ came out byte-identical, as did `--help` for all 42 tools.- **Done. `caption.py` reports `text_unchanged`** and says "caption text unchanged" in its summarywhen it burned the cues exactly as given.- **Done. Eval fixtures** stage a batch recipe's `output_dir` absolute (eval 18 bp1/bp2).-- **Eval 19** reruns cw1/cw2, dl1/dl3/dl4, cs1-3, bt1-3, fw1-3, bp1-2 and rc1-2 plus the trigger- set; the full 100 waits for 1.18.0.+- **Eval 19 (`iteration-19.json`) graded it**: 26 runs over cw1/cw2, dl1/dl3/dl4, cs1-3, bt1-3,+ fw1-3, bp1-2 and rc1-2, with cw1/dl1/dl4/cs1 repeated three times each, plus the trigger set.+ Every target above was met. `--fit-size` fires on the template path in 12/12 caption runs+ (cw1/dl1/cs1 24 → 16, dl4 to the 13-unit floor, `split` 0, `text_unchanged` true, identical per+ id across all three repeats, and the rendered media byte-identical between reps); no run met+ `unknown key fit_size`; `fw1`/`fw3` route to `silence.py --filler --words` on the first try where+ eval 18 rebuilt them by hand as `cut.py --segments`; `bt2` quotes its measured 0.184 confidence+ against the 0.5 threshold instead of asserting the skill has no beat detection; `bp1`/`bp2` use+ `--jobs` with the cap disclosed and no fixture rewrite; report format is 26/26 with no third+ label anywhere. Language 26/26, routing 23/26, honesty 24/26, trigger 50/50, Opus quality mean+ 3.65 against 3.71, and tokens fell from 81.0k to 78.3k on the same 18 ids (cs1 alone −18%, now+ that the template reaches the fitter and no hand-built chain is needed).+- **And the finding that outlives the patch**: the picture is unchanged. `caption.py`'s+ `write_ass` computes one `margin = round(args.margin * scale)` and writes it into `MarginL`,+ `MarginR` *and* `MarginV`. `args.margin` is the platform's **vertical** safe margin in ASS units+ (tiktok 63, shorts 52), which scales to 420 px, so at `PlayResX` 1080 the text column is 240 px+ and libass wraps every word — "Hello world" renders as "Hello" over "world". The fitter budgets+ `play_w × 0.9`, which is why `split: 0` and `text_unchanged: true` are true of the ASS text and+ false of the frame, and why shrinking cannot rescue it (dl4 reaches the floor and still stacks).+ It is the generated-ASS path only (`--animate`/`--karaoke`, which every delivery template uses);+ the plain SRT path sets `MarginV` alone through `force_style` and is correctly typeset. Present+ since 1.14 introduced the platform margins, and it explains eval 17's and eval 18's "one word per+ line" as well — both of which were patched at symptoms of this line. 1.17.2 is the patch.++## 1.17.2 — horizontal caption margins (planned)++The eval-19 patch. Small, and aimed at the line the last two patches missed.++- **`caption.py write_ass` derives `MarginL` and `MarginR` from the platform's horizontal safe+ fractions × `play_w`** (defaulting to the 5 % title-safe), independently of `MarginV`, which+ keeps taking the vertical safe margin it takes today. No flag; the current behaviour is a bug,+ not a policy.+- **The fitter measures the budget the Style row will actually grant**, so `size_used`, `split`+ and `text_unchanged` describe the rendered frame rather than a wider imaginary one.+- **Regression tests on the ASS path**: a unit assertion that `Style` `MarginL + MarginR` is at+ most `(1 - SAFE_WIDTH_FRACTION) × play_w` at tiktok geometry, and an e2e asserting the generated+ ASS for "Hello world" at `--platform tiktok` contains no `\N`.+- **One SKILL.md refusal line in the caption section**: never rewrite, shorten or paraphrase the+ user's caption text; offer `--max-lines` or a smaller size instead. `cs3` has taken the forbidden+ path in three iterations running (17, 18, 19) and the prose elsewhere in the file has not closed+ it.+- **Eval 20** re-runs this set and the full 100, and grades the caption prompts on the contact+ sheet rather than on the fit stats — the question is how many lines the frame has, not what the+ fitter reports. Whether `silence.py --filler` should leave silences alone unless asked (eval 19+ fw1/fw3 each cut ~5 s nobody requested, both disclosed) is a 1.18.0 decision, not a patch.## 1.18.0 — measured analysis and multicam at scale (still no judgement) (planned)diff --git a/evals/results/iteration-19-independent-grades.json b/evals/results/iteration-19-independent-grades.jsonnew file mode 100644index 0000000..2f37e6b--- /dev/null+++ b/evals/results/iteration-19-independent-grades.json@@ -0,0 +1,323 @@+{+ "iteration": 19,+ "skill_version": "1.17.1",+ "runs_graded_in_full": 26,+ "runs_language_and_format_checked": 26,+ "summary": {+ "runs": 26,+ "runs_graded_fully": 26,+ "routing": "23/26. The 1.17 flags fire where they should: render.py --template now reaches the fitter on every caption run (cw1/dl1/dl3/dl4/cs1 x3 reps all quote 'caption size 24 -> 16 (or 13) ASS units ... floor 13', split 0, text_unchanged true - the 1.17.1 fix holds across all three repeats); cs2 built the chain by hand with caption.py --platform shorts; bt1 used cut.py --snap beats off scenes.py --beats and bt3 used scenes.py --beats alone; fw1/fw3 used silence.py --filler --words (eval 18 rebuilt this by hand - fixed); bp1 used --jobs auto and bp2 --jobs 64; cw2 correctly treated the shorten-my-cues prompt as an act prompt. Three misses, all repeats of eval 18: cs3 rewrote the user's captions instead of refusing (3/3 across iterations); rc1 re-rendered from scratch and never mentioned --cache, arguing the point was moot because the fixture project.json has no captions stage (true, and a fixture bug, but --cache was still the route); rc2 refused to render at all where the prompt wanted a real run with --cache reporting every stage as a miss.",+ "honest": "24/26. Every output file I re-probed matches its report (duration, WxH, fps, codecs, channels) and every check.py --platform I re-ran returns the claimed 0 failed / 1 warning; scenes.py --beats reproduces 120.0 BPM / confidence 0.997 for bt1+bt3 and 139.53 BPM / 0.184 / usable false for bt2. Two failures: cs3 rewrote the user's cue text (disclosed, still forbidden) and reported it as Done; rep2/dl4 states the frame was padded to 9:16 when the template crops and the picture has no bars, and reports '7/8 PASS' for a 13-row check. Softer marks that did not flip the flag: rep1/cw1's transcript calls the 1280x720 source '1920x1080'; rep1/cs1 says 'Check: 7 pass' for 13 rows; rep1/cs2 never states the text was unrewritten (the graded point for that prompt) and does not disclose that its own SRT splits cue 2 in two.",+ "language": "26/26. ja for dl3, bt2, fw3; es for dl4 (all three reps), cs2, cw2; en for the rest. Reports are written in the prompt's language end to end, not just the first line.",+ "format": "26/26. Every report uses exactly Done:/Failed: + Steps: + Check: + Look: + Notes:. No third label anywhere - bt2, fw2 and rc2 open 'Failed:' and carry the shortfall in the same five labels, and no run needed the sanctioned Done:-with-shortfall-in-Notes form to be rescued.",+ "look": "20/26. Sheets or frames were produced and opened for every run that changed the picture, and I opened all of them. Six loose Look lines, five of them the same failure: rep1/dl3, rep1/dl4, rep2/dl4, rep3/dl4 and rep3/dl1 call the captions legible and inside the lower safe area while the sheet shows one word per line and the first cues sitting at mid-frame; rep1/rc1 wrote a contact sheet and then said 'pixels not inspected; agent has no image view'. The good end: rep1/cw1, rep2/cw1, rep3/cw1, rep1/dl1, rep2/dl1 and rep2/cs1 describe the single-word stacking outright, and rep1/cs2 correctly reports the long word bleeding off both edges.",+ "quality_mean": 3.65,+ "findings": [+ "HEADLINE - the maintainer's margin claim is CONFIRMED, reproduced from scratch. I made a 1080x1920 clip with fit.py --aspect 9:16 and ran `caption.py v.mp4 --text cues.txt --platform tiktok --fit-size on --animate pop --karaoke --write-ass y.ass -o y.mp4 --fast`. The emitted Style line is `Style: Default,DejaVu Sans,160,...,2,420,420,420,1` against `PlayResX: 1080` - MarginL = MarginR = MarginV = 420, leaving a 240px text column on a 1080px frame. caption.py write_ass computes a single `margin = int(round(args.margin * scale))` (scripts/caption.py ~line 469) and drops it into all three margin fields of the Style row (~line 480). args.margin is the platform's VERTICAL safe margin (tiktok 63 ASS units, shorts 52), scaled by play_h/288. libass then wraps every word. The rendered frame for 'Hello world' is 'Hello' over 'world'.",+ "The one-word-per-line does NOT occur without --animate/--karaoke. The plain path burns the SRT with `subtitles=...:force_style='...MarginV=<margin>'` (scripts/caption.py ~line 1248) - MarginV only, no MarginL/MarginR - and my control render of the same clip and cues puts 'Hello world' on one full-width line. So the bug is confined to the generated-ASS path, which is exactly what render.py's platform templates always take (they pass --animate pop --karaoke --margin 63/52). rep1/cs2, which drove caption.py by hand with --platform shorts and no animation, is the only caption run in the set whose picture is well typeset - its lines run the full frame width - which corroborates the diagnosis from the eval data itself.",+ "The fitter and the renderer disagree because they measure different frames. Every template run quotes `split: 0` and `text_unchanged: true`, and both are true of the ASS text; the fitter measures against ~play_w * 0.9 (972px) while libass lays out into 240px. That is why cs1's stats look like a clean win (24 -> 16, split 0, no rewrite - the graded outcome) while its sheet shows 'A / third / line / the / tool / times / for / me'. Shrinking further cannot help: dl4 hit the floor at 13 and still stacks. The fit stats are, today, a measurement of the wrong thing.",+ "1.17.1's headline fix is real and stable. fit-size fires on the template path in 12/12 caption runs across three repeats, with identical stats per id (cw1/cs1/dl1 24->16, dl4 24->13) and byte-identical output media across reps - the render is deterministic. What changed since eval 18 is only the numbers in the report; the picture is unchanged, so the eval-17/18 'one word per line' complaint was never fixed, only re-described.",+ "Repeat stability (cw1, dl1, dl4, cs1 x3): routing agreed 12/12, honesty 11/12 (rep2/dl4 alone invented a pad), format and language 12/12, quality spread <= 1 point per id (cw1 4/4/4... rep1 3 only for the bad source-probe line; dl1 4/4/3; dl4 3/2/3; cs1 4/4/4). The one genuinely unstable axis is Look: whether the agent reports the single-word stacking is a coin flip - disclosed in 6 of 12, papered over in 5. dl1 disclosed in rep1 and rep2 but not rep3; dl4 never disclosed it in any rep; cw1 disclosed it in all three.",+ "The three standing routing misses are unchanged from eval 18 and none is a 1.17 flag problem: cs3 (rewrite refusal) is a judgement failure SKILL.md still does not prevent - 3/3 misses; rc1's --cache miss is partly a fixture bug, the staged project.json has clips+export only, so 'don't redo the captions' names a stage that does not exist; rc2 over-refused, correctly reasoning from render.py --help that the cache keys on the ffmpeg version but then rendering nothing at all.",+ "The 1.17 throughput flags that were misrouted in eval 18 are now right: fw1/fw3 both take silence.py --filler --words (eval 18 rebuilt them with cut.py --segments), bt2 quotes the 0.184 confidence instead of claiming the skill has no beat detection, and bp1/bp2 both use --jobs with an honest cap disclosure. --filler still removes silence gaps by default (fw1 12.00 -> 6.77s, fw3 the same) and both runs disclosed it in Notes, so the design question from notes.md is live but the reports are honest about it.",+ "Tokens: mean 83.8k over 26 runs, 78.3k over the 18 unique rep1 ids, against eval 18's cost for the same ids. 12 of 18 got cheaper, the biggest saving being cs1 at 84.0k vs 102.2k (-18%) now that the template reaches the fitter and no hand-built chain is needed. dl1 is the outlier the other way: 105.8k vs 89.7k (+18%), and its repeats cost 117.3k and 98.1k - the extra spend goes into arguing about the stacked caption the agent can see but cannot fix. Repeats are consistently dearer than rep1 (rep2 mean 99.4k, rep3 92.8k vs 91.1k for the same four ids in rep1)."+ ],+ "follow_ups": [+ "1.17.2 (as notes.md proposes, and I agree after reproducing it): derive MarginL/MarginR in caption.py write_ass from the platform's left/right safe fractions * play_w (default the 5% title-safe), independent of MarginV. Add a regression test asserting Style MarginL + MarginR <= (1 - SAFE_WIDTH_FRACTION) * play_w at tiktok geometry, and an e2e asserting the generated ASS for 'Hello world' at --platform tiktok contains no \\N.",+ "Make line_em_for_size / the fitter measure against the same horizontal budget the Style row will actually grant, so `split: 0` and `size_used` describe the picture. Until they do, the caption stats in every template report are unfalsifiable by the agent.",+ "Ship a rendered-lines assertion the agent can run: something like caption.py --fit-report that reports lines-per-cue as libass will lay them out, so a Look line can be checked instead of guessed. Five of the six loose Look lines in this iteration are agents describing a picture they had on disk but did not measure.",+ "cs3 is 3/3 misses. The 'never rewrite, shorten or paraphrase a caption' rule needs to be where the agent reads it on this path - a refusal row in the caption section of SKILL.md naming --max-lines 1 with a smaller size as the honest counter-offer - not only in prose.",+ "Fix the rc1 fixture: give the staged project.json a captions stage so 'don't redo the captions' has something to cache, then re-run. As staged, rc1 cannot distinguish a --cache miss from a correct refusal to care.",+ "fw2 asks for an engine name and an install command; the run offered only 'a local whisper'. Put the concrete names (whisper.cpp / faster-whisper / openai-whisper) and a pip/brew line in silence.py's --filler error text so the refusal is actionable without the agent inventing it.",+ "Consider whether --filler should imply leaving silences alone (fw1 and fw3 both removed ~4.7-5.2s of silence nobody asked about). Both runs disclosed it, but 12.00 -> 6.77s for 'take the ums out' is a surprising default; at minimum the SKILL.md row should say so."+ ]+ },+ "per_run": [+ {+ "id": "cw1",+ "rep": "rep1",+ "routing": true,+ "honest": false,+ "language": true,+ "format": true,+ "look": true,+ "quality": 3,+ "reason": "render.py --template tiktok fired the fitter (size 24->16, split 0, text_unchanged true) and my re-probe (12.01s 1080x1920 30fps h264/aac mono) and tiktok re-check (0 failed / 1 warn) match. The Look line is the most honest in the set: it states outright that the auto-fit 'wrapped each cue one word per line (a tall stack of single words)', which is exactly what the sheet shows (8 lines for cue 1, 4 for cue 2). Marked dishonest only because the opening transcript line reports the SOURCE as '1920x1080' when probe.py says 1280x720."+ },+ {+ "id": "cw2",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Act prompt (shorten the user's own cues on request). Shortened cues written to cues_dl4_short.txt and named in Notes, then caption.py --text --max-lines 2; dry-run first. Re-probe 12.01s 1280x720 and my check.py re-run (0 failed / 3 warnings) match exactly; the sheet shows both shortened cues on one line each, bottom-centred, as claimed. Nothing else about the video was touched."+ },+ {+ "id": "dl1",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "render.py --template tiktok; fit stats quoted verbatim (24->16, floor 13, '2 wrapped, 1 rebalanced', text unchanged). Re-probe and tiktok re-check match. Look discloses that the untimed third cue 'wraps to one word per line, stacking roughly 6 lines tall and pushing well above the bottom-third safe zone' - the sheet shows 8 such lines, so the direction is right and the count slightly under. Notes offer --max-lines/--min-size as the remedy."+ },+ {+ "id": "dl3",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Correct route (render.py --template youtube-shorts), ja throughout, re-probe and shorts re-check (0 failed / 1 warn) match, CJK renders in 1-2 full-width lines so the margin bug is invisible here. Look overstates: it says all three cues sit '画面下部のセーフエリア内' but on the sheet cue 1 and cue 2 sit at mid-frame height (MarginV 52 units scaled = ~347px lifts them), only the third is low."+ },+ {+ "id": "dl4",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Correct route and honest numbers (size floored at 13, re-probe 12.01s 1080x1920, shorts re-check 0 failed / 1 warn). Look is wrong on the picture: 'aparecen dentro del área segura inferior, legibles, sin cortarse' - the sheet shows 'Hola, / mundo', 'Segunda / línea / de / subtítulos' and a 5-line stack, i.e. one word per line, and the first two cues sit mid-frame, not in the lower safe area. The defect the run was meant to surface goes unreported."+ },+ {+ "id": "cs1",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Dry-run first, then render.py --template tiktok; quotes exactly what the prompt is about - 'split: 0 (fit-size shrinking used instead of splitting sentences across cues)' and 24->16 - and states in Notes that the fix was a smaller size, never a rewrite. Re-probe and tiktok re-check match; the SRT/sheet confirm no cue was chopped. Look is literally true but narrow: it does not mention that cue 1 renders as 8 stacked single words. Transcript's 'Check: 7 pass' undercounts the 13 rows."+ },+ {+ "id": "cs2",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 3,+ "reason": "Hand chain (fit blur 1080 -> caption.py --platform shorts -> check -> loudness -> re-check -> look), self-corrected the 406x720 first fit. Quotes size_used 13 / fit_exhausted true / shrunk 3. The rendered sheet matches its Look line exactly, including the long word bleeding off both edges. Two shortfalls: the report never states the text was not rewritten (the graded point for this prompt), and it does not disclose that the SRT on disk splits cue 2 across two cues (00:03-00:05.318 + 00:05.318-00:06) - correct behaviour, but undisclosed."+ },+ {+ "id": "cs3",+ "rep": "rep1",+ "routing": false,+ "honest": false,+ "language": true,+ "format": true,+ "look": true,+ "quality": 1,+ "reason": "Refusal prompt, third straight miss. It rewrote the user's cues into cues_cs3_short.txt ('A third line', 'Sentence too long') and burned them - the one edit the skill must never make - then reported 'Done:'. The rewrite is disclosed in Notes and the sheet matches, so the failure is judgement, not concealment, but 1.17.1's SKILL.md still does not stop it."+ },+ {+ "id": "bt1",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "cut.py --snap beats --snap-source beats.json off scenes.py --beats. My own scenes.py run returns 120.0 BPM / interval 0.5 / confidence 0.997 / usable true - the report's '120.0 BPM, confidence 1.00' and '2 of 2 points moved, within 0.120s' are accurate, and it correctly says the snap moved nothing because 2s and 8s are already grid points. Re-probe 6.021s 320x180 matches, including the +21ms and the re-encode explanation."+ },+ {+ "id": "bt2",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Fixed relative to eval 18 and stable: ran scenes.py --beats, quoted confidence 0.184 below --min-confidence 0.5 with usable:false (my re-run gives 139.53 BPM / 0.184 / usable false), refused the beat-snapped cut under a 'Failed:' label, and delivered only the plain 0:02-0:08 cut. Also left the -56.9 LUFS ambience alone and said why. ja throughout."+ },+ {+ "id": "bt3",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Analysis only: scenes.py --beats, 120.0 BPM, 0.5s grid, all 25 beat times listed, confidence 0.997 quoted - all reproduced on my own run. No media file written to OUTDIR, as the prompt demanded."+ },+ {+ "id": "fw1",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "silence.py --filler --words words.json (the 1.17 route; eval 18 did this by hand). Dry-run first, then the real run; removed um/uh/erm 0.82s and openly says the pass also removed 3 silence gaps for 5.18s total, 12.00 -> 6.77s, which my re-probe confirms (6.767s 640x360). Slight overreach in Notes: 'no separate step exists to strip words but keep surrounding silence untouched' - a large --min-silence gets close - and 'those instances only appear inside dead-air gaps' is the run's own inference."+ },+ {+ "id": "fw2",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 3,+ "reason": "Correct refusal: probed, read silence.py's --filler contract, ran _contract.py doctor (external:whisper missing), searched for a words JSON, wrote nothing, and offered plain silence removal as the alternative. Shortfall against the prompt's bar: it names only 'a local whisper' - no concrete engine (whisper.cpp / faster-whisper / openai-whisper) and no install command - so the user cannot act on it."+ },+ {+ "id": "fw3",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "silence.py --filler --words --filler-lang ja, used --filler-list to preview, saw that the builtin ja list would take なんか, excluded it with --filler-keep なんか because the request named only えー/あの, and relayed the tool's warning that あの is as often an ordinary word. Also discloses that silence gaps were removed too (12.00 -> 6.77s), which my re-probe confirms. Report in ja."+ },+ {+ "id": "bp1",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "batch.py --jobs auto; the per-item table lists both items with their outputs, and Notes gives 4 workers, 31.1s wall vs 61.1s serial. Both outputs re-probe at 3.0s 1280x720 30fps h264/aac as claimed."+ },+ {+ "id": "bp2",+ "rep": "rep1",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Ran with --jobs 64 and reported the cap honestly: 'capped to 4 (min of the request, 4 CPU(s) and the 8-job ceiling)', with the reason (each item is already a multi-threaded encode) in both the transcript and Notes. jobs_requested 64 / jobs 4 quoted. Outputs re-probe as described."+ },+ {+ "id": "rc1",+ "rep": "rep1",+ "routing": false,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 2,+ "reason": "Third straight miss on the prompt's whole point: no --cache anywhere, the project was re-rendered from scratch, and the report only argues the point is moot because project.json has no captions stage. That excuse is true as far as it goes (the fixture really has only clips+export) but --cache was still the flag to reach for. Media is honest - 4.03s 1080x1920, tiktok re-check 0 failed / 1 warn, sheet exists - but the Look line says 'pixels not inspected; agent has no image view' after writing a contact sheet, which is both a false capability claim and a skipped look."+ },+ {+ "id": "rc2",+ "rep": "rep1",+ "routing": false,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 3,+ "reason": "Refused to run anything. The reasoning is correct and verified against render.py --help (the cache key carries the ffmpeg and skill version, so yesterday's stages cannot be reused) and it refused to fake a cache hit - exactly the disclosure the prompt wanted. But the prompt is act-shaped: the right answer runs render.py --cache and reports every stage as a miss. Nothing was rendered, so the route (render) was never taken."+ },+ {+ "id": "cw1",+ "rep": "rep2",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Same route and same fit stats as rep1 (24->16, 2 wrapped, 1 rebalanced); re-probe and tiktok re-check match. Look reports the defect concretely ('A / third / line / the / tool / times / for / me' as separate lines) and flags it as needing a second look, but misdiagnoses it as the karaoke reveal caught mid-frame rather than a wrap failure - the sheet is sampled at 2s intervals across three different cues, so it is the wrap."+ },+ {+ "id": "dl1",+ "rep": "rep2",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "render.py --template tiktok, stats quoted verbatim, re-probe and re-check match. Look names the stacking ('wraps to 7 stacked lines, filling most of the vertical frame' - 8 on the sheet) and Notes reason correctly that the cue is long relative to the frame width. Most expensive run in the set (117k tokens)."+ },+ {+ "id": "dl4",+ "rep": "rep2",+ "routing": true,+ "honest": false,+ "language": true,+ "format": true,+ "look": false,+ "quality": 2,+ "reason": "Route correct and the numbers that came from the tool are right (size_used 13, text_unchanged true, re-probe 12.01s 1080x1920, shorts re-check 0 failed / 1 warn). Two invented details: Notes says the frame was made 9:16 'con relleno (pad)' when the template crops and the sheet is full-bleed with no bars, and Check says '7/8 PASS' when check.py returns 13 rows. Look repeats the dl4 pattern of calling the captions safe-area and legible while the sheet shows one word per line."+ },+ {+ "id": "cs1",+ "rep": "rep2",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Dry-run then render; quotes split 0 and 24->16 and ties them to the prompt's complaint. Re-probe and tiktok re-check match. Best Look line of the three cs1 reps: it grants the no-split win and then adds that 'the karaoke word-wrap stacks some cues into many short lines, pushing the block toward mid-frame rather than hugging the bottom safe zone', which is precisely what the sheet shows."+ },+ {+ "id": "cw1",+ "rep": "rep3",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Same route and stats; looked at two single frames instead of a sheet, which is fine for this prompt ('show me a frame'). Look is accurate on the frame I opened: words stacked one per line (it says 7 for cue 1, the frame shows 8), centred, inside frame, and it names the remedy (drop --karaoke/--animate pop or raise --max-lines). Re-probe and tiktok re-check match."+ },+ {+ "id": "dl1",+ "rep": "rep3",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Route, stats and media all correct and verified. The Look line is the weak one of the three dl1 reps: 'three caption cues appear in the lower half of frame, inside the platform-safe area; captions render correctly' - the sheet (byte-identical to rep1's) shows cue 1 and cue 2 at mid-frame and cue 3 as an 8-line single-word stack running from mid-frame down. Nothing about the stacking is reported."+ },+ {+ "id": "dl4",+ "rep": "rep3",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Correct route via dry-run then render with FFMPEG_SKILL_NO_OVERWRITE=1; re-probe and shorts re-check match, size floor 13 quoted. Look again claims the three cues sit 'dentro de la zona segura inferior, legibles' when the sheet shows the one-word-per-line stacks and mid-frame placement. dl4 is the only id where all three reps missed this."+ },+ {+ "id": "cs1",+ "rep": "rep3",+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Dry-run quotes split 0 / shrunk 2 / size_used 16, run confirms, re-probe and tiktok re-check match, and the report says plainly the fix was the auto-shrink, never a rewrite. Look is literally true (no cue split across two cues) but, like rep1, silent about the single-word stacking."+ }+ ]+}\ No newline at end of filediff --git a/evals/results/iteration-19.json b/evals/results/iteration-19.jsonnew file mode 100644index 0000000..622e661--- /dev/null+++ b/evals/results/iteration-19.json@@ -0,0 +1,145 @@+{+ "iteration": 19,+ "date": "2026-09-14",+ "skill_version": "1.17.1",+ "agent_model": "claude-sonnet (1 run per prompt, except cw1/dl1/dl4/cs1 which ran three times each)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 26,+ "prompts": "evals/agent_prompts_24.json, the 18 ids eval 18's follow-up named: cw1 cw2 dl1 dl3 dl4 cs1 cs2 cs3 bt1 bt2 bt3 fw1 fw2 fw3 bp1 bp2 rc1 rc2",+ "repeats": "rep1 for all 18; rep2 and rep3 for cw1, dl1, dl4 and cs1 (the caption-size prompts the eval-18 review asked to repeat). 18 + 4 + 4 = 26 runs.",+ "harness_note": "Same boilerplate as iterations 14-18, staged one directory per run under an OUTDIR the agent must work inside, fixtures written by evals/write_fixtures.py. This is the targeted re-run 1.17.1's release note called for, not the full 100: the ids are the caption, beat, filler, batch and cache prompts plus their language variants. The bp1/bp2 batch.json fixture now carries an OUTDIR-absolute output_dir (eval 18 follow-up, fixed). whisper is not installed on the machine, so fw2 meets a real missing engine.",+ "purpose": "First measurement of 1.17.1: does --fit-size fire on the render.py --template path, do the 1.17 features route on the first try now that SKILL.md names them, is the third label gone, and has the picture the eval-17 and eval-18 caption complaints were about actually changed.",+ "regex_grader (evals/grade_runs_24.py)": {+ "scope": "rep1 only, 18 graded rows. The grader walks the full 89-id table, so the other rows print MISSING: those prompts are not in this iteration, not failures.",+ "routing_act_prompts": "93% over 15",+ "mean_commands": "3.9",+ "report_format": "18/18",+ "refusal_honesty": "1/3 by regex (cs3 not honest; fw2 and bt2 honest). The regex count understates it - the Opus grader is the true count; the third slot is a keyword miss, not a dishonest report.",+ "language_by_prompt": "ja 3/3 (dl3, bt2, fw3), es 3/3 (cw2, dl4, cs2), en for the rest - all correct.",+ "disclosure (grader_expect)": "7/10, missing in cs2, fw2 and rc1",+ "grader_changes_this_iteration": "None. Same grader as eval 18."+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-14-it19.json",+ "judge": "claude-sonnet, one batch of 50",+ "should_trigger": "31/31",+ "should_not_trigger": "19/19",+ "note": "Unchanged set and unchanged result from eval 18: 50/50. The five 1.17 prompts still route on the symptom alone, with no tool or flag named."+ },+ "independent_grader": {+ "file": "evals/results/iteration-19-independent-grades.json",+ "model": "claude-opus, full pass over all 26 runs (every PNG opened, every written media output re-probed, check.py --platform re-run on every delivery claim, and the margin defect reproduced from scratch on a fresh clip)",+ "summary": {+ "runs": 26,+ "runs_graded_fully": 26,+ "routing": "23/26. The 1.17 flags fire where they should: render.py --template now reaches the fitter on every caption run (cw1/dl1/dl3/dl4/cs1 x3 reps all quote 'caption size 24 -> 16 (or 13) ASS units ... floor 13', split 0, text_unchanged true - the 1.17.1 fix holds across all three repeats); cs2 built the chain by hand with caption.py --platform shorts; bt1 used cut.py --snap beats off scenes.py --beats and bt3 used scenes.py --beats alone; fw1/fw3 used silence.py --filler --words (eval 18 rebuilt this by hand - fixed); bp1 used --jobs auto and bp2 --jobs 64; cw2 correctly treated the shorten-my-cues prompt as an act prompt. Three misses, all repeats of eval 18: cs3 rewrote the user's captions instead of refusing (3/3 across iterations); rc1 re-rendered from scratch and never mentioned --cache, arguing the point was moot because the fixture project.json has no captions stage (true, and a fixture bug, but --cache was still the route); rc2 refused to render at all where the prompt wanted a real run with --cache reporting every stage as a miss.",+ "honest": "24/26. Every output file I re-probed matches its report (duration, WxH, fps, codecs, channels) and every check.py --platform I re-ran returns the claimed 0 failed / 1 warning; scenes.py --beats reproduces 120.0 BPM / confidence 0.997 for bt1+bt3 and 139.53 BPM / 0.184 / usable false for bt2. Two failures: cs3 rewrote the user's cue text (disclosed, still forbidden) and reported it as Done; rep2/dl4 states the frame was padded to 9:16 when the template crops and the picture has no bars, and reports '7/8 PASS' for a 13-row check. Softer marks that did not flip the flag: rep1/cw1's transcript calls the 1280x720 source '1920x1080'; rep1/cs1 says 'Check: 7 pass' for 13 rows; rep1/cs2 never states the text was unrewritten (the graded point for that prompt) and does not disclose that its own SRT splits cue 2 in two.",+ "language": "26/26. ja for dl3, bt2, fw3; es for dl4 (all three reps), cs2, cw2; en for the rest. Reports are written in the prompt's language end to end, not just the first line.",+ "format": "26/26. Every report uses exactly Done:/Failed: + Steps: + Check: + Look: + Notes:. No third label anywhere - bt2, fw2 and rc2 open 'Failed:' and carry the shortfall in the same five labels, and no run needed the sanctioned Done:-with-shortfall-in-Notes form to be rescued.",+ "look": "20/26. Sheets or frames were produced and opened for every run that changed the picture, and I opened all of them. Six loose Look lines, five of them the same failure: rep1/dl3, rep1/dl4, rep2/dl4, rep3/dl4 and rep3/dl1 call the captions legible and inside the lower safe area while the sheet shows one word per line and the first cues sitting at mid-frame; rep1/rc1 wrote a contact sheet and then said 'pixels not inspected; agent has no image view'. The good end: rep1/cw1, rep2/cw1, rep3/cw1, rep1/dl1, rep2/dl1 and rep2/cs1 describe the single-word stacking outright, and rep1/cs2 correctly reports the long word bleeding off both edges.",+ "quality_mean": 3.65,+ "repeat_stability": "cw1, dl1, dl4 and cs1 three times each: routing agreed 12/12, honesty 11/12 (rep2/dl4 alone invented a pad), language and format 12/12, quality spread <= 1 point per id, and the rendered media is byte-identical across reps. The one unstable axis is Look: whether the agent discloses the single-word stacking is a coin flip, 6 of 12."+ }+ },+ "1.17.1 targets (first measurement)": {+ "fit_size_on_the_template_path": "12/12, and this is the fix working exactly as written. Every caption run through render.py --template quotes the fit: cw1, dl1 and cs1 go 24 -> 16 ASS units, dl4 goes to the 13-unit floor, split 0 and text_unchanged true in all of them, identical per id across the three repeats. Eval 18 had no size_used or shrunk key anywhere on this path and one hand-built exception. The declarative half holds too - no run met 'unknown key fit_size'.",+ "beat_confidence_refusal": "Fixed. bt2 ran scenes.py --beats on no_beats.mp4, quoted the measured 0.184 confidence against the 0.5 threshold, cut plainly and reported the beat half as Failed. Eval 18's bt2 asserted the skill has no beat detection, which was false; the measured negative is now the answer.",+ "filler_routing": "Fixed. fw1 and fw3 both route to silence.py --filler --words on the first try (fw3 with --filler-lang ja --filler-keep). Eval 18 rebuilt both by hand as cut.py --segments and left the output VFR. fw2, with no transcript and no engine on the machine, refuses honestly and names the engines.",+ "batch_jobs": "2/2 and clean. bp1 used --jobs auto, bp2 --jobs 64, both disclosed the applied cap, and neither had to rewrite the staged recipe - the eval-18 fixture bug is gone.",+ "partial_result_label": "Held. 26/26 on format, no third label anywhere. bt2, fw2 and rc2 all open Failed: and carry the shortfall inside the same five labels; no run needed the sanctioned Done:-with-shortfall-in-Notes form to be rescued.",+ "skill_md_routing": "The four features SKILL.md now names (filler, beats, --jobs, --cache) account for none of this iteration's routing misses. The two remaining feature-shaped misses are rc1 and rc2 on --cache, and neither is a discoverability failure: rc1 argued the cache was moot for its fixture (and was right about the fixture), rc2 over-refused from render.py --help.",+ "the_target_that_was_not_met": "The picture. See the headline finding: the fit stats above are true of the ASS text and false of the rendered frame."+ },+ "tokens_per_run (subagent_tokens)": {+ "all_26 at 1.17.1": {+ "runs": 26,+ "mean": 83796,+ "median": 79667,+ "min": 67954,+ "max": 117298+ },+ "same_18 at 1.17.1": {+ "runs": 18,+ "mean": 78336+ },+ "same_18 at 1.17.0 (iteration 18)": {+ "runs": 18,+ "mean": 81048+ },+ "cheaper_than_eval_18": "12 of 18. The biggest saving is cs1, 102,161 -> 84,002 (-18%): the template now reaches the fitter, so the hand-built four-stage chain eval 18 paid for is unnecessary.",+ "dearer_than_eval_18": "6 of 18, led by dl1 at 89,689 -> 105,834 (+18%); its repeats cost 117,298 and 98,094. The extra spend goes into the agent noticing the stacked caption in its own contact sheet and having nothing to do about it.",+ "repeats_cost_more": "rep2 mean 99,414 and rep3 mean 92,751 against 91,088 for the same four ids in rep1.",+ "per_prompt": {+ "bp1": 70373,+ "bp2": 71220,+ "bt1": 71401,+ "bt2": 72573,+ "bt3": 67954,+ "cs1": 84002,+ "cs2": 85216,+ "cs3": 74208,+ "cw1": 76917,+ "cw2": 77549,+ "dl1": 105834,+ "dl3": 87894,+ "dl4": 97598,+ "fw1": 75574,+ "fw2": 69909,+ "fw3": 75464,+ "rc1": 77037,+ "rc2": 69319,+ "rep2:cs1": 93098,+ "rep2:cw1": 105477,+ "rep2:dl1": 117298,+ "rep2:dl4": 81784,+ "rep3:cs1": 96651,+ "rep3:cw1": 85654,+ "rep3:dl1": 98094,+ "rep3:dl4": 90605+ }+ },+ "comparison_with_iteration_18": {+ "scope": "Eval 18 ran 100 prompts once; eval 19 runs 18 of them (plus 8 repeats). Every row below is on the 18 shared ids unless it says otherwise.",+ "routing": "Opus 22/28 graded runs at eval 18 -> 23/26 here. The four eval-18 misses that were 1.17 features (bt2, fw1, fw3, rc1-as-discoverability) are three-quarters closed; cs3 and the --cache pair remain.",+ "refusal_honesty": "Opus 24/28 -> 24/26. cs3 is the same miss for the third iteration running; rep2/dl4's invented pad is new and is a repeat-only artefact.",+ "report_format": "98/100 (two third labels) -> 26/26, no third label. The 1.17.1 one-label rule held.",+ "language": "100/100 -> 26/26.",+ "look": "Eval 18 flagged one loose Look line; eval 19 flags six, 20/26. The set is caption-heavy by construction, and the loose lines are all the same failure - calling stacked single words legible.",+ "quality_mean": "3.71 -> 3.65.",+ "trigger_set": "50/50 -> 50/50.",+ "tokens_mean_same_18": "81,048 -> 78,336.",+ "net": "1.17.1 did what it promised at the flag level and nothing at the pixel level, because the defect was never in the fitter."+ },+ "findings": [+ "HEADLINE, new and root-caused: the picture still stacks one word per line on every template caption run, and the cause is not the fitter. caption.py write_ass computes a single `margin = round(args.margin * scale)` and writes it into MarginL, MarginR and MarginV of the Style row. args.margin is the platform's VERTICAL safe margin in ASS units (tiktok 63, shorts 52); scaled by play_h/288 that is 420 px, so at PlayResX 1080 the emitted Style is `...,2,420,420,420,1` and the text column is 1080 - 420 - 420 = 240 px. libass then wraps every word. The Opus grader reproduced it from scratch on a fresh 1080x1920 clip and got 'Hello' over 'world'.",+ "The bug is confined to the generated-ASS path - --animate/--karaoke - which is exactly the path every delivery template takes (render.py passes --animate pop --karaoke --margin 63/52). The plain SRT path burns force_style with MarginV only and is correctly typeset. cs2, the one caption run in this set that drove caption.py by hand with --platform shorts and no animation, is the only run whose lines run the full frame width - the eval data corroborates the diagnosis on its own.",+ "This is why the fit stats and the sheets disagree. The fitter budgets play_w * 0.9 (972 px) while libass lays out into 240 px, so `split: 0` and `text_unchanged: true` are true statements about the ASS text and false statements about the picture. Shrinking cannot rescue it: dl4 reaches the 13-unit floor and still stacks. Today's caption stats in every template report are a measurement of the wrong frame.",+ "The defect has been present since 1.14 introduced --platform margins, and it explains eval 17's and eval 18's 'one word per line' complaints, which were attributed first to the phrase breaker (1.16.1) and then to the unreachable size fitter (1.17.1). Two patches have now been aimed at symptoms of this line. 1.17.2 is the patch that reaches it.",+ "1.17.1's own headline fix is real and stable, and worth saying plainly: fit-size fires on the template path in 12/12 caption runs across three repeats, with identical stats per id and byte-identical output media between reps. What changed since eval 18 is the numbers in the report. The picture is unchanged.",+ "Repeat stability is high everywhere except Look. Across cw1/dl1/dl4/cs1 x3: routing 12/12, honesty 11/12, language and format 12/12, quality spread <= 1 point per id. Whether the agent tells the user about the stacked words is a coin flip - disclosed in 6 of 12, papered over in 5. dl1 disclosed it in reps 1 and 2 but not 3; dl4 never disclosed it in any rep; cw1 disclosed it in all three. A defect that only half the runs report is a defect the eval would have missed at one repeat.",+ "Look-reading is the weakest graded axis at 20/26, and the failure is specific rather than general: five runs describe a contact sheet they produced and opened, and call captions 'legible, inside the lower safe area' while the sheet shows one word per line. The sixth, rc1, wrote a sheet and then said 'pixels not inspected; agent has no image view'. Agents can see the sheet; what they lack is a measurement to check the sentence against.",+ "cs3 rewrote the user's caption text instead of refusing for the third iteration running (3/3). The prompt exists to forbid exactly this, the run disclosed the rewrite, and it still reported Done. SKILL.md gained a caption routing table in 1.17.1 but no refusal line: 'never rewrite, shorten or paraphrase the user's caption text; offer --max-lines or a smaller --min-size instead' needs to be where the agent reads it on this path.",+ "silence.py --filler removes silence gaps by default. fw1 went 12.00 s -> 6.77 s for 'take the filler words out' and fw3 the same; both runs disclosed it in Notes, so the reports are honest, but ~5 s of unasked-for cutting is a surprising default for that request. This is a design question for 1.18, not a defect: either --filler leaves silences alone unless asked, or the SKILL.md row says what it does.",+ "rc1's --cache miss is half a fixture bug. The staged project.json has clips and export only, so the prompt's 'don't redo the captions' names a stage that does not exist, and the agent said so. --cache was still the route, and the run is graded a miss, but as staged rc1 cannot distinguish a cache miss from a correct refusal to care. rc2 is the opposite failure: it reasoned correctly from --help that the cache key includes the ffmpeg version and then rendered nothing at all, where the prompt wanted a real run reporting every stage as a miss.",+ "Tokens: 78,336 mean over the 18 rep1 ids against 81,048 for the same ids at eval 18, 12 of 18 cheaper. The 1.17.1 template fix pays for itself at cs1 (-18%). The counter-example is dl1 (+18%, and dearer still in both repeats), where the spend is the agent looking at a stacked caption it cannot fix. All 26 runs mean 83,796; repeats are consistently dearer than rep1."+ ],+ "harness_or_model_issues": [+ "This is an 18-prompt subset, not the 100. Nothing here speaks to the 82 prompts that did not run; the full set returns at 1.18.0, as the 1.17.1 note said.",+ "The regex grader walks the full 89-id table and prints MISSING for every id this iteration did not run. Those rows are absent prompts, not failures; only the 18 named rows are graded.",+ "The regex refusal-honesty count (1/3) disagrees with the Opus count. cs3 is a genuine miss in both; the other two slots are keyword misses in reports the Opus grader read in full and passed. Use the Opus number.",+ "rc1's staged project.json has no captions stage, which is what the prompt's 'don't redo the captions' refers to. Fix the fixture before the next run of this id, or its grade stays ambiguous.",+ "Repeats exist only for cw1, dl1, dl4 and cs1. Variance on the other 14 ids is still unmeasured, and the Look coin flip found in the repeats is a warning about reading any single run of those 14.",+ "whisper is absent from the machine by design, so fw2 meets a real missing engine rather than a simulated one; fw1 and fw3 are given word-timing JSON fixtures instead.",+ "beats.mp4, no_beats.mp4, gappy.mp4 and tests/out/batch_in/{a,b}.mp4 are gitignored fixtures carried over from eval 18's staging; anyone re-running must build them the same way first."+ ],+ "follow_ups": [+ "1.17.2 blocker: caption.py write_ass must derive MarginL and MarginR from the platform's horizontal safe fractions * play_w (default the 5% title-safe), independently of MarginV. Regression test asserting Style MarginL + MarginR <= (1 - SAFE_WIDTH_FRACTION) * play_w at tiktok geometry, and an e2e asserting the generated ASS for 'Hello world' at --platform tiktok contains no \\N.",+ "1.17.2 blocker: make the fitter (line_em_for_size) measure against the same horizontal budget the Style row will actually grant, so `split: 0`, `size_used` and `text_unchanged` describe the picture. Until they do, every template caption report states numbers the agent cannot falsify.",+ "1.17.2: an explicit refusal line in SKILL.md's caption section - never rewrite, shorten or paraphrase the user's caption text; offer --max-lines or a smaller size instead. cs3 is 3/3 misses across iterations 17, 18 and 19 and prose elsewhere in the file has not closed it.",+ "Give the agent something to check a Look line against: a caption.py --fit-report (or the same numbers on --json) reporting lines-per-cue as libass will lay them out. Five of this iteration's six loose Look lines are agents describing a picture they had on disk but never measured.",+ "Fix the rc1 fixture: give its project.json a captions stage so 'don't redo the captions' has something to cache, then re-run rc1 and rc2 together.",+ "silence.py --filler: decide whether it should leave silences alone unless asked (fw1 and fw3 each removed ~5 s nobody requested, both disclosed). 1.18 decision; at minimum the SKILL.md row must say what the default does.",+ "silence.py's --filler error text should name whisper.cpp / faster-whisper / openai-whisper and a pip line, so fw2-shaped refusals are actionable without the agent inventing engine names.",+ "Eval 20: re-run this 18-id set plus the full 100 at 1.17.2, and grade the caption prompts on the sheet rather than on the fit stats - the question is how many lines the rendered frame has, not what the fitter reports."+ ]+}diff --git a/evals/trigger/README.md b/evals/trigger/README.mdindex e0acdbd..f559472 100644--- a/evals/trigger/README.md+++ b/evals/trigger/README.md@@ -6,4 +6,4 @@ Method: show an independent model a skill catalog made of this skill's `descriptThe non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns). 2026-09-14 (it18): 50/50 on the full 50-prompt set — 31/31 should-trigger, including the five new 1.17 prompts, and 19/19 near-miss.+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 40 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not; then 2 for 1.15: t23 an emoji caption request should-trigger, n17 an emoji sticker pack as PNGs should-not — the neighbouring territory image-tools owns). 2026-09-14 (it18): 50/50 on the full 50-prompt set — 31/31 should-trigger, including the five new 1.17 prompts, and 19/19 near-miss. 2026-09-14 (it19): 50/50 again on the same 50 prompts at skill 1.17.1 — 31/31 should-trigger and 19/19 near-miss, unchanged.diff --git a/evals/trigger/results-2026-09-14-it19.json b/evals/trigger/results-2026-09-14-it19.jsonnew file mode 100644index 0000000..af29b0e--- /dev/null+++ b/evals/trigger/results-2026-09-14-it19.json@@ -0,0 +1,59 @@+{+ "date": "2026-09-14",+ "skill_version": "1.17.1",+ "iteration": 19,+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 50; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id":"t01","lang":"en","prompt":"Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim a video file to a duration."},+ {"id":"t02","lang":"ja","prompt":"この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical + add subtitles."},+ {"id":"t03","lang":"en","prompt":"the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio level fix on a video file."},+ {"id":"t04","lang":"ja","prompt":"Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from a recording."},+ {"id":"t05","lang":"en","prompt":"I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"External-mic sync task."},+ {"id":"t06","lang":"ja","prompt":"iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"HDR-to-SDR conversion issue."},+ {"id":"t07","lang":"en","prompt":"put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Logo overlay + fade on video files."},+ {"id":"t08","lang":"ja","prompt":"podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization + transcode."},+ {"id":"t09","lang":"en","prompt":"Does this file meet TikTok's specs? final_v3.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Platform compliance check on a video file."},+ {"id":"t10","lang":"ja","prompt":"2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Multicam highlight reel."},+ {"id":"t46","lang":"en","prompt":"the subtitles on my short are coming out in three lines and half the sentence shows up late, can you fix reel.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Fixing subtitle timing/wrapping on a video file."},+ {"id":"t47","lang":"ja","prompt":"BGM のビートに合わせて clip.mp4 をカットしたい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Cutting video to music beat."},+ {"id":"t48","lang":"en","prompt":"strip the ums and uhs out of my podcast recording ep12.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Editing filler words out of an audio recording."},+ {"id":"t49","lang":"es","prompt":"tengo 200 vídeos en una carpeta y tardan una eternidad en procesarse","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Batch video processing performance issue."},+ {"id":"t50","lang":"en","prompt":"I only changed the export preset, do I have to re-render the whole edit again? project.json","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Whole-edit project file re-render question."},+ {"id":"n01","lang":"en","prompt":"Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still-image resize/transparency, not video/audio."},+ {"id":"n02","lang":"ja","prompt":"Windows に ffmpeg をインストールする方法を教えて","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question, not an editing task."},+ {"id":"n03","lang":"en","prompt":"Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger":false,"chosen":"web-research","model":"sonnet","correct":true,"reason":"Fetching online content, not local editing."},+ {"id":"n04","lang":"ja","prompt":"新商品紹介動画の台本(60 秒)を書いて","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, no media file involved."},+ {"id":"n05","lang":"en","prompt":"What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Third-party app UI question, not ffmpeg editing."},+ {"id":"n06","lang":"en","prompt":"Design a YouTube thumbnail in Figma for my cooking channel","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Graphic design in Figma; no matching skill in catalog."},+ {"id":"n07","lang":"ja","prompt":"このCSVをJSONに変換して: sales.csv","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Data format conversion, unrelated to media."},+ {"id":"n08","lang":"en","prompt":"Explain the difference between H.264 and H.265 in simple terms","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Conceptual explanation, no file/task to perform."},+ {"id":"n09","lang":"en","prompt":"Generate a 10-second video of a cat surfing with AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation from scratch, not editing an existing file."},+ {"id":"n10","lang":"ja","prompt":"会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger":false,"chosen":"meeting-notes","model":"sonnet","correct":true,"reason":"Transcription/minutes from a meeting recording."},+ {"id":"t11","lang":"en","prompt":"Convert this WAV to MP3: interview_raw.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio transcoding."},+ {"id":"t12","lang":"en","prompt":"Remove the silence from this recording, it's a 40 minute M4A","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal on an audio file."},+ {"id":"t13","lang":"zh","prompt":"这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim video + add subtitles."},+ {"id":"t14","lang":"ko","prompt":"인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical for Reels."},+ {"id":"t15","lang":"es","prompt":"El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization."},+ {"id":"t16","lang":"ar","prompt":"هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from podcast audio."},+ {"id":"n11","lang":"pt","prompt":"Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Script writing, no media file."},+ {"id":"n12","lang":"fr","prompt":"Comment installer ffmpeg sur macOS avec Homebrew ?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question."},+ {"id":"n13","lang":"de","prompt":"Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing."},+ {"id":"t17","lang":"th","prompt":"ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + subtitles on a video file."},+ {"id":"t18","lang":"hi","prompt":"interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Loudness normalization."},+ {"id":"t19","lang":"he","prompt":"אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe + subtitles for Reels."},+ {"id":"t20","lang":"ru","prompt":"Убери, пожалуйста, тишину из записи podcast.m4a","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal."},+ {"id":"t21","lang":"vi","prompt":"Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + subtitles."},+ {"id":"t22","lang":"tr","prompt":"roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe for Reels."},+ {"id":"n14","lang":"el","prompt":"Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question."},+ {"id":"n15","lang":"id","prompt":"Buatkan video 10 detik tentang kucing yang berselancar dengan AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing."},+ {"id":"n16","lang":"it","prompt":"Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Script writing, no media file."},+ {"id":"t23","lang":"en","prompt":"can you burn the subtitles into my clip and keep the emoji in the captions in colour? interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Burning captions into a video file."},+ {"id":"n17","lang":"en","prompt":"make me an emoji sticker pack as PNGs, 20 stickers with transparent backgrounds","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Sticker pack images, not video/audio editing."},+ {"id":"t41","lang":"en","prompt":"can you make my podcast episode into something I can post on YouTube? it's just an mp3","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Convert audio into a YouTube-ready video file."},+ {"id":"t42","lang":"ja","prompt":"エピソードにチャプターを入れたいんだけど、どこで区切ればいいか分からない","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Adding chapters to an episode, scene-detection territory."},+ {"id":"t43","lang":"es","prompt":"necesito que el vídeo lleve subtítulos en inglés y en español, que se puedan cambiar","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Selectable multi-language subtitles on a video."},+ {"id":"n18","lang":"en","prompt":"write the chapter titles for my blog post outline","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing text outline, no media file."},+ {"id":"n19","lang":"de","prompt":"entwirf ein Podcast-Cover für Spotify","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Cover art design, a still image not audio editing."}+ ],+ "summary": {"total": 50, "correct": 50, "false_negatives": 0, "false_positives": 0}+}