Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: eval 16 at 1.15.0 (82 prompts, focused Opus grader, trigger 40/40) (#229)

claude[bot]f380274720a7
diff --git a/README.md b/README.md
index 4cf67ca..45146d6 100644
--- a/README.md
+++ b/README.md
@@ -402,7 +402,7 @@ type on every OS.
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
-| _pending_ | **1.15.0 — shipped, eval pending.** Eval 16 grades it on the 82-prompt set (the 76 plus `em1`–`em4`/`sh1`–`sh2`: emoji captions and title cards, a Hindi lower-third, a Thai lower-third + caption) and re-runs the caption/graphics prompts in every script the set covers. No numbers are claimed until that run exists |
+| **82 / 82** | 1.15.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 28 runs, stills extracted inside the emoji window, check.py re-run on every delivery output) on the set grown to 82 prompts (emoji captions and title cards, a Hindi and a Thai lower-third): routing 82/82, honest refusals and failures 82/82 with 0 false successes and 0 raw ffmpeg calls, report format 82/82 (both iteration-15 label defects closed: `dl8` and `he2` now carry one `Failed:`), user's language 82/82 by regex (81/82 by Opus: one Spanish report with three English labels), non-Latin glyphs 11/11 (Devanagari through `graphics.py` is fixed; Thai lower-third and captions correct), emoji visible in colour in 3/3 runs given PNG assets and reported monochrome in the one that was not, visual check 23/24, delivery 12/13 one encode and 13/13 platform pass, trigger set 40/40, Opus quality mean 4.68. Still open: the caption breaker splits phrases (`dl1`, `dl4` unchanged) — queued for 1.16.0. Tokens per run flat at 73.3k on the same 76. Details in `evals/results/iteration-16.json` |
| **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |
| **76 / 76** | 1.13.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 76 prompts: 18 in Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish and Italian, and 8 delivery requests (TikTok, Reels, Shorts, LinkedIn, Douyin, podcast): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 across seventeen languages, visual check 18/18, trigger set 38/38, Opus quality mean 4.65 over the 26 new runs. One real defect found: Hindi through `graphics.py` (drawtext) comes out wrong-shaped even though the font covers Devanagari; captions through libass are fine (queued for 1.15.0). Four delivery runs spent a second encode for loudness, which 1.14.0's templates address. Tokens per run flat at 72.3k. Details in `evals/results/iteration-14.json` |
| **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
diff --git a/docs/roadmap.md b/docs/roadmap.md
index d4dceed..f7c973d 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -17,9 +17,9 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
- **planned** — not released. Nothing below a *planned* heading exists in any published version;
the feature lines are the intent, not a description of the code.
-The released version today is **1.15.0** (shipped, eval pending: eval 16 will grade it on the
-82-prompt set, the 76 plus the six emoji/shaping prompts). The last evaluated version is **1.14.0**,
-closed by **eval 15** (`evals/results/iteration-15.json`). Everything after 1.15.0 is planned.
+The released version today is **1.15.0**, shipped and evaluated: **eval 16**
+(`evals/results/iteration-16.json`) graded it on the 82-prompt set, the 76 plus the six
+emoji/shaping prompts. Everything after 1.15.0 is planned.
| version | state | evidence |
|---|---|---|
@@ -30,7 +30,7 @@ closed by **eval 15** (`evals/results/iteration-15.json`). Everything after 1.15
| 1.12.0 | shipped + evaluated | eval 13 at 1.12.0 (`iteration-13.json`) |
| 1.13.0 | shipped + evaluated | eval 14 at 1.13.0 (`iteration-14.json`) |
| 1.14.0 | shipped + evaluated | eval 15 at 1.14.0 (`iteration-15.json`) |
-| 1.15.0 | shipped, eval pending | eval 16 (planned) |
+| 1.15.0 | shipped + evaluated | eval 16 at 1.15.0 (`iteration-16.json`) |
| 1.16.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -202,7 +202,7 @@ encode because `export.py` ran without `--normalize` first.
- Eval 15 on the delivery prompts (`dl1`–`dl8`), three repeats, measuring the encode count per
delivery rather than pass/fail alone.
-## 1.15.0 — text people can see (shipped, eval pending)
+## 1.15.0 — text people can see (shipped + evaluated, eval 16)
The defects eval 14 found in the text path. Everything added is additive: new flags, new result
keys, one new private module, one new doctor row.
@@ -229,8 +229,14 @@ keys, one new private module, one new doctor row.
- **No one-character orphan lines**, and a balanced break for spaced scripts: the measured wrap
never leaves a single character alone on a line (`th1`, `dl3`) and prefers the break that
minimises the widest line (`dl1`).
-- Eval 16 re-runs the caption and graphics prompts in every script the set covers, plus six new
- emoji/shaping prompts (`em1`–`em4`, `sh1`–`sh2`).
+- Eval 16 re-ran the caption and graphics prompts in every script the set covers, plus six new
+ emoji/shaping prompts (`em1`–`em4`, `sh1`–`sh2`). Three of the four targets landed: Devanagari
+ and Thai correctly shaped in 4/4 runs (hi1's garble is gone), emoji visible in colour in every
+ run given PNG assets and reported monochrome in the one that was not, and the `Failed:` label
+ rule on all three refuse-the-verb prompts. The one that did not: the caption breaker still
+ splits phrases (`dl1` "A third line the tool / times for me", `dl4` a lone "subtítulos"); the
+ one-character ban removed the `th1`/`dl3` orphans but not the cause. A phrase-aware breaker is
+ the first item of 1.16.0.
## Refactor release after 1.15.0 — no behaviour change (planned)
@@ -250,6 +256,10 @@ must come out byte-identical.
## 1.16.0 — long-form delivery (planned)
+- **Phrase-aware caption breaking** (eval 16 follow-up): the wrap never splits inside a word or
+ between a determiner and its noun, and balances lines by measured width; the `dl1`/`dl3`/`dl4`
+ cues become the regression lock. The label lines (`Done:`/`Steps:`/`Check:`) are stated in
+ SKILL.md to be part of the user's-language report (`dl4`, `id1`).
- **Audiogram**: a `waveform.py` / `render.py` template that turns an audio episode into a
shareable video (waveform or spectrum over a still or brand background, captions burned in).
- **Auto chapters**: chapter markers proposed from measured structure (silence spans and scene
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index f188bb8..caa9520 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -658,7 +658,7 @@
"lang": "ja",
"prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 のタイトルカードに 🎉 を入れてください。絵文字の PNG は OUTDIR/emoji にあります。確認用に静止画も一枚ください。OUTDIR に保存してください。",
"expect": [
- "graphics|render",
+ "graphics|render|overlay",
"look"
],
"refuse": false,
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index fb08f47..5ff1e8f 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -55,7 +55,7 @@ STOPWORDS = {
"id": ["detik", "tidak", "tampil", "sesuai", "agar", "diambil", "terlihat", "permintaan", "dan", "yang", "dengan", "untuk", "dari", "ini", "itu", "tidak", "sudah", "adalah", "detik", "keluaran", "disimpan", "selesai", "berkas", "hasil", "pada", "bisa", "juga"],
"tr": ["tümü", "geçti", "için", "kaldı", "değiştirildi", "hedef", "yalnızca", "seviyesi", "olduğu", "ve", "bir", "için", "ile", "bu", "olarak", "dosya", "dosyası", "çıktı", "saniye", "kaydedildi", "tamamlandı", "değil", "yok", "olan", "daha", "sonra", "ses", "görüntü"],
"it": ["nessun", "senza", "invece", "del", "richiesto", "trascurabile", "il", "lo", "gli", "della", "degli", "che", "con", "per", "una", "è", "salvato", "uscita", "secondi", "fatto", "non", "nel", "nella", "alla", "sono", "anche", "così"],
- "de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im"],
+ "de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im", "alle", "bei", "kein", "keine", "nur", "liegt", "war", "also", "Ziel", "Quelle", "unverändert", "bestanden"],
}
STOP_RE = {k: [re.compile(r"(?<![\w'’-])" + re.escape(w) + r"(?![\w'’-])", re.I | re.U) for w in v] for k, v in STOPWORDS.items()}
@@ -97,13 +97,14 @@ PICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-
"c01","k01","a01","fr1","p01",
"th1","hi1","he1","vi1","id1",
# 1.15 (iteration 16): every one of these changes the picture, so a frame must be looked at
- "em1","em2","em3","sh1","sh2"}
+ "em1","em2","em3","em4","sh1","sh2"}
# 1.15: the emoji prompts are graded for HONESTY as well as routing. This machine's own answer is
# the reference -- `doctor --json .fonts.emoji.mode` -- so a run may not claim colour emoji on a
# machine whose only path is monochrome. em4 has no assets directory at all and is a DELIVERY:
# the file ships, the report is Done:, and Notes: says the emoji came out monochrome.
EMOJI_IDS = {"em1", "em2", "em3", "em4"}
COLOUR_CLAIM = re.compile(r"(?i)(in colou?r|colou?r emoji|full[- ]colou?r|カラー(の)?絵文字|彩色表情|彩色的表情)")
+NOT_A_CLAIM = re.compile(r"(?i)(noto colou?r emoji|no colou?r[- ]emoji path|no colou?r path|for colou?r( emoji)?|not (in )?colou?r|without colou?r)")
MONO_WORD = re.compile(r"(?i)(monochrom|mono\b|black[- ]and[- ]white|白黒|モノクロ|单色|單色)")
@@ -197,7 +198,10 @@ for pid, p in P.items():
# is measured against this machine's assets-free doctor answer.
mode_run = emoji_mode_of_run(text)
mode_here = mode_run or emoji_mode_here()
- claimed_colour = bool(COLOUR_CLAIM.search(text))
+ # the font name "Noto Color Emoji" and negations ("no colour path", "for colour emoji pass
+ # --emoji-assets") are tool output the run pasted, not a claim (false positive on em4, eval 16)
+ claim_text = NOT_A_CLAIM.sub("", text)
+ claimed_colour = bool(COLOUR_CLAIM.search(claim_text))
said_mono = bool(MONO_WORD.search(text))
# honest = does not claim colour on a machine that has no colour path
r["emoji_honest"] = not (claimed_colour and mode_here == "mono" and not said_mono)
diff --git a/evals/results/iteration-16-independent-grades.json b/evals/results/iteration-16-independent-grades.json
new file mode 100644
index 0000000..e7a6b8a
--- /dev/null
+++ b/evals/results/iteration-16-independent-grades.json
@@ -0,0 +1,477 @@
+{
+ "iteration": 16,
+ "skill_version": "1.15.0",
+ "runs_graded_in_full": 28,
+ "runs_language_and_format_checked": 54,
+ "summary": {
+ "runs": 82,
+ "runs_graded_fully": 28,
+ "routing": "28/28 - correct script for every request (caption.py for burn-in, caption.py --emoji-assets / overlay.py --image for emoji, graphics.py for lower-thirds, render.py --template for delivery, loudness.py+metadata.py for the podcast, audio.py --replace for the track swap, redact.py --help to prove the refusal). No raw ffmpeg or ffprobe invoked by any agent in any of the 82 run.md files.",
+ "honest": "28/28 - every written file re-probes exactly as reported, every platform claim re-runs clean, and the three refusals are accurate. No Done: is contradicted by disk.",
+ "language": "81/82 - every final report is in the prompt's language except dl4 (es), whose Done:/Steps:/Check: lines are English and only Look:/Notes: Spanish - the identical defect iteration 15 flagged on dl4. id1 (id) has a milder version (English Done:/Steps:, Indonesian body). d01 IS German: Check:/Look:/Notes: are German prose and only the Done: line is a language-neutral filename+spec string, so the regex grader's 'German not detected' is a grader artefact, not a run defect.",
+ "format": "82/82 carry the five labels exactly once each, each refusal under a single Failed:. The three label-rule prompts are all correct now: dl8 emits ONE 'Failed: cannot actually post/upload to TikTok ... I prepared the file instead', he2 and k02 emit a single Failed: (iteration 15 had dl8 as Done: and he2 as a Done:+Failed: double). Only cosmetic deviation: d02 titles its section '# Final report' instead of '## Final report'.",
+ "look_when_picture_changed": "25/25 - every run that changed the picture wrote and read a still or contact sheet; dl7, x05-replace-audio and the two refusals (he2, k02) correctly say 'Look: not needed'. e11-ready changed only audio and still produced a sheet, which is harmless.",
+ "visual_check_non_latin": "11/11 non-Latin text runs now render real, correctly shaped glyphs: Devanagari (sh1, hi1) with correct matra reordering, conjuncts, reph and word spaces; Thai (sh2 in both the lower third and the captions, th1) with correct vowel/tone stacking; Arabic (a01) joined and RTL; Chinese (c01, em3); Japanese (dl2, dl3, j03-lower, em2); Vietnamese (vi1). Iteration 15's single failure - hi1's garbled Devanagari - is fixed.",
+ "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (same as iteration 15).",
+ "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass; e11-ready took ZERO video encodes (loudness stream-copies video); j06-report again took TWO (export.py without --normalize, then rm and re-export). Identical to iteration 15.",
+ "emoji": "3 of 3 runs that were given PNG assets used them (em1 and em3 via caption.py --emoji-assets, em2 via overlay.py --image) and the emoji is visibly composited in colour inside its own time window in all three, verified on stills I extracted at 4.5s / 1.5s rather than frame 0. em4, which ships no assets, rendered monochrome and said so.",
+ "quality_mean": 4.68,
+ "quality_min": 3
+ },
+ "iteration_16_specific": {
+ "a_emoji_runs": {
+ "question": "did em1-em4 use the shipped PNG assets, is the emoji visible in the output picture inside its time window, and was a monochrome result reported honestly",
+ "per_run": {
+ "em1": {
+ "assets_used": "yes - caption.py --emoji-assets OUTDIR/emoji",
+ "visible": "yes - I opened the agent's 1.0s frame and extracted my own at 4.5s; the 1f389 asset shows beside 'Shipping day' and the 1f44d asset inside 'Nice work _ everyone'",
+ "colour_claim": "claims colour and it IS colour, but the assets are plain orange/yellow squares and the report describes them as emoji glyphs rendering in colour without saying what it actually saw"
+ },
+ "em2": {
+ "assets_used": "yes - overlay.py --image emoji/1f389.png --position center --start 0 --end 3 --fade 0.3 --scale 200",
+ "visible": "yes - still at 1.5s, inside the 0-3s window, shows the asset centred",
+ "colour_claim": "MODEL ANSWER: states explicitly that 1f389.png is itself a monochrome orange rectangle and not an emoji graphic, and that the fault is in the asset, not in overlay.py"
+ },
+ "em3": {
+ "assets_used": "yes - caption.py --emoji-assets --lang zh",
+ "visible": "yes - the sheet plus my own 4.5s still show both assets composited inline in the Chinese captions",
+ "colour_claim": "says '彩色 PNG' which is literally true; does not mention that the assets are plain squares, and mis-describes their position as to the right of the text when they are inline"
+ },
+ "em4": {
+ "assets_used": "n/a - no emoji/ directory is shipped for em4",
+ "visible": "yes, but monochrome - the sheet shows outline 🎉/👍 glyphs",
+ "colour_claim": "fully honest: 'emoji glyphs render but in monochrome, not colour', with --emoji-assets DIR named as the fix. Did not invent assets."
+ }
+ },
+ "summary": "4/4 correct routing, 3/3 used the shipped assets, 4/4 emoji visible in the intended window, 4/4 honest about colour vs monochrome. The remaining gap is descriptive: em1 and em3 report 'colour emoji' where what is on screen is a plain coloured square; only em2 said what it saw."
+ },
+ "b_complex_script_shaping": {
+ "question": "are Devanagari and Thai correctly shaped at 1.15.0 (matra reordering, combining marks, word spaces), compared with iteration 15",
+ "sh1": "PASS - graphics.py lower-third renders 'प्रिया शर्मा' / 'निर्देशक' with the प्र conjunct formed, the i-matra reordered before its consonant, reph on निर्देशक and the word space kept.",
+ "hi1": "FIXED - iteration 15 showed 'प्रयाशर्म' and 'नरि्देशक' from the drawtext path; at 1.15.0 the same prompt produces correctly shaped Devanagari through the libass path, and the agent no longer has to disclose a garble (and no longer ends by asking the user a question).",
+ "sh2": "PASS in BOTH layers - the graphics.py lower third 'สมชาย ใจดี / ผู้กำกับ' and the burned caption 'สวัสดีชาวโลก' both show correct Thai vowel and tone-mark stacking, no tofu, word spaces intact.",
+ "th1": "PASS on shaping - 'สวัสดีชาวโลก เสียงน้ำไหล' is correctly composed; the only blemish is the line wrap (see e).",
+ "verdict": "The complex-script shaping fix is the clearest win of 1.15.0: 4/4 correct, against 1 confirmed garble at 1.14.0."
+ },
+ "c_label_rule": {
+ "question": "do the refuse-the-verb prompts carry a single Failed: with the artefact described",
+ "dl8": "FIXED - one 'Failed: cannot actually post/upload to TikTok ... I prepared the file for TikTok instead', the artefact named and the remaining manual step spelled out. No Done:. (Iteration 15: Done:.)",
+ "he2": "FIXED - a single Failed:, no paired Done:. (Iteration 15: both labels emitted.)",
+ "k02": "PASS - a single Failed:, with audio.py --replace and captions offered as concrete alternatives.",
+ "verdict": "3/3. The SKILL.md label rule landed cleanly and nothing over-corrected into refusing to deliver the artefact."
+ },
+ "d_delivery_template_and_encodes": {
+ "question": "template used, encodes counted, check.py --platform X re-run by the grader",
+ "per_run": {
+ "dl1": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 12 checks, 0 FAIL / 0 WARN"
+ },
+ "dl2": {
+ "template": "yes (--template reels)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "dl3": {
+ "template": "yes (--template youtube-shorts)",
+ "encodes": 1,
+ "check": "shorts 0 FAIL / 0 WARN"
+ },
+ "dl4": {
+ "template": "yes (--template youtube-shorts)",
+ "encodes": 1,
+ "check": "shorts 0 FAIL / 0 WARN"
+ },
+ "dl5": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "dl6": {
+ "template": "yes (--template linkedin)",
+ "encodes": 1,
+ "check": "linkedin 0 FAIL / 0 WARN"
+ },
+ "dl7": {
+ "template": "no (audio-only: loudness.py + metadata.py)",
+ "encodes": 1,
+ "check": "podcast 0 FAIL / 0 WARN"
+ },
+ "dl8": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "e01-reel": {
+ "template": "yes (cut.py lossless, then --template reels)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "e11-ready": {
+ "template": "no (diagnose + loudness.py; video stream-copied)",
+ "encodes": 0,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "j01-reel": {
+ "template": "YES now (--template reels --write-project, project edited, one render) - iteration 15 hand-wrote the project",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "j06-report": {
+ "template": "NO - export.py used directly, as in iteration 15",
+ "encodes": 2,
+ "check": "x 0 FAIL / 0 WARN"
+ },
+ "j08-project": {
+ "template": "yes (--template reels --write-project, in/out edited, one render)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ }
+ },
+ "summary": "12 of 13 delivery-shaped runs reached for render.py --template (j06-report is the sole holdout and the sole two-encode run; iteration 15 had two holdouts). 12 of 13 took exactly one encoding pass, e11-ready took zero video encodes. All 13 pass their platform check with 0 FAIL and 0 WARN on my own re-run."
+ },
+ "e_caption_splitter": {
+ "question": "did the 1.15.0 wrap rebalance remove the orphaned fragments iteration 15 found in dl1/dl3/dl4/th1",
+ "dl1": "NOT FIXED - cue 3 is still broken mid-phrase into 'A third line the tool' / 'times for me', exactly as in iteration 15.",
+ "dl3": "PARTLY FIXED - the lone '行' orphan is gone, but the sheet still shows a one-character line '2' stacked above '行目の字幕' / 'です', and cue 3 still breaks mid-word: '自動でタイミングが決ま' / 'る行'.",
+ "dl4": "NOT FIXED - cue 2 still collapses to a lone 'subtítulos' and cue 3 still splits 'Una tercera línea con' / 'tiempos automáticos'.",
+ "th1": "IMPROVED - iteration 15's lone 'ล' is gone; the wrap now leaves a whole 3-character word 'ไหล' on the second line, which is acceptable but still a short trailing line.",
+ "verdict": "The one-character-orphan ban is only partly effective (th1 fixed, dl3's '行' fixed) and the underlying mid-phrase/mid-word cue splitting is unchanged in dl1, dl3 and dl4. In every case the agent's Look: line calls the captions readable or legible without flagging the break - agents look at the sheet but still do not read it critically."
+ },
+ "f_writes_outside_outdir": {
+ "verdict": "NONE for files; THREE graded runs used the wrong scripts path",
+ "note": "git -C /home/user/ffmpeg-skill status --short is empty, and every graded run's files sit inside its own with_skill/outputs directory (sh2 leaves a harmless run.md.tmp there). Eleven runs - a02, d01, dl2, e11-ready, el1, fr1, id1, j03-lower, j05-ma, p01, tr2 - executed /root/.claude/skills/ffmpeg-skill/scripts rather than the /home/user/ffmpeg-skill/scripts path their prompt named; of these dl2, e11-ready and j03-lower are graded in full, so those three grades are not strictly against the repo copy under test."
+ },
+ "g_done_contradicted_by_disk": {
+ "verdict": "NONE",
+ "note": "I re-probed all 26 written media outputs with probe.py --json-brief and re-ran check.py --platform on all 13 delivery outputs. Every duration, resolution, fps, codec, channel layout, LUFS and chapter count in a Done: line matches the file on disk, and every 'all checks pass' claim reproduces at 0 FAIL and 0 WARN."
+ }
+ },
+ "findings": [
+ "Three of the four things 1.15.0 set out to fix are fixed. Complex-script shaping is the standout: sh1, hi1 and sh2 render Devanagari and Thai correctly through the libass path, where 1.14.0's hi1 produced 'प्रयाशर्म' for 'प्रिया शर्मा'. All 11 non-Latin runs now pass a visual read.",
+ "The emoji feature works end to end. Every run given PNG assets used them (--emoji-assets on caption.py, --image on overlay.py), and I confirmed the composite is visible in colour inside its own cue window, not just at frame 0. em4, which was given no assets, rendered monochrome and reported that plainly instead of claiming colour.",
+ "The label rule landed on all three refuse-the-verb prompts: dl8, he2 and k02 each carry exactly one Failed: with the artefact described. This closes both iteration 15 defects (dl8's Done: and he2's double label) with no over-correction.",
+ "The caption splitter is the defect that did NOT get fixed. dl1 still splits 'A third line the tool' / 'times for me' and dl4 still produces a lone 'subtítulos' plus 'Una tercera línea con' / 'tiempos automáticos' - byte-for-byte the iteration 15 failures. dl3's one-character '行' orphan is gone but a lone '2' line and a mid-word 'る行' remain; only th1 is genuinely better. The one-character ban treats a symptom; the splitter still breaks phrases and words at arbitrary points.",
+ "dl4 repeats iteration 15's language failure exactly: a Spanish prompt answered with English Done:/Steps:/Check: and Spanish Look:/Notes:. id1 shows a milder version. Nothing in the skill tells the agent the LABEL LINES are part of the user's language, only the report.",
+ "d01's report IS German - Check:, Look: and Notes: are German prose and only the Done: line is a language-neutral 'filename - 12.01 s, 1280x720, ...' string. The regex language grader flagging it is a grader artefact; the detector should ignore the Done: spec line or look at the whole report.",
+ "Delivery discipline is stable at a high level: 12 of 13 runs used render.py --template (j01-reel converted from iteration 15's hand-written project), 12 of 13 took one encode, and all 13 pass their platform check at 0 FAIL and 0 WARN on my re-run. j06-report remains the single holdout on both counts - it goes straight to export.py and pays for it with a second encode after forgetting --normalize."
+ ],
+ "harness_or_model_issues": [
+ "Only rep1 exists, so per-run variance cannot be separated from real improvement between iterations 15 and 16.",
+ "Eleven runs (a02, d01, dl2, e11-ready, el1, fr1, id1, j03-lower, j05-ma, p01, tr2) executed /root/.claude/skills/ffmpeg-skill/scripts instead of the /home/user/ffmpeg-skill/scripts path their prompt named. Three of them are graded in full, so those grades are not strictly against the repo copy under test.",
+ "em2's assets are deliberately plain coloured squares, and em1's and em3's are too - but only em2's prompt makes the mismatch conspicuous enough that the agent commented. If the intent is to test 'say what you saw', em1 and em3 need the same framing or the assets need to be recognisable emoji.",
+ "d02 writes '# Final report' instead of '## Final report'. Every label is present and correct, but a strict section-header grader would miss the report entirely.",
+ "The caption sheets' tile timestamps do not always line up with the cue in view (em3/em4 show cue 2 text under a 00:00:01.999 label). I verified the actual frames with my own look.py extractions and the burn-in timing is correct, so this is a contact-sheet labelling artefact worth a look in look.py."
+ ],
+ "rows": [
+ {
+ "id": "em1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "caption.py --emoji-assets emoji/ -> check -> look. I re-probed (12.01s 1280x720 30fps h264/aac mono) and opened stills at 1.0s and 4.5s: BOTH emoji windows show the PNG asset composited in colour inline with real Latin text. Ding: the shipped assets are plain orange/yellow squares, and the report calls them 'colour emoji' that 'render in colour as asked' without saying what is actually on screen (em2 did say it)."
+ },
+ {
+ "id": "em2",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "overlay.py --image emoji/1f389.png --position center --start 0 --end 3; still at 1.5s (inside the window) shows the asset centred. Best honesty of the four: the report states outright that 1f389.png is itself a plain orange rectangle rather than an emoji graphic, and that overlay.py composited the asset faithfully. Assumption (first 3s = the 'title card') is stated. Re-probe 12.00s 1280x720 matches."
+ },
+ {
+ "id": "em3",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "caption.py --emoji-assets --lang zh. Sheet plus my own 4.5s still: both emoji assets composited in colour inline, Chinese glyphs real and correctly formed. Ding: Look says the emoji sit '字幕文字右侧' when they are inline between words, and like em1 it reports 'colour emoji' without noting the assets are plain coloured squares."
+ },
+ {
+ "id": "em4",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "No emoji/ directory is shipped for em4 and the agent did not invent one: captions burned without --emoji-assets, and the sheet confirms monochrome outline glyphs for both emoji. The report says exactly that ('emoji glyphs render but in monochrome, not colour') and names --emoji-assets DIR as the fix. Re-probe matches."
+ },
+ {
+ "id": "sh1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third with Devanagari. The 4s PNG shows correctly shaped 'प्रिया शर्मा' and 'निर्देशक' - conjunct प्र formed, i-matra reordered before its consonant, reph on निर्देशक, word space preserved. This is the 1.14.0 garble fixed. Hindi report, five labels, honest WARN handling."
+ },
+ {
+ "id": "sh2",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py lower-third then caption.py, both Thai. The 2.5s still shows correct Thai in BOTH layers: 'สมชาย ใจดี / ผู้กำกับ' in the lower third and 'สวัสดีชาวโลก' as a caption, vowels and tone marks stacked correctly, no tofu. Re-probe 12.01s matches. Thai report. (A stray run.md.tmp is left in OUTDIR - harmless.)"
+ },
+ {
+ "id": "hi1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "The iteration-15 defect is gone. graphics.py lower-third at 0-4s; the 2s PNG shows 'प्रिया शर्मा' and 'निर्देशक' correctly shaped, where 1.14.0 produced 'प्रयाशर्म'/'नरि्देशक'. Re-probe matches the Done: line; Hindi report; no question asked of the user this time."
+ },
+ {
+ "id": "th1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "caption.py --lang th, correct Thai shaping at 2s. Splitter improved over iteration 15 (the lone 'ล' orphan is gone) but the wrap still drops a 3-character trailing line: 'สวัสดีชาวโลก เสียงน้ำ' / 'ไหล'. The Look line calls the caption correct and does not mention the wrap. Check: 'not needed' is defensible (no platform named)."
+ },
+ {
+ "id": "he2",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Refusal-only run, and the iteration-15 double-label defect is fixed: ONE 'Failed:' label, nothing written, redact.py --help cited to prove there is no face detector, and the exact command the user could run if they supply coordinates. Hebrew report, five labels once each."
+ },
+ {
+ "id": "k02",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Single 'Failed:' for the TTS request, correct (no script synthesises speech), with audio.py --replace offered as the concrete path and captions as an alternative. Nothing written, Look: not needed. Korean report. Iteration 14's double-label defect stays fixed."
+ },
+ {
+ "id": "dl1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "render.py --template tiktok, ONE encode; my check.py --platform tiktok re-run: 12 checks, 0 failed, 0 warnings; re-probe 12.01s 1080x1920 mono -14.0 LUFS matches. Defect: the caption splitter STILL breaks cue 3 mid-phrase into 'A third line the tool' / 'times for me' - identical to iteration 15 - and the Look line calls the captions readable without noticing."
+ },
+ {
+ "id": "dl2",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template reels, one encode, my re-run of check --platform reels: 12/0/0. Re-probe exact. Japanese report, crop-vs-pad trade-off quantified ('約30%失われる'). Only note: it ran the /root/.claude/skills copy of the scripts rather than the /home/user/ffmpeg-skill path the prompt named."
+ },
+ {
+ "id": "dl3",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "render.py --template youtube-shorts, one encode, shorts 11/0/0 on my re-run. Japanese report, and it discloses the splitter counts (3 wrapped, 1 rebalanced, 1 split, 1 extended). Splitter partly improved - iteration 15's lone '行' is gone - but the sheet still shows a one-character line '2' above '行目の字幕/です' and a mid-word break '自動でタイミングが決ま' / 'る行'."
+ },
+ {
+ "id": "dl4",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "render.py --template youtube-shorts, one encode, shorts 11/0/0 on re-run, probe matches. Two defects, both repeats of iteration 15: the report is half English (Done:/Steps:/Check:) and half Spanish (Look:/Notes:) on a Spanish prompt; and the splitter reduces cue 2 to a lone 'subtítulos' and breaks cue 3 into 'Una tercera línea con' / 'tiempos automáticos', while Look calls them 'legibles'."
+ },
+ {
+ "id": "dl5",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template tiktok, one encode, tiktok 12/0/0 on my re-run, probe matches. Chinese report; centre-crop loss quantified and the synthetic nature of the source acknowledged in the Look line rather than claimed as good framing."
+ },
+ {
+ "id": "dl6",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template linkedin, one encode, linkedin 12/0/0 on re-run, 1080x1080 confirmed by my probe and the sheet. Japanese report; the 16:9 -> 1:1 side loss is stated."
+ },
+ {
+ "id": "dl7",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Audio-only route chosen correctly: loudness.py -I -16 --tp -1.5 then metadata.py --chapters. My re-probe shows 9.51s AAC mono 48k with 3 chapters, and check --platform podcast re-runs 6/0/0. 'Look: not needed (audio-only)' is right."
+ },
+ {
+ "id": "dl8",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "THE LABEL RULE LANDED. A single 'Failed:' saying the upload is impossible and naming the artefact prepared instead - no second Done:. render.py --template tiktok, one encode, tiktok 12/0/0 on my re-run, probe matches. Honest about the synthetic source giving nothing to judge framing against."
+ },
+ {
+ "id": "e01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "cut.py 0-6 lossless then render.py --template reels (one encode). reels 12/0/0 on my re-run; probe 6.10s 1080x1920 -14.0 LUFS. Sheet shows 'Hello world' / 'Second line' correctly wrapped inside the safe area, and the report explains why the untimed third cue is not visible in a 6s cut."
+ },
+ {
+ "id": "e11-ready",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Diagnosed with check --platform tiktok, fixed only what failed (loudness.py, video stream-copied: ZERO video encodes), re-checked. My re-run: tiktok 12/0/0. The sheet shows the source's letterbox bars and the report says so plainly instead of claiming a reframe. Ran the /root/.claude copy of the scripts."
+ },
+ {
+ "id": "j01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Improved over iteration 15, which hand-wrote a project: this run used render.py --template reels --write-project, edited in/out, and rendered once. reels 12/0/0 on my re-run; probe 6.037s 1080x1920 -14.0 LUFS. Japanese report; the trim-vs-speed decision is justified against the 1.5x threshold."
+ },
+ {
+ "id": "j06-report",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "x 12/0/0 on my re-run and report.html written; but this is the one run that still takes TWO encodes - export.py --preset x first, then rm and re-export with --normalize - and it is the one delivery run that never touches render.py --template. Same shape as iteration 15."
+ },
+ {
+ "id": "j08-project",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template reels --write-project, in/out edited to 2/8, one render from the project; reels 12/0/0 on my re-run, probe 6.037s 1080x1920. The Notes line tells the user exactly which field to change to re-cut, which is what the prompt asked for. Sheet is clean."
+ },
+ {
+ "id": "a01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py burn-in; the 3x2 sheet shows Arabic with correct joining forms and right-to-left order, no tofu. Re-probe matches. Arabic report, five labels, auto-timing of the untimed third cue disclosed with the actual window."
+ },
+ {
+ "id": "j03-lower",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third 0-4s; the 2s frame shows '山田 太郎 / プロデューサー' correctly rendered bottom-left. Re-probe matches; Japanese report; the two loudness WARNs are reported as unrequested judgement rows. Ran the /root/.claude copy of the scripts."
+ },
+ {
+ "id": "c01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py --lang zh burn-in; sheet shows all three Chinese cues clean and correctly placed, font selection (WenQuanYi Zen Hei) reported. Re-probe matches. Chinese report; no platform named so check.py correctly skipped and said so."
+ },
+ {
+ "id": "vi1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py burn-in; the 1s frame shows Vietnamese with every diacritic and tone mark correct on one line. Re-probe matches. Vietnamese report; auto-timing of the third cue disclosed and the .srt pointed at for the exact window."
+ },
+ {
+ "id": "x05-replace-audio",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "audio.py --replace, video stream-copied. My re-probe: 6.00s 640x360 h264 + AAC mono, matching the Done: line. 'Look: not needed (audio-only change)' is correct. Best detail: it noticed the mic track is only 4.516s and reported the ~1.48s silence pad rather than letting it pass silently."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/evals/results/iteration-16.json b/evals/results/iteration-16.json
new file mode 100644
index 0000000..eba65e0
--- /dev/null
+++ b/evals/results/iteration-16.json
@@ -0,0 +1,233 @@
+{
+ "iteration": 16,
+ "date": "2026-09-13",
+ "skill_version": "1.15.0",
+ "agent_model": "claude-sonnet (1 run per prompt, 8-16 in parallel, seven batches)",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 82,
+ "prompts": "evals/agent_prompts_24.json (71) + evals/agent_prompts_exec.json (11)",
+ "repeats": 1,
+ "harness_note": "Same boilerplate as iterations 14-15; the set grew by six 1.15.0 prompts (em1-em4 emoji captions/title cards, em1-em3 shipping a directory of code-point PNGs; sh1 Hindi lower-third, sh2 Thai lower-third plus Thai captions). Tool calls counted from tool_use entries in the agent transcripts. Focused independent grader on 28 runs (emoji, shaping, label-rule, delivery and picture/language runs; every PNG opened, stills extracted inside the emoji window; check.py re-run on every delivery output); language and labels on all 82.",
+ "purpose": "Measure what 1.15.0 changed: complex-script shaping through graphics.py (hi1 was garbled at 1.14.0), emoji people can see (--emoji-assets / overlay), the Failed: label rule on refuse-the-verb prompts (dl8, he2, k02), the caption-wrap orphan fix, and keep the 76-prompt set at iteration-15 levels.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "71-set": {
+ "routing_act_prompts": "100% over 49",
+ "mean_commands": "4.0",
+ "refusal_honesty": "22/22",
+ "language_by_prompt": "all 17 languages 100% (ja 13/13, zh 4/4, ko 2/2, es 3/3, pt 2/2, fr 2/2, de 2/2, ar 2/2, th 3/3, hi 3/3, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2)",
+ "report_format": "71/71",
+ "visual_check_when_picture_changed": "23/24 (e12-vfr conformed VFR to CFR with a re-encode and wrote no still)",
+ "audio_only_handled_as_audio": "4/4",
+ "emoji_honesty": "4/4 (em1-em3 used the shipped assets and reported mode png; em4 had no assets, shipped monochrome and said so)"
+ },
+ "exec-set": {
+ "routing_act_prompts": "100% over 10",
+ "mean_commands": "2.7",
+ "real_execution": "6/6",
+ "honest_failure_reporting": "5/5, false success 0",
+ "audio_only_handled_as_audio": "3/3"
+ },
+ "grader_changes_this_iteration": "em2 accepts overlay.py --image as a route (the prompt hands the agent a PNG, and SKILL.md routes 'put an image on the video' to overlay.py); em4 joined the picture-changed set; the colour-claim regex ignores the font name 'Noto Color Emoji' and negations ('no colour path', 'for colour emoji pass --emoji-assets') that runs paste from tool output (em4 was capped at 0.5 for an honest report); German stopwords extended after d01's honest German report was missed (the Opus grader confirms it is German). Re-graded after each change."
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-13-it16.json",
+ "judge": "claude-sonnet, one batch of 40 (38 + em-caption + sh-lower-third)",
+ "should_trigger": "23/23",
+ "should_not_trigger": "17/17"
+ },
+ "independent_grader": {
+ "file": "evals/results/iteration-16-independent-grades.json",
+ "model": "claude-opus, focused pass (28 runs fully: em1-em4, sh1-sh2, hi1, th1, he2, k02, dl1-dl8, e01, e11, j01, j06, j08, a01, j03, c01, vi1, x05; language and labels on all 82; every PNG opened; stills extracted inside the emoji window; check.py re-run on every delivery output)",
+ "summary": {
+ "runs": 82,
+ "runs_graded_fully": 28,
+ "routing": "28/28 - correct script for every request (caption.py for burn-in, caption.py --emoji-assets / overlay.py --image for emoji, graphics.py for lower-thirds, render.py --template for delivery, loudness.py+metadata.py for the podcast, audio.py --replace for the track swap, redact.py --help to prove the refusal). No raw ffmpeg or ffprobe invoked by any agent in any of the 82 run.md files.",
+ "honest": "28/28 - every written file re-probes exactly as reported, every platform claim re-runs clean, and the three refusals are accurate. No Done: is contradicted by disk.",
+ "language": "81/82 - every final report is in the prompt's language except dl4 (es), whose Done:/Steps:/Check: lines are English and only Look:/Notes: Spanish - the identical defect iteration 15 flagged on dl4. id1 (id) has a milder version (English Done:/Steps:, Indonesian body). d01 IS German: Check:/Look:/Notes: are German prose and only the Done: line is a language-neutral filename+spec string, so the regex grader's 'German not detected' is a grader artefact, not a run defect.",
+ "format": "82/82 carry the five labels exactly once each, each refusal under a single Failed:. The three label-rule prompts are all correct now: dl8 emits ONE 'Failed: cannot actually post/upload to TikTok ... I prepared the file instead', he2 and k02 emit a single Failed: (iteration 15 had dl8 as Done: and he2 as a Done:+Failed: double). Only cosmetic deviation: d02 titles its section '# Final report' instead of '## Final report'.",
+ "look_when_picture_changed": "25/25 - every run that changed the picture wrote and read a still or contact sheet; dl7, x05-replace-audio and the two refusals (he2, k02) correctly say 'Look: not needed'. e11-ready changed only audio and still produced a sheet, which is harmless.",
+ "visual_check_non_latin": "11/11 non-Latin text runs now render real, correctly shaped glyphs: Devanagari (sh1, hi1) with correct matra reordering, conjuncts, reph and word spaces; Thai (sh2 in both the lower third and the captions, th1) with correct vowel/tone stacking; Arabic (a01) joined and RTL; Chinese (c01, em3); Japanese (dl2, dl3, j03-lower, em2); Vietnamese (vi1). Iteration 15's single failure - hi1's garbled Devanagari - is fixed.",
+ "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (same as iteration 15).",
+ "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass; e11-ready took ZERO video encodes (loudness stream-copies video); j06-report again took TWO (export.py without --normalize, then rm and re-export). Identical to iteration 15.",
+ "emoji": "3 of 3 runs that were given PNG assets used them (em1 and em3 via caption.py --emoji-assets, em2 via overlay.py --image) and the emoji is visibly composited in colour inside its own time window in all three, verified on stills I extracted at 4.5s / 1.5s rather than frame 0. em4, which ships no assets, rendered monochrome and said so.",
+ "quality_mean": 4.68,
+ "quality_min": 3
+ }
+ },
+ "1.15.0 targets (first measurement)": {
+ "complex_script_shaping": "4/4 - sh1 and hi1 Devanagari correctly shaped through the libass path (conjunct, i-matra reordered, reph, word space; 1.14.0 produced 'प्रयाशर्म'), sh2 Thai correct in both the lower third and the captions, th1 correct. 11/11 non-Latin runs pass a visual read (iteration 15: 10/11).",
+ "emoji": "em1-em3 used the shipped PNGs (caption.py --emoji-assets twice, overlay.py --image once), the composite is visible in colour inside its own cue window on stills the grader extracted at 1.5 s / 4.5 s; em4 (no assets) rendered monochrome and reported it. Descriptive gap: em1 and em3 call plain coloured squares 'colour emoji'; only em2 said what it saw.",
+ "label_rule": "3/3 - dl8 one Failed: naming the prepared TikTok file (iteration 15: Done:), he2 one Failed: (iteration 15: Done:+Failed:), k02 one Failed:.",
+ "caption_wrap": "NOT fixed as a class - dl1 still 'A third line the tool' / 'times for me', dl4 still a lone 'subtítulos' and 'Una tercera línea con' / 'tiempos automáticos' (byte-for-byte iteration 15); dl3's lone '行' is gone but a lone '2' and a mid-word 'る行' break remain; only th1 is genuinely better. The one-character ban treats a symptom; a phrase-aware breaker is the fix (queued for 1.16.0)."
+ },
+ "delivery_prompts (third measurement)": {
+ "route": "12 of 13 delivery-shaped runs went through render.py --template (j01-reel now uses the template it wrote by hand in iteration 15); j06 used export.py directly",
+ "encodes": "12/13 one encode, e11 zero video encodes, j06 two (export.py without --normalize, then again with) - identical to iteration 15",
+ "platform_check": "13/13 pass check.py --platform X with 0 FAIL / 0 WARN on the grader's own re-run",
+ "tokens_dl": {
+ "runs": 8,
+ "mean": 77755,
+ "median": 76648,
+ "min": 72492,
+ "max": 86938
+ },
+ "tokens_dl_iteration_15": {
+ "runs": 8,
+ "mean": 80368,
+ "median": 76013,
+ "min": 73179,
+ "max": 90441
+ }
+ },
+ "tokens_per_run (subagent_tokens)": {
+ "all_82 at 1.15.0": {
+ "runs": 82,
+ "mean": 73710,
+ "median": 72508,
+ "min": 63975,
+ "max": 104136
+ },
+ "same_76 at 1.15.0": {
+ "runs": 76,
+ "mean": 73253,
+ "median": 72360,
+ "min": 63975,
+ "max": 104136
+ },
+ "same_76 at 1.14.0 (iteration 15)": {
+ "runs": 76,
+ "mean": 73356,
+ "median": 72043,
+ "min": 65066,
+ "max": 97099
+ },
+ "new_6 (em1-em4, sh1-sh2)": {
+ "runs": 6,
+ "mean": 79506,
+ "median": 79928,
+ "min": 75575,
+ "max": 81949
+ },
+ "tool_calls_mean (tool_use entries)": 7.8,
+ "doctor_runs": 4,
+ "json_brief_runs": 36,
+ "template_runs": 18,
+ "per_prompt": {
+ "em4": 75575,
+ "hi1": 71559,
+ "k02": 67890,
+ "he2": 69203,
+ "sh1": 79491,
+ "em1": 79913,
+ "em3": 81949,
+ "dl8": 76365,
+ "th1": 72629,
+ "a01": 79229,
+ "dl7": 72492,
+ "dl1": 77253,
+ "e11-ready": 75325,
+ "em2": 79943,
+ "sh2": 80167,
+ "c01": 72242,
+ "k01": 76091,
+ "dl2": 73658,
+ "dl6": 76932,
+ "j06-report": 74044,
+ "dl4": 83518,
+ "dl3": 86938,
+ "a02": 66919,
+ "c02": 67266,
+ "dl5": 74885,
+ "e01-reel": 78687,
+ "d01": 70132,
+ "j01-reel": 96475,
+ "d02": 70569,
+ "e05-lufs": 72479,
+ "e06-silence": 72141,
+ "e04-sync": 76739,
+ "e02-cut-lossless": 79655,
+ "e03-logo": 80066,
+ "el2": 66937,
+ "el1": 67566,
+ "e07-hdr": 76676,
+ "j08-project": 104136,
+ "f01-missing-input": 75227,
+ "f02-corrupt-input": 74519,
+ "e09-join": 77900,
+ "e08-surround": 81001,
+ "f03-ffmpeg-failure": 76739,
+ "e12-vfr": 72602,
+ "f05-unsupported": 69339,
+ "f04-beyond-duration": 72522,
+ "e10-highlights": 75904,
+ "fr1": 73753,
+ "fr2": 66827,
+ "hi2": 66791,
+ "it2": 66944,
+ "it1": 67513,
+ "id2": 68873,
+ "id1": 72759,
+ "he1": 79234,
+ "r01-face": 69367,
+ "r02-tts": 67698,
+ "j07-probe": 70038,
+ "j04-bgm": 74619,
+ "j05-ma": 71043,
+ "j02-60s": 73385,
+ "j03-lower": 75738,
+ "p02": 72493,
+ "p01": 72911,
+ "r03-download": 63975,
+ "r05-no-raw-ffmpeg": 67262,
+ "r04-translate": 68865,
+ "ru1": 70809,
+ "th2": 68086,
+ "s01": 69286,
+ "s02": 70875,
+ "ru2": 72015,
+ "tr1": 70409,
+ "vi2": 67509,
+ "tr2": 68729,
+ "x03-audio-trim": 68298,
+ "x01-lufs-wav": 69900,
+ "x02-wav-to-mp3": 71539,
+ "vi1": 71980,
+ "x06-reverse": 69961,
+ "x05-replace-audio": 70564,
+ "x04-transcode": 74705
+ }
+ },
+ "comparison_with_iteration_15": {
+ "routing": "100% -> 100%",
+ "refusal_honesty": "22/22 -> 22/22",
+ "report_format": "76/76 -> 82/82 (Opus: 74/76 -> 82/82, both label defects closed)",
+ "language": "Opus 75/76 -> 81/82 (dl4 repeats its English label lines; id1 milder)",
+ "non_latin_glyphs": "10/11 -> 11/11",
+ "delivery": "12/13 one encode and 13/13 platform pass, unchanged",
+ "quality_mean": "4.58 -> 4.68",
+ "tokens_mean_same_76": "73356 -> 73253 (flat)"
+ },
+ "findings": [
+ "Three of the four things 1.15.0 set out to fix are fixed. Complex-script shaping is the standout: sh1, hi1 and sh2 render Devanagari and Thai correctly through the libass path, where 1.14.0's hi1 produced 'प्रयाशर्म' for 'प्रिया शर्मा'. All 11 non-Latin runs now pass a visual read.",
+ "The emoji feature works end to end. Every run given PNG assets used them (--emoji-assets on caption.py, --image on overlay.py), and I confirmed the composite is visible in colour inside its own cue window, not just at frame 0. em4, which was given no assets, rendered monochrome and reported that plainly instead of claiming colour.",
+ "The label rule landed on all three refuse-the-verb prompts: dl8, he2 and k02 each carry exactly one Failed: with the artefact described. This closes both iteration 15 defects (dl8's Done: and he2's double label) with no over-correction.",
+ "The caption splitter is the defect that did NOT get fixed. dl1 still splits 'A third line the tool' / 'times for me' and dl4 still produces a lone 'subtítulos' plus 'Una tercera línea con' / 'tiempos automáticos' - byte-for-byte the iteration 15 failures. dl3's one-character '行' orphan is gone but a lone '2' line and a mid-word 'る行' remain; only th1 is genuinely better. The one-character ban treats a symptom; the splitter still breaks phrases and words at arbitrary points.",
+ "dl4 repeats iteration 15's language failure exactly: a Spanish prompt answered with English Done:/Steps:/Check: and Spanish Look:/Notes:. id1 shows a milder version. Nothing in the skill tells the agent the LABEL LINES are part of the user's language, only the report.",
+ "d01's report IS German - Check:, Look: and Notes: are German prose and only the Done: line is a language-neutral 'filename - 12.01 s, 1280x720, ...' string. The regex language grader flagging it is a grader artefact; the detector should ignore the Done: spec line or look at the whole report.",
+ "Delivery discipline is stable at a high level: 12 of 13 runs used render.py --template (j01-reel converted from iteration 15's hand-written project), 12 of 13 took one encode, and all 13 pass their platform check at 0 FAIL and 0 WARN on my re-run. j06-report remains the single holdout on both counts - it goes straight to export.py and pays for it with a second encode after forgetting --normalize."
+ ],
+ "harness_or_model_issues": [
+ "Only rep1 exists, so per-run variance cannot be separated from real improvement between iterations 15 and 16.",
+ "Eleven runs (a02, d01, dl2, e11-ready, el1, fr1, id1, j03-lower, j05-ma, p01, tr2) executed /root/.claude/skills/ffmpeg-skill/scripts instead of the /home/user/ffmpeg-skill/scripts path their prompt named. Three of them are graded in full, so those grades are not strictly against the repo copy under test.",
+ "em2's assets are deliberately plain coloured squares, and em1's and em3's are too - but only em2's prompt makes the mismatch conspicuous enough that the agent commented. If the intent is to test 'say what you saw', em1 and em3 need the same framing or the assets need to be recognisable emoji.",
+ "d02 writes '# Final report' instead of '## Final report'. Every label is present and correct, but a strict section-header grader would miss the report entirely.",
+ "The caption sheets' tile timestamps do not always line up with the cue in view (em3/em4 show cue 2 text under a 00:00:01.999 label). I verified the actual frames with my own look.py extractions and the burn-in timing is correct, so this is a contact-sheet labelling artefact worth a look in look.py.",
+ "/root/.claude/skills/ffmpeg-skill is a symlink to /home/user/ffmpeg-skill on this box, so the eleven runs that executed that path still ran the 1.15.0 tree under test."
+ ],
+ "follow_ups": [
+ "1.16.0: phrase-aware caption breaking (never split inside a word or between a determiner and its noun; balance lines by width) - dl1/dl3/dl4 evidence in iteration-16-independent-grades.json.",
+ "SKILL.md: the label lines (Done:/Steps:/Check:) are part of the user's-language report (dl4 twice, id1 once).",
+ "Evals: em1/em3 assets should be recognisable emoji or the prompt should ask the agent to say what it saw; d02's '# Final report' heading; look.py sheet tile timestamps vs the cue shown.",
+ "j06-report: 'named platform means --normalize in the same export' sentence for SKILL.md when the byte budget allows."
+ ]
+}
diff --git a/evals/trigger/results-2026-09-13-it16.json b/evals/trigger/results-2026-09-13-it16.json
new file mode 100644
index 0000000..a518838
--- /dev/null
+++ b/evals/trigger/results-2026-09-13-it16.json
@@ -0,0 +1,49 @@
+{
+ "date": "2026-09-13",
+ "skill_version": "1.15.0",
+ "iteration": 16,
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 40; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {"id":"t01","lang":"en","prompt":"Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trims a .MOV video file to a duration"},
+ {"id":"t02","lang":"ja","prompt":"この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical + add captions for Reels"},
+ {"id":"t03","lang":"en","prompt":"the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio level fix on an mp4 file"},
+ {"id":"t04","lang":"ja","prompt":"Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from a recording"},
+ {"id":"t05","lang":"en","prompt":"I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"External-mic sync of audio/video files"},
+ {"id":"t06","lang":"ja","prompt":"iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"HDR-to-SDR conversion issue"},
+ {"id":"t07","lang":"en","prompt":"put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Logo overlay + fade on videos"},
+ {"id":"t08","lang":"ja","prompt":"podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS loudness normalisation and transcode"},
+ {"id":"t09","lang":"en","prompt":"Does this file meet TikTok's specs? final_v3.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Platform compliance check on a video file"},
+ {"id":"t10","lang":"ja","prompt":"2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Multicam highlight reel editing"},
+ {"id":"n01","lang":"en","prompt":"Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still-image resize/transparency, not video/audio"},
+ {"id":"n02","lang":"ja","prompt":"Windows に ffmpeg をインストールする方法を教えて","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Installation how-to question, not an edit request"},
+ {"id":"n03","lang":"en","prompt":"Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger":false,"chosen":"web-research","model":"sonnet","correct":true,"reason":"Downloading online content, no local file to edit"},
+ {"id":"n04","lang":"ja","prompt":"新商品紹介動画の台本(60 秒)を書いて","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script/text, not editing a media file"},
+ {"id":"n05","lang":"en","prompt":"What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Premiere Pro UI question, unrelated to ffmpeg editing"},
+ {"id":"n06","lang":"en","prompt":"Design a YouTube thumbnail in Figma for my cooking channel","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Figma design task, no matching skill in catalog"},
+ {"id":"n07","lang":"en","prompt":"このCSVをJSONに変換して: sales.csv","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Data format conversion, not audio/video"},
+ {"id":"n08","lang":"en","prompt":"Explain the difference between H.264 and H.265 in simple terms","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Conceptual explanation, no file to process"},
+ {"id":"n09","lang":"en","prompt":"Generate a 10-second video of a cat surfing with AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing an existing file"},
+ {"id":"n10","lang":"ja","prompt":"会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger":false,"chosen":"meeting-notes","model":"sonnet","correct":true,"reason":"Transcription/minutes from a meeting recording"},
+ {"id":"t11","lang":"en","prompt":"Convert this WAV to MP3: interview_raw.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio transcoding"},
+ {"id":"t12","lang":"en","prompt":"Remove the silence from this recording, it's a 40 minute M4A","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from audio file"},
+ {"id":"t13","lang":"zh","prompt":"这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim video + add subtitles"},
+ {"id":"t14","lang":"ko","prompt":"인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Reframe to vertical for Reels"},
+ {"id":"t15","lang":"es","prompt":"El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS normalisation of audio"},
+ {"id":"t16","lang":"ar","prompt":"هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from podcast audio"},
+ {"id":"n11","lang":"pt","prompt":"Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, not editing media"},
+ {"id":"n12","lang":"fr","prompt":"Comment installer ffmpeg sur macOS avec Homebrew ?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Installation how-to question"},
+ {"id":"n13","lang":"de","prompt":"Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing"},
+ {"id":"t17","lang":"th","prompt":"ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim video + add subtitles"},
+ {"id":"t18","lang":"hi","prompt":"interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS loudness fix on video's audio"},
+ {"id":"t19","lang":"he","prompt":"אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe + captions for Reels"},
+ {"id":"t20","lang":"ru","prompt":"Убери, пожалуйста, тишину из записи podcast.m4a","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from a recording"},
+ {"id":"t21","lang":"vi","prompt":"Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim video + add subtitles"},
+ {"id":"t22","lang":"tr","prompt":"roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe for Reels"},
+ {"id":"n14","lang":"el","prompt":"Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Installation how-to question"},
+ {"id":"n15","lang":"id","prompt":"Buatkan video 10 detik tentang kucing yang berselancar dengan AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing"},
+ {"id":"n16","lang":"it","prompt":"Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, not editing media"},
+ {"id":"t23","lang":"en","prompt":"can you burn the subtitles into my clip and keep the emoji in the captions in colour? interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Burning subtitles/captions into a video file"},
+ {"id":"n17","lang":"en","prompt":"make me an emoji sticker pack as PNGs, 20 stickers with transparent backgrounds","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still-image sticker pack, explicitly image-tools territory"}
+ ],
+ "summary": {"total": 40, "correct": 40, "should_trigger_correct": "23/23", "should_not_correct": "17/17"}
+}