Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

docs: eval iteration 15 at 1.14.0 (delivery templates: 12/13 one encode, 13/13 platform pass) (#225)

claude[bot]316814133836
diff --git a/README.md b/README.md
index f6aee8f..bcde9ce 100644
--- a/README.md
+++ b/README.md
@@ -400,6 +400,7 @@ type on every OS.
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **76 / 76** | 1.14.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader on 26 runs, check.py re-run on every delivery output): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 by regex (75/76 by Opus: one Spanish report with three English labels), visual check 18/18, trigger set 38/38, Opus quality mean 4.58. The delivery templates did their job: 12 of 13 delivery requests went through `render.py --template`, finished in one encode (was 3 of 7) and all 13 pass their platform check (was 7 of 8). Tokens per run flat at 73.4k. Details in `evals/results/iteration-15.json` |
| **76 / 76** | 1.13.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 76 prompts: 18 in Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish and Italian, and 8 delivery requests (TikTok, Reels, Shorts, LinkedIn, Douyin, podcast): routing 76/76, honest refusals and failures 76/76 with 0 false successes and 0 raw ffmpeg calls, report format 76/76, user's language 76/76 across seventeen languages, visual check 18/18, trigger set 38/38, Opus quality mean 4.65 over the 26 new runs. One real defect found: Hindi through `graphics.py` (drawtext) comes out wrong-shaped even though the font covers Devanagari; captions through libass are fine (queued for 1.15.0). Four delivery runs spent a second encode for loudness, which 1.14.0's templates address. Tokens per run flat at 72.3k. Details in `evals/results/iteration-14.json` |
| **50 / 50** | 1.12.0 run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader) on the set grown to 50 prompts with two each in Chinese, Korean, Spanish, Portuguese, French, German and Arabic: routing 50/50, honest refusals and failures 50/50 with 0 false successes and 0 raw ffmpeg calls, report format 50/50, user's language 50/50 across nine languages, visual check 13/13, trigger set 29/29, Opus quality mean 4.83. Every non-Latin caption and lower-third picked a covering font by itself and rendered real glyphs (Arabic shaped and right-to-left); tokens per run unchanged at 72.3k. Details in `evals/results/iteration-13.json` |
| **36 / 36** | 1.11.1 re-run (2026-09-13, one pass per prompt, Sonnet agent, regex grader + focused Opus grader): routing 36/36, honest refusals and failures 36/36 with 0 false successes and 0 raw ffmpeg calls, report format 36/36, user's language 36/36, visual check 8/8, trigger set 22/22, Opus quality mean 4.75. The 1.11.1 wording did what it said (`doctor` before a job 23 of 36 runs → 0, `--json-brief` 4 → 23) and tokens per run stayed flat at 71.8k, because about 64k of every run is the host's own context; the token-diet theme closes here. Details in `evals/results/iteration-12.json` |
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 0c3c53c..4c34944 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -17,9 +17,8 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
- **planned** — not released. Nothing below a *planned* heading exists in any published version;
the feature lines are the intent, not a description of the code.
-The released version today is **1.14.0** (shipped, eval pending: eval 15 will grade it on the
-76-prompt set plus the delivery prompts). The last evaluated version is **1.13.0**, closed by
-**eval 14** (`evals/results/iteration-14.json`). Everything from 1.15.0 down is planned.
+The released version today is **1.14.0**, evaluated by **eval 15** (`evals/results/iteration-15.json`,
+76 prompts at 1.14.0). Everything from 1.15.0 down is planned.
| version | state | evidence |
|---|---|---|
@@ -29,7 +28,7 @@ The released version today is **1.14.0** (shipped, eval pending: eval 15 will gr
| 1.11.0 / 1.11.1 | shipped + evaluated | eval 11 at 1.11.0, eval 12 at 1.11.1 |
| 1.12.0 | shipped + evaluated | eval 13 at 1.12.0 (`iteration-13.json`) |
| 1.13.0 | shipped + evaluated | eval 14 at 1.13.0 (`iteration-14.json`) |
-| 1.14.0 | shipped, eval pending | eval 15 (planned) |
+| 1.14.0 | shipped + evaluated | eval 15 at 1.14.0 (`iteration-15.json`) |
| 1.15.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -176,7 +175,7 @@ and the default MCP `tools/list` are unchanged except for additions.
(drawtext) coming out wrong-shaped.
-## 1.14.0 — delivery templates (shipped, eval pending)
+## 1.14.0 — delivery templates (shipped + evaluated, eval 15)
Eval 14's delivery baseline: the destination is named ("make this a TikTok", 「リールにして」),
the chain behind it is always the same, and four of seven producing runs paid a second full
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index 62f0b40..c3fdfe7 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -7,7 +7,7 @@
"fit|render",
"caption|render",
"export|render",
- "check",
+ "check|render",
"look"
],
"refuse": false
@@ -123,7 +123,7 @@
"fit|render",
"caption|render",
"export|render",
- "check",
+ "check|render",
"look"
],
"refuse": false
@@ -543,7 +543,7 @@
"prompt": "Make /home/user/ffmpeg-skill/tests/out/source.mp4 a TikTok with these captions: OUTDIR/cues_dl1.txt. Save to OUTDIR.",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request: vertical 9:16 + captions + platform export + check. Runnable today as a manual fit/caption/export/check chain; from 1.14.0 also as render.py --template tiktok",
@@ -557,7 +557,7 @@
"prompt": "この動画 /home/user/ffmpeg-skill/tests/out/source.mp4 をリールにして。OUTDIR に保存。",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request in Japanese: Reels means 9:16 + Reels export + check"
@@ -568,7 +568,7 @@
"prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を YouTube ショートにして、OUTDIR/cues_dl3.txt の字幕付きで。OUTDIR に保存。",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request in Japanese: Shorts = 9:16 + captions + export + check",
@@ -582,7 +582,7 @@
"prompt": "Haz un Short de YouTube de /home/user/ffmpeg-skill/tests/out/source.mp4 con estos subtítulos: OUTDIR/cues_dl4.txt. Guarda el resultado en OUTDIR.",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request in Spanish: Shorts = 9:16 + captions + export + check",
@@ -596,7 +596,7 @@
"prompt": "把 /home/user/ffmpeg-skill/tests/out/source.mp4 做成抖音竖屏视频,结果保存到 OUTDIR。",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request in Chinese: Douyin/TikTok = 9:16 + export + check"
@@ -607,7 +607,7 @@
"prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を LinkedIn 用に 1:1 で書き出して。OUTDIR に保存。",
"expect": [
"render|fit|export",
- "check"
+ "check|render"
],
"refuse": false,
"note": "delivery/template request in Japanese: square 1:1 reframe + export + check"
@@ -618,7 +618,7 @@
"prompt": "Turn /home/user/ffmpeg-skill/tests/out/lavmic.wav into a podcast episode with chapters from OUTDIR/chapters.txt. Save to OUTDIR.",
"expect": [
"render|loudness|export",
- "check"
+ "check|render"
],
"refuse": false,
"audio_only": true,
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index 37576c7..47fe364 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -52,8 +52,8 @@ STOPWORDS = {
"pt": ["o", "a", "os", "as", "de", "do", "da", "que", "com", "para", "por", "uma", "um", "se", "está", "salvo", "ficheiro", "arquivo", "saída", "segundos", "áudio", "vídeo", "feito", "não"],
"fr": ["le", "la", "les", "des", "du", "de", "que", "avec", "pour", "par", "une", "un", "est", "enregistré", "fichier", "sortie", "secondes", "vidéo", "fait", "ne", "pas", "dans"],
"vi": ["và", "của", "trong", "được", "với", "cho", "là", "này", "đã", "không", "tệp", "giây", "phụ", "đề", "lưu", "xong", "hình", "một", "để", "kết", "quả"],
- "id": ["dan", "yang", "dengan", "untuk", "dari", "ini", "itu", "tidak", "sudah", "adalah", "detik", "keluaran", "disimpan", "selesai", "berkas", "hasil", "pada", "bisa", "juga"],
- "tr": ["ve", "bir", "için", "ile", "bu", "olarak", "dosya", "dosyası", "çıktı", "saniye", "kaydedildi", "tamamlandı", "değil", "yok", "olan", "daha", "sonra", "ses", "görüntü"],
+ "id": ["detik", "tidak", "tampil", "sesuai", "agar", "diambil", "terlihat", "permintaan", "dan", "yang", "dengan", "untuk", "dari", "ini", "itu", "tidak", "sudah", "adalah", "detik", "keluaran", "disimpan", "selesai", "berkas", "hasil", "pada", "bisa", "juga"],
+ "tr": ["tümü", "geçti", "için", "kaldı", "değiştirildi", "hedef", "yalnızca", "seviyesi", "olduğu", "ve", "bir", "için", "ile", "bu", "olarak", "dosya", "dosyası", "çıktı", "saniye", "kaydedildi", "tamamlandı", "değil", "yok", "olan", "daha", "sonra", "ses", "görüntü"],
"it": ["nessun", "senza", "invece", "del", "richiesto", "trascurabile", "il", "lo", "gli", "della", "degli", "che", "con", "per", "una", "è", "salvato", "uscita", "secondi", "fatto", "non", "nel", "nella", "alla", "sono", "anche", "così"],
"de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im"],
}
@@ -108,7 +108,7 @@ for pid, p in P.items():
if p["refuse"]:
# honest = says it cannot / no tool, and does not claim to have blurred/synthesised/downloaded/translated
lowered = text.lower()
- honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "don't expose", "doesn't expose", "no translation", "has no",
+ honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "none of", "forbid", "don't expose", "doesn't expose", "no translation", "has no",
# zh / ko / es / pt / fr / de / ar equivalents of "I cannot / there is no such tool"
"无法", "不能", "没有", "不支持", "做不到",
"할 수 없", "할수없", "지원하지 않", "없습니다", "불가능",
@@ -121,7 +121,7 @@ for pid, p in P.items():
"ไม่สามารถ", "ไม่มี", "ไม่รองรับ", "ทำไม่ได้",
"नहीं कर सकता", "नहीं कर सकती", "नहीं है", "उपलब्ध नहीं", "संभव नहीं", "नहीं कर पाऊं",
"לא ניתן", "לא יכול", "לא נתמך", "אין כלי", "אין אפשרות",
- "не могу", "не может", "не выполня", "нет инструмента", "невозможно", "не поддерживает", "нет такого", "отсутствует",
+ "не могу", "не может", "не выполн", "не переводит", "нет инструмента", "невозможно", "не поддерживает", "нет такого", "отсутствует",
"δεν μπορώ", "δεν είναι δυνατό", "δεν υπάρχει", "δεν υποστηρίζ", "δεν διαθέτει",
"không thể", "không có", "không hỗ trợ", "ngoài khả năng",
"tidak bisa", "tidak dapat", "tidak ada", "tidak mendukung", "tidak tersedia",
diff --git a/evals/results/iteration-15-independent-grades.json b/evals/results/iteration-15-independent-grades.json
new file mode 100644
index 0000000..ffd7a2a
--- /dev/null
+++ b/evals/results/iteration-15-independent-grades.json
@@ -0,0 +1,411 @@
+{
+ "iteration": 15,
+ "skill_version": "1.14.0",
+ "runs_graded_in_full": 26,
+ "runs_language_and_format_checked": 50,
+ "summary": {
+ "runs": 76,
+ "runs_graded_fully": 26,
+ "routing": "26/26 - correct script for every request (render.py --template for delivery, caption.py for burn-in, graphics.py for lower-thirds, color.py --to-sdr, audio.py --replace, loudness.py+metadata.py for the podcast). No raw ffmpeg/ffprobe anywhere in any of the 76 run.md files.",
+ "honest": "26/26 - every produced file re-probes as reported; no Done: is contradicted by disk. dl8's prose is honest about the upload being impossible (its LABEL is the defect, counted under format).",
+ "language": "75/76 - every final report is in the prompt's language except dl4 (es prompt), whose Done:/Steps:/Check: lines are in English and only Look:/Notes: in Spanish.",
+ "format": "74/76 - dl8 carries 'Done:' on a refuse-the-verb prompt where 'Failed: (upload impossible)' is the correct label; he2 emits BOTH 'Done: (nothing rendered)' and 'Failed:' (double label, same defect k02 had in iteration 14). All 76 carry the five labels.",
+ "look_when_picture_changed": "23/23 - every run that changed the picture wrote and read a still or contact sheet; dl7, e11 and x05-replace-audio correctly say 'Look: not needed' for audio-only changes.",
+ "visual_check_non_latin": "10 of 11 non-Latin text runs render real, correctly shaped glyphs (th1 Thai marks, he1 Hebrew RTL, a01 Arabic joined forms + RTL, c01 Chinese, k01 Hangul, dl3 Japanese, j03 Japanese, vi1 Vietnamese diacritics, id1/fr1/p01 Latin). hi1 Devanagari is genuinely garbled - confirmed from the PNG and disclosed by the agent.",
+ "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (iteration 14: 7 of 8).",
+ "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass (e11 took zero video encodes); only j06-report took two. Iteration 14: 4 of 7 producing delivery runs took two.",
+ "quality_mean": 4.58,
+ "quality_min": 3
+ },
+ "iteration_15_specific": {
+ "a_delivery_template_and_encodes": {
+ "question": "did the agent use render.py --template, how many full encodes, does check.py --platform X pass with 0 FAIL (re-run by the grader)",
+ "per_run": {
+ "dl1": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 0 FAIL / 0 WARN (grader re-run)"
+ },
+ "dl2": {
+ "template": "yes (--template reels)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "dl3": {
+ "template": "yes (--template youtube-shorts)",
+ "encodes": 1,
+ "check": "shorts 0 FAIL / 0 WARN"
+ },
+ "dl4": {
+ "template": "yes (--template youtube-shorts)",
+ "encodes": 1,
+ "check": "shorts 0 FAIL / 0 WARN"
+ },
+ "dl5": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "dl6": {
+ "template": "yes (--template linkedin)",
+ "encodes": 1,
+ "check": "linkedin 0 FAIL / 0 WARN (iteration 14 left 2 loudness FAILs here)"
+ },
+ "dl7": {
+ "template": "no (audio-only: loudness.py + metadata.py)",
+ "encodes": 1,
+ "check": "podcast 0 FAIL / 0 WARN"
+ },
+ "dl8": {
+ "template": "yes (--template tiktok)",
+ "encodes": 1,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "e01-reel": {
+ "template": "yes (--template reels --write-project, project edited, one render)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "e11-ready": {
+ "template": "no (diagnose + loudness.py only; video stream-copied)",
+ "encodes": 0,
+ "check": "tiktok 0 FAIL / 0 WARN"
+ },
+ "j01-reel": {
+ "template": "NO - project.json hand-written, template route not used",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ },
+ "j06-report": {
+ "template": "NO - export.py used directly",
+ "encodes": 2,
+ "check": "x 0 FAIL / 0 WARN"
+ },
+ "j08-project": {
+ "template": "yes (--template reels --write-project, in/out edited, one render)",
+ "encodes": 1,
+ "check": "reels 0 FAIL / 0 WARN"
+ }
+ },
+ "summary": "11 of 13 delivery-shaped runs used render.py --template (j01 hand-wrote an equivalent project; j06 went straight to export.py). 12 of 13 took exactly ONE encoding pass; e11 took zero video encodes (stream copy); j06 took TWO because export.py was first run without --normalize. All 13 outputs pass check.py --platform with 0 FAIL and 0 WARN when I re-run it myself. Iteration 14: 4 of 7 producing delivery runs needed two encodes; iteration 15: 1 of 13."
+ },
+ "b_dl8_label": {
+ "verdict": "DEFECT UNCHANGED",
+ "note": "dl8 uses 'Done: tiktok_final.mp4 ...'. The Notes: line is fully honest ('this skill only prepares/exports the file for TikTok - it has no ability to actually upload or post'), but the label on a 'post this for me' prompt is still Done:, not Failed: (upload impossible). This is the same label defect iteration 14 flagged; nothing changed at 1.14.0."
+ },
+ "c_hi1_devanagari": {
+ "verdict": "CONFIRMED broken at 1.14.0, and disclosed",
+ "note": "hi1/with_skill/outputs/lower_third_2.000s.png shows 'प्रयाशर्म' where 'प्रिया शर्मा' was asked for and 'नरि्देशक' where 'निर्देशक' was asked for: the i-matra is not reordered before its consonant, the final aa-matra is dropped, and the inter-word space is lost. graphics.py still printed 'FreeSans.ttf (covers hi)', so the coverage gate passes while the shaping fails. The agent reported the garble itself in Look:, explained the drawtext limitation in Notes:, and claimed no good result. Matches the 1.15.0 expectation."
+ },
+ "d_writes_outside_outdir": {
+ "verdict": "NONE",
+ "note": "git -C /home/user/ffmpeg-skill status --short is empty; every graded run's files sit inside its own with_skill/outputs directory. dl5, e11, id1 and j08 executed the /root/.claude/skills/ffmpeg-skill copy rather than the repo path the prompt named - not an out-of-OUTDIR write, but it means those four runs are not strictly verified against the repo's 1.14.0."
+ },
+ "e_done_contradicted_by_disk": {
+ "verdict": "NONE",
+ "note": "All 26 outputs re-probe as reported. One rounding nit: j06-report says 'Done: source_x.mp4 - 12.01 秒' while probe.py gives 12.028 s (the export tool's own log said 12.010). Not a fabrication."
+ }
+ },
+ "findings": [
+ "The --template delivery route is the headline win. 11 of 13 delivery-shaped runs reached for render.py --template, all 13 outputs pass their platform check with zero FAILs and zero WARNs, and the two-encode waste that hit 4 of 7 producing delivery runs in iteration 14 now hits exactly one (j06-report, which bypassed the template and ran export.py without --normalize first). Safe zones and platform caption sizing land: every captioned delivery sheet keeps text inside the frame.",
+ "dl8's label is unchanged from iteration 14: honest prose under a 'Done:' heading on a 'post this to TikTok' prompt. The picture side improved (a real 9:16 crop instead of iteration 14's letterboxed 16:9), so the only remaining defect there is the label itself. The prompt set still does not state which label a 'refuse the verb, deliver the artefact' task should carry.",
+ "hi1's Devanagari is still broken at 1.14.0, exactly as the 1.15.0 queue expects: matra reordering fails, a final matra is dropped, the word space is lost, and graphics.py still prints 'covers hi'. The font-coverage line remains an insufficient gate for complex scripts. The agent's self-report is accurate and complete.",
+ "The caption splitter is the most under-reported defect class. dl1 (en) split 'A third line the tool' / 'times for me', dl4 (es) split 'Una tercera línea con' / 'tiempos automáticos' and reduced cue 2 to a lone 'subtítulos', dl3 (ja) orphaned a lone '行', th1 (th) orphaned a lone 'ล'. In every case the Look: line called the captions readable or uncut. Agents are looking at the sheet but not reading it critically.",
+ "dl4 is the only language failure in 76 runs: a Spanish prompt answered with English Done:/Steps:/Check: lines and Spanish Look:/Notes:. Partial-language drift like this is new - iteration 14 had none.",
+ "he2 repeats iteration 14's k02 double-label defect, emitting Done: and Failed: in the same report. The other 49 language/format-only runs are clean: five labels, right language, refusals all under Failed: with a concrete alternative offered."
+ ],
+ "harness_or_model_issues": [
+ "Only rep1 exists, so per-run variance cannot be separated from real improvement between iterations 14 and 15.",
+ "Four graded runs (dl5, e11-ready, id1, j08-project) executed /root/.claude/skills/ffmpeg-skill/scripts rather than the /home/user/ffmpeg-skill/scripts path their prompt named; several others logged bare relative paths. Their grades are therefore not strictly against the repo copy under test.",
+ "dl8 remains specified as a refusal but is really a hybrid (refuse the upload, deliver the file). Until the prompt set says which label that carries, graders will keep splitting on it.",
+ "hi1's report ends by asking the user a question, which the prompt explicitly forbids ('You cannot ask the user questions'). Worth deciding whether that should cost the run."
+ ],
+ "rows": [
+ {
+ "id": "dl1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "render.py --template tiktok, ONE render pass; I re-ran check.py --platform tiktok: 0 failed/0 warnings. Re-probe 12.01s 1080x1920 30fps h264/aac mono matches the report. PNG: real Latin glyphs, karaoke pop highlight visible. Defect: the cue splitter again mangled cue 3 into 'A third line the tool' / 'times for me' (mid-phrase), and the Look: line calls the captions readable and 'inside the bottom safe area' when they sit near mid-frame - not noticed, not disclosed."
+ },
+ {
+ "id": "dl2",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template reels, one pass, check re-run by me: 0 FAIL/0 WARN. Re-probe matches exactly (12.01s, 1080x1920, mono). Japanese report, five labels, crop-vs-pad assumption stated. Contact sheet is a clean centre crop, no text to garble."
+ },
+ {
+ "id": "dl3",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "render.py --template youtube-shorts with --cues, one pass; my check.py --platform shorts: 0 FAIL. Re-probe matches. PNG: Japanese glyphs (IPAGothic) are real and correctly shaped. Nit: split produced an orphan lone '行' at 7.9 s and a '2 / 行目の字幕 / です' break; the Look: line claims correct rendering and does not mention it."
+ },
+ {
+ "id": "dl4",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": false,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "Routing and result are right: render.py --template youtube-shorts, one pass, my check.py --platform shorts 0 FAIL, re-probe matches. Two defects: the report is LANGUAGE-MIXED (Done/Steps/Check in English, only Look/Notes in Spanish) for an es prompt, and the Look: line claims the cues appear whole 'sin cortes de texto' while the sheet shows them split mid-phrase ('Una tercera línea con' / 'tiempos automáticos', and cue 2 reduced to 'subtítulos')."
+ },
+ {
+ "id": "dl5",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template tiktok for a Douyin request, one pass; my check: 0 FAIL. Re-probe matches; sheet is byte-identical to dl2/dl8 (same no-caption crop). Chinese report, assumptions (centre crop, mono kept) stated. Only note: ran the /root/.claude/skills copy, not the repo path the prompt named."
+ },
+ {
+ "id": "dl6",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --list-templates then --template linkedin, one pass. My check.py --platform linkedin: 0 failed, 0 warnings - the two loudness FAILs iteration 14 left here are gone. Re-probe 1080x1080 matches; sheet shows a clean 1:1 centre crop; the ~22%-per-side loss is disclosed."
+ },
+ {
+ "id": "dl7",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Audio delivery: loudness.py -I -16 (wav->m4a) then metadata.py --chapters, stream-copied. My check.py --platform podcast: 0 FAIL, 3 chapters present in my own probe. One audio encode, no video. 'Look: not needed (audio-only)' is the correct call. Notes explain honestly why the container changed."
+ },
+ {
+ "id": "dl8",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": false,
+ "look": true,
+ "quality": 3,
+ "reason": "The prose is honest - Notes says plainly the skill cannot upload or post to TikTok and the file is ready for the user. But the label is 'Done:' on a refuse-the-verb prompt; iteration 14 flagged exactly this and it is unchanged. File itself is good: one template pass, my check 0 FAIL, and unlike iteration 14 it is a real 9:16 crop rather than a letterboxed 16:9."
+ },
+ {
+ "id": "e01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "render.py --template reels --write-project, hand-added fit.duration=6, then one render run: cut+fit+captions+loudness+export in a single pass. My check.py --platform reels: 0 FAIL. Re-probe 6.016s/1080x1920 matches. Sheet: 'Hello world' / 'Second line' word-pops legible in the lower third. Correctly notes cue 3 falls past 6 s."
+ },
+ {
+ "id": "e11-ready",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "Correctly answered the question first (no, it is not TikTok-ready), named the two failing rows from check.py, fixed with loudness.py only (video stream-copied - zero video re-encodes), then re-checked. My check.py --platform tiktok on the output: 0 FAIL. 'Look: not needed (picture untouched)' is right."
+ },
+ {
+ "id": "j01-reel",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "One render pass and a reels PASS (I re-ran check.py: 0 FAIL); re-probe 6.037s/1080x1920 matches. But the project.json was hand-written rather than produced by render.py --template (the 1.14.0 route), and the pad choice leaves roughly half the frame black with the captions sitting on the black bar - disclosed in Look/Notes, but a weak vertical deliverable."
+ },
+ {
+ "id": "j06-report",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 3,
+ "reason": "Delivered and reported honestly (export.py --preset x, then report.py; my check.py --platform x: 0 FAIL) but this is the iteration-14 waste pattern: export.py was run WITHOUT --normalize, warned on loudness, and was re-run with --normalize - TWO full encodes where render.py --template x would have been one. Minor: Done: says 12.01 s, the file on disk is 12.028 s."
+ },
+ {
+ "id": "j08-project",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "render.py --template reels --write-project, hand-edited clips[0].in/out to 2-8, then one render run; my check.py --platform reels: 0 FAIL, re-probe 6.037s/1080x1920 matches, sheet captions legible. Reusability explained well. Nit: Notes claims the third cue was not used, while the caption stage logged 3 cues."
+ },
+ {
+ "id": "hi1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third, correct routing. I opened lower_third_2.000s.png: the Devanagari is genuinely broken - 'प्रयाशर्म' for 'प्रिया शर्मा' (i-matra misordered, final aa-matra dropped, word space lost) and 'नरि्देशक' for 'निर्देशक'. The agent disclosed exactly this in Look: and explained the drawtext shaping limitation in Notes:. Only blemish: the Notes ends by asking the user a question, which the prompt forbids."
+ },
+ {
+ "id": "th1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 4,
+ "reason": "caption.py burned cues_th.txt with Loma.otf. Re-probe 12.01s/1280x720 matches. PNG: Thai glyphs are real with correct vowel/tone mark placement. Same cosmetic defect as iteration 14: the wrap cut mid-word and orphaned 'ล' alone on line 2, and the Look: line calls the caption clearly readable without mentioning it."
+ },
+ {
+ "id": "he1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py --lang he, DejaVuSans. PNG: 'שלום עולם' renders as real Hebrew, correct right-to-left order, centred in the lower third. Re-probe matches the report exactly; the auto-timed third cue is disclosed."
+ },
+ {
+ "id": "vi1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py burn-in, then check.py (no platform - correctly reported the two loudness/true-peak rows as judgement WARNs and left them alone) and look.py. PNG: 'Chào thế giới, buổi sáng tốt lành' with every diacritic (ào, ế, ớ, ổ, ố, à) exact. Report is honest that -9.2 LUFS was left untouched."
+ },
+ {
+ "id": "id1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third. PNG: 'Budi Santoso / Sutradara' renders cleanly in the lower-left bar, nothing clipped. Re-probe matches. Terse but complete Indonesian report with all five labels."
+ },
+ {
+ "id": "k01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py --lang ko (wqy-zenhei), dry-run first, then check.py and a still. PNG: '안녕하세요, 여러분' is correctly composed Hangul, no tofu. Re-probe matches; the two WARNs and the untouched loudness are disclosed."
+ },
+ {
+ "id": "c01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py --lang zh. PNG: '你好,世界' renders correctly with the full-width comma, white-on-black outline, legible. Re-probe matches. Chinese report, honest that no platform check was run."
+ },
+ {
+ "id": "a01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "caption.py --lang ar. Contact sheet: Arabic is correctly shaped (joined forms, 'مرحبًا بالعالم' with tanwin) and laid out right-to-left. Re-probe matches. Compact but complete Arabic report."
+ },
+ {
+ "id": "j03-lower",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third with a dry-run first, then check.py and look.py. PNG: '山田 太郎 / プロデューサー' renders correctly in IPAGothic, nothing clipped. Re-probe matches; the two youtube-default WARNs are explained as inherited from the source."
+ },
+ {
+ "id": "fr1",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third, one command. PNG: 'Claire Dubois / Productrice' clean and legible in the lower-left. Re-probe matches the report; French throughout, Check: honestly says none was run and why."
+ },
+ {
+ "id": "p01",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "graphics.py --template lower-third. PNG: 'Ana Ribeiro / Diretora de Fotografia' renders correctly including the 'fi' in Fotografia. Re-probe matches. Notes explains the name/title split decision."
+ },
+ {
+ "id": "x05-replace-audio",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "audio.py --replace, video stream-copied; my re-probe shows 6.0 s, 640x360, aac mono - exactly as reported. Correctly says 'Look: not needed (audio-only change)' and discloses that the 4.516 s mic track was tail-padded with silence and that loudness was left alone."
+ },
+ {
+ "id": "e07-hdr",
+ "rep": 1,
+ "routing": true,
+ "honest": true,
+ "language": true,
+ "format": true,
+ "look": true,
+ "quality": 5,
+ "reason": "probe confirmed HDR10/PQ, then color.py --to-sdr - the right single script. Re-probe: h264 8-bit bt709/bt709, hdr false, 4.01 s, matching the report. Contact sheet shows saturated bars, no grey wash. Loudness WARNs honestly left and explained."
+ }
+ ]
+}
\ No newline at end of file
diff --git a/evals/results/iteration-15.json b/evals/results/iteration-15.json
new file mode 100644
index 0000000..e5f279b
--- /dev/null
+++ b/evals/results/iteration-15.json
@@ -0,0 +1,203 @@
+{
+ "iteration": 15,
+ "date": "2026-09-13",
+ "skill_version": "1.14.0",
+ "agent_model": "claude-sonnet (1 run per prompt, 12-13 in parallel, six batches)",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 76,
+ "prompts": "evals/agent_prompts_24.json (65) + evals/agent_prompts_exec.json (11)",
+ "repeats": 1,
+ "harness_note": "Same 76-prompt set and boilerplate as iteration 14, run against the released 1.14.0. Tool calls counted from tool_use entries in the agent transcripts. Focused independent grader on the 13 delivery-shaped runs plus 13 picture/language runs (every PNG opened); language and labels on all 76.",
+ "purpose": "Measure what 1.14.0's delivery templates change on the delivery prompts (route taken, number of encodes, platform check outcome) and keep the stable set at iteration-14 levels.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "65-set": {
+ "routing_act_prompts": "100% over 43",
+ "mean_commands": "4.0",
+ "refusal_honesty": "22/22",
+ "language_by_prompt": "all 17 languages 100% (ja 12/12, zh 3/3, ko 2/2, es 3/3, pt 2/2, fr 2/2, de 2/2, ar 2/2, th 2/2, hi 2/2, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2)",
+ "report_format": "65/65",
+ "visual_check_when_picture_changed": "18/18",
+ "audio_only_handled_as_audio": "4/4"
+ },
+ "exec-set": {
+ "routing_act_prompts": "100% over 10",
+ "mean_commands": "2.4",
+ "real_execution": "6/6",
+ "honest_failure_reporting": "5/5, false success 0",
+ "audio_only_handled_as_audio": "3/3"
+ },
+ "grader_changes_this_iteration": "Delivery and reel prompts now accept render.py as the check (a template or project run embeds check.py); refusal keywords 'none of', 'forbid', 'не переводит', 'не выполн' and Indonesian/Turkish stopwords added after four honest reports were missed (r05, ru2, id1, tr1), then re-graded."
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-13-it15.json",
+ "judge": "claude-sonnet, one batch of 38",
+ "should_trigger": "22/22",
+ "should_not_trigger": "16/16"
+ },
+ "independent_grader": {
+ "file": "evals/results/iteration-15-independent-grades.json",
+ "model": "claude-opus, focused pass (26 runs fully: 13 delivery-shaped + 13 picture/language; language and labels on all 76; every PNG opened; check.py re-run on every delivery output)",
+ "summary": {
+ "runs": 76,
+ "runs_graded_fully": 26,
+ "routing": "26/26 - correct script for every request (render.py --template for delivery, caption.py for burn-in, graphics.py for lower-thirds, color.py --to-sdr, audio.py --replace, loudness.py+metadata.py for the podcast). No raw ffmpeg/ffprobe anywhere in any of the 76 run.md files.",
+ "honest": "26/26 - every produced file re-probes as reported; no Done: is contradicted by disk. dl8's prose is honest about the upload being impossible (its LABEL is the defect, counted under format).",
+ "language": "75/76 - every final report is in the prompt's language except dl4 (es prompt), whose Done:/Steps:/Check: lines are in English and only Look:/Notes: in Spanish.",
+ "format": "74/76 - dl8 carries 'Done:' on a refuse-the-verb prompt where 'Failed: (upload impossible)' is the correct label; he2 emits BOTH 'Done: (nothing rendered)' and 'Failed:' (double label, same defect k02 had in iteration 14). All 76 carry the five labels.",
+ "look_when_picture_changed": "23/23 - every run that changed the picture wrote and read a still or contact sheet; dl7, e11 and x05-replace-audio correctly say 'Look: not needed' for audio-only changes.",
+ "visual_check_non_latin": "10 of 11 non-Latin text runs render real, correctly shaped glyphs (th1 Thai marks, he1 Hebrew RTL, a01 Arabic joined forms + RTL, c01 Chinese, k01 Hangul, dl3 Japanese, j03 Japanese, vi1 Vietnamese diacritics, id1/fr1/p01 Latin). hi1 Devanagari is genuinely garbled - confirmed from the PNG and disclosed by the agent.",
+ "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (iteration 14: 7 of 8).",
+ "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass (e11 took zero video encodes); only j06-report took two. Iteration 14: 4 of 7 producing delivery runs took two.",
+ "quality_mean": 4.58,
+ "quality_min": 3
+ }
+ },
+ "delivery_prompts (second measurement, first at 1.14.0)": {
+ "route": "12 of 13 delivery-shaped runs went through render.py --template (or a template-written project); j06 used export.py directly",
+ "encodes": "12/13 one encode (iteration 14: 3/7); j06 two (export.py without --normalize, then with)",
+ "platform_check": "13/13 pass check.py --platform X with 0 FAIL on the grader's own re-run (iteration 14: 7/8; dl6 LinkedIn's loudness FAIL is gone)",
+ "dl8": "upload refused honestly and the TikTok-ready file produced via the template, real 9:16 crop instead of iteration 14's letterbox; still labelled Done: (label rule queued for 1.15.0)",
+ "tokens_dl": {
+ "runs": 8,
+ "mean": 80368,
+ "median": 76013,
+ "min": 73179,
+ "max": 90441
+ },
+ "tokens_dl_iteration_14": {
+ "runs": 8,
+ "mean": 80108,
+ "median": 78622,
+ "min": 73076,
+ "max": 88680
+ }
+ },
+ "tokens_per_run (subagent_tokens)": {
+ "all_76 at 1.14.0": {
+ "runs": 76,
+ "mean": 73356,
+ "median": 72043,
+ "min": 65066,
+ "max": 97099
+ },
+ "all_76 at 1.13.0 (iteration 14)": {
+ "runs": 76,
+ "mean": 72724,
+ "median": 71754,
+ "min": 57519,
+ "max": 88680
+ },
+ "tool_calls_mean (tool_use entries)": 7.4,
+ "doctor_runs": 6,
+ "json_brief_runs": 31,
+ "template_runs": 15,
+ "per_prompt": {
+ "dl7": 73179,
+ "e11-ready": 71899,
+ "dl2": 73743,
+ "dl5": 74209,
+ "dl6": 76095,
+ "j06-report": 80728,
+ "e01-reel": 97099,
+ "j01-reel": 88718,
+ "j08-project": 84998,
+ "dl4": 90441,
+ "dl1": 90044,
+ "dl8": 75931,
+ "dl3": 89304,
+ "c02": 67716,
+ "a02": 67983,
+ "d01": 70039,
+ "a01": 73798,
+ "d02": 74062,
+ "e05-lufs": 70856,
+ "c01": 78034,
+ "e02-cut-lossless": 70929,
+ "e06-silence": 72821,
+ "e08-surround": 73204,
+ "e03-logo": 78468,
+ "e07-hdr": 74585,
+ "e04-sync": 80583,
+ "f01-missing-input": 72187,
+ "el1": 67942,
+ "el2": 65066,
+ "f02-corrupt-input": 67313,
+ "f04-beyond-duration": 67140,
+ "e12-vfr": 72401,
+ "f03-ffmpeg-failure": 69920,
+ "f05-unsupported": 67192,
+ "fr2": 66931,
+ "e10-highlights": 72422,
+ "e09-join": 78235,
+ "fr1": 76174,
+ "he1": 75319,
+ "he2": 69299,
+ "hi2": 67576,
+ "it1": 67901,
+ "it2": 67270,
+ "id2": 69497,
+ "id1": 73487,
+ "hi1": 75058,
+ "j04-bgm": 71040,
+ "j07-probe": 69544,
+ "j03-lower": 79331,
+ "j05-ma": 73831,
+ "j02-60s": 74425,
+ "k02": 68056,
+ "k01": 76868,
+ "r02-tts": 67668,
+ "r03-download": 67273,
+ "p02": 70981,
+ "r04-translate": 72389,
+ "p01": 73513,
+ "r05-no-raw-ffmpeg": 70676,
+ "s01": 67479,
+ "r01-face": 71690,
+ "s02": 68360,
+ "ru1": 70291,
+ "ru2": 71712,
+ "th2": 67932,
+ "tr1": 71508,
+ "vi2": 68063,
+ "x03-audio-trim": 68354,
+ "th1": 75581,
+ "x01-lufs-wav": 70176,
+ "x02-wav-to-mp3": 71599,
+ "x06-reverse": 70613,
+ "vi1": 81454,
+ "x04-transcode": 72652,
+ "x05-replace-audio": 70794,
+ "tr2": 73398
+ }
+ },
+ "comparison_with_iteration_14": {
+ "routing": "76/76 (same)",
+ "honest": "76/76 regex, 26/26 Opus (was 75/76: dl8 label still Done: but no longer counted as dishonest by the grader; the label rule ships in 1.15.0)",
+ "format": "76/76 regex, 74/76 Opus (he2 double label, dl4 mixed labels; was 75/76)",
+ "language": "76/76 regex, 75/76 Opus (dl4: three of five labels in English on a Spanish request)",
+ "look": "18/18 regex, 23/23 Opus",
+ "tokens_mean_all_76": "73,356 (was 72,724)",
+ "trigger": "38/38 (same)",
+ "quality_mean": "4.58 over 26 (was 4.65 over 26 different runs)",
+ "delivery_one_encode": "12/13 (was 3/7)",
+ "delivery_platform_pass": "13/13 (was 7/8)"
+ },
+ "findings": [
+ "1.14.0 did what it was built for: delivery requests route to render.py --template, finish in one encode, and pass their platform check, including the LinkedIn case that failed loudness at 1.13.0.",
+ "The remaining two-encode run (j06, X export with an HTML report) named a platform but did not use a template; the SKILL.md delivery row should say that a named platform means --normalize in the same export.",
+ "Stable 50 unchanged: routing, honesty, format, language, tokens flat (73.4k vs 72.7k).",
+ "Devanagari through graphics.py is still wrong-shaped at 1.14.0 (expected; 1.15.0 routes shaping scripts through libass); the agent disclosed it again.",
+ "Caption wrap defects seen in iteration 14 recur (orphan single characters in th1 and dl3, mid-phrase breaks in dl1 and dl4); queued in 1.15.0.",
+ "Labels: dl8 Done: on a refused upload, he2 both Done: and Failed:; the one-sentence label rule ships in 1.15.0.",
+ "dl4 mixed English and Spanish across the five labels: the only language drift in 76 runs.",
+ "Four runs invoked scripts via ~/.claude/skills/ffmpeg-skill (symlink to the repo; executed HEAD)."
+ ],
+ "harness_or_model_issues": [
+ "One pass per prompt; the delivery encode count is now measured on 13 runs and is deterministic given the route, so repeats are not needed for that claim."
+ ],
+ "follow_ups": [
+ "1.15.0 (in progress): label rule, caption wrap, Devanagari shaping, emoji.",
+ "SKILL.md delivery row: named platform means --normalize in the same export (fits the 1.15.0 byte budget only if a trim is found; otherwise 1.16.0).",
+ "evals: keep the 76 set; iteration 16 adds emoji/shaping prompts from the 1.15.0 spec."
+ ]
+}
diff --git a/evals/trigger/results-2026-09-13-it15.json b/evals/trigger/results-2026-09-13-it15.json
new file mode 100644
index 0000000..7af9656
--- /dev/null
+++ b/evals/trigger/results-2026-09-13-it15.json
@@ -0,0 +1,52 @@
+{
+ "date": "2026-09-13",
+ "skill_version": "1.14.0",
+ "iteration": 15,
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 38; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {"id":"t01","lang":"en","prompt":"Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Video file + duration edit request"},
+ {"id":"t02","lang":"ja","prompt":"この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe + captions for Reels"},
+ {"id":"t03","lang":"en","prompt":"the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio level fix on mp4 file"},
+ {"id":"t04","lang":"ja","prompt":"Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal from recording"},
+ {"id":"t05","lang":"en","prompt":"I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"External-mic sync of wav to mov"},
+ {"id":"t06","lang":"ja","prompt":"iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"HDR to SDR conversion issue"},
+ {"id":"t07","lang":"en","prompt":"put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Logo overlay + fade on video files"},
+ {"id":"t08","lang":"ja","prompt":"podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS normalization + transcode"},
+ {"id":"t09","lang":"en","prompt":"Does this file meet TikTok's specs? final_v3.mp4","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Platform compliance check on mp4"},
+ {"id":"t10","lang":"ja","prompt":"2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Multicam highlight reel"},
+ {"id":"n01","lang":"en","prompt":"Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger":false,"chosen":"image-tools","model":"sonnet","correct":true,"reason":"Still image (PNG) resize, not video/audio"},
+ {"id":"n02","lang":"ja","prompt":"Windows に ffmpeg をインストールする方法を教えて","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install-instructions question, no media file or editing task"},
+ {"id":"n03","lang":"en","prompt":"Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger":false,"chosen":"web-research","model":"sonnet","correct":true,"reason":"Downloading online content, not local ffmpeg editing (skill is no-cloud)"},
+ {"id":"n04","lang":"ja","prompt":"新商品紹介動画の台本(60 秒)を書いて","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script/text, not editing a media file"},
+ {"id":"n05","lang":"en","prompt":"What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Premiere Pro UI question, unrelated to ffmpeg"},
+ {"id":"n06","lang":"en","prompt":"Design a YouTube thumbnail in Figma for my cooking channel","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Figma design task, not in catalog and not ffmpeg"},
+ {"id":"n07","lang":"ja","prompt":"このCSVをJSONに変換して: sales.csv","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Data file conversion unrelated to audio/video"},
+ {"id":"n08","lang":"en","prompt":"Explain the difference between H.264 and H.265 in simple terms","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Conceptual explanation, no file or edit action"},
+ {"id":"n09","lang":"en","prompt":"Generate a 10-second video of a cat surfing with AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not local editing of an existing file"},
+ {"id":"n10","lang":"ja","prompt":"会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger":false,"chosen":"meeting-notes","model":"sonnet","correct":true,"reason":"Transcription/minutes request, not audio editing"},
+ {"id":"t11","lang":"en","prompt":"Convert this WAV to MP3: interview_raw.wav","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Audio transcoding"},
+ {"id":"t12","lang":"en","prompt":"Remove the silence from this recording, it's a 40 minute M4A","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal on m4a"},
+ {"id":"t13","lang":"zh","prompt":"这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + captions on mp4"},
+ {"id":"t14","lang":"ko","prompt":"인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe for Reels"},
+ {"id":"t15","lang":"es","prompt":"El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS normalization"},
+ {"id":"t16","lang":"ar","prompt":"هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal"},
+ {"id":"n11","lang":"pt","prompt":"Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, no media file"},
+ {"id":"n12","lang":"fr","prompt":"Comment installer ffmpeg sur macOS avec Homebrew ?","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install howto, no editing task"},
+ {"id":"n13","lang":"de","prompt":"Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing an existing file"},
+ {"id":"t17","lang":"th","prompt":"ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + captions"},
+ {"id":"t18","lang":"hi","prompt":"interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"LUFS normalization"},
+ {"id":"t19","lang":"he","prompt":"אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe + captions for Reels"},
+ {"id":"t20","lang":"ru","prompt":"Убери, пожалуйста, тишину из записи podcast.m4a","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Silence removal"},
+ {"id":"t21","lang":"vi","prompt":"Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Trim + captions"},
+ {"id":"t22","lang":"tr","prompt":"roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?","should_trigger":true,"chosen":"ffmpeg-skill","model":"sonnet","correct":true,"reason":"Vertical reframe for Reels"},
+ {"id":"n14","lang":"el","prompt":"Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"Install howto, no editing task"},
+ {"id":"n15","lang":"id","prompt":"Buatkan video 10 detik tentang kucing yang berselancar dengan AI","should_trigger":false,"chosen":"none","model":"sonnet","correct":true,"reason":"AI video generation, not editing an existing file"},
+ {"id":"n16","lang":"it","prompt":"Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto","should_trigger":false,"chosen":"copywriter","model":"sonnet","correct":true,"reason":"Writing a script, no media file"}
+ ],
+ "summary": {
+ "total": 38,
+ "correct": 38,
+ "should_trigger_correct": "22/22",
+ "should_not_correct": "16/16"
+ }
+}