Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

15.3 KB
Raw Blame
{
 "iteration": 16,
 "date": "2026-09-13",
 "skill_version": "1.15.0",
 "agent_model": "claude-sonnet (1 run per prompt, 8-16 in parallel, seven batches)",
 "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
 "runs": 82,
 "prompts": "evals/agent_prompts_24.json (71) + evals/agent_prompts_exec.json (11)",
 "repeats": 1,
 "harness_note": "Same boilerplate as iterations 14-15; the set grew by six 1.15.0 prompts (em1-em4 emoji captions/title cards, em1-em3 shipping a directory of code-point PNGs; sh1 Hindi lower-third, sh2 Thai lower-third plus Thai captions). Tool calls counted from tool_use entries in the agent transcripts. Focused independent grader on 28 runs (emoji, shaping, label-rule, delivery and picture/language runs; every PNG opened, stills extracted inside the emoji window; check.py re-run on every delivery output); language and labels on all 82.",
 "purpose": "Measure what 1.15.0 changed: complex-script shaping through graphics.py (hi1 was garbled at 1.14.0), emoji people can see (--emoji-assets / overlay), the Failed: label rule on refuse-the-verb prompts (dl8, he2, k02), the caption-wrap orphan fix, and keep the 76-prompt set at iteration-15 levels.",
 "regex_grader (evals/grade_runs_24.py)": {
  "71-set": {
   "routing_act_prompts": "100% over 49",
   "mean_commands": "4.0",
   "refusal_honesty": "22/22",
   "language_by_prompt": "all 17 languages 100% (ja 13/13, zh 4/4, ko 2/2, es 3/3, pt 2/2, fr 2/2, de 2/2, ar 2/2, th 3/3, hi 3/3, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2)",
   "report_format": "71/71",
   "visual_check_when_picture_changed": "23/24 (e12-vfr conformed VFR to CFR with a re-encode and wrote no still)",
   "audio_only_handled_as_audio": "4/4",
   "emoji_honesty": "4/4 (em1-em3 used the shipped assets and reported mode png; em4 had no assets, shipped monochrome and said so)"
  },
  "exec-set": {
   "routing_act_prompts": "100% over 10",
   "mean_commands": "2.7",
   "real_execution": "6/6",
   "honest_failure_reporting": "5/5, false success 0",
   "audio_only_handled_as_audio": "3/3"
  },
  "grader_changes_this_iteration": "em2 accepts overlay.py --image as a route (the prompt hands the agent a PNG, and SKILL.md routes 'put an image on the video' to overlay.py); em4 joined the picture-changed set; the colour-claim regex ignores the font name 'Noto Color Emoji' and negations ('no colour path', 'for colour emoji pass --emoji-assets') that runs paste from tool output (em4 was capped at 0.5 for an honest report); German stopwords extended after d01's honest German report was missed (the Opus grader confirms it is German). Re-graded after each change."
 },
 "trigger_set": {
  "file": "evals/trigger/results-2026-09-13-it16.json",
  "judge": "claude-sonnet, one batch of 40 (38 + em-caption + sh-lower-third)",
  "should_trigger": "23/23",
  "should_not_trigger": "17/17"
 },
 "independent_grader": {
  "file": "evals/results/iteration-16-independent-grades.json",
  "model": "claude-opus, focused pass (28 runs fully: em1-em4, sh1-sh2, hi1, th1, he2, k02, dl1-dl8, e01, e11, j01, j06, j08, a01, j03, c01, vi1, x05; language and labels on all 82; every PNG opened; stills extracted inside the emoji window; check.py re-run on every delivery output)",
  "summary": {
   "runs": 82,
   "runs_graded_fully": 28,
   "routing": "28/28 - correct script for every request (caption.py for burn-in, caption.py --emoji-assets / overlay.py --image for emoji, graphics.py for lower-thirds, render.py --template for delivery, loudness.py+metadata.py for the podcast, audio.py --replace for the track swap, redact.py --help to prove the refusal). No raw ffmpeg or ffprobe invoked by any agent in any of the 82 run.md files.",
   "honest": "28/28 - every written file re-probes exactly as reported, every platform claim re-runs clean, and the three refusals are accurate. No Done: is contradicted by disk.",
   "language": "81/82 - every final report is in the prompt's language except dl4 (es), whose Done:/Steps:/Check: lines are English and only Look:/Notes: Spanish - the identical defect iteration 15 flagged on dl4. id1 (id) has a milder version (English Done:/Steps:, Indonesian body). d01 IS German: Check:/Look:/Notes: are German prose and only the Done: line is a language-neutral filename+spec string, so the regex grader's 'German not detected' is a grader artefact, not a run defect.",
   "format": "82/82 carry the five labels exactly once each, each refusal under a single Failed:. The three label-rule prompts are all correct now: dl8 emits ONE 'Failed: cannot actually post/upload to TikTok ... I prepared the file instead', he2 and k02 emit a single Failed: (iteration 15 had dl8 as Done: and he2 as a Done:+Failed: double). Only cosmetic deviation: d02 titles its section '# Final report' instead of '## Final report'.",
   "look_when_picture_changed": "25/25 - every run that changed the picture wrote and read a still or contact sheet; dl7, x05-replace-audio and the two refusals (he2, k02) correctly say 'Look: not needed'. e11-ready changed only audio and still produced a sheet, which is harmless.",
   "visual_check_non_latin": "11/11 non-Latin text runs now render real, correctly shaped glyphs: Devanagari (sh1, hi1) with correct matra reordering, conjuncts, reph and word spaces; Thai (sh2 in both the lower third and the captions, th1) with correct vowel/tone stacking; Arabic (a01) joined and RTL; Chinese (c01, em3); Japanese (dl2, dl3, j03-lower, em2); Vietnamese (vi1). Iteration 15's single failure - hi1's garbled Devanagari - is fixed.",
   "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (same as iteration 15).",
   "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass; e11-ready took ZERO video encodes (loudness stream-copies video); j06-report again took TWO (export.py without --normalize, then rm and re-export). Identical to iteration 15.",
   "emoji": "3 of 3 runs that were given PNG assets used them (em1 and em3 via caption.py --emoji-assets, em2 via overlay.py --image) and the emoji is visibly composited in colour inside its own time window in all three, verified on stills I extracted at 4.5s / 1.5s rather than frame 0. em4, which ships no assets, rendered monochrome and said so.",
   "quality_mean": 4.68,
   "quality_min": 3
  }
 },
 "1.15.0 targets (first measurement)": {
  "complex_script_shaping": "4/4 - sh1 and hi1 Devanagari correctly shaped through the libass path (conjunct, i-matra reordered, reph, word space; 1.14.0 produced 'प्रयाशर्म'), sh2 Thai correct in both the lower third and the captions, th1 correct. 11/11 non-Latin runs pass a visual read (iteration 15: 10/11).",
  "emoji": "em1-em3 used the shipped PNGs (caption.py --emoji-assets twice, overlay.py --image once), the composite is visible in colour inside its own cue window on stills the grader extracted at 1.5 s / 4.5 s; em4 (no assets) rendered monochrome and reported it. Descriptive gap: em1 and em3 call plain coloured squares 'colour emoji'; only em2 said what it saw.",
  "label_rule": "3/3 - dl8 one Failed: naming the prepared TikTok file (iteration 15: Done:), he2 one Failed: (iteration 15: Done:+Failed:), k02 one Failed:.",
  "caption_wrap": "NOT fixed as a class - dl1 still 'A third line the tool' / 'times for me', dl4 still a lone 'subtítulos' and 'Una tercera línea con' / 'tiempos automáticos' (byte-for-byte iteration 15); dl3's lone '行' is gone but a lone '2' and a mid-word 'る行' break remain; only th1 is genuinely better. The one-character ban treats a symptom; a phrase-aware breaker is the fix (queued for 1.16.0)."
 },
 "delivery_prompts (third measurement)": {
  "route": "12 of 13 delivery-shaped runs went through render.py --template (j01-reel now uses the template it wrote by hand in iteration 15); j06 used export.py directly",
  "encodes": "12/13 one encode, e11 zero video encodes, j06 two (export.py without --normalize, then again with) - identical to iteration 15",
  "platform_check": "13/13 pass check.py --platform X with 0 FAIL / 0 WARN on the grader's own re-run",
  "tokens_dl": {
   "runs": 8,
   "mean": 77755,
   "median": 76648,
   "min": 72492,
   "max": 86938
  },
  "tokens_dl_iteration_15": {
   "runs": 8,
   "mean": 80368,
   "median": 76013,
   "min": 73179,
   "max": 90441
  }
 },
 "tokens_per_run (subagent_tokens)": {
  "all_82 at 1.15.0": {
   "runs": 82,
   "mean": 73710,
   "median": 72508,
   "min": 63975,
   "max": 104136
  },
  "same_76 at 1.15.0": {
   "runs": 76,
   "mean": 73253,
   "median": 72360,
   "min": 63975,
   "max": 104136
  },
  "same_76 at 1.14.0 (iteration 15)": {
   "runs": 76,
   "mean": 73356,
   "median": 72043,
   "min": 65066,
   "max": 97099
  },
  "new_6 (em1-em4, sh1-sh2)": {
   "runs": 6,
   "mean": 79506,
   "median": 79928,
   "min": 75575,
   "max": 81949
  },
  "tool_calls_mean (tool_use entries)": 7.8,
  "doctor_runs": 4,
  "json_brief_runs": 36,
  "template_runs": 18,
  "per_prompt": {
   "em4": 75575,
   "hi1": 71559,
   "k02": 67890,
   "he2": 69203,
   "sh1": 79491,
   "em1": 79913,
   "em3": 81949,
   "dl8": 76365,
   "th1": 72629,
   "a01": 79229,
   "dl7": 72492,
   "dl1": 77253,
   "e11-ready": 75325,
   "em2": 79943,
   "sh2": 80167,
   "c01": 72242,
   "k01": 76091,
   "dl2": 73658,
   "dl6": 76932,
   "j06-report": 74044,
   "dl4": 83518,
   "dl3": 86938,
   "a02": 66919,
   "c02": 67266,
   "dl5": 74885,
   "e01-reel": 78687,
   "d01": 70132,
   "j01-reel": 96475,
   "d02": 70569,
   "e05-lufs": 72479,
   "e06-silence": 72141,
   "e04-sync": 76739,
   "e02-cut-lossless": 79655,
   "e03-logo": 80066,
   "el2": 66937,
   "el1": 67566,
   "e07-hdr": 76676,
   "j08-project": 104136,
   "f01-missing-input": 75227,
   "f02-corrupt-input": 74519,
   "e09-join": 77900,
   "e08-surround": 81001,
   "f03-ffmpeg-failure": 76739,
   "e12-vfr": 72602,
   "f05-unsupported": 69339,
   "f04-beyond-duration": 72522,
   "e10-highlights": 75904,
   "fr1": 73753,
   "fr2": 66827,
   "hi2": 66791,
   "it2": 66944,
   "it1": 67513,
   "id2": 68873,
   "id1": 72759,
   "he1": 79234,
   "r01-face": 69367,
   "r02-tts": 67698,
   "j07-probe": 70038,
   "j04-bgm": 74619,
   "j05-ma": 71043,
   "j02-60s": 73385,
   "j03-lower": 75738,
   "p02": 72493,
   "p01": 72911,
   "r03-download": 63975,
   "r05-no-raw-ffmpeg": 67262,
   "r04-translate": 68865,
   "ru1": 70809,
   "th2": 68086,
   "s01": 69286,
   "s02": 70875,
   "ru2": 72015,
   "tr1": 70409,
   "vi2": 67509,
   "tr2": 68729,
   "x03-audio-trim": 68298,
   "x01-lufs-wav": 69900,
   "x02-wav-to-mp3": 71539,
   "vi1": 71980,
   "x06-reverse": 69961,
   "x05-replace-audio": 70564,
   "x04-transcode": 74705
  }
 },
 "comparison_with_iteration_15": {
  "routing": "100% -> 100%",
  "refusal_honesty": "22/22 -> 22/22",
  "report_format": "76/76 -> 82/82 (Opus: 74/76 -> 82/82, both label defects closed)",
  "language": "Opus 75/76 -> 81/82 (dl4 repeats its English label lines; id1 milder)",
  "non_latin_glyphs": "10/11 -> 11/11",
  "delivery": "12/13 one encode and 13/13 platform pass, unchanged",
  "quality_mean": "4.58 -> 4.68",
  "tokens_mean_same_76": "73356 -> 73253 (flat)"
 },
 "findings": [
  "Three of the four things 1.15.0 set out to fix are fixed. Complex-script shaping is the standout: sh1, hi1 and sh2 render Devanagari and Thai correctly through the libass path, where 1.14.0's hi1 produced 'प्रयाशर्म' for 'प्रिया शर्मा'. All 11 non-Latin runs now pass a visual read.",
  "The emoji feature works end to end. Every run given PNG assets used them (--emoji-assets on caption.py, --image on overlay.py), and I confirmed the composite is visible in colour inside its own cue window, not just at frame 0. em4, which was given no assets, rendered monochrome and reported that plainly instead of claiming colour.",
  "The label rule landed on all three refuse-the-verb prompts: dl8, he2 and k02 each carry exactly one Failed: with the artefact described. This closes both iteration 15 defects (dl8's Done: and he2's double label) with no over-correction.",
  "The caption splitter is the defect that did NOT get fixed. dl1 still splits 'A third line the tool' / 'times for me' and dl4 still produces a lone 'subtítulos' plus 'Una tercera línea con' / 'tiempos automáticos' - byte-for-byte the iteration 15 failures. dl3's one-character '行' orphan is gone but a lone '2' line and a mid-word 'る行' remain; only th1 is genuinely better. The one-character ban treats a symptom; the splitter still breaks phrases and words at arbitrary points.",
  "dl4 repeats iteration 15's language failure exactly: a Spanish prompt answered with English Done:/Steps:/Check: and Spanish Look:/Notes:. id1 shows a milder version. Nothing in the skill tells the agent the LABEL LINES are part of the user's language, only the report.",
  "d01's report IS German - Check:, Look: and Notes: are German prose and only the Done: line is a language-neutral 'filename - 12.01 s, 1280x720, ...' string. The regex language grader flagging it is a grader artefact; the detector should ignore the Done: spec line or look at the whole report.",
  "Delivery discipline is stable at a high level: 12 of 13 runs used render.py --template (j01-reel converted from iteration 15's hand-written project), 12 of 13 took one encode, and all 13 pass their platform check at 0 FAIL and 0 WARN on my re-run. j06-report remains the single holdout on both counts - it goes straight to export.py and pays for it with a second encode after forgetting --normalize."
 ],
 "harness_or_model_issues": [
  "Only rep1 exists, so per-run variance cannot be separated from real improvement between iterations 15 and 16.",
  "Eleven runs (a02, d01, dl2, e11-ready, el1, fr1, id1, j03-lower, j05-ma, p01, tr2) executed /root/.claude/skills/ffmpeg-skill/scripts instead of the /home/user/ffmpeg-skill/scripts path their prompt named. Three of them are graded in full, so those grades are not strictly against the repo copy under test.",
  "em2's assets are deliberately plain coloured squares, and em1's and em3's are too - but only em2's prompt makes the mismatch conspicuous enough that the agent commented. If the intent is to test 'say what you saw', em1 and em3 need the same framing or the assets need to be recognisable emoji.",
  "d02 writes '# Final report' instead of '## Final report'. Every label is present and correct, but a strict section-header grader would miss the report entirely.",
  "The caption sheets' tile timestamps do not always line up with the cue in view (em3/em4 show cue 2 text under a 00:00:01.999 label). I verified the actual frames with my own look.py extractions and the burn-in timing is correct, so this is a contact-sheet labelling artefact worth a look in look.py.",
  "/root/.claude/skills/ffmpeg-skill is a symlink to /home/user/ffmpeg-skill on this box, so the eleven runs that executed that path still ran the 1.15.0 tree under test."
 ],
 "follow_ups": [
  "1.16.0: phrase-aware caption breaking (never split inside a word or between a determiner and its noun; balance lines by width) - dl1/dl3/dl4 evidence in iteration-16-independent-grades.json.",
  "SKILL.md: the label lines (Done:/Steps:/Check:) are part of the user's-language report (dl4 twice, id1 once).",
  "Evals: em1/em3 assets should be recognisable emoji or the prompt should ask the agent to say what it saw; d02's '# Final report' heading; look.py sheet tile timestamps vs the cue shown.",
  "j06-report: 'named platform means --normalize in the same export' sentence for SKILL.md when the byte budget allows."
 ]
}