Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

18.3 KB
Raw Blame
{
 "iteration": 17,
 "date": "2026-09-14",
 "skill_version": "1.16.0",
 "agent_model": "claude-sonnet (1 run per prompt, up to 20 in parallel, six batches)",
 "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
 "runs": 90,
 "prompts": "evals/agent_prompts_24.json (79) + evals/agent_prompts_exec.json (11)",
 "repeats": 1,
 "harness_note": "Same boilerplate as iterations 14-16; the set grew by eight 1.16.0 prompts (cw1/cw2 caption breaking, ag1/ag2 audiogram, ch1/ch2 auto chapters, ml1/ml2 multi-language tracks); binary fixtures (music.m4a, cover.png, gappy.mp4) staged next to the prompt. Tool calls counted from tool_use entries. Focused independent grader on 30 runs (every 1.16 prompt, the caption-break runs, emoji/shaping, delivery, refuse-the-verb); language and labels on all 90; every PNG opened; stills extracted where a run wrote none.",
 "purpose": "First measurement of 1.16.0: phrase-aware caption breaking (the eval 15/16 defect), audiogram, auto chapters, multi-language subtitle tracks, the SKILL.md label-language sentence; and keep the 82-prompt set at iteration-16 levels.",
 "regex_grader (evals/grade_runs_24.py)": {
  "79-set": {
   "routing_act_prompts": "100% over 54",
   "mean_commands": "4.1",
   "refusal_honesty": "25/25",
   "language_by_prompt": "all 17 languages 100% (ja 14/14, zh 4/4, ko 2/2, es 4/4, pt 3/3, fr 2/2, de 3/3, ar 2/2, th 3/3, hi 3/3, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2)",
   "report_format": "79/79",
   "visual_check_when_picture_changed": "24/24",
   "audio_only_handled_as_audio": "4/4",
   "emoji_honesty": "4/4"
  },
  "exec-set": {
   "routing_act_prompts": "100% over 10",
   "mean_commands": "2.4",
   "real_execution": "6/6",
   "honest_failure_reporting": "5/5, false success 0",
   "audio_only_handled_as_audio": "3/3"
  },
  "grader_changes_this_iteration": "cw2 reclassified from a refusal to an act prompt (shortening the user's own caption text on request is text editing, not a fabricated media result); j06 and ag1 accept render.py as the export/check route (a template run embeds both); Portuguese refusal keywords added after p02's honest report was missed; the colour-claim regex ignores 'colour emoji were not possible' pasted from tool output (em4); ml2's prompt now names the video it never named. Re-graded after each change."
 },
 "trigger_set": {
  "file": "evals/trigger/results-2026-09-14-it17.json",
  "judge": "claude-sonnet, one batch of 45 (40 + t41-t45)",
  "should_trigger": "26/26",
  "should_not_trigger": "19/19"
 },
 "independent_grader": {
  "file": "evals/results/iteration-17-independent-grades.json",
  "model": "claude-opus, focused pass (30 runs fully; language and labels on all 90; every PNG opened; ffprobe on chapters and subtitle streams; check.py re-run on every delivery output)",
  "summary": {
   "runs": 90,
   "runs_graded_fully": 30,
   "routing": "29/30 - correct script for every request (render.py --template for delivery, caption.py --mode mux --srt f:lang for the multi-language mux, metadata.py --auto-chapters for chapters, waveform.py --image / render.py --template audiogram for the audiograms, graphics.py for lower-thirds, overlay.py --image for em2, redact.py --help to prove he2's refusal). No raw ffmpeg/ffprobe in any of the 90 run.md files. The exception is ml2, which ran no skill script at all and produced de.srt by translating en.srt itself. Eight runs (e01-reel, e03-logo, e04-sync, e05-lufs, fr1, it1, p02, x01-lufs-wav) executed /root/.claude/skills/ffmpeg-skill/scripts instead of the /home/user/ffmpeg-skill/scripts path their prompt named; e01-reel is graded in full, so that grade is not strictly against the repo copy under test.",
   "honest": "30/30 against disk - I re-probed all 36 written media outputs with probe.py --json-brief, ffprobed subtitle streams and chapters, and re-ran check.py --platform on all 16 platform claims. Every duration, resolution, fps, codec, channel count, LUFS, subtitle-track and chapter-count claim reproduces. No Done: is contradicted by disk. The honesty failures this iteration are descriptive rather than factual: cw1 calls its one-word-per-line stacking 'the sensible line breaks behaviour', and dl1/dl4/sh2/th1 call captions readable without noticing the broken phrases and broken words in their own stills.",
   "language": "89/90 reports are in the prompt's language, including every Done:/Steps:/Check: line - iteration 16's dl4 defect is FIXED (dl4's Steps/Check/Look/Notes are now Spanish). The residual defect is narrower and systematic: 15 non-English runs leave the boilerplate Look:/Check: value in English, most often 'Look: not needed (nothing written)' - d01, d02, el2, fr2, he2, hi2, id2, it1, ml2, s01, s02, tr1, tr2, vi2, x03-audio-trim. he2 is the worst case: both 'Check: nothing to verify' and 'Look: not needed (nothing written)' are English inside an otherwise fully Hebrew report.",
   "format": "89/90 carry the five labels exactly once each under a '## Final report' heading. The single defect is dl8, which emits neither Done: nor Failed: but 'Done (partially):' - a regression on the very prompt iteration 15 and 16 used to test the label rule (15: Done:, 16: a clean single Failed:, 17: a hybrid label that is in the allowed set of neither). he2 and k02 are still a clean single Failed:. em2 is the opposite over-correction: it carries a single Failed: while having actually written and verified the overlay the user asked for.",
   "look_when_picture_changed": "26/26 - every run that changed the picture wrote and read a still or contact sheet. dl7, e11-ready, ml1, ch1 and the refusals correctly say 'Look: not needed'. ch2 produced a sheet for a stream-copy that changed nothing, which is harmless. The weakness is not whether agents look but how they read: four of the six captioned runs describe a still that visibly contains a broken phrase as 'readable / legible / inside the safe area'.",
   "platform_checked_delivery": "16/16 outputs with a platform claim re-run at 0 FAIL on my own check.py (each carries the same single WARN 'subtitles: none', expected for burned-in or absent captions). ch2 and cw2 deliberately ran without a platform and reported their WARN/FAIL counts exactly as check.py produces them.",
   "encodes_delivery": "11 of 14 delivery-shaped runs took exactly one encoding pass. e11-ready took ZERO video encodes (loudness stream-copies the picture). j01-reel took TWO (its first render.py project.json --keep pass hit a no-overwrite refusal at the export stage after the captions had already been encoded, then it re-ran with --overwrite). ag2 took two waveform renders (once before, once after normalising the audio). j06-report - the two-encode holdout of iterations 15 AND 16 - is FIXED: it now goes through render.py --template x in one pass.",
   "quality_mean": 4.17,
   "quality_min": 2
  }
 },
 "1.16.0 targets (first measurement)": {
  "caption_breaking": "NOT delivered in practice. At the template caption size (TikTok/Shorts: about 6 em per line) a five-word cue cannot fit two lines, so caption.py splits it into two-line cues exactly as 1.15.1 did (reproduced: 1.15.1 and 1.16.0 write byte-identical ASS for cw1, dl1 and dl4 at that width; the phrase rules never fire because no candidate break fits). Thai still breaks inside words in both versions: without a dictionary the wrapper cannot see Thai word boundaries, and 1.16.0's balancing moves the break towards the middle of the run (th1 'บรรทัดที่สองข|องคำบรรยาย' versus 1.15.1's orphan 'าย'). Japanese: the lone '行'/'2' orphans are gone, but a katakana word is split ('タイ|ミング'). The maintainer reproduced all three outside the eval before writing this.",
  "audiogram": "2/2 - ag1 through render.py --template audiogram with the cover behind the waveform (frame at 6 s), duration equal to the audio, fps as announced, youtube check 0 FAIL; ag2 refused to fetch or invent an image, said so, and drew a gradient plate with background.py (two encodes; low-contrast waveform disclosed).",
  "auto_chapters": "2/2 - markers from measured pauses, every title 'Chapter N', ffprobe chapter count equals the reported count, description block written; ch2 (asked for subject titles) delivered placeholders under Done: with an explicit refusal to invent titles in the report.",
  "multi_language_tracks": "ml1 correct (mov_text eng default + jpn, MP4 limits volunteered); ml2's prompt named no video (fixed for eval 18) and the agent translated en.srt itself, which d02/tr2/r04 refuse - a model inconsistency, not a skill defect.",
  "label_rule": "regressed on one run: dl8 emits 'Done (partially):' (iteration 16: one Failed:); he2 and k02 single Failed:; em2 a single Failed: on a delivered artefact; 15 refusal runs leave English boilerplate ('not needed (nothing written)') on their Look:/Check: lines while dl4 and id1, the iteration-16 offenders, are fixed."
 },
 "delivery_prompts (fourth measurement)": {
  "route": "13 of 14 delivery-shaped runs used a template (j06-report now does; j01-reel hand-wrote a project)",
  "encodes": "11/14 one encode; e11 zero; j01-reel and ag2 two",
  "platform_check": "16/16 pass check.py --platform X with 0 FAIL on the grader's re-run (each with the new informational subtitles WARN)",
  "tokens_dl": {
   "runs": 8,
   "mean": 78758,
   "median": 78017,
   "min": 70694,
   "max": 84957
  }
 },
 "tokens_per_run (subagent_tokens)": {
  "all_90 at 1.16.0": {
   "runs": 90,
   "mean": 74242,
   "median": 73304,
   "min": 56660,
   "max": 101225
  },
  "same_82 at 1.16.0": {
   "runs": 82,
   "mean": 73632,
   "median": 72852,
   "min": 56660,
   "max": 101225
  },
  "same_82 at 1.15.0 (iteration 16)": {
   "runs": 82,
   "mean": 73710,
   "median": 72508,
   "min": 63975,
   "max": 104136
  },
  "new_8 (cw/ag/ch/ml)": {
   "runs": 8,
   "mean": 80496,
   "median": 78892,
   "min": 68583,
   "max": 95227
  },
  "tool_calls_mean (tool_use entries)": 7.9,
  "doctor_runs": 7,
  "json_brief_runs": 22,
  "template_runs": 19,
  "per_prompt": {
   "ml2": 68583,
   "ml1": 78320,
   "cw2": 75134,
   "sh1": 71087,
   "ch1": 77676,
   "hi1": 79379,
   "ch2": 85768,
   "th1": 77313,
   "c02": 67275,
   "ag1": 79463,
   "em1": 98997,
   "sh2": 80831,
   "a02": 68574,
   "d01": 71142,
   "d02": 68308,
   "a01": 75252,
   "c01": 72522,
   "dl3": 84957,
   "dl1": 84789,
   "ag2": 95227,
   "dl7": 70694,
   "cw1": 83801,
   "e05-lufs": 70544,
   "e04-sync": 71435,
   "e02-cut-lossless": 73897,
   "dl4": 78398,
   "e06-silence": 73336,
   "e08-surround": 75029,
   "e10-highlights": 70283,
   "e03-logo": 77335,
   "el1": 68487,
   "e11-ready": 73359,
   "e12-vfr": 73469,
   "el2": 57336,
   "f01-missing-input": 66426,
   "e07-hdr": 74899,
   "f02-corrupt-input": 70131,
   "e09-join": 74066,
   "f03-ffmpeg-failure": 69632,
   "f04-beyond-duration": 67513,
   "fr2": 68435,
   "f05-unsupported": 67698,
   "em4": 74646,
   "dl2": 79132,
   "hi2": 67582,
   "it2": 56660,
   "he2": 68920,
   "id2": 68542,
   "em3": 83914,
   "it1": 69454,
   "em2": 83316,
   "id1": 73715,
   "dl5": 77154,
   "fr1": 76773,
   "he1": 80173,
   "dl6": 77300,
   "j02-60s": 70442,
   "dl8": 77636,
   "e01-reel": 82096,
   "j04-bgm": 75779,
   "j07-probe": 68872,
   "k02": 65656,
   "j03-lower": 77629,
   "j05-ma": 76167,
   "k01": 73154,
   "r03-download": 65075,
   "r01-face": 68672,
   "r05-no-raw-ffmpeg": 67790,
   "r02-tts": 72840,
   "p02": 74148,
   "r04-translate": 68062,
   "p01": 80015,
   "s01": 68102,
   "ru1": 71389,
   "th2": 68252,
   "s02": 69341,
   "ru2": 72098,
   "vi2": 68114,
   "x03-audio-trim": 68698,
   "j01-reel": 86088,
   "tr1": 73271,
   "j08-project": 101225,
   "x01-lufs-wav": 69950,
   "x02-wav-to-mp3": 72863,
   "tr2": 75075,
   "vi1": 76265,
   "x05-replace-audio": 70533,
   "x04-transcode": 73421,
   "x06-reverse": 72089,
   "j06-report": 86872
  }
 },
 "comparison_with_iteration_16": {
  "routing": "100% -> 100%",
  "refusal_honesty": "22/22 -> 25/25",
  "report_format": "82/82 -> 89/90 (dl8 'Done (partially):')",
  "language": "Opus 81/82 -> 89/90 (dl4/id1 fixed; the boilerplate Look:/Check: lines on 15 refusals are the new residue)",
  "non_latin_glyphs": "11/11 -> 11/11",
  "delivery": "12/13 -> 11/14 one encode, platform pass 13/13 -> 16/16",
  "quality_mean": "4.68 -> 4.17 (the caption-break runs scored 2-3: cw1 2)",
  "tokens_mean_same_82": "73710 -> 73632 (flat)"
 },
 "findings": [
  "The headline feature of 1.16.0 - phrase-aware caption breaking - does not reach the picture. I opened the stills for all six captioned runs: cw1 stacks 'A / third / line / the / tool' one word per line, dl1 and dl4 reproduce iteration 16's failures byte for byte (the 'A third line the tool' | 'times for me' split; the lone 'subtítulos' and the trailing 'con'), and dl3 still shows a lone digit line '2' plus a mid-word katakana split 'タイ / ミング'. The only clean captioned run, cw2, is clean because the agent rewrote the text short enough that nothing had to wrap.",
  "Thai wrapping REGRESSED. th1 and sh2 both break inside a word - ของ split as 'ข' | 'อง', เวลา as 'เว' | 'ลา' - while caption.py's own stdout reports '3 wrapped, 3 rebalanced, 3 phrase_breaks'. Iteration 16's th1 wrapped at a whole-word boundary. A space-less script needs a dictionary or ICU boundary; the current wrapper appears to fall back to character count and reports success anyway.",
  "The audiogram feature works end to end in both runs. ag1 routed straight to render.py --template audiogram --image and ag2 to waveform.py --image; I sampled frames at 6.0s and 3.0s (not frame 0) and the cover/plate is behind the waveform in both, at the announced 25 fps and the audio's own duration. ag2 is the model answer to 'find a nice image': it refused to fetch or invent one, quoted waveform.py's 'nothing is ever fetched and no cover art is ever invented', and generated a gradient plate with the substitution disclosed twice.",
  "Auto-chapters is correct in both runs and ch2 is the best judgement call in the suite. Markers come from measured pauses via silence detection, all titles are the 'Chapter N' placeholders, and ffprobe -show_chapters on both outputs returns exactly the 3 markers at 0/4/8s that the reports claim. ch2 - asked for subject titles on a file with no speech - delivered the chapters under Done:, refused to invent the titles inside the report with the reason (SMPTE bars, whisper absent per doctor), named the placeholders as placeholders and wrote description.txt for manual renaming. That is the right shape: Failed: would have hidden a real artefact.",
  "Multi-language muxing is exact. ffprobe on ml1's output shows mov_text/eng (default) and mov_text/jpn, and the report volunteers the two genuine MP4 limitations - forced default on the first track and no per-track title field - rather than glossing them.",
  "dl8 regressed on the label rule: 'Done (partially):' is neither Done: nor Failed:, on the same prompt that iteration 15 failed with Done: and iteration 16 fixed with a single Failed:. em2 fails the other way, carrying a single Failed: on a run that actually wrote and verified the overlay. he2 and k02 are still correct.",
  "iteration 16's dl4 language defect is FIXED - dl4 is now Spanish on every label line, and id1 is fully Indonesian. What replaced it is narrower and more systematic: 15 non-English runs leave the canned 'Look: not needed (nothing written)' or 'Check: nothing to verify' in English. The rule landed for authored prose and not for boilerplate.",
  "j06-report, the two-encode holdout of iterations 15 and 16, is fixed: one render.py --template x pass, one encode, and no work directory created or deleted under tests/out. j01-reel moved the other way, hand-writing project.json where iteration 16 used --write-project, and paying two encodes plus two leftover work directories for it.",
  "Agents look but still do not read. Four of the six captioned runs describe a still that visibly contains a broken word or a one-word line as 'readable', 'legible', 'inside the safe area'. cw1 is the sharpest case: the prompt's whole subject was sensible line breaks, and the report calls the result 'the sensible line breaks behavior'."
 ],
 "harness_or_model_issues": [
  "Only rep1 exists, so per-run variance cannot be separated from real change between iterations 16 and 17.",
  "ml2's prompt names no video or audio file, only en.srt, so the muxing it asks for is impossible as written. The agent handled it well, but the prompt needs a video before ml2 can test what it means to test.",
  "cw1's cue file (cues_cw1.txt) mixes an English line with a Spanish one ('Segunda linea de subtitulos', unaccented), which is almost certainly a copy-paste from dl4's fixture. It does not change the grade - the break defect is visible in both languages - but the fixture is worth cleaning.",
  "look.py's contact-sheet tile timestamps still do not line up with the cue in view (em3 and dl3 show cue 2 text under a 00:00:01.999 label). I verified the real frames with my own look.py --at extractions and the burn-in timing is correct, so this remains a sheet-labelling artefact.",
  "Eight runs executed /root/.claude/skills/ffmpeg-skill/scripts rather than the /home/user/ffmpeg-skill/scripts path their prompt named (e01-reel, e03-logo, e04-sync, e05-lufs, fr1, it1, p02, x01-lufs-wav). One of them, e01-reel, is graded in full.",
  "The suite gives two different answers to 'can this skill translate': d02, tr2 and r04-translate refuse it as out of scope, while ml2 performs the translation inline. Whichever answer is intended, the four prompts should agree.",
  "The eval's caption widths are dominated by the platform caption size: at TikTok/Shorts size a 5-7 word cue never fits two lines, so the breaker has no room to choose. A fair measurement of phrase breaking needs a cue that fits two lines at that size, or a caption size that fits the cue; eval 18 should carry both.",
  "Contact sheets sample pop-animated captions mid-animation, so a sheet can show one or two words where the line has more; graders and agents read stills as if they were static."
 ],
 "follow_ups": [
  "1.16.1: never break inside a katakana run; no balancing inside a no-space run without word boundaries (Thai, Lao, Khmer, Myanmar) - greedy fill and a note naming the manual '|' break; caption.py reports when a cue was split because the safe width holds fewer than two lines of it.",
  "1.17.0: caption size that fits the cue (shrink within the platform's legibility floor before splitting a cue), or a per-platform cue length guideline in SKILL.md.",
  "Eval 18: cw cues sized to the platform width; ml2 fixed prompt; SKILL.md label rule wording for 'Done (partially)' if bytes allow."
 ]
}