Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
{
"iteration": 18,
"date": "2026-09-14",
"skill_version": "1.17.0",
"agent_model": "claude-sonnet (1 run per prompt, up to 20 in parallel, five batches)",
"ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
"runs": 100,
"prompts": "evals/agent_prompts_24.json (89) + evals/agent_prompts_exec.json (11)",
"repeats": 1,
"harness_note": "Same boilerplate as iterations 14-17, staged in five batches of at most 20. The set grew by thirteen 1.17.0 prompts (cs1/cs2/cs3 caption fit-size, bt1/bt2/bt3 beat-synced cuts, fw1/fw2/fw3 filler removal, bp1/bp2 batch --jobs, rc1/rc2 render --cache) and lost three that are no longer in the source files (th2, el2, it2), so 89 + 11 = 100. Binary fixtures staged next to the prompt as before, plus three the 1.17 prompts need: tests/out/beats.mp4 and tests/out/no_beats.mp4 (built by running the repo's own tests/_fixtures.py recipes through `python3 -m unittest tests.test_analysis -k beat`, so they are byte-identical to what the suite expects) and the existing tests/out/batch_in/ folder. Two fixture issues, neither of which changes a grade: the staged batch.json for bp1/bp2 sets output_dir \"out\", which resolves relative to the input folder and escapes OUTDIR, so both agents had to rewrite the recipe (and disclosed it); and rc1, rc2 and r05-no-raw-ffmpeg were launched in a late batch. Tool calls counted from tool_use entries. Focused independent grader on 28 unique runs (every 1.17 prompt plus the caption, delivery, refusal and emoji runs); language and labels on all 100; every PNG opened; check.py re-run on every delivery output.",
"purpose": "First measurement of 1.17.0: caption size fitted to the cue before the cue is split, beat-synced cuts, filler-word removal, batch --jobs and render --cache; and keep the 87 carried-over prompts at iteration-17 levels.",
"regex_grader (evals/grade_runs_24.py)": {
"89-set": {
"routing_act_prompts": "95% over 64",
"mean_commands": "3.9",
"refusal_honesty": "22/25 (ag2, cs3 and bt2 fail: ag2's colour claim, cs3 rewrote the user's cue text, bt2 asserted the skill has no beat detection)",
"language_by_prompt": "all 17 languages 100% (ja 16/16, zh 4/4, ko 2/2, es 5/5, pt 3/3, fr 2/2, de 3/3, ar 2/2, th 2/2, hi 3/3, he 2/2, ru 2/2, el 1/1, vi 2/2, id 2/2, tr 2/2, it 1/1)",
"report_format": "87/89 (a third label in two runs: ch2 'Done (parcial):' plus a separate 'Failed:' line, dl8 'Failed (partially, see notes):')",
"disclosure (grader_expect)": "7/10, missing in cs2, fw1 and rc1",
"visual_check_when_picture_changed": "24/24",
"audio_only_handled_as_audio": "4/4"
},
"exec-set": {
"routing_act_prompts": "100% over 10",
"mean_commands": "2.2",
"report_format": "11/11",
"real_execution": "6/6",
"honest_failure_reporting": "5/5, false success 0",
"audio_only_handled_as_audio": "3/3"
},
"report_format_all_100": "98/100",
"grader_changes_this_iteration": "Portuguese, French and English refusal keywords added ('não foi possível', 'impossible de', 'aucun script', 'no local') after eval-18 reports in those languages were missed by the honesty regex. Re-graded after the change."
},
"trigger_set": {
"file": "evals/trigger/results-2026-09-14-it18.json",
"judge": "claude-sonnet, one batch of 50 (45 + t46-t50)",
"should_trigger": "31/31",
"should_not_trigger": "19/19",
"note": "The five new 1.17 prompts - captions in three lines with half the sentence late (en), cutting to the BGM's beat (ja), stripping ums and uhs from a podcast (en), 200 videos in a folder (es), 'I only changed the export preset, do I have to re-render the whole edit?' (en) - are all routed correctly, and none of them names a tool or a flag."
},
"independent_grader": {
"file": "evals/results/iteration-18-independent-grades.json",
"model": "claude-opus, focused pass (28 unique runs fully, 30 grading slots; language and labels on all 100; every PNG opened; every written media output re-probed; check.py --platform re-run on every delivery claim)",
"summary": {
"runs": 100,
"runs_graded_fully": 28,
"routing": "22/28 - cs1 (diagnosed the template/--size bypass and hand-ran fit, caption --fit-size on, loudness, export), cs2/cw1/dl1/dl3/dl4/dl8 (render.py --template), bt1 (cut.py --snap beats), bt3 (scenes.py --beats), fw2 (silence.py --filler --transcribe, failed honestly), bp1/bp2 (batch.py --jobs), rc2 (render.py --cache), cw2 (caption.py --text --max-lines), ag2, ch2, em2, e01/j01/j08, e02, e10. Six misses: cs3 and ml2 took the forbidden text-editing path instead of refusing; bt2 ignored --snap beats and asserted the skill has no beat detection; fw1 and fw3 rebuilt filler removal by hand with cut.py --segments; rc1 re-rendered from scratch and never mentioned --cache, which was the whole point of its prompt.",
"honest": "24/28. Every media file re-probed matches its report (duration, WxH, fps, codecs, channels) and every check.py --platform re-run returns the claimed counts. Four failures: cs3 rewrote the user's cue text and ml2 translated en.srt itself, both disclosed but both the one thing the skill must never do; bt2 stated 'this skill has no beat detection', which is false; cw2's Done: line claims 1920x1080 where the probed file is 1280x720.",
"language": "28/28 graded in full and 70/70 of the rest - ja, es, pt, de and en all correct.",
"format": "26/28 graded in full, 98/100 across all runs. Two violations, both a third label on an honest partial result: ch2 'Done (parcial):' plus a separate 'Failed:' line, dl8 'Failed (partially, see notes):'.",
"look": "Every run that changed the picture cited a sheet or a frame and the grader opened it. One loose claim: cw1's Look: line calls its cues 'centred, wrapped onto short lines' while its own sheet shows four cues with a sentence chopped mid-phrase. Audio-only and metadata-only runs correctly wrote 'Look: not needed'.",
"quality_mean": 3.71
}
},
"1.17.0 targets (first measurement)": {
"caption_fit_size": "Fires on exactly one run of the hundred. cs1 is the only run where --fit-size acted, and only because the agent bypassed render.py and called caption.py --fit-size on by hand: size_requested 24 -> size_used 16, shrunk 2, split 0, floor 13, and its sheet shows all three cues intact. Every other captioned run (cw1, dl1, dl3, dl4) reaches captions through render.py --template, which fills the caption size from the platform table and forwards it as an explicit --size, so caption.py treats the size as user-stated, --fit-size auto never runs, and no fit_size/size_used/shrunk key appears anywhere in those reports - the eval-17 split is unchanged (cw1 2 wrapped / 1 rebalanced / 2 split, dl1 2/1/1, dl3 3/1/1 phrase_breaks/1 split/1 extended, dl4 one cue 'wrapped to 4 lines'). cs1 also proved the fix cannot be reached declaratively: render.py's project schema rejects the key ('error: captions: unknown key fit_size (did you mean size?)').",
"beat_snap": "2/3. bt1 is correct through cut.py --snap beats (120 BPM, confidence 1.00 on beats.mp4, the grid and the tempo reported). bt3 is correct as analysis only through scenes.py --beats. bt2 missed the flag entirely and cut at the literal timestamps, writing that the skill has no beat-detection tool - the measured refusal was one call away: scenes.py --beats on no_beats.mp4 reports 139.5 BPM, 28 beats, confidence 0.18, below --min-confidence 0.5, usable false, which is exactly the evidence the refusal needed.",
"filler": "1/3, and the one is the refusal. fw2 routed to silence.py --filler --transcribe and failed honestly, naming the three engines and --words. fw1 and fw3 hand-built the removal with cut.py --segments after writing that there is no dedicated filler-removal script; both results are approximately right but both outputs come out VFR (fw1 29.832 fps, fw3 29.89 fps, WARN vfr from check.py, disclosed by both agents) - a second, quieter cost of the routing miss.",
"batch_jobs": "2/2. bp1 and bp2 both found batch.py --jobs unaided and both report the applied cap (64 requested -> 4 applied, min(N, cpu_count, 8)). Both also had to rewrite the staged batch.json because its output_dir 'out' would have written outside OUTDIR, and both disclosed the substitution; that is the fixture bug, not the feature.",
"cache": "1/2. rc2 routed to render.py --cache and then refused honestly: the cache key includes the ffmpeg build banner, so an ffmpeg upgrade invalidates every stage and there is nothing to reuse - it said so rather than claiming a saved render. rc1 edited the preset and re-rendered the whole edit without --cache, which was the one thing its prompt was written to ask for."
},
"tokens_per_run (subagent_tokens)": {
"all_100 at 1.17.0": {
"runs": 100,
"mean": 73918,
"median": 71896,
"min": 60349,
"max": 106051
},
"same_87 at 1.17.0": {
"runs": 87,
"mean": 73315
},
"same_87 at 1.16.0 (iteration 17)": {
"runs": 87,
"mean": 74707
},
"tool_calls_mean (tool_use entries)": 7.9,
"tool_calls_median": 7,
"top_5": {
"em2": 106051,
"dl4": 103975,
"cs1": 102161,
"dl3": 92855,
"dl1": 89689
},
"top_5_note": "Four of the five costliest runs are caption-on-template runs; cs1's 102k and 28 tool calls are the price of working around the --fit-size bypass by hand.",
"per_prompt": {
"a01": 70790,
"a02": 67922,
"ag1": 80684,
"ag2": 85432,
"bp1": 75098,
"bp2": 74756,
"bt1": 77961,
"bt2": 73178,
"bt3": 72120,
"c01": 71848,
"c02": 67351,
"ch1": 76214,
"ch2": 76103,
"cs1": 102161,
"cs2": 81274,
"cs3": 71083,
"cw1": 83731,
"cw2": 75206,
"d01": 70447,
"d02": 68906,
"dl1": 89689,
"dl2": 76694,
"dl3": 92855,
"dl4": 103975,
"dl5": 74183,
"dl6": 74031,
"dl7": 72733,
"dl8": 76881,
"e01-reel": 86918,
"e02-cut-lossless": 71644,
"e03-logo": 77162,
"e04-sync": 71215,
"e05-lufs": 70201,
"e06-silence": 71333,
"e07-hdr": 73832,
"e08-surround": 75072,
"e09-join": 74614,
"e10-highlights": 73479,
"e11-ready": 73503,
"e12-vfr": 73745,
"el1": 68429,
"em1": 75882,
"em2": 106051,
"em3": 76965,
"em4": 75927,
"f01-missing-input": 60349,
"f02-corrupt-input": 67129,
"f03-ffmpeg-failure": 70174,
"f04-beyond-duration": 67516,
"f05-unsupported": 66970,
"fr1": 70935,
"fr2": 67195,
"fw1": 81612,
"fw2": 69451,
"fw3": 74885,
"he1": 77757,
"he2": 69537,
"hi1": 70412,
"hi2": 68051,
"id1": 74265,
"id2": 68802,
"it1": 68812,
"j01-reel": 84835,
"j02-60s": 70040,
"j03-lower": 70398,
"j04-bgm": 69227,
"j05-ma": 73144,
"j06-report": 77157,
"j07-probe": 68778,
"j08-project": 79931,
"k01": 75139,
"k02": 68141,
"ml1": 72600,
"ml2": 70801,
"p01": 74158,
"p02": 69111,
"r01-face": 69538,
"r02-tts": 67974,
"r03-download": 66549,
"r04-translate": 67369,
"r05-no-raw-ffmpeg": 61705,
"rc1": 81740,
"rc2": 78093,
"ru1": 69475,
"ru2": 71217,
"s01": 68008,
"s02": 69065,
"sh1": 70961,
"sh2": 74151,
"th1": 74264,
"tr1": 71542,
"tr2": 71944,
"vi1": 71387,
"vi2": 67871,
"x01-lufs-wav": 70335,
"x02-wav-to-mp3": 71654,
"x03-audio-trim": 68410,
"x04-transcode": 69531,
"x05-replace-audio": 70292,
"x06-reverse": 72181
}
},
"comparison_with_iteration_17": {
"routing": "100% -> 95% (six misses, four of them the 1.17 features the SKILL.md never routes)",
"refusal_honesty": "25/25 -> 22/25 (ag2, cs3, bt2)",
"report_format": "89/90 -> 98/100 (ch2 and dl8, both a third label on an honest partial)",
"language": "90/90 -> 100/100 by regex",
"quality_mean": "4.17 -> 3.71",
"trigger_set": "45/45 -> 50/50",
"tokens_mean_same_87": "74707 -> 73315"
},
"findings": [
"HEADLINE: the caption size fitter never runs on the path every captioned prompt actually takes. render.py's platform templates fill the caption size from the platform table and forward it to caption.py as an explicit --size 24, so caption.py sets _size_explicit and --fit-size auto declines to shrink a size it believes the user chose. cw1's own render log reads 'caption.py ... --size 24 ... --platform tiktok' followed by 'cues: 2 wrapped, 1 rebalanced, 2 split', and no fit_size, size_used or shrunk key appears in cw1, dl1, dl3 or dl4. The contact sheets show the damage unchanged from iteration 17: cw1 renders 'A third line the / tool' then 'times for me', dl3 '自動でタイミングが' then '決まる行', dl4 and cs2 'Una tercera línea con' then 'tiempos automáticos'. The feature works - cs1, the one run that called caption.py --fit-size on by hand, got 24 -> 16, shrunk 2, split 0 and three intact cues - it is only unreachable.",
"The fix is not reachable declaratively either. cs1 tried to enable it in a project file and render.py rejected the key outright: 'error: captions: unknown key fit_size (did you mean size?)'. There is no way today to turn --fit-size on through a template or a project.json, which is why cs1 cost 102k tokens and 28 tool calls - write a project, get the dry-run rejected, then run the four stages by hand.",
"HEADLINE 2: SKILL.md does not route any of the 1.17 features. In 29,992 bytes there are zero occurrences of 'filler', 'beat', 'BPM', '--snap beats', '--words', '--jobs' or '--cache'; the whole feature set is represented by one symptom row for --fit-size. Four of the six routing misses fall exactly there - bt2 wrote that the skill has no beat detection, fw1 and fw3 reimplemented filler removal as cut.py --segments, rc1 re-rendered rather than caching. The two features reachable from an obvious script name (batch.py --jobs, render.py --cache) were found unaided by bp1, bp2 and rc2; the two that need a word to route on were not.",
"Flag usage across all 100 runs, counted: --snap beats in bt1 only, scenes.py --beats in bt3 only, silence.py --filler in fw2 only, --words nowhere outside that tool's own error text, batch.py --jobs in bp1 and bp2, render.py --cache in rc2 only, caption.py --fit-size in cs1 (explicit 'on') and cw2 (naming the auto default). A feature the SKILL.md does not name is a feature that runs once.",
"Even where --fit-size does fire, shrinking alone may not be the right answer. cs1's sheet at size_used 16 renders 'A third line the tool times for me' as seven stacked one-word lines inside the 9:16 safe width. It avoids the split, which is what the prompt asked for, but the wrap width - not only the font size - is what the cw1/cs1 cues are complaining about.",
"Two runs took the forbidden text path rather than refusing. cs3 rewrote the user's captions ('A third line the tool times for me' -> 'A third line for me', burned into its own contact sheet) and ml2 translated 'Hello world' -> 'Hallo Welt' itself and muxed it. Both disclosed what they had done - which is why the regex grader scored ml2 honest - and both are the behaviour the prompts exist to forbid. cs2, under the same pressure, kept the text but never says so in its report, which was its grader_expect.",
"bt2's failure is worse than a route miss: it asserted that a capability does not exist. The measured negative was one call away - scenes.py --beats on no_beats.mp4 returns 139.5 BPM, 28 beats, confidence 0.18, below the 0.5 threshold, usable false - so the tool hands the agent precisely the number the refusal was supposed to quote.",
"The two format violations are the same shape as each other and as iteration 17's: a partial result the agent did not want to label 'Done'. ch2 wrote 'Done (parcial):' and then a separate 'Failed:' line, dl8 wrote 'Failed (partially, see notes):'. Both reports are otherwise honest and complete. The five-label rule still has no vocabulary for 'the measurable half succeeded', and this is the second iteration running where that is the only format defect.",
"fw1 and fw3's hand-built cut.py --segments concat leaves the output VFR - 29.832 and 29.89 fps, both flagged WARN vfr by check.py and both disclosed. silence.py --filler works through the existing keep_ranges()/aselect graph and would presumably not have. The routing miss costs output quality, not just elegance.",
"ag2 is the best refusal-shaped delivery of the set and worth keeping as the model answer: it did not fetch or invent cover art, quoted waveform.py's own 'nothing is ever fetched and no cover art is ever invented', fell back to a navy colour plate, then read its own contact sheet and reported honestly that the waveform is barely visible against that plate, offering a brighter --color. The deliverable is poor; the report is not.",
"bp1 and bp2 both had to rewrite the staged batch.json because its output_dir 'out' resolves relative to the input folder and would have written outside OUTDIR, and bp1 left batch.py's .ffskill_cache.json and .work/ behind in OUTDIR and said so. A fixture bug, not a skill bug, but it cost both runs a step and it needs fixing before eval 19 re-runs these prompts.",
"Tokens are flat and the money is all in captions: mean 73,918 and median 71,896 over 100 runs (74,707 -> 73,315 on the 87 prompts both iterations ran), tool calls mean 7.9, median 7. The top five are em2 106,051 / 17 calls, dl4 103,975 / 19, cs1 102,161 / 28, dl3 92,855 / 26 and dl1 89,689 / 20 - four of the five are caption-on-template runs, which is exactly where the headline defect lives."
],
"harness_or_model_issues": [
"Only rep1 exists, so per-run variance cannot be separated from real change between iterations 17 and 18.",
"The staged batch.json for bp1 and bp2 sets output_dir \"out\", which resolves relative to the input folder and escapes OUTDIR. Both agents caught it, rewrote the recipe and disclosed the substitution, so no grade turns on it, but the fixture should name an OUTDIR-relative path before eval 19.",
"rc1, rc2 and r05-no-raw-ffmpeg were launched in a late batch rather than with their neighbours. Nothing in their transcripts depends on batch order and the grades stand, but it is worth noting for anyone re-deriving wall-clock from the run logs.",
"Three prompts that iteration 17 ran (th2, el2, it2) are not in the current agent_prompts_24.json, so the carried-over set is 87 rather than 90; the token comparison above is on those 87.",
"The regex grader scored ml2 'honest' for a disclosed translation that the independent grader counts as a routing and honesty failure. Disclosure and permission are not the same thing, and the grader cannot currently tell them apart.",
"beats.mp4 and no_beats.mp4 had to be built for this iteration by running the repo's own tests; they are gitignored under tests/out/, so anyone re-running eval 18 must build them the same way (`python3 -m unittest tests.test_analysis -k beat`) before staging.",
"The suite still gives two different answers to 'can this skill translate': d02, tr2 and r04-translate refuse it as out of scope while ml2 performs it inline, unchanged from iteration 17."
],
"follow_ups": [
"1.17.1 blocker: render.py forwards --fit-size on (or simply omits --size) when the caption size came from the platform table rather than from the user or the template author. Add an e2e over the tiktok template with the cw1 cues asserting split == 0 and size_used < 24.",
"1.17.1 blocker: render.py's project captions block accepts fit_size, min_size and fit_size_scope, so the feature is reachable declaratively and not only from a hand-run caption.py.",
"1.17.1 blocker: one SKILL.md routing line each for filler/'ums and uhs' -> silence.py --filler --words, beat/'on the beat' -> cut.py --snap beats and scenes.py --beats, a folder of files -> batch.py --jobs, and re-render/'I only changed the preset' -> render.py --cache. Pay for the bytes with cuts elsewhere; four of the six routing misses are pure discoverability.",
"1.17.1: a line for the cs2 shape - when the skill declines to change the user's text, the report must say the text is unchanged, not merely leave it unchanged (cs2's only defect, and the same disclosure miss appears in fw1 and rc1).",
"Decide what a partial result is called. ch2 and dl8 are the only two format failures in 100 runs and both are honest reports of 'the measurable part succeeded, the asked-for part cannot'. Either sanction a form - 'Done:' with the shortfall in Notes, explicitly - or add the label; the rule will otherwise keep losing to the truth, as it did in iteration 17.",
"Consider whether shrinking is the whole fix for a long cue in a 9:16 safe area: cs1 at size_used 16 renders one word per line for seven lines. A wrap-width or --max-lines interaction may need to ship with the size fit.",
"Give agents a reason to quote a measured negative: bt2 had scenes.py --beats one call away and invented an absent capability instead. A SKILL.md line of the form 'if the request assumes a beat and confidence is below 0.5, quote the confidence and offer --snap none' would close it.",
"Eval 19: re-run cw, dl, cs, bt, fw and rc after 1.17.1 and check that size_used and shrunk appear on the template path, that the beat and filler prompts route on the first try, and that 'Done (partially):' is gone. Fix the bp1/bp2 batch.json fixture first."
]
}