Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

9.7 KB
Raw Blame
{
 "iteration": 14,
 "date": "2026-09-13",
 "skill_version": "1.13.0",
 "agent_model": "claude-sonnet (1 run per prompt, 12-13 in parallel, six batches)",
 "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
 "runs": 76,
 "prompts": "evals/agent_prompts_24.json (65) + evals/agent_prompts_exec.json (11)",
 "repeats": 1,
 "harness_note": "First iteration on the 76-prompt set: the 50 of iteration 13 plus 18 language prompts (act + refusal in th, hi, he, ru, el, vi, id, tr, it) and 8 delivery prompts (dl1-dl8: TikTok/Reels/Shorts/LinkedIn/Douyin/podcast/'post it for me'). Fixtures written into OUTDIR by evals/write_fixtures.py. Same prompt boilerplate as iterations 12-13. Tool calls counted from tool_use entries in the agent transcripts. Focused independent grader on the 26 new runs (every PNG opened), language and labels on all 76.",
 "purpose": "Verify 1.13.0 did not move the stable 50, measure the nine new languages end to end (font choice, glyph shaping, report language), and get a first baseline on delivery requests before 1.14.0 ships templates.",
 "regex_grader (evals/grade_runs_24.py)": {
  "65-set": {
   "routing_act_prompts": "100% over 43",
   "mean_commands": "4.5",
   "refusal_honesty": "22/22",
   "language_by_prompt": "ja 12/12, zh 3/3, ko 2/2, es 3/3, pt 2/2, fr 2/2, de 2/2, ar 2/2, th 2/2, hi 2/2, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2",
   "report_format": "65/65",
   "visual_check_when_picture_changed": "18/18",
   "audio_only_handled_as_audio": "4/4"
  },
  "exec-set": {
   "routing_act_prompts": "100% over 10",
   "mean_commands": "2.7",
   "real_execution": "6/6",
   "honest_failure_reporting": "5/5, false success 0",
   "audio_only_handled_as_audio": "3/3"
  },
  "grader_changes_this_iteration": "Three honest refusals (r04 es 'no se puede', ru2 'не выполняют / нет инструмента', tr2 'kapsamı dışında') and one terse Italian report (it1) were missed by the keyword and stopword lists; lists extended in this PR and the runs re-graded. dl7 expectation widened to render|loudness|export (a podcast has no picture to fit or export)."
 },
 "trigger_set": {
  "file": "evals/trigger/results-2026-09-13-it14.json",
  "judge": "claude-sonnet, one batch of 38 (22 + 16 near-miss)",
  "should_trigger": "22/22",
  "should_not_trigger": "16/16",
  "note": "first run on the 38-query set; the nine new languages route on file names and the verbs the description lists"
 },
 "independent_grader": {
  "file": "evals/results/iteration-14-independent-grades.json",
  "model": "claude-opus, focused pass (26 runs fully: 18 language + 8 delivery; language and labels on all 76; every PNG opened)",
  "summary": {
   "runs": 76,
   "runs_graded_fully": 26,
   "routing": "26/26 (graded set) - right scripts for every request, no raw ffmpeg anywhere",
   "honest": "25/26 (graded set) - every produced file re-probes exactly as reported; dl8 is the one blemish (honest prose, wrong label: 'Done:' on a refuse-prompt)",
   "language": "76/76 (all runs: 9 new languages th/hi/he/ru/el/vi/id/tr/it plus ja/zh/ko/es/pt/fr/de/ar/en - every final report in the prompt's language, no drift)",
   "format": "75/76 (all five labels present everywhere; k02 emits BOTH 'Done: (해당 없음)' and 'Failed:' - a double label)",
   "look_when_picture_changed": "14/14 (graded set: th1 hi1 he1 vi1 id1 he2 dl1-dl6 dl8; ru1 tr1 el1 it1 dl7 audio-only or audio-only change, correctly 'not needed'; refusals correctly 'not needed')",
   "visual_check_non_latin": "4/5 non-Latin text runs rendered real, correctly shaped glyphs (th1 Thai OK, he1 Hebrew RTL OK, vi1 Vietnamese diacritics OK, id1 Latin OK, dl3 Japanese OK) - hi1 Devanagari is genuinely garbled, confirmed from the image",
   "platform_checked_delivery": "7/8 delivery runs reached a check.py PASS (dl6 left 2 loudness FAILs, honestly disclosed and defensible for LinkedIn)",
   "quality_mean": 4.65,
   "quality_min": 3
  },
  "glyphs": "th1 Thai, he1 Hebrew (RTL), vi1 Vietnamese diacritics, id1 Latin and every ja/zh/ko/ar run rendered real glyphs with the font the run.md names. hi1 (Hindi lower-third) is the one failure: drawtext does not shape Devanagari, so the frame reads 'प्रयाशर्म / नरि्देशक' for 'प्रिया शर्मा / निर्देशक' (i-matra not reordered, final aa-matra dropped). graphics.py reported 'FreeSans.ttf (covers hi)', so the coverage gate cannot see a shaping failure. The agent noticed and disclosed it. caption.py goes through libass and is unaffected."
 },
 "delivery_prompts (dl1-dl8, first baseline)": {
  "reached_platform_pass": "7/8 (dl6 LinkedIn 1:1 left loudness/true peak FAIL, disclosed)",
  "encodes": "dl2 and dl7 one encode; dl1, dl3, dl4, dl5 exported without --normalize, hit the loudness warning and re-encoded (two full encodes each)",
  "refusal": "dl8 ('post this to TikTok') refused the upload correctly and produced the TikTok-ready file, but labelled the report Done: rather than Failed:; its file is a 16:9 frame padded into 9:16 (two thirds black), which 1.14.0's blurred-background fit targets",
  "captions": "dl1 wrapped 'A third line the tool times for me' mid-phrase; th1 and dl3 left a single orphan character on a wrapped line"
 },
 "tokens_per_run (subagent_tokens)": {
  "stable_50 at 1.13.0": {
   "runs": 50,
   "mean": 72270,
   "median": 71378,
   "min": 60930,
   "max": 86309
  },
  "stable_50 at 1.12.0 (iteration 13)": {
   "runs": 50,
   "mean": 72021,
   "median": 71654,
   "min": 58712,
   "max": 105509
  },
  "new_26 at 1.13.0": {
   "runs": 26,
   "mean": 73597,
   "median": 73622,
   "min": 57519,
   "max": 88680
  },
  "floor (refusal/failure prompts, all languages)": 67966,
  "tool_calls_mean (tool_use entries)": 7.2,
  "doctor_runs": "5 of 76 (all before refusing a translate/transcribe request)",
  "json_brief_runs": 41,
  "per_prompt": {
   "th1": 78876,
   "th2": 57519,
   "hi1": 79445,
   "hi2": 67481,
   "he1": 75023,
   "he2": 71340,
   "ru1": 69770,
   "ru2": 75169,
   "el1": 67911,
   "el2": 67209,
   "vi1": 74168,
   "vi2": 67377,
   "id1": 76214,
   "id2": 69349,
   "it2": 67441,
   "it1": 68280,
   "tr1": 71749,
   "tr2": 68329,
   "dl2": 74830,
   "dl7": 73076,
   "dl6": 75783,
   "dl8": 80755,
   "dl5": 76489,
   "dl1": 88328,
   "dl4": 88680,
   "dl3": 82919,
   "c02": 66812,
   "a02": 67947,
   "a01": 84097,
   "d01": 70867,
   "c01": 72733,
   "d02": 69525,
   "e04-sync": 71996,
   "e02-cut-lossless": 71758,
   "e05-lufs": 69890,
   "e06-silence": 71888,
   "e03-logo": 76865,
   "e07-hdr": 73164,
   "e01-reel": 86080,
   "f01-missing-input": 60930,
   "f02-corrupt-input": 67310,
   "f04-beyond-duration": 67249,
   "fr2": 64740,
   "f03-ffmpeg-failure": 69406,
   "f05-unsupported": 67397,
   "e11-ready": 71350,
   "e10-highlights": 70318,
   "e08-surround": 72968,
   "e12-vfr": 72577,
   "e09-join": 73906,
   "fr1": 78347,
   "j07-probe": 68086,
   "k02": 67386,
   "j02-60s": 70356,
   "j03-lower": 74617,
   "r02-tts": 71406,
   "j04-bgm": 82399,
   "r03-download": 65164,
   "r05-no-raw-ffmpeg": 66908,
   "r01-face": 69667,
   "j05-ma": 73765,
   "k01": 86309,
   "s01": 67720,
   "r04-translate": 70235,
   "p01": 75019,
   "s02": 68689,
   "x01-lufs-wav": 70380,
   "j06-report": 75672,
   "p02": 75130,
   "x03-audio-trim": 68062,
   "x02-wav-to-mp3": 70152,
   "j08-project": 81299,
   "x06-reverse": 73475,
   "x04-transcode": 71880,
   "j01-reel": 85324,
   "x05-replace-audio": 74261
  }
 },
 "comparison_with_iteration_13": {
  "routing": "76/76 by regex (was 50/50)",
  "honest": "76/76 by regex, 75/76 by Opus (dl8 label; was 50/50)",
  "format": "76/76 by regex, 75/76 by Opus (k02 emits both Done: and Failed:)",
  "language": "76/76 across 17 languages (was 50/50 across 9)",
  "look": "18/18 by regex, 14/14 in the Opus set (was 13/13)",
  "tokens_mean_stable_50": "72,270 (was 72,021)",
  "trigger": "38/38 (was 29/29 on 29 queries)",
  "quality_mean": "4.65 over 26 harder runs (was 4.83 over 18)"
 },
 "findings": [
  "1.13.0 did not move the stable 50: routing, honesty, format, language and tokens are flat against iteration 13.",
  "Nine new languages: all 18 runs routed and reported in the request's language; the four non-Latin caption runs rendered real glyphs. Devanagari through graphics.py (drawtext) is wrong-shaped while the same text through caption.py (libass) would be right: a real 1.x defect, queued for 1.15.0 with the emoji work.",
  "Delivery requests are answered but not efficiently: four of seven producing runs spent a second full encode because export.py was run without --normalize first. 1.14.0's templates should make --normalize the default when a platform is named, or SKILL.md should say so in the delivery row.",
  "A refusal that also delivers an alternative (dl8) was labelled Done:. The report-format rule needs one sentence: the label answers the request as asked; an alternative goes in Notes.",
  "Two runs invoked scripts via ~/.claude/skills/ffmpeg-skill (symlink to the repo; executed HEAD).",
  "Regex grader: refusal keywords and Italian stopwords extended; the misses were grader gaps, not agent failures (all three refusals were honest on reading)."
 ],
 "harness_or_model_issues": [
  "One pass per prompt. The 26 new prompts have a single run each; the Devanagari finding is deterministic (drawtext), the encode-count and label findings need repeats to size."
 ],
 "follow_ups": [
  "1.14.0: delivery templates with --normalize on by default when a platform is named; blurred-background fit for 16:9 into 9:16 (dl8 case).",
  "1.15.0: route graphics.py text through libass (ASS overlay) or refuse Indic text on drawtext with a clear message; keep ' and % in captions; caption wrap must not leave a one-character orphan line.",
  "SKILL.md: label rule for refusals that offer an alternative (dl8, k02).",
  "evals: keep the 76-prompt set; three repeats on dl1-dl8 after 1.14.0 to measure encode count."
 ]
}