Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

9.3 KB
Raw Blame
{
 "iteration": 15,
 "date": "2026-09-13",
 "skill_version": "1.14.0",
 "agent_model": "claude-sonnet (1 run per prompt, 12-13 in parallel, six batches)",
 "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
 "runs": 76,
 "prompts": "evals/agent_prompts_24.json (65) + evals/agent_prompts_exec.json (11)",
 "repeats": 1,
 "harness_note": "Same 76-prompt set and boilerplate as iteration 14, run against the released 1.14.0. Tool calls counted from tool_use entries in the agent transcripts. Focused independent grader on the 13 delivery-shaped runs plus 13 picture/language runs (every PNG opened); language and labels on all 76.",
 "purpose": "Measure what 1.14.0's delivery templates change on the delivery prompts (route taken, number of encodes, platform check outcome) and keep the stable set at iteration-14 levels.",
 "regex_grader (evals/grade_runs_24.py)": {
  "65-set": {
   "routing_act_prompts": "100% over 43",
   "mean_commands": "4.0",
   "refusal_honesty": "22/22",
   "language_by_prompt": "all 17 languages 100% (ja 12/12, zh 3/3, ko 2/2, es 3/3, pt 2/2, fr 2/2, de 2/2, ar 2/2, th 2/2, hi 2/2, he 2/2, ru 2/2, el 2/2, vi 2/2, id 2/2, tr 2/2, it 2/2)",
   "report_format": "65/65",
   "visual_check_when_picture_changed": "18/18",
   "audio_only_handled_as_audio": "4/4"
  },
  "exec-set": {
   "routing_act_prompts": "100% over 10",
   "mean_commands": "2.4",
   "real_execution": "6/6",
   "honest_failure_reporting": "5/5, false success 0",
   "audio_only_handled_as_audio": "3/3"
  },
  "grader_changes_this_iteration": "Delivery and reel prompts now accept render.py as the check (a template or project run embeds check.py); refusal keywords 'none of', 'forbid', 'не переводит', 'не выполн' and Indonesian/Turkish stopwords added after four honest reports were missed (r05, ru2, id1, tr1), then re-graded."
 },
 "trigger_set": {
  "file": "evals/trigger/results-2026-09-13-it15.json",
  "judge": "claude-sonnet, one batch of 38",
  "should_trigger": "22/22",
  "should_not_trigger": "16/16"
 },
 "independent_grader": {
  "file": "evals/results/iteration-15-independent-grades.json",
  "model": "claude-opus, focused pass (26 runs fully: 13 delivery-shaped + 13 picture/language; language and labels on all 76; every PNG opened; check.py re-run on every delivery output)",
  "summary": {
   "runs": 76,
   "runs_graded_fully": 26,
   "routing": "26/26 - correct script for every request (render.py --template for delivery, caption.py for burn-in, graphics.py for lower-thirds, color.py --to-sdr, audio.py --replace, loudness.py+metadata.py for the podcast). No raw ffmpeg/ffprobe anywhere in any of the 76 run.md files.",
   "honest": "26/26 - every produced file re-probes as reported; no Done: is contradicted by disk. dl8's prose is honest about the upload being impossible (its LABEL is the defect, counted under format).",
   "language": "75/76 - every final report is in the prompt's language except dl4 (es prompt), whose Done:/Steps:/Check: lines are in English and only Look:/Notes: in Spanish.",
   "format": "74/76 - dl8 carries 'Done:' on a refuse-the-verb prompt where 'Failed: (upload impossible)' is the correct label; he2 emits BOTH 'Done: (nothing rendered)' and 'Failed:' (double label, same defect k02 had in iteration 14). All 76 carry the five labels.",
   "look_when_picture_changed": "23/23 - every run that changed the picture wrote and read a still or contact sheet; dl7, e11 and x05-replace-audio correctly say 'Look: not needed' for audio-only changes.",
   "visual_check_non_latin": "10 of 11 non-Latin text runs render real, correctly shaped glyphs (th1 Thai marks, he1 Hebrew RTL, a01 Arabic joined forms + RTL, c01 Chinese, k01 Hangul, dl3 Japanese, j03 Japanese, vi1 Vietnamese diacritics, id1/fr1/p01 Latin). hi1 Devanagari is genuinely garbled - confirmed from the PNG and disclosed by the agent.",
   "platform_checked_delivery": "13/13 delivery-shaped outputs pass check.py --platform X with 0 FAIL and 0 WARN when the grader re-runs it (iteration 14: 7 of 8).",
   "encodes_delivery": "12 of 13 delivery runs took exactly one encoding pass (e11 took zero video encodes); only j06-report took two. Iteration 14: 4 of 7 producing delivery runs took two.",
   "quality_mean": 4.58,
   "quality_min": 3
  }
 },
 "delivery_prompts (second measurement, first at 1.14.0)": {
  "route": "12 of 13 delivery-shaped runs went through render.py --template (or a template-written project); j06 used export.py directly",
  "encodes": "12/13 one encode (iteration 14: 3/7); j06 two (export.py without --normalize, then with)",
  "platform_check": "13/13 pass check.py --platform X with 0 FAIL on the grader's own re-run (iteration 14: 7/8; dl6 LinkedIn's loudness FAIL is gone)",
  "dl8": "upload refused honestly and the TikTok-ready file produced via the template, real 9:16 crop instead of iteration 14's letterbox; still labelled Done: (label rule queued for 1.15.0)",
  "tokens_dl": {
   "runs": 8,
   "mean": 80368,
   "median": 76013,
   "min": 73179,
   "max": 90441
  },
  "tokens_dl_iteration_14": {
   "runs": 8,
   "mean": 80108,
   "median": 78622,
   "min": 73076,
   "max": 88680
  }
 },
 "tokens_per_run (subagent_tokens)": {
  "all_76 at 1.14.0": {
   "runs": 76,
   "mean": 73356,
   "median": 72043,
   "min": 65066,
   "max": 97099
  },
  "all_76 at 1.13.0 (iteration 14)": {
   "runs": 76,
   "mean": 72724,
   "median": 71754,
   "min": 57519,
   "max": 88680
  },
  "tool_calls_mean (tool_use entries)": 7.4,
  "doctor_runs": 6,
  "json_brief_runs": 31,
  "template_runs": 15,
  "per_prompt": {
   "dl7": 73179,
   "e11-ready": 71899,
   "dl2": 73743,
   "dl5": 74209,
   "dl6": 76095,
   "j06-report": 80728,
   "e01-reel": 97099,
   "j01-reel": 88718,
   "j08-project": 84998,
   "dl4": 90441,
   "dl1": 90044,
   "dl8": 75931,
   "dl3": 89304,
   "c02": 67716,
   "a02": 67983,
   "d01": 70039,
   "a01": 73798,
   "d02": 74062,
   "e05-lufs": 70856,
   "c01": 78034,
   "e02-cut-lossless": 70929,
   "e06-silence": 72821,
   "e08-surround": 73204,
   "e03-logo": 78468,
   "e07-hdr": 74585,
   "e04-sync": 80583,
   "f01-missing-input": 72187,
   "el1": 67942,
   "el2": 65066,
   "f02-corrupt-input": 67313,
   "f04-beyond-duration": 67140,
   "e12-vfr": 72401,
   "f03-ffmpeg-failure": 69920,
   "f05-unsupported": 67192,
   "fr2": 66931,
   "e10-highlights": 72422,
   "e09-join": 78235,
   "fr1": 76174,
   "he1": 75319,
   "he2": 69299,
   "hi2": 67576,
   "it1": 67901,
   "it2": 67270,
   "id2": 69497,
   "id1": 73487,
   "hi1": 75058,
   "j04-bgm": 71040,
   "j07-probe": 69544,
   "j03-lower": 79331,
   "j05-ma": 73831,
   "j02-60s": 74425,
   "k02": 68056,
   "k01": 76868,
   "r02-tts": 67668,
   "r03-download": 67273,
   "p02": 70981,
   "r04-translate": 72389,
   "p01": 73513,
   "r05-no-raw-ffmpeg": 70676,
   "s01": 67479,
   "r01-face": 71690,
   "s02": 68360,
   "ru1": 70291,
   "ru2": 71712,
   "th2": 67932,
   "tr1": 71508,
   "vi2": 68063,
   "x03-audio-trim": 68354,
   "th1": 75581,
   "x01-lufs-wav": 70176,
   "x02-wav-to-mp3": 71599,
   "x06-reverse": 70613,
   "vi1": 81454,
   "x04-transcode": 72652,
   "x05-replace-audio": 70794,
   "tr2": 73398
  }
 },
 "comparison_with_iteration_14": {
  "routing": "76/76 (same)",
  "honest": "76/76 regex, 26/26 Opus (was 75/76: dl8 label still Done: but no longer counted as dishonest by the grader; the label rule ships in 1.15.0)",
  "format": "76/76 regex, 74/76 Opus (he2 double label, dl4 mixed labels; was 75/76)",
  "language": "76/76 regex, 75/76 Opus (dl4: three of five labels in English on a Spanish request)",
  "look": "18/18 regex, 23/23 Opus",
  "tokens_mean_all_76": "73,356 (was 72,724)",
  "trigger": "38/38 (same)",
  "quality_mean": "4.58 over 26 (was 4.65 over 26 different runs)",
  "delivery_one_encode": "12/13 (was 3/7)",
  "delivery_platform_pass": "13/13 (was 7/8)"
 },
 "findings": [
  "1.14.0 did what it was built for: delivery requests route to render.py --template, finish in one encode, and pass their platform check, including the LinkedIn case that failed loudness at 1.13.0.",
  "The remaining two-encode run (j06, X export with an HTML report) named a platform but did not use a template; the SKILL.md delivery row should say that a named platform means --normalize in the same export.",
  "Stable 50 unchanged: routing, honesty, format, language, tokens flat (73.4k vs 72.7k).",
  "Devanagari through graphics.py is still wrong-shaped at 1.14.0 (expected; 1.15.0 routes shaping scripts through libass); the agent disclosed it again.",
  "Caption wrap defects seen in iteration 14 recur (orphan single characters in th1 and dl3, mid-phrase breaks in dl1 and dl4); queued in 1.15.0.",
  "Labels: dl8 Done: on a refused upload, he2 both Done: and Failed:; the one-sentence label rule ships in 1.15.0.",
  "dl4 mixed English and Spanish across the five labels: the only language drift in 76 runs.",
  "Four runs invoked scripts via ~/.claude/skills/ffmpeg-skill (symlink to the repo; executed HEAD)."
 ],
 "harness_or_model_issues": [
  "One pass per prompt; the delivery encode count is now measured on 13 runs and is deterministic given the route, so repeats are not needed for that claim."
 ],
 "follow_ups": [
  "1.15.0 (in progress): label rule, caption wrap, Devanagari shaping, emoji.",
  "SKILL.md delivery row: named platform means --normalize in the same export (fits the 1.15.0 byte budget only if a trim is found; otherwise 1.16.0).",
  "evals: keep the 76 set; iteration 16 adds emoji/shaping prompts from the 1.15.0 spec."
 ]
}