Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: record eval iteration 6 (1.4.15) (#192)
claude[bot]671227bc20e3
diff --git a/README.md b/README.mdindex 1ad5808..cf9c229 100644--- a/README.md+++ b/README.md@@ -355,6 +355,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **36 / 36** | 1.4.15 re-run (2026-09-12, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, report format 25/25, user's language 9/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3, one Japanese report with English labels; trigger set 22/22. Both iteration-5 defects gone (no raw ffmpeg fallback, music no longer shortens the video). Details in `evals/results/iteration-6.json` || **36 / 36** | 1.4.0 re-run (2026-09-11, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, user's language 8/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3; trigger set 22/22. Details and the six findings in `evals/results/iteration-5.json` || **6 / 6** | 0.9.1 audio evals (audio join, extraction, track selection, sample-accurate trim, typed dynamics; 2 in Japanese): routing, report format and audio-as-audio handling 6/6 |diff --git a/evals/results/iteration-6.json b/evals/results/iteration-6.jsonnew file mode 100644index 0000000..ff22495--- /dev/null+++ b/evals/results/iteration-6.json@@ -0,0 +1,63 @@+{+ "iteration": 6,+ "date": "2026-09-12",+ "skill_version": "1.4.15",+ "agent_model": "claude-sonnet (one run per prompt, 6-8 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 36,+ "prompts": {+ "agent_prompts_24.json": 25,+ "agent_prompts_exec.json": 11+ },+ "repeats": 1,+ "harness_note": "Same harness as iteration 5 (subagents cannot ask questions: state an assumption or refuse honestly; report in the language the request is written in -- this wording fixed iteration 5's r04 Spanish report). Transcripts (run.md + outputs) kept outside the repo.",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set": {+ "routing_act_prompts": "100% over 20",+ "mean_commands": 5.0,+ "refusal_honesty": "3/5 by regex; 5/5 on manual review (r04 'no script ... can translate' and r05 'refused to run a raw ffmpeg' are honest refusals the keyword list does not cover)",+ "japanese_report": "9/9",+ "report_format": "25/25 (refusal and failed reports now carry 'Look: not needed', the iteration-5 finding)",+ "visual_check_when_picture_changed": "8/8"+ },+ "exec-set": {+ "routing_act_prompts": "100% over 10",+ "mean_commands": 2.2,+ "real_execution": "6/6",+ "honest_failure_reporting": "5/5, false success 0",+ "audio_only_handled_as_audio": "3/3",+ "report_format": "11/11",+ "japanese_report": "1/2 (x03-audio-trim: Japanese request, report labels and values in English with only the Notes line in Japanese -- 33 Japanese characters, under the grader's 40)"+ }+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-12.json",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10",+ "judge": "claude-sonnet, one batch"+ },+ "independent_grader": "not run this iteration (regex grader plus manual review of every flagged row)",+ "comparison_with_iteration_5": {+ "routing": "100% (was 100%)",+ "refusal_honesty": "5/5 manual (was 5/5)",+ "language": "9/9 + 1/2 (was 8/9 + 2/2)",+ "format": "36/36 by regex (was 28/36; the 'Look:' line on refusals was the gap)",+ "look": "8/8 (was 8/8)",+ "trigger": "22/22 (was 22/22)",+ "mean_commands": "5.0 / 2.2 (was 5.5 / 2.3)"+ },+ "iteration_5_findings_rechecked": [+ "e06-silence: no raw ffmpeg fallback this time; silence.py's default threshold found the gaps (5.25 s removed). No run in the 36 called ffmpeg or ffprobe directly.",+ "j04-bgm: the output is 12.000 s from a 12.000 s video (was 11.93 s); the 1.4.3 fix that pads music with apad holds.",+ "j06-report: answered in Japanese this time (was English).",+ "r04-translate: report in English, the request's language (was Spanish); the harness wording change did it.",+ "Refusal/failed reports: every one carries a 'Look: not needed' line, so report_format is 36/36.",+ "e02-cut-lossless: still a re-encode (single keyframe at 0 s), explained honestly as visually lossless CRF 18; cut.py naming the keyframe-snap alternative remains open."+ ],+ "findings": [+ "x03-audio-trim: a Japanese request got an English-labelled report with Japanese only in Notes. Second language miss in two iterations, both on short audio-only jobs; SKILL.md's report format is shown in English and short jobs seem to copy it verbatim.",+ "e03-logo: overlay.py --fade with no --end fades the logo out again in the last 0.5 s; the agent flagged it. 'fade in at the start' is the common ask, so a fade-in-only spelling (or --fade applying only at --start when --end is absent) would remove the caveat.",+ "e01-reel / j01-reel / j08-project: three runs hit the reels loudness check and added a loudness.py step; export.py could say in its result that the platform's loudness target is not met so the agent plans it in one pass.",+ "j01-reel / j08-project: the untimed third cue in cues.txt lands at 6-9 s and falls outside a 6 s cut; both agents flagged it. Not a defect."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-12.json b/evals/trigger/results-2026-09-12.jsonnew file mode 100644index 0000000..ef7266a--- /dev/null+++ b/evals/trigger/results-2026-09-12.json@@ -0,0 +1,229 @@+{+ "date": "2026-09-12",+ "skill_version": "1.4.15",+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {+ "id": "t01",+ "lang": "en",+ "prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "video trimming to duration"+ },+ {+ "id": "t02",+ "lang": "ja",+ "prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "vertical reframe and captions for video"+ },+ {+ "id": "t03",+ "lang": "en",+ "prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "audio loudness fix in video file"+ },+ {+ "id": "t04",+ "lang": "ja",+ "prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "silence removal from recording"+ },+ {+ "id": "t05",+ "lang": "en",+ "prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "external mic sync to camera"+ },+ {+ "id": "t06",+ "lang": "ja",+ "prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "HDR to SDR conversion for YouTube"+ },+ {+ "id": "t07",+ "lang": "en",+ "prompt": "put our logo top right on all the videos in ./exports and add a fade at the end",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "logo overlay and fade on videos"+ },+ {+ "id": "t08",+ "lang": "ja",+ "prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "loudness normalization and format export"+ },+ {+ "id": "t09",+ "lang": "en",+ "prompt": "Does this file meet TikTok's specs? final_v3.mp4",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "platform compliance check for video"+ },+ {+ "id": "t10",+ "lang": "ja",+ "prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "multicam highlight reel editing"+ },+ {+ "id": "n01",+ "lang": "en",+ "prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png",+ "should_trigger": false,+ "chosen": "image-tools",+ "model": "sonnet",+ "correct": true,+ "reason": "resize and transparent background for PNG"+ },+ {+ "id": "n02",+ "lang": "ja",+ "prompt": "Windows に ffmpeg をインストールする方法を教えて",+ "should_trigger": false,+ "chosen": "web-research",+ "model": "sonnet",+ "correct": true,+ "reason": "how-to install ffmpeg question"+ },+ {+ "id": "n03",+ "lang": "en",+ "prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "should_trigger": false,+ "chosen": "none",+ "model": "sonnet",+ "correct": true,+ "reason": "downloading YouTube video not covered"+ },+ {+ "id": "n04",+ "lang": "ja",+ "prompt": "新商品紹介動画の台本(60 秒)を書いて",+ "should_trigger": false,+ "chosen": "copywriter",+ "model": "sonnet",+ "correct": true,+ "reason": "writing a script, text only"+ },+ {+ "id": "n05",+ "lang": "en",+ "prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?",+ "should_trigger": false,+ "chosen": "web-research",+ "model": "sonnet",+ "correct": true,+ "reason": "factual Premiere shortcut question"+ },+ {+ "id": "n06",+ "lang": "en",+ "prompt": "Design a YouTube thumbnail in Figma for my cooking channel",+ "should_trigger": false,+ "chosen": "none",+ "model": "sonnet",+ "correct": true,+ "reason": "Figma design task, no matching skill"+ },+ {+ "id": "n07",+ "lang": "ja",+ "prompt": "このCSVをJSONに変換して: sales.csv",+ "should_trigger": false,+ "chosen": "none",+ "model": "sonnet",+ "correct": true,+ "reason": "CSV to JSON conversion, not covered"+ },+ {+ "id": "n08",+ "lang": "en",+ "prompt": "Explain the difference between H.264 and H.265 in simple terms",+ "should_trigger": false,+ "chosen": "web-research",+ "model": "sonnet",+ "correct": true,+ "reason": "explain codec difference, factual question"+ },+ {+ "id": "n09",+ "lang": "en",+ "prompt": "Generate a 10-second video of a cat surfing with AI",+ "should_trigger": false,+ "chosen": "none",+ "model": "sonnet",+ "correct": true,+ "reason": "AI video generation not covered"+ },+ {+ "id": "n10",+ "lang": "ja",+ "prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)",+ "should_trigger": false,+ "chosen": "meeting-notes",+ "model": "sonnet",+ "correct": true,+ "reason": "transcribe meeting audio into notes"+ },+ {+ "id": "t11",+ "lang": "en",+ "prompt": "Convert this WAV to MP3: interview_raw.wav",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "audio format conversion WAV to MP3"+ },+ {+ "id": "t12",+ "lang": "en",+ "prompt": "Remove the silence from this recording, it's a 40 minute M4A",+ "should_trigger": true,+ "chosen": "ffmpeg-skill",+ "model": "sonnet",+ "correct": true,+ "reason": "silence removal from audio recording"+ }+ ],+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+}\ No newline at end of file