Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

evals: iteration 5 against 1.4.0 (24-set, exec set, trigger set) and README row (#162)

Shogo Kajiyama29530318884d
diff --git a/README.md b/README.md
index 9f4ea85..3f3da70 100644
--- a/README.md
+++ b/README.md
@@ -340,6 +340,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th
| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold |
| **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) |
| **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |
+| **36 / 36** | 1.4.0 re-run (2026-09-11, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, user's language 8/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3; trigger set 22/22. Details and the six findings in `evals/results/iteration-5.json` |
| **6 / 6** | 0.9.1 audio evals (audio join, extraction, track selection, sample-accurate trim, typed dynamics; 2 in Japanese): routing, report format and audio-as-audio handling 6/6 |
```bash
diff --git a/evals/results/iteration-5.json b/evals/results/iteration-5.json
new file mode 100644
index 0000000..2112155
--- /dev/null
+++ b/evals/results/iteration-5.json
@@ -0,0 +1,56 @@
+{
+ "iteration": 5,
+ "date": "2026-09-11",
+ "skill_version": "1.4.0",
+ "agent_model": "claude-sonnet (one run per prompt, 6 in parallel)",
+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",
+ "runs": 36,
+ "prompts": {
+ "agent_prompts_24.json": 25,
+ "agent_prompts_exec.json": 11
+ },
+ "repeats": 1,
+ "harness_note": "Subagents could not ask the user questions in this harness; the prompt told them to state an assumption instead, or to refuse honestly when the skill's tools cannot do the job. Transcripts (run.md + outputs) were kept outside the repo.",
+ "regex_grader (evals/grade_runs_24.py)": {
+ "24-set": {
+ "routing_act_prompts": "100% over 20",
+ "mean_commands": 5.5,
+ "refusal_honesty": "3/5 by regex; 5/5 on manual review (r03 said 'no ... capability / has no script', r04 refused correctly but wrote its report in Spanish, which the keyword list does not cover)",
+ "japanese_report": "8/9 (j06-report answered a Japanese prompt in English)",
+ "report_format": "21/25 (the 4 misses are refusal/failed reports without a 'Look:' line, which the grader requires even for refusals)",
+ "visual_check_when_picture_changed": "8/8"
+ },
+ "exec-set": {
+ "routing_act_prompts": "100% over 10",
+ "mean_commands": 2.3,
+ "real_execution": "6/6",
+ "honest_failure_reporting": "5/5, false success 0",
+ "audio_only_handled_as_audio": "3/3",
+ "report_format": "7/11 (the 4 failed-input reports carry no 'Look:' line)"
+ }
+ },
+ "trigger_set": {
+ "file": "evals/trigger/results-2026-09-11.json",
+ "should_trigger": "12/12",
+ "should_not_trigger": "10/10",
+ "judge": "claude-sonnet, one batch"
+ },
+ "independent_grader": "not run this iteration (regex grader plus manual review of every flagged row)",
+ "comparison_with_0.8.x-0.9.1": {
+ "routing": "100% (was 100/99/100)",
+ "refusal_honesty": "5/5 manual (was 12/12)",
+ "language": "8/9 (was 27/27)",
+ "format": "21/25 by regex (was 71/72 by independent grader; the regex grader is stricter on refusals)",
+ "look": "8/8 (was 24/24)",
+ "trigger": "22/22 (was 22/22)",
+ "mean_commands": "5.5 (was 6.5-7.0; render.py and check-inside-export absorb steps)"
+ },
+ "findings": [
+ "e06-silence: the agent ran a raw `ffmpeg -af silencedetect` probe when silence.py's default -35 dBFS threshold found no gaps, then re-ran silence.py at -20 dBFS. SKILL.md forbids raw ffmpeg as a fallback; a `--threshold auto` or a 'no silence found at -35, try -20' hint in silence.py's output would remove the temptation.",
+ "j04-bgm: audio.py --music --duck --music-loop wrote an 11.93 s file from a 12.00 s video (shortest-stream trimming). The picture lost 70 ms; the agent reported it but did not treat it as a defect. Worth a test: adding music must never shorten the video.",
+ "j06-report: Japanese prompt answered in English (report and HTML). First language miss across iterations 2-5; sample of one.",
+ "r04-translate: the refusal was correct (no STT engine, no translation tool, 'never invent dialogue'), but the report was written in Spanish because the harness said 'in the language of the user's request' and the request mentioned Spanish subtitles. Harness wording, not a skill regression.",
+ "Refusal and failed reports omit the 'Look:' line; SKILL.md's report format lists it, so either the format should say it is only for completed jobs or the agents should write 'Look: not needed'.",
+ "e02-cut-lossless: 'without losing quality' at 0:02 produced a re-encode because the nearest keyframe was 2.09 s away; the agent explained the trade-off honestly (visually lossless CRF 18). Consider cut.py printing the keyframe-snap alternative in its report so the agent can offer it."
+ ]
+}
\ No newline at end of file
diff --git a/evals/trigger/results-2026-09-11.json b/evals/trigger/results-2026-09-11.json
new file mode 100644
index 0000000..1d6b7b7
--- /dev/null
+++ b/evals/trigger/results-2026-09-11.json
@@ -0,0 +1,231 @@
+{
+ "date": "2026-09-11",
+ "skill_version": "1.4.0",
+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",
+ "results": [
+ {
+ "id": "t01",
+ "lang": "en",
+ "prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Trimming a video clip to a duration."
+ },
+ {
+ "id": "t02",
+ "lang": "ja",
+ "prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Reframe vertical for Reels plus captions."
+ },
+ {
+ "id": "t03",
+ "lang": "en",
+ "prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Fixing audio levels in a video file."
+ },
+ {
+ "id": "t04",
+ "lang": "ja",
+ "prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Removing silence from a recording."
+ },
+ {
+ "id": "t05",
+ "lang": "en",
+ "prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Syncing external mic audio to camera video."
+ },
+ {
+ "id": "t06",
+ "lang": "ja",
+ "prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "HDR to SDR conversion for YouTube."
+ },
+ {
+ "id": "t07",
+ "lang": "en",
+ "prompt": "put our logo top right on all the videos in ./exports and add a fade at the end",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Logo overlay and fade on videos."
+ },
+ {
+ "id": "t08",
+ "lang": "ja",
+ "prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Loudness normalization to LUFS and format export."
+ },
+ {
+ "id": "t09",
+ "lang": "en",
+ "prompt": "Does this file meet TikTok's specs? final_v3.mp4",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Compliance check against TikTok specs."
+ },
+ {
+ "id": "t10",
+ "lang": "ja",
+ "prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Multicam highlight reel editing."
+ },
+ {
+ "id": "n01",
+ "lang": "en",
+ "prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png",
+ "should_trigger": false,
+ "chosen": "image-tools",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Resizing a PNG and making background transparent."
+ },
+ {
+ "id": "n02",
+ "lang": "ja",
+ "prompt": "Windows に ffmpeg をインストールする方法を教えて",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Asking how to install ffmpeg, not an editing task."
+ },
+ {
+ "id": "n03",
+ "lang": "en",
+ "prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "should_trigger": false,
+ "chosen": "web-research",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Downloading a YouTube video from a URL."
+ },
+ {
+ "id": "n04",
+ "lang": "ja",
+ "prompt": "新商品紹介動画の台本(60 秒)を書いて",
+ "should_trigger": false,
+ "chosen": "copywriter",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Writing a 60-second script, deliverable is words."
+ },
+ {
+ "id": "n05",
+ "lang": "en",
+ "prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Question about Premiere Pro shortcuts, unrelated to catalog."
+ },
+ {
+ "id": "n06",
+ "lang": "en",
+ "prompt": "Design a YouTube thumbnail in Figma for my cooking channel",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Figma thumbnail design request, not covered by catalog skills."
+ },
+ {
+ "id": "n07",
+ "lang": "ja",
+ "prompt": "このCSVをJSONに変換して: sales.csv",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "CSV to JSON conversion, no matching skill."
+ },
+ {
+ "id": "n08",
+ "lang": "en",
+ "prompt": "Explain the difference between H.264 and H.265 in simple terms",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Conceptual explanation question, not a task."
+ },
+ {
+ "id": "n09",
+ "lang": "en",
+ "prompt": "Generate a 10-second video of a cat surfing with AI",
+ "should_trigger": false,
+ "chosen": "none",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "AI video generation, not local ffmpeg editing or download."
+ },
+ {
+ "id": "n10",
+ "lang": "ja",
+ "prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)",
+ "should_trigger": false,
+ "chosen": "meeting-notes",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Transcribing meeting audio into minutes."
+ },
+ {
+ "id": "t11",
+ "lang": "en",
+ "prompt": "Convert this WAV to MP3: interview_raw.wav",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Audio format conversion WAV to MP3."
+ },
+ {
+ "id": "t12",
+ "lang": "en",
+ "prompt": "Remove the silence from this recording, it's a 40 minute M4A",
+ "should_trigger": true,
+ "chosen": "ffmpeg-skill",
+ "model": "sonnet",
+ "correct": true,
+ "reason": "Removing silence from an audio recording."
+ }
+ ],
+ "summary": {
+ "should_trigger": "12/12",
+ "should_not_trigger": "10/10"
+ }
+}
\ No newline at end of file