Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
test: promote eval 22 Set B (b1-b8) into evals/agent_prompts_24.json (#264)
Shogo Kajiyama351e23ef4229
diff --git a/evals/README.md b/evals/README.mdindex e96e38d..32bf481 100644--- a/evals/README.md+++ b/evals/README.md@@ -7,12 +7,13 @@ Two kinds of eval live here:`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.-## The 100-prompt agent set+## The 108-prompt agent set-`agent_prompts_24.json` (89 prompts) and `agent_prompts_exec.json` (11 prompts) together make the-100-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,-8 delivery/template prompts, 6 emoji/shaping prompts, the 8 long-form prompts 1.16 added and the-13 prompts 1.17 added — minus three retired duplicates (`th2`, `el2`, `it2`, the "download this+`agent_prompts_24.json` (97 prompts) and `agent_prompts_exec.json` (11 prompts) together make the+108-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,+8 delivery/template prompts, 6 emoji/shaping prompts, the 8 long-form prompts 1.16 added, the+13 prompts 1.17 added and the 8 prompts (`b1`–`b8`) 1.18 added — minus three retired duplicates+(`th2`, `el2`, `it2`, the "download thisYouTube video" refusal in a fourth, fifth and sixth language; `r03`, `c02` and `fr2` still measurethat class in English, Chinese and French, and `th1`, `el1` and `it1` still carry those threelanguages). Both use the same field shape:@@ -28,6 +29,7 @@ languages). Both use the same field shape:| `audio_only` | true when the input and output are audio: no picture-only script, no `look.py`, and the report marks the visual check not needed || `expect_output` | true when an ffprobe-readable media file must exist in OUTDIR afterwards || `fixtures` | text files the harness writes into OUTDIR **before** the run (see below) |+| `media_fixtures` | (1.18) filenames of synthetic video/audio the harness ffmpeg-generates into OUTDIR **before** the run (see below); unlike `fixtures`, the content is built by `write_fixtures.py`'s `build_media_fixture()`, not carried inline || `grader_expect` | (1.17) a regex the run's report text must match. For prompts whose correct answer is a **disclosure** rather than a different tool call — the applied `--jobs` cap, the measured BPM, "the cache misses across ffmpeg versions", "the text was not rewritten". A miss halves the score || `grader_not` | (1.17) a regex the report must **not** match. A hit is a zero: a report claiming something the run did not do is worse than one that says too little || `note` | grading hint for a human reader |@@ -125,6 +127,25 @@ something a "starts with done" regex let through. SKILL.md forbids the third labthree come from the test fixtures — run the test suite once before an iteration, as the othermedia prompts already require.+### 1.18 prompts (8, `b1`–`b8`), promoted from the eval 22 scratchpad++Symptom-only, one per 1.18.0 analysis/multicam feature, none naming a tool or a flag. Media is+built on demand by `write_fixtures.py`'s `media_fixtures` key (see below) rather than committed,+and every ground-truth number quoted in a `note` was verified against the exact synthetic clip+the harness builds — re-verify with the commands in `evals/write_fixtures.py`'s+`build_media_fixture()` docstring if the fixture construction ever changes.++| id | lang | what it asks | the answer being measured |+|---|---|---|---|+| `b1` | en | "which stretches are locked off, which is the camera moving, where's stuff moving in frame" | `scenes.py --shots`, section by section with timecodes |+| `b2` | en | tighten a voiceover, "keep the breaths, lose the long gaps" | `silence.py --speech-aware`; 3 breaths kept (0.90s), 2 pauses cut, 8.60s of 11.00s |+| `b3` | en | same voiceover, also remove "ums and ahs", "ONE removal pass" | `silence.py --filler --speech-aware` in one pass; no local whisper engine in this environment, so a disclosed partial refusal of the filler half is an acceptable PASS |+| `b4` | en | "where's the action, I'll crop by hand later — don't crop or re-encode" | `cropdetect.py --motion-centre`, report-only; FAIL if it crops anyway |+| `b5` | en | sync three recorders of the same take | `sync.py` with 3 sources; -2.500s / +1.180s measured offsets |+| `b6` | en | two cameras, "always on whoever's loudest, no faster than a second" | `multicam.py --switch energy --min-shot 1` |+| `b7` | ja | Japanese variant of `b4` | same as `b4`; report in Japanese, five English labels |+| `b8` | es | Spanish variant of `b6` | same as `b6`; report in Spanish, five English labels |+### Fixtures`c01`, `k01`, `a01`, `th1`, `he1`, `vi1` refer to a cue file in their own language (`cues_zh.txt`,@@ -141,6 +162,13 @@ committed, the same way no media fixture is committed. Each is a real 3-line cuePrompts without a `fixtures` key need nothing written.+`b1`–`b8` use a second mechanism, `media_fixtures` (a list of filenames, not inline text), for+the synthetic video/audio those prompts need (`shots.mp4`, `speech_breaths.m4a`, `camA.mp4`,+`camB.mp4`, `lav.m4a`, `cam3.m4a`) — ffmpeg-generated by `write_fixtures.py`'s+`build_media_fixture()` the first time each is needed, same "never committed" rule as every other+generated fixture. It shares the same CLI shown above; `python3 evals/write_fixtures.py OUTDIR b1`+also builds `OUTDIR/shots.mp4`.+## Gradingpython3 evals/grade_runs_24.py ITERATION_DIR [PROMPTS_JSON]diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.jsonindex ec4c452..fd88046 100644--- a/evals/agent_prompts_24.json+++ b/evals/agent_prompts_24.json@@ -958,5 +958,130 @@"project.json": "{\n \"output\": \"OUTDIR/final2.mp4\",\n \"clips\": [\n {\n \"src\": \"/home/user/ffmpeg-skill/tests/out/source.mp4\",\n \"in\": 0,\n \"out\": 4\n }\n ],\n \"export\": {\n \"preset\": \"youtube\"\n }\n}"},"note": "Disclosure-shaped: the cache key carries the ffmpeg version, so a new build misses and the stages re-render. A report claiming the old artifacts were reused is the failure."+ },+ {+ "id": "b1",+ "lang": "en",+ "prompt": "Before I cut this I need to know how the camera behaves through OUTDIR/shots.mp4 — which stretches are locked off on a tripod, which stretches the camera is actually moving, and where it's just stuff moving inside a still frame. Give me it section by section with timecodes. Write anything you produce into OUTDIR.",+ "expect": [+ "scenes"+ ],+ "refuse": false,+ "media_fixtures": [+ "shots.mp4"+ ],+ "grader_expect": "(?i)--shots",+ "grader_not": "(?i)\\bI (guess|assume)\\b",+ "note": "scenes.py --shots discoverability, symptom only, no flag named. Ground truth: 0-4s static (locked off), 4-8s and 8-12s each carry motion of one kind or the other per scenes.py's own flow-based classifier -- routing PASS requires scenes.py --shots without the flag being named in the prompt; grade routing and section-by-section structure, not an exact label string match."+ },+ {+ "id": "b2",+ "lang": "en",+ "prompt": "Tighten up this voiceover: OUTDIR/speech_breaths.m4a. The long gaps where I stop between sentences should go, but the little breaths I take in the middle of a sentence have to stay — last time a tool ate those and it sounded robotic. Save the result in OUTDIR.",+ "expect": [+ "silence"+ ],+ "refuse": false,+ "media_fixtures": [+ "speech_breaths.m4a"+ ],+ "audio_only": true,+ "expect_output": true,+ "grader_expect": "(?i)--speech-aware",+ "note": "silence.py --speech-aware. Ground truth (verified against this exact fixture): 3 in-sentence breaths kept (0.90s total, at 1.2-1.5/2.6-2.9/6.7-7.0), 2 sentence-boundary pauses cut, 8.60s of 11.00s kept."+ },+ {+ "id": "b3",+ "lang": "en",+ "prompt": "Same voiceover, OUTDIR/speech_breaths.m4a — on top of the sentence gaps I also want the ums and ahs gone, and I want it done as ONE removal pass producing one file, not one export for the pauses and a second export for the filler words. OUTDIR.",+ "expect": [+ "silence"+ ],+ "refuse": false,+ "media_fixtures": [+ "speech_breaths.m4a"+ ],+ "audio_only": true,+ "grader_expect": "(?i)--filler",+ "grader_not": "(?i)(two|second) (pass|export|render)",+ "note": "silence.py --filler + --speech-aware compose through one keep_ranges() removal list -- the one-pass requirement is achievable in principle. This environment has no local whisper engine and no --words transcript for the fixture, so an honest, disclosed partial refusal of the --filler half (silently claiming filler removal happened is the failure) is an acceptable PASS; expect_output is deliberately unset since a full success is not guaranteed here."+ },+ {+ "id": "b4",+ "lang": "en",+ "prompt": "I'm going to reframe OUTDIR/shots.mp4 to 9:16 by hand later. Where in the frame is the action over time? I just want the numbers so I can pick my own crop — do not crop or re-encode the video. OUTDIR.",+ "expect": [+ "cropdetect"+ ],+ "refuse": false,+ "media_fixtures": [+ "shots.mp4"+ ],+ "grader_expect": "(?i)--motion-centre",+ "grader_not": "(?i)(crop\\.py|fit\\.py|-o .*\\.mp4)",+ "note": "cropdetect.py --motion-centre is report-only. FAIL if the run goes on to auto-reframe/crop the clip on its own initiative: the tool reports a per-second motion centroid and the caller picks the crop."+ },+ {+ "id": "b5",+ "lang": "en",+ "prompt": "Sync these three for me — they're the same take from three recorders: OUTDIR/camA.mp4, OUTDIR/lav.m4a and OUTDIR/cam3.m4a. OUTDIR.",+ "expect": [+ "sync"+ ],+ "refuse": false,+ "media_fixtures": [+ "camA.mp4",+ "lav.m4a",+ "cam3.m4a"+ ],+ "grader_expect": "(?i)-2\\.5|2\\.50",+ "note": "sync.py with three sources (N>=1, additive more_sources). Ground truth (verified against these exact fixtures): lav.m4a -2.500s, cam3.m4a +1.180s, both relative to camA.mp4."+ },+ {+ "id": "b6",+ "lang": "en",+ "prompt": "Two cameras on the same conversation: OUTDIR/camA.mp4 and OUTDIR/camB.mp4. Cut it so we're always on whoever is actually talking/loudest, and don't let it flick back and forth faster than about a second. OUTDIR.",+ "expect": [+ "multicam"+ ],+ "refuse": false,+ "media_fixtures": [+ "camA.mp4",+ "camB.mp4"+ ],+ "expect_output": true,+ "grader_expect": "(?i)--switch energy",+ "note": "multicam.py --switch energy --min-shot 1. Ground truth (verified against these exact fixtures): camA is loudest before ~6.25s, camB after -- one real switch, never faster than the requested ~1s floor."+ },+ {+ "id": "b7",+ "lang": "ja",+ "prompt": "この動画 OUTDIR/shots.mp4 を後で自分で9:16に切り抜きたいので、時間ごとに画面のどのあたりで動きが起きているか、数値だけ教えてください。動画自体は切らないでください。OUTDIR。",+ "expect": [+ "cropdetect"+ ],+ "refuse": false,+ "media_fixtures": [+ "shots.mp4"+ ],+ "grader_expect": "(?i)--motion-centre",+ "grader_not": "(?i)(crop\\.py|fit\\.py)",+ "note": "Japanese variant of b4. Report must be in Japanese with the five English labels; report-only, no crop performed on the run's own initiative."+ },+ {+ "id": "b8",+ "lang": "es",+ "prompt": "Tengo dos cámaras de la misma charla: OUTDIR/camA.mp4 y OUTDIR/camB.mp4. Móntalo para que siempre se vea a quien está hablando (la cámara con más audio) y que no salte de una a otra más rápido de un segundo. OUTDIR.",+ "expect": [+ "multicam"+ ],+ "refuse": false,+ "media_fixtures": [+ "camA.mp4",+ "camB.mp4"+ ],+ "expect_output": true,+ "grader_expect": "(?i)--switch energy",+ "note": "Spanish variant of b6. Report in Spanish, five English labels."}]diff --git a/evals/write_fixtures.py b/evals/write_fixtures.pyindex 19fe7ed..27328d1 100644--- a/evals/write_fixtures.py+++ b/evals/write_fixtures.py@@ -67,9 +67,83 @@ def write(outdir: Path, prompt: dict) -> list:"-i", "color=c=%s:s=72x72:d=0.04" % palette[i % len(palette)],"-vf", "format=rgba", "-frames:v", "1", str(target)], check=True)written.append(target)+ for name in prompt.get("media_fixtures") or []:+ target = outdir / name+ if not target.exists():+ outdir.mkdir(parents=True, exist_ok=True)+ build_media_fixture(name, target)+ written.append(target)return written+def build_media_fixture(name: str, target: Path) -> None:+ """Synthetic media eval 22's Set B prompts need, ffmpeg-generated on demand (never+ committed) so the repo stays free of generated media. Ground truth for each file is+ documented next to its build here, and was verified against the tool it exercises+ (scenes.py --shots, silence.py --speech-aware, sync.py, multicam.py --switch energy)+ when these prompts were staged as scratchpad/eval22."""+ import subprocess++ def ffmpeg(*args):+ subprocess.run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", *[str(a) for a in args]], check=True)++ if name == "shots.mp4":+ # 0-4s locked-off (static, a Life pattern with no camera motion); 4-8s a crop window+ # sliding across a wide testsrc (measured scenes.py --shots label: "motion"); 8-12s a+ # box crossing an otherwise static frame (measured label: "pan"). The labels below are+ # what scenes.py's flow-based classifier actually returns for this synthetic footage --+ # verified once with scenes.py --shots --json against this exact build -- not a semantic+ # guess about which construction "should" read as camera motion vs in-frame motion.+ ffmpeg("-f", "lavfi", "-i",+ "life=size=80x45:rate=30:mold=32:ratio=0.1:death_color=#101030:life_color=#39ff88:seed=1",+ "-t", "4", "-vf", "scale=640:360:flags=neighbor", target.parent / "_shots_a.mp4")+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=640x360:rate=30",+ "-t", "4", "-vf", "crop=640:360:'80*t':0", target.parent / "_shots_b.mp4")+ ffmpeg("-f", "lavfi", "-i", "color=c=0x101832:s=640x360:rate=30",+ "-f", "lavfi", "-i", "color=c=0xffcc00:s=80x80:rate=30",+ "-filter_complex", "[0:v][1:v]overlay=x='(640-80)*t/4':y=140", "-t", "4",+ target.parent / "_shots_c.mp4")+ list_file = target.parent / "_shots_concat.txt"+ list_file.write_text("\n".join(f"file '{target.parent / n}'" for n in+ ("_shots_a.mp4", "_shots_b.mp4", "_shots_c.mp4")) + "\n",+ encoding="utf-8")+ ffmpeg("-f", "concat", "-safe", "0", "-i", list_file, "-c:v", "libx264",+ "-preset", "ultrafast", "-pix_fmt", "yuv420p", target)+ for n in ("_shots_a.mp4", "_shots_b.mp4", "_shots_c.mp4"):+ (target.parent / n).unlink(missing_ok=True)+ list_file.unlink(missing_ok=True)+ elif name == "speech_breaths.m4a":+ # 11.0s: speech bursts with three 0.30s in-sentence breaths (1.2-1.5, 2.6-2.9, 6.7-7.0)+ # and two 1.50s sentence-boundary pauses (4.0-5.5, 8.2-9.7). silence.py --speech-aware+ # keeps the three breaths (0.90s) and cuts only the two pauses (8.60s of 11.00s kept).+ expr = ("0.5*sin(2*PI*180*t)*(lt(t\\,1.2)+between(t\\,1.5\\,2.6)+between(t\\,2.9\\,4.0)"+ "+between(t\\,5.5\\,6.7)+between(t\\,7.0\\,8.2)+gt(t\\,9.7))")+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{expr}':s=48000", "-t", "11", "-c:a", "aac", target)+ elif name in ("camA.mp4", "camB.mp4"):+ # Two cameras on one reference timeline: camA loud 0-6s / quiet 6-12s, camB the reverse+ # -- multicam.py --switch energy should pick camA for the first half, camB the second.+ # Same construction as tests/test_orchestration.py's own _loud_cams().+ loud_first = name == "camA.mp4"+ hue = "" if loud_first else ",hue=h=90"+ vol_expr = ("0.8*sin(2*PI*440*t)*lt(t\\,6)+0.01*sin(2*PI*440*t)*gt(t\\,6)" if loud_first else+ "0.01*sin(2*PI*440*t)*lt(t\\,6)+0.8*sin(2*PI*440*t)*gt(t\\,6)")+ ffmpeg("-f", "lavfi", "-i", f"testsrc2=size=160x90:rate=30{hue}",+ "-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000",+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",+ "-c:a", "aac", target)+ elif name == "lav.m4a":+ # camA's audio (loud 0-6s / quiet 6-12s tone) delayed 2.500s -- sync ground truth -2.500s.+ vol_expr = "0.8*sin(2*PI*440*t)*lt(t\\,6)+0.01*sin(2*PI*440*t)*gt(t\\,6)"+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "9.5",+ "-af", "adelay=2500|2500", "-c:a", "aac", target)+ elif name == "cam3.m4a":+ # camA's audio advanced 1.180s -- sync ground truth +1.18s.+ vol_expr = "0.8*sin(2*PI*440*(t+1.18))*lt(t+1.18\\,6)+0.01*sin(2*PI*440*(t+1.18))*gt(t+1.18\\,6)"+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "10.82", "-c:a", "aac", target)+ else:+ raise SystemExit(f"no media fixture builder for {name!r}")++def main(argv) -> int:args = [a for a in argv if a != "--all"]every = "--all" in argv