Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

test: promote eval 22 Set B (b1-b8) into evals/agent_prompts_24.json (#264)

Shogo Kajiyama351e23ef4229
diff --git a/evals/README.md b/evals/README.md
index e96e38d..32bf481 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -7,12 +7,13 @@ Two kinds of eval live here:
`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.
-## The 100-prompt agent set
+## The 108-prompt agent set
-`agent_prompts_24.json` (89 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
-100-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,
-8 delivery/template prompts, 6 emoji/shaping prompts, the 8 long-form prompts 1.16 added and the
-13 prompts 1.17 added — minus three retired duplicates (`th2`, `el2`, `it2`, the "download this
+`agent_prompts_24.json` (97 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
+108-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,
+8 delivery/template prompts, 6 emoji/shaping prompts, the 8 long-form prompts 1.16 added, the
+13 prompts 1.17 added and the 8 prompts (`b1`–`b8`) 1.18 added — minus three retired duplicates
+(`th2`, `el2`, `it2`, the "download this
YouTube video" refusal in a fourth, fifth and sixth language; `r03`, `c02` and `fr2` still measure
that class in English, Chinese and French, and `th1`, `el1` and `it1` still carry those three
languages). Both use the same field shape:
@@ -28,6 +29,7 @@ languages). Both use the same field shape:
| `audio_only` | true when the input and output are audio: no picture-only script, no `look.py`, and the report marks the visual check not needed |
| `expect_output` | true when an ffprobe-readable media file must exist in OUTDIR afterwards |
| `fixtures` | text files the harness writes into OUTDIR **before** the run (see below) |
+| `media_fixtures` | (1.18) filenames of synthetic video/audio the harness ffmpeg-generates into OUTDIR **before** the run (see below); unlike `fixtures`, the content is built by `write_fixtures.py`'s `build_media_fixture()`, not carried inline |
| `grader_expect` | (1.17) a regex the run's report text must match. For prompts whose correct answer is a **disclosure** rather than a different tool call — the applied `--jobs` cap, the measured BPM, "the cache misses across ffmpeg versions", "the text was not rewritten". A miss halves the score |
| `grader_not` | (1.17) a regex the report must **not** match. A hit is a zero: a report claiming something the run did not do is worse than one that says too little |
| `note` | grading hint for a human reader |
@@ -125,6 +127,25 @@ something a "starts with done" regex let through. SKILL.md forbids the third lab
three come from the test fixtures — run the test suite once before an iteration, as the other
media prompts already require.
+### 1.18 prompts (8, `b1`–`b8`), promoted from the eval 22 scratchpad
+
+Symptom-only, one per 1.18.0 analysis/multicam feature, none naming a tool or a flag. Media is
+built on demand by `write_fixtures.py`'s `media_fixtures` key (see below) rather than committed,
+and every ground-truth number quoted in a `note` was verified against the exact synthetic clip
+the harness builds — re-verify with the commands in `evals/write_fixtures.py`'s
+`build_media_fixture()` docstring if the fixture construction ever changes.
+
+| id | lang | what it asks | the answer being measured |
+|---|---|---|---|
+| `b1` | en | "which stretches are locked off, which is the camera moving, where's stuff moving in frame" | `scenes.py --shots`, section by section with timecodes |
+| `b2` | en | tighten a voiceover, "keep the breaths, lose the long gaps" | `silence.py --speech-aware`; 3 breaths kept (0.90s), 2 pauses cut, 8.60s of 11.00s |
+| `b3` | en | same voiceover, also remove "ums and ahs", "ONE removal pass" | `silence.py --filler --speech-aware` in one pass; no local whisper engine in this environment, so a disclosed partial refusal of the filler half is an acceptable PASS |
+| `b4` | en | "where's the action, I'll crop by hand later — don't crop or re-encode" | `cropdetect.py --motion-centre`, report-only; FAIL if it crops anyway |
+| `b5` | en | sync three recorders of the same take | `sync.py` with 3 sources; -2.500s / +1.180s measured offsets |
+| `b6` | en | two cameras, "always on whoever's loudest, no faster than a second" | `multicam.py --switch energy --min-shot 1` |
+| `b7` | ja | Japanese variant of `b4` | same as `b4`; report in Japanese, five English labels |
+| `b8` | es | Spanish variant of `b6` | same as `b6`; report in Spanish, five English labels |
+
### Fixtures
`c01`, `k01`, `a01`, `th1`, `he1`, `vi1` refer to a cue file in their own language (`cues_zh.txt`,
@@ -141,6 +162,13 @@ committed, the same way no media fixture is committed. Each is a real 3-line cue
Prompts without a `fixtures` key need nothing written.
+`b1`–`b8` use a second mechanism, `media_fixtures` (a list of filenames, not inline text), for
+the synthetic video/audio those prompts need (`shots.mp4`, `speech_breaths.m4a`, `camA.mp4`,
+`camB.mp4`, `lav.m4a`, `cam3.m4a`) — ffmpeg-generated by `write_fixtures.py`'s
+`build_media_fixture()` the first time each is needed, same "never committed" rule as every other
+generated fixture. It shares the same CLI shown above; `python3 evals/write_fixtures.py OUTDIR b1`
+also builds `OUTDIR/shots.mp4`.
+
## Grading
python3 evals/grade_runs_24.py ITERATION_DIR [PROMPTS_JSON]
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index ec4c452..fd88046 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -958,5 +958,130 @@
"project.json": "{\n \"output\": \"OUTDIR/final2.mp4\",\n \"clips\": [\n {\n \"src\": \"/home/user/ffmpeg-skill/tests/out/source.mp4\",\n \"in\": 0,\n \"out\": 4\n }\n ],\n \"export\": {\n \"preset\": \"youtube\"\n }\n}"
},
"note": "Disclosure-shaped: the cache key carries the ffmpeg version, so a new build misses and the stages re-render. A report claiming the old artifacts were reused is the failure."
+ },
+ {
+ "id": "b1",
+ "lang": "en",
+ "prompt": "Before I cut this I need to know how the camera behaves through OUTDIR/shots.mp4 — which stretches are locked off on a tripod, which stretches the camera is actually moving, and where it's just stuff moving inside a still frame. Give me it section by section with timecodes. Write anything you produce into OUTDIR.",
+ "expect": [
+ "scenes"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "shots.mp4"
+ ],
+ "grader_expect": "(?i)--shots",
+ "grader_not": "(?i)\\bI (guess|assume)\\b",
+ "note": "scenes.py --shots discoverability, symptom only, no flag named. Ground truth: 0-4s static (locked off), 4-8s and 8-12s each carry motion of one kind or the other per scenes.py's own flow-based classifier -- routing PASS requires scenes.py --shots without the flag being named in the prompt; grade routing and section-by-section structure, not an exact label string match."
+ },
+ {
+ "id": "b2",
+ "lang": "en",
+ "prompt": "Tighten up this voiceover: OUTDIR/speech_breaths.m4a. The long gaps where I stop between sentences should go, but the little breaths I take in the middle of a sentence have to stay — last time a tool ate those and it sounded robotic. Save the result in OUTDIR.",
+ "expect": [
+ "silence"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "speech_breaths.m4a"
+ ],
+ "audio_only": true,
+ "expect_output": true,
+ "grader_expect": "(?i)--speech-aware",
+ "note": "silence.py --speech-aware. Ground truth (verified against this exact fixture): 3 in-sentence breaths kept (0.90s total, at 1.2-1.5/2.6-2.9/6.7-7.0), 2 sentence-boundary pauses cut, 8.60s of 11.00s kept."
+ },
+ {
+ "id": "b3",
+ "lang": "en",
+ "prompt": "Same voiceover, OUTDIR/speech_breaths.m4a — on top of the sentence gaps I also want the ums and ahs gone, and I want it done as ONE removal pass producing one file, not one export for the pauses and a second export for the filler words. OUTDIR.",
+ "expect": [
+ "silence"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "speech_breaths.m4a"
+ ],
+ "audio_only": true,
+ "grader_expect": "(?i)--filler",
+ "grader_not": "(?i)(two|second) (pass|export|render)",
+ "note": "silence.py --filler + --speech-aware compose through one keep_ranges() removal list -- the one-pass requirement is achievable in principle. This environment has no local whisper engine and no --words transcript for the fixture, so an honest, disclosed partial refusal of the --filler half (silently claiming filler removal happened is the failure) is an acceptable PASS; expect_output is deliberately unset since a full success is not guaranteed here."
+ },
+ {
+ "id": "b4",
+ "lang": "en",
+ "prompt": "I'm going to reframe OUTDIR/shots.mp4 to 9:16 by hand later. Where in the frame is the action over time? I just want the numbers so I can pick my own crop — do not crop or re-encode the video. OUTDIR.",
+ "expect": [
+ "cropdetect"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "shots.mp4"
+ ],
+ "grader_expect": "(?i)--motion-centre",
+ "grader_not": "(?i)(crop\\.py|fit\\.py|-o .*\\.mp4)",
+ "note": "cropdetect.py --motion-centre is report-only. FAIL if the run goes on to auto-reframe/crop the clip on its own initiative: the tool reports a per-second motion centroid and the caller picks the crop."
+ },
+ {
+ "id": "b5",
+ "lang": "en",
+ "prompt": "Sync these three for me — they're the same take from three recorders: OUTDIR/camA.mp4, OUTDIR/lav.m4a and OUTDIR/cam3.m4a. OUTDIR.",
+ "expect": [
+ "sync"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "camA.mp4",
+ "lav.m4a",
+ "cam3.m4a"
+ ],
+ "grader_expect": "(?i)-2\\.5|2\\.50",
+ "note": "sync.py with three sources (N>=1, additive more_sources). Ground truth (verified against these exact fixtures): lav.m4a -2.500s, cam3.m4a +1.180s, both relative to camA.mp4."
+ },
+ {
+ "id": "b6",
+ "lang": "en",
+ "prompt": "Two cameras on the same conversation: OUTDIR/camA.mp4 and OUTDIR/camB.mp4. Cut it so we're always on whoever is actually talking/loudest, and don't let it flick back and forth faster than about a second. OUTDIR.",
+ "expect": [
+ "multicam"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "camA.mp4",
+ "camB.mp4"
+ ],
+ "expect_output": true,
+ "grader_expect": "(?i)--switch energy",
+ "note": "multicam.py --switch energy --min-shot 1. Ground truth (verified against these exact fixtures): camA is loudest before ~6.25s, camB after -- one real switch, never faster than the requested ~1s floor."
+ },
+ {
+ "id": "b7",
+ "lang": "ja",
+ "prompt": "この動画 OUTDIR/shots.mp4 を後で自分で9:16に切り抜きたいので、時間ごとに画面のどのあたりで動きが起きているか、数値だけ教えてください。動画自体は切らないでください。OUTDIR。",
+ "expect": [
+ "cropdetect"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "shots.mp4"
+ ],
+ "grader_expect": "(?i)--motion-centre",
+ "grader_not": "(?i)(crop\\.py|fit\\.py)",
+ "note": "Japanese variant of b4. Report must be in Japanese with the five English labels; report-only, no crop performed on the run's own initiative."
+ },
+ {
+ "id": "b8",
+ "lang": "es",
+ "prompt": "Tengo dos cámaras de la misma charla: OUTDIR/camA.mp4 y OUTDIR/camB.mp4. Móntalo para que siempre se vea a quien está hablando (la cámara con más audio) y que no salte de una a otra más rápido de un segundo. OUTDIR.",
+ "expect": [
+ "multicam"
+ ],
+ "refuse": false,
+ "media_fixtures": [
+ "camA.mp4",
+ "camB.mp4"
+ ],
+ "expect_output": true,
+ "grader_expect": "(?i)--switch energy",
+ "note": "Spanish variant of b6. Report in Spanish, five English labels."
}
]
diff --git a/evals/write_fixtures.py b/evals/write_fixtures.py
index 19fe7ed..27328d1 100644
--- a/evals/write_fixtures.py
+++ b/evals/write_fixtures.py
@@ -67,9 +67,83 @@ def write(outdir: Path, prompt: dict) -> list:
"-i", "color=c=%s:s=72x72:d=0.04" % palette[i % len(palette)],
"-vf", "format=rgba", "-frames:v", "1", str(target)], check=True)
written.append(target)
+ for name in prompt.get("media_fixtures") or []:
+ target = outdir / name
+ if not target.exists():
+ outdir.mkdir(parents=True, exist_ok=True)
+ build_media_fixture(name, target)
+ written.append(target)
return written
+def build_media_fixture(name: str, target: Path) -> None:
+ """Synthetic media eval 22's Set B prompts need, ffmpeg-generated on demand (never
+ committed) so the repo stays free of generated media. Ground truth for each file is
+ documented next to its build here, and was verified against the tool it exercises
+ (scenes.py --shots, silence.py --speech-aware, sync.py, multicam.py --switch energy)
+ when these prompts were staged as scratchpad/eval22."""
+ import subprocess
+
+ def ffmpeg(*args):
+ subprocess.run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", *[str(a) for a in args]], check=True)
+
+ if name == "shots.mp4":
+ # 0-4s locked-off (static, a Life pattern with no camera motion); 4-8s a crop window
+ # sliding across a wide testsrc (measured scenes.py --shots label: "motion"); 8-12s a
+ # box crossing an otherwise static frame (measured label: "pan"). The labels below are
+ # what scenes.py's flow-based classifier actually returns for this synthetic footage --
+ # verified once with scenes.py --shots --json against this exact build -- not a semantic
+ # guess about which construction "should" read as camera motion vs in-frame motion.
+ ffmpeg("-f", "lavfi", "-i",
+ "life=size=80x45:rate=30:mold=32:ratio=0.1:death_color=#101030:life_color=#39ff88:seed=1",
+ "-t", "4", "-vf", "scale=640:360:flags=neighbor", target.parent / "_shots_a.mp4")
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=640x360:rate=30",
+ "-t", "4", "-vf", "crop=640:360:'80*t':0", target.parent / "_shots_b.mp4")
+ ffmpeg("-f", "lavfi", "-i", "color=c=0x101832:s=640x360:rate=30",
+ "-f", "lavfi", "-i", "color=c=0xffcc00:s=80x80:rate=30",
+ "-filter_complex", "[0:v][1:v]overlay=x='(640-80)*t/4':y=140", "-t", "4",
+ target.parent / "_shots_c.mp4")
+ list_file = target.parent / "_shots_concat.txt"
+ list_file.write_text("\n".join(f"file '{target.parent / n}'" for n in
+ ("_shots_a.mp4", "_shots_b.mp4", "_shots_c.mp4")) + "\n",
+ encoding="utf-8")
+ ffmpeg("-f", "concat", "-safe", "0", "-i", list_file, "-c:v", "libx264",
+ "-preset", "ultrafast", "-pix_fmt", "yuv420p", target)
+ for n in ("_shots_a.mp4", "_shots_b.mp4", "_shots_c.mp4"):
+ (target.parent / n).unlink(missing_ok=True)
+ list_file.unlink(missing_ok=True)
+ elif name == "speech_breaths.m4a":
+ # 11.0s: speech bursts with three 0.30s in-sentence breaths (1.2-1.5, 2.6-2.9, 6.7-7.0)
+ # and two 1.50s sentence-boundary pauses (4.0-5.5, 8.2-9.7). silence.py --speech-aware
+ # keeps the three breaths (0.90s) and cuts only the two pauses (8.60s of 11.00s kept).
+ expr = ("0.5*sin(2*PI*180*t)*(lt(t\\,1.2)+between(t\\,1.5\\,2.6)+between(t\\,2.9\\,4.0)"
+ "+between(t\\,5.5\\,6.7)+between(t\\,7.0\\,8.2)+gt(t\\,9.7))")
+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{expr}':s=48000", "-t", "11", "-c:a", "aac", target)
+ elif name in ("camA.mp4", "camB.mp4"):
+ # Two cameras on one reference timeline: camA loud 0-6s / quiet 6-12s, camB the reverse
+ # -- multicam.py --switch energy should pick camA for the first half, camB the second.
+ # Same construction as tests/test_orchestration.py's own _loud_cams().
+ loud_first = name == "camA.mp4"
+ hue = "" if loud_first else ",hue=h=90"
+ vol_expr = ("0.8*sin(2*PI*440*t)*lt(t\\,6)+0.01*sin(2*PI*440*t)*gt(t\\,6)" if loud_first else
+ "0.01*sin(2*PI*440*t)*lt(t\\,6)+0.8*sin(2*PI*440*t)*gt(t\\,6)")
+ ffmpeg("-f", "lavfi", "-i", f"testsrc2=size=160x90:rate=30{hue}",
+ "-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", target)
+ elif name == "lav.m4a":
+ # camA's audio (loud 0-6s / quiet 6-12s tone) delayed 2.500s -- sync ground truth -2.500s.
+ vol_expr = "0.8*sin(2*PI*440*t)*lt(t\\,6)+0.01*sin(2*PI*440*t)*gt(t\\,6)"
+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "9.5",
+ "-af", "adelay=2500|2500", "-c:a", "aac", target)
+ elif name == "cam3.m4a":
+ # camA's audio advanced 1.180s -- sync ground truth +1.18s.
+ vol_expr = "0.8*sin(2*PI*440*(t+1.18))*lt(t+1.18\\,6)+0.01*sin(2*PI*440*(t+1.18))*gt(t+1.18\\,6)"
+ ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "10.82", "-c:a", "aac", target)
+ else:
+ raise SystemExit(f"no media fixture builder for {name!r}")
+
+
def main(argv) -> int:
args = [a for a in argv if a != "--all"]
every = "--all" in argv