Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

fix: route "timeline for Premiere" from multicam through --write-project; add eval 25 kit (#293)

Shogo Kajiyamaabbe6419dc6c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ba41e30..4151796 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,10 @@
## Unreleased
-(nothing yet)
+- SKILL.md and README: `multicam.py --edl` is a plain cut list, not an EDL an editor opens. The multicam
+ row now routes "for Premiere/Resolve" through `--write-project p.json` then `render.py --export-timeline`, and
+ the "open it in Premiere / Resolve / Final Cut" row says to write a project first when there is none (found
+ reading the eval-25 tl2 prompt).
## 2.2.4
diff --git a/README.md b/README.md
index 2a4064f..de614a4 100644
--- a/README.md
+++ b/README.md
@@ -259,7 +259,7 @@ These are the rules the skill file gives the agent and the code enforces.
|---|---|
| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`, `--cache DIR`/`--from STAGE` (reuse the stages that did not change); the captions block takes the fit-size policy (`fit_size`, `min_size`, `fit_size_scope`) and the run reports the caption stage's counts as `caption`; `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table); `--export-timeline edit.fcpxml\|.edl\|.otio` (2.1) writes the cut as an editor timeline for Final Cut, DaVinci Resolve or Premiere instead of rendering it — clips, speed, dissolves, the music bed and chapter markers carry over, and `not_exported` names what stays behind |
| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch`; `--jobs N` processes several files at once under one shared `--timeout` |
-| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list; `--switch energy` auto-cuts to whichever camera is loudest/talking instead of a hand-built list, with `--min-shot` (minimum shot length) and `--edl` (write the cut list) |
+| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list; `--switch energy` auto-cuts to whichever camera is loudest/talking instead of a hand-built list, with `--min-shot` (minimum shot length) and `--edl` (write a plain START-END cut list, not a CMX EDL; for Premiere/Resolve use `--write-project` then `render.py --export-timeline`) |
| `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |
Not tools, but part of the surface: `mcp/server.py` (the MCP transport) and `scripts/_contract.py` (`contract --json`, `doctor`). Per-flag reference for every tool: [references/scripts.md](references/scripts.md).
diff --git a/SKILL.md b/SKILL.md
index 92848a2..93e3f41 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -5,11 +5,11 @@ description: 'Edit video and audio with local FFmpeg from natural-language reque
# ffmpeg-skill
-Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script about to run is the cheapest full flag list. A reference file costs as much to read as this file; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end). MCP: `tools/list` shows only the core 12 by default; the other 42 (per this table) stay callable by name via `tools/call`, described by `contract --json`; set `FFMPEG_SKILL_MCP_FULL=1` for all 42.
+Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script about to run is the cheapest full flag list. A reference file costs as much as this one; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end). MCP: `tools/list` shows only the core 12 by default; the other 42 (per this table) stay callable by name via `tools/call`, described by `contract --json`; set `FFMPEG_SKILL_MCP_FULL=1` for all 42.
Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (status/output/verified plus a `summary`; prefer on writing steps); `--fast` (preview quality); `--progress`; `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (step 7); `--plan FILE` (the dry run as a plan `render.py FILE` runs later; refuses if an input changed). Re-encoding tools also take `--codec h264|hevc|av1|prores` and `--quality N`: unset, SDR is x264, HDR is x265 Main10; `prores` needs `-o NAME.mov`, `h264` refuses HDR (`color.py --to-sdr` first).
-Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe — they just skip the artifact and side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per-tool: `contract --json`'s `dry_run` field.
+Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe, skipping artifacts/side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per tool: `contract --json` `dry_run`.
## Workflow (always follow this order)
@@ -96,7 +96,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
| "add subtitles from this SRT", "burn in captions" | `caption.py input.mp4 --srt subs.srt` |
| "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` |
| "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |
-| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` for 1.16 behaviour, `--min-size` sets the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it |
+| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` = 1.16 behaviour, `--min-size` the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it |
| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (needs a local whisper; else `--text`) |
| "TikTok-style captions with the words popping" | `caption.py input.mp4 --text cues.txt --animate pop --karaoke` |
| "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |
@@ -139,9 +139,9 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
| "stitch these clips", "add a crossfade" | `join.py a.mp4 b.mp4 c.mp4 --transition fade --duration 0.5` (TTS parts: `--list parts.txt`, `--on-missing skip`) |
| "several changes to one edit", 3+ steps | `render.py --init project.json`, edit, `render.py project.json` |
| "I changed one stage, don't redo the rest" | `render.py project.json --cache DIR` — identical stages reused (`--from STAGE` starts there) |
-| "open it in Premiere / Resolve / Final Cut" | `render.py project.json --export-timeline edit.fcpxml\|.edl\|.otio` — the cut as a timeline, nothing rendered; read `not_exported` |
+| "open it in Premiere / Resolve / Final Cut" | `render.py project.json --export-timeline edit.fcpxml\|.edl\|.otio` — renders nothing; read `not_exported`; no project? `--write-project` |
| "do this to every file in the folder", "use all the cores" | `batch.py FOLDER --recipe batch.json --jobs auto` (steps or a render project; cached) |
-| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` (manual) or `--switch energy` (auto-cuts to the loudest camera, `--min-shot`, `--edl`) |
+| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` (manual) or `--switch energy` (`--min-shot`); `--edl` = cut list; editor: `--write-project p.json` → `--export-timeline` |
| "what's in this file", "how long is it" | `probe.py input.mp4` |
| "show me what it looks like", "are the captions readable" | `look.py output.mp4 --tiles 3x2`, then view the PNG |
| "what would you run?", "don't render yet" | any script with `--dry-run` |
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 225d03a..4f9baea 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -657,6 +657,16 @@ from the MCP schema. Two planned items were not done, deliberately:
The 1.21.0 "2.0 freeze" above did not happen as written either: 1.21-1.26 kept shipping
features, and the deprecation window's 90-day half was waived (docs/contract.md). Nothing else.
+## Eval 25 — do natural requests reach 2.1/2.2? (evals only, partial)
+
+`evals/agent_prompts_25.json` (tt1 ja TTS parts with one missing wav, tt2 a video list with one
+missing segment, tl1 hand a project to Resolve, tl2 a multicam edit for Premiere), run with
+`evals/run_agents_25.sh` and graded on each run's `--json` result document by
+`evals/grade_runs_25.py`. The trigger set (`evals/trigger/prompts_25.json`) ran: 3/3. The agent
+prompts did **not** run (the sandbox refused headless tool-using agents), so routing is still
+unmeasured; `evals/results/iteration-25.json` records the hand-run tool check, the grader
+self-test and a proposed (unapplied) SKILL.md fix for multicam's cut-list `--edl` vs a CMX EDL.
+
## Not planned
- A feature that needs 2.0: features land in 1.x behind flags (1.4.0, 1.6.0, 1.7.0 did).
diff --git a/evals/agent_prompts_25.json b/evals/agent_prompts_25.json
new file mode 100644
index 0000000..fc1c9ce
--- /dev/null
+++ b/evals/agent_prompts_25.json
@@ -0,0 +1,102 @@
+[
+ {
+ "id": "tt1",
+ "lang": "ja",
+ "prompt": "TTSで作った音声が parts.txt に並んでるので順番につないで1本のwavにして。1個くらい欠けてても止まらないで",
+ "expect": [
+ "join.py",
+ "--list",
+ "--on-missing skip"
+ ],
+ "fixtures": {
+ "parts.txt": "tts_01.wav\ntts_02.wav\ntts_03.wav\ntts_04.wav\n"
+ },
+ "media_fixtures": [
+ "tts_01.wav",
+ "tts_02.wav",
+ "tts_04.wav"
+ ],
+ "missing": "tts_03.wav",
+ "result_expect": {
+ "tool": "join",
+ "ok": true,
+ "skipped_names": "tts_03.wav"
+ },
+ "grader_not": [
+ "Traceback"
+ ]
+ },
+ {
+ "id": "tt2",
+ "lang": "en",
+ "prompt": "I have clips listed in list.txt from my TTS+render pipeline; join them into final.mp4 and tell me if any segment is missing before doing anything",
+ "expect": [
+ "join.py",
+ "--list"
+ ],
+ "fixtures": {
+ "list.txt": "seg_01.mp4\nseg_02.mp4\nseg_03.mp4\n"
+ },
+ "media_fixtures": [
+ "seg_01.mp4",
+ "seg_03.mp4"
+ ],
+ "missing": "seg_02.mp4",
+ "result_expect": {
+ "tool": "join",
+ "names": "seg_02.mp4"
+ },
+ "report_expect": "seg_02",
+ "must_not_write": "final.mp4",
+ "grader_not": [
+ "Traceback",
+ "--on-missing skip"
+ ]
+ },
+ {
+ "id": "tl1",
+ "lang": "en",
+ "prompt": "Hand this edit (project.json) to DaVinci Resolve so my editor can finish it",
+ "expect": [
+ "render.py",
+ "--export-timeline"
+ ],
+ "grader_expect": [
+ "--export-timeline[ =]\\S+\\.(fcpxml|otio|edl)"
+ ],
+ "fixtures": {
+ "project.json": "{\n \"output\": \"final.mp4\",\n \"clips\": [\n {\n \"src\": \"a.mp4\",\n \"in\": 0.5,\n \"out\": 2.5\n },\n {\n \"src\": \"b.mp4\",\n \"in\": 0,\n \"out\": 2\n }\n ],\n \"transition\": {\n \"type\": \"fade\",\n \"duration\": 0.5\n }\n}\n"
+ },
+ "media_fixtures": [
+ "a.mp4",
+ "b.mp4"
+ ],
+ "result_expect": {
+ "tool": "render",
+ "ok": true
+ },
+ "must_not_write": "final.mp4",
+ "grader_not": [
+ "Traceback"
+ ]
+ },
+ {
+ "id": "tl2",
+ "lang": "en",
+ "prompt": "Make a timeline of this multicam edit (camA.mp4, camB.mp4) that I can open in Premiere",
+ "expect": [
+ "multicam.py|render.py"
+ ],
+ "grader_expect": [
+ "\\.(edl|fcpxml|otio)\\b"
+ ],
+ "media_fixtures": [
+ "camA.mp4",
+ "camB.mp4"
+ ],
+ "record_only": true,
+ "grader_not": [
+ "Traceback"
+ ]
+ }
+]
diff --git a/evals/grade_runs_25.py b/evals/grade_runs_25.py
new file mode 100644
index 0000000..7f9b861
--- /dev/null
+++ b/evals/grade_runs_25.py
@@ -0,0 +1,154 @@
+#!/usr/bin/env python3
+"""Grade eval 25 runs (TTS segment joins, editor-timeline handoff) on the result documents.
+
+ python3 evals/grade_runs_25.py RUN_DIR [PROMPTS_JSON] [--json]
+
+RUN_DIR/<id>/ is what evals/run_agents_25.sh leaves: transcript.jsonl (claude -p stream-json)
+or transcript.txt (any harness's plain log), plus .before/.after file listings. For every
+prompt in PROMPTS_JSON (default evals/agent_prompts_25.json):
+
+ expect every slot ('a|b' = either) appears in a command the agent ran
+ grader_expect every regex matches a command the agent ran
+ grader_not no regex matches the transcript (commands, tool output, report) -- 'Traceback'
+ catches a crashed script; '--on-missing skip' in tt2 catches joining before asking
+ result_expect checked against the LAST --json result document the named tool printed, not the
+ command line: tool=<script stem> selects it, ok=true needs status completed,
+ skipped_names=<file> needs that file in skipped[], names=<file> needs it in
+ skipped[], pending[] or problems[]
+ report_expect substring of the final report
+ must_not_write a file that must not appear in .after (a refused/handoff run renders nothing)
+ record_only routing is recorded, verdict 'recorded' instead of pass/fail
+
+A missing run directory is MISSING, never a crash.
+"""
+import json
+import re
+import sys
+from pathlib import Path
+
+W = Path(__file__).resolve().parent
+
+
+def load_run(d: Path):
+ """(commands, tool_outputs, final_report) from a run folder."""
+ cmds, outs, final = [], [], ""
+ jl = d / "transcript.jsonl"
+ if jl.exists():
+ for line in jl.read_text(encoding="utf-8", errors="replace").splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "result":
+ final = ev.get("result") or final
+ content = (ev.get("message") or {}).get("content")
+ if not isinstance(content, list):
+ continue
+ for c in content:
+ if c.get("type") == "tool_use" and c.get("name") == "Bash":
+ cmds.append((c.get("input") or {}).get("command", ""))
+ elif c.get("type") == "tool_result":
+ body = c.get("content")
+ if isinstance(body, list):
+ body = "\n".join(x.get("text", "") for x in body if isinstance(x, dict))
+ outs.append(body or "")
+ elif c.get("type") == "text" and ev.get("type") == "assistant":
+ final = c.get("text") or final
+ return cmds, outs, final
+ tx = d / "transcript.txt"
+ if tx.exists():
+ text = tx.read_text(encoding="utf-8", errors="replace")
+ cmds = [m.group(1) for m in re.finditer(r"^\$ (.+)$", text, re.M)]
+ rep = text.split("\n## Report\n", 1)
+ return cmds, [rep[0]], rep[1] if len(rep) > 1 else ""
+ return None
+
+
+def result_docs(outs):
+ """Every top-level JSON object printed in tool output (a script's --json document)."""
+ docs = []
+ dec = json.JSONDecoder()
+ for o in outs:
+ i = 0
+ while True:
+ j = o.find("{", i)
+ if j < 0:
+ break
+ try:
+ doc, end = dec.raw_decode(o, j)
+ except ValueError:
+ i = j + 1
+ continue
+ if isinstance(doc, dict) and "status" in doc:
+ docs.append(doc)
+ i = end
+ return docs
+
+
+def _names(doc, *keys):
+ return [Path(str(x.get("path", x) if isinstance(x, dict) else x)).name
+ for k in keys for x in (doc.get(k) or [])]
+
+
+def grade(p, d: Path):
+ run = load_run(d)
+ if run is None:
+ return {"id": p["id"], "verdict": "MISSING"}
+ cmds, outs, final = run
+ cmdtext = "\n".join(cmds)
+ alltext = "\n".join([cmdtext, *outs, final])
+ fails = []
+ for slot in p.get("expect", []):
+ if not any(alt in cmdtext for alt in slot.split("|")):
+ fails.append("expect %r not in any command" % slot)
+ for rx in p.get("grader_expect", []):
+ if not re.search(rx, cmdtext):
+ fails.append("grader_expect %r" % rx)
+ for rx in p.get("grader_not", []):
+ if re.search(re.escape(rx) if rx.startswith("--") else rx, alltext if rx == "Traceback" else cmdtext):
+ fails.append("grader_not %r matched" % rx)
+ re_exp = p.get("result_expect")
+ doc = None
+ if re_exp:
+ tool = re_exp.get("tool")
+ tool_outs = [o for c, o in zip(cmds, outs) if tool + ".py" in c and "--json" in c] if len(cmds) == len(outs) else outs
+ docs = result_docs(tool_outs)
+ doc = docs[-1] if docs else None
+ if doc is None:
+ fails.append("no %s --json result document" % tool)
+ else:
+ if re_exp.get("ok") and doc.get("status") != "completed":
+ fails.append("result status %r" % doc.get("status"))
+ if "skipped_names" in re_exp and re_exp["skipped_names"] not in _names(doc, "skipped"):
+ fails.append("%s not in skipped[]" % re_exp["skipped_names"])
+ if "names" in re_exp and re_exp["names"] not in _names(doc, "skipped", "pending", "problems"):
+ fails.append("%s not in skipped/pending/problems" % re_exp["names"])
+ if p.get("report_expect") and p["report_expect"] not in final:
+ fails.append("report does not name %r" % p["report_expect"])
+ after = d / ".after"
+ if p.get("must_not_write") and after.exists() and p["must_not_write"] in after.read_text().split():
+ fails.append("%s was written" % p["must_not_write"])
+ verdict = "recorded" if p.get("record_only") else ("pass" if not fails else "fail")
+ return {"id": p["id"], "verdict": verdict, "failures": fails,
+ "commands": [c for c in cmds if ".py" in c],
+ "result_status": doc.get("status") if doc else None}
+
+
+def main(argv):
+ as_json = "--json" in argv
+ args = [a for a in argv if a != "--json"]
+ if not args:
+ raise SystemExit(__doc__)
+ root = Path(args[0])
+ prompts = json.loads(Path(args[1] if len(args) > 1 else W / "agent_prompts_25.json").read_text(encoding="utf-8"))
+ rows = [grade(p, root / p["id"]) for p in prompts]
+ if as_json:
+ print(json.dumps(rows, indent=2, ensure_ascii=False))
+ else:
+ for r in rows:
+ print("%-4s %-8s %s" % (r["id"], r["verdict"], "; ".join(r.get("failures") or [])))
+ return 0 if all(r["verdict"] in ("pass", "recorded") for r in rows) else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/evals/results/iteration-25.json b/evals/results/iteration-25.json
new file mode 100644
index 0000000..05c3a7b
--- /dev/null
+++ b/evals/results/iteration-25.json
@@ -0,0 +1,96 @@
+{
+ "iteration": "25",
+ "at_version": "2.2.4",
+ "date": "2026-09-24",
+ "status": "partial: trigger set run for real; agent prompts NOT run (harness blocked in this sandbox)",
+ "scope": "Do natural requests reach the 2.1/2.2 features: join.py --list / --on-missing skip (TTS segments) and render.py --export-timeline (editor handoff)? Four agent prompts (evals/agent_prompts_25.json) and three trigger prompts (evals/trigger/prompts_25.json).",
+ "harness": {
+ "agent_runner": "evals/run_agents_25.sh RUN_DIR [MODEL]: stages fixtures with evals/write_fixtures.py --all, links this checkout in as a project skill, runs `claude -p` once per prompt, saves transcript.jsonl and .before/.after listings",
+ "grader": "evals/grade_runs_25.py RUN_DIR: routing from the Bash commands, pass/fail from the LAST --json result document of the named tool (status, skipped[], problems[]), grader_not 'Traceback' over the whole transcript, must_not_write from .after",
+ "fixtures": "write_fixtures.py builders added for tts_NN.wav (1 s mono 24 kHz tone) and seg_NN.mp4 / a.mp4 / b.mp4 (3 s 320x180 testsrc2 + tone); parts.txt and list.txt each name one file that is not built; project.json is a 2-clip project with a fade"
+ },
+ "agent_runs": {
+ "ran": false,
+ "why": "`claude` 2.x is installed and answers `claude -p`, but launching headless tool-using agents (claude -p --permission-mode bypassPermissions in the fixture folders) was refused by this sandbox's permission classifier. Not worked around. The maintainer can run `sh evals/run_agents_25.sh <dir>` then `python3 evals/grade_runs_25.py <dir>`.",
+ "detail": [
+ {
+ "id": "tt1",
+ "verdict": "not run",
+ "expected": "join.py --list parts.txt --on-missing skip -o <x>.wav --json; skipped[] names tts_03.wav"
+ },
+ {
+ "id": "tt2",
+ "verdict": "not run",
+ "expected": "join.py --list list.txt (fail mode or --dry-run); report names seg_02.mp4; final.mp4 not written"
+ },
+ {
+ "id": "tl1",
+ "verdict": "not run",
+ "expected": "render.py project.json --export-timeline <x>.fcpxml|.otio|.edl; final.mp4 not written"
+ },
+ {
+ "id": "tl2",
+ "verdict": "not run (record-only)",
+ "expected": "multicam.py ... --write-project p.json, then render.py p.json --export-timeline <x>.edl|.fcpxml|.otio"
+ }
+ ]
+ },
+ "tool_check_not_an_eval": {
+ "note": "The expected commands were run BY HAND on the fixtures to confirm the target path works and to feed the grader self-test. This shows the tools behave; it says nothing about whether an agent routes to them.",
+ "tt1": "join.py --list parts.txt --on-missing skip --transition none -o voice.wav --json: status completed, 3.000 s, skipped[0] = {index 2, tts_03.wav, missing}",
+ "tt2": "join.py --list list.txt -o final.mp4 --json: exit 1, kind input INPUT_INVALID, problems[0] = seg_02.mp4 missing, hint names --on-missing skip, final.mp4 not written",
+ "tl1": "render.py project.json --export-timeline edit.fcpxml --json: completed, 2 clips, 3.5 s at 30 fps, fade 15 frames, not_exported [], no media rendered",
+ "tl2": "multicam.py has no timeline export of its own: its --edl writes a START-END cut list (cut.py --segments format), not a CMX 3600 EDL. The path to Premiere is multicam.py --write-project p.json, then render.py p.json --export-timeline x.edl. Under --dry-run multicam writes neither mc.json nor the cut list."
+ },
+ "grader_self_test_not_an_eval": {
+ "note": "Hand-made transcripts (real tool output from the hand runs above for the 'good' set; fabricated failures for the 'bad' set), used only to check the grader.",
+ "good": {
+ "tt1": "pass",
+ "tt2": "pass",
+ "tl1": "pass",
+ "tl2": "recorded"
+ },
+ "bad": {
+ "tt1": "fail (no --on-missing skip; Traceback; no join result doc)",
+ "tt2": "fail (--on-missing skip used; seg_02 not in result doc; report silent; final.mp4 written)",
+ "tl1": "fail (no --export-timeline; final.mp4 written)",
+ "tl2": "MISSING"
+ }
+ },
+ "trigger": {
+ "ran": true,
+ "result_file": "evals/trigger/results-2026-09-24-it25.json",
+ "correct": 3,
+ "total": 3,
+ "detail": [
+ {
+ "id": "g1",
+ "should_trigger": true,
+ "choice": "ffmpeg-skill",
+ "correct": true
+ },
+ {
+ "id": "g2",
+ "should_trigger": true,
+ "choice": "ffmpeg-skill",
+ "correct": true
+ },
+ {
+ "id": "g3",
+ "should_trigger": false,
+ "choice": "none",
+ "correct": true
+ }
+ ]
+ },
+ "findings": [
+ "Trigger: 3/3. The TTS-segment prompt and the editor-handoff prompt (neither names a media extension) both chose ffmpeg-skill; the project-management 'timeline' prompt chose none.",
+ "Routing could not be measured: no agent run happened.",
+ "tl2 risk (from reading, not from a run): SKILL.md's multicam row lists `--edl` among multicam's options, and 'EDL' is the word an agent reaches for when asked for Premiere. multicam's --edl is a cut list Premiere cannot open. The only 'Premiere' row is render.py project.json --export-timeline, which assumes a project file already exists."
+ ],
+ "proposed_fixes_not_applied": [
+ "SKILL.md, multicam row: add '(`--edl` is a cut list, not CMX; for Premiere/Resolve: `--write-project p.json`, then `render.py p.json --export-timeline x.edl`)', and trim a word elsewhere in the row to keep the budget.",
+ "SKILL.md, 'open it in Premiere / Resolve / Final Cut' row: add 'no project yet? multicam.py --write-project / render.py --write-project first'.",
+ "Optional: the description has no 'Premiere/Resolve/Final Cut timeline' or 'TTS segments' words; trigger passed without them, so no description change is proposed on this evidence."
+ ]
+}
diff --git a/evals/run_agents_25.sh b/evals/run_agents_25.sh
new file mode 100644
index 0000000..196f13b
--- /dev/null
+++ b/evals/run_agents_25.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# Eval 25: run each agent prompt once, headless, with the skill loaded from this checkout.
+# sh evals/run_agents_25.sh RUN_DIR [MODEL]
+# Stages fixtures (write_fixtures.py), links the skill in as a project skill under
+# RUN_DIR/<id>/.claude/skills/ffmpeg-skill, runs `claude -p` in that folder and saves the
+# stream-json log to RUN_DIR/<id>/transcript.jsonl. Grade with evals/grade_runs_25.py RUN_DIR.
+set -eu
+RUN=$1
+MODEL=${2:-sonnet}
+REPO=$(cd "$(dirname "$0")/.." && pwd)
+python3 "$REPO/evals/write_fixtures.py" "$RUN" --all "$REPO/evals/agent_prompts_25.json" >/dev/null
+for id in $(python3 -c 'import json,sys; print(" ".join(p["id"] for p in json.load(open(sys.argv[1]))))' "$REPO/evals/agent_prompts_25.json"); do
+ d="$RUN/$id"
+ mkdir -p "$d/.claude/skills"
+ ln -sfn "$REPO" "$d/.claude/skills/ffmpeg-skill"
+ ls "$d" > "$d/.before"
+ prompt=$(python3 -c 'import json,sys; print([p["prompt"] for p in json.load(open(sys.argv[1])) if p["id"]==sys.argv[2]][0])' "$REPO/evals/agent_prompts_25.json" "$id")
+ (cd "$d" && claude -p "$prompt" --model "$MODEL" --output-format stream-json --verbose \
+ --permission-mode bypassPermissions --max-turns 30 > transcript.jsonl 2> stderr.txt) || true
+ ls "$d" > "$d/.after"
+done
diff --git a/evals/trigger/prompts_25.json b/evals/trigger/prompts_25.json
new file mode 100644
index 0000000..f8253b7
--- /dev/null
+++ b/evals/trigger/prompts_25.json
@@ -0,0 +1,17 @@
+[
+ {
+ "id": "g1",
+ "should_trigger": true,
+ "prompt": "My TTS step spits out 40 numbered wav segments and sometimes one fails to generate. Can you stitch whatever's there into a single narration track?"
+ },
+ {
+ "id": "g2",
+ "should_trigger": true,
+ "prompt": "I've got a rough cut described in a JSON project; my editor wants it as something she can open in Final Cut or Resolve to finish the grade."
+ },
+ {
+ "id": "g3",
+ "should_trigger": false,
+ "prompt": "Build me a project management timeline for the Q4 website relaunch: milestones, owners and a Gantt-style view in a spreadsheet."
+ }
+]
diff --git a/evals/trigger/results-2026-09-24-it25.json b/evals/trigger/results-2026-09-24-it25.json
new file mode 100644
index 0000000..6e43822
--- /dev/null
+++ b/evals/trigger/results-2026-09-24-it25.json
@@ -0,0 +1,30 @@
+{
+ "date": "2026-09-24",
+ "iteration": "25",
+ "skill_version": "2.2.4",
+ "judge": "claude -p --model sonnet, no tools, one call per prompt",
+ "method": "evals/trigger/README.md catalog method (skill description + four decoys: image-tools, web-research, copywriter, meeting-notes)",
+ "prompts": "evals/trigger/prompts_25.json",
+ "correct": 3,
+ "total": 3,
+ "detail": [
+ {
+ "id": "g1",
+ "should_trigger": true,
+ "choice": "ffmpeg-skill",
+ "correct": true
+ },
+ {
+ "id": "g2",
+ "should_trigger": true,
+ "choice": "ffmpeg-skill",
+ "correct": true
+ },
+ {
+ "id": "g3",
+ "should_trigger": false,
+ "choice": "none",
+ "correct": true
+ }
+ ]
+}
diff --git a/evals/write_fixtures.py b/evals/write_fixtures.py
index 27328d1..35aa97a 100644
--- a/evals/write_fixtures.py
+++ b/evals/write_fixtures.py
@@ -13,6 +13,7 @@ import json
import ntpath
import os
import posixpath
+import re
import sys
from pathlib import Path
@@ -140,6 +141,17 @@ def build_media_fixture(name: str, target: Path) -> None:
# camA's audio advanced 1.180s -- sync ground truth +1.18s.
vol_expr = "0.8*sin(2*PI*440*(t+1.18))*lt(t+1.18\\,6)+0.01*sin(2*PI*440*(t+1.18))*gt(t+1.18\\,6)"
ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "10.82", "-c:a", "aac", target)
+ elif re.fullmatch(r"tts_\d+\.wav", name):
+ # Eval 25: a TTS segment -- 1.0s of mono 24 kHz tone (pitch varies by index so the joined
+ # file is audibly in order). parts.txt names one more segment than is built.
+ idx = int(re.search(r"\d+", name).group())
+ ffmpeg("-f", "lavfi", "-i", f"sine=frequency={300 + 60 * idx}:sample_rate=24000",
+ "-t", "1", "-ac", "1", target)
+ elif re.fullmatch(r"(seg|cam)_?\w*\d+\.mp4|[ab]\.mp4", name):
+ # Eval 25: a short rendered segment / project clip -- 3.0s 320x180 30 fps with a tone.
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",
+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000", "-t", "3",
+ "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", target)
else:
raise SystemExit(f"no media fixture builder for {name!r}")