Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
fix: route "timeline for Premiere" from multicam through --write-project; add eval 25 kit (#293)
Shogo Kajiyamaabbe6419dc6c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ba41e30..4151796 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,7 +4,10 @@## Unreleased-(nothing yet)+- SKILL.md and README: `multicam.py --edl` is a plain cut list, not an EDL an editor opens. The multicam+ row now routes "for Premiere/Resolve" through `--write-project p.json` then `render.py --export-timeline`, and+ the "open it in Premiere / Resolve / Final Cut" row says to write a project first when there is none (found+ reading the eval-25 tl2 prompt).## 2.2.4diff --git a/README.md b/README.mdindex 2a4064f..de614a4 100644--- a/README.md+++ b/README.md@@ -259,7 +259,7 @@ These are the rules the skill file gives the agent and the code enforces.|---|---|| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`, `--cache DIR`/`--from STAGE` (reuse the stages that did not change); the captions block takes the fit-size policy (`fit_size`, `min_size`, `fit_size_scope`) and the run reports the caption stage's counts as `caption`; `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table); `--export-timeline edit.fcpxml\|.edl\|.otio` (2.1) writes the cut as an editor timeline for Final Cut, DaVinci Resolve or Premiere instead of rendering it — clips, speed, dissolves, the music bed and chapter markers carry over, and `not_exported` names what stays behind || `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch`; `--jobs N` processes several files at once under one shared `--timeout` |-| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list; `--switch energy` auto-cuts to whichever camera is loudest/talking instead of a hand-built list, with `--min-shot` (minimum shot length) and `--edl` (write the cut list) |+| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list; `--switch energy` auto-cuts to whichever camera is loudest/talking instead of a hand-built list, with `--min-shot` (minimum shot length) and `--edl` (write a plain START-END cut list, not a CMX EDL; for Premiere/Resolve use `--write-project` then `render.py --export-timeline`) || `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |Not tools, but part of the surface: `mcp/server.py` (the MCP transport) and `scripts/_contract.py` (`contract --json`, `doctor`). Per-flag reference for every tool: [references/scripts.md](references/scripts.md).diff --git a/SKILL.md b/SKILL.mdindex 92848a2..93e3f41 100644--- a/SKILL.md+++ b/SKILL.md@@ -5,11 +5,11 @@ description: 'Edit video and audio with local FFmpeg from natural-language reque# ffmpeg-skill-Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script about to run is the cheapest full flag list. A reference file costs as much to read as this file; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end). MCP: `tools/list` shows only the core 12 by default; the other 42 (per this table) stay callable by name via `tools/call`, described by `contract --json`; set `FFMPEG_SKILL_MCP_FULL=1` for all 42.+Scripts live in `scripts/` next to this file; run them with `python3 <skill-dir>/scripts/<name>.py`, and delivery templates in `templates/`. This file is enough to do a job: the table below routes the request, and `--help` on the script about to run is the cheapest full flag list. A reference file costs as much as this one; open one only for a question you have: `references/scripts.md` (every flag of all 42 scripts), `references/devices.md` (iPhone HDR, GoPro, DJI, screen recordings, Zoom), `references/gotchas.md` (the long form of the one-line rules at the end). MCP: `tools/list` shows only the core 12 by default; the other 42 (per this table) stay callable by name via `tools/call`, described by `contract --json`; set `FFMPEG_SKILL_MCP_FULL=1` for all 42.Shared flags, on every script: `--dry-run`; `--json` (output path, a probe of the output, the commands run); `--json-brief` (status/output/verified plus a `summary`; prefer on writing steps); `--fast` (preview quality); `--progress`; `--timeout SECONDS` (`kind: timeout`, default 1800); `--overwrite` (step 7); `--plan FILE` (the dry run as a plan `render.py FILE` runs later; refuses if an input changed). Re-encoding tools also take `--codec h264|hevc|av1|prores` and `--quality N`: unset, SDR is x264, HDR is x265 Main10; `prores` needs `-o NAME.mov`, `h264` refuses HDR (`color.py --to-sdr` first).-Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe — they just skip the artifact and side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per-tool: `contract --json`'s `dry_run` field.+Writing tools run nothing under `--dry-run`; the measuring tools (`probe`, `check`, `sync`, `multicam`, `scenes`, `cropdetect`, `report`, `silence`, `loudness`, `stabilize`) may still run ffmpeg/ffprobe, skipping artifacts/side files (`--edl`, `--sheet`, a generated `.ass`); `verify` ignores the flag. Per tool: `contract --json` `dry_run`.## Workflow (always follow this order)@@ -96,7 +96,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,| "add subtitles from this SRT", "burn in captions" | `caption.py input.mp4 --srt subs.srt` || "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` || "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |-| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` for 1.16 behaviour, `--min-size` sets the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it |+| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` = 1.16 behaviour, `--min-size` the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it || "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (needs a local whisper; else `--text`) || "TikTok-style captions with the words popping" | `caption.py input.mp4 --text cues.txt --animate pop --karaoke` || "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |@@ -139,9 +139,9 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,| "stitch these clips", "add a crossfade" | `join.py a.mp4 b.mp4 c.mp4 --transition fade --duration 0.5` (TTS parts: `--list parts.txt`, `--on-missing skip`) || "several changes to one edit", 3+ steps | `render.py --init project.json`, edit, `render.py project.json` || "I changed one stage, don't redo the rest" | `render.py project.json --cache DIR` — identical stages reused (`--from STAGE` starts there) |-| "open it in Premiere / Resolve / Final Cut" | `render.py project.json --export-timeline edit.fcpxml\|.edl\|.otio` — the cut as a timeline, nothing rendered; read `not_exported` |+| "open it in Premiere / Resolve / Final Cut" | `render.py project.json --export-timeline edit.fcpxml\|.edl\|.otio` — renders nothing; read `not_exported`; no project? `--write-project` || "do this to every file in the folder", "use all the cores" | `batch.py FOLDER --recipe batch.json --jobs auto` (steps or a render project; cached) |-| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` (manual) or `--switch energy` (auto-cuts to the loudest camera, `--min-shot`, `--edl`) |+| "three cameras, cut between them" | `multicam.py camA.mp4 camB.mp4 camC.mp4 --switch "0-20:0,20-40:1,40-60:2"` (manual) or `--switch energy` (`--min-shot`); `--edl` = cut list; editor: `--write-project p.json` → `--export-timeline` || "what's in this file", "how long is it" | `probe.py input.mp4` || "show me what it looks like", "are the captions readable" | `look.py output.mp4 --tiles 3x2`, then view the PNG || "what would you run?", "don't render yet" | any script with `--dry-run` |diff --git a/docs/roadmap.md b/docs/roadmap.mdindex 225d03a..4f9baea 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -657,6 +657,16 @@ from the MCP schema. Two planned items were not done, deliberately:The 1.21.0 "2.0 freeze" above did not happen as written either: 1.21-1.26 kept shippingfeatures, and the deprecation window's 90-day half was waived (docs/contract.md). Nothing else.+## Eval 25 — do natural requests reach 2.1/2.2? (evals only, partial)++`evals/agent_prompts_25.json` (tt1 ja TTS parts with one missing wav, tt2 a video list with one+missing segment, tl1 hand a project to Resolve, tl2 a multicam edit for Premiere), run with+`evals/run_agents_25.sh` and graded on each run's `--json` result document by+`evals/grade_runs_25.py`. The trigger set (`evals/trigger/prompts_25.json`) ran: 3/3. The agent+prompts did **not** run (the sandbox refused headless tool-using agents), so routing is still+unmeasured; `evals/results/iteration-25.json` records the hand-run tool check, the grader+self-test and a proposed (unapplied) SKILL.md fix for multicam's cut-list `--edl` vs a CMX EDL.+## Not planned- A feature that needs 2.0: features land in 1.x behind flags (1.4.0, 1.6.0, 1.7.0 did).diff --git a/evals/agent_prompts_25.json b/evals/agent_prompts_25.jsonnew file mode 100644index 0000000..fc1c9ce--- /dev/null+++ b/evals/agent_prompts_25.json@@ -0,0 +1,102 @@+[+ {+ "id": "tt1",+ "lang": "ja",+ "prompt": "TTSで作った音声が parts.txt に並んでるので順番につないで1本のwavにして。1個くらい欠けてても止まらないで",+ "expect": [+ "join.py",+ "--list",+ "--on-missing skip"+ ],+ "fixtures": {+ "parts.txt": "tts_01.wav\ntts_02.wav\ntts_03.wav\ntts_04.wav\n"+ },+ "media_fixtures": [+ "tts_01.wav",+ "tts_02.wav",+ "tts_04.wav"+ ],+ "missing": "tts_03.wav",+ "result_expect": {+ "tool": "join",+ "ok": true,+ "skipped_names": "tts_03.wav"+ },+ "grader_not": [+ "Traceback"+ ]+ },+ {+ "id": "tt2",+ "lang": "en",+ "prompt": "I have clips listed in list.txt from my TTS+render pipeline; join them into final.mp4 and tell me if any segment is missing before doing anything",+ "expect": [+ "join.py",+ "--list"+ ],+ "fixtures": {+ "list.txt": "seg_01.mp4\nseg_02.mp4\nseg_03.mp4\n"+ },+ "media_fixtures": [+ "seg_01.mp4",+ "seg_03.mp4"+ ],+ "missing": "seg_02.mp4",+ "result_expect": {+ "tool": "join",+ "names": "seg_02.mp4"+ },+ "report_expect": "seg_02",+ "must_not_write": "final.mp4",+ "grader_not": [+ "Traceback",+ "--on-missing skip"+ ]+ },+ {+ "id": "tl1",+ "lang": "en",+ "prompt": "Hand this edit (project.json) to DaVinci Resolve so my editor can finish it",+ "expect": [+ "render.py",+ "--export-timeline"+ ],+ "grader_expect": [+ "--export-timeline[ =]\\S+\\.(fcpxml|otio|edl)"+ ],+ "fixtures": {+ "project.json": "{\n \"output\": \"final.mp4\",\n \"clips\": [\n {\n \"src\": \"a.mp4\",\n \"in\": 0.5,\n \"out\": 2.5\n },\n {\n \"src\": \"b.mp4\",\n \"in\": 0,\n \"out\": 2\n }\n ],\n \"transition\": {\n \"type\": \"fade\",\n \"duration\": 0.5\n }\n}\n"+ },+ "media_fixtures": [+ "a.mp4",+ "b.mp4"+ ],+ "result_expect": {+ "tool": "render",+ "ok": true+ },+ "must_not_write": "final.mp4",+ "grader_not": [+ "Traceback"+ ]+ },+ {+ "id": "tl2",+ "lang": "en",+ "prompt": "Make a timeline of this multicam edit (camA.mp4, camB.mp4) that I can open in Premiere",+ "expect": [+ "multicam.py|render.py"+ ],+ "grader_expect": [+ "\\.(edl|fcpxml|otio)\\b"+ ],+ "media_fixtures": [+ "camA.mp4",+ "camB.mp4"+ ],+ "record_only": true,+ "grader_not": [+ "Traceback"+ ]+ }+]diff --git a/evals/grade_runs_25.py b/evals/grade_runs_25.pynew file mode 100644index 0000000..7f9b861--- /dev/null+++ b/evals/grade_runs_25.py@@ -0,0 +1,154 @@+#!/usr/bin/env python3+"""Grade eval 25 runs (TTS segment joins, editor-timeline handoff) on the result documents.++ python3 evals/grade_runs_25.py RUN_DIR [PROMPTS_JSON] [--json]++RUN_DIR/<id>/ is what evals/run_agents_25.sh leaves: transcript.jsonl (claude -p stream-json)+or transcript.txt (any harness's plain log), plus .before/.after file listings. For every+prompt in PROMPTS_JSON (default evals/agent_prompts_25.json):++ expect every slot ('a|b' = either) appears in a command the agent ran+ grader_expect every regex matches a command the agent ran+ grader_not no regex matches the transcript (commands, tool output, report) -- 'Traceback'+ catches a crashed script; '--on-missing skip' in tt2 catches joining before asking+ result_expect checked against the LAST --json result document the named tool printed, not the+ command line: tool=<script stem> selects it, ok=true needs status completed,+ skipped_names=<file> needs that file in skipped[], names=<file> needs it in+ skipped[], pending[] or problems[]+ report_expect substring of the final report+ must_not_write a file that must not appear in .after (a refused/handoff run renders nothing)+ record_only routing is recorded, verdict 'recorded' instead of pass/fail++A missing run directory is MISSING, never a crash.+"""+import json+import re+import sys+from pathlib import Path++W = Path(__file__).resolve().parent+++def load_run(d: Path):+ """(commands, tool_outputs, final_report) from a run folder."""+ cmds, outs, final = [], [], ""+ jl = d / "transcript.jsonl"+ if jl.exists():+ for line in jl.read_text(encoding="utf-8", errors="replace").splitlines():+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ if ev.get("type") == "result":+ final = ev.get("result") or final+ content = (ev.get("message") or {}).get("content")+ if not isinstance(content, list):+ continue+ for c in content:+ if c.get("type") == "tool_use" and c.get("name") == "Bash":+ cmds.append((c.get("input") or {}).get("command", ""))+ elif c.get("type") == "tool_result":+ body = c.get("content")+ if isinstance(body, list):+ body = "\n".join(x.get("text", "") for x in body if isinstance(x, dict))+ outs.append(body or "")+ elif c.get("type") == "text" and ev.get("type") == "assistant":+ final = c.get("text") or final+ return cmds, outs, final+ tx = d / "transcript.txt"+ if tx.exists():+ text = tx.read_text(encoding="utf-8", errors="replace")+ cmds = [m.group(1) for m in re.finditer(r"^\$ (.+)$", text, re.M)]+ rep = text.split("\n## Report\n", 1)+ return cmds, [rep[0]], rep[1] if len(rep) > 1 else ""+ return None+++def result_docs(outs):+ """Every top-level JSON object printed in tool output (a script's --json document)."""+ docs = []+ dec = json.JSONDecoder()+ for o in outs:+ i = 0+ while True:+ j = o.find("{", i)+ if j < 0:+ break+ try:+ doc, end = dec.raw_decode(o, j)+ except ValueError:+ i = j + 1+ continue+ if isinstance(doc, dict) and "status" in doc:+ docs.append(doc)+ i = end+ return docs+++def _names(doc, *keys):+ return [Path(str(x.get("path", x) if isinstance(x, dict) else x)).name+ for k in keys for x in (doc.get(k) or [])]+++def grade(p, d: Path):+ run = load_run(d)+ if run is None:+ return {"id": p["id"], "verdict": "MISSING"}+ cmds, outs, final = run+ cmdtext = "\n".join(cmds)+ alltext = "\n".join([cmdtext, *outs, final])+ fails = []+ for slot in p.get("expect", []):+ if not any(alt in cmdtext for alt in slot.split("|")):+ fails.append("expect %r not in any command" % slot)+ for rx in p.get("grader_expect", []):+ if not re.search(rx, cmdtext):+ fails.append("grader_expect %r" % rx)+ for rx in p.get("grader_not", []):+ if re.search(re.escape(rx) if rx.startswith("--") else rx, alltext if rx == "Traceback" else cmdtext):+ fails.append("grader_not %r matched" % rx)+ re_exp = p.get("result_expect")+ doc = None+ if re_exp:+ tool = re_exp.get("tool")+ tool_outs = [o for c, o in zip(cmds, outs) if tool + ".py" in c and "--json" in c] if len(cmds) == len(outs) else outs+ docs = result_docs(tool_outs)+ doc = docs[-1] if docs else None+ if doc is None:+ fails.append("no %s --json result document" % tool)+ else:+ if re_exp.get("ok") and doc.get("status") != "completed":+ fails.append("result status %r" % doc.get("status"))+ if "skipped_names" in re_exp and re_exp["skipped_names"] not in _names(doc, "skipped"):+ fails.append("%s not in skipped[]" % re_exp["skipped_names"])+ if "names" in re_exp and re_exp["names"] not in _names(doc, "skipped", "pending", "problems"):+ fails.append("%s not in skipped/pending/problems" % re_exp["names"])+ if p.get("report_expect") and p["report_expect"] not in final:+ fails.append("report does not name %r" % p["report_expect"])+ after = d / ".after"+ if p.get("must_not_write") and after.exists() and p["must_not_write"] in after.read_text().split():+ fails.append("%s was written" % p["must_not_write"])+ verdict = "recorded" if p.get("record_only") else ("pass" if not fails else "fail")+ return {"id": p["id"], "verdict": verdict, "failures": fails,+ "commands": [c for c in cmds if ".py" in c],+ "result_status": doc.get("status") if doc else None}+++def main(argv):+ as_json = "--json" in argv+ args = [a for a in argv if a != "--json"]+ if not args:+ raise SystemExit(__doc__)+ root = Path(args[0])+ prompts = json.loads(Path(args[1] if len(args) > 1 else W / "agent_prompts_25.json").read_text(encoding="utf-8"))+ rows = [grade(p, root / p["id"]) for p in prompts]+ if as_json:+ print(json.dumps(rows, indent=2, ensure_ascii=False))+ else:+ for r in rows:+ print("%-4s %-8s %s" % (r["id"], r["verdict"], "; ".join(r.get("failures") or [])))+ return 0 if all(r["verdict"] in ("pass", "recorded") for r in rows) else 1+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/evals/results/iteration-25.json b/evals/results/iteration-25.jsonnew file mode 100644index 0000000..05c3a7b--- /dev/null+++ b/evals/results/iteration-25.json@@ -0,0 +1,96 @@+{+ "iteration": "25",+ "at_version": "2.2.4",+ "date": "2026-09-24",+ "status": "partial: trigger set run for real; agent prompts NOT run (harness blocked in this sandbox)",+ "scope": "Do natural requests reach the 2.1/2.2 features: join.py --list / --on-missing skip (TTS segments) and render.py --export-timeline (editor handoff)? Four agent prompts (evals/agent_prompts_25.json) and three trigger prompts (evals/trigger/prompts_25.json).",+ "harness": {+ "agent_runner": "evals/run_agents_25.sh RUN_DIR [MODEL]: stages fixtures with evals/write_fixtures.py --all, links this checkout in as a project skill, runs `claude -p` once per prompt, saves transcript.jsonl and .before/.after listings",+ "grader": "evals/grade_runs_25.py RUN_DIR: routing from the Bash commands, pass/fail from the LAST --json result document of the named tool (status, skipped[], problems[]), grader_not 'Traceback' over the whole transcript, must_not_write from .after",+ "fixtures": "write_fixtures.py builders added for tts_NN.wav (1 s mono 24 kHz tone) and seg_NN.mp4 / a.mp4 / b.mp4 (3 s 320x180 testsrc2 + tone); parts.txt and list.txt each name one file that is not built; project.json is a 2-clip project with a fade"+ },+ "agent_runs": {+ "ran": false,+ "why": "`claude` 2.x is installed and answers `claude -p`, but launching headless tool-using agents (claude -p --permission-mode bypassPermissions in the fixture folders) was refused by this sandbox's permission classifier. Not worked around. The maintainer can run `sh evals/run_agents_25.sh <dir>` then `python3 evals/grade_runs_25.py <dir>`.",+ "detail": [+ {+ "id": "tt1",+ "verdict": "not run",+ "expected": "join.py --list parts.txt --on-missing skip -o <x>.wav --json; skipped[] names tts_03.wav"+ },+ {+ "id": "tt2",+ "verdict": "not run",+ "expected": "join.py --list list.txt (fail mode or --dry-run); report names seg_02.mp4; final.mp4 not written"+ },+ {+ "id": "tl1",+ "verdict": "not run",+ "expected": "render.py project.json --export-timeline <x>.fcpxml|.otio|.edl; final.mp4 not written"+ },+ {+ "id": "tl2",+ "verdict": "not run (record-only)",+ "expected": "multicam.py ... --write-project p.json, then render.py p.json --export-timeline <x>.edl|.fcpxml|.otio"+ }+ ]+ },+ "tool_check_not_an_eval": {+ "note": "The expected commands were run BY HAND on the fixtures to confirm the target path works and to feed the grader self-test. This shows the tools behave; it says nothing about whether an agent routes to them.",+ "tt1": "join.py --list parts.txt --on-missing skip --transition none -o voice.wav --json: status completed, 3.000 s, skipped[0] = {index 2, tts_03.wav, missing}",+ "tt2": "join.py --list list.txt -o final.mp4 --json: exit 1, kind input INPUT_INVALID, problems[0] = seg_02.mp4 missing, hint names --on-missing skip, final.mp4 not written",+ "tl1": "render.py project.json --export-timeline edit.fcpxml --json: completed, 2 clips, 3.5 s at 30 fps, fade 15 frames, not_exported [], no media rendered",+ "tl2": "multicam.py has no timeline export of its own: its --edl writes a START-END cut list (cut.py --segments format), not a CMX 3600 EDL. The path to Premiere is multicam.py --write-project p.json, then render.py p.json --export-timeline x.edl. Under --dry-run multicam writes neither mc.json nor the cut list."+ },+ "grader_self_test_not_an_eval": {+ "note": "Hand-made transcripts (real tool output from the hand runs above for the 'good' set; fabricated failures for the 'bad' set), used only to check the grader.",+ "good": {+ "tt1": "pass",+ "tt2": "pass",+ "tl1": "pass",+ "tl2": "recorded"+ },+ "bad": {+ "tt1": "fail (no --on-missing skip; Traceback; no join result doc)",+ "tt2": "fail (--on-missing skip used; seg_02 not in result doc; report silent; final.mp4 written)",+ "tl1": "fail (no --export-timeline; final.mp4 written)",+ "tl2": "MISSING"+ }+ },+ "trigger": {+ "ran": true,+ "result_file": "evals/trigger/results-2026-09-24-it25.json",+ "correct": 3,+ "total": 3,+ "detail": [+ {+ "id": "g1",+ "should_trigger": true,+ "choice": "ffmpeg-skill",+ "correct": true+ },+ {+ "id": "g2",+ "should_trigger": true,+ "choice": "ffmpeg-skill",+ "correct": true+ },+ {+ "id": "g3",+ "should_trigger": false,+ "choice": "none",+ "correct": true+ }+ ]+ },+ "findings": [+ "Trigger: 3/3. The TTS-segment prompt and the editor-handoff prompt (neither names a media extension) both chose ffmpeg-skill; the project-management 'timeline' prompt chose none.",+ "Routing could not be measured: no agent run happened.",+ "tl2 risk (from reading, not from a run): SKILL.md's multicam row lists `--edl` among multicam's options, and 'EDL' is the word an agent reaches for when asked for Premiere. multicam's --edl is a cut list Premiere cannot open. The only 'Premiere' row is render.py project.json --export-timeline, which assumes a project file already exists."+ ],+ "proposed_fixes_not_applied": [+ "SKILL.md, multicam row: add '(`--edl` is a cut list, not CMX; for Premiere/Resolve: `--write-project p.json`, then `render.py p.json --export-timeline x.edl`)', and trim a word elsewhere in the row to keep the budget.",+ "SKILL.md, 'open it in Premiere / Resolve / Final Cut' row: add 'no project yet? multicam.py --write-project / render.py --write-project first'.",+ "Optional: the description has no 'Premiere/Resolve/Final Cut timeline' or 'TTS segments' words; trigger passed without them, so no description change is proposed on this evidence."+ ]+}diff --git a/evals/run_agents_25.sh b/evals/run_agents_25.shnew file mode 100644index 0000000..196f13b--- /dev/null+++ b/evals/run_agents_25.sh@@ -0,0 +1,21 @@+#!/bin/sh+# Eval 25: run each agent prompt once, headless, with the skill loaded from this checkout.+# sh evals/run_agents_25.sh RUN_DIR [MODEL]+# Stages fixtures (write_fixtures.py), links the skill in as a project skill under+# RUN_DIR/<id>/.claude/skills/ffmpeg-skill, runs `claude -p` in that folder and saves the+# stream-json log to RUN_DIR/<id>/transcript.jsonl. Grade with evals/grade_runs_25.py RUN_DIR.+set -eu+RUN=$1+MODEL=${2:-sonnet}+REPO=$(cd "$(dirname "$0")/.." && pwd)+python3 "$REPO/evals/write_fixtures.py" "$RUN" --all "$REPO/evals/agent_prompts_25.json" >/dev/null+for id in $(python3 -c 'import json,sys; print(" ".join(p["id"] for p in json.load(open(sys.argv[1]))))' "$REPO/evals/agent_prompts_25.json"); do+ d="$RUN/$id"+ mkdir -p "$d/.claude/skills"+ ln -sfn "$REPO" "$d/.claude/skills/ffmpeg-skill"+ ls "$d" > "$d/.before"+ prompt=$(python3 -c 'import json,sys; print([p["prompt"] for p in json.load(open(sys.argv[1])) if p["id"]==sys.argv[2]][0])' "$REPO/evals/agent_prompts_25.json" "$id")+ (cd "$d" && claude -p "$prompt" --model "$MODEL" --output-format stream-json --verbose \+ --permission-mode bypassPermissions --max-turns 30 > transcript.jsonl 2> stderr.txt) || true+ ls "$d" > "$d/.after"+donediff --git a/evals/trigger/prompts_25.json b/evals/trigger/prompts_25.jsonnew file mode 100644index 0000000..f8253b7--- /dev/null+++ b/evals/trigger/prompts_25.json@@ -0,0 +1,17 @@+[+ {+ "id": "g1",+ "should_trigger": true,+ "prompt": "My TTS step spits out 40 numbered wav segments and sometimes one fails to generate. Can you stitch whatever's there into a single narration track?"+ },+ {+ "id": "g2",+ "should_trigger": true,+ "prompt": "I've got a rough cut described in a JSON project; my editor wants it as something she can open in Final Cut or Resolve to finish the grade."+ },+ {+ "id": "g3",+ "should_trigger": false,+ "prompt": "Build me a project management timeline for the Q4 website relaunch: milestones, owners and a Gantt-style view in a spreadsheet."+ }+]diff --git a/evals/trigger/results-2026-09-24-it25.json b/evals/trigger/results-2026-09-24-it25.jsonnew file mode 100644index 0000000..6e43822--- /dev/null+++ b/evals/trigger/results-2026-09-24-it25.json@@ -0,0 +1,30 @@+{+ "date": "2026-09-24",+ "iteration": "25",+ "skill_version": "2.2.4",+ "judge": "claude -p --model sonnet, no tools, one call per prompt",+ "method": "evals/trigger/README.md catalog method (skill description + four decoys: image-tools, web-research, copywriter, meeting-notes)",+ "prompts": "evals/trigger/prompts_25.json",+ "correct": 3,+ "total": 3,+ "detail": [+ {+ "id": "g1",+ "should_trigger": true,+ "choice": "ffmpeg-skill",+ "correct": true+ },+ {+ "id": "g2",+ "should_trigger": true,+ "choice": "ffmpeg-skill",+ "correct": true+ },+ {+ "id": "g3",+ "should_trigger": false,+ "choice": "none",+ "correct": true+ }+ ]+}diff --git a/evals/write_fixtures.py b/evals/write_fixtures.pyindex 27328d1..35aa97a 100644--- a/evals/write_fixtures.py+++ b/evals/write_fixtures.py@@ -13,6 +13,7 @@ import jsonimport ntpathimport osimport posixpath+import reimport sysfrom pathlib import Path@@ -140,6 +141,17 @@ def build_media_fixture(name: str, target: Path) -> None:# camA's audio advanced 1.180s -- sync ground truth +1.18s.vol_expr = "0.8*sin(2*PI*440*(t+1.18))*lt(t+1.18\\,6)+0.01*sin(2*PI*440*(t+1.18))*gt(t+1.18\\,6)"ffmpeg("-f", "lavfi", "-i", f"aevalsrc='{vol_expr}':s=48000", "-t", "10.82", "-c:a", "aac", target)+ elif re.fullmatch(r"tts_\d+\.wav", name):+ # Eval 25: a TTS segment -- 1.0s of mono 24 kHz tone (pitch varies by index so the joined+ # file is audibly in order). parts.txt names one more segment than is built.+ idx = int(re.search(r"\d+", name).group())+ ffmpeg("-f", "lavfi", "-i", f"sine=frequency={300 + 60 * idx}:sample_rate=24000",+ "-t", "1", "-ac", "1", target)+ elif re.fullmatch(r"(seg|cam)_?\w*\d+\.mp4|[ab]\.mp4", name):+ # Eval 25: a short rendered segment / project clip -- 3.0s 320x180 30 fps with a tone.+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000", "-t", "3",+ "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", target)else:raise SystemExit(f"no media fixture builder for {name!r}")