Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
evals: cross-vendor regex-only eval runner, runnable from Cursor/Codex (#281)
Shogo Kajiyamaf1c5eb5c81ae
diff --git a/README.md b/README.mdindex 0af5100..5edaa11 100644--- a/README.md+++ b/README.md@@ -504,7 +504,7 @@ python3 tests/test_picture.py # one tool group on its own (analysis, editing, aunpm run release-check # pack, install, contract from the installed copy, MCP == contract, doctor, tests, contract evalsnpm run demo # python3 demos/build.py: synthetic footage -> every before/after demo + docs/demos/*.gifnpm run demo:pipeline # examples/make_demo.sh: the older single end-to-end run of every script-python3 evals/run.py --list # agent eval prompts (see evals/)+python3 evals/run.py --list # agent eval prompts (see evals/); regex-only grading, runnable from any agent harness -- see evals/README.md "Running this from Cursor, Codex, or another harness"node bin/install.js --dir /tmp/skills # try the installer without touching ~/.claude```diff --git a/docs/design-decisions.md b/docs/design-decisions.mdindex 73ceb9b..cd946f1 100644--- a/docs/design-decisions.md+++ b/docs/design-decisions.md@@ -206,9 +206,15 @@ decided. Recorded here so none of it is re-proposed from scratch.`cut`, `audio`, `loudness`, `graphics`, `silence`, `probe`), chosen from eval iterations 17-20's`expect` frequency, not taste. `FFMPEG_SKILL_MCP_FULL=1` lists all 42; the other 30 stayreachable by name through `tools/call` either way, and the contract still describes all 42.-- **P1-8 — at least one routing run on a non-Claude model.** Accepted as an `evals/run.py` task:- the harness reads transcripts and files and is not Claude-specific; the routing/refusal set is- what a Cursor or Codex run would publish.+- **P1-8 — at least one routing run on a non-Claude model.** Accepted as an `evals/run.py` task,+ and **shipped**: `run.py` grades any transcript file against `tasks.json` or the fuller+ `agent_prompts_24.json`/`agent_prompts_exec.json` sets with substring and regex checks only+ (`--json` output, an `a|b` alternation in `expect`, and the `grader_expect`/`grader_not`+ regexes), no `claude` CLI or LLM-judge dependency anywhere in the path. `evals/README.md`'s+ "Running this from Cursor, Codex, or another harness" section documents the walkthrough. What+ is still true, and stated rather than fixed: the maintainer has not run another vendor's model+ from this environment, so no cross-vendor number is published here yet -- the harness is ready+ for whoever does.- **P2-9 — issue #143 (real-device corpus) has no acceptance criteria.** Accepted as written inthe issue: the corpus run is the criterion, one row per device family.- **P2-10 — `--cache` has no failure-path tests.** Accepted and **closed in this patch**: the@@ -245,10 +251,14 @@ Each item is recorded here so it is not re-proposed from scratch.- **The evaluation is Claude-only.** True, and it is stated rather than fixed: the agent runs usea Sonnet agent, the independent grader is Opus, and the trigger judge is Sonnet. The maintainercannot run other vendors' models from this environment, so a cross-vendor number would be- invented, not measured. What is being done instead: `evals/run.py` and the regex grader- (`evals/grade_runs_24.py`) are being made runnable from Cursor and Codex, so anyone with access- to another model can run the routing/refusal set — the original 50 prompts — there and publish- the result. The harness reads transcripts and files; nothing in it is Claude-specific by design.+ invented, not measured. What is done instead, **shipped**: `evals/run.py` is a regex/substring-+ only grader runnable from Cursor, Codex, or any harness that can produce a transcript file —+ `--prompts` selects `tasks.json` (the 29-prompt routing/refusal set) or the fuller+ `agent_prompts_24.json`/`agent_prompts_exec.json` sets, and `evals/README.md` documents the+ walkthrough. Anyone with access to another model can run it there and publish the result,+ reported as its own number (see that section's "Comparing results across vendors") rather than+ a delta against this repo's own Claude-based iterations. The harness reads transcripts and+ files; nothing in it is Claude-specific by design.- **`references/` is rarely read during evals.** Measured and expected. The reference files arethe long-form detail, and SKILL.md's request→script table is what carries a job: iteration 11showed that pointing agents at the reference files cost tokens without changing outcomes, anddiff --git a/docs/roadmap.md b/docs/roadmap.mdindex 30ba3ed..136df27 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -22,7 +22,9 @@ Dialogue event per word (active word scaled/emboldened, past/upcoming colours) i`\kf` colour-only sweep (#276), and `loudness.py`'s video-stream-copy branch now writes`-movflags +faststart` like every other mp4-writing path, fixing a `render.py --template`delivery that lost faststart on its `--normalize` pass (#275). 1.25.0 is a `chore:` release bump-that closes the remaining mp4 stream-copy paths over to `+faststart` too (#279); no script API+that closes the remaining mp4 stream-copy paths — `audio.py`, `caption.py --mode mux`, `sync.py`+and `cut.py` — over to `+faststart` too (#279); `metadata.py`'s own `-c copy` tag/chapter write is+left alone, since its documented guarantee is the container's metadata only. No script APIchange, tool count still 42.1.19.0 (the `--write-project` feature) is a `feat:` release, hence the minor bump; it does notstart the "1.19.0: observability, portability" theme further down this document, which remainsdiff --git a/evals/README.md b/evals/README.mdindex 32bf481..95e6bf8 100644--- a/evals/README.md+++ b/evals/README.md@@ -221,6 +221,71 @@ anything.Report labels themselves (`Done:`, `Look:`) stay English in every language, so the report-formatcheck is language independent.+## Running this from Cursor, Codex, or another harness++`evals/run.py` and `evals/tasks.json`/`agent_prompts_24.json`/`agent_prompts_exec.json` have no+dependency on Claude Code, the `claude` CLI, or any particular model: `run.py` only reads a+transcript file (or a folder of them) and a prompts JSON, and grades with substrings and+`re.search` — nothing in the grading path calls an LLM. That makes the corpus and the grader+usable from any agent harness that can (a) hand the skill's prompts to an agent with the+`ffmpeg-skill` scripts on `PATH`/importable, and (b) save what the agent did to a text file.++1. **Get the prompts.** `python3 evals/run.py --list` prints the 29-prompt routing set+ (`tasks.json`); add `--prompts evals/agent_prompts_24.json` (97 prompts) or+ `--prompts evals/agent_prompts_exec.json` (11 prompts, real execution required) for the fuller+ sets. Each line is `<id>. <request> -> <expected script(s)>`.+2. **Run each prompt in your own harness.** Paste the `request`/`prompt` text to your Cursor or+ Codex session with `ffmpeg-skill` available the way that harness exposes local tools (MCP+ server, a `scripts/*.py` CLI on `PATH`, or however it drives shell commands) exactly as you+ normally would. Where a prompt says `OUTDIR`, substitute a real directory you created for the+ run; where a prompt has a `fixtures` or `media_fixtures` key (see above), write those files+ into that directory first — `evals/write_fixtures.py OUTDIR <id>` does this from any Python+ environment, no agent needed.+3. **Save the transcript.** Capture whatever your harness logs for that turn — the commands it+ ran and the final report it gave the user — to `RESULTS/<id>.txt` (or `<id>.md`). It does not+ need to be a Claude-shaped transcript; `run.py` only checks for substrings and regexes in the+ text, so any log that shows the tool invocations and the final report works.+4. **Grade it.**++ ```bash+ python3 evals/run.py RESULTS # tasks.json (default)+ python3 evals/run.py RESULTS --prompts evals/agent_prompts_24.json # the 97-prompt set+ python3 evals/run.py RESULTS/17.txt --task 17 # one transcript+ python3 evals/run.py RESULTS --json > my-vendor-results.json # machine-readable+ ```++ A prompt passes when every `expect` slot (an `a|b` alternation means either satisfies it)+ appears in the transcript, any `grader_expect` regex the prompt carries also matches, and any+ `grader_not` regex does not.++### What regex-only grading can and can't tell you++It can tell you, mechanically and reproducibly: whether the agent called the scripts the prompt+expects (routing), whether a report contains a required disclosure phrase (`grader_expect` — e.g.+the applied `--jobs` cap, the measured BPM, "the cache misses across ffmpeg versions"), and+whether it contains a phrase it must not (`grader_not` — e.g. claiming to have rewritten caption+text the skill never rewrites). That is exactly what this repo's own grading measures too — see+"Grading" above.++It **cannot** tell you whether the output is actually good: whether a crop is well-composed,+whether a caption break reads naturally, whether a color grade looks right, or whether a report+that happens to contain the right substrings is otherwise coherent. This repo's own eval process+pairs the same kind of regex/substring checks with a qualitative pass (a second model, or a human,+reading the transcript and the produced media) for exactly that reason — see+`docs/design-decisions.md`'s "The evaluation is Claude-only" entry. Regex grading alone is a+floor, not a substitute for that judgment call; it catches "used the wrong tool" or "claimed+something it didn't do", not "did an acceptable job".++### Comparing results across vendors++A pass rate from a Cursor or Codex run is **not directly comparable** to a pass rate in+`evals/results/iteration-*.json`: those are produced by a different agent (this repo's own+Claude-based harness), a different grader script (`grade_runs_24.py`, which adds language checks+and the `with_skill`/`old_skill` iteration layout this repo uses to compare against no-skill+baselines) and a different corpus subset in some cases. Report a cross-vendor result as its own+number, with the model, harness and `--prompts` file named, rather than as a delta against this+repo's own eval iterations.+## Results`results/` and `trigger/results-*.json` hold past runs.diff --git a/evals/run.py b/evals/run.pyindex e71338b..eac9470 100755--- a/evals/run.py+++ b/evals/run.py@@ -1,61 +1,167 @@#!/usr/bin/env python3-"""Score an agent transcript against evals/tasks.json.+"""Score an agent transcript against an eval prompt corpus, with regex-only grading.Usage:python3 evals/run.py transcript.txt --task 5 # did the transcript for task 5 call the expected script(s)?- python3 evals/run.py --list # print the prompts (paste them to the agent one by one)- python3 evals/run.py results/ # folder of <id>.txt transcripts -> pass rate+ python3 evals/run.py --list # print the prompts (paste them to the agent one by one)+ python3 evals/run.py results/ # folder of <id>.txt transcripts -> pass rate+ python3 evals/run.py results/ --prompts evals/agent_prompts_24.json # the 108-prompt set++A transcript "passes" when:+ - every expected script name (or `a|b` alternation slot) appears in it, and+ - if the prompt carries `grader_expect`, that regex also matches the transcript, and+ - if the prompt carries `grader_not`, that regex does NOT match the transcript.++Every check here is a substring or regex match against the transcript text -- there is no LLM+grading step and no dependency on any particular model or vendor. That makes this script usable+from any agent harness that can produce a transcript file: Claude Code, Cursor, Codex, or a+plain script that pipes an agent's tool-call log to a text file. See evals/README.md's+"Running this from Cursor, Codex, or another harness" section for the walkthrough, and be aware+of what regex-only grading can't catch (also documented there) -- it checks that the right+scripts were invoked and that a few disclosure phrases are present or absent; it does not judge+whether the output is actually good.-A transcript "passes" when every expected script name (or flag) appears in it.Pair with skill-creator's eval runner for automated, multi-run measurement."""import argparseimport json+import reimport sysfrom pathlib import PathHERE = Path(__file__).resolve().parent-TASKS = json.loads((HERE / "tasks.json").read_text(encoding="utf-8"))["tasks"]+DEFAULT_PROMPTS = HERE / "tasks.json"+++def load_tasks(prompts_path: Path) -> list:+ """Load an eval corpus and normalise it to a common shape.++ Two on-disk shapes are supported:+ - tasks.json: {"tasks": [{"id": int, "request": str, "expect": [...]}, ...]}+ - agent_prompts*.json: [{"id": str, "prompt": str, "expect": [...], ...}, ...]++ Both are normalised to a list of dicts with `id`, `request`, `expect`,+ `grader_expect` (optional regex, or None) and `grader_not` (optional regex, or None).+ `expect` entries may use `a|b` alternation (with or without a `.py` suffix on each side)+ to mean "either satisfies this slot".+ """+ raw = json.loads(prompts_path.read_text(encoding="utf-8"))+ items = raw["tasks"] if isinstance(raw, dict) and "tasks" in raw else raw+ tasks = []+ for t in items:+ tasks.append({+ "id": t["id"],+ "request": t.get("request") or t.get("prompt") or "",+ "expect": t.get("expect") or [],+ "grader_expect": t.get("grader_expect"),+ "grader_not": t.get("grader_not"),+ "refuse": bool(t.get("refuse")),+ })+ return tasks+++def _slot_hit(slot: str, text: str) -> bool:+ """One `expect` entry, e.g. "fit.py|render.py" or "fit|render": true if any alternative+ (with or without a trailing ".py") appears in the transcript text."""+ for alt in slot.split("|"):+ alt = alt.strip()+ if not alt:+ continue+ if alt in text:+ return True+ stem = alt[:-3] if alt.endswith(".py") else alt+ if f"{stem}.py" in text:+ return True+ return Falsedef score(text: str, task: dict) -> tuple:- missing = [e for e in task["expect"] if e not in text]- return (not missing, missing)+ """Regex/substring-only grading. Returns (passed, reasons) where reasons lists every check+ that failed: missing `expect` slots, a `grader_expect` pattern that never matched, or a+ `grader_not` pattern that matched when it must not have."""+ reasons = []+ missing = [e for e in task["expect"] if not _slot_hit(e, text)]+ if missing:+ reasons.append(f"missing: {', '.join(missing)}")+ ge = task.get("grader_expect")+ if ge and not re.search(ge, text):+ reasons.append(f"grader_expect not matched: {ge}")+ gn = task.get("grader_not")+ if gn and re.search(gn, text):+ reasons.append(f"grader_not matched (should not have): {gn}")+ return (not reasons, reasons)+++def _find_transcript(folder: Path, task_id) -> Path:+ """Accept either a flat `<id>.txt`/`<id>.md` file in `folder`, or this repo's own+ `<id>/with_skill/outputs/run.md` iteration layout, whichever exists."""+ for name in (f"{task_id}.txt", f"{task_id}.md"):+ f = folder / name+ if f.exists():+ return f+ nested = folder / str(task_id) / "with_skill" / "outputs" / "run.md"+ if nested.exists():+ return nested+ return Nonedef main() -> int:ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)- ap.add_argument("target", nargs="?", help="transcript file or folder of <id>.txt files")- ap.add_argument("--task", type=int, help="task id for a single transcript")+ ap.add_argument("target", nargs="?", help="transcript file or folder of <id>.txt/<id>.md files")+ ap.add_argument("--task", help="task id for a single transcript (int for tasks.json, string for agent_prompts*.json)")ap.add_argument("--list", action="store_true")+ ap.add_argument("--prompts", type=Path, default=DEFAULT_PROMPTS,+ help="eval corpus JSON to grade against (default: evals/tasks.json)")+ ap.add_argument("--json", action="store_true", help="print machine-readable results instead of text")args = ap.parse_args()++ tasks = load_tasks(args.prompts)+if args.list:- for t in TASKS:- print(f"{t['id']:2d}. {t['request']} -> {', '.join(t['expect'])}")+ for t in tasks:+ print(f"{t['id']}. {t['request']} -> {', '.join(t['expect']) or '(refusal, no expected script)'}")return 0if not args.target:ap.error("give a transcript file/folder or --list")+p = Path(args.target)results = []if p.is_dir():- for t in TASKS:- f = p / f"{t['id']}.txt"- if f.exists():- ok, missing = score(f.read_text(encoding="utf-8", errors="replace"), t)- results.append((t, ok, missing))+ for t in tasks:+ f = _find_transcript(p, t["id"])+ if f is not None:+ ok, reasons = score(f.read_text(encoding="utf-8", errors="replace"), t)+ results.append((t, ok, reasons))else:- if not args.task:+ if args.task is None:ap.error("--task ID is required for a single transcript")- t = next((x for x in TASKS if x["id"] == args.task), None)+ task_id = args.task+ t = next((x for x in tasks if str(x["id"]) == str(task_id)), None)if not t:- ap.error(f"no task {args.task}")- ok, missing = score(p.read_text(encoding="utf-8", errors="replace"), t)- results.append((t, ok, missing))+ ap.error(f"no task {task_id}")+ ok, reasons = score(p.read_text(encoding="utf-8", errors="replace"), t)+ results.append((t, ok, reasons))+passed = sum(1 for _, ok, _ in results if ok)- for t, ok, missing in results:- print(f"{'PASS' if ok else 'FAIL'} {t['id']:2d} {t['request'][:60]:60s}" + ("" if ok else f" missing: {', '.join(missing)}"))- print(f"{passed}/{len(results)} passed")- return 0 if passed == len(results) else 1++ if args.json:+ out = {+ "prompts": str(args.prompts),+ "passed": passed,+ "total": len(results),+ "results": [+ {"id": t["id"], "request": t["request"], "passed": ok, "reasons": reasons}+ for t, ok, reasons in results+ ],+ }+ print(json.dumps(out, indent=2))+ else:+ for t, ok, reasons in results:+ print(f"{'PASS' if ok else 'FAIL'} {str(t['id']):>4s} {t['request'][:60]:60s}" ++ ("" if ok else " " + "; ".join(reasons)))+ print(f"{passed}/{len(results)} passed (grading: regex-only, {args.prompts.name})")++ return 0 if results and passed == len(results) else 1if __name__ == "__main__":diff --git a/tests/test_contract.py b/tests/test_contract.pyindex b6ea7c8..7f0fb19 100644--- a/tests/test_contract.py+++ b/tests/test_contract.py@@ -3138,5 +3138,125 @@ class DoctorDetectionTests(unittest.TestCase):self.assertEqual(d["tools"][tool_name]["usable"], "yes")+class EvalRunnerTests(unittest.TestCase):+ """evals/run.py's regex-only grading -- the cross-vendor path (Cursor, Codex, or any+ harness that produces a transcript file). No LLM judge, no `claude` CLI dependency: every+ assertion here is against the pure-Python score()/load_tasks() functions."""++ @classmethod+ def setUpClass(cls):+ sys.path.insert(0, str(ROOT / "evals"))+ import importlib+ cls.evals_run = importlib.import_module("run")++ def test_score_passes_when_every_expect_slot_is_present(self):+ run = self.evals_run+ task = {"id": 1, "request": "r", "expect": ["probe.py", "cut.py"],+ "grader_expect": None, "grader_not": None}+ ok, reasons = run.score("ran probe.py then cut.py", task)+ self.assertTrue(ok)+ self.assertEqual(reasons, [])++ def test_score_reports_missing_expect_slots(self):+ run = self.evals_run+ task = {"id": 2, "request": "r", "expect": ["probe.py", "cut.py"],+ "grader_expect": None, "grader_not": None}+ ok, reasons = run.score("ran probe.py only", task)+ self.assertFalse(ok)+ self.assertIn("cut.py", reasons[0])++ def test_score_alternation_slot_accepts_either_side(self):+ run = self.evals_run+ task = {"id": 3, "request": "r", "expect": ["fit|render"], "grader_expect": None, "grader_not": None}+ self.assertTrue(run.score("used render.py --template", task)[0])+ self.assertTrue(run.score("used fit.py --duration 2x", task)[0])+ self.assertFalse(run.score("used caption.py", task)[0])++ def test_score_alternation_slot_accepts_dot_py_or_bare_name(self):+ run = self.evals_run+ # tasks.json's expect entries carry ".py"; agent_prompts_24.json's don't. Either spelling+ # of the slot must match a transcript that names the script with ".py".+ task = {"id": 4, "request": "r", "expect": ["render"], "grader_expect": None, "grader_not": None}+ self.assertTrue(run.score("called render.py --template reel", task)[0])++ def test_score_grader_expect_regex_must_match(self):+ run = self.evals_run+ task = {"id": 5, "request": "r", "expect": [], "grader_expect": r"(?i)\d+\s*bpm", "grader_not": None}+ ok, reasons = run.score("tempo is 128 bpm", task)+ self.assertTrue(ok)+ ok, reasons = run.score("tempo unknown", task)+ self.assertFalse(ok)+ self.assertIn("grader_expect", reasons[0])++ def test_score_grader_not_regex_must_not_match(self):+ run = self.evals_run+ task = {"id": 6, "request": "r", "expect": [], "grader_expect": None,+ "grader_not": r"(?i)rewrote the caption"}+ ok, reasons = run.score("Done: burned the captions as written", task)+ self.assertTrue(ok)+ ok, reasons = run.score("Done: rewrote the caption to fit", task)+ self.assertFalse(ok)+ self.assertIn("grader_not", reasons[0])++ def test_load_tasks_normalises_tasks_json_shape(self):+ run = self.evals_run+ tasks = run.load_tasks(ROOT / "evals" / "tasks.json")+ self.assertTrue(tasks)+ t = tasks[0]+ self.assertIn("id", t)+ self.assertIn("request", t)+ self.assertIsInstance(t["expect"], list)+ self.assertIsNone(t["grader_expect"])++ def test_load_tasks_normalises_agent_prompts_shape(self):+ run = self.evals_run+ tasks = run.load_tasks(ROOT / "evals" / "agent_prompts_24.json")+ self.assertTrue(tasks)+ by_id = {t["id"]: t for t in tasks}+ self.assertIn("e01-reel", by_id)+ self.assertEqual(by_id["e01-reel"]["expect"][0], "fit|render")+ # at least one prompt in this set carries a grader_expect/grader_not regex+ self.assertTrue(any(t["grader_expect"] for t in tasks))+ self.assertTrue(any(t["grader_not"] for t in tasks))++ def test_cli_grades_a_flat_directory_of_transcripts(self):+ run = self.evals_run+ with tempfile.TemporaryDirectory() as tmp:+ outdir = Path(tmp)+ (outdir / "1.txt").write_text("ran probe.py then fit.py then export.py", encoding="utf-8")+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json")+ doc = json.loads(proc.stdout)+ self.assertEqual(doc["passed"], 1)+ self.assertEqual(doc["total"], 1)+ self.assertEqual(proc.returncode, 0)++ def test_cli_exit_code_is_nonzero_on_failure(self):+ with tempfile.TemporaryDirectory() as tmp:+ outdir = Path(tmp)+ (outdir / "2.txt").write_text("ran probe.py only", encoding="utf-8")+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json", check=False)+ self.assertNotEqual(proc.returncode, 0)++ def test_cli_accepts_a_prompts_flag_pointing_at_the_108_prompt_set(self):+ with tempfile.TemporaryDirectory() as tmp:+ outdir = Path(tmp)+ (outdir / "e02-cut-lossless.txt").write_text("ran probe.py then cut.py --start 0:02 --end 0:08", encoding="utf-8")+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json",+ "--prompts", str(ROOT / "evals" / "agent_prompts_24.json"))+ doc = json.loads(proc.stdout)+ self.assertEqual(doc["total"], 1)+ self.assertEqual(doc["passed"], 1)++ def test_run_py_has_no_claude_cli_or_llm_judge_dependency(self):+ """The cross-vendor contract: grading is regex/substring only, so a Cursor or Codex+ session can run this without a `claude` binary or an LLM call in the loop."""+ src = (ROOT / "evals" / "run.py").read_text(encoding="utf-8")+ for needle in ("subprocess", "anthropic", "openai", '"claude"', "'claude'", "ANTHROPIC_API_KEY"):+ self.assertNotIn(needle, src, f"evals/run.py should not depend on {needle!r}")+++HERE_EVALS_RUN = str(ROOT / "evals" / "run.py")++if __name__ == "__main__":unittest.main(verbosity=2)