Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

evals: cross-vendor regex-only eval runner, runnable from Cursor/Codex (#281)

Shogo Kajiyamaf1c5eb5c81ae
diff --git a/README.md b/README.md
index 0af5100..5edaa11 100644
--- a/README.md
+++ b/README.md
@@ -504,7 +504,7 @@ python3 tests/test_picture.py # one tool group on its own (analysis, editing, au
npm run release-check # pack, install, contract from the installed copy, MCP == contract, doctor, tests, contract evals
npm run demo # python3 demos/build.py: synthetic footage -> every before/after demo + docs/demos/*.gif
npm run demo:pipeline # examples/make_demo.sh: the older single end-to-end run of every script
-python3 evals/run.py --list # agent eval prompts (see evals/)
+python3 evals/run.py --list # agent eval prompts (see evals/); regex-only grading, runnable from any agent harness -- see evals/README.md "Running this from Cursor, Codex, or another harness"
node bin/install.js --dir /tmp/skills # try the installer without touching ~/.claude
```
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 73ceb9b..cd946f1 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -206,9 +206,15 @@ decided. Recorded here so none of it is re-proposed from scratch.
`cut`, `audio`, `loudness`, `graphics`, `silence`, `probe`), chosen from eval iterations 17-20's
`expect` frequency, not taste. `FFMPEG_SKILL_MCP_FULL=1` lists all 42; the other 30 stay
reachable by name through `tools/call` either way, and the contract still describes all 42.
-- **P1-8 — at least one routing run on a non-Claude model.** Accepted as an `evals/run.py` task:
- the harness reads transcripts and files and is not Claude-specific; the routing/refusal set is
- what a Cursor or Codex run would publish.
+- **P1-8 — at least one routing run on a non-Claude model.** Accepted as an `evals/run.py` task,
+ and **shipped**: `run.py` grades any transcript file against `tasks.json` or the fuller
+ `agent_prompts_24.json`/`agent_prompts_exec.json` sets with substring and regex checks only
+ (`--json` output, an `a|b` alternation in `expect`, and the `grader_expect`/`grader_not`
+ regexes), no `claude` CLI or LLM-judge dependency anywhere in the path. `evals/README.md`'s
+ "Running this from Cursor, Codex, or another harness" section documents the walkthrough. What
+ is still true, and stated rather than fixed: the maintainer has not run another vendor's model
+ from this environment, so no cross-vendor number is published here yet -- the harness is ready
+ for whoever does.
- **P2-9 — issue #143 (real-device corpus) has no acceptance criteria.** Accepted as written in
the issue: the corpus run is the criterion, one row per device family.
- **P2-10 — `--cache` has no failure-path tests.** Accepted and **closed in this patch**: the
@@ -245,10 +251,14 @@ Each item is recorded here so it is not re-proposed from scratch.
- **The evaluation is Claude-only.** True, and it is stated rather than fixed: the agent runs use
a Sonnet agent, the independent grader is Opus, and the trigger judge is Sonnet. The maintainer
cannot run other vendors' models from this environment, so a cross-vendor number would be
- invented, not measured. What is being done instead: `evals/run.py` and the regex grader
- (`evals/grade_runs_24.py`) are being made runnable from Cursor and Codex, so anyone with access
- to another model can run the routing/refusal set — the original 50 prompts — there and publish
- the result. The harness reads transcripts and files; nothing in it is Claude-specific by design.
+ invented, not measured. What is done instead, **shipped**: `evals/run.py` is a regex/substring-
+ only grader runnable from Cursor, Codex, or any harness that can produce a transcript file —
+ `--prompts` selects `tasks.json` (the 29-prompt routing/refusal set) or the fuller
+ `agent_prompts_24.json`/`agent_prompts_exec.json` sets, and `evals/README.md` documents the
+ walkthrough. Anyone with access to another model can run it there and publish the result,
+ reported as its own number (see that section's "Comparing results across vendors") rather than
+ a delta against this repo's own Claude-based iterations. The harness reads transcripts and
+ files; nothing in it is Claude-specific by design.
- **`references/` is rarely read during evals.** Measured and expected. The reference files are
the long-form detail, and SKILL.md's request→script table is what carries a job: iteration 11
showed that pointing agents at the reference files cost tokens without changing outcomes, and
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 30ba3ed..136df27 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -22,7 +22,9 @@ Dialogue event per word (active word scaled/emboldened, past/upcoming colours) i
`\kf` colour-only sweep (#276), and `loudness.py`'s video-stream-copy branch now writes
`-movflags +faststart` like every other mp4-writing path, fixing a `render.py --template`
delivery that lost faststart on its `--normalize` pass (#275). 1.25.0 is a `chore:` release bump
-that closes the remaining mp4 stream-copy paths over to `+faststart` too (#279); no script API
+that closes the remaining mp4 stream-copy paths — `audio.py`, `caption.py --mode mux`, `sync.py`
+and `cut.py` — over to `+faststart` too (#279); `metadata.py`'s own `-c copy` tag/chapter write is
+left alone, since its documented guarantee is the container's metadata only. No script API
change, tool count still 42.
1.19.0 (the `--write-project` feature) is a `feat:` release, hence the minor bump; it does not
start the "1.19.0: observability, portability" theme further down this document, which remains
diff --git a/evals/README.md b/evals/README.md
index 32bf481..95e6bf8 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -221,6 +221,71 @@ anything.
Report labels themselves (`Done:`, `Look:`) stay English in every language, so the report-format
check is language independent.
+## Running this from Cursor, Codex, or another harness
+
+`evals/run.py` and `evals/tasks.json`/`agent_prompts_24.json`/`agent_prompts_exec.json` have no
+dependency on Claude Code, the `claude` CLI, or any particular model: `run.py` only reads a
+transcript file (or a folder of them) and a prompts JSON, and grades with substrings and
+`re.search` — nothing in the grading path calls an LLM. That makes the corpus and the grader
+usable from any agent harness that can (a) hand the skill's prompts to an agent with the
+`ffmpeg-skill` scripts on `PATH`/importable, and (b) save what the agent did to a text file.
+
+1. **Get the prompts.** `python3 evals/run.py --list` prints the 29-prompt routing set
+ (`tasks.json`); add `--prompts evals/agent_prompts_24.json` (97 prompts) or
+ `--prompts evals/agent_prompts_exec.json` (11 prompts, real execution required) for the fuller
+ sets. Each line is `<id>. <request> -> <expected script(s)>`.
+2. **Run each prompt in your own harness.** Paste the `request`/`prompt` text to your Cursor or
+ Codex session with `ffmpeg-skill` available the way that harness exposes local tools (MCP
+ server, a `scripts/*.py` CLI on `PATH`, or however it drives shell commands) exactly as you
+ normally would. Where a prompt says `OUTDIR`, substitute a real directory you created for the
+ run; where a prompt has a `fixtures` or `media_fixtures` key (see above), write those files
+ into that directory first — `evals/write_fixtures.py OUTDIR <id>` does this from any Python
+ environment, no agent needed.
+3. **Save the transcript.** Capture whatever your harness logs for that turn — the commands it
+ ran and the final report it gave the user — to `RESULTS/<id>.txt` (or `<id>.md`). It does not
+ need to be a Claude-shaped transcript; `run.py` only checks for substrings and regexes in the
+ text, so any log that shows the tool invocations and the final report works.
+4. **Grade it.**
+
+ ```bash
+ python3 evals/run.py RESULTS # tasks.json (default)
+ python3 evals/run.py RESULTS --prompts evals/agent_prompts_24.json # the 97-prompt set
+ python3 evals/run.py RESULTS/17.txt --task 17 # one transcript
+ python3 evals/run.py RESULTS --json > my-vendor-results.json # machine-readable
+ ```
+
+ A prompt passes when every `expect` slot (an `a|b` alternation means either satisfies it)
+ appears in the transcript, any `grader_expect` regex the prompt carries also matches, and any
+ `grader_not` regex does not.
+
+### What regex-only grading can and can't tell you
+
+It can tell you, mechanically and reproducibly: whether the agent called the scripts the prompt
+expects (routing), whether a report contains a required disclosure phrase (`grader_expect` — e.g.
+the applied `--jobs` cap, the measured BPM, "the cache misses across ffmpeg versions"), and
+whether it contains a phrase it must not (`grader_not` — e.g. claiming to have rewritten caption
+text the skill never rewrites). That is exactly what this repo's own grading measures too — see
+"Grading" above.
+
+It **cannot** tell you whether the output is actually good: whether a crop is well-composed,
+whether a caption break reads naturally, whether a color grade looks right, or whether a report
+that happens to contain the right substrings is otherwise coherent. This repo's own eval process
+pairs the same kind of regex/substring checks with a qualitative pass (a second model, or a human,
+reading the transcript and the produced media) for exactly that reason — see
+`docs/design-decisions.md`'s "The evaluation is Claude-only" entry. Regex grading alone is a
+floor, not a substitute for that judgment call; it catches "used the wrong tool" or "claimed
+something it didn't do", not "did an acceptable job".
+
+### Comparing results across vendors
+
+A pass rate from a Cursor or Codex run is **not directly comparable** to a pass rate in
+`evals/results/iteration-*.json`: those are produced by a different agent (this repo's own
+Claude-based harness), a different grader script (`grade_runs_24.py`, which adds language checks
+and the `with_skill`/`old_skill` iteration layout this repo uses to compare against no-skill
+baselines) and a different corpus subset in some cases. Report a cross-vendor result as its own
+number, with the model, harness and `--prompts` file named, rather than as a delta against this
+repo's own eval iterations.
+
## Results
`results/` and `trigger/results-*.json` hold past runs.
diff --git a/evals/run.py b/evals/run.py
index e71338b..eac9470 100755
--- a/evals/run.py
+++ b/evals/run.py
@@ -1,61 +1,167 @@
#!/usr/bin/env python3
-"""Score an agent transcript against evals/tasks.json.
+"""Score an agent transcript against an eval prompt corpus, with regex-only grading.
Usage:
python3 evals/run.py transcript.txt --task 5 # did the transcript for task 5 call the expected script(s)?
- python3 evals/run.py --list # print the prompts (paste them to the agent one by one)
- python3 evals/run.py results/ # folder of <id>.txt transcripts -> pass rate
+ python3 evals/run.py --list # print the prompts (paste them to the agent one by one)
+ python3 evals/run.py results/ # folder of <id>.txt transcripts -> pass rate
+ python3 evals/run.py results/ --prompts evals/agent_prompts_24.json # the 108-prompt set
+
+A transcript "passes" when:
+ - every expected script name (or `a|b` alternation slot) appears in it, and
+ - if the prompt carries `grader_expect`, that regex also matches the transcript, and
+ - if the prompt carries `grader_not`, that regex does NOT match the transcript.
+
+Every check here is a substring or regex match against the transcript text -- there is no LLM
+grading step and no dependency on any particular model or vendor. That makes this script usable
+from any agent harness that can produce a transcript file: Claude Code, Cursor, Codex, or a
+plain script that pipes an agent's tool-call log to a text file. See evals/README.md's
+"Running this from Cursor, Codex, or another harness" section for the walkthrough, and be aware
+of what regex-only grading can't catch (also documented there) -- it checks that the right
+scripts were invoked and that a few disclosure phrases are present or absent; it does not judge
+whether the output is actually good.
-A transcript "passes" when every expected script name (or flag) appears in it.
Pair with skill-creator's eval runner for automated, multi-run measurement.
"""
import argparse
import json
+import re
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
-TASKS = json.loads((HERE / "tasks.json").read_text(encoding="utf-8"))["tasks"]
+DEFAULT_PROMPTS = HERE / "tasks.json"
+
+
+def load_tasks(prompts_path: Path) -> list:
+ """Load an eval corpus and normalise it to a common shape.
+
+ Two on-disk shapes are supported:
+ - tasks.json: {"tasks": [{"id": int, "request": str, "expect": [...]}, ...]}
+ - agent_prompts*.json: [{"id": str, "prompt": str, "expect": [...], ...}, ...]
+
+ Both are normalised to a list of dicts with `id`, `request`, `expect`,
+ `grader_expect` (optional regex, or None) and `grader_not` (optional regex, or None).
+ `expect` entries may use `a|b` alternation (with or without a `.py` suffix on each side)
+ to mean "either satisfies this slot".
+ """
+ raw = json.loads(prompts_path.read_text(encoding="utf-8"))
+ items = raw["tasks"] if isinstance(raw, dict) and "tasks" in raw else raw
+ tasks = []
+ for t in items:
+ tasks.append({
+ "id": t["id"],
+ "request": t.get("request") or t.get("prompt") or "",
+ "expect": t.get("expect") or [],
+ "grader_expect": t.get("grader_expect"),
+ "grader_not": t.get("grader_not"),
+ "refuse": bool(t.get("refuse")),
+ })
+ return tasks
+
+
+def _slot_hit(slot: str, text: str) -> bool:
+ """One `expect` entry, e.g. "fit.py|render.py" or "fit|render": true if any alternative
+ (with or without a trailing ".py") appears in the transcript text."""
+ for alt in slot.split("|"):
+ alt = alt.strip()
+ if not alt:
+ continue
+ if alt in text:
+ return True
+ stem = alt[:-3] if alt.endswith(".py") else alt
+ if f"{stem}.py" in text:
+ return True
+ return False
def score(text: str, task: dict) -> tuple:
- missing = [e for e in task["expect"] if e not in text]
- return (not missing, missing)
+ """Regex/substring-only grading. Returns (passed, reasons) where reasons lists every check
+ that failed: missing `expect` slots, a `grader_expect` pattern that never matched, or a
+ `grader_not` pattern that matched when it must not have."""
+ reasons = []
+ missing = [e for e in task["expect"] if not _slot_hit(e, text)]
+ if missing:
+ reasons.append(f"missing: {', '.join(missing)}")
+ ge = task.get("grader_expect")
+ if ge and not re.search(ge, text):
+ reasons.append(f"grader_expect not matched: {ge}")
+ gn = task.get("grader_not")
+ if gn and re.search(gn, text):
+ reasons.append(f"grader_not matched (should not have): {gn}")
+ return (not reasons, reasons)
+
+
+def _find_transcript(folder: Path, task_id) -> Path:
+ """Accept either a flat `<id>.txt`/`<id>.md` file in `folder`, or this repo's own
+ `<id>/with_skill/outputs/run.md` iteration layout, whichever exists."""
+ for name in (f"{task_id}.txt", f"{task_id}.md"):
+ f = folder / name
+ if f.exists():
+ return f
+ nested = folder / str(task_id) / "with_skill" / "outputs" / "run.md"
+ if nested.exists():
+ return nested
+ return None
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("target", nargs="?", help="transcript file or folder of <id>.txt files")
- ap.add_argument("--task", type=int, help="task id for a single transcript")
+ ap.add_argument("target", nargs="?", help="transcript file or folder of <id>.txt/<id>.md files")
+ ap.add_argument("--task", help="task id for a single transcript (int for tasks.json, string for agent_prompts*.json)")
ap.add_argument("--list", action="store_true")
+ ap.add_argument("--prompts", type=Path, default=DEFAULT_PROMPTS,
+ help="eval corpus JSON to grade against (default: evals/tasks.json)")
+ ap.add_argument("--json", action="store_true", help="print machine-readable results instead of text")
args = ap.parse_args()
+
+ tasks = load_tasks(args.prompts)
+
if args.list:
- for t in TASKS:
- print(f"{t['id']:2d}. {t['request']} -> {', '.join(t['expect'])}")
+ for t in tasks:
+ print(f"{t['id']}. {t['request']} -> {', '.join(t['expect']) or '(refusal, no expected script)'}")
return 0
if not args.target:
ap.error("give a transcript file/folder or --list")
+
p = Path(args.target)
results = []
if p.is_dir():
- for t in TASKS:
- f = p / f"{t['id']}.txt"
- if f.exists():
- ok, missing = score(f.read_text(encoding="utf-8", errors="replace"), t)
- results.append((t, ok, missing))
+ for t in tasks:
+ f = _find_transcript(p, t["id"])
+ if f is not None:
+ ok, reasons = score(f.read_text(encoding="utf-8", errors="replace"), t)
+ results.append((t, ok, reasons))
else:
- if not args.task:
+ if args.task is None:
ap.error("--task ID is required for a single transcript")
- t = next((x for x in TASKS if x["id"] == args.task), None)
+ task_id = args.task
+ t = next((x for x in tasks if str(x["id"]) == str(task_id)), None)
if not t:
- ap.error(f"no task {args.task}")
- ok, missing = score(p.read_text(encoding="utf-8", errors="replace"), t)
- results.append((t, ok, missing))
+ ap.error(f"no task {task_id}")
+ ok, reasons = score(p.read_text(encoding="utf-8", errors="replace"), t)
+ results.append((t, ok, reasons))
+
passed = sum(1 for _, ok, _ in results if ok)
- for t, ok, missing in results:
- print(f"{'PASS' if ok else 'FAIL'} {t['id']:2d} {t['request'][:60]:60s}" + ("" if ok else f" missing: {', '.join(missing)}"))
- print(f"{passed}/{len(results)} passed")
- return 0 if passed == len(results) else 1
+
+ if args.json:
+ out = {
+ "prompts": str(args.prompts),
+ "passed": passed,
+ "total": len(results),
+ "results": [
+ {"id": t["id"], "request": t["request"], "passed": ok, "reasons": reasons}
+ for t, ok, reasons in results
+ ],
+ }
+ print(json.dumps(out, indent=2))
+ else:
+ for t, ok, reasons in results:
+ print(f"{'PASS' if ok else 'FAIL'} {str(t['id']):>4s} {t['request'][:60]:60s}" +
+ ("" if ok else " " + "; ".join(reasons)))
+ print(f"{passed}/{len(results)} passed (grading: regex-only, {args.prompts.name})")
+
+ return 0 if results and passed == len(results) else 1
if __name__ == "__main__":
diff --git a/tests/test_contract.py b/tests/test_contract.py
index b6ea7c8..7f0fb19 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -3138,5 +3138,125 @@ class DoctorDetectionTests(unittest.TestCase):
self.assertEqual(d["tools"][tool_name]["usable"], "yes")
+class EvalRunnerTests(unittest.TestCase):
+ """evals/run.py's regex-only grading -- the cross-vendor path (Cursor, Codex, or any
+ harness that produces a transcript file). No LLM judge, no `claude` CLI dependency: every
+ assertion here is against the pure-Python score()/load_tasks() functions."""
+
+ @classmethod
+ def setUpClass(cls):
+ sys.path.insert(0, str(ROOT / "evals"))
+ import importlib
+ cls.evals_run = importlib.import_module("run")
+
+ def test_score_passes_when_every_expect_slot_is_present(self):
+ run = self.evals_run
+ task = {"id": 1, "request": "r", "expect": ["probe.py", "cut.py"],
+ "grader_expect": None, "grader_not": None}
+ ok, reasons = run.score("ran probe.py then cut.py", task)
+ self.assertTrue(ok)
+ self.assertEqual(reasons, [])
+
+ def test_score_reports_missing_expect_slots(self):
+ run = self.evals_run
+ task = {"id": 2, "request": "r", "expect": ["probe.py", "cut.py"],
+ "grader_expect": None, "grader_not": None}
+ ok, reasons = run.score("ran probe.py only", task)
+ self.assertFalse(ok)
+ self.assertIn("cut.py", reasons[0])
+
+ def test_score_alternation_slot_accepts_either_side(self):
+ run = self.evals_run
+ task = {"id": 3, "request": "r", "expect": ["fit|render"], "grader_expect": None, "grader_not": None}
+ self.assertTrue(run.score("used render.py --template", task)[0])
+ self.assertTrue(run.score("used fit.py --duration 2x", task)[0])
+ self.assertFalse(run.score("used caption.py", task)[0])
+
+ def test_score_alternation_slot_accepts_dot_py_or_bare_name(self):
+ run = self.evals_run
+ # tasks.json's expect entries carry ".py"; agent_prompts_24.json's don't. Either spelling
+ # of the slot must match a transcript that names the script with ".py".
+ task = {"id": 4, "request": "r", "expect": ["render"], "grader_expect": None, "grader_not": None}
+ self.assertTrue(run.score("called render.py --template reel", task)[0])
+
+ def test_score_grader_expect_regex_must_match(self):
+ run = self.evals_run
+ task = {"id": 5, "request": "r", "expect": [], "grader_expect": r"(?i)\d+\s*bpm", "grader_not": None}
+ ok, reasons = run.score("tempo is 128 bpm", task)
+ self.assertTrue(ok)
+ ok, reasons = run.score("tempo unknown", task)
+ self.assertFalse(ok)
+ self.assertIn("grader_expect", reasons[0])
+
+ def test_score_grader_not_regex_must_not_match(self):
+ run = self.evals_run
+ task = {"id": 6, "request": "r", "expect": [], "grader_expect": None,
+ "grader_not": r"(?i)rewrote the caption"}
+ ok, reasons = run.score("Done: burned the captions as written", task)
+ self.assertTrue(ok)
+ ok, reasons = run.score("Done: rewrote the caption to fit", task)
+ self.assertFalse(ok)
+ self.assertIn("grader_not", reasons[0])
+
+ def test_load_tasks_normalises_tasks_json_shape(self):
+ run = self.evals_run
+ tasks = run.load_tasks(ROOT / "evals" / "tasks.json")
+ self.assertTrue(tasks)
+ t = tasks[0]
+ self.assertIn("id", t)
+ self.assertIn("request", t)
+ self.assertIsInstance(t["expect"], list)
+ self.assertIsNone(t["grader_expect"])
+
+ def test_load_tasks_normalises_agent_prompts_shape(self):
+ run = self.evals_run
+ tasks = run.load_tasks(ROOT / "evals" / "agent_prompts_24.json")
+ self.assertTrue(tasks)
+ by_id = {t["id"]: t for t in tasks}
+ self.assertIn("e01-reel", by_id)
+ self.assertEqual(by_id["e01-reel"]["expect"][0], "fit|render")
+ # at least one prompt in this set carries a grader_expect/grader_not regex
+ self.assertTrue(any(t["grader_expect"] for t in tasks))
+ self.assertTrue(any(t["grader_not"] for t in tasks))
+
+ def test_cli_grades_a_flat_directory_of_transcripts(self):
+ run = self.evals_run
+ with tempfile.TemporaryDirectory() as tmp:
+ outdir = Path(tmp)
+ (outdir / "1.txt").write_text("ran probe.py then fit.py then export.py", encoding="utf-8")
+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json")
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["passed"], 1)
+ self.assertEqual(doc["total"], 1)
+ self.assertEqual(proc.returncode, 0)
+
+ def test_cli_exit_code_is_nonzero_on_failure(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ outdir = Path(tmp)
+ (outdir / "2.txt").write_text("ran probe.py only", encoding="utf-8")
+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json", check=False)
+ self.assertNotEqual(proc.returncode, 0)
+
+ def test_cli_accepts_a_prompts_flag_pointing_at_the_108_prompt_set(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ outdir = Path(tmp)
+ (outdir / "e02-cut-lossless.txt").write_text("ran probe.py then cut.py --start 0:02 --end 0:08", encoding="utf-8")
+ proc = sh(sys.executable, HERE_EVALS_RUN, str(outdir), "--json",
+ "--prompts", str(ROOT / "evals" / "agent_prompts_24.json"))
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["total"], 1)
+ self.assertEqual(doc["passed"], 1)
+
+ def test_run_py_has_no_claude_cli_or_llm_judge_dependency(self):
+ """The cross-vendor contract: grading is regex/substring only, so a Cursor or Codex
+ session can run this without a `claude` binary or an LLM call in the loop."""
+ src = (ROOT / "evals" / "run.py").read_text(encoding="utf-8")
+ for needle in ("subprocess", "anthropic", "openai", '"claude"', "'claude'", "ANTHROPIC_API_KEY"):
+ self.assertNotIn(needle, src, f"evals/run.py should not depend on {needle!r}")
+
+
+HERE_EVALS_RUN = str(ROOT / "evals" / "run.py")
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)