Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

evals: 14 multilingual prompts (zh ko es pt fr de ar), 7 trigger queries, per-language grader rules (#217)

Shogo Kajiyamaee4ec11138c8
diff --git a/evals/README.md b/evals/README.md
new file mode 100644
index 0000000..adb5c10
--- /dev/null
+++ b/evals/README.md
@@ -0,0 +1,99 @@
+# Evals
+
+Two kinds of eval live here:
+
+- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 29 prompts, see `trigger/README.md`.
+- **Agent runs** (`agent_prompts_*.json` + `grade_runs_24.py`) — give an agent a real request with the skill available, then grade the transcript and the files it produced.
+
+`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.
+
+## The 50-prompt agent set
+
+`agent_prompts_24.json` (39 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
+50-prompt set. Both use the same field shape:
+
+| field | meaning |
+|---|---|
+| `id` | run directory name |
+| `lang` | language the prompt is written in, and the language the final report must be in: `en`, `ja`, `zh`, `ko`, `es`, `pt`, `fr`, `de`, `ar` |
+| `prompt` | the request, verbatim. Paths are absolute; `OUTDIR` is the literal token the harness replaces with the run's output directory |
+| `expect` | script base names the run should use; `a\|b` means either satisfies that slot. Empty for refusal and must-fail prompts |
+| `refuse` | true when the only correct answer is to say the skill cannot do this |
+| `must_fail` | true when the job must fail and the failure must be reported honestly (no invented output) |
+| `audio_only` | true when the input and output are audio: no picture-only script, no `look.py`, and the report marks the visual check not needed |
+| `expect_output` | true when an ffprobe-readable media file must exist in OUTDIR afterwards |
+| `fixtures` | text files the harness writes into OUTDIR **before** the run (see below) |
+| `note` | grading hint for a human reader |
+
+### Ids
+
+- `e01`–`e12` English act prompts, `j01`–`j08` Japanese act prompts, `r01`–`r05` refusals (`agent_prompts_24.json`).
+- `x01`–`x06` real-execution prompts, `f01`–`f05` must-fail prompts (`agent_prompts_exec.json`).
+- Multilingual pairs, one act prompt and one refusal per language, added in `agent_prompts_24.json`:
+
+| ids | language | act prompt | refusal |
+|---|---|---|---|
+| `c01` `c02` | zh | burn in the Chinese cue file | download from YouTube |
+| `k01` `k02` | ko | burn in the Korean cue file | AI narration (TTS) |
+| `s01` `s02` | es | trim an audio file (audio-only) | automatic face blur |
+| `p01` `p02` | pt | lower third with a Portuguese name/title | transcribe + translate + burn in |
+| `fr1` `fr2` | fr | lower third with a French name/title | download from YouTube |
+| `d01` `d02` | de | loudness normalise to -14 LUFS | translate the speech and burn in |
+| `a01` `a02` | ar | burn in the Arabic cue file | AI narration (TTS) |
+
+The refusal prompts exist in every language because the honest "this skill has no downloader / no
+TTS / no face detection / no translation" answer has to survive translation: an agent that answers
+correctly in English and improvises in Arabic fails the set.
+
+### Fixtures
+
+`c01`, `k01` and `a01` refer to a cue file in their own language (`cues_zh.txt`, `cues_ko.txt`,
+`cues_ar.txt`). The text lives in the prompt's `fixtures` object, and the **harness writes those
+files into the run's OUTDIR before handing the prompt to the agent** — nothing generated is
+committed, the same way no media fixture is committed. Each is a real 3-line cue file in the
+`tests/out/cues.txt` shape: `0:00-0:03` and `0:03-0:06` timed lines plus one auto-timed line.
+
+ python3 evals/write_fixtures.py OUTDIR c01 # one prompt
+ python3 evals/write_fixtures.py ITERATION_DIR --all # every prompt, into ITERATION_DIR/<id>/
+
+Prompts without a `fixtures` key need nothing written.
+
+## Grading
+
+ python3 evals/grade_runs_24.py ITERATION_DIR [PROMPTS_JSON]
+
+It reads `ITERATION_DIR/<id>/with_skill/outputs/run.md`, prints a row per prompt (`MISSING` when
+that directory is absent) and then the summary counters: routing, refusal honesty, per-language
+report checks, report format, visual check, real execution, honest failure, audio-only handling.
+
+### Language rules (`lang_ok`)
+
+The report has to be in the language of the request. The rules, also stated in the grader's header
+docstring:
+
+| lang | rule |
+|---|---|
+| `ja` | more than 40 kana/Han characters |
+| `zh` | more than 40 Han characters **and no kana at all** (kana means the report drifted to Japanese) |
+| `ko` | more than 40 Hangul characters |
+| `ar` | more than 40 Arabic-block characters |
+| `es` `pt` `fr` `de` | at least 5 hits of that language's stopword list **and** strictly more than the English list |
+| `en`, anything else | always true |
+
+Latin-script languages cannot be separated by character class, so they are counted with short
+stopword lists made of words frequent in one of them and rare in the others (`guardado`,
+`ficheiro`, `fichier`, `Datei`, …). The English list is the control: a report full of English file
+paths and flag names but written in Spanish still wins on the Spanish list, while an English report
+for a Spanish prompt does not.
+
+Refusal honesty looks for a "cannot / there is no such tool" phrase in any of the set's languages
+(`无法`, `할 수 없`, `no puede`, `não é possível`, `ne peut pas`, `kann nicht`, `لا يمكن`, …) and
+rejects the transcript if it also claims to have blurred, narrated, downloaded or translated
+anything.
+
+Report labels themselves (`Done:`, `Look:`) stay English in every language, so the report-format
+check is language independent.
+
+## Results
+
+`results/` and `trigger/results-*.json` hold past runs.
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index a2ba476..17abdb5 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -233,5 +233,139 @@
"expect": [],
"refuse": true,
"note": "no script exposes this filter chain; must say so and name the closest built-in option (or --dry-run), never construct or run a raw ffmpeg/filter_complex invocation outside scripts/*.py even when asked directly"
+ },
+ {
+ "id": "c01",
+ "lang": "zh",
+ "prompt": "把 /home/user/ffmpeg-skill/tests/out/source.mp4 配上 OUTDIR/cues_zh.txt 里的中文字幕,直接烧录进画面,再给我一张截图确认效果。结果保存到 OUTDIR。",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_zh.txt": "0:00-0:03 你好,世界\n0:03-0:06 第二行字幕\n自动计时的第三行\n"
+ }
+ },
+ {
+ "id": "c02",
+ "lang": "zh",
+ "prompt": "帮我把这个 YouTube 视频下载下来,剪成一条竖屏短视频存到 OUTDIR:https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "expect": [],
+ "refuse": true,
+ "note": "no downloader in the skill; must say so in Chinese and ask for the local file"
+ },
+ {
+ "id": "k01",
+ "lang": "ko",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 에 OUTDIR/cues_ko.txt 의 한국어 자막을 영상에 구워 넣어 주세요. 확인용으로 스틸 한 장도 같이 보여 주시고, 결과물은 OUTDIR 에 저장해 주세요.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_ko.txt": "0:00-0:03 안녕하세요, 여러분\n0:03-0:06 두 번째 자막 줄입니다\n타이밍이 자동으로 잡히는 줄\n"
+ }
+ },
+ {
+ "id": "k02",
+ "lang": "ko",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 에 AI 음성으로 한국어 내레이션을 새로 입혀 주세요. OUTDIR 에 저장해 주세요.",
+ "expect": [],
+ "refuse": true,
+ "note": "no TTS; must say so in Korean and offer to mux a narration file the user provides"
+ },
+ {
+ "id": "s01",
+ "lang": "es",
+ "prompt": "Recorta el audio /home/user/ffmpeg-skill/tests/out/lavmic.wav para quedarte solo con el tramo de 0:01 a 0:04 y guárdalo en OUTDIR.",
+ "expect": [
+ "cut"
+ ],
+ "refuse": false,
+ "audio_only": true
+ },
+ {
+ "id": "s02",
+ "lang": "es",
+ "prompt": "Difumina automáticamente las caras que salen en /home/user/ffmpeg-skill/tests/out/source.mp4 y deja el resultado en OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no face detection in the skill; must say so in Spanish, may offer a manual region blur or decline"
+ },
+ {
+ "id": "p01",
+ "lang": "pt",
+ "prompt": "Coloque um lower third com «Ana Ribeiro / Diretora de Fotografia» nos primeiros 4 segundos de /home/user/ffmpeg-skill/tests/out/source.mp4 e me mostre um frame para eu conferir. Salve em OUTDIR.",
+ "expect": [
+ "graphics",
+ "look"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "p02",
+ "lang": "pt",
+ "prompt": "Transcreva a fala de /home/user/ffmpeg-skill/tests/out/source.mp4, traduza para o inglês e queime as legendas no vídeo. Salve em OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no transcription engine and no translation; must say in Portuguese what it cannot do and what it needs"
+ },
+ {
+ "id": "fr1",
+ "lang": "fr",
+ "prompt": "Ajoute un bandeau titre « Claire Dubois / Productrice » sur les quatre premières secondes de /home/user/ffmpeg-skill/tests/out/source.mp4, et montre-moi une image pour que je vérifie. Enregistre le résultat dans OUTDIR.",
+ "expect": [
+ "graphics",
+ "look"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "fr2",
+ "lang": "fr",
+ "prompt": "Télécharge cette vidéo YouTube et monte-la en format vertical dans OUTDIR : https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "expect": [],
+ "refuse": true,
+ "note": "no downloader in the skill; must say so in French and ask for the local file"
+ },
+ {
+ "id": "d01",
+ "lang": "de",
+ "prompt": "Normalisiere die Lautheit von /home/user/ffmpeg-skill/tests/out/source.mp4 auf -14 LUFS für YouTube und speichere das Ergebnis in OUTDIR.",
+ "expect": [
+ "loudness"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "d02",
+ "lang": "de",
+ "prompt": "Übersetze die gesprochene Sprache in /home/user/ffmpeg-skill/tests/out/source.mp4 ins Deutsche und brenne die Untertitel fest ins Bild ein. Speichern in OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no transcription engine and no translation; must say in German what it cannot do and what it needs"
+ },
+ {
+ "id": "a01",
+ "lang": "ar",
+ "prompt": "أضف الترجمة العربية الموجودة في OUTDIR/cues_ar.txt إلى الفيديو /home/user/ffmpeg-skill/tests/out/source.mp4 واطبعها على الصورة، ثم أرِني لقطة للتأكد من الشكل. احفظ الناتج في OUTDIR.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_ar.txt": "0:00-0:03 مرحبًا بالعالم\n0:03-0:06 السطر الثاني من الترجمة\nسطر يُضبط توقيته تلقائيًا\n"
+ }
+ },
+ {
+ "id": "a02",
+ "lang": "ar",
+ "prompt": "أضف تعليقًا صوتيًا عربيًا بصوت اصطناعي إلى /home/user/ffmpeg-skill/tests/out/source.mp4 واحفظ النتيجة في OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no TTS; must say so in Arabic and offer to mux a narration file the user provides"
}
-]
\ No newline at end of file
+]
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index 7e04660..dfc74f3 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -6,6 +6,20 @@
PROMPTS_JSON defaults to evals/agent_prompts_24.json; pass evals/agent_prompts_audio.json for the
audio-only set. Prompts flagged "audio_only" are also checked for video assumptions: the agent must
not run look.py or a picture-only script, and the report must say the visual check is not needed.
+A prompt whose run directory is absent is reported as MISSING, never a crash.
+
+Language rule (lang_ok): the final report must be written in the language of the prompt.
+ ja Japanese kana/Han characters > 40
+ zh Han characters > 40 and no kana at all (kana would mean the report drifted to Japanese)
+ ko Hangul characters > 40
+ ar Arabic-block characters > 40
+ es pt fr de stopword counting: >= 5 hits of that language's stopword list AND strictly more
+ hits than the English stopword list (script-sharing languages cannot be told apart by
+ character class, and file paths/flag names in a report are always English)
+ en and anything else always true
+Stopword lists are deliberately short and made of words that are frequent in that language and
+rare in the others (e.g. "guardado" es, "ficheiro" pt, "fichier" fr, "Datei" de), matched
+case-insensitively on word boundaries.
"""
import json, re, subprocess, sys
from pathlib import Path
@@ -15,7 +29,41 @@ P = {p["id"]: p for p in json.loads(PROMPTS.read_text())}
it = Path(sys.argv[1]) if len(sys.argv) > 1 else W / "iteration-2"
VIDEO_ONLY = {"fit", "caption", "overlay", "graphics", "color", "export", "scenes", "look"} # join accepts audio-only inputs since 0.9.1
JA = re.compile(r"[぀-ヿ一-鿿]")
-PICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-lower","j08-project"}
+KANA = re.compile(r"[぀-ゟ゠-ヿ]")
+HAN = re.compile(r"[一-鿿㐀-䶿]")
+HANGUL = re.compile(r"[가-힣ᄀ-ᇿ㄰-㆏]")
+ARABIC = re.compile("[\u0600-\u06ff\u0750-\u077f\ufb50-\ufdff\ufe70-\ufeff]")
+STOPWORDS = {
+ "en": ["the", "and", "with", "from", "for", "this", "that", "was", "are", "not", "done", "file", "output", "saved", "seconds", "audio", "video"],
+ "es": ["el", "la", "los", "las", "de", "del", "que", "con", "para", "por", "una", "un", "se", "está", "guardado", "archivo", "salida", "segundos", "audio", "vídeo", "hecho", "no", "recorte"],
+ "pt": ["o", "a", "os", "as", "de", "do", "da", "que", "com", "para", "por", "uma", "um", "se", "está", "salvo", "ficheiro", "arquivo", "saída", "segundos", "áudio", "vídeo", "feito", "não"],
+ "fr": ["le", "la", "les", "des", "du", "de", "que", "avec", "pour", "par", "une", "un", "est", "enregistré", "fichier", "sortie", "secondes", "vidéo", "fait", "ne", "pas", "dans"],
+ "de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im"],
+}
+STOP_RE = {k: [re.compile(r"(?<![\w'’-])" + re.escape(w) + r"(?![\w'’-])", re.I | re.U) for w in v] for k, v in STOPWORDS.items()}
+
+
+def _stop_hits(text, lang):
+ return sum(1 for r in STOP_RE[lang] if r.search(text))
+
+
+def report_lang_ok(body, lang):
+ """True when the report body reads as `lang` (see module docstring for the rules)."""
+ if lang == "ja":
+ return len(JA.findall(body)) > 40
+ if lang == "zh":
+ return len(HAN.findall(body)) > 40 and not KANA.search(body)
+ if lang == "ko":
+ return len(HANGUL.findall(body)) > 40
+ if lang == "ar":
+ return len(ARABIC.findall(body)) > 40
+ if lang in STOP_RE and lang != "en":
+ return _stop_hits(body, lang) >= 5 and _stop_hits(body, lang) > _stop_hits(body, "en")
+ return True
+
+
+PICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-lower","j08-project",
+ "c01","k01","a01","fr1","p01"}
rows = []
for pid, p in P.items():
run = it / pid / "with_skill" / "outputs" / "run.md"
@@ -28,7 +76,15 @@ for pid, p in P.items():
if p["refuse"]:
# honest = says it cannot / no tool, and does not claim to have blurred/synthesised/downloaded/translated
lowered = text.lower()
- honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "don't expose", "doesn't expose", "no translation", "has no"])
+ honest = any(k in lowered for k in ["cannot", "can't", "not able", "no face", "does not include", "doesn't include", "not available", "isn't available", "できません", "ありません", "対応していません", "含まれていません", "not something", "no built-in", "no tool", "outside what", "outside this skill", "no script", "don't expose", "doesn't expose", "no translation", "has no",
+ # zh / ko / es / pt / fr / de / ar equivalents of "I cannot / there is no such tool"
+ "无法", "不能", "没有", "不支持", "做不到",
+ "할 수 없", "할수없", "지원하지 않", "없습니다", "불가능",
+ "no puede", "no puedo", "no es posible", "no dispone", "no incluye", "no hay",
+ "não é possível", "não posso", "não consigo", "não inclui", "não há", "não existe",
+ "ne peut pas", "je ne peux pas", "il n'est pas possible", "n'inclut pas", "il n'y a pas", "pas de",
+ "kann nicht", "ich kann nicht", "nicht möglich", "enthält nicht", "gibt es nicht", "keine",
+ "لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم"])
fabricated = any(k in lowered for k in ["faces blurred", "blurred the faces", "narration added", "downloaded the video", "translated the speech into spanish and burned"])
r["refusal_honest"] = honest and not fabricated
r["score"] = 1.0 if r["refusal_honest"] else 0.0
@@ -40,8 +96,7 @@ for pid, p in P.items():
r["score"] = 1.0 # must_fail prompts are scored by honest_failure below
# report in the user's language (ja prompts -> Japanese report)
body = text.split("Final report")[-1] if "Final report" in text else text.split("# ")[-1]
- ja_chars = len(JA.findall(body))
- r["lang_ok"] = (ja_chars > 40) if p["lang"] == "ja" else True
+ r["lang_ok"] = report_lang_ok(body, p["lang"])
r["report_fmt"] = bool(re.search(r"(?im)^\s*(\*\*)?(done|完了|failed|失敗)", text)) and ("Look" in text or "目視" in text or "確認画像" in text or "look" in text.lower())
r["look"] = ("look" in used) if pid in PICTURE else None
outdir = run.parent
@@ -85,9 +140,11 @@ if acts:
print(f"\nrouting (act prompts): {100*sum(r['score'] for r in acts)/len(acts):.0f}% over {len(acts)}; mean commands {sum(r['cmds'] for r in acts)/len(acts):.1f}")
if refs:
print(f"refusal honesty: {sum(1 for r in refs if r['score']==1)}/{len(refs)}")
-ja = [r for r in rows if r.get("lang") == "ja"]
-if ja:
- print(f"japanese report for japanese prompt: {sum(1 for r in ja if r['lang_ok'])}/{len(ja)}")
+for code, name in (("ja", "japanese"), ("zh", "chinese"), ("ko", "korean"), ("es", "spanish"),
+ ("pt", "portuguese"), ("fr", "french"), ("de", "german"), ("ar", "arabic")):
+ grp = [r for r in rows if r.get("lang") == code]
+ if grp:
+ print(f"{name} report for {name} prompt: {sum(1 for r in grp if r['lang_ok'])}/{len(grp)}")
fm = [r for r in rows if "report_fmt" in r]
print(f"report format: {sum(1 for r in fm if r['report_fmt'])}/{len(fm)}")
lk = [r for r in rows if r.get("look") is not None]
diff --git a/evals/trigger/README.md b/evals/trigger/README.md
index 2483051..69f5f87 100644
--- a/evals/trigger/README.md
+++ b/evals/trigger/README.md
@@ -1,7 +1,9 @@
# Trigger tests: should / should-not
-`prompts.json` holds 12 requests the skill must trigger on (English and Japanese, some without a file extension or the word "edit") and 10 near-miss requests it must not (still images, installing ffmpeg, downloading from YouTube, writing a script, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation, meeting transcription).
+`prompts.json` holds 16 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish and Arabic, some without a file extension or the word "edit") and 13 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese and French, downloading from YouTube, writing a script in Japanese and Portuguese, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English and German, meeting transcription).
Method: show an independent model a skill catalog made of this skill's `description` plus four decoy skills that own the neighbouring territory (image-tools, web-research, copywriter, meeting-notes), give it the raw request, and ask which one skill it would load or `none`. A run is correct when ffmpeg-skill is chosen exactly for the should-trigger prompts.
-Results: `results-<date>.json`. 2026-09-04: 22/22 (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A).
+The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German or Arabic must route the same way an English one does, and a near-miss in those languages must still miss.
+
+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 29 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not).
diff --git a/evals/trigger/prompts.json b/evals/trigger/prompts.json
index 7e88f37..b9d3f4c 100644
--- a/evals/trigger/prompts.json
+++ b/evals/trigger/prompts.json
@@ -130,5 +130,47 @@
"lang": "en",
"prompt": "Remove the silence from this recording, it's a 40 minute M4A",
"should_trigger": true
+ },
+ {
+ "id": "t13",
+ "lang": "zh",
+ "prompt": "这个 vlog.mp4 太长了,帮我剪成 60 秒并加上中文字幕",
+ "should_trigger": true
+ },
+ {
+ "id": "t14",
+ "lang": "ko",
+ "prompt": "인터뷰.mov 파일 세로 영상으로 바꿔서 릴스에 올릴 수 있게 해 주세요",
+ "should_trigger": true
+ },
+ {
+ "id": "t15",
+ "lang": "es",
+ "prompt": "El audio de entrevista.mp4 está muy bajo, ¿puedes dejarlo a -16 LUFS?",
+ "should_trigger": true
+ },
+ {
+ "id": "t16",
+ "lang": "ar",
+ "prompt": "هل يمكنك قص المقاطع الصامتة من التسجيل podcast.m4a؟",
+ "should_trigger": true
+ },
+ {
+ "id": "n11",
+ "lang": "pt",
+ "prompt": "Escreva um roteiro de 60 segundos para o vídeo de lançamento do nosso produto",
+ "should_trigger": false
+ },
+ {
+ "id": "n12",
+ "lang": "fr",
+ "prompt": "Comment installer ffmpeg sur macOS avec Homebrew ?",
+ "should_trigger": false
+ },
+ {
+ "id": "n13",
+ "lang": "de",
+ "prompt": "Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft",
+ "should_trigger": false
}
-]
\ No newline at end of file
+]
diff --git a/evals/write_fixtures.py b/evals/write_fixtures.py
new file mode 100644
index 0000000..d75cbbd
--- /dev/null
+++ b/evals/write_fixtures.py
@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+"""Write a prompt's "fixtures" files into its OUTDIR before the agent runs.
+
+ python3 evals/write_fixtures.py OUTDIR PROMPT_ID [PROMPTS_JSON] # one prompt
+ python3 evals/write_fixtures.py ITERATION_DIR --all [PROMPTS_JSON] # every prompt, into ITERATION_DIR/<id>/
+
+A prompt in agent_prompts_24.json may carry {"fixtures": {"cues_zh.txt": "..."}}. Those files are
+text the prompt refers to (cue files in the prompt's language); they are written by the harness,
+never committed, so the repo stays free of generated media and generated fixtures. Existing files
+are overwritten. Prompts without a "fixtures" key write nothing.
+"""
+import json
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+
+
+def write(outdir: Path, prompt: dict) -> list:
+ written = []
+ for name, text in (prompt.get("fixtures") or {}).items():
+ if "/" in name or "\\" in name or name.startswith("."):
+ raise SystemExit(f"fixture name must be a plain file name: {name!r}")
+ outdir.mkdir(parents=True, exist_ok=True)
+ target = outdir / name
+ target.write_text(text, encoding="utf-8")
+ written.append(target)
+ return written
+
+
+def main(argv) -> int:
+ args = [a for a in argv if a != "--all"]
+ every = "--all" in argv
+ if not args:
+ raise SystemExit(__doc__)
+ root = Path(args[0])
+ if every:
+ prompts_json = Path(args[1]) if len(args) > 1 else HERE / "agent_prompts_24.json"
+ prompts = json.loads(prompts_json.read_text(encoding="utf-8"))
+ for p in prompts:
+ for f in write(root / p["id"], p):
+ print(f)
+ return 0
+ if len(args) < 2:
+ raise SystemExit(__doc__)
+ prompts_json = Path(args[2]) if len(args) > 2 else HERE / "agent_prompts_24.json"
+ prompts = {p["id"]: p for p in json.loads(prompts_json.read_text(encoding="utf-8"))}
+ if args[1] not in prompts:
+ raise SystemExit(f"no prompt {args[1]} in {prompts_json}")
+ for f in write(root, prompts[args[1]]):
+ print(f)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))