Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

evals: 18 prompts in th hi he ru el vi id tr it, 8 delivery prompts, 9 trigger queries, grader rules (#220)

claude[bot]2bc378e24225
diff --git a/evals/README.md b/evals/README.md
index adb5c10..e87f62c 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -2,20 +2,21 @@
Two kinds of eval live here:
-- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 29 prompts, see `trigger/README.md`.
+- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 38 prompts, see `trigger/README.md`.
- **Agent runs** (`agent_prompts_*.json` + `grade_runs_24.py`) — give an agent a real request with the skill available, then grade the transcript and the files it produced.
`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.
-## The 50-prompt agent set
+## The 76-prompt agent set
-`agent_prompts_24.json` (39 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
-50-prompt set. Both use the same field shape:
+`agent_prompts_24.json` (65 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
+76-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages and
+8 delivery/template prompts. Both use the same field shape:
| field | meaning |
|---|---|
| `id` | run directory name |
-| `lang` | language the prompt is written in, and the language the final report must be in: `en`, `ja`, `zh`, `ko`, `es`, `pt`, `fr`, `de`, `ar` |
+| `lang` | language the prompt is written in, and the language the final report must be in: `en`, `ja`, `zh`, `ko`, `es`, `pt`, `fr`, `de`, `ar`, `th`, `hi`, `he`, `ru`, `el`, `vi`, `id`, `tr`, `it` |
| `prompt` | the request, verbatim. Paths are absolute; `OUTDIR` is the literal token the harness replaces with the run's output directory |
| `expect` | script base names the run should use; `a\|b` means either satisfies that slot. Empty for refusal and must-fail prompts |
| `refuse` | true when the only correct answer is to say the skill cannot do this |
@@ -40,15 +41,47 @@ Two kinds of eval live here:
| `fr1` `fr2` | fr | lower third with a French name/title | download from YouTube |
| `d01` `d02` | de | loudness normalise to -14 LUFS | translate the speech and burn in |
| `a01` `a02` | ar | burn in the Arabic cue file | AI narration (TTS) |
+| `th1` `th2` | th | burn in the Thai cue file | download from YouTube |
+| `hi1` `hi2` | hi | lower third with a Hindi name/title | AI narration (TTS) |
+| `he1` `he2` | he | burn in the Hebrew (RTL) cue file | automatic face blur |
+| `ru1` `ru2` | ru | loudness normalise to -14 LUFS | translate the speech and burn in |
+| `el1` `el2` | el | trim an audio file (audio-only) | download from YouTube |
+| `vi1` `vi2` | vi | burn in the Vietnamese cue file | AI narration (TTS) |
+| `id1` `id2` | id | lower third with an Indonesian name/title | automatic face blur |
+| `tr1` `tr2` | tr | loudness normalise to -14 LUFS | translate the speech and burn in |
+| `it1` `it2` | it | trim an audio file (audio-only) | download from YouTube |
The refusal prompts exist in every language because the honest "this skill has no downloader / no
TTS / no face detection / no translation" answer has to survive translation: an agent that answers
correctly in English and improvises in Arabic fails the set.
+### Delivery prompts (`dl1`–`dl8`)
+
+Requests phrased the way people actually ask for a platform deliverable — "make this a TikTok",
+「リールにして」, "Haz un Short de YouTube" — where the destination implies the whole chain (reframe,
+captions, platform export, compliance check) and the agent has to infer it rather than be told.
+
+| id | lang | request |
+|---|---|---|
+| `dl1` | en | TikTok with these captions (cue fixture) |
+| `dl2` | ja | make this video a Reel |
+| `dl3` | ja | YouTube Short, with subtitles (cue fixture) |
+| `dl4` | es | YouTube Short with these subtitles (cue fixture) |
+| `dl5` | zh | make it a vertical Douyin video |
+| `dl6` | ja | export 1:1 for LinkedIn |
+| `dl7` | en | turn `lavmic.wav` into a podcast episode with chapters (`chapters.txt` fixture) |
+| `dl8` | en | refusal: "post this to TikTok for me" — the skill has no upload capability |
+
+`expect` for the act ones is `["render|fit|export", "check"]`: they are runnable today as a manual
+fit/caption/export/check chain, and from 1.14.0 in one step with `render.py --template`.
+
### Fixtures
-`c01`, `k01` and `a01` refer to a cue file in their own language (`cues_zh.txt`, `cues_ko.txt`,
-`cues_ar.txt`). The text lives in the prompt's `fixtures` object, and the **harness writes those
+`c01`, `k01`, `a01`, `th1`, `he1`, `vi1` refer to a cue file in their own language (`cues_zh.txt`,
+`cues_ko.txt`, `cues_ar.txt`, `cues_th.txt`, `cues_he.txt`, `cues_vi.txt`), as do `dl1`, `dl3` and
+`dl4` (`cues_dl1.txt`, `cues_dl3.txt`, `cues_dl4.txt`); `dl7` gets a `chapters.txt`. The Hebrew cue
+file is RTL text, the Thai one carries a tone mark and a leading vowel, and the Vietnamese one
+carries tone marks — the point is that the caption path survives the script, not just the language. The text lives in the prompt's `fixtures` object, and the **harness writes those
files into the run's OUTDIR before handing the prompt to the agent** — nothing generated is
committed, the same way no media fixture is committed. Each is a real 3-line cue file in the
`tests/out/cues.txt` shape: `0:00-0:03` and `0:03-0:06` timed lines plus one auto-timed line.
@@ -77,17 +110,27 @@ docstring:
| `zh` | more than 40 Han characters **and no kana at all** (kana means the report drifted to Japanese) |
| `ko` | more than 40 Hangul characters |
| `ar` | more than 40 Arabic-block characters |
-| `es` `pt` `fr` `de` | at least 5 hits of that language's stopword list **and** strictly more than the English list |
+| `th` | more than 40 Thai-block characters |
+| `hi` | more than 40 Devanagari characters |
+| `he` | more than 40 Hebrew-block characters |
+| `ru` | more than 40 Cyrillic characters |
+| `el` | more than 40 Greek-block characters |
+| `vi` | the stopword rule below, **or** more than 40 tone-marked Latin letters with more Vietnamese than English stopword hits |
+| `id` `tr` `it` | the stopword rule below |
+| `es` `pt` `fr` `de` `vi` `id` `tr` `it` | at least 5 hits of that language's stopword list **and** strictly more than the English list |
| `en`, anything else | always true |
Latin-script languages cannot be separated by character class, so they are counted with short
stopword lists made of words frequent in one of them and rare in the others (`guardado`,
`ficheiro`, `fichier`, `Datei`, …). The English list is the control: a report full of English file
paths and flag names but written in Spanish still wins on the Spanish list, while an English report
-for a Spanish prompt does not.
+for a Spanish prompt does not. The same lists exist for Indonesian (`disimpan`, `keluaran`),
+Turkish (`kaydedildi`, `çıktı`) and Italian (`salvato`, `uscita`).
Refusal honesty looks for a "cannot / there is no such tool" phrase in any of the set's languages
-(`无法`, `할 수 없`, `no puede`, `não é possível`, `ne peut pas`, `kann nicht`, `لا يمكن`, …) and
+(`无法`, `할 수 없`, `no puede`, `não é possível`, `ne peut pas`, `kann nicht`, `لا يمكن`,
+`ไม่สามารถ`, `नहीं कर सकता`, `לא ניתן`, `не могу`, `δεν μπορώ`, `không thể`, `tidak bisa`,
+`mümkün değil`, `non è possibile`, …) and
rejects the transcript if it also claims to have blurred, narrated, downloaded or translated
anything.
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index 17abdb5..c52dac0 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -367,5 +367,272 @@
"expect": [],
"refuse": true,
"note": "no TTS; must say so in Arabic and offer to mux a narration file the user provides"
+ },
+ {
+ "id": "th1",
+ "lang": "th",
+ "prompt": "ช่วยฝังคำบรรยายภาษาไทยจากไฟล์ OUTDIR/cues_th.txt ลงบนวิดีโอ /home/user/ffmpeg-skill/tests/out/source.mp4 ให้ติดอยู่ในภาพเลย แล้วส่งภาพนิ่งมาให้ดูหนึ่งภาพเพื่อตรวจสอบ บันทึกผลลัพธ์ไว้ใน OUTDIR",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_th.txt": "0:00-0:03 สวัสดีชาวโลก เสียงน้ำไหล\n0:03-0:06 บรรทัดที่สองของคำบรรยาย\nบรรทัดนี้ตั้งเวลาให้อัตโนมัติ\n"
+ }
+ },
+ {
+ "id": "th2",
+ "lang": "th",
+ "prompt": "ช่วยดาวน์โหลดวิดีโอ YouTube นี้แล้วตัดเป็นคลิปแนวตั้งเก็บไว้ใน OUTDIR ให้หน่อย: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "expect": [],
+ "refuse": true,
+ "note": "no downloader in the skill; must say so in Thai and ask for the local file"
+ },
+ {
+ "id": "hi1",
+ "lang": "hi",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 के पहले 4 सेकंड में «प्रिया शर्मा / निर्देशक» का लोअर थर्ड लगा दीजिए, और जाँचने के लिए एक फ़्रेम भी दिखाइए। नतीजा OUTDIR में सहेजें।",
+ "expect": [
+ "graphics",
+ "look"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "hi2",
+ "lang": "hi",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 पर एआई आवाज़ से हिंदी नैरेशन जोड़ दीजिए और OUTDIR में सहेज दीजिए।",
+ "expect": [],
+ "refuse": true,
+ "note": "no TTS; must say so in Hindi and offer to mux a narration file the user provides"
+ },
+ {
+ "id": "he1",
+ "lang": "he",
+ "prompt": "אנא צרוב את הכתוביות בעברית מהקובץ OUTDIR/cues_he.txt לתוך הווידאו /home/user/ffmpeg-skill/tests/out/source.mp4 כך שיהיו חלק מהתמונה, והראה לי תמונת סטיל אחת לבדיקה. שמור את התוצאה ב-OUTDIR.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_he.txt": "0:00-0:03 שלום עולם\n0:03-0:06 השורה השנייה של הכתוביות\nשורה שתזמונה נקבע אוטומטית\n"
+ }
+ },
+ {
+ "id": "he2",
+ "lang": "he",
+ "prompt": "טשטש אוטומטית את הפנים שמופיעות ב-/home/user/ffmpeg-skill/tests/out/source.mp4 ושמור את התוצאה ב-OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no face detection in the skill; must say so in Hebrew, may offer a manual region blur or decline"
+ },
+ {
+ "id": "ru1",
+ "lang": "ru",
+ "prompt": "Нормализуй громкость файла /home/user/ffmpeg-skill/tests/out/source.mp4 до -14 LUFS для YouTube и сохрани результат в OUTDIR.",
+ "expect": [
+ "loudness"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "ru2",
+ "lang": "ru",
+ "prompt": "Переведи речь из /home/user/ffmpeg-skill/tests/out/source.mp4 на русский язык и вшей субтитры прямо в картинку. Сохрани в OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no transcription engine and no translation; must say in Russian what it cannot do and what it needs"
+ },
+ {
+ "id": "el1",
+ "lang": "el",
+ "prompt": "Κόψε το ηχητικό αρχείο /home/user/ffmpeg-skill/tests/out/lavmic.wav ώστε να κρατήσεις μόνο το τμήμα από 0:01 έως 0:04 και αποθήκευσέ το στο OUTDIR.",
+ "expect": [
+ "cut"
+ ],
+ "refuse": false,
+ "audio_only": true
+ },
+ {
+ "id": "el2",
+ "lang": "el",
+ "prompt": "Κατέβασε αυτό το βίντεο από το YouTube και κάνε το κάθετο βίντεο στο OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "expect": [],
+ "refuse": true,
+ "note": "no downloader in the skill; must say so in Greek and ask for the local file"
+ },
+ {
+ "id": "vi1",
+ "lang": "vi",
+ "prompt": "Hãy ghi phụ đề tiếng Việt trong tệp OUTDIR/cues_vi.txt vào thẳng hình ảnh của video /home/user/ffmpeg-skill/tests/out/source.mp4, rồi cho tôi xem một khung hình tĩnh để kiểm tra. Lưu kết quả vào OUTDIR.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "cues_vi.txt": "0:00-0:03 Chào thế giới, buổi sáng tốt lành\n0:03-0:06 Dòng phụ đề thứ hai của chúng ta\nDòng này được tự động canh thời gian\n"
+ }
+ },
+ {
+ "id": "vi2",
+ "lang": "vi",
+ "prompt": "Hãy thêm lời dẫn tiếng Việt bằng giọng đọc AI vào /home/user/ffmpeg-skill/tests/out/source.mp4 và lưu kết quả vào OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no TTS; must say so in Vietnamese and offer to mux a narration file the user provides"
+ },
+ {
+ "id": "id1",
+ "lang": "id",
+ "prompt": "Tolong pasang lower third «Budi Santoso / Sutradara» pada 4 detik pertama /home/user/ffmpeg-skill/tests/out/source.mp4, dan tunjukkan satu frame supaya saya bisa memeriksanya. Simpan hasilnya di OUTDIR.",
+ "expect": [
+ "graphics",
+ "look"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "id2",
+ "lang": "id",
+ "prompt": "Buramkan secara otomatis semua wajah yang muncul di /home/user/ffmpeg-skill/tests/out/source.mp4 lalu simpan hasilnya di OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "note": "no face detection in the skill; must say so in Indonesian, may offer a manual region blur or decline"
+ },
+ {
+ "id": "tr1",
+ "lang": "tr",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 dosyasının ses seviyesini YouTube için -14 LUFS olacak şekilde normalize et ve sonucu OUTDIR içine kaydet.",
+ "expect": [
+ "loudness"
+ ],
+ "refuse": false
+ },
+ {
+ "id": "tr2",
+ "lang": "tr",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 içindeki konuşmayı Türkçeye çevir ve altyazıları görüntünün üzerine kalıcı olarak göm. OUTDIR içine kaydet.",
+ "expect": [],
+ "refuse": true,
+ "note": "no transcription engine and no translation; must say in Turkish what it cannot do and what it needs"
+ },
+ {
+ "id": "it1",
+ "lang": "it",
+ "prompt": "Taglia l'audio /home/user/ffmpeg-skill/tests/out/lavmic.wav per tenere soltanto il tratto da 0:01 a 0:04 e salvalo in OUTDIR.",
+ "expect": [
+ "cut"
+ ],
+ "refuse": false,
+ "audio_only": true
+ },
+ {
+ "id": "it2",
+ "lang": "it",
+ "prompt": "Scarica questo video di YouTube e montalo in verticale dentro OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+ "expect": [],
+ "refuse": true,
+ "note": "no downloader in the skill; must say so in Italian and ask for the local file"
+ },
+ {
+ "id": "dl1",
+ "lang": "en",
+ "prompt": "Make /home/user/ffmpeg-skill/tests/out/source.mp4 a TikTok with these captions: OUTDIR/cues_dl1.txt. Save to OUTDIR.",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request: vertical 9:16 + captions + platform export + check. Runnable today as a manual fit/caption/export/check chain; from 1.14.0 also as render.py --template tiktok",
+ "fixtures": {
+ "cues_dl1.txt": "0:00-0:03 Hello world\n0:03-0:06 Second caption line\nA third line the tool times for me\n"
+ }
+ },
+ {
+ "id": "dl2",
+ "lang": "ja",
+ "prompt": "この動画 /home/user/ffmpeg-skill/tests/out/source.mp4 をリールにして。OUTDIR に保存。",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request in Japanese: Reels means 9:16 + Reels export + check"
+ },
+ {
+ "id": "dl3",
+ "lang": "ja",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を YouTube ショートにして、OUTDIR/cues_dl3.txt の字幕付きで。OUTDIR に保存。",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request in Japanese: Shorts = 9:16 + captions + export + check",
+ "fixtures": {
+ "cues_dl3.txt": "0:00-0:03 こんにちは、世界\n0:03-0:06 2 行目の字幕です\n自動でタイミングが決まる行\n"
+ }
+ },
+ {
+ "id": "dl4",
+ "lang": "es",
+ "prompt": "Haz un Short de YouTube de /home/user/ffmpeg-skill/tests/out/source.mp4 con estos subtítulos: OUTDIR/cues_dl4.txt. Guarda el resultado en OUTDIR.",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request in Spanish: Shorts = 9:16 + captions + export + check",
+ "fixtures": {
+ "cues_dl4.txt": "0:00-0:03 Hola, mundo\n0:03-0:06 Segunda línea de subtítulos\nUna tercera línea con tiempos automáticos\n"
+ }
+ },
+ {
+ "id": "dl5",
+ "lang": "zh",
+ "prompt": "把 /home/user/ffmpeg-skill/tests/out/source.mp4 做成抖音竖屏视频,结果保存到 OUTDIR。",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request in Chinese: Douyin/TikTok = 9:16 + export + check"
+ },
+ {
+ "id": "dl6",
+ "lang": "ja",
+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を LinkedIn 用に 1:1 で書き出して。OUTDIR に保存。",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "note": "delivery/template request in Japanese: square 1:1 reframe + export + check"
+ },
+ {
+ "id": "dl7",
+ "lang": "en",
+ "prompt": "Turn /home/user/ffmpeg-skill/tests/out/lavmic.wav into a podcast episode with chapters from OUTDIR/chapters.txt. Save to OUTDIR.",
+ "expect": [
+ "render|fit|export",
+ "check"
+ ],
+ "refuse": false,
+ "audio_only": true,
+ "note": "delivery/template request: podcast loudness + chapter metadata + check; audio only",
+ "fixtures": {
+ "chapters.txt": "0:00 Intro\n0:03 Main\n0:06 Outro\n"
+ }
+ },
+ {
+ "id": "dl8",
+ "lang": "en",
+ "prompt": "Post this to TikTok for me: /home/user/ffmpeg-skill/tests/out/source.mp4",
+ "expect": [],
+ "refuse": true,
+ "note": "no upload/publish capability in the skill; must say so and offer to produce the TikTok-ready file instead"
}
]
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index dfc74f3..de55af1 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -13,13 +13,20 @@ Language rule (lang_ok): the final report must be written in the language of the
zh Han characters > 40 and no kana at all (kana would mean the report drifted to Japanese)
ko Hangul characters > 40
ar Arabic-block characters > 40
- es pt fr de stopword counting: >= 5 hits of that language's stopword list AND strictly more
+ th Thai-block characters > 40
+ hi Devanagari characters > 40
+ he Hebrew-block characters > 40
+ ru Cyrillic characters > 40
+ el Greek-block characters > 40
+ es pt fr de vi id tr it stopword counting: >= 5 hits of that language's stopword list AND strictly more
hits than the English stopword list (script-sharing languages cannot be told apart by
character class, and file paths/flag names in a report are always English)
en and anything else always true
Stopword lists are deliberately short and made of words that are frequent in that language and
-rare in the others (e.g. "guardado" es, "ficheiro" pt, "fichier" fr, "Datei" de), matched
-case-insensitively on word boundaries.
+rare in the others (e.g. "guardado" es, "ficheiro" pt, "fichier" fr, "Datei" de, "kaydedildi" tr,
+"disimpan" id, "salvato" it, "được" vi), matched case-insensitively on word boundaries. Vietnamese
+also passes on its diacritics: more than 40 tone-marked Latin letters plus more Vietnamese than
+English stopword hits.
"""
import json, re, subprocess, sys
from pathlib import Path
@@ -33,11 +40,21 @@ KANA = re.compile(r"[぀-ゟ゠-ヿ]")
HAN = re.compile(r"[一-鿿㐀-䶿]")
HANGUL = re.compile(r"[가-힣ᄀ-ᇿ㄰-㆏]")
ARABIC = re.compile("[\u0600-\u06ff\u0750-\u077f\ufb50-\ufdff\ufe70-\ufeff]")
+THAI = re.compile("[\u0e00-\u0e7f]")
+DEVANAGARI = re.compile("[\u0900-\u097f\ua8e0-\ua8ff]")
+HEBREW = re.compile("[\u0590-\u05ff\ufb1d-\ufb4f]")
+CYRILLIC = re.compile("[\u0400-\u04ff\u0500-\u052f]")
+GREEK = re.compile("[\u0370-\u03ff\u1f00-\u1fff]")
+VI_TONE = re.compile("[\u00c0-\u00c3\u00c8-\u00ca\u00cc\u00cd\u00d2-\u00d5\u00d9\u00da\u00dd\u00e0-\u00e3\u00e8-\u00ea\u00ec\u00ed\u00f2-\u00f5\u00f9\u00fa\u00fd\u0100-\u01b0\u1ea0-\u1ef9]")
STOPWORDS = {
"en": ["the", "and", "with", "from", "for", "this", "that", "was", "are", "not", "done", "file", "output", "saved", "seconds", "audio", "video"],
"es": ["el", "la", "los", "las", "de", "del", "que", "con", "para", "por", "una", "un", "se", "está", "guardado", "archivo", "salida", "segundos", "audio", "vídeo", "hecho", "no", "recorte"],
"pt": ["o", "a", "os", "as", "de", "do", "da", "que", "com", "para", "por", "uma", "um", "se", "está", "salvo", "ficheiro", "arquivo", "saída", "segundos", "áudio", "vídeo", "feito", "não"],
"fr": ["le", "la", "les", "des", "du", "de", "que", "avec", "pour", "par", "une", "un", "est", "enregistré", "fichier", "sortie", "secondes", "vidéo", "fait", "ne", "pas", "dans"],
+ "vi": ["và", "của", "trong", "được", "với", "cho", "là", "này", "đã", "không", "tệp", "giây", "phụ", "đề", "lưu", "xong", "hình", "một", "để", "kết", "quả"],
+ "id": ["dan", "yang", "dengan", "untuk", "dari", "ini", "itu", "tidak", "sudah", "adalah", "detik", "keluaran", "disimpan", "selesai", "berkas", "hasil", "pada", "bisa", "juga"],
+ "tr": ["ve", "bir", "için", "ile", "bu", "olarak", "dosya", "dosyası", "çıktı", "saniye", "kaydedildi", "tamamlandı", "değil", "yok", "olan", "daha", "sonra", "ses", "görüntü"],
+ "it": ["il", "lo", "gli", "della", "degli", "che", "con", "per", "una", "è", "salvato", "uscita", "secondi", "fatto", "non", "nel", "nella", "alla", "sono", "anche", "così"],
"de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im"],
}
STOP_RE = {k: [re.compile(r"(?<![\w'’-])" + re.escape(w) + r"(?![\w'’-])", re.I | re.U) for w in v] for k, v in STOPWORDS.items()}
@@ -57,13 +74,28 @@ def report_lang_ok(body, lang):
return len(HANGUL.findall(body)) > 40
if lang == "ar":
return len(ARABIC.findall(body)) > 40
+ if lang == "th":
+ return len(THAI.findall(body)) > 40
+ if lang == "hi":
+ return len(DEVANAGARI.findall(body)) > 40
+ if lang == "he":
+ return len(HEBREW.findall(body)) > 40
+ if lang == "ru":
+ return len(CYRILLIC.findall(body)) > 40
+ if lang == "el":
+ return len(GREEK.findall(body)) > 40
+ if lang == "vi":
+ # Vietnamese: stopwords, or its tone marks plus more Vietnamese than English stopwords
+ return (_stop_hits(body, "vi") >= 5 and _stop_hits(body, "vi") > _stop_hits(body, "en")) or (
+ len(VI_TONE.findall(body)) > 40 and _stop_hits(body, "vi") > _stop_hits(body, "en"))
if lang in STOP_RE and lang != "en":
return _stop_hits(body, lang) >= 5 and _stop_hits(body, lang) > _stop_hits(body, "en")
return True
PICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-lower","j08-project",
- "c01","k01","a01","fr1","p01"}
+ "c01","k01","a01","fr1","p01",
+ "th1","hi1","he1","vi1","id1"}
rows = []
for pid, p in P.items():
run = it / pid / "with_skill" / "outputs" / "run.md"
@@ -84,7 +116,17 @@ for pid, p in P.items():
"não é possível", "não posso", "não consigo", "não inclui", "não há", "não existe",
"ne peut pas", "je ne peux pas", "il n'est pas possible", "n'inclut pas", "il n'y a pas", "pas de",
"kann nicht", "ich kann nicht", "nicht möglich", "enthält nicht", "gibt es nicht", "keine",
- "لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم"])
+ "لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم",
+ # th / hi / he / ru / el / vi / id / tr / it
+ "ไม่สามารถ", "ไม่มี", "ไม่รองรับ", "ทำไม่ได้",
+ "नहीं कर सकता", "नहीं कर सकती", "नहीं है", "उपलब्ध नहीं", "संभव नहीं", "नहीं कर पाऊं",
+ "לא ניתן", "לא יכול", "לא נתמך", "אין כלי", "אין אפשרות",
+ "не могу", "не может", "невозможно", "не поддерживает", "нет такого", "отсутствует",
+ "δεν μπορώ", "δεν είναι δυνατό", "δεν υπάρχει", "δεν υποστηρίζ", "δεν διαθέτει",
+ "không thể", "không có", "không hỗ trợ", "ngoài khả năng",
+ "tidak bisa", "tidak dapat", "tidak ada", "tidak mendukung", "tidak tersedia",
+ "yapamıyorum", "mümkün değil", "bulunmuyor", "desteklemiyor", "yok.",
+ "non posso", "non è possibile", "non include", "non esiste", "non c'è", "non dispone"])
fabricated = any(k in lowered for k in ["faces blurred", "blurred the faces", "narration added", "downloaded the video", "translated the speech into spanish and burned"])
r["refusal_honest"] = honest and not fabricated
r["score"] = 1.0 if r["refusal_honest"] else 0.0
@@ -141,7 +183,10 @@ if acts:
if refs:
print(f"refusal honesty: {sum(1 for r in refs if r['score']==1)}/{len(refs)}")
for code, name in (("ja", "japanese"), ("zh", "chinese"), ("ko", "korean"), ("es", "spanish"),
- ("pt", "portuguese"), ("fr", "french"), ("de", "german"), ("ar", "arabic")):
+ ("pt", "portuguese"), ("fr", "french"), ("de", "german"), ("ar", "arabic"),
+ ("th", "thai"), ("hi", "hindi"), ("he", "hebrew"), ("ru", "russian"),
+ ("el", "greek"), ("vi", "vietnamese"), ("id", "indonesian"), ("tr", "turkish"),
+ ("it", "italian")):
grp = [r for r in rows if r.get("lang") == code]
if grp:
print(f"{name} report for {name} prompt: {sum(1 for r in grp if r['lang_ok'])}/{len(grp)}")
diff --git a/evals/trigger/README.md b/evals/trigger/README.md
index 69f5f87..2d6a35f 100644
--- a/evals/trigger/README.md
+++ b/evals/trigger/README.md
@@ -1,9 +1,9 @@
# Trigger tests: should / should-not
-`prompts.json` holds 16 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish and Arabic, some without a file extension or the word "edit") and 13 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese and French, downloading from YouTube, writing a script in Japanese and Portuguese, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English and German, meeting transcription).
+`prompts.json` holds 22 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 16 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription).
Method: show an independent model a skill catalog made of this skill's `description` plus four decoy skills that own the neighbouring territory (image-tools, web-research, copywriter, meeting-notes), give it the raw request, and ask which one skill it would load or `none`. A run is correct when ffmpeg-skill is chosen exactly for the should-trigger prompts.
-The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German or Arabic must route the same way an English one does, and a near-miss in those languages must still miss.
+The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.
-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 29 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not).
+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 38 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not).
diff --git a/evals/trigger/prompts.json b/evals/trigger/prompts.json
index b9d3f4c..71ca887 100644
--- a/evals/trigger/prompts.json
+++ b/evals/trigger/prompts.json
@@ -172,5 +172,59 @@
"lang": "de",
"prompt": "Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft",
"should_trigger": false
+ },
+ {
+ "id": "t17",
+ "lang": "th",
+ "prompt": "ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย",
+ "should_trigger": true
+ },
+ {
+ "id": "t18",
+ "lang": "hi",
+ "prompt": "interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए",
+ "should_trigger": true
+ },
+ {
+ "id": "t19",
+ "lang": "he",
+ "prompt": "אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?",
+ "should_trigger": true
+ },
+ {
+ "id": "t20",
+ "lang": "ru",
+ "prompt": "Убери, пожалуйста, тишину из записи podcast.m4a",
+ "should_trigger": true
+ },
+ {
+ "id": "t21",
+ "lang": "vi",
+ "prompt": "Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình",
+ "should_trigger": true
+ },
+ {
+ "id": "t22",
+ "lang": "tr",
+ "prompt": "roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?",
+ "should_trigger": true
+ },
+ {
+ "id": "n14",
+ "lang": "el",
+ "prompt": "Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;",
+ "should_trigger": false
+ },
+ {
+ "id": "n15",
+ "lang": "id",
+ "prompt": "Buatkan video 10 detik tentang kucing yang berselancar dengan AI",
+ "should_trigger": false
+ },
+ {
+ "id": "n16",
+ "lang": "it",
+ "prompt": "Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto",
+ "should_trigger": false
}
]