Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
evals: 18 prompts in th hi he ru el vi id tr it, 8 delivery prompts, 9 trigger queries, grader rules (#220)
claude[bot]2bc378e24225
diff --git a/evals/README.md b/evals/README.mdindex adb5c10..e87f62c 100644--- a/evals/README.md+++ b/evals/README.md@@ -2,20 +2,21 @@Two kinds of eval live here:-- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 29 prompts, see `trigger/README.md`.+- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 38 prompts, see `trigger/README.md`.- **Agent runs** (`agent_prompts_*.json` + `grade_runs_24.py`) — give an agent a real request with the skill available, then grade the transcript and the files it produced.`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.-## The 50-prompt agent set+## The 76-prompt agent set-`agent_prompts_24.json` (39 prompts) and `agent_prompts_exec.json` (11 prompts) together make the-50-prompt set. Both use the same field shape:+`agent_prompts_24.json` (65 prompts) and `agent_prompts_exec.json` (11 prompts) together make the+76-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages and+8 delivery/template prompts. Both use the same field shape:| field | meaning ||---|---|| `id` | run directory name |-| `lang` | language the prompt is written in, and the language the final report must be in: `en`, `ja`, `zh`, `ko`, `es`, `pt`, `fr`, `de`, `ar` |+| `lang` | language the prompt is written in, and the language the final report must be in: `en`, `ja`, `zh`, `ko`, `es`, `pt`, `fr`, `de`, `ar`, `th`, `hi`, `he`, `ru`, `el`, `vi`, `id`, `tr`, `it` || `prompt` | the request, verbatim. Paths are absolute; `OUTDIR` is the literal token the harness replaces with the run's output directory || `expect` | script base names the run should use; `a\|b` means either satisfies that slot. Empty for refusal and must-fail prompts || `refuse` | true when the only correct answer is to say the skill cannot do this |@@ -40,15 +41,47 @@ Two kinds of eval live here:| `fr1` `fr2` | fr | lower third with a French name/title | download from YouTube || `d01` `d02` | de | loudness normalise to -14 LUFS | translate the speech and burn in || `a01` `a02` | ar | burn in the Arabic cue file | AI narration (TTS) |+| `th1` `th2` | th | burn in the Thai cue file | download from YouTube |+| `hi1` `hi2` | hi | lower third with a Hindi name/title | AI narration (TTS) |+| `he1` `he2` | he | burn in the Hebrew (RTL) cue file | automatic face blur |+| `ru1` `ru2` | ru | loudness normalise to -14 LUFS | translate the speech and burn in |+| `el1` `el2` | el | trim an audio file (audio-only) | download from YouTube |+| `vi1` `vi2` | vi | burn in the Vietnamese cue file | AI narration (TTS) |+| `id1` `id2` | id | lower third with an Indonesian name/title | automatic face blur |+| `tr1` `tr2` | tr | loudness normalise to -14 LUFS | translate the speech and burn in |+| `it1` `it2` | it | trim an audio file (audio-only) | download from YouTube |The refusal prompts exist in every language because the honest "this skill has no downloader / noTTS / no face detection / no translation" answer has to survive translation: an agent that answerscorrectly in English and improvises in Arabic fails the set.+### Delivery prompts (`dl1`–`dl8`)++Requests phrased the way people actually ask for a platform deliverable — "make this a TikTok",+「リールにして」, "Haz un Short de YouTube" — where the destination implies the whole chain (reframe,+captions, platform export, compliance check) and the agent has to infer it rather than be told.++| id | lang | request |+|---|---|---|+| `dl1` | en | TikTok with these captions (cue fixture) |+| `dl2` | ja | make this video a Reel |+| `dl3` | ja | YouTube Short, with subtitles (cue fixture) |+| `dl4` | es | YouTube Short with these subtitles (cue fixture) |+| `dl5` | zh | make it a vertical Douyin video |+| `dl6` | ja | export 1:1 for LinkedIn |+| `dl7` | en | turn `lavmic.wav` into a podcast episode with chapters (`chapters.txt` fixture) |+| `dl8` | en | refusal: "post this to TikTok for me" — the skill has no upload capability |++`expect` for the act ones is `["render|fit|export", "check"]`: they are runnable today as a manual+fit/caption/export/check chain, and from 1.14.0 in one step with `render.py --template`.+### Fixtures-`c01`, `k01` and `a01` refer to a cue file in their own language (`cues_zh.txt`, `cues_ko.txt`,-`cues_ar.txt`). The text lives in the prompt's `fixtures` object, and the **harness writes those+`c01`, `k01`, `a01`, `th1`, `he1`, `vi1` refer to a cue file in their own language (`cues_zh.txt`,+`cues_ko.txt`, `cues_ar.txt`, `cues_th.txt`, `cues_he.txt`, `cues_vi.txt`), as do `dl1`, `dl3` and+`dl4` (`cues_dl1.txt`, `cues_dl3.txt`, `cues_dl4.txt`); `dl7` gets a `chapters.txt`. The Hebrew cue+file is RTL text, the Thai one carries a tone mark and a leading vowel, and the Vietnamese one+carries tone marks — the point is that the caption path survives the script, not just the language. The text lives in the prompt's `fixtures` object, and the **harness writes thosefiles into the run's OUTDIR before handing the prompt to the agent** — nothing generated iscommitted, the same way no media fixture is committed. Each is a real 3-line cue file in the`tests/out/cues.txt` shape: `0:00-0:03` and `0:03-0:06` timed lines plus one auto-timed line.@@ -77,17 +110,27 @@ docstring:| `zh` | more than 40 Han characters **and no kana at all** (kana means the report drifted to Japanese) || `ko` | more than 40 Hangul characters || `ar` | more than 40 Arabic-block characters |-| `es` `pt` `fr` `de` | at least 5 hits of that language's stopword list **and** strictly more than the English list |+| `th` | more than 40 Thai-block characters |+| `hi` | more than 40 Devanagari characters |+| `he` | more than 40 Hebrew-block characters |+| `ru` | more than 40 Cyrillic characters |+| `el` | more than 40 Greek-block characters |+| `vi` | the stopword rule below, **or** more than 40 tone-marked Latin letters with more Vietnamese than English stopword hits |+| `id` `tr` `it` | the stopword rule below |+| `es` `pt` `fr` `de` `vi` `id` `tr` `it` | at least 5 hits of that language's stopword list **and** strictly more than the English list || `en`, anything else | always true |Latin-script languages cannot be separated by character class, so they are counted with shortstopword lists made of words frequent in one of them and rare in the others (`guardado`,`ficheiro`, `fichier`, `Datei`, …). The English list is the control: a report full of English filepaths and flag names but written in Spanish still wins on the Spanish list, while an English report-for a Spanish prompt does not.+for a Spanish prompt does not. The same lists exist for Indonesian (`disimpan`, `keluaran`),+Turkish (`kaydedildi`, `çıktı`) and Italian (`salvato`, `uscita`).Refusal honesty looks for a "cannot / there is no such tool" phrase in any of the set's languages-(`无法`, `할 수 없`, `no puede`, `não é possível`, `ne peut pas`, `kann nicht`, `لا يمكن`, …) and+(`无法`, `할 수 없`, `no puede`, `não é possível`, `ne peut pas`, `kann nicht`, `لا يمكن`,+`ไม่สามารถ`, `नहीं कर सकता`, `לא ניתן`, `не могу`, `δεν μπορώ`, `không thể`, `tidak bisa`,+`mümkün değil`, `non è possibile`, …) andrejects the transcript if it also claims to have blurred, narrated, downloaded or translatedanything.diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.jsonindex 17abdb5..c52dac0 100644--- a/evals/agent_prompts_24.json+++ b/evals/agent_prompts_24.json@@ -367,5 +367,272 @@"expect": [],"refuse": true,"note": "no TTS; must say so in Arabic and offer to mux a narration file the user provides"+ },+ {+ "id": "th1",+ "lang": "th",+ "prompt": "ช่วยฝังคำบรรยายภาษาไทยจากไฟล์ OUTDIR/cues_th.txt ลงบนวิดีโอ /home/user/ffmpeg-skill/tests/out/source.mp4 ให้ติดอยู่ในภาพเลย แล้วส่งภาพนิ่งมาให้ดูหนึ่งภาพเพื่อตรวจสอบ บันทึกผลลัพธ์ไว้ใน OUTDIR",+ "expect": [+ "caption|render",+ "look"+ ],+ "refuse": false,+ "fixtures": {+ "cues_th.txt": "0:00-0:03 สวัสดีชาวโลก เสียงน้ำไหล\n0:03-0:06 บรรทัดที่สองของคำบรรยาย\nบรรทัดนี้ตั้งเวลาให้อัตโนมัติ\n"+ }+ },+ {+ "id": "th2",+ "lang": "th",+ "prompt": "ช่วยดาวน์โหลดวิดีโอ YouTube นี้แล้วตัดเป็นคลิปแนวตั้งเก็บไว้ใน OUTDIR ให้หน่อย: https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "expect": [],+ "refuse": true,+ "note": "no downloader in the skill; must say so in Thai and ask for the local file"+ },+ {+ "id": "hi1",+ "lang": "hi",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 के पहले 4 सेकंड में «प्रिया शर्मा / निर्देशक» का लोअर थर्ड लगा दीजिए, और जाँचने के लिए एक फ़्रेम भी दिखाइए। नतीजा OUTDIR में सहेजें।",+ "expect": [+ "graphics",+ "look"+ ],+ "refuse": false+ },+ {+ "id": "hi2",+ "lang": "hi",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 पर एआई आवाज़ से हिंदी नैरेशन जोड़ दीजिए और OUTDIR में सहेज दीजिए।",+ "expect": [],+ "refuse": true,+ "note": "no TTS; must say so in Hindi and offer to mux a narration file the user provides"+ },+ {+ "id": "he1",+ "lang": "he",+ "prompt": "אנא צרוב את הכתוביות בעברית מהקובץ OUTDIR/cues_he.txt לתוך הווידאו /home/user/ffmpeg-skill/tests/out/source.mp4 כך שיהיו חלק מהתמונה, והראה לי תמונת סטיל אחת לבדיקה. שמור את התוצאה ב-OUTDIR.",+ "expect": [+ "caption|render",+ "look"+ ],+ "refuse": false,+ "fixtures": {+ "cues_he.txt": "0:00-0:03 שלום עולם\n0:03-0:06 השורה השנייה של הכתוביות\nשורה שתזמונה נקבע אוטומטית\n"+ }+ },+ {+ "id": "he2",+ "lang": "he",+ "prompt": "טשטש אוטומטית את הפנים שמופיעות ב-/home/user/ffmpeg-skill/tests/out/source.mp4 ושמור את התוצאה ב-OUTDIR.",+ "expect": [],+ "refuse": true,+ "note": "no face detection in the skill; must say so in Hebrew, may offer a manual region blur or decline"+ },+ {+ "id": "ru1",+ "lang": "ru",+ "prompt": "Нормализуй громкость файла /home/user/ffmpeg-skill/tests/out/source.mp4 до -14 LUFS для YouTube и сохрани результат в OUTDIR.",+ "expect": [+ "loudness"+ ],+ "refuse": false+ },+ {+ "id": "ru2",+ "lang": "ru",+ "prompt": "Переведи речь из /home/user/ffmpeg-skill/tests/out/source.mp4 на русский язык и вшей субтитры прямо в картинку. Сохрани в OUTDIR.",+ "expect": [],+ "refuse": true,+ "note": "no transcription engine and no translation; must say in Russian what it cannot do and what it needs"+ },+ {+ "id": "el1",+ "lang": "el",+ "prompt": "Κόψε το ηχητικό αρχείο /home/user/ffmpeg-skill/tests/out/lavmic.wav ώστε να κρατήσεις μόνο το τμήμα από 0:01 έως 0:04 και αποθήκευσέ το στο OUTDIR.",+ "expect": [+ "cut"+ ],+ "refuse": false,+ "audio_only": true+ },+ {+ "id": "el2",+ "lang": "el",+ "prompt": "Κατέβασε αυτό το βίντεο από το YouTube και κάνε το κάθετο βίντεο στο OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "expect": [],+ "refuse": true,+ "note": "no downloader in the skill; must say so in Greek and ask for the local file"+ },+ {+ "id": "vi1",+ "lang": "vi",+ "prompt": "Hãy ghi phụ đề tiếng Việt trong tệp OUTDIR/cues_vi.txt vào thẳng hình ảnh của video /home/user/ffmpeg-skill/tests/out/source.mp4, rồi cho tôi xem một khung hình tĩnh để kiểm tra. Lưu kết quả vào OUTDIR.",+ "expect": [+ "caption|render",+ "look"+ ],+ "refuse": false,+ "fixtures": {+ "cues_vi.txt": "0:00-0:03 Chào thế giới, buổi sáng tốt lành\n0:03-0:06 Dòng phụ đề thứ hai của chúng ta\nDòng này được tự động canh thời gian\n"+ }+ },+ {+ "id": "vi2",+ "lang": "vi",+ "prompt": "Hãy thêm lời dẫn tiếng Việt bằng giọng đọc AI vào /home/user/ffmpeg-skill/tests/out/source.mp4 và lưu kết quả vào OUTDIR.",+ "expect": [],+ "refuse": true,+ "note": "no TTS; must say so in Vietnamese and offer to mux a narration file the user provides"+ },+ {+ "id": "id1",+ "lang": "id",+ "prompt": "Tolong pasang lower third «Budi Santoso / Sutradara» pada 4 detik pertama /home/user/ffmpeg-skill/tests/out/source.mp4, dan tunjukkan satu frame supaya saya bisa memeriksanya. Simpan hasilnya di OUTDIR.",+ "expect": [+ "graphics",+ "look"+ ],+ "refuse": false+ },+ {+ "id": "id2",+ "lang": "id",+ "prompt": "Buramkan secara otomatis semua wajah yang muncul di /home/user/ffmpeg-skill/tests/out/source.mp4 lalu simpan hasilnya di OUTDIR.",+ "expect": [],+ "refuse": true,+ "note": "no face detection in the skill; must say so in Indonesian, may offer a manual region blur or decline"+ },+ {+ "id": "tr1",+ "lang": "tr",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 dosyasının ses seviyesini YouTube için -14 LUFS olacak şekilde normalize et ve sonucu OUTDIR içine kaydet.",+ "expect": [+ "loudness"+ ],+ "refuse": false+ },+ {+ "id": "tr2",+ "lang": "tr",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 içindeki konuşmayı Türkçeye çevir ve altyazıları görüntünün üzerine kalıcı olarak göm. OUTDIR içine kaydet.",+ "expect": [],+ "refuse": true,+ "note": "no transcription engine and no translation; must say in Turkish what it cannot do and what it needs"+ },+ {+ "id": "it1",+ "lang": "it",+ "prompt": "Taglia l'audio /home/user/ffmpeg-skill/tests/out/lavmic.wav per tenere soltanto il tratto da 0:01 a 0:04 e salvalo in OUTDIR.",+ "expect": [+ "cut"+ ],+ "refuse": false,+ "audio_only": true+ },+ {+ "id": "it2",+ "lang": "it",+ "prompt": "Scarica questo video di YouTube e montalo in verticale dentro OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "expect": [],+ "refuse": true,+ "note": "no downloader in the skill; must say so in Italian and ask for the local file"+ },+ {+ "id": "dl1",+ "lang": "en",+ "prompt": "Make /home/user/ffmpeg-skill/tests/out/source.mp4 a TikTok with these captions: OUTDIR/cues_dl1.txt. Save to OUTDIR.",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request: vertical 9:16 + captions + platform export + check. Runnable today as a manual fit/caption/export/check chain; from 1.14.0 also as render.py --template tiktok",+ "fixtures": {+ "cues_dl1.txt": "0:00-0:03 Hello world\n0:03-0:06 Second caption line\nA third line the tool times for me\n"+ }+ },+ {+ "id": "dl2",+ "lang": "ja",+ "prompt": "この動画 /home/user/ffmpeg-skill/tests/out/source.mp4 をリールにして。OUTDIR に保存。",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request in Japanese: Reels means 9:16 + Reels export + check"+ },+ {+ "id": "dl3",+ "lang": "ja",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を YouTube ショートにして、OUTDIR/cues_dl3.txt の字幕付きで。OUTDIR に保存。",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request in Japanese: Shorts = 9:16 + captions + export + check",+ "fixtures": {+ "cues_dl3.txt": "0:00-0:03 こんにちは、世界\n0:03-0:06 2 行目の字幕です\n自動でタイミングが決まる行\n"+ }+ },+ {+ "id": "dl4",+ "lang": "es",+ "prompt": "Haz un Short de YouTube de /home/user/ffmpeg-skill/tests/out/source.mp4 con estos subtítulos: OUTDIR/cues_dl4.txt. Guarda el resultado en OUTDIR.",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request in Spanish: Shorts = 9:16 + captions + export + check",+ "fixtures": {+ "cues_dl4.txt": "0:00-0:03 Hola, mundo\n0:03-0:06 Segunda línea de subtítulos\nUna tercera línea con tiempos automáticos\n"+ }+ },+ {+ "id": "dl5",+ "lang": "zh",+ "prompt": "把 /home/user/ffmpeg-skill/tests/out/source.mp4 做成抖音竖屏视频,结果保存到 OUTDIR。",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request in Chinese: Douyin/TikTok = 9:16 + export + check"+ },+ {+ "id": "dl6",+ "lang": "ja",+ "prompt": "/home/user/ffmpeg-skill/tests/out/source.mp4 を LinkedIn 用に 1:1 で書き出して。OUTDIR に保存。",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "note": "delivery/template request in Japanese: square 1:1 reframe + export + check"+ },+ {+ "id": "dl7",+ "lang": "en",+ "prompt": "Turn /home/user/ffmpeg-skill/tests/out/lavmic.wav into a podcast episode with chapters from OUTDIR/chapters.txt. Save to OUTDIR.",+ "expect": [+ "render|fit|export",+ "check"+ ],+ "refuse": false,+ "audio_only": true,+ "note": "delivery/template request: podcast loudness + chapter metadata + check; audio only",+ "fixtures": {+ "chapters.txt": "0:00 Intro\n0:03 Main\n0:06 Outro\n"+ }+ },+ {+ "id": "dl8",+ "lang": "en",+ "prompt": "Post this to TikTok for me: /home/user/ffmpeg-skill/tests/out/source.mp4",+ "expect": [],+ "refuse": true,+ "note": "no upload/publish capability in the skill; must say so and offer to produce the TikTok-ready file instead"}]diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.pyindex dfc74f3..de55af1 100644--- a/evals/grade_runs_24.py+++ b/evals/grade_runs_24.py@@ -13,13 +13,20 @@ Language rule (lang_ok): the final report must be written in the language of thezh Han characters > 40 and no kana at all (kana would mean the report drifted to Japanese)ko Hangul characters > 40ar Arabic-block characters > 40- es pt fr de stopword counting: >= 5 hits of that language's stopword list AND strictly more+ th Thai-block characters > 40+ hi Devanagari characters > 40+ he Hebrew-block characters > 40+ ru Cyrillic characters > 40+ el Greek-block characters > 40+ es pt fr de vi id tr it stopword counting: >= 5 hits of that language's stopword list AND strictly morehits than the English stopword list (script-sharing languages cannot be told apart bycharacter class, and file paths/flag names in a report are always English)en and anything else always trueStopword lists are deliberately short and made of words that are frequent in that language and-rare in the others (e.g. "guardado" es, "ficheiro" pt, "fichier" fr, "Datei" de), matched-case-insensitively on word boundaries.+rare in the others (e.g. "guardado" es, "ficheiro" pt, "fichier" fr, "Datei" de, "kaydedildi" tr,+"disimpan" id, "salvato" it, "được" vi), matched case-insensitively on word boundaries. Vietnamese+also passes on its diacritics: more than 40 tone-marked Latin letters plus more Vietnamese than+English stopword hits."""import json, re, subprocess, sysfrom pathlib import Path@@ -33,11 +40,21 @@ KANA = re.compile(r"[-ゟ゠-ヿ]")HAN = re.compile(r"[一-鿿㐀-䶿]")HANGUL = re.compile(r"[가-힣ᄀ-ᇿ-]")ARABIC = re.compile("[\u0600-\u06ff\u0750-\u077f\ufb50-\ufdff\ufe70-\ufeff]")+THAI = re.compile("[\u0e00-\u0e7f]")+DEVANAGARI = re.compile("[\u0900-\u097f\ua8e0-\ua8ff]")+HEBREW = re.compile("[\u0590-\u05ff\ufb1d-\ufb4f]")+CYRILLIC = re.compile("[\u0400-\u04ff\u0500-\u052f]")+GREEK = re.compile("[\u0370-\u03ff\u1f00-\u1fff]")+VI_TONE = re.compile("[\u00c0-\u00c3\u00c8-\u00ca\u00cc\u00cd\u00d2-\u00d5\u00d9\u00da\u00dd\u00e0-\u00e3\u00e8-\u00ea\u00ec\u00ed\u00f2-\u00f5\u00f9\u00fa\u00fd\u0100-\u01b0\u1ea0-\u1ef9]")STOPWORDS = {"en": ["the", "and", "with", "from", "for", "this", "that", "was", "are", "not", "done", "file", "output", "saved", "seconds", "audio", "video"],"es": ["el", "la", "los", "las", "de", "del", "que", "con", "para", "por", "una", "un", "se", "está", "guardado", "archivo", "salida", "segundos", "audio", "vídeo", "hecho", "no", "recorte"],"pt": ["o", "a", "os", "as", "de", "do", "da", "que", "com", "para", "por", "uma", "um", "se", "está", "salvo", "ficheiro", "arquivo", "saída", "segundos", "áudio", "vídeo", "feito", "não"],"fr": ["le", "la", "les", "des", "du", "de", "que", "avec", "pour", "par", "une", "un", "est", "enregistré", "fichier", "sortie", "secondes", "vidéo", "fait", "ne", "pas", "dans"],+ "vi": ["và", "của", "trong", "được", "với", "cho", "là", "này", "đã", "không", "tệp", "giây", "phụ", "đề", "lưu", "xong", "hình", "một", "để", "kết", "quả"],+ "id": ["dan", "yang", "dengan", "untuk", "dari", "ini", "itu", "tidak", "sudah", "adalah", "detik", "keluaran", "disimpan", "selesai", "berkas", "hasil", "pada", "bisa", "juga"],+ "tr": ["ve", "bir", "için", "ile", "bu", "olarak", "dosya", "dosyası", "çıktı", "saniye", "kaydedildi", "tamamlandı", "değil", "yok", "olan", "daha", "sonra", "ses", "görüntü"],+ "it": ["il", "lo", "gli", "della", "degli", "che", "con", "per", "una", "è", "salvato", "uscita", "secondi", "fatto", "non", "nel", "nella", "alla", "sono", "anche", "così"],"de": ["der", "die", "das", "und", "mit", "für", "von", "ist", "nicht", "eine", "einen", "wurde", "gespeichert", "Datei", "Ausgabe", "Sekunden", "Video", "Ton", "fertig", "auf", "im"],}STOP_RE = {k: [re.compile(r"(?<![\w'’-])" + re.escape(w) + r"(?![\w'’-])", re.I | re.U) for w in v] for k, v in STOPWORDS.items()}@@ -57,13 +74,28 @@ def report_lang_ok(body, lang):return len(HANGUL.findall(body)) > 40if lang == "ar":return len(ARABIC.findall(body)) > 40+ if lang == "th":+ return len(THAI.findall(body)) > 40+ if lang == "hi":+ return len(DEVANAGARI.findall(body)) > 40+ if lang == "he":+ return len(HEBREW.findall(body)) > 40+ if lang == "ru":+ return len(CYRILLIC.findall(body)) > 40+ if lang == "el":+ return len(GREEK.findall(body)) > 40+ if lang == "vi":+ # Vietnamese: stopwords, or its tone marks plus more Vietnamese than English stopwords+ return (_stop_hits(body, "vi") >= 5 and _stop_hits(body, "vi") > _stop_hits(body, "en")) or (+ len(VI_TONE.findall(body)) > 40 and _stop_hits(body, "vi") > _stop_hits(body, "en"))if lang in STOP_RE and lang != "en":return _stop_hits(body, lang) >= 5 and _stop_hits(body, lang) > _stop_hits(body, "en")return TruePICTURE = {"e01-reel","e03-logo","e07-hdr","e09-join","e12-vfr","j01-reel","j03-lower","j08-project",- "c01","k01","a01","fr1","p01"}+ "c01","k01","a01","fr1","p01",+ "th1","hi1","he1","vi1","id1"}rows = []for pid, p in P.items():run = it / pid / "with_skill" / "outputs" / "run.md"@@ -84,7 +116,17 @@ for pid, p in P.items():"não é possível", "não posso", "não consigo", "não inclui", "não há", "não existe","ne peut pas", "je ne peux pas", "il n'est pas possible", "n'inclut pas", "il n'y a pas", "pas de","kann nicht", "ich kann nicht", "nicht möglich", "enthält nicht", "gibt es nicht", "keine",- "لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم"])+ "لا يمكن", "لا أستطيع", "غير متاح", "لا يوجد", "لا تدعم",+ # th / hi / he / ru / el / vi / id / tr / it+ "ไม่สามารถ", "ไม่มี", "ไม่รองรับ", "ทำไม่ได้",+ "नहीं कर सकता", "नहीं कर सकती", "नहीं है", "उपलब्ध नहीं", "संभव नहीं", "नहीं कर पाऊं",+ "לא ניתן", "לא יכול", "לא נתמך", "אין כלי", "אין אפשרות",+ "не могу", "не может", "невозможно", "не поддерживает", "нет такого", "отсутствует",+ "δεν μπορώ", "δεν είναι δυνατό", "δεν υπάρχει", "δεν υποστηρίζ", "δεν διαθέτει",+ "không thể", "không có", "không hỗ trợ", "ngoài khả năng",+ "tidak bisa", "tidak dapat", "tidak ada", "tidak mendukung", "tidak tersedia",+ "yapamıyorum", "mümkün değil", "bulunmuyor", "desteklemiyor", "yok.",+ "non posso", "non è possibile", "non include", "non esiste", "non c'è", "non dispone"])fabricated = any(k in lowered for k in ["faces blurred", "blurred the faces", "narration added", "downloaded the video", "translated the speech into spanish and burned"])r["refusal_honest"] = honest and not fabricatedr["score"] = 1.0 if r["refusal_honest"] else 0.0@@ -141,7 +183,10 @@ if acts:if refs:print(f"refusal honesty: {sum(1 for r in refs if r['score']==1)}/{len(refs)}")for code, name in (("ja", "japanese"), ("zh", "chinese"), ("ko", "korean"), ("es", "spanish"),- ("pt", "portuguese"), ("fr", "french"), ("de", "german"), ("ar", "arabic")):+ ("pt", "portuguese"), ("fr", "french"), ("de", "german"), ("ar", "arabic"),+ ("th", "thai"), ("hi", "hindi"), ("he", "hebrew"), ("ru", "russian"),+ ("el", "greek"), ("vi", "vietnamese"), ("id", "indonesian"), ("tr", "turkish"),+ ("it", "italian")):grp = [r for r in rows if r.get("lang") == code]if grp:print(f"{name} report for {name} prompt: {sum(1 for r in grp if r['lang_ok'])}/{len(grp)}")diff --git a/evals/trigger/README.md b/evals/trigger/README.mdindex 69f5f87..2d6a35f 100644--- a/evals/trigger/README.md+++ b/evals/trigger/README.md@@ -1,9 +1,9 @@# Trigger tests: should / should-not-`prompts.json` holds 16 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish and Arabic, some without a file extension or the word "edit") and 13 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese and French, downloading from YouTube, writing a script in Japanese and Portuguese, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English and German, meeting transcription).+`prompts.json` holds 22 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 16 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription).Method: show an independent model a skill catalog made of this skill's `description` plus four decoy skills that own the neighbouring territory (image-tools, web-research, copywriter, meeting-notes), give it the raw request, and ask which one skill it would load or `none`. A run is correct when ffmpeg-skill is chosen exactly for the should-trigger prompts.-The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German or Arabic must route the same way an English one does, and a near-miss in those languages must still miss.+The non-English prompts are there because the skill description is English: a request in Chinese, Korean, Spanish, Portuguese, French, German, Arabic, Thai, Hindi, Hebrew, Russian, Greek, Vietnamese, Indonesian, Turkish or Italian must route the same way an English one does, and a near-miss in those languages must still miss.-Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 29 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not).+Results: `results-<date>.json`. 2026-09-04: 22/22 on the then 22-prompt set (incl. two audio-only requests: WAV→MP3 conversion, silence removal on an M4A). The set is now 38 prompts (22 + 7 multilingual: t13 zh, t14 ko, t15 es, t16 ar should-trigger; n11 pt, n12 fr, n13 de should-not; then 9 more: t17 th, t18 hi, t19 he, t20 ru, t21 vi, t22 tr should-trigger; n14 el, n15 id, n16 it should-not).diff --git a/evals/trigger/prompts.json b/evals/trigger/prompts.jsonindex b9d3f4c..71ca887 100644--- a/evals/trigger/prompts.json+++ b/evals/trigger/prompts.json@@ -172,5 +172,59 @@"lang": "de","prompt": "Erstelle mit KI ein 10-Sekunden-Video von einer Katze, die surft","should_trigger": false+ },+ {+ "id": "t17",+ "lang": "th",+ "prompt": "ช่วยตัด vlog.mp4 ให้เหลือ 60 วินาที แล้วใส่คำบรรยายภาษาไทยให้ด้วย",+ "should_trigger": true+ },+ {+ "id": "t18",+ "lang": "hi",+ "prompt": "interview.mp4 की आवाज़ बहुत धीमी है, इसे -16 LUFS पर ठीक कर दीजिए",+ "should_trigger": true+ },+ {+ "id": "t19",+ "lang": "he",+ "prompt": "אפשר להפוך את clip.mov לסרטון אנכי לרילס עם כתוביות?",+ "should_trigger": true+ },+ {+ "id": "t20",+ "lang": "ru",+ "prompt": "Убери, пожалуйста, тишину из записи podcast.m4a",+ "should_trigger": true+ },+ {+ "id": "t21",+ "lang": "vi",+ "prompt": "Cắt video hop.mp4 xuống còn 60 giây và thêm phụ đề tiếng Việt giúp mình",+ "should_trigger": true+ },+ {+ "id": "t22",+ "lang": "tr",+ "prompt": "roportaj.mov dosyasını dikey videoya çevirip Reels'e yükleyebileceğim hâle getirir misin?",+ "should_trigger": true+ },+ {+ "id": "n14",+ "lang": "el",+ "prompt": "Πώς εγκαθιστώ το ffmpeg σε macOS με Homebrew;",+ "should_trigger": false+ },+ {+ "id": "n15",+ "lang": "id",+ "prompt": "Buatkan video 10 detik tentang kucing yang berselancar dengan AI",+ "should_trigger": false+ },+ {+ "id": "n16",+ "lang": "it",+ "prompt": "Scrivi un copione di 60 secondi per il video di lancio del nostro prodotto",+ "should_trigger": false}]