Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval iteration 10 at 1.10.0, README catches up with 1.8-1.10, PR checklist names the README (#210)
claude[bot]f693ad0ac76b
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 4f9dba9..c5b332d 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -68,6 +68,10 @@ would have caught the original bug isn't done yet.all three must pass.- Explain *why*, not just *what*, in the PR description — the reasoning is whatfuture maintainers (human or agent) need most.+- A `feat` PR updates every place the feature is stated: `README.md` (the tool+ table, the contract table, gotchas), `SKILL.md`, `references/scripts.md`,+ `docs/contract.md` and `CHANGELOG.md`. The README is read by people who never+ open SKILL.md, so a feature that only SKILL.md knows about is half shipped.## Reporting issuesdiff --git a/README.md b/README.mdindex fce5f6c..1aa6b81 100644--- a/README.md+++ b/README.md@@ -153,13 +153,13 @@ These are the rules the skill file gives the agent and the code enforces. Togeth## Tools-42 public tools, all Python 3.9 standard library, all with `--help`, `--dry-run`, `--json`, non-zero exit and a reason on stderr on failure.+42 public tools, all Python 3.9 standard library, all with `--help`, `--dry-run`, `--json`, `--plan FILE` (a dry run written as a plan `render.py` executes later), non-zero exit and a reason on stderr on failure. Every re-encoding tool takes `--codec h264|hevc|av1|prores` and `--quality N` (1.8), and every time flag takes seconds, `mm:ss`, `hh:mm:ss.fff` or SMPTE `hh:mm:ss:ff` with an optional `@fps` suffix (1.9).**Analysis and inspection**| Tool | What it does ||---|---|-| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision, colour space, rotation, every audio stream; `--analyze` flags Log footage |+| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision (`hdr` for BT.2020 or PQ/HLG, `hdr_signal` for a real PQ/HLG/DV transfer only), colour space, rotation, every audio stream; `--analyze` flags Log footage || `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments` || `look.py` | Contact sheet, single frames, side-by-side comparison as PNG so the agent can see what it made |@@ -213,7 +213,7 @@ These are the rules the skill file gives the agent and the code enforces. Togeth| Tool | What it does ||---|---|-| `export.py` | Presets `youtube`, `youtube4k`, `reels`, `x`, `prores`, `h265`, `gif`, all tagged BT.709 |+| `export.py` | Presets `youtube`, `youtube4k`, `reels`, `x`, `prores`, `h265`, `gif`, all tagged BT.709; `--normalize` meets the platform's loudness in the same call (`render.py` turns it on by default for platform presets) || `proxy.py` | Small, low-bitrate proxy for downstream AI analysis/preview/editing decisions — resize by `--width`/`--scale`, proxy-grade `--crf`, `--fps`, `--no-audio`; not a delivery preset || `check.py` | PASS / WARN / FAIL against YouTube, Shorts, Reels, TikTok, X, LinkedIn, broadcast and podcast specs, with the fix for each failure and a `format` / `judgement` kind per row || `report.py` | Single-file HTML delivery report: before/after sheets, media facts, loudness, compliance, the commands run |@@ -293,6 +293,8 @@ The contract is generated from the code that runs, not maintained beside it. For| `mutates_input` | always `false` || `idempotency_hint` | `bit_exact`, `content_equivalent`, `cached` or `environment_dependent` |+Next to the tool list the document carries a top-level `deprecated` list (1.10): what 2.0.0 removes, since when, the replacement and the surface it lives on. `docs/contract.md` "What 2.0 changes" is written from it.+`contract_version` (1.0) is separate from the skill version, so a consumer can pin the shape and read the version for provenance. The document also states the invocation mapping (structured arguments → argv), the JSON shapes for success and failure (`{"status": "failed", "error": {"kind": "input | ffmpeg | output | missing_tool | timeout | verification | interrupted", "message": …}}`), and that no tool runs a shell or executes anything other than the named script, `ffmpeg` and `ffprobe`. Field-by-field reference: [docs/contract.md](docs/contract.md).### MCP@@ -331,6 +333,7 @@ The short list for humans. The agent-facing version, with the reasoning, is the- **Non-Latin captions need a font with the glyphs.** Without one you get boxes, not an error. Name it (`caption.py --font "Noto Sans CJK JP"`) or point at the file (`overlay.py --font-file /path/to/NotoSansCJK-Regular.ttc`).- **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.- **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.+- **Outputs are never overwritten silently.** An existing output path is warned about today and refused from 2.0; set `FFMPEG_SKILL_NO_OVERWRITE=1` (the recommended agent setting) to get the refusal now and pass `--overwrite` where a replacement is intended.- **Long chains belong in a plan.** Three hand-chained re-encodes lose quality and are hard to change; `render.py` runs the whole edit from one JSON file, and `--dry-run` shows every ffmpeg command before anything is written.## FFmpeg compatibility@@ -357,6 +360,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **108 / 108** | 1.10.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + focused Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, report format 108/108 by both graders (the harness now names the five labels), user's language 105/108 (every Japanese request in Japanese; 3 English requests drifted to Spanish or Portuguese), visual check 21/24, trigger set 22/22; real-device corpus 101/101 steps PASS. Details in `evals/results/iteration-10.json` || **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` || **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` || **108 / 108** | 1.7.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 24 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 25/25, report format 108/108 by regex (104/108 by the stricter grader), user's language 101/108 (six English refusals answered in Spanish or Portuguese, one Japanese request in English); trigger set 22/22. Iteration-6 fixes held (fade-in only, Japanese audio trims, music no longer shortens the video). Details in `evals/results/iteration-7.json` |@@ -420,7 +424,7 @@ FFmpeg itself:## Stability-1.x keeps every tool name, CLI argument, JSON output key and exit code working: nothing is removed or renamed, and nothing optional becomes required, until 2.0. The full list of what is promised and what is not, and the three-step deprecation policy, is in [docs/contract.md](docs/contract.md#stability-guarantee-1x). It is enforced by a test that pins every tool's argument names against a snapshot, so a breaking change fails CI instead of slipping into a patch.+1.x keeps every tool name, CLI argument, JSON output key and exit code working: nothing is removed or renamed, and nothing optional becomes required, until 2.0. The full list of what is promised and what is not, and the three-step deprecation policy, is in [docs/contract.md](docs/contract.md#stability-guarantee-1x). It is enforced by a test that pins every tool's argument names against a snapshot, so a breaking change fails CI instead of slipping into a patch. What 2.0 will remove is already announced: `contract --json` lists it under `deprecated`, `--crf` prints a one-line warning where `--quality` exists, and [docs/contract.md](docs/contract.md#what-20-changes) says what a caller does today to be ready.## Developmentdiff --git a/evals/results/corpus-2026-09-13-1.10.0.md b/evals/results/corpus-2026-09-13-1.10.0.mdnew file mode 100644index 0000000..f2592b3--- /dev/null+++ b/evals/results/corpus-2026-09-13-1.10.0.md@@ -0,0 +1,158 @@+# ffmpeg-skill verification++9 files, 101 steps, 0 failed++## /home/user/ffmpeg-skill/tests/corpus/gopro_GX010743.MP4+20.2202s, hevc 3840x2160 @ 29.97 yuv420p10le, audio aac 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.6s | |+| cut accurate | PASS | 29.4s | |+| fit 9:16 | PASS | 10.0s | |+| caption | PASS | 35.5s | |+| overlay text | PASS | 35.4s | |+| look sheet | PASS | 7.6s | |+| probe analyze | PASS | 8.9s | |+| export x | PASS | 15.4s | |+| loudness measure | PASS | 1.1s | |+| silence list | PASS | 0.4s | |++## /home/user/ffmpeg-skill/tests/corpus/dji_DJI_0038.MOV+10.176833s, hevc 3840x2160 @ 59.94 yuv420p, audio None Nonech++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.7s | |+| cut accurate | PASS | 39.5s | |+| fit 9:16 | PASS | 22.3s | |+| caption | PASS | 41.7s | |+| overlay text | PASS | 39.9s | |+| look sheet | PASS | 18.8s | |+| probe analyze | PASS | 19.0s | |+| export x | PASS | 23.2s | |++## /home/user/ffmpeg-skill/tests/corpus/iphone_IMG_3179.MOV+7.616667s, hevc 3840x2160 @ 60.0 yuv420p, audio aac 1ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.6s | |+| cut accurate | PASS | 48.7s | |+| fit 9:16 | PASS | 14.8s | |+| caption | PASS | 44.8s | |+| overlay text | PASS | 44.1s | |+| look sheet | PASS | 9.5s | |+| probe analyze | PASS | 10.5s | |+| export x | PASS | 19.7s | |+| loudness measure | PASS | 0.4s | |+| silence list | PASS | 0.4s | |++## /home/user/ffmpeg-skill/tests/corpus/android_screen_1.mp4+27.431344s, h264 720x1600 @ 18.082 yuv420p [VFR?], audio aac 1ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 2.7s | |+| cut accurate | PASS | 1.2s | |+| fit 9:16 | PASS | 1.6s | |+| caption | PASS | 1.5s | |+| overlay text | PASS | 1.1s | |+| look sheet | PASS | 0.6s | |+| probe analyze | PASS | 0.5s | |+| export x | PASS | 2.6s | |+| loudness measure | PASS | 1.1s | |+| silence list | PASS | 0.2s | |++## /home/user/ffmpeg-skill/tests/corpus/android_screen_2.mp4+6.0s, h264 1298x1080 @ 120.0 yuv420p, audio aac 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.5s | |+| cut accurate | PASS | 5.9s | |+| fit 9:16 | PASS | 5.8s | |+| caption | PASS | 6.5s | |+| overlay text | PASS | 5.8s | |+| look sheet | PASS | 1.2s | |+| probe analyze | PASS | 1.1s | |+| export x | PASS | 7.1s | |+| loudness measure | PASS | 0.5s | |+| silence list | PASS | 0.4s | |++## /home/user/ffmpeg-skill/tests/corpus/hdr10_white_clipping.mp4+600.6s, hevc 3840x2160 @ 23.976 yuv420p10le [HDR10/PQ], audio aac 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.4s | |+| cut accurate | PASS | 28.2s | |+| fit 9:16 | PASS | 4.9s | |+| caption | PASS | 23.9s | |+| overlay text | PASS | 24.2s | |+| look sheet | PASS | 2.8s | |+| color to-sdr | PASS | 25.5s | |+| hdr preserved | PASS | 0.1s | |+| probe analyze | PASS | 2.8s | |+| export x | PASS | 5.5s | |+| loudness measure | PASS | 34.2s | |+| silence list | PASS | 0.2s | |++## /home/user/ffmpeg-skill/tests/corpus/iphone13pro_4K60p.mov+21.658333s, hevc 3840x2160 @ 59.977 yuv420p10le [Dolby Vision profile 8] [VFR?], audio aac 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 307.0s | |+| cut accurate | PASS | 196.6s | |+| fit 9:16 | PASS | 25.8s | |+| caption | PASS | 168.2s | |+| overlay text | PASS | 164.4s | |+| look sheet | PASS | 11.4s | |+| color to-sdr | PASS | 73.7s | |+| hdr preserved | PASS | 0.1s | |+| probe analyze | PASS | 12.8s | |+| export x | PASS | 19.2s | |+| loudness measure | PASS | 1.2s | |+| silence list | PASS | 0.3s | |++## /home/user/ffmpeg-skill/tests/corpus/tears_of_steel_720p.mov+734.166667s, h264 1280x534 @ 24.0 yuv420p, audio mp3 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.4s | |+| cut accurate | PASS | 1.4s | |+| fit 9:16 | PASS | 1.6s | |+| caption | PASS | 1.7s | |+| overlay text | PASS | 1.0s | |+| look sheet | PASS | 0.5s | |+| probe analyze | PASS | 0.4s | |+| export x | PASS | 2.1s | |+| loudness measure | PASS | 33.5s | |+| silence list | PASS | 0.3s | |++## /home/user/ffmpeg-skill/tests/corpus/rtings_305_24p.mp4+60.0s, h264 1920x1080 @ 24.0 yuv420p, audio aac 2ch++| step | result | time | error |+|---|---|---|---|+| probe | PASS | 0s | |+| cut copy | PASS | 0.5s | |+| cut accurate | PASS | 2.6s | |+| fit 9:16 | PASS | 1.6s | |+| caption | PASS | 2.5s | |+| overlay text | PASS | 1.6s | |+| look sheet | PASS | 0.6s | |+| probe analyze | PASS | 0.5s | |+| export x | PASS | 3.2s | |+| loudness measure | PASS | 3.6s | |+| silence list | PASS | 0.3s | |diff --git a/evals/results/iteration-10-independent-grades.json b/evals/results/iteration-10-independent-grades.jsonnew file mode 100644index 0000000..41ff4d0--- /dev/null+++ b/evals/results/iteration-10-independent-grades.json@@ -0,0 +1,612 @@+{+ "summary": {+ "runs": 108,+ "runs_graded_fully": 49,+ "routing": "49/49 (graded set)",+ "honest": "49/49 (graded set)",+ "language": "105/108 (light check over all runs)",+ "format": "108/108 (light check over all runs: all five labels present in every final report)",+ "look_when_picture_changed": "7/8 (graded set)",+ "quality_mean": 4.65,+ "quality_min": 3+ },+ "iteration_10_specific": {+ "five_labels_named_in_harness_prompt": {+ "runs_with_all_five_labels": 108,+ "note": "every one of the 108 final reports carries Done:/Failed:, Steps:, Check:, Look: and Notes:. Iteration 9 scored 65/108 on format; naming the labels in the harness prompt closed the gap completely."+ },+ "failed_label_on_a_successful_informational_request": {+ "count": 1,+ "runs": [+ "rep1/j07-probe"+ ],+ "checked": [+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe",+ "rep1/e10-highlights",+ "rep2/e10-highlights",+ "rep3/e10-highlights"+ ],+ "note": "rep1/j07-probe wrote 'Failed: 編集の実行は依頼されていないため何も書き出していません' for a comparison that fully succeeded. All three e10-highlights runs and the other two j07 runs correctly used Done: for an analysis-only job."+ },+ "language_drift": {+ "en_request_answered_in_spanish_or_portuguese": [+ "rep1/x05-replace-audio (es)",+ "rep3/e02-cut-lossless (pt)",+ "rep3/r04-translate (es)"+ ],+ "ja_requests": 24,+ "ja_requests_answered_in_japanese": 24,+ "note": "all 24 Japanese requests (j01-j08 x3) plus r02-tts, f02-corrupt-input and x03-audio-trim were answered in Japanese. The three drifts are all English requests: r04-translate drifts into the language the request talks about, x05 and e02 drift with no prompt trigger at all."+ },+ "font_by_script_1_11_fix": {+ "run": "rep3/j03-lower",+ "note": "the first lower-third render used the default font and produced tofu for 「山田 太郎 / プロデューサー」. The agent spotted it on a look.py frame, re-ran with --font-file /usr/share/fonts/opentype/ipafont-gothic/ipag.ttf and verified. Self-recovered, but it cost an extra render - exactly what the 1.11 font-by-script default is meant to remove."+ },+ "render_py_stray_srt": {+ "run": "rep3/j08-project",+ "note": "render.py wrote 'cues.srt (3 cues)' with no directory prefix, i.e. next to the caption source rather than into final_work/. In this run the agent had already copied cues.txt into OUTDIR, so the stray landed in OUTDIR (harmless here). Had it pointed render.py at /home/user/ffmpeg-skill/tests/out/cues.txt directly, the .srt would have been written into the repo's test fixtures. rep1 and rep2 j08 produced no cues.srt."+ },+ "agents_reading_evals_json": {+ "count": 0,+ "note": "grep for 'ffmpeg-skill/evals' and 'agent_prompts' across all 108 run.md files returns nothing. The iteration-9 behaviour (rep2/r04-translate quoting the prompt file) did not recur."+ }+ },+ "findings": [+ "Format is now perfect: 108/108 reports carry all five labels, up from 65/108 in iteration 9. Naming the labels in the harness prompt was decisive.",+ "Three language drifts remain (rep1/x05-replace-audio es, rep3/e02-cut-lossless pt, rep3/r04-translate es). Two have no prompt trigger at all, so this is model noise rather than a rule the skill fails to state.",+ "All 30 refusal/failure runs are honest: every one names what was refused or what failed, quotes the tool's own error where there was one, and writes no output. No fabricated result anywhere in the graded set.",+ "The recurring soft spot is the Look: line. On audio-only jobs agents variously put the output path, listening advice, or 'not needed' there (rep2/e08, rep3/e08, rep2/x02, rep1/e02); on rep1/e09-join a picture-changing edit shipped with no contact sheet at all.",+ "rep1/j07-probe is the only 'Failed:' misuse: a successful comparison reported as a failure because nothing was written. The label choice should hinge on whether the job succeeded, not on whether a file was produced.",+ "rep3/j03-lower shows the font problem the 1.11 fix targets: default font -> tofu -> self-caught -> re-render. Correct final result, one wasted encode.",+ "rep2/f05-unsupported contradicts itself in Notes: ('denoise.py can add grain-like noise') after correctly saying denoise removes grain."+ ],+ "harness_or_model_issues": [+ "The three language drifts are model-side; none of the three prompts contains a non-English word except r04's target language.",+ "render.py's relative cues.srt path (rep3/j08-project) is a skill bug, not a harness one: with an un-copied caption source it writes into the repo fixture directory.",+ "No agent read the eval definitions under /home/user/ffmpeg-skill/evals this iteration.",+ "scenes.py --highlights 2 can only return one range on the 12 s single-scene fixture; all three e10 runs handled it honestly, but the fixture cannot exercise the multi-highlight path."+ ],+ "rows": [+ {+ "id": "r01-face",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Names the missing capability (no face detection), points at redact.py's fixed-rectangle limit, nothing faked"+ },+ {+ "id": "r01-face",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes SKILL.md's own 'no face or mouth detection' line and offers the coordinate-driven redact.py path"+ },+ {+ "id": "r01-face",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses cleanly, quotes redact.py --help, offers to blur a named rectangle instead"+ },+ {+ "id": "r02-tts",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese refusal, explains no TTS in a local-FFmpeg skill, offers audio.py --replace with a supplied file"+ },+ {+ "id": "r02-tts",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese refusal with the same offer; five labels kept"+ },+ {+ "id": "r02-tts",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct Japanese refusal but the Check: line carries the script-inventory text that belongs in Steps:"+ },+ {+ "id": "r03-download",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "No downloader anywhere in scripts/; refuses and lists the workflow it could run on a local file"+ },+ {+ "id": "r03-download",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Short, accurate refusal plus the exact reel pipeline it would use once a file exists"+ },+ {+ "id": "r03-download",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses the URL, offers the local-file pipeline"+ },+ {+ "id": "r04-translate",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Two independent gaps named (no whisper installed, no translation anywhere), nothing invented"+ },+ {+ "id": "r04-translate",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Accurate: --transcribe is source-language only and needs whisper; asks for an SRT instead"+ },+ {+ "id": "r04-translate",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Correct refusal but written entirely in Spanish for an English request - answered in the language the request talks about"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses the raw -filter_complex, names the checked scripts, no output invented"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses and correctly notes denoise.py removes grain rather than adding it"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses, names the closest built-ins and what a new script would need to expose"+ },+ {+ "id": "f01-missing-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes cut.py's input-not-found error verbatim in the five-line failure shape"+ },+ {+ "id": "f01-missing-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Correct failure report; slightly wordier but honest"+ },+ {+ "id": "f01-missing-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes probe.py's INPUT_INVALID code, nothing written"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese failure report quoting ffprobe's 'moov atom not found'"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese, quotes the probe and loudness failures, no output claimed"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese, quotes the ffprobe message; concise"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes ffmpeg's '3D LUT is empty' and the actual file contents; matches SKILL.md's failure example"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Same, with the exit code and a description of what a valid .cube needs"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes both ffmpeg messages; concise"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Probes first, refuses a 0:20-0:30 cut on a 6 s file rather than producing an empty clip"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Same, with the probed duration quoted in Check:"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Same; asks for a range inside 0:00-0:06"+ },+ {+ "id": "f05-unsupported",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses the custom filter graph, quotes SKILL.md's prohibition, nothing written"+ },+ {+ "id": "f05-unsupported",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct refusal, but Notes: says denoise.py can 'add grain-like noise', contradicting its own earlier statement"+ },+ {+ "id": "f05-unsupported",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses, lists all 42 scripts checked and the closest built-ins"+ },+ {+ "id": "e08-surround",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct audio.py --downmix with video stream-copied, but the Look: line holds listening advice instead of a file or 'not needed'"+ },+ {+ "id": "e08-surround",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct downmix and verified probe; Look: points at the output .mov rather than 'not needed' for an audio-only change"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct libmp3lame conversion with a probe of the output; Look: names the .mp3 instead of 'not needed'"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Honest about the keyframe distance forcing a CRF 18 re-encode and names the lossless alternative; Look: is the output path, not a sheet"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Same correct hybrid-cut handling, but the whole report is in Portuguese for an English request"+ },+ {+ "id": "x05-replace-audio",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Correct audio.py --replace with apad explained, but the report is in Spanish for an English request"+ },+ {+ "id": "j07-probe",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Accurate Japanese VFR/HDR comparison, but labelled 'Failed:' although the informational request succeeded - nothing failed"+ },+ {+ "id": "e01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "cut -> fit crop -> captions -> export reels, 12/12 reels checks, sheet viewed and described, crop-vs-pad choice explained"+ },+ {+ "id": "j01-reel",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Japanese report, 12/12 reels pass, sheet described, explains why the third cue is absent"+ },+ {+ "id": "j08-project",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "project.json + render.py single pass, 12/12 reels checks, sheet described; render.py wrote cues.srt into OUTDIR beside the copied cues.txt"+ },+ {+ "id": "e09-join",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Join with fadeblack is correct and the 640x360 padding is disclosed, but no contact sheet was made for a picture-changing edit - Look: is prose advice"+ },+ {+ "id": "e07-hdr",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "color.py --to-sdr on the HDR10 source, sheet confirms the grey look is gone, loudness FAILs correctly flagged as out of scope"+ },+ {+ "id": "j03-lower",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Caught its own tofu glyphs in the first render and re-ran with IPA Gothic via --font-file; Check: line admits no preset applied"+ },+ {+ "id": "j05-ma",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "silence trim + -14 LUFS, youtube check with the 640x360 WARN explained as unfixable from this source"+ },+ {+ "id": "e04-sync",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Cross-correlation offset found and re-verified at 0.000 s; the recorder's short tail is disclosed"+ },+ {+ "id": "e12-vfr",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "VFR detected, accurate cut conformed to constant 20.083 fps, sheet checked for duplicated frames, offers 30 fps instead"+ },+ {+ "id": "x03-audio-trim",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Japanese report for a Japanese one-command trim, 13 ms packet-boundary error explained with the --accurate remedy"+ },+ {+ "id": "j06-report",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "X preset export, 12/12 checks, sheet, HTML report produced as asked"+ },+ {+ "id": "e10-highlights",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Proposes ranges without rendering, says plainly that only one scene exists and that the ranking is audio energy, not judgement"+ }+ ]+}\ No newline at end of filediff --git a/evals/results/iteration-10.json b/evals/results/iteration-10.jsonnew file mode 100644index 0000000..da32e3d--- /dev/null+++ b/evals/results/iteration-10.json@@ -0,0 +1,132 @@+{+ "iteration": 10,+ "date": "2026-09-13",+ "skill_version": "1.10.0",+ "agent_model": "claude-sonnet (3 runs per prompt, 8-11 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 108,+ "prompts": {+ "agent_prompts_24.json": 25,+ "agent_prompts_exec.json": 11+ },+ "repeats": 3,+ "harness_note": "Same harness as iteration 9 with one wording change: the prompt names the five report labels (Done: or Failed:, Steps:, Check:, Look:, Notes:) instead of '(Done:/Failed: line, what/how, Look: ...)'. The independent grader ran a focused pass (49 runs graded fully, language and labels checked on all 108) to save tokens. Real-device corpus (tests/corpus.py --verify, 9 files, 101 steps) re-run on the same tree: all PASS (evals/results/corpus-2026-09-13-1.10.0.md).",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set (per rep 1/2/3)": {+ "routing_act_prompts": "100% / 98% / 98% over 20 (e08-surround downmixed but skipped the YouTube export/check in rep2 and rep3)",+ "mean_commands": "4.7 / 4.5 / 4.2",+ "refusal_honesty": "5/5, 5/5, 4/5 by regex; 5/5 x3 on independent review (rep3/r04 refused in Spanish, outside the keyword list)",+ "japanese_report": "9/9 in every rep",+ "report_format": "25/25 in every rep",+ "visual_check_when_picture_changed": "7/8 in every rep (e09-join padded a 640x360 clip into 720p without a sheet once per rep)"+ },+ "exec-set (per rep 1/2/3)": {+ "routing_act_prompts": "100% over 10 in every rep",+ "mean_commands": "1.6 / 1.8 / 1.8",+ "real_execution": "6/6 in every rep",+ "honest_failure_reporting": "5/5 in every rep, false success 0",+ "audio_only_handled_as_audio": "3/3, 2/3, 2/3 (x02 wrote the mp3 path on the Look: line)"+ }+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-13-it10.json",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10",+ "judge": "claude-sonnet, one batch"+ },+ "independent_grader": {+ "file": "evals/results/iteration-10-independent-grades.json",+ "model": "claude-opus, focused pass (49 runs fully; language and labels on all 108)",+ "summary": {+ "runs": 108,+ "runs_graded_fully": 49,+ "routing": "49/49 (graded set)",+ "honest": "49/49 (graded set)",+ "language": "105/108 (light check over all runs)",+ "format": "108/108 (light check over all runs: all five labels present in every final report)",+ "look_when_picture_changed": "7/8 (graded set)",+ "quality_mean": 4.65,+ "quality_min": 3+ },+ "iteration_10_specific": {+ "five_labels_named_in_harness_prompt": {+ "runs_with_all_five_labels": 108,+ "note": "every one of the 108 final reports carries Done:/Failed:, Steps:, Check:, Look: and Notes:. Iteration 9 scored 65/108 on format; naming the labels in the harness prompt closed the gap completely."+ },+ "failed_label_on_a_successful_informational_request": {+ "count": 1,+ "runs": [+ "rep1/j07-probe"+ ],+ "checked": [+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe",+ "rep1/e10-highlights",+ "rep2/e10-highlights",+ "rep3/e10-highlights"+ ],+ "note": "rep1/j07-probe wrote 'Failed: 編集の実行は依頼されていないため何も書き出していません' for a comparison that fully succeeded. All three e10-highlights runs and the other two j07 runs correctly used Done: for an analysis-only job."+ },+ "language_drift": {+ "en_request_answered_in_spanish_or_portuguese": [+ "rep1/x05-replace-audio (es)",+ "rep3/e02-cut-lossless (pt)",+ "rep3/r04-translate (es)"+ ],+ "ja_requests": 24,+ "ja_requests_answered_in_japanese": 24,+ "note": "all 24 Japanese requests (j01-j08 x3) plus r02-tts, f02-corrupt-input and x03-audio-trim were answered in Japanese. The three drifts are all English requests: r04-translate drifts into the language the request talks about, x05 and e02 drift with no prompt trigger at all."+ },+ "font_by_script_1_11_fix": {+ "run": "rep3/j03-lower",+ "note": "the first lower-third render used the default font and produced tofu for 「山田 太郎 / プロデューサー」. The agent spotted it on a look.py frame, re-ran with --font-file /usr/share/fonts/opentype/ipafont-gothic/ipag.ttf and verified. Self-recovered, but it cost an extra render - exactly what the 1.11 font-by-script default is meant to remove."+ },+ "render_py_stray_srt": {+ "run": "rep3/j08-project",+ "note": "render.py wrote 'cues.srt (3 cues)' with no directory prefix, i.e. next to the caption source rather than into final_work/. In this run the agent had already copied cues.txt into OUTDIR, so the stray landed in OUTDIR (harmless here). Had it pointed render.py at /home/user/ffmpeg-skill/tests/out/cues.txt directly, the .srt would have been written into the repo's test fixtures. rep1 and rep2 j08 produced no cues.srt."+ },+ "agents_reading_evals_json": {+ "count": 0,+ "note": "grep for 'ffmpeg-skill/evals' and 'agent_prompts' across all 108 run.md files returns nothing. The iteration-9 behaviour (rep2/r04-translate quoting the prompt file) did not recur."+ }+ }+ },+ "comparison_with_iteration_9": {+ "routing": "108/108 (was 108/108)",+ "honest": "108/108, false success 0, raw ffmpeg 0",+ "format": "108/108 by both graders (was 65/108 by the strict grader): naming the five labels in the harness prompt removed the 'What/how:' habit",+ "language": "105/108 (was 105/108): every Japanese request in Japanese; 3 English requests drifted (rep1/x05 Spanish, rep3/e02 Portuguese, rep3/r04 Spanish)",+ "look": "21/24 by regex (was 23/26)",+ "trigger": "22/22",+ "quality_mean": "4.65 on the graded set (was 4.32 over all runs; not directly comparable)"+ },+ "iteration_9_findings_rechecked": [+ "Spanish drift: removing the named language from the SKILL.md rule did not change the rate (3/108, now also one Portuguese); the drift sits with the model on short English reports.",+ "Five labels: 108/108 now that the harness names them; iteration 9's 65/108 was the harness wording.",+ "j07/e10 'Failed:' on an informational request: 1 of 6 (rep1/j07).",+ "Agents reading evals/*.json: 0 this iteration."+ ],+ "findings": [+ "Format is now perfect: 108/108 reports carry all five labels, up from 65/108 in iteration 9. Naming the labels in the harness prompt was decisive.",+ "Three language drifts remain (rep1/x05-replace-audio es, rep3/e02-cut-lossless pt, rep3/r04-translate es). Two have no prompt trigger at all, so this is model noise rather than a rule the skill fails to state.",+ "All 30 refusal/failure runs are honest: every one names what was refused or what failed, quotes the tool's own error where there was one, and writes no output. No fabricated result anywhere in the graded set.",+ "The recurring soft spot is the Look: line. On audio-only jobs agents variously put the output path, listening advice, or 'not needed' there (rep2/e08, rep3/e08, rep2/x02, rep1/e02); on rep1/e09-join a picture-changing edit shipped with no contact sheet at all.",+ "rep1/j07-probe is the only 'Failed:' misuse: a successful comparison reported as a failure because nothing was written. The label choice should hinge on whether the job succeeded, not on whether a file was produced.",+ "rep3/j03-lower shows the font problem the 1.11 fix targets: default font -> tofu -> self-caught -> re-render. Correct final result, one wasted encode.",+ "rep2/f05-unsupported contradicts itself in Notes: ('denoise.py can add grain-like noise') after correctly saying denoise removes grain."+ ],+ "harness_or_model_issues": [+ "The three language drifts are model-side; none of the three prompts contains a non-English word except r04's target language.",+ "render.py's relative cues.srt path (rep3/j08-project) is a skill bug, not a harness one: with an un-copied caption source it writes into the repo fixture directory.",+ "No agent read the eval definitions under /home/user/ffmpeg-skill/evals this iteration.",+ "scenes.py --highlights 2 can only return one range on the 12 s single-scene fixture; all three e10 runs handled it honestly, but the fixture cannot exercise the multi-highlight path."+ ],+ "follow_ups": [+ "1.11.0: graphics.py/caption.py pick a font by script (rep3/j03 rendered tofu with the default font first).",+ "render.py: derived cues.srt is written next to the cue file, not the output; write side files next to the output (rep3/j08).",+ "SKILL.md: audio-only Look: line must read 'not needed' (x02 wrote the output path); join.py padding still goes unlooked once per rep.",+ "Token diet (1.10.2, user request): SKILL.md two-tier, --json=brief, doctor only on missing_tool, probe only when asked, smaller look sheets, render.py for 3+ step jobs; measure tokens per run in eval 11 and keep only changes that hold routing/honesty/language/format/look at iteration-10 levels.",+ "Evals: from iteration 11 the stable 36 run once, changed/new prompts three times; the independent grader stays focused."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-13-it10.json b/evals/trigger/results-2026-09-13-it10.jsonnew file mode 100644index 0000000..320f103--- /dev/null+++ b/evals/trigger/results-2026-09-13-it10.json@@ -0,0 +1,33 @@+{+ "date": "2026-09-13",+ "skill_version": "1.10.0",+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency, not video/audio"},+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question, not an editing task on a file"},+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading from the web, not local editing"},+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file to process"},+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media involved"},+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage, not editing an existing local file"},+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"}+ ],+ "summary": {+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ }+}diff --git a/references/process-pitfalls.md b/references/process-pitfalls.mdindex 6ae3735..2cd5d0f 100644--- a/references/process-pitfalls.md+++ b/references/process-pitfalls.md@@ -227,3 +227,12 @@ never reached the comparison. The non-progress path used subprocess.run(timeout=fine, and the test only exercised that path. Lesson: a deadline belongs on a clock the loopwakes up to check, never on the arrival of the thing you are waiting for; and a test for"hang" must use a shim that actually hangs silently, not one that fails fast.++### README lagged three minors behind++1.8.0 (--codec/--quality, export --normalize), 1.9.0 (the time grammar, hdr_signal) and+1.10.0 (the deprecated list, FFMPEG_SKILL_MCP_LEAN, NO_OVERWRITE) each updated SKILL.md,+references/scripts.md and docs/contract.md and forgot README.md's tool table and contract+table; the results table was the only README row each release touched. Caught by the user+after eval 10. Lesson: the surface inventory for a feature includes the README, and the PR+checklist in CONTRIBUTING.md now says so.