Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: eval iteration 9 at 1.9.0 (108 runs, independent grader, trigger set) (#208)
claude[bot]d7baf64a605e
diff --git a/README.md b/README.mdindex 03b2346..3b5376e 100644--- a/README.md+++ b/README.md@@ -355,6 +355,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **108 / 108** | 1.9.0 re-run (2026-09-13, three passes per prompt, Sonnet agent, regex grader + independent Opus grader): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 23/26, user's language 105/108 (every Japanese request answered in Japanese; 3 English requests drifted to Spanish), report format 108/108 by regex (65/108 by the stricter grader, which now counts any missing label), trigger set 22/22; 12 of 15 platform jobs were one encode and `render.py` rendered once in 3/3 (was 1/3). The 1.9.0 time grammar was not used by any agent. Details in `evals/results/iteration-9.json` || **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` || **108 / 108** | 1.7.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 24 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 25/25, report format 108/108 by regex (104/108 by the stricter grader), user's language 101/108 (six English refusals answered in Spanish or Portuguese, one Japanese request in English); trigger set 22/22. Iteration-6 fixes held (fade-in only, Japanese audio trims, music no longer shortens the video). Details in `evals/results/iteration-7.json` || **36 / 36** | 1.4.15 re-run (2026-09-12, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, report format 25/25, user's language 9/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3, one Japanese report with English labels; trigger set 22/22. Both iteration-5 defects gone (no raw ffmpeg fallback, music no longer shortens the video). Details in `evals/results/iteration-6.json` |diff --git a/evals/results/iteration-9-independent-grades.json b/evals/results/iteration-9-independent-grades.jsonnew file mode 100644index 0000000..9e69bcb--- /dev/null+++ b/evals/results/iteration-9-independent-grades.json@@ -0,0 +1,1320 @@+{+ "summary": {+ "runs": 108,+ "routing": "108/108",+ "honest": "108/108",+ "language": "105/108",+ "format": "65/108",+ "look_when_picture_changed": "23/26",+ "quality_mean": 4.32,+ "quality_min": 2+ },+ "iteration_9_specific": {+ "time_grammar_hhmmssff_or_at_fps": {+ "runs_where_the_agent_used_it_in_a_command": 0,+ "runs_where_it_only_appears_in_tool_output": [+ "rep2/e01-reel",+ "rep2/j01-reel",+ "rep3/e01-reel",+ "rep3/j01-reel"+ ],+ "note": "the hh:mm:ss:ff@fps form shows up only inside caption.py's own 'wrote .srt (3 cues, 00:00:00:00-00:00:09:00 @ 30fps)' line. Every agent-written --start/--end used 0, 2, 0:01 or 0:02. In 108 runs no agent ever typed a frame-accurate or @fps time, so the 1.9.0 grammar is invisible in practice; other hits of the regex (rep1/e09, rep1/x05, rep2/e09, rep3/e09) are '1280x720 @ 30fps' resolution strings, not times."+ },+ "hdr_signal_vs_hdr": {+ "runs_that_should_surface_it": [+ "rep1/e07-hdr",+ "rep2/e07-hdr",+ "rep3/e07-hdr",+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ],+ "mentioned_hdr_signal": [+ "rep1/e07-hdr",+ "rep3/e07-hdr",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ],+ "showed_only_hdr": [+ "rep2/e07-hdr",+ "rep1/j07-probe"+ ],+ "note": "4 of 6; where it appears it is quoted straight out of probe --json and never explained to the user as a separate signal, so it reads as noise rather than as information"+ },+ "render_py_runs": {+ "runs": [+ "rep1/j08-project",+ "rep2/j08-project",+ "rep3/j08-project"+ ],+ "renders_per_run": {+ "rep1/j08": 1,+ "rep2/j08": 1,+ "rep3/j08": 1+ },+ "normalize_applied": 3,+ "normalize_applied_automatically_by_default": 0,+ "note": "all three project.json files were hand-written with an explicit \"normalize\": true (rep2/rep3) or an export block edited to include normalize (rep1). No run's log contains the phrase '--normalize on by default', so the 1.9.0 default was never observable; the iteration-8 double-render failure is gone, but because the agents set the flag themselves it cannot be attributed to the default."+ },+ "platform_export_runs": {+ "count": 15,+ "encodes_per_run": {+ "rep1/e01": 1,+ "rep2/e01": 2,+ "rep3/e01": 1,+ "rep1/j01": 1,+ "rep2/j01": 1,+ "rep3/j01": 1,+ "rep1/j05": 1,+ "rep2/j05": 1,+ "rep3/j05": 1,+ "rep1/j06": 1,+ "rep2/j06": 2,+ "rep3/j06": 2,+ "rep1/j08": 1,+ "rep2/j08": 1,+ "rep3/j08": 1+ },+ "single_encode": 12,+ "two_encode": [+ "rep2/e01-reel (loudness.py then export.py)",+ "rep2/j06-report (export, saw the loudness warning, re-exported with --normalize)",+ "rep3/j06-report (same pattern)"+ ],+ "note": "counts export.py + loudness.py + render.py's export stage. 12 of 15 are single-encode, up from 10 of 15 in iteration 8; the remaining waste is j06, where the agent exports first and only adds --normalize after reading the warning."+ },+ "check_without_platform_runs": {+ "runs": [+ "rep1/e12-vfr",+ "rep2/e04-sync",+ "rep2/e12-vfr",+ "rep3/e12-vfr"+ ],+ "warn_rows_named_in_check_line": [+ "rep1/e12-vfr",+ "rep3/e12-vfr"+ ],+ "handled_correctly": 3,+ "detail": [+ "rep1/e12-vfr: 2 WARN rows (loudness -9.8 LUFS, true peak -0.4 dBTP) named in Check: and explicitly left alone because no destination was given",+ "rep3/e12-vfr: same two WARNs named and left alone, with the reason",+ "rep2/e12-vfr: ran the platform-less check but the Check: line only says 'all checks PASS' -- the log itself lists no WARN row, so nothing was hidden, but nothing was surfaced either",+ "rep2/e04-sync: the WARN rows were acted on -- an unrequested loudness normalisation was applied to a sync job, which is the judgement-row behaviour SKILL.md asks agents not to chase"+ ]+ },+ "cases_where_a_regex_grader_would_disagree": [+ "rep3/r05-no-raw-ffmpeg: refuses with 'The ffmpeg-skill's scripts have no tool for this request' -- a correct refusal that a keyword list looking for 'cannot'/'not supported' would miss",+ "rep1/r05 and rep2/r05 similarly refuse without ever naming a closest built-in option, which a keyword grader would score identically to rep3 even though rep1's is a single collapsed paragraph",+ "rep2/x01, rep2/x02, rep2/x03, rep1/x04, rep1/x05, rep1/x06, rep3/x04: the Look: line for an audio-only or picture-unchanged job names the output file (or, in rep3/e10, run.md) instead of 'not needed' -- semantically honest, but a regex looking for 'not needed' would flag them, and rep3/e10's Look: naming run.md is genuinely wrong",+ "rep1/f01 and rep2/r01 and rep3/e10 are graded language failures (Spanish reply to an English request) that a language-agnostic keyword grader would pass",+ "rep1/j07 and rep3/j07 invent script names ('hdr_to_sdr', 'encode.py') that no routing regex would catch because no wrong command was actually run"+ ]+ },+ "findings": [+ "Format is still the weakest axis: 43 of 108 reports drop at least one of the five labels, and the pattern is unchanged from iteration 8 -- 'What/how:' or 'How:' replaces Steps:, and then Check: and Notes: vanish. It clusters almost perfectly on short jobs and refusals (x02/x04/x05/x06, r01-r05, f01/f04/f05) where the work itself was correct.",+ "Language drift persists at exactly the same 3-run rate and in the same shape: Spanish prose for an English request on reports where little or nothing was produced (rep1/f01-missing-input, rep2/r01-face, rep3/e10-highlights). The Japanese side is now clean -- every ja prompt got a Japanese report, which is an improvement over iteration 8's English-labels-with-one-Japanese-word failures.",+ "The 1.9.0 time grammar was never exercised. No agent wrote an hh:mm:ss:ff or @fps argument in 108 runs; the only occurrences are inside caption.py's own output. Nothing in SKILL.md's workflow gives an agent a reason to reach for frame-accurate times, so the feature is currently dead weight.",+ "probe's hdr_signal appears in 4 of the 6 HDR-relevant runs but is always pasted from --json and never used: no report distinguishes 'tagged as HDR' from 'actually HDR', which is what the field exists to express.",+ "render.py's normalize default cannot be confirmed: all three j08 runs wrote \"normalize\": true into project.json themselves and no log printed '--normalize on by default'. The good news is the iteration-8 double-render regression is gone (1 render per j08 run, down from 2 in two of three).",+ "Single-encode platform exports improved to 12/15. The residual waste is entirely j06 (2 of 3 runs) and rep2/e01, where the agent exports first and only then reacts to the loudness warning -- the warning works, but a habit of exporting with --normalize up front would save the encode.",+ "Look: discipline for join.py and colour work is still the blind spot. rep2/e09, rep3/e09 (a 640x360 clip padded into a 720p frame) and rep3/e07 (HDR->SDR tone map) changed the picture and produced no contact sheet; rep1/x06 and rep3/x06 reversed a clip with nothing looked at. SKILL.md step 8 lists transitions and colour but agents read 'join' and 'transcode' as picture-neutral.",+ "Two runs invented tool names while refusing nothing: rep1/j07 recommends an 'hdr_to_sdr conversion script' and rep3/j07 recommends 'encode.py --codec hevc'. Neither ran a wrong command, but both hand the user a command that does not exist -- a new failure mode the routing check does not see.",+ "rep2/e04-sync applied an unrequested loudness normalisation after a platform-less check WARN, the one clear case of chasing a judgement row this iteration."+ ],+ "harness_or_model_issues": [+ "run.md remains a model-written summary, not a transcript: encode counts, 'viewed the sheet' claims and quoted --json fragments are self-reported and cannot be independently verified per command.",+ "Scripts were again invoked through three different path spellings (/home/user/ffmpeg-skill/scripts, /root/.claude/skills/ffmpeg-skill/scripts, and bare relative paths), so the skill is installed in more than one place.",+ "rep2/r04-translate ran a bare `which whisper-cli whisper faster-whisper` shell probe and rep1/j03, rep2/j03, rep3/j03 ran `fc-list`; harmless and not raw ffmpeg, but the eval's routing rule does not state whether non-ffmpeg shell probes are allowed.",+ "rep3/e10-highlights wrote its Look: line pointing at run.md itself, which suggests the agent treated the run log as a deliverable."+ ],+ "rows": [+ {+ "id": "e01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Single-encode export --normalize reel, 12/12 reels pass, sheet viewed and described"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Honest lossless-vs-accurate tradeoff, but delivered an 8.09 s file that is not the requested 0:02-0:08 window"+ },+ {+ "id": "e03-logo",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "overlay.py top-right/opacity/fade plus a contact sheet named in Look:"+ },+ {+ "id": "e04-sync",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "sync.py offset 2.5 s with confidence reported and the apad tail flagged"+ },+ {+ "id": "e05-lufs",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "loudness -16/-1.5 then check podcast, all 7 pass"+ },+ {+ "id": "e06-silence",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "silence.py defaults, 5.25 s removed and reported as asked"+ },+ {+ "id": "e07-hdr",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "probe reports hdr_signal, color.py --to-sdr --codec h264, sheet inspected for the grey wash"+ },+ {+ "id": "e08-surround",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "downmix then youtube check, loudness fixed and re-checked"+ },+ {+ "id": "e09-join",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "fade join with the 640x360 clip padded to 720p and a contact sheet named"+ },+ {+ "id": "e10-highlights",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "scenes.py only, ranges proposed with the audio-proxy caveat, nothing rendered"+ },+ {+ "id": "e11-ready",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check tiktok, loudness fix, re-check all pass"+ },+ {+ "id": "e12-vfr",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "VFR detected and conformed; check.py without --platform, both WARN rows named and left alone"+ },+ {+ "id": "f01-missing-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Honest missing-input failure but written in Spanish for an English request and with no Steps/Check/Notes labels"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct ffprobe failure in Japanese but only Failed/what-how/Look; no Check: or Notes:"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Exact SKILL.md failure shape with ffmpeg's own LUT error quoted"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Refused on probe numbers without fabricating, but prose headings replace Steps/Check/Notes"+ },+ {+ "id": "f05-unsupported",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Correct refusal after auditing scripts/, but no Check:/Notes: lines"+ },+ {+ "id": "j01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Single-encode reels pipeline, Japanese report, sheet described"+ },+ {+ "id": "j02-60s",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "fit.py --duration 8 at 1.5x speed so no words are lost, reasoning stated in Japanese"+ },+ {+ "id": "j03-lower",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "graphics.py lower-third with an explicit CJK font and a confirmation frame"+ },+ {+ "id": "j04-bgm",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "audio.py --music --duck --music-loop correct; ran an unrequested youtube check but labelled it informational"+ },+ {+ "id": "j05-ma",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "silence then export youtube --normalize in one encode, 11/11 pass"+ },+ {+ "id": "j06-report",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "export x --normalize in one encode plus report.html, sheet viewed"+ },+ {+ "id": "j07-probe",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Good VFR/HDR comparison but invents an 'hdr_to_sdr' script instead of color.py --to-sdr, no hdr_signal, and no Steps/Check/Notes labels"+ },+ {+ "id": "j08-project",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "render.py project.json in one pass with normalize applied, sheet viewed, re-edit path explained"+ },+ {+ "id": "r01-face",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Refusal grounded in redact.py --help; missing Steps/Check/Notes labels"+ },+ {+ "id": "r02-tts",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Correct no-TTS refusal in Japanese offering --replace; label shape incomplete"+ },+ {+ "id": "r03-download",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "No downloader, asks for a local file; label shape incomplete"+ },+ {+ "id": "r04-translate",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Names caption.py --transcribe's limits, full five-label shape, English as requested"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Refuses raw ffmpeg but never names a closest built-in option and collapses the report into one paragraph"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Correct -16 LUFS normalise but only Done/Steps/Look, no Check: or Notes:"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "audio.py extension-driven conversion with full label shape"+ },+ {+ "id": "x03-audio-trim",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Sample-accurate 3.000 s wav cut reported in Japanese with all five labels"+ },+ {+ "id": "x04-transcode",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "h265 preset correct, but Done/How/Look only and Look: names the mp4 instead of 'not needed'"+ },+ {+ "id": "x05-replace-audio",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct track swap; Look: points at the mp4 and Check:/Notes: are missing"+ },+ {+ "id": "x06-reverse",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "reverse.py verified by probe only; no sheet for a reversed picture and no Check:/Notes:"+ },+ {+ "id": "e01-reel",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Correct reel and sheet, but loudness.py then export.py = two encodes instead of export --normalize"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Accepted cut.py's automatic re-encode and explained the lossless alternative"+ },+ {+ "id": "e03-logo",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 4,+ "reason": "Overlay and sheet correct; Done/How/Look only"+ },+ {+ "id": "e04-sync",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Sync verified by re-running sync.py, but an unrequested loudness change was applied and there is no Look: line"+ },+ {+ "id": "e05-lufs",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Loudness and podcast check correct; Done/How/Look only and Look: names the mp4"+ },+ {+ "id": "e06-silence",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Silence removal reported with numbers; Look: names the mp4 rather than 'not needed'"+ },+ {+ "id": "e07-hdr",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "HDR->SDR with a look.py --compare frame; probe output shown without hdr_signal"+ },+ {+ "id": "e08-surround",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Downmix correct but no check.py --platform youtube although YouTube was the stated destination"+ },+ {+ "id": "e09-join",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": false,+ "quality": 3,+ "reason": "fadeblack join correct but no contact sheet for a clip that was scaled/padded, and Done/How/Look only"+ },+ {+ "id": "e10-highlights",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Good analysis-only answer with caveats; Check:/Notes: labels missing"+ },+ {+ "id": "e11-ready",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "check -> loudness -> re-check all correct; no Look: line"+ },+ {+ "id": "e12-vfr",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "VFR conformed with --accurate; check.py without --platform reported as a generic sanity pass"+ },+ {+ "id": "f01-missing-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Honest missing-input failure in English; Failed/What-how/Look only"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Both failures quoted with the ffprobe message, full label shape in Japanese"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Accurate LUT failure with the file's real contents explained; no Check: line"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Refused on probe numbers, offered a valid range; label shape incomplete"+ },+ {+ "id": "f05-unsupported",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Quotes SKILL.md's no-raw-ffmpeg rule and rules out color.py; label shape incomplete"+ },+ {+ "id": "j01-reel",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "fit --duration trim, captions, export reels --normalize in one encode, sheet described in Japanese"+ },+ {+ "id": "j02-60s",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Speed change chosen over trimming with the reasoning and an alternative offered"+ },+ {+ "id": "j03-lower",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Lower-third with CJK font checked first and a confirmation frame"+ },+ {+ "id": "j04-bgm",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Ducked BGM with loop and fade-out; Done/How/Look only and Look: names the mp4"+ },+ {+ "id": "j05-ma",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Silence then export youtube --normalize in one encode, 11/11 pass"+ },+ {+ "id": "j06-report",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Correct X export and report.html, but exported once without --normalize and had to re-export"+ },+ {+ "id": "j07-probe",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "VFR vs HDR10 comparison quoting hdr_signal and the --codec h264 HDR refusal, full label shape"+ },+ {+ "id": "j08-project",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "project.json with normalize, dry-run then one render, sheet viewed"+ },+ {+ "id": "r01-face",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Correct no-face-detection refusal but answered in Spanish to an English request with no Steps/Check/Notes"+ },+ {+ "id": "r02-tts",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Honest no-TTS refusal in Japanese but the report is one Failed: paragraph with no other labels"+ },+ {+ "id": "r03-download",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "No downloader, asks for a local file and names the follow-up tools; label shape incomplete"+ },+ {+ "id": "r04-translate",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Notes both the missing whisper binary and the absent translation path; label shape incomplete"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Greps the scripts to prove no filter_complex passthrough; no Check:/Notes: labels"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Normalised and re-measured with the full five-label shape"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Conversion correct; Done/How/Look only and Look: names the mp3"+ },+ {+ "id": "x03-audio-trim",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Lossless packet-boundary cut with the -13 ms error explained; Look: names the wav instead of 'not needed'"+ },+ {+ "id": "x04-transcode",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "h265 preset with the archival assumption stated; Check:/Notes: missing and Look: names the mp4"+ },+ {+ "id": "x05-replace-audio",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Swap correct with the short-track padding flagged; Done/How/Look only"+ },+ {+ "id": "x06-reverse",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "reverse.py plus a look.py --compare frame proving the reversal, full label shape"+ },+ {+ "id": "e01-reel",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Single-encode reels pipeline with blurred pad, 12/12 pass, sheet described"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Took cut.py's frame-accurate re-encode and spelled out the lossless alternative"+ },+ {+ "id": "e03-logo",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 4,+ "reason": "Overlay plus sheet correct; Done/How/Look only"+ },+ {+ "id": "e04-sync",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Offset applied and verified by a second sync.py run, full label shape"+ },+ {+ "id": "e05-lufs",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Dry-run then normalise then podcast check, all pass"+ },+ {+ "id": "e06-silence",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "--list then cut with a sheet confirming the jump cuts"+ },+ {+ "id": "e07-hdr",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": false,+ "quality": 3,+ "reason": "Correct --to-sdr --codec h264 with hdr_signal quoted, but no contact sheet for a colour change and Look: names the mp4"+ },+ {+ "id": "e08-surround",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Downmix correct but no youtube check although YouTube was the destination"+ },+ {+ "id": "e09-join",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": false,+ "quality": 3,+ "reason": "fadeblack join correct but no sheet for the rescaled clip and Done/What-how/Look only"+ },+ {+ "id": "e10-highlights",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Sound analysis-only answer, but written in Spanish for an English request and Look: points at run.md"+ },+ {+ "id": "e11-ready",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check tiktok, loudness fix, re-check, verdict stated plainly"+ },+ {+ "id": "e12-vfr",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "VFR conformed, sheet inspected for stutter, both WARN rows from the platform-less check named"+ },+ {+ "id": "f01-missing-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Honest missing-input failure; Failed/What-how/Look only"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "ffprobe error quoted with the full five-label shape in Japanese"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Exact failure shape with the invalid .cube explained"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refused before running cut.py using probe numbers, full label shape"+ },+ {+ "id": "f05-unsupported",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Correct refusal citing SKILL.md and look.py --help; Check:/Notes: missing"+ },+ {+ "id": "j01-reel",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Crop-based 9:16 reel in one encode, sheet described including the crop cost"+ },+ {+ "id": "j02-60s",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "1.5x speed change so nothing is cut, reasoning in Japanese"+ },+ {+ "id": "j03-lower",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Caught tofu glyphs on the first frame check and re-ran with IPAGothic"+ },+ {+ "id": "j04-bgm",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Ducked, looped BGM at -22 dB; Done/how/Look only and Look: names the mp4"+ },+ {+ "id": "j05-ma",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Silence then export youtube --normalize in one encode with a sheet"+ },+ {+ "id": "j06-report",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Correct X export plus report.html, but the first export ran without --normalize and had to be repeated"+ },+ {+ "id": "j07-probe",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Full label shape and hdr_signal quoted, but invents an 'encode.py' tool twice as the CFR/HDR path"+ },+ {+ "id": "j08-project",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "project.json dry-run then one render with normalize, sheet viewed and the pad explained"+ },+ {+ "id": "r01-face",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "No-face-detection refusal with the full five-label shape and a concrete manual alternative"+ },+ {+ "id": "r02-tts",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Honest no-TTS refusal in Japanese offering --replace/--mix; label shape incomplete"+ },+ {+ "id": "r03-download",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "No downloader, asks for the local file; label shape incomplete"+ },+ {+ "id": "r04-translate",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Both blockers named with the full label shape and a burn-in offer"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Greps prove no vignette/grain/filter_complex and says there is no tool for this request; only Failed:/Look:"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Normalise plus podcast check with the full label shape"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Minimal correct conversion with all five labels"+ },+ {+ "id": "x03-audio-trim",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Sample-accurate 3.000 s cut reported in Japanese with all five labels"+ },+ {+ "id": "x04-transcode",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Dry-run then h265 export; Done/How/Look only and Look: names the mp4"+ },+ {+ "id": "x05-replace-audio",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Swap with both inputs probed, silence tail flagged, full label shape"+ },+ {+ "id": "x06-reverse",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct reverse with all five labels, but no frame or sheet to confirm the reversal by eye"+ }+ ]+}\ No newline at end of filediff --git a/evals/results/iteration-9.json b/evals/results/iteration-9.jsonnew file mode 100644index 0000000..011eb2c--- /dev/null+++ b/evals/results/iteration-9.json@@ -0,0 +1,195 @@+{+ "iteration": 9,+ "date": "2026-09-13",+ "skill_version": "1.9.0",+ "agent_model": "claude-sonnet (3 runs per prompt, 8-11 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 108,+ "prompts": {+ "agent_prompts_24.json": 25,+ "agent_prompts_exec.json": 11+ },+ "repeats": 3,+ "harness_note": "Same harness as iteration 8 (per-run prompt.json, subagents cannot ask questions, report in the request's language). Transcripts kept outside the repo. The eighth audit ran in parallel on the same tree and its fixes (1.9.1) were not in the tree the agents used.",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set (per rep 1/2/3)": {+ "routing_act_prompts": "100% / 98% / 98% over 20 (e08-surround downmixed but skipped the export/check in rep2 and rep3)",+ "mean_commands": "4.9 / 4.0 / 4.2",+ "refusal_honesty": "5/5, 5/5, 4/5 by regex; 5/5 x3 on independent review (rep3/r05 says 'no tool for this request', a phrase the keyword list misses)",+ "japanese_report": "9/9 in every rep",+ "report_format": "25/25, 22/25, 25/25 (rep2: e04, e11, r02 dropped a label)",+ "visual_check_when_picture_changed": "8/8, 6/8, 6/8 (e09-join padded a 640x360 clip into 720p without a sheet in rep2 and rep3; rep3/e07 tone-mapped without a sheet)"+ },+ "exec-set (per rep 1/2/3)": {+ "routing_act_prompts": "100% over 10 in every rep",+ "mean_commands": "1.3 / 1.8 / 1.8",+ "real_execution": "6/6 in every rep",+ "honest_failure_reporting": "5/5 in every rep, false success 0",+ "audio_only_handled_as_audio": "3/3, 0/3, 3/3 (rep2 x01-x03 wrote a Look: line other than 'not needed' for a WAV: 'audio-only file, no picture to inspect'; correct in substance, the regex wants the exact phrase)"+ }+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-12-it9.json",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10",+ "judge": "claude-sonnet, one batch"+ },+ "independent_grader": {+ "file": "evals/results/iteration-9-independent-grades.json",+ "model": "claude-opus, every run read",+ "summary": {+ "runs": 108,+ "routing": "108/108",+ "honest": "108/108",+ "language": "105/108",+ "format": "65/108",+ "look_when_picture_changed": "23/26",+ "quality_mean": 4.32,+ "quality_min": 2+ },+ "strictness_note": "Same rules as iteration 8's pass for language. Format was counted more strictly than iteration 8: any missing label among Done/Failed, Steps, Check, Look, Notes is a miss (65/108), where iteration 8 counted 'What/how:' in place of Steps: (91/108). The harness prompt itself says '(Done:/Failed: line, what/how, Look: ...)', which induces that label, so the format column measures the harness as much as the skill.",+ "iteration_9_specific": {+ "time_grammar_hhmmssff_or_at_fps": {+ "runs_where_the_agent_used_it_in_a_command": 0,+ "runs_where_it_only_appears_in_tool_output": [+ "rep2/e01-reel",+ "rep2/j01-reel",+ "rep3/e01-reel",+ "rep3/j01-reel"+ ],+ "note": "the hh:mm:ss:ff@fps form shows up only inside caption.py's own 'wrote .srt (3 cues, 00:00:00:00-00:00:09:00 @ 30fps)' line. Every agent-written --start/--end used 0, 2, 0:01 or 0:02. In 108 runs no agent ever typed a frame-accurate or @fps time, so the 1.9.0 grammar is invisible in practice; other hits of the regex (rep1/e09, rep1/x05, rep2/e09, rep3/e09) are '1280x720 @ 30fps' resolution strings, not times."+ },+ "hdr_signal_vs_hdr": {+ "runs_that_should_surface_it": [+ "rep1/e07-hdr",+ "rep2/e07-hdr",+ "rep3/e07-hdr",+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ],+ "mentioned_hdr_signal": [+ "rep1/e07-hdr",+ "rep3/e07-hdr",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ],+ "showed_only_hdr": [+ "rep2/e07-hdr",+ "rep1/j07-probe"+ ],+ "note": "4 of 6; where it appears it is quoted straight out of probe --json and never explained to the user as a separate signal, so it reads as noise rather than as information"+ },+ "render_py_runs": {+ "runs": [+ "rep1/j08-project",+ "rep2/j08-project",+ "rep3/j08-project"+ ],+ "renders_per_run": {+ "rep1/j08": 1,+ "rep2/j08": 1,+ "rep3/j08": 1+ },+ "normalize_applied": 3,+ "normalize_applied_automatically_by_default": 0,+ "note": "all three project.json files were hand-written with an explicit \"normalize\": true (rep2/rep3) or an export block edited to include normalize (rep1). No run's log contains the phrase '--normalize on by default', so the 1.9.0 default was never observable; the iteration-8 double-render failure is gone, but because the agents set the flag themselves it cannot be attributed to the default."+ },+ "platform_export_runs": {+ "count": 15,+ "encodes_per_run": {+ "rep1/e01": 1,+ "rep2/e01": 2,+ "rep3/e01": 1,+ "rep1/j01": 1,+ "rep2/j01": 1,+ "rep3/j01": 1,+ "rep1/j05": 1,+ "rep2/j05": 1,+ "rep3/j05": 1,+ "rep1/j06": 1,+ "rep2/j06": 2,+ "rep3/j06": 2,+ "rep1/j08": 1,+ "rep2/j08": 1,+ "rep3/j08": 1+ },+ "single_encode": 12,+ "two_encode": [+ "rep2/e01-reel (loudness.py then export.py)",+ "rep2/j06-report (export, saw the loudness warning, re-exported with --normalize)",+ "rep3/j06-report (same pattern)"+ ],+ "note": "counts export.py + loudness.py + render.py's export stage. 12 of 15 are single-encode, up from 10 of 15 in iteration 8; the remaining waste is j06, where the agent exports first and only adds --normalize after reading the warning."+ },+ "check_without_platform_runs": {+ "runs": [+ "rep1/e12-vfr",+ "rep2/e04-sync",+ "rep2/e12-vfr",+ "rep3/e12-vfr"+ ],+ "warn_rows_named_in_check_line": [+ "rep1/e12-vfr",+ "rep3/e12-vfr"+ ],+ "handled_correctly": 3,+ "detail": [+ "rep1/e12-vfr: 2 WARN rows (loudness -9.8 LUFS, true peak -0.4 dBTP) named in Check: and explicitly left alone because no destination was given",+ "rep3/e12-vfr: same two WARNs named and left alone, with the reason",+ "rep2/e12-vfr: ran the platform-less check but the Check: line only says 'all checks PASS' -- the log itself lists no WARN row, so nothing was hidden, but nothing was surfaced either",+ "rep2/e04-sync: the WARN rows were acted on -- an unrequested loudness normalisation was applied to a sync job, which is the judgement-row behaviour SKILL.md asks agents not to chase"+ ]+ },+ "cases_where_a_regex_grader_would_disagree": [+ "rep3/r05-no-raw-ffmpeg: refuses with 'The ffmpeg-skill's scripts have no tool for this request' -- a correct refusal that a keyword list looking for 'cannot'/'not supported' would miss",+ "rep1/r05 and rep2/r05 similarly refuse without ever naming a closest built-in option, which a keyword grader would score identically to rep3 even though rep1's is a single collapsed paragraph",+ "rep2/x01, rep2/x02, rep2/x03, rep1/x04, rep1/x05, rep1/x06, rep3/x04: the Look: line for an audio-only or picture-unchanged job names the output file (or, in rep3/e10, run.md) instead of 'not needed' -- semantically honest, but a regex looking for 'not needed' would flag them, and rep3/e10's Look: naming run.md is genuinely wrong",+ "rep1/f01 and rep2/r01 and rep3/e10 are graded language failures (Spanish reply to an English request) that a language-agnostic keyword grader would pass",+ "rep1/j07 and rep3/j07 invent script names ('hdr_to_sdr', 'encode.py') that no routing regex would catch because no wrong command was actually run"+ ]+ }+ },+ "comparison_with_iteration_8": {+ "routing": "108/108 (was 108/108)",+ "honest": "108/108, false success 0, raw ffmpeg 0 (was the same)",+ "look": "23/26 (was 22/24; join.py padding and one HDR tone-map without a sheet)",+ "trigger": "22/22 (was 22/22)",+ "mean_commands": "4.4 / 1.6 (was 4.4 / 2.0)",+ "platform_exports": "12 of 15 platform jobs were one encode (was 10 of 15); j08 rendered once in 3/3 (was 1/3)",+ "language": "105/108 (was 98/108): every Japanese request got a Japanese report; 3 English requests still answered in Spanish (rep1/f01, rep2/r01, rep3/e10)",+ "quality_mean": "4.32 (was 4.46; the stricter format count pulls short correct jobs down to 3)"+ },+ "iteration_8_findings_rechecked": [+ "Japanese labels-only reports: gone (0 of 27 Japanese runs, was 7).",+ "Spanish drift on English refusal-shaped prompts: still 3/108, same shape (little or nothing produced). SKILL.md's language paragraph itself uses 'Spanish subtitles' as its example.",+ "render.py double render: gone (1 render per j08 run); but every agent wrote \"normalize\": true itself, so the 1.9.0 default was never exercised.",+ "--codec / --quality: used in 3 runs (e07-hdr --codec h264 x2, one advice mention); --quality 0."+ ],+ "findings": [+ "Format is still the weakest axis: 43 of 108 reports drop at least one of the five labels, and the pattern is unchanged from iteration 8 -- 'What/how:' or 'How:' replaces Steps:, and then Check: and Notes: vanish. It clusters almost perfectly on short jobs and refusals (x02/x04/x05/x06, r01-r05, f01/f04/f05) where the work itself was correct.",+ "Language drift persists at exactly the same 3-run rate and in the same shape: Spanish prose for an English request on reports where little or nothing was produced (rep1/f01-missing-input, rep2/r01-face, rep3/e10-highlights). The Japanese side is now clean -- every ja prompt got a Japanese report, which is an improvement over iteration 8's English-labels-with-one-Japanese-word failures.",+ "The 1.9.0 time grammar was never exercised. No agent wrote an hh:mm:ss:ff or @fps argument in 108 runs; the only occurrences are inside caption.py's own output. Nothing in SKILL.md's workflow gives an agent a reason to reach for frame-accurate times, so the feature is currently dead weight.",+ "probe's hdr_signal appears in 4 of the 6 HDR-relevant runs but is always pasted from --json and never used: no report distinguishes 'tagged as HDR' from 'actually HDR', which is what the field exists to express.",+ "render.py's normalize default cannot be confirmed: all three j08 runs wrote \"normalize\": true into project.json themselves and no log printed '--normalize on by default'. The good news is the iteration-8 double-render regression is gone (1 render per j08 run, down from 2 in two of three).",+ "Single-encode platform exports improved to 12/15. The residual waste is entirely j06 (2 of 3 runs) and rep2/e01, where the agent exports first and only then reacts to the loudness warning -- the warning works, but a habit of exporting with --normalize up front would save the encode.",+ "Look: discipline for join.py and colour work is still the blind spot. rep2/e09, rep3/e09 (a 640x360 clip padded into a 720p frame) and rep3/e07 (HDR->SDR tone map) changed the picture and produced no contact sheet; rep1/x06 and rep3/x06 reversed a clip with nothing looked at. SKILL.md step 8 lists transitions and colour but agents read 'join' and 'transcode' as picture-neutral.",+ "Two runs invented tool names while refusing nothing: rep1/j07 recommends an 'hdr_to_sdr conversion script' and rep3/j07 recommends 'encode.py --codec hevc'. Neither ran a wrong command, but both hand the user a command that does not exist -- a new failure mode the routing check does not see.",+ "rep2/e04-sync applied an unrequested loudness normalisation after a platform-less check WARN, the one clear case of chasing a judgement row this iteration."+ ],+ "harness_or_model_issues": [+ "run.md remains a model-written summary, not a transcript: encode counts, 'viewed the sheet' claims and quoted --json fragments are self-reported and cannot be independently verified per command.",+ "Scripts were again invoked through three different path spellings (/home/user/ffmpeg-skill/scripts, /root/.claude/skills/ffmpeg-skill/scripts, and bare relative paths), so the skill is installed in more than one place.",+ "rep2/r04-translate ran a bare `which whisper-cli whisper faster-whisper` shell probe and rep1/j03, rep2/j03, rep3/j03 ran `fc-list`; harmless and not raw ffmpeg, but the eval's routing rule does not state whether non-ffmpeg shell probes are allowed.",+ "rep3/e10-highlights wrote its Look: line pointing at run.md itself, which suggests the agent treated the run log as a deliverable."+ ],+ "follow_ups": [+ "SKILL.md language paragraph: drop the 'Spanish subtitles' example or phrase it without naming a language; the three Spanish drifts follow the only foreign-language word in the rule.",+ "SKILL.md: name a reason to use hh:mm:ss:ff / @fps (an editor's timecode list, a cue sheet from an NLE) or the grammar stays unexercised; 0 uses in 108 runs.",+ "SKILL.md probe section: say what hdr_signal means in one sentence so the report can tell 'tagged HDR' from 'HDR signal'; 4 of 6 runs pasted it unexplained.",+ "SKILL.md step 8 (Look): join.py that scales or pads and color.py --to-sdr change the picture; still read as picture-neutral in 3 runs.",+ "Two j07 runs recommended scripts that do not exist (hdr_to_sdr, encode.py); a 'name only scripts that exist' line near the request table, and the doctor list as the source of truth.",+ "Harness: replace '(Done:/Failed: line, what/how, Look: ...)' in the prompt with the five labels so the format column measures the skill.",+ "grade_runs_24.py: accept 'no tool' in the honesty keyword list and any Look: line that says audio-only for WAV inputs."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-12-it9.json b/evals/trigger/results-2026-09-12-it9.jsonnew file mode 100644index 0000000..41ec0af--- /dev/null+++ b/evals/trigger/results-2026-09-12-it9.json@@ -0,0 +1,33 @@+{+ "date": "2026-09-13",+ "skill_version": "1.9.0",+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency, not video/audio"},+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to question, not an editing task on a file"},+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading from the web, not local editing"},+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file to process"},+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media involved"},+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation of new footage, not editing an existing local file"},+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"}+ ],+ "summary": {+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ }+}