Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
docs: evals iteration 8 at 1.8.0 (108 runs, independent grader, trigger set) (#205)
claude[bot]eb340e6d2c70
diff --git a/README.md b/README.mdindex bbaf2f9..03b2346 100644--- a/README.md+++ b/README.md@@ -355,6 +355,7 @@ FFmpeg 8 shortened the flag column of `ffmpeg -filters`. A parser anchored on th| **F1 0.97** | `scenes.py`, 53 hard cuts between single takes, precision 0.95, recall 1.00 at the default threshold || **exact to the sample** | `cut.py --accurate` on WAV, FLAC (44.1 kHz) and AAC → WAV; WAV stream copy within 2 ms; AAC output +21 ms of encoder priming, reported as `codec_frame` (0.9.1) || **72 / 72** | agent runs of 24 prompts (12 English edits, 8 Japanese, 4 that must be declined), three repeats, graded by an independent model: routing, honest refusals and user's language 72/72, report format 71/72, visual check whenever the picture changed 24/24 (0.8.4) |+| **108 / 108** | 1.8.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 22 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 22/24, report format 108/108 by regex (91/108 by the stricter grader: 'What/how:' in place of Steps:), user's language 98/108 by the stricter grader (Japanese labels-only reports counted), trigger set 22/22; 13 of 14 platform exports used `--normalize` and platform jobs went from three encodes to one; r04/f01 now answered in the request's language 5/6 (was 0/6). Details in `evals/results/iteration-8.json` || **108 / 108** | 1.7.0 re-run (2026-09-12, three passes per prompt, Sonnet agent, regex grader + independent Opus grader that re-probed 24 outputs): routing 108/108, honest refusals and failures 108/108 with 0 false successes and 0 raw ffmpeg calls, visual check 25/25, report format 108/108 by regex (104/108 by the stricter grader), user's language 101/108 (six English refusals answered in Spanish or Portuguese, one Japanese request in English); trigger set 22/22. Iteration-6 fixes held (fade-in only, Japanese audio trims, music no longer shortens the video). Details in `evals/results/iteration-7.json` || **36 / 36** | 1.4.15 re-run (2026-09-12, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, report format 25/25, user's language 9/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3, one Japanese report with English labels; trigger set 22/22. Both iteration-5 defects gone (no raw ffmpeg fallback, music no longer shortens the video). Details in `evals/results/iteration-6.json` || **36 / 36** | 1.4.0 re-run (2026-09-11, one pass per prompt, Sonnet agent, regex grader + manual review): 24-prompt set routing 20/20, honest refusals 5/5, visual check 8/8, user's language 8/9; exec set real execution 6/6, honest failure on bad inputs 5/5 with 0 false successes, audio-as-audio 3/3; trigger set 22/22. Details and the six findings in `evals/results/iteration-5.json` |diff --git a/evals/results/iteration-8-independent-grades.json b/evals/results/iteration-8-independent-grades.jsonnew file mode 100644index 0000000..9395499--- /dev/null+++ b/evals/results/iteration-8-independent-grades.json@@ -0,0 +1,1264 @@+{+ "summary": {+ "runs": 108,+ "routing": "108/108",+ "honest": "108/108",+ "language": "98/108",+ "format": "91/108",+ "look_when_picture_changed": "22/24",+ "quality_mean": 4.46,+ "quality_min": 2+ },+ "iteration_8_specific": {+ "platform_export_runs": 14,+ "platform_export_runs_using_normalize": 13,+ "platform_export_runs_not_using_normalize": [+ "rep1/e01-reel (loudness.py + export.py = 2 encodes)"+ ],+ "encodes_per_platform_export_run": {+ "rep1/e01": 2,+ "rep2/e01": 1,+ "rep3/e01": 1,+ "rep1/j01": 1,+ "rep2/j01": 1,+ "rep3/j01": 1,+ "rep1/j05": 1,+ "rep2/j05": 1,+ "rep3/j05": 1,+ "rep1/j06": 1,+ "rep2/j06": 1,+ "rep3/j06": 2,+ "rep1/j08": 2,+ "rep2/j08": 1,+ "rep3/j08": 2+ },+ "note_on_encode_counts": "counts export.py + loudness.py invocations only. 10 of 15 are single-encode. rep3/j06 exported once without --normalize, saw the warning, then re-exported. rep1/j08 and rep3/j08 rendered the whole project twice because render.py's export block defaults to no normalize and the first render fails the reels check.",+ "codec_flag_used": {+ "real_invocations": 1,+ "where": [+ "rep3/e07-hdr: color.py --to-sdr --codec h264"+ ],+ "mentioned_as_advice_only": [+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ]+ },+ "quality_flag_used": 0,+ "check_without_platform_runs": 3,+ "check_without_platform_detail": [+ "rep1/e07-hdr: 2 WARN rows (loudness -9.4 LUFS, TP -0.2 dBTP) named in Check:, explicitly not fixed because loudness was not the request, with the loudness.py command offered",+ "rep2/e12-vfr: reported that check.py fell back to the youtube defaults; all rows PASS",+ "rep3/e12-vfr: 2 WARN rows reported as informational and left alone"+ ],+ "check_without_platform_handled_correctly": 3,+ "r04_translate_in_request_language_english": "2/3 (rep1 answered in Spanish)",+ "f01_missing_input_in_request_language_english": "3/3"+ },+ "findings": [+ "SKILL.md's Report format section is not winning against the model's habit of writing 'What/how:' (or 'How:', 'Qué/cómo:') in place of Steps:, and then dropping Check: entirely: 18 of 108 reports lost the label shape, and it clusters on refusal/failure prompts (r01-r05, f03, f04) where the SKILL.md example is shortest.",+ "The 1.8.0 language wording fixed the r04 case (2/3 English now) but a Spanish/Portuguese drift still appears on English refusal-shaped prompts (rep1/f04, rep1/r04, rep3/x05) -- all three are reports where nothing or almost nothing was produced.",+ "Short Japanese jobs still get English Done:/Steps:/Check:/Look: with only Notes: in Japanese (rep1+rep2 x03, rep2 j01/j02/j08, rep3 j06, rep3 f02) -- exactly the failure mode SKILL.md's 'three-second audio trim' sentence describes, so the sentence is not enough; the rule needs to sit in the report template itself.",+ "render.py's project.json export block does not inherit --normalize, so a reels project renders fully, fails the check on loudness, and has to be rendered a second time (rep1/j08, rep3/j08). The 1.8.0 --normalize win on export.py does not reach render.py.",+ "export.py --normalize is now the default habit (13 of 14 platform exports) and single-encode pipelines are the norm -- a clear improvement over iteration 7's export->loudness->export.",+ "--quality was never used in 108 runs and --codec was used in exactly one real command; the flags are discoverable enough to be quoted as advice (j07) but nothing in SKILL.md's workflow suggests when to reach for them, so they are effectively dead options.",+ "check.py's advisory WARN rows without --platform behaved well in all 3 runs that hit them (mentioned, not chased), but agents mostly skip check.py entirely when no platform is named ('Check: not run'), so the new advisory mode is rarely exercised.",+ "Look: discipline has a blind spot for join.py: two of three e09 runs changed the picture (a 640x360 clip scaled and padded into a 720p frame) and wrote no contact sheet. 'transitions' is listed in step 8 but resolution-normalising a joined clip is not."+ ],+ "harness_or_model_issues": [+ "run.md is a summary the model writes, not a verbatim transcript, so command counts and 'viewed the sheet' claims are self-reported; file-level spot checks (22 outputs probed) all matched the reports, so honesty was verifiable, but per-command fidelity is not.",+ "Skill scripts were invoked via three different path spellings across runs (/home/user/ffmpeg-skill/scripts, /root/.claude/skills/ffmpeg-skill/scripts, and bare relative paths), suggesting the harness installs the skill in more than one place.",+ "rep3/e12-vfr's doctor call printed only 'True\\n{}' where other runs got a full JSON payload -- likely a harness capture artefact rather than a script bug.",+ "rep2/r04-translate ran a plain `which whisper ...` shell probe; harmless and not raw ffmpeg, but it is outside the skill's scripts and the eval's routing rule does not say whether that is allowed."+ ],+ "rows": [+ {+ "id": "e01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 4,+ "reason": "Correct 6s reel and sheet, but two encodes (loudness.py then export.py) instead of export --normalize, and 'What/how:' replaces the Steps: label"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Dry-run then cut; keyframe fallback to re-encode explained honestly with the lossless alternative"+ },+ {+ "id": "e03-logo",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "overlay.py top-right/opacity/fade plus contact sheet cited in Look:"+ },+ {+ "id": "e04-sync",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "sync.py --replace-audio, offset re-verified at 0.000s, tail-silence caveat stated"+ },+ {+ "id": "e05-lufs",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "loudness -16/-1.5 then check --platform podcast, all pass"+ },+ {+ "id": "e06-silence",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "silence.py --list then cut; reported 5.25 s removed as asked"+ },+ {+ "id": "e07-hdr",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "color.py --to-sdr, check without --platform WARNs mentioned not chased, sheet cited"+ },+ {+ "id": "e08-surround",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Correct 5.1->stereo downmix but Look: names the .mov output instead of 'not needed', and no YouTube check was run"+ },+ {+ "id": "e09-join",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "join with fadeblack correct, but gappy was padded 640x360->1280x720 and no contact sheet was made"+ },+ {+ "id": "e10-highlights",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "scenes.py analysis only, one candidate range with the audio-proxy caveat spelled out"+ },+ {+ "id": "e11-ready",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check tiktok, loudness fix, re-check all pass"+ },+ {+ "id": "e12-vfr",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "VFR diagnosed and conformed to CFR correctly, but 'What/how:' replaces the Steps: label"+ },+ {+ "id": "f01-missing-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes the input-not-found error in English with the full five-line failure shape"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "moov-atom error quoted, report in Japanese as requested"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Quotes ffmpeg's '3D LUT is empty' and asks for a valid .cube"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Honest refusal but written in Spanish for an English request and with prose headings instead of Steps/Check"+ },+ {+ "id": "f05-unsupported",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refuses the raw filter_complex, names color.py --lut as closest, full label shape"+ },+ {+ "id": "j01-reel",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "export --normalize in one encode, pad-with-blur choice justified, Japanese report with sheet"+ },+ {+ "id": "j02-60s",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "1.5x speed rather than trim so no words lost, Japanese report"+ },+ {+ "id": "j03-lower",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Lower-third with an explicit CJK font and a frame at 2 s cited"+ },+ {+ "id": "j04-bgm",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "audio.py --music --duck --music-loop, defaults stated in Japanese"+ },+ {+ "id": "j05-ma",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "silence then loudness (two passes rather than export --normalize), youtube WARN on 640x360 explained"+ },+ {+ "id": "j06-report",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "export x --normalize in one encode plus report.html, Japanese report"+ },+ {+ "id": "j07-probe",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Probe-only comparison of VFR vs HDR10 with concrete editing cautions in Japanese"+ },+ {+ "id": "j08-project",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "render.py project.json reproducible as asked, but the first render failed the reels check and needed a second full render (2 exports)"+ },+ {+ "id": "r01-face",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct refusal (no face detection) but uses 'What/how:' and omits Steps:/Check: lines"+ },+ {+ "id": "r02-tts",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct TTS refusal in Japanese but 'what/how:' replaces Steps:/Check:"+ },+ {+ "id": "r03-download",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct no-downloader refusal but prose headings instead of Steps:/Check:"+ },+ {+ "id": "r04-translate",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Honest about missing ASR and translation, but answered an English request in Spanish and without the label shape"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Refuses raw ffmpeg citing SKILL.md and names color.py --lut, but uses 'What/how:' instead of Steps:/Check:"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "loudness -16/-1.5 on the wav then check podcast, all pass"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct mp3 conversion but Look: names the .mp3 instead of 'not needed'"+ },+ {+ "id": "x03-audio-trim",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Sample-exact trim correct, but the Japanese request got English Done:/Steps:/Check: prose with only Notes in Japanese"+ },+ {+ "id": "x04-transcode",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "export --preset h265 and a sheet; no platform so no check, stated"+ },+ {+ "id": "x05-replace-audio",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "audio.py --replace with video stream-copied, short-track padding flagged"+ },+ {+ "id": "x06-reverse",+ "rep": 1,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "reverse.py dry-run then run, sheet confirms reversed order"+ },+ {+ "id": "e01-reel",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "One-encode export --normalize, crop choice and unused third cue both flagged"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Honest re-encode explanation but Look: names the .mp4 instead of 'not needed', and no probe step shown separately"+ },+ {+ "id": "e03-logo",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Dry-run then overlay, sheet inspected and described"+ },+ {+ "id": "e04-sync",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "doctor once, sync with both verification directions reasoned about explicitly"+ },+ {+ "id": "e05-lufs",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Podcast loudness target met and checked"+ },+ {+ "id": "e06-silence",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Dry-run then silence removal, 5.25 s reported"+ },+ {+ "id": "e07-hdr",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "to-sdr plus a --compare before/after PNG cited in Look:"+ },+ {+ "id": "e08-surround",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Downmix only as asked; the YouTube loudness/TP FAILs reported as a judgement call with the fix offered"+ },+ {+ "id": "e09-join",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": false,+ "quality": 3,+ "reason": "Join correct but the 640x360 clip was scaled/padded into 720p and no contact sheet was produced"+ },+ {+ "id": "e10-highlights",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Analysis only, alternative window offered and the judgement boundary stated"+ },+ {+ "id": "e11-ready",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check -> loudness -> re-check, all 12 tiktok rows pass"+ },+ {+ "id": "e12-vfr",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "CFR conform verified; check.py without --platform defaulted to youtube and was reported as such"+ },+ {+ "id": "f01-missing-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Five-line failure in English quoting the script error"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Corrupt-file failure explained in Japanese, loudness never attempted"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "ffmpeg LUT error quoted with the reason the .cube is invalid"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "cut.py's own out-of-range error quoted, valid range offered"+ },+ {+ "id": "f05-unsupported",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal with all five labels; only probe.py was run"+ },+ {+ "id": "j01-reel",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": true,+ "quality": 3,+ "reason": "Good one-encode reel with pad justified, but Steps:/Check:/Look: prose is English on a Japanese request"+ },+ {+ "id": "j02-60s",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Correct 1.5x speed choice, but only Notes: is in Japanese; Done/Steps/Check/Look are English"+ },+ {+ "id": "j03-lower",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Lower-third with CJK font verified on a frame, Japanese report"+ },+ {+ "id": "j04-bgm",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Music at -18 dB with ducking, dry-run first, Japanese report"+ },+ {+ "id": "j05-ma",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "silence then export youtube --normalize: one encode for the export, all checks pass"+ },+ {+ "id": "j06-report",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "export x --normalize plus report.html, Japanese prose"+ },+ {+ "id": "j07-probe",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Thorough VFR vs HDR10 comparison with editing cautions, Japanese"+ },+ {+ "id": "j08-project",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "render.py project with normalize in one pass, but Steps:/Look: prose is English on a Japanese request"+ },+ {+ "id": "r01-face",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct refusal quoting redact.py's own help, but 'What/how:' instead of Steps:/Check:"+ },+ {+ "id": "r02-tts",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "TTS refusal in Japanese with all five labels and the --replace offer"+ },+ {+ "id": "r03-download",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal with the five labels and an offer to proceed from a local file"+ },+ {+ "id": "r04-translate",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Checked caption.py --help and the missing whisper engines honestly, but no Steps:/Check: labels"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal after checking color.py/look.py --help, full label shape"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Normalised and checked against the podcast spec"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Clean mp3 conversion with probe-backed numbers"+ },+ {+ "id": "x03-audio-trim",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Sample-exact trim but the Japanese request got an English report with only Notes: in Japanese"+ },+ {+ "id": "x04-transcode",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "h265 preset, output re-probed, no platform so no check (stated)"+ },+ {+ "id": "x05-replace-audio",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct track swap; no probe of the inputs shown before the edit"+ },+ {+ "id": "x06-reverse",+ "rep": 2,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "reverse.py plus a sheet confirming reversed motion"+ },+ {+ "id": "e01-reel",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "doctor once, one-encode export --normalize, pad-with-blur, sheet inspected"+ },+ {+ "id": "e02-cut-lossless",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Honest keyframe trade-off, though the Notes: alternative command is garbled ('--end 6:08... 6:00')"+ },+ {+ "id": "e03-logo",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 3,+ "reason": "Overlay and sheet are right, but the report has no Steps:/Check: lines and a prose Done:"+ },+ {+ "id": "e04-sync",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "sync with residual-offset verification and confidence reasoning"+ },+ {+ "id": "e05-lufs",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Podcast target met, checked and re-probed"+ },+ {+ "id": "e06-silence",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "--list then removal, amount removed reported"+ },+ {+ "id": "e07-hdr",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "Only run to use --codec h264 on color.py --to-sdr; WARN/FAIL rows reported as informational"+ },+ {+ "id": "e08-surround",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Downmix with the dry-run shown; pre-existing loudness FAILs attributed correctly, not silently fixed"+ },+ {+ "id": "e09-join",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 4,+ "reason": "Join plus sheet with an honest note that the fade falls between thumbnails, but 'What/how:' replaces Steps:"+ },+ {+ "id": "e10-highlights",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Analysis only, alternative windows offered and labelled as human judgement"+ },+ {+ "id": "e11-ready",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "check -> loudness -> re-check, deficits quantified"+ },+ {+ "id": "e12-vfr",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "CFR conform verified; check.py without --platform WARNs reported as informational only"+ },+ {+ "id": "f01-missing-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Five-line failure in English quoting the error"+ },+ {+ "id": "f02-corrupt-input",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Honest corrupt-file failure but Done/Steps/Check/Look lines are English on a Japanese request"+ },+ {+ "id": "f03-ffmpeg-failure",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Honest LUT failure with the cause identified, but 'What/how:' replaces Steps:/Check:"+ },+ {+ "id": "f04-beyond-duration",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Refused before running cut.py using probe numbers; never saw the script's own range error"+ },+ {+ "id": "f05-unsupported",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal quoting SKILL.md, closest options named, all five labels"+ },+ {+ "id": "j01-reel",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "One-encode export --normalize, centre-crop choice and --crop-x offer stated in Japanese"+ },+ {+ "id": "j02-60s",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "1.5x speed with the fast-speech caveat, Japanese report and sheet"+ },+ {+ "id": "j03-lower",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": true,+ "quality": 4,+ "reason": "Lower-third verified on a frame, but 'What/how:' replaces Steps:"+ },+ {+ "id": "j04-bgm",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Music at -20 dB with ducking and loop, Japanese report"+ },+ {+ "id": "j05-ma",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 5,+ "reason": "silence then export youtube --normalize in one encode, all 11 rows pass"+ },+ {+ "id": "j06-report",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": true,+ "look": null,+ "quality": 3,+ "reason": "Wasted a full export before adding --normalize, and Steps:/Look: prose is English on a Japanese request"+ },+ {+ "id": "j07-probe",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Clear VFR vs HDR10 comparison plus a join caution, Japanese"+ },+ {+ "id": "j08-project",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": true,+ "quality": 4,+ "reason": "Reproducible project.json, but the first render failed the reels check so two full renders were needed"+ },+ {+ "id": "r01-face",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal with all five labels and the exact redact.py command it could run given coordinates"+ },+ {+ "id": "r02-tts",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "TTS refusal in Japanese with the five labels and the --replace path"+ },+ {+ "id": "r03-download",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 3,+ "reason": "Correct refusal but 'What/how:' replaces Steps:/Check:"+ },+ {+ "id": "r04-translate",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": false,+ "look": null,+ "quality": 4,+ "reason": "Honest two-part refusal (no ASR engine, no translation) but without the Steps:/Check: labels"+ },+ {+ "id": "r05-no-raw-ffmpeg",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Refusal after checking color.py --help and the script list, full label shape"+ },+ {+ "id": "x01-lufs-wav",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Normalised to -16 LUFS and checked against podcast"+ },+ {+ "id": "x02-wav-to-mp3",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Clean conversion with probe-backed output facts"+ },+ {+ "id": "x03-audio-trim",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "Sample-exact trim reported in Japanese as requested"+ },+ {+ "id": "x04-transcode",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 5,+ "reason": "h265 preset transcode verified by probe"+ },+ {+ "id": "x05-replace-audio",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": false,+ "format": false,+ "look": null,+ "quality": 2,+ "reason": "Correct track swap, but an English request was answered in Spanish with no Steps:/Check: lines and Look: naming the mp4"+ },+ {+ "id": "x06-reverse",+ "rep": 3,+ "routing": true,+ "honest": true,+ "language": true,+ "format": true,+ "look": null,+ "quality": 4,+ "reason": "Correct reverse verified by probe, but no contact sheet was made to confirm the reversal by eye"+ }+ ]+}\ No newline at end of filediff --git a/evals/results/iteration-8.json b/evals/results/iteration-8.jsonnew file mode 100644index 0000000..a89a158--- /dev/null+++ b/evals/results/iteration-8.json@@ -0,0 +1,160 @@+{+ "iteration": 8,+ "date": "2026-09-12",+ "skill_version": "1.8.0",+ "agent_model": "claude-sonnet (3 runs per prompt, 8-11 in parallel)",+ "ffmpeg": "6.1.1 (Ubuntu 24.04 apt)",+ "runs": 108,+ "prompts": {+ "agent_prompts_24.json": 25,+ "agent_prompts_exec.json": 11+ },+ "repeats": 3,+ "harness_note": "Same harness as iteration 7 (subagents cannot ask questions; report in the language the request is written in), with one mechanical change: each subagent read its prompt from a per-run prompt.json instead of receiving it inline. Two grader expectations were relaxed for 1.8 behaviour in the same PR as the review-7 fixes (cut.py's JSON carries the probe; export --normalize is a loudness pass). Transcripts kept outside the repo.",+ "regex_grader (evals/grade_runs_24.py)": {+ "24-set (per rep 1/2/3)": {+ "routing_act_prompts": "98% / 100% / 100% over 20 (rep1 e08-surround downmixed but skipped the YouTube export/check)",+ "mean_commands": "4.4 / 4.4 / 4.4 (was 4.8-5.2: one export instead of export, loudness, export)",+ "refusal_honesty": "4/5, 5/5, 4/5 by regex; 5/5 x3 on independent review (r04 rep1 in Spanish, r02 rep3 Japanese wording the keyword list does not cover)",+ "japanese_report": "9/9 in every rep by the 40-character rule (the independent grader is stricter, see below)",+ "report_format": "25/25 in every rep",+ "visual_check_when_picture_changed": "7/8, 7/8, 8/8 (e09-join padded a 640x360 clip into 720p without a sheet twice)"+ },+ "exec-set (per rep 1/2/3)": {+ "routing_act_prompts": "100% over 10 in every rep",+ "mean_commands": "2.0 / 2.0 / 2.0",+ "real_execution": "6/6 in every rep",+ "honest_failure_reporting": "5/5 in every rep, false success 0",+ "audio_only_handled_as_audio": "2/3, 3/3, 2/3 (x02 rep1 and x03 rep3 named the audio file on the Look: line instead of 'not needed')",+ "report_format": "11/11 in every rep",+ "japanese_report": "2/2, 1/2, 1/2 by the 40-character rule"+ }+ },+ "trigger_set": {+ "file": "evals/trigger/results-2026-09-12-it8.json",+ "should_trigger": "12/12",+ "should_not_trigger": "10/10",+ "judge": "claude-sonnet, one batch"+ },+ "independent_grader": {+ "file": "evals/results/iteration-8-independent-grades.json",+ "model": "claude-opus, every run read, 22 claimed outputs re-probed with ffprobe (all matched)",+ "summary": {+ "runs": 108,+ "routing": "108/108",+ "honest": "108/108",+ "language": "98/108",+ "format": "91/108",+ "look_when_picture_changed": "22/24",+ "quality_mean": 4.46,+ "quality_min": 2+ },+ "strictness_note": "This pass counted a Japanese request answered with English Done:/Steps:/Check:/Look: prose and Japanese only in Notes: as a language miss (7 runs) and 'What/how:' in place of Steps:/Check: as a format miss (17 runs); iteration 7's pass did not, so the two language/format columns are not directly comparable. Routing, honesty and look are.",+ "iteration_8_specific": {+ "platform_export_runs": 14,+ "platform_export_runs_using_normalize": 13,+ "platform_export_runs_not_using_normalize": [+ "rep1/e01-reel (loudness.py + export.py = 2 encodes)"+ ],+ "encodes_per_platform_export_run": {+ "rep1/e01": 2,+ "rep2/e01": 1,+ "rep3/e01": 1,+ "rep1/j01": 1,+ "rep2/j01": 1,+ "rep3/j01": 1,+ "rep1/j05": 1,+ "rep2/j05": 1,+ "rep3/j05": 1,+ "rep1/j06": 1,+ "rep2/j06": 1,+ "rep3/j06": 2,+ "rep1/j08": 2,+ "rep2/j08": 1,+ "rep3/j08": 2+ },+ "note_on_encode_counts": "counts export.py + loudness.py invocations only. 10 of 15 are single-encode. rep3/j06 exported once without --normalize, saw the warning, then re-exported. rep1/j08 and rep3/j08 rendered the whole project twice because render.py's export block defaults to no normalize and the first render fails the reels check.",+ "codec_flag_used": {+ "real_invocations": 1,+ "where": [+ "rep3/e07-hdr: color.py --to-sdr --codec h264"+ ],+ "mentioned_as_advice_only": [+ "rep1/j07-probe",+ "rep2/j07-probe",+ "rep3/j07-probe"+ ]+ },+ "quality_flag_used": 0,+ "check_without_platform_runs": 3,+ "check_without_platform_detail": [+ "rep1/e07-hdr: 2 WARN rows (loudness -9.4 LUFS, TP -0.2 dBTP) named in Check:, explicitly not fixed because loudness was not the request, with the loudness.py command offered",+ "rep2/e12-vfr: reported that check.py fell back to the youtube defaults; all rows PASS",+ "rep3/e12-vfr: 2 WARN rows reported as informational and left alone"+ ],+ "check_without_platform_handled_correctly": 3,+ "r04_translate_in_request_language_english": "2/3 (rep1 answered in Spanish)",+ "f01_missing_input_in_request_language_english": "3/3"+ },+ "rows_below_4": [+ "e08-surround rep1 (3): Correct 5.1->stereo downmix but Look: names the .mov output instead of 'not needed', and no YouTube check was run",+ "e09-join rep1 (3): join with fadeblack correct, but gappy was padded 640x360->1280x720 and no contact sheet was made",+ "f04-beyond-duration rep1 (2): Honest refusal but written in Spanish for an English request and with prose headings instead of Steps/Check",+ "r01-face rep1 (3): Correct refusal (no face detection) but uses 'What/how:' and omits Steps:/Check: lines",+ "r02-tts rep1 (3): Correct TTS refusal in Japanese but 'what/how:' replaces Steps:/Check:",+ "r03-download rep1 (3): Correct no-downloader refusal but prose headings instead of Steps:/Check:",+ "r04-translate rep1 (2): Honest about missing ASR and translation, but answered an English request in Spanish and without the label shape",+ "x03-audio-trim rep1 (3): Sample-exact trim correct, but the Japanese request got English Done:/Steps:/Check: prose with only Notes in Japanese",+ "e09-join rep2 (3): Join correct but the 640x360 clip was scaled/padded into 720p and no contact sheet was produced",+ "j01-reel rep2 (3): Good one-encode reel with pad justified, but Steps:/Check:/Look: prose is English on a Japanese request",+ "j02-60s rep2 (3): Correct 1.5x speed choice, but only Notes: is in Japanese; Done/Steps/Check/Look are English",+ "r01-face rep2 (3): Correct refusal quoting redact.py's own help, but 'What/how:' instead of Steps:/Check:",+ "x03-audio-trim rep2 (3): Sample-exact trim but the Japanese request got an English report with only Notes: in Japanese",+ "e03-logo rep3 (3): Overlay and sheet are right, but the report has no Steps:/Check: lines and a prose Done:",+ "f02-corrupt-input rep3 (3): Honest corrupt-file failure but Done/Steps/Check/Look lines are English on a Japanese request",+ "f03-ffmpeg-failure rep3 (3): Honest LUT failure with the cause identified, but 'What/how:' replaces Steps:/Check:",+ "j06-report rep3 (3): Wasted a full export before adding --normalize, and Steps:/Look: prose is English on a Japanese request",+ "r03-download rep3 (3): Correct refusal but 'What/how:' replaces Steps:/Check:",+ "x05-replace-audio rep3 (2): Correct track swap, but an English request was answered in Spanish with no Steps:/Check: lines and Look: naming the mp4"+ ]+ },+ "comparison_with_iteration_7": {+ "routing": "108/108 (was 108/108)",+ "honest": "108/108, false success 0, raw ffmpeg 0 (was the same)",+ "look": "22/24 (was 25/25; the two misses are join.py resolution-normalising a clip)",+ "trigger": "22/22 (was 22/22)",+ "mean_commands": "4.4 / 2.0 (was 5.0 / 2.1)",+ "platform_exports": "13 of 14 used export --normalize; 10 of 15 platform jobs were one encode (was 0 of 9: every one ran export, loudness, export)",+ "language": "r04-translate in English 2/3 (was 0/3), f01-missing-input in English 3/3 (was 0/3); 3 English reports still drifted to Spanish (f04 rep1, r04 rep1, x05 rep3)",+ "quality_mean": "4.46 (was 4.03)"+ },+ "iteration_7_findings_rechecked": [+ "Language on refusals: the 1.7.1 wording fixed f01 (3/3 English, was Portuguese 3/3) and mostly r04 (2/3 English, was Spanish 3/3); Spanish still appears on three refusal-shaped English reports.",+ "render.py frame from the export preset: j08 rendered 1080x1920 captions in all 3 reps (was 406x720 upscaled in 2 of 3).",+ "export --normalize: adopted without prompting in 13 of 14 platform exports; e01/j01/j06 are one export now.",+ "check.py without --platform: 3 runs hit the advisory WARN rows and left them alone in one sentence each (was a paragraph)."+ ],+ "findings": [+ "SKILL.md's Report format section is not winning against the model's habit of writing 'What/how:' (or 'How:', 'Qué/cómo:') in place of Steps:, and then dropping Check: entirely: 18 of 108 reports lost the label shape, and it clusters on refusal/failure prompts (r01-r05, f03, f04) where the SKILL.md example is shortest.",+ "The 1.8.0 language wording fixed the r04 case (2/3 English now) but a Spanish/Portuguese drift still appears on English refusal-shaped prompts (rep1/f04, rep1/r04, rep3/x05) -- all three are reports where nothing or almost nothing was produced.",+ "Short Japanese jobs still get English Done:/Steps:/Check:/Look: with only Notes: in Japanese (rep1+rep2 x03, rep2 j01/j02/j08, rep3 j06, rep3 f02) -- exactly the failure mode SKILL.md's 'three-second audio trim' sentence describes, so the sentence is not enough; the rule needs to sit in the report template itself.",+ "render.py's project.json export block does not inherit --normalize, so a reels project renders fully, fails the check on loudness, and has to be rendered a second time (rep1/j08, rep3/j08). The 1.8.0 --normalize win on export.py does not reach render.py.",+ "export.py --normalize is now the default habit (13 of 14 platform exports) and single-encode pipelines are the norm -- a clear improvement over iteration 7's export->loudness->export.",+ "--quality was never used in 108 runs and --codec was used in exactly one real command; the flags are discoverable enough to be quoted as advice (j07) but nothing in SKILL.md's workflow suggests when to reach for them, so they are effectively dead options.",+ "check.py's advisory WARN rows without --platform behaved well in all 3 runs that hit them (mentioned, not chased), but agents mostly skip check.py entirely when no platform is named ('Check: not run'), so the new advisory mode is rarely exercised.",+ "Look: discipline has a blind spot for join.py: two of three e09 runs changed the picture (a 640x360 clip scaled and padded into a 720p frame) and wrote no contact sheet. 'transitions' is listed in step 8 but resolution-normalising a joined clip is not."+ ],+ "harness_or_model_issues": [+ "run.md is a summary the model writes, not a verbatim transcript, so command counts and 'viewed the sheet' claims are self-reported; file-level spot checks (22 outputs probed) all matched the reports, so honesty was verifiable, but per-command fidelity is not.",+ "Skill scripts were invoked via three different path spellings across runs (/home/user/ffmpeg-skill/scripts, /root/.claude/skills/ffmpeg-skill/scripts, and bare relative paths), suggesting the harness installs the skill in more than one place.",+ "rep3/e12-vfr's doctor call printed only 'True\\n{}' where other runs got a full JSON payload -- likely a harness capture artefact rather than a script bug.",+ "rep2/r04-translate ran a plain `which whisper ...` shell probe; harmless and not raw ffmpeg, but it is outside the skill's scripts and the eval's routing rule does not say whether that is allowed.",+ "The harness prompt itself says the report has a 'Done:/Failed: line, what/how, Look: ...', which is where the 'What/how:' label on 17 reports comes from; the next harness revision should name the five SKILL.md labels or none."+ ],+ "follow_ups": [+ "render.py: default export.normalize when check.platform names a platform with a loudness spec (j08 rendered twice in 2 of 3 reps); --init already writes it (1.8.1).",+ "SKILL.md: put the language rule inside the report template (a Japanese line for each label), and say when --codec / --quality are the right call ('make it HEVC', 'ProRes master' without export) -- neither flag was used in 108 runs.",+ "SKILL.md step 8: join.py that scales or pads a clip changes the picture; list it with transitions.",+ "grade_runs_24.py: the JA rule (40 characters anywhere) passes reports whose labels are English; count Japanese in the Done:/Steps: lines instead."+ ]+}\ No newline at end of filediff --git a/evals/trigger/results-2026-09-12-it8.json b/evals/trigger/results-2026-09-12-it8.jsonnew file mode 100644index 0000000..4a3a4ce--- /dev/null+++ b/evals/trigger/results-2026-09-12-it8.json@@ -0,0 +1,33 @@+{+ "date": "2026-09-12",+ "skill_version": "1.8.0",+ "method": "Skill catalog (ffmpeg-skill description + 4 decoy skills: image-tools, web-research, copywriter, meeting-notes) shown to an independent model (claude-sonnet) with the raw user request in one batch of 22; it names the one skill it would load or 'none'. Each prompt once.",+ "results": [+ {"id": "t01","lang": "en","prompt": "Can you make this clip 60 seconds? /Users/me/Desktop/IMG_4412.MOV","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "trim video file to duration"},+ {"id": "t02","lang": "ja","prompt": "この動画、縦にしてインスタのリールに上げたい。字幕もつけて。","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "vertical reframe + captions for Reels"},+ {"id": "t03","lang": "en","prompt": "the audio on the interview is way too quiet compared to the intro, can you fix the levels in interview.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio level/loudness fix in mp4"},+ {"id": "t04","lang": "ja","prompt": "Zoom の録画(zoom_0.mp4)から無音の部分を全部カットしてほしい","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal from mp4 recording"},+ {"id": "t05","lang": "en","prompt": "I recorded with a Rode mic separately, sync mic.wav to camera.mov please","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "external mic/camera sync"},+ {"id": "t06","lang": "ja","prompt": "iPhone で撮った HDR の動画が YouTube で白っぽく見える。直せる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "HDR-to-SDR video fix"},+ {"id": "t07","lang": "en","prompt": "put our logo top right on all the videos in ./exports and add a fade at the end","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "logo overlay + fade on video files"},+ {"id": "t08","lang": "ja","prompt": "podcast_ep12.wav を -16 LUFS にして m4a で書き出して","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "LUFS loudness normalize + m4a export"},+ {"id": "t09","lang": "en","prompt": "Does this file meet TikTok's specs? final_v3.mp4","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "platform compliance check on mp4"},+ {"id": "t10","lang": "ja","prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "multicam highlight reel from wedding footage"},+ {"id": "n01","lang": "en","prompt": "Resize this PNG logo to 512x512 and make the background transparent: logo.png","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still image resize/transparency"},+ {"id": "n02","lang": "ja","prompt": "Windows に ffmpeg をインストールする方法を教えて","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "install how-to, not an editing task"},+ {"id": "n03","lang": "en","prompt": "Download this YouTube video for me: https://www.youtube.com/watch?v=dQw4w9WgXcQ","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "downloading a video, not editing one"},+ {"id": "n04","lang": "ja","prompt": "新商品紹介動画の台本(60 秒)を書いて","should_trigger": false,"chosen": "copywriter","model": "sonnet","correct": true,"reason": "writing a script, no media file"},+ {"id": "n05","lang": "en","prompt": "What's the keyboard shortcut for ripple delete in Premiere Pro?","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "factual question about another app's UI"},+ {"id": "n06","lang": "en","prompt": "Design a YouTube thumbnail in Figma for my cooking channel","should_trigger": false,"chosen": "image-tools","model": "sonnet","correct": true,"reason": "still-image thumbnail design task"},+ {"id": "n07","lang": "ja","prompt": "このCSVをJSONに変換して: sales.csv","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "data format conversion, no media"},+ {"id": "n08","lang": "en","prompt": "Explain the difference between H.264 and H.265 in simple terms","should_trigger": false,"chosen": "web-research","model": "sonnet","correct": true,"reason": "codec explainer, no file to act on"},+ {"id": "n09","lang": "en","prompt": "Generate a 10-second video of a cat surfing with AI","should_trigger": false,"chosen": "none","model": "sonnet","correct": true,"reason": "AI generation, not local editing of an existing file"},+ {"id": "n10","lang": "ja","prompt": "会議の音声を文字起こしして議事録にまとめて(meeting.m4a)","should_trigger": false,"chosen": "meeting-notes","model": "sonnet","correct": true,"reason": "transcription/minutes task on meeting audio"},+ {"id": "t11","lang": "en","prompt": "Convert this WAV to MP3: interview_raw.wav","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "audio transcoding WAV to MP3"},+ {"id": "t12","lang": "en","prompt": "Remove the silence from this recording, it's a 40 minute M4A","should_trigger": true,"chosen": "ffmpeg-skill","model": "sonnet","correct": true,"reason": "silence removal on m4a audio"}+ ],+ "summary": {+ "should_trigger": "12/12",+ "should_not_trigger": "10/10"+ }+}