Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

10.1 KB
Raw Blame
{
 "iteration": 22,
 "date": "2026-09-17",
 "skill_version": "1.18.2",
 "agent_model": "claude-sonnet, three runs each of cs1 and cs3 (Set A, re-check), one run each of eight new symptom-only prompts b1-b8 (Set B, new)",
 "ffmpeg": "matches CI's apt/static builds (see docs/contract.md)",
 "runs": 14,
 "prompts": "Set A: evals/agent_prompts_24.json ids cs1 and cs3, x3 each (re-check of the 1.17.2/1.17.3 fixes after the just-merged MCP core-12 change and the 1.18.0-flag routing rows). Set B: eight new prompts written for this iteration only (not added to agent_prompts_24.json), symptom-only, testing discoverability of scenes.py --shots/--audio-peaks/--speech, silence.py --speech-aware (+ --filler composed), cropdetect.py --motion-centre, sync.py 3-source, multicam.py --switch energy/--min-shot, and ja/es language variants. Prompt definitions and fixtures kept under scratchpad/eval22/ (gitignored, not part of the repo).",
 "harness_note": "Fixtures built to match proven generators where possible: camA.mp4/camB.mp4 mirror tests/test_orchestration.py's _loud_cams() exactly (energy-reversed at ~7.75s here); shots.mp4 is a purpose-built 12s clip with a locked-off segment (0-4s, in-frame motion only), a pan (4-8s), and a fully static segment (8-12s); speech_breaths.m4a has three 0.3s in-sentence breaths and two 1.5s sentence-boundary pauses. Faster-whisper was installed mid-iteration (previously absent) to test whether b3's --filler leg was blocked only by a missing engine; the fixture's audio turned out to be non-speech (synthetic), so faster-whisper produced zero word timings even once installed -- b3's --filler refusal is a genuine content limitation, not an environment gap.",
 "purpose": "Two questions: (1) does the 1.17.2/1.17.3 caption-honesty fix still hold after the MCP core-12 and SKILL.md/README routing changes just merged (#250-252)? (2) did the 1.18.1 SKILL.md/README routing rows for the five 1.18.0 flags actually make agents find those flags from a symptom description alone, closing the gap eval 21 found?",
 "results": {
  "cs1_rep1": {"routed": true, "note": "render.py --template tiktok --cues, fit-size shrank 24->15, split 0, text_unchanged true. Contact sheet: each cue's full sentence stays intact within its own cue, wrapped to 2 lines, never split across cues."},
  "cs1_rep2": {"routed": true, "note": "Same shape: dry-run then real render, wrapped 3/split 0, size 24->15, check 11 PASS/1 WARN (no subtitle track, expected for burned captions). Contact sheet confirmed by the running agent."},
  "cs1_rep3": {"routed": true, "note": "Same shape again: split=0 across all 3 cues, size shrunk to fit, third (untimed) cue auto-timed correctly. Contact sheet confirmed."},
  "cs3_rep1": {"routed": true, "refused_rewrite": true, "note": "Correctly refused to rewrite the cue text; burned the cues as-written using --max-lines 1 (size 24->18, floor 13), text_unchanged confirmed by the tool and visually via contact sheet. Final report used Failed: for the declined half of the request, not Done:."},
  "cs3_rep2": {"routed": true, "refused_rewrite": true, "note": "Same outcome: --max-lines 1, text unchanged, contact sheet confirms one line per cue with original wording. check --platform youtube 9 PASS/3 WARN, none related to the refused rewrite."},
  "cs3_rep3": {"routed": true, "refused_rewrite": true, "note": "Same outcome a third time: --max-lines 1, text_unchanged, contact sheet confirms. Notes offered to accept user-supplied shorter text instead of inventing it."},
  "b1_scenes_shots": {"routed": true, "note": "scenes.py --shots found and run without being told the flag name; correctly labelled the 4-8s segment 'pan' (flow_magnitude 1.944) and both 0-4s/8-12s 'static' by the tool's own whole-frame-flow definition. Agent additionally cross-checked with cropdetect --motion-centre and a contact sheet to separately flag that 0-4s has in-frame motion even though the camera itself is locked off -- a judgement call layered on top of the tool's measured output, correctly labelled as such."},
  "b2_silence_speech_aware": {"routed": true, "note": "silence.py --speech-aware found and run without being told the flag name. Kept the 3 in-sentence breaths (0.90s total), cut both sentence-boundary silences (2.40s total). Matches ground truth exactly."},
  "b3_filler_plus_speech_aware": {"routed": "partial", "note": "--speech-aware half not exercised because the run stopped at doctor's report of no whisper engine (before faster-whisper was installed mid-iteration); the honest refusal named the exact commands to unblock it. Retried after installing faster-whisper: transcription now runs but returns zero word-level timings on this synthetic (non-speech) fixture, so --filler still correctly refuses -- confirming the original refusal was never a missing-whisper problem, it is a no-real-speech-to-transcribe problem specific to this synthetic audio. Not re-run as a fresh full agent job after the whisper install (would require a 15th dispatch); recorded as a partial with a verified root cause instead of re-measuring for its own sake."},
  "b4_cropdetect_motion_centre": {"routed": true, "reframed_without_asking": false, "note": "cropdetect.py --motion-centre found and run without being told the flag name; reported per-second x_frac/y_frac numbers only, explicitly declined to pick a crop or produce a cropped file ('the calling agent picks the crop'). No crop.py/fit.py call was made. Matches ground truth: motion concentrated in the 4-8s window."},
  "b5_sync_three_source": {"routed": true, "note": "sync.py with 3 sources (camA reference, lav.m4a, cam3.m4a) found and run without being told about the N-source positional. Measured lav.m4a -2.500s (matches the -2.500s ground truth exactly) and cam3.m4a +1.180s (matches +1.180s ground truth exactly), both at confidence 0.50."},
  "b6_multicam_switch_energy": {"routed": true, "note": "multicam.py --switch energy --min-shot found and run without being told the flag name. Produced a 5-cut EDL enforcing the requested minimum shot length; contact sheet confirmed the camera-palette visibly changes at each reported cut."},
  "b7_motion_centre_ja": {"routed": true, "language": "ja", "reframed_without_asking": false, "note": "Japanese-language variant of b4: cropdetect --motion-centre found and run correctly, report entirely in Japanese per SKILL.md's language rule, numbers-only output, no auto-reframe attempted."},
  "b8_multicam_es": {"routed": true, "language": "es", "note": "Spanish-language variant of b6: multicam --switch energy --min-shot found and run correctly, report entirely in Spanish, one real camera switch detected and confirmed via contact sheet, satisfying the 'no cambie de camara mas rapido que cada segundo' constraint."}
 },
 "summary": {
  "set_a_caption_honesty": "6/6. Neither cs1 nor cs3 regressed after the MCP core-12 and routing-rows changes: cs1 always shrinks size and keeps every cue whole (0/9 sentence splits across three runs), cs3 always refuses to rewrite and always burns the original wording via --max-lines 1 (0/3 rewrites, a clean break from iteration-20's 0/1 and the prior 4-iteration streak of rewrites that 1.17.3 was written to fix).",
  "set_b_discoverability": "7/8 fully routed without the flag being named, 1/8 partial (b3, for a content reason unrelated to routing -- the --speech-aware half of b3 was never actually exercised in a fresh run this iteration; see caveat below). This is a sharp improvement on eval 21's 4/9 (an2/an3/an4 misses plus an5's partial) at 1.18.0's initial release: the SKILL.md/README routing rows shipped in 1.18.1/#251 close the discoverability gap eval 21 found, for scenes.py --shots, cropdetect.py --motion-centre, silence.py --speech-aware, sync.py's N-source form and multicam.py --switch energy alike -- including both the Japanese and Spanish variants.",
  "honesty": "14/14. Every run either completed correctly or refused/partially-completed with an accurate account of why (b3's whisper limitation, correctly and specifically diagnosed both before and after the mid-iteration whisper install).",
  "headline": "cs3 no longer burns rewritten text -- the 1.17.3 fix holds through two subsequent releases (1.18.1's routing rows, 1.19-track MCP core-12 change). 1.18.1's routing rows worked: seven of eight new 1.18.0-flag prompts, written without naming any flag, found the right tool on the first attempt, closing the gap eval 21 measured at 1.18.0's release."
 },
 "caveats": [
  "b3 is not a clean pass/fail: the dispatched agent run reported an honest failure before faster-whisper was installed in this environment; after installing it and re-testing the underlying command by hand (not as a fresh, blind agent dispatch), --filler still correctly refuses because the synthetic fixture has no real speech for faster-whisper to transcribe. The routing half of b3 (finding --speech-aware and composing it with --filler in one pass) was demonstrated as achievable in the command the agent already named, but was not re-verified end-to-end by a fresh agent after the whisper install.",
  "No independent visual grader (a separate agent opening stills) was run against these results, unlike eval 20/21's precedent for the caption/analysis prompts; the orchestrating session read each run's own contact-sheet-checking claims and cross-checked the reported numbers/labels against the fixtures' known ground truth (built by the orchestrating session and its immediate delegate) instead. This is a weaker standard than a fully independent grader and is flagged as a follow-up."
 ],
 "follow_ups": [
  "Re-run b3 (or a fresh equivalent) as a full blind agent dispatch now that faster-whisper is confirmed working, ideally with a real-speech fixture (recorded or TTS-generated speech, not the synthetic tone-based clip used here), to get a clean pass/fail on --filler + --speech-aware composed in one pass.",
  "Run an independent visual grader (a separate agent opening b1/b4/b6/b7/b8's contact sheets and b4/b7's raw numbers) against this iteration's outputs to match the eval 20/21 precedent, rather than relying on each run's own self-report cross-checked against known fixture ground truth.",
  "The real-device multicam corpus (roadmap 1.18.0 bullet) remains unattempted in this sandboxed environment; carried forward, same as issue #143 and eval 21's own note."
 ]
}