Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
scenes.py: add --rank-by, fix false "motion ranking" claim in docstring (#26)
Shogo Kajiyama19aebc28db8a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c9fd107..1ebfd98 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,7 @@## Unreleased — FFmpeg 8+ / Windows compatibility for caption and color+- **`scenes.py --rank-by`.** `--highlights` ranked candidate scenes by audio energy only, and the docstring falsely claimed motion was also used (it never was — dead documentation). `--rank-by {audio,duration}` (default audio, unchanged) makes the criterion explicit and adds a real second option (longest scene first); the result JSON now reports `highlights_rank_by`.- **`fit.py --crop-x` / `--crop-y`.** `--fit crop` always cropped from the centre, so reframing a wide shot to 9:16 could cut off a subject held to one side (a product, a person off-centre). `--crop-x`/`--crop-y` (0=left/top, 0.5=centre default, 1=right/bottom) pick which edge to keep instead; range-checked before ffmpeg runs. No change to the default (centre) behaviour.- **Filter file paths on Windows.** `subtitles=`, `ass=`, `lut3d=file=`, `fontfile=` and `fontsdir=` values are parsed twice by ffmpeg (graph, then filter options), so a drive letter needs two levels of colon escaping (`D\\\\:/x.srt`). 0.9.1 escaped once and every caption / LUT job on Windows failed with `Unable to parse "original_size" option value` or `Error parsing a filter description`. `escape_filter_path` now escapes for both passes; `;` is escaped too. Reproduced and tested on Linux with a directory literally named `D:` plus spaces and non-ASCII in the path.- **`overlay.py --image` length on FFmpeg 7+.** `-shortest` alone left up to 2 s of the looped still after the video ended (8.1 / 9.0); the command now also passes `-t <video duration>`.diff --git a/README.md b/README.mdindex 27af8d2..94fdf63 100644--- a/README.md+++ b/README.md@@ -115,7 +115,7 @@ These are the rules the skill file gives the agent and the code enforces. Togeth5. **Contract-derived MCP.** `mcp/server.py` builds its `tools/list` from the contract. Tool names, order and `inputSchema` cannot drift from the scripts; a test keeps the two byte-identical.6. **Capability detection.** `doctor` reads `ffmpeg -encoders / -filters / -bsfs` and reports which of the components the tools need are present on this build (libx264, libass, zscale, loudnorm, xfade, …), before a job fails inside ffmpeg.7. **Unknown is not missing.** When a listing cannot be read (a layout the parser does not know, ffmpeg exiting non-zero) the affected capabilities are `unknown`: never `missing`, never silently `available`. An installed filter is not reported absent; a failed detection is not a pass.-8. **Verify the result.** The output is probed, and when the picture changed (captions, overlays, crops, colour, transitions) the agent runs `look.py` and inspects the PNG. The report is not finished until its `Look:` line names that image; audio-only jobs say `Look: not needed`. **"Inspects" means the calling agent's own vision, not a feature of this skill:** `look.py` only renders a PNG; nothing in this repository detects faces, products or any other subject. When a crop or reframe needs to keep a specific part of the frame (`fit.py --fit crop --crop-x/-y`, see [Tools](#tools)), it is the multimodal agent looking at that PNG and choosing the anchor — a non-visual caller (a script, a CLI user without eyes on the sheet) has to supply that decision itself; the default is a plain centre crop.+8. **Verify the result.** The output is probed, and when the picture changed (captions, overlays, crops, colour, transitions) the agent runs `look.py` and inspects the PNG. The report is not finished until its `Look:` line names that image; audio-only jobs say `Look: not needed`. **"Inspects" means the calling agent's own vision, not a feature of this skill:** `look.py` only renders a PNG; nothing in this repository detects faces, products, subjects, or "the interesting part" of a frame or a scene. When a crop or reframe needs to keep a specific part of the frame (`fit.py --fit crop --crop-x/-y`, see [Tools](#tools)), it is the multimodal agent looking at that PNG and choosing the anchor — a non-visual caller (a script, a CLI user without eyes on the sheet) has to supply that decision itself, and the default is a plain centre crop. Likewise `scenes.py --highlights` ranks candidate scenes by a measured proxy (`--rank-by audio` or `--rank-by duration`), never by content; it is the agent that turns a look at the sheet into a judgement.9. **Keep originals.** No tool overwrites its input. Outputs are new files named `<input>_<operation>.<ext>` unless told otherwise, and a test hashes every input after the run.## Tools@@ -127,7 +127,7 @@ These are the rules the skill file gives the agent and the code enforces. Togeth| Tool | What it does ||---|---|| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision, colour space, rotation, every audio stream; `--analyze` flags Log footage |-| `scenes.py` | Scene changes, audio peaks, highlight proposals and a per-scene sheet; cut list for `cut.py --segments` |+| `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments` || `look.py` | Contact sheet, single frames, side-by-side comparison as PNG so the agent can see what it made |**Editing**diff --git a/SKILL.md b/SKILL.mdindex 8755df0..db735ef 100644--- a/SKILL.md+++ b/SKILL.md@@ -186,6 +186,7 @@ Keep it to those five lines plus anything the user must decide. Attach the conta- `--fit crop` to reach 9:16 from 16:9 throws away 70 % of the width: a wide shot loses people at the edges. Check the sheet; pad (bars), `--crop-x`/`--crop-y` toward the subject, or a reframe is often the honest answer — a silent centre crop is a guess, not a decision.- Conforming 60 fps to 30 halves the motion samples: fine for a talking head, visibly choppy for sports, gaming, drone pans. Keep 60 when the platform allows it.- "Make it 60 seconds" on a 3-minute talk by speed change is unwatchable (3×); by trim it drops two thirds of the words. Ask which, or propose a highlight cut with `scenes.py`.+- `scenes.py --highlights` defaults to the loudest scenes (`--rank-by audio`): a quiet but important moment (a confession, a punchline landing in silence) is skipped, and pure crowd noise or a mic bump can outrank it. `--rank-by duration` picks the longest unbroken scenes instead. Neither is "the best parts" — check the contact sheet (`--sheet`) before treating the picks as final.## Gotchasdiff --git a/scripts/scenes.py b/scripts/scenes.pyindex 62111b4..27558a7 100755--- a/scripts/scenes.py+++ b/scripts/scenes.py@@ -3,14 +3,19 @@agent can plan an edit or a digest without watching the whole file.Scene cuts come from ffmpeg's scdet; energy peaks from a 0.5 s RMS envelope-of the audio. Highlight candidates are the scenes ranked by audio energy-(and, optionally, by motion).+of the audio. Highlight candidates are scenes ranked by --rank-by: "audio"+(default, loudest first) or "duration" (longest first). Both are proxies,+not a judgement of what matters: "audio" misses a quiet but important+moment (a confession, a punchline landing in silence) and can surface pure+crowd noise; "duration" just finds long unbroken takes. Neither replaces+watching the contact sheet (--sheet) before committing to a cut.Examples:python3 scenes.py talk.mp4 # scenes + peaks, JSONpython3 scenes.py event.mp4 --highlights 5 --target 60 # 5 candidate ranges summing to ~60 spython3 scenes.py event.mp4 --highlights 4 --edl picks.txt # cut.py --segments compatible listpython3 scenes.py event.mp4 --sheet scenes.png # one thumbnail per scene+ python3 scenes.py talk.mp4 --highlights 5 --rank-by duration # longest unbroken scenes, not loudest"""import argparseimport math@@ -96,6 +101,8 @@ def main() -> int:ap.add_argument("--ratio", type=float, default=3.0, help="a cut must exceed this multiple of the neighbouring frames' median score (default 3; lower = more cuts)")ap.add_argument("--min-scene", type=float, default=1.0, help="ignore cuts closer than this in seconds (default 1)")ap.add_argument("--highlights", type=int, default=0, help="number of highlight ranges to propose")+ ap.add_argument("--rank-by", choices=["audio", "duration"], default="audio",+ help="how to rank scenes for --highlights: audio energy (default) or scene duration")ap.add_argument("--target", type=float, help="with --highlights: total seconds the picks should add up to (trims long scenes)")ap.add_argument("--max-scene", type=float, default=15.0, help="cap a highlight range at this many seconds (default 15)")ap.add_argument("--edl", help="write highlight ranges as START-END lines (cut.py --segments format)")@@ -136,7 +143,11 @@ def main() -> int:info(f"{len(scenes)} scenes, {len(peaks)} audio peaks over {dur:.1f}s")if args.highlights:- ranked = sorted(scenes, key=lambda sc: (-sc["audio_rms"], sc["start"]))[: args.highlights]+ if args.rank_by == "duration":+ rank_key = lambda sc: (-sc["duration"], sc["start"])+ else:+ rank_key = lambda sc: (-sc["audio_rms"], sc["start"])+ ranked = sorted(scenes, key=rank_key)[: args.highlights]picks: List[Tuple[float, float]] = []budget = args.target if args.target else Noneper = (budget / max(1, len(ranked))) if budget else args.max_scene@@ -156,6 +167,7 @@ def main() -> int:picks.sort()result["highlights"] = [{"start": s, "end": e, "duration": round(e - s, 2)} for s, e in picks]result["highlights_total"] = round(sum(e - s for s, e in picks), 2)+ result["highlights_rank_by"] = args.rank_byinfo(f"proposed {len(picks)} highlight ranges totalling {result['highlights_total']:.1f}s")if args.edl:with open(args.edl, "w", encoding="utf-8") as fh:diff --git a/tests/test_all.py b/tests/test_all.pyindex f0c10e9..e47141b 100644--- a/tests/test_all.py+++ b/tests/test_all.py@@ -670,6 +670,24 @@ class FFmpegSkillTests(unittest.TestCase):script("cut.py", src, "--segments", segs, "--accurate", "--fast", "-o", out)self.assertClose(probe(str(out))["duration"], 6.0, 0.6)+ def test_scenes_rank_by_duration_picks_longest_scenes_not_loudest(self):+ src = OUT / "scenes_rank_src.mp4"+ # three scenes: 2s, 6s (longest, silent), 4s (loud tone) -- audio and duration ranking must disagree+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",+ "-f", "lavfi", "-i", "testsrc2=size=640x360:rate=30:d=2", "-f", "lavfi", "-i", "smptebars=size=640x360:rate=30:d=6",+ "-f", "lavfi", "-i", "mandelbrot=size=640x360:rate=30",+ "-f", "lavfi", "-i", "aevalsrc='0.6*sin(2*PI*440*t)*between(t\\,8\\,11)':s=48000",+ "-filter_complex", "[2:v]trim=0:4,setpts=PTS-STARTPTS[m];[0:v][1:v][m]concat=n=3:v=1:a=0[v]",+ "-map", "[v]", "-map", "3:a", "-t", "12", "-c:v", "libx264", "-preset", "veryfast", "-c:a", "aac", src)+ by_audio = json.loads(script("scenes.py", src, "--highlights", "1", "--json").stdout)+ by_duration = json.loads(script("scenes.py", src, "--highlights", "1", "--rank-by", "duration", "--json").stdout)+ self.assertEqual(by_audio["highlights_rank_by"], "audio")+ self.assertEqual(by_duration["highlights_rank_by"], "duration")+ # audio ranking picks the loud scene (8-11s window, inside scene 3 at 8-12s)+ self.assertTrue(any(h["start"] >= 7.5 for h in by_audio["highlights"]), by_audio["highlights"])+ # duration ranking picks the longest (silent) scene, which starts at 2s+ self.assertTrue(any(round(h["start"]) == 2 for h in by_duration["highlights"]), by_duration["highlights"])+def test_render_project(self):proj = OUT / "project.json"proj.write_text(json.dumps({