Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
fix(join): a plain cut leaves no hole where a clip's sound outruns its picture (#302)
Ryan Maliad6f3318f2825
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cc3764c..a803fae 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,7 +4,7 @@## Unreleased-(nothing yet)+- fix(join): a plain cut (`--transition none`) no longer leaves a hole in the video where a clip's sound runs past its picture. The concat filter starts each clip after the *longer* stream of the one before, so a music bed padded to the container left 0.3 s of missing video (an odd average rate downstream), and even a 20 ms AAC tail left a one-frame gap; the run succeeded with `verified` false. Every clip now gets the crossfade path's one length for both streams: the last frame held when the sound runs more than a frame past the picture, the sound trimmed when it runs less. `expected_duration` counts those lengths, not container durations.## 2.4.0diff --git a/docs/design-decisions.md b/docs/design-decisions.mdindex da94ae3..3a0865a 100644--- a/docs/design-decisions.md+++ b/docs/design-decisions.md@@ -814,3 +814,14 @@ not a new file format this tool would have to maintain.- **"Found no speech" is not "no engine".** An engine that ran and returned no cue is refusedas `kind: input`, `reason: "no_speech"`, naming the engine and the caller's input rather thanthe engine's deleted temporary SRT. Tests: `AsrNoSpeechTests`.++## Unreleased++- **A plain `join.py` cut gives each clip one length, as a crossfade does.** The concat filter+ starts the next clip after the *longer* stream of this one, so the picture of a clip whose sound+ runs past it -- a music bed padded to the container, or only an AAC tail -- got a hole (measured:+ 0.3 s, and one frame from a 20 ms tail). `clip_length` decides the length for both paths: the+ sound's when it runs more than a frame past the picture (the last frame is held; narration is never+ cut), the picture's otherwise (the tail is trimmed). Tests:+ `test_join_none_holds_a_picture_shorter_than_its_sound`,+ `test_join_none_trims_a_sound_tail_under_a_frame`.diff --git a/references/scripts.md b/references/scripts.mdindex 6a2af10..8928da1 100644--- a/references/scripts.md+++ b/references/scripts.md@@ -591,10 +591,10 @@ are always present (`[]` when none) and noted.Normalises every clip to one frame size, fps, `yuv420p`, 48 kHz and onechannel layout (the widest clip's -- a 5.1 clip keeps 5.1 -- or `--channels`;silent track generated for clips without audio), then chains `xfade` +-`acrossfade`. With a transition each clip has one length for picture and sound+`acrossfade` (or `concat` for `--transition none`). Every clip has one length for picture and sound-- the video stream's, or the audio's when it runs more than a frame longer --and the shorter stream is padded (silence / last frame held), so every clip's-sound stays with its picture. Output length = sum of clip lengths −+sound stays with its picture and a plain cut leaves no hole in the video. Output length = sum of clip lengths −transition × (n−1); `verified` checks the output's video-stream length against it. Clips must belonger than 2 × the transition. Use `--transition none` for a plain cut.A subtitle/data track in the source is not carried into the retimed/concatenateddiff --git a/scripts/join.py b/scripts/join.pyindex afb9bed..c44d472 100755--- a/scripts/join.py+++ b/scripts/join.py@@ -81,7 +81,7 @@ def unpending_length(durs: List[float], d: float) -> Optional[float]:def clip_length(meta: Dict[str, Any], fps: float) -> float:- """The one length a clip holds in a crossfaded join, for its picture and its sound alike: the+ """The one length a clip holds in a join, for its picture and its sound alike: thevideo stream's, or the audio stream's when the sound runs more than a frame past the picture(narration is never cut; an AAC tail adds no frame). The container duration (audio priming,a longer track) is only the fallback when a stream length is unknown (a pending input)."""@@ -443,9 +443,12 @@ def main() -> int:note_short_segments(args.inputs, metas, MIN_SEGMENT_FRAMES / fps, f"{MIN_SEGMENT_FRAMES} frames at {fps:g} fps")durs = [m.get("duration") or 0.0 for m in metas]d = args.duration if args.transition != "none" else 0.0- # a crossfaded join gives each clip one length for both streams, so the offsets that place- # every picture also place its sound; the plain cut (concat) keeps the streams paired itself- lens = [clip_length(m, fps) for m in metas] if d else durs+ # every clip gets one length for both streams, so whatever places the next clip -- xfade's+ # offsets, or the concat filter, which starts it after the LONGER stream of this one -- places+ # its picture and its sound together. The plain cut used to take the raw streams, and a sound+ # past the picture (a music bed padded to the container, even a 20 ms AAC tail) left a hole in+ # the video there.+ lens = [clip_length(m, fps) for m in metas]place_silent(lens, d)for p, dur in zip(args.inputs, lens):if d and dur <= d * 2 and not STATE.dry_run:@@ -497,18 +500,17 @@ def main() -> int:parts.append(f"[{i}:v]{geo},setsar=1,fps={fps:g},format={pixfmt},settb=AVTB[v{i}]")parts.append(f"[{audio_src[i]}]aformat=sample_rates=48000:channel_layouts={layout},asetpts=PTS-STARTPTS[a{i}]")+ # pad the shorter stream to the clip's length: the audio with silence, the picture by holding+ # its last frame, then cut both there -- each clip boundary is then the same for both chains+ for i in range(n):+ vlen = (metas[i].get("video") or {}).get("duration") or lens[i]+ hold = f"tpad=stop_mode=clone:stop_duration={lens[i] - vlen:.3f}," if lens[i] > vlen else ""+ parts.append(f"[v{i}]{hold}trim=duration={lens[i]:.3f}[vp{i}]")+ parts.append(f"[a{i}]apad=whole_dur={lens[i]:.3f},atrim=duration={lens[i]:.3f}[ap{i}]")if args.transition == "none":- chain = "".join(f"[v{i}][a{i}]" for i in range(n))+ chain = "".join(f"[vp{i}][ap{i}]" for i in range(n))parts.append(f"{chain}concat=n={n}:v=1:a=1[vout][aout]")else:- # pad the shorter stream to the clip's length: the audio with silence, the picture by- # holding its last frame, then cut both there -- each xfade and acrossfade then starts- # at the same clip boundary instead of the audio chain drifting by every clip's gap- for i in range(n):- vlen = (metas[i].get("video") or {}).get("duration") or lens[i]- hold = f"tpad=stop_mode=clone:stop_duration={lens[i] - vlen:.3f}," if lens[i] > vlen else ""- parts.append(f"[v{i}]{hold}trim=duration={lens[i]:.3f}[vp{i}]")- parts.append(f"[a{i}]apad=whole_dur={lens[i]:.3f},atrim=duration={lens[i]:.3f}[ap{i}]")vprev, aprev = "vp0", "ap0"offset = 0.0for i in range(1, n):@@ -529,11 +531,11 @@ def main() -> int:"by its measured length")output = args.output or default_output(args.inputs[0], "joined", "mp4")cmd += ["-filter_complex", ";".join(parts), "-map", "[vout]", "-map", "[aout]"]- if d:- # the fps filter already made the picture constant-rate and xfade places every frame: FFmpeg- # 5.1's default cfr output mode still duplicated the last one to meet the AAC tail (a 51-frame- # dissolve came out 52), which newer builds no longer do -- pass the frames through as built- cmd += ["-fps_mode", "passthrough"]+ # the fps filter already made the picture constant-rate, and the per-clip trims (then xfade or+ # concat) place every frame: FFmpeg 5.1's default cfr output mode still duplicated the last one+ # to meet the AAC tail (a 51-frame dissolve came out 52), which newer builds no longer do -- pass+ # the frames through as built, for a plain cut as much as for a transition+ cmd += ["-fps_mode", "passthrough"]cmd += video_args(hdr_meta or metas[0], args.crf, args.preset) + aac_args() + [output]run(cmd)expected = unpending_length(lens, d)diff --git a/tests/test_editing.py b/tests/test_editing.pyindex 5e8a794..9bc1f37 100644--- a/tests/test_editing.py+++ b/tests/test_editing.py@@ -7,6 +7,7 @@import osimport platformimport json+import mathimport reimport shutilimport subprocess@@ -861,6 +862,74 @@ class EditingTests(MediaFixtures):self.assertClose(probe(str(out))["video"]["duration"], 4.5, 0.05)self.assertClose(self._tone_onset(out, 1900), 2.5, 0.1)+ def _moving_clip(self, out, picture, tone, freq, fps=30):+ """Like _av_clip, but every pixel changes on every frame (a luma ramp stepped by the frame+ number), so a held frame is told apart from the next one by its content, not assumed.+ testsrc2 is not enough at this size: its moving parts are a few pixels, and two adjacent+ frames measured 65 dB apart."""+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i",+ f"nullsrc=s=64x48:r={fps}:d={picture},geq=lum='mod(N*23+X*2,256)':cb=128:cr=128",+ "-f", "lavfi", "-i", f"sine=f={freq}:d={tone}", "-c:v", "libx264", "-preset", "ultrafast", "-c:a", "aac", out)+ return out++ @staticmethod+ def _gray_frames(path):+ """Every frame of `path`, decoded to 64x48 luma bytes."""+ raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(path), "-fps_mode", "passthrough", "-vf",+ "scale=64:48,format=gray", "-f", "rawvideo", "-"], stdout=subprocess.PIPE, check=True).stdout+ return [raw[i:i + 64 * 48] for i in range(0, len(raw), 64 * 48)]++ @staticmethod+ def _frame_psnr(a, b):+ """PSNR of two decoded luma frames (bytes), 99 when identical. MediaFixtures._psnr compares files."""+ mse = sum((x - y) ** 2 for x, y in zip(a, b)) / len(a)+ return 99.0 if mse == 0 else 10 * math.log10(255 * 255 / mse)++ @staticmethod+ def _pts_steps(path):+ out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "frame=pts_time",+ "-of", "csv=p=0", str(path)], stdout=subprocess.PIPE, text=True, check=True).stdout+ pts = [float(x.strip(",")) for x in out.split()]+ return {round(b - a, 4) for a, b in zip(pts, pts[1:])}++ def test_join_none_holds_a_picture_shorter_than_its_sound(self):+ """A plain cut (--transition none) of a clip whose sound runs 0.3 s past its+ picture. The concat filter starts the next clip after the longer stream, so the video had+ a 0.3 s hole (an odd average rate downstream) and the run still succeeded. Each clip now+ lasts as long as its sound, its last frame held, as the crossfade path already does."""+ d = OUT / "join_none_hold"+ d.mkdir(exist_ok=True)+ a = self._moving_clip(d / "a.mp4", 3, 3.3, 700)+ b = self._moving_clip(d / "b.mp4", 3, 3.3, 1900)+ out = d / "held.mp4"+ doc = json.loads(script("join.py", a, b, "--transition", "none", "--preset", "ultrafast", "--json",+ "-o", out).stdout)+ self.assertEqual(doc["expected_duration"], 6.6, "two clips of 3.3 s: each lasts as long as its sound")+ self.assertTrue(doc["verified"], doc["verification"])+ self.assertEqual(self._pts_steps(out), {0.0333}, "no hole in the video timeline")+ frames = self._gray_frames(out)+ self.assertEqual(len(frames), 2 * 99, "3.3 s per clip at 30 fps")+ for i in range(90, 99):+ self.assertGreaterEqual(self._frame_psnr(frames[i], frames[89]), 45.0, f"frame {i} holds clip a's last frame")+ self.assertLess(self._frame_psnr(frames[89], frames[88]), 35.0, "the premise: the picture moves every frame")+ self.assertLess(self._frame_psnr(frames[99], frames[89]), 35.0, "clip b starts with its own picture")+ self.assertClose(self._tone_onset(out, 1900), 3.3, 0.1)++ def test_join_none_trims_a_sound_tail_under_a_frame(self):+ """The other side of the hold rule: an AAC tail of 20 ms is not held for -- each clip keeps its+ 90 frames and nothing is inserted between them."""+ d = OUT / "join_none_tail"+ d.mkdir(exist_ok=True)+ a = self._moving_clip(d / "a.mp4", 3, 3.02, 700)+ b = self._moving_clip(d / "b.mp4", 3, 3.02, 1900)+ out = d / "tail.mp4"+ doc = json.loads(script("join.py", a, b, "--transition", "none", "--preset", "ultrafast", "--json",+ "-o", out).stdout)+ self.assertEqual(doc["expected_duration"], 6.0, "the 20 ms tails are trimmed, not counted")+ self.assertTrue(doc["verified"], doc["verification"])+ self.assertEqual(self._pts_steps(out), {0.0333})+ self.assertEqual(len(self._gray_frames(out)), 2 * 90)+def test_join_dissolve_of_short_parts_keeps_the_frame_count(self):"""Two 1 s parts at 30 fps whose AAC sound runs 20 ms past the picture (a cut part) dissolvedover 0.3 s: 30 + 30 - 9 = 51 frames. Offsetting by the container length added frames and