Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.
Python agent-skillsai-agentsaudio-processingcaptionsclaude-codecodexcoding-agentcursorffmpegloudness-normalizationmcpmcp-servermodel-context-protocolpythonsubtitlestranscodingvideovideo-editingvideo-processingwhisper
caption.py: --audio-stream for explicit multi-audio-track selection (#55) (#64)
Shogo Kajiyama03cc9c490d42
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3756c58..a9726de 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,7 @@## Unreleased+- **`caption.py --audio-stream N`: explicit multi-audio-track selection.** Confirmed `caption.py` did silently pick a track on a multi-audio-track input (dubbed languages, M&E stems): burn mode had no `-map` at all (ffmpeg's own automatic stream-selection heuristic, not necessarily index 0, decided), mux mode and the karaoke energy-timing/`--transcribe` audio extraction both hardcoded `0:a:0`. `--audio-stream N` (default 0, matching `audio.py`'s existing flag and unchanged prior behaviour) now threads the same explicit track index through all four: burn's re-encoded audio, mux's stream-copied audio, `--transcribe`'s speech-to-text source, and karaoke's energy-timing analysis, refusing an out-of-range index the same way `audio.py --audio-stream` already does. Closes [#55](https://github.com/kajisho5/ffmpeg-skill/issues/55).- **`caption.py --text`: SMPTE non-drop-frame timecode cues.** Cues were positioned by decimal seconds only; broadcast-style deliverables often supply cue timing as `hh:mm:ss:ff` frame timecode instead. `--text` cue lines now also accept that format (e.g. `00:00:03:15 --> 00:00:06:00 ...`) — the frame count is converted to seconds with `--fps`, or the input video's own probed fps when `--input` is given and `--fps` is not. A cue that is shaped like a timecode but has no fps available (no `--fps`, no `--input`) is refused with a clear error naming the missing `--fps`, rather than silently misread as a plain text line the way an ordinary unparseable cue line already is. `_common.py` gains the reusable pieces other tools can build on later: `parse_time()` takes an optional `fps` argument for the `hh:mm:ss:ff` case (raising the new `MissingFpsError`, a `ValueError` subclass, when fps is needed but absent), and `fmt_smpte_time()` formats seconds back to `hh:mm:ss:ff` — used here to echo the interpreted cue range in the `wrote ... .srt` report line so a caller can confirm the timecode was read correctly. The written `.srt` itself stays decimal-millisecond SRT timing, since that is the only timing SRT/mux subtitle codecs actually carry; nothing claims frame-exact precision it can't hold. Drop-frame (29.97/59.94 fps) counting is out of scope. Closes [#54](https://github.com/kajisho5/ffmpeg-skill/issues/54).- **`references/process-pitfalls.md`: development-process mistakes already made once.** Distinct from `references/ci-platform-pitfalls.md` (ffmpeg/CI behaviour differences): this is about the process of making a change safely, not FFmpeg itself. Three entries to start: narrowing a `TOOL_META[...]["required"]` capability list without first grepping `tests/test_contract.py`'s `DoctorDetectionTests` for the fixture-pinned `usable` outcome it protects (nearly broken twice, in #51 and #52, caught before landing both times); retrying a git tag push or GitHub Release creation in this environment, where both are scoped out (branch pushes work, tag pushes 403 at the git-receive-pack level; no `create_release`/`create_tag` MCP tool exists; the outbound proxy itself blocks a raw REST API call to the releases endpoint) rather than accepting it after one confirmation; and redesigning a test fixture a third time instead of recognising, after two independently-redesigned fixtures failed differently on the same platform, that the platform's real behaviour (not the fixture) is the actual cause (`stabilize.py`'s macOS libvidstab test). A living document — add to it whenever one of these recurs. Docs-only; no behaviour changed.- **`doctor --json`: `gpu_encoders`.** No tool here uses GPU-accelerated encoding — every tool assumes CPU x264/x265 — but `doctor` had no way to answer "is GPU encoding available on this machine" at all, unlike every other capability it already reports `yes`/`no`/`unknown` for. `gpu_encoders` reports GPU-backed encoders (`nvenc`, `videotoolbox`, `qsv`, `vaapi`, `amf`) present in this ffmpeg *build*, read from the same `-encoders` listing `doctor` already parses — `{"status": "parsed"|"unparsed"|"failed"|"missing", "present": [...]}`. Deliberately build-presence only: proving a real GPU/driver will accept a job would need an actual encode, which `doctor`'s introspection never runs (matching its existing 10s-timeout, listing-only philosophy). Purely informational — no tool declares or requires a GPU encoder, so this field never affects `ok` or any tool's `usable`. The human-readable `doctor` output gets one line naming what's present (or "none"). Closes [#52](https://github.com/kajisho5/ffmpeg-skill/issues/52).diff --git a/references/scripts.md b/references/scripts.mdindex 2dae950..5db083c 100644--- a/references/scripts.md+++ b/references/scripts.md@@ -283,7 +283,7 @@ frame like an editor would. Use `--compare` to show before/after to the user.### caption.py — subtitles (static, animated, karaoke)```caption.py INPUT --srt FILE | --ass FILE | --text CUES.txt [--write-srt OUT.srt]- [--mode burn|mux] [--fps N]+ [--mode burn|mux] [--audio-stream N] [--fps N][--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB][--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N][--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass] [-o OUT]@@ -306,6 +306,12 @@ takes only a plain SRT (`--srt`/`--text`/`--transcribe`, not `--ass`, sincestyling has no soft-subtitle equivalent) and no `--animate`/`--karaoke`. Thesubtitle codec follows the output container: `mov_text` for `.mp4`/`.m4v`/`.mov`,`srt` for `.mkv`, `webvtt` for `.webm`.+`--audio-stream N` (default 0, the first track) picks which audio stream of a+multi-track input (dubbed languages, M&E stems) is kept — applies to burn's+re-encoded audio, mux's stream-copied audio, `--transcribe`'s speech-to-text+source, and karaoke's energy-timing analysis alike, so all four agree on the+same track instead of each silently defaulting to whichever one ffmpeg's own+stream selection would have picked.### overlay.py — logo, image, title, video picture-in-picture, chroma key```diff --git a/scripts/caption.py b/scripts/caption.pyindex fb51c58..e3883d5 100755--- a/scripts/caption.py+++ b/scripts/caption.py@@ -75,7 +75,7 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float, fps: Optional[flreturn cues-def transcribe(video: str, out_srt: str, language: Optional[str], model: str) -> List[Tuple[float, float, str]]:+def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0) -> List[Tuple[float, float, str]]:"""Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.No engine installed -> clear error with install hints; the skill never depends on one."""@@ -86,7 +86,8 @@ def transcribe(video: str, out_srt: str, language: Optional[str], model: str) ->ffmpeg = require_tool("ffmpeg")tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")wav = os.path.join(tmpdir, "audio.wav")- subprocess.run([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav], check=True)+ subprocess.run([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,+ "-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav], check=True)# 1. whisper.cppcli = shutil.which("whisper-cli") or shutil.which("whisper-cpp") or shutil.which("main")if cli and (shutil.which("whisper-cli") or shutil.which("whisper-cpp")):@@ -167,7 +168,7 @@ def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")-def word_durations_from_audio(video: str, start: float, end: float, n_words: int) -> List[int]:+def word_durations_from_audio(video: str, start: float, end: float, n_words: int, audio_stream: int = 0) -> List[int]:"""Split a cue's time across n_words in proportion to speech energy (centiseconds each).Decodes the cue window to 8 kHz mono, builds a 10 ms RMS envelope, removes the noise floor,@@ -182,7 +183,7 @@ def word_durations_from_audio(video: str, start: float, end: float, n_words: intreturn [total_cs]ffmpeg = require_tool("ffmpeg")cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-ss", f"{start:.3f}", "-i", video,- "-t", f"{end - start:.3f}", "-vn", "-ac", "1", "-ar", "8000", "-f", "s16le", "-"]+ "-map", f"0:a:{audio_stream}", "-t", f"{end - start:.3f}", "-vn", "-ac", "1", "-ar", "8000", "-f", "s16le", "-"]proc = sp.run(cmd, stdout=sp.PIPE, stderr=sp.PIPE)n = len(proc.stdout) // 2if proc.returncode != 0 or n < 800:@@ -266,7 +267,7 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: intsegments = body.split("\\N")words = [w for seg in segments for w in seg.split(" ") if w]if getattr(args, "karaoke_timing", "even") == "energy" and video:- durs = word_durations_from_audio(video, start, end, len(words))+ durs = word_durations_from_audio(video, start, end, len(words), getattr(args, "audio_stream", 0))else:per = max(1, dur_cs // max(1, len(words)))durs = [per] * len(words)@@ -308,6 +309,10 @@ def main() -> int:ap.add_argument("--mode", choices=["burn", "mux"], default="burn",help="'burn' renders subtitles into the picture (default); ""'mux' copies video/audio untouched and adds the SRT as a soft, toggleable subtitle stream")+ ap.add_argument("--audio-stream", type=int, default=0,+ help="which audio stream of the input to keep, 0-based in file order (probe.py lists them under "+ "audio_streams) -- matters on a multi-track input (dubbed languages, M&E stems); default 0, "+ "the first track, same as leaving it unset always did")src = ap.add_argument_group("subtitle source")src.add_argument("--srt", help="SRT file to burn")src.add_argument("--ass", help="ASS file to burn (styles inside the file are used)")@@ -373,16 +378,26 @@ def main() -> int:if args.animate != "none" or args.karaoke:die("--animate/--karaoke render pixels into the picture and require --mode burn")+ meta = None+ if args.input:+ meta = probe(args.input)+ if not meta.get("video"):+ die("input has no video stream")+ audio_streams = meta.get("audio_streams") or []+ if audio_streams and not (0 <= args.audio_stream < len(audio_streams)):+ die(f"--audio-stream {args.audio_stream}: input has {len(audio_streams)} audio stream(s), 0..{len(audio_streams) - 1}")+ if args.audio_stream and not audio_streams:+ die("--audio-stream needs an input with audio streams")fps_for_tc = args.fps- if fps_for_tc is None and args.input:- fps_for_tc = probe(args.input).get("video", {}).get("fps")+ if fps_for_tc is None and meta is not None:+ fps_for_tc = meta.get("video", {}).get("fps")srt_path = args.srtif args.transcribe:if not args.input:die("--transcribe needs the input video")srt_path = args.write_srt or os.path.splitext(args.input)[0] + ".srt"- cues = transcribe(args.input, srt_path, args.language, args.model)+ cues = transcribe(args.input, srt_path, args.language, args.model, args.audio_stream)info(f"wrote {srt_path} ({len(cues)} cues)")args.text = Noneif args.text:@@ -405,9 +420,6 @@ def main() -> int:if not args.input:die("input video is required unless you only use --text/--write-srt")- meta = probe(args.input)- if not meta.get("video"):- die("input has no video stream")output = args.output or default_output(args.input, "captioned")@@ -418,7 +430,7 @@ def main() -> int:maps = ["-map", "0:v:0"]cmd = ffmpeg_base() + ["-i", args.input, "-i", srt_path]if meta.get("audio"):- maps += ["-map", "0:a:0"]+ maps += ["-map", f"0:a:{args.audio_stream}"]maps += ["-map", "1:0"]cmd += maps + ["-c:v", "copy"] + (["-c:a", "copy"] if meta.get("audio") else []) + ["-c:s", codec]if args.language:@@ -467,7 +479,10 @@ def main() -> int:if args.fonts_dir:vf += f":fontsdir={escape_filter_path(args.fonts_dir)}"- cmd = ffmpeg_base() + ["-i", args.input, "-vf", vf] + video_args(meta, args.crf, args.preset) + cfr_args(meta)+ cmd = ffmpeg_base() + ["-i", args.input, "-map", "0:v:0"]+ if meta.get("audio"):+ cmd += ["-map", f"0:a:{args.audio_stream}"]+ cmd += ["-vf", vf] + video_args(meta, args.crf, args.preset) + cfr_args(meta)cmd += (aac_args() if meta.get("audio") else ["-an"]) + [output]run(cmd)result = probe(output, role="output")diff --git a/tests/test_all.py b/tests/test_all.pyindex e303aac..b7a1173 100644--- a/tests/test_all.py+++ b/tests/test_all.py@@ -568,6 +568,38 @@ class FFmpegSkillTests(unittest.TestCase):script("caption.py", "--text", self.cues, "--write-srt", srt)script("caption.py", self.src, "--srt", srt, "--mode", "mux", "-o", OUT / "x.avi", expect_fail=True)+ def test_caption_audio_stream_selects_the_requested_track_not_always_the_first(self):+ two = OUT / "cap_two_streams.mkv"+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000", "-f", "lavfi", "-i", "sine=frequency=880:sample_rate=44100",+ "-t", "4", "-map", "0:v", "-map", "1:a", "-map", "2:a", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", two)+ self.assertEqual(len(probe(str(two))["audio_streams"]), 2)+ cue = OUT / "cap_two_cue.txt"+ cue.write_text("0:00-0:02 Hello\n", encoding="utf-8")+ # mux mode: the picked track survives a stream copy untouched, verified by its own probed sample rate+ out_mux = OUT / "cap_two_mux.mp4"+ script("caption.py", two, "--text", cue, "--audio-stream", "1", "--mode", "mux", "-o", out_mux, "--write-srt", OUT / "cap_two_mux.srt")+ self.assertEqual(probe(str(out_mux))["audio"]["sample_rate"], 44100, "mux must keep the requested track (index 1), not default to index 0")+ # burn mode: the picked track survives re-encoding to AAC too (sample rate normally changes on re-encode,+ # so compare against the same tool re-encoding the default track 0 instead)+ out_burn1 = OUT / "cap_two_burn1.mp4"+ script("caption.py", two, "--text", cue, "--audio-stream", "1", "-o", out_burn1, "--preset", "veryfast")+ out_burn0 = OUT / "cap_two_burn0.mp4"+ script("caption.py", two, "--text", cue, "-o", out_burn0, "--preset", "veryfast")+ # both re-encode to AAC, but decoding each and comparing peak frequency would be overkill here --+ # the -map argument itself is what this test protects, already proven correct in mux mode above;+ # this just confirms burn mode doesn't crash or silently drop audio when --audio-stream is given+ self.assertIsNotNone(probe(str(out_burn1))["audio"])+ self.assertIsNotNone(probe(str(out_burn0))["audio"])++ def test_caption_audio_stream_out_of_range_refused(self):+ cue = OUT / "cap_range_cue.txt"+ cue.write_text("0:00-0:02 Hello\n", encoding="utf-8")+ proc = script("caption.py", self.src, "--text", cue, "--audio-stream", "5", "-o", OUT / "x.mp4", "--json", expect_fail=True)+ doc = json.loads(proc.stdout)+ self.assertEqual(doc["error"]["kind"], "input")+ self.assertIn("audio-stream", doc["error"]["message"])+# ---------------------------------------------------------------- overlaydef test_overlay_image_and_text(self):out1 = OUT / "ov_img.mp4"