Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

caption.py: --audio-stream for explicit multi-audio-track selection (#55) (#64)

Shogo Kajiyama03cc9c490d42
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3756c58..a9726de 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
## Unreleased
+- **`caption.py --audio-stream N`: explicit multi-audio-track selection.** Confirmed `caption.py` did silently pick a track on a multi-audio-track input (dubbed languages, M&E stems): burn mode had no `-map` at all (ffmpeg's own automatic stream-selection heuristic, not necessarily index 0, decided), mux mode and the karaoke energy-timing/`--transcribe` audio extraction both hardcoded `0:a:0`. `--audio-stream N` (default 0, matching `audio.py`'s existing flag and unchanged prior behaviour) now threads the same explicit track index through all four: burn's re-encoded audio, mux's stream-copied audio, `--transcribe`'s speech-to-text source, and karaoke's energy-timing analysis, refusing an out-of-range index the same way `audio.py --audio-stream` already does. Closes [#55](https://github.com/kajisho5/ffmpeg-skill/issues/55).
- **`caption.py --text`: SMPTE non-drop-frame timecode cues.** Cues were positioned by decimal seconds only; broadcast-style deliverables often supply cue timing as `hh:mm:ss:ff` frame timecode instead. `--text` cue lines now also accept that format (e.g. `00:00:03:15 --> 00:00:06:00 ...`) — the frame count is converted to seconds with `--fps`, or the input video's own probed fps when `--input` is given and `--fps` is not. A cue that is shaped like a timecode but has no fps available (no `--fps`, no `--input`) is refused with a clear error naming the missing `--fps`, rather than silently misread as a plain text line the way an ordinary unparseable cue line already is. `_common.py` gains the reusable pieces other tools can build on later: `parse_time()` takes an optional `fps` argument for the `hh:mm:ss:ff` case (raising the new `MissingFpsError`, a `ValueError` subclass, when fps is needed but absent), and `fmt_smpte_time()` formats seconds back to `hh:mm:ss:ff` — used here to echo the interpreted cue range in the `wrote ... .srt` report line so a caller can confirm the timecode was read correctly. The written `.srt` itself stays decimal-millisecond SRT timing, since that is the only timing SRT/mux subtitle codecs actually carry; nothing claims frame-exact precision it can't hold. Drop-frame (29.97/59.94 fps) counting is out of scope. Closes [#54](https://github.com/kajisho5/ffmpeg-skill/issues/54).
- **`references/process-pitfalls.md`: development-process mistakes already made once.** Distinct from `references/ci-platform-pitfalls.md` (ffmpeg/CI behaviour differences): this is about the process of making a change safely, not FFmpeg itself. Three entries to start: narrowing a `TOOL_META[...]["required"]` capability list without first grepping `tests/test_contract.py`'s `DoctorDetectionTests` for the fixture-pinned `usable` outcome it protects (nearly broken twice, in #51 and #52, caught before landing both times); retrying a git tag push or GitHub Release creation in this environment, where both are scoped out (branch pushes work, tag pushes 403 at the git-receive-pack level; no `create_release`/`create_tag` MCP tool exists; the outbound proxy itself blocks a raw REST API call to the releases endpoint) rather than accepting it after one confirmation; and redesigning a test fixture a third time instead of recognising, after two independently-redesigned fixtures failed differently on the same platform, that the platform's real behaviour (not the fixture) is the actual cause (`stabilize.py`'s macOS libvidstab test). A living document — add to it whenever one of these recurs. Docs-only; no behaviour changed.
- **`doctor --json`: `gpu_encoders`.** No tool here uses GPU-accelerated encoding — every tool assumes CPU x264/x265 — but `doctor` had no way to answer "is GPU encoding available on this machine" at all, unlike every other capability it already reports `yes`/`no`/`unknown` for. `gpu_encoders` reports GPU-backed encoders (`nvenc`, `videotoolbox`, `qsv`, `vaapi`, `amf`) present in this ffmpeg *build*, read from the same `-encoders` listing `doctor` already parses — `{"status": "parsed"|"unparsed"|"failed"|"missing", "present": [...]}`. Deliberately build-presence only: proving a real GPU/driver will accept a job would need an actual encode, which `doctor`'s introspection never runs (matching its existing 10s-timeout, listing-only philosophy). Purely informational — no tool declares or requires a GPU encoder, so this field never affects `ok` or any tool's `usable`. The human-readable `doctor` output gets one line naming what's present (or "none"). Closes [#52](https://github.com/kajisho5/ffmpeg-skill/issues/52).
diff --git a/references/scripts.md b/references/scripts.md
index 2dae950..5db083c 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -283,7 +283,7 @@ frame like an editor would. Use `--compare` to show before/after to the user.
### caption.py — subtitles (static, animated, karaoke)
```
caption.py INPUT --srt FILE | --ass FILE | --text CUES.txt [--write-srt OUT.srt]
- [--mode burn|mux] [--fps N]
+ [--mode burn|mux] [--audio-stream N] [--fps N]
[--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB]
[--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N]
[--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass] [-o OUT]
@@ -306,6 +306,12 @@ takes only a plain SRT (`--srt`/`--text`/`--transcribe`, not `--ass`, since
styling has no soft-subtitle equivalent) and no `--animate`/`--karaoke`. The
subtitle codec follows the output container: `mov_text` for `.mp4`/`.m4v`/`.mov`,
`srt` for `.mkv`, `webvtt` for `.webm`.
+`--audio-stream N` (default 0, the first track) picks which audio stream of a
+multi-track input (dubbed languages, M&E stems) is kept — applies to burn's
+re-encoded audio, mux's stream-copied audio, `--transcribe`'s speech-to-text
+source, and karaoke's energy-timing analysis alike, so all four agree on the
+same track instead of each silently defaulting to whichever one ffmpeg's own
+stream selection would have picked.
### overlay.py — logo, image, title, video picture-in-picture, chroma key
```
diff --git a/scripts/caption.py b/scripts/caption.py
index fb51c58..e3883d5 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -75,7 +75,7 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float, fps: Optional[fl
return cues
-def transcribe(video: str, out_srt: str, language: Optional[str], model: str) -> List[Tuple[float, float, str]]:
+def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0) -> List[Tuple[float, float, str]]:
"""Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),
whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.
No engine installed -> clear error with install hints; the skill never depends on one."""
@@ -86,7 +86,8 @@ def transcribe(video: str, out_srt: str, language: Optional[str], model: str) ->
ffmpeg = require_tool("ffmpeg")
tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")
wav = os.path.join(tmpdir, "audio.wav")
- subprocess.run([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav], check=True)
+ subprocess.run([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,
+ "-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav], check=True)
# 1. whisper.cpp
cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp") or shutil.which("main")
if cli and (shutil.which("whisper-cli") or shutil.which("whisper-cpp")):
@@ -167,7 +168,7 @@ def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
-def word_durations_from_audio(video: str, start: float, end: float, n_words: int) -> List[int]:
+def word_durations_from_audio(video: str, start: float, end: float, n_words: int, audio_stream: int = 0) -> List[int]:
"""Split a cue's time across n_words in proportion to speech energy (centiseconds each).
Decodes the cue window to 8 kHz mono, builds a 10 ms RMS envelope, removes the noise floor,
@@ -182,7 +183,7 @@ def word_durations_from_audio(video: str, start: float, end: float, n_words: int
return [total_cs]
ffmpeg = require_tool("ffmpeg")
cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-ss", f"{start:.3f}", "-i", video,
- "-t", f"{end - start:.3f}", "-vn", "-ac", "1", "-ar", "8000", "-f", "s16le", "-"]
+ "-map", f"0:a:{audio_stream}", "-t", f"{end - start:.3f}", "-vn", "-ac", "1", "-ar", "8000", "-f", "s16le", "-"]
proc = sp.run(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
n = len(proc.stdout) // 2
if proc.returncode != 0 or n < 800:
@@ -266,7 +267,7 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
segments = body.split("\\N")
words = [w for seg in segments for w in seg.split(" ") if w]
if getattr(args, "karaoke_timing", "even") == "energy" and video:
- durs = word_durations_from_audio(video, start, end, len(words))
+ durs = word_durations_from_audio(video, start, end, len(words), getattr(args, "audio_stream", 0))
else:
per = max(1, dur_cs // max(1, len(words)))
durs = [per] * len(words)
@@ -308,6 +309,10 @@ def main() -> int:
ap.add_argument("--mode", choices=["burn", "mux"], default="burn",
help="'burn' renders subtitles into the picture (default); "
"'mux' copies video/audio untouched and adds the SRT as a soft, toggleable subtitle stream")
+ ap.add_argument("--audio-stream", type=int, default=0,
+ help="which audio stream of the input to keep, 0-based in file order (probe.py lists them under "
+ "audio_streams) -- matters on a multi-track input (dubbed languages, M&E stems); default 0, "
+ "the first track, same as leaving it unset always did")
src = ap.add_argument_group("subtitle source")
src.add_argument("--srt", help="SRT file to burn")
src.add_argument("--ass", help="ASS file to burn (styles inside the file are used)")
@@ -373,16 +378,26 @@ def main() -> int:
if args.animate != "none" or args.karaoke:
die("--animate/--karaoke render pixels into the picture and require --mode burn")
+ meta = None
+ if args.input:
+ meta = probe(args.input)
+ if not meta.get("video"):
+ die("input has no video stream")
+ audio_streams = meta.get("audio_streams") or []
+ if audio_streams and not (0 <= args.audio_stream < len(audio_streams)):
+ die(f"--audio-stream {args.audio_stream}: input has {len(audio_streams)} audio stream(s), 0..{len(audio_streams) - 1}")
+ if args.audio_stream and not audio_streams:
+ die("--audio-stream needs an input with audio streams")
fps_for_tc = args.fps
- if fps_for_tc is None and args.input:
- fps_for_tc = probe(args.input).get("video", {}).get("fps")
+ if fps_for_tc is None and meta is not None:
+ fps_for_tc = meta.get("video", {}).get("fps")
srt_path = args.srt
if args.transcribe:
if not args.input:
die("--transcribe needs the input video")
srt_path = args.write_srt or os.path.splitext(args.input)[0] + ".srt"
- cues = transcribe(args.input, srt_path, args.language, args.model)
+ cues = transcribe(args.input, srt_path, args.language, args.model, args.audio_stream)
info(f"wrote {srt_path} ({len(cues)} cues)")
args.text = None
if args.text:
@@ -405,9 +420,6 @@ def main() -> int:
if not args.input:
die("input video is required unless you only use --text/--write-srt")
- meta = probe(args.input)
- if not meta.get("video"):
- die("input has no video stream")
output = args.output or default_output(args.input, "captioned")
@@ -418,7 +430,7 @@ def main() -> int:
maps = ["-map", "0:v:0"]
cmd = ffmpeg_base() + ["-i", args.input, "-i", srt_path]
if meta.get("audio"):
- maps += ["-map", "0:a:0"]
+ maps += ["-map", f"0:a:{args.audio_stream}"]
maps += ["-map", "1:0"]
cmd += maps + ["-c:v", "copy"] + (["-c:a", "copy"] if meta.get("audio") else []) + ["-c:s", codec]
if args.language:
@@ -467,7 +479,10 @@ def main() -> int:
if args.fonts_dir:
vf += f":fontsdir={escape_filter_path(args.fonts_dir)}"
- cmd = ffmpeg_base() + ["-i", args.input, "-vf", vf] + video_args(meta, args.crf, args.preset) + cfr_args(meta)
+ cmd = ffmpeg_base() + ["-i", args.input, "-map", "0:v:0"]
+ if meta.get("audio"):
+ cmd += ["-map", f"0:a:{args.audio_stream}"]
+ cmd += ["-vf", vf] + video_args(meta, args.crf, args.preset) + cfr_args(meta)
cmd += (aac_args() if meta.get("audio") else ["-an"]) + [output]
run(cmd)
result = probe(output, role="output")
diff --git a/tests/test_all.py b/tests/test_all.py
index e303aac..b7a1173 100644
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -568,6 +568,38 @@ class FFmpegSkillTests(unittest.TestCase):
script("caption.py", "--text", self.cues, "--write-srt", srt)
script("caption.py", self.src, "--srt", srt, "--mode", "mux", "-o", OUT / "x.avi", expect_fail=True)
+ def test_caption_audio_stream_selects_the_requested_track_not_always_the_first(self):
+ two = OUT / "cap_two_streams.mkv"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",
+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000", "-f", "lavfi", "-i", "sine=frequency=880:sample_rate=44100",
+ "-t", "4", "-map", "0:v", "-map", "1:a", "-map", "2:a", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", two)
+ self.assertEqual(len(probe(str(two))["audio_streams"]), 2)
+ cue = OUT / "cap_two_cue.txt"
+ cue.write_text("0:00-0:02 Hello\n", encoding="utf-8")
+ # mux mode: the picked track survives a stream copy untouched, verified by its own probed sample rate
+ out_mux = OUT / "cap_two_mux.mp4"
+ script("caption.py", two, "--text", cue, "--audio-stream", "1", "--mode", "mux", "-o", out_mux, "--write-srt", OUT / "cap_two_mux.srt")
+ self.assertEqual(probe(str(out_mux))["audio"]["sample_rate"], 44100, "mux must keep the requested track (index 1), not default to index 0")
+ # burn mode: the picked track survives re-encoding to AAC too (sample rate normally changes on re-encode,
+ # so compare against the same tool re-encoding the default track 0 instead)
+ out_burn1 = OUT / "cap_two_burn1.mp4"
+ script("caption.py", two, "--text", cue, "--audio-stream", "1", "-o", out_burn1, "--preset", "veryfast")
+ out_burn0 = OUT / "cap_two_burn0.mp4"
+ script("caption.py", two, "--text", cue, "-o", out_burn0, "--preset", "veryfast")
+ # both re-encode to AAC, but decoding each and comparing peak frequency would be overkill here --
+ # the -map argument itself is what this test protects, already proven correct in mux mode above;
+ # this just confirms burn mode doesn't crash or silently drop audio when --audio-stream is given
+ self.assertIsNotNone(probe(str(out_burn1))["audio"])
+ self.assertIsNotNone(probe(str(out_burn0))["audio"])
+
+ def test_caption_audio_stream_out_of_range_refused(self):
+ cue = OUT / "cap_range_cue.txt"
+ cue.write_text("0:00-0:02 Hello\n", encoding="utf-8")
+ proc = script("caption.py", self.src, "--text", cue, "--audio-stream", "5", "-o", OUT / "x.mp4", "--json", expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("audio-stream", doc["error"]["message"])
+
# ---------------------------------------------------------------- overlay
def test_overlay_image_and_text(self):
out1 = OUT / "ov_img.mp4"