Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

Release 0.2.0: colour, audio post, drift sync, animated captions, VFR handling (#2)

kajisho52d03cb18c0e5
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..e5074ef
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.2.0
+
+- `color.py` (new): HDR10/HLG → SDR BT.709 tone mapping (zscale + tonemap), 3D `.cube` LUTs with strength blending, metadata-only colour retagging.
+- `audio.py` (new): voice clean-up chain, FFT denoise, music bed with sidechain ducking, fades, 5.1 → stereo downmix, mono/stereo layout, track replacement.
+- `sync.py`: coarse-to-fine search (20 ms FFT → 1 ms direct) and `--fix-drift` clock-drift measurement and correction by resampling; roughly 5x faster on the default window.
+- `caption.py`: `--animate fade|pop|slide` and `--karaoke` word-by-word highlight, generated as a styled ASS sized to the video; SRT input can be animated too.
+- `fit.py`: `--smooth blend|interpolate` for slow motion.
+- VFR sources are conformed to constant frame rate automatically by every re-encoding script; `cut.py` switches to accurate mode on VFR.
+- `probe.py`: `hdr`, `hdr_format`, `bit_depth` fields.
+- `export.py`: warns when an HDR source is exported without tone mapping.
+- Tests now cover VFR, rotated, 5.1, 10-bit HDR10 HEVC and drifting sources.
+
+## 0.1.0
+
+- Initial release: probe, cut, caption, fit, sync, loudness, overlay, export; npx installer; demo and tests.
diff --git a/README.md b/README.md
index 21fa783..e86d075 100644
--- a/README.md
+++ b/README.md
@@ -17,9 +17,12 @@ npx ffmpeg-skill
- **Probe first, verify last** — the skill forces the agent to read real duration/fps/resolution before editing and to check the result after, so you get "final.mp4: 59.98 s, 1080×1920, 30 fps" instead of guesses.
- **Lossless when possible** — cuts and joins use stream copy by default; re-encoding only happens when it must (frame-accurate cuts, filters, format changes).
- **Cut & join** segments with `mm:ss` / `hh:mm:ss.ms` times.
-- **Captions** — burn SRT/ASS with font, size, colour, outline and position control; generate SRT from a plain timed-text file.
-- **Fit** to an exact duration (pitch-preserving speed change or trim) and to 16:9 / 9:16 / 1:1 / 4:5 by padding or cropping.
-- **Multicam / external-audio sync** — offset detection by cross-correlation implemented in pure Python (no numpy).
+- **Captions** — burn SRT/ASS with font, size, colour, outline and position control; generate SRT from a plain timed-text file; animated (fade/pop/slide) and word-by-word karaoke highlight styles for short-form video.
+- **Fit** to an exact duration (pitch-preserving speed change or trim) and to 16:9 / 9:16 / 1:1 / 4:5 by padding or cropping; motion-interpolated or blended slow motion.
+- **Real-world footage handling** — variable-frame-rate phone clips are conformed to constant fps automatically, rotation metadata is honoured, 10-bit HEVC and 5.1 sources are handled.
+- **Multicam / external-audio sync** — offset detection by cross-correlation implemented in pure Python (no numpy), 1 ms resolution, plus clock-drift correction for long takes.
+- **Colour management** — real HDR10/HLG → SDR BT.709 tone mapping, 3D LUT (.cube) for Log footage and looks, and metadata-only retagging.
+- **Audio post** — voice clean-up chain (highpass, de-esser, FFT denoise, compressor), background music with sidechain ducking, fades, 5.1 → stereo downmix, track replacement.
- **Loudness** — two-pass EBU R128 normalisation to −14 LUFS (or any target) with true-peak ceiling.
- **Overlays** — logos, watermarks and titles with position, time range, opacity and fades.
- **Export presets** — YouTube, Instagram Reels/Shorts/TikTok, X, ProRes 422 HQ master, H.265, GIF — all tagged BT.709.
@@ -59,14 +62,14 @@ Once installed, just talk to your agent. Five things you can say to Claude Code:
1. **"Take `interview.mp4`, keep 0:45–3:10 and 5:00–6:30, and make it exactly 60 seconds for Reels."**
→ `probe.py` → `cut.py --segments 0:45-3:10,5:00-6:30` → `fit.py --duration 60 --aspect 9:16 --fit crop` → `export.py --preset reels` → `probe.py` to confirm 60.0 s at 1080×1920.
-2. **"Burn these captions in, white text with a black outline at the top, in Japanese."**
- → `caption.py --text cues.txt --font "Noto Sans CJK JP" --position top --outline 3`.
-3. **"The lav mic recording is out of sync with the camera — fix it and normalise to −14 LUFS."**
- → `sync.py camera.mp4 lav.wav --replace-audio` → `loudness.py` → report the detected offset and final LUFS.
+2. **"Burn these captions in TikTok style, words popping in with a yellow highlight, in Japanese."**
+ → `caption.py --text cues.txt --font "Noto Sans CJK JP" --animate pop --karaoke --highlight-color FFD200`.
+3. **"The lav mic recording is out of sync with the camera and drifts over the hour — fix it, clean up the hiss and normalise to −14 LUFS."**
+ → `sync.py camera.mp4 lav.wav --fix-drift --replace-audio` → `audio.py --voice` → `loudness.py` → report the detected offset, drift ppm and final LUFS.
4. **"Put our logo in the top-right corner for the whole video at 80% opacity, and a title card for the first 4 seconds."**
→ `overlay.py --image logo.png --position top-right --scale 220 --opacity 0.8` → `overlay.py --text "…" --start 0 --end 4 --fade 0.4`.
-5. **"Give me a ProRes master and an H.265 copy of the final cut."**
- → `export.py --preset prores` and `export.py --preset h265`.
+5. **"This iPhone HDR clip looks washed out on YouTube — fix it and give me a ProRes master too."**
+ → `probe.py` (shows `hdr: true`) → `color.py --to-sdr` → `export.py --preset youtube` and `export.py --preset prores`.
The scripts also work on their own:
@@ -81,11 +84,13 @@ More examples: [examples/README.md](examples/README.md). To see everything run e
| Script | What it does |
|--------|--------------|
-| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, colour space, audio channels as JSON |
+| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format, colour space, rotation, audio channels as JSON |
| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact |
-| `caption.py` | Burn SRT/ASS (font, size, colour, outline, position); build SRT from timed plain text |
-| `fit.py` | Fit to a duration (speed or trim) and/or aspect ratio (pad or crop), force constant fps |
-| `sync.py` | Detect offset between two recordings by audio cross-correlation; output aligned video/audio |
+| `caption.py` | Burn SRT/ASS (font, size, colour, outline, position); build SRT from timed plain text; animated + karaoke ASS |
+| `fit.py` | Fit to a duration (speed or trim, smooth slow-mo) and/or aspect ratio (pad or crop), force constant fps |
+| `sync.py` | Detect offset between two recordings by audio cross-correlation (1 ms), correct clock drift; output aligned video/audio |
+| `color.py` | HDR10/HLG → SDR BT.709 tone mapping, 3D LUT application, colour-tag rewriting |
+| `audio.py` | Denoise / voice chain, music bed with auto-ducking, fades, downmix, replace track |
| `loudness.py` | Two-pass EBU R128 `loudnorm` to −14 LUFS / −1 dBTP (or custom), video stream-copied |
| `overlay.py` | Composite image/logo or drawtext title with position, time range, opacity, fade |
| `export.py` | Presets: `youtube`, `youtube4k`, `reels`, `x`, `prores`, `h265`, `gif` |
@@ -94,7 +99,7 @@ All scripts: Python 3.9+, standard library only, `--help`, non-zero exit + stder
## Requirements
-- FFmpeg 5.0+ with `libx264`, `libx265`, `libass` and `prores_ks` (the default builds from Homebrew, apt and gyan.dev include all of them)
+- FFmpeg 5.0+ with `libx264`, `libx265`, `libass`, `prores_ks` and `libzimg` (for `color.py --to-sdr`); the default builds from Homebrew, apt and gyan.dev include all of them
- Python 3.9+
- Node 16+ only for the `npx` installer
@@ -102,7 +107,7 @@ All scripts: Python 3.9+, standard library only, `--help`, non-zero exit + stder
```bash
bash examples/make_demo.sh # generates footage, runs every script, rebuilds assets/demo.gif
-python3 tests/test_all.py # end-to-end tests (needs ffmpeg)
+python3 tests/test_all.py # end-to-end tests incl. VFR, rotated, 5.1, 10-bit HDR10 and drifting sources (needs ffmpeg)
node bin/install.js --dir /tmp/skills # try the installer without touching ~/.claude
```
diff --git a/SKILL.md b/SKILL.md
index ea10807..f39cfad 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -1,6 +1,6 @@
---
name: ffmpeg-skill
-description: Professional video editing with local FFmpeg — cut, caption, fit to duration/aspect, sync multicam audio, normalise loudness, overlay logos and export platform presets, all from Python stdlib scripts with no cloud or API keys.
+description: Professional video editing with local FFmpeg — cut, caption (animated/karaoke), fit to duration/aspect, sync multicam audio with drift correction, HDR-to-SDR and LUT colour, denoise/duck/mix audio, normalise loudness, overlay logos and export platform presets, all from Python stdlib scripts with no cloud or API keys.
---
# ffmpeg-skill
@@ -23,8 +23,8 @@ path on stdout, and defaults the output name to `<input>_<operation>.<ext>`.
(plain cuts on keyframes, remuxing, audio-only changes), do not re-encode.
`cut.py` and `loudness.py` stream-copy video by default; only pass
`--accurate` to `cut.py` when the user needs frame-exact cuts.
-3. **Chain operations in a sensible order.** Cut → fit → caption/overlay →
- sync/loudness → export. Do the destructive/aspect changes before burning
+3. **Chain operations in a sensible order.** Colour (HDR→SDR / LUT) → cut →
+ fit → caption/overlay → sync → audio → loudness → export. Do the destructive/aspect changes before burning
text so captions are sized for the final frame. Re-encode as few times as
possible: if several re-encoding steps are needed, keep intermediates at
CRF 18 (the default) and only use `export.py` for the last step.
@@ -51,6 +51,17 @@ path on stdout, and defaults the output name to `<input>_<operation>.<ext>`.
| "fix the audio levels", "normalise to -14 LUFS" | `loudness.py input.mp4` (`-I -16 --tp -1.5` for podcasts, `-I -23` for broadcast) |
| "export for YouTube / Reels / X", "give me a ProRes master", "make it HEVC" | `export.py input.mp4 --preset youtube|reels|x|prores|h265` |
| "make a GIF preview" | `export.py input.mp4 --preset gif` |
+| "the colours look washed out / it's an iPhone HDR video" | `color.py input.mov --to-sdr` (probe shows `hdr: true`) |
+| "apply this LUT", "convert the S-Log / V-Log footage" | `color.py input.mp4 --lut grade.cube [--lut-strength 0.7]` |
+| "the colours are tagged wrong" | `color.py input.mp4 --retag bt709` (no re-encode) |
+| "clean up the audio", "remove the hiss / room noise" | `audio.py input.mp4 --voice` (speech) or `--denoise` |
+| "add background music under the talking" | `audio.py input.mp4 --music bed.mp3 --duck --fade-out 3` |
+| "convert the 5.1 to stereo" | `audio.py input.mov --downmix` |
+| "swap in the narration track" | `audio.py input.mp4 --replace narration.wav` |
+| "the audio drifts out of sync over the hour" | `sync.py camera.mp4 recorder.wav --fix-drift --replace-audio` |
+| "smooth slow motion", "half speed but fluid" | `fit.py input.mp4 --duration 2x --smooth interpolate` (slow) or `--smooth blend` |
+| "TikTok-style captions with the words popping / highlighted" | `caption.py input.mp4 --text cues.txt --animate pop --karaoke` |
+| "it's a phone video with variable frame rate" | nothing extra: every re-encoding script conforms VFR to constant fps automatically; `fit.py --fps 30` to pick the rate |
## Scripts
@@ -79,18 +90,27 @@ fit.py INPUT [--duration T --method speed|trim [--from-center] [--max-speed 4]]
[--fps N] [-o OUT]
```
`speed` retimes video and audio together (pitch-preserving `atempo`); it
-refuses factors beyond `--max-speed`. `trim` keeps the head (or the middle with
-`--from-center`). `--fps` forces constant frame rate — use it on VFR sources.
+refuses factors beyond `--max-speed`. For slow motion add `--smooth blend`
+(frame blending, fast) or `--smooth interpolate` (motion-compensated
+`minterpolate`, fluid but roughly 10-20x slower than realtime). `trim` keeps
+the head (or the middle with `--from-center`). `--fps` forces a constant frame
+rate; VFR sources are conformed automatically even without it.
-### caption.py — subtitles
+### caption.py — subtitles (static, animated, karaoke)
```
caption.py INPUT --srt FILE | --ass FILE | --text CUES.txt [--write-srt OUT.srt]
[--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB]
- [--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N] [-o OUT]
+ [--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N]
+ [--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass] [-o OUT]
caption.py --text CUES.txt --write-srt OUT.srt # generate the SRT only
```
Text cue format, one per line: `0:00-0:03 Hello`, `00:00:03.500 --> 00:00:06 Two | lines`.
Lines without a time run for `--auto-seconds` (3 s) after the previous cue. `|` is a line break.
+`--animate`/`--karaoke` generate a styled ASS (PlayRes = video size) from the
+SRT/text cues: `pop` is the short-form "bouncy" entrance, `--karaoke` fills each
+word from `--color` to `--highlight-color` evenly across the cue (word timing
+is distributed, not transcribed). The ASS is kept next to the output so the
+user can hand-tune timings and re-run with `--ass`.
### overlay.py — logo, image, title
```
@@ -99,17 +119,47 @@ overlay.py INPUT --image PNG [--scale W | --scale-percent P] | --text "..." [--f
```
Alpha in PNGs is respected. Fades apply to the overlay only; the video keeps playing.
-### sync.py — offset detection and alignment
+### sync.py — offset detection, alignment, drift correction
```
-sync.py REFERENCE SECOND [--json] [--max-offset 30] [--analyze-seconds 120]
+sync.py REFERENCE SECOND [--json] [--max-offset 30] [--analyze-seconds 120] [--fix-drift [--drift-window 60]]
[--replace-audio | --trim-second] [-o OUT]
```
-Cross-correlates loudness envelopes (pure Python FFT, ~5 ms resolution).
-Positive offset = the second recording started later. `--replace-audio` writes
-the reference video with the second file's audio aligned (video stream copied).
+Cross-correlates loudness envelopes: coarse FFT search (20 ms), then a direct
+1 ms refinement (pure Python, a 2-minute window takes ~1-3 s). Positive offset
+= the second recording started later. `--replace-audio` writes the reference
+video with the second file's audio aligned (video stream copied).
`--trim-second` writes the second file shifted to the reference timeline.
-Check `confidence` (0–1); below ~0.3 the match is doubtful — try a longer
-`--analyze-seconds` or a window that contains a clear event (clap).
+`--fix-drift` measures the offset again near the end of the overlap, reports
+the clock difference in ppm, and resamples the second file so a 60-minute
+take stays in sync (typical consumer devices drift 20-500 ppm = up to 1.8 s/h).
+Use it whenever the recording is longer than ~10 minutes. Check `confidence`
+(0–1); below ~0.3 the match is doubtful — use a window with a clear event.
+
+### color.py — HDR to SDR, LUTs, colour tags
+```
+color.py INPUT --to-sdr [--tonemap hable|mobius|reinhard|bt2390] [--peak 1000] [--desat 0] [-o OUT]
+color.py INPUT --lut grade.cube [--lut-strength 0..1] [-o OUT]
+color.py INPUT --retag bt709|bt2020-pq|bt2020-hlg|bt601 [-o OUT] # metadata only, stream copy
+```
+`--to-sdr` does a real conversion: linearise (zscale, PQ or HLG), tone-map
+(default `hable`, `mobius` keeps more highlight detail, `bt2390` is the
+broadcast standard), then BT.709 gamma + matrix. Refuses when probe says the
+input is not HDR unless `--force`. `--lut` applies a 3D .cube with
+tetrahedral interpolation (Log→709 conversions, creative looks); blend with
+`--lut-strength`. Everything else in the skill assumes SDR BT.709, so run this
+first on HDR or Log sources.
+
+### audio.py — clean-up, music, ducking, layout
+```
+audio.py INPUT [--voice | --denoise [--denoise-strength 25]] [--gain dB]
+ [--music FILE [--music-volume -14] [--duck [--duck-amount 12]] [--music-loop]]
+ [--fade-in S] [--fade-out S] [--stereo | --mono | --downmix] [--replace FILE] [-o OUT]
+```
+`--voice` = highpass 80 Hz → de-esser → FFT denoise → gentle compressor, the
+standard talking-head chain. `--duck` uses a sidechain compressor keyed by the
+speech so music dips under dialogue and swells in pauses. `--downmix` uses the
+ITU centre/LFE weights for 5.1/7.1 → stereo. Video is always stream-copied.
+Run `loudness.py` after this for final levels.
### loudness.py — EBU R128 normalisation
```
@@ -131,21 +181,22 @@ trims to platform maximums (Reels 90 s, X 140 s) unless `--allow-long`.
- **Variable frame rate (phone/screen recordings).** `probe.py` sets
`variable_frame_rate_suspected` when `r_frame_rate` and `avg_frame_rate`
- disagree. VFR causes audio drift and bad concat joins. Fix it early with
- `fit.py --fps 30` (or 60) before cutting with `--accurate`, captioning or
- syncing; add `--accurate` to `cut.py` because copy-cuts on VFR are unreliable.
+ disagree. Every re-encoding script then adds `-fps_mode cfr` at the source's
+ average rate, and `cut.py` switches itself to `--accurate` (copy-cuts on VFR
+ are unreliable). Pick the rate explicitly with `fit.py --fps 30|60` when the
+ average is odd (e.g. 23.4 fps from dropped frames).
- **Audio drift / sync.** Don't mix files with different frame rates or sample
rates in one `cut.py --segments` join without re-encoding (`--accurate`).
- After `sync.py`, verify by running it again on the output: the offset should
- be ~0. If dialogue drifts over a long recording, the clocks differ; align the
- start with `sync.py`, then fix drift with `fit.py --duration` on the audio-only
- track to match the reference length.
-- **Colour.** All H.264/H.265 outputs are tagged BT.709 (`-colorspace`,
- `-color_primaries`, `-color_trc`) and `yuv420p`. If `probe.py` shows a
- BT.2020/HDR source (`color_transfer` smpte2084 or arib-std-b67), tell the
- user the output will be tagged as SDR BT.709 without tone mapping, and add
- `-vf zscale=t=linear:npl=100,tonemap=hable,zscale=p=bt709:t=bt709:m=bt709`
- manually with ffmpeg if they need a proper SDR conversion.
+ After `sync.py`, verify by running it again on the output: offset (and drift
+ ppm with `--fix-drift`) should be ~0. Recordings longer than ~10 minutes from
+ separate devices: always use `--fix-drift`.
+- **Colour.** All H.264/H.265 outputs are tagged BT.709 and `yuv420p`. When
+ `probe.py` reports `hdr: true` (`hdr_format` HDR10/PQ, HLG or BT.2020), run
+ `color.py --to-sdr` **first**; other scripts would tag the HDR picture as
+ BT.709 and it would look flat and desaturated (`export.py` warns about this).
+ For Log footage (S-Log, V-Log, C-Log: looks grey and low-contrast but is
+ tagged SDR) apply the manufacturer's `.cube` with `color.py --lut`. Keep
+ ProRes masters at source colour: `export.py --preset prores` does not retag.
- **CJK and other non-Latin text.** libass and drawtext need a font that has
the glyphs. Check with `fc-list | grep -i cjk`. Then either name it
(`caption.py --font "Noto Sans CJK JP"`) or point at the file
diff --git a/assets/demo.gif b/assets/demo.gif
index d0b6de3..7eafe2e 100644
Binary files a/assets/demo.gif and b/assets/demo.gif differ
diff --git a/examples/README.md b/examples/README.md
index b835488..47abedd 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -74,6 +74,40 @@ python3 $S/overlay.py final.mp4 --image logo.png --position top-right --scale 22
python3 $S/overlay.py final.mp4 --text "Episode 12 — Clean Audio" --position top-left --font-size 48 --box --start 0 --end 4 --fade 0.4
```
+### "The iPhone HDR footage looks washed out"
+```bash
+python3 $S/probe.py IMG_0231.MOV --field video.hdr_format # HDR10/PQ, HLG, ...
+python3 $S/color.py IMG_0231.MOV --to-sdr # real tone mapping to BT.709
+python3 $S/color.py IMG_0231.MOV --to-sdr --tonemap mobius --peak 1200
+```
+
+### "Apply the S-Log3 conversion LUT" / "give it this look"
+```bash
+python3 $S/color.py a7s.mp4 --lut SLog3SGamut3.CineToLC-709TypeA.cube
+python3 $S/color.py final.mp4 --lut teal_orange.cube --lut-strength 0.6
+```
+
+### "Clean up the audio and add music under it"
+```bash
+python3 $S/audio.py talk.mp4 --voice # highpass, de-ess, denoise, compress
+python3 $S/audio.py talk.mp4 --music bed.mp3 --duck --music-volume -16 --fade-out 3
+python3 $S/audio.py surround.mov --downmix # 5.1 -> stereo
+python3 $S/audio.py clip.mp4 --replace narration.wav --fade-in 0.5
+```
+
+### "TikTok-style captions"
+```bash
+python3 $S/caption.py final.mp4 --text cues.txt --animate pop --karaoke --bold --size 30
+python3 $S/caption.py final.mp4 --srt subs.srt --animate fade --position center
+# the generated .ass sits next to the output; tweak it and re-run with --ass
+```
+
+### "Smooth half-speed slow motion"
+```bash
+python3 $S/fit.py shot.mp4 --duration 10 --smooth interpolate # 5 s clip -> fluid 10 s (slow to render)
+python3 $S/fit.py shot.mp4 --duration 10 --smooth blend # quick alternative
+```
+
### "Sync the lav mic to the camera"
```bash
python3 $S/sync.py camera.mp4 lav.wav --json # just tell me the offset
@@ -86,6 +120,12 @@ python3 $S/sync.py camera_synced.mp4 lav.wav --json # verify: off
python3 $S/sync.py camA.mp4 camB.mp4 --trim-second -o camB_aligned.mp4
```
+### "It's an hour-long recording and the audio drifts"
+```bash
+python3 $S/sync.py camera.mp4 zoom_h5.wav --fix-drift --json # reports offset + drift ppm
+python3 $S/sync.py camera.mp4 zoom_h5.wav --fix-drift --replace-audio -o camera_synced.mp4
+```
+
### "Fix the levels"
```bash
python3 $S/loudness.py final.mp4 # -14 LUFS / -1 dBTP (YouTube, Spotify)
diff --git a/examples/make_demo.sh b/examples/make_demo.sh
index 76d96b7..bd4d242 100755
--- a/examples/make_demo.sh
+++ b/examples/make_demo.sh
@@ -39,6 +39,10 @@ cat > "$OUT/cues.txt" <<'CUES'
0:08-0:12 No API keys. No cloud. No dependencies.
CUES
+# HDR10 clip (10-bit HEVC, PQ / BT.2020) and a 5.1 clip, to show colour + audio handling
+ffmpeg -y -hide_banner -loglevel error -f lavfi -i "testsrc2=size=1920x1080:rate=30" -f lavfi -i "aevalsrc='$TONES':s=48000" \
+ -t 4 -vf format=yuv420p10le -c:v libx265 -preset ultrafast -x265-params "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:log-level=error" -tag:v hvc1 -c:a aac "$OUT/hdr10.mp4"
+
# ------------------------------------------------------------------ pipeline
step "2/10 probe.py"
"$PY" "$SCRIPTS/probe.py" "$OUT/source.mp4" --compact
@@ -49,9 +53,9 @@ step "3/10 cut.py (two segments, lossless copy + concat)"
step "4/10 fit.py (retime to 8 s, pad to 9:16 at 720 px wide)"
"$PY" "$SCRIPTS/fit.py" "$OUT/source.mp4" --duration 8 --aspect 9:16 --fit pad --width 720 -o "$OUT/02_fit.mp4"
-step "5/10 caption.py (text cues -> SRT -> burn in)"
+step "5/10 caption.py (text cues -> animated ASS with word highlight -> burn in)"
"$PY" "$SCRIPTS/caption.py" "$OUT/source.mp4" --text "$OUT/cues.txt" --write-srt "$OUT/cues.srt" \
- --size 26 --bold --position bottom --margin 40 -o "$OUT/03_captioned.mp4"
+ --size 26 --bold --position bottom --margin 40 --animate pop --karaoke -o "$OUT/03_captioned.mp4"
step "6/10 overlay.py (logo top-right with fade, plus a title)"
"$PY" "$SCRIPTS/overlay.py" "$OUT/03_captioned.mp4" --image "$OUT/logo.png" --position top-right \
@@ -63,8 +67,13 @@ step "7/10 sync.py (detect the 2.5 s lav-mic offset, replace camera audio)"
"$PY" "$SCRIPTS/sync.py" "$OUT/source.mp4" "$OUT/lavmic.wav" --json | tee "$OUT/sync.json"
"$PY" "$SCRIPTS/sync.py" "$OUT/05_titled.mp4" "$OUT/lavmic.wav" --replace-audio -o "$OUT/06_synced.mp4"
-step "8/10 loudness.py (-14 LUFS, -1 dBTP, two-pass)"
-"$PY" "$SCRIPTS/loudness.py" "$OUT/06_synced.mp4" -o "$OUT/07_loudnorm.mp4"
+step "8/10 audio.py (voice clean-up + ducked music bed) and loudness.py (-14 LUFS, two-pass)"
+ffmpeg -y -hide_banner -loglevel error -f lavfi -i "aevalsrc='0.3*sin(2*PI*110*t)+0.2*sin(2*PI*165*t)*gt(sin(2*PI*2*t)\,0)':s=48000" -t 12 -c:a aac "$OUT/music.m4a"
+"$PY" "$SCRIPTS/audio.py" "$OUT/06_synced.mp4" --voice --music "$OUT/music.m4a" --duck --fade-out 2 -o "$OUT/06b_audio.mp4"
+"$PY" "$SCRIPTS/loudness.py" "$OUT/06b_audio.mp4" -o "$OUT/07_loudnorm.mp4"
+
+step "8b/10 color.py (HDR10 -> SDR BT.709 tone mapping)"
+"$PY" "$SCRIPTS/color.py" "$OUT/hdr10.mp4" --to-sdr --preset veryfast -o "$OUT/07b_sdr.mp4"
step "9/10 export.py (YouTube, Reels, X, ProRes, H.265)"
"$PY" "$SCRIPTS/export.py" "$OUT/07_loudnorm.mp4" --preset youtube -o "$OUT/08_youtube.mp4"
diff --git a/package.json b/package.json
index 2a71413..7e31e3a 100644
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
{
"name": "ffmpeg-skill",
- "version": "0.1.0",
- "description": "Agent Skill that lets coding agents (Claude Code, Cursor, Codex) do professional video editing with local FFmpeg. No API keys, no cloud, no dependencies.",
+ "version": "0.2.0",
+ "description": "Agent Skill that lets coding agents (Claude Code, Cursor, Codex) do professional video editing with local FFmpeg: cut, captions, sync with drift correction, HDR to SDR, LUTs, audio clean-up and ducking, loudness, platform exports. No API keys, no cloud, no dependencies.",
"keywords": ["ffmpeg", "video", "agent-skill", "claude-code", "cursor", "codex", "skill", "video-editing"],
"license": "MIT",
"author": "kajisho5",
diff --git a/scripts/_common.py b/scripts/_common.py
index d3b37f1..7a37381 100755
--- a/scripts/_common.py
+++ b/scripts/_common.py
@@ -128,6 +128,10 @@ def probe(path: str) -> Dict[str, Any]:
rotation = int(video["tags"]["rotate"])
except ValueError:
pass
+ pix = video.get("pix_fmt") or ""
+ trc = video.get("color_transfer") or ""
+ prim = video.get("color_primaries") or ""
+ hdr = trc in ("smpte2084", "arib-std-b67") or prim == "bt2020"
out["video"] = {
"codec": video.get("codec_name"),
"profile": video.get("profile"),
@@ -139,6 +143,9 @@ def probe(path: str) -> Dict[str, Any]:
"avg_frame_rate": video.get("avg_frame_rate"),
"variable_frame_rate_suspected": vfr,
"pix_fmt": video.get("pix_fmt"),
+ "bit_depth": 10 if "10" in pix else (12 if "12" in pix else 8),
+ "hdr": hdr,
+ "hdr_format": ("HDR10/PQ" if trc == "smpte2084" else "HLG" if trc == "arib-std-b67" else "BT.2020 SDR" if hdr else None),
"color_space": video.get("color_space"),
"color_primaries": video.get("color_primaries"),
"color_transfer": video.get("color_transfer"),
@@ -208,6 +215,20 @@ def escape_drawtext(text: str) -> str:
)
+def cfr_args(meta: Optional[Dict[str, Any]], fps: Optional[float] = None) -> List[str]:
+ """Force a constant frame rate on output when the source looks VFR (or fps is given).
+
+ VFR sources (phone/screen recordings) drift against audio after cuts and joins,
+ so every re-encoding script passes this to conform them automatically.
+ """
+ v = (meta or {}).get("video") or {}
+ if fps is None and not v.get("variable_frame_rate_suspected"):
+ return []
+ rate = fps or v.get("fps") or 30.0
+ rate = round(rate) if abs(rate - round(rate)) < 0.02 else rate
+ return ["-fps_mode", "cfr", "-r", f"{rate:g}"]
+
+
def x264_args(crf: int = 18, preset: str = "medium", keep_bt709: bool = True) -> List[str]:
args = ["-c:v", "libx264", "-preset", preset, "-crf", str(crf), "-pix_fmt", "yuv420p", "-movflags", "+faststart"]
if keep_bt709:
diff --git a/scripts/audio.py b/scripts/audio.py
new file mode 100755
index 0000000..3e3246e
--- /dev/null
+++ b/scripts/audio.py
@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+"""Audio post: denoise, voice clean-up, background music with auto-ducking,
+fades and stereo/mono handling. Video is stream-copied.
+
+Examples:
+ python3 audio.py interview.mp4 --denoise # FFT noise reduction
+ python3 audio.py interview.mp4 --voice # highpass + de-esser + compressor + denoise
+ python3 audio.py talk.mp4 --music bed.mp3 --duck # music under speech, auto-ducked
+ python3 audio.py talk.mp4 --music bed.mp3 --music-volume -18 --fade-out 3
+ python3 audio.py clip.mp4 --fade-in 0.5 --fade-out 1 --stereo
+ python3 audio.py surround.mov --downmix # 5.1 -> stereo with proper centre/LFE weights
+ python3 audio.py clip.mp4 --replace narration.wav # swap the audio track entirely
+"""
+import argparse
+import sys
+from typing import List
+
+from _common import audio_codec_for, default_output, die, ffmpeg_base, info, probe, run
+
+VOICE_CHAIN = "highpass=f=80,deesser=i=0.4,afftdn=nf=-25:tn=1,acompressor=threshold=-18dB:ratio=3:attack=5:release=80:makeup=2"
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("input")
+ ap.add_argument("-o", "--output", help="output file (default: <name>_audio.<ext>)")
+ clean = ap.add_argument_group("clean-up")
+ clean.add_argument("--denoise", action="store_true", help="FFT noise reduction (afftdn, adaptive)")
+ clean.add_argument("--denoise-strength", type=float, default=25.0, help="noise floor in dB to remove, 10..60 (default 25)")
+ clean.add_argument("--voice", action="store_true", help="speech preset: highpass 80 Hz, de-esser, denoise, gentle compression")
+ clean.add_argument("--gain", type=float, help="gain in dB applied to the main track")
+ music = ap.add_argument_group("music")
+ music.add_argument("--music", help="music file to mix underneath")
+ music.add_argument("--music-volume", type=float, default=-14.0, help="music level in dB relative to full scale (default -14)")
+ music.add_argument("--duck", action="store_true", help="auto-duck the music when the main track has speech (sidechain compressor)")
+ music.add_argument("--duck-amount", type=float, default=12.0, help="how many dB to duck (default 12)")
+ music.add_argument("--music-loop", action="store_true", help="loop the music if shorter than the video")
+ fades = ap.add_argument_group("fades / layout")
+ fades.add_argument("--fade-in", type=float, default=0.0, help="seconds")
+ fades.add_argument("--fade-out", type=float, default=0.0, help="seconds")
+ fades.add_argument("--stereo", action="store_true", help="force 2-channel output (mono is duplicated to both sides)")
+ fades.add_argument("--mono", action="store_true", help="force 1-channel output")
+ fades.add_argument("--downmix", action="store_true", help="downmix 5.1/7.1 to stereo using standard weights")
+ fades.add_argument("--replace", help="replace the audio with this file (trimmed/padded to the video)")
+ ap.add_argument("--bitrate", default="192k")
+ args = ap.parse_args()
+
+ meta = probe(args.input)
+ dur = meta.get("duration") or 0.0
+ has_video = bool(meta.get("video"))
+ if not meta.get("audio") and not args.replace:
+ die("input has no audio stream (use --replace to add one)")
+ output = args.output or default_output(args.input, "audio")
+
+ inputs: List[str] = ["-i", args.input]
+ main_src = "0:a:0"
+ idx = 1
+ if args.replace:
+ probe(args.replace)
+ inputs += ["-i", args.replace]
+ main_src = f"{idx}:a:0"
+ idx += 1
+
+ fx: List[str] = []
+ if args.downmix:
+ fx.append("pan=stereo|FL=0.707*FC+FL+0.5*BL+0.5*SL+0.5*LFE|FR=0.707*FC+FR+0.5*BR+0.5*SR+0.5*LFE")
+ if args.voice:
+ fx.append(VOICE_CHAIN)
+ elif args.denoise:
+ fx.append(f"afftdn=nf=-{args.denoise_strength:g}:tn=1")
+ if args.gain:
+ fx.append(f"volume={args.gain:g}dB")
+ if args.mono:
+ fx.append("pan=mono|c0=0.5*c0+0.5*c1")
+ elif args.stereo:
+ fx.append("aformat=channel_layouts=stereo")
+
+ graph: List[str] = []
+ graph.append(f"[{main_src}]{','.join(fx) if fx else 'anull'}[main]")
+ last = "main"
+
+ if args.music:
+ probe(args.music)
+ if args.music_loop:
+ inputs += ["-stream_loop", "-1", "-i", args.music]
+ else:
+ inputs += ["-i", args.music]
+ m = f"{idx}:a:0"
+ idx += 1
+ mfx = [f"volume={args.music_volume:g}dB", f"atrim=0:{dur:.3f}" if dur else "anull"]
+ if args.fade_out:
+ mfx.append(f"afade=t=out:st={max(0.0, dur - args.fade_out):.3f}:d={args.fade_out:g}")
+ graph.append(f"[{m}]{','.join(mfx)}[music]")
+ if args.duck:
+ graph.append("[main]asplit=2[mainA][sc]")
+ graph.append(
+ f"[music][sc]sidechaincompress=threshold=0.05:ratio={max(2.0, args.duck_amount / 3):.1f}:attack=20:release=400:makeup=1[ducked]"
+ )
+ graph.append("[mainA][ducked]amix=inputs=2:duration=first:dropout_transition=2:normalize=0[mix]")
+ else:
+ graph.append("[main][music]amix=inputs=2:duration=first:dropout_transition=2:normalize=0[mix]")
+ last = "mix"
+
+ post: List[str] = []
+ if args.fade_in:
+ post.append(f"afade=t=in:st=0:d={args.fade_in:g}")
+ if args.fade_out and dur:
+ post.append(f"afade=t=out:st={max(0.0, dur - args.fade_out):.3f}:d={args.fade_out:g}")
+ if args.replace and dur:
+ post.append(f"apad,atrim=0:{dur:.3f}")
+ if post:
+ graph.append(f"[{last}]{','.join(post)}[out]")
+ last = "out"
+
+ cmd = ffmpeg_base() + inputs + ["-filter_complex", ";".join(graph), "-map", f"[{last}]"]
+ if has_video:
+ cmd += ["-map", "0:v:0", "-c:v", "copy"]
+ cmd += audio_codec_for(output, args.bitrate) + ["-shortest", output]
+ run(cmd)
+ r = probe(output)
+ a = r["audio"]
+ info(f"wrote {output} ({r['duration']:.3f}s, audio {a['codec']} {a['channels']}ch {a['sample_rate']}Hz)")
+ print(output)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/caption.py b/scripts/caption.py
index 4dd8ca9..42caba9 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -11,6 +11,7 @@ Text-to-SRT input format (one cue per line, blank lines ignored):
Examples:
python3 caption.py input.mp4 --srt subs.srt
+ python3 caption.py input.mp4 --text cues.txt --animate pop --karaoke # word-by-word highlight, TikTok style
python3 caption.py input.mp4 --srt subs.srt --font "Noto Sans CJK JP" --size 28 --position top
python3 caption.py --text cues.txt --write-srt cues.srt # only produce the SRT
python3 caption.py input.mp4 --text cues.txt # generate + burn in one go
@@ -21,7 +22,7 @@ import re
import sys
from typing import List, Tuple
-from _common import aac_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, info, parse_time, probe, run, x264_args
+from _common import aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, info, parse_time, probe, run, x264_args
ALIGN = {"bottom": 2, "top": 8, "center": 5, "bottom-left": 1, "bottom-right": 3, "top-left": 7, "top-right": 9}
@@ -58,12 +59,84 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float) -> List[Tuple[fl
return cues
+def parse_srt(path: str) -> List[Tuple[float, float, str]]:
+ cues: List[Tuple[float, float, str]] = []
+ block: List[str] = []
+ with open(path, encoding="utf-8-sig") as fh:
+ content = fh.read().replace("\r\n", "\n") + "\n\n"
+ for line in content.split("\n"):
+ if line.strip():
+ block.append(line)
+ continue
+ if block:
+ times = next((b for b in block if "-->" in b), None)
+ if times:
+ a, b = times.split("-->")
+ text = "\n".join(block[block.index(times) + 1:]).strip()
+ cues.append((parse_time(a), parse_time(b), text))
+ block = []
+ if not cues:
+ die(f"no cues found in {path}")
+ return cues
+
+
def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
with open(path, "w", encoding="utf-8") as fh:
for i, (s, e, t) in enumerate(cues, 1):
fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
+def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int, play_h: int) -> None:
+ """Write a styled ASS file with optional animation and word-by-word highlight."""
+ def t(sec: float) -> str:
+ cs = int(round(sec * 100))
+ h, rem = divmod(cs, 360000)
+ m, rem = divmod(rem, 6000)
+ s_, cs = divmod(rem, 100)
+ return f"{h}:{m:02d}:{s_:02d}.{cs:02d}"
+
+ scale = play_h / 288.0 # our --size is relative to a 288-line script like force_style
+ size = int(round(args.size * scale))
+ margin = int(round(args.margin * scale))
+ # karaoke: PrimaryColour is the "sung" colour, SecondaryColour the "not yet sung" one
+ primary = ass_color(args.highlight_color if args.karaoke else args.color)
+ secondary = ass_color(args.color)
+ outline = ass_color(args.outline_color)
+ back = ass_color(args.outline_color, 0x80)
+ header = [
+ "[Script Info]", "ScriptType: v4.00+", f"PlayResX: {play_w}", f"PlayResY: {play_h}", "WrapStyle: 0", "ScaledBorderAndShadow: yes", "",
+ "[V4+ Styles]",
+ "Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding",
+ f"Style: Default,{args.font},{size},{primary},{secondary},{outline},{back},{-1 if args.bold else 0},0,0,0,100,100,0,0,{3 if args.box else 1},{args.outline * scale:.1f},{args.shadow * scale:.1f},{ALIGN[args.position]},{margin},{margin},{margin},1",
+ "", "[Events]", "Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text",
+ ]
+ lines = []
+ for start, end, text in cues:
+ text = text.replace("\n", "\\N")
+ fx = ""
+ if args.animate == "fade":
+ fx = "{\\fad(200,200)}"
+ elif args.animate == "pop":
+ fx = "{\\fad(80,120)\\fscx60\\fscy60\\t(0,120,\\fscx110\\fscy110)\\t(120,200,\\fscx100\\fscy100)}"
+ elif args.animate == "slide":
+ fx = "{\\fad(150,150)\\move(%d,%d,%d,%d,0,250)}" % (play_w // 2, play_h - margin + int(30 * scale), play_w // 2, play_h - margin)
+ body = text
+ if args.karaoke:
+ # split each line into words and give every word an equal share of the cue (\k is in centiseconds)
+ dur_cs = max(1, int(round((end - start) * 100)))
+ segments = body.split("\\N")
+ words = [w for seg in segments for w in seg.split(" ") if w]
+ per = max(1, dur_cs // max(1, len(words)))
+ out_segments = []
+ for seg in segments:
+ ws = [w for w in seg.split(" ") if w]
+ out_segments.append(" ".join(f"{{\\kf{per}}}{w}" for w in ws))
+ body = "\\N".join(out_segments)
+ lines.append(f"Dialogue: 0,{t(start)},{t(end)},Default,,0,0,0,,{fx}{body}")
+ with open(path, "w", encoding="utf-8-sig") as fh:
+ fh.write("\n".join(header + lines) + "\n")
+
+
def ass_color(hex_rgb: str, alpha: int = 0) -> str:
h = hex_rgb.lstrip("#")
if len(h) != 6:
@@ -95,6 +168,11 @@ def main() -> int:
sty.add_argument("--position", choices=sorted(ALIGN), default="bottom", help="on-screen placement (default bottom)")
sty.add_argument("--margin", type=int, default=30, help="vertical margin from the edge (default 30)")
sty.add_argument("--box", action="store_true", help="draw an opaque box behind text instead of an outline")
+ anim = ap.add_argument_group("animation (generates ASS; needs --text or --srt input)")
+ anim.add_argument("--animate", choices=["none", "fade", "pop", "slide"], default="none", help="per-cue entrance animation")
+ anim.add_argument("--karaoke", action="store_true", help="word-by-word highlight (fills from --color to --highlight-color across each cue)")
+ anim.add_argument("--highlight-color", default="FFD200", help="karaoke fill colour RRGGBB (default FFD200)")
+ anim.add_argument("--write-ass", help="where to save the generated ASS (default: next to the output)")
enc = ap.add_argument_group("encoding")
enc.add_argument("--crf", type=int, default=18)
enc.add_argument("--preset", default="medium")
@@ -115,7 +193,20 @@ def main() -> int:
if not args.input:
die("input video is required unless you only use --text/--write-srt")
- probe(args.input)
+ meta = probe(args.input)
+ if not meta.get("video"):
+ die("input has no video stream")
+
+ output = args.output or default_output(args.input, "captioned")
+ if (args.animate != "none" or args.karaoke) and not args.ass:
+ cues_for_ass = cues if args.text else parse_srt(srt_path)
+ ass_path = args.write_ass or os.path.splitext(output)[0] + ".ass"
+ w, h = meta["video"]["width"], meta["video"]["height"]
+ if meta["video"].get("rotation") in (90, -90, 270, -270):
+ w, h = h, w
+ write_ass(cues_for_ass, ass_path, args, w, h)
+ info(f"wrote {ass_path} ({len(cues_for_ass)} cues, animate={args.animate}, karaoke={args.karaoke})")
+ args.ass = ass_path
if args.ass:
if not os.path.exists(args.ass):
@@ -144,8 +235,8 @@ def main() -> int:
if args.fonts_dir:
vf += f":fontsdir={escape_filter_path(args.fonts_dir)}"
- output = args.output or default_output(args.input, "captioned")
- cmd = ffmpeg_base() + ["-i", args.input, "-vf", vf] + x264_args(args.crf, args.preset) + aac_args() + [output]
+ cmd = ffmpeg_base() + ["-i", args.input, "-vf", vf] + x264_args(args.crf, args.preset) + cfr_args(meta)
+ cmd += (aac_args() if meta.get("audio") else ["-an"]) + [output]
run(cmd)
result = probe(output)
info(f"wrote {output} ({result.get('duration'):.3f}s)")
diff --git a/scripts/color.py b/scripts/color.py
new file mode 100755
index 0000000..df8083d
--- /dev/null
+++ b/scripts/color.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Colour management: convert HDR (HDR10/PQ, HLG, BT.2020) to SDR BT.709 with
+real tone mapping, apply a .cube LUT (Log footage, creative grades), or fix
+wrong colour tags without re-encoding.
+
+Examples:
+ python3 color.py iphone_hdr.mov --to-sdr # PQ/HLG -> BT.709 SDR, hable tonemap
+ python3 color.py iphone_hdr.mov --to-sdr --tonemap mobius --peak 1000
+ python3 color.py slog3.mp4 --lut SLog3_to_Rec709.cube # apply LUT (any Log -> 709 or a look)
+ python3 color.py clip.mp4 --lut look.cube --lut-strength 0.6
+ python3 color.py wrongly_tagged.mp4 --retag bt709 # metadata only, stream copy
+"""
+import argparse
+import os
+import sys
+from typing import List
+
+from _common import aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, info, probe, run, x264_args
+
+TONEMAPS = ["hable", "mobius", "reinhard", "bt2390", "clip", "linear", "gamma"]
+
+
+def hdr_to_sdr_chain(meta: dict, tonemap: str, peak: float, desat: float) -> str:
+ v = meta["video"]
+ trc = v.get("color_transfer") or "smpte2084"
+ prim = v.get("color_primaries") or "bt2020"
+ space = v.get("color_space") or "bt2020nc"
+ # zscale needs explicit input tags when the file lacks them
+ chain: List[str] = [
+ f"zscale=tin={trc}:pin={prim}:min={space}:rin={v.get('color_range') or 'tv'}:t=linear:npl={peak:g}",
+ "format=gbrpf32le",
+ "zscale=p=bt709",
+ f"tonemap=tonemap={tonemap}:desat={desat:g}" + (":peak=%g" % (peak / 100.0) if tonemap in ("bt2390",) else ""),
+ "zscale=t=bt709:m=bt709:r=tv",
+ "format=yuv420p",
+ ]
+ return ",".join(chain)
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("input")
+ ap.add_argument("-o", "--output", help="output file (default: <name>_sdr / _lut / _retag)")
+ mode = ap.add_mutually_exclusive_group(required=True)
+ mode.add_argument("--to-sdr", action="store_true", help="tone-map HDR (PQ/HLG/BT.2020) to SDR BT.709")
+ mode.add_argument("--lut", help=".cube LUT to apply (3D)")
+ mode.add_argument("--retag", choices=["bt709", "bt2020-pq", "bt2020-hlg", "bt601"], help="rewrite colour tags only (no re-encode)")
+ ap.add_argument("--tonemap", choices=TONEMAPS, default="hable", help="tone-mapping curve (default hable)")
+ ap.add_argument("--peak", type=float, default=1000.0, help="source peak brightness in nits used for PQ (default 1000)")
+ ap.add_argument("--desat", type=float, default=0.0, help="tonemap desaturation strength (default 0)")
+ ap.add_argument("--lut-strength", type=float, default=1.0, help="blend LUT result with the original, 0..1 (default 1)")
+ ap.add_argument("--force", action="store_true", help="run --to-sdr even if the file is not tagged as HDR (treat as PQ)")
+ ap.add_argument("--crf", type=int, default=18)
+ ap.add_argument("--preset", default="medium")
+ args = ap.parse_args()
+
+ meta = probe(args.input)
+ if not meta.get("video"):
+ die("input has no video stream")
+ v = meta["video"]
+ has_audio = bool(meta.get("audio"))
+
+ if args.retag:
+ tags = {
+ "bt709": ["bt709", "bt709", "bt709"],
+ "bt2020-pq": ["bt2020nc", "bt2020", "smpte2084"],
+ "bt2020-hlg": ["bt2020nc", "bt2020", "arib-std-b67"],
+ "bt601": ["smpte170m", "smpte170m", "smpte170m"],
+ }[args.retag]
+ output = args.output or default_output(args.input, "retag")
+ ext = os.path.splitext(output)[1].lower()
+ cmd = ffmpeg_base() + ["-i", args.input, "-map", "0", "-c", "copy",
+ "-colorspace", tags[0], "-color_primaries", tags[1], "-color_trc", tags[2]]
+ if ext in (".mp4", ".mov", ".m4v"):
+ cmd += ["-movflags", "+faststart"]
+ cmd.append(output)
+ proc = run(cmd, check=False)
+ if proc.returncode != 0:
+ # some codecs cannot carry retagged colour info without a bitstream filter; fall back to re-encode
+ info("stream copy could not rewrite tags, re-encoding")
+ cmd = ffmpeg_base() + ["-i", args.input, "-map", "0:v:0", "-map", "0:a?"] + x264_args(args.crf, args.preset, keep_bt709=False)
+ cmd += ["-colorspace", tags[0], "-color_primaries", tags[1], "-color_trc", tags[2]] + (aac_args() if has_audio else []) + [output]
+ run(cmd)
+ info(f"wrote {output} (tags -> {args.retag})")
+ print(output)
+ return 0
+
+ if args.to_sdr:
+ if not v.get("hdr") and not args.force:
+ die(f"{args.input} is not tagged as HDR (transfer={v.get('color_transfer')}, primaries={v.get('color_primaries')}). Use --force to tone-map anyway.")
+ vf = hdr_to_sdr_chain(meta, args.tonemap, args.peak, args.desat)
+ output = args.output or default_output(args.input, "sdr")
+ tag = "sdr"
+ else:
+ if not os.path.exists(args.lut):
+ die(f"LUT not found: {args.lut}")
+ lut = f"lut3d=file={escape_filter_path(args.lut)}:interp=tetrahedral"
+ if 0 < args.lut_strength < 1:
+ # blend graded and original
+ vf = f"split[o][g];[g]{lut}[g2];[o][g2]blend=all_mode=normal:all_opacity={args.lut_strength:g},format=yuv420p"
+ else:
+ vf = f"{lut},format=yuv420p"
+ output = args.output or default_output(args.input, "lut")
+ tag = "lut"
+
+ cmd = ffmpeg_base() + ["-i", args.input, "-vf", vf, "-map", "0:v:0", "-map", "0:a?"]
+ cmd += x264_args(args.crf, args.preset) + cfr_args(meta) + (aac_args() if has_audio else []) + [output]
+ run(cmd)
+ r = probe(output)
+ info(f"wrote {output} ({r['duration']:.3f}s, {r['video']['width']}x{r['video']['height']}, "
+ f"{r['video']['color_transfer']}/{r['video']['color_primaries']}, {tag})")
+ print(output)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/cut.py b/scripts/cut.py
index cd3f03e..8e5296b 100755
--- a/scripts/cut.py
+++ b/scripts/cut.py
@@ -16,7 +16,7 @@ import sys
import tempfile
from typing import List, Tuple
-from _common import aac_args, default_output, die, ffmpeg_base, info, parse_time, probe, run, x264_args
+from _common import aac_args, cfr_args, default_output, die, ffmpeg_base, info, parse_time, probe, run, x264_args
def parse_segments(spec: str) -> List[Tuple[float, float]]:
@@ -37,26 +37,27 @@ def parse_segments(spec: str) -> List[Tuple[float, float]]:
return segs
-def cut_one(src: str, start: float, end: float, dst: str, reencode: bool, crf: int, preset: str, tolerance: float = 0.5) -> bool:
+def cut_one(src: str, start: float, end: float, dst: str, reencode: bool, crf: int, preset: str, tolerance: float = 0.5, meta: dict = None) -> bool:
"""Cut one segment. Returns True if the result was re-encoded."""
dur = end - start
+ meta = meta or probe(src)
if reencode:
cmd = ffmpeg_base() + ["-ss", f"{start:.3f}", "-i", src, "-t", f"{dur:.3f}"]
- cmd += x264_args(crf, preset) + aac_args() + ["-avoid_negative_ts", "make_zero", dst]
+ cmd += x264_args(crf, preset) + cfr_args(meta) + aac_args() + ["-avoid_negative_ts", "make_zero", dst]
else:
cmd = ffmpeg_base() + ["-ss", f"{start:.3f}", "-i", src, "-t", f"{dur:.3f}", "-c", "copy", "-avoid_negative_ts", "make_zero", dst]
proc = run(cmd, check=False)
if proc.returncode != 0:
if not reencode:
info("stream copy failed, falling back to re-encode")
- return cut_one(src, start, end, dst, True, crf, preset, tolerance)
+ return cut_one(src, start, end, dst, True, crf, preset, tolerance, meta)
die(f"ffmpeg failed:\n{proc.stderr.strip()}")
if not reencode and tolerance >= 0:
got = probe(dst).get("duration") or 0.0
if abs(got - dur) > tolerance:
info(f"stream copy landed on a keyframe {abs(got - dur):.2f}s away from the requested cut "
f"(> {tolerance:.2f}s tolerance); re-encoding this segment for accuracy")
- return cut_one(src, start, end, dst, True, crf, preset, tolerance)
+ return cut_one(src, start, end, dst, True, crf, preset, tolerance, meta)
return reencode
@@ -77,6 +78,9 @@ def main() -> int:
meta = probe(args.input)
total = meta.get("duration") or 0.0
+ if meta.get("video", {}) and meta["video"].get("variable_frame_rate_suspected") and not args.accurate:
+ info("source looks variable-frame-rate; lossless cuts on VFR are unreliable, switching to --accurate")
+ args.accurate = True
if args.segments:
segments = parse_segments(args.segments)
@@ -104,13 +108,13 @@ def main() -> int:
reencoded = False
if len(segments) == 1:
- reencoded = cut_one(args.input, segments[0][0], segments[0][1], output, args.accurate, args.crf, args.preset, args.tolerance)
+ reencoded = cut_one(args.input, segments[0][0], segments[0][1], output, args.accurate, args.crf, args.preset, args.tolerance, meta)
else:
with tempfile.TemporaryDirectory(prefix="ffskill_cut_") as tmp:
parts = []
for i, (s, e) in enumerate(segments):
part = os.path.join(tmp, f"part{i:03d}{ext}")
- reencoded |= cut_one(args.input, s, e, part, args.accurate, args.crf, args.preset, args.tolerance)
+ reencoded |= cut_one(args.input, s, e, part, args.accurate, args.crf, args.preset, args.tolerance, meta)
parts.append(part)
listfile = os.path.join(tmp, "list.txt")
with open(listfile, "w", encoding="utf-8") as fh:
@@ -120,7 +124,7 @@ def main() -> int:
proc = run(cmd, check=False)
if proc.returncode != 0:
info("concat with stream copy failed, re-encoding the join")
- cmd = ffmpeg_base() + ["-f", "concat", "-safe", "0", "-i", listfile] + x264_args(args.crf, args.preset) + aac_args() + [output]
+ cmd = ffmpeg_base() + ["-f", "concat", "-safe", "0", "-i", listfile] + x264_args(args.crf, args.preset) + cfr_args(meta) + aac_args() + [output]
run(cmd)
result = probe(output)
diff --git a/scripts/export.py b/scripts/export.py
index 902bd7e..5b1d918 100755
--- a/scripts/export.py
+++ b/scripts/export.py
@@ -22,7 +22,7 @@ import argparse
import sys
from typing import Dict, List
-from _common import default_output, die, ffmpeg_base, info, probe, run
+from _common import cfr_args, default_output, die, ffmpeg_base, info, probe, run
PRESETS: Dict[str, Dict] = {
"youtube": {"w": 1920, "h": 1080, "ext": "mp4", "video": ["-c:v", "libx264", "-preset", "slow", "-crf", "18", "-profile:v", "high", "-pix_fmt", "yuv420p"], "audio": ["-c:a", "aac", "-b:a", "192k", "-ar", "48000"], "max": None, "desc": "1080p H.264, AAC 192k"},
@@ -61,6 +61,8 @@ def main() -> int:
meta = probe(args.input)
if not meta.get("video"):
die("input has no video stream")
+ if meta["video"].get("hdr") and args.preset != "prores":
+ info("warning: source is HDR (%s). This preset outputs SDR BT.709 tags without tone mapping; run color.py --to-sdr first for correct colours." % meta["video"].get("hdr_format"))
has_audio = bool(meta.get("audio"))
output = args.output or default_output(args.input, args.preset, p["ext"])
@@ -91,6 +93,8 @@ def main() -> int:
if args.crf is not None and "-crf" in video:
video[video.index("-crf") + 1] = str(args.crf)
cmd += video
+ if "-r" not in video:
+ cmd += cfr_args(meta)
if args.preset not in ("prores",):
cmd += BT709
if p["ext"] == "mp4":
diff --git a/scripts/fit.py b/scripts/fit.py
index d67f199..e239537 100755
--- a/scripts/fit.py
+++ b/scripts/fit.py
@@ -19,7 +19,7 @@ import sys
from fractions import Fraction
from typing import List
-from _common import aac_args, default_output, die, ffmpeg_base, info, parse_time, probe, run, x264_args
+from _common import aac_args, cfr_args, default_output, die, ffmpeg_base, info, parse_time, probe, run, x264_args
ASPECT_PRESETS = {"16:9": Fraction(16, 9), "9:16": Fraction(9, 16), "1:1": Fraction(1, 1), "4:5": Fraction(4, 5), "4:3": Fraction(4, 3), "21:9": Fraction(21, 9)}
@@ -63,6 +63,8 @@ def main() -> int:
d.add_argument("--method", choices=["speed", "trim"], default="speed", help="how to reach the duration (default speed)")
d.add_argument("--from-center", action="store_true", help="with --method trim, keep the middle instead of the start")
d.add_argument("--max-speed", type=float, default=4.0, help="refuse speed factors above this (default 4x)")
+ d.add_argument("--smooth", choices=["none", "blend", "interpolate"], default="none",
+ help="slow-motion quality: blend (frame blending) or interpolate (motion-compensated, slow but fluid). default none = duplicate frames")
a = ap.add_argument_group("aspect")
a.add_argument("--aspect", help="target aspect ratio, e.g. 16:9, 9:16, 1:1, 4:5")
a.add_argument("--fit", choices=["pad", "crop"], default="pad", help="pad (letterbox) or crop to reach the aspect (default pad)")
@@ -103,6 +105,12 @@ def main() -> int:
die(f"required speed factor {factor:.2f}x exceeds --max-speed {args.max_speed}x; use --method trim or raise the limit")
if abs(factor - 1.0) > 1e-4:
vf.append(f"setpts={1/factor:.8f}*PTS")
+ src_fps = meta["video"].get("fps") or 30.0
+ if factor < 1.0 and args.smooth == "interpolate":
+ vf.append(f"minterpolate=fps={src_fps:g}:mi_mode=mci:mc_mode=aobmc:me_mode=bidir:vsbmc=1")
+ elif factor < 1.0 and args.smooth == "blend":
+ vf.append(f"fps={src_fps:g}")
+ vf.append("tblend=all_mode=average")
if has_audio:
af.append(atempo_chain(factor))
post += ["-t", f"{target:.3f}"]
@@ -133,6 +141,8 @@ def main() -> int:
if args.fps:
vf.append(f"fps={args.fps:g}")
+ elif meta["video"].get("variable_frame_rate_suspected"):
+ info("source looks variable-frame-rate; conforming to constant fps automatically")
output = args.output or default_output(args.input, "fit")
cmd = ffmpeg_base() + pre_input + ["-i", args.input]
@@ -141,6 +151,7 @@ def main() -> int:
if af:
cmd += ["-af", ",".join(af)]
cmd += x264_args(args.crf, args.preset)
+ cmd += cfr_args(meta, args.fps) if not args.fps else []
if has_audio:
cmd += aac_args()
else:
diff --git a/scripts/overlay.py b/scripts/overlay.py
index 3b846ab..fce6147 100755
--- a/scripts/overlay.py
+++ b/scripts/overlay.py
@@ -15,7 +15,7 @@ import argparse
import sys
from typing import List, Optional
-from _common import aac_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, x264_args
+from _common import aac_args, cfr_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, x264_args
POS = {
"top-left": ("{m}", "{m}"),
@@ -157,7 +157,7 @@ def main() -> int:
opts.append(f"enable='{enable}'")
cmd += ["-vf", "drawtext=" + ":".join(opts)]
- cmd += x264_args(args.crf, args.preset)
+ cmd += x264_args(args.crf, args.preset) + cfr_args(meta)
cmd += aac_args() if meta.get("audio") else ["-an"]
cmd.append(output)
run(cmd)
diff --git a/scripts/sync.py b/scripts/sync.py
index fb002b6..9a93477 100755
--- a/scripts/sync.py
+++ b/scripts/sync.py
@@ -1,11 +1,10 @@
#!/usr/bin/env python3
"""Detect the time offset between two recordings by audio cross-correlation
-and (optionally) write a synced output.
+and (optionally) write a synced output, with optional clock-drift correction.
Pure standard library: both tracks are decoded by ffmpeg to mono 8 kHz PCM,
-reduced to a coarse loudness envelope, and cross-correlated with an FFT
-implemented in Python. Precision is roughly +/- one envelope step (default
-5 ms), which is plenty for lining up a lav mic or a second camera.
+reduced to a loudness envelope, cross-correlated with an FFT implemented in
+Python (coarse, 20 ms), then refined by direct correlation at 1 ms.
Offset semantics: a positive offset means the SECOND input starts LATER
than the reference, i.e. `second` must be shifted earlier by that amount.
@@ -15,6 +14,7 @@ Examples:
python3 sync.py camera.mp4 lavmic.wav --replace-audio -o synced.mp4
python3 sync.py camA.mp4 camB.mp4 --trim-second -o camB_synced.mp4
python3 sync.py cam.mp4 mic.wav --max-offset 60 --json
+ python3 sync.py cam.mp4 recorder.wav --fix-drift --replace-audio # long takes: fix clock drift too
"""
import argparse
import cmath
@@ -31,9 +31,9 @@ from _common import aac_args, audio_codec_for, default_output, die, ffmpeg_base,
SR = 8000 # decode sample rate
-def decode_mono(path: str, seconds: float) -> List[float]:
+def decode_mono(path: str, seconds: float, start: float = 0.0) -> List[float]:
ffmpeg = require_tool("ffmpeg")
- cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-i", path, "-t", f"{seconds:.3f}",
+ cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-ss", f"{start:.3f}", "-i", path, "-t", f"{seconds:.3f}",
"-vn", "-ac", "1", "-ar", str(SR), "-f", "s16le", "-"]
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if proc.returncode != 0 or not proc.stdout:
@@ -110,6 +110,45 @@ def cross_correlate(ref: List[float], other: List[float], max_lag: int):
return best_lag, best_val / energy
+def refine(ref_s: List[float], oth_s: List[float], coarse_offset: float, fine_step: int, window_s: float) -> float:
+ """Direct correlation at fine resolution around a coarse estimate (+/- window_s)."""
+ ref_e = envelope(ref_s, fine_step)
+ oth_e = envelope(oth_s, fine_step)
+ centre = int(round(coarse_offset * SR / fine_step))
+ span = int(window_s * SR / fine_step)
+ best_lag, best_val = centre, -float("inf")
+ n = min(len(ref_e), len(oth_e))
+ for lag in range(centre - span, centre + span + 1):
+ # ref[i] ~ oth[i - lag]
+ lo, hi = max(0, lag), min(n, n + lag)
+ if hi - lo < 10:
+ continue
+ val = 0.0
+ for i in range(lo, hi):
+ val += ref_e[i] * oth_e[i - lag]
+ val /= (hi - lo)
+ if val > best_val:
+ best_val, best_lag = val, lag
+ return best_lag * fine_step / SR
+
+
+def measure_offset(ref_path: str, oth_path: str, start: float, seconds: float, step_ms: float, max_offset: float, fine_ms: float):
+ """Return (offset_seconds, confidence) for a window starting at `start` in both files."""
+ ref_s = decode_mono(ref_path, seconds, start)
+ oth_s = decode_mono(oth_path, seconds, start)
+ step = max(1, int(SR * step_ms / 1000))
+ ref = envelope(ref_s, step)
+ oth = envelope(oth_s, step)
+ if len(ref) < 10 or len(oth) < 10:
+ die("not enough audio to analyse")
+ max_lag = int(max_offset * SR / step)
+ lag, score = cross_correlate(ref, oth, max_lag)
+ offset = lag * step / SR
+ if fine_ms and fine_ms < step_ms:
+ offset = refine(ref_s, oth_s, offset, max(1, int(SR * fine_ms / 1000)), step_ms / 1000 * 2)
+ return offset, max(0.0, min(1.0, score))
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("reference", help="reference recording (usually the camera video)")
@@ -117,7 +156,10 @@ def main() -> int:
ap.add_argument("-o", "--output", help="output file when writing a synced result")
ap.add_argument("--max-offset", type=float, default=30.0, help="largest offset to search in seconds (default 30)")
ap.add_argument("--analyze-seconds", type=float, default=120.0, help="how much audio to analyse from each file (default 120)")
- ap.add_argument("--step-ms", type=float, default=5.0, help="envelope resolution in ms (default 5)")
+ ap.add_argument("--step-ms", type=float, default=20.0, help="coarse envelope resolution in ms for the FFT search (default 20)")
+ ap.add_argument("--fine-ms", type=float, default=1.0, help="fine resolution in ms for the refinement pass, 0 to skip (default 1)")
+ ap.add_argument("--fix-drift", action="store_true", help="also measure the offset near the END and correct clock drift by resampling the second file")
+ ap.add_argument("--drift-window", type=float, default=60.0, help="seconds of audio analysed at each end for drift (default 60)")
ap.add_argument("--json", action="store_true", help="print the result as JSON")
mode = ap.add_mutually_exclusive_group()
mode.add_argument("--replace-audio", action="store_true", help="write reference video with the second file's audio, aligned")
@@ -129,18 +171,47 @@ def main() -> int:
if not probe(p).get("audio"):
die(f"{p} has no audio stream to correlate")
- step = max(1, int(SR * args.step_ms / 1000))
- ref = envelope(decode_mono(args.reference, args.analyze_seconds), step)
- oth = envelope(decode_mono(args.second, args.analyze_seconds), step)
- if len(ref) < 10 or len(oth) < 10:
- die("not enough audio to analyse")
+ offset, score = measure_offset(args.reference, args.second, 0.0, args.analyze_seconds, args.step_ms, args.max_offset, args.fine_ms)
- max_lag = int(args.max_offset * SR / step)
- lag, score = cross_correlate(ref, oth, max_lag)
- # With prod = FFT(ref) * conj(FFT(other)), the peak sits at lag k where ref[i] ~ other[i - k]:
- # the same event happens k steps later in the reference than in the second file, which
- # means the second recording STARTED k steps later. Positive offset = second starts later.
- offset = lag * step / SR
+ drift_ratio = 1.0
+ drift_info = None
+ if args.fix_drift:
+ ref_dur = probe(args.reference)["duration"] or 0.0
+ sec_dur = probe(args.second)["duration"] or 0.0
+ overlap_end = min(ref_dur, sec_dur + offset) # last reference time both files cover
+ head_len = min(args.analyze_seconds, overlap_end)
+ tail_start = overlap_end - args.drift_window
+ if tail_start <= head_len / 2 + 5:
+ info("warning: files too short to measure drift reliably; skipping drift correction")
+ else:
+ ref_start = tail_start
+ sec_start = tail_start - offset
+ if sec_start < 0:
+ ref_start -= sec_start
+ sec_start = 0.0
+ ref_s = decode_mono(args.reference, args.drift_window, ref_start)
+ oth_s = decode_mono(args.second, args.drift_window, sec_start)
+ step = max(1, int(SR * args.step_ms / 1000))
+ lag, end_score = cross_correlate(envelope(ref_s, step), envelope(oth_s, step), int(2.0 * SR / step))
+ residual = lag * step / SR
+ if args.fine_ms:
+ residual = refine(ref_s, oth_s, residual, max(1, int(SR * args.fine_ms / 1000)), args.step_ms / 1000 * 2)
+ # both measurements represent the offset at the centre of their windows
+ head_mid = head_len / 2
+ tail_mid = ref_start + args.drift_window / 2
+ elapsed = tail_mid - head_mid
+ if elapsed > 0 and end_score > 0.1:
+ # offset(T) = offset0 - (ratio - 1) * T, where ratio is how fast the second file's clock
+ # runs relative to the reference (ratio > 1 = the second file is too long / plays slow)
+ drift_ratio = 1.0 - residual / elapsed
+ offset = offset + (drift_ratio - 1.0) * head_mid # extrapolate back to T = 0
+ drift_info = {"residual_at_end_seconds": round(residual, 4), "measured_over_seconds": round(elapsed, 2),
+ "drift_ppm": round((drift_ratio - 1) * 1e6, 1),
+ "meaning": "second file runs %.1f ppm %s (%.3fs over %.0fs); it will be resampled to match" % (
+ abs(drift_ratio - 1) * 1e6, "long/slow" if drift_ratio > 1 else "short/fast", abs(residual), elapsed),
+ "confidence": round(end_score, 3)}
+ else:
+ info("warning: could not measure drift with confidence; skipping drift correction")
result = {
"reference": args.reference,
@@ -149,6 +220,8 @@ def main() -> int:
"confidence": round(max(0.0, min(1.0, score)), 3),
"meaning": ("second starts %.3fs %s than reference" % (abs(offset), "later" if offset > 0 else "earlier")),
}
+ if drift_info:
+ result["drift"] = drift_info
if result["confidence"] < 0.1:
info("warning: low correlation confidence; check that both files contain the same audio event")
@@ -160,14 +233,20 @@ def main() -> int:
head_trim = -offset if offset < 0 else 0.0
second_meta = probe(args.second)
has_video = bool(second_meta.get("video"))
+ sec_sr = (second_meta.get("audio") or {}).get("sample_rate") or 48000
+ drift_af: List[str] = []
+ if abs(drift_ratio - 1.0) > 1e-7:
+ # the second file runs long by drift_ratio -> play it faster by that ratio (pitch shift is ~ppm, inaudible)
+ drift_af = [f"asetrate={sec_sr * drift_ratio:.6f}", f"aresample={sec_sr}"]
if args.replace_audio:
cmd = ffmpeg_base() + ["-i", args.reference]
if head_trim > 0:
cmd += ["-ss", f"{head_trim:.4f}"]
cmd += ["-i", args.second, "-map", "0:v:0", "-map", "1:a:0", "-c:v", "copy"]
- if delay_ms > 0:
- cmd += ["-af", f"adelay={delay_ms}:all=1"]
+ af_parts = drift_af + ([f"adelay={delay_ms}:all=1"] if delay_ms > 0 else [])
+ if af_parts:
+ cmd += ["-af", ",".join(af_parts)]
cmd += aac_args() + ["-shortest", output]
proc = run(cmd, check=False)
if proc.returncode != 0:
@@ -176,18 +255,32 @@ def main() -> int:
cmd = cmd[:-1] + x264_args(args.crf) + [output]
run(cmd)
else:
- if head_trim > 0:
+ if head_trim > 0 and not drift_af:
cmd = ffmpeg_base() + ["-ss", f"{head_trim:.4f}", "-i", args.second, "-c", "copy", "-avoid_negative_ts", "make_zero", output]
proc = run(cmd, check=False)
if proc.returncode != 0:
cmd = ffmpeg_base() + ["-ss", f"{head_trim:.4f}", "-i", args.second] + (x264_args(args.crf) if has_video else []) + audio_codec_for(output) + [output]
run(cmd)
else:
- af = f"adelay={delay_ms}:all=1"
- cmd = ffmpeg_base() + ["-i", args.second]
+ cmd = ffmpeg_base()
+ if head_trim > 0:
+ cmd += ["-ss", f"{head_trim:.4f}"]
+ cmd += ["-i", args.second]
+ af_parts = list(drift_af)
+ if delay_ms > 0:
+ af_parts.append(f"adelay={delay_ms}:all=1")
if has_video:
- cmd += ["-vf", f"tpad=start_duration={offset:.4f}"] + x264_args(args.crf)
- cmd += ["-af", af] + audio_codec_for(output) + [output]
+ vf = []
+ if delay_ms > 0:
+ vf.append(f"tpad=start_duration={offset:.4f}")
+ if drift_af:
+ vf.append(f"setpts=PTS/{drift_ratio:.9f}")
+ if vf:
+ cmd += ["-vf", ",".join(vf)]
+ cmd += x264_args(args.crf)
+ if af_parts:
+ cmd += ["-af", ",".join(af_parts)]
+ cmd += audio_codec_for(output) + [output]
run(cmd)
result["output"] = output
info(f"wrote {output}")
@@ -196,6 +289,8 @@ def main() -> int:
print(json.dumps(result, indent=2))
else:
print(f"offset: {result['offset_seconds']:+.3f}s ({result['meaning']}), confidence {result['confidence']:.2f}")
+ if drift_info:
+ print(f"drift: {drift_info['drift_ppm']:+.1f} ppm ({drift_info['residual_at_end_seconds']:+.3f}s over {drift_info['measured_over_seconds']:.0f}s), confidence {drift_info['confidence']:.2f}")
if "output" in result:
print(result["output"])
return 0
diff --git a/tests/test_all.py b/tests/test_all.py
index d57dd09..21214bc 100644
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -53,6 +53,31 @@ class FFmpegSkillTests(unittest.TestCase):
cls.cues = OUT / "cues.txt"
cls.cues.write_text("0:00-0:03 Hello world\n0:03-0:06 Second | line\nAuto timed cue\n", encoding="utf-8")
+ # --- "real world" material: VFR, rotated phone clip, 5.1 audio, 10-bit HDR10 HEVC, long drifting pair
+ cls.vfr = OUT / "vfr.mp4"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=1280x720:rate=30", "-f", "lavfi", "-i", f"aevalsrc='{TONES}':s=48000",
+ "-t", "12", "-vf", "select='gt(random(1)\\,0.3)'", "-fps_mode", "vfr",
+ "-c:v", "libx264", "-preset", "veryfast", "-c:a", "aac", cls.vfr)
+ cls.rot = OUT / "rot.mp4"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-display_rotation", "90", "-i", cls.src, "-t", "6", "-c", "copy", cls.rot)
+ cls.surround = OUT / "surround.mov"
+ six = "|".join([TONES, TONES, "0.5*" + TONES, "0.2*sin(2*PI*60*t)", "0.3*" + TONES, "0.3*" + TONES])
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=1280x720:rate=30", "-f", "lavfi", "-i", f"aevalsrc='{six}':s=48000:c=5.1",
+ "-t", "6", "-c:v", "libx264", "-preset", "veryfast", "-c:a", "aac", "-ac", "6", cls.surround)
+ cls.hdr = OUT / "hdr10.mp4"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "testsrc2=size=1920x1080:rate=30", "-f", "lavfi", "-i", f"aevalsrc='{TONES}':s=48000",
+ "-t", "4", "-vf", "format=yuv420p10le", "-c:v", "libx265", "-preset", "ultrafast",
+ "-x265-params", "colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:log-level=error",
+ "-tag:v", "hvc1", "-c:a", "aac", cls.hdr)
+ cls.long_ref = OUT / "long_ref.wav"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", f"aevalsrc='{TONES}':s=48000", "-t", "200", "-c:a", "pcm_s16le", cls.long_ref)
+ cls.long_drift = OUT / "long_drift.wav"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-ss", "1.2", "-i", cls.long_ref,
+ "-af", "asetrate=48000*0.9995,aresample=48000", "-c:a", "pcm_s16le", cls.long_drift)
+
def assertClose(self, a, b, tol, msg=""):
self.assertIsNotNone(a, msg)
self.assertLessEqual(abs(a - b), tol, f"{msg}: {a} vs {b} (tol {tol})")
@@ -201,6 +226,120 @@ class FFmpegSkillTests(unittest.TestCase):
def test_export_list(self):
self.assertIn("youtube", script("export.py", "--list").stdout)
+ # ---------------------------------------------------------------- real-world material
+ def test_probe_detects_vfr_rotation_surround_hdr(self):
+ self.assertTrue(probe(str(self.vfr))["video"]["variable_frame_rate_suspected"])
+ self.assertEqual(probe(str(self.rot))["video"]["rotation"], 90)
+ self.assertEqual(probe(str(self.surround))["audio"]["channels"], 6)
+ h = probe(str(self.hdr))["video"]
+ self.assertTrue(h["hdr"])
+ self.assertEqual(h["hdr_format"], "HDR10/PQ")
+ self.assertEqual(h["bit_depth"], 10)
+ self.assertEqual(h["codec"], "hevc")
+
+ def test_vfr_is_conformed_to_cfr_on_cut_and_fit(self):
+ out = OUT / "vfr_cut.mp4"
+ proc = script("cut.py", self.vfr, "--start", "2", "--end", "6", "-o", out)
+ self.assertIn("variable-frame-rate", proc.stderr)
+ m = probe(str(out))
+ self.assertFalse(m["video"]["variable_frame_rate_suspected"])
+ self.assertClose(m["duration"], 4.0, 0.2)
+ out2 = OUT / "vfr_fit.mp4"
+ script("fit.py", self.vfr, "--fps", "30", "--aspect", "1:1", "-o", out2)
+ m2 = probe(str(out2))
+ self.assertClose(m2["video"]["fps"], 30.0, 0.05)
+ self.assertFalse(m2["video"]["variable_frame_rate_suspected"])
+
+ def test_rotated_source_uses_display_orientation(self):
+ out = OUT / "rot_fit.mp4"
+ script("fit.py", self.rot, "--aspect", "9:16", "--fit", "pad", "--width", "540", "-o", out)
+ m = probe(str(out))
+ self.assertEqual((m["video"]["width"], m["video"]["height"]), (540, 960))
+ self.assertEqual(m["video"]["rotation"], 0)
+
+ def test_hdr_to_sdr_tonemap(self):
+ out = OUT / "sdr.mp4"
+ script("color.py", self.hdr, "--to-sdr", "--preset", "veryfast", "-o", out)
+ v = probe(str(out))["video"]
+ self.assertFalse(v["hdr"])
+ self.assertEqual((v["color_transfer"], v["color_primaries"], v["pix_fmt"]), ("bt709", "bt709", "yuv420p"))
+ self.assertEqual((v["width"], v["height"]), (1920, 1080))
+ # refuses on SDR input unless forced
+ script("color.py", self.src, "--to-sdr", expect_fail=True)
+
+ def test_color_retag_is_stream_copy(self):
+ out = OUT / "retag.mp4"
+ proc = script("color.py", self.src, "--retag", "bt601", "-o", out)
+ self.assertIn("-c copy", proc.stderr)
+ self.assertEqual(probe(str(out))["video"]["color_transfer"], "smpte170m")
+
+ def test_color_lut(self):
+ lut = OUT / "invert.cube"
+ lines = ["LUT_3D_SIZE 2"]
+ for b in (0, 1):
+ for g in (0, 1):
+ for r in (0, 1):
+ lines.append(f"{1 - r} {1 - g} {1 - b}")
+ lut.write_text("\n".join(lines) + "\n")
+ out = OUT / "lut.mp4"
+ script("color.py", self.src, "--lut", lut, "--lut-strength", "0.5", "--preset", "veryfast", "-o", out)
+ self.assertClose(probe(str(out))["duration"], 12.0, 0.2)
+
+ def test_export_warns_on_hdr(self):
+ out = OUT / "hdr_youtube.mp4"
+ proc = script("export.py", self.hdr, "--preset", "x", "-o", out)
+ self.assertIn("HDR", proc.stderr)
+
+ def test_audio_downmix_voice_and_ducking(self):
+ out = OUT / "downmix.mp4"
+ script("audio.py", self.surround, "--downmix", "--voice", "-o", out)
+ a = probe(str(out))["audio"]
+ self.assertEqual(a["channels"], 2)
+ out2 = OUT / "ducked.mp4"
+ proc = script("audio.py", self.src, "--music", self.long_ref, "--duck", "--fade-out", "2", "-o", out2)
+ self.assertIn("sidechaincompress", proc.stderr)
+ self.assertClose(probe(str(out2))["duration"], 12.0, 0.2)
+ out3 = OUT / "replaced.mp4"
+ script("audio.py", self.src, "--replace", self.mic, "--stereo", "-o", out3)
+ m3 = probe(str(out3))
+ self.assertEqual(m3["audio"]["channels"], 2)
+ self.assertClose(m3["duration"], 12.0, 0.2)
+
+ def test_sync_fine_resolution_and_drift(self):
+ data = json.loads(script("sync.py", self.long_ref, self.long_drift, "--fix-drift", "--json").stdout)
+ self.assertClose(data["offset_seconds"], 1.2, 0.01, "offset extrapolated to t=0 at 1 ms resolution")
+ self.assertClose(data["drift"]["drift_ppm"], 500.0, 40.0)
+ out = OUT / "drift_fixed.wav"
+ script("sync.py", self.long_ref, self.long_drift, "--fix-drift", "--trim-second", "-o", out)
+ again = json.loads(script("sync.py", self.long_ref, out, "--fix-drift", "--json").stdout)
+ self.assertClose(again["offset_seconds"], 0.0, 0.01)
+ self.assertClose(again["drift"]["drift_ppm"], 0.0, 40.0)
+
+ def test_fit_smooth_slow_motion_blend(self):
+ out = OUT / "slow.mp4"
+ script("fit.py", self.src, "--duration", "18", "--smooth", "blend", "--preset", "veryfast", "-o", out)
+ m = probe(str(out))
+ self.assertClose(m["duration"], 18.0, 0.2)
+ self.assertClose(m["video"]["fps"], 30.0, 0.05, "frame rate preserved while slowing down")
+
+ def test_caption_animated_karaoke_ass(self):
+ out = OUT / "karaoke.mp4"
+ ass = OUT / "karaoke.ass"
+ script("caption.py", self.src, "--text", self.cues, "--animate", "pop", "--karaoke", "--write-ass", ass, "--preset", "veryfast", "-o", out)
+ text = ass.read_text(encoding="utf-8-sig")
+ self.assertIn("PlayResX: 1280", text)
+ self.assertIn("\\kf", text)
+ self.assertIn("\\fscx", text)
+ self.assertEqual(text.count("Dialogue:"), 3)
+ self.assertClose(probe(str(out))["duration"], 12.0, 0.2)
+ # from an existing SRT too
+ srt = OUT / "cues.srt"
+ if not srt.exists():
+ script("caption.py", "--text", self.cues, "--write-srt", srt)
+ out2 = OUT / "fade.mp4"
+ script("caption.py", self.src, "--srt", srt, "--animate", "fade", "--preset", "veryfast", "-o", out2)
+ self.assertTrue((OUT / "fade.ass").exists())
+
# ---------------------------------------------------------------- help
def test_every_script_has_help(self):
for name in sorted(p.name for p in SCRIPTS.glob("*.py") if not p.name.startswith("_")):