Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

feat: 1.18.0 — measured analysis and multicam at scale (#246)

Shogo Kajiyama6058a4d84d98
diff --git a/docs/contract.md b/docs/contract.md
index 3c1dc5f..87ad179 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -458,6 +458,18 @@ Per-tool keys added in 1.17.3, all additive:
| `text_unchanged` | `caption.py` | a sibling inside the `caption` block, **burn mode only** (`--mode mux` never touches the text and omits the key): `true` when the drawn text equals the cues that were handed in — nothing transcribed, no cue dropped, no cue **split** across two consecutive cues and no glyph stripped (`--emoji none`). Wrapping, line breaks and timing do not count: the words are the same. This tool never rewrites, shortens or translates a cue, so the key is a statement of what happened, not a judgement of the text |
+Per-tool keys added in 1.18.0, all additive:
+
+| key | tool | what it holds |
+|---|---|---|
+| `shots` | `scenes.py --shots` | `[{start, end, label, flow_magnitude}]` per detected scene, `label` one of `static`/`pan`/`motion` from a lightweight block-matching optical-flow proxy (frames decoded at 4 fps, 48x27, no external dependency). A shot too short to sample two frames is `static` with `flow_magnitude: 0` |
+| `audio_peaks_db` | `scenes.py --audio-peaks` | `[{time, level}]`, measured dBFS loudness peaks. A **new** key: the pre-existing `audio_peaks` (always reported, unrelated unitless RMS figures used for `--highlights` scoring) keeps its 1.0 meaning unchanged |
+| `speech` | `scenes.py --speech` | `[{time, speech_music_ratio}]`, a per-second zero-crossing-rate ratio against the file's own median — a measured proxy, not a speech/music classification |
+| `motion_centre` | `cropdetect.py --motion-centre` | `[{time, x, y, x_frac, y_frac, motion}]` per second, sampled over the same windows as the crop-bar detection. `x`/`y` are source pixels, `x_frac`/`y_frac` a 0..1 fraction of `source_width`/`source_height`; a window with no measured motion reports `x`/`y`/`x_frac`/`y_frac: null`. Report only — this tool never picks a reframe |
+| `speech_aware`, `speech_aware.breaths` | `silence.py --speech-aware` | `{min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}`. `breaths` are the sub-`--min-silence` gaps kept because they sit inside a sentence; the removal list (`silences`, `keep`, `removed_seconds`) already reflects the speech-aware classification. Composes with `--filler` through the same `keep_ranges()`/`merge_spans()` pipeline, so `--speech-aware --filler` produces one removal list |
+| `sources` | `sync.py` | `[{path, offset_s, confidence, drift_ppm}]`, one entry per SOURCE. Present for every run, including the original single-SOURCE shape (where it mirrors the top-level `second`/`offset_seconds`/`confidence` additively). With 2+ SOURCEs it is the *only* per-source shape: there is no top-level `second`/`offset_seconds` because there is no single pair to put there |
+| `switch_mode`, `min_shot` | `multicam.py --switch energy` | `"energy"` and the `--min-shot` value used (default 1.5s), alongside the existing `cuts` (`[[start, end, camera], ...]`) which already carries the camera index for `--edl`'s companion cut list |
+
`check.py` also gains an informational `subtitles` row on **every** platform:
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
does not (or when there are none). Like `channels` and `chapters` it is never
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 549b23a..8332fc2 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -445,3 +445,94 @@ folder" → `batch.py`, "a 60 s highlight" → `scenes.py`), `references/scripts
flags, and SKILL.md has single digits of headroom under its 30,000-byte budget. Only the caption
size got a row, because that one is a *different answer to a request the table already claims to
route*. If eval 18 shows agents missing `--filler` or `--snap beats`, 1.18.0 buys the rows.
+
+**`--shots`' flow measurement is scoped to stay dependency-free and fast: 48x27 grayscale frames
+at 4 fps, a 4x4 grid of block matches, +/-3 px search.** A real optical-flow library was never on
+the table — the zero-dependency rule stdlib+ffmpeg covers every other script, and a Lucas-Kanade
+or dense-flow implementation in pure Python would be both slower and no more honest for what
+`--shots` actually needs, which is three labels, not a per-pixel field. 48x27 keeps a whole shot's
+decode at a few KB (a feature-length input never risks memory the way `--beats`' 22050 Hz PCM
+does), and 4 fps is enough to see whether the frame is panning, static or churning without
+sampling every frame ffmpeg decodes. The zero-shift tie-break in `_block_match` (see the code
+comment) exists because a textureless block — sky, an out-of-focus background, `--shots`' own
+`test_scenes_shots_static_clip_is_labelled_static` fixture — ties every candidate offset on SAD,
+and without an explicit bias towards "no motion" the scan reported the search window's first
+corner as the measured displacement: a still frame read as steady motion in one direction, every
+time. `agreement` (how consistently the 16 blocks agree on direction) is what tells a pan from
+motion-inside-a-static-frame: a camera move shifts the whole picture one way, a subject moving in
+front of a still background does not, and the two look identical in `magnitude` alone.
+
+**`--audio-peaks` writes a new `audio_peaks_db` key rather than changing `audio_peaks`.** The
+unconditional `audio_peaks` list scenes.py has always reported (`[{time, rms}]`) is a different
+measurement in a different unit, used to score `--highlights`; the stability guarantee's "no key
+given a different type" means an explicit request for dBFS-unit peaks needed its own name, not a
+unit change to a key a caller has been reading since 1.0.
+
+**`--speech` reports a ratio, not a label.** A zero-crossing-rate proxy (this window's ZCR over
+the file's own median) says nothing about whether a stretch *is* speech or music — it says speech
+transients cross zero faster than sustained tones, which is true often enough to be a usable
+number and not true often enough to be a classifier. Naming the key `speech_music_ratio` rather
+than `is_speech` keeps the tool on its side of the "analysis tools report numbers, they never
+decide what's interesting" line: the calling agent reads the number and decides what a high or
+low ratio means for its own request, the same way it already reads `--rank-by audio`'s RMS
+figures without this skill calling any of them "the best scene".
+
+**`--speech-aware` composes with `--filler` by feeding the same `keep_ranges()`/`merge_spans()`
+pipeline `--filler` already uses, not a second removal pass.** The composition risk was two
+independent cut lists disagreeing — a breath kept by `--speech-aware` getting cut anyway because
+`--filler` found a word inside it, or the reverse. `speech_aware_silences()` returns two lists
+(sentence-boundary silences to remove, breaths to keep) instead of a single filtered list, so a
+breath is simply never added to the removal side: it is not "removed then added back", which
+would leave a seam. The one designed interaction is a filler word *inside* a kept breath: the
+breath itself is not cut, but `merge_spans()` still unions the filler span into the removal list
+the same way it has always merged a filler word sitting inside a plain silencedetect gap (the
+1.17.0 regression fix), so a real disfluency is still removed even from air `--speech-aware`
+would otherwise keep whole. `BREATH_FLOOR` (0.12 s) exists because plain `detect()` only sees
+gaps at or above `--min-silence`; without a second, shorter silencedetect pass the breaths inside
+a sentence are invisible to begin with, not merely unclassified.
+
+**sync.py's N-source shape adds a new optional `more_sources` positional rather than renaming
+`second` to a list.** The stability guarantee treats an argparse dest as a CLI surface: renaming
+`second` to `sources` (even as a 1+ `nargs`) would have been a rename of a positional the contract
+promises never to rename, and the MCP `inputSchema` derives its property names directly from
+argparse dests (see `_contract.input_schema`), so the rename would have propagated into a removed
+MCP property too. `second` therefore keeps meaning exactly what it always has — the one other
+recording a 2-source call aligns — and `more_sources` (`nargs="*"`, default `[]`) is purely
+additive: omitted, the CLI and its JSON are unchanged from 1.17. `args.sources = [second] +
+more_sources` is built once, right after parsing, so the rest of `main()` (and `measure_one()`,
+factored out of the single-pair drift code that used to live inline) never has to know which
+positional a given source came from.
+
+**Why `sources` carries the single-source measurement too, additively, instead of only appearing
+at N>1.** A caller that standardises on reading `sources` for a multi-camera job should not also
+need a special case for the 2-source call — `sources[0]` is always the same measurement as the
+top-level `offset_seconds`/`confidence` when there is exactly one. The reverse (a flat `second`
+key materialising when there are three sources) is not offered, because there is no single "the"
+second source to put there once there are two or three, and inventing one would be worse than
+leaving the key out.
+
+**`multicam.py --switch energy`'s window and merge rule.** Loudness is measured every 0.25 s (four
+times finer than the default `--min-shot`, so a real cut point is not missed by more than 0.25 s)
+and picking the winner is a plain argmax over cameras with a video stream — the same "biggest
+number wins" rule `scenes.py --rank-by audio` already uses, not a smarter voice-activity model,
+because a second measurement method would be a second thing to keep honest and this one is easy to
+audit from the `cuts` list alone. Folding a run shorter than `--min-shot` into its neighbour
+(rather than, say, discarding it or holding the previous camera) was chosen because it never
+invents a cut that was not there and never drops the fact that a switch was measured at that
+instant — it only refuses to *act* on a switch too brief to be a readable shot. The direction of
+the fold (into the next run, or the previous one if it is the last) is arbitrary in the sense that
+either choice is defensible; it is documented here rather than left to be rediscovered from the
+code, and pinned by `test_multicam_switch_energy_respects_min_shot`.
+
+**The multicam timeline needed no new render.py project stage.** A `--switch` cut list is already
+exactly what `clips[]` expresses: one clip per cut, `src` the camera's own file, `in`/`out` on
+that camera's own timeline (the reference cut, shifted by the camera's measured offset). Inventing
+a `"multicam"` stage type would have meant a second way to say "play this camera from A to B" that
+`render.py` would have to keep in sync with `clips[]` forever; reusing `clips[]` means a switch
+list can be re-rendered with a different `--min-shot` by re-running `multicam.py --switch energy`
+and rewriting the same project's `clips` array, with every other stage (captions, audio, export,
+check) working on the multicam edit exactly as it would on any other project. `--edl` writes the
+plain `cut.py --segments` file (`START-END` per line) rather than a `render.py` project directly,
+because the camera index a project's clips need is already sitting in the JSON `cuts` field
+(`[[start, end, camera], ...]`) — turning that into `clips[]` is a few lines in the calling agent,
+not a new file format this tool would have to maintain.
diff --git a/references/scripts.md b/references/scripts.md
index 6f6552a..d7f1a60 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -161,7 +161,7 @@ rounded, if they aren't.
### cropdetect.py — measure black bars, report the crop rectangle
```
-cropdetect.py INPUT [--seconds N] [--samples N] [--limit F] [--round N]
+cropdetect.py INPUT [--seconds N] [--samples N] [--limit F] [--round N] [--motion-centre]
```
Measurement only -- writes no file. Samples `--samples` windows spread
across the file (default 5, totalling `--seconds` 10s of footage) and
@@ -174,6 +174,14 @@ removing detected bars is wanted -- genuine letterboxed content (a
scope-ratio film in a 16:9 frame) "detects" the same way as accidental
bars; look at the frame before cropping it away.
+`--motion-centre` (1.18) reports the motion centroid once per second, sampled
+across the same windows: `motion_centre: [{time, x, y, x_frac, y_frac, motion}]`,
+`x`/`y` in source pixels and `x_frac`/`y_frac` as a 0..1 fraction of
+`source_width`/`source_height`. A frame with no measured motion in a window
+reports `x`/`y`/`x_frac`/`y_frac: null` and `motion: 0`. Report only, like the
+crop rectangle above -- this hands the calling agent numbers to reframe a 9:16
+crop with; it never picks the subject or the crop box itself.
+
### deinterlace.py — deinterlace interlaced footage
```
deinterlace.py INPUT [--mode frame|field] [--parity auto|tff|bff] [--only-interlaced] [-o OUT]
@@ -465,7 +473,7 @@ each shorter clip's last frame (with silence) out to the longest.
### silence.py — remove dead air / jump cuts
```
-silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [-o OUT]
+silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [--speech-aware] [-o OUT]
silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
[--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
[--transcribe] [--filler-list] [--max-cuts 400]
@@ -531,6 +539,17 @@ additive sibling covering everything that went. A filler word quiet enough to
sit inside a detected silence is merged into it rather than counted twice, so
the two figures can be equal.
+**`--speech-aware` (1.18)** keeps breaths shorter than `--min-silence` when
+they sit inside a sentence, and only cuts at sentence-boundary pauses
+(`--min-silence` or longer). It re-runs silence detection at a much shorter
+floor to measure the short gaps at all, then classifies each one: shorter than
+`--min-silence` is an in-sentence breath (kept, listed in
+`speech_aware.breaths`); `--min-silence` or longer is a sentence boundary (cut,
+same as the plain flag). It composes with `--filler` through the same
+`keep_ranges()`/`merge_spans()` pipeline the filler-inside-a-silence fix
+already uses, so `--speech-aware --filler` produces one removal list. `--edl`
+writes the resulting cut list exactly as it does today.
+
### join.py — concatenate with transitions
```
join.py CLIP1 CLIP2 [...] [--transition fade|dissolve|wipeleft|slideleft|fadeblack|fadewhite|circleopen|none]
@@ -701,6 +720,7 @@ already has a picture.
```
scenes.py INPUT [--threshold 10] [--min-scene 1] [--highlights N [--target SECONDS] [--max-scene 15]] [--edl picks.txt] [--sheet scenes.png] [--json]
scenes.py INPUT --beats [--beat-step 0.01] [--beat-range 60-200] [--min-confidence 0.5] [--json]
+scenes.py INPUT [--shots] [--audio-peaks] [--speech] [--json]
```
Lists scenes with audio energy, the loudest moments, and (with
`--highlights`) proposes N ranges that add up to `--target` seconds, biased to
@@ -751,6 +771,29 @@ would change a file on the strength of it (`cut.py --snap beats`). `--edl` with
`--beats` is unchanged: beats are never written as an EDL, because a beat is
not a cut. No audio stream is a `kind: input` refusal.
+**`--shots`, `--audio-peaks`, `--speech` (1.18)** are three independent
+measurements that combine with each other and with `--beats`/`--highlights`:
+
+- `--shots` labels each already-detected scene `static`/`pan`/`motion` by a
+ lightweight block-matching optical-flow proxy (frames decoded at 4 fps,
+ 48x27, no external dependency), reporting `shots: [{start, end, label,
+ flow_magnitude}]`. A shot too short to sample two frames is `static` with
+ `flow_magnitude: 0` — there is nothing to measure motion between. A proxy,
+ the same spirit as `--rank-by`: it reports what a coarse block match saw,
+ not what is interesting about the shot.
+- `--audio-peaks` reports loudness peaks as measured dBFS: `audio_peaks_db:
+ [{time, level}]`. This is a new key, kept separate from the pre-existing
+ (always-on) `audio_peaks` list, whose entries are an unrelated unitless RMS
+ figure used for `--highlights` scoring — a different unit needed a
+ different key so `audio_peaks`'s meaning does not change underneath a
+ caller reading it since 1.0.
+- `--speech` reports a per-second zero-crossing-rate ratio,
+ `speech: [{time, speech_music_ratio}]` — speech's rapid consonant
+ transients raise the zero-crossing rate; sustained tones (music, room
+ tone) cross zero at a steadier rate. `1.0` means "typical for this file's
+ own median," not an absolute threshold. This is a measured number, not a
+ speech/music classification — nothing here decides which stretch is which.
+
### check.py — pre-delivery compliance
```
check.py INPUT --platform youtube|shorts|reels|tiktok|x|linkedin|facebook|broadcast|podcast|custom [--no-loudness] [--json]
@@ -905,7 +948,8 @@ hand the path to the user together with the numbers.
### multicam.py — align several cameras and switch between them
```
-multicam.py REF CAM2 [CAM3 ...] [--switch "START-END:CAM,..."] | [--auto N] [--audio IDX] [--fix-drift]
+multicam.py REF CAM2 [CAM3 ...] [--switch "START-END:CAM,..."|energy [--min-shot 1.5]] | [--auto N]
+ [--audio IDX] [--fix-drift] [--edl cuts.txt]
[--offsets-only] [--width W --height H --fps N] [-o OUT]
```
All inputs are aligned to the first one by audio (same engine as `sync.py`,
@@ -915,6 +959,23 @@ simply alternates every N seconds. Audio comes from the reference unless
`--audio` picks another input, e.g. an external recorder that has no video.
`--offsets-only` reports offsets and confidence without rendering.
+**`--switch energy` (1.18)** auto-switches to whichever camera (of those with
+a video stream) measures the loudest audio at each 0.25 s window on the
+reference timeline, then folds any run shorter than `--min-shot` (default
+1.5 s) into its neighbour so the cut never lingers on a shot too short to
+read. A measured loudest-camera pick, the same spirit as `scenes.py
+--rank-by audio`: a proxy for who is talking, not a judgement — a loud crowd
+or a hot mic wins over a quiet subject exactly like the scene ranking does.
+`--edl` (1.18, any switch mode) writes the resulting cut list as `cut.py
+--segments`-format `START-END` lines; the camera index for each cut is
+already in the JSON `cuts` field (`[[start, end, camera], ...]`), which is
+what `--edl` leaves out on purpose — one file for an NLE's cut list, one
+field for the camera it came from. The multicam timeline needs no dedicated
+project format: each cut is a `render.py` clip (`{"src": ..., "in": ...,
+"out": ...}` on that camera's own timeline, shifted by its measured offset),
+so a switch list can be re-rendered with different `--min-shot` values by
+editing `project.json`'s `clips`, not by inventing a second timeline schema.
+
### verify.py — real-footage verification kit
```
verify.py FILES_OR_FOLDERS [--quick] [--report verify.md] [--out DIR --keep] [--seconds 6] [--json]
@@ -1185,18 +1246,18 @@ draws whatever glyph the text font has; `--emoji none` strips them).
### sync.py — offset detection, alignment, drift correction
```
-sync.py REFERENCE SECOND [--json] [--max-offset 30] [--analyze-seconds 120] [--fix-drift [--drift-window 60]]
- [--replace-audio | --trim-second] [-o OUT]
+sync.py REFERENCE SOURCE [SOURCE ...] [--json] [--max-offset 30] [--analyze-seconds 120]
+ [--fix-drift [--drift-window 60]] [--replace-audio | --trim-second] [-o OUT]
```
Cross-correlates loudness envelopes: coarse FFT search (20 ms), then a direct
1 ms refinement (pure Python, a 2-minute window takes ~1-3 s). Positive offset
-= the second recording started later. `--replace-audio` writes the reference
-video with the second file's audio aligned (video stream copied); the output
-keeps the reference's full length -- a shorter or head-trimmed second file is
+= the source recording started later. `--replace-audio` writes the reference
+video with the source file's audio aligned (video stream copied); the output
+keeps the reference's full length -- a shorter or head-trimmed source file is
padded with silence, never allowed to cut the picture.
-`--trim-second` writes the second file shifted to the reference timeline.
+`--trim-second` writes the source file shifted to the reference timeline.
`--fix-drift` measures the offset again near the end of the overlap, reports
-the clock difference in ppm, and resamples the second file so a 60-minute
+the clock difference in ppm, and resamples the source file so a 60-minute
take stays in sync (typical consumer devices drift 20-500 ppm = up to 1.8 s/h).
Use it whenever the recording is longer than ~10 minutes. Check `confidence`
(0–1, normalised correlation with a runner-up penalty); below 0.3 the match is
@@ -1206,6 +1267,20 @@ window 95 %, misses flagged below 0.3. Keep `--analyze-seconds` at least 4×
`--max-offset` (default 120 s vs 30 s): lags with under 35 % overlap are
ignored, so an offset larger than ~60 % of the window cannot be found.
+**One reference, 1+ SOURCE (1.18).** With exactly one SOURCE the CLI keeps its
+original shape byte for byte -- `second`, `offset_seconds`, `confidence`,
+`meaning`, `drift` at the top level, `--replace-audio`/`--trim-second`
+available -- with the same measurement additively mirrored under a new
+`sources: [{path, offset_s, confidence, drift_ppm}]` list. With 2+ SOURCEs the
+result is one offsets JSON, `{reference, sources: [...]}`, and no flat
+`second`/`offset_seconds` (there is no single pair to put there);
+`--replace-audio`/`--trim-second` refuse with `kind: input`, since each writes
+one synced output and there is more than one source to choose from. This is
+the same offset/drift measurement `multicam.py` already loops over per camera
+internally (it imports `sync.measure_offset`); `sync.py`'s own N-source CLI
+now exposes that loop directly, for aligning cameras without cutting between
+them.
+
### color.py — HDR to SDR, LUTs, colour tags, Dolby Vision
```
color.py INPUT --to-sdr [--tonemap hable|mobius|reinhard|bt2390] [--peak 1000] [--desat 0] [-o OUT]
diff --git a/scripts/_common/__init__.py b/scripts/_common/__init__.py
index 853ea65..ae77739 100644
--- a/scripts/_common/__init__.py
+++ b/scripts/_common/__init__.py
@@ -68,12 +68,13 @@ from _common.emit import (
)
from _common.probe import (
analyze_levels, _aspect_string, _bit_depth, decode_pcm_mono, detect_scenes, detect_silences, fingerprint,
- _fraction, keyframes_near, SCORE_RE, SIL_RE,
+ _fraction, keyframes_near, SCORE_RE, SIL_RE, decode_gray_frames,
measured_level_dbfs, MEDIA_EXT, _output_failed, probe, rms_envelope, _to_float, _to_int, verify_output
)
from _common.decision import (
aac_args, add_pad_fill_args, audio_codec_for, AUDIO_CODECS, brand_caption_style, BRAND_DEFAULTS,
description_block, _evidence_rank, fmt_chapter_time, propose_chapters,
+ frame_flow, label_shot_flow, MOTION_GRID, MOTION_SEARCH, MOTION_STATIC_PX, MOTION_PAN_SPREAD,
filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS, FILLER_MAX_WORD,
FILLER_MIN_GAP, FILLER_PAD, normalise_filler_token,
beat_grid, snap_points, BEAT_MIN_CONFIDENCE, BEAT_ONSET_K, BEAT_OCTAVE_MARGIN,
@@ -205,6 +206,7 @@ __all__ = [
"place_output", "_plan_at_exit", "_plan_inputs", "_PLAN_STRIP", "PLAN_VERSION", "PREFERRED_FAMILIES",
"print_json", "probe", "PROBE_TIMEOUT", "_progress_line", "read_text_or_die", "refuse_output_is_input",
"_remember_output", "require_tool", "resolve_emoji_assets", "_result_v2", "rms_envelope", "run", "run_analysis",
+ "decode_gray_frames", "frame_flow", "label_shot_flow", "MOTION_GRID", "MOTION_SEARCH", "MOTION_STATIC_PX", "MOTION_PAN_SPREAD",
"_run_captured", "run_keeping_subtitles", "run_tool", "_run_with_progress", "_SCRIPT_FONT_CACHE",
"_script_font_entry", "script_font_for_text", "script_font_status", "_script_font_uncached", "_SCRIPT_RANGES",
"SCRIPTS", "_sdr_bt709", "_set_current_ctx", "_SHAPING_BUILD_CACHE", "SHAPING_SCRIPTS", "shell_quote",
diff --git a/scripts/_common/decision.py b/scripts/_common/decision.py
index 8afdbb4..e990ec2 100644
--- a/scripts/_common/decision.py
+++ b/scripts/_common/decision.py
@@ -864,3 +864,89 @@ def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = F
m["start"] = round(m["start"], 4)
m["end"] = round(m["end"], 4)
return merged
+
+
+# --- 1.18.0: lightweight block-matching motion estimate (scenes.py --shots, cropdetect.py
+# --motion-centre) ---------------------------------------------------------------------------
+
+MOTION_GRID = 4 # NxN anchor blocks per frame
+MOTION_SEARCH = 3 # +/- pixels searched per block, at the decoded (low) resolution
+MOTION_STATIC_PX = 0.35 # average per-frame displacement below this, at decode resolution, is "static"
+MOTION_PAN_SPREAD = 0.6 # block-to-block direction agreement above this (0..1) reads as a pan
+
+
+def _block_match(prev: bytes, cur: bytes, w: int, h: int, cx: int, cy: int, half: int, search: int) -> "Tuple[float, float]":
+ """(dx, dy) that best aligns a `half*2` square centred at (cx, cy) in `prev` to `cur`,
+ searched over +/- `search` px by sum-of-absolute-differences. Coordinates and the returned
+ offset are in decoded-frame pixels (a handful of pixels a side at 1.18.0's sample size)."""
+ x0, y0 = max(half, min(w - half - 1, cx)), max(half, min(h - half - 1, cy))
+ ref = [prev[(y0 + dy) * w + (x0 + dx)] for dy in range(-half, half + 1) for dx in range(-half, half + 1)]
+
+ def sad_at(xx: int, yy: int) -> "Optional[int]":
+ if xx - half < 0 or xx + half >= w or yy - half < 0 or yy + half >= h:
+ return None
+ sad = 0
+ for dy in range(-half, half + 1):
+ row = (yy + dy) * w
+ for dx in range(-half, half + 1):
+ sad += abs(ref[(dy + half) * (2 * half + 1) + (dx + half)] - cur[row + xx + dx])
+ return sad
+
+ # Zero shift is the tie-break candidate, not the search order's first cell: on a textureless
+ # block (a flat colour, sky, an out-of-focus background) every offset scores the same SAD, and
+ # without an explicit tie towards "no motion" the scan used to report the search window's
+ # first corner as the measured displacement -- a still frame with nothing to match against
+ # read as steady motion in one direction, every time.
+ best_sad, best = sad_at(x0, y0), (0.0, 0.0)
+ if best_sad is None:
+ best_sad = float("inf")
+ for sy in range(-search, search + 1):
+ for sx in range(-search, search + 1):
+ if sx == 0 and sy == 0:
+ continue
+ sad = sad_at(x0 + sx, y0 + sy)
+ if sad is not None and sad < best_sad:
+ best_sad, best = sad, (float(sx), float(sy))
+ return best
+
+
+def frame_flow(prev: bytes, cur: bytes, w: int, h: int, *, grid: int = MOTION_GRID,
+ search: int = MOTION_SEARCH) -> "Dict[str, Any]":
+ """One measurement between two consecutive decoded grayscale frames: the mean block
+ displacement vector, its magnitude, and how consistently the blocks agree on direction
+ (0 = every block moved a different way, 1 = every block agrees -- a pan or dolly moves the
+ whole frame one way, on-screen motion inside a mostly-static frame does not)."""
+ half = max(1, min(w, h) // (grid * 3))
+ vecs: "List[Tuple[float, float]]" = []
+ for gy in range(grid):
+ for gx in range(grid):
+ cx = int((gx + 0.5) * w / grid)
+ cy = int((gy + 0.5) * h / grid)
+ vecs.append(_block_match(prev, cur, w, h, cx, cy, half, search))
+ mdx = sum(v[0] for v in vecs) / len(vecs)
+ mdy = sum(v[1] for v in vecs) / len(vecs)
+ magnitude = math.hypot(mdx, mdy)
+ mean_len = sum(math.hypot(*v) for v in vecs) / len(vecs)
+ agreement = (magnitude / mean_len) if mean_len > 1e-6 else 1.0 # 1.0 = every block agrees
+ return {"dx": mdx, "dy": mdy, "magnitude": magnitude, "agreement": min(1.0, agreement)}
+
+
+def label_shot_flow(flows: "Sequence[Dict[str, Any]]") -> "Dict[str, Any]":
+ """{label, flow_magnitude} for one shot from its per-frame-pair flow measurements.
+
+ static: mean displacement below MOTION_STATIC_PX. pan: above it, and blocks agree on
+ direction (a camera move shifts the whole frame). motion: above it, blocks disagree (motion
+ inside an otherwise still frame -- handheld jitter, or a subject moving across a static
+ background). This is a measured proxy, the same spirit as scenes.py --rank-by: it reports
+ what a coarse block match saw, not what is interesting about the shot."""
+ if not flows:
+ return {"label": "static", "flow_magnitude": 0.0}
+ magnitude = sum(f["magnitude"] for f in flows) / len(flows)
+ agreement = sum(f["agreement"] for f in flows) / len(flows)
+ if magnitude < MOTION_STATIC_PX:
+ label = "static"
+ elif agreement >= MOTION_PAN_SPREAD:
+ label = "pan"
+ else:
+ label = "motion"
+ return {"label": label, "flow_magnitude": round(magnitude, 3)}
diff --git a/scripts/_common/probe.py b/scripts/_common/probe.py
index aa1559f..3c58a3d 100644
--- a/scripts/_common/probe.py
+++ b/scripts/_common/probe.py
@@ -77,6 +77,32 @@ MEDIA_EXT = {".mp4", ".mov", ".mkv", ".webm", ".m4v", ".avi", ".ts", ".mts", ".m
".wav", ".flac", ".mp3", ".m4a", ".aac", ".ogg", ".opus", ".aif", ".aiff", ".caf", ".wma", ".png", ".jpg", ".jpeg", ".webp"}
+def decode_gray_frames(path: str, fps: float, width: int, height: int, *, start: float = 0.0,
+ seconds: "Optional[float]" = None, check: bool = True) -> "List[bytes]":
+ """Decode `path` to raw 8-bit grayscale frames at a low `fps`/`width`x`height`, one ffmpeg
+ pass to stdout. Used by scenes.py --shots (per-shot flow label) and cropdetect.py
+ --motion-centre (motion centroid): both need pixel data, not a filter's own summary number,
+ but at 1.18.0's resolutions (tens of pixels a side, a few fps) a whole shot is a few KB, so
+ piping raw frames through Python stays cheap and needs no extra dependency."""
+ ffmpeg = require_tool("ffmpeg")
+ cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin"]
+ if start:
+ cmd += ["-ss", f"{start:.3f}"]
+ cmd += ["-i", path]
+ if seconds is not None:
+ cmd += ["-t", f"{max(0.0, seconds):.3f}"]
+ cmd += ["-vf", f"fps={fps:g},scale={width}:{height}:flags=area,format=gray",
+ "-f", "rawvideo", "-"]
+ proc = run_analysis(cmd, check=False, text=False)
+ if proc.returncode != 0 or not proc.stdout:
+ if check:
+ die(f"could not decode frames from {path}:\n{proc.stderr.decode(errors='replace').strip()}", kind="ffmpeg")
+ return []
+ frame_size = width * height
+ data = proc.stdout
+ return [data[i:i + frame_size] for i in range(0, len(data) - frame_size + 1, frame_size)]
+
+
def _output_failed(path: str, why: str) -> "None":
"""An ffmpeg run reported success but the artifact is not usable: say so, and do not leave a
0-byte file behind that a later step could mistake for a result."""
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 42dea62..2a6bb52 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -411,11 +411,16 @@ def output_schema(name: str, meta: Dict[str, Any]) -> Dict[str, Any]:
extra = {"audiogram": {"type": "object", "description": "{style, background ('image' or 'color'), image, position, vis_height, platform, captions, title, stages, verified} -- present on every run, so a plain waveform answers background 'color' (1.16)"},
"notes": {"type": "array", "items": {"type": "string"}}}
elif name == "scenes":
- extra = {"file": {"type": "string"}, "duration": {"type": "number"}, "scene_count": {"type": "integer"}, "scenes": {"type": "array"}, "audio_peaks": {"type": "array"}}
+ extra = {"file": {"type": "string"}, "duration": {"type": "number"}, "scene_count": {"type": "integer"}, "scenes": {"type": "array"}, "audio_peaks": {"type": "array"},
+ "shots": {"type": "array", "description": "--shots (1.18): [{start, end, label, flow_magnitude}]"},
+ "audio_peaks_db": {"type": "array", "description": "--audio-peaks (1.18): [{time, level}] measured dBFS, distinct from audio_peaks above"},
+ "speech": {"type": "array", "description": "--speech (1.18): [{time, speech_music_ratio}]"}}
elif name == "silence":
- extra = {"silences": {"type": "array"}, "keep": {"type": "array"}, "input_duration": {"type": "number"}, "kept_duration": {"type": "number"}, "removed_seconds": {"type": "number"}}
+ extra = {"silences": {"type": "array"}, "keep": {"type": "array"}, "input_duration": {"type": "number"}, "kept_duration": {"type": "number"}, "removed_seconds": {"type": "number"},
+ "speech_aware": {"type": "object", "description": "--speech-aware (1.18): {min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}"}}
elif name == "sync":
- extra = {"reference": {"type": "string"}, "second": {"type": "string"}, "offset_seconds": {"type": "number"}, "confidence": {"type": "number"}, "meaning": {"type": "string"}, "drift": {"type": "object"}}
+ extra = {"reference": {"type": "string"}, "second": {"type": "string"}, "offset_seconds": {"type": "number"}, "confidence": {"type": "number"}, "meaning": {"type": "string"}, "drift": {"type": "object"},
+ "sources": {"type": "array", "description": "1.18: [{path, offset_s, confidence, drift_ppm}], one per SOURCE; the only per-source shape once more than one SOURCE is given"}}
elif name == "look":
extra = {"outputs": {"type": "array", "items": {"type": "string"}}}
elif name == "render":
diff --git a/scripts/cropdetect.py b/scripts/cropdetect.py
index 0cf8e2a..bd5ea26 100644
--- a/scripts/cropdetect.py
+++ b/scripts/cropdetect.py
@@ -33,7 +33,10 @@ import sys
from collections import Counter
from typing import Dict, List, Tuple
-from _common import add_common, apply_common, die, emit, info, print_json, probe, require_tool, run_analysis
+from _common import add_common, apply_common, die, emit, info, print_json, probe, require_tool, run_analysis, decode_gray_frames, frame_flow
+
+MOTION_CENTRE_FPS = 2.0
+MOTION_CENTRE_W, MOTION_CENTRE_H = 64, 36
CROP_RE = re.compile(r"crop=(\d+):(\d+):(\d+):(\d+)")
@@ -61,6 +64,52 @@ def detect(path: str, seconds: float, samples: int, limit: float, round_to: int,
return rects
+def motion_centre(path: str, seconds: float, samples: int, duration: float, sw: int, sh: int) -> List[Dict]:
+ """Per-second motion centroid: {time, x, y, x_frac, y_frac} -- a report-only measurement,
+ the same sampled-window approach as detect() above. x/y are pixel coordinates in the SOURCE
+ frame (consistent with the --round crop rectangle this tool already reports in source
+ pixels); x_frac/y_frac are the same position as a 0..1 fraction of source_width/height, for a
+ caller that wants to reframe without first knowing the source size. This never picks a
+ subject -- it reports where in the frame the measured pixel motion was concentrated, which is
+ not the same thing as where the interesting subject is (a moving background behind a still
+ speaker centres the motion on the background)."""
+ per_window = max(0.5, seconds / max(1, samples))
+ cell = 8 # NxN diff grid at decode resolution
+ cw, ch = MOTION_CENTRE_W / cell, MOTION_CENTRE_H / cell
+ out: List[Dict] = []
+ for i in range(samples):
+ start = 0.0 if duration <= 0 else (duration - per_window) * i / max(1, samples - 1) if samples > 1 else 0.0
+ start = max(0.0, start)
+ frames = decode_gray_frames(path, MOTION_CENTRE_FPS, MOTION_CENTRE_W, MOTION_CENTRE_H,
+ start=start, seconds=per_window, check=False)
+ for k in range(len(frames) - 1):
+ prev, cur = frames[k], frames[k + 1]
+ wsum = wx = wy = 0.0
+ for gy in range(cell):
+ for gx in range(cell):
+ x0, x1 = int(gx * cw), int((gx + 1) * cw)
+ y0, y1 = int(gy * ch), int((gy + 1) * ch)
+ diff = 0
+ for y in range(y0, y1):
+ row = y * MOTION_CENTRE_W
+ for x in range(x0, x1):
+ diff += abs(prev[row + x] - cur[row + x])
+ cx = (gx + 0.5) / cell
+ cy = (gy + 0.5) / cell
+ wsum += diff
+ wx += diff * cx
+ wy += diff * cy
+ t = start + (k + 1) / MOTION_CENTRE_FPS
+ if wsum <= 0:
+ out.append({"time": round(t, 2), "x": None, "y": None, "x_frac": None, "y_frac": None, "motion": 0.0})
+ continue
+ xf, yf = wx / wsum, wy / wsum
+ out.append({"time": round(t, 2), "x": round(xf * sw), "y": round(yf * sh),
+ "x_frac": round(xf, 3), "y_frac": round(yf, 3), "motion": round(wsum / (MOTION_CENTRE_W * MOTION_CENTRE_H), 2)})
+ out.sort(key=lambda r: r["time"])
+ return out
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("input")
@@ -68,6 +117,10 @@ def main() -> int:
ap.add_argument("--samples", type=int, default=5, help="number of windows spread across the file (default 5)")
ap.add_argument("--limit", type=float, default=0.0941176, help="black-pixel threshold, 0..1 (default ~0.094, cropdetect's own default)")
ap.add_argument("--round", type=int, default=16, dest="round_to", help="the reported width/height are rounded to a multiple of this (default 16)")
+ ap.add_argument("--motion-centre", action="store_true",
+ help="report the motion centroid per second, sampled the same way as the crop "
+ "detection above (report only -- this tool never picks a reframe, it hands "
+ "the calling agent numbers to reframe with)")
add_common(ap)
args = ap.parse_args()
apply_common(args)
@@ -102,6 +155,11 @@ def main() -> int:
info(f"detected crop={w}:{h}:{x}:{y} (source {sw}x{sh}, confidence {result['confidence']:.0%}) -- "
f"crop.py {args.input} --x {x} --y {y} --width {w} --height {h}")
+ if args.motion_centre:
+ centre = motion_centre(args.input, args.seconds, args.samples, duration, sw, sh)
+ result["motion_centre"] = centre
+ info(f"--motion-centre: {len(centre)} measured points")
+
if args.json:
emit(None, **result)
else:
diff --git a/scripts/multicam.py b/scripts/multicam.py
index 2104abe..317a813 100755
--- a/scripts/multicam.py
+++ b/scripts/multicam.py
@@ -26,12 +26,16 @@ Examples:
python3 multicam.py camA.mp4 camB.mp4 --auto 8 -o edit.mp4 # alternate cameras every 8 s
"""
import argparse
+import math
import sys
-from typing import List, Tuple
+from typing import Dict, List, Tuple
-from _common import video_args, aac_args, add_common, apply_common, default_output, die, emit, ffmpeg_base, info, time_arg, probe, run, x264_args, X264_PRESETS, fmt_secs
+from _common import video_args, aac_args, add_common, apply_common, default_output, die, emit, ffmpeg_base, info, time_arg, probe, run, x264_args, X264_PRESETS, fmt_secs, decode_pcm_mono, rms_envelope, STATE
from sync import measure_offset
+ENERGY_WINDOW_S = 0.25 # --switch energy: loudness measured over this window on the reference timeline
+ENERGY_RATE = 8000 # decode rate for the per-camera loudness envelope
+
def parse_switch(spec: str, n: int) -> List[Tuple[float, float, int]]:
out = []
@@ -55,12 +59,75 @@ def parse_switch(spec: str, n: int) -> List[Tuple[float, float, int]]:
return out
+def energy_switch(inputs: List[str], metas: List["Dict"], offsets: List[float], ratios: List[float],
+ ref_dur: float, min_shot: float, window_s: float = ENERGY_WINDOW_S) -> List[Tuple[float, float, int]]:
+ """Auto-switch cuts: at every `window_s` step on the reference timeline, cut to whichever
+ camera (of those WITH video) measures the loudest audio at that moment, then merge the
+ winners into runs and fold any run shorter than `min_shot` into its neighbour. This is a
+ measured loudest-camera pick, the same spirit as scenes.py --rank-by audio: it is a proxy for
+ "who is talking", not a judgement -- a loud crowd or a camera with a hot mic wins over a
+ quiet subject exactly like scenes.py's own audio ranking does."""
+ candidates = [c for c, m in enumerate(metas) if m.get("video")]
+ if not candidates:
+ die("no input has video; --switch energy needs at least one camera with a picture")
+ envs: "Dict[int, List[float]]" = {}
+ step = max(1, int(ENERGY_RATE * window_s))
+ for c in candidates:
+ samples = decode_pcm_mono(inputs[c], ENERGY_RATE, check=False)
+ envs[c] = rms_envelope(samples, step) if samples else []
+ n_windows = max(1, int(math.ceil(ref_dur / window_s)))
+ winners: List[int] = []
+ for i in range(n_windows):
+ t = i * window_s
+ best_c, best_v = candidates[0], -1.0
+ for c in candidates:
+ src_t = (t - offsets[c]) * ratios[c]
+ idx = int(src_t / window_s)
+ env = envs[c]
+ val = env[idx] if 0 <= idx < len(env) else -1.0
+ if val > best_v:
+ best_v, best_c = val, c
+ winners.append(best_c)
+ # collapse into runs
+ runs: List[List] = []
+ for i, c in enumerate(winners):
+ t0, t1 = i * window_s, min(ref_dur, (i + 1) * window_s)
+ if runs and runs[-1][2] == c:
+ runs[-1][1] = t1
+ else:
+ runs.append([t0, t1, c])
+ # fold a run shorter than min_shot into its neighbour: the next run if there is one, else the
+ # previous one -- a single pass is enough because folding only ever lengthens a run, and a
+ # run just extended is re-checked on the next iteration through the while loop.
+ changed = True
+ while changed and len(runs) > 1:
+ changed = False
+ for i, r in enumerate(runs):
+ if r[1] - r[0] < min_shot:
+ if i + 1 < len(runs):
+ runs[i + 1][0] = r[0]
+ else:
+ runs[i - 1][1] = r[1]
+ del runs[i]
+ changed = True
+ break
+ return [(s, e, c) for s, e, c in runs]
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("inputs", nargs="+", help="reference camera first, then other cameras / recorders")
ap.add_argument("-o", "--output", help="output file (default: <reference>_multicam.mp4)")
- ap.add_argument("--switch", help="switch list START-END:CAM,... on the reference timeline")
+ ap.add_argument("--switch", help="switch list START-END:CAM,... on the reference timeline, or "
+ "the literal 'energy' to auto-switch to whichever camera is "
+ "loudest at each moment (respecting --min-shot)")
ap.add_argument("--auto", type=float, help="no switch list: alternate through the cameras every N seconds")
+ ap.add_argument("--min-shot", type=float, default=1.5,
+ help="--switch energy: never cut to a new camera for less than this many "
+ "seconds (default 1.5)")
+ ap.add_argument("--edl", help="write the resulting cut list to this file, one START-END per line "
+ "(cut.py --segments format); the camera index for each cut is in "
+ "the JSON `cuts` field alongside it")
ap.add_argument("--audio", type=int, default=0, help="input index to take audio from (default 0 = reference)")
ap.add_argument("--offsets-only", action="store_true", help="print the measured offsets and exit")
ap.add_argument("--max-offset", type=float, default=30.0)
@@ -137,8 +204,13 @@ def main() -> int:
print(f"{p}: {o:+.3f}s (confidence {c:.2f})")
return 0
+ if args.min_shot <= 0:
+ die(f"--min-shot must be positive, got {args.min_shot:g}")
ref_dur = metas[0]["duration"] or 0.0
- if args.switch:
+ energy_mode = args.switch == "energy"
+ if energy_mode:
+ cuts = energy_switch(args.inputs, metas, offsets, ratios, ref_dur, args.min_shot)
+ elif args.switch:
cuts = parse_switch(args.switch, n)
elif args.auto:
if args.auto <= 0:
@@ -167,6 +239,13 @@ def main() -> int:
if not metas[c].get("video"):
die(f"camera {c} ({args.inputs[c]}) has no video; it can only be used with --audio")
+ if args.edl:
+ if not STATE.dry_run:
+ with open(args.edl, "w", encoding="utf-8") as fh:
+ for s, e, _c in filled:
+ fh.write(f"{s:.3f}-{e:.3f}\n")
+ info(f"wrote {args.edl}")
+
v0 = metas[0]["video"]
w, h = args.width or v0["width"], args.height or v0["height"]
if v0.get("rotation") in (90, -90, 270, -270) and not (args.width or args.height):
@@ -212,7 +291,8 @@ def main() -> int:
run(cmd)
r = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(r['duration'])}, {len(filled)} cuts, audio from input {a})")
- emit(output, cuts=[[round(s, 3), round(e, 3), c] for s, e, c in filled], **report)
+ extra = {"switch_mode": "energy", "min_shot": args.min_shot} if energy_mode else {}
+ emit(output, cuts=[[round(s, 3), round(e, 3), c] for s, e, c in filled], **report, **extra)
return 0
diff --git a/scripts/scenes.py b/scripts/scenes.py
index 91aff53..088664b 100755
--- a/scripts/scenes.py
+++ b/scripts/scenes.py
@@ -25,7 +25,11 @@ from typing import Dict, List, Optional, Tuple
# `detect_scenes` moved into _common/probe.py in 1.16.0 (see silence.py); the body is unchanged.
from _common import (detect_scenes, STATE, add_common, apply_common, beat_grid, default_font_file, die, emit,
escape_filter_path, ffmpeg_base, info, print_json, probe, run, decode_pcm_mono,
- rms_envelope, BEAT_MIN_CONFIDENCE)
+ rms_envelope, BEAT_MIN_CONFIDENCE, decode_gray_frames, frame_flow, label_shot_flow)
+
+SHOT_FPS = 4.0 # frames/second sampled for --shots' flow estimate
+SHOT_W, SHOT_H = 48, 27 # decode size for --shots (16:9-ish; enough blocks, still a few KB/shot)
+SPEECH_STEP_S = 1.0 # --speech energy-ratio window
@@ -54,6 +58,51 @@ def parse_beat_range(text: str) -> "tuple":
return (lo, hi)
+def shot_flow_label(path: str, start: float, end: float) -> Dict:
+ """{start, end, label, flow_magnitude} for one shot: decode it at SHOT_FPS/SHOT_W x SHOT_H
+ and block-match consecutive frames (see _common.decision.frame_flow / label_shot_flow). A
+ shot under two sampled frames has nothing to compare and is reported static with
+ flow_magnitude 0 -- there is no motion measurement to make on a single frame."""
+ frames = decode_gray_frames(path, SHOT_FPS, SHOT_W, SHOT_H, start=start, seconds=max(0.0, end - start), check=False)
+ flows = [frame_flow(frames[i], frames[i + 1], SHOT_W, SHOT_H) for i in range(len(frames) - 1)]
+ label = label_shot_flow(flows)
+ return {"start": round(start, 3), "end": round(end, 3), "label": label["label"], "flow_magnitude": label["flow_magnitude"]}
+
+
+def audio_peaks_db(samples: "List[float]", rate: int, step_s: float = 0.25) -> List[Dict]:
+ """--audio-peaks: local maxima of the loudness envelope reported as measured dBFS, not the
+ unitless RMS scenes.py has always put in the (unconditional) `audio_peaks` key -- a
+ different unit needs a different key so the existing one keeps meaning what it always has."""
+ env = rms_envelope(samples, int(rate * step_s))
+ peaks = []
+ for i, val in enumerate(env):
+ if val <= 1e-6:
+ continue
+ if (i == 0 or env[i - 1] <= val) and (i == len(env) - 1 or env[i + 1] <= val):
+ level = round(20 * math.log10(val), 1)
+ peaks.append({"time": round(i * step_s, 2), "level": level})
+ return peaks
+
+
+def speech_music_ratio(samples: "List[float]", rate: int, step_s: float = SPEECH_STEP_S) -> List[Dict]:
+ """--speech: a per-second zero-crossing-rate ratio, reported as a measured number, not a
+ speech/music label. Speech's rapid consonant transients drive the zero-crossing rate up and
+ make it jump window to window; sustained tones (music, a held note, room tone) cross zero at
+ a steadier rate. ratio = this window's ZCR / the file's median ZCR, so 1.0 is "typical for
+ this file" regardless of its overall noisiness -- a proxy, in the same spirit as scenes.py's
+ --rank-by, not a classifier: nothing here decides what is speech."""
+ step = max(1, int(rate * step_s))
+ zcrs: List[float] = []
+ for i in range(0, len(samples) - step + 1, step):
+ block = samples[i:i + step]
+ crossings = sum(1 for a, b in zip(block, block[1:]) if (a >= 0) != (b >= 0))
+ zcrs.append(crossings / max(1, len(block) - 1))
+ if not zcrs:
+ return []
+ med = sorted(zcrs)[len(zcrs) // 2] or 1e-9
+ return [{"time": round(i * step_s, 2), "speech_music_ratio": round(z / med, 3)} for i, z in enumerate(zcrs)]
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("input")
@@ -76,6 +125,15 @@ def main() -> int:
ap.add_argument("--min-confidence", type=float, default=BEAT_MIN_CONFIDENCE,
help="with --beats: below this confidence the grid is still reported, marked "
f"usable: false (default {BEAT_MIN_CONFIDENCE})")
+ ap.add_argument("--shots", action="store_true",
+ help="label each detected shot static / pan / motion by a measured optical-flow "
+ "proxy (lightweight block matching over sampled frames); reports flow_magnitude too")
+ ap.add_argument("--audio-peaks", action="store_true",
+ help="report loudness peaks as {time, level} in measured dBFS (separate from the "
+ "always-on `audio_peaks` RMS list used to pick --highlights)")
+ ap.add_argument("--speech", action="store_true",
+ help="report a per-second speech-vs-music energy ratio (zero-crossing-rate proxy, "
+ "a measurement, not a speech/music classification)")
ap.add_argument("--sheet", help="write a contact sheet PNG with the first frame of every scene")
ap.add_argument("--no-timecode", action="store_true", help="--sheet without the burnt-in timecode stamp (a way out if drawtext itself is unusable, see doctor)")
add_common(ap)
@@ -100,7 +158,8 @@ def main() -> int:
# With --beats the file is decoded once, at the finer rate, and both envelopes come from that
# one pass: the 0.5 s scene blocks are an exact multiple of the 10 ms onset blocks.
beat_rate = 22050
- fine_samples = decode_pcm_mono(args.input, beat_rate, check=False) if (args.beats and meta.get("audio")) else None
+ need_fine = args.beats or args.audio_peaks or args.speech
+ fine_samples = decode_pcm_mono(args.input, beat_rate, check=False) if (need_fine and meta.get("audio")) else None
if fine_samples is not None:
env = audio_envelope(args.input, step_s, rate=beat_rate, samples=fine_samples)
else:
@@ -128,6 +187,32 @@ def main() -> int:
result: Dict = {"file": args.input, "duration": round(dur, 3), "scene_count": len(scenes), "scenes": scenes, "audio_peaks": peaks}
info(f"{len(scenes)} scenes, {len(peaks)} audio peaks over {dur:.1f}s")
+ if args.shots:
+ shots = [shot_flow_label(args.input, s, e) for s, e in zip(bounds[:-1], bounds[1:]) if e - s > 0.05]
+ result["shots"] = shots
+ counts = {}
+ for sh in shots:
+ counts[sh["label"]] = counts.get(sh["label"], 0) + 1
+ info(f"--shots: {len(shots)} shots (" + ", ".join(f"{v} {k}" for k, v in sorted(counts.items())) + ")")
+
+ if args.audio_peaks:
+ if not meta.get("audio"):
+ result["audio_peaks_db"] = []
+ info("--audio-peaks: no audio stream, nothing to measure")
+ else:
+ db_samples = fine_samples if fine_samples is not None else decode_pcm_mono(args.input, 22050, check=False)
+ result["audio_peaks_db"] = audio_peaks_db(db_samples, 22050 if fine_samples is not None else 22050)
+ info(f"--audio-peaks: {len(result['audio_peaks_db'])} peaks")
+
+ if args.speech:
+ if not meta.get("audio"):
+ result["speech"] = []
+ info("--speech: no audio stream, nothing to measure")
+ else:
+ sp_samples = fine_samples if fine_samples is not None else decode_pcm_mono(args.input, 22050, check=False)
+ result["speech"] = speech_music_ratio(sp_samples, 22050)
+ info(f"--speech: {len(result['speech'])} one-second windows")
+
if args.beats:
# A beat grid is a measurement of the music's periodicity, not a statement about where a
# cut belongs. scenes.py reports what it measured, including a low confidence: reporting a
diff --git a/scripts/silence.py b/scripts/silence.py
index 4728fc7..c909c02 100755
--- a/scripts/silence.py
+++ b/scripts/silence.py
@@ -42,6 +42,32 @@ def merge_spans(spans: List[Tuple[float, float]]) -> List[Tuple[float, float]]:
return out
+BREATH_FLOOR = 0.12 # shortest gap silencedetect is asked for under --speech-aware
+
+
+def speech_aware_silences(path: str, threshold: float, min_silence: float) -> "Tuple[List[Tuple[float, float]], List[Tuple[float, float]]]":
+ """(sentence-boundary silences to remove, breaths kept) for --speech-aware.
+
+ Re-runs the same silencedetect() already used everywhere else in this file, but with a much
+ shorter minimum duration, so short in-sentence breaths are measured at all -- the plain
+ `detect()` call above never sees them because its own --min-silence floor filters them out
+ before they reach Python. Every gap silencedetect measured that is still shorter than
+ --min-silence sits *inside* a sentence -- the two speech-flagged stretches on either side of
+ it are close together in time because nothing longer separated them from the gaps around
+ them -- and is kept rather than cut; --min-silence keeps the meaning it already has for
+ --filler and the plain run: the shortest gap this tool will remove."""
+ floor = min(BREATH_FLOOR, min_silence)
+ fine = detect(path, threshold, floor)
+ boundaries, breaths = [], []
+ for s, e in fine:
+ length = (e - s) if e != float("inf") else float("inf")
+ if length >= min_silence:
+ boundaries.append((s, e))
+ else:
+ breaths.append((s, e))
+ return boundaries, breaths
+
+
def keep_ranges(silences: List[Tuple[float, float]], duration: float, margin: float, min_keep: float) -> List[Tuple[float, float]]:
keeps: List[Tuple[float, float]] = []
cursor = 0.0
@@ -182,6 +208,11 @@ def main() -> int:
ap.add_argument("--min-silence", type=float, default=0.6, help="only remove gaps at least this long in seconds (default 0.6)")
ap.add_argument("--margin", type=float, default=0.15, help="seconds of silence to keep on each side of speech (default 0.15)")
ap.add_argument("--min-keep", type=float, default=0.2, help="drop kept pieces shorter than this (default 0.2)")
+ ap.add_argument("--speech-aware", action="store_true",
+ help="keep breaths shorter than --min-silence when they sit inside a sentence "
+ "(measured by re-running silence detection at a much shorter floor), and "
+ "only cut at sentence-boundary pauses (--min-silence or longer). Composes "
+ "with --filler through the same keep_ranges() removal list.")
ap.add_argument("--list", action="store_true", help="only print silences and the kept ranges")
ap.add_argument("--edl", help="write the kept ranges to this file, one START-END per line")
fil = ap.add_argument_group("filler words (1.17)")
@@ -229,7 +260,11 @@ def main() -> int:
if not meta.get("audio"):
die("input has no audio stream to analyse")
duration = meta.get("duration") or 0.0
- silences = detect(args.input, args.threshold, args.min_silence)
+ breaths: List[Tuple[float, float]] = []
+ if args.speech_aware:
+ silences, breaths = speech_aware_silences(args.input, args.threshold, args.min_silence)
+ else:
+ silences = detect(args.input, args.threshold, args.min_silence)
filler_info, filler_ranges = resolve_filler(args, meta) if args.filler else (None, [])
# One sorted, merged removal list through the graph the tool already has: filler removal IS
# time-range removal, so it reuses keep_ranges() and the same aselect/concat chain.
@@ -253,6 +288,16 @@ def main() -> int:
"kept_duration": round(kept, 3),
"removed_seconds": round(silence_only, 3),
}
+ if args.speech_aware:
+ summary["speech_aware"] = {
+ "min_silence": args.min_silence, "floor": min(BREATH_FLOOR, args.min_silence),
+ "breaths_kept": len(breaths),
+ "breaths_kept_seconds": round(sum((e - s) for s, e in breaths if e != float("inf")), 3),
+ "breaths": [[round(s, 3), None if e == float("inf") else round(e, 3)] for s, e in breaths],
+ }
+ info(f"--speech-aware: {len(breaths)} breath(s) kept "
+ f"({summary['speech_aware']['breaths_kept_seconds']:.2f}s), "
+ f"{len(silences)} sentence-boundary silence(s) cut")
if filler_info is not None:
# removed_seconds above is the silence-only figure, unchanged in meaning; the filler share
# is reported inside `filler`, and removed_seconds_total is the additive sibling that
diff --git a/scripts/sync.py b/scripts/sync.py
index 6023e16..5057109 100755
--- a/scripts/sync.py
+++ b/scripts/sync.py
@@ -30,7 +30,7 @@ import json
import math
import os
import sys
-from typing import List
+from typing import Dict, List
from _common import video_args, add_common, apply_common, emit, aac_args, audio_codec_for, default_output, die, ffmpeg_base, info, probe, require_tool, run, run_analysis, x264_args, decode_pcm_mono, rms_envelope
@@ -195,84 +195,126 @@ def measure_offset(ref_path: str, oth_path: str, start: float, seconds: float, s
return offset, max(0.0, min(1.0, score))
-def main() -> int:
- ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("reference", help="reference recording (usually the camera video)")
- ap.add_argument("second", help="recording to align (external audio or second camera)")
- ap.add_argument("-o", "--output", help="output file when writing a synced result")
- ap.add_argument("--max-offset", type=float, default=30.0, help="largest offset to search in seconds (default 30)")
- ap.add_argument("--analyze-seconds", type=float, default=120.0, help="how much audio to analyse from each file (default 120)")
- ap.add_argument("--step-ms", type=float, default=20.0, help="coarse envelope resolution in ms for the FFT search (default 20)")
- ap.add_argument("--fine-ms", type=float, default=1.0, help="fine resolution in ms for the refinement pass, 0 to skip (default 1)")
- ap.add_argument("--fix-drift", action="store_true", help="also measure the offset near the END and correct clock drift by resampling the second file")
- ap.add_argument("--drift-window", type=float, default=60.0, help="seconds of audio analysed at each end for drift (default 60)")
- mode = ap.add_mutually_exclusive_group()
- mode.add_argument("--replace-audio", action="store_true", help="write reference video with the second file's audio, aligned")
- mode.add_argument("--trim-second", action="store_true", help="write the second file shifted so it lines up with the reference")
- ap.add_argument("--crf", type=int, default=18)
- add_common(ap)
- args = ap.parse_args()
- apply_common(args)
- if args.analyze_seconds > 900:
- die(f"--analyze-seconds {args.analyze_seconds:g}: the window is decoded into memory; 900 s is the ceiling")
-
- for p in (args.reference, args.second):
- if not probe(p).get("audio"):
- die(f"{p} has no audio stream to correlate")
-
- offset, score = measure_offset(args.reference, args.second, 0.0, args.analyze_seconds, args.step_ms, args.max_offset, args.fine_ms)
-
+def measure_one(reference: str, path: str, args) -> Dict:
+ """Everything sync.py measures for ONE other source against the reference: offset,
+ confidence and (with --fix-drift) the same end-of-file drift measurement main() has always
+ made for a single pair. Used for every source when N>1, and for the historical single-pair
+ CLI shape when N==1 -- one measurement, so the two shapes cannot disagree."""
+ offset, score = measure_offset(reference, path, 0.0, args.analyze_seconds, args.step_ms, args.max_offset, args.fine_ms)
drift_ratio = 1.0
drift_info = None
if args.fix_drift:
- ref_dur = probe(args.reference)["duration"] or 0.0
- sec_dur = probe(args.second)["duration"] or 0.0
- overlap_end = min(ref_dur, sec_dur + offset) # last reference time both files cover
+ ref_dur = probe(reference)["duration"] or 0.0
+ sec_dur = probe(path)["duration"] or 0.0
+ overlap_end = min(ref_dur, sec_dur + offset)
head_len = min(args.analyze_seconds, overlap_end)
tail_start = overlap_end - args.drift_window
if tail_start <= head_len / 2 + 5:
- info("warning: files too short to measure drift reliably; skipping drift correction")
+ info(f"warning: {path} too short to measure drift reliably; skipping drift correction")
else:
ref_start = tail_start
sec_start = tail_start - offset
if sec_start < 0:
ref_start -= sec_start
sec_start = 0.0
- ref_s = decode_mono(args.reference, args.drift_window, ref_start)
- oth_s = decode_mono(args.second, args.drift_window, sec_start)
+ ref_s = decode_mono(reference, args.drift_window, ref_start)
+ oth_s = decode_mono(path, args.drift_window, sec_start)
step = max(1, int(SR * args.step_ms / 1000))
lag, end_score = cross_correlate(envelope(ref_s, step), envelope(oth_s, step), int(2.0 * SR / step))
residual = lag * step / SR
if args.fine_ms:
residual = refine(ref_s, oth_s, residual, max(1, int(SR * args.fine_ms / 1000)), args.step_ms / 1000 * 2)
- # both measurements represent the offset at the centre of their windows
head_mid = head_len / 2
tail_mid = ref_start + args.drift_window / 2
elapsed = tail_mid - head_mid
if elapsed > 0 and end_score > 0.1:
- # offset(T) = offset0 - (ratio - 1) * T, where ratio is how fast the second file's clock
- # runs relative to the reference (ratio > 1 = the second file is too long / plays slow)
drift_ratio = 1.0 - residual / elapsed
- offset = offset + (drift_ratio - 1.0) * head_mid # extrapolate back to T = 0
+ offset = offset + (drift_ratio - 1.0) * head_mid
drift_info = {"residual_at_end_seconds": round(residual, 4), "measured_over_seconds": round(elapsed, 2),
"drift_ppm": round((drift_ratio - 1) * 1e6, 1),
"meaning": "second file runs %.1f ppm %s (%.3fs over %.0fs); it will be resampled to match" % (
abs(drift_ratio - 1) * 1e6, "long/slow" if drift_ratio > 1 else "short/fast", abs(residual), elapsed),
"confidence": round(end_score, 3)}
else:
- info("warning: could not measure drift with confidence; skipping drift correction")
-
- result = {
- "reference": args.reference,
- "second": args.second,
- "offset_seconds": round(offset, 4),
- "confidence": round(max(0.0, min(1.0, score)), 3),
- "meaning": ("second starts %.3fs %s than reference" % (abs(offset), "later" if offset > 0 else "earlier")),
- }
- if drift_info:
- result["drift"] = drift_info
- if result["confidence"] < 0.1:
- info("warning: low correlation confidence; check that both files contain the same audio event")
+ info(f"warning: could not measure drift with confidence for {path}; skipping drift correction")
+ return {"path": path, "offset_seconds": round(offset, 4), "offset_s": round(offset, 4),
+ "confidence": round(max(0.0, min(1.0, score)), 3), "drift_ratio": drift_ratio, "drift": drift_info,
+ "drift_ppm": (drift_info["drift_ppm"] if drift_info else None)}
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("reference", help="reference recording (usually the camera video)")
+ ap.add_argument("second", help="recording to align (external audio or second camera)")
+ ap.add_argument("more_sources", nargs="*", metavar="SOURCE3...", default=[],
+ help="(1.18) additional recordings beyond `second` to align to the same "
+ "reference -- a third camera, a second external recorder. With none of "
+ "these, sync.py keeps its original 2-source shape (offset_seconds/"
+ "confidence/drift at the top level, --replace-audio/--trim-second "
+ "available). With 1+, all sources (second plus these) are measured and "
+ "reported as one offsets JSON {reference, sources: [{path, offset_s, "
+ "confidence, drift_ppm}]}, and --replace-audio/--trim-second refuse: "
+ "each writes ONE synced output and there is more than one source")
+ ap.add_argument("-o", "--output", help="output file when writing a synced result (one-source runs only)")
+ ap.add_argument("--max-offset", type=float, default=30.0, help="largest offset to search in seconds (default 30)")
+ ap.add_argument("--analyze-seconds", type=float, default=120.0, help="how much audio to analyse from each file (default 120)")
+ ap.add_argument("--step-ms", type=float, default=20.0, help="coarse envelope resolution in ms for the FFT search (default 20)")
+ ap.add_argument("--fine-ms", type=float, default=1.0, help="fine resolution in ms for the refinement pass, 0 to skip (default 1)")
+ ap.add_argument("--fix-drift", action="store_true", help="also measure the offset near the END and correct clock drift by resampling the second file")
+ ap.add_argument("--drift-window", type=float, default=60.0, help="seconds of audio analysed at each end for drift (default 60)")
+ mode = ap.add_mutually_exclusive_group()
+ mode.add_argument("--replace-audio", action="store_true", help="write reference video with the second file's audio, aligned")
+ mode.add_argument("--trim-second", action="store_true", help="write the second file shifted so it lines up with the reference")
+ ap.add_argument("--crf", type=int, default=18)
+ add_common(ap)
+ args = ap.parse_args()
+ apply_common(args)
+ if args.analyze_seconds > 900:
+ die(f"--analyze-seconds {args.analyze_seconds:g}: the window is decoded into memory; 900 s is the ceiling")
+ args.sources = [args.second] + list(args.more_sources)
+
+ for p in (args.reference,) + tuple(args.sources):
+ if not probe(p).get("audio"):
+ die(f"{p} has no audio stream to correlate")
+
+ n_sources = len(args.sources)
+ if (args.replace_audio or args.trim_second) and n_sources > 1:
+ die("--replace-audio/--trim-second write ONE synced output and need exactly one SOURCE; "
+ f"{n_sources} were given. Run sync.py once per source, or drop the flag and read the "
+ "offsets JSON.", kind="input")
+
+ measured = [measure_one(args.reference, p, args) for p in args.sources]
+ for p, m in zip(args.sources, measured):
+ if m["confidence"] < 0.1:
+ info(f"warning: {p}: low correlation confidence; check that it shares an audio event with the reference")
+
+ sources_block = [{"path": p, "offset_s": m["offset_s"], "confidence": m["confidence"],
+ **({"drift_ppm": m["drift_ppm"]} if args.fix_drift else {})}
+ for p, m in zip(args.sources, measured)]
+
+ if n_sources == 1:
+ # The original 2-source shape, unchanged, plus (additively) the same measurement under
+ # `sources` so a caller reading the new key gets the same numbers for a single source.
+ args.second = args.sources[0]
+ m = measured[0]
+ offset, score, drift_ratio, drift_info = m["offset_seconds"], m["confidence"], m["drift_ratio"], m["drift"]
+ result = {
+ "reference": args.reference,
+ "second": args.second,
+ "offset_seconds": offset,
+ "confidence": score,
+ "meaning": ("second starts %.3fs %s than reference" % (abs(offset), "later" if offset > 0 else "earlier")),
+ "sources": sources_block,
+ }
+ if drift_info:
+ result["drift"] = drift_info
+ if score < 0.1:
+ info("warning: low correlation confidence; check that both files contain the same audio event")
+ else:
+ offset = drift_ratio = drift_info = None # not used below; output writing is 1-source only
+ result = {"reference": args.reference, "sources": sources_block}
+ info(f"measured {n_sources} sources against {args.reference}: " +
+ ", ".join(f"{s['path']}: {s['offset_s']:+.3f}s (confidence {s['confidence']:.2f})" for s in sources_block))
if args.replace_audio or args.trim_second:
output = args.output or default_output(args.reference if args.replace_audio else args.second, "synced", "mp4")
@@ -339,12 +381,18 @@ def main() -> int:
if args.json:
emit(result.get("output"), **{k: v for k, v in result.items() if k != "output"})
- else:
+ elif n_sources == 1:
print(f"offset: {result['offset_seconds']:+.3f}s ({result['meaning']}), confidence {result['confidence']:.2f}")
if drift_info:
print(f"drift: {drift_info['drift_ppm']:+.1f} ppm ({drift_info['residual_at_end_seconds']:+.3f}s over {drift_info['measured_over_seconds']:.0f}s), confidence {drift_info['confidence']:.2f}")
if "output" in result:
print(result["output"])
+ else:
+ for s in sources_block:
+ line = f"{s['path']}: {s['offset_s']:+.3f}s, confidence {s['confidence']:.2f}"
+ if args.fix_drift:
+ line += f", drift {s['drift_ppm']:+.1f} ppm" if s["drift_ppm"] is not None else ""
+ print(line)
return 0
diff --git a/tests/fixtures/mcp_tools.json b/tests/fixtures/mcp_tools.json
index 7fe836c..35966f5 100644
--- a/tests/fixtures/mcp_tools.json
+++ b/tests/fixtures/mcp_tools.json
@@ -302,6 +302,7 @@
"json",
"json_brief",
"limit",
+ "motion_centre",
"overwrite",
"plan",
"progress",
@@ -741,6 +742,7 @@
"codec",
"crf",
"dry_run",
+ "edl",
"fast",
"fix_drift",
"fps",
@@ -749,6 +751,7 @@
"json",
"json_brief",
"max_offset",
+ "min_shot",
"offsets_only",
"output",
"overwrite",
@@ -994,6 +997,7 @@
"name": "scenes",
"properties": [
"argv",
+ "audio_peaks",
"beat_range",
"beat_step",
"beats",
@@ -1014,6 +1018,8 @@
"rank_by",
"ratio",
"sheet",
+ "shots",
+ "speech",
"target",
"threshold",
"timeout"
@@ -1076,6 +1082,7 @@
"preset",
"progress",
"quality",
+ "speech_aware",
"threshold",
"timeout",
"transcribe",
@@ -1205,6 +1212,7 @@
"json",
"json_brief",
"max_offset",
+ "more_sources",
"output",
"overwrite",
"plan",
diff --git a/tests/test_analysis.py b/tests/test_analysis.py
index 2b69848..b0bd0ab 100644
--- a/tests/test_analysis.py
+++ b/tests/test_analysis.py
@@ -64,6 +64,30 @@ class AnalysisTests(MediaFixtures):
def test_cropdetect_bad_seconds_refused(self):
script("cropdetect.py", self.src, "--seconds", "0", expect_fail=True)
+ # ---------------------------------------------------------------- 1.18.0: cropdetect.py --motion-centre
+ def test_cropdetect_motion_centre_reports_per_second_points(self):
+ data = json.loads(script("cropdetect.py", self.src, "--seconds", "4", "--samples", "2",
+ "--motion-centre", "--json").stdout)
+ self.assertIn("motion_centre", data)
+ self.assertGreater(len(data["motion_centre"]), 0)
+ for pt in data["motion_centre"]:
+ self.assertIn("time", pt)
+ if pt["x"] is not None:
+ self.assertGreaterEqual(pt["x"], 0)
+ self.assertLessEqual(pt["x"], data["source_width"])
+ self.assertGreaterEqual(pt["x_frac"], 0.0)
+ self.assertLessEqual(pt["x_frac"], 1.0)
+
+ def test_cropdetect_motion_centre_writes_no_file(self):
+ before = set(OUT.iterdir())
+ script("cropdetect.py", self.src, "--seconds", "1", "--samples", "1", "--motion-centre")
+ self.assertEqual(before, set(OUT.iterdir()))
+
+ def test_cropdetect_motion_centre_is_off_by_default(self):
+ data = json.loads(script("cropdetect.py", self.src, "--seconds", "1", "--samples", "1",
+ "--json").stdout)
+ self.assertNotIn("motion_centre", data)
+
def test_fonts_dir_does_not_bypass_the_coverage_check(self):
"""--fonts-dir says "also look here", not "this exact face": a directory that does not
cover the script must not silently switch the guarantee off."""
@@ -119,6 +143,59 @@ class AnalysisTests(MediaFixtures):
again = json.loads(script("sync.py", self.src, out, "--json").stdout)
self.assertClose(again["offset_seconds"], 0.0, 0.05)
+ # ---------------------------------------------------------------- 1.18.0: sync.py 3+ sources
+ def _cam3(self):
+ """A third source: the reference shifted 1.2 s later, so it has a real measurable offset
+ distinct from self.mic's 2.5 s."""
+ cam3 = OUT / "sync_cam3.wav"
+ if not cam3.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "aevalsrc=0:s=48000:d=1.2", "-i", self.src,
+ "-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1", cam3)
+ return cam3
+
+ def test_sync_two_sources_keeps_the_original_shape(self):
+ """The 2-source CLI shape is a special case of the N-source one, byte-for-byte: the same
+ top-level offset_seconds/confidence/second keys tested since 1.0, plus (additively) the
+ new `sources` list."""
+ data = json.loads(script("sync.py", self.src, self.mic, "--json").stdout)
+ self.assertIn("offset_seconds", data)
+ self.assertIn("second", data)
+ self.assertIn("confidence", data)
+ self.assertIn("sources", data)
+ self.assertEqual(len(data["sources"]), 1)
+ self.assertEqual(data["sources"][0]["path"], str(self.mic))
+ self.assertAlmostEqual(data["sources"][0]["offset_s"], data["offset_seconds"], delta=0.001)
+
+ def test_sync_three_sources_writes_one_offsets_json(self):
+ data = json.loads(script("sync.py", self.src, self.mic, self._cam3(), "--json").stdout)
+ self.assertNotIn("second", data) # no single-pair shape for N>1
+ self.assertNotIn("offset_seconds", data)
+ self.assertEqual(data["reference"], str(self.src))
+ self.assertEqual(len(data["sources"]), 2)
+ by_path = {s["path"]: s for s in data["sources"]}
+ self.assertClose(by_path[str(self.mic)]["offset_s"], 2.5, 0.05)
+ self.assertClose(by_path[str(self._cam3())]["offset_s"], -1.2, 0.05)
+ for s in data["sources"]:
+ self.assertIn("confidence", s)
+
+ def test_sync_three_sources_reports_drift_ppm_per_source(self):
+ data = json.loads(script("sync.py", self.long_ref, self.long_drift, self.long_ref,
+ "--fix-drift", "--json").stdout)
+ drifts = {s["path"]: s["drift_ppm"] for s in data["sources"]}
+ self.assertClose(drifts[str(self.long_drift)], 500.0, 60.0)
+ self.assertClose(drifts[str(self.long_ref)], 0.0, 60.0)
+
+ def test_sync_replace_audio_refuses_multiple_sources(self):
+ r = script("sync.py", self.src, self.mic, self._cam3(), "--replace-audio",
+ "-o", OUT / "nope.mp4", "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_sync_three_sources_text_output(self):
+ proc = script("sync.py", self.src, self.mic, self._cam3())
+ self.assertIn(str(self.mic), proc.stdout)
+ self.assertIn(str(self._cam3()), proc.stdout)
+
# ---------------------------------------------------------------- real-world material
def test_probe_detects_vfr_rotation_surround_hdr(self):
self.assertTrue(probe(str(self.vfr))["video"]["variable_frame_rate_suspected"])
@@ -579,6 +656,53 @@ class AnalysisTests(MediaFixtures):
"--json").stdout)
self.assertNotIn("beat_grid", data)
+ # --------------------------------------------------------------- 1.18.0: --shots / --audio-peaks / --speech
+ def test_scenes_shots_labels_every_scene(self):
+ data = json.loads(script("scenes.py", self.src, "--shots", "--json").stdout)
+ self.assertIn("shots", data)
+ self.assertEqual(len(data["shots"]), data["scene_count"])
+ for sh in data["shots"]:
+ self.assertIn(sh["label"], ("static", "pan", "motion"))
+ self.assertGreaterEqual(sh["flow_magnitude"], 0.0)
+ self.assertIn("start", sh)
+ self.assertIn("end", sh)
+
+ def test_scenes_shots_static_clip_is_labelled_static(self):
+ """A single unmoving frame held for the whole clip has nothing to flow: static, magnitude 0."""
+ still = OUT / "still.mp4"
+ if not still.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "color=c=gray:s=320x180:rate=10", "-f", "lavfi", "-i", "sine=f=440",
+ "-t", "4", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", still)
+ data = json.loads(script("scenes.py", still, "--shots", "--json").stdout)
+ self.assertTrue(all(sh["label"] == "static" for sh in data["shots"]))
+ self.assertTrue(all(sh["flow_magnitude"] == 0.0 for sh in data["shots"]))
+
+ def test_scenes_audio_peaks_db_is_measured_in_dbfs_and_separate_from_the_rms_list(self):
+ data = json.loads(script("scenes.py", self.src, "--audio-peaks", "--json").stdout)
+ self.assertIn("audio_peaks_db", data)
+ self.assertIn("audio_peaks", data) # the pre-existing key, unchanged
+ for p in data["audio_peaks_db"]:
+ self.assertIn("time", p)
+ self.assertIn("level", p)
+ self.assertLess(p["level"], 0) # dBFS, not linear RMS
+
+ def test_scenes_speech_reports_a_ratio_not_a_label(self):
+ data = json.loads(script("scenes.py", self.src, "--speech", "--json").stdout)
+ self.assertIn("speech", data)
+ self.assertGreater(len(data["speech"]), 0)
+ for w in data["speech"]:
+ self.assertIn("time", w)
+ self.assertIn("speech_music_ratio", w)
+ self.assertGreaterEqual(w["speech_music_ratio"], 0.0)
+
+ def test_scenes_shots_audio_peaks_speech_combine_with_beats_and_highlights(self):
+ data = json.loads(script("scenes.py", self._beats(), "--shots", "--audio-peaks", "--speech",
+ "--beats", "--highlights", "2", "--json").stdout)
+ for key in ("shots", "audio_peaks_db", "speech", "beats", "highlights"):
+ self.assertIn(key, data)
+
class ProposeChaptersTests(unittest.TestCase):
"""1.16: the pure half of metadata.py --auto-chapters. No media, no subprocess."""
diff --git a/tests/test_editing.py b/tests/test_editing.py
index 89ffa2e..ac1104d 100644
--- a/tests/test_editing.py
+++ b/tests/test_editing.py
@@ -1272,6 +1272,72 @@ class EditingTests(MediaFixtures):
"--json").stdout)["filler"]
self.assertIn("so", extra["removed_words"])
+ # ------------------------------------------------ 1.18.0: silence.py --speech-aware
+ def _breathy(self):
+ """10 s: speech 0-3s, a 0.25 s breath (well under the 0.6 s default --min-silence),
+ speech 3.25-6s, a 1 s sentence-boundary pause 6-7s, speech 7-10s."""
+ clip = OUT / "breathy.mp4"
+ if not clip.exists():
+ expr = "0.5*sin(2*PI*440*t)*(lt(t\\,3)+between(t\\,3.25\\,6)+gt(t\\,7))"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", f"aevalsrc='{expr}':s=48000", "-t", "10", clip)
+ return clip
+
+ def test_speech_aware_keeps_a_short_breath_and_cuts_the_sentence_boundary(self):
+ data = json.loads(script("silence.py", self._breathy(), "--speech-aware", "--list",
+ "--margin", "0.05", "--json").stdout)
+ sa = data["speech_aware"]
+ self.assertEqual(sa["breaths_kept"], 1)
+ self.assertAlmostEqual(sa["breaths"][0][0], 3.0, delta=0.1)
+ self.assertAlmostEqual(sa["breaths"][0][1], 3.25, delta=0.1)
+ # the sentence-boundary pause (6-7s) is the only one removed
+ self.assertEqual(len(data["silences"]), 1)
+ self.assertAlmostEqual(data["silences"][0][0], 6.0, delta=0.1)
+
+ def test_speech_aware_without_the_flag_would_have_cut_the_breath_too(self):
+ """With a --min-silence short enough to catch the breath, the plain (non-speech-aware)
+ run removes it; --speech-aware is what keeps it."""
+ plain = json.loads(script("silence.py", self._breathy(), "--min-silence", "0.2",
+ "--threshold", "-35", "--list", "--json").stdout)
+ aware = json.loads(script("silence.py", self._breathy(), "--min-silence", "0.6",
+ "--speech-aware", "--list", "--json").stdout)
+ self.assertEqual(len(plain["silences"]), 2) # breath + sentence pause, both cut
+ self.assertEqual(len(aware["silences"]), 1) # only the sentence pause
+ self.assertEqual(aware["speech_aware"]["breaths_kept"], 1)
+
+ def test_speech_aware_writes_an_edl_like_the_plain_flag(self):
+ edl = OUT / "speech_aware.edl"
+ script("silence.py", self._breathy(), "--speech-aware", "--edl", edl, "--list", "--json")
+ lines = edl.read_text(encoding="utf-8").strip().splitlines()
+ self.assertGreaterEqual(len(lines), 1)
+ for line in lines:
+ a, b = line.split("-")
+ self.assertLess(float(a), float(b))
+
+ def test_speech_aware_composes_with_filler_into_one_removal_list(self):
+ """The trickiest part of 1.18.0: --speech-aware and --filler must flow through the same
+ keep_ranges() and produce one removal list, not two independent cuts. A filler word
+ placed inside the kept breath must still be removed -- and the breath must still not be
+ cut just because --filler ran."""
+ clip = self._breathy()
+ words = OUT / "words_breath.json"
+ # a filler word squarely inside the kept breath (3.0-3.25s)
+ words.write_text(json.dumps({"language": "en", "segments": [
+ {"words": [{"word": "um", "start": 3.05, "end": 3.15}]}]}), encoding="utf-8")
+ out = OUT / "speech_aware_filler.mp4"
+ data = json.loads(script("silence.py", clip, "--speech-aware", "--filler",
+ "--words", words, "-o", out, "--margin", "0.02", "--json").stdout)
+ self.assertIn("speech_aware", data)
+ self.assertIn("filler", data)
+ # the breath is still reported as kept by --speech-aware...
+ self.assertEqual(data["speech_aware"]["breaths_kept"], 1)
+ # ...but the filler word inside it is removed anyway: one composed list, not two cuts
+ # that ignore each other. removed_seconds_total covers both the sentence pause and the
+ # filler word.
+ self.assertGreater(data["removed_seconds_total"], data["removed_seconds"])
+ m = probe(str(out))
+ self.assertLess(m["duration"], data["input_duration"])
+
def test_filler_warns_about_a_japanese_discourse_marker(self):
words = [("\u305d\u308c\u306f", 0.2, 0.5), ("\u306a\u3093\u304b", 0.6, 0.85), ("\u3044\u3044", 1.0, 1.3)]
data = json.loads(script("silence.py", self._gappy(), "--filler", "--filler-list",
diff --git a/tests/test_orchestration.py b/tests/test_orchestration.py
index 84cf0c8..e26e5ac 100644
--- a/tests/test_orchestration.py
+++ b/tests/test_orchestration.py
@@ -253,6 +253,96 @@ class OrchestrationTests(MediaFixtures):
self.assertClose(probe(str(auto))["duration"], 12.0, 0.2)
script("multicam.py", self.src, camB, "--switch", "0-3:5", expect_fail=True)
+ # ------------------------------------------------------------ 1.18.0: --switch energy / --edl
+ def _loud_cams(self):
+ """Two cameras on one reference timeline: camA loud 0-6s / quiet 6-12s, camB the reverse
+ -- --switch energy should pick camA for the first half and camB for the second."""
+ camA = OUT / "mc_energy_a.mp4"
+ camB = OUT / "mc_energy_b.mp4"
+ if not camA.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "testsrc2=size=160x90:rate=30", "-f", "lavfi",
+ "-i", "aevalsrc='0.8*sin(2*PI*440*t)*lt(t\\,6)+0.01*sin(2*PI*440*t)*gt(t\\,6)':s=48000",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", camA)
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "testsrc2=size=160x90:rate=30,hue=h=90", "-f", "lavfi",
+ "-i", "aevalsrc='0.01*sin(2*PI*440*t)*lt(t\\,6)+0.8*sin(2*PI*440*t)*gt(t\\,6)':s=48000",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", camB)
+ return camA, camB
+
+ def test_multicam_switch_energy_picks_the_loudest_camera(self):
+ camA, camB = self._loud_cams()
+ out = OUT / "mc_energy.mp4"
+ data = json.loads(script("multicam.py", camA, camB, "--switch", "energy", "--min-shot", "1",
+ "--fast", "-o", out, "--json").stdout)
+ self.assertEqual(data["switch_mode"], "energy")
+ cams_by_time = {}
+ for s, e, c in data["cuts"]:
+ cams_by_time[(s + e) / 2] = c
+ early = [c for t, c in cams_by_time.items() if t < 5]
+ late = [c for t, c in cams_by_time.items() if t > 7]
+ self.assertTrue(early and all(c == 0 for c in early), f"expected camera 0 (louder) early: {data['cuts']}")
+ self.assertTrue(late and all(c == 1 for c in late), f"expected camera 1 (louder) late: {data['cuts']}")
+
+ def test_multicam_switch_energy_respects_min_shot(self):
+ camA, camB = self._loud_cams()
+ out = OUT / "mc_energy_minshot.mp4"
+ data = json.loads(script("multicam.py", camA, camB, "--switch", "energy", "--min-shot", "3",
+ "--fast", "-o", out, "--json").stdout)
+ for s, e, _c in data["cuts"]:
+ # the very first/last cut may be shorter (it borders the clip edge), interior cuts
+ # must respect --min-shot
+ if s > 0.01 and e < 11.99:
+ self.assertGreaterEqual(round(e - s, 2), 3.0 - 0.05, data["cuts"])
+
+ def test_multicam_edl_matches_the_reported_cuts(self):
+ camA, camB = self._loud_cams()
+ edl = OUT / "mc_energy.edl"
+ out = OUT / "mc_energy_edl.mp4"
+ data = json.loads(script("multicam.py", camA, camB, "--switch", "energy", "--min-shot", "1",
+ "--edl", edl, "--fast", "-o", out, "--json").stdout)
+ lines = edl.read_text(encoding="utf-8").strip().splitlines()
+ self.assertEqual(len(lines), len(data["cuts"]))
+ for line, (s, e, _c) in zip(lines, data["cuts"]):
+ a, b = (float(x) for x in line.split("-"))
+ self.assertAlmostEqual(a, s, delta=0.01)
+ self.assertAlmostEqual(b, e, delta=0.01)
+ self.assertLess(a, b)
+
+ def test_multicam_switch_energy_needs_at_least_one_video_camera(self):
+ wav_a = OUT / "mc_energy_audio_only_a.wav"
+ wav_b = OUT / "mc_energy_audio_only_b.wav"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", f"aevalsrc='{TONES}':s=48000", "-t", "4", wav_a)
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", f"aevalsrc='{TONES}':s=48000", "-t", "4", wav_b)
+ script("multicam.py", wav_a, wav_b, "--switch", "energy", "-o", OUT / "mc_nope.mp4", expect_fail=True)
+
+ def test_multicam_min_shot_must_be_positive(self):
+ camA, camB = self._loud_cams()
+ script("multicam.py", camA, camB, "--switch", "energy", "--min-shot", "0", "-o", OUT / "mc_nope2.mp4", expect_fail=True)
+
+ def test_multicam_edl_is_renderable_as_a_render_project(self):
+ """The multicam timeline is expressible as a render.py project without a new stage type:
+ each cut becomes a clip that points at its own camera's file with in/out on that camera's
+ OWN timeline (reference time shifted by the measured offset), using render.py's existing
+ clips array -- see docs/contract.md and docs/design-decisions.md for why no new project
+ schema was needed."""
+ camA, camB = self._loud_cams()
+ data = json.loads(script("multicam.py", camA, camB, "--switch", "energy", "--min-shot", "1",
+ "--fast", "-o", OUT / "mc_energy_proj.mp4", "--json").stdout)
+ offsets = data["offsets_seconds"]
+ clips = [{"src": [str(camA), str(camB)][c], "in": round(s - offsets[c], 3), "out": round(e - offsets[c], 3)}
+ for s, e, c in data["cuts"]]
+ project = OUT / "mc_project.json"
+ project.write_text(json.dumps({"output": "mc_rendered.mp4", "clips": clips}), encoding="utf-8")
+ out = OUT / "mc_rendered.mp4"
+ script("render.py", project, "--fast", "-o", out)
+ self.assertTrue(out.exists())
+ m = probe(str(out))
+ # the measured per-camera offset carries its own small error, so this is a looser bound
+ # than the multicam output's own duration check -- the point is that render.py accepts the
+ # clip list at all, not a frame-accurate re-derivation of multicam's own timeline math.
+ self.assertGreater(m["duration"], 8.0)
+
def test_multicam_fix_drift_trims_before_resample_not_after(self):
"""--fix-drift's audio path computes a_start (an atrim start point) in the source's own
pre-correction time axis, but the filter chain used to apply asetrate/aresample (the drift