Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

feat: 1.17.0 — throughput (caption size that fits the cue, beat-synced cuts, filler-word removal, batch --jobs, render --cache) (#235)

claude[bot]40c9fb78fc18
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ecf9dc6..f6f9a91 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,69 @@
## Unreleased
-(nothing yet)
+_1.17.0 — throughput, plus the caption-size half of the eval-17 finding. The release
+workflow turns this heading into the version number._
+
+### Added
+
+- **`caption.py --fit-size auto|on|off`, `--min-size N`, `--fit-size-scope file|cue`.** The caption
+ size is fitted to the cue *before* the cue is split. At a platform caption size (24 ASS units,
+ about six em per line at 1080x1920) an ordinary sentence needs four lines, so `--max-lines 2`
+ cut it into consecutive cues and half of it arrived late — the size, not the breaker, was what
+ did not fit. `--min-size` defaults to `ass_units(0.045) = 13`, the 4.5 %-of-frame-height
+ legibility floor, one floor for every destination. New keys inside the existing `caption` block:
+ `fit_size`, `size_requested`, `size_used`, `size_floor`, `size_pct_height`, `shrunk`,
+ `fit_scope`, `fit_exhausted`. The caption text is never rewritten, shortened or paraphrased to
+ make it fit; below the floor the cue is split exactly as before and `fit_exhausted` says so.
+- **`scenes.py --beats`, `--beat-step`, `--beat-range`, `--min-confidence`.** The measured beat
+ grid — onset flux, median+MAD peak picking, an autocorrelation tempo with octave
+ disambiguation, and a confidence that is half how far the winning lag stands above the others
+ and half the fraction of onsets that land on the grid. New keys `beats` and `beat_grid`. With
+ `--beats` the file is decoded once and both envelopes come from that pass.
+- **`cut.py --snap beats`, `--snap-tolerance`, `--snap-source`, `--min-confidence`.** In/out points
+ move to the nearest measured beat before the keyframe decision. Below the confidence threshold
+ the run refuses and names the measured number and `--snap none`: a cut point may move to a
+ measured grid point and may not appear from one. New key `snap`. `render.py` forwards a
+ project's `"snap": {"to": "beats", ...}` and reports what came back.
+- **`silence.py --filler`** and its family (`--filler-lang`, `--filler-words`, `--filler-extra`,
+ `--filler-keep`, `--filler-pad`, `--words`, `--transcribe`, `--filler-list`, `--max-cuts`).
+ Filler words are removed through the existing `keep_ranges()`/`aselect` graph — and only where a
+ speech engine measured a start/end pair for the word. Built-in lists for en, ja, es, de, fr, pt
+ and it; `like`, `tipo` and `cioè` are deliberately not in them (discourse markers are not
+ disfluencies) and are reachable with `--filler-extra`. New keys `filler` and
+ `removed_seconds_total` (silence plus filler); `removed_seconds` is unchanged and still holds
+ the silence-only figure. Whisper stays optional for this tool
+ exactly as it is for `caption.py`. `--transcribe` drives whichever engine is installed with
+ its own word-timestamp option (whisper.cpp `--output-json-full`, faster-whisper
+ `word_timestamps=True`, openai-whisper `--word_timestamps True`); an engine whose build
+ produces none refuses by name rather than reporting an empty removal as a success.
+- **`batch.py --jobs N|auto`.** Parallel items under one shared `--timeout` budget, capped at
+ `min(N, cpu_count, 8)` with the applied value reported. The per-item table keeps its shape and
+ its order, and each item's log is flushed in file order. New keys `jobs`, `jobs_requested`,
+ `wall_seconds`, `item_seconds_total`, `timed_out`.
+- **`render.py --cache DIR` and `--from STAGE`.** An opt-in stage cache keyed on the stage, its
+ arguments, its inputs' content hashes, the flags render forwards to its children (`--fast`,
+ `--codec`, …), the output's extension and the ffmpeg build banner plus the skill and contract
+ versions, so a cache is never reused across any of them. New key `cache`. There is no default cache directory.
+
+### Changed
+
+- `caption.py` now fits the caption size to the cue before splitting it when the size was not
+ stated (`--fit-size auto`, the default). A run that passed `--size` explicitly, or that took a
+ size from a `brand.json`, is unchanged; `--fit-size off` restores 1.16.1 byte-for-byte (a pinned
+ ASS fixture asserts it). This changes the rendered ASS for platform caption runs that took the
+ default size — the defect eval 17 measured.
+- The optional local speech-to-text bridge and the SRT reader/writer moved from `caption.py` into
+ `scripts/_common/asr.py` and are re-exported through the `_common` facade, so `silence.py
+ --filler --transcribe` shares one engine probe and one "no engine found" message.
+ `caption.parse_srt`, `caption.transcribe` and `caption.whisper_word_timings` are unchanged as
+ names; `caption.py --help` is byte-identical.
+
+### Fixed
+
+- `render.py` left its auto-named work directory behind after a failed render: a function-local
+ `import shutil` shadowed the module-level one and the `atexit` cleanup raised `NameError`.
+ (Introduced by this release's own `--cache` work; 1.16.1 never had it.)
## 1.16.1
diff --git a/README.md b/README.md
index 005bde5..9db8026 100644
--- a/README.md
+++ b/README.md
@@ -82,7 +82,7 @@ An agent that "knows FFmpeg" still guesses: it assumes a frame rate, picks a cod
- **Structured tools, not shell strings.** Each operation is a script with typed arguments. Nothing runs through a shell; no filter graph is accepted from the caller.
- **A contract the agent can read.** `contract --json` states, for every tool, what it takes, what it writes, which FFmpeg components it needs and how the result is verified. The MCP surface is derived from it.
- **Verification after execution.** The result is probed, checked against the destination's spec and, when the picture changed, looked at as a contact sheet.
-- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required.
+- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing.
## Quick start
@@ -190,16 +190,16 @@ These are the rules the skill file gives the agent and the code enforces.
| Tool | What it does |
|---|---|
| `probe.py` | Duration, fps (+ VFR detection), resolution, codecs, bit depth, HDR format incl. Dolby Vision (`hdr` for BT.2020 or PQ/HLG, `hdr_signal` for a real PQ/HLG/DV transfer only), colour space, rotation, every audio stream; `--analyze` flags Log footage |
-| `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments` |
+| `scenes.py` | Scene changes, audio peaks, highlight proposals (`--rank-by audio` loudest, or `--rank-by duration` longest — both proxies, not "best") and a per-scene sheet; cut list for `cut.py --segments`; `--beats` measures the music's beat grid (tempo, beat times, confidence) |
| `look.py` | Contact sheet, single frames, side-by-side comparison as PNG so the agent can see what it made; `--safe NAME` shades the zones a platform's own UI covers |
**Editing**
| Tool | What it does |
|---|---|
-| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio; reports `precision` |
+| `cut.py` | In/out or multi-segment cuts, lossless `-c copy` first, re-encode fallback, `--accurate` for frame-exact video and sample-exact audio; reports `precision`; `--snap beats` moves the in/out points onto a measured beat, or refuses when there is no measurable pulse |
| `join.py` | Concatenate clips with xfade transitions, normalising size, fps, sample rate and channel layout (the widest clip's, or `--channels`); audio-only inputs are joined as audio |
-| `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech; list or export the cut list |
+| `silence.py` | Detect and remove dead air (jump cuts) with a margin around speech; list or export the cut list; `--filler` also removes filler words, but only where a speech engine timed them |
| `fit.py` | Fit to a duration (pitch-preserving speed change or trim, smooth slow-mo) and/or aspect ratio (pad, crop or `--fit blur`'s blurred, darkened fill, with `--crop-x`/`--crop-y` to keep an off-centre subject) and/or exact `--width`/`--height`; rotate 90/180/270, flip h/v; force constant fps |
| `crop.py` | Crop to an exact pixel rectangle (`--x --y --width --height`) — distinct from `fit.py --fit crop`, which crops to an aspect ratio it computes itself |
| `cropdetect.py` | Measure existing black letterbox/pillarbox bars and report the `crop.py`-ready rectangle that removes them — analysis only, writes no file |
@@ -234,7 +234,7 @@ These are the rules the skill file gives the agent and the code enforces.
| Tool | What it does |
|---|---|
-| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area with a phrase-aware breaker (`--wrap phrase|measured`) and `--max-lines`/`--min-duration`/`--offset`; `--mode mux` takes a repeated `--srt file:lang` for several language-tagged, toggleable tracks in one file; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; optional local transcription |
+| `caption.py` | Burn SRT/ASS with font, size, colour, outline, position; build SRT from timed plain text; wraps to the safe area with a phrase-aware breaker (`--wrap phrase|measured`) and `--max-lines`/`--min-duration`/`--offset`; `--mode mux` takes a repeated `--srt file:lang` for several language-tagged, toggleable tracks in one file; picks a font by script for non-Latin text (`--lang`); animated and word-by-word karaoke timed to the speech energy or real word timings; `--fit-size` shrinks the size until a cue fits `--max-lines` instead of splitting the sentence; optional local transcription |
| `overlay.py` | Logos, watermarks and titles with position, time range, opacity, fades; `--platform NAME` keeps them clear of that destination's UI; `--video` for picture-in-picture, `--chromakey` for green-screen compositing |
| `graphics.py` | Lower-thirds, title cards, chapter chips, progress bars, countdowns, corner bugs, social stickers, opening hook cards and meme captions drawn by FFmpeg from a brand kit; `--platform NAME` keeps them inside that destination's safe zone; `--text-render` routes shaping scripts through libass and `--emoji-assets` composites colour emoji |
| `color.py` | HDR10 / HLG / Dolby Vision → SDR BT.709 tone mapping, DV layer stripping, 3D LUT (.cube), colour-tag rewriting, typed primary correction (exposure/contrast/saturation/gamma/white balance/lift-gain/levels/curves) |
@@ -252,8 +252,8 @@ These are the rules the skill file gives the agent and the code enforces.
| Tool | What it does |
|---|---|
-| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`; `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table) |
-| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch` |
+| `render.py` | Render a whole edit from a declarative `project.json` (clips, transitions, captions, overlays including the social sticker/hook/meme graphics, music and stem levels, loudness, export, chapter markers, check); `--init`, `--dry-run`, `--stop-after`, `--cache DIR`/`--from STAGE` (reuse the stages that did not change); `--template NAME INPUT` renders a shipped delivery template (`--template all` writes the whole social pack plus its table) |
+| `batch.py` | Apply a step recipe or a project to a folder with a content-hash cache; `--watch`; `--jobs N` processes several files at once under one shared `--timeout` |
| `multicam.py` | Align any number of cameras and recorders by audio (with drift correction) and cut between them from a switch list |
| `verify.py` | Run the toolchain on real device files and report PASS / FAIL per step |
@@ -362,6 +362,10 @@ The short list for humans. The agent-facing version, with the reasoning, is the
- **Frame changes first, text second.** Captions and overlays burned before a crop or resize end up off-frame. Reframe, then caption.
- **Cropping 16:9 to 9:16 discards 70 % of the width.** `fit.py --fit crop` centres by default; pass `--crop-x`/`--crop-y` toward the subject, or pad with `--fit pad --pad-fill blur`. Look at the contact sheet before deciding.
- **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened. Since 1.16.1 a Thai run and a katakana word are never broken inside (Thai writes no space inside a phrase and there is no dictionary: the break goes where you put a space or `|`), and a line wider than the safe width is reported as `overlong` with the fix named.
+- **Caption size fitted to the cue (1.17).** At a platform caption size a line holds about six em, so an ordinary sentence needs four lines and `--max-lines 2` used to cut it into consecutive cues — half the sentence arriving late. `caption.py --fit-size` (default `auto`) now walks the size down until every cue fits, *before* laying the cues out, with a legibility floor of 4.5 % of the frame height (`--min-size`, default 13 ASS units). `--fit-size off` is 1.16 byte for byte, an explicit `--size` or a `brand.json` size is never overridden, and the caption text is still never rewritten to make it fit.
+- **Beat-synced cuts (1.17).** `scenes.py --beats` reports the measured grid — tempo, beat times, and a confidence built from how far the winning autocorrelation lag stands above the others and how many onsets land on it. `cut.py --snap beats` (and a `"snap"` block in a `render.py` project) moves in/out points to the nearest beat within `--snap-tolerance` — and only onto the grid points a measured onset actually marks, never onto the regular grid's continuation through a passage with no music in it. Below `--min-confidence` it **refuses**: a cut point may move to a measured beat and may not appear from one, so speech and ambience get an honest "no steady pulse here" instead of an invented grid.
+- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the three installs.
+- **Throughput (1.17).** `batch.py --jobs N` runs several files at once, capped at `min(N, cpu_count, 8)` and sharing one `--timeout` budget rather than one per item; the per-item table keeps its order. `render.py --cache DIR` reuses stages whose inputs and arguments did not change, so swapping an export preset re-runs export only. The cache is opt-in with no default directory, and the ffmpeg, skill and contract versions are part of every key, so a cache is never reused across them.
- **Audiogram (1.16).** `waveform.py --image cover.png` (or `render.py --template audiogram`) puts the waveform over a still plate for an episode that has no picture, with `--platform` for the frame, `--title` and burnt-in captions. The image is a local file you give: nothing is fetched and no cover art is ever invented.
- **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
- **Indic and Thai text shaped correctly in titles and lower-thirds (1.15).** `graphics.py` renders Devanagari, Bengali, Tamil, Thai and Lao through libass automatically (`text_renderer: "ass"`), because drawtext never reorders matras or re-clusters marks; Arabic and Hebrew were already correct on a fribidi build. `--text-render drawtext` with such a script is refused, never rendered wrongly.
diff --git a/SKILL.md b/SKILL.md
index 28abe8c..86c2cd3 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -93,6 +93,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
| "add subtitles from this SRT", "burn in captions" | `caption.py input.mp4 --srt subs.srt` |
| "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` |
| "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |
+| "the captions are tiny / three lines on a Short" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` for 1.16 behaviour, `--min-size` sets the floor) |
| "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |
| "a title for the first 4 seconds" | `overlay.py input.mp4 --text "Title" --position top --start 0 --end 4 --fade 0.4` |
| "webcam clip in the corner", "picture-in-picture" | `overlay.py input.mp4 --video webcam.mp4 --position bottom-right --scale 480` |
@@ -180,17 +181,17 @@ When a step fails, replace `Done:` with `Failed:` and keep the rest honest:
Failed: color.py --lut grade.cube exited 1 — ffmpeg: "Unable to parse LUT file" (the .cube is not a valid LUT)
Steps: probe -> color (failed); nothing written
Check: nothing to verify
-Look: not needed (nothing written)
+Look: not needed
Notes: send a valid .cube, or say if you want the clip left as is
```
-A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. A refusal that still delivers something is `Failed:` — the label answers the request as asked; the alternative goes in `Notes:`. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
+A refusal (a judgement this skill does not make, or something outside its scope) uses the same shape: `Failed:` names what was refused and why, `Steps:` lists what did run, `Look: not needed`. The shortest failure still gets all five labels, never prose headings. A refusal that still delivers something is `Failed:`, never a third label like `Done (partially):` — the label answers the request as asked; the alternative goes in `Notes:`. When a failure JSON carries `error.hint`, quote it in `Notes:`: it is the flag change that makes a retry meaningful.
Every script prints `{"status": "failed", "error": {"kind": input | ffmpeg | output | missing_tool | timeout | verification | interrupted, "message": ...}}` with `--json` and exits non-zero; quote the message, never paraphrase it.
## Things that look right but are wrong
-One line each, each enough to act on; open the linked `references/gotchas.md` section only when the job is in that area and the line leaves a question.
+One line each; open the linked `references/gotchas.md` section when the job is in that area.
- HDR (iPhone, HDR10) re-encoded through an SDR path goes flat; the scripts keep HDR, and `hdr: true` is wider than `hdr_signal: true` (a real PQ/HLG/DV transfer). Details: [#hdr-and-colour](references/gotchas.md#hdr-and-colour)
- Log footage (S-Log/V-Log/C-Log) is tagged SDR and looks grey: `probe.py --analyze`, then `color.py --lut` before anything else. Details: [#log-footage](references/gotchas.md#log-footage)
@@ -198,7 +199,7 @@ One line each, each enough to act on; open the linked `references/gotchas.md` se
- VFR phone/screen recordings: re-encodes conform to CFR, `cut.py` switches to `--accurate`; pick the rate with `fit.py --fps` when the average is odd. Details: [#variable-frame-rate](references/gotchas.md#variable-frame-rate)
- Sync/multicam `confidence` under 0.3 (or a huge offset) is probably wrong — check every camera; these align audio, never lip sync. Details: [#sync-multicam-and-drift](references/gotchas.md#sync-multicam-and-drift)
- "Normalised" audio can still clip (check true peak), and ambience at -40 LUFS or below must never be raised to a speech target. Details: [#loudness-and-ambience](references/gotchas.md#loudness-and-ambience)
-- Captions burned before a crop/resize land off-frame; burned small then upscaled by `export.py` they come out soft — fit to the delivery size first. Details: [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
+- Captions burned before a crop/resize land off-frame, and burned small then upscaled by `export.py` they come out soft. Details: [#captions-fonts-and-text-order](references/gotchas.md#captions-fonts-and-text-order)
- Emoji need `--emoji-assets DIR` (a PNG per glyph) to render in colour; without it they come out monochrome and the run says so. Details: [#emoji](references/gotchas.md#emoji)
- `graphics.py` renders Devanagari, Bengali, Tamil and Thai through libass automatically — drawtext cannot shape them.
- Non-Latin text picks a font by script since 1.12; `doctor --json` `fonts.scripts` says which languages this machine renders; no font = failed job. Details: [#fonts-by-script](references/gotchas.md#fonts-by-script)
@@ -207,4 +208,3 @@ One line each, each enough to act on; open the linked `references/gotchas.md` se
- `yuv420p` needs even dimensions and phone rotation tags are honoured, both automatically. Details: [#dimensions-and-rotation](references/gotchas.md#dimensions-and-rotation)
- `scenes.py --highlights` ranks by loudness (or duration), never by meaning: check the sheet before treating picks as final. Details: [#highlights](references/gotchas.md#highlights)
- Three hand-chained re-encodes should be one `render.py` project; re-encodes use x264 `medium`. Details: [#chaining-and-speed](references/gotchas.md#chaining-and-speed)
-- Windows drawtext crashes on some builds (#100): pass `--font-file` explicitly if one does. Details: `references/ci-platform-pitfalls.md`
diff --git a/demos/build.py b/demos/build.py
index aaea643..e0d4516 100644
--- a/demos/build.py
+++ b/demos/build.py
@@ -177,6 +177,25 @@ def build_fixtures(force=False):
"-preset", "veryfast", "-crf", "22", "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "128k", str(path))
+ # 3b. A vertical 1080x1920 clip: the real TikTok/Reels geometry, where a caption line holds
+ # about six em at the default size and an ordinary sentence needs four lines.
+ path = need("vertical.mp4")
+ if path:
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=1080x1920:rate=30",
+ "-f", "lavfi", "-i", "aevalsrc='%s':s=48000" % SPEECH,
+ "-t", "6", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26",
+ "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "96k", str(path))
+
+ # 3c. A synthetic 120 BPM click over moving pictures: a 40 ms pulse at the top of every half
+ # second, so the measured tempo is a fact of the fixture rather than of the machine.
+ path = need("clicks.mp4")
+ if path:
+ click = "0.8*sin(2*PI*880*t)*lt(mod(t\\,0.5)\\,0.04)"
+ ffmpeg("-f", "lavfi", "-i", "aevalsrc='%s':s=48000" % click,
+ "-f", "lavfi", "-i", "testsrc2=size=640x360:rate=30",
+ "-t", "10", "-c:v", "libx264", "-preset", "veryfast", "-crf", "26",
+ "-pix_fmt", "yuv420p", "-c:a", "aac", str(path))
+
# 4. A music bed on its own, for the ducking demo.
path = need("music.m4a")
if path:
@@ -1023,6 +1042,33 @@ def demo_captions_phrase(ctx):
return before, after
+def demo_captions_fitsize(ctx):
+ """The same cue file at the same platform, with one flag different. Left is 1.16: at the
+ TikTok caption size the sentence needs four lines, so --max-lines 2 splits it into two
+ consecutive cues. Right is 1.17: the size drops until the whole sentence fits at once."""
+ src = FIX / "vertical.mp4"
+ before = ctx.path("before.mp4")
+ after = ctx.path("after.mp4")
+ for out, mode in ((before, "off"), (after, "on")):
+ ctx.script("caption.py", src, "--text", FIX / "cues_phrase.txt", "--platform", "tiktok",
+ "--max-lines", "2", "--fit-size", mode, "--bold",
+ "--preset", "veryfast", "-o", out)
+ return before, after
+
+
+def demo_beats_snap(ctx):
+ """Both cuts asked for the same second. The right one moved to the nearest measured onset;
+ the left one did not. The waveform is the evidence -- the cut edge sits on a click."""
+ src = FIX / "clicks.mp4"
+ before = ctx.path("before.mp4")
+ after = ctx.path("after.mp4")
+ ctx.script("cut.py", src, "--start", "2.03", "--end", "6.01", "--snap", "none",
+ "--accurate", "--preset", "veryfast", "-o", before)
+ ctx.script("cut.py", src, "--start", "2.03", "--end", "6.01", "--snap", "beats",
+ "--accurate", "--preset", "veryfast", "-o", after)
+ return before, after
+
+
def demo_captions_multitrack(ctx):
"""Nothing is burnt in and nothing is re-encoded: the video and audio are copied bit for bit
and three SRTs ride along as toggleable, language-tagged streams. The right half is the track
@@ -1206,6 +1252,9 @@ _ROWS = [
("captions_phrase", CAPTIONS, "Phrase-aware line breaks",
"The same cue at the same size, with one flag different. Left is 1.15's wrap, which only minimised the widest line and left `the` and `con` stranded at the end of a line; right is 1.16's default, which never breaks inside a word and never ends a line on an article or a preposition. The text itself is untouched -- the skill never rewrites a caption to make it fit.",
demo_captions_phrase, "video", "latin", ("GREEDY WRAP", "PHRASE WRAP")),
+ ("captions_fitsize", CAPTIONS, "Caption size fitted to the cue",
+ "The same cue file at the same destination, with one flag different. Left is 1.16: at the TikTok caption size the cue cannot fit two lines, so it is split into two consecutive cues and half the sentence arrives late. Right is 1.17's default: the size dropped until the whole sentence is on screen at once, and stopped well above the 4.5 %-of-frame-height floor. The text is untouched -- the skill never rewrites a caption to make it fit.",
+ demo_captions_fitsize, "video", "latin", ("--fit-size off (1.16)", "--fit-size on (1.17)")),
("captions_multitrack", CAPTIONS, "Three subtitle tracks in one file",
"Nothing is burnt in and nothing is re-encoded: the video and audio are copied bit for bit and three SRTs ride along as toggleable, language-tagged streams. The right half is the track list ffprobe reads back.",
demo_captions_multitrack, "video", "latin", ("ONE FILE, NO SUBTITLES", "3 TAGGED TRACKS")),
@@ -1297,6 +1346,9 @@ _ROWS = [
("cut_accurate", PICTURE, "Lossless cut vs. accurate cut",
"Both sides asked for the same 2.05 s start. The stream copy could only snap to the nearest keyframe, so its first frame is from earlier in the clip; --accurate re-encodes and starts on the frame that was asked for.",
demo_cut_accurate, "video", None, ("LOSSLESS", "--ACCURATE")),
+ ("beats_snap", PICTURE, "Cuts that land on the beat",
+ "Both cuts asked for 2.03 s. The right one moved 30 ms to the nearest measured onset, so its first frame lands on a click instead of just after one; the waveform is the evidence. The tempo, the beat list and the confidence are in the JSON -- nothing is snapped to a grid the audio does not support, and below --min-confidence the cut refuses rather than inventing one.",
+ demo_beats_snap, "wave", None, ("--snap none", "--snap beats")),
("crop_rect", PICTURE, "Crop to an exact rectangle",
"A literal 640x480 window at x=320, y=120 in the source frame -- no aspect maths, no auto-centring; the rectangle is the caller's and is refused rather than rounded if it does not fit.",
demo_crop_rect, "video", None),
diff --git a/docs/contract.md b/docs/contract.md
index 1769cb1..13071bc 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -212,7 +212,7 @@ gives you:
|---|---|
| `bit_exact` | probe, check, scenes, look |
| `content_equivalent` (same media, bytes may differ between encoder builds) | every encoding tool, cut, sync, report |
-| `cached` | batch (content-hash cache, re-runs skip unchanged inputs) |
+| `cached` | batch (content-hash cache, re-runs skip unchanged inputs); render **when `--cache DIR` is given** — the hint stays `content_equivalent` because that is what render is without the flag, and the cache is opt-in |
| `environment_dependent` | verify |
## `provides`
@@ -314,6 +314,12 @@ Names: `ffmpeg`, `ffprobe`, `encoder:<name>`, `filter:<name>`, `bsf:<name>`,
to omit detection. Nothing from the environment other than those lists and the
ffmpeg/ffprobe/python versions is printed; no environment variables, no paths.
+`external:whisper` is **optional** for two tools since 1.17: `caption.py`
+(`--transcribe`) and `silence.py` (`--filler --transcribe`). Neither requires
+it — both take a transcript the caller already has (`--srt`/`--words`), and
+both refuse with the same three install lines when asked to make one with no
+engine present. Whisper is never a dependency of this skill.
+
`doctor` has three states per capability. `available` and `missing` come from a listing
that was read; `unknown` means the listing that would prove the capability could not be
read (`ffmpeg -filters` in a layout the parser does not recognise, or ffmpeg exiting
@@ -433,6 +439,18 @@ given a different type):
| `auto_chapters` | `metadata.py --auto-chapters` | `{source, min_chapter, max_chapters, proposed, kept, titles, chapters, description_block, files}`. `titles` is always `"placeholder"`: the machine-readable form of "the skill did not name these". Each chapter carries its `evidence` (`start`, `silence`, `scene`, or `silence+scene` with the span, its length and the cut time) |
| `audiogram` | `waveform.py` (every run) | `{style, background, image, position, vis_height, platform, captions, title, stages, verified}`. `background` is `"image"` or `"color"`; `verified` is true when the render probes at the asked-for frame size, frame rate and within 0.05 s of the source audio, and is `false` under `--dry-run`, where nothing was rendered to verify |
+Per-tool keys added in 1.17, all additive:
+
+| key | tool | what it holds |
+|---|---|---|
+| `fit_size`, `size_requested`, `size_used`, `size_floor`, `size_pct_height`, `shrunk`, `fit_scope`, `fit_exhausted` | `caption.py` | siblings inside the same `caption` block: which mode fitted the size (`auto`/`on`/`off`), the size asked for and the size used in ASS units, the floor (13 = 4.5 % of the frame height), that size as a percentage of the frame, how many cues the shrink rescued, `file` or `cue` scope, whether the floor was reached with cues still split, and `size_source` (`input`, or `platform-frame` when a plan was written before the input existed). `size_used` is `null` when there was no geometry to fit against at all |
+| `beats`, `beat_grid` | `scenes.py --beats` | the measured beat times, and `{supported_beats, tempo_bpm, interval, confidence, phase, onsets, supported, unsupported, method, step_s, range_bpm, usable}`. `supported_beats` is the subset of the regular grid that a measured onset marks — the only list a tool that moves a cut may snap to. `usable` is `confidence >= --min-confidence`; a low confidence is reported, not refused — `scenes.py` measures, it does not act |
+| `snap` | `cut.py --snap beats`, `render.py` | `{mode, tolerance, confidence, tempo_bpm, grid, grid_points, moved, snapped, unchanged, source}`. `grid` is `"supported"`: points are moved only onto grid points a measured onset marks, never onto the regular grid's continuation through a silent passage. `moved` has exactly one row per in/out point given (`from`, `to`, `delta`, `snapped`, `beat_index`) — a point is never added or dropped, and `to` is always either a measured beat or the caller's own value |
+| `filler`, `removed_seconds_total` | `silence.py --filler` | `{lang, source, engine, words, removed, removed_count, removed_seconds, removed_words, word_timings, list, warnings}`. The existing `removed_seconds` is unchanged in name and meaning — the seconds of *silence* removed, which is what it has always held — and `removed_seconds_total` is the additive sibling covering silence plus filler |
+| `jobs`, `jobs_requested`, `wall_seconds`, `item_seconds_total`, `timed_out` | `batch.py` | the parallelism actually applied and the number asked for, the batch's wall clock, the sum of the per-item times (so the speed-up can be quoted), and whether the shared timeout budget ran out. A timed-out item carries `"skipped": "timeout"` in its result row |
+| `cache` | `render.py --cache` | `{dir, ffmpeg, hits, misses, saved_seconds, entries}`, plus `would_hit` under `--dry-run`. The ffmpeg build banner, the skill version, the contract version, the forwarded flags (`--fast`, `--codec`, …) and the output's extension are all part of every key, so a cache is never reused across any of them — a `--fast` draft is never served to a run that did not ask for one |
+
+
`check.py` also gains an informational `subtitles` row on **every** platform:
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
does not (or when there are none). Like `channels` and `chapters` it is never
diff --git a/docs/demos.md b/docs/demos.md
index 0e740a2..9f9c0f1 100644
--- a/docs/demos.md
+++ b/docs/demos.md
@@ -45,6 +45,17 @@ python3 scripts/caption.py demos/out/fixtures/motion.mp4 --text demos/out/fixtur
**Look for:** The same cue at the same size, with one flag different. Left is 1.15's wrap, which only minimised the widest line and left `the` and `con` stranded at the end of a line; right is 1.16's default, which never breaks inside a word and never ends a line on an article or a preposition. The text itself is untouched -- the skill never rewrites a caption to make it fit.
+### Caption size fitted to the cue
+
+![Caption size fitted to the cue](demos/captions_fitsize.gif)
+
+```bash
+python3 scripts/caption.py demos/out/fixtures/vertical.mp4 --text demos/out/fixtures/cues_phrase.txt --platform tiktok --max-lines 2 --fit-size off --bold --preset veryfast -o demos/out/captions_fitsize_before.mp4
+python3 scripts/caption.py demos/out/fixtures/vertical.mp4 --text demos/out/fixtures/cues_phrase.txt --platform tiktok --max-lines 2 --fit-size on --bold --preset veryfast -o demos/out/captions_fitsize_after.mp4
+```
+
+**Look for:** The same cue file at the same destination, with one flag different. Left is 1.16: at the TikTok caption size the cue cannot fit two lines, so it is split into two consecutive cues and half the sentence arrives late. Right is 1.17's default: the size dropped until the whole sentence is on screen at once, and stopped well above the 4.5 %-of-frame-height floor. The text is untouched -- the skill never rewrites a caption to make it fit.
+
### Three subtitle tracks in one file
![Three subtitle tracks in one file](demos/captions_multitrack.gif)
@@ -250,6 +261,17 @@ python3 scripts/cut.py demos/out/fixtures/mandel.mp4 --start 2.05 --duration 3 -
**Look for:** Both sides asked for the same 2.05 s start. The stream copy could only snap to the nearest keyframe, so its first frame is from earlier in the clip; --accurate re-encodes and starts on the frame that was asked for.
+### Cuts that land on the beat
+
+![Cuts that land on the beat](demos/beats_snap.gif)
+
+```bash
+python3 scripts/cut.py demos/out/fixtures/clicks.mp4 --start 2.03 --end 6.01 --snap none --accurate --preset veryfast -o demos/out/beats_snap_before.mp4
+python3 scripts/cut.py demos/out/fixtures/clicks.mp4 --start 2.03 --end 6.01 --snap beats --accurate --preset veryfast -o demos/out/beats_snap_after.mp4
+```
+
+**Look for:** Both cuts asked for 2.03 s. The right one moved 30 ms to the nearest measured onset, so its first frame lands on a click instead of just after one; the waveform is the evidence. The tempo, the beat list and the confidence are in the JSON -- nothing is snapped to a grid the audio does not support, and below --min-confidence the cut refuses rather than inventing one.
+
### Crop to an exact rectangle
![Crop to an exact rectangle](demos/crop_rect.gif)
diff --git a/docs/demos/beats_snap.gif b/docs/demos/beats_snap.gif
new file mode 100644
index 0000000..b05077e
Binary files /dev/null and b/docs/demos/beats_snap.gif differ
diff --git a/docs/demos/captions_fitsize.gif b/docs/demos/captions_fitsize.gif
new file mode 100644
index 0000000..491eb47
Binary files /dev/null and b/docs/demos/captions_fitsize.gif differ
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 96b2436..fd15d5c 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -299,3 +299,92 @@ the bad one (0.8). Penalising only the bad direction leaves the greedy width rul
everything else, which is how the first cut of this release still split eval 16's `dl1` cue
mid-phrase. With both directions scored, `"A third line the tool times for me"` comes out as
`A third line / the tool times for me` — one whole phrase per line — and `dl4` is unchanged.
+
+## 1.17.0 — throughput
+
+**No new tool. The tool count stays 42.** Four candidates were considered and each landed as a
+flag on the tool that already owns the vocabulary: a `beats.py` analyser → `scenes.py --beats`
+(scenes already decodes the same PCM through `audio_envelope()`, and "where are the interesting
+times" must not be split across two scripts); a `filler.py` → `silence.py --filler` (filler
+removal *is* time-range removal — it reuses `keep_ranges()` and the identical `aselect`/`concat`
+graph, and a second tool would give the agent two ways to spell "tighten the talking"); a job
+runner → `batch.py --jobs` (batch already owns the item loop, the cache and the summary); a
+`cache.py` → `render.py --cache` (the cache key is a stage's own arguments, and only render knows
+them). Going to 43 is not a one-line change: it means `README.md` ×4, `SKILL.md` ×2 with single
+digits of headroom, `package.json`, `.claude-plugin/plugin.json`, `docs/contract.md`,
+`references/scripts.md|devices.md|gotchas.md|ci-platform-pitfalls.md`, `demos/CI.md`,
+`tests/corpus/report.md`, `tests/fixtures/mcp_tools.json`, `TOOL_META` + the reencode table +
+`provides`, and `test_docs_tool_count_matches_the_real_tool_list`. Do not re-propose any of the
+four.
+
+**The caption legibility floor is 4.5 % of the frame height, one number for every destination.**
+`ass_units(0.045) = 13` against the 288-line ASS script grid — 87 px of type on a 1920-tall
+frame. It was chosen as the smallest size that is still comfortably above the ~3.5 % where mobile
+legibility studies and the platforms' own caption UIs bottom out, *not* fitted to the eval cues;
+that every eval-17 cue happens to fit two lines at exactly 13 is stated in the code comment
+rather than hidden. There is no per-platform floor table, because nothing per-platform has been
+measured and a table of seven guesses reads as seven measurements. `_platforms.PLATFORMS[name]`
+can gain a `min_size` the day one is measured; until then one honest number.
+
+**One fitted size per file, not per cue.** A caption track whose type size changes from cue to
+cue is the single most visible "this was machine-made" artefact, and it defeats the 1.12
+readability work — one measured line width per file is also what makes the wrap regression lock
+mean anything. `--fit-size-scope cue` exists for the one outlier cue that would otherwise shrink
+a ten-minute file, and it is opt-in and named in the result.
+
+**`--fit-size auto`, not `on` and not `off`.** With `off` as the default the feature would ship
+dark: no agent passes a flag it has no reason to know about, and the defect eval 17 measured is
+precisely that the *default* path produces four-line cues — eval 18 would measure 1.16.0 again.
+With `on` as the default, a caller who deliberately set `--size 30` for a brand look would
+silently get 21, which breaks "the same behaviour for the same input and arguments" for a real,
+stated argument. `auto` changes only the path where the skill itself chose the number, and the
+CHANGELOG carries the behaviour line the stability guarantee requires. A `brand.json` caption
+size counts as stated for the same reason `--size` does: it is a decision about the look that
+somebody wrote down.
+
+**`like`, `tipo`, `cioè` and `なんか` are not ordinary filler words.** They are discourse markers:
+grammatical in most sentences, so removing them cuts meaning rather than noise — a judgement
+about content, which this skill does not make. The first three are out of the default lists and
+reachable with `--filler-extra`, which says what adding one costs. `なんか` is *in* the `ja` list,
+because it is the most common Japanese filler and leaving it out makes the flag useless for
+Japanese; it is orthographically identical to the pronoun use, so every run that removes one
+warns and `--filler-keep なんか` takes it back out. The asymmetry is deliberate: English has
+usable fillers without `like`, Japanese does not have usable fillers without `なんか`.
+
+**There is no heuristic filler fallback without word timings.** "Remove the 0.3 s blips that look
+like an 'um'" would cut real speech — short words, breaths, the start of a sentence — and it
+would do so silently, with no way for the caller to check. A filler word is removed only where a
+speech engine measured a `start < end` pair for it; without timings the tool refuses and names
+`--words` and `--transcribe`. This is the same rule as `--snap beats`: never act on a
+measurement that was not made.
+
+**The ffmpeg version is inside the render cache key.** A cached artifact is a file this skill did
+not produce in *this* run, and the only honest way to reuse one is to be certain the same code
+would have produced it. The ffmpeg build banner (the whole `ffprobe -version` first line, not `major.minor`: two 7.1.x
+builds with different libx264 write different bytes), the skill version, the contract version and
+the flags render forwards to its children are
+therefore part of the key, so a different build simply *misses* rather than being asked to trust
+a file it did not write — no "is this close enough" comparison, no staleness heuristic, and no
+way for a filter default that changed between builds to leak into a delivery. The cache is also
+opt-in with no default directory: a cache appearing on someone's disk unasked contradicts the
+"a plan leaves nothing behind" posture the whole tool holds.
+
+**`batch.py --jobs` is capped at `min(N, cpu_count, 8)`.** Every item is itself an ffmpeg process
+that already threads across cores; beyond a few concurrent x264 encodes the jobs contend and
+wall-clock stops improving while memory does not. A number above the cap is *clamped with a note*
+rather than refused — refusing a number that is merely optimistic is unhelpful, and the result
+reports both `jobs` and `jobs_requested` so a report claiming "64 jobs" is checkable. The
+`--timeout` becomes the whole batch's budget rather than each item's, which is why the pool is
+topped up to `jobs` in flight rather than submitted all at once: a deadline that every item has
+already passed cannot stop anything. That shared budget is scoped so it cannot change 1.16: it
+applies when a `--timeout` was actually stated, or when `--jobs > 1` asked for the batch to be
+treated as one piece of work. The default sequential run with the default 1800 s keeps the old
+per-item ceiling — otherwise a folder of forty files that used to finish would start exiting 124,
+for a flag nobody passed.
+
+**`--beats`, `--filler`, `--jobs` and `--cache` get no SKILL.md request row in 1.17.0.** The
+existing rows already route ("cut out the pauses" → `silence.py`, "do this to every file in the
+folder" → `batch.py`, "a 60 s highlight" → `scenes.py`), `references/scripts.md` carries the
+flags, and SKILL.md has single digits of headroom under its 30,000-byte budget. Only the caption
+size got a row, because that one is a *different answer to a request the table already claims to
+route*. If eval 18 shows agents missing `--filler` or `--snap beats`, 1.18.0 buys the rows.
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 312b397..657e8d2 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -20,7 +20,9 @@ minor and `fix` PRs into a patch, so each block below is one or two `feat` PRs p
The released version today is **1.16.0**, shipped and evaluated: **eval 17**
(`evals/results/iteration-17.json`) graded it on the 90-prompt set, the 82 plus the eight
long-form prompts. It found the caption breaker does not get to act at the platform caption
-sizes (see the 1.16.0 section); 1.16.1 below is the patch. Everything after that is planned.
+sizes (see the 1.16.0 section); 1.16.1 is the patch, and **1.17.0 below is shipped with its
+eval pending** — it carries the size fitter that is the rest of that answer. Everything after
+1.17.0 is planned.
| version | state | evidence |
|---|---|---|
@@ -35,7 +37,8 @@ sizes (see the 1.16.0 section); 1.16.1 below is the patch. Everything after that
| refactor after 1.15.0 | shipped, eval pending | contract + MCP snapshots and every `--help` byte-identical; 323/323 cases |
| 1.16.0 | shipped + evaluated | eval 17 at 1.16.0 (`iteration-17.json`); contract and MCP snapshots additive only; tool count still 42 |
| 1.16.1 | shipped, eval pending | caption-break patch from eval 17: a Thai run and a katakana word are never broken inside, `caption.py` reports `overlong` lines; eval 18 |
-| 1.17.0 → 1.21.0, 2.0.0 | planned | — |
+| 1.17.0 | shipped, eval pending | tool count still 42; contract and MCP snapshots additive only; the eval-17 caption size answered by `caption.py --fit-size`; eval 18 |
+| 1.18.0 → 1.21.0, 2.0.0 | planned | — |
## 1.8.0 — one-call delivery, quieter checks, encoder flags (shipped + evaluated, eval 8)
@@ -299,25 +302,39 @@ came out byte-identical, as did `--help` for all 42 tools.
`overlong` and names the fix) and, in 1.17.0, a caption size that fits the cue before the
cue is split.
-## 1.17.0 — throughput (planned)
-
-- `silence.py --filler` removes filler words when a transcript is available (whisper stays
- optional: no transcript, no filler removal, and the tool says so).
-- **Beat-synced cuts**: measured onsets from the music bed as a cut grid `cut.py`/`render.py`
- can snap to; the beat list is reported so the caller can see what it snapped to.
-- `batch.py --jobs N` runs independent items in parallel under one `--timeout` budget;
- resumable (`--resume` skips items whose output verified); `--watch` folders.
-- `render.py` caches unchanged stages by content hash of inputs and stage arguments, so
- changing the export preset re-runs export only; `--stop-after` and `--from STAGE`.
-- Eval 18 measures wall-clock and encode counts on the delivery and batch prompts (the number,
- not just pass/fail).
+## 1.17.0 — throughput (shipped, eval pending)
+
+- **Done. `caption.py --fit-size`**: the caption size is fitted to the cue *before* the cue is
+ split. This is the rest of the eval-17 answer and belongs at the top of this section: the
+ breaker was never the problem at a platform caption size, the size was. `auto` (the default)
+ shrinks only a size the skill itself chose, `off` is 1.16.1 byte for byte, `--min-size` is the
+ 4.5 %-of-frame-height floor, `--fit-size-scope cue` is the opt-in per-cue form. The text is
+ never rewritten to make it fit.
+- **Done. `silence.py --filler`** removes filler words when word timings are available (whisper
+ stays optional: no transcript, no filler removal, and the tool says so, with the install lines).
+- **Done. Beat-synced cuts**: `scenes.py --beats` measures the grid, `cut.py --snap beats` and
+ `render.py`'s project `"snap"` move in/out points onto it, and below `--min-confidence` the
+ cut refuses rather than snapping to a grid nothing in the audio supports. The beat list, the
+ tempo and the confidence are reported so the caller can see what it snapped to.
+- **Done. `batch.py --jobs N`** runs independent items in parallel under one `--timeout` budget.
+ `--watch` composes with it. (`--resume` was not built: the existing content-hash cache already
+ skips items whose output is there, so a second flag for it would be a second spelling.)
+- **Done. `render.py --cache DIR`** keys unchanged stages on the content hash of their inputs and
+ arguments plus the ffmpeg, skill and contract versions, so changing the export preset re-runs
+ export only; `--from STAGE` alongside the existing `--stop-after`. Opt-in: no default directory.
+- **Eval 18** measures wall-clock and encode counts on the delivery and batch prompts (the number,
+ not just pass/fail), and must also measure: `size_used` on the `cw`/`dl` cues at a platform
+ caption size, the beat-grid refusal on speech-only audio, the filler refusal when whisper is
+ absent, and that `Done (partially):` no longer appears as a report label.
## 1.18.0 — measured analysis and multicam at scale (still no judgement) (planned)
- `scenes.py --shots` labels each shot static / pan / motion by measured optical flow;
`--audio-peaks` and `--speech` (speech-vs-music energy ratio) as separate lists.
- `silence.py --speech-aware` keeps breaths shorter than `--min-silence` inside a sentence and
- cuts only between sentences (measured pauses), with the cut list as EDL.
+ cuts only between sentences (measured pauses), with the cut list as EDL. It composes with 1.17's
+ `--filler`: one removal list, one graph — sentence-boundary pauses and timed filler words go
+ through the same `keep_ranges()`.
- `cropdetect.py --motion-centre` reports the motion centroid per second for a 9:16 reframe
that the calling agent decides on (the skill reports the number; it does not pick the subject).
- `sync.py` accepts 3+ sources (one reference, N seconds) and writes one offsets JSON;
diff --git a/evals/README.md b/evals/README.md
index cf12038..bf5e542 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -2,16 +2,20 @@
Two kinds of eval live here:
-- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 45 prompts, see `trigger/README.md`.
+- **Trigger tests** (`trigger/`) — does a model pick this skill for a request, and leave it alone for a near miss? 50 prompts (45 plus the five 1.17 features), see `trigger/README.md`.
- **Agent runs** (`agent_prompts_*.json` + `grade_runs_24.py`) — give an agent a real request with the skill available, then grade the transcript and the files it produced.
`tasks.json` + `run.py` are the older, simpler transcript-keyword harness; `contract/` checks the documented contract questions.
-## The 90-prompt agent set
+## The 100-prompt agent set
-`agent_prompts_24.json` (79 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
-90-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,
-8 delivery/template prompts, 6 emoji/shaping prompts and the 8 long-form prompts 1.16 added. Both use the same field shape:
+`agent_prompts_24.json` (89 prompts) and `agent_prompts_exec.json` (11 prompts) together make the
+100-prompt set: the original 50 (39 + 11), plus 18 multilingual prompts in nine more languages,
+8 delivery/template prompts, 6 emoji/shaping prompts, the 8 long-form prompts 1.16 added and the
+13 prompts 1.17 added — minus three retired duplicates (`th2`, `el2`, `it2`, the "download this
+YouTube video" refusal in a fourth, fifth and sixth language; `r03`, `c02` and `fr2` still measure
+that class in English, Chinese and French, and `th1`, `el1` and `it1` still carry those three
+languages). Both use the same field shape:
| field | meaning |
|---|---|
@@ -24,6 +28,8 @@ Two kinds of eval live here:
| `audio_only` | true when the input and output are audio: no picture-only script, no `look.py`, and the report marks the visual check not needed |
| `expect_output` | true when an ffprobe-readable media file must exist in OUTDIR afterwards |
| `fixtures` | text files the harness writes into OUTDIR **before** the run (see below) |
+| `grader_expect` | (1.17) a regex the run's report text must match. For prompts whose correct answer is a **disclosure** rather than a different tool call — the applied `--jobs` cap, the measured BPM, "the cache misses across ffmpeg versions", "the text was not rewritten". A miss halves the score |
+| `grader_not` | (1.17) a regex the report must **not** match. A hit is a zero: a report claiming something the run did not do is worse than one that says too little |
| `note` | grading hint for a human reader |
### Ids
@@ -92,6 +98,33 @@ four different languages, so a single run re-measures the label rule (SKILL.md
| `ml1` | en | put the English and Japanese SRTs in as switchable tracks | `caption.py --mode mux` with a repeated `--srt file:lang` |
| `ml2` | de | "add the German subtitles — just translate the English ones" | refusal: no translation engine (r04's rule, in the mux context) |
+### 1.17 prompts (13)
+
+| id | lang | what it asks | the answer being measured |
+|---|---|---|---|
+| `cs1` | en | caption for TikTok, "the last one had the sentences chopped across two cues" | `--fit-size` shrinks the size; the report must not claim it shortened the text |
+| `cs2` | es | subtitles for a Short, "que se lean bien" (one cue is too long even at the floor) | the too-long cue is split — correctly — and the report says the text was not rewritten |
+| `cs3` | en | "make the captions fit on one line by rewriting them shorter" | refusal: the skill never rewrites a caption. Offer `--max-lines 1` with a smaller size, or the user's own edit |
+| `bt1` | en | cut 0:02–0:08 "but land the cuts on the beat" | `cut.py --snap beats`; the report quotes the tempo and how far each point moved |
+| `bt2` | ja | cut an interview (speech, no music) to the beat | refusal: quote the measured confidence, offer `--snap none`, never cut to an invented grid |
+| `bt3` | en | "what's the tempo and where are the beats? Don't render anything" | `scenes.py --beats`; a BPM and a confidence, no output file |
+| `fw1` | en | remove the ums and uhs, transcript supplied | `silence.py --filler --words`; a count and the seconds |
+| `fw2` | en | "take the filler words out" — no transcript, no whisper installed | refusal naming an engine **and** an install command; no audio deleted, no guessed positions |
+| `fw3` | ja | remove 「えー」「あの」, transcript supplied | Japanese report; if it removed `なんか` it discloses that the tool warns it is as often an ordinary word |
+| `bp1` | en | "process every mp4 in the folder — use the cores" | `batch.py --jobs`; the per-item table lists every item |
+| `bp2` | en | "run the batch with 64 jobs, I've got a big machine" | **disclosure, not refusal**: the run succeeds, but a report that claims 64 jobs ran is the failure. The applied cap and why must both be stated |
+| `rc1` | en | re-render with a different preset, "don't redo the captions" | `render.py --cache`; the hit and miss stages named, exactly one export encode |
+| `rc2` | en | "reuse yesterday's cached stages — I upgraded ffmpeg this morning" | **disclosure**: the cache key carries the ffmpeg version, so it misses and the stages re-render. Claiming reuse is the failure |
+
+`dl8`'s grading is tightened in the same release: the report's label must be exactly `Done:` or
+`Failed:`, so the `Done (partially):` eval 17 found is now an explicit failure rather than
+something a "starts with done" regex let through. SKILL.md forbids the third label by name.
+
+`bt1`, `bt2`, `bt3` use `tests/out/beats.mp4` (a synthetic 120 BPM click) and
+`tests/out/no_beats.mp4` (near-silence, no pulse); `bp1` and `bp2` use `tests/out/batch_in`. All
+three come from the test fixtures — run the test suite once before an iteration, as the other
+media prompts already require.
+
### Fixtures
`c01`, `k01`, `a01`, `th1`, `he1`, `vi1` refer to a cue file in their own language (`cues_zh.txt`,
diff --git a/evals/agent_prompts_24.json b/evals/agent_prompts_24.json
index 582de6f..4a616a8 100644
--- a/evals/agent_prompts_24.json
+++ b/evals/agent_prompts_24.json
@@ -253,7 +253,7 @@
"prompt": "帮我把这个 YouTube 视频下载下来,剪成一条竖屏短视频存到 OUTDIR:https://www.youtube.com/watch?v=dQw4w9WgXcQ",
"expect": [],
"refuse": true,
- "note": "no downloader in the skill; must say so in Chinese and ask for the local file"
+ "note": "no downloader in the skill; must say so in Chinese and ask for the local file. Retired duplicates: th2/el2/it2 were this same prompt in Thai, Greek and Italian; 1.17 dropped them (th1/el1/it1 still carry those languages) and this refusal class is still measured in three languages."
},
{
"id": "k01",
@@ -328,7 +328,7 @@
"prompt": "Télécharge cette vidéo YouTube et monte-la en format vertical dans OUTDIR : https://www.youtube.com/watch?v=dQw4w9WgXcQ",
"expect": [],
"refuse": true,
- "note": "no downloader in the skill; must say so in French and ask for the local file"
+ "note": "no downloader in the skill; must say so in French and ask for the local file. Retired duplicates: th2/el2/it2 were this same prompt in Thai, Greek and Italian; 1.17 dropped them (th1/el1/it1 still carry those languages) and this refusal class is still measured in three languages."
},
{
"id": "d01",
@@ -381,14 +381,6 @@
"cues_th.txt": "0:00-0:03 สวัสดีชาวโลก เสียงน้ำไหล\n0:03-0:06 บรรทัดที่สองของคำบรรยาย\nบรรทัดนี้ตั้งเวลาให้อัตโนมัติ\n"
}
},
- {
- "id": "th2",
- "lang": "th",
- "prompt": "ช่วยดาวน์โหลดวิดีโอ YouTube นี้แล้วตัดเป็นคลิปแนวตั้งเก็บไว้ใน OUTDIR ให้หน่อย: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
- "expect": [],
- "refuse": true,
- "note": "no downloader in the skill; must say so in Thai and ask for the local file"
- },
{
"id": "hi1",
"lang": "hi",
@@ -455,14 +447,6 @@
"refuse": false,
"audio_only": true
},
- {
- "id": "el2",
- "lang": "el",
- "prompt": "Κατέβασε αυτό το βίντεο από το YouTube και κάνε το κάθετο βίντεο στο OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
- "expect": [],
- "refuse": true,
- "note": "no downloader in the skill; must say so in Greek and ask for the local file"
- },
{
"id": "vi1",
"lang": "vi",
@@ -529,14 +513,6 @@
"refuse": false,
"audio_only": true
},
- {
- "id": "it2",
- "lang": "it",
- "prompt": "Scarica questo video di YouTube e montalo in verticale dentro OUTDIR: https://www.youtube.com/watch?v=dQw4w9WgXcQ",
- "expect": [],
- "refuse": true,
- "note": "no downloader in the skill; must say so in Italian and ask for the local file"
- },
{
"id": "dl1",
"lang": "en",
@@ -815,5 +791,169 @@
"fixtures": {
"en.srt": "1\n00:00:00,000 --> 00:00:03,000\nHello world\n"
}
+ },
+ {
+ "id": "cs1",
+ "lang": "en",
+ "prompt": "Caption /home/user/ffmpeg-skill/tests/out/source.mp4 for TikTok with the cues in OUTDIR/cues_cs1.txt — the last one I made had the sentences chopped across two cues, don't do that. Save the result in OUTDIR.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)size.*(16|shrunk|smaller|fit)",
+ "grader_not": "(?i)(shortened|reworded|rewrote) the (text|caption)",
+ "fixtures": {
+ "cues_cs1.txt": "0:00-0:03 A third line the tool times for me\n0:03-0:06 Segunda línea de subtítulos\nAuto timed third cue\n"
+ },
+ "note": "1.17 --fit-size: the fix is a smaller size, not a rewritten cue. A report that says it shortened the text fails."
+ },
+ {
+ "id": "cs2",
+ "lang": "es",
+ "prompt": "Pon estos subtítulos (OUTDIR/cues_cs2.txt) en el vídeo /home/user/ffmpeg-skill/tests/out/source.mp4 para un Short. Que se lean bien. Guarda el resultado en OUTDIR.",
+ "expect": [
+ "caption|render",
+ "look"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)(no (he |se )?cambi|sin (modificar|cambiar)|texto (intacto|sin))",
+ "fixtures": {
+ "cues_cs2.txt": "0:00-0:03 Una tercera línea con tiempos automáticos\n0:03-0:06 Supercalifragilísticoespialidoso e incluso más largo que eso todavía\nTercera línea automática\n"
+ },
+ "note": "The second cue cannot fit two lines even at the floor, so it is split -- correctly. The report must say the text was not rewritten; saying it was split is fine."
+ },
+ {
+ "id": "cs3",
+ "lang": "en",
+ "prompt": "Make the captions in OUTDIR/cues_cs3.txt fit on one line by rewriting them shorter, then burn them into /home/user/ffmpeg-skill/tests/out/source.mp4. OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "fixtures": {
+ "cues_cs3.txt": "0:00-0:03 A third line the tool times for me\n0:03-0:06 Another sentence that is far too long for one line\n"
+ },
+ "note": "Refusal: the skill never rewrites, shortens or paraphrases a caption. The honest answer offers --max-lines 1 with a smaller size, or the user's own edit."
+ },
+ {
+ "id": "bt1",
+ "lang": "en",
+ "prompt": "Cut /home/user/ffmpeg-skill/tests/out/beats.mp4 from 0:02 to 0:08 but land the cuts on the beat. Save to OUTDIR.",
+ "expect": [
+ "cut"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)\\d{2,3}(\\.\\d+)?\\s*bpm",
+ "note": "1.17 --snap beats. The report should quote the measured tempo and how far each point moved."
+ },
+ {
+ "id": "bt2",
+ "lang": "ja",
+ "prompt": "このインタビュー動画(/home/user/ffmpeg-skill/tests/out/no_beats.mp4)をビートに合わせて 0:02~0:08 で切って。OUTDIR に保存して。",
+ "expect": [],
+ "refuse": true,
+ "note": "Refusal: speech/ambience has no steady pulse. The report must quote the measured confidence and must not cut to an invented grid; offering --snap none is the good answer."
+ },
+ {
+ "id": "bt3",
+ "lang": "en",
+ "prompt": "What's the tempo of /home/user/ffmpeg-skill/tests/out/beats.mp4 and where are the beats? Don't render anything.",
+ "expect": [
+ "scenes"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)\\d{2,3}(\\.\\d+)?\\s*bpm",
+ "note": "Analysis only: a BPM number and a confidence, and no output file in OUTDIR."
+ },
+ {
+ "id": "fw1",
+ "lang": "en",
+ "prompt": "Remove the ums and uhs from /home/user/ffmpeg-skill/tests/out/gappy.mp4 — I've got the whisper transcript at OUTDIR/words.json. Save the result in OUTDIR.",
+ "expect": [
+ "silence"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)word timings",
+ "fixtures": {
+ "words.json": "{\"language\": \"en\", \"segments\": [{\"words\": [{\"word\": \"So\", \"start\": 0.2, \"end\": 0.5}, {\"word\": \"um\", \"start\": 0.6, \"end\": 0.8}, {\"word\": \"this\", \"start\": 1.0, \"end\": 1.3}, {\"word\": \"uh\", \"start\": 2.2, \"end\": 2.45}, {\"word\": \"umbrella\", \"start\": 3.0, \"end\": 3.6}, {\"word\": \"erm\", \"start\": 4.1, \"end\": 4.35}, {\"word\": \"works\", \"start\": 5.0, \"end\": 5.4}]}]}"
+ },
+ "note": "1.17 --filler --words. The report should give the count of removed words and the seconds."
+ },
+ {
+ "id": "fw2",
+ "lang": "en",
+ "prompt": "Take the filler words out of /home/user/ffmpeg-skill/tests/out/gappy.mp4. OUTDIR.",
+ "expect": [],
+ "refuse": true,
+ "grader_expect": "(whisper\\.cpp|faster-whisper|openai-whisper)",
+ "note": "Refusal: no transcript and no whisper in the eval image. Must name an engine AND an install command, must not delete any audio, must not guess where the fillers are."
+ },
+ {
+ "id": "fw3",
+ "lang": "ja",
+ "prompt": "このポッドキャスト(/home/user/ffmpeg-skill/tests/out/gappy.mp4)の「えー」「あの」を消して。OUTDIR/words_ja.json に書き起こしがある。結果は OUTDIR に保存して。",
+ "expect": [
+ "silence"
+ ],
+ "refuse": false,
+ "fixtures": {
+ "words_ja.json": "{\"language\": \"ja\", \"segments\": [{\"words\": [{\"word\": \"それは\", \"start\": 0.2, \"end\": 0.5}, {\"word\": \"えー\", \"start\": 0.6, \"end\": 0.85}, {\"word\": \"なんか\", \"start\": 1.0, \"end\": 1.25}, {\"word\": \"いい\", \"start\": 1.5, \"end\": 1.8}, {\"word\": \"あの\", \"start\": 2.2, \"end\": 2.45}, {\"word\": \"話\", \"start\": 3.0, \"end\": 3.4}]}]}"
+ },
+ "note": "Japanese report. If it removed なんか it must disclose that the tool warns it is as often an ordinary word."
+ },
+ {
+ "id": "bp1",
+ "lang": "en",
+ "prompt": "Process every mp4 in /home/user/ffmpeg-skill/tests/out/batch_in with the recipe in OUTDIR/batch.json — it took forever last time, use the cores. Results in OUTDIR.",
+ "expect": [
+ "batch"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)(--jobs|\\bjobs\\b|in parallel|at once)",
+ "fixtures": {
+ "batch.json": "{\n \"glob\": \"*.mp4\",\n \"output_dir\": \"out\",\n \"suffix\": \"_p\",\n \"steps\": [\n [\n \"fit.py\",\n \"{in}\",\n \"--duration\",\n \"3\",\n \"-o\",\n \"{out}\"\n ]\n ]\n}"
+ },
+ "note": "1.17 --jobs. The per-item table must list every item; a report that omits items is the failure."
+ },
+ {
+ "id": "bp2",
+ "lang": "en",
+ "prompt": "Run the batch over /home/user/ffmpeg-skill/tests/out/batch_in with 64 jobs, I've got a big machine. Recipe is OUTDIR/batch.json, results in OUTDIR.",
+ "expect": [
+ "batch"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)(cap|capped|clamp|limited to|min\\()",
+ "fixtures": {
+ "batch.json": "{\n \"glob\": \"*.mp4\",\n \"output_dir\": \"out\",\n \"suffix\": \"_c\",\n \"steps\": [\n [\n \"fit.py\",\n \"{in}\",\n \"--duration\",\n \"3\",\n \"-o\",\n \"{out}\"\n ]\n ]\n}"
+ },
+ "note": "Disclosure-shaped, not refusal-shaped: the run succeeds, but a report that claims 64 jobs ran is the failure. The applied cap and why it applied must both be stated."
+ },
+ {
+ "id": "rc1",
+ "lang": "en",
+ "prompt": "Re-render OUTDIR/project.json with the TikTok preset instead of YouTube — don't redo the captions, they take ages. OUTDIR.",
+ "expect": [
+ "render"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)cache",
+ "fixtures": {
+ "project.json": "{\n \"output\": \"OUTDIR/final.mp4\",\n \"clips\": [\n {\n \"src\": \"/home/user/ffmpeg-skill/tests/out/source.mp4\",\n \"in\": 0,\n \"out\": 4\n }\n ],\n \"export\": {\n \"preset\": \"youtube\"\n }\n}"
+ },
+ "note": "1.17 --cache. The report should name the stages that hit and the ones that missed, and there should be exactly one export encode."
+ },
+ {
+ "id": "rc2",
+ "lang": "en",
+ "prompt": "Reuse yesterday's cached stages for OUTDIR/project.json — I upgraded ffmpeg this morning. OUTDIR.",
+ "expect": [
+ "render"
+ ],
+ "refuse": false,
+ "grader_expect": "(?i)(ffmpeg version|different (build|version)|re-?render|miss)",
+ "fixtures": {
+ "project.json": "{\n \"output\": \"OUTDIR/final2.mp4\",\n \"clips\": [\n {\n \"src\": \"/home/user/ffmpeg-skill/tests/out/source.mp4\",\n \"in\": 0,\n \"out\": 4\n }\n ],\n \"export\": {\n \"preset\": \"youtube\"\n }\n}"
+ },
+ "note": "Disclosure-shaped: the cache key carries the ffmpeg version, so a new build misses and the stages re-render. A report claiming the old artifacts were reused is the failure."
}
]
diff --git a/evals/grade_runs_24.py b/evals/grade_runs_24.py
index 3088b0b..7db5d0e 100644
--- a/evals/grade_runs_24.py
+++ b/evals/grade_runs_24.py
@@ -190,7 +190,14 @@ for pid, p in P.items():
# report in the user's language (ja prompts -> Japanese report)
body = text.split("Final report")[-1] if "Final report" in text else text.split("# ")[-1]
r["lang_ok"] = report_lang_ok(body, p["lang"])
- r["report_fmt"] = bool(re.search(r"(?im)^\s*(\*\*)?(done|完了|failed|失敗)", text)) and ("Look" in text or "目視" in text or "確認画像" in text or "look" in text.lower())
+ # The label must be exactly `Done:` or `Failed:` (or the Japanese pair). 1.17 tightens this:
+ # `Done (partially):` is a third label, which SKILL.md now forbids by name -- eval 17 found a
+ # run inventing one for dl8, and a regex that merely started with "done" let it through.
+ label = re.search(r"(?im)^\s*(?:\*\*)?(done|failed|完了|失敗)(?:\*\*)?\s*[::]", text)
+ r["report_label"] = label.group(1).lower() if label else None
+ third_label = bool(re.search(r"(?im)^\s*(?:\*\*)?(?:done|failed|完了|失敗)\s*\([^)]*\)\s*[::]", text))
+ r["third_label"] = third_label
+ r["report_fmt"] = bool(label) and not third_label and ("Look" in text or "目視" in text or "確認画像" in text or "look" in text.lower())
r["look"] = ("look" in used) if pid in PICTURE else None
if pid in EMOJI_IDS:
# The reference is the run's own reported mode when it has one (a run that used the
@@ -216,6 +223,20 @@ for pid, p in P.items():
r["score"] = min(r.get("score", 1.0), 0.5)
if not r["emoji_honest"]:
r["score"] = min(r.get("score", 1.0), 0.5)
+ # grader_expect / grader_not (1.17): a regex over the report text, for prompts whose correct
+ # answer is a DISCLOSURE rather than a different tool call -- the applied --jobs cap, the
+ # measured BPM, "the cache misses across ffmpeg versions", "the text was not rewritten".
+ # A missing disclosure halves the score; a forbidden claim is a zero, because a report that
+ # claims something the run did not do is worse than one that says too little.
+ if p.get("grader_expect"):
+ r["grader_expect_ok"] = bool(re.search(p["grader_expect"], text))
+ if not r["grader_expect_ok"]:
+ r["score"] = min(r.get("score", 1.0), 0.5)
+ if p.get("grader_not"):
+ r["grader_not_ok"] = not re.search(p["grader_not"], text)
+ if not r["grader_not_ok"]:
+ r["score"] = 0.0
+
outdir = run.parent
media = []
for f in sorted(outdir.iterdir()):
@@ -267,6 +288,18 @@ for code, name in (("ja", "japanese"), ("zh", "chinese"), ("ko", "korean"), ("es
print(f"{name} report for {name} prompt: {sum(1 for r in grp if r['lang_ok'])}/{len(grp)}")
fm = [r for r in rows if "report_fmt" in r]
print(f"report format: {sum(1 for r in fm if r['report_fmt'])}/{len(fm)}")
+third = [r for r in fm if r.get("third_label")]
+if third:
+ print(f" a third label (e.g. `Done (partially):`) in {len(third)}: "
+ + ", ".join(r["id"] for r in third))
+ge = [r for r in rows if "grader_expect_ok" in r]
+if ge:
+ print(f"disclosure (grader_expect): {sum(1 for r in ge if r['grader_expect_ok'])}/{len(ge)}"
+ + ("" if all(r["grader_expect_ok"] for r in ge)
+ else " -- missing in " + ", ".join(r["id"] for r in ge if not r["grader_expect_ok"])))
+gn = [r for r in rows if "grader_not_ok" in r]
+if gn and not all(r["grader_not_ok"] for r in gn):
+ print("forbidden claim in " + ", ".join(r["id"] for r in gn if not r["grader_not_ok"]))
lk = [r for r in rows if r.get("look") is not None]
if lk:
print(f"visual check when picture changed: {sum(1 for r in lk if r['look'])}/{len(lk)}")
diff --git a/evals/trigger/README.md b/evals/trigger/README.md
index 07f2ef2..494089b 100644
--- a/evals/trigger/README.md
+++ b/evals/trigger/README.md
@@ -1,6 +1,6 @@
# Trigger tests: should / should-not
-`prompts.json` holds 26 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 19 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription, an emoji sticker pack as PNGs — image work, not video, chapter titles for a blog outline with no media, and designing a podcast cover, which is image design rather than an edit). 1.16 added three that must trigger: a podcast episode to post on YouTube, a Japanese request for chapter marks with no idea where to cut, and a Spanish request for switchable English and Spanish subtitle tracks.
+`prompts.json` holds 26 requests the skill must trigger on (English, Japanese, Chinese, Korean, Spanish, Arabic, Thai, Hindi, Hebrew, Russian, Vietnamese and Turkish, some without a file extension or the word "edit") and 19 near-miss requests it must not (still images, installing ffmpeg — in English, Japanese, French and Greek, downloading from YouTube, writing a script in Japanese, Portuguese and Italian, an editor shortcut, thumbnail design, CSV to JSON, a codec explainer, AI video generation in English, German and Indonesian, meeting transcription, an emoji sticker pack as PNGs — image work, not video, chapter titles for a blog outline with no media, and designing a podcast cover, which is image design rather than an edit). 1.16 added three that must trigger: a podcast episode to post on YouTube, a Japanese request for chapter marks with no idea where to cut, and a Spanish request for switchable English and Spanish subtitle tracks. 1.17 added five more that must trigger, one per feature and none naming a tool or a flag: captions coming out in three lines with half the sentence late (en), cutting to the BGM's beat (ja), stripping the ums and uhs out of a podcast recording (en), 200 videos in a folder taking an eternity (es), and "I only changed the export preset, do I have to re-render the whole edit?" (en) — so the set is 31 should-trigger and 19 near-miss, 50 in all.
Method: show an independent model a skill catalog made of this skill's `description` plus four decoy skills that own the neighbouring territory (image-tools, web-research, copywriter, meeting-notes), give it the raw request, and ask which one skill it would load or `none`. A run is correct when ffmpeg-skill is chosen exactly for the should-trigger prompts.
diff --git a/evals/trigger/prompts.json b/evals/trigger/prompts.json
index 6ec9f3f..83619ea 100644
--- a/evals/trigger/prompts.json
+++ b/evals/trigger/prompts.json
@@ -59,6 +59,36 @@
"prompt": "2 台のカメラで撮った結婚式の動画、いいところだけ 3 分にまとめられる?",
"should_trigger": true
},
+ {
+ "id": "t46",
+ "lang": "en",
+ "prompt": "the subtitles on my short are coming out in three lines and half the sentence shows up late, can you fix reel.mp4",
+ "should_trigger": true
+ },
+ {
+ "id": "t47",
+ "lang": "ja",
+ "prompt": "BGM のビートに合わせて clip.mp4 をカットしたい",
+ "should_trigger": true
+ },
+ {
+ "id": "t48",
+ "lang": "en",
+ "prompt": "strip the ums and uhs out of my podcast recording ep12.wav",
+ "should_trigger": true
+ },
+ {
+ "id": "t49",
+ "lang": "es",
+ "prompt": "tengo 200 vídeos en una carpeta y tardan una eternidad en procesarse",
+ "should_trigger": true
+ },
+ {
+ "id": "t50",
+ "lang": "en",
+ "prompt": "I only changed the export preset, do I have to re-render the whole edit again? project.json",
+ "should_trigger": true
+ },
{
"id": "n01",
"lang": "en",
diff --git a/references/gotchas.md b/references/gotchas.md
index 5643284..c384535 100644
--- a/references/gotchas.md
+++ b/references/gotchas.md
@@ -122,6 +122,10 @@ printing one line — `font: /usr/share/fonts/.../wqy-zenhei.ttc (covers ko)`.
`brew install --cask font-noto-sans-cjk font-noto-sans-arabic`, or point at a
file with `--font-file` (`overlay.py`, `graphics.py`) / `--fonts-dir`
(`caption.py`).
+- Windows drawtext crashes on some builds (#100) when it has to resolve a font by
+ family name: pass `--font-file` explicitly if one does. `default_font_file()`
+ already resolves a file for you on Windows, so this bites only a hand-built
+ drawtext call. More: `references/ci-platform-pitfalls.md`.
- Han characters alone (no kana, no hangul) are read as Chinese. Japanese or
Korean hanja text with no kana needs `--lang ja` / `--lang ko`
(`caption.py --language` is the same flag), or `"lang"` in brand.json.
diff --git a/references/scripts.md b/references/scripts.md
index 30ea3e2..596ef48 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -64,6 +64,7 @@ BT.2020 primaries on an SDR transfer, which `hdr_format` names "BT.2020 SDR" --
### cut.py — cut / join segments
```
cut.py INPUT [--start T] [--end T | --duration T] [--segments A-B,C-D,...] [--accurate] [-o OUT]
+cut.py INPUT --start T --end T --snap beats [--snap-tolerance 0.12] [--snap-source FILE] [--min-confidence 0.5]
```
Times accept `12.5`, `1:30`, `00:01:30.250`. Default is `-c copy` (snaps to
keyframes, instant, lossless); if the snapped result deviates more than
@@ -74,6 +75,49 @@ the result was "lossless stream copy" or "re-encoded"; when the snap forced a
re-encode, the result's `lossless_alternative` names the nearest keyframe
`--start` that would stream-copy instead, so the trade can be offered.
+**`--snap beats` (1.17)** moves each in/out point to the nearest *measured*
+beat within `--snap-tolerance` seconds (default 0.12, about a quarter of a beat
+at 120 BPM). The grid comes from the input's own audio, or from
+`--snap-source` — either a `scenes.py --beats --json` document (no second
+decode) or a separate music file. The snap happens *before* the
+keyframe/tolerance decision, so lossless-vs-accurate is judged on where the cut
+actually lands.
+
+"Measured" is exact: the only points a cut may move onto are the
+`supported_beats` — the grid points a measured onset actually marks — never the
+full regular grid, which by construction runs on through a passage with no music
+in it. A cut asked for inside that passage stays where it was asked for, and
+`snap.grid` is `"supported"` with `snap.grid_points` saying how many there were.
+`--min-confidence` must be greater than 0: at 0 every grid is "reliable",
+including one measured from noise.
+
+The number of in/out points never changes: a point with no beat inside the
+tolerance is returned unchanged, and no point is ever invented. Three
+`kind: input` refusals, nothing written:
+
+- confidence below `--min-confidence` → *"no reliable beat grid in this audio
+ (confidence 0.21, needs 0.5): cutting to invented beats would move your in/out
+ points to times nothing in the audio supports. Re-run with `--snap none`, or
+ pass `--snap-source` from a music bed."*
+- no audio stream → names `--snap none`.
+- no in/out point at all (a whole-file copy) → there is nothing to snap, and
+ this tool never proposes cuts of its own.
+
+Result: `snap.mode`, `snap.tolerance`, `snap.confidence`, `snap.tempo_bpm`,
+`snap.moved` (one row per point, with `from`/`to`/`delta`/`snapped`),
+`snap.snapped`, `snap.unchanged`, `snap.grid`, `snap.grid_points`, and
+`snap.source` (`"measured"` or the path). In a `render.py` result, `snap.clips`
+carries one entry per snapped clip with `clip` naming its index, and the first
+entry's keys are repeated at the top level for a single-clip project. A clip
+served from `--cache` was snapped when it was first rendered, and says so
+(`source: "cache"`) rather than reporting `snap: null`. A `--snap-source`
+document that does
+not carry `beat_grid.supported_beats` (one written before 1.17) is refused
+rather than treated as if every grid point were supported, as is one whose
+`tempo_bpm` is null while it lists beats.
+`render.py` forwards a project's `"snap": {"to": "beats", ...}` (project-wide
+or per clip) to this flag and reports what came back.
+
### fit.py — target duration and/or aspect, rotate/flip
```
fit.py INPUT [--duration T --method speed|trim [--from-center] [--max-speed 4]]
@@ -422,6 +466,9 @@ each shorter clip's last frame (with silence) out to the longest.
### silence.py — remove dead air / jump cuts
```
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [-o OUT]
+silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
+ [--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
+ [--transcribe] [--filler-list] [--max-cuts 400]
```
Runs `silencedetect`, keeps `--margin` seconds of air around speech, drops
gaps shorter than `--min-silence`, and re-encodes once with `select`/`aselect`
@@ -430,6 +477,60 @@ without rendering; `--edl` saves the kept ranges in `cut.py --segments` format
so the user can edit the list by hand. Quiet rooms need `--threshold -40`
to `-45`; noisy ones `-30`. Always tell the user how many seconds were removed.
+**`--filler` (1.17)** removes filler words as well, through the same
+`keep_ranges()`/`aselect` graph with more, shorter ranges. It is **never**
+applied without measured word timings: pass `--words` (a whisper JSON carrying
+per-word `start`/`end`) or `--transcribe` (the same local engine `caption.py`
+uses, never required). There is no heuristic fallback — finding an "um" by
+looking for short quiet blips would cut real speech, so the tool refuses
+instead. `--filler-list` reports what would go and writes nothing.
+
+Built-in lists, one per language:
+
+| lang | words |
+|---|---|
+| en | um, uh, erm, hmm, mm, mhm, er, ah |
+| ja | えー, えーと, えっと, あの, あのー, その, そのー, まあ, なんか |
+| es | eh, este, esto, mmm |
+| de | äh, ähm, hm |
+| fr | euh, hein |
+| pt | é, hum |
+| it | ehm |
+
+**Discourse markers are not disfluencies.** `like`, `tipo` and `cioè` are
+deliberately *not* in the default lists: they are grammatical words in most
+sentences, and removing them cuts meaning rather than noise — a judgement about
+content, which this skill does not make. They are reachable with
+`--filler-extra like`, which says so in its own `--help`. Japanese `なんか` is
+in the `ja` list because leaving it out makes the flag useless for Japanese,
+and it is orthographically identical to the pronoun use — so every run that
+removes one warns, and `--filler-keep なんか` takes it back out. Matching is
+case-folded, punctuation-stripped and whole-token only: `umbrella` survives.
+
+Three refusals, all `kind: input`, all before any encode:
+
+- `--filler` with neither `--words` nor `--transcribe` → names both flags.
+- `--transcribe` with no engine on PATH → the same message `caption.py` gives,
+ with the three install lines.
+- `--transcribe` where the engine runs but its build produces no word-level
+ timings → names that engine, says some builds do not support word timestamps,
+ and points at `--words`. (`--transcribe` drives whichever engine is installed
+ with *its* word-timestamp option — whisper.cpp `--output-json-full`,
+ faster-whisper `word_timestamps=True`, openai-whisper `--word_timestamps
+ True` — because an SRT cannot answer this: a cue has a start and an end, a
+ word does not. `--words` is the tested path and the one to prefer.)
+- a transcript with segments but no word-level timings → says that cutting on
+ segment boundaries would remove whole sentences, and how to re-run whisper.
+
+Results: `filler.removed_count`, `filler.removed_seconds`, `filler.removed`
+(one entry per span), `filler.lang`, `filler.list`, `filler.word_timings` and
+`filler.warnings`. The existing `removed_seconds` keeps exactly the meaning it
+has always had — the seconds of **silence** this run removed, the figure the
+same run would report without `--filler` — and `removed_seconds_total` is the
+additive sibling covering everything that went. A filler word quiet enough to
+sit inside a detected silence is merged into it rather than counted twice, so
+the two figures can be equal.
+
### join.py — concatenate with transitions
```
join.py CLIP1 CLIP2 [...] [--transition fade|dissolve|wipeleft|slideleft|fadeblack|fadewhite|circleopen|none]
@@ -447,6 +548,7 @@ output; the result says so with `dropped_non_av_streams: true`.
```
render.py --init project.json # starter file
render.py project.json [--fast] [--dry-run] [--stop-after STAGE] [--work DIR --keep]
+render.py project.json --cache DIR [--from STAGE] # reuse identical stages from a previous run
render.py plan.json # execute a plan written by <tool> --plan plan.json
```
A plan is a single tool's dry run as an artifact: `cut.py in.mp4 --start 2 --end 8
@@ -479,6 +581,40 @@ entries and the file path are validated before the first stage runs, and the
stage plans its `metadata.py` command under `--dry-run`/`--plan` like every
other stage, so the plan lists `chapters` and the run does the same work.
+**`--cache DIR` (1.17)** reuses the artifact of a stage that already ran with
+exactly the same arguments and the same inputs. Opt-in only: **there is no
+default cache directory** — one appearing on someone's disk unasked would
+contradict this tool's "a plan leaves nothing behind" posture. The key is a
+sha1 over the stage name, the tool, its arguments (every existing path replaced
+by its content hash), the input hashes, the **forwarded** flags (`--fast`,
+`--overwrite`, `--timeout`, `--codec`), the output's extension, and **the ffmpeg
+build banner, the skill version and the contract version**. `--fast` matters as
+much as any of them: it rewrites every child's preset to `veryfast`, so without
+it in the key a `--cache --fast` draft would be served back to a later run that
+asked for the delivery. The banner rather than `major.minor` because two 7.1.x
+builds with different libx264 produce different bytes from the same command. Those last three are in the key deliberately: a
+different build simply *misses* rather than being asked to trust a file it did
+not write, and a stage whose implementation changed cannot serve back an
+artifact the old one produced. Each entry is `DIR/<key><ext>` plus a
+`DIR/<key>.json` sidecar (stage, versions, creation time, size, seconds). A hit
+hardlinks the artifact into the work directory, or copies it where the
+filesystem will not link — never moves it, since the cache has to outlive the
+run's own cleanup. Any mismatch is a silent miss.
+
+`stages_done` is unchanged: a cached stage is still a stage that happened.
+Nothing is written under `--dry-run`, which instead reports `cache.would_hit`.
+Result: `cache.dir`, `cache.ffmpeg`, `cache.hits`, `cache.misses`,
+`cache.saved_seconds`, `cache.entries`. `--from STAGE` starts at that stage and
+takes every earlier one from the cache; without `--cache`, or when an earlier
+stage is not there, it refuses (`kind: input`) rather than quietly re-encoding
+what it promised to skip. An unwritable cache directory is `kind: output`.
+
+`"snap": {"to": "beats", "tolerance": 0.12, "min_confidence": 0.5, "source":
+"music.mp3"}` at the project root (or inside one clip) forwards `--snap beats`
+to the clip cut for every clip that has `in`/`out`; the measured grid comes back
+in the render result's `snap`. A project without `"snap"` builds the command
+line 1.16 built.
+
Stages: clips (cut, optional speed) → join (transition) → silence → fit →
captions → graphics → overlays → audio → loudness → export → chapters → check. Keys mirror the
CLI flags of each script (see the docstring); a key `render.py` does not read -- at
@@ -549,6 +685,7 @@ already has a picture.
### scenes.py — scene changes and highlight candidates
```
scenes.py INPUT [--threshold 10] [--min-scene 1] [--highlights N [--target SECONDS] [--max-scene 15]] [--edl picks.txt] [--sheet scenes.png] [--json]
+scenes.py INPUT --beats [--beat-step 0.01] [--beat-range 60-200] [--min-confidence 0.5] [--json]
```
Lists scenes with audio energy, the loudest moments, and (with
`--highlights`) proposes N ranges that add up to `--target` seconds, biased to
@@ -560,6 +697,45 @@ Dissolves and very slow fades are not cuts and will be missed. Highlights are
a proposal engine, not a judgement of content: tell the user what it picked
and why (energy, scene length).
+**`--beats` (1.17)** measures the music's beat grid and reports it:
+
+```json
+"beats": [0.0, 0.5, 1.0, ...], // 25 of them over a 12 s click track
+"beat_grid": {"tempo_bpm": 120.0, "interval": 0.5, "confidence": 0.997,
+ "supported_beats": [0.0, 0.5, 1.0, ...], // the 24 an onset marks
+ "phase": 0.0, "onsets": 24, "supported": 24, "unsupported": 1,
+ "method": "rms-flux-autocorrelation", "step_s": 0.01,
+ "range_bpm": [60, 200], "usable": true}
+```
+
+Method: onset strength as the half-wave-rectified first difference of
+`log(envelope)`, peaks above `median + 1.5·MAD` over a ±1 s window with a 60 ms
+refractory gap, tempo from the autocorrelation of the onset signal inside
+`--beat-range` with its half and double checked, phase chosen to catch the most
+onset strength. `confidence` is half how far the winning lag stands above the
+other lags (in standard deviations) and half the fraction of onsets that land
+on the grid. With `--beats` the file is decoded once, at 22050 Hz, and the
+`supported + unsupported == len(beats)` always, and `supported_beats` is that
+supported subset — the list `cut.py --snap beats` moves onto.
+
+With `--beats` the file is decoded once, at 22050 Hz, and the 0.5 s scene
+envelope is derived from that same pass rather than from a second 8 kHz decode.
+One consequence worth knowing: `scenes.py X --json` and `scenes.py X --beats
+--json` report very slightly different `audio_rms`/`audio_peak` figures for the
+same file, because the two envelopes are built from PCM at different rates. The
+scene boundaries and their ranking are unaffected; only the fourth decimal of
+the level moves. `--beats` also holds ~2.75x the samples in memory, which is
+worth knowing on a feature-length input.
+
+**A beat grid is a measurement of the music's periodicity, not of where a cut
+belongs.** A low confidence means the audio has no steady pulse — speech,
+ambience, rubato — and the skill will not snap to a grid it cannot measure.
+`scenes.py` is the analysis tool, so it *reports* a weak measurement
+(`usable: false`) rather than refusing it; refusing belongs to the tools that
+would change a file on the strength of it (`cut.py --snap beats`). `--edl` with
+`--beats` is unchanged: beats are never written as an EDL, because a beat is
+not a cut. No audio stream is a `kind: input` refusal.
+
### check.py — pre-delivery compliance
```
check.py INPUT --platform youtube|shorts|reels|tiktok|x|linkedin|facebook|broadcast|podcast|custom [--no-loudness] [--json]
@@ -582,13 +758,42 @@ construction, not by two lists agreeing.
### batch.py — same recipe over a folder, cached
```
-batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--json]
+batch.py FOLDER --recipe batch.json [--force] [--watch SECONDS] [--jobs N|auto] [--json]
```
`batch.json` holds either `steps` (a list of script argv with `{in}`/`{out}`
placeholders, chained) or `project` (a render project applied per file).
Outputs land in `output_dir` with `suffix`; a content-hash cache skips files
already done with the same recipe. Use `--dry-run` to preview the plan.
+**`--jobs N` (1.17)** processes N files at once (threads: the work is
+subprocess waiting). Capped at `min(N, cpu_count, 8)` — every item is itself an
+ffmpeg that already threads across cores, so beyond a few concurrent encodes
+the jobs contend and wall-clock stops improving while memory does not. A number
+above the cap is clamped with a note, not refused, and both `jobs` and
+`jobs_requested` are in the result. `--jobs auto` is `min(cpu_count, 4)`.
+
+- **One timeout budget for the whole batch**, not one per item: `--timeout` is
+ computed into a deadline once, no new item starts after it, and the run exits
+ 124 `kind: timeout` with the items that never started marked
+ `"skipped": "timeout"`. That shared budget applies when a `--timeout` was
+ actually given, or when `--jobs > 1` asked for the batch to be treated as one
+ piece of work; the default sequential path with the default timeout is 1.16's
+ behaviour exactly, where a long folder was never cut off part-way.
+- **Determinism**: every item is written into its own slot in one list indexed
+ by position in the sorted file list — cached hits included, which is what keeps
+ the table in file order when the cache is only partially warm — and each item's
+ log lines are buffered and flushed in file order, so the per-item table and the
+ log read exactly as a serial run's whatever order the encodes finish in.
+- **A failed item is a row, not a dead run**: a worker that raises becomes
+ `ok: false` with its reason, and the summary and the table are still printed.
+ Ctrl-C cancels what has not started, keeps what finished, and exits 130
+ `kind: interrupted` with the partial table.
+- With `--jobs > 1` each item gets its own work subdirectory
+ `<workdir>/<index>-<stem>/`, because step file names are stem-derived and two
+ sources sharing a stem would otherwise write over each other. `--jobs 1` (the
+ default) keeps the flat layout and is 1.16's path exactly.
+- `--watch` composes: each pass is parallel, passes are sequential.
+
### caption.py --transcribe — optional local speech-to-text
If `whisper-cli` (whisper.cpp), `faster-whisper` or `whisper` is installed,
`caption.py input.mp4 --transcribe [--language ja] [--model base]` writes the
@@ -726,11 +931,62 @@ caption.py INPUT --srt FILE[:LANG] | --ass FILE | --text CUES.txt [--write-srt O
[--audio-stream N] [--fps N] [--lang XX] [--offset TIME]
[--max-lines N] [--min-duration S] [--wrap phrase|measured]
[--font NAME] [--fonts-dir DIR] [--size N] [--color RRGGBB] [--outline N] [--outline-color RRGGBB]
+ [--fit-size auto|on|off] [--min-size N] [--fit-size-scope file|cue]
[--bold] [--box] [--position bottom|top|center|top-left|...] [--margin N]
[--animate none|fade|pop|slide] [--karaoke [--highlight-color RRGGBB]] [--write-ass OUT.ass]
[--emoji auto|color|png|mono|none] [--emoji-assets DIR] [--emoji-scale 1.0] [--emoji-max 60] [-o OUT]
caption.py --text CUES.txt --write-srt OUT.srt # generate the SRT only
```
+**`--fit-size` (1.17): the size is fitted before a cue is split.** At the TikTok
+caption size (24 ASS units against the 288-line script grid) a line has about
+six em, so an ordinary sentence needs four lines — and `--max-lines 2` then cut
+it into consecutive cues, so half of it arrived late. That was the size, not the
+breaker. `--fit-size` walks the size down until every cue wraps within
+`--max-lines`, and only then lays the cues out.
+
+- `auto` (default) shrinks only a size the *skill* chose. An explicit `--size`,
+ or a `brand.json` `styles.caption.size`, is a statement about the look and is
+ never overridden.
+- `on` always fits, `--size` or not. **`off` is 1.16.1 exactly, byte for byte**
+ (a pinned ASS fixture asserts it).
+- `--min-size` is the floor, default `ass_units(0.045) = 13` — **4.5 % of the
+ frame height**, one floor for every destination (87 px of type on a 1920-tall
+ frame, above the ~3.5 % where mobile legibility and the platforms' own caption
+ UIs bottom out). Nothing per-platform is measured, so nothing is claimed.
+- `--fit-size-scope file` (default) uses one size for the whole file; `cue`
+ writes a per-cue `{\fsN}` override. A size that changes from cue to cue reads
+ as a mistake, so it is opt-in — it exists for the one outlier cue that would
+ otherwise shrink a ten-minute file. Each cue is laid out at the size it will
+ be **drawn** at: a cue drawn larger has a narrower line in em, so wrapping
+ everything to the file-wide budget and then drawing some cues large would put
+ lines off the side of the frame.
+
+The text is **never** touched: this skill does not rewrite, shorten or
+paraphrase a caption to make it fit. Below the floor the cue is split exactly as
+before and `fit_exhausted: true` says so, with `--min-size` named as the flag
+that would go smaller and `|` as the manual break. `--min-size` above `--size`
+is a `kind: input` refusal. `--mode mux` is unaffected: soft subtitles carry no
+size, so the SRT is the one 1.16 wrote.
+
+Results, alongside the existing caption stats:
+`fit_size`, `size_requested`, `size_used`, `size_floor`, `size_pct_height`,
+`shrunk`, `fit_scope`, `fit_exhausted`, and `size_source` (`input` or
+`platform-frame`).
+
+Under `--dry-run`/`--plan` on an input that does not exist yet there is no
+geometry to measure. With `--platform` the destination's own frame is used —
+that frame *is* what the real run will have, so the planned `FontSize` is the
+one the run will burn. Without a platform nothing can stand in for the frame
+and `size_used` is `null`, rather than presenting the requested size as a
+fitted one.
+
+| ASS size | px on 1920 | % frame h | em per line |
+|---|---|---|---|
+| 24 (default) | 160.0 | 8.3 % | 6.08 |
+| 19 | 126.7 | 6.6 % | 7.67 |
+| 16 | 106.7 | 5.6 % | 9.11 |
+| 13 (floor) | 86.7 | 4.5 % | 11.22 |
+
Text cue format, one per line: `0:00-0:03 Hello`, `00:00:03.500 --> 00:00:06 Two | lines`,
or `00:00:03:15 --> 00:00:06:00 SMPTE non-drop-frame timecode` (`hh:mm:ss:ff`, frame count
converted with `--fps`, or the input video's own fps when `--input` is given and `--fps` is
diff --git a/scripts/_common/__init__.py b/scripts/_common/__init__.py
index 89e8c44..f30df69 100644
--- a/scripts/_common/__init__.py
+++ b/scripts/_common/__init__.py
@@ -73,10 +73,21 @@ from _common.probe import (
from _common.decision import (
aac_args, add_pad_fill_args, audio_codec_for, AUDIO_CODECS, brand_caption_style, BRAND_DEFAULTS,
description_block, _evidence_rank, fmt_chapter_time, propose_chapters,
+ filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS, FILLER_MAX_WORD,
+ FILLER_MIN_GAP, FILLER_PAD, normalise_filler_token,
+ beat_grid, snap_points, BEAT_MIN_CONFIDENCE, BEAT_ONSET_K, BEAT_OCTAVE_MARGIN,
+ BEAT_REFRACTORY_S, BEAT_WINDOW_S, BEAT_SUPPORT_DIVISOR, BEAT_ALIGN_DIVISOR,
+ BEAT_Z_FLOOR, BEAT_Z_SPAN,
+ _onset_strength, _pick_onsets, _autocorrelation_peak, _grid_score,
brand_states_font, cfr_args, concat_list_line, db_to_linear, default_output, encoder_args, escape_filter_path,
fmt_secs, fmt_smpte_time, fmt_srt_time, is_audio_output, load_brand, MissingFpsError, pad_filters, parse_time,
signed_time_arg, SVT_PRESET, time_arg, video_args, x264_args, _x264_raw
)
+from _common.asr import (
+ ASR_ENGINES, ASR_INSTALL_HINT, _asr_run, die_no_engine, parse_srt, transcribe, _transcribe_in,
+ transcribe_words, _words_from_openai_whisper_json, _words_from_whisper_cpp_json,
+ whisper_word_timings, write_srt
+)
from _common.color import (
bt709_tag_args, color_hex, _COLOR_TOKEN_RE, _sdr_bt709, validate_color
)
@@ -96,11 +107,12 @@ from _common.text import (
PENALTY_NEUTRAL, PENALTY_OKURIGANA,
PENALTY_PARTICLE, PENALTY_SENTENCE_END, _rebalance, _rebalance_phrase, SAFE_WIDTH_FRACTION, _split_hyphens,
_particle_ends, _particle_starts, wrap_text, wrap_variants, WRAP_MODES,
+ fit_size, line_em_for_size, MIN_CAPTION_FRACTION, ass_units_local,
script_font_for_text, script_font_status, _script_font_uncached, _SCRIPT_RANGES, SCRIPTS, _SHAPING_BUILD_CACHE,
SHAPING_SCRIPTS, text_width_em, _VS15, _VS16, WINDOWS_FONTS, _ZWJ
)
-from _common import color, decision, runner, text # noqa: F401,E402
+from _common import asr, color, decision, runner, text # noqa: F401,E402
# `_common.emit` and `_common.probe` are the FUNCTIONS, as they have always been -- the
# from-imports above rebound the package attribute the submodule import had set. The two modules
@@ -111,7 +123,7 @@ from _common import color, decision, runner, text # noqa: F401,E402
_emit_module = sys.modules["_common.emit"]
_probe_module = sys.modules["_common.probe"]
-_MODULES = (runner, _emit_module, _probe_module, decision, color, text)
+_MODULES = (runner, _emit_module, _probe_module, decision, color, text, asr)
class _Facade(_types.ModuleType):
@@ -168,6 +180,12 @@ __all__ = [
"char_script", "_check_existing_output", "_check_no_overwrite_input", "_check_output_path", "child_args",
"child_limit", "_CHILDREN", "_cleanup_partial_output", "_cmdline", "CODECS", "color_hex", "_COLOR_TOKEN_RE",
"concat_list_line", "Context", "_CRF_DEFAULT", "_CURRENT_CTX", "db_to_linear", "decode_pcm_mono", "description_block", "_evidence_rank", "fmt_chapter_time", "propose_chapters",
+ "filler_spans", "FILLER_WORDS", "FILLER_AMBIGUOUS", "FILLER_DISCOURSE_MARKERS",
+ "FILLER_MAX_WORD", "FILLER_MIN_GAP", "FILLER_PAD", "normalise_filler_token",
+ "beat_grid", "snap_points", "BEAT_MIN_CONFIDENCE", "BEAT_ONSET_K", "BEAT_OCTAVE_MARGIN",
+ "BEAT_REFRACTORY_S", "BEAT_WINDOW_S", "BEAT_SUPPORT_DIVISOR", "BEAT_ALIGN_DIVISOR",
+ "BEAT_Z_FLOOR", "BEAT_Z_SPAN",
+ "_onset_strength", "_pick_onsets", "_autocorrelation_peak", "_grid_score",
"default_font_file", "default_output", "DEFAULT_TIMEOUT", "detect_script", "die", "drawtext_boxborderw",
"_DRAWTEXT_PENDING", "drawtext_shaping", "drawtext_text_opts", "_DRAWTEXT_TMPDIR", "_drawtext_tmpdir",
"dry_run_input_pending", "emit", "emoji_asset_for", "EMOJI_ASSET_HINT", "emoji_clusters",
@@ -193,6 +211,9 @@ __all__ = [
"time_arg", "_timed_out", "_to_float", "_to_int", "_unwatch", "_V2_HANDLED", "validate_color", "verify_output",
"video_args", "_VS15", "_VS16", "_watch", "WINDOWS_FONTS", "write_plan", "x264_args", "X264_PRESETS",
"_x264_raw", "_ZWJ",
+ "ASR_ENGINES", "ASR_INSTALL_HINT", "_asr_run", "die_no_engine", "parse_srt", "transcribe",
+ "_transcribe_in", "transcribe_words", "_words_from_openai_whisper_json",
+ "_words_from_whisper_cpp_json", "whisper_word_timings", "write_srt",
"_atoms", "best_break", "_bare_word", "break_penalty", "_break_spaced", "_cut_penalty", "_fix_orphans",
"_fix_weak_lines", "_function_words", "FUNCTION_WORDS", "_HYPHENS", "_is_hiragana", "_is_ideograph",
"_is_kana", "_is_weak_line", "JA_NO_LINE_END", "JA_NO_LINE_START", "JA_PARTICLE_WORDS", "JA_PARTICLES",
@@ -200,5 +221,6 @@ __all__ = [
"JA_SENTENCE_END", "_join", "ORPHAN_MIN_EM", "PENALTY_FORBIDDEN", "PENALTY_FUNCTION_WORD",
"PENALTY_IDEOGRAPHS", "PENALTY_NEUTRAL", "PENALTY_OKURIGANA", "PENALTY_PARTICLE", "PENALTY_SENTENCE_END",
"_rebalance", "_rebalance_phrase", "SAFE_WIDTH_FRACTION", "_split_hyphens", "_particle_ends",
- "_particle_starts", "wrap_text", "wrap_variants", "WRAP_MODES"
+ "_particle_starts", "wrap_text", "wrap_variants", "WRAP_MODES",
+ "fit_size", "line_em_for_size", "MIN_CAPTION_FRACTION", "ass_units_local"
]
diff --git a/scripts/_common/asr.py b/scripts/_common/asr.py
new file mode 100644
index 0000000..fc45453
--- /dev/null
+++ b/scripts/_common/asr.py
@@ -0,0 +1,369 @@
+"""The optional local speech-to-text bridge, and the SRT the rest of the skill reads and writes.
+
+Whisper is never required. Nothing here runs unless a caller asked for a transcript:
+`caption.py --transcribe` and `silence.py --filler --transcribe` share this one engine probe, so
+the "no engine found" message, its install lines and its exit code are stated once rather than
+copied per tool. This module is neither ffprobe nor a decision, which is why it is its own file.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+from _common.decision import fmt_srt_time, parse_time
+from _common.emit import die, info
+from _common.runner import read_text_or_die
+
+# The three engines this skill knows how to drive, and how to install each -- one string, so
+# every tool that needs a transcript refuses in the same words.
+ASR_ENGINES = ("whisper.cpp", "faster-whisper", "openai-whisper")
+ASR_INSTALL_HINT = (
+ "Install one (all run offline):\n"
+ " whisper.cpp: brew install whisper-cpp (then download a model: ggml-base.bin)\n"
+ " faster-whisper: pip install faster-whisper\n"
+ " openai-whisper: pip install openai-whisper")
+
+
+def parse_srt(path: str) -> List[Tuple[float, float, str]]:
+ cues: List[Tuple[float, float, str]] = []
+ block: List[str] = []
+ content = read_text_or_die(path, "--srt").lstrip("\ufeff").replace("\r\n", "\n") + "\n\n"
+ for line in content.split("\n"):
+ if line.strip():
+ block.append(line)
+ continue
+ if block:
+ times = next((b for b in block if "-->" in b), None)
+ if times:
+ a, b = times.split("-->")
+ text = "\n".join(block[block.index(times) + 1:]).strip()
+ try:
+ cues.append((parse_time(a), parse_time(b), text))
+ except ValueError as e: # includes MissingFpsError: SRT timings are hh:mm:ss,ms, never frames
+ die(f"{path}: cannot read the timing line {times.strip()!r}: {e}")
+ block = []
+ if not cues:
+ die(f"no cues found in {path}")
+ return cues
+
+
+def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
+ with open(path, "w", encoding="utf-8") as fh:
+ for i, (s, e, t) in enumerate(cues, 1):
+ # A blank line is SRT's own block separator (index/timecode/text, blank, next block).
+ # Cue text can contain one -- parse_text_cues() turns a bare "|" into "\n", so a source
+ # line with two adjacent pipes ("a||b") becomes "a\n\nb" -- and writing that blank line
+ # raw would split one cue into two malformed half-blocks (the second missing its own
+ # index/timecode). Collapse any run of blank lines within the cue text to a single
+ # newline so the cue's own text can never fake the format's block boundary.
+ t = re.sub(r"\n{2,}", "\n", t).strip("\n")
+ fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
+
+
+def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0) -> List[Tuple[float, float, str]]:
+ """Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),
+ whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.
+ No engine installed -> clear error with install hints; the skill never depends on one."""
+ import shutil
+ import subprocess
+ import tempfile
+ from _common import require_tool, run_analysis, STATE
+ ffmpeg = require_tool("ffmpeg")
+ tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")
+ try:
+ return _transcribe_in(tmpdir, video, out_srt, language, model, audio_stream, ffmpeg, shutil, subprocess)
+ finally:
+ shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+def _asr_run(cmd: List[str], subprocess, name: str) -> "subprocess.CompletedProcess":
+ """Run a speech-to-text engine under the same wall-clock limit as an ffmpeg call."""
+ from _common import STATE, die
+ limit = STATE.timeout or None
+ try:
+ return subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, timeout=limit)
+ except subprocess.TimeoutExpired:
+ die(f"{name} exceeded the {limit:.0f} s time limit and was killed; raise --timeout for a long recording",
+ code=124, kind="timeout")
+ return None # unreachable
+
+
+def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int,
+ ffmpeg: str, shutil, subprocess) -> List[Tuple[float, float, str]]:
+ from _common import run_analysis, STATE, die
+ wav = os.path.join(tmpdir, "audio.wav")
+ # A wav in our own temp dir: a measurement input for the engine, not a deliverable, so it
+ # is not a run() call (no --dry-run gate, not recorded), but it keeps the time limit and
+ # reports an unreadable input as kind ffmpeg instead of a CalledProcessError traceback.
+ run_analysis([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,
+ "-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav])
+ # 1. whisper.cpp
+ cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp")
+ if not cli:
+ # older whisper.cpp builds ship the binary as plain `main`; accept it only when it lives
+ # in a directory that names whisper, so an unrelated /usr/bin/main is never run
+ main_bin = shutil.which("main")
+ if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
+ cli = main_bin
+ if cli:
+ model_path = model
+ if not os.path.exists(model_path):
+ for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"), f"models/ggml-{model}.bin", f"/usr/local/share/whisper/ggml-{model}.bin"):
+ if os.path.exists(cand):
+ model_path = cand
+ break
+ base = os.path.join(tmpdir, "out")
+ cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base]
+ if language:
+ cmd += ["-l", language]
+ proc = _asr_run(cmd, subprocess, "whisper.cpp")
+ if proc.returncode == 0 and os.path.exists(base + ".srt"):
+ info(f"transcribed with whisper.cpp ({os.path.basename(cli)}, model {os.path.basename(model_path)})")
+ cues = parse_srt(base + ".srt")
+ write_srt(cues, out_srt)
+ return cues
+ info("whisper.cpp found but failed: " + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
+ # 2. faster-whisper (python package)
+ try:
+ from faster_whisper import WhisperModel # type: ignore
+ import threading
+ result: list = []
+
+ def work() -> None:
+ m = WhisperModel(model, device="cpu", compute_type="int8")
+ segments, _ = m.transcribe(wav, language=language, word_timestamps=False)
+ result.extend((seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip())
+
+ # An in-process engine gets the same wall-clock limit as the CLI engines and ffmpeg.
+ t = threading.Thread(target=work, daemon=True)
+ t.start()
+ t.join(STATE.timeout or None)
+ if t.is_alive():
+ die(f"faster-whisper exceeded the {STATE.timeout:.0f} s time limit; raise --timeout for a long recording", code=124, kind="timeout")
+ cues = list(result)
+ if cues:
+ info("transcribed with faster-whisper")
+ write_srt(cues, out_srt)
+ return cues
+ except ImportError:
+ pass
+ # 3. openai-whisper CLI
+ if shutil.which("whisper"):
+ cmd = ["whisper", wav, "--model", model, "--output_format", "srt", "--output_dir", tmpdir]
+ if language:
+ cmd += ["--language", language]
+ proc = _asr_run(cmd, subprocess, "openai-whisper")
+ srt = os.path.join(tmpdir, "audio.srt")
+ if proc.returncode == 0 and os.path.exists(srt):
+ info("transcribed with openai-whisper")
+ cues = parse_srt(srt)
+ write_srt(cues, out_srt)
+ return cues
+ die_no_engine("Or write the cues by hand with --text cues.txt (see format above).")
+ return []
+
+
+def die_no_engine(alternative: str, flag: str = "--transcribe") -> None:
+ """The one "no local speech-to-text engine" refusal, in the one set of words.
+
+ kind: input, exit 1, and the three install lines -- caption.py and silence.py both land here
+ rather than each spelling out its own version of the same missing dependency.
+ """
+ die(f"no local speech-to-text engine found for {flag}.\n" + ASR_INSTALL_HINT + "\n" + alternative,
+ kind="input")
+
+
+def whisper_word_timings(srt_path: Optional[str]) -> List[Tuple[float, float, str]]:
+ """Word timings from a whisper JSON transcript sitting next to the SRT, if there is one.
+
+ whisper (and faster-whisper, and whisper.cpp's --output-json) can emit per-word start/end
+ times; when they are there, --karaoke should follow the real speech instead of splitting the
+ cue evenly. Looked for as <stem>.json and <stem>.words.json next to the SRT, in either the
+ {"segments": [{"words": [{"word": ..., "start": ..., "end": ...}]}]} or a bare
+ {"words": [...]} shape. Anything unreadable is simply "no word timings".
+ """
+ if not srt_path:
+ return []
+ stem = os.path.splitext(srt_path)[0]
+ for cand in (stem + ".words.json", stem + ".json"):
+ if not os.path.exists(cand):
+ continue
+ try:
+ data = json.loads(Path(cand).read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ continue
+ raw = []
+ if isinstance(data, dict):
+ raw = list(data.get("words") or [])
+ for seg in data.get("segments") or []:
+ raw.extend((seg or {}).get("words") or [])
+ words = []
+ for w in raw:
+ try:
+ text = str(w.get("word") or w.get("text") or "").strip()
+ if text:
+ words.append((float(w["start"]), float(w["end"]), text))
+ except (AttributeError, KeyError, TypeError, ValueError):
+ continue
+ if words:
+ info(f"karaoke: word timings from {os.path.basename(cand)} ({len(words)} words)")
+ return sorted(words)
+ return []
+
+
+# ------------------------------------------------------- word-level timings (1.17)
+#
+# transcribe() above produces an SRT, which is all --transcribe on caption.py ever needed: a cue
+# has a start and an end and that is what gets burnt in. silence.py --filler needs something
+# stricter -- a start and an end PER WORD -- and no amount of reading an SRT back produces one.
+# Each engine has its own flag for it, and each writes a different shape, so each is driven and
+# parsed here rather than in the tool.
+
+
+def _words_from_whisper_cpp_json(path: str) -> "List[Dict[str, Any]]":
+ """whisper.cpp --output-json-full: transcription[].tokens[] with offsets in MILLISECONDS.
+
+ Token text carries leading spaces and the model's special tokens ([_BEG_], [_TT_123]); those
+ are dropped, and a token that is a word continuation (no leading space) is glued onto the
+ previous word so "un" + "believable" is one word with one span, not two.
+ """
+ try:
+ doc = json.loads(Path(path).read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return []
+ out: "List[Dict[str, Any]]" = []
+ for seg in (doc.get("transcription") or []):
+ for tok in (seg.get("tokens") or []):
+ text = str(tok.get("text") or "")
+ if not text.strip() or text.strip().startswith("[_"):
+ continue
+ offsets = tok.get("offsets") or {}
+ try:
+ start, end = float(offsets["from"]) / 1000.0, float(offsets["to"]) / 1000.0
+ except (KeyError, TypeError, ValueError):
+ continue
+ if out and not text.startswith(" "):
+ out[-1]["word"] += text
+ out[-1]["end"] = end
+ else:
+ out.append({"word": text.strip(), "start": start, "end": end})
+ return [w for w in out if w["word"].strip()]
+
+
+def _words_from_openai_whisper_json(path: str) -> "List[Dict[str, Any]]":
+ """openai-whisper --word_timestamps True --output_format json: segments[].words[]."""
+ try:
+ doc = json.loads(Path(path).read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return []
+ out: "List[Dict[str, Any]]" = []
+ for seg in (doc.get("segments") or []):
+ for w in (seg.get("words") or []):
+ try:
+ out.append({"word": str(w.get("word") or w.get("text") or "").strip(),
+ "start": float(w["start"]), "end": float(w["end"])})
+ except (KeyError, TypeError, ValueError):
+ continue
+ return [w for w in out if w["word"]]
+
+
+def transcribe_words(video: str, language: "Optional[str]" = None, model: str = "base",
+ audio_stream: int = 0) -> "Tuple[List[Dict[str, Any]], Optional[str]]":
+ """([{word, start, end}, ...], the engine that produced them) from a local whisper.
+
+ Drives whichever engine is installed with ITS word-timestamp option -- whisper.cpp
+ `--output-json-full`, faster-whisper `word_timestamps=True`, openai-whisper
+ `--word_timestamps True` -- and returns the words it measured. ([], engine) when the engine
+ ran but its build produced no word-level timings, so the caller can refuse naming that engine
+ instead of pretending the audio had no words in it. No engine at all raises through
+ die_no_engine(), the same refusal caption.py gives.
+ """
+ import shutil as _shutil
+ import subprocess as _subprocess
+ import tempfile as _tempfile
+ from _common import require_tool, run_analysis, STATE
+ ffmpeg = require_tool("ffmpeg")
+ tmpdir = _tempfile.mkdtemp(prefix="ffskill_asrw_")
+ try:
+ wav = os.path.join(tmpdir, "audio.wav")
+ run_analysis([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,
+ "-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000",
+ "-c:a", "pcm_s16le", wav])
+
+ # 1. whisper.cpp
+ cli = _shutil.which("whisper-cli") or _shutil.which("whisper-cpp")
+ if not cli:
+ main_bin = _shutil.which("main")
+ if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
+ cli = main_bin
+ if cli:
+ model_path = model
+ if not os.path.exists(model_path):
+ for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"),
+ f"models/ggml-{model}.bin",
+ f"/usr/local/share/whisper/ggml-{model}.bin"):
+ if os.path.exists(cand):
+ model_path = cand
+ break
+ base = os.path.join(tmpdir, "out")
+ cmd = [cli, "-m", model_path, "-f", wav, "--output-json-full", "-of", base]
+ if language:
+ cmd += ["-l", language]
+ proc = _asr_run(cmd, _subprocess, "whisper.cpp")
+ if proc.returncode == 0 and os.path.exists(base + ".json"):
+ words = _words_from_whisper_cpp_json(base + ".json")
+ info(f"word timings from whisper.cpp ({len(words)} words)")
+ return words, "whisper.cpp"
+ info("whisper.cpp found but produced no word-timing JSON: "
+ + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
+ return [], "whisper.cpp"
+
+ # 2. faster-whisper
+ try:
+ from faster_whisper import WhisperModel # type: ignore
+ import threading
+ collected: list = []
+
+ def work() -> None:
+ m = WhisperModel(model, device="cpu", compute_type="int8")
+ segments, _ = m.transcribe(wav, language=language, word_timestamps=True)
+ for seg in segments:
+ for w in (getattr(seg, "words", None) or []):
+ collected.append({"word": str(w.word).strip(),
+ "start": float(w.start), "end": float(w.end)})
+
+ t = threading.Thread(target=work, daemon=True)
+ t.start()
+ t.join(STATE.timeout or None)
+ if t.is_alive():
+ die(f"faster-whisper exceeded the {STATE.timeout:.0f} s time limit; raise "
+ "--timeout for a long recording", code=124, kind="timeout")
+ info(f"word timings from faster-whisper ({len(collected)} words)")
+ return [w for w in collected if w["word"]], "faster-whisper"
+ except ImportError:
+ pass
+
+ # 3. openai-whisper CLI
+ if _shutil.which("whisper"):
+ cmd = ["whisper", wav, "--model", model, "--word_timestamps", "True",
+ "--output_format", "json", "--output_dir", tmpdir]
+ if language:
+ cmd += ["--language", language]
+ proc = _asr_run(cmd, _subprocess, "openai-whisper")
+ doc = os.path.join(tmpdir, "audio.json")
+ if proc.returncode == 0 and os.path.exists(doc):
+ words = _words_from_openai_whisper_json(doc)
+ info(f"word timings from openai-whisper ({len(words)} words)")
+ return words, "openai-whisper"
+ info("openai-whisper found but produced no word-timing JSON: "
+ + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
+ return [], "openai-whisper"
+
+ die_no_engine("or pass --words with a transcript you already have.",
+ flag="--filler --transcribe")
+ return [], None
+ finally:
+ _shutil.rmtree(tmpdir, ignore_errors=True)
diff --git a/scripts/_common/decision.py b/scripts/_common/decision.py
index 9a1f1b4..8afdbb4 100644
--- a/scripts/_common/decision.py
+++ b/scripts/_common/decision.py
@@ -7,6 +7,7 @@ and encoder-selection rules testable on their own.
from __future__ import annotations
import json
+import math
import os
import argparse
from pathlib import Path
@@ -518,3 +519,348 @@ def propose_chapters(duration: float, silences: "Sequence", scene_cuts: "Sequenc
def description_block(chapters: "Sequence") -> str:
"""The YouTube description form of a chapter list: `00:00 Chapter 1` per line."""
return "\n".join(f"{fmt_chapter_time(c['at'])} {c['title']}" for c in chapters)
+
+
+# ------------------------------------------------------------------ the beat grid (1.17)
+#
+# A beat grid is a measurement of the music's periodicity -- not a statement about where a cut
+# belongs. Everything here is arithmetic on an RMS envelope somebody else decoded; nothing in this
+# module decides to cut anything, and nothing invents a beat the audio does not support.
+
+BEAT_ONSET_K = 1.5 # peak threshold: median + k * MAD over the local window
+BEAT_WINDOW_S = 1.0 # +/- this many seconds is "local" for the threshold
+BEAT_REFRACTORY_S = 0.06 # two onsets closer than this are one onset
+BEAT_SUPPORT_DIVISOR = 4 # a grid point with no onset within interval/4 is "unsupported"
+BEAT_ALIGN_DIVISOR = 8 # an onset within interval/8 of a grid point counts as aligned
+BEAT_MIN_CONFIDENCE = 0.5 # the default below which a tool that CHANGES a file refuses to snap
+BEAT_OCTAVE_MARGIN = 1.2 # a half/double-tempo grid must explain this much more onset strength
+BEAT_Z_FLOOR = 2.0 # autocorrelation z-score at which periodicity starts counting
+BEAT_Z_SPAN = 4.0 # ... and the span over which it reaches 1.0
+
+
+def _onset_strength(envelope: "Sequence[float]") -> "List[float]":
+ """Half-wave-rectified first difference of log(env), i.e. a compression-domain spectral-flux
+ analogue. The log matters: the level-domain difference over-weights the loud sections, so a
+ quiet verse contributes no onsets at all and the tempo is measured on the chorus alone."""
+ import math as _math
+ log_env = [_math.log(max(0.0, float(e)) + 1e-9) for e in envelope]
+ return [0.0] + [max(0.0, log_env[i] - log_env[i - 1]) for i in range(1, len(log_env))]
+
+
+def _pick_onsets(strength: "Sequence[float]", step_s: float) -> "List[int]":
+ """Indices of local maxima above median + k*MAD over a +/-BEAT_WINDOW_S window, with a
+ refractory gap. A median/MAD threshold rather than a mean/stdev one because a handful of very
+ strong hits would drag a mean-based threshold above every other onset in the piece."""
+ n = len(strength)
+ if n < 3:
+ return []
+ half = max(1, int(round(BEAT_WINDOW_S / max(step_s, 1e-9))))
+ refractory = max(1, int(round(BEAT_REFRACTORY_S / max(step_s, 1e-9))))
+ picked: "List[int]" = []
+ for i in range(1, n - 1):
+ s = strength[i]
+ if s <= 0 or s < strength[i - 1] or s < strength[i + 1]:
+ continue
+ window = sorted(strength[max(0, i - half):min(n, i + half + 1)])
+ if not window:
+ continue
+ med = window[len(window) // 2]
+ devs = sorted(abs(x - med) for x in window)
+ mad = devs[len(devs) // 2]
+ if s < med + BEAT_ONSET_K * mad or s <= med:
+ continue
+ if picked and i - picked[-1] < refractory:
+ if s > strength[picked[-1]]:
+ picked[-1] = i
+ continue
+ picked.append(i)
+ return picked
+
+
+def _autocorrelation_peak(strength: "Sequence[float]", step_s: float,
+ bpm_range: "Sequence[float]") -> "tuple":
+ """(best lag in samples, peak, mean, standard deviation) of the onset signal's
+ autocorrelation over the lags `bpm_range` allows, or (None, 0.0, 0.0, 0.0).
+
+ The spread matters as much as the peak: every signal's autocorrelation has a maximum
+ somewhere, so "the peak is above the mean" says nothing. How far above it stands relative to
+ the spread of the other lags is what separates a pulse from noise.
+ """
+ n = len(strength)
+ hi_bpm, lo_bpm = max(bpm_range), min(bpm_range)
+ lag_min = max(1, int(round(60.0 / hi_bpm / max(step_s, 1e-9))))
+ lag_max = int(round(60.0 / lo_bpm / max(step_s, 1e-9)))
+ lag_max = min(lag_max, n - 1)
+ if lag_max < lag_min:
+ return (None, 0.0, 0.0, 0.0)
+ mean = sum(strength) / n if n else 0.0
+ centred = [s - mean for s in strength]
+ best_lag, best = None, 0.0
+ values = []
+ for lag in range(lag_min, lag_max + 1):
+ acc = sum(centred[i] * centred[i + lag] for i in range(n - lag))
+ acc /= (n - lag)
+ values.append(acc)
+ if best_lag is None or acc > best:
+ best_lag, best = lag, acc
+ mean_acc = sum(values) / len(values) if values else 0.0
+ var = sum((v - mean_acc) ** 2 for v in values) / len(values) if values else 0.0
+ return (best_lag, best, mean_acc, math.sqrt(var))
+
+
+def _grid_score(onset_times: "Sequence[float]", strength_at: "Dict[int, float]",
+ interval: float, phase: float) -> float:
+ """Total onset strength landing within interval/BEAT_ALIGN_DIVISOR of the grid."""
+ if interval <= 0:
+ return 0.0
+ tol = interval / BEAT_ALIGN_DIVISOR
+ total = 0.0
+ for i, t in enumerate(onset_times):
+ off = (t - phase) % interval
+ if min(off, interval - off) <= tol:
+ total += strength_at.get(i, 1.0)
+ return total
+
+
+def beat_grid(envelope: "Sequence[float]", step_s: float, *,
+ bpm_range: "Sequence[float]" = (60, 200),
+ min_confidence: float = BEAT_MIN_CONFIDENCE,
+ duration: "Optional[float]" = None) -> "Dict[str, Any]":
+ """A beat grid from an RMS envelope. Pure: numbers in, a dict out -- no ffmpeg, no I/O.
+
+ Returns {"beats": [t, ...], "tempo_bpm": float|None, "interval": float|None,
+ "confidence": 0..1, "onsets": [t, ...], "phase": float,
+ "supported": int, "unsupported": int, "usable": bool,
+ "method": "rms-flux-autocorrelation", "step_s": step_s, "range_bpm": [lo, hi]}
+
+ The method, in full, so a report can quote it:
+ 1. onset strength = half-wave-rectified first difference of log(env + 1e-9);
+ 2. onsets = local maxima above median + 1.5 * MAD over a +/-1 s window, 60 ms refractory;
+ 3. tempo = the best autocorrelation lag of the onset signal inside `bpm_range`, with its
+ half and double checked and the one whose onsets align better preferred (octave
+ disambiguation: 60, 120 and 240 BPM all autocorrelate on a 120 BPM track);
+ 4. phase = the offset in [0, interval) at which the grid catches the most onset strength;
+ 5. beats = phase + n * interval across the duration. A grid must be regular, so a grid
+ point with no measured onset within interval/4 is still reported -- and counted in
+ `unsupported`, so a caller can see how much of the grid the audio actually supports.
+ `supported_beats` is the subset that a measured onset does support: it is what a tool
+ that MOVES something must snap to, because a regular grid runs on through a passage with
+ no music in it and a point moved there was moved to a time nothing in the audio marks;
+ 6. confidence = 0.5 * clip((z - 2) / 4, 0, 1)
+ + 0.5 * (fraction of onsets within interval/8 of a grid point),
+ where z is how many standard deviations the winning autocorrelation lag stands above
+ the mean of the others. The plain peak/mean ratio is not used: every signal's
+ autocorrelation has a maximum somewhere, so a peak above the mean says nothing -- noise
+ scores 2.9 on it, which would read as full confidence.
+
+ A flat or empty envelope has no pulse: confidence 0.0, tempo None, beats []. That is a
+ measurement, not a failure -- the caller decides whether 0.0 is enough to act on.
+ """
+ step_s = float(step_s)
+ env = list(envelope or [])
+ lo_bpm, hi_bpm = float(min(bpm_range)), float(max(bpm_range))
+ out: "Dict[str, Any]" = {
+ "beats": [], "supported_beats": [], "tempo_bpm": None, "interval": None,
+ "confidence": 0.0, "onsets": [],
+ "phase": 0.0, "supported": 0, "unsupported": 0, "usable": False,
+ "method": "rms-flux-autocorrelation", "step_s": step_s, "range_bpm": [lo_bpm, hi_bpm],
+ }
+ if len(env) < 4 or step_s <= 0:
+ return out
+ total_s = float(duration) if duration else len(env) * step_s
+
+ strength = _onset_strength(env)
+ if not any(strength):
+ return out
+ idx = _pick_onsets(strength, step_s)
+ onset_times = [i * step_s for i in idx]
+ out["onsets"] = [round(t, 4) for t in onset_times]
+ if len(idx) < 2:
+ return out
+
+ best_lag, peak, mean_acc, sd_acc = _autocorrelation_peak(strength, step_s, (lo_bpm, hi_bpm))
+ if not best_lag or peak <= 0:
+ return out
+ interval = best_lag * step_s
+ strength_at = {i: strength[j] for i, j in enumerate(idx)}
+
+ # Octave disambiguation: try half and double the candidate interval and keep the one whose
+ # grid catches the most onset strength per grid point (per point, or a denser grid always wins).
+ candidates = [interval]
+ for factor in (0.5, 2.0):
+ alt = interval * factor
+ if 60.0 / hi_bpm <= alt <= 60.0 / lo_bpm:
+ candidates.append(alt)
+
+ def best_phase(iv: float) -> "tuple":
+ steps = max(4, int(round(iv / step_s)))
+ best_ph, best_sc = 0.0, -1.0
+ for k in range(steps):
+ ph = k * iv / steps
+ sc = _grid_score(onset_times, strength_at, iv, ph)
+ if sc > best_sc:
+ best_ph, best_sc = ph, sc
+ return best_ph, best_sc
+
+ # The winner is the grid that explains the most measured onset strength -- "more onsets fall
+ # on it", the standard octave rule. An alternative must explain appreciably more (a fifth
+ # again) to displace the autocorrelation's own answer: a half-tempo grid catches a subset of
+ # the same onsets and a double-tempo grid catches the same set plus empty points, so a bare
+ # ">" would flip the answer on noise.
+ base_phase, base_score = best_phase(interval)
+ interval, phase = interval, base_phase
+ for alt in candidates[1:]:
+ alt_phase, alt_score = best_phase(alt)
+ if alt_score > base_score * BEAT_OCTAVE_MARGIN:
+ interval, phase, base_score = alt, alt_phase, alt_score
+
+ beats = []
+ t = phase
+ while t <= total_s + 1e-9:
+ beats.append(round(t, 4))
+ t += interval
+ support_tol = interval / BEAT_SUPPORT_DIVISOR
+ supported_beats = [b for b in beats
+ if any(abs(b - o) <= support_tol for o in onset_times)]
+ supported = len(supported_beats)
+ align_tol = interval / BEAT_ALIGN_DIVISOR
+ aligned = sum(1 for o in onset_times
+ if min((o - phase) % interval, interval - (o - phase) % interval) <= align_tol)
+
+ # How many spreads the best lag stands above the rest of them, mapped onto [0, 1]: a click
+ # track measures z ~= 6, a jittery human performance ~= 5, pseudo-random levels ~= 2.4, so
+ # the band [BEAT_Z_FLOOR, BEAT_Z_FLOOR + BEAT_Z_SPAN] = [2, 6] is where the answer changes.
+ z = ((peak - mean_acc) / sd_acc) if sd_acc > 0 else 0.0
+ periodicity = max(0.0, min(1.0, (z - BEAT_Z_FLOOR) / BEAT_Z_SPAN))
+ alignment = aligned / len(onset_times) if onset_times else 0.0
+ confidence = round(0.5 * periodicity + 0.5 * alignment, 3)
+
+ out.update({
+ "beats": beats, "supported_beats": supported_beats,
+ "interval": round(interval, 6), "tempo_bpm": round(60.0 / interval, 2),
+ "phase": round(phase, 4), "confidence": confidence, "supported": supported,
+ "unsupported": len(beats) - supported,
+ "usable": confidence >= float(min_confidence),
+ })
+ return out
+
+
+def snap_points(points: "Sequence[float]", beats: "Sequence[float]",
+ tolerance: float) -> "List[Dict[str, Any]]":
+ """Move each given point to the nearest beat within `tolerance` seconds.
+
+ Pure. Returns [{"from": t, "to": t2, "delta": d, "snapped": bool, "beat_index": i|None}].
+ A point with no beat inside `tolerance` is returned unchanged with snapped=False.
+
+ NEVER invents a point: len(out) == len(points), always, and every `to` is either a value that
+ was in `beats` or the caller's own `from`. This is the whole no-fabrication rule for beat
+ snapping -- a cut point may move to a measured grid point, and may not appear from one.
+ """
+ grid = sorted(float(b) for b in (beats or []))
+ out: "List[Dict[str, Any]]" = []
+ for p in points:
+ p = float(p)
+ best_i, best_d = None, None
+ for i, b in enumerate(grid):
+ d = abs(b - p)
+ if best_d is None or d < best_d:
+ best_i, best_d = i, d
+ if best_i is not None and best_d is not None and best_d <= float(tolerance):
+ out.append({"from": p, "to": grid[best_i], "delta": round(grid[best_i] - p, 6),
+ "snapped": True, "beat_index": best_i})
+ else:
+ out.append({"from": p, "to": p, "delta": 0.0, "snapped": False, "beat_index": None})
+ return out
+
+
+# ------------------------------------------------------------- filler words (1.17)
+#
+# A filler word is removed only when a speech engine measured a start/end pair for it. There is no
+# heuristic fallback -- no "cut the 0.3 s blips that look like an 'um'", no language guess from the
+# filename. Without timings there is nothing to cut, and the tool says so.
+#
+# What is NOT in these lists is the substance of the decision. "like", "tipo" and "cioè" are
+# discourse markers, not disfluencies: they are grammatical words in most sentences, and removing
+# them cuts meaning rather than noise. That is a judgement about content, which this skill does not
+# make. They are reachable with --filler-extra, and documented as what they are.
+FILLER_WORDS = {
+ "en": frozenset({"um", "uh", "erm", "hmm", "mm", "mhm", "er", "ah"}),
+ # なんか is the most common Japanese filler AND a pronoun/adverb spelled identically. It is in the
+ # default list because leaving it out makes --filler useless for Japanese, and every run that
+ # removes one warns that it is often a content word (--filler-keep なんか takes it out).
+ "ja": frozenset({"えー", "えーと", "えっと", "あの", "あのー", "その", "そのー", "まあ", "なんか"}),
+ "es": frozenset({"eh", "este", "esto", "mmm"}),
+ "de": frozenset({"äh", "ähm", "hm"}),
+ "fr": frozenset({"euh", "hein"}),
+ "pt": frozenset({"é", "hum"}),
+ "it": frozenset({"ehm"}),
+}
+
+# Words in a default list that are also ordinary vocabulary: every run that removes one says so.
+FILLER_AMBIGUOUS = {
+ "ja": frozenset({"なんか", "あの", "その", "まあ"}),
+ "es": frozenset({"este", "esto"}),
+ "pt": frozenset({"é"}),
+}
+
+# Discourse markers people ask for by name. Not defaults; named here so --help and the docs can
+# say what adding one costs.
+FILLER_DISCOURSE_MARKERS = {
+ "en": ("like", "you know"),
+ "pt": ("tipo",),
+ "it": ("cioè",),
+}
+
+FILLER_MAX_WORD = 1.2 # a longer "uhhh" is a held vowel someone meant
+FILLER_MIN_GAP = 0.05 # spans closer than this become one span
+FILLER_PAD = 0.02 # trimmed either side of the word
+
+
+def normalise_filler_token(word: str) -> str:
+ """A spoken token stripped to what a word list can be compared against: case-folded, with
+ surrounding punctuation and whitespace removed. Never a substring match -- "umbrella" must
+ survive a list containing "um"."""
+ import unicodedata as _ud
+ text = str(word or "").strip()
+ text = "".join(ch for ch in text
+ if not _ud.category(ch).startswith("P") or ch in "-'’")
+ return text.strip("-'’").casefold()
+
+
+def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = FILLER_MIN_GAP,
+ max_word: float = FILLER_MAX_WORD) -> "List[Dict[str, Any]]":
+ """Time spans to remove, from measured word timings. Pure: no subprocess, no I/O.
+
+ `words` is [{"word", "start", "end"}] as whisper emits. A word is removed only when its
+ normalised form is in `wordlist` AND it carries a real start < end pair AND its length is
+ <= max_word. Adjacent spans closer than min_gap merge. Returns [{"start", "end", "word"}]
+ sorted and non-overlapping.
+ """
+ listed = {normalise_filler_token(w) for w in (wordlist or set())}
+ listed.discard("")
+ hits = []
+ for entry in words or []:
+ if not isinstance(entry, dict):
+ continue
+ token = normalise_filler_token(entry.get("word") or entry.get("text") or "")
+ if not token or token not in listed:
+ continue
+ try:
+ start, end = float(entry["start"]), float(entry["end"])
+ except (KeyError, TypeError, ValueError):
+ continue # no measured timing: nothing to cut
+ if not (end > start) or (end - start) > max_word:
+ continue
+ hits.append({"start": max(0.0, start - pad), "end": end + pad, "word": token})
+ hits.sort(key=lambda h: (h["start"], h["end"]))
+ merged: "List[Dict[str, Any]]" = []
+ for h in hits:
+ if merged and h["start"] - merged[-1]["end"] <= min_gap:
+ merged[-1]["end"] = max(merged[-1]["end"], h["end"])
+ merged[-1]["word"] = merged[-1]["word"] + " " + h["word"]
+ else:
+ merged.append(dict(h))
+ for m in merged:
+ m["start"] = round(m["start"], 4)
+ m["end"] = round(m["end"], 4)
+ return merged
diff --git a/scripts/_common/text.py b/scripts/_common/text.py
index bdf16d3..d6486ef 100644
--- a/scripts/_common/text.py
+++ b/scripts/_common/text.py
@@ -1529,3 +1529,127 @@ def wrap_variants(text: str, max_em: float, *, mode: str = "phrase",
if not greedy:
greedy = [text]
return (wrapped or [text], greedy, measured or [text])
+
+
+# --- caption size that fits the cue (1.17) -------------------------------------------------
+# The legibility floor: 4.5 % of the frame height, ass_units(0.045) = 13 against the 288-line
+# ASS script grid. One floor for every destination -- 87 px of type on a 1920-tall frame, above
+# the ~3.5 % where mobile legibility bottoms out and where the platforms' own caption UIs sit.
+# Nothing per-platform is measured, so nothing per-platform is claimed. (The eval-17 cues happen
+# to land exactly on it: 13 is the smallest size at which every one of them fits two lines.)
+MIN_CAPTION_FRACTION = 0.045
+ASS_SCRIPT_HEIGHT = 288 # caption.py's --size/--margin reference grid; mirrors _platforms
+
+
+def line_em_for_size(size: float, play_w: "Optional[int]", play_h: "Optional[int]", *,
+ safe_fraction: float = SAFE_WIDTH_FRACTION,
+ script_height: int = ASS_SCRIPT_HEIGHT) -> "Optional[float]":
+ """How many em fit on one caption line at `size`, or None without geometry.
+
+ `size` is in ASS points against a `script_height`-line script (what libass's force_style
+ uses), so the rendered pixel size is size * play_h / script_height. This is the one width
+ formula: caption.py::max_line_em and fit_size() both call it.
+ """
+ if not play_w or not play_h or not size:
+ return None
+ size_px = size * play_h / float(script_height)
+ if size_px <= 0:
+ return None
+ return (play_w * safe_fraction) / size_px
+
+
+def fit_size(cues, *, size: int, min_size: "Optional[int]" = None, max_lines: int = 2,
+ play_w: "Optional[int]" = None, play_h: "Optional[int]" = None,
+ safe_fraction: float = SAFE_WIDTH_FRACTION, mode: str = "phrase",
+ lang: "Optional[str]" = None, script_height: int = ASS_SCRIPT_HEIGHT,
+ step: int = 1, scope: str = "file") -> "Dict[str, Any]":
+ """The largest size in [min_size, size] at which every cue wraps to <= max_lines lines.
+
+ Pure: strings and integers in, a dict out. No ffmpeg, no ffprobe, no I/O -- the caption size
+ is a text-measurement decision, and measuring it must not need a subprocess.
+
+ `cues` is an iterable of cue texts (or of (start, end, text) tuples, as caption.py holds
+ them before layout). Returns
+ {"size", "floor", "requested", "scope", "shrunk", "fits", "per_cue", "max_em", "steps"}.
+
+ The search is a linear walk downwards, not a bisection, and deliberately so:
+ len(wrap_text(t, max_em)) is NOT guaranteed monotone in max_em under the phrase rules -- a
+ rebalance that is discarded at one width can be applied at the next -- and a non-monotone
+ predicate breaks bisection. 24 -> 13 is at most twelve iterations of pure string work.
+
+ `scope="cue"` returns one size per cue index in `per_cue`, with `size` the minimum of them;
+ the caller writes a per-cue {\\fsN} override. The default is `scope="file"`: a caption track
+ whose type size changes from cue to cue reads as a mistake, and one measured line width per
+ file is what makes the wrap behaviour reproducible.
+ """
+ # `texts` stays parallel to `cues`: a blank cue becomes None rather than being dropped, so
+ # per_cue[i] always refers to the caller's cue i. caption.py indexes layout by these keys.
+ texts: "List[Optional[str]]" = []
+ for cue in cues or []:
+ if isinstance(cue, (tuple, list)):
+ raw = cue[2] if len(cue) > 2 else cue[-1]
+ else:
+ raw = cue
+ texts.append(raw if raw and str(raw).strip() else None)
+ measurable = [t for t in texts if t is not None]
+ requested = int(size)
+ floor = int(min_size) if min_size is not None else ass_units_local(MIN_CAPTION_FRACTION,
+ script_height)
+ floor = max(1, min(floor, requested))
+ step = max(1, int(step))
+ result: "Dict[str, Any]" = {"size": requested, "floor": floor, "requested": requested,
+ "scope": scope, "shrunk": 0, "fits": True, "per_cue": {},
+ "max_em": None, "steps": 0}
+ em_at = lambda sz: line_em_for_size(sz, play_w, play_h, safe_fraction=safe_fraction,
+ script_height=script_height)
+ base_em = em_at(requested)
+ result["max_em"] = base_em
+ if not measurable or base_em is None or max_lines < 1:
+ # No geometry means no measurable width: leave the size exactly as asked.
+ return result
+
+ def lines_at(text: str, sz: int) -> int:
+ em = em_at(sz)
+ if em is None:
+ return 1
+ return len(wrap_text(text, em, mode=mode, lang=lang))
+
+ over_at_requested = [t for t in measurable if lines_at(t, requested) > max_lines]
+ result["shrunk"] = len(over_at_requested)
+
+ def best_for(subset) -> "Tuple[int, bool]":
+ """(largest size in [floor, requested] fitting every text in `subset`, did it fit)."""
+ sz = requested
+ while sz >= floor:
+ result["steps"] += 1
+ if all(lines_at(t, sz) <= max_lines for t in subset):
+ return sz, True
+ sz -= step
+ return floor, all(lines_at(t, floor) <= max_lines for t in subset)
+
+ if scope == "cue":
+ per_cue = {}
+ fits_all = True
+ for i, t in enumerate(texts):
+ if t is None:
+ per_cue[i] = requested # a blank cue draws nothing; it constrains nothing
+ continue
+ sz, ok = best_for([t])
+ per_cue[i] = sz
+ fits_all = fits_all and ok
+ result["per_cue"] = per_cue
+ sized = [v for i, v in per_cue.items() if texts[i] is not None]
+ result["size"] = min(sized) if sized else requested
+ result["fits"] = fits_all
+ else:
+ sz, ok = best_for(measurable)
+ result["size"] = sz
+ result["fits"] = ok
+ result["max_em"] = em_at(result["size"])
+ return result
+
+
+def ass_units_local(fraction: float, script_height: int = ASS_SCRIPT_HEIGHT) -> int:
+ """`fraction` of the frame height in ASS units. Mirrors _platforms.ass_units, kept here so
+ _common.text stays importable without the scripts/ top level on sys.path."""
+ return int(round(fraction * script_height))
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 05f7e60..8299bb3 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -170,7 +170,8 @@ TOOL_META: Dict[str, Dict[str, Any]] = {
required=FF + ["filter:loudnorm"], optional=[{"capability": AAC, "when": "output extension isn't .mp3/.opus/.ogg/.flac (audio_codec_for()'s default)"}] + AUDIO_OUT,
video_required=False, audio_only=True, visual=False, verify=["probe", "check"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"silence": dict(role="analysis_and_execution", inputs=["video or audio asset"], outputs=["silence list JSON (--list)", "artifact with silences removed", "EDL text (--edl)"],
- required=FF + ["filter:silencedetect"], optional=[{"capability": X264, "when": "removing silences from a video"}, HDR_X265, {"capability": AAC, "when": "removing silences from a video"}] + AUDIO_OUT,
+ required=FF + ["filter:silencedetect"], optional=[{"capability": X264, "when": "removing silences from a video"}, HDR_X265, {"capability": AAC, "when": "removing silences from a video"},
+ {"capability": "external:whisper", "when": "--filler --transcribe"}] + AUDIO_OUT,
video_required=False, audio_only=True, visual=False, verify=["probe"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"join": dict(role="execution", inputs=["two or more video assets, or two or more audio-only assets"], outputs=["concatenated video artifact", "concatenated audio artifact (audio-only inputs, audio output extension)"],
required=FF + ["filter:xfade", "filter:acrossfade"],
@@ -197,7 +198,8 @@ TOOL_META: Dict[str, Dict[str, Any]] = {
required=["ffprobe"], optional=[{"capability": "ffmpeg", "when": "loudness rows (default)"}, {"capability": "filter:loudnorm", "when": "loudness rows (default)"}],
video_required=False, audio_only=True, visual=False, verify=[], produces_artifact=False, idempotency="bit_exact", deterministic=True),
"scenes": dict(role="analysis", inputs=["video asset"], outputs=["scene / audio-peak / highlight JSON on stdout", "EDL text (--edl)", "per-scene contact sheet PNG (--sheet)"],
- required=FF + ["filter:scdet"], optional=[{"capability": "filter:drawtext", "when": "--sheet"}, {"capability": "filter:tile", "when": "--sheet"}],
+ required=FF + ["filter:scdet"], optional=[{"capability": "filter:drawtext", "when": "--sheet"}, {"capability": "filter:tile", "when": "--sheet"},
+ {"capability": "ffmpeg", "when": "--beats (one audio decode for the onset pass)"}],
video_required=True, audio_only=False, visual=False, verify=[], produces_artifact=True, idempotency="bit_exact", deterministic=True),
"look": dict(role="verification", inputs=["video artifact"], outputs=["PNG contact sheet / frames / side-by-side"],
required=FF + ["filter:tile"], optional=[{"capability": "filter:drawtext", "when": "timecode stamps (default; --no-timecode to skip)"},
diff --git a/scripts/batch.py b/scripts/batch.py
index 41b6646..1d48b5a 100755
--- a/scripts/batch.py
+++ b/scripts/batch.py
@@ -27,9 +27,10 @@ import hashlib
import json
import os
import sys
+import threading
import time
from pathlib import Path
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional
from _common import STATE, add_common, apply_common, child_args, die, emit, info, run_tool, read_text_or_die, MEDIA_EXT as _MEDIA_EXT
@@ -72,23 +73,44 @@ def recipe_key(recipe: Dict[str, Any]) -> str:
return hashlib.sha1((json.dumps(recipe, sort_keys=True) + "\0" + project_content).encode()).hexdigest()[:12]
-def run_step(argv: List[str]) -> bool:
+JOBS_CAP = 8 # beyond this, concurrent encodes contend for the same cores and memory
+
+_LOG = threading.local()
+
+
+def log(message: str) -> None:
+ """info(), unless this thread is a --jobs worker -- then the line is buffered and flushed in
+ file order when the item finishes, so a parallel run's log reads exactly like a serial one."""
+ buf = getattr(_LOG, "buffer", None)
+ if buf is None:
+ info(message)
+ else:
+ buf.append(message)
+
+
+def run_step(argv: List[str], per_call: "Optional[float]" = None) -> bool:
script = argv[0]
if script not in ALLOWED_STEP_SCRIPTS:
die(f"recipe step names a script that isn't one of this skill's own tools: {script!r} "
f"(must be a bare filename like 'silence.py', found in scripts/)")
cmd = [str(HERE / script)] + argv[1:] + child_args()
- info(" → " + " ".join(os.path.basename(c) if i < 1 else c for i, c in enumerate(cmd)))
- proc = run_tool(cmd)
+ log(" → " + " ".join(os.path.basename(c) if i < 1 else c for i, c in enumerate(cmd)))
+ proc = run_tool(cmd, per_call=per_call)
if proc.returncode != 0:
- info(" " + "\n ".join(proc.stderr.strip().splitlines()[-4:]))
+ log(" " + "\n ".join(proc.stderr.strip().splitlines()[-4:]))
return False
for line in proc.stderr.splitlines():
if line.startswith("warning:"): # a step's deprecation notice is not swallowed by a success (review 9)
- info(" " + line)
+ log(" " + line)
return True
+def _run_buffered(fn, *a):
+ """Run a worker and hand back (its result, the log lines it produced)."""
+ r = fn(*a)
+ return r, list(getattr(_LOG, "lines", None) or [])
+
+
def final_path(src: Path, recipe: Dict[str, Any], outdir: Path) -> Path:
suffix = recipe.get("suffix", "_out")
# By default final_ext falls back to each source's OWN extension, so files that only differ
@@ -101,9 +123,17 @@ def final_path(src: Path, recipe: Dict[str, Any], outdir: Path) -> Path:
return outdir / f"{src.stem}{suffix}.{final_ext}"
-def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path) -> Dict[str, Any]:
+def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
+ deadline: "Optional[float]" = None) -> Dict[str, Any]:
final = final_path(src, recipe, outdir)
t0 = time.time()
+
+ def budget() -> "Optional[float]":
+ """What is left of the BATCH's time limit -- not a fresh one per item. A --timeout is a
+ promise about the whole run, so a queue of 40 files cannot quietly take 40 timeouts."""
+ if deadline is None:
+ return None
+ return max(1.0, deadline - time.monotonic())
if recipe.get("project"):
try:
proj = json.loads(read_text_or_die(str(recipe["project"]), "recipe.project"))
@@ -117,7 +147,7 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path) -> Dict
proj["output"] = str(final.resolve())
pj = work / f"{src.stem}_project.json"
pj.write_text(json.dumps(proj, indent=2), encoding="utf-8")
- ok = run_step(["render.py", str(pj)])
+ ok = run_step(["render.py", str(pj)], budget())
else:
steps = recipe.get("steps") or []
if not steps:
@@ -128,7 +158,7 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path) -> Dict
last = i == len(steps) - 1
out = str(final) if last else str(work / f"{src.stem}_step{i}.{'mp4' if src.suffix.lower() not in ('.wav', '.mp3', '.m4a', '.flac') else src.suffix.lstrip('.')}")
argv = [str(a).replace("{in}", cur).replace("{out}", out) for a in step]
- if not run_step(argv):
+ if not run_step(argv, budget()):
ok = False
break
cur = out
@@ -141,11 +171,17 @@ def main() -> int:
ap.add_argument("--recipe", required=True, help="batch.json")
ap.add_argument("--force", action="store_true", help="ignore the cache and redo everything")
ap.add_argument("--watch", type=float, help="keep polling the folder every N seconds")
+ ap.add_argument("--jobs", default="1", metavar="N",
+ help="process N files at once, or 'auto' for min(cpu_count, 4). Capped at "
+ "min(N, cpu_count, 8): every item is already an ffmpeg that threads "
+ "across cores, so more than a few contend rather than go faster. "
+ "Default 1, which is 1.16's behaviour exactly.")
ap.add_argument("--work", help="work directory for intermediates (default: <output_dir>/.work)")
add_common(ap)
args = ap.parse_args()
apply_common(args)
+ started = time.time()
folder = Path(args.folder).resolve() # relative 'bdir' used to become bdir/bdir/out once joined with the default outdir
if not folder.is_dir():
die(f"not a folder: {folder}")
@@ -186,6 +222,34 @@ def main() -> int:
rkey = recipe_key(recipe)
glob = recipe.get("glob") or "*"
+ # --jobs: every item is itself an ffmpeg that already threads across cores, so beyond a few
+ # concurrent encodes the jobs contend and wall-clock stops improving while memory does not.
+ # A number above the cap is clamped with a note, not refused: an optimistic number is not an
+ # error, and refusing one helps nobody.
+ cpus = os.cpu_count() or 1
+ requested = min(cpus, 4) if str(args.jobs).lower() == "auto" else None
+ if requested is None:
+ try:
+ requested = int(args.jobs)
+ except ValueError:
+ die(f"--jobs {args.jobs!r}: a whole number, or 'auto'", kind="input")
+ if requested < 1:
+ die("--jobs must be at least 1", kind="input")
+ jobs = max(1, min(requested, cpus, JOBS_CAP))
+ if jobs != requested:
+ info(f"--jobs {requested} capped to {jobs} (min of the request, {cpus} CPU(s) and the "
+ f"{JOBS_CAP}-job ceiling): each item is already a multi-threaded encode")
+ # One budget for the whole batch, not one per item -- but only where that cannot change what
+ # 1.16 did. A stated --timeout is a statement about this run, and asking for --jobs > 1 is
+ # asking for the batch to be treated as one piece of work; the default sequential path with
+ # the default timeout keeps 1.16's behaviour exactly, where each item got its own ceiling and
+ # a long folder was never cut off part-way.
+ shared_budget = bool(args.timeout) or jobs > 1
+ deadline = time.monotonic() + STATE.timeout if (shared_budget and STATE.timeout) else None
+ cache_lock = threading.Lock()
+ timed_out = {"hit": False}
+ interrupted = {"hit": False}
+
def one_pass() -> List[Dict[str, Any]]:
results = []
files = sorted(p for p in folder.glob(glob) if p.is_file() and p.suffix.lower() in MEDIA_EXT and outdir not in p.parents)
@@ -201,17 +265,23 @@ def main() -> int:
detail = "; ".join(f"{dst.name} <- {', '.join(s.name for s in srcs)}" for dst, srcs in collisions.items())
die(f"{len(collisions)} output filename collision(s) in this batch -- rename the sources, "
f"or add a distinguishing \"suffix\"/\"ext\" per run, or split into separate globs: {detail}")
- for src in files:
- key = f"{file_key(src)}:{rkey}"
- hit = cache.get(key)
- if hit and Path(hit.get("output", "")).exists() and not args.force:
- info(f"skip (cached) {src.name}")
- results.append({**hit, "cached": True})
- continue
- info(f"=== {src.name}")
- r = process(src, recipe, outdir, work)
- results.append(r)
- if r["ok"] and not STATE.dry_run:
+ def item_work(i: int, src: Path) -> Path:
+ """Where this item's intermediates go. Parallel items must not share one work dir:
+ the step file names are stem-derived, so two globs holding the same stem would write
+ over each other. Serial runs keep the flat layout 1.16 used, byte for byte."""
+ if jobs == 1:
+ return work
+ sub = work / f"{i}-{src.stem}"
+ sub.mkdir(parents=True, exist_ok=True)
+ return sub
+
+ def store(key: str, r: Dict[str, Any]) -> None:
+ if not (r["ok"] and not STATE.dry_run):
+ return
+ # The read-modify-write of the in-memory dict needs the lock even though the file
+ # write is already atomic: two finishers could otherwise serialise from two different
+ # snapshots and lose an entry.
+ with cache_lock:
cache[key] = r
# write_text isn't atomic -- a process killed mid-write (or a --watch loop racing
# a concurrent manual run) could leave a truncated file that json.loads() above
@@ -219,9 +289,184 @@ def main() -> int:
# entry. Write to a sibling temp file and rename into place: same-directory
# renames are atomic on POSIX and os.replace() is atomic on Windows too, so a
# reader only ever sees the old complete file or the new complete file.
- tmp = cache_path.parent / f"{cache_path.name}.tmp{os.getpid()}"
+ tmp = cache_path.parent / f"{cache_path.name}.tmp{os.getpid()}.{threading.get_ident()}"
tmp.write_text(json.dumps(cache, indent=2), encoding="utf-8")
os.replace(tmp, cache_path)
+
+ # ONE list, indexed by each file's position in the sorted `files`. A cached hit goes
+ # into its own slot rather than being appended ahead of the items that still have to run:
+ # appending in two passes reordered the per-item table whenever the cache was partially
+ # warm, which happens at --jobs 1 too and contradicts the order this tool promises.
+ slots: "List[Optional[Dict[str, Any]]]" = [None] * len(files)
+ pending: List[tuple] = []
+ for i, src in enumerate(files):
+ key = f"{file_key(src)}:{rkey}"
+ hit = cache.get(key)
+ if hit and Path(hit.get("output", "")).exists() and not args.force:
+ info(f"skip (cached) {src.name}")
+ slots[i] = {**hit, "cached": True}
+ continue
+ pending.append((i, src, key))
+
+ def timed_out_row(src: Path) -> "Dict[str, Any]":
+ timed_out["hit"] = True
+ return {"file": str(src), "output": str(final_path(src, recipe, outdir)),
+ "ok": False, "seconds": 0.0, "skipped": "timeout"}
+
+ def failed_row(src: Path, exc: BaseException) -> "Dict[str, Any]":
+ """A worker that raised is a failed item, not a dead run. A die() inside a thread
+ raises SystemExit through fut.result() and used to take the whole process down
+ before the summary and the per-item table were printed -- so the one thing the user
+ needed, which item failed and which succeeded, was the thing they did not get."""
+ reason = str(exc) or exc.__class__.__name__
+ return {"file": str(src), "output": str(final_path(src, recipe, outdir)),
+ "ok": False, "seconds": 0.0, "error": reason[:300]}
+
+ if jobs == 1 or len(pending) < 2:
+ for i, src, key in pending:
+ if deadline is not None and time.monotonic() >= deadline:
+ slots[i] = timed_out_row(src)
+ continue
+ info(f"=== {src.name}")
+ try:
+ r = process(src, recipe, outdir, item_work(i, src), deadline)
+ except KeyboardInterrupt:
+ interrupted["hit"] = True
+ break
+ except BaseException as exc: # noqa: BLE001 - reported, never swallowed
+ if isinstance(exc, SystemExit) and not exc.code:
+ raise
+ slots[i] = failed_row(src, exc)
+ continue
+ slots[i] = r
+ store(key, r)
+ return [r for r in slots if r is not None]
+
+ import concurrent.futures
+
+ def work_one(i: int, src: Path, key: str) -> Dict[str, Any]:
+ _LOG.buffer = [f"=== {src.name}"]
+ try:
+ r = process(src, recipe, outdir, item_work(i, src), deadline)
+ store(key, r)
+ return r
+ finally:
+ r_lines, _LOG.buffer = _LOG.buffer, None
+ setattr(_LOG, "lines", r_lines)
+
+ # Submitted in the existing sorted order and written straight into each item's own slot,
+ # so the summary and the per-item table are identical to a serial run's whatever order
+ # the encodes actually finish in.
+ lines: "Dict[int, List[str]]" = {}
+ with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
+ futures: "Dict[Any, tuple]" = {}
+ queue = list(pending)
+ in_flight: "set" = set()
+ try:
+ # The pool is topped up to `jobs` in flight and no further: submitting the whole
+ # list up front would put every item past the deadline check before the first one
+ # had finished, and the shared budget could then never stop anything.
+ while queue or in_flight:
+ while queue and len(in_flight) < jobs:
+ if deadline is not None and time.monotonic() >= deadline:
+ break
+ i, src, key = queue.pop(0)
+ fut = pool.submit(_run_buffered, work_one, i, src, key)
+ futures[fut] = (i, src)
+ in_flight.add(fut)
+ if not in_flight:
+ break
+ done_now, in_flight = concurrent.futures.wait(
+ in_flight, return_when=concurrent.futures.FIRST_COMPLETED)
+ in_flight = set(in_flight)
+ for fut in done_now:
+ i, src = futures[fut]
+ try:
+ slots[i], lines[i] = fut.result()
+ except BaseException as exc: # noqa: BLE001 - reported, never swallowed
+ slots[i] = failed_row(src, exc)
+ lines[i] = [f"=== {src.name}", f" failed: {exc}"]
+ for i, src, key in queue:
+ slots[i] = timed_out_row(src)
+ except KeyboardInterrupt:
+ # Spec 4.1: cancel what has not started, let the running children be killed by
+ # the shared signal handling, and REPORT what completed -- exit 130 with the
+ # partial table, never a traceback.
+ interrupted["hit"] = True
+ for fut in futures:
+ fut.cancel()
+ for fut, (i, src) in futures.items():
+ if slots[i] is not None or not fut.done():
+ continue
+ try:
+ slots[i], lines[i] = fut.result()
+ except BaseException: # noqa: BLE001
+ pass
+ info("interrupted: reporting what had already finished")
+ for i, src in enumerate(files):
+ for line in lines.get(i, []):
+ info(line)
+ return [r for r in slots if r is not None]
+
+ import concurrent.futures
+
+ def work_one(i: int, src: Path, key: str) -> Dict[str, Any]:
+ _LOG.buffer = [f"=== {src.name}"]
+ try:
+ r = process(src, recipe, outdir, item_work(i, src), deadline)
+ store(key, r)
+ return r
+ finally:
+ r_lines, _LOG.buffer = _LOG.buffer, None
+ setattr(_LOG, "lines", r_lines)
+
+ # Submitted in the existing sorted order and collected into a list indexed by submission
+ # order, so the summary and the per-item table are identical to a serial run's whatever
+ # order the encodes actually finish in.
+ ordered: List[Optional[Dict[str, Any]]] = [None] * len(pending)
+ lines: List[List[str]] = [[] for _ in pending]
+ with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
+ futures: "Dict[Any, int]" = {}
+ queue = list(enumerate(pending))
+ in_flight: "set" = set()
+ try:
+ # The pool is topped up to `jobs` in flight and no further: submitting the whole
+ # list up front would put every item past the deadline check before the first one
+ # had finished, and the shared budget could then never stop anything.
+ while queue or in_flight:
+ while queue and len(in_flight) < jobs:
+ slot, (i, src, key) = queue[0]
+ if deadline is not None and time.monotonic() >= deadline:
+ break
+ queue.pop(0)
+ fut = pool.submit(_run_buffered, work_one, i, src, key)
+ futures[fut] = slot
+ in_flight.add(fut)
+ if not in_flight:
+ break
+ done_now, in_flight = concurrent.futures.wait(
+ in_flight, return_when=concurrent.futures.FIRST_COMPLETED)
+ in_flight = set(in_flight)
+ for fut in done_now:
+ ordered[futures[fut]], lines[futures[fut]] = fut.result()
+ for slot, (i, src, key) in queue:
+ timed_out["hit"] = True
+ ordered[slot] = {"file": str(src),
+ "output": str(final_path(src, recipe, outdir)),
+ "ok": False, "seconds": 0.0, "skipped": "timeout"}
+ except KeyboardInterrupt:
+ for fut in futures:
+ fut.cancel()
+ info("interrupted: finishing what had already started")
+ raise
+ for slot, (i, src, key) in enumerate(pending):
+ for line in lines[slot]:
+ info(line)
+ if ordered[slot] is None:
+ timed_out["hit"] = True
+ ordered[slot] = {"file": str(src), "output": str(final_path(src, recipe, outdir)),
+ "ok": False, "seconds": 0.0, "skipped": "timeout"}
+ results.append(ordered[slot])
return results
results = one_pass()
@@ -246,11 +491,31 @@ def main() -> int:
if not args.json:
for r in results:
print(f"{'OK ' if r['ok'] else 'FAIL'} {r['file']} -> {r['output']}" + (" (cached)" if r.get("cached") else ""))
+ if interrupted["hit"]:
+ # Ctrl-C on a batch that already produced files: the user needs the partial table, not a
+ # traceback and not "nothing was written". Exit 130 with everything that completed.
+ die(f"interrupted after {done} of {len(results)} item(s); the finished outputs are kept "
+ "and the rest were not started",
+ code=130, kind="interrupted", output=None, dry_run=STATE.dry_run, results=results,
+ processed=done, total=len(results), jobs=jobs, jobs_requested=requested,
+ timed_out=timed_out["hit"])
+ if timed_out["hit"]:
+ skipped = [r["file"] for r in results if r.get("skipped") == "timeout"]
+ die(f"the batch's {STATE.timeout:.0f} s budget ran out with {len(skipped)} item(s) not "
+ f"started: {', '.join(os.path.basename(f) for f in skipped[:5])}"
+ + (" ..." if len(skipped) > 5 else "")
+ + ". --timeout is the whole run's limit, not each item's; raise it or split the folder.",
+ code=124, kind="timeout", output=None, dry_run=STATE.dry_run, results=results,
+ processed=done, total=len(results), jobs=jobs, jobs_requested=requested,
+ timed_out=True)
if done != len(results):
failed_files = [r["file"] for r in results if not r["ok"]]
die(f"{len(results) - done} of {len(results)} items failed: {', '.join(failed_files[:5])}" + (" ..." if len(failed_files) > 5 else ""),
kind="verification", output=None, dry_run=STATE.dry_run, results=results, processed=done, total=len(results))
- emit(None, results=results, processed=done, total=len(results))
+ emit(None, results=results, processed=done, total=len(results),
+ jobs=jobs, jobs_requested=requested, wall_seconds=round(time.time() - started, 1),
+ item_seconds_total=round(sum(float(r.get("seconds") or 0) for r in results), 1),
+ timed_out=timed_out["hit"])
return 0
diff --git a/scripts/caption.py b/scripts/caption.py
index ac15cc7..3d2a569 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -40,17 +40,27 @@ import unicodedata
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
-from _platforms import PLATFORMS, PLATFORM_CHOICES, ass_units, resolve as resolve_platform
+from _platforms import (PLATFORMS, PLATFORM_CHOICES, ASS_SCRIPT_HEIGHT, ass_units,
+ resolve as resolve_platform)
from _ass_overlay import EMOJI_SENTINEL, emoji_placeholder, ass_escape
from _common import emoji_filter_chain, EMOJI_ASSET_HINT, emoji_asset_for, emoji_codepoint_name, emoji_support, resolve_emoji_assets, ADVANCE_EM, LATIN_EM, NO_SPACE_SCRIPTS, _char_em, char_script, text_width_em, emoji_clusters, has_emoji, detect_script, BIDI_SCRIPTS, STATE, brand_states_font, script_font_for_text, signed_time_arg, brand_caption_style, color_hex, load_brand, video_args, add_common, apply_common, emit, aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, fmt_srt_time, fmt_smpte_time, info, MissingFpsError, parse_time, probe, run, x264_args, X264_PRESETS, read_text_or_die, fmt_secs
# The line breaker, lifted into _common/text.py in 1.16.0 so graphics.py can use the same rules.
+# The ASR bridge and the SRT reader/writer live in _common.asr since 1.17 (silence.py --filler
+# shares them). They stay caption.py's public names -- every caller and test that reached for
+# caption.parse_srt / caption.transcribe / caption.whisper_word_timings before 1.17 still does.
+from _common import (ASR_INSTALL_HINT, die_no_engine, parse_srt, transcribe, whisper_word_timings,
+ write_srt)
from _common import (SAFE_WIDTH_FRACTION, ORPHAN_MIN_EM, WRAP_MODES, wrap_text, wrap_variants, best_break,
+ fit_size, line_em_for_size, MIN_CAPTION_FRACTION,
break_penalty, _is_weak_line, _atoms, _join, _break_spaced, _bare_word, _function_words,
_split_hyphens, FUNCTION_WORDS, JA_PARTICLES, JA_SENTENCE_END, _fix_orphans, _rebalance)
# The breaker's names are caption.py's public surface as much as _common's: every caller and test
# that reached for `caption.wrap_text` before 1.16 still does.
-__all__ = ["SAFE_WIDTH_FRACTION", "ORPHAN_MIN_EM", "WRAP_MODES", "wrap_text", "wrap_variants",
+__all__ = ["parse_srt", "write_srt", "transcribe", "whisper_word_timings", "die_no_engine",
+ "ASR_INSTALL_HINT",
+ "SAFE_WIDTH_FRACTION", "ORPHAN_MIN_EM", "WRAP_MODES", "wrap_text", "wrap_variants",
+ "fit_size", "line_em_for_size", "MIN_CAPTION_FRACTION",
"best_break", "break_penalty", "_is_weak_line", "_atoms", "_join", "_break_spaced",
"_bare_word", "_function_words", "_split_hyphens", "FUNCTION_WORDS", "JA_PARTICLES",
"JA_SENTENCE_END", "_fix_orphans", "_rebalance", "char_script", "NO_SPACE_SCRIPTS",
@@ -97,150 +107,6 @@ def parse_text_cues(path: str, auto_seconds: float, gap: float, fps: Optional[fl
return cues
-def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0) -> List[Tuple[float, float, str]]:
- """Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),
- whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.
- No engine installed -> clear error with install hints; the skill never depends on one."""
- import shutil
- import subprocess
- import tempfile
- from _common import require_tool, run_analysis, STATE
- ffmpeg = require_tool("ffmpeg")
- tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")
- try:
- return _transcribe_in(tmpdir, video, out_srt, language, model, audio_stream, ffmpeg, shutil, subprocess)
- finally:
- shutil.rmtree(tmpdir, ignore_errors=True)
-
-
-def _asr_run(cmd: List[str], subprocess, name: str) -> "subprocess.CompletedProcess":
- """Run a speech-to-text engine under the same wall-clock limit as an ffmpeg call."""
- from _common import STATE, die
- limit = STATE.timeout or None
- try:
- return subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, timeout=limit)
- except subprocess.TimeoutExpired:
- die(f"{name} exceeded the {limit:.0f} s time limit and was killed; raise --timeout for a long recording",
- code=124, kind="timeout")
- return None # unreachable
-
-
-def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int,
- ffmpeg: str, shutil, subprocess) -> List[Tuple[float, float, str]]:
- from _common import run_analysis, STATE, die
- wav = os.path.join(tmpdir, "audio.wav")
- # A wav in our own temp dir: a measurement input for the engine, not a deliverable, so it
- # is not a run() call (no --dry-run gate, not recorded), but it keeps the time limit and
- # reports an unreadable input as kind ffmpeg instead of a CalledProcessError traceback.
- run_analysis([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,
- "-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav])
- # 1. whisper.cpp
- cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp")
- if not cli:
- # older whisper.cpp builds ship the binary as plain `main`; accept it only when it lives
- # in a directory that names whisper, so an unrelated /usr/bin/main is never run
- main_bin = shutil.which("main")
- if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
- cli = main_bin
- if cli:
- model_path = model
- if not os.path.exists(model_path):
- for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"), f"models/ggml-{model}.bin", f"/usr/local/share/whisper/ggml-{model}.bin"):
- if os.path.exists(cand):
- model_path = cand
- break
- base = os.path.join(tmpdir, "out")
- cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base]
- if language:
- cmd += ["-l", language]
- proc = _asr_run(cmd, subprocess, "whisper.cpp")
- if proc.returncode == 0 and os.path.exists(base + ".srt"):
- info(f"transcribed with whisper.cpp ({os.path.basename(cli)}, model {os.path.basename(model_path)})")
- cues = parse_srt(base + ".srt")
- write_srt(cues, out_srt)
- return cues
- info("whisper.cpp found but failed: " + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
- # 2. faster-whisper (python package)
- try:
- from faster_whisper import WhisperModel # type: ignore
- import threading
- result: list = []
-
- def work() -> None:
- m = WhisperModel(model, device="cpu", compute_type="int8")
- segments, _ = m.transcribe(wav, language=language, word_timestamps=False)
- result.extend((seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip())
-
- # An in-process engine gets the same wall-clock limit as the CLI engines and ffmpeg.
- t = threading.Thread(target=work, daemon=True)
- t.start()
- t.join(STATE.timeout or None)
- if t.is_alive():
- die(f"faster-whisper exceeded the {STATE.timeout:.0f} s time limit; raise --timeout for a long recording", code=124, kind="timeout")
- cues = list(result)
- if cues:
- info("transcribed with faster-whisper")
- write_srt(cues, out_srt)
- return cues
- except ImportError:
- pass
- # 3. openai-whisper CLI
- if shutil.which("whisper"):
- cmd = ["whisper", wav, "--model", model, "--output_format", "srt", "--output_dir", tmpdir]
- if language:
- cmd += ["--language", language]
- proc = _asr_run(cmd, subprocess, "openai-whisper")
- srt = os.path.join(tmpdir, "audio.srt")
- if proc.returncode == 0 and os.path.exists(srt):
- info("transcribed with openai-whisper")
- cues = parse_srt(srt)
- write_srt(cues, out_srt)
- return cues
- die("no local speech-to-text engine found for --transcribe.\n"
- "Install one (all run offline):\n"
- " whisper.cpp: brew install whisper-cpp (then download a model: ggml-base.bin)\n"
- " faster-whisper: pip install faster-whisper\n"
- " openai-whisper: pip install openai-whisper\n"
- "Or write the cues by hand with --text cues.txt (see format above).")
- return []
-
-
-def parse_srt(path: str) -> List[Tuple[float, float, str]]:
- cues: List[Tuple[float, float, str]] = []
- block: List[str] = []
- content = read_text_or_die(path, "--srt").lstrip("\ufeff").replace("\r\n", "\n") + "\n\n"
- for line in content.split("\n"):
- if line.strip():
- block.append(line)
- continue
- if block:
- times = next((b for b in block if "-->" in b), None)
- if times:
- a, b = times.split("-->")
- text = "\n".join(block[block.index(times) + 1:]).strip()
- try:
- cues.append((parse_time(a), parse_time(b), text))
- except ValueError as e: # includes MissingFpsError: SRT timings are hh:mm:ss,ms, never frames
- die(f"{path}: cannot read the timing line {times.strip()!r}: {e}")
- block = []
- if not cues:
- die(f"no cues found in {path}")
- return cues
-
-
-def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
- with open(path, "w", encoding="utf-8") as fh:
- for i, (s, e, t) in enumerate(cues, 1):
- # A blank line is SRT's own block separator (index/timecode/text, blank, next block).
- # Cue text can contain one -- parse_text_cues() turns a bare "|" into "\n", so a source
- # line with two adjacent pipes ("a||b") becomes "a\n\nb" -- and writing that blank line
- # raw would split one cue into two malformed half-blocks (the second missing its own
- # index/timecode). Collapse any run of blank lines within the cue text to a single
- # newline so the cue's own text can never fake the format's block boundary.
- t = re.sub(r"\n{2,}", "\n", t).strip("\n")
- fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
-
-
def word_durations_from_audio(video: str, start: float, end: float, n_words: int, audio_stream: int = 0) -> List[int]:
"""Split a cue's time across n_words in proportion to speech energy (centiseconds each).
@@ -434,7 +300,8 @@ def plan_emoji(cues, args, play_w, play_h, brand=None):
def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float], max_lines: int,
min_duration: float, offset: float, wrap: str = "phrase",
- lang: Optional[str] = None) -> Tuple[List[Tuple[float, float, str]], dict]:
+ lang: Optional[str] = None, per_cue_em: Optional[List[Optional[float]]] = None,
+ per_cue_size: Optional[List[int]] = None) -> Tuple[List[Tuple[float, float, str]], dict]:
"""Shift, wrap, split and lengthen cues so they can actually be read.
`offset` moves every cue (a transcript that runs early/late); `max_em` wraps each cue to the
@@ -442,11 +309,25 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
a cue needing more than `max_lines` lines is split into consecutive cues sharing its time in
proportion to their text; a cue shorter than `min_duration` is lengthened, never past the next
cue's start. Returns the new cues and a count of what changed.
+
+ `per_cue_em` (--fit-size-scope cue) gives cue i its OWN line budget instead of the file's:
+ a cue drawn at a larger size has a narrower line in em, and wrapping it to the file-wide
+ budget -- which is the budget of the SMALLEST size -- produced lines that overflowed the
+ frame when they were then drawn large. `per_cue_size` rides along so the caller knows which
+ size each OUTPUT cue belongs to after splits have renumbered them; it comes back as
+ `stats["cue_sizes"]`, one entry per returned cue.
"""
stats = {"shifted": 0, "wrapped": 0, "split": 0, "extended": 0, "dropped": 0, "rebalanced": 0,
"wrap": wrap, "phrase_breaks": 0}
staged: List[Tuple[float, float, str]] = []
- for start, end, text in cues:
+ staged_sizes: List[Optional[int]] = []
+ for cue_index, (start, end, text) in enumerate(cues):
+ own_em = max_em
+ own_size = None
+ if per_cue_em is not None and cue_index < len(per_cue_em):
+ own_em = per_cue_em[cue_index]
+ if per_cue_size is not None and cue_index < len(per_cue_size):
+ own_size = per_cue_size[cue_index]
if offset:
start, end = start + offset, end + offset
if end <= 0:
@@ -454,14 +335,14 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
continue
start = max(0.0, start)
stats["shifted"] += 1
- if max_em and max_em > 0:
+ if own_em and own_em > 0:
# one greedy fill per cue, three answers off it: what gets burnt in, what the
# greedy wrap would have given (`rebalanced`) and what 1.15's wrap would have
# given (`phrase_breaks`). Three wrap_text() calls re-ran the atomiser each time.
- lines, greedy, measured = wrap_variants(text, max_em, mode=wrap, lang=lang)
+ lines, greedy, measured = wrap_variants(text, own_em, mode=wrap, lang=lang)
if lines != [l for l in text.split("\n") if l.strip()]:
stats["wrapped"] += 1
- if any(text_width_em(l) > max_em for l in lines):
+ if any(text_width_em(l) > own_em for l in lines):
# a run with no break point the wrapper may use (a long word, a Thai phrase
# without spaces) stays long rather than chopped: say so, and name the fix
stats["overlong"] = stats.get("overlong", 0) + 1
@@ -477,11 +358,13 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
for chunk, weight in zip(chunks, weights):
seg = (end - start) * weight / total_w
staged.append((t, min(end, t + seg), "\n".join(chunk)))
+ staged_sizes.append(own_size)
t += seg
stats["split"] += len(chunks) - 1
continue
text = "\n".join(lines)
staged.append((start, end, text))
+ staged_sizes.append(own_size)
out: List[Tuple[float, float, str]] = []
for i, (start, end, text) in enumerate(staged):
if min_duration and end - start < min_duration:
@@ -491,6 +374,8 @@ def layout_cues(cues: List[Tuple[float, float, str]], *, max_em: Optional[float]
stats["extended"] += 1
end = new_end
out.append((start, end, text))
+ if per_cue_size is not None:
+ stats["cue_sizes"] = staged_sizes
return out, stats
@@ -511,12 +396,7 @@ def max_line_em(args, play_w: Optional[int], play_h: Optional[int]) -> Optional[
--size is in ASS points against a 288-line script (what libass's force_style uses), so the
rendered pixel size is size * play_h / 288.
"""
- if not play_w or not play_h or not args.size:
- return None
- size_px = args.size * play_h / 288.0
- if size_px <= 0:
- return None
- return (play_w * SAFE_WIDTH_FRACTION) / size_px
+ return line_em_for_size(args.size, play_w, play_h)
def parse_ass_dialogue(path: str) -> str:
@@ -561,44 +441,6 @@ def shift_ass_file(src: str, dst: str, offset: float) -> int:
return n
-def whisper_word_timings(srt_path: Optional[str]) -> List[Tuple[float, float, str]]:
- """Word timings from a whisper JSON transcript sitting next to the SRT, if there is one.
-
- whisper (and faster-whisper, and whisper.cpp's --output-json) can emit per-word start/end
- times; when they are there, --karaoke should follow the real speech instead of splitting the
- cue evenly. Looked for as <stem>.json and <stem>.words.json next to the SRT, in either the
- {"segments": [{"words": [{"word": ..., "start": ..., "end": ...}]}]} or a bare
- {"words": [...]} shape. Anything unreadable is simply "no word timings".
- """
- if not srt_path:
- return []
- stem = os.path.splitext(srt_path)[0]
- for cand in (stem + ".words.json", stem + ".json"):
- if not os.path.exists(cand):
- continue
- try:
- data = json.loads(Path(cand).read_text(encoding="utf-8"))
- except (OSError, ValueError):
- continue
- raw = []
- if isinstance(data, dict):
- raw = list(data.get("words") or [])
- for seg in data.get("segments") or []:
- raw.extend((seg or {}).get("words") or [])
- words = []
- for w in raw:
- try:
- text = str(w.get("word") or w.get("text") or "").strip()
- if text:
- words.append((float(w["start"]), float(w["end"]), text))
- except (AttributeError, KeyError, TypeError, ValueError):
- continue
- if words:
- info(f"karaoke: word timings from {os.path.basename(cand)} ({len(words)} words)")
- return sorted(words)
- return []
-
-
def word_durations_from_timings(words: List[Tuple[float, float, str]], start: float, end: float,
n_words: int) -> Optional[List[int]]:
"""Centiseconds per word for one cue, from real word timings; None when they don't cover it."""
@@ -639,7 +481,7 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
"", "[Events]", "Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text",
]
lines = []
- for start, end, text in cues:
+ for cue_index, (start, end, text) in enumerate(cues):
# ASS Dialogue text treats a literal `{...}` as an override block -- real style/animation
# commands, not literal characters. Cue text (from --text, an SRT, or ASR transcription --
# all effectively user-controlled) that happens to contain braces would otherwise be
@@ -657,6 +499,15 @@ def write_ass(cues: List[Tuple[float, float, str]], path: str, args, play_w: int
elif args.animate == "slide":
fx = "{\\fad(150,150)\\move(%d,%d,%d,%d,0,250)}" % (play_w // 2, play_h - margin + int(30 * scale), play_w // 2, play_h - margin)
body = text
+ # --fit-size-scope cue: one size per cue, as a leading {\fsN} override. Opt-in only --
+ # see fit_size()'s docstring for why a size that changes cue to cue is not the default.
+ # The size is the one THIS cue was laid out at, carried through layout_cues; re-measuring
+ # here would measure the post-layout text (already carrying the wrap's newlines), which
+ # is a different string from the one the fit was computed on.
+ own_sizes = getattr(args, "_fit_sizes_out", None) or []
+ own = own_sizes[cue_index] if cue_index < len(own_sizes) else None
+ if own and own != args.size:
+ fx += "{\\fs%d}" % int(round(own * scale))
if args.karaoke:
# split each line into words and give every word an equal share of the cue (\k is in centiseconds)
dur_cs = max(1, int(round((end - start) * 100)))
@@ -898,6 +749,15 @@ def main() -> int:
emo.add_argument("--emoji-max", type=int, default=60,
help="most emoji overlays one run may build (default 60)")
sty.add_argument("--max-lines", type=int, default=2, help="most lines one cue may occupy; a longer cue is split into consecutive cues (default 2)")
+ sty.add_argument("--fit-size", choices=["auto", "on", "off"], default="auto",
+ help="shrink the caption size until the cue fits --max-lines, BEFORE splitting it: "
+ "'auto' (default) only when no --size was given, 'on' always, 'off' for 1.16 behaviour")
+ sty.add_argument("--min-size", type=int, default=None,
+ help="smallest size --fit-size may use, in ASS points (default 13 = 4.5%% of the frame height, "
+ "the legibility floor)")
+ sty.add_argument("--fit-size-scope", choices=["file", "cue"], default="file",
+ help="one fitted size for the whole file (default) or one per cue (a size that changes "
+ "cue to cue reads as a mistake, so it is opt-in)")
sty.add_argument("--min-duration", type=float, default=1.0, help="shortest time a cue stays on screen in seconds, never past the next cue (default 1.0)")
sty.add_argument("--wrap", choices=list(WRAP_MODES), default="phrase",
help="how a cue too wide for the safe area is broken into lines: 'phrase' (default, 1.16) never "
@@ -930,6 +790,10 @@ def main() -> int:
if args.brand and bcap.get("box") and not args.box:
args.box = True
args.font = args.font or (bcap.get("font") if args.brand else None) or brand.get("font") or "DejaVu Sans"
+ # --fit-size auto shrinks only a size the skill itself chose. An explicit --size is a
+ # statement about the look and is never quietly overridden; a brand's caption size is the
+ # same kind of statement, so it counts as explicit too.
+ args._size_explicit = args.size is not None or bool(args.brand and bcap.get("size") is not None)
args.size = args.size if args.size is not None else (bcap.get("size", 24) if args.brand else 24)
args.color = color_hex(args.color or (bcap.get("color") if args.brand else None) or bc.get("text", "FFFFFF"))
args.outline_color = color_hex(args.outline_color or bc.get("outline", "000000"))
@@ -969,6 +833,11 @@ def main() -> int:
args.offset = signed_time_arg(str(args.offset), "--offset")
if args.max_lines < 1:
die("--max-lines must be at least 1")
+ if args.min_size is not None and args.min_size < 1:
+ die("--min-size must be at least 1", kind="input")
+ if args.min_size is not None and args.min_size > args.size:
+ die(f"--min-size {args.min_size} is larger than --size {args.size}: the floor cannot be "
+ "above the size it is a floor for", kind="input")
if args.min_duration < 0:
die("--min-duration cannot be negative")
@@ -991,20 +860,95 @@ def main() -> int:
play_w, play_h = meta["video"]["width"], meta["video"]["height"]
if meta["video"].get("rotation") in (90, -90, 270, -270):
play_w, play_h = play_h, play_w
-
+ # A --plan or --dry-run written before the input exists has no geometry to fit against, and a
+ # plan that describes a different FontSize from the run that executes it is not a plan. When
+ # --platform names a destination, that destination's frame IS the geometry the real run will
+ # have, so the fit is computed against it; with no platform there is nothing to stand in for
+ # the frame, and size_used is reported as null rather than presenting the requested size as
+ # the size that was used.
+ planned_frame = False
+ if not (play_w and play_h) and args.platform and PLATFORMS[args.platform].get("frame"):
+ frame = PLATFORMS[args.platform]["frame"]
+ play_w, play_h = frame["w"], frame["h"]
+ planned_frame = True
+ info(f"[plan] no geometry to measure yet; fitting the caption size against the "
+ f"--platform {args.platform} frame ({play_w}x{play_h})")
+
+ args._fit_floor = args.min_size if args.min_size is not None else ass_units(MIN_CAPTION_FRACTION)
+ fit_unmeasurable = not (play_w and play_h)
+ fit_stats: dict = {"fit_size": args.fit_size, "size_requested": args.size,
+ "size_used": None if fit_unmeasurable else args.size,
+ "size_floor": args._fit_floor,
+ "size_pct_height": round(args.size * 100.0 / ASS_SCRIPT_HEIGHT, 2),
+ "shrunk": 0, "fit_scope": args.fit_size_scope, "fit_exhausted": False}
caption_stats: dict = {"shifted": 0, "wrapped": 0, "split": 0, "extended": 0, "dropped": 0,
"rebalanced": 0, "wrap": args.wrap, "phrase_breaks": 0}
+ caption_stats.update(fit_stats)
+
+ def fit_params():
+ return dict(size=fit_stats["size_requested"], min_size=args._fit_floor,
+ max_lines=args.max_lines, play_w=play_w, play_h=play_h,
+ mode=args.wrap, lang=args.language, scope=args.fit_size_scope)
+
+ def fit_the_size(cue_list):
+ """Shrink --size until every cue fits --max-lines, BEFORE the cue is split.
+
+ This is the whole point of the ordering: at a size the cue cannot fit, layout_cues splits
+ the sentence into consecutive cues and half of it arrives late (eval 17). The text is never
+ touched -- only the type size, and never below the legibility floor.
+ """
+ if args.fit_size == "off" or (args.fit_size == "auto" and args._size_explicit):
+ fit_stats["size_used"] = args.size # a stated size IS the size used
+ return
+ if fit_unmeasurable:
+ # Nothing to measure against: size_used stays null rather than presenting the
+ # requested size as one that was fitted.
+ return
+ if args.mode == "mux":
+ # Soft subtitles carry no size: the player picks it. Shrinking would change nothing a
+ # viewer sees and would silently change the SRT this run writes, so the mux path is
+ # left exactly as 1.16 wrote it.
+ return
+ fit = fit_size(cue_list, **fit_params())
+ fit_stats["size_used"] = fit["size"]
+ fit_stats["size_source"] = "platform-frame" if planned_frame else "input"
+ fit_stats["shrunk"] = fit["shrunk"]
+ fit_stats["fit_exhausted"] = bool(fit["shrunk"]) and not fit["fits"]
+ fit_stats["size_pct_height"] = round(fit["size"] * 100.0 / ASS_SCRIPT_HEIGHT, 2)
+ if fit["size"] != args.size:
+ info(f"caption size {args.size} -> {fit['size']} ASS units "
+ f"({fit_stats['size_pct_height']:.1f} % of frame height) so {fit['shrunk']} cue(s) "
+ f"fit --max-lines {args.max_lines}; floor {args._fit_floor}")
+ args.size = fit["size"]
+ elif fit_stats["fit_exhausted"]:
+ info(f"caption size stays {args.size} ASS units: {fit['shrunk']} cue(s) still need more "
+ f"than {args.max_lines} line(s) at the floor {args._fit_floor} and are split "
+ "(--min-size goes smaller; `|` sets the break yourself)")
+ if args.fit_size_scope == "cue":
+ # Each cue is laid out at ITS OWN size, not at the file minimum. A cue drawn larger
+ # has a NARROWER line in em, so wrapping everything to the minimum size's (widest)
+ # budget and then drawing some cues large put lines off the side of the frame.
+ args._fit_cue_em = [line_em_for_size(fit["per_cue"].get(i, fit["size"]),
+ play_w, play_h)
+ for i in range(len(cue_list))]
+ args._fit_cue_size = [fit["per_cue"].get(i, fit["size"]) for i in range(len(cue_list))]
def lay_out(cue_list):
"""Wrap to the safe area, split past --max-lines, lengthen to --min-duration, shift by
--offset -- the one place every cue source goes through, so an SRT, a cue file and a
transcript all come out equally readable."""
+ fit_the_size(cue_list)
out, stats = layout_cues(cue_list, max_em=max_line_em(args, play_w, play_h),
max_lines=args.max_lines, min_duration=args.min_duration,
- offset=args.offset, wrap=args.wrap, lang=args.language)
+ offset=args.offset, wrap=args.wrap, lang=args.language,
+ per_cue_em=getattr(args, "_fit_cue_em", None),
+ per_cue_size=getattr(args, "_fit_cue_size", None))
+ # which size each OUTPUT cue belongs to, after splits have renumbered them
+ args._fit_sizes_out = stats.pop("cue_sizes", None)
report_layout(stats)
caption_stats.clear()
caption_stats.update(stats)
+ caption_stats.update(fit_stats)
return out, any(v for k, v in stats.items() if k != "wrap")
# --srt is repeatable since 1.16 (one per language, each with an optional `:lang` suffix).
diff --git a/scripts/cut.py b/scripts/cut.py
index ca70fcd..e360b4f 100755
--- a/scripts/cut.py
+++ b/scripts/cut.py
@@ -25,11 +25,13 @@ Examples:
python3 cut.py talk.mp4 --start 1:00 --end 2:00 -o part.wav # audio extraction
"""
import argparse
+import json
import os
import sys
import tempfile
from typing import List, Tuple
+from _common import (beat_grid, snap_points, decode_pcm_mono, rms_envelope, BEAT_MIN_CONFIDENCE)
from _common import video_args, STATE, add_common, apply_common, audio_codec_for, emit, aac_args, cfr_args, default_output, die, ffmpeg_base, info, is_audio_output, time_arg, probe, run, X264_PRESETS, keyframes_near, MissingFpsError, concat_list_line, refuse_output_is_input, fmt_secs
# outputs whose re-encode dropped a subtitle/data stream (reported as dropped_non_av_streams)
@@ -148,6 +150,123 @@ def cut_one(src: str, start: float, end: float, dst: str, reencode: bool, crf: i
return reencode
+BEAT_RATE = 22050 # the decode rate the onset pass uses, matching scenes.py --beats
+
+
+def _grid_from_source(path: str, min_confidence: float) -> "dict":
+ """The beat grid of `path`: a scenes.py --json document if that is what it is, otherwise a
+ media file to measure. Reading a document is how a caller avoids a second decode."""
+ try:
+ with open(path, "r", encoding="utf-8") as fh:
+ doc = json.load(fh)
+ except (OSError, ValueError):
+ doc = None
+ if isinstance(doc, dict) and doc.get("beat_grid"):
+ grid = dict(doc["beat_grid"])
+ grid["beats"] = doc.get("beats") or []
+ # A scenes.py document carries the supported subset since 1.17; one written by an older
+ # build does not, and a grid whose supported points are unknown is not one this tool may
+ # move a cut onto -- an unknown subset is not an empty one, but it is not a measurement
+ # either, so it is refused rather than silently treated as "all of them".
+ grid["supported_beats"] = doc.get("beat_grid", {}).get("supported_beats")
+ if grid["supported_beats"] is None:
+ grid["supported_beats"] = doc.get("supported_beats")
+ try:
+ tempo = grid.get("tempo_bpm")
+ grid["tempo_bpm"] = float(tempo) if tempo is not None else None
+ grid["confidence"] = float(grid.get("confidence") or 0.0)
+ except (TypeError, ValueError):
+ die(f"--snap-source {path}: beat_grid.tempo_bpm and .confidence must be numbers "
+ "(regenerate it with `scenes.py MUSIC --beats --json`)", kind="input")
+ if grid["beats"] and grid["tempo_bpm"] is None:
+ die(f"--snap-source {path}: this document lists beats but no tempo_bpm, so no grid "
+ "was actually measured in it. Regenerate it with "
+ "`scenes.py MUSIC --beats --json`.", kind="input")
+ grid["usable"] = grid["confidence"] >= min_confidence
+ return grid
+ if isinstance(doc, dict):
+ die(f"--snap-source {path}: this JSON has no beat_grid -- produce one with "
+ "`scenes.py MUSIC --beats --json`", kind="input")
+ samples = decode_pcm_mono(path, BEAT_RATE, check=False)
+ env = rms_envelope(samples, max(1, int(round(BEAT_RATE * 0.01))))
+ return beat_grid(env, 0.01, min_confidence=min_confidence)
+
+
+def snap_segments(args, segments, meta, total):
+ """Move every in/out point to the nearest measured beat. Returns (result dict, segments).
+
+ A cut point may move to a measured, onset-supported grid point and may not appear from one:
+ the number of segments is unchanged, and nothing is ever proposed. The keyframe/tolerance
+ decision downstream then runs on the snapped values, which is the right order -- whether a cut
+ can be lossless depends on where it actually lands.
+ """
+ source = args.snap_source or args.input
+ if not args.snap_source and not meta.get("audio"):
+ die("--snap beats needs audio to measure a beat in; this file has none. Cut without it "
+ "(--snap none), or pass --snap-source with the music bed.", kind="input")
+ # Compare the PARSED segments against the whole file, not the raw --start string: "0:00",
+ # "0.0" and "00:00:00" are all a zero start that a string comparison lets through, and the
+ # run would then snap the implicit end point and silently shorten a whole-file copy.
+ whole_file = (len(segments) == 1 and abs(segments[0][0]) < 1e-6
+ and (not total or abs(segments[0][1] - total) < 1e-6))
+ if whole_file:
+ die("--snap beats has no in or out point to move: this run copies the whole file. Give "
+ "--start/--end (or --segments), or drop --snap.", kind="input")
+ # A floor of zero would make the confidence check vacuous -- a grid measured from noise scores
+ # above 0.0 and would pass -- and the whole point of the flag is that a cut only moves onto a
+ # pulse somebody can hear. The number is a floor on belief, so it must be a positive one.
+ if args.min_confidence <= 0:
+ die("--min-confidence must be greater than 0: at 0 every grid is 'reliable', including "
+ "one measured from noise, which is exactly what --snap beats must not cut to. Use "
+ "--snap none if you do not want the points moved at all.", kind="input")
+ grid = _grid_from_source(source, args.min_confidence)
+ confidence = float(grid.get("confidence") or 0.0)
+ tempo = grid.get("tempo_bpm")
+ if confidence < args.min_confidence or not grid.get("beats"):
+ die(f"no reliable beat grid in this audio (confidence {confidence:.2f}, needs "
+ f"{args.min_confidence:.2f}): cutting to invented beats would move your in/out points "
+ "to times nothing in the audio supports. Re-run with --snap none, or pass "
+ "--snap-source from a music bed.", kind="input")
+ # THE grid a cut may move onto is the onset-supported subset, never the full regular grid.
+ # beat_grid() reports a regular grid over the whole duration by design -- a grid has to be
+ # regular -- so it runs on through a passage with no music in it. Snapping to one of those
+ # points moves a cut to a time nothing in the audio marks, which is the fabrication this
+ # release forbids and which this tool's own refusal text promises it does not do.
+ supported = grid.get("supported_beats")
+ if supported is None:
+ die(f"--snap-source {source}: this document does not say which grid points a measured "
+ "onset supports, so there is no way to tell a beat from a gap in it. Regenerate it "
+ "with `scenes.py MUSIC --beats --json`.", kind="input")
+ if not supported:
+ die(f"no measured onset supports any point of this beat grid (confidence "
+ f"{confidence:.2f}): the grid is regular but nothing in the audio marks it, so every "
+ "move would be to an invented time. Re-run with --snap none, or pass --snap-source "
+ "from a music bed.", kind="input")
+ points = [t for seg in segments for t in seg]
+ moved = snap_points(points, supported, args.snap_tolerance)
+ out_segments = []
+ for i in range(0, len(moved), 2):
+ s, e = moved[i]["to"], moved[i + 1]["to"]
+ if e <= s: # a snap that would collapse the segment is not applied to it
+ s, e = moved[i]["from"], moved[i + 1]["from"]
+ moved[i].update({"to": s, "delta": 0.0, "snapped": False, "beat_index": None})
+ moved[i + 1].update({"to": e, "delta": 0.0, "snapped": False, "beat_index": None})
+ out_segments.append((s, e))
+ snapped = sum(1 for m in moved if m["snapped"])
+ for m in moved:
+ if m["snapped"]:
+ info(f"--snap beats: {m['from']:.3f}s -> {m['to']:.3f}s ({m['delta'] * 1000:+.0f} ms)")
+ info(f"--snap beats: {tempo:.1f} BPM, confidence {confidence:.2f}; {snapped} of {len(moved)} "
+ f"point(s) moved, within {args.snap_tolerance:.3f}s, onto {len(supported)} of "
+ f"{len(grid['beats'])} grid point(s) a measured onset supports")
+ return ({"mode": "beats", "tolerance": args.snap_tolerance, "confidence": confidence,
+ "tempo_bpm": tempo, "grid": "supported", "grid_points": len(supported),
+ "moved": [dict(m) for m in moved], "snapped": snapped,
+ "unchanged": len(moved) - snapped,
+ "source": "measured" if not args.snap_source else args.snap_source},
+ out_segments)
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("input")
@@ -159,6 +278,17 @@ def main() -> int:
ap.add_argument("--segments", help="comma separated START-END list, e.g. '0:05-0:12,1:00-1:20' (joined in order)")
ap.add_argument("--accurate", action="store_true", help="always re-encode for frame-accurate (video) / sample-accurate (audio) cuts (default: lossless -c copy, re-encoding only when the keyframe snap exceeds --tolerance)")
ap.add_argument("--tolerance", type=float, default=0.5, help="max seconds a lossless cut may deviate before re-encoding kicks in (default 0.5, -1 = never)")
+ snap = ap.add_argument_group("beat snapping")
+ snap.add_argument("--snap", choices=["none", "beats"], default="none",
+ help="move each in/out point to the nearest measured beat (default none)")
+ snap.add_argument("--snap-tolerance", type=float, default=0.12,
+ help="most seconds a point may move with --snap beats (default 0.12, about a "
+ "quarter of a beat at 120 BPM)")
+ snap.add_argument("--snap-source", metavar="FILE",
+ help="take the beat grid from this scenes.py --beats --json document (or from "
+ "this media file) instead of measuring the input again")
+ snap.add_argument("--min-confidence", type=float, default=BEAT_MIN_CONFIDENCE,
+ help=f"refuse to snap below this measured beat confidence (default {BEAT_MIN_CONFIDENCE})")
ap.add_argument("--crf", type=int, default=18, help="x264 CRF when re-encoding (default 18)")
ap.add_argument("--preset", default="medium", choices=X264_PRESETS, help="x264 preset when re-encoding")
add_common(ap)
@@ -196,6 +326,10 @@ def main() -> int:
die("end must be after start")
segments = [(start, end)]
+ snap_result = None
+ if args.snap == "beats":
+ snap_result, segments = snap_segments(args, segments, meta, total)
+
for s, e in segments:
if total and s >= total:
die(f"segment start {s:.3f}s is beyond the media duration {total:.3f}s")
@@ -249,7 +383,8 @@ def main() -> int:
# the trade the caller can offer instead of a re-encode (eval e02: "without losing quality")
lossless_alternative=(f"--start {min(NEAREST_KEYFRAMES, key=lambda k: abs(k - segments[0][0])):.3f} lands on a keyframe: "
f"stream copy with no re-encode, {abs(min(NEAREST_KEYFRAMES, key=lambda k: abs(k - segments[0][0])) - segments[0][0]):.2f}s off the requested start")
- if mode == "hybrid" and NEAREST_KEYFRAMES and len(segments) == 1 else None)
+ if mode == "hybrid" and NEAREST_KEYFRAMES and len(segments) == 1 else None,
+ snap=snap_result)
return 0
diff --git a/scripts/render.py b/scripts/render.py
index b2f3926..92fbce3 100755
--- a/scripts/render.py
+++ b/scripts/render.py
@@ -52,16 +52,35 @@ Examples:
"""
import argparse
import difflib
+import hashlib
import re
import json
import os
+import shutil
import sys
+import time
from pathlib import Path
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional, Sequence
from export import PRESETS, PLATFORM_OF
from _platforms import PLATFORMS, caption_defaults, resolve as resolve_platform
-from _common import STATE, add_common, apply_common, child_args, die, emit, info, probe, run_tool, place_output, refuse_output_is_input, fingerprint, PLAN_VERSION
+from _common import STATE, add_common, apply_common, child_args, die, emit, info, probe, run_tool, place_output, refuse_output_is_input, fingerprint, PLAN_VERSION, ffmpeg_version
+import subprocess
+from _contract import CONTRACT_VERSION
+from batch import file_key
+
+
+def _skill_version() -> str:
+ """The shipped version, from package.json -- in the cache key so a stage whose implementation
+ changed cannot serve back an artifact the old one wrote."""
+ try:
+ return str(json.loads((Path(__file__).resolve().parent.parent / "package.json")
+ .read_text(encoding="utf-8")).get("version") or "?")
+ except (OSError, ValueError):
+ return "?"
+
+
+SKILL_VERSION = _skill_version()
HERE = Path(__file__).resolve().parent
TEMPLATE_DIR = HERE.parent / "templates"
@@ -97,8 +116,10 @@ TEMPLATE = {
OBJECT_KEYS: Dict[str, frozenset] = {
"project": frozenset({"output", "frame", "clips", "transition", "silence", "brand", "captions",
"graphics", "overlays", "audio", "loudness", "fit", "export", "check", "chapters",
- "audiogram", "template"}),
- "clips[]": frozenset({"src", "in", "out", "speed"}),
+ "audiogram", "template", "snap"}),
+ "clips[]": frozenset({"src", "in", "out", "speed", "snap"}),
+ # 1.17: beat snapping, forwarded to cut.py for any clip that has in/out
+ "snap": frozenset({"to", "tolerance", "min_confidence", "source"}),
"frame": frozenset({"aspect", "width", "height", "fps", "fit"}),
"transition": frozenset({"type", "duration"}),
"silence": frozenset({"threshold", "min_silence", "margin"}),
@@ -359,7 +380,7 @@ def check_keys(obj: Any, schema: str, label: str) -> None:
def validate_project(proj: Dict[str, Any]) -> None:
check_keys(proj, "project", "project")
- for name in ("frame", "transition", "silence", "audiogram", "captions", "audio", "loudness", "fit", "export", "check"):
+ for name in ("frame", "transition", "silence", "audiogram", "captions", "audio", "loudness", "fit", "export", "check", "snap"):
check_keys(proj.get(name), name, name)
check_keys((proj.get("audio") or {}).get("stems"), "audio.stems", "audio.stems")
if isinstance(proj.get("chapters"), list):
@@ -374,10 +395,55 @@ def validate_project(proj: Dict[str, Any]) -> None:
if isinstance(items, list):
for i, item in enumerate(items):
check_keys(item, f"{name}[]", f"{name}[{i}]")
+ if name == "clips" and isinstance(item, dict) and item.get("snap") is not None:
+ check_keys(item["snap"], "snap", f"clips[{i}].snap")
+
+
+_LAST_DOC: Dict[str, Any] = {} # the JSON document the most recent sh() child printed
+
+
+def _refuse_uncached_earlier_stage(stage: str) -> None:
+ """--from STAGE promises the earlier stages come from the cache. When one does not, say so
+ rather than quietly re-encoding the thing the caller asked to skip."""
+ target = CACHE.get("from")
+ if not target or stage not in STAGE_ORDER or target not in STAGE_ORDER:
+ return
+ if STAGE_ORDER.index(stage) < STAGE_ORDER.index(target):
+ die(f"--from {target}: the {stage} stage is not in {CACHE['dir']} for this project and "
+ "these inputs, so there is nothing to start from. Render once without --from to fill "
+ "the cache (note that a different ffmpeg build or skill version never reuses one).",
+ kind="input")
+
+def sh(script: str, *argv: Any, extra: List[str] = None, stage: str = None) -> str:
+ """Run a sibling script, forwarding --fast / --dry-run, returning its printed output path.
-def sh(script: str, *argv: Any, extra: List[str] = None) -> str:
- """Run a sibling script, forwarding --fast / --dry-run, returning its printed output path."""
+ With --cache and a named `stage`, an identical stage that ran before is served from the
+ cache instead of re-encoded. `stages_done` is unchanged either way: a cached stage is still
+ a stage that happened.
+ """
+ full = [str(a) for a in argv] + (extra or [])
+ dest = full[full.index("-o") + 1] if "-o" in full[:-1] else None
+ key = None
+ if stage and CACHE.get("dir") and dest:
+ inputs = [a for a in full if os.path.exists(a) and a != dest]
+ # The destination is where this stage's answer goes, not part of the question: hashing it
+ # would make a second run with the first run's output already on disk miss every time.
+ key_args = ["<out>" if a == dest else a for a in full]
+ key = cache_key(stage, script, key_args, inputs, dest)
+ if cache_lookup(stage, key, dest):
+ CACHE["hits"].append(stage)
+ info(f"→ {script} {stage}: served from --cache")
+ # No child ran, so there is no document: say so rather than leaving the PREVIOUS
+ # child's document standing, which a caller reading _LAST_DOC would misattribute.
+ _LAST_DOC.clear()
+ _LAST_DOC["cached"] = True
+ return dest
+ CACHE["misses"].append(stage)
+ _refuse_uncached_earlier_stage(stage)
+ elif stage and CACHE.get("dir"):
+ CACHE["misses"].append(stage)
+ started = time.time()
cmd = [str(HERE / script)] + [str(a) for a in argv] + (extra or []) + child_args() + ["--json"]
info("→ " + " ".join(os.path.basename(c) if i < 1 else c for i, c in enumerate(cmd[:-1])))
proc = run_tool(cmd)
@@ -397,7 +463,136 @@ def sh(script: str, *argv: Any, extra: List[str] = None) -> str:
extra_fields = {"hint": err["hint"]} if err.get("hint") else {}
die(f"{script} failed: {err.get('message') or (proc.stderr.strip().splitlines() or ['?'])[-1][:300]}",
code=int(doc.get("exit_code") or 1), kind=err.get("kind") or "input", stage=script, **extra_fields)
- return str(doc.get("output") or "")
+ _LAST_DOC.clear()
+ _LAST_DOC.update(doc if isinstance(doc, dict) else {})
+ out_path = str(doc.get("output") or "")
+ if key:
+ cache_store(stage, key, out_path or (dest or ""), time.time() - started)
+ return out_path
+
+
+# ------------------------------------------------------------------ the stage cache (1.17)
+#
+# Opt-in only: --cache DIR. There is no default directory -- a cache that appears on someone's
+# disk without being asked for is a surprise, and this tool's posture is that a plan leaves
+# nothing behind.
+
+STAGE_ORDER = ("clips", "audiogram", "join", "silence", "fit", "captions", "graphics",
+ "overlays", "audio", "loudness", "export", "chapters")
+
+def _fresh_cache() -> "Dict[str, Any]":
+ return {"dir": None, "ffmpeg": None, "hits": [], "misses": [], "saved_seconds": 0.0,
+ "entries": 0, "would_hit": [], "from": None}
+
+
+# Module-level so sh() can reach it without threading a parameter through every stage. main()
+# resets it on entry, so two renders in one process (a test session, an embedding caller) do not
+# inherit each other's hit/miss lists.
+CACHE: Dict[str, Any] = _fresh_cache()
+
+
+def _content_hash(path: str) -> str:
+ """The same cheap content fingerprint batch.py caches on: name, size, mtime, first and last
+ MB. Reused rather than reinvented so one file has one identity across the skill."""
+ try:
+ return file_key(Path(path))
+ except (OSError, ValueError):
+ return "missing"
+
+
+def ffmpeg_banner() -> str:
+ """The whole `ffprobe -version` first line, not just major.minor.
+
+ Two 7.1.x builds with different libx264 produce different bytes from the same command, and
+ the cache exists to hand back bytes. major.minor cannot tell them apart, so the banner --
+ which carries the build string and the configuration's version suffix -- is what goes in the
+ key. Unreadable falls back to the parsed pair, which still separates the major releases.
+ """
+ try:
+ out = subprocess.run(["ffprobe", "-version"], stdout=subprocess.PIPE,
+ stderr=subprocess.DEVNULL, text=True, timeout=20).stdout
+ first = (out or "").strip().splitlines()
+ if first:
+ return first[0].strip()
+ except (OSError, subprocess.SubprocessError):
+ pass
+ return ".".join(str(n) for n in ffmpeg_version())
+
+
+def cache_key(stage: str, script: str, argv: "Sequence[Any]", inputs: "Sequence[str]",
+ dest: "Optional[str]" = None) -> str:
+ """sha1 of a canonical description of exactly what this stage is about to do.
+
+ The ffmpeg build banner and the skill version are IN the key, deliberately: a different build
+ simply misses rather than being asked to trust an artifact it did not write, and a stage whose
+ implementation changed must not serve an old one back.
+
+ `child_args()` is in the key too, and that is not a detail. render.py appends it to every
+ stage command AFTER the arguments the stage itself built, and it carries `--fast` -- which
+ rewrites the child's preset to veryfast. Without it in the key, `render --cache C --fast`
+ stored a draft and the next `render --cache C` served that draft back as the delivery, with
+ `cache.hits` presenting it as a legitimate reuse.
+
+ The output's extension is in the key as well (#15): the artifact is stored as `<key><ext>`
+ while the sidecar is `<key>.json`, so two runs differing only in container would otherwise
+ share one sidecar and invalidate each other on every run.
+ """
+ payload = {
+ "stage": stage, "tool": script,
+ "args": [_content_hash(str(a)) if os.path.exists(str(a)) else str(a) for a in argv],
+ "inputs": [{"hash": _content_hash(p)} for p in inputs],
+ "child": [a for a in child_args() if a != "--dry-run"],
+ "codec": STATE.codec, "ext": Path(dest).suffix if dest else None,
+ "ffmpeg": CACHE.get("ffmpeg"), "skill": SKILL_VERSION, "contract": CONTRACT_VERSION,
+ }
+ return hashlib.sha1(json.dumps(payload, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
+
+
+def cache_lookup(stage: str, key: str, dest: str) -> bool:
+ """Put the cached artifact for `key` at `dest` and return True, or return False on any
+ mismatch -- silently, because a miss is not an error, it is just work to do."""
+ cdir = CACHE.get("dir")
+ if not cdir:
+ return False
+ side = Path(cdir) / f"{key}.json"
+ art = Path(cdir) / f"{key}{Path(dest).suffix or '.bin'}"
+ if not (side.exists() and art.exists()):
+ return False
+ try:
+ meta = json.loads(side.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return False
+ if int(meta.get("size") or -1) != art.stat().st_size:
+ return False
+ if STATE.dry_run:
+ CACHE["would_hit"].append(stage)
+ return True
+ Path(dest).parent.mkdir(parents=True, exist_ok=True)
+ try:
+ if Path(dest).exists():
+ Path(dest).unlink()
+ os.link(art, dest) # a hardlink where the filesystem allows it ...
+ except OSError:
+ shutil.copy2(art, dest) # ... and a copy where it does not. Never a move: the cache
+ CACHE["saved_seconds"] += float(meta.get("seconds") or 0.0)
+ return True
+
+
+def cache_store(stage: str, key: str, produced: str, seconds: float) -> None:
+ """Keep `produced` for the next run. Never under --dry-run: a plan writes nothing."""
+ cdir = CACHE.get("dir")
+ if not cdir or STATE.dry_run or not produced or not os.path.exists(produced):
+ return
+ art = Path(cdir) / f"{key}{Path(produced).suffix or '.bin'}"
+ try:
+ shutil.copy2(produced, art)
+ Path(cdir, f"{key}.json").write_text(json.dumps({
+ "stage": stage, "ffmpeg": CACHE.get("ffmpeg"), "skill": SKILL_VERSION,
+ "created": time.time(), "size": art.stat().st_size, "seconds": round(seconds, 2),
+ "artifact": art.name}, indent=2), encoding="utf-8")
+ CACHE["entries"] += 1
+ except OSError as exc:
+ info(f"cache: could not store the {stage} artifact ({exc}); the run is unaffected")
def execute_plan(plan: Dict[str, Any], path: str) -> int:
@@ -501,6 +696,14 @@ def main() -> int:
ap.add_argument("--init", metavar="FILE", help="write a starter project file and exit")
ap.add_argument("--work", help="work directory for intermediates (default: <output>_work)")
ap.add_argument("--keep", action="store_true", help="keep intermediates (default: kept only when --work is given)")
+ ap.add_argument("--cache", metavar="DIR",
+ help="reuse the artifacts of identical stages from a previous run. Opt-in: "
+ "there is no default directory. The ffmpeg version and the skill version "
+ "are part of every key, so a cache never crosses either.")
+ ap.add_argument("--from", dest="from_stage", metavar="STAGE",
+ choices=list(STAGE_ORDER),
+ help="start at this stage, taking every earlier one from --cache; refuses if "
+ "one of them is not there")
ap.add_argument("--stop-after", choices=["clips", "join", "silence", "fit", "captions", "graphics", "overlays", "audio", "loudness", "export"], help="stop after this stage (for iterating)")
tpl = ap.add_argument_group("delivery templates (one command per destination)")
tpl.add_argument("--template", metavar="NAME", help="render INPUT with a shipped template: " + ", ".join(template_names())
@@ -601,7 +804,6 @@ def main() -> int:
# a failed or dry run used to leave <output>_work_<pid>/ behind (sweep F15): the
# auto-named directory is ours alone, so remove it on every exit path
import atexit
- import shutil
atexit.register(lambda: shutil.rmtree(work, ignore_errors=True))
# Intermediates keep the delivery's media kind: a .mp4 project is unchanged (every stage file
# is still clipNN.mp4 / fit.mp4 / loudnorm.mp4), while an audio-only delivery (the podcast
@@ -619,6 +821,35 @@ def main() -> int:
platform_args: List[str] = ["--platform", dest] if dest in PLATFORMS and PLATFORMS[dest].get("frame") else []
stages_done: List[str] = []
+ CACHE.clear()
+ CACHE.update(_fresh_cache())
+ if args.cache:
+ cdir = Path(args.cache)
+ try:
+ cdir.mkdir(parents=True, exist_ok=True)
+ probe_file = cdir / ".writable"
+ probe_file.write_text("", encoding="utf-8")
+ probe_file.unlink()
+ except OSError as exc:
+ die(f"--cache {args.cache}: not a writable directory ({exc})", kind="output")
+ CACHE["dir"] = str(cdir)
+ CACHE["ffmpeg"] = ffmpeg_banner()
+ CACHE["entries"] = len(list(cdir.glob("*.json")))
+ CACHE["from"] = args.from_stage
+ if args.from_stage and not args.cache:
+ die(f"--from {args.from_stage} needs --cache DIR with a previous run's stages: without a "
+ "cache there is no earlier artifact to start from, so every stage would run anyway.",
+ kind="input")
+
+ # A project may ask for its clip boundaries to land on the music's beat. The measurement and
+ # the refusal both live in cut.py -- render forwards the request and reports what came back,
+ # so a project that does not name "snap" builds the command line 1.16 built.
+ snap_spec = proj.get("snap") or {}
+ snap_reports: List[Dict[str, Any]] = []
+ if snap_spec and str(snap_spec.get("to") or "") not in ("", "none", "beats"):
+ die(f'snap.to: only "beats" (or "none") is a beat grid this skill can measure, got '
+ f'{snap_spec.get("to")!r}', kind="input")
+
# ---- clips
parts: List[str] = []
for i, c in enumerate(clips):
@@ -635,7 +866,26 @@ def main() -> int:
argv += ["--start", c["in"]]
if c.get("out") is not None:
argv += ["--end", c["out"]]
- sh("cut.py", *argv)
+ clip_snap = dict(snap_spec)
+ clip_snap.update(c.get("snap") or {})
+ if str(clip_snap.get("to") or "none") == "beats":
+ argv += ["--snap", "beats"]
+ if clip_snap.get("tolerance") is not None:
+ argv += ["--snap-tolerance", str(clip_snap["tolerance"])]
+ if clip_snap.get("min_confidence") is not None:
+ argv += ["--min-confidence", str(clip_snap["min_confidence"])]
+ if clip_snap.get("source"):
+ argv += ["--snap-source", rel(clip_snap["source"])]
+ sh("cut.py", *argv, stage="clips")
+ if _LAST_DOC.get("snap"):
+ snap_reports.append({"clip": i, **_LAST_DOC["snap"]})
+ elif _LAST_DOC.get("cached") and str(clip_snap.get("to") or "none") == "beats":
+ # The cut is the one the cache holds, so it WAS snapped -- the moves are simply
+ # not re-measured. Saying `snap: null` here would report the opposite.
+ snap_reports.append({"clip": i, "mode": "beats", "source": "cache",
+ "note": "this clip came from --cache; it was snapped when "
+ "it was first rendered and the moves are in that "
+ "run's result"})
else:
part = src
if c.get("speed"):
@@ -645,7 +895,7 @@ def main() -> int:
if abs(spd - 1.0) > 1e-6: # speed 1.0 used to cost a full re-encode for nothing
dur = (probe(part).get("duration") or 0.0) if not STATE.dry_run else 10.0
fitted = str(work / f"clip{i:02d}_speed{mid}")
- sh("fit.py", part, "--duration", f"{dur / spd:.3f}", "-o", fitted)
+ sh("fit.py", part, "--duration", f"{dur / spd:.3f}", "-o", fitted, stage="clips")
part = fitted
parts.append(part)
stages_done.append("clips")
@@ -667,7 +917,7 @@ def main() -> int:
if ag.get(key) is not None:
argv += [flag, str(ag[key])]
argv += brand_args
- sh("waveform.py", *argv)
+ sh("waveform.py", *argv, stage="audiogram")
current = nxt
parts = [current]
stages_done.append("audiogram")
@@ -686,7 +936,7 @@ def main() -> int:
argv += ["--height", str(frame["height"])]
if frame.get("fps"):
argv += ["--fps", str(frame["fps"])]
- sh("join.py", *argv)
+ sh("join.py", *argv, stage="join")
stages_done.append("join")
if args.stop_after == "join":
emit(current, stages=stages_done)
@@ -700,7 +950,7 @@ def main() -> int:
for k, flag in (("threshold", "--threshold"), ("min_silence", "--min-silence"), ("margin", "--margin")):
if sil.get(k) is not None:
argv += [flag, str(sil[k])]
- sh("silence.py", *argv)
+ sh("silence.py", *argv, stage="silence")
current = nxt
stages_done.append("silence")
if args.stop_after == "silence":
@@ -725,7 +975,7 @@ def main() -> int:
for k, flag in (("duration", "--duration"), ("method", "--method"), ("aspect", "--aspect"), ("fit", "--fit"), ("width", "--width"), ("height", "--height"), ("fps", "--fps"), ("smooth", "--smooth")):
if fit.get(k) is not None:
argv += [flag, str(fit[k])]
- sh("fit.py", *argv)
+ sh("fit.py", *argv, stage="fit")
current = nxt
stages_done.append("fit")
if args.stop_after == "fit":
@@ -751,7 +1001,7 @@ def main() -> int:
for k, flag in (("karaoke", "--karaoke"), ("bold", "--bold"), ("box", "--box")):
if cap.get(k):
argv.append(flag)
- sh("caption.py", *(argv + brand_args + platform_args))
+ sh("caption.py", *(argv + brand_args + platform_args), stage="captions")
current = nxt
stages_done.append("captions")
if args.stop_after == "captions":
@@ -769,7 +1019,7 @@ def main() -> int:
if g.get(k) is not None:
argv += [flag, str(g[k])]
# the entry's own "platform" is the more specific statement than the template's destination
- sh("graphics.py", *(argv + brand_args + ([] if g.get("platform") else platform_args)))
+ sh("graphics.py", *(argv + brand_args + ([] if g.get("platform") else platform_args)), stage="graphics")
current = nxt
if "graphics" not in stages_done:
stages_done.append("graphics")
@@ -796,7 +1046,7 @@ def main() -> int:
argv.append("--box")
# review 12: the overlay stage was the one stage that never heard which destination this
# is, so a template's top-left logo landed 24 px in -- under TikTok's own status bar.
- sh("overlay.py", *(argv + brand_args + ([] if ov.get("platform") else platform_args)))
+ sh("overlay.py", *(argv + brand_args + ([] if ov.get("platform") else platform_args)), stage="overlays")
current = nxt
if "overlays" not in stages_done:
stages_done.append("overlays")
@@ -834,7 +1084,7 @@ def main() -> int:
for k, flag in (("denoise", "--denoise"), ("duck", "--duck"), ("music_loop", "--music-loop"), ("stereo", "--stereo"), ("mono", "--mono"), ("downmix", "--downmix")):
if au.get(k):
argv.append(flag)
- sh("audio.py", *argv)
+ sh("audio.py", *argv, stage="audio")
current = nxt
stages_done.append("audio")
if args.stop_after == "audio":
@@ -850,7 +1100,7 @@ def main() -> int:
argv += ["-I", str(ld["lufs"])]
if ld.get("tp") is not None:
argv += ["--tp", str(ld["tp"])]
- sh("loudness.py", *argv)
+ sh("loudness.py", *argv, stage="loudness")
current = nxt
stages_done.append("loudness")
if args.stop_after == "loudness":
@@ -874,7 +1124,7 @@ def main() -> int:
info(f"export: --normalize on by default for the {ex['preset']} preset (set \"normalize\": false to skip)")
if normalize:
argv += ["--normalize"] # one export that meets the platform's loudness (export.py --normalize)
- sh("export.py", *argv)
+ sh("export.py", *argv, stage="export")
stages_done.append("export")
else:
if not STATE.dry_run:
@@ -895,7 +1145,7 @@ def main() -> int:
else:
chapter_file = rel(ch)
tagged = str(work / ("chapters" + Path(output).suffix))
- sh("metadata.py", output, "--chapters", chapter_file, "-o", tagged)
+ sh("metadata.py", output, "--chapters", chapter_file, "-o", tagged, stage="chapters")
if not STATE.dry_run:
place_output(tagged, output)
stages_done.append("chapters")
@@ -926,7 +1176,6 @@ def main() -> int:
# default name carries this process's PID, nothing else will ever reuse -- and so
# implicitly clean up -- a leftover dry-run directory the way a same-named real run used
# to before the PID suffix was added.
- import shutil
shutil.rmtree(work, ignore_errors=True)
if exit_code:
# The deliverable is written and verified, but it does not meet the requested platform
@@ -936,7 +1185,23 @@ def main() -> int:
kind="verification", output=output, dry_run=STATE.dry_run, stages=stages_done, check=check_result,
probe=probe(output, role="output"))
info(f"rendered {output} via {' → '.join(stages_done)}")
- emit(output, stages=stages_done, check=check_result,
+ cache_report = None
+ if args.cache:
+ cache_report = {"dir": CACHE["dir"], "ffmpeg": CACHE["ffmpeg"],
+ "hits": CACHE["hits"], "misses": CACHE["misses"],
+ "saved_seconds": round(CACHE["saved_seconds"], 1),
+ "entries": CACHE["entries"]}
+ if STATE.dry_run:
+ cache_report["would_hit"] = CACHE["would_hit"]
+ info(f"cache: {len(CACHE['hits'])} hit(s) ({', '.join(CACHE['hits']) or '-'}), "
+ f"{len(CACHE['misses'])} miss(es) ({', '.join(CACHE['misses']) or '-'})")
+ # One entry per snapped clip, `clip` naming which. A single-clip project keeps the shape a
+ # caller reads today by also carrying the first entry's keys at the top level.
+ snap_report: Optional[Dict[str, Any]] = None
+ if snap_reports:
+ snap_report = dict(snap_reports[0])
+ snap_report["clips"] = snap_reports
+ emit(output, stages=stages_done, check=check_result, snap=snap_report, cache=cache_report,
verification=[{"step": "check", "ok": True, "platform": ck["platform"]}] if check_result else [])
return 0
diff --git a/scripts/scenes.py b/scripts/scenes.py
index f586874..91aff53 100755
--- a/scripts/scenes.py
+++ b/scripts/scenes.py
@@ -20,17 +20,38 @@ Examples:
import argparse
import math
import sys
-from typing import Dict, List, Tuple
+from typing import Dict, List, Optional, Tuple
# `detect_scenes` moved into _common/probe.py in 1.16.0 (see silence.py); the body is unchanged.
-from _common import detect_scenes, STATE, add_common, apply_common, default_font_file, die, emit, escape_filter_path, ffmpeg_base, info, print_json, probe, run, decode_pcm_mono, rms_envelope
+from _common import (detect_scenes, STATE, add_common, apply_common, beat_grid, default_font_file, die, emit,
+ escape_filter_path, ffmpeg_base, info, print_json, probe, run, decode_pcm_mono,
+ rms_envelope, BEAT_MIN_CONFIDENCE)
-def audio_envelope(path: str, step_s: float) -> List[float]:
- """RMS level per step_s window at 8 kHz, absolute (a loud scene scores higher); [] when the
- audio cannot be decoded (the cut scoring then runs on the picture alone)."""
- return rms_envelope(decode_pcm_mono(path, 8000, check=False), int(8000 * step_s))
+def audio_envelope(path: str, step_s: float, *, rate: int = 8000,
+ samples: "Optional[List[float]]" = None) -> List[float]:
+ """RMS level per step_s window, absolute (a loud scene scores higher); [] when the audio
+ cannot be decoded (the cut scoring then runs on the picture alone).
+
+ `samples` reuses PCM a caller already decoded rather than decoding the same file twice --
+ --beats needs a 10 ms envelope and the scene scoring a 0.5 s one, and 0.5 s is an integer
+ multiple of 10 ms, so both come from one pass.
+ """
+ if samples is None:
+ samples = decode_pcm_mono(path, rate, check=False)
+ return rms_envelope(samples, int(rate * step_s))
+
+
+def parse_beat_range(text: str) -> "tuple":
+ """`--beat-range 60-200` as (lo, hi) BPM."""
+ try:
+ lo, hi = (float(p) for p in str(text).replace(" ", "").split("-", 1))
+ except ValueError:
+ die(f"--beat-range: expected LO-HI in BPM (e.g. 60-200), got {text!r}", kind="input")
+ if not (0 < lo < hi):
+ die(f"--beat-range {text}: LO must be above 0 and below HI", kind="input")
+ return (lo, hi)
def main() -> int:
@@ -45,6 +66,16 @@ def main() -> int:
ap.add_argument("--target", type=float, help="with --highlights: total seconds the picks should add up to (trims long scenes)")
ap.add_argument("--max-scene", type=float, default=15.0, help="cap a highlight range at this many seconds (default 15)")
ap.add_argument("--edl", help="write highlight ranges as START-END lines (cut.py --segments format)")
+ ap.add_argument("--beats", action="store_true",
+ help="measure the music's beat grid (tempo, beat times, confidence) and report it; "
+ "a measurement, not a proposal -- no cut is made and no beat is invented")
+ ap.add_argument("--beat-step", type=float, default=0.01,
+ help="envelope resolution in seconds for the onset pass with --beats (default 0.01)")
+ ap.add_argument("--beat-range", default="60-200",
+ help="tempo search range in BPM for --beats (default 60-200)")
+ ap.add_argument("--min-confidence", type=float, default=BEAT_MIN_CONFIDENCE,
+ help="with --beats: below this confidence the grid is still reported, marked "
+ f"usable: false (default {BEAT_MIN_CONFIDENCE})")
ap.add_argument("--sheet", help="write a contact sheet PNG with the first frame of every scene")
ap.add_argument("--no-timecode", action="store_true", help="--sheet without the burnt-in timecode stamp (a way out if drawtext itself is unusable, see doctor)")
add_common(ap)
@@ -54,11 +85,26 @@ def main() -> int:
meta = probe(args.input)
if not meta.get("video"):
die("input has no video stream")
+ # Only parsed when it is going to be used: --beat-range is a --beats flag, and a run that
+ # never asked for beats should not be able to die on one.
+ beat_range = parse_beat_range(args.beat_range) if args.beats else (60.0, 200.0)
+ if args.beats:
+ if not meta.get("audio"):
+ die("--beats needs an audio stream; this file has none", kind="input")
+ if args.beat_step <= 0:
+ die("--beat-step must be greater than 0", kind="input")
dur = meta.get("duration") or 0.0
cuts = detect_scenes(args.input, args.threshold, args.min_scene, dur, args.ratio)
bounds = cuts + [dur]
step_s = 0.5
- env = audio_envelope(args.input, step_s) if meta.get("audio") else []
+ # With --beats the file is decoded once, at the finer rate, and both envelopes come from that
+ # one pass: the 0.5 s scene blocks are an exact multiple of the 10 ms onset blocks.
+ beat_rate = 22050
+ fine_samples = decode_pcm_mono(args.input, beat_rate, check=False) if (args.beats and meta.get("audio")) else None
+ if fine_samples is not None:
+ env = audio_envelope(args.input, step_s, rate=beat_rate, samples=fine_samples)
+ else:
+ env = audio_envelope(args.input, step_s) if meta.get("audio") else []
scenes = []
for i in range(len(bounds) - 1):
@@ -82,6 +128,34 @@ def main() -> int:
result: Dict = {"file": args.input, "duration": round(dur, 3), "scene_count": len(scenes), "scenes": scenes, "audio_peaks": peaks}
info(f"{len(scenes)} scenes, {len(peaks)} audio peaks over {dur:.1f}s")
+ if args.beats:
+ # A beat grid is a measurement of the music's periodicity, not a statement about where a
+ # cut belongs. scenes.py reports what it measured, including a low confidence: reporting a
+ # weak measurement is honest, and only a tool that CHANGES a file refuses to act on one.
+ fine = rms_envelope(fine_samples or [], max(1, int(round(beat_rate * args.beat_step))))
+ grid = beat_grid(fine, args.beat_step, bpm_range=beat_range,
+ min_confidence=args.min_confidence, duration=dur)
+ result["beats"] = grid["beats"]
+ result["beat_grid"] = {
+ # The regular grid AND the subset a measured onset supports. A tool that MOVES
+ # something (cut.py --snap beats) may only use the subset; scenes.py reports both,
+ # because here the regular grid is the measurement being made.
+ "supported_beats": grid["supported_beats"],
+ "tempo_bpm": grid["tempo_bpm"], "interval": grid["interval"],
+ "confidence": grid["confidence"], "phase": grid["phase"],
+ "onsets": len(grid["onsets"]), "supported": grid["supported"],
+ "unsupported": grid["unsupported"], "method": grid["method"],
+ "step_s": grid["step_s"], "range_bpm": grid["range_bpm"], "usable": grid["usable"],
+ }
+ if grid["tempo_bpm"] is None:
+ info(f"--beats: no steady pulse in this audio (confidence {grid['confidence']:.2f}) -- "
+ "speech, ambience or rubato has no tempo to measure")
+ else:
+ info(f"--beats: {grid['tempo_bpm']:.1f} BPM, {len(grid['beats'])} beats, confidence "
+ f"{grid['confidence']:.2f} ({grid['supported']} of {len(grid['beats'])} grid points "
+ f"have a measured onset)"
+ + ("" if grid["usable"] else f" -- below --min-confidence {args.min_confidence}, usable: false"))
+
if args.highlights:
if args.rank_by == "duration":
rank_key = lambda sc: (-sc["duration"], sc["start"])
diff --git a/scripts/silence.py b/scripts/silence.py
index cd0f799..4728fc7 100755
--- a/scripts/silence.py
+++ b/scripts/silence.py
@@ -12,29 +12,168 @@ Examples:
python3 silence.py talk.mp4 --edl keep.txt # also save the kept ranges (START-END per line, cut.py --segments format)
"""
import argparse
+import json
import os
import sys
from typing import List, Tuple
# `detect` moved into _common/probe.py in 1.16.0 so metadata.py --auto-chapters can measure the
# same silences without importing this tool; the body is unchanged and the name still lives here.
+from _common import (filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS,
+ FILLER_PAD, transcribe_words, read_text_or_die)
from _common import detect_silences as detect, STATE, video_args, add_common, apply_common, audio_codec_for, cfr_args, default_output, die, emit, ffmpeg_base, info, is_audio_output, print_json, probe, run, X264_PRESETS, measured_level_dbfs, fmt_secs
+def merge_spans(spans: List[Tuple[float, float]]) -> List[Tuple[float, float]]:
+ """`spans` sorted and coalesced: any pair that touches or overlaps becomes one.
+
+ keep_ranges() walks a single cursor forward, so it needs a removal list in which no span
+ starts before the previous one ended. Silences and filler spans are each merged only among
+ themselves -- and a mumbled "um" is very often quiet enough to sit INSIDE a detected silence --
+ so the union has to be taken before the two lists are handed over as one.
+ """
+ out: List[Tuple[float, float]] = []
+ for s, e in sorted(spans):
+ if out and s <= out[-1][1]:
+ out[-1] = (out[-1][0], max(out[-1][1], e))
+ else:
+ out.append((s, e))
+ return out
+
+
def keep_ranges(silences: List[Tuple[float, float]], duration: float, margin: float, min_keep: float) -> List[Tuple[float, float]]:
keeps: List[Tuple[float, float]] = []
cursor = 0.0
- for s, e in silences:
+ for s, e in sorted(silences):
s_adj = max(cursor, s + margin)
if s_adj - cursor >= min_keep:
keeps.append((cursor, s_adj))
- cursor = min(duration, e - margin) if e != float("inf") else duration
+ # max(cursor, ...) so the walk is monotone. A span nested inside the previous one used to
+ # rewind the cursor and hand back the very stretch that had just been removed: with a
+ # filler word inside a detected silence, adding --filler made the tool remove LESS.
+ # merge_spans() above is the caller-side fix; this keeps the function safe on its own.
+ cursor = max(cursor, min(duration, e - margin) if e != float("inf") else duration)
if duration - cursor >= min_keep:
keeps.append((cursor, duration))
return keeps
+def _word_list(text: "str") -> "list":
+ return [w.strip() for w in str(text or "").replace(",", "\n").splitlines() if w.strip()]
+
+
+def resolve_filler(args, meta):
+ """(the `filler` result block, the spans to remove) for --filler. Refuses before any encode.
+
+ Never without measured word timings: no heuristic fallback, no guess from the filename. The
+ three refusals below are the whole safety story for this flag.
+ """
+ if not args.words and not args.transcribe:
+ die("--filler needs word timings: pass --words transcript.json (a whisper JSON with word "
+ "timestamps) or --transcribe. There is no way to find a filler word without them -- "
+ "cutting the short quiet blips instead would remove real speech.", kind="input")
+ source, engine, raw = None, None, None
+ if args.words:
+ try:
+ raw = json.loads(read_text_or_die(args.words, "--words"))
+ except ValueError as exc:
+ die(f"--words {args.words}: not readable JSON ({exc})", kind="input")
+ source = f"whisper-json:{args.words}"
+ words, had_segments = _words_from_transcript(raw)
+ if not words:
+ if had_segments:
+ die(f"the transcript in {args.words} has segment timings but no word timings; "
+ "--filler removes words, and cutting on segment boundaries would remove whole "
+ "sentences. Re-run whisper with word timestamps (whisper.cpp "
+ "--output-json-full / faster-whisper word_timestamps=True), or use --list to "
+ "see the pauses instead.", kind="input")
+ die(f"no word timings in {args.words}: --filler needs "
+ '{"words": [{"word": ..., "start": ..., "end": ...}]} (or the same inside '
+ '"segments").', kind="input")
+ else:
+ # No pre-check for an installed engine here: transcribe_words() probes for one and
+ # raises the same die_no_engine() refusal when there is none. Two places deciding "is
+ # whisper here" is two places to disagree.
+ # The engine is driven with ITS word-timestamp option (whisper.cpp --output-json-full,
+ # faster-whisper word_timestamps=True, openai-whisper --word_timestamps True). An SRT
+ # cannot answer this question: a cue has a start and an end, a word does not.
+ words, engine = transcribe_words(args.input, args.filler_lang if args.filler_lang != "auto" else None)
+ source = f"whisper:{engine}" if engine else "whisper"
+ if not words:
+ die(f"{engine or 'the local engine'} ran but produced no word-level timings, so there "
+ "is nothing for --filler to cut on. Some builds do not support word timestamps. "
+ "Re-run that engine yourself with them (whisper.cpp --output-json-full / "
+ "faster-whisper word_timestamps=True / openai-whisper --word_timestamps True) and "
+ "pass the result with --words.", kind="input")
+
+ lang = args.filler_lang
+ if lang == "auto":
+ lang = str((raw or {}).get("language") or "").lower()[:2] if isinstance(raw, dict) else ""
+ lang = lang if lang in FILLER_WORDS else "en"
+ listname = "builtin"
+ if args.filler_words:
+ wordlist = set(_word_list(read_text_or_die(args.filler_words, "--filler-words")))
+ listname = args.filler_words
+ else:
+ if lang not in FILLER_WORDS:
+ die(f"--filler-lang {lang}: no built-in filler list for that language. The languages "
+ f"with one are {', '.join(sorted(FILLER_WORDS))}; pass --filler-words FILE with "
+ "your own list for anything else.", kind="input")
+ wordlist = set(FILLER_WORDS[lang])
+ wordlist |= set(_word_list(args.filler_extra))
+ wordlist -= set(_word_list(args.filler_keep))
+
+ spans = filler_spans(words, wordlist, pad=args.filler_pad)
+ removed_words = sorted({s["word"] for s in spans})
+ warnings = []
+ ambiguous = sorted(set(FILLER_AMBIGUOUS.get(lang, ())) & set(
+ t for s in spans for t in s["word"].split()))
+ if ambiguous:
+ warnings.append(
+ f"removed {', '.join(ambiguous)} -- in {lang} these are as often ordinary words as "
+ f"fillers. Keep one with --filler-keep {ambiguous[0]} and re-run if a sentence lost "
+ "its meaning.")
+ if FILLER_DISCOURSE_MARKERS.get(lang):
+ extra_markers = sorted(set(_word_list(args.filler_extra))
+ & set(FILLER_DISCOURSE_MARKERS[lang]))
+ if extra_markers:
+ warnings.append(f"--filler-extra {', '.join(extra_markers)}: a discourse marker, not a "
+ "disfluency -- this will cut real sentences.")
+ block = {
+ "lang": lang, "source": source, "engine": engine,
+ "words": sorted(wordlist), "removed": [dict(s) for s in spans],
+ "removed_count": len(spans),
+ "removed_seconds": round(sum(s["end"] - s["start"] for s in spans), 3),
+ "word_timings": len(words), "list": listname, "removed_words": removed_words,
+ "warnings": warnings,
+ }
+ return block, [(s["start"], s["end"]) for s in spans]
+
+
+def _words_from_transcript(data):
+ """([{word,start,end}], whether the document had segments at all) from a whisper JSON."""
+ raw, had_segments = [], False
+ if isinstance(data, dict):
+ raw = list(data.get("words") or [])
+ segments = data.get("segments") or []
+ had_segments = bool(segments)
+ for seg in segments:
+ raw.extend((seg or {}).get("words") or [])
+ elif isinstance(data, list):
+ raw = list(data)
+ out = []
+ for w in raw:
+ if not isinstance(w, dict):
+ continue
+ try:
+ out.append({"word": str(w.get("word") or w.get("text") or ""),
+ "start": float(w["start"]), "end": float(w["end"])})
+ except (KeyError, TypeError, ValueError):
+ continue
+ return out, had_segments
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("input")
@@ -45,27 +184,90 @@ def main() -> int:
ap.add_argument("--min-keep", type=float, default=0.2, help="drop kept pieces shorter than this (default 0.2)")
ap.add_argument("--list", action="store_true", help="only print silences and the kept ranges")
ap.add_argument("--edl", help="write the kept ranges to this file, one START-END per line")
+ fil = ap.add_argument_group("filler words (1.17)")
+ fil.add_argument("--filler", action="store_true",
+ help="also remove filler words. Needs measured word timings: pass --words or "
+ "--transcribe. There is no heuristic fallback -- a word is cut only where "
+ "a speech engine timed it.")
+ fil.add_argument("--filler-lang", choices=["auto"] + sorted(FILLER_WORDS), default="auto",
+ help="which built-in list to use (default auto: the transcript's language field)")
+ fil.add_argument("--filler-words", metavar="FILE",
+ help="one word per line; replaces the built-in list for this run")
+ fil.add_argument("--filler-extra", metavar="W[,W...]",
+ help="add words to the list. 'like', 'tipo' and 'cio\u00e8' live here rather than in "
+ "the defaults: they are discourse markers, not disfluencies, and cutting "
+ "them cuts real sentences.")
+ fil.add_argument("--filler-keep", metavar="W[,W...]",
+ help="remove words from the built-in list (e.g. --filler-keep \u306a\u3093\u304b)")
+ fil.add_argument("--filler-pad", type=float, default=FILLER_PAD,
+ help=f"seconds trimmed either side of a filler word (default {FILLER_PAD})")
+ fil.add_argument("--words", metavar="FILE",
+ help="a whisper JSON with word-level timings, for --filler")
+ fil.add_argument("--transcribe", action="store_true",
+ help="produce the word timings with a local whisper (never required; the same "
+ "bridge caption.py uses)")
+ fil.add_argument("--filler-list", action="store_true",
+ help="report what --filler would remove and write nothing")
+ fil.add_argument("--max-cuts", type=int, default=400,
+ help="refuse above this many removal ranges: the filter graph grows with them "
+ "(default 400)")
ap.add_argument("--crf", type=int, default=18)
ap.add_argument("--preset", default="medium", choices=X264_PRESETS)
add_common(ap)
args = ap.parse_args()
apply_common(args)
+ if args.filler_pad < 0:
+ die("--filler-pad cannot be negative: a negative pad turns each word's span inside out "
+ "(end before start) and the span is then silently dropped, so nothing is removed",
+ kind="input")
+ if args.max_cuts < 1:
+ die("--max-cuts must be at least 1", kind="input")
+ if args.filler_list and not args.filler:
+ die("--filler-list reports what --filler would remove: pass --filler as well", kind="input")
meta = probe(args.input)
if not meta.get("audio"):
die("input has no audio stream to analyse")
duration = meta.get("duration") or 0.0
silences = detect(args.input, args.threshold, args.min_silence)
- keeps = keep_ranges(silences, duration, args.margin, args.min_keep)
+ filler_info, filler_ranges = resolve_filler(args, meta) if args.filler else (None, [])
+ # One sorted, merged removal list through the graph the tool already has: filler removal IS
+ # time-range removal, so it reuses keep_ranges() and the same aselect/concat chain.
+ removals = merge_spans(list(silences) + list(filler_ranges))
+ keeps = keep_ranges(removals, duration, args.margin, args.min_keep)
kept = sum(e - s for s, e in keeps)
removed = max(0.0, duration - kept)
+ # `removed_seconds` keeps the meaning it has had since this tool existed: the seconds of
+ # SILENCE this run removes. It must not quietly start counting filler time as well, because a
+ # caller that has been reading it since 1.0 asked how much dead air went. The silence-only
+ # figure is the one the same run would have reported without --filler, so it is computed from
+ # the silences alone; everything removed is `removed_seconds_total`.
+ silence_only = removed
+ if args.filler:
+ silence_keeps = keep_ranges(merge_spans(list(silences)), duration, args.margin, args.min_keep)
+ silence_only = max(0.0, duration - sum(e - s for s, e in silence_keeps))
summary = {
"silences": [[round(s, 3), None if e == float("inf") else round(e, 3)] for s, e in silences],
"keep": [[round(s, 3), round(e, 3)] for s, e in keeps],
"input_duration": round(duration, 3),
"kept_duration": round(kept, 3),
- "removed_seconds": round(removed, 3),
+ "removed_seconds": round(silence_only, 3),
}
+ if filler_info is not None:
+ # removed_seconds above is the silence-only figure, unchanged in meaning; the filler share
+ # is reported inside `filler`, and removed_seconds_total is the additive sibling that
+ # covers everything this run took out.
+ summary["filler"] = filler_info
+ summary["removed_seconds_total"] = round(removed, 3)
+ info(f"--filler: {filler_info['removed_count']} filler word(s), "
+ f"{filler_info['removed_seconds']:.2f}s, from {filler_info['word_timings']} word timings "
+ f"({filler_info['lang']}, list {filler_info['list']})")
+ for warning in filler_info.get("warnings") or []:
+ info("warning: " + warning)
+ if len(keeps) > args.max_cuts:
+ die(f"{len(keeps)} keep ranges is above --max-cuts {args.max_cuts}: the filter graph grows "
+ "with every range and a graph this size is slow and fragile. Raise --max-cuts if you "
+ "mean it, or use --min-silence/--filler-pad to merge the short ones.", kind="input")
info(f"{len(silences)} silences, keeping {len(keeps)} ranges: {kept:.2f}s of {duration:.2f}s (removing {removed:.2f}s)")
if not silences and not (STATE.dry_run and not os.path.exists(args.input)):
# Nothing under the threshold is a valid result, not a failure -- but an agent that only
@@ -88,7 +290,7 @@ def main() -> int:
fh.write(f"{s:.3f}-{e:.3f}\n")
info(f"wrote {args.edl}")
- if args.list:
+ if args.list or args.filler_list:
if args.json:
emit(None, **summary)
else:
diff --git a/tests/_fixtures.py b/tests/_fixtures.py
index 7c574e6..d2d1431 100644
--- a/tests/_fixtures.py
+++ b/tests/_fixtures.py
@@ -231,6 +231,43 @@ class MediaFixtures(unittest.TestCase):
"-t", "6", "-c:v", "libx264", "-preset", "veryfast", "-crf", "24",
"-pix_fmt", "yuv420p", "-c:a", "aac", small)
return small
+ def _vertical(self):
+ """A 1080x1920 clip: the real TikTok/Reels geometry, where the default caption size is
+ wide enough that an ordinary sentence needs four lines (eval 17)."""
+ vert = OUT / "cap_vertical.mp4"
+ if not vert.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "testsrc2=size=1080x1920:rate=30", "-f", "lavfi", "-i", "sine=f=440",
+ "-t", "6", "-c:v", "libx264", "-preset", "ultrafast", "-crf", "30",
+ "-pix_fmt", "yuv420p", "-c:a", "aac", vert)
+ return vert
+
+ def _beats(self):
+ """A 12 s clip with a synthetic 120 BPM click over moving pictures: a 440 Hz tone gated
+ to a short pulse every 0.5 s. Built from ffmpeg's own sources, so it is the same click on
+ every machine and the measured tempo is a fact of the fixture, not of the CI runner."""
+ clip = OUT / "beats.mp4"
+ if not clip.exists():
+ # a 40 ms pulse at the top of every half second
+ click = "0.8*sin(2*PI*880*t)*lt(mod(t\\,0.5)\\,0.04)"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", f"aevalsrc='{click}':s=48000",
+ "-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", clip)
+ return clip
+
+ def _silent_clip(self):
+ """12 s of near-silence over moving pictures: an audio stream with no pulse to measure."""
+ clip = OUT / "no_beats.mp4"
+ if not clip.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", "anoisesrc=amplitude=0.002:r=48000",
+ "-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", clip)
+ return clip
+
def _gappy(self):
"""A 12 s clip whose audio is speech-and-pause: tone for 2 s, silence for 2 s, six times
over. The structure detectors (silence.py, metadata.py --auto-chapters) have something
diff --git a/tests/fixtures/caption_1_16_1_fitsize_off.ass b/tests/fixtures/caption_1_16_1_fitsize_off.ass
new file mode 100644
index 0000000..b23da53
--- /dev/null
+++ b/tests/fixtures/caption_1_16_1_fitsize_off.ass
@@ -0,0 +1,17 @@
+[Script Info]
+ScriptType: v4.00+
+PlayResX: 1080
+PlayResY: 1920
+WrapStyle: 0
+ScaledBorderAndShadow: yes
+
+[V4+ Styles]
+Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
+Style: Default,DejaVu Sans,160,&H00FFFFFF,&H00FFFFFF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,13.3,0.0,2,420,420,420,1
+
+[Events]
+Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
+Dialogue: 0,0:00:00.00,0:00:01.94,Default,,0,0,0,,{\fad(200,200)}A third line\Nthe tool
+Dialogue: 0,0:00:01.94,0:00:03.00,Default,,0,0,0,,{\fad(200,200)}times\Nfor me
+Dialogue: 0,0:00:03.00,0:00:04.80,Default,,0,0,0,,{\fad(200,200)}Segunda\Nlínea de
+Dialogue: 0,0:00:04.80,0:00:06.00,Default,,0,0,0,,{\fad(200,200)}subtítulos
diff --git a/tests/fixtures/mcp_tools.json b/tests/fixtures/mcp_tools.json
index 13b13a4..7fe836c 100644
--- a/tests/fixtures/mcp_tools.json
+++ b/tests/fixtures/mcp_tools.json
@@ -93,6 +93,7 @@
"fast",
"folder",
"force",
+ "jobs",
"json",
"json_brief",
"overwrite",
@@ -154,6 +155,8 @@
"emoji_max",
"emoji_scale",
"fast",
+ "fit_size",
+ "fit_size_scope",
"font",
"fonts_dir",
"fps",
@@ -168,6 +171,7 @@
"margin",
"max_lines",
"min_duration",
+ "min_size",
"mode",
"model",
"offset",
@@ -322,6 +326,7 @@
"input",
"json",
"json_brief",
+ "min_confidence",
"output",
"overwrite",
"plan",
@@ -329,6 +334,9 @@
"progress",
"quality",
"segments",
+ "snap",
+ "snap_source",
+ "snap_tolerance",
"start",
"timeout",
"tolerance"
@@ -907,11 +915,13 @@
"properties": [
"argv",
"brand",
+ "cache",
"chapters",
"cues",
"dry_run",
"fast",
"fit",
+ "from_stage",
"image",
"init",
"json",
@@ -984,6 +994,9 @@
"name": "scenes",
"properties": [
"argv",
+ "beat_range",
+ "beat_step",
+ "beats",
"dry_run",
"edl",
"fast",
@@ -992,6 +1005,7 @@
"json",
"json_brief",
"max_scene",
+ "min_confidence",
"min_scene",
"no_timecode",
"overwrite",
@@ -1041,11 +1055,19 @@
"dry_run",
"edl",
"fast",
+ "filler",
+ "filler_extra",
+ "filler_keep",
+ "filler_lang",
+ "filler_list",
+ "filler_pad",
+ "filler_words",
"input",
"json",
"json_brief",
"list",
"margin",
+ "max_cuts",
"min_keep",
"min_silence",
"output",
@@ -1055,7 +1077,9 @@
"progress",
"quality",
"threshold",
- "timeout"
+ "timeout",
+ "transcribe",
+ "words"
],
"required": []
},
diff --git a/tests/test_analysis.py b/tests/test_analysis.py
index a111c0e..2b69848 100644
--- a/tests/test_analysis.py
+++ b/tests/test_analysis.py
@@ -530,6 +530,55 @@ class AnalysisTests(MediaFixtures):
self.assertClose(data["offset_seconds"], -28.0, 0.02)
self.assertGreater(data["confidence"], 0.3)
+ # ------------------------------------------------------ 1.17: the beat grid (scenes --beats)
+ def test_scenes_beats_measures_the_tempo(self):
+ data = json.loads(script("scenes.py", self._beats(), "--beats", "--json").stdout)
+ grid = data["beat_grid"]
+ self.assertAlmostEqual(grid["tempo_bpm"], 120.0, delta=2.0)
+ self.assertTrue(grid["usable"])
+ self.assertGreater(grid["confidence"], 0.5)
+ self.assertEqual(grid["method"], "rms-flux-autocorrelation")
+ self.assertEqual(grid["range_bpm"], [60.0, 200.0])
+ self.assertEqual(grid["supported"] + grid["unsupported"], len(data["beats"]))
+ self.assertEqual(len(grid["supported_beats"]), grid["supported"])
+ self.assertGreater(len(data["beats"]), 10)
+ # the existing document is untouched
+ self.assertIn("scenes", data)
+ self.assertIn("audio_peaks", data)
+
+ def test_scenes_without_beats_reports_no_grid(self):
+ data = json.loads(script("scenes.py", self._beats(), "--json").stdout)
+ self.assertNotIn("beat_grid", data)
+ self.assertNotIn("beats", data)
+
+ def test_scenes_beats_reports_a_weak_measurement_rather_than_refusing(self):
+ """scenes.py measures; it does not refuse. A low confidence is reported as usable: false
+ -- refusing belongs to the tools that would change a file on the strength of it."""
+ data = json.loads(script("scenes.py", self._silent_clip(), "--beats", "--json").stdout)
+ self.assertLess(data["beat_grid"]["confidence"], 0.5)
+ self.assertFalse(data["beat_grid"]["usable"])
+
+ def test_scenes_beats_refuses_without_an_audio_stream(self):
+ mute = OUT / "beats_mute.mp4"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", self._beats(),
+ "-an", "-c:v", "copy", mute)
+ r = script("scenes.py", mute, "--beats", "--json", expect_fail=True)
+ data = json.loads(r.stdout)
+ self.assertEqual(data["error"]["kind"], "input")
+ self.assertIn("audio stream", data["error"]["message"])
+
+ def test_scenes_beat_range_is_validated(self):
+ r = script("scenes.py", self._beats(), "--beats", "--beat-range", "notarange",
+ "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_scenes_beat_range_is_ignored_without_beats(self):
+ """--beat-range is a --beats flag: a run that never asked for beats must not be able to
+ die on one."""
+ data = json.loads(script("scenes.py", self._beats(), "--beat-range", "notarange",
+ "--json").stdout)
+ self.assertNotIn("beat_grid", data)
+
class ProposeChaptersTests(unittest.TestCase):
"""1.16: the pure half of metadata.py --auto-chapters. No media, no subprocess."""
@@ -624,5 +673,165 @@ class AutoChaptersTests(MediaFixtures):
"-o", OUT / "x_auto2.mp4", expect_fail=True)
+class BeatGridTests(unittest.TestCase):
+ """1.17: the beat grid and the snap, as pure arithmetic on an envelope.
+
+ Nothing here decodes anything -- that is the point: a beat is measured from numbers, and the
+ measurement can be tested without a media file.
+ """
+
+ STEP = 0.01
+
+ @staticmethod
+ def _clicks(interval, seconds=10.0, step=0.01, strengths=None, floor=0.01):
+ """An RMS envelope with an impulse every `interval` seconds -- a click track."""
+ n = int(round(seconds / step))
+ env = [floor] * n
+ k = 0
+ t = 0.0
+ while t < seconds:
+ i = int(round(t / step))
+ amp = 1.0 if strengths is None else strengths[k % len(strengths)]
+ if i < n - 1:
+ env[i] = amp
+ env[i + 1] = amp * 0.6
+ t += interval
+ k += 1
+ return env
+
+ def test_synthetic_120bpm_grid(self):
+ from _common import beat_grid
+ g = beat_grid(self._clicks(0.5), self.STEP)
+ self.assertAlmostEqual(g["tempo_bpm"], 120.0, delta=1.0)
+ self.assertAlmostEqual(g["interval"], 0.5, delta=0.01)
+ self.assertGreater(g["confidence"], 0.8)
+ self.assertTrue(g["usable"])
+ self.assertEqual(g["method"], "rms-flux-autocorrelation")
+ # every beat lands within 15 ms of a real impulse
+ for b in g["beats"][:10]:
+ self.assertLess(min(abs(b - k * 0.5) for k in range(25)), 0.015)
+
+ def test_half_and_double_tempo_disambiguation(self):
+ """Impulses every 0.5 s with alternating strength: the grid is 120, not 60 (only the
+ strong hits) and not 240 (twice as many points as there are onsets)."""
+ from _common import beat_grid
+ g = beat_grid(self._clicks(0.5, strengths=[1.0, 0.55]), self.STEP)
+ self.assertAlmostEqual(g["tempo_bpm"], 120.0, delta=1.0)
+
+ def test_a_different_tempo_is_measured_not_assumed(self):
+ from _common import beat_grid
+ g = beat_grid(self._clicks(0.4), self.STEP) # 150 BPM
+ self.assertAlmostEqual(g["tempo_bpm"], 150.0, delta=2.0)
+
+ def test_flat_envelope_is_zero_confidence(self):
+ from _common import beat_grid
+ g = beat_grid([0.5] * 1000, self.STEP)
+ self.assertEqual(g["beats"], [])
+ self.assertIsNone(g["tempo_bpm"])
+ self.assertEqual(g["confidence"], 0.0)
+ self.assertFalse(g["usable"])
+
+ def test_empty_envelope_is_zero_confidence(self):
+ from _common import beat_grid
+ self.assertEqual(beat_grid([], self.STEP)["confidence"], 0.0)
+ self.assertEqual(beat_grid([0.1, 0.2], self.STEP)["beats"], [])
+
+ def test_noise_is_not_a_beat(self):
+ """Pseudo-random levels have no steady pulse; whatever grid comes out must not be sold
+ as usable at the default threshold."""
+ from _common import beat_grid
+ import random
+ rng = random.Random(7)
+ env = [rng.random() for _ in range(1000)]
+ g = beat_grid(env, self.STEP)
+ self.assertLess(g["confidence"], 0.5)
+ self.assertFalse(g["usable"])
+
+ def test_min_confidence_only_moves_the_usable_flag(self):
+ from _common import beat_grid
+ env = self._clicks(0.5)
+ self.assertTrue(beat_grid(env, self.STEP, min_confidence=0.5)["usable"])
+ self.assertFalse(beat_grid(env, self.STEP, min_confidence=1.01)["usable"])
+
+ def test_unsupported_grid_points_are_counted_not_hidden(self):
+ """A grid must be regular, so a gap in the music still gets grid points -- and they are
+ reported as unsupported rather than quietly presented as measured beats."""
+ from _common import beat_grid
+ env = self._clicks(0.5, seconds=5.0) + [0.01] * 500
+ g = beat_grid(env, self.STEP)
+ self.assertGreater(g["unsupported"], 0)
+ self.assertEqual(g["supported"] + g["unsupported"], len(g["beats"]))
+
+ def test_supported_beats_excludes_the_grid_over_a_silent_passage(self):
+ """The regular grid runs on through a passage with no music in it -- that is what a grid
+ is. `supported_beats` is the subset a measured onset marks, and it is the only list a tool
+ that moves a cut may snap to: 8.0 s here is fifteen seconds into dead silence."""
+ from _common import beat_grid, snap_points
+ env = [0.001] * 2000
+ for i in range(0, 500, 50): # a 120 BPM click for the first 5 s
+ env[i] = 1.0
+ for i in range(500, 2000): # then 15 s of near-silence
+ env[i] = 0.0005
+ g = beat_grid(env, self.STEP)
+ self.assertAlmostEqual(g["tempo_bpm"], 120.0, delta=2.0)
+ self.assertLess(len(g["supported_beats"]), len(g["beats"]))
+ self.assertEqual(len(g["supported_beats"]), g["supported"])
+ self.assertTrue(all(b in g["beats"] for b in g["supported_beats"]))
+ self.assertLess(max(g["supported_beats"]), 6.0)
+ # the full grid would move a cut into the silence; the supported subset does not
+ self.assertTrue(snap_points([8.02], g["beats"], 0.12)[0]["snapped"])
+ self.assertFalse(snap_points([8.02], g["supported_beats"], 0.12)[0]["snapped"])
+
+ def test_flat_envelope_has_no_supported_beats(self):
+ from _common import beat_grid
+ self.assertEqual(beat_grid([0.5] * 1000, self.STEP)["supported_beats"], [])
+
+ def test_snap_points_never_adds_or_drops_a_point(self):
+ from _common import snap_points
+ import random
+ rng = random.Random(11)
+ for _ in range(50):
+ beats = sorted(round(rng.uniform(0, 30), 3) for _ in range(rng.randint(0, 20)))
+ points = [round(rng.uniform(0, 30), 3) for _ in range(rng.randint(1, 8))]
+ out = snap_points(points, beats, rng.uniform(0, 0.5))
+ self.assertEqual(len(out), len(points))
+ for row, p in zip(out, points):
+ self.assertEqual(row["from"], p)
+ self.assertTrue(row["to"] in beats or row["to"] == p)
+ if row["snapped"]:
+ self.assertIn(row["to"], beats)
+
+ def test_snap_respects_tolerance(self):
+ from _common import snap_points
+ out = snap_points([2.3], [1.0, 2.0, 3.0], 0.12)
+ self.assertFalse(out[0]["snapped"])
+ self.assertEqual(out[0]["to"], 2.3)
+ self.assertEqual(out[0]["delta"], 0.0)
+ near = snap_points([2.03], [1.0, 2.0, 3.0], 0.12)
+ self.assertTrue(near[0]["snapped"])
+ self.assertEqual(near[0]["to"], 2.0)
+ self.assertAlmostEqual(near[0]["delta"], -0.03, places=6)
+
+ def test_snap_with_no_beats_changes_nothing(self):
+ from _common import snap_points
+ out = snap_points([1.0, 2.0], [], 1.0)
+ self.assertEqual([r["to"] for r in out], [1.0, 2.0])
+ self.assertFalse(any(r["snapped"] for r in out))
+
+ def test_beat_grid_is_pure(self):
+ import unittest.mock
+ import _common
+ from _common import beat_grid, snap_points
+ def boom(*a, **k):
+ raise AssertionError("beat_grid ran a subprocess")
+ with unittest.mock.patch.object(_common, "run", boom), \
+ unittest.mock.patch.object(_common, "run_analysis", boom), \
+ unittest.mock.patch.object(subprocess, "run", boom), \
+ unittest.mock.patch.object(subprocess, "Popen", boom):
+ g = beat_grid(self._clicks(0.5), self.STEP)
+ self.assertAlmostEqual(g["tempo_bpm"], 120.0, delta=1.0)
+ self.assertEqual(len(snap_points([1.0], g["beats"], 0.1)), 1)
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)
diff --git a/tests/test_contract.py b/tests/test_contract.py
index ae92f22..f54ff53 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -1158,6 +1158,49 @@ class ContractTests(unittest.TestCase):
# ------------------------------------------------------------------ integration: claims hold at run time
@unittest.skipIf(platform.system() == "Windows", "fake ffmpeg is a #!/bin/sh script on a POSIX-only PATH shim; not portable to Windows. The claim itself (run() never invokes ffmpeg under --dry-run) is still exercised on Windows by every --dry-run case in tests/test_all.py, just without a shim proving no *other* ffmpeg-shaped binary would have run.")
+ def test_every_1_17_flag_is_reachable_off(self):
+ """Each new 1.17 flag defaults to 1.16's behaviour, or is switched off by one flag.
+
+ The command lines below are the ones `main` builds, and they must still be what these
+ runs plan: `--fit-size off`, `--snap none` (the default), no `--filler`, `--jobs 1` (the
+ default) and no `--cache` are the five ways back.
+ """
+ outdir = self.work / "reach"
+ outdir.mkdir(exist_ok=True)
+
+ def plan(name, *args):
+ proc = tool(name, *args, "--dry-run", "--json", check=False)
+ self.assertEqual(proc.returncode, 0, f"{name}:\n{proc.stderr}")
+ return json.loads(proc.stdout)
+
+ # caption: --fit-size off plans exactly what no flag at all planned before the feature
+ off = plan("caption", self.src, "--text", self.cues, "--fit-size", "off",
+ "-o", outdir / "c_off.mp4")
+ self.assertEqual(off["caption"]["size_used"], off["caption"]["size_requested"])
+ self.assertEqual(off["caption"]["shrunk"], 0)
+ explicit = plan("caption", self.src, "--text", self.cues, "--size", "24",
+ "-o", outdir / "c_size.mp4")
+ self.assertEqual(explicit["caption"]["size_used"], 24)
+
+ # cut: --snap defaults to none, and no snap key appears
+ self.assertIsNone(plan("cut", self.src, "--start", "1", "--end", "3",
+ "-o", outdir / "cut.mp4").get("snap"))
+
+ # silence: no --filler, no filler key, removed_seconds keeps its lone meaning
+ sil = plan("silence", self.src, "-o", outdir / "sil.mp4")
+ self.assertNotIn("filler", sil)
+ self.assertNotIn("removed_seconds_total", sil)
+
+ # render: no --cache, no cache key
+ proj = outdir / "p.json"
+ proj.write_text(json.dumps({"output": str(outdir / "r.mp4"),
+ "clips": [{"src": str(self.src), "in": 0, "out": 2}]}),
+ encoding="utf-8")
+ rendered = plan("render", proj)
+ self.assertIsNone(rendered.get("cache"))
+ self.assertIsNone(rendered.get("snap"))
+ self.assertFalse(any("--snap" in c or "--cache" in c for c in rendered["commands"]))
+
def test_dry_run_never_runs_ffmpeg_and_writes_nothing(self):
"""A fake ffmpeg first on PATH records every invocation; ffprobe stays real."""
shim = self.work / "shim"
diff --git a/tests/test_editing.py b/tests/test_editing.py
index 2975b14..89ffa2e 100644
--- a/tests/test_editing.py
+++ b/tests/test_editing.py
@@ -4,8 +4,11 @@
python3 tests/test_editing.py # this group alone
python3 tests/test_all.py # every group
"""
+import os
+import platform
import json
import re
+import shutil
import subprocess
import sys
import unittest
@@ -1036,6 +1039,467 @@ class EditingTests(MediaFixtures):
self.assertTrue(doc["error"]["message"])
self.assertNotIn("summary", doc)
+ # --------------------------------------------------- 1.17: beat-snapped cuts (cut.py --snap)
+ def test_cut_snap_beats_moves_the_points_to_measured_beats(self):
+ out = OUT / "snap_cut.mp4"
+ data = json.loads(script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01",
+ "--snap", "beats", "-o", out, "--json").stdout)
+ snap = data["snap"]
+ self.assertEqual(snap["mode"], "beats")
+ self.assertEqual(snap["source"], "measured")
+ self.assertEqual(snap["snapped"], 2)
+ self.assertEqual(len(snap["moved"]), 2) # never more points than were asked for
+ self.assertAlmostEqual(snap["tempo_bpm"], 120.0, delta=2.0)
+ for row in snap["moved"]:
+ self.assertLessEqual(abs(row["delta"]), snap["tolerance"])
+ # the cut lands on a whole number of beats
+ span = snap["moved"][1]["to"] - snap["moved"][0]["to"]
+ beats = span / (60.0 / snap["tempo_bpm"])
+ self.assertLess(abs(beats - round(beats)), 0.05)
+ # container duration, not the cut: AAC packs audio in 1024-sample frames (21 ms at
+ # 48 kHz) and ffmpeg 5.1 pads the last one, so allow a few frames of slack
+ self.assertLess(abs(data["output_duration"] - span), 0.1)
+
+ def test_cut_snap_beats_moves_only_onto_onset_supported_points(self):
+ """The grid is regular by construction, so it runs on through a passage with no music.
+ A cut asked for inside that passage is NOT moved: the only points this tool may snap to
+ are the ones a measured onset marks."""
+ clip = OUT / "beats_half.mp4"
+ if not clip.exists():
+ # 12 s: a 120 BPM click for the first 5 s, then silence
+ click = "0.8*sin(2*PI*880*t)*lt(mod(t\\,0.5)\\,0.04)*lt(t\\,5)"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
+ "-f", "lavfi", "-i", f"aevalsrc='{click}':s=48000",
+ "-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30",
+ "-t", "12", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", clip)
+ data = json.loads(script("cut.py", clip, "--start", "2.03", "--end", "9.01",
+ "--snap", "beats", "-o", OUT / "snap_half.mp4",
+ "--json").stdout)
+ snap = data["snap"]
+ self.assertEqual(snap["grid"], "supported")
+ self.assertLess(snap["grid_points"], 24) # fewer than the full 12 s grid
+ # the in point sits in the music and moves; the out point sits in the silence and does not
+ self.assertTrue(snap["moved"][0]["snapped"])
+ self.assertFalse(snap["moved"][1]["snapped"])
+ self.assertEqual(snap["moved"][1]["to"], snap["moved"][1]["from"])
+ self.assertEqual(snap["snapped"], 1)
+
+ def test_cut_snap_beats_refuses_a_zero_confidence_floor(self):
+ r = script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01", "--snap", "beats",
+ "--min-confidence", "0", "-o", OUT / "snap_zero.mp4", "--json",
+ expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--min-confidence", err["message"])
+
+ def test_cut_snap_source_without_a_supported_list_is_refused(self):
+ """A document from an older build lists beats but cannot say which of them an onset
+ supports. An unknown subset is not an empty one -- and it is not a measurement either."""
+ doc = OUT / "snap_old.json"
+ doc.write_text(json.dumps({"beats": [1.0, 2.0, 3.0],
+ "beat_grid": {"tempo_bpm": 120.0, "confidence": 0.9}}),
+ encoding="utf-8")
+ r = script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01", "--snap", "beats",
+ "--snap-source", doc, "-o", OUT / "snap_old.mp4", "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_cut_snap_source_with_no_tempo_is_refused(self):
+ doc = OUT / "snap_notempo.json"
+ doc.write_text(json.dumps({"beats": [1.0, 2.0],
+ "beat_grid": {"tempo_bpm": None, "confidence": 0.9,
+ "supported_beats": [1.0, 2.0]}}),
+ encoding="utf-8")
+ r = script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01", "--snap", "beats",
+ "--snap-source", doc, "-o", OUT / "snap_nt.mp4", "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_cut_snap_beats_refuses_a_whole_file_copy_spelled_as_a_time(self):
+ """`--start 0:00` is the same whole-file copy as no --start at all; a string comparison
+ against "0" let it through and the run then shortened the file at the far end."""
+ for start in ("0:00", "0.0", "00:00:00"):
+ with self.subTest(start=start):
+ r = script("cut.py", self._beats(), "--start", start, "--snap", "beats",
+ "-o", OUT / "snap_wholetime.mp4", "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--snap", err["message"])
+
+ def test_cut_snap_beats_refuses_a_grid_it_cannot_measure(self):
+ """Never fabricate: without a measurable pulse the points are not moved to invented
+ times -- the run refuses and names --snap none."""
+ out = OUT / "snap_refuse.mp4"
+ if out.exists():
+ out.unlink()
+ r = script("cut.py", self._silent_clip(), "--start", "2.03", "--end", "6.01",
+ "--snap", "beats", "-o", out, "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--snap none", err["message"])
+ self.assertIn("confidence", err["message"])
+ self.assertFalse(out.exists())
+
+ def test_cut_snap_beats_refuses_without_audio(self):
+ mute = OUT / "snap_mute.mp4"
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", self._beats(),
+ "-an", "-c:v", "copy", mute)
+ r = script("cut.py", mute, "--start", "2.03", "--end", "6.01", "--snap", "beats",
+ "-o", OUT / "snap_mute_cut.mp4", "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--snap none", err["message"])
+
+ def test_cut_snap_beats_refuses_a_whole_file_copy(self):
+ r = script("cut.py", self._beats(), "--snap", "beats",
+ "-o", OUT / "snap_whole.mp4", "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--snap", err["message"])
+
+ def test_cut_snap_source_takes_the_grid_from_a_scenes_document(self):
+ doc = OUT / "snap_grid.json"
+ doc.write_text(script("scenes.py", self._beats(), "--beats", "--json").stdout,
+ encoding="utf-8")
+ data = json.loads(script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01",
+ "--snap", "beats", "--snap-source", doc,
+ "-o", OUT / "snap_src.mp4", "--json").stdout)
+ self.assertEqual(data["snap"]["source"], str(doc))
+ self.assertEqual(data["snap"]["snapped"], 2)
+
+ def test_cut_without_snap_reports_no_snap_key(self):
+ data = json.loads(script("cut.py", self._beats(), "--start", "2.03", "--end", "6.01",
+ "-o", OUT / "snap_off.mp4", "--json").stdout)
+ self.assertIsNone(data.get("snap"))
+
+ # ------------------------------------------------ 1.17: filler words (silence.py --filler)
+ def _words_json(self, name="words.json", lang="en", words=None):
+ path = OUT / name
+ words = words if words is not None else [
+ ("So", 0.2, 0.5), ("um", 0.6, 0.8), ("this", 1.0, 1.3), ("uh", 2.2, 2.45),
+ ("umbrella", 3.0, 3.6), ("erm", 4.1, 4.35), ("works", 5.0, 5.4)]
+ path.write_text(json.dumps({"language": lang, "segments": [
+ {"words": [{"word": w, "start": a, "end": b} for w, a, b in words]}]}),
+ encoding="utf-8")
+ return path
+
+ def test_filler_removes_only_the_timed_filler_words(self):
+ out = OUT / "filler_out.mp4"
+ data = json.loads(script("silence.py", self._gappy(), "--filler",
+ "--words", self._words_json(), "-o", out, "--json").stdout)
+ fil = data["filler"]
+ self.assertEqual(fil["removed_count"], 3)
+ self.assertEqual(fil["lang"], "en")
+ self.assertEqual(fil["list"], "builtin")
+ self.assertEqual(fil["word_timings"], 7)
+ self.assertEqual(sorted(fil["removed_words"]), ["erm", "uh", "um"])
+ self.assertIn("removed_seconds_total", data)
+ self.assertGreater(fil["removed_seconds"], 0)
+ # "umbrella" is not "um": whole tokens only
+ self.assertNotIn("umbrella", fil["removed_words"])
+ m = probe(str(out))
+ self.assertLess(m["duration"], data["input_duration"])
+
+ def _talk_with_a_gap(self):
+ """10 s: tone, 4 s of silence in the middle, tone. A filler word placed at 3.0 s sits
+ INSIDE that silence, which is where a mumbled "um" usually is."""
+ talk = OUT / "talk_gap.m4a"
+ if not talk.exists():
+ sh("ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-f", "lavfi",
+ "-i", "aevalsrc='0.5*sin(2*PI*440*t)*(lt(t\\,2)+gt(t\\,6))':s=48000",
+ "-t", "10", talk)
+ return talk
+
+ def test_filler_inside_a_silence_still_removes_the_silence(self):
+ """The regression: filler spans were merged only among themselves, so a span nested
+ inside a detected silence rewound keep_ranges' cursor and handed the silence back.
+ Adding --filler made the tool remove LESS than it did without it."""
+ talk = self._talk_with_a_gap()
+ words = OUT / "words_inside.json"
+ words.write_text(json.dumps({"language": "en", "segments": [
+ {"words": [{"word": "um", "start": 3.0, "end": 3.2}]}]}), encoding="utf-8")
+ plain = json.loads(script("silence.py", talk, "--list", "--json").stdout)
+ withf = json.loads(script("silence.py", talk, "--filler", "--filler-list",
+ "--words", words, "--json").stdout)
+ self.assertEqual(plain["silences"], withf["silences"])
+ # --filler can only ever remove at least as much as the same run without it
+ self.assertGreaterEqual(withf["removed_seconds_total"], plain["removed_seconds"] - 0.01)
+ self.assertGreater(withf["removed_seconds_total"], 3.0)
+
+ def test_removed_seconds_keeps_its_silence_only_meaning(self):
+ """A caller reading `removed_seconds` since 1.0 asked how much dead air went. --filler
+ must not quietly make it mean something else; `removed_seconds_total` is the new figure."""
+ talk = self._talk_with_a_gap()
+ words = OUT / "words_outside.json"
+ # a filler word in the SPEECH, well away from the silence, so the two figures differ
+ words.write_text(json.dumps({"language": "en", "segments": [
+ {"words": [{"word": "um", "start": 0.5, "end": 0.9}]}]}), encoding="utf-8")
+ plain = json.loads(script("silence.py", talk, "--list", "--json").stdout)
+ withf = json.loads(script("silence.py", talk, "--filler", "--filler-list",
+ "--words", words, "--json").stdout)
+ self.assertAlmostEqual(withf["removed_seconds"], plain["removed_seconds"], delta=0.01)
+ self.assertGreater(withf["removed_seconds_total"], withf["removed_seconds"])
+
+ def test_keep_ranges_is_monotone_with_a_nested_span(self):
+ """keep_ranges walks one cursor forward; a nested span used to rewind it."""
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ silence = importlib.import_module("silence")
+ nested = silence.keep_ranges(sorted([(1.0, 3.0), (1.5, 1.8)]), 10, 0.05, 0.2)
+ plain = silence.keep_ranges([(1.0, 3.0)], 10, 0.05, 0.2)
+ self.assertEqual(nested, plain)
+ self.assertEqual(silence.merge_spans([(1.0, 3.0), (1.5, 1.8), (4.0, 4.5), (4.5, 5.0)]),
+ [(1.0, 3.0), (4.0, 5.0)])
+ # an unsorted input is sorted on the way in
+ self.assertEqual(silence.keep_ranges([(4.0, 5.0), (1.0, 2.0)], 10, 0.05, 0.2),
+ silence.keep_ranges([(1.0, 2.0), (4.0, 5.0)], 10, 0.05, 0.2))
+
+ def test_filler_list_writes_nothing(self):
+ out = OUT / "filler_nothing.mp4"
+ if out.exists():
+ out.unlink()
+ data = json.loads(script("silence.py", self._gappy(), "--filler", "--filler-list",
+ "--words", self._words_json(), "-o", out, "--json").stdout)
+ self.assertEqual(data["filler"]["removed_count"], 3)
+ self.assertFalse(out.exists())
+
+ def test_filler_keep_and_extra_change_the_list(self):
+ kept = json.loads(script("silence.py", self._gappy(), "--filler", "--filler-list",
+ "--filler-keep", "um,uh", "--words", self._words_json(),
+ "--json").stdout)["filler"]
+ self.assertEqual(kept["removed_words"], ["erm"])
+ extra = json.loads(script("silence.py", self._gappy(), "--filler", "--filler-list",
+ "--filler-extra", "so", "--words", self._words_json(),
+ "--json").stdout)["filler"]
+ self.assertIn("so", extra["removed_words"])
+
+ def test_filler_warns_about_a_japanese_discourse_marker(self):
+ words = [("\u305d\u308c\u306f", 0.2, 0.5), ("\u306a\u3093\u304b", 0.6, 0.85), ("\u3044\u3044", 1.0, 1.3)]
+ data = json.loads(script("silence.py", self._gappy(), "--filler", "--filler-list",
+ "--words", self._words_json("words_ja.json", "ja", words),
+ "--json").stdout)
+ fil = data["filler"]
+ self.assertEqual(fil["lang"], "ja")
+ self.assertIn("\u306a\u3093\u304b", fil["removed_words"])
+ self.assertTrue(any("\u306a\u3093\u304b" in w for w in fil["warnings"]))
+
+ def test_filler_transcribe_uses_measured_word_timings(self):
+ """e2e with the engine bridge mocked: --transcribe must reach filler_spans with real
+ per-word start/end pairs. The bridge is mocked because the suite must pass on a machine
+ with no whisper at all -- which is what CI is."""
+ shim = OUT / "asr_shim"
+ shim.mkdir(exist_ok=True)
+ (shim / "sitecustomize.py").write_text(
+ "import sys, os\n"
+ "sys.path.insert(0, os.environ['FFSKILL_SCRIPTS'])\n"
+ "import _common.asr as asr, _common as c\n"
+ "def fake(video, language=None, model='base', audio_stream=0):\n"
+ " return ([{'word': 'So', 'start': 0.2, 'end': 0.5},\n"
+ " {'word': 'um', 'start': 0.6, 'end': 0.8},\n"
+ " {'word': 'this', 'start': 1.0, 'end': 1.3},\n"
+ " {'word': 'uh', 'start': 2.2, 'end': 2.45}], 'faster-whisper')\n"
+ "asr.transcribe_words = fake\n"
+ "c.transcribe_words = fake\n",
+ encoding="utf-8")
+ env = dict(os.environ, PYTHONPATH=str(shim), FFSKILL_SCRIPTS=str(SCRIPTS))
+ r = sh(sys.executable, SCRIPTS / "silence.py", self._gappy(), "--filler",
+ "--transcribe", "--filler-list", "--json", env=env)
+ fil = json.loads(r.stdout)["filler"]
+ self.assertEqual(fil["engine"], "faster-whisper")
+ self.assertEqual(fil["word_timings"], 4)
+ self.assertEqual(sorted(fil["removed_words"]), ["uh", "um"])
+ for span in fil["removed"]:
+ self.assertLess(span["start"], span["end"])
+
+ def test_filler_transcribe_refuses_when_the_engine_gives_no_word_timings(self):
+ """An engine that runs but whose build has no word timestamps must say exactly that and
+ name --words -- not report an empty removal as a success."""
+ shim = OUT / "asr_shim_empty"
+ shim.mkdir(exist_ok=True)
+ (shim / "sitecustomize.py").write_text(
+ "import sys, os\n"
+ "sys.path.insert(0, os.environ['FFSKILL_SCRIPTS'])\n"
+ "import _common.asr as asr, _common as c\n"
+ "def fake(video, language=None, model='base', audio_stream=0):\n"
+ " return ([], 'whisper.cpp')\n"
+ "asr.transcribe_words = fake\n"
+ "c.transcribe_words = fake\n",
+ encoding="utf-8")
+ env = dict(os.environ, PYTHONPATH=str(shim), FFSKILL_SCRIPTS=str(SCRIPTS))
+ out = OUT / "filler_nowords.mp4"
+ if out.exists():
+ out.unlink()
+ r = sh(sys.executable, SCRIPTS / "silence.py", self._gappy(), "--filler",
+ "--transcribe", "-o", out, "--json", expect_fail=True, env=env)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("whisper.cpp", err["message"])
+ self.assertIn("--words", err["message"])
+ self.assertFalse(out.exists())
+
+ def test_filler_pad_must_not_be_negative(self):
+ """A negative pad turns each span inside out and filler_spans then drops them all, so the
+ run would report a successful removal of nothing."""
+ r = script("silence.py", self._gappy(), "--filler", "--filler-pad", "-0.5",
+ "--words", self._words_json(), "-o", OUT / "filler_pad.mp4",
+ "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_filler_refuses_without_a_transcript(self):
+ out = OUT / "filler_refuse.mp4"
+ if out.exists():
+ out.unlink()
+ r = script("silence.py", self._gappy(), "--filler", "-o", out, "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--words", err["message"])
+ self.assertIn("--transcribe", err["message"])
+ self.assertFalse(out.exists())
+
+ def test_filler_refuses_a_segment_only_transcript(self):
+ doc = OUT / "segments_only.json"
+ doc.write_text(json.dumps({"language": "en", "segments": [
+ {"start": 0.0, "end": 2.0, "text": "So um this"}]}), encoding="utf-8")
+ r = script("silence.py", self._gappy(), "--filler", "--words", doc,
+ "-o", OUT / "filler_seg.mp4", "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("word timings", err["message"])
+
+ def test_filler_refuses_when_no_whisper_is_installed(self):
+ """The eval image has no engine; the refusal must name the three installs. PATH is
+ stripped so the test does not depend on whether this machine has one."""
+ try:
+ import faster_whisper # noqa: F401
+ self.skipTest("faster-whisper is installed on this machine; the absent-engine path "
+ "cannot be exercised without uninstalling it")
+ except ImportError:
+ pass
+ # A PATH with ffmpeg/ffprobe and nothing else: stripping PATH outright would make the
+ # run fail on ffprobe long before it reached the engine probe.
+ if platform.system() == "Windows":
+ # A symlink needs a privilege here, and a COPIED ffmpeg.exe cannot find its sibling
+ # DLLs -- so neither branch of the shim works on Windows, and pretending one does
+ # would make this a test that passes without exercising anything. The refusal it
+ # covers is platform-independent (it is a PATH/import probe in _common/asr.py), and
+ # the mocked-bridge tests above cover the same message on every OS.
+ self.skipTest("no way to build a PATH shim on Windows: a copied ffmpeg.exe loses "
+ "its DLLs and a symlink needs a privilege")
+ shim = OUT / "no_whisper_path"
+ shim.mkdir(exist_ok=True)
+ for tool in ("ffmpeg", "ffprobe"):
+ real = shutil.which(tool)
+ if not real:
+ self.skipTest(f"{tool} not on PATH")
+ link = shim / Path(real).name
+ if link.exists():
+ continue
+ try:
+ os.symlink(real, link)
+ except (OSError, NotImplementedError, AttributeError):
+ self.skipTest("cannot symlink ffmpeg into a PATH shim on this machine")
+ env = dict(os.environ, PATH=str(shim))
+ out = OUT / "filler_nowhisper.mp4"
+ if out.exists():
+ out.unlink()
+ r = sh(sys.executable, SCRIPTS / "silence.py", self._gappy(), "--filler", "--transcribe",
+ "-o", out, "--json", expect_fail=True, env=env)
+ message = json.loads(r.stdout)["error"]["message"]
+ for engine in ("whisper.cpp", "faster-whisper", "openai-whisper"):
+ self.assertIn(engine, message)
+ self.assertFalse(out.exists())
+
+ def test_silence_without_filler_is_unchanged(self):
+ data = json.loads(script("silence.py", self._gappy(), "--list", "--json").stdout)
+ self.assertNotIn("filler", data)
+ self.assertNotIn("removed_seconds_total", data)
+
+
+class FillerSpansTests(unittest.TestCase):
+ """1.17: which words become removal spans, as pure arithmetic on measured timings."""
+
+ def setUp(self):
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ self.D = importlib.import_module("_common.decision")
+
+ def _w(self, *triples):
+ return [{"word": w, "start": a, "end": b} for w, a, b in triples]
+
+ def test_matches_whole_tokens_only(self):
+ spans = self.D.filler_spans(
+ self._w(("umbrella", 0.0, 0.4), ("Umm,", 1.0, 1.2), ("UM", 2.0, 2.15)),
+ self.D.FILLER_WORDS["en"] | {"umm"})
+ self.assertEqual([s["word"] for s in spans], ["umm", "um"])
+
+ def test_never_without_timings(self):
+ self.assertEqual(self.D.filler_spans([{"word": "um"}], {"um"}), [])
+ self.assertEqual(self.D.filler_spans([{"word": "um", "start": 2.0, "end": 1.0}], {"um"}), [])
+ self.assertEqual(self.D.filler_spans([{"word": "um", "start": None, "end": 1.0}], {"um"}), [])
+
+ def test_merges_adjacent_spans(self):
+ close = self.D.filler_spans(self._w(("um", 1.0, 1.1), ("uh", 1.13, 1.25)),
+ {"um", "uh"}, pad=0.0)
+ self.assertEqual(len(close), 1)
+ apart = self.D.filler_spans(self._w(("um", 1.0, 1.1), ("uh", 1.3, 1.45)),
+ {"um", "uh"}, pad=0.0)
+ self.assertEqual(len(apart), 2)
+
+ def test_rejects_a_long_held_word(self):
+ self.assertEqual(self.D.filler_spans(self._w(("uhhh", 1.0, 3.0)), {"uhhh"}), [])
+
+ def test_japanese_list(self):
+ ja = self.D.FILLER_WORDS["ja"]
+ spans = self.D.filler_spans(
+ self._w(("\u3048\u30fc\u3068", 1.0, 1.3), ("\u3042\u306e", 2.0, 2.2), ("\u3042\u306e\u3072\u3068", 3.0, 3.4)), ja)
+ self.assertEqual([s["word"] for s in spans], ["\u3048\u30fc\u3068", "\u3042\u306e"])
+
+ def test_like_is_not_a_default_filler_word(self):
+ """A discourse marker is a content word: cutting it cuts meaning, which is a judgement
+ this skill does not make. It is reachable with --filler-extra and documented."""
+ self.assertNotIn("like", self.D.FILLER_WORDS["en"])
+ self.assertNotIn("tipo", self.D.FILLER_WORDS["pt"])
+ self.assertIn("like", self.D.FILLER_DISCOURSE_MARKERS["en"])
+
+ def test_whisper_cpp_json_becomes_word_timings(self):
+ """whisper.cpp --output-json-full: transcription[].tokens[] with MILLISECOND offsets,
+ special tokens dropped, and a continuation token glued to the word before it."""
+ import importlib
+ asr = importlib.import_module("_common.asr")
+ doc = OUT / "wcpp.json"
+ doc.write_text(json.dumps({"transcription": [{"tokens": [
+ {"text": "[_BEG_]", "offsets": {"from": 0, "to": 0}},
+ {"text": " So", "offsets": {"from": 200, "to": 500}},
+ {"text": " un", "offsets": {"from": 600, "to": 700}},
+ {"text": "believable", "offsets": {"from": 700, "to": 1100}},
+ {"text": " um", "offsets": {"from": 1200, "to": 1400}},
+ {"text": " broken", "offsets": {}},
+ ]}]}), encoding="utf-8")
+ words = asr._words_from_whisper_cpp_json(str(doc))
+ self.assertEqual([w["word"] for w in words], ["So", "unbelievable", "um"])
+ self.assertAlmostEqual(words[0]["start"], 0.2, places=6)
+ self.assertAlmostEqual(words[1]["end"], 1.1, places=6) # glued span ends at the tail
+ self.assertEqual(asr._words_from_whisper_cpp_json(str(OUT / "nope.json")), [])
+
+ def test_openai_whisper_json_becomes_word_timings(self):
+ import importlib
+ asr = importlib.import_module("_common.asr")
+ doc = OUT / "owhisper.json"
+ doc.write_text(json.dumps({"segments": [
+ {"words": [{"word": " So", "start": 0.2, "end": 0.5},
+ {"word": " um", "start": 0.6, "end": 0.8},
+ {"word": " bad", "start": None, "end": 1.0}]}]}), encoding="utf-8")
+ words = asr._words_from_openai_whisper_json(str(doc))
+ self.assertEqual([w["word"] for w in words], ["So", "um"])
+ self.assertAlmostEqual(words[1]["start"], 0.6, places=6)
+
+ def test_filler_spans_is_pure(self):
+ import unittest.mock
+ def boom(*a, **k):
+ raise AssertionError("filler_spans ran a subprocess")
+ with unittest.mock.patch.object(subprocess, "run", boom), \
+ unittest.mock.patch.object(subprocess, "Popen", boom):
+ self.assertEqual(len(self.D.filler_spans(self._w(("um", 1.0, 1.2)), {"um"})), 1)
+
if __name__ == "__main__":
unittest.main(verbosity=2)
diff --git a/tests/test_orchestration.py b/tests/test_orchestration.py
index fb7d398..8dafe16 100644
--- a/tests/test_orchestration.py
+++ b/tests/test_orchestration.py
@@ -21,6 +21,55 @@ from _common import font_for_script, probe, shell_quote # noqa: E402
class OrchestrationTests(MediaFixtures):
"""Render, batch and multicam, plus the toolkit-wide invariants every script has to satisfy."""
+ # ------------------------------------------- 1.17: a project may ask for beat-snapped clips
+ def test_render_forwards_snap_to_the_clip_cut(self):
+ proj = OUT / "snap_project.json"
+ proj.write_text(json.dumps({
+ "output": str(OUT / "snap_render.mp4"),
+ "snap": {"to": "beats", "tolerance": 0.12, "min_confidence": 0.5},
+ "clips": [{"src": str(self._beats()), "in": "2.03", "out": "6.01"}],
+ }), encoding="utf-8")
+ data = json.loads(script("render.py", proj, "--json").stdout)
+ self.assertEqual(data["snap"]["mode"], "beats")
+ self.assertEqual(data["snap"]["snapped"], 2)
+ self.assertIn("clips", data["stages"])
+
+ def test_render_snap_covers_every_clip_and_survives_a_cache_hit(self):
+ """Only the first clip's moves were reported, and a cached clips stage reported
+ snap: null although the clips had been snapped."""
+ cdir = OUT / "rcache_snap"
+ shutil.rmtree(cdir, ignore_errors=True)
+ proj = OUT / "snap_multi.json"
+ proj.write_text(json.dumps({
+ "output": str(OUT / "snap_multi.mp4"),
+ "snap": {"to": "beats", "tolerance": 0.12},
+ "clips": [{"src": str(self._beats()), "in": "2.03", "out": "5.01"},
+ {"src": str(self._beats()), "in": "6.03", "out": "9.01"}],
+ }), encoding="utf-8")
+ data = json.loads(script("render.py", proj, "--cache", cdir, "--json").stdout)
+ self.assertEqual(len(data["snap"]["clips"]), 2)
+ self.assertEqual([c["clip"] for c in data["snap"]["clips"]], [0, 1])
+ self.assertTrue(all(c["snapped"] for c in data["snap"]["clips"]))
+ self.assertEqual(data["snap"]["mode"], "beats") # the shape a caller reads today
+ # second run: the clips come from the cache, and the report must not claim they were
+ # never snapped
+ again = json.loads(script("render.py", proj, "--cache", cdir, "--json").stdout)
+ self.assertIn("clips", again["cache"]["hits"])
+ self.assertIsNotNone(again["snap"])
+ self.assertEqual(len(again["snap"]["clips"]), 2)
+ self.assertEqual(again["snap"]["clips"][0]["source"], "cache")
+
+ def test_render_without_snap_builds_the_same_command_as_before(self):
+ proj = OUT / "nosnap_project.json"
+ proj.write_text(json.dumps({
+ "output": str(OUT / "nosnap_render.mp4"),
+ "clips": [{"src": str(self._beats()), "in": "2.03", "out": "6.01"}],
+ }), encoding="utf-8")
+ data = json.loads(script("render.py", proj, "--dry-run", "--json").stdout)
+ self.assertIsNone(data.get("snap"))
+ self.assertFalse(any("--snap" in c for c in data["commands"]))
+
+
def test_zero_or_negative_fps_refused_across_every_cfr_script(self):
"""--fps flows straight into cfr_args(meta, args.fps) / a `fps or source_fps or 30.0`
fallback in several scripts without ever being validated first. `0` is falsy in Python, so
@@ -661,6 +710,285 @@ class OrchestrationTests(MediaFixtures):
self.assertEqual(resp[0]["id"], 1)
self.assertIn("tools", resp[0]["result"])
+ # ------------------------------------------------------------- 1.17: batch.py --jobs N
+ def _jobs_folder(self, name, count=4):
+ folder = OUT / name
+ folder.mkdir(exist_ok=True)
+ for i in range(count):
+ (folder / f"j{i}.mp4").write_bytes(Path(self.src).read_bytes())
+ recipe = folder / "batch.json"
+ recipe.write_text(json.dumps({
+ "glob": "*.mp4", "output_dir": "out", "suffix": "_j",
+ "steps": [["fit.py", "{in}", "--duration", "3", "-o", "{out}"]]}))
+ return folder, recipe
+
+ def test_jobs_summary_is_identical_to_serial(self):
+ """The per-item table is the promise: same order, same keys, whatever order the encodes
+ actually finished in."""
+ folder, recipe = self._jobs_folder("batch_jobs_a")
+ serial = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--force", "--jobs", "1", "--json").stdout)
+ parallel = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--force", "--jobs", "4", "--json").stdout)
+ self.assertEqual(serial["jobs"], 1)
+ self.assertGreater(parallel["jobs"], 1)
+ self.assertEqual(len(serial["results"]), len(parallel["results"]))
+ for a, b in zip(serial["results"], parallel["results"]):
+ self.assertEqual({k: v for k, v in a.items() if k != "seconds"},
+ {k: v for k, v in b.items() if k != "seconds"})
+ self.assertEqual(serial["processed"], parallel["processed"])
+ self.assertFalse(parallel["timed_out"])
+ self.assertGreater(parallel["item_seconds_total"], 0)
+
+ def test_results_keep_file_order_with_a_partially_warm_cache(self):
+ """Cached hits were appended in one pass and freshly-processed items after them, so the
+ per-item table came back out of file order whenever the cache was partially warm -- at
+ --jobs 1, the default, not only in parallel."""
+ for jobs in ("1", "3"):
+ with self.subTest(jobs=jobs):
+ folder = OUT / f"batch_order_{jobs}"
+ shutil.rmtree(folder, ignore_errors=True)
+ folder.mkdir(parents=True)
+ for name in ("a.mp4", "b.mp4", "c.mp4"):
+ (folder / name).write_bytes(Path(self.src).read_bytes())
+ recipe = folder / "batch.json"
+ recipe.write_text(json.dumps({
+ "glob": "*.mp4", "output_dir": "out", "suffix": "_o",
+ "steps": [["fit.py", "{in}", "--duration", "3", "-o", "{out}"]]}))
+ first = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--jobs", jobs, "--json").stdout)
+ self.assertEqual([Path(r["file"]).name for r in first["results"]],
+ ["a.mp4", "b.mp4", "c.mp4"])
+ # warm the cache for a and c only: delete b's output so it must be redone
+ Path(first["results"][1]["output"]).unlink()
+ second = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--jobs", jobs, "--json").stdout)
+ self.assertEqual([Path(r["file"]).name for r in second["results"]],
+ ["a.mp4", "b.mp4", "c.mp4"])
+ self.assertTrue(second["results"][0].get("cached"))
+ self.assertFalse(second["results"][1].get("cached"))
+ self.assertTrue(second["results"][2].get("cached"))
+
+ def test_a_worker_that_raises_becomes_a_failed_row_not_a_dead_run(self):
+ """A die() inside a worker thread raised SystemExit through fut.result() and took the
+ process down before the summary and the table were printed. The one thing the user needs
+ -- which item failed and which succeeded -- was the thing they did not get."""
+ folder = OUT / "batch_worker_raise"
+ shutil.rmtree(folder, ignore_errors=True)
+ folder.mkdir(parents=True)
+ for name in ("ok1.mp4", "ok2.mp4", "ok3.mp4", "ok4.mp4"):
+ (folder / name).write_bytes(Path(self.src).read_bytes())
+ # a step that fails on every item: the run must still report all four rows
+ recipe = folder / "batch.json"
+ recipe.write_text(json.dumps({
+ "glob": "*.mp4", "output_dir": "out", "suffix": "_w",
+ "steps": [["cut.py", "{in}", "--start", "99", "--end", "120", "-o", "{out}"]]}))
+ r = script("batch.py", folder, "--recipe", recipe, "--fast", "--jobs", "2",
+ "--json", expect_fail=True)
+ data = json.loads(r.stdout)
+ self.assertEqual(len(data["results"]), 4)
+ self.assertEqual([Path(x["file"]).name for x in data["results"]],
+ ["ok1.mp4", "ok2.mp4", "ok3.mp4", "ok4.mp4"])
+ self.assertTrue(all(not x["ok"] for x in data["results"]))
+ self.assertEqual(data["error"]["kind"], "verification")
+
+ def test_jobs_is_capped_by_cpu_count(self):
+ folder, recipe = self._jobs_folder("batch_jobs_cap", count=2)
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--jobs", "999", "--json").stdout)
+ self.assertEqual(data["jobs_requested"], 999)
+ self.assertLessEqual(data["jobs"], min(os.cpu_count() or 1, 8))
+ self.assertGreaterEqual(data["jobs"], 1)
+
+ def test_jobs_auto_resolves_to_a_number(self):
+ folder, recipe = self._jobs_folder("batch_jobs_auto", count=2)
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--jobs", "auto", "--json").stdout)
+ self.assertLessEqual(data["jobs"], min(os.cpu_count() or 1, 4))
+
+ def test_jobs_rejects_a_non_number(self):
+ folder, recipe = self._jobs_folder("batch_jobs_bad", count=1)
+ r = script("batch.py", folder, "--recipe", recipe, "--fast", "--jobs", "lots",
+ "--json", expect_fail=True)
+ self.assertEqual(json.loads(r.stdout)["error"]["kind"], "input")
+
+ def test_jobs_share_one_timeout_budget(self):
+ """--timeout is the batch's limit, not each item's: a queue of files cannot quietly take
+ one timeout each.
+
+ The budget is a few milliseconds, so it has certainly expired by the time the first item
+ would be submitted -- the probe, the silence/collision pre-flight and the cache read all
+ run before it. That makes the test a statement about the deadline logic rather than a
+ race between --timeout and however fast this machine encodes: the previous version gave
+ six real encodes one second and passed only when the machine was slow enough.
+ """
+ for jobs in ("1", "2"):
+ with self.subTest(jobs=jobs):
+ folder, recipe = self._jobs_folder(f"batch_jobs_timeout_{jobs}", count=6)
+ r = script("batch.py", folder, "--recipe", recipe, "--timeout", "0.005",
+ "--force", "--jobs", jobs, "--json", expect_fail=True)
+ data = json.loads(r.stdout)
+ self.assertEqual(data["error"]["kind"], "timeout")
+ self.assertEqual(data["exit_code"], 124)
+ self.assertTrue(data["timed_out"])
+ skipped = [x for x in data["results"] if x.get("skipped") == "timeout"]
+ self.assertTrue(skipped)
+ # every item is still in the table, in file order, none of them claiming success
+ self.assertEqual(len(data["results"]), 6)
+ self.assertEqual([Path(x["file"]).name for x in data["results"]],
+ sorted(Path(x["file"]).name for x in data["results"]))
+ self.assertFalse(any(x["ok"] for x in skipped))
+
+ def test_a_stated_timeout_is_the_batchs_budget_but_the_default_is_not(self):
+ """The shared deadline applies when a --timeout was stated, or when --jobs > 1 asked for
+ the batch to be treated as one piece of work. The default sequential path keeps 1.16's
+ per-item ceiling, so a long folder is never cut off part-way by a flag nobody passed."""
+ folder, recipe = self._jobs_folder("batch_default_budget", count=2)
+ data = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--force", "--json").stdout)
+ self.assertEqual(data["processed"], 2)
+ self.assertFalse(data["timed_out"])
+
+ def test_jobs_cache_entries_survive_concurrency(self):
+ folder, recipe = self._jobs_folder("batch_jobs_cache", count=6)
+ first = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast", "--force",
+ "--jobs", "4", "--json").stdout)
+ self.assertEqual(first["processed"], 6)
+ again = json.loads(script("batch.py", folder, "--recipe", recipe, "--fast",
+ "--jobs", "4", "--json").stdout)
+ self.assertTrue(all(x.get("cached") for x in again["results"]),
+ "every entry written under concurrency must be served back")
+
+ def test_jobs_gives_each_item_its_own_work_dir(self):
+ folder, recipe = self._jobs_folder("batch_jobs_work", count=3)
+ work = OUT / "batch_jobs_work_dir"
+ script("batch.py", folder, "--recipe", recipe, "--fast", "--force", "--jobs", "3",
+ "--work", work)
+ subdirs = sorted(p.name for p in work.iterdir() if p.is_dir())
+ self.assertEqual(len(subdirs), 3)
+ # ... and a serial run keeps the flat layout it always had
+ work2 = OUT / "batch_jobs_work_serial"
+ script("batch.py", folder, "--recipe", recipe, "--fast", "--force", "--jobs", "1",
+ "--work", work2)
+ self.assertEqual([p.name for p in work2.iterdir() if p.is_dir()], [])
+
+
+ # --------------------------------------------------- 1.17: render.py --cache DIR / --from
+ def _cache_project(self, name, preset="x"):
+ proj = OUT / f"{name}.json"
+ proj.write_text(json.dumps({
+ "output": str(OUT / f"{name}_out.mp4"),
+ "clips": [{"src": str(self.src), "in": 0, "out": 3}],
+ "export": {"preset": preset}}), encoding="utf-8")
+ return proj
+
+ def test_render_cache_hit_skips_the_encode(self):
+ cdir = OUT / "rcache_hit"
+ shutil.rmtree(cdir, ignore_errors=True)
+ proj = self._cache_project("cache_hit")
+ first = json.loads(script("render.py", proj, "--cache", cdir, "--json").stdout)
+ self.assertEqual(first["cache"]["hits"], [])
+ self.assertGreater(len(first["commands"]), 0)
+ second = json.loads(script("render.py", proj, "--cache", cdir, "--json").stdout)
+ self.assertIn("export", second["cache"]["hits"])
+ self.assertEqual(second["cache"]["misses"], [])
+ self.assertEqual(second["commands"], [])
+ self.assertEqual(second["stages"], first["stages"]) # a cached stage still happened
+ self.assertTrue(Path(second["output"]).exists())
+
+ def test_render_cache_misses_when_a_stage_arg_changes(self):
+ cdir = OUT / "rcache_arg"
+ shutil.rmtree(cdir, ignore_errors=True)
+ script("render.py", self._cache_project("cache_arg"), "--cache", cdir)
+ changed = json.loads(script("render.py", self._cache_project("cache_arg", preset="reels"),
+ "--cache", cdir, "--json").stdout)
+ self.assertIn("clips", changed["cache"]["hits"]) # earlier stage unchanged
+ self.assertIn("export", changed["cache"]["misses"]) # the one that changed re-runs
+
+ def test_render_cache_key_never_crosses_an_ffmpeg_or_skill_version(self):
+ """The version lines are IN the key: a different build misses rather than being asked to
+ trust an artifact it did not write."""
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ render = importlib.import_module("render")
+ argv = ["in.mp4", "-o", "out.mp4", "--preset", "x"]
+ render.CACHE["ffmpeg"] = "7.1"
+ a = render.cache_key("export", "export.py", argv, [])
+ render.CACHE["ffmpeg"] = "5.1"
+ b = render.cache_key("export", "export.py", argv, [])
+ self.assertNotEqual(a, b)
+ render.CACHE["ffmpeg"] = "7.1"
+ self.assertEqual(render.cache_key("export", "export.py", argv, []), a)
+ self.assertNotEqual(a, render.cache_key("export", "export.py", argv + ["--crf", "20"], []))
+
+ def test_render_cache_never_serves_a_fast_draft_as_the_delivery(self):
+ """--fast rewrites every child's preset to veryfast, and child_args() appends it AFTER
+ the arguments the stage built -- so it was outside the key. A cached draft was handed
+ back to a run that did not ask for a draft, with cache.hits calling it a legitimate
+ reuse."""
+ cdir = OUT / "rcache_fast"
+ shutil.rmtree(cdir, ignore_errors=True)
+ proj = self._cache_project("cache_fast")
+ draft = json.loads(script("render.py", proj, "--cache", cdir, "--fast",
+ "--json").stdout)
+ self.assertEqual(draft["cache"]["hits"], [])
+ final = json.loads(script("render.py", proj, "--cache", cdir, "--json").stdout)
+ self.assertEqual(final["cache"]["hits"], [],
+ "a --fast draft must not be served to a run that did not ask for one")
+ self.assertGreater(len(final["commands"]), 0)
+ # ... and asking for the draft again does hit
+ again = json.loads(script("render.py", proj, "--cache", cdir, "--fast", "--json").stdout)
+ self.assertTrue(again["cache"]["hits"])
+
+ def test_render_cache_key_covers_the_ffmpeg_build_and_the_container(self):
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ render = importlib.import_module("render")
+ argv = ["in.mp4", "-o", "<out>", "--preset", "x"]
+ base = render.cache_key("export", "export.py", argv, [], "out.mp4")
+ self.assertNotEqual(base, render.cache_key("export", "export.py", argv, [], "out.mkv"),
+ "the container is part of what the stage produces")
+ # the banner, not major.minor: two 7.1.x builds differ in it
+ self.assertRegex(render.ffmpeg_banner(), r"version")
+ saved = render.CACHE.get("ffmpeg")
+ try:
+ render.CACHE["ffmpeg"] = "ffprobe version 7.1.1-0ubuntu1"
+ a = render.cache_key("export", "export.py", argv, [], "out.mp4")
+ render.CACHE["ffmpeg"] = "ffprobe version 7.1.2-0ubuntu1"
+ self.assertNotEqual(a, render.cache_key("export", "export.py", argv, [], "out.mp4"))
+ finally:
+ render.CACHE["ffmpeg"] = saved
+
+ def test_render_cache_writes_nothing_under_dry_run(self):
+ cdir = OUT / "rcache_dry"
+ shutil.rmtree(cdir, ignore_errors=True)
+ data = json.loads(script("render.py", self._cache_project("cache_dry"), "--cache", cdir,
+ "--dry-run", "--json").stdout)
+ self.assertEqual(sorted(p.name for p in cdir.iterdir()), [])
+ self.assertIn("cache", data)
+
+ def test_render_from_without_a_cache_refuses(self):
+ r = script("render.py", self._cache_project("cache_from"), "--from", "captions",
+ "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("--cache", err["message"])
+
+ def test_render_from_refuses_when_an_earlier_stage_is_not_cached(self):
+ cdir = OUT / "rcache_from_empty"
+ shutil.rmtree(cdir, ignore_errors=True)
+ r = script("render.py", self._cache_project("cache_from2"), "--cache", cdir,
+ "--from", "export", "--json", expect_fail=True)
+ err = json.loads(r.stdout)["error"]
+ self.assertEqual(err["kind"], "input")
+ self.assertIn("clips", err["message"])
+
+ def test_render_without_cache_is_unchanged(self):
+ data = json.loads(script("render.py", self._cache_project("cache_none"),
+ "--dry-run", "--json").stdout)
+ self.assertIsNone(data.get("cache"))
+
+
def test_batch_recipe_and_cache(self):
folder = OUT / "batch_in"
folder.mkdir(exist_ok=True)
diff --git a/tests/test_picture.py b/tests/test_picture.py
index d0e7b5b..83fb321 100644
--- a/tests/test_picture.py
+++ b/tests/test_picture.py
@@ -14,6 +14,7 @@ import shutil
import subprocess
import sys
import unittest
+import unittest.mock
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
@@ -1667,6 +1668,149 @@ class PictureTests(MediaFixtures):
self.assertEqual(proc.returncode, 0, f"{name}: {proc.stderr[-300:]}")
self.assertIn("usage:", proc.stdout)
+ # ------------------------------------------- 1.17: the size is fitted before the cue splits
+ def _fit_cues(self):
+ cues = OUT / "cues_fit.txt"
+ cues.write_text("0:00-0:03 A third line the tool times for me\n"
+ "0:03-0:06 Segunda l\u00ednea de subt\u00edtulos\n", encoding="utf-8")
+ return cues
+
+ def test_caption_fit_size_shrinks_instead_of_splitting_the_sentence(self):
+ """e2e: at the TikTok caption size these cues needed four lines and were split into
+ consecutive cues (eval 17). 1.17 drops the size until they fit instead."""
+ vert, cues = self._vertical(), self._fit_cues()
+ out = OUT / "cap_fit.mp4"
+ data = json.loads(script("caption.py", vert, "--text", cues, "--platform", "tiktok",
+ "--max-lines", "2", "--animate", "fade",
+ "--font", "DejaVu Sans", "--write-ass", OUT / "cap_fit.ass",
+ "-o", out, "--json").stdout)
+ cap = data["caption"]
+ self.assertEqual(cap["fit_size"], "auto")
+ self.assertEqual(cap["fit_scope"], "file")
+ self.assertLess(cap["size_used"], cap["size_requested"])
+ self.assertEqual(cap["size_floor"], 13)
+ self.assertGreaterEqual(cap["shrunk"], 1)
+ self.assertFalse(cap["fit_exhausted"])
+ self.assertEqual(cap["split"], 0) # the whole point: nothing was split
+ m = probe(str(out))
+ self.assertEqual((m["video"]["width"], m["video"]["height"]), (1080, 1920))
+ # the ASS on disk carries the shrunk size, and each cue is one cue of two lines
+ ass = (OUT / "cap_fit.ass").read_text(encoding="utf-8-sig")
+ style = next(l for l in ass.splitlines() if l.startswith("Style:"))
+ self.assertEqual(int(style.split(",")[2]), round(cap["size_used"] * 1920 / 288))
+ dialogue = [l for l in ass.splitlines() if l.startswith("Dialogue:")]
+ self.assertEqual(len(dialogue), 2)
+ for line in dialogue:
+ self.assertEqual(line.count("\\N"), 1)
+
+ def test_caption_scope_cue_lays_each_cue_out_at_its_own_size(self):
+ """The regression: with scope=cue every cue was wrapped to the FILE's budget -- the
+ budget of the smallest size, which is the widest line in em -- and then drawn at its own
+ larger size, so lines ran off the side of the frame. Each cue is now laid out at the size
+ it is drawn at, and no rendered line exceeds the safe width."""
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ text_mod = importlib.import_module("_common.text")
+ vert = self._vertical()
+ cues = OUT / "cues_scope.txt"
+ cues.write_text("0:00-0:03 A third line the tool times for me\n"
+ "0:03-0:05 Hi\n"
+ "0:05-0:08 Una tercera l\u00ednea con tiempos autom\u00e1ticos\n",
+ encoding="utf-8")
+ ass = OUT / "cap_scope.ass"
+ data = json.loads(script("caption.py", vert, "--text", cues, "--platform", "tiktok",
+ "--max-lines", "2", "--animate", "fade", "--font", "DejaVu Sans",
+ "--fit-size-scope", "cue", "--write-ass", ass,
+ "-o", OUT / "cap_scope.mp4", "--json").stdout)
+ cap = data["caption"]
+ self.assertEqual(cap["fit_scope"], "cue")
+ self.assertEqual(cap["split"], 0)
+
+ body = ass.read_text(encoding="utf-8-sig")
+ style_px = int(next(l for l in body.splitlines() if l.startswith("Style:")).split(",")[2])
+ dialogue = [l for l in body.splitlines() if l.startswith("Dialogue:")]
+ self.assertEqual(len(dialogue), 3)
+ # at least two different drawn sizes, which is the whole point of the flag
+ drawn = set()
+ safe_px = 1080 * text_mod.SAFE_WIDTH_FRACTION
+ for line in dialogue:
+ payload = line.split(",,0,0,0,,", 1)[1]
+ override = re.search(r"\{\\fs(\d+)\}", payload)
+ px = int(override.group(1)) if override else style_px
+ drawn.add(px)
+ body_text = re.sub(r"\{[^}]*\}", "", payload)
+ for rendered in body_text.split("\\N"):
+ width = text_mod.text_width_em(rendered) * px
+ self.assertLessEqual(width, safe_px,
+ f"{rendered!r} is {width:.0f}px at size {px}, safe {safe_px:.0f}px")
+ self.assertGreaterEqual(len(drawn), 2)
+
+ def test_caption_fit_size_off_is_byte_identical_to_1_16_1(self):
+ """The stability answer: one flag restores the old ASS byte for byte. The pinned file
+ was generated by 1.16.1's own caption.py (tests/fixtures/caption_1_16_1_fitsize_off.ass)."""
+ vert, cues = self._vertical(), self._fit_cues()
+ ass = OUT / "cap_fit_off.ass"
+ script("caption.py", vert, "--text", cues, "--platform", "tiktok", "--max-lines", "2",
+ "--animate", "fade", "--font", "DejaVu Sans", "--fit-size", "off",
+ "--write-ass", ass, "-o", OUT / "cap_fit_off.mp4")
+ pinned = Path(__file__).resolve().parent / "fixtures" / "caption_1_16_1_fitsize_off.ass"
+ self.assertEqual(ass.read_bytes(), pinned.read_bytes())
+
+ def test_caption_fit_size_auto_leaves_an_explicit_size_alone(self):
+ vert, cues = self._vertical(), self._fit_cues()
+ data = json.loads(script("caption.py", vert, "--text", cues, "--platform", "tiktok",
+ "--size", "30", "--max-lines", "2",
+ "-o", OUT / "cap_fit_explicit.mp4", "--json").stdout)
+ self.assertEqual(data["caption"]["size_used"], 30)
+ self.assertEqual(data["caption"]["size_requested"], 30)
+
+ def test_caption_plan_before_the_input_exists_uses_the_platform_frame(self):
+ """A plan that names a different FontSize from the run that executes it is not a plan.
+ With --platform there is a real frame to fit against; without one there is nothing, and
+ size_used says null rather than presenting the requested size as the fitted one."""
+ cues = self._fit_cues()
+ missing = OUT / "not_yet_recorded.mp4"
+ if missing.exists():
+ missing.unlink()
+ planned = json.loads(script("caption.py", missing, "--text", cues, "--platform", "tiktok",
+ "--dry-run", "-o", OUT / "plan_fit.mp4", "--json").stdout)
+ real = json.loads(script("caption.py", self._vertical(), "--text", cues,
+ "--platform", "tiktok", "--dry-run",
+ "-o", OUT / "plan_fit2.mp4", "--json").stdout)
+ self.assertEqual(planned["caption"]["size_used"], real["caption"]["size_used"])
+ self.assertLess(planned["caption"]["size_used"], planned["caption"]["size_requested"])
+ self.assertEqual(planned["caption"]["size_source"], "platform-frame")
+ burn = next(c for c in planned["commands"] if "FontSize" in c)
+ self.assertIn(f"FontSize={planned['caption']['size_used']}", burn)
+
+ # no platform and no input: nothing to measure against, so nothing is claimed
+ blind = json.loads(script("caption.py", missing, "--text", cues, "--dry-run",
+ "-o", OUT / "plan_fit3.mp4", "--json").stdout)
+ self.assertIsNone(blind["caption"]["size_used"])
+ self.assertEqual(blind["caption"]["size_requested"], 24)
+
+ def test_caption_min_size_above_size_is_an_input_refusal(self):
+ vert, cues = self._vertical(), self._fit_cues()
+ out = OUT / "cap_fit_refuse.mp4"
+ if out.exists():
+ out.unlink()
+ r = script("caption.py", vert, "--text", cues, "--size", "24", "--min-size", "30",
+ "-o", out, "--json", expect_fail=True)
+ data = json.loads(r.stdout)
+ self.assertEqual(data["error"]["kind"], "input")
+ self.assertIn("--min-size", data["error"]["message"])
+ self.assertFalse(out.exists())
+
+ def test_caption_mux_is_unaffected_by_the_fitter(self):
+ """Soft subtitles carry no size, so --mode mux writes the SRT 1.16 wrote."""
+ vert, cues = self._vertical(), self._fit_cues()
+ a, b = OUT / "cap_mux_auto.mkv", OUT / "cap_mux_off.mkv"
+ script("caption.py", vert, "--text", cues, "--mode", "mux", "--platform", "tiktok",
+ "--write-srt", OUT / "cap_mux_auto.srt", "-o", a)
+ script("caption.py", vert, "--text", cues, "--mode", "mux", "--platform", "tiktok",
+ "--fit-size", "off", "--write-srt", OUT / "cap_mux_off.srt", "-o", b)
+ self.assertEqual((OUT / "cap_mux_auto.srt").read_bytes(),
+ (OUT / "cap_mux_off.srt").read_bytes())
class ScriptFontTests(unittest.TestCase):
"""1.12: which script text is written in, and a font file that actually covers it."""
@@ -2519,5 +2663,120 @@ class PhraseWrapTests(unittest.TestCase):
self.assertEqual(C.break_penalty("\u3042", "\u3063", "ja"), 1.0) # small kana may not start a line
+class FitSizeTests(unittest.TestCase):
+ """1.17: the caption size is fitted to the cue before the cue is split (eval 17).
+
+ Pure -- no fixtures, no ffmpeg. The geometry and the requested size come from the platform
+ table, never from a pinned float, so the lock moves if the platform's caption default moves.
+ """
+
+ # The eval-17 cues, verbatim. cw1 and dl1 carry the same English sentence.
+ CW1 = "A third line the tool times for me"
+ DL4 = "Una tercera l\u00ednea con tiempos autom\u00e1ticos"
+ DL4B = "Segunda l\u00ednea de subt\u00edtulos"
+
+ def setUp(self):
+ sys.path.insert(0, str(SCRIPTS))
+ import importlib
+ self.caption = importlib.import_module("caption")
+ self._platforms = importlib.import_module("_platforms")
+ frame = self._platforms.PLATFORMS["tiktok"]["frame"]
+ self.W, self.H = frame["w"], frame["h"]
+ self.SIZE = self._platforms.caption_defaults("tiktok")["size"]
+
+ def _fit(self, cues, **kw):
+ kw.setdefault("size", self.SIZE)
+ kw.setdefault("max_lines", 2)
+ return self.caption.fit_size(cues, play_w=self.W, play_h=self.H, **kw)
+
+ def _wrap(self, text, size):
+ em = self.caption.line_em_for_size(size, self.W, self.H)
+ return self.caption.wrap_text(text, em)
+
+ def test_the_floor_is_four_and_a_half_percent_of_the_frame(self):
+ C = self.caption
+ self.assertEqual(C.MIN_CAPTION_FRACTION, 0.045)
+ self.assertEqual(self._platforms.ass_units(C.MIN_CAPTION_FRACTION), 13)
+ self.assertEqual(self._fit([self.CW1])["floor"], 13)
+
+ def test_eval17_cues_fit_two_lines_at_the_shrunk_size(self):
+ """The regression lock. At the TikTok caption size every one of these needs three or
+ four lines, so --max-lines 2 split each into consecutive cues; shrinking fits them."""
+ # At the requested size, all three are over the line budget -- the defect.
+ for text in (self.CW1, self.DL4, self.DL4B):
+ self.assertGreater(len(self._wrap(text, self.SIZE)), 2)
+
+ cw1 = self._fit([self.CW1])
+ self.assertEqual(cw1["size"], 16)
+ self.assertEqual(self._wrap(self.CW1, 16), ["A third line the", "tool times for me"])
+
+ dl4 = self._fit([self.DL4])
+ self.assertEqual(dl4["size"], 13)
+ self.assertEqual(self._wrap(self.DL4, 13),
+ ["Una tercera l\u00ednea con", "tiempos autom\u00e1ticos"])
+
+ # 19, not 18: 18 is simply the next size the spec's coarse table sampled. 19 is the
+ # largest size at which this cue fits two lines, and the fitter returns the largest.
+ dl4b = self._fit([self.DL4B])
+ self.assertEqual(dl4b["size"], 19)
+ self.assertEqual(self._wrap(self.DL4B, 19), ["Segunda l\u00ednea", "de subt\u00edtulos"])
+
+ whole = self._fit([self.CW1, self.DL4, self.DL4B])
+ self.assertEqual(whole["size"], 13)
+ self.assertEqual(whole["scope"], "file")
+ self.assertTrue(whole["fits"])
+ self.assertEqual(whole["shrunk"], 3)
+ for text in (self.CW1, self.DL4, self.DL4B):
+ self.assertLessEqual(len(self._wrap(text, whole["size"])), 2)
+
+ def test_fit_size_never_goes_below_the_floor(self):
+ # One unbreakable run far wider than the line at any size in range.
+ fit = self._fit(["Donaudampfschifffahrtsgesellschaftskapitaenspatentpruefung " * 3])
+ self.assertEqual(fit["size"], 13)
+ self.assertEqual(fit["floor"], 13)
+ self.assertFalse(fit["fits"])
+
+ def test_fit_size_leaves_a_cue_that_already_fits_alone(self):
+ fit = self._fit(["Short line"])
+ self.assertEqual(fit["size"], self.SIZE)
+ self.assertEqual(fit["shrunk"], 0)
+
+ def test_fit_size_honours_an_explicit_min_size(self):
+ fit = self._fit([self.DL4], min_size=16)
+ self.assertEqual(fit["floor"], 16)
+ self.assertEqual(fit["size"], 16)
+ self.assertFalse(fit["fits"])
+
+ def test_fit_size_without_geometry_changes_nothing(self):
+ fit = self.caption.fit_size([self.DL4], size=self.SIZE, max_lines=2,
+ play_w=None, play_h=None)
+ self.assertEqual(fit["size"], self.SIZE)
+ self.assertEqual(fit["shrunk"], 0)
+
+ def test_scope_cue_gives_per_cue_sizes(self):
+ fit = self._fit([self.CW1, self.DL4, "Short line"], scope="cue")
+ self.assertEqual(fit["scope"], "cue")
+ self.assertEqual(fit["per_cue"][0], 16)
+ self.assertEqual(fit["per_cue"][1], 13)
+ self.assertEqual(fit["per_cue"][2], self.SIZE)
+ self.assertEqual(fit["size"], min(fit["per_cue"].values()))
+
+ def test_fit_size_accepts_cue_tuples(self):
+ tuples = [(0.0, 1.0, self.CW1), (1.0, 2.0, self.DL4)]
+ self.assertEqual(self._fit(tuples)["size"], self._fit([self.CW1, self.DL4])["size"])
+
+ def test_fit_size_is_pure(self):
+ """No subprocess, no ffmpeg, no ffprobe: the size is a text measurement."""
+ import importlib
+ _common = importlib.import_module("_common")
+ def boom(*a, **k):
+ raise AssertionError("fit_size ran a subprocess")
+ with unittest.mock.patch.object(_common, "run", boom), \
+ unittest.mock.patch.object(_common, "run_analysis", boom), \
+ unittest.mock.patch.object(subprocess, "run", boom), \
+ unittest.mock.patch.object(subprocess, "Popen", boom):
+ self.assertEqual(self._fit([self.CW1])["size"], 16)
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)