Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

feat: Parakeet speech engines for --transcribe (#312)

Shogo Kajiyama231130bf69de
diff --git a/CHANGELOG.md b/CHANGELOG.md
index afd2868..2a23f93 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,12 @@
## Unreleased
-(nothing yet)
+- feat: Parakeet speech engines, built by [@rymalia](https://github.com/rymalia) in #305. `parakeet-mlx` and `parakeet.cpp` (`parakeet-cli` + a `.gguf`) for `caption.py --transcribe` and `silence.py --filler --transcribe`; `--engine` / `FFMPEG_SKILL_ASR_ENGINE` pick one, and a named engine runs as asked. `auto` runs Parakeet for English speech (an English `--language`, else whisper.cpp's language detector) and Whisper otherwise. Speech whose language nobody named and nothing here could detect goes to a Whisper engine that can run; Parakeet runs on English assumed only when there is none or every one failed, and the result's top-level `notes` then says the transcript is wrong if the speech is not English (new for `silence.py`). Results carry `transcription` (engine, model, language, routing) in every caption `--mode`; `silence.py`'s `filler.source` is `parakeet:ENGINE` for them. A Parakeet engine whose output is not a readable transcript is a failed run and the next engine is tried; its own empty answer is the `no_speech` refusal. `transcribe_result()` / `transcribe_words_result()` return what a call used as a `Transcription`, and `transcribe()` / `transcribe_words()` keep their return shapes. New optional contract capability `external:parakeet`.
+- fix: whisper.cpp is told to detect the language (`-l auto`) when `--language` / `--filler-lang` is not given. It was given no `-l`, and its own default is English, so `caption.py --transcribe` and `silence.py --filler --transcribe` decoded speech in any other language as English. The result's `transcription.language` stays `null` for a language the engine identified itself.
+- fix: the "no local speech-to-text engine found" refusal, with its install lines, now only means no engine was found. A whisper.cpp that was installed but failed, with nothing after it, got that refusal; now the refusal names each engine that was found and why it did not transcribe (`kind: input`, `reason: "engine_failed"` or `"english_only"`, `engines[].detail` with the engine's own error line).
+- fix: a faster-whisper that raises while loading or running its model (an offline first run, a broken install) is a failed engine: `caption.py --transcribe` logs "faster-whisper found but failed" and tries the next engine instead of refusing with "faster-whisper found no speech", and `silence.py --filler --transcribe` can fall back to Parakeet. The exception used to be swallowed in the engine's thread. An engine that ran and found no segment is still the `no_speech` refusal.
+- docs: parakeet-mlx downloads its default model (`mlx-community/parakeet-tdt-0.6b-v2`) from Hugging Face on first use, as faster-whisper does, and openai-whisper downloads its model from OpenAI. The install hint, README, docs/contract.md and references/scripts.md now say so, and that the audio never leaves the machine (inference is local). The skill's own code opens no connection, but faster-whisper runs inside the skill's process, so its first-run download is made from that process. The refusal's "all run offline" is gone: a first run that downloads is not offline.
+- feat(contract): `execution.model_downloads` says which speech engine fetches its model on first use, from where, and in which process (faster-whisper in the skill's own; parakeet-mlx and openai-whisper in their own child processes; whisper.cpp and parakeet.cpp never). `execution.network: false` keeps its meaning, the skill's own code, and no longer reads as a promise that a first `--transcribe` is network-free.
## 2.6.0
diff --git a/README.md b/README.md
index 88973b9..6ae843b 100644
--- a/README.md
+++ b/README.md
@@ -82,7 +82,7 @@ An agent that "knows FFmpeg" still guesses: it assumes a frame rate, picks a cod
- **Structured tools, not shell strings.** Each operation is a script with typed arguments. Nothing runs through a shell; no filter graph is accepted from the caller.
- **A contract the agent can read.** `contract --json` states, for every tool, what it takes, what it writes, which FFmpeg components it needs and how the result is verified. The MCP surface is derived from it.
- **Verification after execution.** The result is probed, checked against the destination's spec and, when the picture changed, looked at as a contact sheet.
-- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a whisper is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing.
+- **Local first.** No cloud, no API keys, no Python dependencies. Optional local transcription is used when a speech engine (Parakeet or a whisper) is installed, never required — by `caption.py --transcribe` and, since 1.17, by `silence.py --filler --transcribe`; both take a transcript you already have instead, and both refuse with the install lines rather than guessing. The engine runs on your machine and the audio is never uploaded; parakeet-mlx, faster-whisper and openai-whisper download their model weights the first time they run (parakeet-mlx's default from Hugging Face).
## Quick start
@@ -371,7 +371,7 @@ The short list for humans. The agent-facing version, with the reasoning, is the
- **Phrase-aware caption breaking (1.16).** `caption.py`/`graphics.py --wrap phrase` (the default) never breaks inside a word or on the wrong side of a hyphen, never leaves a lone digit, kana or punctuation pair on a line, prefers Japanese sentence ends and particles over a mid-word break, and never ends a line on an article or preposition. All four are penalties over break positions that already fit, so no line is widened and the line count never changes; `--wrap measured` restores 1.15's width-only wrap. The text itself is never rewritten or shortened. Since 1.16.1 a Thai run and a katakana word are never broken inside (Thai writes no space inside a phrase and there is no dictionary: the break goes where you put a space or `|`), and a line wider than the safe width is reported as `overlong` with the fix named.
- **Caption size fitted to the cue (1.17, reachable from the templates since 1.17.1).** At a platform caption size a line holds about six em, so an ordinary sentence needs four lines and `--max-lines 2` used to cut it into consecutive cues — half the sentence arriving late. `caption.py --fit-size` (default `auto`) walks the size down until every cue fits, *before* laying the cues out, with a legibility floor of 4.5 % of the frame height (`--min-size`, default 13 ASS units). An explicit `--size` or a `brand.json` size is a statement about the look and is never overridden — which in 1.17.0 also silenced the fitter on every `render.py --template` run, since a template fills the size from the platform table; 1.17.1 marks that size as the default it is (`"fit_size": "on"` in the filled project), so the type shrinks and no cue is split. `--fit-size off` restores that earlier behaviour byte for byte, and the caption text is still never rewritten to make it fit.
- **Beat-synced cuts (1.17).** `scenes.py --beats` reports the measured grid — tempo, beat times, and a confidence built from how far the winning autocorrelation lag stands above the others and how many onsets land on it. `cut.py --snap beats` (and a `"snap"` block in a `render.py` project) moves in/out points to the nearest beat within `--snap-tolerance` — and only onto the grid points a measured onset actually marks, never onto the regular grid's continuation through a passage with no music in it. Below `--min-confidence` it **refuses**: a cut point may move to a measured beat and may not appear from one, so speech and ambience get an honest "no steady pulse here" instead of an invented grid.
-- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the three installs.
+- **Filler words (1.17).** `silence.py --filler --words transcript.json` removes "um" and "uh" through the same cut graph the silences use. Never without measured word timings — there is no heuristic that finds an "um" without them that would not also cut real speech — and `like`, `tipo` and `cioè` are deliberately not in the built-in lists, because a discourse marker is a content word. Whisper stays optional: `--transcribe` with no engine installed refuses and names the five installs.
- **Throughput (1.17).** `batch.py --jobs N` runs several files at once, capped at `min(N, cpu_count, 8)` and sharing one `--timeout` budget rather than one per item; the per-item table keeps its order. `render.py --cache DIR` reuses stages whose inputs and arguments did not change, so swapping an export preset re-runs export only. The cache is opt-in with no default directory, and the ffmpeg, skill and contract versions are part of every key, so a cache is never reused across them.
- **Audiogram (1.16).** `waveform.py --image cover.png` (or `render.py --template audiogram`) puts the waveform over a still plate for an episode that has no picture, with `--platform` for the frame, `--title` and burnt-in captions. The image is a local file you give: nothing is fetched and no cover art is ever invented.
- **Emoji in captions and titles (1.15).** `caption.py`/`graphics.py --emoji-assets DIR` composites a PNG per emoji (Twemoji/Noto naming, `1f389.png`) on top of the text, because drawtext cannot load a colour emoji font at all and an installed one does not prove libass will draw it in colour — `doctor --json`'s `fonts.emoji` answers that from a render probe. Without assets the run still succeeds and reports `mode: mono`. Nothing is ever downloaded.
diff --git a/SKILL.md b/SKILL.md
index d23e38d..e858d2a 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -32,7 +32,7 @@ Ask a short question only when the answer changes the output materially and the
- **Destination** decides aspect, length limit, loudness and codec; "for Reels" answers all four and names a template. No destination and a plain cut/caption: keep the source format and say so. If the user says "export", "post" or "deliver", ask where.
- **Duration** ("make it 60 s") without a method: speed up for ≤1.5× changes, trim otherwise, say which. Ask when the content is a talk (trimming loses words) and the change is large.
-- **Captions** without a text source: `--transcribe` if a local whisper exists, else ask for the text or a timed file; never invent dialogue.
+- **Captions** without a text source: `--transcribe` if a local engine exists, else ask for the text or a timed file; never invent dialogue.
- **Fonts and brand**: if the user mentions a brand, colours or "our font", ask for or create `brand.json` once, reuse it.
- **CJK / non-Latin text**: let the tool pick the font by script (`--font` turns it off); `--lang ja|ko` for Han-only text. `doctor --json` `.fonts.scripts` says what renders here. Tofu is a failed job.
- **Crop position** for `--fit crop`: centre by default, but when the request or the source names an off-centre subject ("keep the product on the right", someone visibly off-centre in the sheet) use `--crop-x`/`--crop-y` (0=left/top, 1=right/bottom) instead of guessing centre, or ask which edge to keep.
@@ -97,7 +97,7 @@ Timestamp flags (`--start`, `--end`, `--at`, `--from`, `--duration`, `--offset`,
| "caption it with these lines" (text with times) | `caption.py input.mp4 --text cues.txt` |
| "keep the subtitles toggleable", "mux in an SRT" | `caption.py input.mp4 --srt subs.srt --mode mux`; repeat `--srt file:lang` for several languages, `.mkv` for more than two |
| "the captions are tiny / three lines on a Short", "don't chop the sentence" | `caption.py` shrinks the size until the cue fits `--max-lines` before splitting it (`--fit-size off` keeps `--size` and splits instead, `--min-size` the floor). Keep the template's `--max-lines` (2 on a vertical) and let the size drop; raising it to dodge a shrink stacks two words per line. A word still wider than the column at the floor is sliced at the edge (hyphen preferred), never rewritten, automatic; `broken_inside_word` in the JSON counts it |
-| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (needs a local whisper; else `--text`) |
+| "transcribe it and caption it" | `caption.py input.mp4 --transcribe --animate pop --karaoke` (Parakeet/whisper; else `--text`) |
| "TikTok-style captions with the words popping" | `caption.py input.mp4 --text cues.txt --animate pop --karaoke` |
| "our logo top-right", "a watermark" | `overlay.py input.mp4 --image logo.png --position top-right --scale 200` |
| "a title for the first 4 seconds" | `overlay.py input.mp4 --text "Title" --position top --start 0 --end 4 --fade 0.4` |
diff --git a/docs/contract.md b/docs/contract.md
index 78e4b64..d4f3e61 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -123,7 +123,11 @@ JSON would have been rewritten for that. The flat keys, typed per tool in `outpu
"entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
"not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
"requirements": {"python": ">=3.9 (standard library only)", "ffmpeg": ">=5.0", "ffprobe": ">=5.0"},
- "execution": {"shell": false, "arbitrary_executables": false, "network": false, "input_mutation": false}
+ "execution": {"shell": false, "arbitrary_executables": false, "network": false,
+ "model_downloads": {"when": "...", "in_process": {"faster-whisper": "..."},
+ "child_process": {"parakeet-mlx": "...", "openai-whisper": "..."},
+ "none": ["whisper.cpp", "parakeet.cpp"], "offline": "..."},
+ "input_mutation": false}
}
```
@@ -349,10 +353,10 @@ A project may now carry `"template"` (the name it was filled from) and `"frame":
## Capabilities
Names: `ffmpeg`, `ffprobe`, `encoder:<name>`, `filter:<name>`, `bsf:<name>`,
-`external:whisper`. `capabilities.required` is the union of every tool's required list;
+`external:whisper`, `external:parakeet`. `capabilities.required` is the union of every tool's required list;
`optional` the union of the conditional ones. With detection (the default)
`available`, `missing` and `missing_optional` are added from `doctor`, which reads
-`ffmpeg -encoders / -filters / -bsfs` and looks for a local whisper. Pass `--static`
+`ffmpeg -encoders / -filters / -bsfs` and looks for a local whisper or Parakeet. Pass `--static`
to omit detection. Nothing from the environment other than those lists and the
ffmpeg/ffprobe/python versions is printed; no environment variables, no paths (`doctor`'s
`hw.default_on` is a boolean derived from `FFMPEG_SKILL_HW`, never its value).
@@ -360,8 +364,29 @@ ffmpeg/ffprobe/python versions is printed; no environment variables, no paths (`
`external:whisper` is **optional** for two tools since 1.17: `caption.py`
(`--transcribe`) and `silence.py` (`--filler --transcribe`). Neither requires
it — both take a transcript the caller already has (`--srt`/`--words`), and
-both refuse with the same three install lines when asked to make one with no
+both refuse with the same install lines when asked to make one with no
engine present. Whisper is never a dependency of this skill.
+`external:parakeet` is the same kind of optional capability for the same two flags:
+parakeet-mlx, or parakeet-cli with a `.gguf` model it can find. Either engine family
+satisfies `--transcribe`; `--engine auto` (the default, or `FFMPEG_SKILL_ASR_ENGINE`) runs
+Parakeet for English speech and Whisper for every other language. When no `--language` is given
+and no detector can name it, a Whisper engine that can run takes it (whisper.cpp is passed
+`-l auto`: its own default is English). Parakeet then runs only on English *assumed* -- when no
+Whisper engine is installed, or every one failed, or `--model` names an English-only Parakeet
+model -- and the result's top-level `notes` says the transcript is wrong if the speech is not
+English; a multilingual (v3) Parakeet model assumes nothing and gets no note. The result's
+`transcription` says which engine, model and routing decision produced the cues (in every
+`--mode`, `mux` included).
+Every engine transcribes on the machine running the skill, and the audio is never uploaded. The
+skill's own code opens no network connection (`execution.network: false`), but an engine may
+fetch its model weights on first use: parakeet-mlx downloads its default
+`mlx-community/parakeet-tdt-0.6b-v2` from Hugging Face and openai-whisper its model from OpenAI,
+each in its own child process. faster-whisper is a Python library the skill runs inside its own
+process, so its first-run download from Hugging Face is made by the skill's process;
+`HF_HUB_OFFLINE=1` keeps it (and parakeet-mlx) on the cached copy after that. parakeet.cpp and
+whisper.cpp read a model file the user downloaded. `execution.model_downloads` says the same in the
+contract, engine by engine, so a caller that sandboxes the network can tell before a first
+`--transcribe` that faster-whisper's fetch comes from the skill's own process.
`doctor` has three states per capability. `available` and `missing` come from a listing
that was read; `unknown` means the listing that would prove the capability could not be
@@ -546,11 +571,19 @@ Per-tool keys added after 2.2.2, all additive:
| `short_segments`, `duplicates` | `join.py` (every run) | warnings, never refusals, and the join command is unchanged: `short_segments: [{index, path, duration}]` names each measured input shorter than 2 frames at the join's fps (an audio-only join: shorter than 0.05 s); `duplicates: [{path, indices}]` names a path (compared resolved) listed more than once — repeating a clip can be intended. `[]` when none; each non-empty one adds a `notes` line |
| `silent` | `waveform.py` (every run) | `true` when the input audio's whole-file peak is at or below `--silence-threshold` (default -50 dBFS), which draws a flat line; `--on-silent warn` (default) renders it anyway with a `notes` line, `fail` refuses (`kind: input`) before ffmpeg runs. `false` when audible, `null` under `--dry-run` (nothing measured). `verified` is unaffected |
| `encoder`, `hw`, `notes` | every re-encoding tool, `export.py`, `render.py`, `batch.py` (and its rows), `waveform.py` | `encoder`: the video encoder the last command ran (after any GPU→CPU fallback; under `--dry-run`, the planned one), `copy` for a stream copy; absent when no command encoded video. A tool that runs other tools reports its stages': `render.py` its stages and an executed plan's tool, `waveform.py` the title/caption stage that writes its file (its `commands` include that stage's). `hw` is present when VideoToolbox was asked for (`--hw`, `FFMPEG_SKILL_HW=1`, or a stage or `batch.py` item that reported it) and on `export.py` when `FFMPEG_SKILL_HW=1` is set but its delivery preset does not take it: `{requested, source: flag\|env, used, fallback, notes}`. `requested` says whether this run's encodes were asked onto VideoToolbox (`false` for that export), `used` whether the encoder that ran last is VideoToolbox (`null` under `--dry-run`, which ran nothing, and when the tool encoded nothing itself: `batch.py`, whose top-level `hw` gathers its items' `requested`/`source`/`fallback`/`notes` while each `results` row carries the item's own `encoder`, and `hw`/`notes` when VideoToolbox was asked for), `fallback` whether a job VideoToolbox refused was re-encoded on the CPU, and `notes` why. Only a VideoToolbox failure (the encoder could not open, or its compression session failed) is retried on the CPU; any other failure is reported as it happened, once. Every VideoToolbox encode adds a top-level `notes` line: its quality is not CRF-equivalent (`--quality` / a preset's CRF maps to `-q:v` approximately), and `verified` stays what it is for every tool, the measured properties of the output. A `kind: ffmpeg` failure under `--hw` / `FFMPEG_SKILL_HW` carries `encoder` and `hw` for the command that ran last (a stage's, when `render.py`/`waveform.py` re-raise it); when the command itself failed inside the run (not a tool's own check afterwards, such as `cut.py`'s), its message also names that encoder |
+| `transcription` | `caption.py --transcribe` (every `--mode`); inside `filler` for `silence.py --filler --transcribe` | `{routing, detected_language?, engine, model, language}`: the engine and model that made the transcript; `language` is the language the engine was told -- the one named (`--language`, `--filler-lang`), else the one auto's detector named -- and `null` when the engine identified it itself; `detected_language` is present when auto's whisper.cpp detector named one; `routing` says why `--engine auto` chose the engine (`"requested"` for a named engine), ending `, no Parakeet engine transcribed it` when auto chose Parakeet and a Whisper engine made the transcript after every Parakeet engine failed |
+| `notes` | `silence.py --filler --transcribe` (new for `silence.py`; `caption.py` already had it) | present when `--engine auto` ran the English-only Parakeet model on English *assumed* (no language given or detectable, and no Whisper engine installed or none that succeeded): the transcript is wrong if the speech is not English. `caption.py --transcribe` puts the same line in its `notes` |
`caption.py --transcribe` and `silence.py --filler --transcribe`: when a local speech engine ran
and produced no cue, the refusal is `"<engine> found no speech in <input>"` with `kind: input`,
`reason: "no_speech"` and `engine` in the failure document — distinct from the "no local
-speech-to-text engine found" refusal, which now only means no engine was found.
+speech-to-text engine found" refusal, which now only means no engine was found. When `--engine
+auto` found engines and none transcribed, the refusal is `kind: input` with `reason:
+"engine_failed"` (one ran and failed) or `"english_only"` (the only engines found were Parakeet
+engines auto passed over for a language that is not English), `engine` (the first one it names)
+and `engines: [{engine, reason, detail}]`, where `detail` carries an engine's own error line. A
+named `--engine` that is not installed is `kind: missing_tool`; one that is installed and failed
+is `kind: input`, `reason: "engine_failed"`, with `engine` and `detail`.
`check.py` also gains an informational `subtitles` row on **every** platform:
`PASS` when every soft subtitle stream carries a language tag, `WARN` when one
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index bdd4cd0..e76c106 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -949,3 +949,60 @@ not a new file format this tool would have to maintain.
re-encoded on the CPU by the fallback above, with ffmpeg's line in the note. Code:
`_common.runner.apple_silicon`; `doctor`'s `hw.platform_ok` uses the same check. Test:
`test_hw_runs_on_apple_silicon_hardware_under_a_rosetta_python`.
+
+## Unreleased — a second speech engine
+
+- **`auto` picks Parakeet only for English.** The default Parakeet model (tdt-0.6b-v2) is
+ English-only and transcribes other speech as English-shaped nonsense. A named `--language`
+ decides; without one whisper.cpp's detector (the smallest multilingual ggml model, a second
+ or less) decides. When neither can name the language, a Whisper engine that can run goes first
+ (Parakeet only if every Whisper engine fails): a fast English transcript is not worth a
+ confidently wrong one. Only when no Whisper engine is installed, or every one failed, does
+ Parakeet run on English assumed, and then the result's top-level `notes` says the transcript is
+ wrong if the speech is not English -- `transcription.routing` alone was too quiet for a caller
+ that reads `verified`. "Goes first" means a Whisper engine that detects the language: whisper.cpp
+ is passed `-l auto` when nothing named the language (its own default is `-l en`, which decoded
+ every unnamed language as English, in 2.5.1 too), else the named or detected one, and an
+ English-only Whisper model (`base.en`, `ggml-*.en.bin`) does not count as one.
+ A named `--engine parakeet-*` runs as asked, with no note, and so does a Parakeet model named
+ with `--model` or `PARAKEET_MODEL`: a multilingual (v3) one assumes nothing and gets no note; an
+ English-only one named with `--model` gets the note, pointing at a v3 model rather than at a
+ Whisper engine, which cannot load it. Measured on 8 min of LibriSpeech
+ (contributor's figures, Apple Silicon): parakeet-mlx v2 2.8% WER at ~120× real time, whisper
+ large-v3-turbo 2.4% at ~39×. Tests: `ParakeetRoutingTests`, `ParakeetEngineTests`,
+ `UndetectedLanguageTests`, `WhisperLanguageTests`, `AutoRoutingFallbackTests`.
+- **A Whisper engine that failed is a failure, not silence.** Undetected speech goes to Whisper
+ first, so a Whisper failure has to read as one for Parakeet to be tried after it. A
+ faster-whisper whose model would not load (an offline first run) raised inside its worker
+ thread, the exception was swallowed and the run was refused as "found no speech"; it now logs
+ "found but failed" and the next engine is tried, as a failed whisper.cpp always was. An engine
+ that ran and returned no segment is still `no_speech` (`AsrNoSpeechTests`). For word timings,
+ undetected speech gives every Whisper engine its turn before Parakeet, as captions do; with no
+ Parakeet waiting, the first Whisper engine that ran and failed is still the one silence.py's
+ "no word-level timings" refusal names, as in 2.5.1. Tests: `WhisperFailureTests`,
+ `AutoRoutingFallbackTests`.
+- **"No engine found" means none was found.** Its install lines are for a machine with no engine;
+ 2.5.1 also gave it when an installed whisper.cpp failed, and auto would have given it after
+ passing over an installed Parakeet engine for French. When engines were found and none
+ transcribed, the refusal names each and why (`kind: input`, `reason` `engine_failed` or
+ `english_only`, `engines[].detail` with the engine's own error line). A named `--engine` is `kind: missing_tool`
+ only when it is not installed; an installed one that failed is `kind: input`, `engine_failed`.
+ The line comes from stderr, or for parakeet-mlx from its stdout, where it prints its errors.
+ Test: `EngineRefusalTests`.
+- **The transcribe call returns what it used; nothing is left in module state.**
+ `transcribe_result()` / `transcribe_words_result()` return a `Transcription` (cues, words,
+ engine, the `transcription` facts, notes), so two transcriptions in one process never read each
+ other's engine or words. `transcribe()` and `transcribe_words()` keep their old return shapes
+ for existing callers. Test: `TranscriptionStateTests`.
+- **The skill's code downloads nothing; an engine may fetch its model once.** parakeet-mlx's
+ default `mlx-community/parakeet-tdt-0.6b-v2` comes from Hugging Face on first use, as
+ faster-whisper's does, and openai-whisper's from OpenAI. The audio never leaves the machine and
+ inference is local, so "no cloud" stays true. `execution.network: false` is about the skill's
+ own code: parakeet-mlx and openai-whisper fetch in their own child processes, but faster-whisper
+ is a library the skill runs in its own process, so its first-run fetch is made from that process
+ (`HF_HUB_OFFLINE=1` keeps it on the cached copy afterwards). Loading it with `local_files_only`
+ by default would break the first run 2.5.1 users rely on. The install hint, docs/contract.md and
+ references/scripts.md say so, and the contract's `execution.model_downloads` names
+ each engine's source and process, so `network: false` cannot be read as "a first
+ `--transcribe` is network-free". Tests: `InstallHintTests`,
+ `test_the_network_claim_matches_where_each_engine_runs`.
diff --git a/references/scripts.md b/references/scripts.md
index 4581b18..18abcb5 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -490,7 +490,8 @@ each shorter clip's last frame (with silence) out to the longest.
silence.py INPUT [--threshold -35] [--min-silence 0.6] [--margin 0.15] [--min-keep 0.2] [--list] [--edl keep.txt] [--speech-aware] [-o OUT]
silence.py INPUT --filler --words transcript.json [--filler-lang auto|en|ja|es|de|fr|pt|it]
[--filler-words FILE] [--filler-extra W,W] [--filler-keep W,W] [--filler-pad 0.02]
- [--transcribe] [--filler-list] [--max-cuts 400]
+ [--transcribe] [--engine auto|parakeet-mlx|parakeet.cpp|whisper.cpp|faster-whisper|openai-whisper]
+ [--filler-list] [--max-cuts 400]
```
Runs `silencedetect`, keeps `--margin` seconds of air around speech, drops
gaps shorter than `--min-silence`, and re-encodes once with `select`/`aselect`
@@ -533,7 +534,10 @@ Three refusals, all `kind: input`, all before any encode:
- `--filler` with neither `--words` nor `--transcribe` → names both flags.
- `--transcribe` with no engine on PATH → the same message `caption.py` gives,
- with the three install lines.
+ with its install lines (Parakeet and whisper engines). Engines that were found
+ but did not transcribe are named instead, with why (`reason: engine_failed`,
+ or `english_only` for a Parakeet engine auto passed over for `--filler-lang`
+ that is not English; see `caption.py --transcribe` below).
- `--transcribe` where the engine runs but its build produces no word-level
timings → names that engine, says some builds do not support word timestamps,
and points at `--words`. (`--transcribe` drives whichever engine is installed
@@ -546,7 +550,11 @@ Three refusals, all `kind: input`, all before any encode:
Results: `filler.removed_count`, `filler.removed_seconds`, `filler.removed`
(one entry per span), `filler.lang`, `filler.list`, `filler.word_timings` and
-`filler.warnings`. The existing `removed_seconds` keeps exactly the meaning it
+`filler.warnings`. With `--transcribe`, `filler.source` is `whisper:ENGINE` or
+`parakeet:ENGINE` and `filler.transcription` says which engine, model, language
+and routing made the words; a top-level `notes` line says when `--engine auto`
+ran the English-only Parakeet model on English assumed (no `--filler-lang`, and
+no Whisper engine that could take it). The existing `removed_seconds` keeps exactly the meaning it
has always had — the seconds of **silence** this run removed, the figure the
same run would report without `--filler` — and `removed_seconds_total` is the
additive sibling covering everything that went. A filler word quiet enough to
@@ -928,12 +936,38 @@ above the cap is clamped with a note, not refused, and both `jobs` and
- `--watch` composes: each pass is parallel, passes are sequential.
### caption.py --transcribe — optional local speech-to-text
-If `whisper-cli` (whisper.cpp), `faster-whisper` or `whisper` is installed,
-`caption.py input.mp4 --transcribe [--language ja] [--model base]` writes the
-SRT from the audio and burns it (combine with `--animate pop --karaoke`).
-Nothing is downloaded and nothing is required: without an engine it prints
-install hints and the user can supply `--text` cues instead. Always tell the
-user which engine was used, and treat the transcript as a draft to review.
+If `parakeet-mlx`, `parakeet-cli` (parakeet.cpp, with a `.gguf`), `whisper-cli` (whisper.cpp),
+`faster-whisper` or `whisper` is installed,
+`caption.py input.mp4 --transcribe [--language ja] [--model base] [--engine auto]` writes the
+SRT from the audio and burns it (combine with `--animate pop --karaoke`). `--engine auto` (default,
+or `FFMPEG_SKILL_ASR_ENGINE`) runs Parakeet for English speech -- an English `--language`, else
+English from whisper.cpp's language detector -- and Whisper for any other language. When neither
+can name the language, a Whisper engine that can run goes first; Parakeet runs on English
+*assumed* only when no Whisper engine is installed or every one failed, and the result's
+top-level `notes` then says so: that transcript is wrong for non-English speech, so pass `--language`
+(`--filler-lang` for `silence.py`) when the speech may not be English.
+`--engine parakeet-mlx|parakeet.cpp|whisper.cpp|faster-whisper|openai-whisper` forces one (a
+missing one is `kind: missing_tool`; an installed one that fails is `kind: input`, `reason:
+engine_failed`, with its own error line in `detail`). Without a language whisper.cpp is passed
+`-l auto` (its own default is English), else the named or detected language. `--model` is the
+whisper model; a Parakeet engine takes it only when it names a Parakeet model
+(`mlx-community/parakeet-*` or a `.gguf`), else `PARAKEET_MODEL` / `PARAKEET_CPP_MODEL`, else the
+English `tdt-0.6b-v2`. A non-English `--language` on an English-only Parakeet model is refused; a
+multilingual (v3) model assumes no language, so it gets no English-assumed note. The result's
+`transcription` names the engine, model and routing. `silence.py --filler --transcribe` takes
+the same `--engine`.
+Nothing is required: without an engine it prints install hints and the user can supply `--text`
+cues instead. When engines are installed but none transcribed, the refusal names each and why
+instead (`reason: engine_failed`, or `english_only` for a Parakeet engine auto passed over for
+another language). The skill's own code downloads nothing, and every engine transcribes on this
+machine -- the audio is never uploaded. An engine may fetch its *model weights* the
+first time it runs: parakeet-mlx downloads `mlx-community/parakeet-tdt-0.6b-v2` (or the `--model` /
+`PARAKEET_MODEL` repo) from Hugging Face into its cache; faster-whisper (also from Hugging Face)
+and openai-whisper (from OpenAI) fetch theirs on first use too. faster-whisper is a library that
+runs inside the skill's own process, so its download is made from that process.
+`HF_HUB_OFFLINE=1` keeps the Hugging Face ones on the cached copy afterwards. parakeet.cpp and
+whisper.cpp only read a model file you downloaded. Always tell the user which engine was used,
+and treat the transcript as a draft to review.
### MCP server — the toolkit for any MCP client
`python3 mcp/server.py` speaks MCP over stdio; each script is a tool taking
diff --git a/scripts/_common/asr.py b/scripts/_common/asr.py
index 788e956..c6fc54f 100644
--- a/scripts/_common/asr.py
+++ b/scripts/_common/asr.py
@@ -1,9 +1,11 @@
"""The optional local speech-to-text bridge, and the SRT the rest of the skill reads and writes.
-Whisper is never required. Nothing here runs unless a caller asked for a transcript:
+No speech engine is ever required. Nothing here runs unless a caller asked for a transcript:
`caption.py --transcribe` and `silence.py --filler --transcribe` share this one engine probe, so
the "no engine found" message, its install lines and its exit code are stated once rather than
-copied per tool. This module is neither ffprobe nor a decision, which is why it is its own file.
+copied per tool. What a call used (engine, model, routing, words, notes) comes back in its
+Transcription, never in module state. This module is neither ffprobe nor a decision, which is
+why it is its own file.
"""
from __future__ import annotations
@@ -11,20 +13,67 @@ import json
import os
import re
from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Dict, List, NamedTuple, Optional, Tuple
from _common.decision import fmt_srt_time, parse_time
from _common.emit import die, info
from _common.runner import read_text_or_die
-# The three engines this skill knows how to drive, and how to install each -- one string, so
-# every tool that needs a transcript refuses in the same words.
-ASR_ENGINES = ("whisper.cpp", "faster-whisper", "openai-whisper")
+# The engines this skill knows how to drive, and how to install each -- one string, so every
+# tool that needs a transcript refuses in the same words. The two Parakeet engines are
+# English-first (the default v2 model is English-only): under `--engine auto` they run only for
+# English speech and the Whisper engines take every other language.
+PARAKEET_ENGINES = ("parakeet-mlx", "parakeet.cpp")
+WHISPER_ENGINES = ("whisper.cpp", "faster-whisper", "openai-whisper")
+ASR_ENGINES = PARAKEET_ENGINES + WHISPER_ENGINES
+ENGINE_CHOICES = ("auto",) + ASR_ENGINES
+ASR_ENGINE_ENV = "FFMPEG_SKILL_ASR_ENGINE"
+PARAKEET_MLX_DEFAULT_MODEL = "mlx-community/parakeet-tdt-0.6b-v2"
+# Every engine transcribes on this machine: the audio is never uploaded. Three of them fetch
+# their model weights the first time they run, which the hint says, so "offline" is never
+# promised for a first run that is not. The skill's own code opens no connection, but
+# faster-whisper is a library run inside this process, so its first-run fetch is made from here.
ASR_INSTALL_HINT = (
- "Install one (all run offline):\n"
+ "Install one (each transcribes on this machine; the audio never leaves it):\n"
+ " parakeet-mlx: uv tool install parakeet-mlx (Apple Silicon; English; the first run\n"
+ " downloads the model from Hugging Face, " + PARAKEET_MLX_DEFAULT_MODEL + ")\n"
+ " parakeet.cpp: parakeet-cli from github.com/mudler/parakeet.cpp releases, plus a\n"
+ " tdt-0.6b-v2 .gguf in ~/.cache/parakeet.cpp/ (English)\n"
" whisper.cpp: brew install whisper-cpp (then download a model: ggml-base.bin)\n"
- " faster-whisper: pip install faster-whisper\n"
- " openai-whisper: pip install openai-whisper")
+ " faster-whisper: pip install faster-whisper (the first run downloads its model)\n"
+ " openai-whisper: pip install openai-whisper (the first run downloads its model)")
+
+
+class Transcription:
+ """What one transcription produced, and what produced it.
+
+ transcribe_result() and transcribe_words_result() return one instead of leaving it in module
+ state, so two transcriptions in one process (mcp/server.py, batch.py, a test) can never read
+ each other's engine, words or routing.
+
+ cues [(start, end, text)]: what transcribe_result() wrote to the SRT (a Parakeet word
+ run fills them too; a whisper word run leaves them [])
+ words [{word, start, end}]: transcribe_words_result()'s word timings; from
+ transcribe_result(), a Parakeet engine's word timings for --karaoke ([] for a
+ whisper engine, whose word timings are read from a JSON next to the SRT instead)
+ engine the engine that produced them, or None
+ facts the result document's `transcription` block, in its key order:
+ {routing, detected_language?, engine, model, language}
+ notes sentences for the result document's top-level `notes`
+ """
+
+ def __init__(self, cues: Optional[List[Tuple[float, float, str]]] = None,
+ words: "Optional[List[Dict[str, Any]]]" = None, engine: Optional[str] = None,
+ facts: Optional[Dict[str, Any]] = None, notes: Optional[List[str]] = None) -> None:
+ self.cues: List[Tuple[float, float, str]] = list(cues or [])
+ self.words: "List[Dict[str, Any]]" = [dict(w) for w in (words or [])]
+ self.engine = engine
+ self.facts: Dict[str, Any] = dict(facts or {})
+ self.notes: List[str] = list(notes or [])
+
+ def __repr__(self) -> str:
+ return (f"Transcription(engine={self.engine!r}, cues={len(self.cues)}, words={len(self.words)}, "
+ f"facts={self.facts!r}, notes={self.notes!r})")
def parse_srt(path: str) -> List[Tuple[float, float, str]]:
@@ -61,6 +110,67 @@ def die_no_speech(engine: str, video: str) -> None:
hint="check that the input (and --audio-stream) carries the speech, or write the cues by hand with --text")
+ENGINE_FAILED_REASON = "engine_failed"
+ENGLISH_ONLY_REASON = "english_only"
+
+
+def die_engine_failed(engine: str, video: str, tried: "List[Dict[str, str]]", installed: bool) -> None:
+ """A named --engine that did not transcribe. kind missing_tool only when it is not installed;
+ one that is installed and failed is kind input, `reason: "engine_failed"`, with the engine's
+ own last line in `detail` -- a JSON caller never sees the log line above it."""
+ if not installed:
+ die(f"--engine {engine} is not installed" + (
+ " (parakeet-cli and a .gguf model it can find: --model, PARAKEET_CPP_MODEL or ~/.cache/parakeet.cpp/)"
+ if engine == "parakeet.cpp" else ""),
+ kind="missing_tool", hint="install it (see --help), or use --engine auto to use whichever engine is installed")
+ detail = next((t["detail"] for t in tried if t["engine"] == engine), "failed")
+ die(f"--engine {engine} could not transcribe {video}: {engine} {detail}", kind="input",
+ reason=ENGINE_FAILED_REASON, engine=engine, detail=detail,
+ hint="fix the engine (its own message is in detail), or use --engine auto to fall back to another installed engine")
+
+
+def die_engines_failed(tried: "List[Dict[str, str]]", video: str, flag: str, alternative: str) -> None:
+ """--engine auto found engines and none transcribed: name each and why. Not die_no_engine():
+ its install lines would tell the caller to install what is installed. kind input;
+ `reason` is "english_only" when the only engines found were Parakeet engines auto passed over
+ for a language that is not English, else "engine_failed"; `engines` lists them all."""
+ failed = [t for t in tried if t["reason"] != ENGLISH_ONLY_REASON]
+ if failed:
+ hint = "fix the engine that failed (its own message is in engines[].detail)"
+ else:
+ hint = ("install a Whisper engine for this language (see --help), or name a Parakeet engine with --engine "
+ "and give it a multilingual (v3) model")
+ die(f"no installed speech-to-text engine could transcribe {video} for {flag}:\n"
+ + "\n".join(f" {t['engine']} {t['detail']}" for t in tried) + "\n" + alternative,
+ kind="input", reason=ENGINE_FAILED_REASON if failed else ENGLISH_ONLY_REASON,
+ engine=(failed or tried)[0]["engine"], engines=[dict(t) for t in tried], hint=hint)
+
+
+def _failed(engine: str, detail: str) -> "Dict[str, str]":
+ return {"engine": engine, "reason": ENGINE_FAILED_REASON, "detail": detail}
+
+
+def _failure_line(proc: Any, errors_on_stdout: bool = False) -> str:
+ """The line that says why an engine run failed, for the log and the failure document.
+
+ Its last stderr line, else its last stdout line. parakeet-mlx (`errors_on_stdout`) prints its
+ errors to stdout through rich -- "Error loading model ...", wrapped onto continuation lines --
+ while stderr may hold only a download's progress bars, so its last error line comes first."""
+ if errors_on_stdout:
+ lines = [ln.strip() for ln in (proc.stdout or "").splitlines()]
+ for i in range(len(lines) - 1, -1, -1):
+ if "error" in lines[i].lower():
+ end = i + 1
+ while end < len(lines) and lines[end]:
+ end += 1
+ return " ".join(lines[i:end])[:200]
+ for text in (proc.stderr, proc.stdout):
+ lines = [ln.strip() for ln in (text or "").splitlines() if ln.strip()]
+ if lines:
+ return lines[-1][:200]
+ return "?"
+
+
def _engine_cues(srt: str, engine: str, video: str) -> List[Tuple[float, float, str]]:
"""The cues of an SRT an engine wrote; an SRT with no cue in it is die_no_speech()."""
try:
@@ -85,79 +195,179 @@ def write_srt(cues: List[Tuple[float, float, str]], path: str) -> None:
fh.write(f"{i}\n{fmt_srt_time(s)} --> {fmt_srt_time(e)}\n{t}\n\n")
-def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0) -> List[Tuple[float, float, str]]:
- """Optional local ASR bridge. Tries, in order: whisper-cli / main (whisper.cpp), faster-whisper (python),
- whisper (openai-whisper CLI). Produces an SRT with word timings where the engine supports it.
- No engine installed -> clear error with install hints; the skill never depends on one."""
+def transcribe_result(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0,
+ engine: Optional[str] = None) -> Transcription:
+ """Optional local ASR bridge: the SRT written to `out_srt`, and a Transcription saying what
+ produced it. `engine` (else $FFMPEG_SKILL_ASR_ENGINE, else auto) picks the engine; auto tries
+ Parakeet (parakeet-mlx, parakeet.cpp) for English speech, then whisper-cli / main
+ (whisper.cpp), faster-whisper (python), whisper (openai-whisper CLI). No engine installed ->
+ clear error with install hints; the skill never depends on one."""
import shutil
import subprocess
import tempfile
- from _common import require_tool, run_analysis, STATE
+ from _common import require_tool
ffmpeg = require_tool("ffmpeg")
+ wanted = requested_engine(engine)
tmpdir = tempfile.mkdtemp(prefix="ffskill_asr_")
try:
- return _transcribe_in(tmpdir, video, out_srt, language, model, audio_stream, ffmpeg, shutil, subprocess)
+ return _transcribe_in(tmpdir, video, out_srt, language, model, audio_stream, ffmpeg, shutil, subprocess, wanted)
finally:
shutil.rmtree(tmpdir, ignore_errors=True)
-def _asr_run(cmd: List[str], subprocess, name: str) -> "subprocess.CompletedProcess":
+def transcribe(video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int = 0,
+ engine: Optional[str] = None) -> List[Tuple[float, float, str]]:
+ """transcribe_result()'s cues alone: the call shape caption.transcribe has always had. A caller
+ that reports which engine ran (caption.py does) calls transcribe_result() instead."""
+ return transcribe_result(video, out_srt, language, model, audio_stream, engine).cues
+
+
+def _asr_run(cmd: List[str], subprocess, name: str, env: "Optional[Dict[str, str]]" = None) -> "subprocess.CompletedProcess":
"""Run a speech-to-text engine under the same wall-clock limit as an ffmpeg call."""
from _common import STATE, die
limit = STATE.timeout or None
try:
- return subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", timeout=limit)
+ return subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace",
+ timeout=limit, **({"env": env} if env is not None else {}))
except subprocess.TimeoutExpired:
die(f"{name} exceeded the {limit:.0f} s time limit and was killed; raise --timeout for a long recording",
code=124, kind="timeout")
return None # unreachable
+def assumed_english_note(language_flag: str = "--language", whisper_failed: bool = False,
+ parakeet_model: bool = False) -> str:
+ """The top-level `notes` line for a transcript the English-only Parakeet model made from speech
+ whose language nobody named and nothing here could detect. The fix it names: the Whisper
+ engine that failed (`whisper_failed`), a multilingual Parakeet model when --model named a
+ Parakeet one (`parakeet_model`), else a Whisper engine to install."""
+ if whisper_failed:
+ fix = "or fix the Whisper engine that failed (the log names it)"
+ elif parakeet_model:
+ fix = "or name a multilingual (v3) Parakeet model with --model"
+ else:
+ fix = "or install a Whisper engine with a multilingual model, which --engine auto then uses for speech it cannot identify"
+ return (f"English was assumed: no {language_flag} was given and the spoken language could not be "
+ "detected here (that needs whisper.cpp with a multilingual ggml model), so the English-only "
+ "Parakeet model made this transcript. If the speech is not English the transcript is wrong: "
+ f"name the language with {language_flag}, " + fix)
+
+
+def _parakeet_attempt(engines: "List[str]", facts: Dict[str, Any], wanted: str, language: Optional[str],
+ wav: str, tmpdir: str, model: Optional[str], shutil, subprocess, video: str,
+ language_flag: str, need_words: bool = False,
+ tried: "Optional[List[Dict[str, str]]]" = None) -> Optional[Tuple[Transcription, str]]:
+ """(Transcription, the model's file name) from the first of `engines` that transcribes, or None.
+
+ Under --engine auto with no language named or detected, the English-only model ran on an
+ assumption: the Transcription then carries assumed_english_note(), and a warning is logged. A
+ multilingual (v3) model assumed nothing, so it gets no note and its routing drops "assumed
+ English". `need_words`: an engine whose cues read but whose word times do not is passed over
+ (silence.py --filler can cut nothing on cues), unless it was asked for by name. An engine that
+ ran and did not transcribe is added to `tried`."""
+ for eng in engines:
+ got = run_parakeet(eng, wav, tmpdir, model, language, shutil, subprocess, video, tried)
+ if got and need_words and not got[1] and wanted not in PARAKEET_ENGINES:
+ # cues read but no word times (a caption could use them; this caller cannot); a
+ # named engine keeps its own "no word-level timings" refusal instead
+ info(f"{eng} gave cues but no readable word timings; trying the next engine")
+ if tried is not None:
+ tried.append(_failed(eng, "gave cues but no readable word timings"))
+ got = None
+ if not got:
+ continue
+ cues, words, chosen = got
+ run = Transcription(cues=cues, words=words, engine=eng,
+ facts=dict(facts, engine=eng, model=chosen, language=language or facts.get("detected_language")))
+ if wanted == "auto" and not language and not facts.get("detected_language"):
+ if _parakeet_multilingual(chosen):
+ run.facts["routing"] = run.facts["routing"].replace(", assumed English", "; the Parakeet model is multilingual")
+ else:
+ run.notes.append(assumed_english_note(language_flag, whisper_failed=facts.get("routing") == ROUTING_LAST_RESORT,
+ parakeet_model=facts.get("routing") == ROUTING_PARAKEET_MODEL))
+ info("warning: " + run.notes[-1])
+ return run, os.path.basename(chosen)
+ return None
+
+
+def _skipped_parakeet(route: "Route", language: Optional[str], language_flag: str) -> "List[Dict[str, str]]":
+ """The installed Parakeet engines --engine auto passed over for a language that is not English."""
+ what = f"{language_flag} {language}" if language else f"the detected language {route.facts.get('detected_language')}"
+ return [{"engine": e, "reason": ENGLISH_ONLY_REASON,
+ "detail": f"is installed, but --engine auto runs Parakeet only for English speech, and {what} is not English"}
+ for e in route.skipped]
+
+
def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str], model: str, audio_stream: int,
- ffmpeg: str, shutil, subprocess) -> List[Tuple[float, float, str]]:
- from _common import run_analysis, STATE, die
+ ffmpeg: str, shutil, subprocess, wanted: str = "auto") -> Transcription:
+ from _common import run_analysis, STATE
wav = os.path.join(tmpdir, "audio.wav")
# A wav in our own temp dir: a measurement input for the engine, not a deliverable, so it
# is not a run() call (no --dry-run gate, not recorded), but it keeps the time limit and
# reports an unreadable input as kind ffmpeg instead of a CalledProcessError traceback.
run_analysis([ffmpeg, "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-i", video,
"-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav])
+ # every engine found that did not transcribe, and why: the refusal names them rather than
+ # telling the caller to install what is installed
+ tried: "List[Dict[str, str]]" = []
+
+ def parakeet(engines: "List[str]", facts: Dict[str, Any]) -> Optional[Transcription]:
+ done = _parakeet_attempt(engines, facts, wanted, language, wav, tmpdir, model, shutil, subprocess, video, "--language",
+ tried=tried)
+ if not done:
+ return None
+ run, model_name = done
+ info(f"transcribed with {run.engine} (model {model_name})")
+ write_srt(run.cues, out_srt)
+ return run
+
+ # 0. Parakeet (English), when auto-routing picks it or it was asked for by name
+ route = parakeet_route(wanted, language, wav, shutil, subprocess, model)
+ done = parakeet(route.first, route.facts)
+ if done:
+ return done
+ tried.extend(_skipped_parakeet(route, language, "--language"))
+ if wanted in PARAKEET_ENGINES:
+ die_engine_failed(wanted, video, tried, _parakeet_available(wanted, shutil, model))
+ facts = route.facts
+ if route.first:
+ # auto chose Parakeet and no Parakeet engine transcribed it: the routing says why Whisper ran
+ facts = dict(facts, routing=facts["routing"] + ROUTING_PARAKEET_FAILED)
+ # the language each Whisper engine is told: the named one, else the one auto's detector named
+ lang = language or facts.get("detected_language")
# 1. whisper.cpp
- cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp")
- if not cli:
- # older whisper.cpp builds ship the binary as plain `main`; accept it only when it lives
- # in a directory that names whisper, so an unrelated /usr/bin/main is never run
- main_bin = shutil.which("main")
- if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
- cli = main_bin
+ cli = _whisper_cpp_cli(shutil) if wanted in ("auto", "whisper.cpp") else None
if cli:
- model_path = model
- if not os.path.exists(model_path):
- for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"), f"models/ggml-{model}.bin", f"/usr/local/share/whisper/ggml-{model}.bin"):
- if os.path.exists(cand):
- model_path = cand
- break
+ model_path = _whisper_cpp_model(model)
base = os.path.join(tmpdir, "out")
- cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base]
- if language:
- cmd += ["-l", language]
+ # whisper.cpp's own default is -l en: without "auto" it decodes any speech as English
+ cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base, "-l", lang or "auto"]
proc = _asr_run(cmd, subprocess, "whisper.cpp")
if proc.returncode == 0 and os.path.exists(base + ".srt"):
info(f"transcribed with whisper.cpp ({os.path.basename(cli)}, model {os.path.basename(model_path)})")
cues = _engine_cues(base + ".srt", "whisper.cpp", video)
write_srt(cues, out_srt)
- return cues
- info("whisper.cpp found but failed: " + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
+ return Transcription(cues=cues, engine="whisper.cpp",
+ facts=dict(facts, engine="whisper.cpp", model=model_path, language=lang))
+ line = _failure_line(proc)
+ info("whisper.cpp found but failed: " + line)
+ tried.append(_failed("whisper.cpp", "found but failed: " + line))
# 2. faster-whisper (python package)
try:
+ if wanted not in ("auto", "faster-whisper"):
+ raise ImportError
from faster_whisper import WhisperModel # type: ignore
import threading
result: list = []
+ crashed: list = []
def work() -> None:
- m = WhisperModel(model, device="cpu", compute_type="int8")
- segments, _ = m.transcribe(wav, language=language, word_timestamps=False)
- result.extend((seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip())
+ try:
+ m = WhisperModel(model, device="cpu", compute_type="int8")
+ segments, _ = m.transcribe(wav, language=lang, word_timestamps=False)
+ result.extend((seg.start, seg.end, seg.text.strip()) for seg in segments if seg.text.strip())
+ except Exception as exc: # noqa: BLE001 -- a model that would not load or run: a failed engine
+ crashed.append(exc)
# An in-process engine gets the same wall-clock limit as the CLI engines and ffmpeg.
t = threading.Thread(target=work, daemon=True)
@@ -165,28 +375,50 @@ def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str
t.join(STATE.timeout or None)
if t.is_alive():
die(f"faster-whisper exceeded the {STATE.timeout:.0f} s time limit; raise --timeout for a long recording", code=124, kind="timeout")
- cues = list(result)
- if not cues:
- die_no_speech("faster-whisper", video)
- info("transcribed with faster-whisper")
- write_srt(cues, out_srt)
- return cues
+ if crashed:
+ # it failed (a model it could not fetch or load), which is not "found no speech":
+ # move on to the next engine, as a failed whisper.cpp does
+ line = str(crashed[0])[:200] or type(crashed[0]).__name__
+ info("faster-whisper found but failed: " + line)
+ tried.append(_failed("faster-whisper", "found but failed: " + line))
+ else:
+ cues = list(result)
+ if not cues:
+ die_no_speech("faster-whisper", video)
+ info("transcribed with faster-whisper")
+ write_srt(cues, out_srt)
+ return Transcription(cues=cues, engine="faster-whisper",
+ facts=dict(facts, engine="faster-whisper", model=model, language=lang))
except ImportError:
pass
# 3. openai-whisper CLI
- if shutil.which("whisper"):
+ if wanted in ("auto", "openai-whisper") and shutil.which("whisper"):
cmd = ["whisper", wav, "--model", model, "--output_format", "srt", "--output_dir", tmpdir]
- if language:
- cmd += ["--language", language]
+ if lang:
+ cmd += ["--language", lang]
proc = _asr_run(cmd, subprocess, "openai-whisper")
srt = os.path.join(tmpdir, "audio.srt")
if proc.returncode == 0 and os.path.exists(srt):
info("transcribed with openai-whisper")
cues = _engine_cues(srt, "openai-whisper", video)
write_srt(cues, out_srt)
- return cues
+ return Transcription(cues=cues, engine="openai-whisper",
+ facts=dict(facts, engine="openai-whisper", model=model, language=lang))
+ line = _failure_line(proc)
+ info("openai-whisper found but failed: " + line)
+ tried.append(_failed("openai-whisper", "found but failed: " + line))
+ # 4. Parakeet after all, when the language was never known and every Whisper engine failed
+ if route.last:
+ info("no Whisper engine transcribed it; trying Parakeet with English assumed")
+ done = parakeet(list(route.last), {"routing": ROUTING_LAST_RESORT})
+ if done:
+ return done
+ if wanted != "auto":
+ die_engine_failed(wanted, video, tried, any(t["engine"] == wanted for t in tried))
+ if tried:
+ die_engines_failed(tried, video, "--transcribe", "Or write the cues by hand with --text cues.txt (see format above).")
die_no_engine("Or write the cues by hand with --text cues.txt (see format above).")
- return []
+ return Transcription()
def die_no_engine(alternative: str, flag: str = "--transcribe") -> None:
@@ -293,22 +525,395 @@ def _words_from_openai_whisper_json(path: str) -> "List[Dict[str, Any]]":
return [w for w in out if w["word"]]
+# ------------------------------------------------------- Parakeet engines and routing
+
+
+def _words_from_parakeet_mlx_json(path: str) -> "List[Dict[str, Any]]":
+ """parakeet-mlx --output-format json: sentences[].tokens[] of SUB-word pieces, times in seconds.
+
+ A piece with a leading space starts a word; a bare " " piece is a separator, so the piece
+ after it starts a word too (" m" "is" "ter" " " "Q" "u" "il" "ter" -> "mister", "Quilter").
+ """
+ try:
+ doc = json.loads(Path(path).read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return []
+ out: "List[Dict[str, Any]]" = []
+ for sent in _dicts(doc.get("sentences")) if isinstance(doc, dict) else []:
+ brk = True # every sentence starts a new word
+ for tok in _dicts(sent.get("tokens")):
+ text = str(tok.get("text") or "")
+ try:
+ start, end = float(tok["start"]), float(tok["end"])
+ except (KeyError, TypeError, ValueError):
+ continue
+ if not text.strip():
+ brk = True
+ continue
+ if brk or text.startswith(" ") or not out:
+ out.append({"word": text.strip(), "start": start, "end": end})
+ else:
+ out[-1]["word"] += text
+ out[-1]["end"] = end
+ brk = False
+ return [w for w in out if w["word"]]
+
+
+def _cues_from_parakeet_mlx_json(path: str) -> List[Tuple[float, float, str]]:
+ """parakeet-mlx's own sentence segmentation, as cues."""
+ try:
+ doc = json.loads(Path(path).read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return []
+ cues: List[Tuple[float, float, str]] = []
+ for sent in _dicts(doc.get("sentences")) if isinstance(doc, dict) else []:
+ try:
+ text = str(sent.get("text") or "").strip()
+ if text:
+ cues.append((float(sent["start"]), float(sent["end"]), text))
+ except (KeyError, TypeError, ValueError):
+ continue
+ return cues
+
+
+def _dicts(value: Any) -> "List[Dict[str, Any]]":
+ """The dict items of a JSON list; anything else (a number, a string, null) is no items. The
+ engines' JSON is parsed defensively: a wrong shape yields no words, never a traceback."""
+ return [v for v in value if isinstance(v, dict)] if isinstance(value, list) else []
+
+
+def _words_from_parakeet_cpp_json(text: str) -> "List[Dict[str, Any]]":
+ """parakeet-cli transcribe --json: {"words": [{"w", "start", "end", "conf"}]}, times in seconds."""
+ try:
+ doc = json.loads(text)
+ except ValueError:
+ return []
+ out: "List[Dict[str, Any]]" = []
+ for w in _dicts(doc.get("words")) if isinstance(doc, dict) else []:
+ try:
+ word = str(w.get("w") or w.get("word") or "").strip()
+ if word:
+ out.append({"word": word, "start": float(w["start"]), "end": float(w["end"])})
+ except (AttributeError, KeyError, TypeError, ValueError):
+ continue
+ return out
+
+
+# Cue boundaries for an engine that gives words but no sentences (parakeet.cpp): the same limits
+# parakeet-mlx's own segmentation is run with, so both Parakeet engines cut cues alike.
+CUE_MAX_SECONDS = 7.0
+CUE_MAX_CHARS = 84
+CUE_GAP_SECONDS = 0.8
+
+
+def cues_from_words(words: "List[Dict[str, Any]]") -> List[Tuple[float, float, str]]:
+ """Group timed words into cues: a cue ends after sentence-final punctuation, before a pause of
+ CUE_GAP_SECONDS or more, or before it would pass CUE_MAX_SECONDS / CUE_MAX_CHARS."""
+ cues: List[Tuple[float, float, str]] = []
+ cur: "List[Dict[str, Any]]" = []
+
+ def flush() -> None:
+ if cur:
+ cues.append((cur[0]["start"], cur[-1]["end"], " ".join(w["word"] for w in cur)))
+ cur.clear()
+
+ for w in words:
+ if cur:
+ text_len = len(" ".join(x["word"] for x in cur)) + 1 + len(w["word"])
+ if (w["start"] - cur[-1]["end"] >= CUE_GAP_SECONDS or w["end"] - cur[0]["start"] > CUE_MAX_SECONDS
+ or text_len > CUE_MAX_CHARS):
+ flush()
+ cur.append(w)
+ if w["word"][-1:] in ".?!":
+ flush()
+ flush()
+ return cues
+
+
+def requested_engine(engine: Optional[str]) -> str:
+ """--engine, else $FFMPEG_SKILL_ASR_ENGINE, else "auto"; an unknown env value is refused."""
+ name = engine or os.environ.get(ASR_ENGINE_ENV, "").strip() or "auto"
+ if name not in ENGINE_CHOICES:
+ die(f"{ASR_ENGINE_ENV}={name!r} is not a speech engine this skill drives (one of {', '.join(ENGINE_CHOICES)})",
+ kind="input", hint=f"unset {ASR_ENGINE_ENV} or set it to one of {', '.join(ENGINE_CHOICES)}")
+ return name
+
+
+def _is_english(language: Optional[str]) -> bool:
+ # "eng" too: caption.py's --language is also the mux track tag, where ISO 639-2 is natural
+ return (language or "").lower().split("-")[0].split("_")[0] in ("en", "eng", "english")
+
+
+def _whisper_cpp_cli(shutil) -> Optional[str]:
+ cli = shutil.which("whisper-cli") or shutil.which("whisper-cpp")
+ if not cli:
+ # older whisper.cpp builds ship the binary as plain `main`; accept it only when it lives
+ # in a directory that names whisper, so an unrelated /usr/bin/main is never run
+ main_bin = shutil.which("main")
+ if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
+ cli = main_bin
+ return cli
+
+
+def _whisper_cpp_model(model: str) -> str:
+ if os.path.exists(model):
+ return model
+ for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"), f"models/ggml-{model}.bin",
+ f"/usr/local/share/whisper/ggml-{model}.bin"):
+ if os.path.exists(cand):
+ return cand
+ return model
+
+
+def detect_language(wav: str, shutil, subprocess) -> Optional[str]:
+ """The spoken language of `wav` from whisper.cpp's detector (-dl), or None when it cannot tell.
+
+ The smallest installed ggml model is used: detection reads the first 30 s once, and a small
+ model answers it in well under a second on the GPU.
+ """
+ cli = _whisper_cpp_cli(shutil)
+ if not cli:
+ return None
+ model = next((p for p in (_whisper_cpp_model(m) for m in ("tiny", "base", "small", "tiny.en"))
+ if os.path.exists(p) and not p.endswith(".en.bin")), None)
+ if not model:
+ return None
+ proc = _asr_run([cli, "-m", model, "-f", wav, "-dl", "-l", "auto"], subprocess, "whisper.cpp language detection")
+ m = re.search(r"auto-detected language:\s*([a-z]{2,3})", (proc.stderr or "") + (proc.stdout or ""))
+ return m.group(1) if m else None
+
+
+ROUTING_ASSUMED_ENGLISH = "auto: language not detectable here and no Whisper engine, assumed English"
+ROUTING_UNDETECTED_WHISPER = "auto: language not detectable here, Whisper first (Parakeet is English-only)"
+ROUTING_LAST_RESORT = "auto: language not detectable here and no Whisper engine transcribed it, assumed English"
+ROUTING_PARAKEET_MODEL = "auto: --model names a Parakeet model and the language is not detectable here, assumed English"
+# appended to the routing auto chose Parakeet for, when no Parakeet engine transcribed it and a
+# Whisper engine did
+ROUTING_PARAKEET_FAILED = ", no Parakeet engine transcribed it"
+
+
+class Route(NamedTuple):
+ """parakeet_route()'s answer: which Parakeet engines run, and when."""
+ first: List[str] # Parakeet engines to try before any Whisper engine
+ facts: Dict[str, Any] # the routing facts for the result document's `transcription`
+ last: Tuple[str, ...] = () # Parakeet engines to try once every Whisper engine has failed
+ skipped: Tuple[str, ...] = () # installed Parakeet engines auto passed over: the language is not English
+
+
+def parakeet_route(engine: str, language: Optional[str], wav: str, shutil, subprocess,
+ model: Optional[str] = None, whisper_model: Optional[str] = None) -> Route:
+ """Route(the Parakeet engines to try first, routing facts for the result document, the ones to
+ try last, the installed ones passed over).
+
+ `auto` tries Parakeet only for English: an explicit --language, else whisper.cpp's detector.
+ When neither can say, the English-only model is not trusted with it: a Whisper engine that
+ can run (`whisper_model`, default `model`) goes first and Parakeet only after every Whisper
+ engine failed; with no Whisper engine (or a --model that names a Parakeet model, which no
+ Whisper engine can load), Parakeet runs on English assumed and the caller's result carries
+ assumed_english_note(). A named Parakeet engine always runs; with a non-English --language it
+ needs a multilingual (v3) model, which _parakeet_model_for() checks.
+ """
+ if engine in PARAKEET_ENGINES:
+ return Route([engine], {"routing": "requested"})
+ if engine != "auto":
+ return Route([], {"routing": "requested"})
+ installed = tuple(e for e in PARAKEET_ENGINES if _parakeet_available(e, shutil, model))
+ if not installed:
+ return Route([], {"routing": "auto: no Parakeet engine installed"})
+ if language:
+ if _is_english(language):
+ return Route(list(PARAKEET_ENGINES), {"routing": "auto: --language is English"})
+ info(f"{' and '.join(installed)} passed over: --engine auto runs Parakeet only for English speech (--language {language})")
+ return Route([], {"routing": f"auto: --language {language} is not English"}, (), installed)
+ detected = detect_language(wav, shutil, subprocess)
+ if detected is None:
+ wmodel = whisper_model if whisper_model is not None else model
+ if _names_parakeet_model(wmodel or "base"):
+ return Route(list(PARAKEET_ENGINES), {"routing": ROUTING_PARAKEET_MODEL})
+ if _whisper_ready(shutil, wmodel):
+ info("the spoken language could not be detected here; a Whisper engine takes it (Parakeet's model is English-only)")
+ return Route([], {"routing": ROUTING_UNDETECTED_WHISPER}, PARAKEET_ENGINES)
+ return Route(list(PARAKEET_ENGINES), {"routing": ROUTING_ASSUMED_ENGLISH})
+ if _is_english(detected):
+ return Route(list(PARAKEET_ENGINES), {"routing": "auto: detected English", "detected_language": detected})
+ info(f"{' and '.join(installed)} passed over: --engine auto runs Parakeet only for English speech (detected {detected})")
+ return Route([], {"routing": f"auto: detected {detected}, not English", "detected_language": detected}, (), installed)
+
+
+def _module_importable(name: str) -> bool:
+ """`import name` would find something, without importing it (faster-whisper pulls in ctranslate2)."""
+ import importlib.util
+ import sys
+ try:
+ return importlib.util.find_spec(name) is not None
+ except (ImportError, ValueError): # a module already in sys.modules with no __spec__ (a stub)
+ return sys.modules.get(name) is not None
+
+
+def _names_parakeet_model(name: str) -> bool:
+ """`name` is a Parakeet model (an mlx-community/parakeet-* repo or a .gguf), which no Whisper engine loads."""
+ return "parakeet" in name.lower() or name.lower().endswith(".gguf")
+
+
+def _english_only_whisper_model(name: str) -> bool:
+ """An English-only Whisper model: tiny.en, base.en, ... or a ggml-*.en.bin file."""
+ base = os.path.basename(name).lower()
+ return (base[:-4] if base.endswith(".bin") else base).endswith(".en")
+
+
+def _whisper_ready(shutil, model: Optional[str]) -> bool:
+ """A Whisper engine that --engine auto would run with `model` on speech in any language:
+ whisper.cpp with that model on disk, faster-whisper importable, or the openai-whisper CLI. A
+ Parakeet model name is no Whisper model, and an English-only one (base.en, ggml-*.en.bin)
+ handles English alone, so both answer False."""
+ name = (model or "base")
+ if _names_parakeet_model(name) or _english_only_whisper_model(name):
+ return False
+ if _whisper_cpp_cli(shutil):
+ path = _whisper_cpp_model(name)
+ if os.path.exists(path) and not _english_only_whisper_model(path):
+ return True
+ return _module_importable("faster_whisper") or bool(shutil.which("whisper"))
+
+
+def _parakeet_available(engine: str, shutil, model: Optional[str] = None) -> bool:
+ if engine == "parakeet-mlx":
+ return bool(shutil.which("parakeet-mlx"))
+ gguf = _parakeet_cpp_gguf(model)
+ return bool(shutil.which("parakeet-cli")) and gguf is not None and os.path.isfile(gguf)
+
+
+def _parakeet_cpp_gguf(model: Optional[str]) -> Optional[str]:
+ """--model when it is a .gguf, else $PARAKEET_CPP_MODEL, else an English tdt-0.6b-v2 .gguf in
+ ~/.cache/parakeet.cpp/ (f16 first, then q8_0, then any other quantisation)."""
+ if model and model.lower().endswith(".gguf"):
+ return os.path.expanduser(model)
+ env = os.environ.get("PARAKEET_CPP_MODEL", "").strip()
+ if env:
+ return os.path.expanduser(env)
+ cache = Path(os.path.expanduser("~/.cache/parakeet.cpp"))
+ found = sorted(cache.glob("tdt-0.6b-v2-*.gguf")) if cache.is_dir() else []
+ for pref in ("-f16.gguf", "-q8_0.gguf"):
+ for p in found:
+ if p.name.endswith(pref):
+ return str(p)
+ return str(found[0]) if found else None
+
+
+def _parakeet_model_for(engine: str, model: Optional[str], language: Optional[str]) -> str:
+ """The model the Parakeet engine runs: --model when it names a Parakeet model, else the
+ engine's own environment variable, else the English v2 default. A non-English --language on a
+ model without "v3" in its name is refused: v2 would transcribe it as English gibberish."""
+ if engine == "parakeet-mlx":
+ chosen = model if model and "parakeet" in model.lower() and not model.lower().endswith(".gguf") else \
+ (os.environ.get("PARAKEET_MODEL", "").strip() or PARAKEET_MLX_DEFAULT_MODEL)
+ else:
+ chosen = _parakeet_cpp_gguf(model) or ""
+ if language and not _is_english(language) and not _parakeet_multilingual(chosen):
+ die(f"{engine} with model {os.path.basename(chosen) or '?'} is English-only; --language {language} needs a multilingual (v3) Parakeet model",
+ kind="input", hint="pass --model with a parakeet-tdt-0.6b-v3 model, or --engine auto / whisper.cpp for this language")
+ return chosen
+
+
+def _parakeet_multilingual(model: str) -> bool:
+ """A multilingual Parakeet model (tdt-0.6b-v3, 25 European languages) rather than the English-only v2."""
+ return "v3" in os.path.basename(model or "").lower()
+
+
+def run_parakeet(engine: str, wav: str, tmpdir: str, model: Optional[str], language: Optional[str], shutil, subprocess,
+ video: str, tried: "Optional[List[Dict[str, str]]]" = None,
+ ) -> Optional[Tuple[List[Tuple[float, float, str]], "List[Dict[str, Any]]", str]]:
+ """(cues, words, model) from one Parakeet engine, or None when it is not installed or failed
+ (an info line says which, and a failed run is added to `tried`), so the caller moves on to the
+ next engine. Output that yields no words is only "no speech" when it is the engine's own empty
+ answer; anything else it could not be read, which is a failed run too."""
+ if not _parakeet_available(engine, shutil, model):
+ info(f"{engine} not available" + (" (no readable .gguf: --model, PARAKEET_CPP_MODEL, or tdt-0.6b-v2 in ~/.cache/parakeet.cpp)"
+ if engine == "parakeet.cpp" and shutil.which("parakeet-cli") else ""))
+ return None
+ chosen = _parakeet_model_for(engine, model, language)
+
+ def failed(detail: str) -> None:
+ info(f"{engine} {detail}")
+ if tried is not None:
+ tried.append(_failed(engine, detail))
+
+ if engine == "parakeet-mlx":
+ outdir = os.path.join(tmpdir, "pmlx")
+ cmd = ["parakeet-mlx", wav, "--model", chosen, "--output-format", "json", "--output-dir", outdir,
+ "--max-duration", f"{CUE_MAX_SECONDS:g}", "--silence-gap", f"{CUE_GAP_SECONDS:g}"]
+ # the JSON is looked for as <outdir>/audio.json: a host PARAKEET_OUTPUT_TEMPLATE would name it otherwise
+ env = {k: v for k, v in os.environ.items() if k != "PARAKEET_OUTPUT_TEMPLATE"}
+ proc = _asr_run(cmd, subprocess, "parakeet-mlx", env=env)
+ doc = os.path.join(outdir, os.path.splitext(os.path.basename(wav))[0] + ".json")
+ if proc.returncode != 0 or not os.path.exists(doc):
+ failed("found but failed: " + _failure_line(proc, errors_on_stdout=True))
+ return None
+ cues, words = _cues_from_parakeet_mlx_json(doc), _words_from_parakeet_mlx_json(doc)
+ try:
+ raw, key = Path(doc).read_text(encoding="utf-8", errors="replace"), "sentences"
+ except OSError:
+ raw, key = "", "sentences"
+ else:
+ cmd = ["parakeet-cli", "transcribe", "--model", chosen, "--input", wav, "--json"]
+ proc = _asr_run(cmd, subprocess, "parakeet.cpp")
+ if proc.returncode != 0:
+ failed("found but failed: " + _failure_line(proc))
+ return None
+ words = _words_from_parakeet_cpp_json(proc.stdout)
+ cues = cues_from_words(words)
+ raw, key = proc.stdout or "", "words"
+ if not cues:
+ if _heard_nothing(raw, key):
+ die_no_speech(engine, video)
+ failed(f"ran but wrote output this skill cannot read as a transcript ({(raw.strip() or 'nothing')[:80]!r})")
+ return None
+ return cues, words, chosen
+
+
+def _heard_nothing(text: str, key: str) -> bool:
+ """True when an engine's JSON is its well-formed empty answer, {key: []}: what parakeet-cli
+ ("words") and parakeet-mlx ("sentences") both write for silence (measured on 3 s of it).
+ Anything else that yields no words -- not JSON, the wrong shape, entries that do not parse --
+ is output that could not be read, not a finding that nobody spoke."""
+ try:
+ doc = json.loads(text)
+ except ValueError:
+ return False
+ return isinstance(doc, dict) and doc.get(key) == []
+
+
def transcribe_words(video: str, language: "Optional[str]" = None, model: str = "base",
- audio_stream: int = 0) -> "Tuple[List[Dict[str, Any]], Optional[str]]":
- """([{word, start, end}, ...], the engine that produced them) from a local whisper.
+ audio_stream: int = 0, engine: "Optional[str]" = None) -> "Tuple[List[Dict[str, Any]], Optional[str]]":
+ """transcribe_words_result()'s (words, engine) alone: the call shape this function has had
+ since 1.17. A caller that reports which engine ran (silence.py does) calls
+ transcribe_words_result() instead."""
+ run = transcribe_words_result(video, language, model, audio_stream, engine)
+ return run.words, run.engine
+
+
+def transcribe_words_result(video: str, language: "Optional[str]" = None, model: str = "base",
+ audio_stream: int = 0, engine: "Optional[str]" = None,
+ language_flag: str = "--filler-lang") -> Transcription:
+ """[{word, start, end}, ...] from a local speech engine, as a Transcription naming the engine.
- Drives whichever engine is installed with ITS word-timestamp option -- whisper.cpp
+ `engine` (else $FFMPEG_SKILL_ASR_ENGINE, else auto) picks it; auto tries Parakeet for English
+ speech first (parakeet-mlx sub-word tokens merged into words, parakeet.cpp `--json` words),
+ then drives whichever whisper is installed with ITS word-timestamp option -- whisper.cpp
`--output-json-full`, faster-whisper `word_timestamps=True`, openai-whisper
- `--word_timestamps True` -- and returns the words it measured. ([], engine) when the engine
- ran but its build produced no word-level timings, so the caller can refuse naming that engine
- instead of pretending the audio had no words in it. No engine at all raises through
- die_no_engine(), the same refusal caption.py gives.
+ `--word_timestamps True` -- and returns the words it measured. Words [] with the engine set
+ when the engine ran but its build produced no word-level timings, so the caller can refuse
+ naming that engine instead of pretending the audio had no words in it. No engine at all
+ raises through die_no_engine(), the same refusal caption.py gives. `language_flag` is the
+ caller's flag for the language, named in assumed_english_note().
"""
import shutil as _shutil
import subprocess as _subprocess
import tempfile as _tempfile
from _common import require_tool, run_analysis, STATE
ffmpeg = require_tool("ffmpeg")
+ wanted = requested_engine(engine)
+ parakeet_model = None if model == "base" else model
tmpdir = _tempfile.mkdtemp(prefix="ffskill_asrw_")
try:
wav = os.path.join(tmpdir, "audio.wav")
@@ -316,47 +921,72 @@ def transcribe_words(video: str, language: "Optional[str]" = None, model: str =
"-map", f"0:a:{audio_stream}", "-vn", "-ac", "1", "-ar", "16000",
"-c:a", "pcm_s16le", wav])
+ tried: "List[Dict[str, str]]" = [] # engines found that gave no words, and why
+
+ def parakeet(engines: "List[str]", facts: Dict[str, Any]) -> Optional[Transcription]:
+ done = _parakeet_attempt(engines, facts, wanted, language, wav, tmpdir, parakeet_model,
+ _shutil, _subprocess, video, language_flag, need_words=True, tried=tried)
+ if not done:
+ return None
+ run, model_name = done
+ info(f"word timings from {run.engine} ({len(run.words)} words, model {model_name})")
+ return run
+
+ # 0. Parakeet (English), when auto-routing picks it or it was asked for by name
+ route = parakeet_route(wanted, language, wav, _shutil, _subprocess, parakeet_model, whisper_model=model)
+ done = parakeet(route.first, route.facts)
+ if done:
+ return done
+ tried.extend(_skipped_parakeet(route, language, language_flag))
+ if wanted in PARAKEET_ENGINES:
+ die_engine_failed(wanted, video, tried, _parakeet_available(wanted, _shutil, parakeet_model))
+ facts = route.facts
+ if route.first:
+ facts = dict(facts, routing=facts["routing"] + ROUTING_PARAKEET_FAILED)
+ lang = language or facts.get("detected_language")
+ failed: Optional[str] = None # the first Whisper engine that ran and wrote no word-timing JSON
+ # Undetected speech (Parakeet waits in route.last) was routed to Whisper because some
+ # Whisper engine can take it, so each one gets its turn; otherwise the first that ran
+ # and failed is the one the caller's refusal names, as before.
+ every_whisper = bool(route.last)
+
# 1. whisper.cpp
- cli = _shutil.which("whisper-cli") or _shutil.which("whisper-cpp")
- if not cli:
- main_bin = _shutil.which("main")
- if main_bin and "whisper" in os.path.dirname(os.path.realpath(main_bin)).lower():
- cli = main_bin
+ cli = _whisper_cpp_cli(_shutil) if wanted in ("auto", "whisper.cpp") else None
if cli:
- model_path = model
- if not os.path.exists(model_path):
- for cand in (os.path.expanduser(f"~/.cache/whisper.cpp/ggml-{model}.bin"),
- f"models/ggml-{model}.bin",
- f"/usr/local/share/whisper/ggml-{model}.bin"):
- if os.path.exists(cand):
- model_path = cand
- break
+ model_path = _whisper_cpp_model(model)
base = os.path.join(tmpdir, "out")
- cmd = [cli, "-m", model_path, "-f", wav, "--output-json-full", "-of", base]
- if language:
- cmd += ["-l", language]
+ # whisper.cpp's own default is -l en: without "auto" it decodes any speech as English
+ cmd = [cli, "-m", model_path, "-f", wav, "--output-json-full", "-of", base, "-l", lang or "auto"]
proc = _asr_run(cmd, _subprocess, "whisper.cpp")
if proc.returncode == 0 and os.path.exists(base + ".json"):
words = _words_from_whisper_cpp_json(base + ".json")
info(f"word timings from whisper.cpp ({len(words)} words)")
- return words, "whisper.cpp"
- info("whisper.cpp found but produced no word-timing JSON: "
- + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
- return [], "whisper.cpp"
+ return Transcription(words=words, engine="whisper.cpp",
+ facts=dict(facts, engine="whisper.cpp", model=model_path, language=lang))
+ line = _failure_line(proc)
+ info("whisper.cpp found but produced no word-timing JSON: " + line)
+ tried.append(_failed("whisper.cpp", "found but produced no word-timing JSON: " + line))
+ failed = "whisper.cpp"
# 2. faster-whisper
try:
+ if (failed and not every_whisper) or wanted not in ("auto", "faster-whisper"):
+ raise ImportError
from faster_whisper import WhisperModel # type: ignore
import threading
collected: list = []
+ crashed: list = []
def work() -> None:
- m = WhisperModel(model, device="cpu", compute_type="int8")
- segments, _ = m.transcribe(wav, language=language, word_timestamps=True)
- for seg in segments:
- for w in (getattr(seg, "words", None) or []):
- collected.append({"word": str(w.word).strip(),
- "start": float(w.start), "end": float(w.end)})
+ try:
+ m = WhisperModel(model, device="cpu", compute_type="int8")
+ segments, _ = m.transcribe(wav, language=lang, word_timestamps=True)
+ for seg in segments:
+ for w in (getattr(seg, "words", None) or []):
+ collected.append({"word": str(w.word).strip(),
+ "start": float(w.start), "end": float(w.end)})
+ except Exception as exc: # noqa: BLE001 -- a model that would not load or run
+ crashed.append(exc)
t = threading.Thread(target=work, daemon=True)
t.start()
@@ -364,29 +994,52 @@ def transcribe_words(video: str, language: "Optional[str]" = None, model: str =
if t.is_alive():
die(f"faster-whisper exceeded the {STATE.timeout:.0f} s time limit; raise "
"--timeout for a long recording", code=124, kind="timeout")
- info(f"word timings from faster-whisper ({len(collected)} words)")
- return [w for w in collected if w["word"]], "faster-whisper"
+ if crashed:
+ line = str(crashed[0])[:200] or type(crashed[0]).__name__
+ info("faster-whisper found but failed: " + line)
+ tried.append(_failed("faster-whisper", "found but failed: " + line))
+ failed = failed or "faster-whisper"
+ else:
+ info(f"word timings from faster-whisper ({len(collected)} words)")
+ return Transcription(words=[w for w in collected if w["word"]], engine="faster-whisper",
+ facts=dict(facts, engine="faster-whisper", model=model, language=lang))
except ImportError:
pass
# 3. openai-whisper CLI
- if _shutil.which("whisper"):
+ if (not failed or every_whisper) and wanted in ("auto", "openai-whisper") and _shutil.which("whisper"):
cmd = ["whisper", wav, "--model", model, "--word_timestamps", "True",
"--output_format", "json", "--output_dir", tmpdir]
- if language:
- cmd += ["--language", language]
+ if lang:
+ cmd += ["--language", lang]
proc = _asr_run(cmd, _subprocess, "openai-whisper")
doc = os.path.join(tmpdir, "audio.json")
if proc.returncode == 0 and os.path.exists(doc):
words = _words_from_openai_whisper_json(doc)
info(f"word timings from openai-whisper ({len(words)} words)")
- return words, "openai-whisper"
- info("openai-whisper found but produced no word-timing JSON: "
- + (proc.stderr.strip().splitlines() or ["?"])[-1][:200])
- return [], "openai-whisper"
+ return Transcription(words=words, engine="openai-whisper",
+ facts=dict(facts, engine="openai-whisper", model=model, language=lang))
+ line = _failure_line(proc)
+ info("openai-whisper found but produced no word-timing JSON: " + line)
+ tried.append(_failed("openai-whisper", "found but produced no word-timing JSON: " + line))
+ failed = failed or "openai-whisper"
+ # 4. Parakeet after all, when the language was never known and no Whisper engine gave words
+ if route.last:
+ info("no Whisper engine produced word timings; trying Parakeet with English assumed")
+ done = parakeet(list(route.last), {"routing": ROUTING_LAST_RESORT})
+ if done:
+ return done
+ if failed and not route.last:
+ # it ran: the caller refuses naming it ("no word-level timings"), not "no engine".
+ # After a last-resort Parakeet run, the refusal below names every engine tried.
+ return Transcription(engine=failed, facts=facts)
+ if wanted != "auto":
+ die_engine_failed(wanted, video, tried, False)
+ if tried:
+ die_engines_failed(tried, video, "--filler --transcribe", "or pass --words with a transcript you already have.")
die_no_engine("or pass --words with a transcript you already have.",
flag="--filler --transcribe")
- return [], None
+ return Transcription()
finally:
_shutil.rmtree(tmpdir, ignore_errors=True)
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 47edb42..f90ab43 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -54,6 +54,8 @@ ROLES = {
# filter:<name> `ffmpeg -filters`
# bsf:<name> `ffmpeg -bsfs`
# external:whisper a local whisper engine (whisper.cpp / faster-whisper / openai-whisper)
+# external:parakeet a local Parakeet engine (parakeet-mlx, or parakeet-cli with a .gguf model);
+# English. Either external engine satisfies --transcribe.
# "optional" entries name the flag or condition under which the capability is needed.
FF = ["ffmpeg", "ffprobe"]
X264 = "encoder:libx264"
@@ -145,7 +147,7 @@ TOOL_META: Dict[str, Dict[str, Any]] = {
required=FF + [X264], optional=[],
video_required=True, audio_only=False, visual=True, verify=["probe", "look"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"caption": dict(role="execution", inputs=["video asset", "SRT/ASS file or timed text (--text)"], outputs=["video artifact with burnt-in captions (--mode burn)", "video artifact with one or several language-tagged soft subtitle streams (--mode mux, --srt file:lang repeated)", "generated .srt / .ass sidecar"],
- required=FF + [X264, AAC, "filter:subtitles"], optional=[{"capability": "filter:ass", "when": "--animate / --karaoke"}, HDR_X265, {"capability": "external:whisper", "when": "--transcribe"},
+ required=FF + [X264, AAC, "filter:subtitles"], optional=[{"capability": "filter:ass", "when": "--animate / --karaoke"}, HDR_X265, {"capability": "external:whisper", "when": "--transcribe"}, {"capability": "external:parakeet", "when": "--transcribe (English; either engine family satisfies it)"},
{"capability": "encoder:mov_text", "when": "--mode mux with a .mp4/.m4v/.mov output"}, {"capability": "encoder:webvtt", "when": "--mode mux with a .webm output"}, {"capability": "encoder:srt", "when": "--mode mux with a .mkv output"}],
video_required=True, audio_only=False, visual=True, verify=["probe", "look"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"overlay": dict(role="execution", inputs=["video asset", "image (--image / --logo), text (--text), or a second video (--video) to composite"], outputs=["video artifact with the overlay composited"],
@@ -172,7 +174,8 @@ TOOL_META: Dict[str, Dict[str, Any]] = {
video_required=False, audio_only=True, visual=False, verify=["probe", "check"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"silence": dict(role="analysis_and_execution", inputs=["video or audio asset"], outputs=["silence list JSON (--list)", "artifact with silences removed", "EDL text (--edl)"],
required=FF + ["filter:silencedetect"], optional=[{"capability": X264, "when": "removing silences from a video"}, HDR_X265, {"capability": AAC, "when": "removing silences from a video"},
- {"capability": "external:whisper", "when": "--filler --transcribe"}] + AUDIO_OUT,
+ {"capability": "external:whisper", "when": "--filler --transcribe"},
+ {"capability": "external:parakeet", "when": "--filler --transcribe (English; either engine family satisfies it)"}] + AUDIO_OUT,
video_required=False, audio_only=True, visual=False, verify=["probe"], produces_artifact=True, idempotency="content_equivalent", deterministic=True),
"join": dict(role="execution", inputs=["two or more video assets, or two or more audio-only assets"], outputs=["concatenated video artifact", "concatenated audio artifact (audio-only inputs, audio output extension)"],
required=FF + ["filter:xfade", "filter:acrossfade"],
@@ -409,7 +412,8 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"notes": {"type": "array", "items": {"type": "string"}, "description": "present when no --platform was named: youtube was assumed and judgement rows are WARN"},
"checks": {"type": "array", "items": {"type": "object", "properties": {"check": {"type": "string", "description": "row name; --content adds black, frozen and silence; audio FAILs when the track is present but silent (peak <= -50 dBFS)"}, "status": {"enum": ["PASS", "WARN", "FAIL"]}, "value": {}, "expected": {}, "fix": {"type": "string"}, "kind": {"enum": ["format", "judgement"]}}}}}
elif name == "caption":
- extra = {"caption": {"type": "object", "description": "cue layout: shifted / wrapped / rebalanced / split / extended / dropped counts, plus wrap ('phrase' or 'measured'), phrase_breaks (1.16), broken_inside_word -- atoms hard-sliced at the column edge because they did not fit alone even at the size floor (1.18.4) -- and overlong -- now residual: a single character alone wider than the column. Burn mode: cues_burned -- non-blank cues that overlap [0, duration] and are drawn -- and cues_outside -- non-blank cues wholly outside it (2.2.6); a burn with no visible cue is refused, kind input"},
+ extra = {"transcription": {"type": "object", "description": "with --transcribe: {engine, model, language, routing (why auto chose it), detected_language when auto detected it}; when auto ran the English-only Parakeet model on English assumed, notes says so"},
+ "caption": {"type": "object", "description": "cue layout: shifted / wrapped / rebalanced / split / extended / dropped counts, plus wrap ('phrase' or 'measured'), phrase_breaks (1.16), broken_inside_word -- atoms hard-sliced at the column edge because they did not fit alone even at the size floor (1.18.4) -- and overlong -- now residual: a single character alone wider than the column. Burn mode: cues_burned -- non-blank cues that overlap [0, duration] and are drawn -- and cues_outside -- non-blank cues wholly outside it (2.2.6); a burn with no visible cue is refused, kind input"},
"tracks": {"type": "array", "description": "--mode mux: one entry per subtitle stream in the output ({index, file, language, title, codec, default, cues, kept_from_input}); a stream the input already carried has file null and kept_from_input true (1.16)"},
"subtitle_tracks": {"type": "integer", "description": "--mode mux: how many subtitle streams the output carries"},
"emoji": {"type": "object", "description": "how the emoji in the text were drawn (mode, overlays, missing)"},
@@ -430,7 +434,9 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"speech": {"type": "array", "description": "--speech (1.18): [{time, speech_music_ratio}]"}}
elif name == "silence":
extra = {"silences": {"type": "array"}, "keep": {"type": "array"}, "input_duration": {"type": "number"}, "kept_duration": {"type": "number"}, "removed_seconds": {"type": "number"},
- "speech_aware": {"type": "object", "description": "--speech-aware (1.18): {min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}"}}
+ "speech_aware": {"type": "object", "description": "--speech-aware (1.18): {min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}"},
+ "filler": {"type": "object", "description": "--filler: {lang, source (whisper-json:FILE | whisper:ENGINE | parakeet:ENGINE), engine, transcription (with --transcribe: {engine, model, language, routing}), removed, removed_count, ...}"},
+ "notes": {"type": "array", "items": {"type": "string"}, "description": "--filler --transcribe under --engine auto: present when the English-only Parakeet model ran on English assumed (no --filler-lang, no language detected, no Whisper engine installed or none that succeeded); the transcript is wrong if the speech is not English"}}
elif name == "sync":
extra = {"reference": {"type": "string"}, "second": {"type": "string"}, "offset_seconds": {"type": "number"}, "confidence": {"type": "number"}, "meaning": {"type": "string"}, "drift": {"type": "object"},
"sources": {"type": "array", "description": "1.18: [{path, offset_s, confidence, drift_ppm}], one per SOURCE; the only per-source shape once more than one SOURCE is given"}}
@@ -672,6 +678,13 @@ def _whisper_available() -> bool:
return importlib.util.find_spec("faster_whisper") is not None or importlib.util.find_spec("whisper") is not None
+def _parakeet_available() -> bool:
+ """parakeet-mlx on PATH, or parakeet-cli with a model it can find (PARAKEET_CPP_MODEL or a
+ .gguf in ~/.cache/parakeet.cpp) -- the same test the speech bridge applies before running it."""
+ from _common.asr import _parakeet_available as runtime_check # the bridge's own test, not a copy of it
+ return any(runtime_check(e, shutil) for e in ("parakeet-mlx", "parakeet.cpp"))
+
+
def _hw_default() -> Dict[str, Any]:
"""--hw's machine facts, normalised (never the raw environment value): whether VideoToolbox
constant-quality encoding can run here (the tools' own check: Apple Silicon hardware, also
@@ -870,6 +883,8 @@ def doctor() -> Dict[str, Any]:
state[cap] = _from("bsfs", cap[4:])
elif cap == "external:whisper":
state[cap] = "available" if _whisper_available() else "missing"
+ elif cap == "external:parakeet":
+ state[cap] = "available" if _parakeet_available() else "missing"
else:
state[cap] = "missing"
available = sorted(c for c, st in state.items() if st == "available")
@@ -997,7 +1012,15 @@ def _capability_fix_hint(cap: str) -> str:
return (f"no installed font covers {LANGUAGE_NAMES.get(script, script)} text on this machine; "
f"{FONT_INSTALL_HINT} (doctor --json .fonts.scripts lists every script)")
if cap == "external:whisper":
- return "install a local whisper (whisper-cli, whisper-cpp, faster-whisper or openai-whisper) for --transcribe"
+ return ("install a local whisper (whisper-cli, whisper-cpp, faster-whisper or openai-whisper) for --transcribe; "
+ "faster-whisper and openai-whisper download their model on first use, and every engine transcribes "
+ "on this machine (the audio is never uploaded)")
+ if cap == "external:parakeet":
+ from _common.asr import PARAKEET_MLX_DEFAULT_MODEL
+ return ("for English --transcribe, install parakeet-mlx (uv tool install parakeet-mlx; its first run downloads "
+ f"{PARAKEET_MLX_DEFAULT_MODEL} from Hugging Face, then it transcribes on this machine and the audio is "
+ "never uploaded) or parakeet-cli with a tdt-0.6b-v2 .gguf in ~/.cache/parakeet.cpp; a local whisper "
+ "covers every language")
return f"'{cap}' is not available; see docs/contract.md"
@@ -1362,7 +1385,7 @@ def build(detect: bool = True) -> Dict[str, Any]:
version = skill_version()
tools = [tool_spec(n, version) for n in public_tools()]
wanted = required_capabilities()
- caps: Dict[str, Any] = {"required": wanted["required"], "optional": wanted["optional"], "naming": "ffmpeg | ffprobe | encoder:<name> | filter:<name> | bsf:<name> | external:whisper"}
+ caps: Dict[str, Any] = {"required": wanted["required"], "optional": wanted["optional"], "naming": "ffmpeg | ffprobe | encoder:<name> | filter:<name> | bsf:<name> | external:whisper | external:parakeet"}
if detect:
d = doctor()
caps.update({"available": d["available"], "missing": d["missing"], "missing_optional": d["missing_optional"],
@@ -1389,8 +1412,21 @@ def build(detect: bool = True) -> Dict[str, Any]:
"execution": {
"shell": False,
"arbitrary_executables": False,
- "subprocess": "argv list only: [python3, scripts/<tool>.py, ...] and [ffmpeg|ffprobe, ...] resolved from PATH",
+ "subprocess": "argv list only: [python3, scripts/<tool>.py, ...] and [ffmpeg|ffprobe, ...] resolved from PATH; "
+ "with --transcribe, a local speech engine's CLI (whisper.cpp's whisper-cli / whisper-cpp / main, "
+ "openai-whisper's whisper, parakeet-mlx, parakeet-cli) resolved from PATH",
"network": False,
+ # `network` is about the skill's own code. A speech engine may fetch its model weights the
+ # first time it runs; faster-whisper is a library, so its fetch is made from the skill's own
+ # process. A caller that sandboxes the network must know that before a first --transcribe.
+ "model_downloads": {
+ "when": "first use of a speech engine (--transcribe) whose model is not cached yet; media is never uploaded",
+ "in_process": {"faster-whisper": "Hugging Face (Systran/faster-whisper-<model>)"},
+ "child_process": {"parakeet-mlx": "Hugging Face (mlx-community/parakeet-*)",
+ "openai-whisper": "OpenAI (its model checkpoint)"},
+ "none": ["whisper.cpp", "parakeet.cpp"],
+ "offline": "HF_HUB_OFFLINE=1 keeps faster-whisper and parakeet-mlx on the cached copy",
+ },
"input_mutation": False,
},
"invocation": {
diff --git a/scripts/caption.py b/scripts/caption.py
index 7a20349..70714b2 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -49,6 +49,8 @@ from _common import emoji_filter_chain, EMOJI_ASSET_HINT, emoji_asset_for, emoji
# The ASR bridge and the SRT reader/writer live in _common.asr since 1.17 (silence.py --filler
# shares them). They stay caption.py's public names -- every caller and test that reached for
# caption.parse_srt / caption.transcribe / caption.whisper_word_timings before 1.17 still does.
+from _common import asr as _asr
+from _common.asr import ENGINE_CHOICES
from _common import (ASR_INSTALL_HINT, die_no_engine, parse_srt, transcribe, whisper_word_timings,
write_srt)
from _common import (SAFE_WIDTH_FRACTION, ORPHAN_MIN_EM, WRAP_MODES, wrap_text, wrap_variants, best_break,
@@ -901,7 +903,7 @@ def main() -> int:
"them; .mkv is the honest multi-track container and stores the code you give verbatim")
src.add_argument("--ass", help="ASS file to burn (styles inside the file are used)")
src.add_argument("--text", help="plain text cue file to convert into SRT (see format above)")
- src.add_argument("--transcribe", action="store_true", help="generate the SRT from the audio with a local speech-to-text engine if one is installed (whisper-cli / whisper / faster-whisper); never required")
+ src.add_argument("--transcribe", action="store_true", help="generate the SRT from the audio with a local speech-to-text engine if one is installed (parakeet-mlx / parakeet.cpp / whisper-cli / whisper / faster-whisper); never required")
src.add_argument("--language", "--lang", help="language code (e.g. en, ja, zh, ko): the language for --transcribe (default auto), "
"the tag on the subtitle stream with --mode mux, and the hint that says whether Han-only "
"text is Chinese, Japanese or Korean when a font is picked by script")
@@ -914,7 +916,12 @@ def main() -> int:
"(default: none, so no player burns in a language the viewer did not ask for)")
src.add_argument("--offset", default="0", help="shift every cue by TIME (seconds, mm:ss, hh:mm:ss.ms or "
"hh:mm:ss:ff; a leading - shifts earlier); works for --text, --srt and --ass")
- src.add_argument("--model", default="base", help="whisper model name/path for --transcribe (default base)")
+ src.add_argument("--model", default="base",
+ help="speech model for --transcribe: a whisper model name/path (default base), or a Parakeet "
+ "model (an mlx-community/parakeet-* repo or a .gguf) for the Parakeet engines")
+ src.add_argument("--engine", choices=ENGINE_CHOICES, default=None,
+ help="speech engine for --transcribe (default: $FFMPEG_SKILL_ASR_ENGINE, else auto = Parakeet "
+ "for English speech when installed, else whisper.cpp / faster-whisper / openai-whisper)")
src.add_argument("--write-srt", help="where to save the generated SRT (default: <text>.srt)")
src.add_argument("--auto-seconds", type=float, default=3.0, help="duration for cues without timing (default 3)")
src.add_argument("--gap", type=float, default=0.0, help="gap after auto-timed cues in seconds")
@@ -1241,8 +1248,13 @@ def main() -> int:
cues = []
info(f"[dry-run] would transcribe {args.input} and write {srt_path}")
else:
- cues = transcribe(args.input, srt_path, args.language, args.model, args.audio_stream)
- args._word_timings = whisper_word_timings(srt_path)
+ # the call's own result, not module state: engine, routing and words belong to this run
+ asr_run = _asr.transcribe_result(args.input, srt_path, args.language, args.model, args.audio_stream,
+ engine=args.engine)
+ cues = asr_run.cues
+ args._asr = dict(asr_run.facts)
+ args._asr_notes = list(asr_run.notes)
+ args._word_timings = whisper_word_timings(srt_path) or sorted((w["start"], w["end"], w["word"]) for w in asr_run.words)
cues, changed = lay_out(cues)
if changed:
write_srt(cues, srt_path)
@@ -1279,7 +1291,8 @@ def main() -> int:
# WRITING waits for a real run (the rule every side file in this tool follows), which is why
# the cues are kept in hand below for the font sample and the ASS generator.
planned_cues: Optional[List[Tuple[float, float, str]]] = None
- side_notes: List[str] = []
+ # an English-only transcript made on an assumption says so first (asr.assumed_english_note)
+ side_notes: List[str] = list(getattr(args, "_asr_notes", None) or [])
ass_sample_path = args.ass
if args.srt and not (args.text or args.transcribe) and os.path.exists(srt_path or ""):
adjusted, changed = lay_out(parse_srt(srt_path))
@@ -1416,7 +1429,8 @@ def main() -> int:
result = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(result.get('duration'))}, mux, {len(added)} "
f"subtitle track(s) added, codec {codec})")
- emit(output, tracks=tracks, subtitle_tracks=total, **({"notes": notes} if notes else {}))
+ emit(output, tracks=tracks, subtitle_tracks=total, **({"notes": notes} if notes else {}),
+ **({"transcription": args._asr} if getattr(args, "_asr", None) else {}))
return 0
# A font that covers the text, before anything is rendered: non-Latin cues in a Latin-only
@@ -1582,6 +1596,8 @@ def main() -> int:
info("caption text unchanged: the cues were burned exactly as given (line breaks, timing "
"and type size only)")
extra["caption"] = dict(caption_stats)
+ if getattr(args, "_asr", None):
+ extra["transcription"] = args._asr
if emoji_plan:
notes = list(extra.get("notes") or [])
if emoji_plan["mode"] == "mono":
diff --git a/scripts/silence.py b/scripts/silence.py
index 48d9901..1d37694 100755
--- a/scripts/silence.py
+++ b/scripts/silence.py
@@ -20,7 +20,8 @@ from typing import List, Tuple
# `detect` moved into _common/probe.py in 1.16.0 so metadata.py --auto-chapters can measure the
# same silences without importing this tool; the body is unchanged and the name still lives here.
from _common import (filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS,
- FILLER_PAD, transcribe_words, read_text_or_die)
+ FILLER_PAD, read_text_or_die)
+from _common import asr as _asr
from _common import detect_silences as detect, STATE, video_args, add_common, apply_common, audio_codec_for, cfr_args, default_output, die, emit, ffmpeg_base, info, is_audio_output, print_json, probe, run, X264_PRESETS, measured_level_dbfs, fmt_secs
@@ -99,7 +100,7 @@ def resolve_filler(args, meta):
die("--filler needs word timings: pass --words transcript.json (a whisper JSON with word "
"timestamps) or --transcribe. There is no way to find a filler word without them -- "
"cutting the short quiet blips instead would remove real speech.", kind="input")
- source, engine, raw = None, None, None
+ source, engine, raw, transcription = None, None, None, {}
if args.words:
try:
raw = json.loads(read_text_or_die(args.words, "--words"))
@@ -118,14 +119,20 @@ def resolve_filler(args, meta):
'{"words": [{"word": ..., "start": ..., "end": ...}]} (or the same inside '
'"segments").', kind="input")
else:
- # No pre-check for an installed engine here: transcribe_words() probes for one and
+ # No pre-check for an installed engine here: transcribe_words_result() probes for one and
# raises the same die_no_engine() refusal when there is none. Two places deciding "is
# whisper here" is two places to disagree.
# The engine is driven with ITS word-timestamp option (whisper.cpp --output-json-full,
# faster-whisper word_timestamps=True, openai-whisper --word_timestamps True). An SRT
# cannot answer this question: a cue has a start and an end, a word does not.
- words, engine = transcribe_words(args.input, args.filler_lang if args.filler_lang != "auto" else None)
- source = f"whisper:{engine}" if engine else "whisper"
+ # The call's own result, not module state: engine, routing and notes belong to this run.
+ asr_run = _asr.transcribe_words_result(args.input, args.filler_lang if args.filler_lang != "auto" else None,
+ engine=args.engine)
+ words, engine = asr_run.words, asr_run.engine
+ transcription = dict(asr_run.facts)
+ args._asr_notes = list(asr_run.notes)
+ family = "parakeet" if engine in _asr.PARAKEET_ENGINES else "whisper"
+ source = f"{family}:{engine}" if engine else "whisper"
if not words:
die(f"{engine or 'the local engine'} ran but produced no word-level timings, so there "
"is nothing for --filler to cut on. Some builds do not support word timestamps. "
@@ -168,6 +175,7 @@ def resolve_filler(args, meta):
"disfluency -- this will cut real sentences.")
block = {
"lang": lang, "source": source, "engine": engine,
+ **({"transcription": transcription} if engine and transcription else {}),
"words": sorted(wordlist), "removed": [dict(s) for s in spans],
"removed_count": len(spans),
"removed_seconds": round(sum(s["end"] - s["start"] for s in spans), 3),
@@ -235,8 +243,11 @@ def main() -> int:
fil.add_argument("--words", metavar="FILE",
help="a whisper JSON with word-level timings, for --filler")
fil.add_argument("--transcribe", action="store_true",
- help="produce the word timings with a local whisper (never required; the same "
+ help="produce the word timings with a local speech engine, Parakeet or whisper (never required; the same "
"bridge caption.py uses)")
+ fil.add_argument("--engine", choices=_asr.ENGINE_CHOICES, default=None,
+ help="speech engine for --transcribe (default: $FFMPEG_SKILL_ASR_ENGINE, else auto = Parakeet "
+ "for English speech when installed, else whisper)")
fil.add_argument("--filler-list", action="store_true",
help="report what --filler would remove and write nothing")
fil.add_argument("--max-cuts", type=int, default=400,
@@ -311,6 +322,10 @@ def main() -> int:
f"({filler_info['lang']}, list {filler_info['list']})")
for warning in filler_info.get("warnings") or []:
info("warning: " + warning)
+ if getattr(args, "_asr_notes", None):
+ # an English-only transcript made on an assumption (asr.assumed_english_note): the
+ # bridge already logged the warning; the result document carries it at the top level
+ summary["notes"] = list(args._asr_notes)
if len(keeps) > args.max_cuts:
die(f"{len(keeps)} keep ranges is above --max-cuts {args.max_cuts}: the filter graph grows "
"with every range and a graph this size is slow and fragile. Raise --max-cuts if you "
diff --git a/tests/_fixtures.py b/tests/_fixtures.py
index 337317b..f0b94be 100644
--- a/tests/_fixtures.py
+++ b/tests/_fixtures.py
@@ -21,6 +21,11 @@ SCRIPTS = ROOT / "scripts"
HERE = Path(__file__).resolve().parent
OUT = Path(os.environ.get("OUT", ROOT / "tests" / "out"))
+# The host's own defaults must not reach the suite (every tool subprocess inherits os.environ):
+# FFMPEG_SKILL_ASR_ENGINE or a PARAKEET_* setting would change which speech engine a --transcribe
+# test drives, or where parakeet-mlx writes. Tests that exercise those opt in with an explicit env.
+for _k in ("FFMPEG_SKILL_ASR_ENGINE", "PARAKEET_MODEL", "PARAKEET_CPP_MODEL", "PARAKEET_OUTPUT_TEMPLATE"):
+ os.environ.pop(_k, None)
sys.path.insert(0, str(SCRIPTS))
sys.path.insert(0, str(HERE))
from _common import escape_filter_path # noqa: E402
diff --git a/tests/fixtures/mcp_tools.json b/tests/fixtures/mcp_tools.json
index 4689475..3b43a28 100644
--- a/tests/fixtures/mcp_tools.json
+++ b/tests/fixtures/mcp_tools.json
@@ -151,6 +151,7 @@
"emoji_assets",
"emoji_max",
"emoji_scale",
+ "engine",
"fast",
"fit_size",
"fit_size_scope",
@@ -1034,6 +1035,7 @@
"codec",
"dry_run",
"edl",
+ "engine",
"fast",
"filler",
"filler_extra",
diff --git a/tests/test_all.py b/tests/test_all.py
index 6286273..4c37d00 100644
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -4,7 +4,7 @@
python3 tests/test_all.py # or: python3 -m unittest tests/test_all.py
The cases themselves live one module per tool group -- test_analysis, test_editing, test_audio,
-test_picture, test_delivery, test_orchestration, test_accel -- over the fixtures in _fixtures.py. This file
+test_picture, test_delivery, test_orchestration, test_accel, test_asr -- over the fixtures in _fixtures.py. This file
imports all seven so that the entry point every CI job, npm test and release_check.sh already runs
keeps running exactly the same set; each group module is also runnable on its own.
"""
@@ -16,13 +16,14 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
import test_accel # noqa: E402
import test_analysis # noqa: E402
+import test_asr # noqa: E402
import test_audio # noqa: E402
import test_delivery # noqa: E402
import test_editing # noqa: E402
import test_orchestration # noqa: E402
import test_picture # noqa: E402
-MODULES = (test_analysis, test_editing, test_audio, test_picture, test_delivery, test_orchestration, test_accel)
+MODULES = (test_analysis, test_editing, test_audio, test_picture, test_delivery, test_orchestration, test_accel, test_asr)
def load_tests(loader, tests, pattern):
diff --git a/tests/test_asr.py b/tests/test_asr.py
new file mode 100644
index 0000000..1d601da
--- /dev/null
+++ b/tests/test_asr.py
@@ -0,0 +1,976 @@
+#!/usr/bin/env python3
+"""Tests for the Parakeet speech engines.
+
+ python3 tests/test_asr.py # this group alone
+ python3 tests/test_all.py # every group
+
+The Parakeet engines are driven through fake binaries on a PATH that holds nothing else, and a
+faster_whisper the host may have installed is hidden from those runs, so these cases never depend on
+what the host has installed.
+"""
+import json
+import os
+import platform
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import textwrap
+import unittest
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from _fixtures import MediaFixtures, OUT, script, sh # noqa: E402
+from _common import asr # noqa: E402
+
+
+def _assumed(doc):
+ """The result document's top-level notes that say English was assumed."""
+ return [n for n in doc.get("notes") or [] if n.startswith("English was assumed")]
+
+
+class ParakeetCppModelTests(unittest.TestCase):
+ """parakeet.cpp runs only with a .gguf it can find: an explicit --model is enough, for a
+ forced engine and for --engine auto alike."""
+
+ def test_parakeet_cpp_runs_with_only_an_explicit_model(self):
+ with tempfile.TemporaryDirectory() as d, mock.patch.dict(os.environ, {"HOME": d}):
+ os.environ.pop("PARAKEET_CPP_MODEL", None)
+ gguf = Path(d) / "tdt-0.6b-v3-f16.gguf"
+ gguf.write_bytes(b"GGUF")
+ which = mock.Mock(which=lambda n: "/x/parakeet-cli" if n == "parakeet-cli" else None)
+ self.assertFalse(asr._parakeet_available("parakeet.cpp", which))
+ self.assertTrue(asr._parakeet_available("parakeet.cpp", which, str(gguf)))
+ self.assertEqual(asr._parakeet_model_for("parakeet.cpp", str(gguf), "de"), str(gguf))
+ # --engine auto must see it too, not only a forced --engine parakeet.cpp
+ self.assertEqual(asr.parakeet_route("auto", "en", "a.wav", which, subprocess)[0], [])
+ self.assertEqual(asr.parakeet_route("auto", "en", "a.wav", which, subprocess, str(gguf))[0], list(asr.PARAKEET_ENGINES))
+
+
+MLX_DOC = {"text": "So um we start. Here.", "sentences": [
+ {"text": "So um we start.", "start": 0.2, "end": 1.6, "tokens": [
+ {"text": " So", "start": 0.2, "end": 0.4}, {"text": " u", "start": 0.5, "end": 0.6}, {"text": "m", "start": 0.6, "end": 0.7},
+ {"text": " ", "start": 0.7, "end": 0.8}, {"text": "we", "start": 0.9, "end": 1.0}, {"text": " start", "start": 1.1, "end": 1.5},
+ {"text": ".", "start": 1.5, "end": 1.6}]},
+ {"text": "Here.", "start": 2.0, "end": 2.4, "tokens": [{"text": " Here", "start": 2.0, "end": 2.3}, {"text": ".", "start": 2.3, "end": 2.4}]}]}
+CPP_DOC = {"text": "So um we start. Here.", "frame_sec": 0.08, "words": [
+ {"w": "So", "start": 0.2, "end": 0.4, "conf": 0.99}, {"w": "um", "start": 0.5, "end": 0.7, "conf": 0.9},
+ {"w": "we", "start": 0.9, "end": 1.0, "conf": 0.99}, {"w": "start.", "start": 1.1, "end": 1.6, "conf": 0.99},
+ {"w": "Here.", "start": 2.0, "end": 2.4, "conf": 0.99}]}
+
+
+class ParakeetParsingTests(unittest.TestCase):
+ def test_mlx_subword_tokens_become_words(self):
+ with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh:
+ json.dump(MLX_DOC, fh)
+ words = asr._words_from_parakeet_mlx_json(fh.name)
+ self.assertEqual([w["word"] for w in words], ["So", "um", "we", "start.", "Here."])
+ self.assertEqual((words[1]["start"], words[1]["end"]), (0.5, 0.7))
+ self.assertEqual(asr._cues_from_parakeet_mlx_json(fh.name), [(0.2, 1.6, "So um we start."), (2.0, 2.4, "Here.")])
+ os.unlink(fh.name)
+
+ def test_cpp_words_and_the_cues_built_from_them(self):
+ words = asr._words_from_parakeet_cpp_json(json.dumps(CPP_DOC))
+ self.assertEqual(len(words), 5)
+ self.assertEqual(asr.cues_from_words(words), [(0.2, 1.6, "So um we start."), (2.0, 2.4, "Here.")])
+
+ def test_malformed_engine_output_yields_no_words_instead_of_crashing(self):
+ """Garbage, the wrong shape, and words missing fields are skipped, never raised."""
+ for text in ("not json", "[]", "null", json.dumps({"words": "x"}), json.dumps({"words": 5}),
+ json.dumps({"words": [5, None, []]})):
+ self.assertEqual(asr._words_from_parakeet_cpp_json(text), [], text)
+ mixed = {"words": [{"w": "ok", "start": 0.1, "end": 0.3}, {"w": "no-times"}, {"start": 1, "end": 2},
+ {"w": "bad", "start": "x", "end": 1}, "str", {"w": "fine", "start": 1.0, "end": 1.2}]}
+ self.assertEqual([w["word"] for w in asr._words_from_parakeet_cpp_json(json.dumps(mixed))], ["ok", "fine"])
+ with tempfile.TemporaryDirectory() as d:
+ p = Path(d) / "a.json"
+ for body in ("{truncated", "[1, 2]", json.dumps({"sentences": [{"text": "no times"}]}),
+ json.dumps({"sentences": 5}), json.dumps({"sentences": [5, None]}),
+ json.dumps({"sentences": [{"tokens": 5}]}), json.dumps({"sentences": [{"tokens": [5, None]}]})):
+ p.write_text(body)
+ self.assertEqual(asr._cues_from_parakeet_mlx_json(str(p)), [], body)
+ self.assertEqual(asr._words_from_parakeet_mlx_json(str(p)), [], body)
+ self.assertEqual(asr._words_from_parakeet_mlx_json(str(Path(d) / "missing.json")), [])
+
+ def test_cues_split_on_a_pause_and_on_length(self):
+ words = [{"word": "a", "start": 0.0, "end": 0.2}, {"word": "b", "start": 1.5, "end": 1.7}]
+ self.assertEqual(len(asr.cues_from_words(words)), 2)
+ # 0.5 s apart, 0.4 s long: w13 ends at 6.9 s, w14 would stretch the cue to 7.4 s (> 7 s)
+ long = [{"word": f"w{i}", "start": i * 0.5, "end": i * 0.5 + 0.4} for i in range(20)]
+ self.assertEqual(asr.cues_from_words(long), [
+ (0.0, 6.9, " ".join(f"w{i}" for i in range(14))),
+ (7.0, 9.9, " ".join(f"w{i}" for i in range(14, 20)))])
+
+
+class ParakeetRoutingTests(unittest.TestCase):
+ def test_auto_routes_by_language(self):
+ which = mock.Mock(side_effect=lambda n: "/x/" + n if n == "parakeet-mlx" else None)
+ sh_ = mock.Mock(which=which)
+ self.assertEqual(asr.parakeet_route("auto", "en", "a.wav", sh_, subprocess)[0], ["parakeet-mlx", "parakeet.cpp"])
+ # ISO 639-2, the spelling caption.py's --language also tags a muxed track with
+ self.assertEqual(asr.parakeet_route("auto", "eng", "a.wav", sh_, subprocess)[0], ["parakeet-mlx", "parakeet.cpp"])
+ self.assertEqual(asr.parakeet_route("auto", "fr", "a.wav", sh_, subprocess)[0], [])
+ self.assertEqual(asr.parakeet_route("auto", "fr", "a.wav", sh_, subprocess).skipped, ("parakeet-mlx",))
+ self.assertEqual(asr.parakeet_route("whisper.cpp", None, "a.wav", sh_, subprocess)[0], [])
+ self.assertEqual(asr.parakeet_route("parakeet.cpp", "de", "a.wav", sh_, subprocess)[0], ["parakeet.cpp"])
+ with mock.patch.object(asr, "detect_language", return_value="ja"):
+ self.assertEqual(asr.parakeet_route("auto", None, "a.wav", sh_, subprocess)[0], [])
+ with mock.patch.object(asr, "detect_language", return_value=None), \
+ mock.patch.object(asr, "_whisper_ready", return_value=False):
+ route = asr.parakeet_route("auto", None, "a.wav", sh_, subprocess)
+ self.assertEqual(route.first, ["parakeet-mlx", "parakeet.cpp"])
+ self.assertEqual(route.last, ())
+ self.assertIn("assumed English", route.facts["routing"])
+
+ def test_english_only_model_refuses_another_language(self):
+ with mock.patch.dict(os.environ, {}, clear=False):
+ os.environ.pop("PARAKEET_MODEL", None)
+ with self.assertRaises(SystemExit):
+ asr._parakeet_model_for("parakeet-mlx", None, "fr")
+ self.assertEqual(asr._parakeet_model_for("parakeet-mlx", None, "eng"), asr.PARAKEET_MLX_DEFAULT_MODEL)
+ self.assertIn("v3", asr._parakeet_model_for("parakeet-mlx", "mlx-community/parakeet-tdt-0.6b-v3", "fr"))
+ self.assertEqual(asr._parakeet_model_for("parakeet-mlx", "large-v3-turbo", "en"), asr.PARAKEET_MLX_DEFAULT_MODEL)
+
+ def test_unknown_engine_in_the_environment_is_refused(self):
+ with mock.patch.dict(os.environ, {asr.ASR_ENGINE_ENV: "vosk"}):
+ with self.assertRaises(SystemExit):
+ asr.requested_engine(None)
+
+ def test_engine_flag_beats_the_environment_which_beats_auto(self):
+ with mock.patch.dict(os.environ, {}, clear=False):
+ os.environ.pop(asr.ASR_ENGINE_ENV, None)
+ self.assertEqual(asr.requested_engine(None), "auto")
+ os.environ[asr.ASR_ENGINE_ENV] = "parakeet.cpp"
+ self.assertEqual(asr.requested_engine(None), "parakeet.cpp")
+ self.assertEqual(asr.requested_engine("whisper.cpp"), "whisper.cpp")
+
+
+class UndetectedLanguageTests(unittest.TestCase):
+ """Review of #305: with no --language and nothing to detect one, auto handed the speech to the
+ English-only Parakeet model and the only trace was `transcription.routing`. A Whisper engine
+ that can run now goes first; only with none does Parakeet run on English assumed, and then
+ the caller gets a top-level `notes` line saying the transcript may be wrong."""
+
+ def setUp(self):
+ self.mlx_only = mock.Mock(which=lambda n: "/x/parakeet-mlx" if n == "parakeet-mlx" else None)
+
+ def test_a_runnable_whisper_goes_first_and_parakeet_waits(self):
+ with mock.patch.object(asr, "detect_language", return_value=None), \
+ mock.patch.object(asr, "_whisper_ready", return_value=True):
+ route = asr.parakeet_route("auto", None, "a.wav", self.mlx_only, subprocess)
+ self.assertEqual(route.first, [])
+ self.assertEqual(route.last, asr.PARAKEET_ENGINES)
+ self.assertEqual(route.facts, {"routing": asr.ROUTING_UNDETECTED_WHISPER})
+
+ def test_only_parakeet_runs_on_english_assumed(self):
+ with mock.patch.object(asr, "detect_language", return_value=None), \
+ mock.patch.object(asr, "_whisper_ready", return_value=False):
+ route = asr.parakeet_route("auto", None, "a.wav", self.mlx_only, subprocess)
+ self.assertEqual(route.first, list(asr.PARAKEET_ENGINES))
+ self.assertEqual(route.facts, {"routing": asr.ROUTING_ASSUMED_ENGLISH})
+
+ def test_a_named_language_or_a_detected_one_never_asks_for_whisper(self):
+ with mock.patch.object(asr, "_whisper_ready", side_effect=AssertionError("not consulted")):
+ self.assertEqual(asr.parakeet_route("auto", "en", "a.wav", self.mlx_only, subprocess).first,
+ list(asr.PARAKEET_ENGINES))
+ with mock.patch.object(asr, "detect_language", return_value="en"):
+ self.assertEqual(asr.parakeet_route("auto", None, "a.wav", self.mlx_only, subprocess).last, ())
+ # a named engine runs as asked
+ self.assertEqual(asr.parakeet_route("parakeet-mlx", None, "a.wav", self.mlx_only, subprocess),
+ asr.Route(["parakeet-mlx"], {"routing": "requested"}))
+
+ def test_whisper_ready_means_an_engine_auto_would_run(self):
+ with tempfile.TemporaryDirectory() as d, mock.patch.dict(sys.modules, {"faster_whisper": None}):
+ model = Path(d) / "ggml-base.bin"
+ cli_only = mock.Mock(which=lambda n: "/x/whisper-cli" if n == "whisper-cli" else None)
+ self.assertFalse(asr._whisper_ready(cli_only, str(Path(d) / "missing.bin")), "whisper-cli with no model")
+ model.write_bytes(b"")
+ self.assertTrue(asr._whisper_ready(cli_only, str(model)))
+ self.assertFalse(asr._whisper_ready(cli_only, "mlx-community/parakeet-tdt-0.6b-v3"), "a Parakeet model")
+ self.assertFalse(asr._whisper_ready(cli_only, str(Path(d) / "tdt-0.6b-v2-f16.gguf")))
+ self.assertTrue(asr._whisper_ready(mock.Mock(which=lambda n: "/x/whisper" if n == "whisper" else None), "base"))
+ self.assertFalse(asr._whisper_ready(self.mlx_only, "base"))
+ stub = type(sys)("faster_whisper") # a module with no __spec__, as a test stub is
+ with mock.patch.dict(sys.modules, {"faster_whisper": stub}):
+ self.assertTrue(asr._whisper_ready(self.mlx_only, "base"))
+
+ def test_the_note_names_the_callers_flag(self):
+ self.assertIn("--filler-lang", asr.assumed_english_note("--filler-lang"))
+ self.assertIn("install a Whisper engine", asr.assumed_english_note())
+ self.assertIn("fix the Whisper engine", asr.assumed_english_note(whisper_failed=True))
+
+
+class _FakeEngines:
+ """shutil / subprocess stand-ins for the bridge: `which` answers only for `names`, and `run`
+ writes what the named engine would have written where its arguments say. An engine in `fail`
+ exits 1 the way the real one does: parakeet-mlx prints its error to stdout (through rich, with
+ a download's progress bar on stderr), the others to stderr. `calls` records (argv, env)."""
+
+ MLX_ERROR = ("Error loading model mlx-community/parakeet-tdt-0.6b-v2: cannot reach\n"
+ "huggingface.co (offline)\n\nparakeet-tdt-0.6b-v2 transcription complete.\n")
+
+ def __init__(self, names, fail=()):
+ self.names = set(names)
+ self.fail = set(fail)
+ self.calls = []
+
+ def which(self, name):
+ return f"/x/{name}" if name in self.names or name in ("ffmpeg", "ffprobe") else None
+
+ def ran(self, exe):
+ """The argv of every run of `exe`."""
+ return [c for c, _ in self.calls if os.path.basename(c[0]) == exe]
+
+ def run(self, cmd, **kw):
+ self.calls.append((list(cmd), kw.get("env")))
+ exe = os.path.basename(cmd[0])
+ if exe in self.fail:
+ if exe == "parakeet-mlx":
+ return subprocess.CompletedProcess(cmd, 1, self.MLX_ERROR, "Fetching 5 files: 0%| | 0/5\n")
+ return subprocess.CompletedProcess(cmd, 1, "", f"{exe}: failed to load model\n")
+ if exe == "parakeet-mlx":
+ out = cmd[cmd.index("--output-dir") + 1]
+ os.makedirs(out, exist_ok=True)
+ # the real CLI names its file by --output-template, else $PARAKEET_OUTPUT_TEMPLATE
+ template = (kw["env"] if kw.get("env") is not None else os.environ).get("PARAKEET_OUTPUT_TEMPLATE") or "{filename}"
+ name = template.format(filename="audio", parent=out, date="20261009", index="1")
+ Path(out, name + ".json").write_text(json.dumps(MLX_DOC), encoding="utf-8")
+ elif exe == "whisper-cli" and "-osrt" in cmd:
+ Path(cmd[cmd.index("-of") + 1] + ".srt").write_text(
+ "1\n00:00:00,000 --> 00:00:01,500\nBonjour tout le monde\n", encoding="utf-8")
+ elif exe == "whisper-cli" and "--output-json-full" in cmd:
+ Path(cmd[cmd.index("-of") + 1] + ".json").write_text(json.dumps({"transcription": [{"tokens": [
+ {"text": " Bonjour", "offsets": {"from": 0, "to": 600}},
+ {"text": " euh", "offsets": {"from": 700, "to": 900}}]}]}), encoding="utf-8")
+ return subprocess.CompletedProcess(cmd, 0, "", "")
+
+
+class _BridgeHarness:
+ """Runs the real bridge in process against _FakeEngines: no binary, no ffmpeg, every OS."""
+
+ def _with(self, engines, call, faster_whisper=None, fail=(), env=None, fake=None):
+ import _common
+ fake = fake or _FakeEngines(engines, fail)
+ stack = [mock.patch.object(_common, "run_analysis", lambda *a, **k: None),
+ mock.patch.object(shutil, "which", fake.which),
+ mock.patch.object(subprocess, "run", fake.run),
+ # None makes `import faster_whisper` fail; a stub module stands in for an install
+ mock.patch.dict(sys.modules, {"faster_whisper": faster_whisper}),
+ # a host ggml model in ~/.cache/whisper.cpp must not change the model named below
+ mock.patch.object(asr, "_whisper_cpp_model", lambda m: m),
+ mock.patch.dict(os.environ, dict({"PARAKEET_MODEL": "", "PARAKEET_CPP_MODEL": "", asr.ASR_ENGINE_ENV: ""},
+ **(env or {})))]
+ for p in stack:
+ p.start()
+ try:
+ return call()
+ finally:
+ for p in reversed(stack):
+ p.stop()
+
+ def _refusal(self, engines, call, **kw):
+ """The one die() a bridge call ends in: (message, kind, extra fields)."""
+ calls = []
+
+ def fake_die(msg, code=1, kind="input", **extra):
+ calls.append((msg, kind, extra))
+ raise SystemExit(code)
+ with mock.patch.object(asr, "die", fake_die), self.assertRaises(SystemExit):
+ self._with(engines, call, **kw)
+ self.assertEqual(len(calls), 1, calls)
+ return calls[0]
+
+
+def _faster_whisper_with_words():
+ """A faster_whisper that loads and returns one segment with word timings."""
+ mod = type(sys)("faster_whisper")
+
+ class Word:
+ def __init__(self, word, start, end):
+ self.word, self.start, self.end = word, start, end
+
+ class Segment:
+ start, end, text = 0.0, 0.9, " Bonjour euh"
+ words = [Word(" Bonjour", 0.0, 0.6), Word(" euh", 0.7, 0.9)]
+
+ class WhisperModel:
+ def __init__(self, *a, **k):
+ pass
+
+ def transcribe(self, wav, language=None, word_timestamps=False):
+ mod.languages.append(language)
+ return iter([Segment()]), None
+ mod.WhisperModel = WhisperModel
+ mod.languages = []
+ return mod
+
+
+def _crashing_faster_whisper():
+ """A faster_whisper whose model cannot be loaded -- what an offline first run looks like."""
+ mod = type(sys)("faster_whisper")
+
+ class WhisperModel:
+ def __init__(self, *a, **k):
+ raise RuntimeError("cannot fetch Systran/faster-whisper-base: offline")
+ mod.WhisperModel = WhisperModel
+ return mod
+
+
+class WhisperFailureTests(_BridgeHarness, unittest.TestCase):
+ """Undetected speech goes to Whisper first, so a Whisper engine that fails must read as a
+ failure: a faster-whisper whose model would not load used to be swallowed in its thread and
+ reported as "found no speech", which also kept Parakeet from ever being tried."""
+
+ def test_a_crashed_faster_whisper_is_not_no_speech(self):
+ """...and not "no engine found" either: it was found. The refusal names it and its own error."""
+ with tempfile.TemporaryDirectory() as d:
+ msg, kind, extra = self._refusal([], lambda: asr.transcribe_result("talk.mp4", os.path.join(d, "a.srt"), None, "base"),
+ faster_whisper=_crashing_faster_whisper())
+ self.assertEqual((kind, extra.get("reason"), extra.get("engine")), ("input", asr.ENGINE_FAILED_REASON, "faster-whisper"))
+ self.assertNotIn("no local speech-to-text engine found", msg)
+ self.assertNotIn("pip install", msg, "no install line for an engine that is installed")
+ self.assertIn("faster-whisper found but failed: cannot fetch Systran/faster-whisper-base: offline", msg)
+ self.assertEqual(extra["engines"], [{"engine": "faster-whisper", "reason": asr.ENGINE_FAILED_REASON,
+ "detail": "found but failed: cannot fetch Systran/faster-whisper-base: offline"}])
+
+ def test_parakeet_runs_last_after_a_crashed_faster_whisper(self):
+ with tempfile.TemporaryDirectory() as d:
+ run = self._with(["parakeet-mlx"], lambda: asr.transcribe_result("talk.mp4", os.path.join(d, "a.srt"), None, "base"),
+ faster_whisper=_crashing_faster_whisper())
+ self.assertEqual(run.engine, "parakeet-mlx")
+ self.assertEqual(run.facts["routing"], asr.ROUTING_LAST_RESORT)
+ self.assertEqual(run.notes, [asr.assumed_english_note(whisper_failed=True)])
+ words = self._with(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4"),
+ faster_whisper=_crashing_faster_whisper())
+ self.assertEqual((words.engine, len(words.words)), ("parakeet-mlx", 5))
+ self.assertEqual(words.notes, [asr.assumed_english_note("--filler-lang", whisper_failed=True)])
+
+ def test_a_crashed_faster_whisper_word_run_is_still_refused_by_name_without_parakeet(self):
+ """No Parakeet to fall back on: silence.py keeps its "<engine> ran but produced no
+ word-level timings" refusal, naming faster-whisper, as before."""
+ run = self._with([], lambda: asr.transcribe_words_result("talk.mp4"), faster_whisper=_crashing_faster_whisper())
+ self.assertEqual((run.engine, run.words), ("faster-whisper", []))
+
+ def test_a_word_run_whose_last_resort_parakeet_also_fails_names_every_engine(self):
+ """Parakeet was tried after the Whisper engine failed: the refusal must name both, not
+ hand back the Whisper failure as if Parakeet had never run (CodeRabbit on #312)."""
+ msg, kind, extra = self._refusal(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4"),
+ faster_whisper=_crashing_faster_whisper(), fail=("parakeet-mlx",))
+ self.assertEqual((kind, extra.get("reason")), ("input", asr.ENGINE_FAILED_REASON))
+ self.assertEqual([e["engine"] for e in extra["engines"]], ["faster-whisper", "parakeet-mlx"])
+ self.assertIn("--filler --transcribe", msg)
+
+
+class TranscriptionStateTests(_BridgeHarness, unittest.TestCase):
+ """Review of #305: the engine, routing and Parakeet word timings were handed back through
+ module-level LAST_RUN / LAST_WORDS, which every call in one process shared (mcp/server.py,
+ batch.py, a test). The calls now return them; nothing is kept in the module."""
+
+ def test_two_transcriptions_in_one_process_do_not_share_state(self):
+ self.assertFalse(hasattr(asr, "LAST_RUN") or hasattr(asr, "LAST_WORDS"),
+ "module-level hand-off state is back")
+ with tempfile.TemporaryDirectory() as d:
+ srt = os.path.join(d, "a.srt")
+ first = self._with(["parakeet-mlx"], lambda: asr.transcribe_result("talk.mp4", srt, "en", "base"))
+ snapshot = (first.engine, [dict(w) for w in first.words], dict(first.facts), list(first.cues))
+ self.assertEqual(first.engine, "parakeet-mlx")
+ self.assertEqual([w["word"] for w in first.words], ["So", "um", "we", "start.", "Here."])
+ self.assertEqual(first.facts, {"routing": "auto: --language is English", "engine": "parakeet-mlx",
+ "model": asr.PARAKEET_MLX_DEFAULT_MODEL, "language": "en"})
+
+ # a whisper caption run after it: no Parakeet words, model or routing carried over
+ second = self._with(["whisper-cli"], lambda: asr.transcribe_result("talk.mp4", srt, None, "base",
+ engine="whisper.cpp"))
+ self.assertEqual(second.engine, "whisper.cpp")
+ self.assertEqual(second.words, [])
+ self.assertEqual(second.cues, [(0.0, 1.5, "Bonjour tout le monde")])
+ self.assertEqual(second.facts, {"routing": "requested", "engine": "whisper.cpp", "model": "base", "language": None})
+ self.assertEqual(second.notes, [])
+
+ # a word run in between and after: its own words, engine and facts only
+ third = self._with(["whisper-cli", "parakeet-mlx"],
+ lambda: asr.transcribe_words_result("talk.mp4", "fr", engine="whisper.cpp"))
+ self.assertEqual([w["word"] for w in third.words], ["Bonjour", "euh"])
+ self.assertEqual(third.facts["language"], "fr")
+
+ # and the earlier results are still exactly what their own calls returned
+ self.assertEqual((first.engine, first.words, first.facts, first.cues), snapshot)
+ self.assertEqual(second.words, [])
+
+ def test_the_old_call_shapes_still_answer(self):
+ """transcribe() -> cues and transcribe_words() -> (words, engine): what callers had."""
+ with tempfile.TemporaryDirectory() as d:
+ cues = self._with(["parakeet-mlx"], lambda: asr.transcribe("talk.mp4", os.path.join(d, "a.srt"), "en", "base"))
+ self.assertEqual(cues, [(0.2, 1.6, "So um we start."), (2.0, 2.4, "Here.")])
+ words, engine = self._with(["parakeet-mlx"], lambda: asr.transcribe_words("talk.mp4", "en"))
+ self.assertEqual((len(words), engine), (5, "parakeet-mlx"))
+
+
+class _SrtDir:
+ def setUp(self):
+ self._dir = tempfile.TemporaryDirectory()
+ self.srt = os.path.join(self._dir.name, "a.srt")
+
+ def tearDown(self):
+ self._dir.cleanup()
+
+
+class EngineRefusalTests(_SrtDir, _BridgeHarness, unittest.TestCase):
+ """Review of #305: "no local speech-to-text engine found", with its install lines, was the refusal
+ whenever nothing transcribed -- also when an installed engine had failed (a whisper.cpp with
+ nothing after it, as in 2.5.1) or when auto had passed over an installed Parakeet engine for a
+ language that is not English. It now means only that no engine was found; the others name each
+ engine and why. A named engine is kind missing_tool only when it is not installed."""
+
+ def test_nothing_installed_is_still_no_engine(self):
+ msg, kind, _ = self._refusal([], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base"))
+ self.assertTrue(msg.startswith("no local speech-to-text engine found for --transcribe."), msg)
+ self.assertEqual(kind, "input")
+ msg, kind, _ = self._refusal([], lambda: asr.transcribe_words_result("talk.mp4"))
+ self.assertTrue(msg.startswith("no local speech-to-text engine found for --filler --transcribe."), msg)
+
+ def test_parakeet_passed_over_for_another_language_is_named(self):
+ msg, kind, extra = self._refusal(["parakeet-mlx"], lambda: asr.transcribe_result("talk.mp4", self.srt, "fr", "base"))
+ self.assertEqual((kind, extra["reason"], extra["engine"]), ("input", asr.ENGLISH_ONLY_REASON, "parakeet-mlx"))
+ self.assertNotIn("no local speech-to-text engine found", msg)
+ self.assertNotIn("uv tool install parakeet-mlx", msg, "it is installed")
+ self.assertIn("parakeet-mlx is installed, but --engine auto runs Parakeet only for English speech, "
+ "and --language fr is not English", msg)
+ self.assertIn("Whisper engine", extra["hint"])
+ msg, _, extra = self._refusal(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4", "fr"))
+ self.assertEqual(extra["reason"], asr.ENGLISH_ONLY_REASON)
+ self.assertIn("--filler-lang fr is not English", msg)
+ with mock.patch.object(asr, "detect_language", return_value="ja"):
+ msg, _, _ = self._refusal(["parakeet-mlx"], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base"))
+ self.assertIn("the detected language ja is not English", msg)
+
+ def test_a_failed_whisper_cpp_with_nothing_after_it_is_named(self):
+ """2.5.1 told the caller to install whisper.cpp when whisper.cpp itself had failed."""
+ msg, kind, extra = self._refusal(["whisper-cli"], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base"),
+ fail=["whisper-cli"])
+ self.assertEqual((kind, extra["reason"], extra["engine"]), ("input", asr.ENGINE_FAILED_REASON, "whisper.cpp"))
+ self.assertIn("whisper.cpp found but failed: whisper-cli: failed to load model", msg)
+ self.assertNotIn("no local speech-to-text engine found", msg)
+
+ def test_a_named_engine_that_is_installed_and_failed_is_not_missing_tool(self):
+ """kind missing_tool (DEPENDENCY_MISSING, "install it") only when the engine is not there; the
+ engine's own error is in the document -- parakeet-mlx prints it to stdout, not stderr."""
+ msg, kind, extra = self._refusal(
+ ["parakeet-mlx"], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base", engine="parakeet-mlx"),
+ fail=["parakeet-mlx"])
+ self.assertEqual((kind, extra["reason"], extra["engine"]), ("input", asr.ENGINE_FAILED_REASON, "parakeet-mlx"))
+ self.assertEqual(extra["detail"], "found but failed: Error loading model mlx-community/parakeet-tdt-0.6b-v2: "
+ "cannot reach huggingface.co (offline)")
+ self.assertIn(extra["detail"], msg)
+ _, kind, extra = self._refusal(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4", engine="parakeet-mlx"),
+ fail=["parakeet-mlx"])
+ self.assertEqual((kind, extra["reason"]), ("input", asr.ENGINE_FAILED_REASON))
+ _, kind, extra = self._refusal(
+ ["whisper-cli"], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base", engine="whisper.cpp"),
+ fail=["whisper-cli"])
+ self.assertEqual((kind, extra["engine"]), ("input", "whisper.cpp"))
+ self.assertIn("failed to load model", extra["detail"])
+
+ def test_a_named_engine_that_is_not_installed_is_missing_tool(self):
+ for engine in asr.ASR_ENGINES:
+ msg, kind, _ = self._refusal([], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base", engine=engine))
+ self.assertEqual(kind, "missing_tool", engine)
+ self.assertIn(f"--engine {engine} is not installed", msg)
+ _, kind, _ = self._refusal([], lambda: asr.transcribe_words_result("talk.mp4", engine=engine))
+ self.assertEqual(kind, "missing_tool", engine)
+
+ def test_the_failure_line_reads_parakeet_mlx_errors_from_stdout(self):
+ proc = subprocess.CompletedProcess([], 1, _FakeEngines.MLX_ERROR, "Fetching 5 files: 0%|\n")
+ self.assertEqual(asr._failure_line(proc, errors_on_stdout=True),
+ "Error loading model mlx-community/parakeet-tdt-0.6b-v2: cannot reach huggingface.co (offline)")
+ self.assertEqual(asr._failure_line(proc), "Fetching 5 files: 0%|")
+ self.assertEqual(asr._failure_line(subprocess.CompletedProcess([], 1, "only stdout\n", "")), "only stdout")
+ self.assertEqual(asr._failure_line(subprocess.CompletedProcess([], 1, "", "")), "?")
+
+
+class WhisperLanguageTests(_SrtDir, _BridgeHarness, unittest.TestCase):
+ """Review of #305: whisper.cpp was given -l only with --language, and its own default is -l en,
+ so it decoded any speech as English -- on main for every run without --language, on the route
+ that sends undetected speech to Whisper because Parakeet is English-only, and after auto's
+ detector had named the language. It is now told the language, or "auto"."""
+
+ @staticmethod
+ def _l(fake):
+ argv = fake.ran("whisper-cli")[-1]
+ return argv[argv.index("-l") + 1]
+
+ def test_no_language_asks_whisper_cpp_to_detect_it(self):
+ for call in (lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base"),
+ lambda: asr.transcribe_words_result("talk.mp4")):
+ fake = _FakeEngines(["whisper-cli"])
+ run = self._with([], call, fake=fake)
+ self.assertEqual((run.engine, self._l(fake)), ("whisper.cpp", "auto"))
+ self.assertIsNone(run.facts["language"], "whisper.cpp detected it; nothing named it")
+
+ def test_the_detected_language_is_passed_on(self):
+ fake = _FakeEngines(["whisper-cli", "parakeet-mlx"])
+ with mock.patch.object(asr, "detect_language", return_value="ja"):
+ run = self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, None, "base"), fake=fake)
+ self.assertEqual(self._l(fake), "ja")
+ self.assertEqual(run.facts, {"routing": "auto: detected ja, not English", "detected_language": "ja",
+ "engine": "whisper.cpp", "model": "base", "language": "ja"})
+ self.assertEqual(fake.ran("parakeet-mlx"), [])
+ fw = _faster_whisper_with_words()
+ run = self._with(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4"), faster_whisper=fw)
+ self.assertEqual((run.engine, fw.languages, run.facts["language"]), ("faster-whisper", ["ja"], "ja"))
+
+ def test_undetected_speech_routed_to_whisper_lets_whisper_cpp_detect(self):
+ fake = _FakeEngines(["whisper-cli", "parakeet-mlx"])
+ with mock.patch.object(asr, "detect_language", return_value=None), \
+ mock.patch.object(asr, "_whisper_ready", return_value=True):
+ run = self._with([], lambda: asr.transcribe_words_result("talk.mp4"), fake=fake)
+ self.assertEqual((run.facts["routing"], self._l(fake)), (asr.ROUTING_UNDETECTED_WHISPER, "auto"))
+
+ def test_a_named_language_is_passed_as_given(self):
+ fake = _FakeEngines(["whisper-cli"])
+ run = self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "fr", "base"), fake=fake)
+ self.assertEqual((self._l(fake), run.facts["language"]), ("fr", "fr"))
+
+ def test_an_english_only_whisper_model_is_no_engine_for_any_language(self):
+ cli = mock.Mock(which=lambda n: "/x/whisper-cli" if n == "whisper-cli" else None)
+ openai = mock.Mock(which=lambda n: "/x/whisper" if n == "whisper" else None)
+ with tempfile.TemporaryDirectory() as d, mock.patch.dict(sys.modules, {"faster_whisper": None}):
+ en = Path(d) / "ggml-base.en.bin"
+ en.write_bytes(b"")
+ self.assertFalse(asr._whisper_ready(cli, str(en)))
+ self.assertFalse(asr._whisper_ready(openai, "base.en"))
+ self.assertTrue(asr._whisper_ready(openai, "base"))
+
+
+class AutoRoutingFallbackTests(_SrtDir, _BridgeHarness, unittest.TestCase):
+ """Review of #305: what --engine auto does when its first choice fails, and what it then says."""
+
+ def test_whisper_after_a_failed_parakeet_says_why_it_ran(self):
+ """The routing said why Parakeet was chosen, next to engine whisper.cpp."""
+ fake = _FakeEngines(["parakeet-mlx", "whisper-cli"], fail=["parakeet-mlx"])
+ run = self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "en", "base"), fake=fake)
+ self.assertEqual(run.engine, "whisper.cpp")
+ self.assertEqual(run.facts["routing"], "auto: --language is English, no Parakeet engine transcribed it")
+ words = self._with(["parakeet-mlx", "whisper-cli"], lambda: asr.transcribe_words_result("talk.mp4", "en"),
+ fail=["parakeet-mlx"])
+ self.assertEqual((words.engine, words.facts["routing"]),
+ ("whisper.cpp", "auto: --language is English" + asr.ROUTING_PARAKEET_FAILED))
+
+ def test_undetected_speech_reaches_faster_whisper_after_a_failed_whisper_cpp(self):
+ """_whisper_ready() counted faster-whisper, but the word path stopped at the failed
+ whisper.cpp and ran Parakeet on English assumed, saying no Whisper engine transcribed it."""
+ fake = _FakeEngines(["whisper-cli", "parakeet-mlx"], fail=["whisper-cli"])
+ run = self._with([], lambda: asr.transcribe_words_result("talk.mp4"), faster_whisper=_faster_whisper_with_words(),
+ fake=fake)
+ self.assertEqual(run.engine, "faster-whisper")
+ self.assertEqual([w["word"] for w in run.words], ["Bonjour", "euh"])
+ self.assertEqual(run.facts["routing"], asr.ROUTING_UNDETECTED_WHISPER)
+ self.assertEqual((run.notes, fake.ran("parakeet-mlx")), ([], []))
+
+ def test_without_parakeet_a_failed_whisper_cpp_word_run_is_named_as_before(self):
+ """No routing decision rests on faster-whisper here: the engine that ran and failed is the
+ one silence.py's "no word-level timings" refusal names, as in 2.5.1."""
+ run = self._with(["whisper-cli"], lambda: asr.transcribe_words_result("talk.mp4"),
+ faster_whisper=_faster_whisper_with_words(), fail=["whisper-cli"])
+ self.assertEqual((run.engine, run.words), ("whisper.cpp", []))
+
+ def test_a_multilingual_parakeet_model_assumes_no_english(self):
+ v3 = "mlx-community/parakeet-tdt-0.6b-v3"
+ run = self._with(["parakeet-mlx", "whisper-cli"], lambda: asr.transcribe_result("talk.mp4", self.srt, None, v3))
+ self.assertEqual((run.engine, run.facts["model"]), ("parakeet-mlx", v3))
+ self.assertEqual(run.facts["routing"], "auto: --model names a Parakeet model and the language is not detectable "
+ "here; the Parakeet model is multilingual")
+ self.assertEqual(run.notes, [])
+ # the model from PARAKEET_MODEL, no Whisper engine: nothing assumed either
+ run = self._with(["parakeet-mlx"], lambda: asr.transcribe_words_result("talk.mp4"), env={"PARAKEET_MODEL": v3})
+ self.assertEqual(run.facts["model"], v3)
+ self.assertNotIn("assumed English", run.facts["routing"])
+ self.assertEqual(run.notes, [])
+
+ def test_an_english_only_parakeet_model_named_with_model_points_at_the_model(self):
+ """Not "no Whisper engine ... install one": a Whisper engine cannot load a Parakeet model."""
+ run = self._with(["parakeet-mlx", "whisper-cli"],
+ lambda: asr.transcribe_result("talk.mp4", self.srt, None, asr.PARAKEET_MLX_DEFAULT_MODEL))
+ self.assertEqual(run.facts["routing"], asr.ROUTING_PARAKEET_MODEL)
+ self.assertEqual(run.notes, [asr.assumed_english_note(parakeet_model=True)])
+ self.assertNotIn("install a Whisper engine", run.notes[0])
+
+ def test_a_host_output_template_does_not_hide_parakeet_mlx_output(self):
+ """parakeet-mlx names its file by $PARAKEET_OUTPUT_TEMPLATE; the bridge reads audio.json."""
+ fake = _FakeEngines(["parakeet-mlx"])
+ run = self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "en", "base"), fake=fake,
+ env={"PARAKEET_OUTPUT_TEMPLATE": "{filename}_{date}"})
+ self.assertEqual(run.engine, "parakeet-mlx")
+ (_, env), = fake.calls
+ self.assertNotIn("PARAKEET_OUTPUT_TEMPLATE", env)
+
+
+class InstallHintTests(unittest.TestCase):
+ """Review of #305: parakeet-mlx fetches its default model from Hugging Face on first use. The
+ hints say so, and say the audio stays on the machine, so "no cloud" stays a true sentence."""
+
+ def test_the_refusal_hint_names_the_first_run_download_and_keeps_audio_local(self):
+ hint = asr.ASR_INSTALL_HINT
+ line = next(ln for ln in hint.splitlines() if ln.strip().startswith("parakeet-mlx:"))
+ rest = hint[hint.index(line):].split("parakeet.cpp:")[0]
+ self.assertIn("first run", rest)
+ self.assertIn("Hugging Face", rest)
+ self.assertIn(asr.PARAKEET_MLX_DEFAULT_MODEL, rest)
+ self.assertIn("never leaves", hint.splitlines()[0])
+ self.assertNotIn("offline", hint, "a first run that downloads is not offline")
+
+ def test_the_capability_fix_hint_says_the_same(self):
+ """_capability_fix_hint() is the `fix` doctor gives for a missing *required* capability; no
+ tool requires a speech engine today, so doctor lists external:* under missing_optional
+ without it. The text is still kept true for the day one does."""
+ import _contract
+ fix = _contract._capability_fix_hint("external:parakeet")
+ for words in ("first run", "Hugging Face", asr.PARAKEET_MLX_DEFAULT_MODEL, "never uploaded"):
+ self.assertIn(words, fix)
+
+ def test_the_network_claim_matches_where_each_engine_runs(self):
+ """Review of #305: "the skill itself opens no network connection" was false for faster-whisper,
+ a library the skill runs in its own process, whose first run downloads its model. And
+ "doctor's fix line" was named as a place that says so, which doctor never shows for an
+ optional capability."""
+ import _contract
+ root = Path(__file__).resolve().parent.parent
+ contract_md = (root / "docs" / "contract.md").read_text(encoding="utf-8")
+ decisions = (root / "docs" / "design-decisions.md").read_text(encoding="utf-8")
+ self.assertNotIn("skill itself opens no network connection", contract_md)
+ self.assertIn("faster-whisper is a Python library the skill runs inside its own process", " ".join(contract_md.split()))
+ self.assertNotIn("(the skill's own process) stay true", decisions)
+ for text in (decisions, (root / "CHANGELOG.md").read_text(encoding="utf-8")):
+ self.assertNotIn("doctor's fix line", text)
+ self.assertNotIn("`doctor`'s fix line", text)
+ # the engines it runs are listed where the contract says what it runs
+ execution = _contract.build(detect=False)["execution"]
+ ran = execution["subprocess"]
+ for exe in ("whisper-cli", "whisper-cpp", "parakeet-mlx", "parakeet-cli", "whisper"):
+ self.assertIn(exe, ran)
+ # ...and the machine-readable contract says which engine downloads, from which process: a
+ # caller that read only `network: false` took a first faster-whisper run for network-free
+ # (CodeRabbit on #312)
+ downloads = execution["model_downloads"]
+ self.assertEqual(set(downloads["in_process"]), {"faster-whisper"})
+ self.assertEqual(set(downloads["child_process"]), {"parakeet-mlx", "openai-whisper"})
+ self.assertEqual(set(downloads["none"]), {"whisper.cpp", "parakeet.cpp"})
+ self.assertIn("model_downloads", contract_md)
+
+ def test_the_reference_page_no_longer_says_nothing_is_downloaded(self):
+ root = Path(__file__).resolve().parent.parent
+ page = (root / "references" / "scripts.md").read_text(encoding="utf-8")
+ section = page[page.index("### caption.py --transcribe"):]
+ section = section[:section.index("\n### ", 5)]
+ self.assertNotIn("Nothing is downloaded", section)
+ for words in ("Hugging Face", "never uploaded", "first time it runs"):
+ self.assertIn(words, section)
+
+
[email protected](platform.system() == "Windows", "the fake engines are #! scripts on a POSIX PATH shim; the routing "
+ "they drive is covered on every OS by the in-process tests above")
+class ParakeetEngineTests(MediaFixtures):
+ """caption.py / silence.py drive fake Parakeet binaries on a PATH that holds only them and ffmpeg."""
+
+ @classmethod
+ def setUpClass(cls):
+ super().setUpClass()
+ cls.bin = Path(tempfile.mkdtemp(prefix="ffskill_fakeasr_"))
+ for tool in ("ffmpeg", "ffprobe", "fc-match", "fc-list"):
+ if shutil.which(tool):
+ os.symlink(shutil.which(tool), cls.bin / tool)
+ mlx = """
+ import json, os, sys
+ a = sys.argv[1:]; out = a[a.index("--output-dir") + 1]; os.makedirs(out, exist_ok=True)
+ json.dump(DOC, open(os.path.join(out, os.path.splitext(os.path.basename(a[0]))[0] + ".json"), "w"))
+ """
+ cpp = """
+ import json, sys
+ assert sys.argv[1] == "transcribe" and "--json" in sys.argv
+ print(json.dumps(DOC))
+ """
+ for name, body, doc in (("parakeet-mlx", mlx, MLX_DOC), ("parakeet-cli", cpp, CPP_DOC)):
+ p = cls.bin / name
+ p.write_text(f"#!{sys.executable}\nDOC = {doc!r}\n" + textwrap.dedent(body))
+ p.chmod(p.stat().st_mode | stat.S_IXUSR)
+ cls.gguf = cls.bin / "tdt-0.6b-v2-f16.gguf"
+ cls.gguf.write_bytes(b"GGUF")
+ # A PATH hides the host's engine binaries, but not a Python module: the tools would import a
+ # faster_whisper the host has installed (a Mac where Parakeet is used often has one) and
+ # Whisper would then go first. This sitecustomize makes `import faster_whisper` fail in
+ # every subprocess, as on CI.
+ cls.shim = Path(tempfile.mkdtemp(prefix="ffskill_nofw_"))
+ (cls.shim / "sitecustomize.py").write_text("import sys\nsys.modules['faster_whisper'] = None\n", encoding="utf-8")
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.bin, ignore_errors=True)
+ shutil.rmtree(cls.shim, ignore_errors=True)
+ super().tearDownClass()
+
+ def env(self, **extra):
+ e = dict(os.environ, PATH=f"{self.bin}{os.pathsep}/usr/bin{os.pathsep}/bin", PARAKEET_CPP_MODEL=str(self.gguf),
+ PYTHONPATH=os.pathsep.join(p for p in (str(self.shim), os.environ.get("PYTHONPATH")) if p))
+ e.update(extra)
+ return e
+
+ def test_a_host_faster_whisper_is_hidden_from_the_tools(self):
+ """With a faster_whisper importable on the host (a stub package on PYTHONPATH stands in for
+ one), the tools still cannot import it."""
+ stub = Path(tempfile.mkdtemp(prefix="ffskill_fwstub_"))
+ try:
+ (stub / "faster_whisper").mkdir()
+ (stub / "faster_whisper" / "__init__.py").write_text("class WhisperModel:\n pass\n", encoding="utf-8")
+ env = self.env()
+ env["PYTHONPATH"] += os.pathsep + str(stub)
+ probe = sh(sys.executable, "-c", "import sys; sys.path.insert(0, sys.argv[1]); from _common import asr; "
+ "print(asr._module_importable('faster_whisper'))", str(Path(__file__).resolve().parent.parent / "scripts"),
+ env=env)
+ finally:
+ shutil.rmtree(stub, ignore_errors=True)
+ self.assertEqual(probe.stdout.strip(), "False")
+
+ def test_caption_auto_picks_parakeet_for_english_and_reports_it(self):
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--language", "en", "--fast", "--json",
+ "-o", OUT / "pk_auto.mp4", env=self.env()).stdout)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet-mlx")
+ self.assertEqual(doc["transcription"]["model"], asr.PARAKEET_MLX_DEFAULT_MODEL)
+ self.assertIn("So um we start.", (OUT / "pk_auto.srt").read_text())
+ self.assertFalse(_assumed(doc), "English was named, not assumed")
+
+ def test_caption_engine_flag_picks_parakeet_cpp(self):
+ """One end-to-end parakeet.cpp run; flag-over-env precedence is a unit test above."""
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--engine", "parakeet.cpp", "--fast", "--json",
+ "-o", OUT / "pk_cpp.mp4", env=self.env()).stdout)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet.cpp")
+ self.assertIn("So um we start.", (OUT / "pk_cpp.srt").read_text())
+ # a named engine runs as asked, with no language: no routing decision, so no warning
+ self.assertEqual(doc["transcription"]["routing"], "requested")
+ self.assertFalse(_assumed(doc))
+
+ def test_caption_mux_reports_the_transcription_too(self):
+ """--mode mux wrote a soft subtitle track from the transcript but left `transcription` out of
+ its result, so a caller could not tell which engine made it."""
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--engine", "parakeet.cpp", "--mode", "mux",
+ "--json", "-o", OUT / "pk_mux.mp4", env=self.env()).stdout)
+ self.assertEqual(doc["subtitle_tracks"], 1)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet.cpp")
+
+ def test_caption_takes_the_engine_from_the_environment(self):
+ """$FFMPEG_SKILL_ASR_ENGINE reaches caption.py through its real parser (no --engine given)."""
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--fast", "--json", "-o", OUT / "pk_cpp_env.mp4",
+ env=self.env(FFMPEG_SKILL_ASR_ENGINE="parakeet.cpp")).stdout)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet.cpp")
+
+ def _run_with_mlx_writing(self, body, tool, *args, expect_fail=False):
+ """`tool` with a parakeet-mlx that exits 0 after writing `body` as its JSON."""
+ fake = Path(tempfile.mkdtemp(prefix="ffskill_oddasr_"))
+ try:
+ for name in os.listdir(self.bin):
+ if name != "parakeet-mlx":
+ os.symlink(self.bin / name, fake / name)
+ mlx = fake / "parakeet-mlx"
+ mlx.write_text(f"#!{sys.executable}\nimport os, sys\na = sys.argv[1:]; out = a[a.index('--output-dir') + 1]\n"
+ "os.makedirs(out, exist_ok=True)\n"
+ "open(os.path.join(out, os.path.splitext(os.path.basename(a[0]))[0] + '.json'), 'w')"
+ f".write({body!r})\n")
+ mlx.chmod(0o755)
+ proc = script(tool, self.src, *args, "--json", env=self.env(PATH=str(fake)), expect_fail=expect_fail)
+ finally:
+ shutil.rmtree(fake, ignore_errors=True)
+ return json.loads(proc.stdout)
+
+ def test_unreadable_engine_output_falls_through_instead_of_claiming_silence(self):
+ """parakeet-mlx exited 0 but wrote something that is not a transcript: that is a failed
+ run, so auto moves on to parakeet.cpp instead of reporting "no speech" in the video."""
+ doc = self._run_with_mlx_writing('{"sentences": 5}', "caption.py", "--transcribe", "--fast", "-o", OUT / "pk_garbage.mp4")
+ self.assertEqual(doc["transcription"]["engine"], "parakeet.cpp")
+
+ def test_filler_words_fall_through_an_engine_with_sentences_but_no_word_times(self):
+ """Sentences whose text and times read but whose tokens do not: cues for a caption, but no
+ word timings, which is all silence.py --filler needs -- so the next engine is tried."""
+ body = json.dumps({"text": "So um we start.", "sentences": [{"text": "So um we start.", "start": 0.2, "end": 1.6,
+ "tokens": [{"text": " So"}, 5]}]})
+ doc = self._run_with_mlx_writing(body, "silence.py", "--filler", "--transcribe", "--filler-list")
+ self.assertEqual(doc["filler"]["source"], "parakeet:parakeet.cpp")
+ self.assertEqual([r["word"] for r in doc["filler"]["removed"]], ["um"])
+
+ def test_a_named_engine_with_no_word_times_is_refused_by_name(self):
+ """--engine parakeet-mlx ran and gave no word timings: say that, not "install it"."""
+ body = json.dumps({"text": "So um we start.", "sentences": [{"text": "So um we start.", "start": 0.2, "end": 1.6,
+ "tokens": [{"text": " So"}, 5]}]})
+ doc = self._run_with_mlx_writing(body, "silence.py", "--filler", "--transcribe", "--filler-list",
+ "--engine", "parakeet-mlx", expect_fail=True)
+ self.assertEqual(doc["error"]["kind"], "input")
+ self.assertIn("parakeet-mlx ran but produced no word-level timings", doc["error"]["message"])
+
+ def test_an_engine_that_heard_nothing_still_reports_no_speech(self):
+ """What both real engines write for 3 s of silence (measured): an empty list. That is an
+ answer, not a failure, so it is still the no-speech refusal and nothing else is tried."""
+ doc = self._run_with_mlx_writing('{"text": "", "sentences": []}', "caption.py", "--transcribe", "--fast",
+ "-o", OUT / "pk_silent.mp4", expect_fail=True)
+ self.assertEqual((doc["error"]["kind"], doc.get("reason"), doc.get("engine")), ("input", "no_speech", "parakeet-mlx"))
+
+ def test_auto_falls_through_a_failing_parakeet_mlx_to_parakeet_cpp(self):
+ """parakeet-mlx is installed but crashes: auto moves on to the next Parakeet engine."""
+ broken = Path(tempfile.mkdtemp(prefix="ffskill_brokenasr_"))
+ try:
+ for name in os.listdir(self.bin):
+ if name != "parakeet-mlx":
+ os.symlink(self.bin / name, broken / name)
+ ran = broken / "mlx-ran"
+ mlx = broken / "parakeet-mlx"
+ mlx.write_text(f"#!/bin/sh\n: > '{ran}'\necho 'Metal device lost' >&2\nexit 3\n")
+ mlx.chmod(0o755)
+ # only the fakes and ffmpeg on PATH: no host whisper can detect a language or take over,
+ # so auto assumes English and the order is parakeet-mlx, then parakeet.cpp
+ doc = json.loads(script("silence.py", self.src, "--filler", "--transcribe", "--filler-list", "--json",
+ env=self.env(PATH=str(broken))).stdout)
+ mlx_ran = ran.exists()
+ finally:
+ shutil.rmtree(broken, ignore_errors=True)
+ self.assertTrue(mlx_ran, "the crashing parakeet-mlx was never run, so nothing fell through")
+ self.assertEqual(doc["filler"]["source"], "parakeet:parakeet.cpp")
+ self.assertEqual([r["word"] for r in doc["filler"]["removed"]], ["um"])
+
+ def _with_whisper_cli(self, fail):
+ """A PATH holding the fake Parakeet engines, ffmpeg and a fake whisper-cli whose language
+ detector names nothing; it transcribes ("Bonjour tout le monde") unless `fail`. HOME
+ holds the ggml-base.bin it is run with. Returns (PATH dir, HOME dir); the caller removes both."""
+ d = Path(tempfile.mkdtemp(prefix="ffskill_whisperasr_"))
+ home = Path(tempfile.mkdtemp(prefix="ffskill_whisperhome_"))
+ for name in os.listdir(self.bin):
+ os.symlink(self.bin / name, d / name)
+ (home / ".cache" / "whisper.cpp").mkdir(parents=True)
+ (home / ".cache" / "whisper.cpp" / "ggml-base.bin").write_bytes(b"")
+ cli = d / "whisper-cli"
+ cli.write_text(f"#!{sys.executable}\nFAIL = {fail!r}\nARGS = {str(home / 'whisper-args.json')!r}\n" + textwrap.dedent("""
+ import json, sys
+ a = sys.argv[1:]
+ if "-dl" not in a:
+ json.dump(a, open(ARGS, "w"))
+ if "-dl" in a:
+ print("whisper_full_with_state: no language line in this build", file=sys.stderr)
+ sys.exit(0)
+ if FAIL:
+ print("whisper_init_from_file_with_params: failed to load model", file=sys.stderr)
+ sys.exit(1)
+ of = a[a.index("-of") + 1]
+ if "-osrt" in a:
+ open(of + ".srt", "w").write("1\\n00:00:00,000 --> 00:00:01,500\\nBonjour tout le monde\\n")
+ if "--output-json-full" in a:
+ json.dump({"transcription": [{"tokens": [{"text": " Bonjour", "offsets": {"from": 0, "to": 600}},
+ {"text": " euh", "offsets": {"from": 700, "to": 900}}]}]},
+ open(of + ".json", "w"))
+ """))
+ cli.chmod(0o755)
+ return d, home
+
+ def test_no_language_and_only_parakeet_says_english_was_assumed(self):
+ """Review of #305: a Mac with parakeet-mlx and no Whisper captioning speech nobody named.
+ Parakeet still runs, but the result says, at the top level, that English was assumed."""
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--fast", "--json",
+ "-o", OUT / "pk_assumed.mp4", env=self.env(PATH=str(self.bin))).stdout)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet-mlx")
+ self.assertEqual(doc["transcription"]["routing"], asr.ROUTING_ASSUMED_ENGLISH)
+ self.assertEqual(len(_assumed(doc)), 1, doc.get("notes"))
+ self.assertIn("--language", _assumed(doc)[0])
+ self.assertTrue(doc["verified"], "the render itself is still verified; the note carries the doubt")
+ fil = json.loads(script("silence.py", self.src, "--filler", "--transcribe", "--filler-list", "--json",
+ env=self.env(PATH=str(self.bin))).stdout)
+ self.assertEqual(fil["filler"]["source"], "parakeet:parakeet-mlx")
+ self.assertEqual(len(_assumed(fil)), 1, fil.get("notes"))
+ self.assertIn("--filler-lang", _assumed(fil)[0])
+
+ def test_no_language_goes_to_whisper_before_parakeet(self):
+ """The same machine with whisper.cpp installed but no language it can detect: Whisper,
+ which handles any language, takes the speech; the English-only model is not trusted with it."""
+ d, home = self._with_whisper_cli(fail=False)
+ args = home / "whisper-args.json"
+ try:
+ env = self.env(PATH=str(d), HOME=str(home))
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--mode", "mux", "--json",
+ "-o", OUT / "pk_whisperfirst.mp4", env=env).stdout)
+ caption_args = json.loads(args.read_text())
+ fil = json.loads(script("silence.py", self.src, "--filler", "--transcribe", "--filler-list", "--json",
+ env=env).stdout)
+ filler_args = json.loads(args.read_text())
+ finally:
+ shutil.rmtree(d, ignore_errors=True)
+ shutil.rmtree(home, ignore_errors=True)
+ self.assertEqual(doc["transcription"], {"routing": asr.ROUTING_UNDETECTED_WHISPER, "engine": "whisper.cpp",
+ "model": str(home / ".cache" / "whisper.cpp" / "ggml-base.bin"),
+ "language": None})
+ # whisper.cpp's own default is -l en: "handles any language" only holds when it is told to detect
+ for argv in (caption_args, filler_args):
+ self.assertEqual(argv[argv.index("-l") + 1], "auto", argv)
+ self.assertIn("Bonjour tout le monde", (OUT / "pk_whisperfirst.srt").read_text())
+ self.assertFalse(_assumed(doc))
+ self.assertEqual(fil["filler"]["source"], "whisper:whisper.cpp")
+ self.assertEqual(fil["filler"]["transcription"]["routing"], asr.ROUTING_UNDETECTED_WHISPER)
+ self.assertNotIn("notes", fil)
+
+ def test_parakeet_runs_last_when_every_whisper_engine_fails(self):
+ """Whisper went first and failed: Parakeet is still better than refusing, on English
+ assumed, and the note says which fix applies."""
+ d, home = self._with_whisper_cli(fail=True)
+ try:
+ env = self.env(PATH=str(d), HOME=str(home))
+ doc = json.loads(script("caption.py", self.src, "--transcribe", "--mode", "mux", "--json",
+ "-o", OUT / "pk_lastresort.mp4", env=env).stdout)
+ fil = json.loads(script("silence.py", self.src, "--filler", "--transcribe", "--filler-list", "--json",
+ env=env).stdout)
+ finally:
+ shutil.rmtree(d, ignore_errors=True)
+ shutil.rmtree(home, ignore_errors=True)
+ self.assertEqual(doc["transcription"]["engine"], "parakeet-mlx")
+ self.assertEqual(doc["transcription"]["routing"], asr.ROUTING_LAST_RESORT)
+ self.assertEqual(len(_assumed(doc)), 1, doc.get("notes"))
+ self.assertIn("fix the Whisper engine", _assumed(doc)[0])
+ self.assertEqual(fil["filler"]["source"], "parakeet:parakeet-mlx")
+ self.assertEqual(fil["filler"]["transcription"]["routing"], asr.ROUTING_LAST_RESORT)
+ self.assertEqual(len(_assumed(fil)), 1, fil.get("notes"))
+
+ def test_caption_another_language_skips_parakeet(self):
+ """Parakeet is installed and auto passes it over for French: the refusal says that, not
+ "no local speech-to-text engine found" with a line telling the caller to install it."""
+ proc = script("caption.py", self.src, "--transcribe", "--language", "fr", "--fast", "--json",
+ "-o", OUT / "pk_fr.mp4", env=self.env(PATH=str(self.bin)), expect_fail=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual((doc["error"]["kind"], doc["reason"], doc["engine"]), ("input", asr.ENGLISH_ONLY_REASON, "parakeet-mlx"))
+ self.assertNotIn("no local speech-to-text engine found", doc["error"]["message"])
+ self.assertIn("--language fr is not English", doc["error"]["message"])
+ self.assertEqual([e["engine"] for e in doc["engines"]], ["parakeet-mlx", "parakeet.cpp"])
+
+ def test_a_named_engine_that_failed_says_why_in_the_document(self):
+ """parakeet-mlx prints its errors to stdout and exits 1: the failure document carries that
+ line (a JSON caller never sees the log), and the kind is not missing_tool -- it is installed."""
+ broken = Path(tempfile.mkdtemp(prefix="ffskill_mlxerr_"))
+ try:
+ for name in os.listdir(self.bin):
+ if name != "parakeet-mlx":
+ os.symlink(self.bin / name, broken / name)
+ mlx = broken / "parakeet-mlx"
+ mlx.write_text("#!/bin/sh\necho 'Fetching 5 files: 0%|' >&2\n"
+ "echo 'Error loading model mlx-community/parakeet-tdt-0.6b-v2: Metal device lost'\nexit 1\n")
+ mlx.chmod(0o755)
+ proc = script("caption.py", self.src, "--transcribe", "--engine", "parakeet-mlx", "--fast", "--json",
+ "-o", OUT / "pk_mlxerr.mp4", env=self.env(PATH=str(broken)), expect_fail=True)
+ finally:
+ shutil.rmtree(broken, ignore_errors=True)
+ doc = json.loads(proc.stdout)
+ self.assertEqual((doc["error"]["kind"], doc["reason"], doc["engine"]), ("input", asr.ENGINE_FAILED_REASON, "parakeet-mlx"))
+ self.assertNotEqual(doc["error"]["code"], "DEPENDENCY_MISSING")
+ self.assertEqual(doc["detail"], "found but failed: Error loading model mlx-community/parakeet-tdt-0.6b-v2: Metal device lost")
+
+ def test_silence_filler_words_from_parakeet(self):
+ doc = json.loads(script("silence.py", self.src, "--filler", "--transcribe", "--filler-list", "--json",
+ "--engine", "parakeet.cpp", env=self.env()).stdout)
+ self.assertEqual(doc["filler"]["source"], "parakeet:parakeet.cpp")
+ self.assertEqual([r["word"] for r in doc["filler"]["removed"]], ["um"])
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/tests/test_contract.py b/tests/test_contract.py
index ad17b8d..3b15252 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -33,6 +33,12 @@ from _common.runner import HW_ENV, HW_FORCED_ENV # noqa: E402
# tests/_fixtures.py cannot drift from what the tools read.
for _k in (HW_ENV, HW_FORCED_ENV):
os.environ.pop(_k, None)
+
+# The host's own speech defaults must not reach the suite either: FFMPEG_SKILL_ASR_ENGINE or a
+# PARAKEET_* setting would change which speech engine a --transcribe test drives, or where
+# parakeet-mlx writes. Tests that exercise those opt in with an explicit env.
+for _k in ("FFMPEG_SKILL_ASR_ENGINE", "PARAKEET_MODEL", "PARAKEET_CPP_MODEL", "PARAKEET_OUTPUT_TEMPLATE"):
+ os.environ.pop(_k, None)
import _common # noqa: E402
import server as mcp_server # noqa: E402
@@ -262,6 +268,20 @@ class ContractTests(unittest.TestCase):
f"docs/roadmap.md says the released version is {m.group(1)!r}, "
f"but package.json is {pkg['version']!r} -- update the roadmap sentence")
+ def test_contract_md_names_only_environment_variables_the_code_reads(self):
+ """docs/contract.md is the authority on flags; a review of #305 found its `cache` row naming
+ FFMPEG_SKILL_HW, carried over from an unmerged branch, which nothing in this tree read.
+ Every FFMPEG_SKILL_* variable it names must be read by a script, the MCP server or the
+ installer."""
+ text = (ROOT / "docs" / "contract.md").read_text(encoding="utf-8")
+ code = "".join(p.read_text(encoding="utf-8", errors="replace")
+ for d in ("scripts", "mcp", "bin") for p in sorted((ROOT / d).rglob("*"))
+ if p.suffix in (".py", ".js") and "__pycache__" not in p.parts)
+ named = sorted(set(re.findall(r"\bFFMPEG_SKILL_[A-Z0-9_]*[A-Z0-9]", text)))
+ self.assertTrue(named, "docs/contract.md names no FFMPEG_SKILL_* variable at all")
+ self.assertEqual([v for v in named if v not in code], [],
+ "docs/contract.md names environment variables nothing in scripts/, mcp/ or bin/ reads")
+
def test_docs_failure_json_example_keys_match_the_real_error_shape(self):
"""docs/contract.md's illustrative failure-JSON example listed only kind/message under
error for years after "code"/"retryable" were added to the real die() output (Hardening
diff --git a/tests/test_editing.py b/tests/test_editing.py
index 5440ee2..ff21592 100644
--- a/tests/test_editing.py
+++ b/tests/test_editing.py
@@ -1896,14 +1896,13 @@ class EditingTests(MediaFixtures):
(shim / "sitecustomize.py").write_text(
"import sys, os\n"
"sys.path.insert(0, os.environ['FFSKILL_SCRIPTS'])\n"
- "import _common.asr as asr, _common as c\n"
- "def fake(video, language=None, model='base', audio_stream=0):\n"
- " return ([{'word': 'So', 'start': 0.2, 'end': 0.5},\n"
+ "import _common.asr as asr\n"
+ "def fake(video, language=None, model='base', audio_stream=0, engine=None, **kw):\n"
+ " return asr.Transcription(words=[{'word': 'So', 'start': 0.2, 'end': 0.5},\n"
" {'word': 'um', 'start': 0.6, 'end': 0.8},\n"
" {'word': 'this', 'start': 1.0, 'end': 1.3},\n"
- " {'word': 'uh', 'start': 2.2, 'end': 2.45}], 'faster-whisper')\n"
- "asr.transcribe_words = fake\n"
- "c.transcribe_words = fake\n",
+ " {'word': 'uh', 'start': 2.2, 'end': 2.45}], engine='faster-whisper')\n"
+ "asr.transcribe_words_result = fake\n",
encoding="utf-8")
env = dict(os.environ, PYTHONPATH=str(shim), FFSKILL_SCRIPTS=str(SCRIPTS))
r = sh(sys.executable, SCRIPTS / "silence.py", self._gappy(), "--filler",
@@ -1923,11 +1922,10 @@ class EditingTests(MediaFixtures):
(shim / "sitecustomize.py").write_text(
"import sys, os\n"
"sys.path.insert(0, os.environ['FFSKILL_SCRIPTS'])\n"
- "import _common.asr as asr, _common as c\n"
- "def fake(video, language=None, model='base', audio_stream=0):\n"
- " return ([], 'whisper.cpp')\n"
- "asr.transcribe_words = fake\n"
- "c.transcribe_words = fake\n",
+ "import _common.asr as asr\n"
+ "def fake(video, language=None, model='base', audio_stream=0, engine=None, **kw):\n"
+ " return asr.Transcription(engine='whisper.cpp')\n"
+ "asr.transcribe_words_result = fake\n",
encoding="utf-8")
env = dict(os.environ, PYTHONPATH=str(shim), FFSKILL_SCRIPTS=str(SCRIPTS))
out = OUT / "filler_nowords.mp4"
diff --git a/tests/test_picture.py b/tests/test_picture.py
index 6185847..64b658c 100644
--- a/tests/test_picture.py
+++ b/tests/test_picture.py
@@ -3601,5 +3601,38 @@ class AsrNoSpeechTests(unittest.TestCase):
self.assertNotIn("no local speech-to-text engine", call[0])
[email protected](platform.system() == "Windows", "the fake whisper-cli is a #!/bin/sh script on a POSIX PATH shim")
+class WhisperDefaultModelTests(unittest.TestCase):
+ """A whisper.cpp install made the way the refusal hint says (brew + ggml-base.bin) must be the
+ model caption.py --transcribe runs by default: the default became large-v3-turbo with the
+ Parakeet engines, and whisper-cli was handed a bare model name it could not open."""
+
+ def test_the_default_model_is_the_one_the_install_hint_names(self):
+ import tempfile
+ from _common import asr
+ self.assertIn("ggml-base.bin", asr.ASR_INSTALL_HINT)
+ with tempfile.TemporaryDirectory() as d:
+ home, bin_dir = Path(d, "home"), Path(d, "bin")
+ (home / ".cache" / "whisper.cpp").mkdir(parents=True)
+ model = home / ".cache" / "whisper.cpp" / "ggml-base.bin"
+ model.write_bytes(b"")
+ bin_dir.mkdir()
+ seen = Path(d, "args.txt")
+ # a whisper-cli that records its arguments and writes one cue where -of says
+ cli = bin_dir / "whisper-cli"
+ cli.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "' + str(seen) + '"\n'
+ 'while [ $# -gt 0 ]; do [ "$1" = "-of" ] && of="$2"; shift; done\n'
+ 'printf "1\\n00:00:00,000 --> 00:00:01,000\\nhello\\n" > "$of.srt"\n', encoding="utf-8")
+ cli.chmod(0o755)
+ clip = Path(d, "talk.mp4")
+ sh("ffmpeg", "-y", "-v", "error", "-f", "lavfi", "-i", "color=c=black:s=320x180:r=30:d=2",
+ "-f", "lavfi", "-i", "sine=d=2", "-c:v", "libx264", "-c:a", "aac", "-shortest", clip)
+ env = dict(os.environ, HOME=str(home), PATH=f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
+ FFMPEG_SKILL_ASR_ENGINE="whisper.cpp")
+ script("caption.py", clip, "--transcribe", "-o", Path(d, "out.mp4"), env=env)
+ args = seen.read_text(encoding="utf-8").split("\n")
+ self.assertEqual(args[args.index("-m") + 1], str(model))
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)