Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

feat(stt): run the word aligner on the GPU only, measured against phase 2

EtienneLescot2c7604e4fc94
diff --git a/electron/stt/whisperServer.test.ts b/electron/stt/whisperServer.test.ts
index 3f495a807..773235577 100644
--- a/electron/stt/whisperServer.test.ts
+++ b/electron/stt/whisperServer.test.ts
@@ -422,11 +422,11 @@ describe("WhisperServerManager", () => {
],
};
- async function run(emissionsReply: Response) {
+ async function run(emissionsReply: Response, reply: object = inference) {
const fetchMock = vi.fn(async (url: string, _init?: RequestInit) =>
url.endsWith("/emissions")
? emissionsReply
- : new Response(JSON.stringify(inference), { status: 200 }),
+ : new Response(JSON.stringify(reply), { status: 200 }),
);
vi.stubGlobal("fetch", fetchMock);
try {
@@ -462,6 +462,19 @@ describe("WhisperServerManager", () => {
expect(result.timing?.rtf).toBeCloseTo(0.3);
});
+ it("leaves the aligner out on the CPU, without even asking for it", async () => {
+ const { result, alignerFor, fetchMock } = await run(
+ new Response(JSON.stringify(emissions), { status: 200 }),
+ { ...inference, backend: "whispercpp-cpu" },
+ );
+ expect(alignerFor).not.toHaveBeenCalled();
+ expect(fetchMock).toHaveBeenCalledOnce();
+ expect(result.wordSegments.map((w) => [w.word, w.startSec, w.endSec])).toEqual([
+ ["ab", 0.2, 0.5],
+ ["ba", 0.5, 1],
+ ]);
+ });
+
it("keeps whisper's times when the aligner fails", async () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
try {
diff --git a/electron/stt/whisperServer.ts b/electron/stt/whisperServer.ts
index 196c9c011..2543a79bb 100644
--- a/electron/stt/whisperServer.ts
+++ b/electron/stt/whisperServer.ts
@@ -646,7 +646,13 @@ export class WhisperServerManager {
)
.filter((w) => w.word.length > 0);
let alignSec = 0;
- if (speech?.length && words.length && opts.alignerFor) {
+ const backend = this.toBackend(json.backend);
+ // GPU only. On the CPU the aligner's forward pass costs +29% (English) to
+ // +58% (French) of a transcription, against +9% to +14% on Vulkan, and
+ // character-level DTW alone is already within 16 ms median there
+ // (tools/stt-eval/word-timing, issue #948). Not asking also means a
+ // CPU-only machine never downloads the model.
+ if (speech?.length && words.length && opts.alignerFor && backend !== "whispercpp-cpu") {
try {
const modelPath = await opts.alignerFor(detectedLanguage);
if (modelPath) {
@@ -670,7 +676,6 @@ export class WhisperServerManager {
// VAD is the better judge of where speech starts after a pause (see
// snapWordBoundaries.ts).
const wordSegments: SttWordSegment[] = anchorWordsOnSpeech(words, speech);
- const backend = this.toBackend(json.backend);
const timing = this.toTiming(json.timing);
// The aligner is part of what this chunk cost.
if (timing && alignSec > 0) {
diff --git a/technical-documentation/architecture/transcription-and-captions.md b/technical-documentation/architecture/transcription-and-captions.md
index 2467004e6..5371502b8 100644
--- a/technical-documentation/architecture/transcription-and-captions.md
+++ b/technical-documentation/architecture/transcription-and-captions.md
@@ -113,7 +113,7 @@ bare `ipcRenderer.invoke`, `fetchWithRetry` has no per-request deadline, and
whisper-server's 30 s readiness budget only starts once the download resolved.
The word aligner (§ Word-level alignment, step 5) is not part of that phase: it
-is fetched in the background the first time a transcription detects English or
+is fetched in the background the first time a transcription on a GPU detects English or
French (109 or 348 MB), and no chunk ever waits for it.
Three edges make that promise hold, and each is load-bearing:
@@ -261,7 +261,7 @@ it verbatim in the response.
the same thing as whisper.cpp's `t_dtw` matching faster-whisper's word
**end** (`tools/stt-eval/whispercpp-dtw-poc/REPORT.md`).
5. **Re-time the words on a CTC aligner** (issue #948, phase 3) — when the
- language whisper detected has one, a wav2vec2 model fine-tuned for CTC scores
+ chunk ran on a GPU and the language whisper detected has an aligner, a wav2vec2 model fine-tuned for CTC scores
every 20 ms frame of the speech against every letter, and
[`electron/stt/ctcAlign.ts`](../../electron/stt/ctcAlign.ts) forces
whisper's own words through those scores:
@@ -287,12 +287,17 @@ it verbatim in the response.
midpoint. Next to a pause (a gap of 100 ms or more between letters) the
edges move further out, 60 ms before and 150 ms after, both capped at
mid-pause: early there is silence, late is an audible attack.
- - **Fallback.** No aligner for the language, a download not landed yet or
+ - **GPU only.** On the CPU the forward pass would add 29% (English) to 58%
+ (French) to a transcription, against 9 to 14% on Vulkan, for a gain over
+ step 2's character-level DTW that is not worth that there. A chunk whisper
+ ran on the CPU skips this step, and a CPU-only machine never downloads the
+ model.
+ - **Fallback.** CPU, no aligner for the language, a download not landed yet or
failed, a helper without `/emissions`, a stretch whose letters do not fit in
its frames: the words keep the times of step 4. Step 6 runs either way.
6. **Anchor phrase edges on the speech** —
[`electron/stt/snapWordBoundaries.ts`](../../electron/stt/snapWordBoundaries.ts)
- leaves boundaries inside a phrase alone: they are within ~16 ms of the audio
+ leaves boundaries inside a phrase alone: they are within ~15 ms of the audio
already, and an energy snap on top of them drags correct boundaries early.
The edges of a phrase are different. The token before a phrase's first word
is the previous phrase's last one, so that word starts in the pause, as
@@ -329,21 +334,29 @@ themselves about 10 to 20 ms from a human's.
|---|---|---|---|---|
| First-token start + 150 ms RMS snap + VAD edges (before #948), TTS | 105 / 275 ms / 32% | 83% | 5% | 84% |
| Previous-token start + two-way VAD edges (phase 1), TTS | 31 / 125 ms / 64% | 89% | 19% | 88% |
-| + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 96% |
-| + CTC aligner (phase 3), TTS noisy | 15 / 39 ms / 95% | 93% | 48% | 93% |
-| Phase 1, LibriSpeech | 40 / 145 ms / 55% | 67% | 14% | 71% |
-| + CTC aligner (phase 3), LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
-
-With the aligner, deleting one word leaves on average 17 ms of it audible and
-cuts 15 ms into its neighbours (TTS clean; 20 and 13 ms on LibriSpeech), against
-40 and 44 ms before. The misses left on phrase-initial starts are mostly the
-reference's: the French TTS voices start a word on its silent stop closure,
-which no aligner hears. On a real French take (25 s, no reference) checked on
-its spectrogram, the aligner moved six boundaries by more than 40 ms: three
-onto a visible boundary that phase 1 had missed by 65 to 250 ms (e.g.
-"montrer | ça" from inside `montrer` to the onset of the /s/), one the right
-way but still about 150 ms early (a drawn-out "Bah… voilà"), and two between
-vowels, where the spectrogram cannot tell.
+| Same, character-level DTW (phase 2), TTS clean | 17 / 60 ms / 86% | 89% | 39% | 94% |
+| + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 95% |
+| Phase 2, TTS noisy | 16 / 60 ms / 86% | 88% | 40% | 92% |
+| + CTC aligner, TTS noisy | 15 / 39 ms / 95% | 94% | 48% | 93% |
+| Phase 2, LibriSpeech | 20 / 65 ms / 82% | 63% | 35% | 81% |
+| + CTC aligner, LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
+
+Per language, character-level DTW gives French 19 ms median and 83% within
+50 ms (was 26 ms, 70%) and English 15 ms and 89% (was 40 ms, 57%). The aligner
+takes French to 14 ms and 95% (P90 66 → 36 ms, clean cuts 33% → 47%) and
+English to 15 ms and 96% (P90 50 → 35 ms, clean cuts 45% → 52%). Deleting one
+word then leaves 17 ms of it audible and cuts 15 ms into its neighbours (TTS
+clean; 20 and 12 ms on LibriSpeech), against 19 and 24 ms with phase 2 alone.
+The misses left on phrase-initial starts are mostly the reference's: the French
+TTS voices start a word on its silent stop closure, which no aligner hears.
+
+On a real French take (25 s, no reference) read on its spectrogram, the
+aligner is not uniformly better than phase 2: it moved "c'est" onto the onset
+of its /s/ (phase 2 was 90 ms early), but put "quoi" after "c'est" 70 to 80 ms
+early twice, inside the vowel of *c'est*, where phase 2 had it on the /k/
+closure. French silent final letters are the likely cause: the model was
+trained on spelling, and the `t` of *c'est* has no sound to sit on (phase 2
+drops such letters before aligning; this pass does not yet).
A +15 ms calibration offset on every boundary gained 4 points of inner
boundaries within 50 ms but dropped noisy phrase deletes to 82%, so it is not
@@ -896,12 +909,15 @@ it deletes data
in `electron/native/whisper-stt/src/main.cpp` are exercised only at
runtime.
- **Word timing without an aligner.** Phrase edges sit on the VAD (step 6),
- and English and French words are re-timed on a CTC aligner (step 5). Every
- other language keeps whisper-small's DTW: about 30 ms median and 125 ms P90
- inside a phrase, so one single-word delete in five is clean. A multilingual
- CTC model (`facebook/omniASR-CTC-300M`) would cover them, at a cost per
+ and on a GPU English and French words are re-timed on a CTC aligner (step 5).
+ Every other language, and every language on the CPU, keeps whisper-small's
+ character-level DTW: about 16 ms median and 60 ms P90 on synthetic speech, so
+ about two single-word deletes in five are clean. A multilingual CTC model
+ (`facebook/omniASR-CTC-300M`) would cover the other languages, at a cost per
language family nobody has measured yet (issue #948).
-- **The aligner on the CPU.** On the GPU it adds about 10% to a transcription.
- On the CPU fallback (16 threads, Ryzen 7 5800X) it adds 28% in English and
- 42 to 61% in French: the French model is a 24-layer wav2vec2 large, and its
- forward pass costs about half of whisper-small's.
+- **The aligner is GPU-only.** On the CPU fallback (16 threads, Ryzen 7 5800X)
+ it would add 29% in English and 58% in French: the French model is a 24-layer
+ wav2vec2 large, about half of whisper-small's cost on its own.
+- **French silent letters.** The aligner spells *c'est* with its `t`, which
+ has no sound; on a real take that put the next word 70 to 80 ms early.
+ Dropping silent final consonants before aligning, as step 2 does, is untried.
diff --git a/tools/stt-eval/word-timing/README.md b/tools/stt-eval/word-timing/README.md
index 9e273aab5..064e3c9bb 100644
--- a/tools/stt-eval/word-timing/README.md
+++ b/tools/stt-eval/word-timing/README.md
@@ -82,7 +82,7 @@ node real-check.mjs <whisper-stt-server.exe> take.wav [--align <aligner.gguf>]
It prints how far each word's first-token time lies after its start, and where
each phrase's first word lands against the VAD onset, raw and after the
-post-pass. With `--align`, every word is listed with its phase-1 and aligner
+post-pass. With `--align`, every word is listed with its DTW and aligner
times, so the boundaries that moved can be checked by ear or on a spectrogram.
## Reading the numbers
diff --git a/tools/stt-eval/word-timing/real-check.mjs b/tools/stt-eval/word-timing/real-check.mjs
index a0d9ea90c..c58a66682 100644
--- a/tools/stt-eval/word-timing/real-check.mjs
+++ b/tools/stt-eval/word-timing/real-check.mjs
@@ -99,7 +99,7 @@ for (const [i, s] of speech.entries()) {
}
if (aligned) {
const moved = [];
- console.log("\nword phase 1 (s) aligner (s) start / end moved");
+ console.log("\nword DTW (s) aligner (s) start / end moved");
raw.forEach((w, j) => {
const a = post[j];
const b = aligned[j];
@@ -111,6 +111,6 @@ if (aligned) {
moved.sort((x, y) => x - y);
const m = (p) => (moved[Math.floor(moved.length * p)] * 1000).toFixed(0);
console.log(
- `\naligner vs phase 1, |moved| per boundary: median ${m(0.5)} ms, p90 ${m(0.9)} ms (${emissions.elapsed_s.toFixed(2)} s on ${emissions.device})`,
+ `\naligner vs DTW, |moved| per boundary: median ${m(0.5)} ms, p90 ${m(0.9)} ms (${emissions.elapsed_s.toFixed(2)} s on ${emissions.device})`,
);
}