Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
docs(stt): correct the French take reading, and record the silent-letter trial
EtienneLescot50e8e64ed2ab
diff --git a/technical-documentation/architecture/transcription-and-captions.md b/technical-documentation/architecture/transcription-and-captions.mdindex 5371502b8..d3b1dfcaf 100644--- a/technical-documentation/architecture/transcription-and-captions.md+++ b/technical-documentation/architecture/transcription-and-captions.md@@ -350,13 +350,20 @@ clean; 20 and 12 ms on LibriSpeech), against 19 and 24 ms with phase 2 alone.The misses left on phrase-initial starts are mostly the reference's: the FrenchTTS voices start a word on its silent stop closure, which no aligner hears.-On a real French take (25 s, no reference) read on its spectrogram, the-aligner is not uniformly better than phase 2: it moved "c'est" onto the onset-of its /s/ (phase 2 was 90 ms early), but put "quoi" after "c'est" 70 to 80 ms-early twice, inside the vowel of *c'est*, where phase 2 had it on the /k/-closure. French silent final letters are the likely cause: the model was-trained on spelling, and the `t` of *c'est* has no sound to sit on (phase 2-drops such letters before aligning; this pass does not yet).+On a real French take (25 s, no reference), every boundary the aligner moved+by 40 ms or more was checked against the audio's energy and zero crossings at+10 ms: each now sits on the acoustic boundary. "c'est" starts on its /s/+(phase 2 was 90 ms early, in the end of the word before); "quoi" starts where+the /k/ closure begins, after the vowel of *c'est* (phase 2 was on the burst,+70 to 80 ms later, which is also a clean cut since the closure is silent);+in "en fait on va" the aligner is on the /f/, the dip before *on* and the /v/,+where phase 2 was 30 to 50 ms off each.++Dropping French silent final consonants before aligning, as step 2 does, was+tried and left out: it changed none of the take's boundaries by more than+15 ms, and on the corpus it traded 3 points of French clean cuts for 1 to 2+points of French phrase deletes (a phrase-final *fois* and *Windows* lost the+letter that held their end).A +15 ms calibration offset on every boundary gained 4 points of innerboundaries within 50 ms but dropped noisy phrase deletes to 82%, so it is not@@ -918,6 +925,3 @@ it deletes data- **The aligner is GPU-only.** On the CPU fallback (16 threads, Ryzen 7 5800X)it would add 29% in English and 58% in French: the French model is a 24-layerwav2vec2 large, about half of whisper-small's cost on its own.-- **French silent letters.** The aligner spells *c'est* with its `t`, which- has no sound; on a real take that put the next word 70 to 80 ms early.- Dropping silent final consonants before aligning, as step 2 does, is untried.