Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
chore(skills): measure capture quality, hotwords and external transcripts
Elian Dorana0e3dfbc992c
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex 56196fc76a..c255dfee50 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -1,6 +1,6 @@---name: working-with-transcription-description: Use when working on Trilium's meeting transcription — the desktop pipeline in `apps/desktop/src/services/transcription/` (capture window, Silero VAD segmentation, Parakeet via sherpa-onnx-node, sinks and audio taps), the `transcript` attachment and its `TranscriptView`, the "Transcribe meeting" launcher, the standalone WebAssembly prototype, speaker detection (diarization), or any change to recognition quality — decoding method, beam search, hotwords, VAD settings, a different model. Includes replay.mjs, which transcribes a development recording again with changed settings, compare.mjs, which diffs two transcripts word by word with an error rate, and diarize.mjs, which labels a recording's speakers live and offline and times both. Don't judge an engine change by ear or from one live meeting; replay a recording and compare.+description: Use when working on Trilium's meeting transcription — the desktop pipeline in `apps/desktop/src/services/transcription/` (capture window, Silero VAD segmentation, Parakeet via sherpa-onnx-node, sinks and audio taps), the `transcript` attachment and its `TranscriptView`, the "Transcribe meeting" launcher, the standalone WebAssembly prototype, speaker detection (diarization), or any change to recognition quality — decoding method, beam search, hotwords, VAD settings, a different model — or to audio capture (stutters, clicks, the capture worklet). Includes replay.mjs, which transcribes a development recording again with changed settings, compare.mjs, which diffs two transcripts word by word with an error rate, external.mjs, which scores transcripts against another tool's export of the same meeting, hotwords.mjs, which builds the BPE vocabulary hotwords need and checks a glossary, diarize.mjs, which labels a recording's speakers live and offline and times both, capture-check.mjs, which finds capture damage in a recording, and capture-test.mjs, which plays a tone through the real capture code in Electron under load. Don't judge an engine change by ear or from one live meeting; replay a recording and compare.---# Working with transcription@@ -11,7 +11,7 @@ Desktop transcribes a meeting on the user's machine: no audio leaves it, and out```capture window (hidden, offscreen) main process- getUserMedia ─ microphone ─┐+ getUserMedia ─ microphone ─┐ AudioWorklet "chunk-collector" (audio thread)getDisplayMedia ─ system ──┴─ IPC "transcription-samples", 4096-sample Float32 chunks│RecordingSession ── audio taps (dev: WavRecorder → *.wav)@@ -27,13 +27,14 @@ capture window (hidden, offscreen) main process| File | Owns ||---|---|-| `apps/desktop/src/services/transcription/capture.ts` | The hidden capture window and its page; microphone and loopback system audio |+| `apps/desktop/src/services/transcription/capture.ts` | The hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) || `…/engine.ts` | `loadSherpaEngine()`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_SETTINGS`, `VAD_SETTINGS`), reported as `settings` || `…/session.ts` | `RecordingSession`: per-source segmentation, the decode queue, sinks, taps, pending sources || `…/attachment_sink.ts` | The `transcript` attachment, rewritten after every utterance || `…/audio_recorder.ts` | Development recordings: `WavRecorder`, `TranscriptFileSink`, `getRecordingDirectory()` || `…/index.ts` | IPC handlers, the state broadcast, which sinks and taps a recording gets |-| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `speakers`, `utterances`), `parseTranscript()`, speaker keys and `setSpeakerName()` |+| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `speakers`, `utterances`), `parseTranscript()`, speaker keys and `setSpeakerName()`; `cleanUtteranceText()`, the filler and stutter cleanup |+| Cleanup | The stored transcript stays verbatim. `TranscriptView` shows each utterance through `cleanUtteranceText()` while the synced option `transcriptCleanupEnabled` (default on, the "Hide filler words" switch above the transcript) holds, and leaves out utterances it empties. The rules are English-only for stutters (a function-word list) and keep numbers, comma-separated repeats and "that that"/"it it" || Speaker names | `TranscriptView` reads the stored transcript, applies `setSpeakerName()` and saves it whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename during a recording survives, and an utterance a rename overwrote comes back with the next write || `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in || `apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx` | The launcher and `useTranscriptionState()` |@@ -47,6 +48,8 @@ Both files go in `<data dir>/transcription-model`; for `pnpm desktop:start` thatWithout the directory, starting fails with `transcription.model-missing`.+The **full-precision** export is not in the releases: it is `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face (`encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt`; 2.4 GB). Keep it in its own directory (here `apps/desktop/data/transcription-model-fp32`, with `silero_vad.onnx` linked in) and pass `--model` to `replay.mjs`, which takes `<part>.int8.onnx` and falls back to `<part>.onnx`. `engine.ts` still names the int8 files.+## Development recordings`pnpm desktop:start` (where `isDev()` holds) keeps every transcription in `apps/desktop/data/transcription-recordings/<created>_<noteId>/`:@@ -89,6 +92,14 @@ Per source it counts substitutions, deletions and insertions over the first tranTo measure accuracy rather than difference, make a reference: copy the baseline to `transcript.reference.json` and correct only the `text` fields by listening to the WAVs. `reference` always counts as the ground truth, whichever side it is given on.+When another tool transcribed the same meeting (a Teams or AppFlowy export, one `Name: text` line per turn), `external.mjs` scores every transcript against it, the microphone against the lines of `--me` and the system source against everyone else, and counts glossary terms:++```bash+node $S/external.mjs <recording> "<export>.txt" --me Elian live fp32 --terms "heap/STM32/Renesas"+```++The export has errors of its own (it wrote "MicroEdge" and "SortMap", and split words between speakers), so only the difference between labels means something. It writes numbers as digits and drops many fillers, which caps how close any of our transcripts can score.+## Speaker detectionThe app labels the system source's utterances live when `speaker/nemo_en_titanet_small.onnx` is installed: `EmbeddingSpeakerTracker` in `engine.ts` embeds each transcribed utterance and `SpeakerClusterer` (`speakers.ts`, settings `SPEAKER_SETTINGS`) assigns it, giving the utterance a `speaker` number that `TranscriptView` shows as "Speaker n". The microphone is never labelled. It runs synchronously on Electron's main thread for now, ~100 ms per utterance. `replay.mjs` does not label speakers; use `diarize.mjs`.@@ -129,7 +140,50 @@ Two distinct voices on a clean recording are the easy case; similar voices, more- **One recording is an anecdote.** Check a change on several, with both sources, a non-native speaker, and another language; Parakeet v3 handles 25.- **A VAD change moves segment boundaries**, and the recognizer sees different context, so part of any difference is boundaries, not recognition.- **Compare RTF only on one machine with nothing else running.** It was 0.125–0.127 for consecutive runs of one recording here; the PoC commit's ~0.03 came from a faster machine.-- **Check hotwords took effect**: they need `modified_beam_search` (with `greedy_search` sherpa ignores them without a word), and the replay's output must show no `Encode hotwords failed` line. With Parakeet's default settings, a hotword is looked up as a single entry of `tokens.txt`, so an ordinary word fails ("Cannot find ID for token androgynous"), and **one failing term drops the whole list** for that utterance. Encoding words into subword pieces needs `--set recognizer.modelingUnit=bpe --set recognizer.bpeVocab=<file>`, a vocabulary the Parakeet archive does not ship; it has to be exported from the NeMo tokenizer first.+- **Check hotwords took effect**: they need `modified_beam_search` (with `greedy_search` sherpa ignores them without a word), and the replay's output must show no `Encode hotwords failed` line. See "Hotwords".++## Hotwords++Without a BPE vocabulary, sherpa looks a hotword up as a single entry of `tokens.txt`, so an ordinary word fails ("Cannot find ID for token androgynous") and **one failing term drops the whole list** for that utterance. The Parakeet archives ship no vocabulary; `hotwords.mjs` builds one:++```bash+node $S/hotwords.mjs vocab # <model dir>/bpe.vocab from NVIDIA's tokenizer.json+node $S/hotwords.mjs vocab apps/desktop/data/transcription-model-fp32+node $S/hotwords.mjs check "heap/STM32/Iconiq" # how each term splits, and which are risky+node $S/replay.mjs <rec> --model <dir> --label hot --hotwords "heap/STM32/Stéphane" \+ --set recognizer.decodingMethod=modified_beam_search --set recognizer.maxActivePaths=4 \+ --set recognizer.modelingUnit=bpe --set recognizer.bpeVocab=<dir>/bpe.vocab \+ --set recognizer.hotwordsScore=1.5+```++`vocab` scores each piece by the rank of the merge that forms it, so a term splits as the model splits it (`▁J ava ▁he ap`). sherpa's own recipe gives every piece the same score (longest match); on the 44-minute call that missed Stéphane and Renesas, which the ranked vocabulary found.++**sherpa's NeMo beam search keeps the boost of a hotword's first piece** (`offline-transducer-modified-beam-search-nemo-decoder.cc`, 1.13.8): the global `hotwordsScore` is added to the logits before the hypothesis score is read from them, and `Finalize()` takes back only the context-graph score of an unfinished match. Every token that starts a hotword is therefore boosted everywhere, finished or not. A per-phrase score (`term :2`) does not avoid it, since the pre-boost uses the global score; a global score of 0 still left stray words, through pruning. What works is keeping such entries out of the list: `check` flags a term whose first piece is a common word on its own (`Iconiq` → `▁I`, which put 19–25 stray "I" into the call; `heap` → `▁he`).++What a glossary needs, from that call:++- **Score 1.5.** At 3, "SOAR" appeared 161 times.+- **Only terms spoken as they are written.** MicroEJ is said "micro-edge" and came out as "MicroE", "MicroEs", "MicroEch", worse than without it; IntelliJ became "Intel J". Those need a rewrite after recognition ("micro edge" → MicroEJ), not a hotword.+- **No names that open like common words or other names**: Claire turned "Clear for me" into "Claire for me"; the Al- names produced "Alan White." and "Albert?".+- **Drop terms the model already gets** (Iconiq was 10/10 without a hotword).+- **The full-precision model.** On int8, beam search costs more than hotwords gain (microphone WER 17.0% against 15.3% greedy).++## Capture quality++```bash+node $S/capture-check.mjs latest # breaks at chunk boundaries, clipping, digital silence+node $S/capture-test.mjs # the capture worklet vs the old ScriptProcessor, page busy+node $S/capture-test.mjs --capture worklet --load all --seconds 30+node $S/capture-test.mjs --stereo # a left-only tone must arrive at amplitude 0.25+```++`capture-check.mjs` compares the step between neighbouring samples at each 4096-sample chunk boundary with the steps around it: about 1.0–1.3 is a clean join, about 2 means the capture corrupts the start of chunks, heard as a crackle four times a second. It also counts **dips**, 10 ms windows that fall to near silence between loud stretches and fade back within 40–70 ms, per minute with the worst minute.++Dips are a second, separate defect, of the system loopback alone. In the Romanian Google Meet call they came at 3.4 a minute, up to 18, in a burst from minute 3 to 13 while the machine shared Figma from Chrome and the user typed in another Trilium; the microphone had none, the English Teams call 0.09 a minute. The user heard the call cleanly, so the audio was lost by the capture, and inside chunks (not at their boundaries), so upstream of `CAPTURE_PROCESSOR`: Chromium reading the loopback, or delivering it to the hidden page, late under load. `backgroundThrottling: false` might help on Windows, where Chromium lowers the priority of hidden windows; unverified, and `capture-test.mjs` cannot reproduce it, since its tone never passes through the loopback.++`capture-test.mjs` runs the capture for real: Electron (Nix-aware, as `scripts/electron-start.mts`) opens a hidden offscreen window like `createCaptureWindow()`, plays a 440 Hz tone from a 48 kHz context into a `MediaStream` and captures it at 16 kHz with `CAPTURE_PROCESSOR` read from `capture.ts`, under a chosen load (`jank`: the page busy 300 ms of every 500; `block`: the main process blocked 150 ms a second; `burn`: every core busy). A pure sine is predictable from its two previous samples, so every break shows. The node options in its page mirror `pipe()` in `capture.ts`; change both together.++The capture used a `ScriptProcessorNode` until the recordings showed breaks at every chunk boundary (ratio 1.6–2.35 on both tracks of three meetings). Its callback runs on the page's main thread, and when that thread is late Chromium hands it input whose first 128–640 samples (whole 128-frame render quanta) are corrupt, without changing the length. Under the `jank` load it broke 6–8 times in 20 s and 13 times in 30 s under `all`; the worklet, which runs on the audio thread, broke none. Saturating the CPU alone broke neither.## Findings so far@@ -155,11 +209,19 @@ On a 44-minute Teams call (both sources, English, non-native speakers), scored a| 20 s / 1.5 s, `blankPenalty` 1 | 15.3% | 26.6% | The penalty fixes the 1.5 s drops too, at more latency || VAD threshold 0.4 / 0.3 | 15.6 / 16.0% | 25.8 / 26.0% | 0.3 recovers one quiet passage; otherwise noise || `--pad` 0.2 / 0.4 s | 17.3 / 24.4% | 24.5 / 25.3% | Parakeet drops whole clauses again |+| Full precision, greedy, no penalty | 13.3% | 24.9% | RTF 0.021 (int8 0.013); peak memory +1.6 GB; drops no clauses, so needs no `blankPenalty` (0.5 only adds junk on the system track) |+| Full precision, beam 4 + a 27-term glossary at 1.5 | 13.4% | 24.8% | "heap" 47 times (the other tool: 47; int8: 29), "hip" 0 (8), STM32 3/3, Stéphane 3/3, Renesas 1/1 |++The same 31-minute Romanian meeting (no reference, judged by reading): int8 is often nonsense ("Pă2, așa, să interinteg navigator"), full precision mostly readable ("Să integrăm navigatorul") but 62% of the microphone words differ between the two. Both slip into Spanish and Portuguese ("só", "como"), as Parakeet guesses the language per segment and cannot be told it; full precision drops diacritics on common words twice as often (82 against 36 of să/și/că/adică…). Whisper with a fixed language (sherpa-onnx supports it) is the untested next step.## Traps- **Electron's V8 memory cage** rejects sherpa-onnx's external buffers: in the app, `vad.front(false)` (and `readWave(…, false)`). Plain Node, as in `replay.mjs`, does not need it.-- **Capture chunks must stay a multiple of 512 samples.** `VadSegmenter.accept()` feeds whole 512-sample windows and drops a remainder; 4096 divides evenly.+- **Capture chunks must stay a multiple of 512 samples.** `VadSegmenter.accept()` feeds whole 512-sample windows and drops a remainder; `CHUNK_SAMPLES` (4096) divides evenly.+- **Never capture with a `ScriptProcessorNode`.** Its callback runs on the page's main thread and corrupts chunk starts when that thread is late (see "Capture quality").+- **The capture worklet needs `channelCount: 1` with `channelCountMode: "explicit"`** to down-mix a stereo loopback to mono; with the default, `inputs[0][0]` is the left channel alone. `numberOfOutputs: 0` is fine: Chromium still runs the node.+- **`ELECTRON_RUN_AS_NODE=1` is set in a VS Code terminal**, and turns the Electron binary into plain Node (`Cannot find module 'electron'`). `capture-test.mjs` clears it; clear it for any Electron script run from there.+- **No top-level `await` in an ESM Electron entry**: it holds back `ready`, and the app hangs without output.- **`maxSpeechDuration` is not a hard cut**: past it, sherpa lowers the minimum silence to 0.1 s and raises the threshold to 0.9, so a segment ends at the next short gap, typically 1–2 s later.- **The capture window must be `offscreen`**: a hidden window renders no frames without it, and the self-capture that yields system audio times out. `getDisplayMedia()` needs a user gesture, which only `executeJavaScript(…, true)` gives a hidden window.- **Chromium's audio processing is applied during capture**, so the WAV already holds it and no replay can switch it off. The system track runs with echo cancellation, noise suppression and auto gain all off; the microphone keeps echo cancellation, so the meeting played through speakers is not transcribed twice, and its auto gain. Earlier recordings, including the one under "Findings so far", had noise suppression on both tracks and echo cancellation on the system track, so do not compare transcripts across that change.@@ -171,6 +233,7 @@ On a 44-minute Teams call (both sources, English, non-native speakers), scored a```bashpnpm --filter @triliumnext/desktop test transcriptionpnpm --filter @triliumnext/client test TranscriptView RecordMeetingButton+pnpm --filter @triliumnext/commons test transcript```-The desktop specs fake the engine and the capture; nothing loads the model. Changes to recognition quality are measured with the scripts above, not with specs.+The desktop specs fake the engine and the capture; nothing loads the model and nothing runs the capture page. Changes to recognition quality are measured with the scripts above, not with specs, and a change to the capture page with `capture-test.mjs`.diff --git a/.claude/skills/working-with-transcription/capture-check.mjs b/.claude/skills/working-with-transcription/capture-check.mjsnew file mode 100644index 0000000000..1912e3ecff--- /dev/null+++ b/.claude/skills/working-with-transcription/capture-check.mjs@@ -0,0 +1,118 @@+#!/usr/bin/env node+// Checks a development recording's WAVs for capture damage: breaks at the 4096-sample chunk+// boundaries, dips, clipping and runs of digital silence.+//+// node capture-check.mjs [recording dir | latest]+//+// The boundary test compares the step between neighbouring samples at each chunk boundary with+// the steps just before and after it, during audible sound. A capture that joins its chunks+// cleanly scores about the same at the boundary as one sample later (~1.0–1.3); one that corrupts+// the start of a chunk scores about twice that.+//+// A dip is a 10 ms window that falls to near silence between loud stretches and comes back within+// tens of milliseconds. One alone can be a stop consonant; many in a minute are audio the capture+// lost and Chromium faded over. Normal speech gives a handful per meeting; the loopback capture+// of a loaded machine gave up to 18 a minute. The worst minute is printed with the rate.++import fs from "node:fs";+import path from "node:path";++import { parseArgs, resolveRecording } from "./recordings.mjs";++const CHUNK_SAMPLES = 4096;+const RATE = 16000;++const { positional } = parseArgs(process.argv.slice(2));+const recording = resolveRecording(positional[0]);+console.log(recording.directory);+console.log("source minutes boundary control verdict dips/min worst minute clipped silent runs ≥ 10 ms");+for (const [ source, { file } ] of Object.entries(recording.manifest.sources)) {+ const samples = readPcm(path.join(recording.directory, file));+ const boundary = stepRatio(samples, 0);+ const control = stepRatio(samples, 1);+ const isBroken = boundary.ratio > 1.4 && boundary.ratio > control.ratio * 1.25;+ const dips = countDipsPerMinute(samples);+ const worst = dips.reduce((best, count, minute) => count > dips[best] ? minute : best, 0);+ const total = dips.reduce((sum, count) => sum + count, 0);+ console.log([+ source.padEnd(12),+ (samples.length / RATE / 60).toFixed(1).padStart(7),+ boundary.ratio.toFixed(2).padStart(9),+ control.ratio.toFixed(2).padStart(8),+ (boundary.count < 100 ? "too little sound" : isBroken ? "chunk breaks" : "clean").padEnd(16),+ (total / Math.max(1, dips.length)).toFixed(2).padStart(8),+ (dips.length ? `${dips[worst]} at ${worst}:00` : "").padStart(12),+ String(countClipped(samples)).padStart(8),+ String(countSilentRuns(samples)).padStart(8)+ ].join(" "));+}++function readPcm(file) {+ const buffer = fs.readFileSync(file);+ const dataOffset = buffer.indexOf("data") + 8;+ return new Int16Array(buffer.buffer, buffer.byteOffset + dataOffset, (buffer.length - dataOffset) >> 1);+}++/** The mean ratio of the step at each `phase` past a chunk boundary to the steps around it. */+function stepRatio(samples, phase) {+ let sum = 0;+ let count = 0;+ for (let i = CHUNK_SAMPLES + phase; i + 1 < samples.length; i += CHUNK_SAMPLES) {+ if (Math.abs(samples[i - 2]) + Math.abs(samples[i + 1]) < 600) {+ continue;+ }+ const step = Math.abs(samples[i] - samples[i - 1]);+ const around = (Math.abs(samples[i - 1] - samples[i - 2]) + Math.abs(samples[i + 1] - samples[i])) / 2;+ sum += step / (around + 1);+ count++;+ }+ return { ratio: count ? sum / count : 0, count };+}++/** Dips per minute: a 10 ms window below RMS 30 entered from, and left for, RMS above 1500. */+function countDipsPerMinute(samples) {+ const window = RATE / 100;+ const levels = [];+ for (let start = 0; start + window <= samples.length; start += window) {+ let sum = 0;+ for (let i = start; i < start + window; i++) {+ sum += samples[i] * samples[i];+ }+ levels.push(Math.sqrt(sum / window));+ }+ const perMinute = new Array(Math.ceil(samples.length / RATE / 60)).fill(0);+ for (let i = 5; i < levels.length - 5; i++) {+ const before = (levels[i - 5] + levels[i - 4] + levels[i - 3]) / 3;+ const after = (levels[i + 3] + levels[i + 4] + levels[i + 5]) / 3;+ if (before > 1500 && after > 1500 && levels[i] < 30 && levels[i - 1] >= 30) {+ perMinute[Math.floor(i * window / RATE / 60)]++;+ }+ }+ return perMinute;+}++function countClipped(samples) {+ let clipped = 0;+ for (const sample of samples) {+ if (sample >= 32767 || sample <= -32768) {+ clipped++;+ }+ }+ return clipped;+}++function countSilentRuns(samples) {+ let runs = 0;+ let length = 0;+ for (const sample of samples) {+ if (sample === 0) {+ length++;+ continue;+ }+ if (length >= RATE / 100) {+ runs++;+ }+ length = 0;+ }+ return runs;+}diff --git a/.claude/skills/working-with-transcription/capture-test.mjs b/.claude/skills/working-with-transcription/capture-test.mjsnew file mode 100644index 0000000000..d545c3abbe--- /dev/null+++ b/.claude/skills/working-with-transcription/capture-test.mjs@@ -0,0 +1,249 @@+#!/usr/bin/env node+// Plays a sine tone inside Electron and captures it the way the transcription capture window does,+// then reports every break in the tone. The capture worklet is read from capture.ts, so this tests+// the code that ships.+//+// node capture-test.mjs # worklet and the old ScriptProcessor, page busy+// node capture-test.mjs --capture worklet --load all --seconds 30+// node capture-test.mjs --stereo # a left-only tone must arrive at half amplitude+//+// --capture <worklet|script|both> default both; "script" is the ScriptProcessorNode capture+// used before the worklet, kept as the known-bad reference+// --load <none|jank|block|burn|all> default jank:+// jank the capture page busy 300 ms of every 500 ms+// block Electron's main process blocked 150 ms every second+// burn every CPU core busy in worker threads+// --seconds <n> capture length, default 20+// --stereo play the tone on the left channel only+// --keep <dir> also write each capture as <dir>/<capture>-<load>.wav+//+// Runs Electron the way scripts/electron-start.mts does (Nix-aware) and clears+// ELECTRON_RUN_AS_NODE, which a VS Code terminal sets and which turns Electron into plain Node.++import { execFileSync } from "node:child_process";+import fs from "node:fs";+import os from "node:os";+import path from "node:path";+import { fileURLToPath } from "node:url";++import { parseArgs } from "./recordings.mjs";++const RATE = 16000;+const CHUNK_SAMPLES = 4096;+const TONE_HZ = 440;+const AMPLITUDE = 0.5;+const REPO = path.resolve(import.meta.dirname, "../../..");+const CAPTURE_TS = path.join(REPO, "apps/desktop/src/services/transcription/capture.ts");+const SELF = fileURLToPath(import.meta.url);++// No top-level await: in an ESM entry it holds back Electron's "ready" event.+const run = process.versions.electron+ ? runInElectron(JSON.parse(process.env.CAPTURE_TEST_CONFIG ?? "{}"))+ : launch();+run.catch((e) => {+ console.error(e);+ process.exit(1);+});++async function launch() {+ const { options } = parseArgs(process.argv.slice(2), { flags: [ "stereo" ] });+ const captures = (options.capture ?? "both") === "both" ? [ "worklet", "script" ] : [ options.capture ];+ const load = options.load ?? "jank";+ const seconds = Number(options.seconds ?? 20);+ const { getElectronPath, getNixLdLibraryPath, isNixOS } = await import(path.join(REPO, "scripts/utils.mts"));+ const env = { ...process.env, LD_LIBRARY_PATH: isNixOS() ? getNixLdLibraryPath() : process.env.LD_LIBRARY_PATH };+ delete env.ELECTRON_RUN_AS_NODE;++ console.log(`${seconds} s of a ${TONE_HZ} Hz tone${options.stereo ? " (left channel only)" : ""}, load: ${load}\n`);+ for (const capture of captures) {+ const config = { capture, load, seconds, stereo: !!options.stereo, keep: options.keep ?? null };+ const output = execFileSync(getElectronPath(), [ SELF, "--no-sandbox" ], {+ env: { ...env, CAPTURE_TEST_CONFIG: JSON.stringify(config) },+ encoding: "utf8",+ stdio: [ "ignore", "pipe", "pipe" ],+ timeout: (seconds + 60) * 1000+ });+ const line = output.split("\n").find((l) => l.startsWith("RESULT "));+ if (!line) {+ console.error(output);+ process.exit(1);+ }+ const result = JSON.parse(line.slice("RESULT ".length));+ const verdict = result.breaks === 0 ? "clean" : `${result.breaks} break(s)`;+ console.log(`${capture.padEnd(8)} ${verdict.padEnd(12)} ${result.breaksAtChunkStart} at a chunk start; `+ + `captured ${result.capturedSeconds} s in ${result.chunks} chunks; amplitude ${result.amplitude}`+ + (result.firstBreaks.length ? `\n first: ${result.firstBreaks.join(", ")}` : ""));+ }+}++async function runInElectron(config) {+ const { app, BrowserWindow, ipcMain } = await import("electron");+ const source = fs.readFileSync(CAPTURE_TS, "utf8");+ const processor = /const CAPTURE_PROCESSOR = `([\s\S]*?)`;/.exec(source)?.[1];+ if (!processor) {+ throw new Error(`No CAPTURE_PROCESSOR in ${CAPTURE_TS}.`);+ }++ const chunks = [];+ ipcMain.on("samples", (_event, samples) => chunks.push(samples));++ await app.whenReady();+ const pagePath = path.join(os.tmpdir(), `trilium-capture-test-${process.pid}.html`);+ fs.writeFileSync(pagePath, buildPage(config, processor));+ // The same window as createCaptureWindow() in capture.ts.+ const window = new BrowserWindow({+ show: false,+ webPreferences: {+ nodeIntegration: true,+ contextIsolation: false,+ sandbox: false,+ autoplayPolicy: "no-user-gesture-required",+ backgroundThrottling: false,+ offscreen: true+ }+ });+ window.webContents.on("console-message", ({ message }) => {+ if (message.startsWith("failed")) {+ console.log(message);+ }+ });+ await window.loadFile(pagePath);++ if (config.load === "block" || config.load === "all") {+ setInterval(() => {+ const end = Date.now() + 150;+ while (Date.now() < end) { /* block the main process */ }+ }, 1000);+ }+ if (config.load === "burn" || config.load === "all") {+ const { Worker } = await import("node:worker_threads");+ for (let i = 0; i < os.availableParallelism(); i++) {+ new Worker("for (;;) {}", { eval: true });+ }+ }++ await new Promise((resolve) => setTimeout(resolve, config.seconds * 1000));+ const samples = new Float32Array(chunks.reduce((n, c) => n + c.length, 0));+ let offset = 0;+ for (const chunk of chunks) {+ samples.set(chunk, offset);+ offset += chunk.length;+ }+ fs.rmSync(pagePath, { force: true });+ if (config.keep) {+ fs.mkdirSync(config.keep, { recursive: true });+ writeWav(path.join(config.keep, `${config.capture}-${config.load}.wav`), samples);+ }+ console.log(`RESULT ${JSON.stringify({ chunks: chunks.length, ...analyze(samples) })}`);+ app.exit(0);+}++/**+ * A 48 kHz context plays the tone into a MediaStream, as a sound card would, and a 16 kHz context+ * captures that stream with the options pipe() in capture.ts uses.+ */+function buildPage(config, processor) {+ const page = function () {+ const { ipcRenderer } = require("electron");+ const config = CONFIG;++ async function start() {+ const device = new AudioContext({ sampleRate: 48000 });+ const tone = device.createOscillator();+ tone.frequency.value = TONE_HZ;+ const gain = device.createGain();+ gain.gain.value = AMPLITUDE;+ const output = device.createMediaStreamDestination();+ if (config.stereo) {+ const merger = device.createChannelMerger(2);+ tone.connect(gain).connect(merger, 0, 0);+ merger.connect(output);+ } else {+ tone.connect(gain).connect(output);+ }+ tone.start();++ const context = new AudioContext({ sampleRate: RATE });+ const source = context.createMediaStreamSource(output.stream);+ if (config.capture === "script") {+ const node = context.createScriptProcessor(CHUNK_SAMPLES, 1, 1);+ node.onaudioprocess = (e) => ipcRenderer.send("samples", e.inputBuffer.getChannelData(0).slice());+ source.connect(node);+ node.connect(context.destination);+ } else {+ await context.audioWorklet.addModule(URL.createObjectURL(+ new Blob([ PROCESSOR ], { type: "text/javascript" })));+ const node = new AudioWorkletNode(context, "chunk-collector", {+ numberOfInputs: 1,+ numberOfOutputs: 0,+ channelCount: 1,+ channelCountMode: "explicit",+ processorOptions: { chunkSamples: CHUNK_SAMPLES }+ });+ node.port.onmessage = (e) => ipcRenderer.send("samples", e.data);+ source.connect(node);+ }++ if (config.load === "jank" || config.load === "all") {+ setInterval(() => {+ const end = performance.now() + 300;+ while (performance.now() < end) { /* keep the page busy */ }+ }, 500);+ }+ }+ start().catch((e) => console.log(`failed: ${e}`));+ };+ const script = `(${page})()`+ .replaceAll("CONFIG", JSON.stringify(config))+ .replaceAll("PROCESSOR", JSON.stringify(processor))+ .replaceAll("TONE_HZ", String(TONE_HZ))+ .replaceAll("AMPLITUDE", String(AMPLITUDE))+ .replaceAll("CHUNK_SAMPLES", String(CHUNK_SAMPLES))+ .replaceAll("RATE", String(RATE));+ return `<!DOCTYPE html><script>${script}</script>`;+}++/**+ * A pure sine obeys x[n] = 2·cos(ω)·x[n−1] − x[n−2], so the prediction residual is ~0 except where+ * samples were lost, repeated or zeroed. The first second, where the resampler settles, is skipped.+ */+function analyze(samples) {+ const c = 2 * Math.cos(2 * Math.PI * TONE_HZ / RATE);+ const breaks = [];+ let sumSquares = 0;+ for (let n = RATE + 2; n < samples.length; n++) {+ sumSquares += samples[n] * samples[n];+ const residual = Math.abs(samples[n] - c * samples[n - 1] + samples[n - 2]);+ if (residual > 0.01 && (!breaks.length || n - breaks.at(-1) > 4)) {+ breaks.push(n);+ }+ }+ const measured = Math.max(0, samples.length - RATE - 2);+ return {+ capturedSeconds: +(samples.length / RATE).toFixed(3),+ amplitude: +(Math.sqrt(sumSquares / Math.max(1, measured)) * Math.SQRT2).toFixed(3),+ breaks: breaks.length,+ breaksAtChunkStart: breaks.filter((n) => n % CHUNK_SAMPLES <= 2).length,+ firstBreaks: breaks.slice(0, 6).map((n) => `${(n / RATE).toFixed(3)} s (+${n % CHUNK_SAMPLES})`)+ };+}++function writeWav(file, samples) {+ const data = Buffer.alloc(44 + samples.length * 2);+ data.write("RIFF", 0);+ data.writeUInt32LE(36 + samples.length * 2, 4);+ data.write("WAVEfmt ", 8);+ data.writeUInt32LE(16, 16);+ data.writeUInt16LE(1, 20);+ data.writeUInt16LE(1, 22);+ data.writeUInt32LE(RATE, 24);+ data.writeUInt32LE(RATE * 2, 28);+ data.writeUInt16LE(2, 32);+ data.writeUInt16LE(16, 34);+ data.write("data", 36);+ data.writeUInt32LE(samples.length * 2, 40);+ for (const [ i, v ] of samples.entries()) {+ data.writeInt16LE(Math.max(-32768, Math.min(32767, Math.round(v * 32767))), 44 + i * 2);+ }+ fs.writeFileSync(file, data);+}diff --git a/.claude/skills/working-with-transcription/external.mjs b/.claude/skills/working-with-transcription/external.mjsnew file mode 100644index 0000000000..df9feec32f--- /dev/null+++ b/.claude/skills/working-with-transcription/external.mjs@@ -0,0 +1,102 @@+#!/usr/bin/env node+// Scores a recording's transcripts against another tool's transcript of the same meeting (a Teams+// or AppFlowy export: one "Name: text" line per turn), and counts glossary terms in each.+//+// node external.mjs <recording dir | latest> <export.txt> --me <name> [labels…]+// [--terms "heap/Java heap/STM32"]+//+// Lines spoken by --me are compared with the microphone, every other speaker with the system+// source, as word error rates over the external text, so the other tool's own errors count too:+// only the difference between labels means something. Casing, punctuation, accents and the+// fillers uh, um, mm, hmm, ah, eh, oh are ignored. Without labels, every transcript is scored.++import fs from "node:fs";++import { listTranscripts, parseArgs, readTranscript, resolveRecording } from "./recordings.mjs";++const FILLERS = new Set([ "uh", "um", "mm", "mmhmm", "uhhuh", "hmm", "ah", "oh", "eh" ]);++const { positional, options } = parseArgs(process.argv.slice(2));+const recording = resolveRecording(positional[0]);+const exportFile = positional[1];+if (!exportFile || !options.me) {+ console.error("Usage: node external.mjs <recording> <export.txt> --me <name> [labels…] [--terms a/b]");+ process.exit(1);+}++const reference = { microphone: [], system: [] };+let referenceText = "";+for (const line of fs.readFileSync(exportFile, "utf8").split(/\r?\n/)) {+ const turn = /^([^:]{1,60}):\s*(.*)$/.exec(line);+ if (!turn) {+ continue;+ }+ reference[turn[1].trim() === options.me ? "microphone" : "system"].push(...toWords(turn[2]));+ referenceText += ` ${turn[2]}`;+}+if (!reference.microphone.length) {+ console.error(`No line of ${exportFile} is spoken by "${options.me}".`);+ process.exit(1);+}++const labels = positional.slice(2);+const transcripts = labels.length+ ? new Map(labels.map((label) => [ label, readTranscript(recording.directory, label) ]))+ : listTranscripts(recording.directory);+const terms = (options.terms ?? "").split("/").map((term) => term.trim()).filter(Boolean);++const rows = [ [ "label", "mic WER", "system WER", "utterances", ...terms ] ];+rows.push([ "(external)", "", "", "", ...terms.map((term) => countTerm(referenceText, term)) ]);+for (const [ label, transcript ] of transcripts) {+ const hypothesis = { microphone: [], system: [] };+ for (const utterance of [ ...transcript.utterances ].sort((a, b) => a.offsetMs - b.offsetMs)) {+ hypothesis[utterance.source]?.push(...toWords(utterance.text));+ }+ const text = transcript.utterances.map((utterance) => utterance.text).join(" ");+ rows.push([+ label,+ percent(wordErrorRate(reference.microphone, hypothesis.microphone)),+ percent(wordErrorRate(reference.system, hypothesis.system)),+ transcript.utterances.length,+ ...terms.map((term) => countTerm(text, term))+ ]);+}+const widths = rows[0].map((_, column) => Math.max(...rows.map((row) => String(row[column]).length)));+for (const row of rows) {+ console.log(row.map((cell, column) => column ? String(cell).padStart(widths[column]) : String(cell).padEnd(widths[column])).join(" "));+}++function toWords(text) {+ return text.toLowerCase()+ .normalize("NFD").replace(/[̀-ͯ]/g, "")+ .replace(/['’]/g, "")+ .replace(/[^\p{L}\p{N}]+/gu, " ")+ .split(" ")+ .filter((word) => word && !FILLERS.has(word));+}++function wordErrorRate(ref, hyp) {+ if (!ref.length) {+ return NaN;+ }+ let previous = Int32Array.from({ length: hyp.length + 1 }, (_, j) => j);+ let current = new Int32Array(hyp.length + 1);+ for (let i = 1; i <= ref.length; i++) {+ current[0] = i;+ for (let j = 1; j <= hyp.length; j++) {+ current[j] = Math.min(previous[j] + 1, current[j - 1] + 1,+ previous[j - 1] + (ref[i - 1] === hyp[j - 1] ? 0 : 1));+ }+ [ previous, current ] = [ current, previous ];+ }+ return previous[hyp.length] / ref.length;+}++function countTerm(text, term) {+ const escaped = term.replace(/[.*+?^${}()|[\]\\]/g, "\\$&").replace(/\s+/g, "\\s+");+ return (text.match(new RegExp(`(?<![\\p{L}\\p{N}])${escaped}(?![\\p{L}\\p{N}])`, "giu")) ?? []).length;+}++function percent(value) {+ return Number.isNaN(value) ? "—" : `${(value * 100).toFixed(1)}%`;+}diff --git a/.claude/skills/working-with-transcription/hotwords.mjs b/.claude/skills/working-with-transcription/hotwords.mjsnew file mode 100644index 0000000000..f8e2fe26fc--- /dev/null+++ b/.claude/skills/working-with-transcription/hotwords.mjs@@ -0,0 +1,114 @@+#!/usr/bin/env node+// Prepares and checks hotwords for Parakeet.+//+// node hotwords.mjs vocab [model dir] # write <model dir>/bpe.vocab from tokenizer.json+// node hotwords.mjs check "heap/STM32/Iconiq" [model dir]+//+// sherpa-onnx encodes a hotword into tokens with a BPE vocabulary (`modelingUnit: "bpe"`,+// `bpeVocab`), which the Parakeet archives do not ship. `vocab` builds one from NVIDIA's+// tokenizer.json, scoring each piece by the rank of the merge that forms it, so that hotwords are+// split as the model itself splits words. The model directory defaults to+// <data dir>/transcription-model; tokens.txt there must match the tokenizer.+//+// `check` prints how each hotword is split and flags the ones whose first piece is a common word+// on its own: sherpa's NeMo beam search keeps the bonus of a hotword's first piece even when the+// word is never completed, so "Iconiq" (▁I c oni q) inserts a stray "I" and "heap" (▁he ap) turns+// other words into "he".++import fs from "node:fs";+import path from "node:path";++const TOKENIZER_URL = "https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/resolve/main/tokenizer.json";+const REPO = path.resolve(import.meta.dirname, "../../..");+const DEFAULT_MODEL_DIR = path.join(REPO, "apps/desktop/data/transcription-model");+const UNMERGED_SCORE = -100000;+/** Words a first piece is risky as: frequent enough that boosting them changes ordinary speech. */+const COMMON_WORDS = new Set([+ "i", "a", "he", "she", "it", "we", "you", "they", "me", "my", "us", "so", "to", "in", "on", "at",+ "of", "or", "is", "be", "do", "go", "no", "an", "as", "by", "up", "if", "the", "and", "but",+ "for", "not", "all", "are", "was", "can", "one", "out", "now", "how", "his", "her", "its"+]);++const [ command, ...args ] = process.argv.slice(2);+if (command === "vocab") {+ await writeVocab(args[0] ?? DEFAULT_MODEL_DIR);+} else if (command === "check" && args[0]) {+ check(args[0], args[1] ?? DEFAULT_MODEL_DIR);+} else {+ console.error("Usage: node hotwords.mjs vocab [model dir] | check \"a/b/c\" [model dir]");+ process.exit(1);+}++async function writeVocab(modelDir) {+ const tokenizerPath = path.join(modelDir, "tokenizer.json");+ if (!fs.existsSync(tokenizerPath)) {+ console.log(`Downloading ${TOKENIZER_URL}…`);+ const response = await fetch(TOKENIZER_URL);+ if (!response.ok) {+ throw new Error(`${TOKENIZER_URL}: HTTP ${response.status}`);+ }+ fs.writeFileSync(tokenizerPath, Buffer.from(await response.arrayBuffer()));+ }+ const { model } = JSON.parse(fs.readFileSync(tokenizerPath, "utf8"));+ const tokens = readTokens(modelDir);+ const mismatched = tokens.filter(({ piece, id }) => piece in model.vocab && model.vocab[piece] !== id);+ if (mismatched.length) {+ throw new Error(`tokens.txt does not match tokenizer.json (${mismatched.length} ids differ).`);+ }+ const rank = new Map();+ for (const [ index, merge ] of model.merges.entries()) {+ const [ left, right ] = Array.isArray(merge) ? merge : merge.split(" ");+ if (!rank.has(left + right)) {+ rank.set(left + right, index);+ }+ }+ const lines = tokens.map(({ piece }) => `${piece}\t${rank.has(piece) ? -rank.get(piece) : UNMERGED_SCORE}`);+ fs.writeFileSync(path.join(modelDir, "bpe.vocab"), `${lines.join("\n")}\n`);+ console.log(`Wrote ${path.join(modelDir, "bpe.vocab")} (${lines.length} pieces).`);+}++function check(hotwords, modelDir) {+ const vocabPath = path.join(modelDir, "bpe.vocab");+ if (!fs.existsSync(vocabPath)) {+ throw new Error(`No ${vocabPath}; run "node hotwords.mjs vocab" first.`);+ }+ const scores = new Map(fs.readFileSync(vocabPath, "utf8").split("\n").filter(Boolean).map((line) => {+ const [ piece, score ] = line.split("\t");+ return [ piece, Number(score) ];+ }));+ for (const hotword of hotwords.split("/").map((word) => word.replace(/\s*:[\d.]+$/, "").trim()).filter(Boolean)) {+ const pieces = encode(hotword, scores);+ const first = pieces[0].replace(/^▁/, "");+ const warning = COMMON_WORDS.has(first.toLowerCase())+ ? ` ← boosts the word "${first}" wherever it is not completed`+ : "";+ console.log(`${hotword.padEnd(20)} ${pieces.join(" ")}${warning}`);+ }+}++/** Splits a word as sherpa's BPE encoder does: merge the best-scoring adjacent pair until none is left. */+function encode(text, scores) {+ const pieces = [ ..."▁" + text.replace(/ /g, "▁") ];+ for (;;) {+ let best = -Infinity;+ let at = -1;+ for (let i = 0; i + 1 < pieces.length; i++) {+ const score = scores.get(pieces[i] + pieces[i + 1]);+ if (score !== undefined && score > UNMERGED_SCORE && score > best) {+ best = score;+ at = i;+ }+ }+ if (at < 0) {+ return pieces;+ }+ pieces.splice(at, 2, pieces[at] + pieces[at + 1]);+ }+}++function readTokens(modelDir) {+ return fs.readFileSync(path.join(modelDir, "tokens.txt"), "utf8").split("\n").filter(Boolean).map((line) => {+ const separator = line.lastIndexOf(" ");+ return { piece: line.slice(0, separator), id: Number(line.slice(separator + 1)) };+ });+}diff --git a/.claude/skills/working-with-transcription/replay.mjs b/.claude/skills/working-with-transcription/replay.mjsindex f3985d1be3..4d9fab202c 100644--- a/.claude/skills/working-with-transcription/replay.mjs+++ b/.claude/skills/working-with-transcription/replay.mjs@@ -9,7 +9,8 @@// --set recognizer.maxActivePaths=8 --set vad.threshold=0.4// (the value is parsed as JSON when it parses, else kept as a string)// --hotwords <a/b/c> per-utterance hotwords, separated by "/" (needs modified_beam_search)-// --model <dir> model directory (default: <data dir>/transcription-model)+// --model <dir> model directory (default: <data dir>/transcription-model); takes+// <part>.int8.onnx, else <part>.onnx, for encoder, decoder and joiner// --source <name> replay only microphone or system// --pad <seconds> extend each segment by this much audio on both sides before decoding// --force overwrite an existing transcript.<label>.json@@ -124,9 +125,9 @@ console.log(`Wrote ${outputPath}`);function buildRecognizerConfig(settings, modelDir) {const modelConfig = {transducer: {- encoder: path.join(modelDir, "encoder.int8.onnx"),- decoder: path.join(modelDir, "decoder.int8.onnx"),- joiner: path.join(modelDir, "joiner.int8.onnx")+ encoder: modelFile(modelDir, "encoder"),+ decoder: modelFile(modelDir, "decoder"),+ joiner: modelFile(modelDir, "joiner")},tokens: path.join(modelDir, "tokens.txt"),numThreads: DEFAULT_THREADS@@ -138,6 +139,12 @@ function buildRecognizerConfig(settings, modelDir) {return config;}+/** The int8 file the app ships, or the full-precision `<part>.onnx` of another export. */+function modelFile(modelDir, part) {+ const int8 = path.join(modelDir, `${part}.int8.onnx`);+ return fs.existsSync(int8) ? int8 : path.join(modelDir, `${part}.onnx`);+}+function applySetting(target, assignment) {const separator = assignment.indexOf("=");if (separator < 1) {