Skip to content

Build your personal knowledge base with Trilium Notes

chore(skills): replay and compare meeting transcriptions

Elian Dorana8aebdf4a546
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
new file mode 100644
index 0000000000..63611f8a7c
--- /dev/null
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -0,0 +1,124 @@
+---
+name: working-with-transcription
+description: Use when working on Trilium's meeting transcription — the desktop pipeline in `apps/desktop/src/services/transcription/` (capture window, Silero VAD segmentation, Parakeet via sherpa-onnx-node, sinks and audio taps), the `transcript` attachment and its `TranscriptView`, the "Transcribe meeting" launcher, the standalone WebAssembly prototype, or any change to recognition quality — decoding method, beam search, hotwords, VAD settings, a different model. Includes replay.mjs, which transcribes a development recording again with changed settings, and compare.mjs, which diffs two transcripts word by word with an error rate. Don't judge an engine change by ear or from one live meeting; replay a recording and compare.
+---
+
+# Working with transcription
+
+Desktop transcribes a meeting on the user's machine: no audio leaves it, and outside development builds none is kept. Standalone has only a prototype (`apps/standalone/transcription-poc/`, see its README).
+
+## The pipeline
+
+```
+capture window (hidden, offscreen) main process
+ getUserMedia ─ microphone ─┐
+ getDisplayMedia ─ system ──┴─ IPC "transcription-samples", 4096-sample Float32 chunks
+ │
+ RecordingSession ── audio taps (dev: WavRecorder → *.wav)
+ │
+ per source: SourcePipeline → VadSegmenter (Silero, 512-sample windows)
+ │ utterances
+ one queue: recognizer.transcribe() (Parakeet TDT 0.6B v3 int8, libuv pool)
+ │
+ sinks: AttachmentSink (transcript attachment), dev: TranscriptFileSink
+ │
+ state broadcast "transcription-state" (status, noteId, pendingSources)
+```
+
+| File | Owns |
+|---|---|
+| `apps/desktop/src/services/transcription/capture.ts` | The hidden capture window and its page; microphone and loopback system audio |
+| `…/engine.ts` | `loadSherpaEngine()`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_METHOD`, `VAD_SETTINGS`), reported as `settings` |
+| `…/session.ts` | `RecordingSession`: per-source segmentation, the decode queue, sinks, taps, pending sources |
+| `…/attachment_sink.ts` | The `transcript` attachment, rewritten after every utterance |
+| `…/audio_recorder.ts` | Development recordings: `WavRecorder`, `TranscriptFileSink`, `getRecordingDirectory()` |
+| `…/index.ts` | IPC handlers, the state broadcast, which sinks and taps a recording gets |
+| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `utterances`) |
+| `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in |
+| `apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx` | The launcher and `useTranscriptionState()` |
+
+## The model
+
+Both files go in `<data dir>/transcription-model`; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`.
+
+- Parakeet: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).
+- `silero_vad.onnx` from the same release.
+
+Without the directory, starting fails with `transcription.model-missing`.
+
+## Development recordings
+
+`pnpm desktop:start` (where `isDev()` holds) keeps every transcription in `apps/desktop/data/transcription-recordings/<created>_<noteId>/`:
+
+| File | Holds |
+|---|---|
+| `microphone.wav`, `system.wav` | Mono 16 kHz 16-bit PCM, ~115 MB per hour each; the header is rewritten every chunk, so a crash leaves them playable |
+| `recording.json` | `noteId`, `startedAt`, `sampleRate`, `engine` (the settings used), each source's `firstSampleOffsetMs` |
+| `transcript.json` | The live transcript, in the attachment's format |
+| `transcript.<label>.json` | Replays, written by `replay.mjs`, with a `replay` block (settings, hotwords, timings) |
+
+There is no clean-up: delete directories by hand.
+
+## Replaying a recording
+
+```bash
+S=.claude/skills/working-with-transcription
+node $S/replay.mjs latest --label greedy # the baseline: recorded settings
+node $S/replay.mjs latest --label beam4 \
+ --set recognizer.decodingMethod=modified_beam_search --set recognizer.maxActivePaths=4
+node $S/replay.mjs latest --label hot --hotwords "Trilium/Becca/Froca" \
+ --set recognizer.decodingMethod=modified_beam_search
+node $S/replay.mjs latest --label vad04 --set vad.threshold=0.4 --set vad.maxSpeechDuration=6
+```
+
+- `latest` is the newest development recording; any recording directory works too.
+- It starts from `recording.json`'s `engine` and applies each `--set` (JSON-parsed when it parses). Recognizer keys sherpa-onnx reads from `modelConfig` (`modelType`, `numThreads`, `provider`, `modelingUnit`, `bpeVocab`) go there; the rest (`decodingMethod`, `maxActivePaths`, `hotwordsFile`, `hotwordsScore`, `blankPenalty`, `ruleFsts`, `ruleFars`) go top level.
+- It runs in plain Node with the desktop app's `sherpa-onnx-node`, segments with Silero exactly as `VadSegmenter` does, and times each decode: RTF is decode time over audio length.
+- `--source system` replays one source; `--force` replaces a label.
+
+## Comparing transcripts
+
+```bash
+node $S/compare.mjs latest # list every transcript with its settings and RTF
+node $S/compare.mjs latest greedy beam4 # word diff of beam4 against greedy
+node $S/compare.mjs latest reference beam4 # beam4's word error rate against a corrected one
+```
+
+Per source it counts substitutions, deletions and insertions over the first transcript's words, and lists each difference with its time: `[-removed-] {+added+}`. Casing and punctuation are ignored unless `--strict`.
+
+To measure accuracy rather than difference, make a reference: copy the baseline to `transcript.reference.json` and correct only the `text` fields by listening to the WAVs. `reference` always counts as the ground truth, whichever side it is given on.
+
+## Measuring a change
+
+- **Compare against a replay, never against `transcript.json`.** The live recognizer decoded Float32 samples; a replay decodes the 16-bit WAV, and that rounding alone changed 4.1% of the words on the first recording with identical settings and identical segments. Replays of the same WAV are deterministic (0.0% between two runs), so the baseline is `replay.mjs --label greedy` with no overrides.
+- **One recording is an anecdote.** Check a change on several, with both sources, a non-native speaker, and another language; Parakeet v3 handles 25.
+- **A VAD change moves segment boundaries**, and the recognizer sees different context, so part of any difference is boundaries, not recognition.
+- **Compare RTF only on one machine with nothing else running.** It was 0.125–0.127 for consecutive runs of one recording here; the PoC commit's ~0.03 came from a faster machine.
+- **Check hotwords took effect**: they need `modified_beam_search` (with `greedy_search` sherpa ignores them without a word), and the replay's output must show no `Encode hotwords failed` line. With Parakeet's default settings, a hotword is looked up as a single entry of `tokens.txt`, so an ordinary word fails ("Cannot find ID for token androgynous"), and **one failing term drops the whole list** for that utterance. Encoding words into subword pieces needs `--set recognizer.modelingUnit=bpe --set recognizer.bpeVocab=<file>`, a vocabulary the Parakeet archive does not ship; it has to be exported from the NeMo tokenizer first.
+
+## Findings so far
+
+On the first recording (two minutes of system audio, English interview), against the `greedy` replay:
+
+| Change | Words changed | RTF | Notes |
+|---|---|---|---|
+| `modified_beam_search`, 4 paths | 1.6% | 0.099 (greedy 0.127) | Faster, but all five changes were deletions of spoken words |
+| Hotwords as plain words | 0% | — | Not applied: the terms did not encode (above) |
+
+## Traps
+
+- **Electron's V8 memory cage** rejects sherpa-onnx's external buffers: in the app, `vad.front(false)` (and `readWave(…, false)`). Plain Node, as in `replay.mjs`, does not need it.
+- **Capture chunks must stay a multiple of 512 samples.** `VadSegmenter.accept()` feeds whole 512-sample windows and drops a remainder; 4096 divides evenly.
+- **`maxSpeechDuration` is not a hard cut**: past it, sherpa lowers the minimum silence to 0.1 s and raises the threshold to 0.9, so a segment ends at the next short gap, typically 1–2 s later.
+- **The capture window must be `offscreen`**: a hidden window renders no frames without it, and the self-capture that yields system audio times out. `getDisplayMedia()` needs a user gesture, which only `executeJavaScript(…, true)` gives a hidden window.
+- **A silent system source** usually means the sink's monitor volume is near zero (seen at -66 dB); the per-source peak logged every 5 s shows it.
+- **Prebuilds**: `sherpa-onnx-node` has none for win-arm64. `provider: "trt"` exits the process with an offline model.
+
+## Tests
+
+```bash
+pnpm --filter @triliumnext/desktop test transcription
+pnpm --filter @triliumnext/client test TranscriptView RecordMeetingButton
+```
+
+The desktop specs fake the engine and the capture; nothing loads the model. Changes to recognition quality are measured with the scripts above, not with specs.
diff --git a/.claude/skills/working-with-transcription/compare.mjs b/.claude/skills/working-with-transcription/compare.mjs
new file mode 100644
index 0000000000..0c1795a555
--- /dev/null
+++ b/.claude/skills/working-with-transcription/compare.mjs
@@ -0,0 +1,245 @@
+#!/usr/bin/env node
+// Compares two transcripts of one development recording, word by word and per source.
+//
+// node compare.mjs <recording dir | latest> list the transcripts
+// node compare.mjs <recording dir | latest> <A> <B> compare A with B
+//
+// A and B are labels: `live` is transcript.json, any other is transcript.<label>.json. When one
+// of them is `reference` (a transcript corrected by hand), the rate is the other's word error
+// rate against it; otherwise it is how much B differs from A.
+//
+// --strict count casing and punctuation, which are ignored by default
+// --context <n> words of context around each difference (default 4)
+// --max-hunks <n> differences listed per source (default 30, 0 for all)
+
+import { listTranscripts, parseArgs, readTranscript, resolveRecording } from "./recordings.mjs";
+
+/** Beyond this many edits the transcripts are unrelated, and the diff would take too much memory. */
+const MAX_EDITS = 20_000;
+
+const { positional, options } = parseArgs(process.argv.slice(2), { flags: [ "strict" ] });
+const recording = resolveRecording(positional[0]);
+const [ labelA, labelB ] = positional.slice(1);
+
+if (!labelA) {
+ listAll();
+} else if (!labelB) {
+ console.error("Give two labels to compare, or none to list the transcripts.");
+ process.exit(1);
+} else {
+ compare(labelA, labelB);
+}
+
+function listAll() {
+ console.log(`Recording ${recording.directory}\n`);
+ for (const [ label, transcript ] of listTranscripts(recording.directory)) {
+ const counts = Object.entries(wordsBySource(transcript))
+ .map(([ source, words ]) => `${source} ${words.length} words`).join(", ");
+ console.log(`${label.padEnd(16)} ${transcript.utterances.length} utterances; ${counts}`);
+ console.log(`${"".padEnd(16)} ${describeEngine(transcript)}`);
+ }
+}
+
+function compare(a, b) {
+ const transcriptA = readTranscript(recording.directory, a);
+ const transcriptB = readTranscript(recording.directory, b);
+ // A reference is the ground truth, so it goes first: the error rate counts against its words.
+ const [ refLabel, hypLabel, ref, hyp ] = b === "reference"
+ ? [ b, a, transcriptB, transcriptA ]
+ : [ a, b, transcriptA, transcriptB ];
+ const rateName = refLabel === "reference" ? "WER" : "differs";
+
+ console.log(`${hypLabel} against ${refLabel}`
+ + `${options.strict ? "" : " (ignoring casing and punctuation)"}\n`);
+ for (const [ label, transcript ] of [ [ refLabel, ref ], [ hypLabel, hyp ] ]) {
+ console.log(`${label}: ${describeEngine(transcript)}`);
+ }
+
+ const refWords = wordsBySource(ref);
+ const hypWords = wordsBySource(hyp);
+ const sources = [ ...new Set([ ...Object.keys(refWords), ...Object.keys(hypWords) ]) ].sort();
+ const results = sources.map((source) => ({
+ source, ...diffWords(refWords[source] ?? [], hypWords[source] ?? [])
+ }));
+
+ console.log(`\n${"source".padEnd(12)}${"words".padStart(8)}${"sub".padStart(6)}`
+ + `${"del".padStart(6)}${"ins".padStart(6)}${rateName.padStart(10)}`);
+ for (const { source, refCount, substitutions, deletions, insertions } of results) {
+ const rate = refCount ? (substitutions + deletions + insertions) / refCount : 0;
+ console.log(`${source.padEnd(12)}${String(refCount).padStart(8)}`
+ + `${String(substitutions).padStart(6)}${String(deletions).padStart(6)}`
+ + `${String(insertions).padStart(6)}${`${(rate * 100).toFixed(1)}%`.padStart(10)}`);
+ }
+
+ const maxHunks = Number(options["max-hunks"] ?? 30);
+ for (const { source, hunks } of results) {
+ if (!hunks.length) {
+ continue;
+ }
+ console.log(`\n${source} — ${refLabel} [-removed-] ${hypLabel} {+added+}`);
+ const shown = maxHunks > 0 ? hunks.slice(0, maxHunks) : hunks;
+ for (const hunk of shown) {
+ console.log(` [${formatOffset(hunk.offsetMs)}] ${hunk.text}`);
+ }
+ if (shown.length < hunks.length) {
+ console.log(` … ${hunks.length - shown.length} more (--max-hunks 0 shows all)`);
+ }
+ }
+}
+
+/** Every word of a transcript per source, with the offset of the utterance it belongs to. */
+function wordsBySource(transcript) {
+ const words = {};
+ for (const { source, offsetMs, text } of transcript.utterances) {
+ for (const display of text.split(/\s+/)) {
+ const key = options.strict
+ ? display
+ : display.toLowerCase().replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
+ if (key) {
+ (words[source] ??= []).push({ key, display, offsetMs });
+ }
+ }
+ }
+ return words;
+}
+
+/**
+ * Aligns two word lists with Myers' diff and counts the edits. Within a run of changes, deleted
+ * and inserted words pair up as substitutions, which approximates a Levenshtein alignment.
+ */
+function diffWords(ref, hyp) {
+ const ops = myersDiff(ref.map((word) => word.key), hyp.map((word) => word.key));
+ let substitutions = 0;
+ let deletions = 0;
+ let insertions = 0;
+ const hunks = [];
+ const context = Number(options.context ?? 4);
+
+ let i = 0;
+ while (i < ops.length) {
+ if (ops[i].type === "equal") {
+ i++;
+ continue;
+ }
+ const start = i;
+ while (i < ops.length && ops[i].type !== "equal") {
+ i++;
+ }
+ const run = ops.slice(start, i);
+ const removed = run.filter((op) => op.type === "delete").map((op) => ref[op.index]);
+ const added = run.filter((op) => op.type === "insert").map((op) => hyp[op.index]);
+ const paired = Math.min(removed.length, added.length);
+ substitutions += paired;
+ deletions += removed.length - paired;
+ insertions += added.length - paired;
+
+ const before = ops.slice(Math.max(0, start - context), start)
+ .map((op) => ref[op.index].display);
+ const nextChange = ops.findIndex((op, index) => index >= i && op.type !== "equal");
+ const after = ops.slice(i, Math.min(i + context, nextChange === -1 ? ops.length : nextChange))
+ .map((op) => ref[op.index].display);
+ const parts = [
+ ...before,
+ removed.length ? `[-${removed.map((word) => word.display).join(" ")}-]` : "",
+ added.length ? `{+${added.map((word) => word.display).join(" ")}+}` : "",
+ ...after
+ ].filter(Boolean);
+ hunks.push({
+ offsetMs: (removed[0] ?? added[0]).offsetMs,
+ text: parts.join(" ")
+ });
+ }
+ return { refCount: ref.length, substitutions, deletions, insertions, hunks };
+}
+
+/**
+ * The shortest edit script from `a` to `b`, as `equal`/`delete`/`insert` operations whose `index`
+ * points into `a` (equal, delete) or `b` (insert).
+ */
+function myersDiff(a, b) {
+ const n = a.length;
+ const m = b.length;
+ const max = n + m;
+ const offset = max + 1;
+ const v = new Int32Array(2 * max + 3);
+ // trace[d] holds v for k in [-d - 1, d + 1] before step d, at index k + d + 1.
+ const trace = [];
+ for (let d = 0; d <= max; d++) {
+ if (d > MAX_EDITS) {
+ throw new Error(`The transcripts differ by more than ${MAX_EDITS} words.`);
+ }
+ trace.push(v.slice(offset - d - 1, offset + d + 2));
+ for (let k = -d; k <= d; k += 2) {
+ let x = k === -d || (k !== d && v[offset + k - 1] < v[offset + k + 1])
+ ? v[offset + k + 1]
+ : v[offset + k - 1] + 1;
+ let y = x - k;
+ while (x < n && y < m && a[x] === b[y]) {
+ x++;
+ y++;
+ }
+ v[offset + k] = x;
+ if (x >= n && y >= m) {
+ return backtrack(trace, n, m, a, b);
+ }
+ }
+ }
+ return [];
+}
+
+function backtrack(trace, n, m) {
+ const ops = [];
+ let x = n;
+ let y = m;
+ for (let d = trace.length - 1; d >= 0; d--) {
+ const snapshot = trace[d];
+ const at = (k) => snapshot[k + d + 1];
+ const k = x - y;
+ const previousK = k === -d || (k !== d && at(k - 1) < at(k + 1)) ? k + 1 : k - 1;
+ const previousX = at(previousK);
+ const previousY = previousX - previousK;
+ while (x > previousX && y > previousY) {
+ x--;
+ y--;
+ ops.push({ type: "equal", index: x });
+ }
+ if (d > 0) {
+ if (x === previousX) {
+ ops.push({ type: "insert", index: previousY });
+ } else {
+ ops.push({ type: "delete", index: previousX });
+ }
+ }
+ x = previousX;
+ y = previousY;
+ }
+ return ops.reverse();
+}
+
+function describeEngine(transcript) {
+ const replay = transcript.replay;
+ if (!replay) {
+ return "live, as recorded (settings in recording.json)";
+ }
+ const recognizer = replay.engine?.recognizer ?? {};
+ const parts = [ recognizer.decodingMethod ?? "?" ];
+ if (recognizer.maxActivePaths) {
+ parts.push(`${recognizer.maxActivePaths} paths`);
+ }
+ if (replay.hotwords) {
+ parts.push(`hotwords ${replay.hotwords}`);
+ }
+ const vad = replay.engine?.vad ?? {};
+ parts.push(`VAD threshold ${vad.threshold}, silence ${vad.minSilenceDuration} s, `
+ + `max ${vad.maxSpeechDuration} s`);
+ const timings = Object.entries(replay.sources ?? {})
+ .filter(([ , stats ]) => stats.segments)
+ .map(([ source, stats ]) => `${source} RTF ${stats.rtf.toFixed(3)}`);
+ return [ ...parts, ...timings ].join("; ");
+}
+
+function formatOffset(milliseconds) {
+ const totalSeconds = Math.max(0, Math.floor(milliseconds / 1000));
+ const minutes = Math.floor(totalSeconds / 60);
+ return `${minutes}:${String(totalSeconds % 60).padStart(2, "0")}`;
+}
diff --git a/.claude/skills/working-with-transcription/recordings.mjs b/.claude/skills/working-with-transcription/recordings.mjs
new file mode 100644
index 0000000000..7271974556
--- /dev/null
+++ b/.claude/skills/working-with-transcription/recordings.mjs
@@ -0,0 +1,88 @@
+// Shared by replay.mjs and compare.mjs: finding a development recording and its transcripts.
+
+import fs from "node:fs";
+import path from "node:path";
+
+export const TRANSCRIPT_VERSION = 1;
+
+/** Where `pnpm desktop:start` keeps recordings (its TRILIUM_DATA_DIR is apps/desktop/data). */
+const DEV_RECORDINGS_DIR = path.resolve(
+ import.meta.dirname, "../../../apps/desktop/data/transcription-recordings");
+
+/**
+ * Splits argv into positional arguments and `--name value` options. Names in `multiple` collect
+ * every value into an array; names in `flags` take no value.
+ */
+export function parseArgs(argv, { multiple = [], flags = [] } = {}) {
+ const positional = [];
+ const options = {};
+ for (let i = 0; i < argv.length; i++) {
+ const arg = argv[i];
+ if (!arg.startsWith("--")) {
+ positional.push(arg);
+ continue;
+ }
+ const name = arg.slice(2);
+ if (flags.includes(name)) {
+ options[name] = true;
+ continue;
+ }
+ const value = argv[++i];
+ if (value === undefined) {
+ throw new Error(`--${name} needs a value.`);
+ }
+ if (multiple.includes(name)) {
+ (options[name] ??= []).push(value);
+ } else {
+ options[name] = value;
+ }
+ }
+ return { positional, options };
+}
+
+/**
+ * Reads the recording in `argument`, a recording directory or `latest` (the newest recording of
+ * the development data directory).
+ */
+export function resolveRecording(argument) {
+ let directory = argument;
+ if (!argument || argument === "latest") {
+ const names = fs.existsSync(DEV_RECORDINGS_DIR)
+ ? fs.readdirSync(DEV_RECORDINGS_DIR).sort()
+ : [];
+ if (!names.length) {
+ throw new Error(`No recordings in ${DEV_RECORDINGS_DIR}.`);
+ }
+ directory = path.join(DEV_RECORDINGS_DIR, names.at(-1));
+ }
+ const manifestPath = path.join(directory, "recording.json");
+ if (!fs.existsSync(manifestPath)) {
+ throw new Error(`${directory} is not a recording: it has no recording.json.`);
+ }
+ return { directory, manifest: JSON.parse(fs.readFileSync(manifestPath, "utf8")) };
+}
+
+/** The transcript file of `label`: `live` is the one written while recording. */
+export function transcriptPath(directory, label) {
+ return path.join(directory, label === "live" ? "transcript.json" : `transcript.${label}.json`);
+}
+
+/** Every transcript in a recording directory, by label. */
+export function listTranscripts(directory) {
+ const transcripts = new Map();
+ for (const name of fs.readdirSync(directory).sort()) {
+ const match = /^transcript(?:\.(.+))?\.json$/.exec(name);
+ if (match) {
+ transcripts.set(match[1] ?? "live", readTranscript(directory, match[1] ?? "live"));
+ }
+ }
+ return transcripts;
+}
+
+export function readTranscript(directory, label) {
+ const filePath = transcriptPath(directory, label);
+ if (!fs.existsSync(filePath)) {
+ throw new Error(`No transcript "${label}" in ${directory} (${path.basename(filePath)}).`);
+ }
+ return JSON.parse(fs.readFileSync(filePath, "utf8"));
+}
diff --git a/.claude/skills/working-with-transcription/replay.mjs b/.claude/skills/working-with-transcription/replay.mjs
new file mode 100644
index 0000000000..1050152465
--- /dev/null
+++ b/.claude/skills/working-with-transcription/replay.mjs
@@ -0,0 +1,191 @@
+#!/usr/bin/env node
+// Transcribes a development recording again, with the settings it was recorded with plus any
+// overrides, and writes transcript.<label>.json beside the live transcript.json.
+//
+// node replay.mjs <recording dir | latest> --label <name> [options]
+//
+// --set <path>=<value> override a setting from recording.json's `engine`, e.g.
+// --set recognizer.decodingMethod=modified_beam_search
+// --set recognizer.maxActivePaths=8 --set vad.threshold=0.4
+// (the value is parsed as JSON when it parses, else kept as a string)
+// --hotwords <a/b/c> per-utterance hotwords, separated by "/" (needs modified_beam_search)
+// --model <dir> model directory (default: <data dir>/transcription-model)
+// --source <name> replay only microphone or system
+// --force overwrite an existing transcript.<label>.json
+//
+// Runs in plain Node (no Electron) with the sherpa-onnx-node the desktop app depends on.
+
+import fs from "node:fs";
+import { createRequire } from "node:module";
+import path from "node:path";
+
+import {
+ parseArgs, resolveRecording, TRANSCRIPT_VERSION, transcriptPath
+} from "./recordings.mjs";
+
+const SAMPLE_RATE = 16000;
+const DEFAULT_THREADS = 2;
+/** Recognizer settings that sherpa-onnx reads from `modelConfig`; the rest sit at the top level. */
+const MODEL_CONFIG_KEYS = new Set([
+ "modelType", "numThreads", "provider", "modelingUnit", "bpeVocab", "debug"
+]);
+
+const { positional, options } = parseArgs(process.argv.slice(2), {
+ multiple: [ "set" ], flags: [ "force" ]
+});
+const recording = resolveRecording(positional[0]);
+const label = options.label;
+if (!label || !/^[\w.-]+$/.test(label) || label === "live") {
+ fail("Give a --label of letters, digits, '.', '_' or '-' (not 'live').");
+}
+const outputPath = transcriptPath(recording.directory, label);
+if (fs.existsSync(outputPath) && !options.force) {
+ fail(`${outputPath} exists; pass --force to replace it.`);
+}
+
+const engine = structuredClone(recording.manifest.engine ?? {});
+for (const assignment of options.set ?? []) {
+ applySetting(engine, assignment);
+}
+const modelDirectory = options.model
+ ?? path.join(path.dirname(path.dirname(recording.directory)), "transcription-model");
+const hotwords = options.hotwords;
+if (hotwords && engine.recognizer?.decodingMethod !== "modified_beam_search") {
+ console.warn("Warning: hotwords apply only with "
+ + "--set recognizer.decodingMethod=modified_beam_search; they will be ignored.");
+}
+
+const sherpa = loadSherpa();
+console.log(`Loading the model from ${modelDirectory}…`);
+const loadStarted = performance.now();
+const recognizer = await sherpa.OfflineRecognizer.createAsync(
+ buildRecognizerConfig(engine.recognizer ?? {}, modelDirectory));
+const loadMs = performance.now() - loadStarted;
+
+const utterances = [];
+const sourceStats = {};
+for (const [ source, { file, firstSampleOffsetMs } ] of Object.entries(recording.manifest.sources)) {
+ if (options.source && options.source !== source) {
+ continue;
+ }
+ const wave = sherpa.readWave(path.join(recording.directory, file));
+ if (wave.sampleRate !== SAMPLE_RATE) {
+ fail(`${file} is ${wave.sampleRate} Hz; the engine expects ${SAMPLE_RATE} Hz.`);
+ }
+
+ let decodeMs = 0;
+ let segmentCount = 0;
+ for (const segment of segmentSource(wave.samples, engine.vad ?? {}, modelDirectory)) {
+ const stream = hotwords ? recognizer.createStream(hotwords) : recognizer.createStream();
+ stream.acceptWaveform({ samples: segment.samples, sampleRate: SAMPLE_RATE });
+ const decodeStarted = performance.now();
+ const text = (await recognizer.decodeAsync(stream)).text.trim();
+ decodeMs += performance.now() - decodeStarted;
+ segmentCount++;
+ if (text) {
+ utterances.push({
+ source,
+ offsetMs: firstSampleOffsetMs + segment.start / SAMPLE_RATE * 1000,
+ durationMs: segment.samples.length / SAMPLE_RATE * 1000,
+ text
+ });
+ }
+ }
+ const audioSeconds = wave.samples.length / SAMPLE_RATE;
+ sourceStats[source] = {
+ audioSeconds, segments: segmentCount, decodeMs, rtf: decodeMs / 1000 / audioSeconds
+ };
+ console.log(`${source}: ${segmentCount} segments, ${audioSeconds.toFixed(1)} s of audio `
+ + `decoded in ${(decodeMs / 1000).toFixed(1)} s (RTF ${sourceStats[source].rtf.toFixed(3)})`);
+}
+utterances.sort((a, b) => a.offsetMs - b.offsetMs);
+
+fs.writeFileSync(outputPath, JSON.stringify({
+ version: TRANSCRIPT_VERSION,
+ startedAt: recording.manifest.startedAt,
+ utterances,
+ replay: {
+ label,
+ createdAt: new Date().toISOString(),
+ engine: { ...engine, modelDirectory },
+ hotwords: hotwords ?? null,
+ loadMs,
+ sources: sourceStats
+ }
+}, null, 2));
+console.log(`Wrote ${outputPath}`);
+
+/**
+ * Cuts one source into utterances with Silero VAD, as the live `VadSegmenter` does: 512-sample
+ * windows, then a flush at the end.
+ */
+function* segmentSource(samples, vadSettings, modelDir) {
+ const windowSize = vadSettings.windowSize ?? 512;
+ const vad = new sherpa.Vad({
+ sileroVad: { model: path.join(modelDir, "silero_vad.onnx"), ...vadSettings, windowSize },
+ sampleRate: SAMPLE_RATE,
+ numThreads: 1,
+ provider: "cpu"
+ }, 60);
+ const drain = function* () {
+ while (!vad.isEmpty()) {
+ yield vad.front();
+ vad.pop();
+ }
+ };
+ for (let i = 0; i + windowSize <= samples.length; i += windowSize) {
+ vad.acceptWaveform(samples.subarray(i, i + windowSize));
+ yield* drain();
+ }
+ vad.flush();
+ yield* drain();
+}
+
+function buildRecognizerConfig(settings, modelDir) {
+ const modelConfig = {
+ transducer: {
+ encoder: path.join(modelDir, "encoder.int8.onnx"),
+ decoder: path.join(modelDir, "decoder.int8.onnx"),
+ joiner: path.join(modelDir, "joiner.int8.onnx")
+ },
+ tokens: path.join(modelDir, "tokens.txt"),
+ numThreads: DEFAULT_THREADS
+ };
+ const config = { featConfig: { sampleRate: SAMPLE_RATE, featureDim: 80 }, modelConfig };
+ for (const [ key, value ] of Object.entries(settings)) {
+ (MODEL_CONFIG_KEYS.has(key) ? modelConfig : config)[key] = value;
+ }
+ return config;
+}
+
+function applySetting(target, assignment) {
+ const separator = assignment.indexOf("=");
+ if (separator < 1) {
+ fail(`--set takes <path>=<value>, got "${assignment}".`);
+ }
+ const keys = assignment.slice(0, separator).split(".");
+ const raw = assignment.slice(separator + 1);
+ let value;
+ try {
+ value = JSON.parse(raw);
+ } catch {
+ value = raw;
+ }
+ let node = target;
+ for (const key of keys.slice(0, -1)) {
+ node[key] ??= {};
+ node = node[key];
+ }
+ node[keys.at(-1)] = value;
+}
+
+function loadSherpa() {
+ const repoRoot = path.resolve(import.meta.dirname, "../../..");
+ const require = createRequire(path.join(repoRoot, "apps/desktop/package.json"));
+ return require("sherpa-onnx-node");
+}
+
+function fail(message) {
+ console.error(message);
+ process.exit(1);
+}
diff --git a/apps/desktop/src/services/transcription/engine.ts b/apps/desktop/src/services/transcription/engine.ts
index 4d49bf0b2f..0eca27eda7 100644
--- a/apps/desktop/src/services/transcription/engine.ts
+++ b/apps/desktop/src/services/transcription/engine.ts
@@ -101,7 +101,7 @@ export async function loadSherpaEngine(): Promise<SpeechEngine> {
provider: "cpu"
}, 60)),
settings: {
- modelDirectory: MODEL_DIR,
+ modelDirectory: path.resolve(MODEL_DIR),
recognizer: { ...RECOGNIZER_SETTINGS, decodingMethod: DECODING_METHOD },
vad: VAD_SETTINGS
}