Skip to content

Build your personal knowledge base with Trilium Notes

chore(skills): measure speaker detection on a recording, live and offline

Elian Doran12a248dd0375
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index 9445eef34f..2ca7369a33 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -1,6 +1,6 @@
---
name: working-with-transcription
-description: Use when working on Trilium's meeting transcription — the desktop pipeline in `apps/desktop/src/services/transcription/` (capture window, Silero VAD segmentation, Parakeet via sherpa-onnx-node, sinks and audio taps), the `transcript` attachment and its `TranscriptView`, the "Transcribe meeting" launcher, the standalone WebAssembly prototype, or any change to recognition quality — decoding method, beam search, hotwords, VAD settings, a different model. Includes replay.mjs, which transcribes a development recording again with changed settings, and compare.mjs, which diffs two transcripts word by word with an error rate. Don't judge an engine change by ear or from one live meeting; replay a recording and compare.
+description: Use when working on Trilium's meeting transcription — the desktop pipeline in `apps/desktop/src/services/transcription/` (capture window, Silero VAD segmentation, Parakeet via sherpa-onnx-node, sinks and audio taps), the `transcript` attachment and its `TranscriptView`, the "Transcribe meeting" launcher, the standalone WebAssembly prototype, speaker detection (diarization), or any change to recognition quality — decoding method, beam search, hotwords, VAD settings, a different model. Includes replay.mjs, which transcribes a development recording again with changed settings, compare.mjs, which diffs two transcripts word by word with an error rate, and diarize.mjs, which labels a recording's speakers live and offline and times both. Don't judge an engine change by ear or from one live meeting; replay a recording and compare.
---
# Working with transcription
@@ -88,6 +88,38 @@ Per source it counts substitutions, deletions and insertions over the first tran
To measure accuracy rather than difference, make a reference: copy the baseline to `transcript.reference.json` and correct only the `text` fields by listening to the WAVs. `reference` always counts as the ground truth, whichever side it is given on.
+## Speaker detection
+
+Not in the app yet. `diarize.mjs` labels one source's utterances by speaker in two ways and times both:
+
+- **live**: one embedding per utterance, in arrival order, joined to the closest speaker heard so far when the cosine similarity reaches `--threshold` (0.5), else a new speaker; utterances shorter than `--min-duration` (1.5 s) never start one. This is what can run during a meeting.
+- **offline**: sherpa-onnx's `OfflineSpeakerDiarization` over the whole source (pyannote segmentation, embeddings, clustering), mapped onto the utterances by overlap; `+Dn` marks an utterance in which a second speaker talks ≥ 1 s.
+
+```bash
+node $S/diarize.mjs latest # both, TitaNet, system source
+node $S/diarize.mjs latest --mode offline --clusters 2 # when the speaker count is known
+node $S/diarize.mjs latest --embedding campplus --threads 2
+```
+
+Models, in `<data dir>/transcription-model/speaker/`, from `k2-fsa/sherpa-onnx` releases:
+
+- `nemo_en_titanet_small.onnx` (40 MB) and `3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx` (30 MB) from `speaker-recongition-models` (the tag is misspelled upstream);
+- `segmentation.onnx` (6 MB): `model.onnx` of `sherpa-onnx-pyannote-segmentation-3-0.tar.bz2` from `speaker-segmentation-models`, renamed.
+
+`SpeakerEmbeddingExtractor.compute()` and `OfflineSpeakerDiarization.process()` are **synchronous**, with no async variant: in the app they must not run on Electron's main thread (a worker thread, or the capture window).
+
+Results on the first recording (two minutes, interviewer and curator on the system track, one thread):
+
+| Method | Speakers found | Correct | Time for 117 s | CPU |
+|---|---|---|---|---|
+| live, TitaNet | 2 | Every single-speaker utterance; the other speaker scored 0.25 against 0.60–0.90 | 1.2 s (RTF 0.010), ~100 ms per utterance | ≈ wall |
+| live, CAM++ | 4 | Split both speakers | 1.4 s | ≈ wall |
+| offline, TitaNet, threshold 0.5 (default) | 6 | Over-split | 16.8 s (RTF 0.143) | ≈ wall |
+| offline, TitaNet, threshold 1.0 or `--clusters 2` | 2 | As live, and flagged the 4 utterances where the interviewer cuts in | 20.7 s (RTF 0.177); 11.7 s on 2 threads | 20.6 s; 23.3 s on 2 threads |
+| offline, CAM++, `--clusters 2` | 2 | Wrong split | 18.4 s | ≈ wall |
+
+Two distinct voices on a clean recording are the easy case; similar voices, more speakers and call-quality audio are untested.
+
## Measuring a change
- **Compare against a replay, never against `transcript.json`.** The live recognizer decoded Float32 samples; a replay decodes the 16-bit WAV, and that rounding alone changed 4.1% of the words on the first recording with identical settings and identical segments. Replays of the same WAV are deterministic (0.0% between two runs), so the baseline is `replay.mjs --label greedy` with no overrides.
diff --git a/.claude/skills/working-with-transcription/diarize.mjs b/.claude/skills/working-with-transcription/diarize.mjs
new file mode 100644
index 0000000000..ba0d64b7d8
--- /dev/null
+++ b/.claude/skills/working-with-transcription/diarize.mjs
@@ -0,0 +1,281 @@
+#!/usr/bin/env node
+// Labels the utterances of one source of a development recording by speaker, two ways, and times
+// both:
+// live one embedding per utterance as it would arrive, matched against the speakers heard so
+// far (the method that can run during a meeting)
+// offline sherpa-onnx's whole-recording diarization: pyannote segmentation, embeddings,
+// clustering (the method that needs the finished recording)
+//
+// node diarize.mjs <recording dir | latest> [options]
+//
+// --source <name> microphone or system (default system)
+// --transcript <label> whose utterances to label (default greedy, else live)
+// --embedding <name|file> titanet (default), campplus, or a model file
+// --mode <live|offline|both> (default both)
+// --threshold <n> live: cosine similarity that joins a known speaker (default 0.5)
+// --min-duration <s> live: shorter utterances never start a speaker (default 1.5)
+// --clusters <n> offline: the number of speakers, when known
+// --cluster-threshold <n> offline: clustering threshold otherwise (default 0.5)
+// --threads <n> threads per model (default 1)
+//
+// Models: <data dir>/transcription-model/speaker/ holds the embedding models and
+// segmentation.onnx; see SKILL.md, "Speaker detection".
+
+import fs from "node:fs";
+import path from "node:path";
+
+import { listTranscripts, parseArgs, readTranscript, resolveRecording } from "./recordings.mjs";
+import { loadSherpa, SAMPLE_RATE } from "./sherpa.mjs";
+
+const EMBEDDING_MODELS = {
+ titanet: "nemo_en_titanet_small.onnx",
+ campplus: "3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx"
+};
+/** An utterance whose second speaker talks this long within it counts as a speaker change. */
+const CHANGE_SECONDS = 1;
+
+const { positional, options } = parseArgs(process.argv.slice(2));
+const recording = resolveRecording(positional[0]);
+const sourceName = options.source ?? "system";
+const source = recording.manifest.sources[sourceName];
+if (!source) {
+ fail(`The recording has no ${sourceName} source.`);
+}
+const transcriptLabel = options.transcript
+ ?? (listTranscripts(recording.directory).has("greedy") ? "greedy" : "live");
+const mode = options.mode ?? "both";
+const threads = Number(options.threads ?? 1);
+const speakerDir = path.join(
+ path.dirname(path.dirname(recording.directory)), "transcription-model", "speaker");
+const embeddingModel = EMBEDDING_MODELS[options.embedding ?? "titanet"]
+ ? path.join(speakerDir, EMBEDDING_MODELS[options.embedding ?? "titanet"])
+ : options.embedding;
+if (!fs.existsSync(embeddingModel)) {
+ fail(`No embedding model at ${embeddingModel}.`);
+}
+
+const sherpa = loadSherpa();
+const wave = sherpa.readWave(path.join(recording.directory, source.file));
+const audioSeconds = wave.samples.length / SAMPLE_RATE;
+const utterances = readTranscript(recording.directory, transcriptLabel).utterances
+ .filter((utterance) => utterance.source === sourceName)
+ .map((utterance) => {
+ const startSeconds = (utterance.offsetMs - source.firstSampleOffsetMs) / 1000;
+ return { ...utterance, startSeconds, endSeconds: startSeconds + utterance.durationMs / 1000 };
+ });
+
+console.log(`${sourceName} of ${path.basename(recording.directory)}: ${audioSeconds.toFixed(1)} s, `
+ + `${utterances.length} utterances from "${transcriptLabel}", `
+ + `${path.basename(embeddingModel)}, ${threads} thread(s)\n`);
+
+const live = mode !== "offline" ? labelLive() : null;
+const offline = mode !== "live" ? labelOffline() : null;
+
+for (const [ index, utterance ] of utterances.entries()) {
+ const columns = [ formatSeconds(utterance.startSeconds).padStart(6),
+ `${(utterance.durationMs / 1000).toFixed(1)}s`.padStart(6) ];
+ if (live) {
+ const { speaker, similarity } = live.labels[index];
+ columns.push(`${speaker} ${similarity === null ? "" : similarity.toFixed(2)}`.padEnd(10));
+ }
+ if (offline) {
+ const { speaker, change } = offline.labels[index];
+ columns.push(`${speaker}${change ? ` +${change}` : ""}`.padEnd(10));
+ }
+ columns.push(utterance.text.length > 70 ? `${utterance.text.slice(0, 69)}…` : utterance.text);
+ console.log(columns.join(" "));
+}
+
+console.log("");
+for (const [ name, result ] of [ [ "live", live ], [ "offline", offline ] ]) {
+ if (!result) {
+ continue;
+ }
+ const speakers = new Set(result.labels.map((label) => label.speaker).filter((s) => s !== "?"));
+ console.log(`${name.padEnd(8)} ${speakers.size} speakers; ${result.timing}`);
+}
+if (live && offline) {
+ const changes = offline.labels.filter((label) => label.change).length;
+ console.log(`agreement ${(agreement(live.labels, offline.labels) * 100).toFixed(0)}% of speech `
+ + `after matching speakers; ${changes} utterance(s) hold a speaker change `
+ + `(offline, ≥${CHANGE_SECONDS} s of a second speaker)`);
+}
+
+/** One embedding per utterance, in arrival order, matched by cosine similarity to centroids. */
+function labelLive() {
+ const threshold = Number(options.threshold ?? 0.5);
+ const minDuration = Number(options["min-duration"] ?? 1.5);
+ const loadStarted = performance.now();
+ const extractor = new sherpa.SpeakerEmbeddingExtractor({
+ model: embeddingModel, numThreads: threads, provider: "cpu"
+ });
+ const loadMs = performance.now() - loadStarted;
+
+ const centroids = [];
+ const labels = [];
+ const perUtteranceMs = [];
+ const cpuBefore = process.cpuUsage();
+ const started = performance.now();
+ for (const utterance of utterances) {
+ const utteranceStarted = performance.now();
+ const embedding = normalize(embed(extractor, utterance));
+ perUtteranceMs.push(performance.now() - utteranceStarted);
+
+ let best = -1;
+ let bestSimilarity = -1;
+ for (const [ index, centroid ] of centroids.entries()) {
+ const similarity = dot(normalize(centroid), embedding);
+ if (similarity > bestSimilarity) {
+ best = index;
+ bestSimilarity = similarity;
+ }
+ }
+ const isLongEnough = utterance.durationMs / 1000 >= minDuration;
+ if (best >= 0 && bestSimilarity >= threshold) {
+ if (isLongEnough) {
+ addInto(centroids[best], embedding);
+ }
+ labels.push({ speaker: `S${best + 1}`, similarity: bestSimilarity });
+ } else if (isLongEnough) {
+ centroids.push(Float32Array.from(embedding));
+ labels.push({ speaker: `S${centroids.length}`, similarity: best >= 0 ? bestSimilarity : null });
+ } else {
+ labels.push({ speaker: "?", similarity: best >= 0 ? bestSimilarity : null });
+ }
+ }
+ const wallMs = performance.now() - started;
+ const cpuMs = cpuMillis(process.cpuUsage(cpuBefore));
+ const sorted = [ ...perUtteranceMs ].sort((a, b) => a - b);
+ return {
+ labels,
+ timing: `load ${loadMs.toFixed(0)} ms; ${(wallMs / 1000).toFixed(2)} s wall, `
+ + `${(cpuMs / 1000).toFixed(2)} s CPU for ${audioSeconds.toFixed(0)} s of audio `
+ + `(RTF ${(wallMs / 1000 / audioSeconds).toFixed(3)}); per utterance median `
+ + `${sorted[Math.floor(sorted.length / 2)].toFixed(0)} ms, max ${sorted.at(-1).toFixed(0)} ms`
+ };
+}
+
+/** sherpa-onnx's diarization over the whole source, mapped onto the utterances by overlap. */
+function labelOffline() {
+ const clustering = options.clusters
+ ? { numClusters: Number(options.clusters) }
+ : { threshold: Number(options["cluster-threshold"] ?? 0.5) };
+ const loadStarted = performance.now();
+ const diarization = new sherpa.OfflineSpeakerDiarization({
+ segmentation: {
+ pyannote: { model: path.join(speakerDir, "segmentation.onnx") },
+ numThreads: threads
+ },
+ embedding: { model: embeddingModel, numThreads: threads },
+ clustering,
+ minDurationOn: 0.2,
+ minDurationOff: 0.5
+ });
+ const loadMs = performance.now() - loadStarted;
+
+ const cpuBefore = process.cpuUsage();
+ const started = performance.now();
+ const segments = diarization.process(wave.samples);
+ const wallMs = performance.now() - started;
+ const cpuMs = cpuMillis(process.cpuUsage(cpuBefore));
+
+ const labels = utterances.map((utterance) => {
+ const overlap = new Map();
+ for (const segment of segments) {
+ const seconds = Math.min(segment.end, utterance.endSeconds)
+ - Math.max(segment.start, utterance.startSeconds);
+ if (seconds > 0) {
+ overlap.set(segment.speaker, (overlap.get(segment.speaker) ?? 0) + seconds);
+ }
+ }
+ const ranked = [ ...overlap ].sort((a, b) => b[1] - a[1]);
+ if (!ranked.length) {
+ return { speaker: "?", change: null };
+ }
+ const second = ranked[1];
+ return {
+ speaker: `D${ranked[0][0] + 1}`,
+ change: second && second[1] >= CHANGE_SECONDS ? `D${second[0] + 1}` : null
+ };
+ });
+ return {
+ labels,
+ timing: `load ${loadMs.toFixed(0)} ms; ${(wallMs / 1000).toFixed(2)} s wall, `
+ + `${(cpuMs / 1000).toFixed(2)} s CPU for ${audioSeconds.toFixed(0)} s of audio `
+ + `(RTF ${(wallMs / 1000 / audioSeconds).toFixed(3)}); ${segments.length} segments, `
+ + `clustering ${JSON.stringify(clustering)}`
+ };
+}
+
+function embed(extractor, utterance) {
+ const start = Math.max(0, Math.round(utterance.startSeconds * SAMPLE_RATE));
+ const end = Math.min(wave.samples.length, Math.round(utterance.endSeconds * SAMPLE_RATE));
+ const stream = extractor.createStream();
+ stream.acceptWaveform({ samples: wave.samples.subarray(start, end), sampleRate: SAMPLE_RATE });
+ stream.inputFinished();
+ if (!extractor.isReady(stream)) {
+ fail(`The utterance at ${formatSeconds(utterance.startSeconds)} is too short to embed.`);
+ }
+ return extractor.compute(stream);
+}
+
+/**
+ * The share of speech, weighted by utterance length, on which live and offline agree, after
+ * mapping each live speaker to the offline speaker it overlaps most.
+ */
+function agreement(liveLabels, offlineLabels) {
+ const weights = new Map();
+ let total = 0;
+ for (const [ index, utterance ] of utterances.entries()) {
+ const key = `${liveLabels[index].speaker}|${offlineLabels[index].speaker}`;
+ weights.set(key, (weights.get(key) ?? 0) + utterance.durationMs);
+ total += utterance.durationMs;
+ }
+ const mapping = new Map();
+ for (const [ key ] of [ ...weights ].sort((a, b) => b[1] - a[1])) {
+ const [ liveSpeaker, offlineSpeaker ] = key.split("|");
+ if (!mapping.has(liveSpeaker) && ![ ...mapping.values() ].includes(offlineSpeaker)) {
+ mapping.set(liveSpeaker, offlineSpeaker);
+ }
+ }
+ let agreed = 0;
+ for (const [ index, utterance ] of utterances.entries()) {
+ if (mapping.get(liveLabels[index].speaker) === offlineLabels[index].speaker) {
+ agreed += utterance.durationMs;
+ }
+ }
+ return total ? agreed / total : 0;
+}
+
+function normalize(vector) {
+ const length = Math.hypot(...vector) || 1;
+ return vector.map((value) => value / length);
+}
+
+function dot(a, b) {
+ let sum = 0;
+ for (let i = 0; i < a.length; i++) {
+ sum += a[i] * b[i];
+ }
+ return sum;
+}
+
+function addInto(target, vector) {
+ for (let i = 0; i < target.length; i++) {
+ target[i] += vector[i];
+ }
+}
+
+function cpuMillis({ user, system }) {
+ return (user + system) / 1000;
+}
+
+function formatSeconds(seconds) {
+ const total = Math.max(0, Math.floor(seconds));
+ return `${Math.floor(total / 60)}:${String(total % 60).padStart(2, "0")}`;
+}
+
+function fail(message) {
+ console.error(message);
+ process.exit(1);
+}
diff --git a/.claude/skills/working-with-transcription/replay.mjs b/.claude/skills/working-with-transcription/replay.mjs
index 1050152465..8992804c7a 100644
--- a/.claude/skills/working-with-transcription/replay.mjs
+++ b/.claude/skills/working-with-transcription/replay.mjs
@@ -16,14 +16,13 @@
// Runs in plain Node (no Electron) with the sherpa-onnx-node the desktop app depends on.
import fs from "node:fs";
-import { createRequire } from "node:module";
import path from "node:path";
import {
parseArgs, resolveRecording, TRANSCRIPT_VERSION, transcriptPath
} from "./recordings.mjs";
+import { loadSherpa, SAMPLE_RATE, segmentSource } from "./sherpa.mjs";
-const SAMPLE_RATE = 16000;
const DEFAULT_THREADS = 2;
/** Recognizer settings that sherpa-onnx reads from `modelConfig`; the rest sit at the top level. */
const MODEL_CONFIG_KEYS = new Set([
@@ -75,7 +74,7 @@ for (const [ source, { file, firstSampleOffsetMs } ] of Object.entries(recording
let decodeMs = 0;
let segmentCount = 0;
- for (const segment of segmentSource(wave.samples, engine.vad ?? {}, modelDirectory)) {
+ for (const segment of segmentSource(sherpa, wave.samples, engine.vad ?? {}, modelDirectory)) {
const stream = hotwords ? recognizer.createStream(hotwords) : recognizer.createStream();
stream.acceptWaveform({ samples: segment.samples, sampleRate: SAMPLE_RATE });
const decodeStarted = performance.now();
@@ -115,32 +114,6 @@ fs.writeFileSync(outputPath, JSON.stringify({
}, null, 2));
console.log(`Wrote ${outputPath}`);
-/**
- * Cuts one source into utterances with Silero VAD, as the live `VadSegmenter` does: 512-sample
- * windows, then a flush at the end.
- */
-function* segmentSource(samples, vadSettings, modelDir) {
- const windowSize = vadSettings.windowSize ?? 512;
- const vad = new sherpa.Vad({
- sileroVad: { model: path.join(modelDir, "silero_vad.onnx"), ...vadSettings, windowSize },
- sampleRate: SAMPLE_RATE,
- numThreads: 1,
- provider: "cpu"
- }, 60);
- const drain = function* () {
- while (!vad.isEmpty()) {
- yield vad.front();
- vad.pop();
- }
- };
- for (let i = 0; i + windowSize <= samples.length; i += windowSize) {
- vad.acceptWaveform(samples.subarray(i, i + windowSize));
- yield* drain();
- }
- vad.flush();
- yield* drain();
-}
-
function buildRecognizerConfig(settings, modelDir) {
const modelConfig = {
transducer: {
@@ -179,12 +152,6 @@ function applySetting(target, assignment) {
node[keys.at(-1)] = value;
}
-function loadSherpa() {
- const repoRoot = path.resolve(import.meta.dirname, "../../..");
- const require = createRequire(path.join(repoRoot, "apps/desktop/package.json"));
- return require("sherpa-onnx-node");
-}
-
function fail(message) {
console.error(message);
process.exit(1);
diff --git a/.claude/skills/working-with-transcription/sherpa.mjs b/.claude/skills/working-with-transcription/sherpa.mjs
new file mode 100644
index 0000000000..36c124d07e
--- /dev/null
+++ b/.claude/skills/working-with-transcription/sherpa.mjs
@@ -0,0 +1,39 @@
+// Shared by replay.mjs and diarize.mjs: loading sherpa-onnx-node and cutting audio as the app does.
+
+import { createRequire } from "node:module";
+import path from "node:path";
+
+export const SAMPLE_RATE = 16000;
+
+/** The desktop app's sherpa-onnx-node, which runs in plain Node too. */
+export function loadSherpa() {
+ const repoRoot = path.resolve(import.meta.dirname, "../../..");
+ const require = createRequire(path.join(repoRoot, "apps/desktop/package.json"));
+ return require("sherpa-onnx-node");
+}
+
+/**
+ * Cuts one source into utterances with Silero VAD, as the live `VadSegmenter` does: 512-sample
+ * windows, then a flush at the end.
+ */
+export function* segmentSource(sherpa, samples, vadSettings, modelDir) {
+ const windowSize = vadSettings.windowSize ?? 512;
+ const vad = new sherpa.Vad({
+ sileroVad: { model: path.join(modelDir, "silero_vad.onnx"), ...vadSettings, windowSize },
+ sampleRate: SAMPLE_RATE,
+ numThreads: 1,
+ provider: "cpu"
+ }, 60);
+ const drain = function* () {
+ while (!vad.isEmpty()) {
+ yield vad.front();
+ vad.pop();
+ }
+ };
+ for (let i = 0; i + windowSize <= samples.length; i += windowSize) {
+ vad.acceptWaveform(samples.subarray(i, i + windowSize));
+ yield* drain();
+ }
+ vad.flush();
+ yield* drain();
+}