Skip to content

Build your personal knowledge base with Trilium Notes

feat(desktop): download each transcription model as a resource of its own

Elian Dorand2fc4fd52ebe
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index ea1dc11b77..2b979dfa91 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -36,12 +36,12 @@ The main process opens the capture window, starts the transcription process and
|---|---|
| `apps/desktop/src/services/transcription/capture.ts` | Main process: `CaptureWindow`, the hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) |
| `…/index.ts` | Main process: IPC handlers, the state broadcast, `getRecordingDirectory()`; registers the model resource, starts a recording's process and capture window and connects them |
-| `…/model.ts` | `TRANSCRIPTION_MODEL`, the model as a downloadable resource (URLs pinned to a revision, sizes, SHA-256), and `LOCAL_MODEL_DIR` |
+| `…/model.ts` | Main process: the three models as downloadable resources (`RECOGNIZER_MODEL`, `VAD_MODEL`, `SPEAKER_MODEL`: URLs pinned to a revision, sizes, SHA-256), `LOCAL_MODEL_DIR`, and `getModelPaths()`, which resolves the `ModelPaths` a recording starts with |
| `…/worker_client.ts` | Main process: `TranscriptionWorker`, which forks the transcription process, relays its events and stops or kills it |
| `…/attachment_sink.ts` | Main process: the `transcript` attachment, rewritten after every utterance |
| `…/protocol.ts` | Both: the messages between the processes, `SAMPLE_RATE`, `Logger` |
| `…/transcription_worker.ts` | The transcription process's entry (bundled as `transcription_worker.cjs`): `PortCapture`, the session, the dev sinks and taps |
-| `…/engine.ts` | Transcription process: `loadSherpaEngine(modelDir)`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_SETTINGS`, `VAD_SETTINGS`), reported as `settings` |
+| `…/engine.ts` | Transcription process: `loadSherpaEngine(models)`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_SETTINGS`, `VAD_SETTINGS`), reported as `settings` |
| `…/session.ts` | Transcription process: `RecordingSession`: per-source segmentation, the decode queue, sinks, taps, pending sources |
| `…/audio_recorder.ts` | Transcription process: development recordings, `WavRecorder` and `TranscriptFileSink` |
@@ -54,9 +54,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri
## The model
-Users download the model from the app: the launcher calls `requireResources([TRANSCRIPTION_MODEL_RESOURCE_ID])` before it creates the meeting note, which shows the download dialog when the model is missing. The resource manager (`apps/server/src/services/resources.ts`) puts the int8 model, Silero VAD and TitaNet (~711 MB) in `<data dir>/resources/transcription.model/<version>/`, each file checked against the SHA-256 in `model.ts`. Changing a file there means a new `version`, so the new files download beside the old ones.
+Users download the models from the app: the launcher calls `requireResources(Object.values(TRANSCRIPTION_RESOURCES))` before it creates the meeting note, which shows the download dialog for those missing. They are three resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>/<version>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). Changing a file of one means a new `version` for it alone, so only that one downloads again. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.
-A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the download; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
+A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself (Silero when `silero_vad.onnx` is there, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.
- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).
diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjs
index 6e74c85644..db7721b9ac 100644
--- a/.claude/skills/working-with-transcription/worker-test.mjs
+++ b/.claude/skills/working-with-transcription/worker-test.mjs
@@ -114,7 +114,12 @@ async function runInElectron(config) {
log: { info: () => logLines++, error: (message) => console.log(`worker error: ${message}`) }
});
const startedAt = performance.now();
- await worker.start({ noteId: "worker-test", modelDirectory: MODEL_DIR, recordingDirectory: null });
+ const models = {
+ recognizer: MODEL_DIR,
+ vad: path.join(MODEL_DIR, "silero_vad.onnx"),
+ speaker: path.join(MODEL_DIR, "speaker", "nemo_en_titanet_small.onnx")
+ };
+ await worker.start({ noteId: "worker-test", models, recordingDirectory: null });
const readyMs = Math.round(performance.now() - startedAt);
const pagePath = path.join(os.tmpdir(), `trilium-worker-test-${process.pid}.html`);
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
index 13893940a0..fb7434a93b 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
@@ -100,7 +100,8 @@ describe("toggleRecording", () => {
it("asks for the model first, and creates no note when it is not downloaded", async () => {
await toggleRecording("idle");
- expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith([ "transcription.model" ]);
+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith(
+ [ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);
expect(transcription.start).toHaveBeenCalledOnce();
mocks.requireResources.mockResolvedValueOnce(false);
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
index 03ce4e43c5..44dd00127e 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
@@ -1,7 +1,7 @@
import "./RecordMeetingButton.css";
import {
- dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_MODEL_RESOURCE_ID, type TranscriptionState,
+ dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_RESOURCES, type TranscriptionState,
type TranscriptionStatus
} from "@triliumnext/commons";
import { useCallback, useEffect, useState } from "preact/hooks";
@@ -104,7 +104,7 @@ export async function toggleRecording(status: TranscriptionStatus) {
if (status !== "idle") {
return;
}
- if (!await requireResources([ TRANSCRIPTION_MODEL_RESOURCE_ID ])) {
+ if (!await requireResources(Object.values(TRANSCRIPTION_RESOURCES))) {
return;
}
diff --git a/apps/desktop/src/services/transcription/engine.spec.ts b/apps/desktop/src/services/transcription/engine.spec.ts
index ccdefa5d31..90cb7e121a 100644
--- a/apps/desktop/src/services/transcription/engine.spec.ts
+++ b/apps/desktop/src/services/transcription/engine.spec.ts
@@ -91,6 +91,11 @@ vi.mock("fs", () => ({
}));
const MODEL_DIR = "/models/parakeet";
+const MODELS = {
+ recognizer: MODEL_DIR,
+ vad: "/models/silero/silero_vad.onnx",
+ speaker: "/models/titanet/nemo_en_titanet_small.onnx"
+};
beforeEach(() => {
h.installed = [];
@@ -101,8 +106,8 @@ beforeEach(() => {
});
describe("loadSherpaEngine", () => {
- it("loads Parakeet and Silero from the model directory and reports their settings", async () => {
- const engine = await loadSherpaEngine(MODEL_DIR);
+ it("loads Parakeet and Silero from their paths and reports their settings", async () => {
+ const engine = await loadSherpaEngine(MODELS);
expect(h.recognizerConfig).toMatchObject({
featConfig: { sampleRate: 16000, featureDim: 80 },
@@ -134,7 +139,7 @@ describe("loadSherpaEngine", () => {
it("prefers the full-precision model, decoding it without a blank penalty", async () => {
h.installed = [ "encoder.onnx" ];
- const engine = await loadSherpaEngine(MODEL_DIR);
+ const engine = await loadSherpaEngine(MODELS);
expect(h.recognizerConfig).toMatchObject({
modelConfig: {
@@ -152,11 +157,11 @@ describe("loadSherpaEngine", () => {
});
it("segments a source in whole 512-sample windows, handing out what the VAD completes", async () => {
- const engine = await loadSherpaEngine(MODEL_DIR);
+ const engine = await loadSherpaEngine(MODELS);
const segmenter = engine.createSegmenter();
const vad = vads[0];
expect(h.vadConfig).toMatchObject({
- sileroVad: { model: path.join(MODEL_DIR, "silero_vad.onnx"), threshold: 0.5 },
+ sileroVad: { model: MODELS.vad, threshold: 0.5 },
sampleRate: 16000
});
@@ -176,9 +181,8 @@ describe("loadSherpaEngine", () => {
it("tells speakers apart with the speaker model, when it is installed", async () => {
h.installed = [ "nemo_en_titanet_small.onnx" ];
- const engine = await loadSherpaEngine(MODEL_DIR);
- expect(h.extractorConfig?.model)
- .toBe(path.join(MODEL_DIR, "speaker", "nemo_en_titanet_small.onnx"));
+ const engine = await loadSherpaEngine(MODELS);
+ expect(h.extractorConfig?.model).toBe(MODELS.speaker);
expect(engine.settings.speakers).toMatchObject({
model: "nemo_en_titanet_small.onnx", threshold: 0.5, minDurationSeconds: 1.5
});
@@ -190,5 +194,9 @@ describe("loadSherpaEngine", () => {
h.isEmbeddingReady = false;
expect(tracker?.identify(new Float32Array(100))).toBeNull();
+
+ const withoutSpeakers = await loadSherpaEngine({ ...MODELS, speaker: null });
+ expect(withoutSpeakers.createSpeakerTracker).toBeUndefined();
+ expect(withoutSpeakers.settings.speakers).toBeNull();
});
});
diff --git a/apps/desktop/src/services/transcription/engine.ts b/apps/desktop/src/services/transcription/engine.ts
index b478a8b2af..aa262081ff 100644
--- a/apps/desktop/src/services/transcription/engine.ts
+++ b/apps/desktop/src/services/transcription/engine.ts
@@ -2,7 +2,7 @@ import fs from "fs";
import path from "path";
import type { SpeakerEmbeddingExtractor, Vad } from "sherpa-onnx-node";
-import { SAMPLE_RATE } from "./protocol";
+import { type ModelPaths, SAMPLE_RATE } from "./protocol";
import { SpeakerClusterer, type SpeakerClustererOptions } from "./speakers";
/** Turns one utterance into text. */
@@ -44,9 +44,6 @@ export interface SpeechEngine {
settings: Record<string, unknown>;
}
-/** The speaker embedding model, optional: without it, utterances carry no speaker. */
-const SPEAKER_MODEL = path.join("speaker", "nemo_en_titanet_small.onnx");
-
/** Silero VAD consumes 512-sample windows at 16 kHz. */
const VAD_WINDOW_SIZE = 512;
@@ -85,14 +82,15 @@ const VAD_SETTINGS = {
};
/**
- * Loads the local engine from `modelDir`: Silero VAD for segmentation (`silero_vad.onnx`) and
- * Parakeet TDT 0.6B v3 for recognition, both through sherpa-onnx. Recognition uses the
- * full-precision model (`encoder.onnx` with `encoder.weights`) when the directory holds it, else
- * the int8 one (`encoder.int8.onnx`). Recognition runs on the libuv thread pool; the VAD and the
- * speaker embeddings run on the calling thread.
+ * Loads the local engine from `models`: Silero VAD for segmentation and Parakeet TDT 0.6B v3 for
+ * recognition, both through sherpa-onnx, and the speaker model when it is given and installed.
+ * Recognition uses the full-precision model (`encoder.onnx` with `encoder.weights`) when the
+ * recognizer directory holds it, else the int8 one (`encoder.int8.onnx`). Recognition runs on the
+ * libuv thread pool; the VAD and the speaker embeddings run on the calling thread.
*/
-export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine> {
- const speakerModel = path.join(modelDir, SPEAKER_MODEL);
+export async function loadSherpaEngine(models: ModelPaths): Promise<SpeechEngine> {
+ const modelDir = models.recognizer;
+ const speakerModel = models.speaker && fs.existsSync(models.speaker) ? models.speaker : null;
const precision: ModelPrecision = fs.existsSync(path.join(modelDir, "encoder.onnx"))
? "fp32"
: "int8";
@@ -114,7 +112,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>
...decodingSettings
});
- const speakerExtractor = fs.existsSync(speakerModel)
+ const speakerExtractor = speakerModel
? new sherpa.SpeakerEmbeddingExtractor({
model: speakerModel, numThreads: 1, provider: "cpu"
})
@@ -130,7 +128,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>
},
createSegmenter: () => new VadSegmenter(new sherpa.Vad({
sileroVad: {
- model: path.join(modelDir, "silero_vad.onnx"),
+ model: models.vad,
...VAD_SETTINGS
},
sampleRate: SAMPLE_RATE,
@@ -145,7 +143,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>
precision,
recognizer: { ...RECOGNIZER_SETTINGS, ...decodingSettings },
vad: VAD_SETTINGS,
- speakers: speakerExtractor
+ speakers: speakerModel
? { model: path.basename(speakerModel), ...SPEAKER_SETTINGS }
: null
}
diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.ts
index c38f6c0290..e8751f6539 100644
--- a/apps/desktop/src/services/transcription/index.spec.ts
+++ b/apps/desktop/src/services/transcription/index.spec.ts
@@ -1,4 +1,5 @@
import type { TranscriptionState } from "@triliumnext/commons";
+import path from "path";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { CaptureWindowOptions } from "./capture";
@@ -14,7 +15,7 @@ const h = vi.hoisted(() => ({
isProtectedSessionAvailable: false,
coreListeners: new Map<string, () => void>(),
isDev: false,
- isModelInstalled: true,
+ installedResources: [] as string[],
registerResource: vi.fn(),
log: { info: vi.fn(), error: vi.fn() },
listeners: null as WorkerListeners | null,
@@ -72,7 +73,7 @@ vi.mock("@triliumnext/server/src/services/data_dir.js", () => ({
vi.mock("@triliumnext/server/src/services/resources.js", () => ({
default: {
register: h.registerResource,
- getPath: (id: string) => h.isModelInstalled ? `/data/resources/${id}` : null
+ getPath: (id: string) => h.installedResources.includes(id) ? `/data/resources/${id}` : null
}
}));
vi.mock("./capture", () => ({
@@ -128,7 +129,9 @@ beforeEach(() => {
h.isProtectedSessionAvailable = false;
h.broadcast.length = 0;
h.isDev = false;
- h.isModelInstalled = true;
+ h.installedResources = [
+ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet"
+ ];
h.listeners = null;
h.worker.start.mockResolvedValue({ type: "ready", startedAt: STARTED_AT, settings: {} });
h.capture.start.mockResolvedValue([ "microphone", "system" ]);
@@ -179,7 +182,11 @@ describe("transcription IPC", () => {
expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({
noteId: "meeting",
- modelDirectory: "/data/resources/transcription.model",
+ models: {
+ recognizer: "/data/resources/transcription.parakeet",
+ vad: path.join("/data/resources/transcription.silero-vad", "silero_vad.onnx"),
+ speaker: path.join("/data/resources/transcription.titanet", "nemo_en_titanet_small.onnx")
+ },
recordingDirectory: null
});
expect(h.sink.open).toHaveBeenCalledExactlyOnceWith(STARTED_AT);
@@ -322,13 +329,23 @@ describe("transcription IPC", () => {
expect(await invoke("transcription-start", "deleted")).toBe("transcription.note-not-found");
expect(await invoke("transcription-start", { noteId: "meeting" }))
.toBe("transcription.note-not-found");
- h.isModelInstalled = false;
+ h.installedResources = [ "transcription.parakeet", "transcription.titanet" ];
expect(await invoke("transcription-start", "meeting"))
.toBe("transcription.model-not-downloaded");
expect(h.listeners).toBeNull();
expect(h.broadcast).toEqual([]);
});
+ it("registers the three models, and records without the speaker model", async () => {
+ const invoke = await loadService();
+ expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id))
+ .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);
+
+ h.installedResources = [ "transcription.parakeet", "transcription.silero-vad" ];
+ expect(await invoke("transcription-start", "meeting")).toBe("");
+ expect(h.worker.start.mock.calls[0][0].models.speaker).toBeNull();
+ });
+
it("explains why it could not start, ends the process and returns to idle", async () => {
const invoke = await loadService();
diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.ts
index 7f3d881579..1418ad5048 100644
--- a/apps/desktop/src/services/transcription/index.ts
+++ b/apps/desktop/src/services/transcription/index.ts
@@ -8,7 +8,7 @@ import path from "path";
import { AttachmentSink } from "./attachment_sink";
import { CaptureWindow } from "./capture";
-import { TRANSCRIPTION_MODEL } from "./model";
+import { getModelPaths, TRANSCRIPTION_MODELS } from "./model";
import { TranscriptionWorker } from "./worker_client";
/** Where development builds keep the audio of each recording, one directory per recording. */
@@ -39,7 +39,7 @@ const unsavedSinks = new Set<AttachmentSink>();
let quitting: "no" | "waiting" | "done" = "no";
export function setupTranscriptionHandlers() {
- resources.register(TRANSCRIPTION_MODEL);
+ resources.register(...TRANSCRIPTION_MODELS);
ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));
ipcMain.handle("transcription-stop", () => stopRecording());
ipcMain.handle("transcription-get-state", () => getState());
@@ -62,8 +62,8 @@ export async function startRecording(noteId: unknown): Promise<string> {
if (typeof noteId !== "string" || !becca.getNote(noteId)) {
return t("transcription.note-not-found");
}
- const modelDirectory = resources.getPath(TRANSCRIPTION_MODEL.id);
- if (!modelDirectory) {
+ const models = getModelPaths();
+ if (!models) {
return t("transcription.model-not-downloaded");
}
@@ -84,7 +84,7 @@ export async function startRecording(noteId: unknown): Promise<string> {
try {
const { startedAt } = await recording.worker.start({
noteId,
- modelDirectory,
+ models,
recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null
});
sink.open(startedAt);
diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.ts
index 3ae1926a7b..b03767b6b3 100644
--- a/apps/desktop/src/services/transcription/model.ts
+++ b/apps/desktop/src/services/transcription/model.ts
@@ -1,25 +1,30 @@
-import { TRANSCRIPTION_MODEL_RESOURCE_ID } from "@triliumnext/commons";
+import { TRANSCRIPTION_RESOURCES } from "@triliumnext/commons";
import dataDirs from "@triliumnext/server/src/services/data_dir.js";
-import type { ResourceDefinition } from "@triliumnext/server/src/services/resources.js";
+import resources, { type ResourceDefinition } from "@triliumnext/server/src/services/resources.js";
import fs from "fs";
import { t } from "i18next";
import path from "path";
+import type { ModelPaths } from "./protocol";
+
/**
- * A model directory installed by hand, which takes precedence over the downloaded one: it can
+ * A model directory installed by hand, which takes precedence over the downloaded models: it can
* hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.
*/
export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");
+const VAD_FILE = "silero_vad.onnx";
+const SPEAKER_FILE = "nemo_en_titanet_small.onnx";
+
const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";
const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";
-/** The files `loadSherpaEngine()` reads from its model directory, with the int8 Parakeet model. */
-export const TRANSCRIPTION_MODEL: ResourceDefinition = {
- id: TRANSCRIPTION_MODEL_RESOURCE_ID,
- version: "parakeet-v3-int8-1",
- title: () => t("transcription.model-title"),
- description: () => t("transcription.model-description"),
+/** Parakeet TDT 0.6B v3, int8: the files `loadSherpaEngine()` reads from its recognizer directory. */
+export const RECOGNIZER_MODEL: ResourceDefinition = {
+ id: TRANSCRIPTION_RESOURCES.recognizer,
+ version: "v3-int8-2bda32ec",
+ title: () => t("transcription.recognizer-title"),
+ description: () => t("transcription.recognizer-description"),
findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,
files: [
{
@@ -45,18 +50,57 @@ export const TRANSCRIPTION_MODEL: ResourceDefinition = {
url: `${PARAKEET}/tokens.txt`,
size: 93939,
sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"
- },
- {
- path: "silero_vad.onnx",
- url: `${SHERPA_RELEASES}/asr-models/silero_vad.onnx`,
- size: 643854,
- sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"
- },
- {
- path: "speaker/nemo_en_titanet_small.onnx",
- url: `${SHERPA_RELEASES}/speaker-recongition-models/nemo_en_titanet_small.onnx`,
- size: 40257283,
- sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"
}
]
};
+
+export const VAD_MODEL: ResourceDefinition = {
+ id: TRANSCRIPTION_RESOURCES.vad,
+ version: "1",
+ title: () => t("transcription.vad-title"),
+ description: () => t("transcription.vad-description"),
+ findLocalCopy: () => fs.existsSync(path.join(LOCAL_MODEL_DIR, VAD_FILE)) ? LOCAL_MODEL_DIR : null,
+ files: [ {
+ path: VAD_FILE,
+ url: `${SHERPA_RELEASES}/asr-models/${VAD_FILE}`,
+ size: 643854,
+ sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"
+ } ]
+};
+
+export const SPEAKER_MODEL: ResourceDefinition = {
+ id: TRANSCRIPTION_RESOURCES.speaker,
+ version: "1",
+ title: () => t("transcription.speaker-title"),
+ description: () => t("transcription.speaker-description"),
+ findLocalCopy: () => {
+ const directory = path.join(LOCAL_MODEL_DIR, "speaker");
+ return fs.existsSync(path.join(directory, SPEAKER_FILE)) ? directory : null;
+ },
+ files: [ {
+ path: SPEAKER_FILE,
+ url: `${SHERPA_RELEASES}/speaker-recongition-models/${SPEAKER_FILE}`,
+ size: 40257283,
+ sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"
+ } ]
+};
+
+export const TRANSCRIPTION_MODELS = [ RECOGNIZER_MODEL, VAD_MODEL, SPEAKER_MODEL ];
+
+/**
+ * Returns where the installed models are, or `null` while the recognizer or the VAD is missing.
+ * Without the speaker model, the transcript labels no speakers.
+ */
+export function getModelPaths(): ModelPaths | null {
+ const recognizer = resources.getPath(RECOGNIZER_MODEL.id);
+ const vad = resources.getPath(VAD_MODEL.id);
+ const speaker = resources.getPath(SPEAKER_MODEL.id);
+ if (!recognizer || !vad) {
+ return null;
+ }
+ return {
+ recognizer,
+ vad: path.join(vad, VAD_FILE),
+ speaker: speaker ? path.join(speaker, SPEAKER_FILE) : null
+ };
+}
diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.ts
index 653bf9343c..5023d4c25a 100644
--- a/apps/desktop/src/services/transcription/protocol.ts
+++ b/apps/desktop/src/services/transcription/protocol.ts
@@ -9,12 +9,22 @@ export interface Logger {
error(message: string): void;
}
+/** Where the engine's models are. */
+export interface ModelPaths {
+ /** The directory holding Parakeet: `tokens.txt` and the encoder, decoder and joiner. */
+ recognizer: string;
+ /** Silero VAD's `.onnx` file. */
+ vad: string;
+ /** The speaker embedding model's `.onnx` file, or `null` to label no speakers. */
+ speaker: string | null;
+}
+
/** What the main process sends the transcription process, in this order. */
export type WorkerRequest =
| {
type: "start";
noteId: string;
- modelDirectory: string;
+ models: ModelPaths;
/** Where development builds keep the audio and the transcript as produced, else `null`. */
recordingDirectory: string | null;
}
diff --git a/apps/desktop/src/services/transcription/transcription_worker.spec.ts b/apps/desktop/src/services/transcription/transcription_worker.spec.ts
index d8e077a8fd..03d6dc704b 100644
--- a/apps/desktop/src/services/transcription/transcription_worker.spec.ts
+++ b/apps/desktop/src/services/transcription/transcription_worker.spec.ts
@@ -69,8 +69,10 @@ function capturePort() {
return Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });
}
+const MODELS = { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null };
+
const START: WorkerRequest = {
- type: "start", noteId: "meeting", modelDirectory: "/models", recordingDirectory: null
+ type: "start", noteId: "meeting", models: MODELS, recordingDirectory: null
};
beforeEach(() => {
@@ -91,7 +93,7 @@ describe("transcription process", () => {
parent.send(START);
const ready = await parent.next("ready");
expect(ready).toMatchObject({ type: "ready", settings: engine.settings });
- expect(h.loadEngine).toHaveBeenCalledExactlyOnceWith("/models");
+ expect(h.loadEngine).toHaveBeenCalledExactlyOnceWith(MODELS);
expect(h.fileSinks).toEqual([]);
const port = capturePort();
diff --git a/apps/desktop/src/services/transcription/transcription_worker.ts b/apps/desktop/src/services/transcription/transcription_worker.ts
index a646115632..f40c5cfba3 100644
--- a/apps/desktop/src/services/transcription/transcription_worker.ts
+++ b/apps/desktop/src/services/transcription/transcription_worker.ts
@@ -42,8 +42,8 @@ process.parentPort.on("message", ({ data, ports }) => {
}
});
-async function start({ noteId, modelDirectory, recordingDirectory }: StartRequest) {
- const engine = await loadSherpaEngine(modelDirectory);
+async function start({ noteId, models, recordingDirectory }: StartRequest) {
+ const engine = await loadSherpaEngine(models);
const sinks: TranscriptSink[] = [
{ write: (utterance) => post({ type: "utterance", utterance }) }
];
diff --git a/apps/desktop/src/services/transcription/worker_client.spec.ts b/apps/desktop/src/services/transcription/worker_client.spec.ts
index 948cac1dd8..31f780caee 100644
--- a/apps/desktop/src/services/transcription/worker_client.spec.ts
+++ b/apps/desktop/src/services/transcription/worker_client.spec.ts
@@ -53,7 +53,11 @@ afterEach(() => {
vi.useRealTimers();
});
-const REQUEST = { noteId: "meeting", modelDirectory: "/models", recordingDirectory: null };
+const REQUEST = {
+ noteId: "meeting",
+ models: { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null },
+ recordingDirectory: null
+};
describe("TranscriptionWorker", () => {
it("starts the bundled worker, or the sources when there is no bundle", () => {
diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.json
index c0dcc4714f..68bcf778b2 100644
--- a/apps/server/src/assets/translations/en/server.json
+++ b/apps/server/src/assets/translations/en/server.json
@@ -571,9 +571,13 @@
"save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.",
"capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.",
"attachment-title": "Transcript",
- "model-title": "Speech recognition model",
- "model-description": "Transcribes meetings on this device: Parakeet TDT 0.6B v3 for speech recognition, Silero VAD to find speech and TitaNet to tell speakers apart. Nothing you record leaves your computer.",
- "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it."
+ "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it.",
+ "recognizer-title": "Parakeet TDT 0.6B v3",
+ "recognizer-description": "NVIDIA's speech recognition model, which turns what is said into text in 25 European languages. It runs on this device: nothing you record leaves your computer.",
+ "vad-title": "Silero VAD",
+ "vad-description": "Finds the stretches of speech in the audio, so that only those are transcribed.",
+ "speaker-title": "TitaNet small",
+ "speaker-description": "Tells apart the other people in the meeting, so that the transcript shows who said what."
},
"resources": {
"unknown": "There is no resource named {{id}}.",
diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.ts
index e89ee68c1f..a7da875aba 100644
--- a/packages/commons/src/lib/transcript.ts
+++ b/packages/commons/src/lib/transcript.ts
@@ -40,8 +40,12 @@ export const TRANSCRIPT_MIME = "application/json";
/** The longest speaker name a transcript accepts. */
export const MAX_SPEAKER_NAME_LENGTH = 100;
-/** The resource holding the models transcription runs on, `TRANSCRIPTION_MODEL` on desktop. */
-export const TRANSCRIPTION_MODEL_RESOURCE_ID = "transcription.model";
+/** The resources holding the models transcription runs on, defined in the desktop app. */
+export const TRANSCRIPTION_RESOURCES = {
+ recognizer: "transcription.parakeet",
+ vad: "transcription.silero-vad",
+ speaker: "transcription.titanet"
+} as const;
/** Reads a transcript attachment's content, or returns `null` for content of another shape. */
export function parseTranscript(content: string | null | undefined): Transcript | null {