Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(desktop): download each transcription model as a resource of its own
Elian Dorand2fc4fd52ebe
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex ea1dc11b77..2b979dfa91 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -36,12 +36,12 @@ The main process opens the capture window, starts the transcription process and|---|---|| `apps/desktop/src/services/transcription/capture.ts` | Main process: `CaptureWindow`, the hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) || `…/index.ts` | Main process: IPC handlers, the state broadcast, `getRecordingDirectory()`; registers the model resource, starts a recording's process and capture window and connects them |-| `…/model.ts` | `TRANSCRIPTION_MODEL`, the model as a downloadable resource (URLs pinned to a revision, sizes, SHA-256), and `LOCAL_MODEL_DIR` |+| `…/model.ts` | Main process: the three models as downloadable resources (`RECOGNIZER_MODEL`, `VAD_MODEL`, `SPEAKER_MODEL`: URLs pinned to a revision, sizes, SHA-256), `LOCAL_MODEL_DIR`, and `getModelPaths()`, which resolves the `ModelPaths` a recording starts with || `…/worker_client.ts` | Main process: `TranscriptionWorker`, which forks the transcription process, relays its events and stops or kills it || `…/attachment_sink.ts` | Main process: the `transcript` attachment, rewritten after every utterance || `…/protocol.ts` | Both: the messages between the processes, `SAMPLE_RATE`, `Logger` || `…/transcription_worker.ts` | The transcription process's entry (bundled as `transcription_worker.cjs`): `PortCapture`, the session, the dev sinks and taps |-| `…/engine.ts` | Transcription process: `loadSherpaEngine(modelDir)`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_SETTINGS`, `VAD_SETTINGS`), reported as `settings` |+| `…/engine.ts` | Transcription process: `loadSherpaEngine(models)`: recognizer and VAD settings (`RECOGNIZER_SETTINGS`, `DECODING_SETTINGS`, `VAD_SETTINGS`), reported as `settings` || `…/session.ts` | Transcription process: `RecordingSession`: per-source segmentation, the decode queue, sinks, taps, pending sources || `…/audio_recorder.ts` | Transcription process: development recordings, `WavRecorder` and `TranscriptFileSink` |@@ -54,9 +54,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri## The model-Users download the model from the app: the launcher calls `requireResources([TRANSCRIPTION_MODEL_RESOURCE_ID])` before it creates the meeting note, which shows the download dialog when the model is missing. The resource manager (`apps/server/src/services/resources.ts`) puts the int8 model, Silero VAD and TitaNet (~711 MB) in `<data dir>/resources/transcription.model/<version>/`, each file checked against the SHA-256 in `model.ts`. Changing a file there means a new `version`, so the new files download beside the old ones.+Users download the models from the app: the launcher calls `requireResources(Object.values(TRANSCRIPTION_RESOURCES))` before it creates the meeting note, which shows the download dialog for those missing. They are three resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>/<version>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). Changing a file of one means a new `version` for it alone, so only that one downloads again. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.-A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the download; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").+A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself (Silero when `silero_vad.onnx` is there, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjsindex 6e74c85644..db7721b9ac 100644--- a/.claude/skills/working-with-transcription/worker-test.mjs+++ b/.claude/skills/working-with-transcription/worker-test.mjs@@ -114,7 +114,12 @@ async function runInElectron(config) {log: { info: () => logLines++, error: (message) => console.log(`worker error: ${message}`) }});const startedAt = performance.now();- await worker.start({ noteId: "worker-test", modelDirectory: MODEL_DIR, recordingDirectory: null });+ const models = {+ recognizer: MODEL_DIR,+ vad: path.join(MODEL_DIR, "silero_vad.onnx"),+ speaker: path.join(MODEL_DIR, "speaker", "nemo_en_titanet_small.onnx")+ };+ await worker.start({ noteId: "worker-test", models, recordingDirectory: null });const readyMs = Math.round(performance.now() - startedAt);const pagePath = path.join(os.tmpdir(), `trilium-worker-test-${process.pid}.html`);diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsxindex 13893940a0..fb7434a93b 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx@@ -100,7 +100,8 @@ describe("toggleRecording", () => {it("asks for the model first, and creates no note when it is not downloaded", async () => {await toggleRecording("idle");- expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith([ "transcription.model" ]);+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith(+ [ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);expect(transcription.start).toHaveBeenCalledOnce();mocks.requireResources.mockResolvedValueOnce(false);diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsxindex 03ce4e43c5..44dd00127e 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx@@ -1,7 +1,7 @@import "./RecordMeetingButton.css";import {- dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_MODEL_RESOURCE_ID, type TranscriptionState,+ dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_RESOURCES, type TranscriptionState,type TranscriptionStatus} from "@triliumnext/commons";import { useCallback, useEffect, useState } from "preact/hooks";@@ -104,7 +104,7 @@ export async function toggleRecording(status: TranscriptionStatus) {if (status !== "idle") {return;}- if (!await requireResources([ TRANSCRIPTION_MODEL_RESOURCE_ID ])) {+ if (!await requireResources(Object.values(TRANSCRIPTION_RESOURCES))) {return;}diff --git a/apps/desktop/src/services/transcription/engine.spec.ts b/apps/desktop/src/services/transcription/engine.spec.tsindex ccdefa5d31..90cb7e121a 100644--- a/apps/desktop/src/services/transcription/engine.spec.ts+++ b/apps/desktop/src/services/transcription/engine.spec.ts@@ -91,6 +91,11 @@ vi.mock("fs", () => ({}));const MODEL_DIR = "/models/parakeet";+const MODELS = {+ recognizer: MODEL_DIR,+ vad: "/models/silero/silero_vad.onnx",+ speaker: "/models/titanet/nemo_en_titanet_small.onnx"+};beforeEach(() => {h.installed = [];@@ -101,8 +106,8 @@ beforeEach(() => {});describe("loadSherpaEngine", () => {- it("loads Parakeet and Silero from the model directory and reports their settings", async () => {- const engine = await loadSherpaEngine(MODEL_DIR);+ it("loads Parakeet and Silero from their paths and reports their settings", async () => {+ const engine = await loadSherpaEngine(MODELS);expect(h.recognizerConfig).toMatchObject({featConfig: { sampleRate: 16000, featureDim: 80 },@@ -134,7 +139,7 @@ describe("loadSherpaEngine", () => {it("prefers the full-precision model, decoding it without a blank penalty", async () => {h.installed = [ "encoder.onnx" ];- const engine = await loadSherpaEngine(MODEL_DIR);+ const engine = await loadSherpaEngine(MODELS);expect(h.recognizerConfig).toMatchObject({modelConfig: {@@ -152,11 +157,11 @@ describe("loadSherpaEngine", () => {});it("segments a source in whole 512-sample windows, handing out what the VAD completes", async () => {- const engine = await loadSherpaEngine(MODEL_DIR);+ const engine = await loadSherpaEngine(MODELS);const segmenter = engine.createSegmenter();const vad = vads[0];expect(h.vadConfig).toMatchObject({- sileroVad: { model: path.join(MODEL_DIR, "silero_vad.onnx"), threshold: 0.5 },+ sileroVad: { model: MODELS.vad, threshold: 0.5 },sampleRate: 16000});@@ -176,9 +181,8 @@ describe("loadSherpaEngine", () => {it("tells speakers apart with the speaker model, when it is installed", async () => {h.installed = [ "nemo_en_titanet_small.onnx" ];- const engine = await loadSherpaEngine(MODEL_DIR);- expect(h.extractorConfig?.model)- .toBe(path.join(MODEL_DIR, "speaker", "nemo_en_titanet_small.onnx"));+ const engine = await loadSherpaEngine(MODELS);+ expect(h.extractorConfig?.model).toBe(MODELS.speaker);expect(engine.settings.speakers).toMatchObject({model: "nemo_en_titanet_small.onnx", threshold: 0.5, minDurationSeconds: 1.5});@@ -190,5 +194,9 @@ describe("loadSherpaEngine", () => {h.isEmbeddingReady = false;expect(tracker?.identify(new Float32Array(100))).toBeNull();++ const withoutSpeakers = await loadSherpaEngine({ ...MODELS, speaker: null });+ expect(withoutSpeakers.createSpeakerTracker).toBeUndefined();+ expect(withoutSpeakers.settings.speakers).toBeNull();});});diff --git a/apps/desktop/src/services/transcription/engine.ts b/apps/desktop/src/services/transcription/engine.tsindex b478a8b2af..aa262081ff 100644--- a/apps/desktop/src/services/transcription/engine.ts+++ b/apps/desktop/src/services/transcription/engine.ts@@ -2,7 +2,7 @@ import fs from "fs";import path from "path";import type { SpeakerEmbeddingExtractor, Vad } from "sherpa-onnx-node";-import { SAMPLE_RATE } from "./protocol";+import { type ModelPaths, SAMPLE_RATE } from "./protocol";import { SpeakerClusterer, type SpeakerClustererOptions } from "./speakers";/** Turns one utterance into text. */@@ -44,9 +44,6 @@ export interface SpeechEngine {settings: Record<string, unknown>;}-/** The speaker embedding model, optional: without it, utterances carry no speaker. */-const SPEAKER_MODEL = path.join("speaker", "nemo_en_titanet_small.onnx");-/** Silero VAD consumes 512-sample windows at 16 kHz. */const VAD_WINDOW_SIZE = 512;@@ -85,14 +82,15 @@ const VAD_SETTINGS = {};/**- * Loads the local engine from `modelDir`: Silero VAD for segmentation (`silero_vad.onnx`) and- * Parakeet TDT 0.6B v3 for recognition, both through sherpa-onnx. Recognition uses the- * full-precision model (`encoder.onnx` with `encoder.weights`) when the directory holds it, else- * the int8 one (`encoder.int8.onnx`). Recognition runs on the libuv thread pool; the VAD and the- * speaker embeddings run on the calling thread.+ * Loads the local engine from `models`: Silero VAD for segmentation and Parakeet TDT 0.6B v3 for+ * recognition, both through sherpa-onnx, and the speaker model when it is given and installed.+ * Recognition uses the full-precision model (`encoder.onnx` with `encoder.weights`) when the+ * recognizer directory holds it, else the int8 one (`encoder.int8.onnx`). Recognition runs on the+ * libuv thread pool; the VAD and the speaker embeddings run on the calling thread.*/-export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine> {- const speakerModel = path.join(modelDir, SPEAKER_MODEL);+export async function loadSherpaEngine(models: ModelPaths): Promise<SpeechEngine> {+ const modelDir = models.recognizer;+ const speakerModel = models.speaker && fs.existsSync(models.speaker) ? models.speaker : null;const precision: ModelPrecision = fs.existsSync(path.join(modelDir, "encoder.onnx"))? "fp32": "int8";@@ -114,7 +112,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>...decodingSettings});- const speakerExtractor = fs.existsSync(speakerModel)+ const speakerExtractor = speakerModel? new sherpa.SpeakerEmbeddingExtractor({model: speakerModel, numThreads: 1, provider: "cpu"})@@ -130,7 +128,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>},createSegmenter: () => new VadSegmenter(new sherpa.Vad({sileroVad: {- model: path.join(modelDir, "silero_vad.onnx"),+ model: models.vad,...VAD_SETTINGS},sampleRate: SAMPLE_RATE,@@ -145,7 +143,7 @@ export async function loadSherpaEngine(modelDir: string): Promise<SpeechEngine>precision,recognizer: { ...RECOGNIZER_SETTINGS, ...decodingSettings },vad: VAD_SETTINGS,- speakers: speakerExtractor+ speakers: speakerModel? { model: path.basename(speakerModel), ...SPEAKER_SETTINGS }: null}diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.tsindex c38f6c0290..e8751f6539 100644--- a/apps/desktop/src/services/transcription/index.spec.ts+++ b/apps/desktop/src/services/transcription/index.spec.ts@@ -1,4 +1,5 @@import type { TranscriptionState } from "@triliumnext/commons";+import path from "path";import { beforeEach, describe, expect, it, vi } from "vitest";import type { CaptureWindowOptions } from "./capture";@@ -14,7 +15,7 @@ const h = vi.hoisted(() => ({isProtectedSessionAvailable: false,coreListeners: new Map<string, () => void>(),isDev: false,- isModelInstalled: true,+ installedResources: [] as string[],registerResource: vi.fn(),log: { info: vi.fn(), error: vi.fn() },listeners: null as WorkerListeners | null,@@ -72,7 +73,7 @@ vi.mock("@triliumnext/server/src/services/data_dir.js", () => ({vi.mock("@triliumnext/server/src/services/resources.js", () => ({default: {register: h.registerResource,- getPath: (id: string) => h.isModelInstalled ? `/data/resources/${id}` : null+ getPath: (id: string) => h.installedResources.includes(id) ? `/data/resources/${id}` : null}}));vi.mock("./capture", () => ({@@ -128,7 +129,9 @@ beforeEach(() => {h.isProtectedSessionAvailable = false;h.broadcast.length = 0;h.isDev = false;- h.isModelInstalled = true;+ h.installedResources = [+ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet"+ ];h.listeners = null;h.worker.start.mockResolvedValue({ type: "ready", startedAt: STARTED_AT, settings: {} });h.capture.start.mockResolvedValue([ "microphone", "system" ]);@@ -179,7 +182,11 @@ describe("transcription IPC", () => {expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({noteId: "meeting",- modelDirectory: "/data/resources/transcription.model",+ models: {+ recognizer: "/data/resources/transcription.parakeet",+ vad: path.join("/data/resources/transcription.silero-vad", "silero_vad.onnx"),+ speaker: path.join("/data/resources/transcription.titanet", "nemo_en_titanet_small.onnx")+ },recordingDirectory: null});expect(h.sink.open).toHaveBeenCalledExactlyOnceWith(STARTED_AT);@@ -322,13 +329,23 @@ describe("transcription IPC", () => {expect(await invoke("transcription-start", "deleted")).toBe("transcription.note-not-found");expect(await invoke("transcription-start", { noteId: "meeting" })).toBe("transcription.note-not-found");- h.isModelInstalled = false;+ h.installedResources = [ "transcription.parakeet", "transcription.titanet" ];expect(await invoke("transcription-start", "meeting")).toBe("transcription.model-not-downloaded");expect(h.listeners).toBeNull();expect(h.broadcast).toEqual([]);});+ it("registers the three models, and records without the speaker model", async () => {+ const invoke = await loadService();+ expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id))+ .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);++ h.installedResources = [ "transcription.parakeet", "transcription.silero-vad" ];+ expect(await invoke("transcription-start", "meeting")).toBe("");+ expect(h.worker.start.mock.calls[0][0].models.speaker).toBeNull();+ });+it("explains why it could not start, ends the process and returns to idle", async () => {const invoke = await loadService();diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.tsindex 7f3d881579..1418ad5048 100644--- a/apps/desktop/src/services/transcription/index.ts+++ b/apps/desktop/src/services/transcription/index.ts@@ -8,7 +8,7 @@ import path from "path";import { AttachmentSink } from "./attachment_sink";import { CaptureWindow } from "./capture";-import { TRANSCRIPTION_MODEL } from "./model";+import { getModelPaths, TRANSCRIPTION_MODELS } from "./model";import { TranscriptionWorker } from "./worker_client";/** Where development builds keep the audio of each recording, one directory per recording. */@@ -39,7 +39,7 @@ const unsavedSinks = new Set<AttachmentSink>();let quitting: "no" | "waiting" | "done" = "no";export function setupTranscriptionHandlers() {- resources.register(TRANSCRIPTION_MODEL);+ resources.register(...TRANSCRIPTION_MODELS);ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));ipcMain.handle("transcription-stop", () => stopRecording());ipcMain.handle("transcription-get-state", () => getState());@@ -62,8 +62,8 @@ export async function startRecording(noteId: unknown): Promise<string> {if (typeof noteId !== "string" || !becca.getNote(noteId)) {return t("transcription.note-not-found");}- const modelDirectory = resources.getPath(TRANSCRIPTION_MODEL.id);- if (!modelDirectory) {+ const models = getModelPaths();+ if (!models) {return t("transcription.model-not-downloaded");}@@ -84,7 +84,7 @@ export async function startRecording(noteId: unknown): Promise<string> {try {const { startedAt } = await recording.worker.start({noteId,- modelDirectory,+ models,recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null});sink.open(startedAt);diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.tsindex 3ae1926a7b..b03767b6b3 100644--- a/apps/desktop/src/services/transcription/model.ts+++ b/apps/desktop/src/services/transcription/model.ts@@ -1,25 +1,30 @@-import { TRANSCRIPTION_MODEL_RESOURCE_ID } from "@triliumnext/commons";+import { TRANSCRIPTION_RESOURCES } from "@triliumnext/commons";import dataDirs from "@triliumnext/server/src/services/data_dir.js";-import type { ResourceDefinition } from "@triliumnext/server/src/services/resources.js";+import resources, { type ResourceDefinition } from "@triliumnext/server/src/services/resources.js";import fs from "fs";import { t } from "i18next";import path from "path";+import type { ModelPaths } from "./protocol";+/**- * A model directory installed by hand, which takes precedence over the downloaded one: it can+ * A model directory installed by hand, which takes precedence over the downloaded models: it can* hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.*/export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");+const VAD_FILE = "silero_vad.onnx";+const SPEAKER_FILE = "nemo_en_titanet_small.onnx";+const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";-/** The files `loadSherpaEngine()` reads from its model directory, with the int8 Parakeet model. */-export const TRANSCRIPTION_MODEL: ResourceDefinition = {- id: TRANSCRIPTION_MODEL_RESOURCE_ID,- version: "parakeet-v3-int8-1",- title: () => t("transcription.model-title"),- description: () => t("transcription.model-description"),+/** Parakeet TDT 0.6B v3, int8: the files `loadSherpaEngine()` reads from its recognizer directory. */+export const RECOGNIZER_MODEL: ResourceDefinition = {+ id: TRANSCRIPTION_RESOURCES.recognizer,+ version: "v3-int8-2bda32ec",+ title: () => t("transcription.recognizer-title"),+ description: () => t("transcription.recognizer-description"),findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,files: [{@@ -45,18 +50,57 @@ export const TRANSCRIPTION_MODEL: ResourceDefinition = {url: `${PARAKEET}/tokens.txt`,size: 93939,sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"- },- {- path: "silero_vad.onnx",- url: `${SHERPA_RELEASES}/asr-models/silero_vad.onnx`,- size: 643854,- sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"- },- {- path: "speaker/nemo_en_titanet_small.onnx",- url: `${SHERPA_RELEASES}/speaker-recongition-models/nemo_en_titanet_small.onnx`,- size: 40257283,- sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"}]};++export const VAD_MODEL: ResourceDefinition = {+ id: TRANSCRIPTION_RESOURCES.vad,+ version: "1",+ title: () => t("transcription.vad-title"),+ description: () => t("transcription.vad-description"),+ findLocalCopy: () => fs.existsSync(path.join(LOCAL_MODEL_DIR, VAD_FILE)) ? LOCAL_MODEL_DIR : null,+ files: [ {+ path: VAD_FILE,+ url: `${SHERPA_RELEASES}/asr-models/${VAD_FILE}`,+ size: 643854,+ sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"+ } ]+};++export const SPEAKER_MODEL: ResourceDefinition = {+ id: TRANSCRIPTION_RESOURCES.speaker,+ version: "1",+ title: () => t("transcription.speaker-title"),+ description: () => t("transcription.speaker-description"),+ findLocalCopy: () => {+ const directory = path.join(LOCAL_MODEL_DIR, "speaker");+ return fs.existsSync(path.join(directory, SPEAKER_FILE)) ? directory : null;+ },+ files: [ {+ path: SPEAKER_FILE,+ url: `${SHERPA_RELEASES}/speaker-recongition-models/${SPEAKER_FILE}`,+ size: 40257283,+ sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"+ } ]+};++export const TRANSCRIPTION_MODELS = [ RECOGNIZER_MODEL, VAD_MODEL, SPEAKER_MODEL ];++/**+ * Returns where the installed models are, or `null` while the recognizer or the VAD is missing.+ * Without the speaker model, the transcript labels no speakers.+ */+export function getModelPaths(): ModelPaths | null {+ const recognizer = resources.getPath(RECOGNIZER_MODEL.id);+ const vad = resources.getPath(VAD_MODEL.id);+ const speaker = resources.getPath(SPEAKER_MODEL.id);+ if (!recognizer || !vad) {+ return null;+ }+ return {+ recognizer,+ vad: path.join(vad, VAD_FILE),+ speaker: speaker ? path.join(speaker, SPEAKER_FILE) : null+ };+}diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.tsindex 653bf9343c..5023d4c25a 100644--- a/apps/desktop/src/services/transcription/protocol.ts+++ b/apps/desktop/src/services/transcription/protocol.ts@@ -9,12 +9,22 @@ export interface Logger {error(message: string): void;}+/** Where the engine's models are. */+export interface ModelPaths {+ /** The directory holding Parakeet: `tokens.txt` and the encoder, decoder and joiner. */+ recognizer: string;+ /** Silero VAD's `.onnx` file. */+ vad: string;+ /** The speaker embedding model's `.onnx` file, or `null` to label no speakers. */+ speaker: string | null;+}+/** What the main process sends the transcription process, in this order. */export type WorkerRequest =| {type: "start";noteId: string;- modelDirectory: string;+ models: ModelPaths;/** Where development builds keep the audio and the transcript as produced, else `null`. */recordingDirectory: string | null;}diff --git a/apps/desktop/src/services/transcription/transcription_worker.spec.ts b/apps/desktop/src/services/transcription/transcription_worker.spec.tsindex d8e077a8fd..03d6dc704b 100644--- a/apps/desktop/src/services/transcription/transcription_worker.spec.ts+++ b/apps/desktop/src/services/transcription/transcription_worker.spec.ts@@ -69,8 +69,10 @@ function capturePort() {return Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });}+const MODELS = { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null };+const START: WorkerRequest = {- type: "start", noteId: "meeting", modelDirectory: "/models", recordingDirectory: null+ type: "start", noteId: "meeting", models: MODELS, recordingDirectory: null};beforeEach(() => {@@ -91,7 +93,7 @@ describe("transcription process", () => {parent.send(START);const ready = await parent.next("ready");expect(ready).toMatchObject({ type: "ready", settings: engine.settings });- expect(h.loadEngine).toHaveBeenCalledExactlyOnceWith("/models");+ expect(h.loadEngine).toHaveBeenCalledExactlyOnceWith(MODELS);expect(h.fileSinks).toEqual([]);const port = capturePort();diff --git a/apps/desktop/src/services/transcription/transcription_worker.ts b/apps/desktop/src/services/transcription/transcription_worker.tsindex a646115632..f40c5cfba3 100644--- a/apps/desktop/src/services/transcription/transcription_worker.ts+++ b/apps/desktop/src/services/transcription/transcription_worker.ts@@ -42,8 +42,8 @@ process.parentPort.on("message", ({ data, ports }) => {}});-async function start({ noteId, modelDirectory, recordingDirectory }: StartRequest) {- const engine = await loadSherpaEngine(modelDirectory);+async function start({ noteId, models, recordingDirectory }: StartRequest) {+ const engine = await loadSherpaEngine(models);const sinks: TranscriptSink[] = [{ write: (utterance) => post({ type: "utterance", utterance }) }];diff --git a/apps/desktop/src/services/transcription/worker_client.spec.ts b/apps/desktop/src/services/transcription/worker_client.spec.tsindex 948cac1dd8..31f780caee 100644--- a/apps/desktop/src/services/transcription/worker_client.spec.ts+++ b/apps/desktop/src/services/transcription/worker_client.spec.ts@@ -53,7 +53,11 @@ afterEach(() => {vi.useRealTimers();});-const REQUEST = { noteId: "meeting", modelDirectory: "/models", recordingDirectory: null };+const REQUEST = {+ noteId: "meeting",+ models: { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null },+ recordingDirectory: null+};describe("TranscriptionWorker", () => {it("starts the bundled worker, or the sources when there is no bundle", () => {diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.jsonindex c0dcc4714f..68bcf778b2 100644--- a/apps/server/src/assets/translations/en/server.json+++ b/apps/server/src/assets/translations/en/server.json@@ -571,9 +571,13 @@"save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.","capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.","attachment-title": "Transcript",- "model-title": "Speech recognition model",- "model-description": "Transcribes meetings on this device: Parakeet TDT 0.6B v3 for speech recognition, Silero VAD to find speech and TitaNet to tell speakers apart. Nothing you record leaves your computer.",- "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it."+ "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it.",+ "recognizer-title": "Parakeet TDT 0.6B v3",+ "recognizer-description": "NVIDIA's speech recognition model, which turns what is said into text in 25 European languages. It runs on this device: nothing you record leaves your computer.",+ "vad-title": "Silero VAD",+ "vad-description": "Finds the stretches of speech in the audio, so that only those are transcribed.",+ "speaker-title": "TitaNet small",+ "speaker-description": "Tells apart the other people in the meeting, so that the transcript shows who said what."},"resources": {"unknown": "There is no resource named {{id}}.",diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.tsindex e89ee68c1f..a7da875aba 100644--- a/packages/commons/src/lib/transcript.ts+++ b/packages/commons/src/lib/transcript.ts@@ -40,8 +40,12 @@ export const TRANSCRIPT_MIME = "application/json";/** The longest speaker name a transcript accepts. */export const MAX_SPEAKER_NAME_LENGTH = 100;-/** The resource holding the models transcription runs on, `TRANSCRIPTION_MODEL` on desktop. */-export const TRANSCRIPTION_MODEL_RESOURCE_ID = "transcription.model";+/** The resources holding the models transcription runs on, defined in the desktop app. */+export const TRANSCRIPTION_RESOURCES = {+ recognizer: "transcription.parakeet",+ vad: "transcription.silero-vad",+ speaker: "transcription.titanet"+} as const;/** Reads a transcript attachment's content, or returns `null` for content of another shape. */export function parseTranscript(content: string | null | undefined): Transcript | null {