Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(desktop): download the transcription model on first use
Elian Doran2c57a81518bf
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex df27d13ef4..ea1dc11b77 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -35,7 +35,8 @@ The main process opens the capture window, starts the transcription process and| File | Owns ||---|---|| `apps/desktop/src/services/transcription/capture.ts` | Main process: `CaptureWindow`, the hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) |-| `…/index.ts` | Main process: IPC handlers, the state broadcast, `MODEL_DIR`, `getRecordingDirectory()`; starts a recording's process and capture window and connects them |+| `…/index.ts` | Main process: IPC handlers, the state broadcast, `getRecordingDirectory()`; registers the model resource, starts a recording's process and capture window and connects them |+| `…/model.ts` | `TRANSCRIPTION_MODEL`, the model as a downloadable resource (URLs pinned to a revision, sizes, SHA-256), and `LOCAL_MODEL_DIR` || `…/worker_client.ts` | Main process: `TranscriptionWorker`, which forks the transcription process, relays its events and stops or kills it || `…/attachment_sink.ts` | Main process: the `transcript` attachment, rewritten after every utterance || `…/protocol.ts` | Both: the messages between the processes, `SAMPLE_RATE`, `Logger` |@@ -53,7 +54,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri## The model-The files go in `<data dir>/transcription-model`; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").+Users download the model from the app: the launcher calls `requireResources([TRANSCRIPTION_MODEL_RESOURCE_ID])` before it creates the meeting note, which shows the download dialog when the model is missing. The resource manager (`apps/server/src/services/resources.ts`) puts the int8 model, Silero VAD and TitaNet (~711 MB) in `<data dir>/resources/transcription.model/<version>/`, each file checked against the SHA-256 in `model.ts`. Changing a file there means a new `version`, so the new files download beside the old ones.++A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the download; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).@@ -68,7 +71,7 @@ curl -fL -o silero_vad.onnx "$R/asr-models/silero_vad.onnx"curl -fL -o speaker/nemo_en_titanet_small.onnx "$R/speaker-recongition-models/nemo_en_titanet_small.onnx"```-Without the directory, starting fails with `transcription.model-missing`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.+Without either directory, starting fails with `transcription.model-not-downloaded`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.## Development recordingsdiff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsxindex 25215f6a7c..13893940a0 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx@@ -8,7 +8,8 @@ import { renderInto } from "../../test/render";const mocks = vi.hoisted(() => ({getInboxNote: vi.fn(),createNote: vi.fn(),- showError: vi.fn()+ showError: vi.fn(),+ requireResources: vi.fn()}));vi.mock("../../services/date_notes", () => ({@@ -17,6 +18,9 @@ vi.mock("../../services/date_notes", () => ({vi.mock("../../services/note_create", () => ({default: { createNote: mocks.createNote }}));+vi.mock("../../services/dialog", () => ({+ requireResources: mocks.requireResources+}));vi.mock("../../services/toast", () => ({default: { showError: mocks.showError }}));@@ -69,6 +73,7 @@ describe("toggleRecording", () => {window.electronApi = { transcription } as unknown as typeof window.electronApi;mocks.getInboxNote.mockResolvedValue({ noteId: "inbox", isProtected: false });mocks.createNote.mockResolvedValue({ note: { noteId: "meeting" } });+ mocks.requireResources.mockResolvedValue(true);});it("starts recording into a new meeting note, and shows why it could not", async () => {@@ -93,6 +98,17 @@ describe("toggleRecording", () => {expect(mocks.createNote).not.toHaveBeenCalled();});+ it("asks for the model first, and creates no note when it is not downloaded", async () => {+ await toggleRecording("idle");+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith([ "transcription.model" ]);+ expect(transcription.start).toHaveBeenCalledOnce();++ mocks.requireResources.mockResolvedValueOnce(false);+ await toggleRecording("idle");+ expect(mocks.createNote).toHaveBeenCalledOnce();+ expect(transcription.start).toHaveBeenCalledOnce();+ });+it("starts nothing when no meeting note could be created", async () => {mocks.getInboxNote.mockResolvedValue(null);vi.spyOn(console, "warn").mockImplementation(() => {});diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsxindex 34b18e7fd7..03ce4e43c5 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx@@ -1,11 +1,13 @@import "./RecordMeetingButton.css";import {- dayjs, MEETING_RECORDING_LABEL, type TranscriptionState, type TranscriptionStatus+ dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_MODEL_RESOURCE_ID, type TranscriptionState,+ type TranscriptionStatus} from "@triliumnext/commons";import { useCallback, useEffect, useState } from "preact/hooks";import dateNoteService from "../../services/date_notes";+import { requireResources } from "../../services/dialog";import { t } from "../../services/i18n";import noteCreateService from "../../services/note_create";import toast from "../../services/toast";@@ -102,6 +104,9 @@ export async function toggleRecording(status: TranscriptionStatus) {if (status !== "idle") {return;}+ if (!await requireResources([ TRANSCRIPTION_MODEL_RESOURCE_ID ])) {+ return;+ }const note = (await createMeetingNote())?.note;if (!note) {diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.tsindex 53e2560b48..c38f6c0290 100644--- a/apps/desktop/src/services/transcription/index.spec.ts+++ b/apps/desktop/src/services/transcription/index.spec.ts@@ -15,6 +15,7 @@ const h = vi.hoisted(() => ({coreListeners: new Map<string, () => void>(),isDev: false,isModelInstalled: true,+ registerResource: vi.fn(),log: { info: vi.fn(), error: vi.fn() },listeners: null as WorkerListeners | null,captureOptions: null as CaptureWindowOptions | null,@@ -49,7 +50,6 @@ vi.mock("electron", () => ({port2 = "worker end";}}));-vi.mock("fs", () => ({ default: { existsSync: () => h.isModelInstalled } }));vi.mock("@triliumnext/core", () => ({becca: {getNote: (noteId: string) =>@@ -69,6 +69,12 @@ vi.mock("i18next", () => ({vi.mock("@triliumnext/server/src/services/data_dir.js", () => ({default: { TRILIUM_DATA_DIR: "/data" }}));+vi.mock("@triliumnext/server/src/services/resources.js", () => ({+ default: {+ register: h.registerResource,+ getPath: (id: string) => h.isModelInstalled ? `/data/resources/${id}` : null+ }+}));vi.mock("./capture", () => ({CaptureWindow: class {start = h.capture.start;@@ -173,7 +179,7 @@ describe("transcription IPC", () => {expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({noteId: "meeting",- modelDirectory: "/data/transcription-model",+ modelDirectory: "/data/resources/transcription.model",recordingDirectory: null});expect(h.sink.open).toHaveBeenCalledExactlyOnceWith(STARTED_AT);@@ -318,7 +324,7 @@ describe("transcription IPC", () => {.toBe("transcription.note-not-found");h.isModelInstalled = false;expect(await invoke("transcription-start", "meeting"))- .toBe(`transcription.model-missing {"directory":"/data/transcription-model"}`);+ .toBe("transcription.model-not-downloaded");expect(h.listeners).toBeNull();expect(h.broadcast).toEqual([]);});diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.tsindex 28c3003a88..7f3d881579 100644--- a/apps/desktop/src/services/transcription/index.ts+++ b/apps/desktop/src/services/transcription/index.ts@@ -1,18 +1,16 @@import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";import dataDirs from "@triliumnext/server/src/services/data_dir.js";+import resources from "@triliumnext/server/src/services/resources.js";import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";-import fs from "fs";import { t } from "i18next";import path from "path";import { AttachmentSink } from "./attachment_sink";import { CaptureWindow } from "./capture";+import { TRANSCRIPTION_MODEL } from "./model";import { TranscriptionWorker } from "./worker_client";-/** Where the models live: Parakeet TDT 0.6B v3 (int8) plus `silero_vad.onnx`. */-export const MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");-/** Where development builds keep the audio of each recording, one directory per recording. */export const RECORDINGS_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-recordings");@@ -41,6 +39,7 @@ const unsavedSinks = new Set<AttachmentSink>();let quitting: "no" | "waiting" | "done" = "no";export function setupTranscriptionHandlers() {+ resources.register(TRANSCRIPTION_MODEL);ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));ipcMain.handle("transcription-stop", () => stopRecording());ipcMain.handle("transcription-get-state", () => getState());@@ -63,8 +62,9 @@ export async function startRecording(noteId: unknown): Promise<string> {if (typeof noteId !== "string" || !becca.getNote(noteId)) {return t("transcription.note-not-found");}- if (!fs.existsSync(MODEL_DIR)) {- return t("transcription.model-missing", { directory: MODEL_DIR });+ const modelDirectory = resources.getPath(TRANSCRIPTION_MODEL.id);+ if (!modelDirectory) {+ return t("transcription.model-not-downloaded");}setState({ status: "starting", noteId, pendingSources: [] });@@ -84,7 +84,7 @@ export async function startRecording(noteId: unknown): Promise<string> {try {const { startedAt } = await recording.worker.start({noteId,- modelDirectory: MODEL_DIR,+ modelDirectory,recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null});sink.open(startedAt);diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.tsnew file mode 100644index 0000000000..3ae1926a7b--- /dev/null+++ b/apps/desktop/src/services/transcription/model.ts@@ -0,0 +1,62 @@+import { TRANSCRIPTION_MODEL_RESOURCE_ID } from "@triliumnext/commons";+import dataDirs from "@triliumnext/server/src/services/data_dir.js";+import type { ResourceDefinition } from "@triliumnext/server/src/services/resources.js";+import fs from "fs";+import { t } from "i18next";+import path from "path";++/**+ * A model directory installed by hand, which takes precedence over the downloaded one: it can+ * hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.+ */+export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");++const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";+const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";++/** The files `loadSherpaEngine()` reads from its model directory, with the int8 Parakeet model. */+export const TRANSCRIPTION_MODEL: ResourceDefinition = {+ id: TRANSCRIPTION_MODEL_RESOURCE_ID,+ version: "parakeet-v3-int8-1",+ title: () => t("transcription.model-title"),+ description: () => t("transcription.model-description"),+ findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,+ files: [+ {+ path: "encoder.int8.onnx",+ url: `${PARAKEET}/encoder.int8.onnx`,+ size: 652184281,+ sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"+ },+ {+ path: "decoder.int8.onnx",+ url: `${PARAKEET}/decoder.int8.onnx`,+ size: 11845275,+ sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"+ },+ {+ path: "joiner.int8.onnx",+ url: `${PARAKEET}/joiner.int8.onnx`,+ size: 6355277,+ sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"+ },+ {+ path: "tokens.txt",+ url: `${PARAKEET}/tokens.txt`,+ size: 93939,+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"+ },+ {+ path: "silero_vad.onnx",+ url: `${SHERPA_RELEASES}/asr-models/silero_vad.onnx`,+ size: 643854,+ sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"+ },+ {+ path: "speaker/nemo_en_titanet_small.onnx",+ url: `${SHERPA_RELEASES}/speaker-recongition-models/nemo_en_titanet_small.onnx`,+ size: 40257283,+ sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"+ }+ ]+};diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.jsonindex 69dab37b4a..c0dcc4714f 100644--- a/apps/server/src/assets/translations/en/server.json+++ b/apps/server/src/assets/translations/en/server.json@@ -565,13 +565,15 @@"transcription": {"already-recording": "A meeting is already being transcribed.","note-not-found": "The meeting note no longer exists.",- "model-missing": "No speech recognition model was found in {{- directory}}.","start-failed": "The transcription could not start: {{- message}}","stopped-unexpectedly": "The transcription stopped unexpectedly. What was transcribed until then is kept.","save-locked": "The transcript cannot be saved while the protected session is locked. The transcription keeps it and saves it once you enter the protected session, as long as Trilium stays open.","save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.","capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.",- "attachment-title": "Transcript"+ "attachment-title": "Transcript",+ "model-title": "Speech recognition model",+ "model-description": "Transcribes meetings on this device: Parakeet TDT 0.6B v3 for speech recognition, Silero VAD to find speech and TitaNet to tell speakers apart. Nothing you record leaves your computer.",+ "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it."},"resources": {"unknown": "There is no resource named {{id}}.",diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.tsindex 386e565de0..e89ee68c1f 100644--- a/packages/commons/src/lib/transcript.ts+++ b/packages/commons/src/lib/transcript.ts@@ -40,6 +40,9 @@ export const TRANSCRIPT_MIME = "application/json";/** The longest speaker name a transcript accepts. */export const MAX_SPEAKER_NAME_LENGTH = 100;+/** The resource holding the models transcription runs on, `TRANSCRIPTION_MODEL` on desktop. */+export const TRANSCRIPTION_MODEL_RESOURCE_ID = "transcription.model";+/** Reads a transcript attachment's content, or returns `null` for content of another shape. */export function parseTranscript(content: string | null | undefined): Transcript | null {if (!content) {