Skip to content

Build your personal knowledge base with Trilium Notes

feat(desktop): download the transcription model on first use

Elian Doran2c57a81518bf
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index df27d13ef4..ea1dc11b77 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -35,7 +35,8 @@ The main process opens the capture window, starts the transcription process and
| File | Owns |
|---|---|
| `apps/desktop/src/services/transcription/capture.ts` | Main process: `CaptureWindow`, the hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) |
-| `…/index.ts` | Main process: IPC handlers, the state broadcast, `MODEL_DIR`, `getRecordingDirectory()`; starts a recording's process and capture window and connects them |
+| `…/index.ts` | Main process: IPC handlers, the state broadcast, `getRecordingDirectory()`; registers the model resource, starts a recording's process and capture window and connects them |
+| `…/model.ts` | `TRANSCRIPTION_MODEL`, the model as a downloadable resource (URLs pinned to a revision, sizes, SHA-256), and `LOCAL_MODEL_DIR` |
| `…/worker_client.ts` | Main process: `TranscriptionWorker`, which forks the transcription process, relays its events and stops or kills it |
| `…/attachment_sink.ts` | Main process: the `transcript` attachment, rewritten after every utterance |
| `…/protocol.ts` | Both: the messages between the processes, `SAMPLE_RATE`, `Logger` |
@@ -53,7 +54,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri
## The model
-The files go in `<data dir>/transcription-model`; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
+Users download the model from the app: the launcher calls `requireResources([TRANSCRIPTION_MODEL_RESOURCE_ID])` before it creates the meeting note, which shows the download dialog when the model is missing. The resource manager (`apps/server/src/services/resources.ts`) puts the int8 model, Silero VAD and TitaNet (~711 MB) in `<data dir>/resources/transcription.model/<version>/`, each file checked against the SHA-256 in `model.ts`. Changing a file there means a new `version`, so the new files download beside the old ones.
+
+A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the download; for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.
- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).
@@ -68,7 +71,7 @@ curl -fL -o silero_vad.onnx "$R/asr-models/silero_vad.onnx"
curl -fL -o speaker/nemo_en_titanet_small.onnx "$R/speaker-recongition-models/nemo_en_titanet_small.onnx"
```
-Without the directory, starting fails with `transcription.model-missing`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.
+Without either directory, starting fails with `transcription.model-not-downloaded`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.
## Development recordings
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
index 25215f6a7c..13893940a0 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
@@ -8,7 +8,8 @@ import { renderInto } from "../../test/render";
const mocks = vi.hoisted(() => ({
getInboxNote: vi.fn(),
createNote: vi.fn(),
- showError: vi.fn()
+ showError: vi.fn(),
+ requireResources: vi.fn()
}));
vi.mock("../../services/date_notes", () => ({
@@ -17,6 +18,9 @@ vi.mock("../../services/date_notes", () => ({
vi.mock("../../services/note_create", () => ({
default: { createNote: mocks.createNote }
}));
+vi.mock("../../services/dialog", () => ({
+ requireResources: mocks.requireResources
+}));
vi.mock("../../services/toast", () => ({
default: { showError: mocks.showError }
}));
@@ -69,6 +73,7 @@ describe("toggleRecording", () => {
window.electronApi = { transcription } as unknown as typeof window.electronApi;
mocks.getInboxNote.mockResolvedValue({ noteId: "inbox", isProtected: false });
mocks.createNote.mockResolvedValue({ note: { noteId: "meeting" } });
+ mocks.requireResources.mockResolvedValue(true);
});
it("starts recording into a new meeting note, and shows why it could not", async () => {
@@ -93,6 +98,17 @@ describe("toggleRecording", () => {
expect(mocks.createNote).not.toHaveBeenCalled();
});
+ it("asks for the model first, and creates no note when it is not downloaded", async () => {
+ await toggleRecording("idle");
+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith([ "transcription.model" ]);
+ expect(transcription.start).toHaveBeenCalledOnce();
+
+ mocks.requireResources.mockResolvedValueOnce(false);
+ await toggleRecording("idle");
+ expect(mocks.createNote).toHaveBeenCalledOnce();
+ expect(transcription.start).toHaveBeenCalledOnce();
+ });
+
it("starts nothing when no meeting note could be created", async () => {
mocks.getInboxNote.mockResolvedValue(null);
vi.spyOn(console, "warn").mockImplementation(() => {});
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
index 34b18e7fd7..03ce4e43c5 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
@@ -1,11 +1,13 @@
import "./RecordMeetingButton.css";
import {
- dayjs, MEETING_RECORDING_LABEL, type TranscriptionState, type TranscriptionStatus
+ dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_MODEL_RESOURCE_ID, type TranscriptionState,
+ type TranscriptionStatus
} from "@triliumnext/commons";
import { useCallback, useEffect, useState } from "preact/hooks";
import dateNoteService from "../../services/date_notes";
+import { requireResources } from "../../services/dialog";
import { t } from "../../services/i18n";
import noteCreateService from "../../services/note_create";
import toast from "../../services/toast";
@@ -102,6 +104,9 @@ export async function toggleRecording(status: TranscriptionStatus) {
if (status !== "idle") {
return;
}
+ if (!await requireResources([ TRANSCRIPTION_MODEL_RESOURCE_ID ])) {
+ return;
+ }
const note = (await createMeetingNote())?.note;
if (!note) {
diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.ts
index 53e2560b48..c38f6c0290 100644
--- a/apps/desktop/src/services/transcription/index.spec.ts
+++ b/apps/desktop/src/services/transcription/index.spec.ts
@@ -15,6 +15,7 @@ const h = vi.hoisted(() => ({
coreListeners: new Map<string, () => void>(),
isDev: false,
isModelInstalled: true,
+ registerResource: vi.fn(),
log: { info: vi.fn(), error: vi.fn() },
listeners: null as WorkerListeners | null,
captureOptions: null as CaptureWindowOptions | null,
@@ -49,7 +50,6 @@ vi.mock("electron", () => ({
port2 = "worker end";
}
}));
-vi.mock("fs", () => ({ default: { existsSync: () => h.isModelInstalled } }));
vi.mock("@triliumnext/core", () => ({
becca: {
getNote: (noteId: string) =>
@@ -69,6 +69,12 @@ vi.mock("i18next", () => ({
vi.mock("@triliumnext/server/src/services/data_dir.js", () => ({
default: { TRILIUM_DATA_DIR: "/data" }
}));
+vi.mock("@triliumnext/server/src/services/resources.js", () => ({
+ default: {
+ register: h.registerResource,
+ getPath: (id: string) => h.isModelInstalled ? `/data/resources/${id}` : null
+ }
+}));
vi.mock("./capture", () => ({
CaptureWindow: class {
start = h.capture.start;
@@ -173,7 +179,7 @@ describe("transcription IPC", () => {
expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({
noteId: "meeting",
- modelDirectory: "/data/transcription-model",
+ modelDirectory: "/data/resources/transcription.model",
recordingDirectory: null
});
expect(h.sink.open).toHaveBeenCalledExactlyOnceWith(STARTED_AT);
@@ -318,7 +324,7 @@ describe("transcription IPC", () => {
.toBe("transcription.note-not-found");
h.isModelInstalled = false;
expect(await invoke("transcription-start", "meeting"))
- .toBe(`transcription.model-missing {"directory":"/data/transcription-model"}`);
+ .toBe("transcription.model-not-downloaded");
expect(h.listeners).toBeNull();
expect(h.broadcast).toEqual([]);
});
diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.ts
index 28c3003a88..7f3d881579 100644
--- a/apps/desktop/src/services/transcription/index.ts
+++ b/apps/desktop/src/services/transcription/index.ts
@@ -1,18 +1,16 @@
import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";
import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";
import dataDirs from "@triliumnext/server/src/services/data_dir.js";
+import resources from "@triliumnext/server/src/services/resources.js";
import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";
-import fs from "fs";
import { t } from "i18next";
import path from "path";
import { AttachmentSink } from "./attachment_sink";
import { CaptureWindow } from "./capture";
+import { TRANSCRIPTION_MODEL } from "./model";
import { TranscriptionWorker } from "./worker_client";
-/** Where the models live: Parakeet TDT 0.6B v3 (int8) plus `silero_vad.onnx`. */
-export const MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");
-
/** Where development builds keep the audio of each recording, one directory per recording. */
export const RECORDINGS_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-recordings");
@@ -41,6 +39,7 @@ const unsavedSinks = new Set<AttachmentSink>();
let quitting: "no" | "waiting" | "done" = "no";
export function setupTranscriptionHandlers() {
+ resources.register(TRANSCRIPTION_MODEL);
ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));
ipcMain.handle("transcription-stop", () => stopRecording());
ipcMain.handle("transcription-get-state", () => getState());
@@ -63,8 +62,9 @@ export async function startRecording(noteId: unknown): Promise<string> {
if (typeof noteId !== "string" || !becca.getNote(noteId)) {
return t("transcription.note-not-found");
}
- if (!fs.existsSync(MODEL_DIR)) {
- return t("transcription.model-missing", { directory: MODEL_DIR });
+ const modelDirectory = resources.getPath(TRANSCRIPTION_MODEL.id);
+ if (!modelDirectory) {
+ return t("transcription.model-not-downloaded");
}
setState({ status: "starting", noteId, pendingSources: [] });
@@ -84,7 +84,7 @@ export async function startRecording(noteId: unknown): Promise<string> {
try {
const { startedAt } = await recording.worker.start({
noteId,
- modelDirectory: MODEL_DIR,
+ modelDirectory,
recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null
});
sink.open(startedAt);
diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.ts
new file mode 100644
index 0000000000..3ae1926a7b
--- /dev/null
+++ b/apps/desktop/src/services/transcription/model.ts
@@ -0,0 +1,62 @@
+import { TRANSCRIPTION_MODEL_RESOURCE_ID } from "@triliumnext/commons";
+import dataDirs from "@triliumnext/server/src/services/data_dir.js";
+import type { ResourceDefinition } from "@triliumnext/server/src/services/resources.js";
+import fs from "fs";
+import { t } from "i18next";
+import path from "path";
+
+/**
+ * A model directory installed by hand, which takes precedence over the downloaded one: it can
+ * hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.
+ */
+export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");
+
+const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";
+const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";
+
+/** The files `loadSherpaEngine()` reads from its model directory, with the int8 Parakeet model. */
+export const TRANSCRIPTION_MODEL: ResourceDefinition = {
+ id: TRANSCRIPTION_MODEL_RESOURCE_ID,
+ version: "parakeet-v3-int8-1",
+ title: () => t("transcription.model-title"),
+ description: () => t("transcription.model-description"),
+ findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,
+ files: [
+ {
+ path: "encoder.int8.onnx",
+ url: `${PARAKEET}/encoder.int8.onnx`,
+ size: 652184281,
+ sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"
+ },
+ {
+ path: "decoder.int8.onnx",
+ url: `${PARAKEET}/decoder.int8.onnx`,
+ size: 11845275,
+ sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"
+ },
+ {
+ path: "joiner.int8.onnx",
+ url: `${PARAKEET}/joiner.int8.onnx`,
+ size: 6355277,
+ sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"
+ },
+ {
+ path: "tokens.txt",
+ url: `${PARAKEET}/tokens.txt`,
+ size: 93939,
+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"
+ },
+ {
+ path: "silero_vad.onnx",
+ url: `${SHERPA_RELEASES}/asr-models/silero_vad.onnx`,
+ size: 643854,
+ sha256: "9e2449e1087496d8d4caba907f23e0bd3f78d91fa552479bb9c23ac09cbb1fd6"
+ },
+ {
+ path: "speaker/nemo_en_titanet_small.onnx",
+ url: `${SHERPA_RELEASES}/speaker-recongition-models/nemo_en_titanet_small.onnx`,
+ size: 40257283,
+ sha256: "ad4a1802485d8b34c722d2a9d04249662f2ece5d28a7a039063ca22f515a789e"
+ }
+ ]
+};
diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.json
index 69dab37b4a..c0dcc4714f 100644
--- a/apps/server/src/assets/translations/en/server.json
+++ b/apps/server/src/assets/translations/en/server.json
@@ -565,13 +565,15 @@
"transcription": {
"already-recording": "A meeting is already being transcribed.",
"note-not-found": "The meeting note no longer exists.",
- "model-missing": "No speech recognition model was found in {{- directory}}.",
"start-failed": "The transcription could not start: {{- message}}",
"stopped-unexpectedly": "The transcription stopped unexpectedly. What was transcribed until then is kept.",
"save-locked": "The transcript cannot be saved while the protected session is locked. The transcription keeps it and saves it once you enter the protected session, as long as Trilium stays open.",
"save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.",
"capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.",
- "attachment-title": "Transcript"
+ "attachment-title": "Transcript",
+ "model-title": "Speech recognition model",
+ "model-description": "Transcribes meetings on this device: Parakeet TDT 0.6B v3 for speech recognition, Silero VAD to find speech and TitaNet to tell speakers apart. Nothing you record leaves your computer.",
+ "model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it."
},
"resources": {
"unknown": "There is no resource named {{id}}.",
diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.ts
index 386e565de0..e89ee68c1f 100644
--- a/packages/commons/src/lib/transcript.ts
+++ b/packages/commons/src/lib/transcript.ts
@@ -40,6 +40,9 @@ export const TRANSCRIPT_MIME = "application/json";
/** The longest speaker name a transcript accepts. */
export const MAX_SPEAKER_NAME_LENGTH = 100;
+/** The resource holding the models transcription runs on, `TRANSCRIPTION_MODEL` on desktop. */
+export const TRANSCRIPTION_MODEL_RESOURCE_ID = "transcription.model";
+
/** Reads a transcript attachment's content, or returns `null` for content of another shape. */
export function parseTranscript(content: string | null | undefined): Transcript | null {
if (!content) {