Skip to content

Build your personal knowledge base with Trilium Notes

feat(transcription): choose the speech recognition model in the settings

Elian Doranf6408f8f11df
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index bfc07627e7..c5738c77bc 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -55,9 +55,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri
## The model
-Users download the models from the app: the launcher calls `requireResources(Object.values(TRANSCRIPTION_RESOURCES))` before it creates the meeting note, which shows the download dialog for those missing. They are three resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>-<hash>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). The hash in the directory name covers every file's path and SHA-256, so changing a file of one model downloads that model alone again, into a new directory. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.
+Users choose the recognizer under Options → Transcription: the non-synced option `transcriptionModel` holds a key of `TRANSCRIPTION_MODELS` (commons `transcript.ts`), `parakeet-int8` (the default, "Standard") or `parakeet-fp32` ("High accuracy"), each naming its resource and precision; an unknown value reads as the default (`getTranscriptionModel()`). The launcher calls `requireResources(getTranscriptionResources(<option>))` before it creates the meeting note, which shows the download dialog for those missing, and `startRecording()` resolves the same option through `getModelPaths()`. The models are resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>-<hash>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.parakeet-fp32` (2.5 GB, pinned to a Hugging Face revision), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). The hash in the directory name covers every file's path and SHA-256, so changing a file of one model downloads that model alone again, into a new directory. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.
-A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself (Silero when `silero_vad.onnx` is there, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
+A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself, when it holds that model's files (each Parakeet precision when every one of its files is there, Silero when `silero_vad.onnx` is, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. `ModelPaths.recognizer` carries the precision with the directory, and `loadSherpaEngine()` loads that precision's files; it records it as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.
- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).
diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjs
index 7a1ff357b4..1198556ef9 100644
--- a/.claude/skills/working-with-transcription/worker-test.mjs
+++ b/.claude/skills/working-with-transcription/worker-test.mjs
@@ -10,7 +10,8 @@
// --source <name> which WAV of the recording to play, default system
// --seconds <n> how much of it, default 120
// --speed <n> how much faster than real time to post it, default 4
-// --model <dir> the model directory, default apps/desktop/data/transcription-model
+// --model <dir> the model directory, default apps/desktop/data/transcription-model; its
+// full-precision Parakeet when it holds encoder.weights, else the int8 one
// --out <file> writes the utterances there, in the transcript format
//
// Like `pnpm desktop:start`, it loads the TypeScript through NODE_OPTIONS="--import tsx", which
@@ -125,7 +126,10 @@ async function runInElectron(config) {
});
const startedAt = performance.now();
const models = {
- recognizer: config.model,
+ recognizer: {
+ directory: config.model,
+ precision: fs.existsSync(path.join(config.model, "encoder.weights")) ? "fp32" : "int8"
+ },
vad: path.join(config.model, "silero_vad.onnx"),
speaker: path.join(config.model, "speaker", "nemo_en_titanet_small.onnx")
};
diff --git a/apps/client/src/translations/en/translation.json b/apps/client/src/translations/en/translation.json
index 62c579372c..bc90595c6d 100644
--- a/apps/client/src/translations/en/translation.json
+++ b/apps/client/src/translations/en/translation.json
@@ -4625,6 +4625,10 @@
"license": "{{license}} license"
},
"transcription_options": {
- "placeholder": "Settings for transcribing meetings will appear here."
+ "recognition_title": "Speech recognition",
+ "european_languages": "European languages",
+ "european_languages_description": "Parakeet TDT 0.6B v3 by NVIDIA recognizes 25 European languages and works out which one is spoken. High accuracy makes fewer mistakes, but it is a 2.4 GB download instead of 0.6 GB and needs about 1.6 GB more memory while recording. A model that is not downloaded yet is downloaded when you start a recording.",
+ "model_standard": "Standard",
+ "model_high_accuracy": "High accuracy"
}
}
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
index fb7434a93b..0ea8060998 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx
@@ -9,7 +9,8 @@ const mocks = vi.hoisted(() => ({
getInboxNote: vi.fn(),
createNote: vi.fn(),
showError: vi.fn(),
- requireResources: vi.fn()
+ requireResources: vi.fn(),
+ transcriptionModel: null as string | null
}));
vi.mock("../../services/date_notes", () => ({
@@ -21,6 +22,9 @@ vi.mock("../../services/note_create", () => ({
vi.mock("../../services/dialog", () => ({
requireResources: mocks.requireResources
}));
+vi.mock("../../services/options", () => ({
+ default: { get: (name: string) => name === "transcriptionModel" ? mocks.transcriptionModel : null }
+}));
vi.mock("../../services/toast", () => ({
default: { showError: mocks.showError }
}));
@@ -38,6 +42,7 @@ import RecordMeetingButton, { createMeetingNote, toggleRecording } from "./Recor
beforeEach(() => {
vi.clearAllMocks();
+ mocks.transcriptionModel = null;
vi.useFakeTimers({ now: new Date(2026, 9, 13, 14, 30), toFake: [ "Date" ] });
});
@@ -110,6 +115,13 @@ describe("toggleRecording", () => {
expect(transcription.start).toHaveBeenCalledOnce();
});
+ it("asks for the model the settings select", async () => {
+ mocks.transcriptionModel = "parakeet-fp32";
+ await toggleRecording("idle");
+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith(
+ [ "transcription.parakeet-fp32", "transcription.silero-vad", "transcription.titanet" ]);
+ });
+
it("starts nothing when no meeting note could be created", async () => {
mocks.getInboxNote.mockResolvedValue(null);
vi.spyOn(console, "warn").mockImplementation(() => {});
diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
index 44dd00127e..0d4d435739 100644
--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx
@@ -1,7 +1,7 @@
import "./RecordMeetingButton.css";
import {
- dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_RESOURCES, type TranscriptionState,
+ dayjs, getTranscriptionResources, MEETING_RECORDING_LABEL, type TranscriptionState,
type TranscriptionStatus
} from "@triliumnext/commons";
import { useCallback, useEffect, useState } from "preact/hooks";
@@ -10,6 +10,7 @@ import dateNoteService from "../../services/date_notes";
import { requireResources } from "../../services/dialog";
import { t } from "../../services/i18n";
import noteCreateService from "../../services/note_create";
+import options from "../../services/options";
import toast from "../../services/toast";
import {
LaunchBarActionButton,
@@ -104,7 +105,7 @@ export async function toggleRecording(status: TranscriptionStatus) {
if (status !== "idle") {
return;
}
- if (!await requireResources(Object.values(TRANSCRIPTION_RESOURCES))) {
+ if (!await requireResources(getTranscriptionResources(options.get("transcriptionModel")))) {
return;
}
diff --git a/apps/client/src/widgets/type_widgets/options/transcription.spec.tsx b/apps/client/src/widgets/type_widgets/options/transcription.spec.tsx
new file mode 100644
index 0000000000..769bfd43ba
--- /dev/null
+++ b/apps/client/src/widgets/type_widgets/options/transcription.spec.tsx
@@ -0,0 +1,80 @@
+import { render } from "preact";
+import { act } from "preact/test-utils";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+const mocks = vi.hoisted(() => ({
+ electron: true,
+ stored: "" as string,
+ saved: [] as string[]
+}));
+
+vi.mock("../../../services/utils", async (importOriginal) => ({
+ ...(await importOriginal<typeof import("../../../services/utils")>()),
+ isElectron: () => mocks.electron
+}));
+
+vi.mock("../../../services/i18n", () => ({ t: (key: string) => key }));
+
+vi.mock("../../react/hooks", async (importOriginal) => ({
+ ...(await importOriginal<typeof import("../../react/hooks")>()),
+ useTriliumOption: (name: string) => [
+ name === "transcriptionModel" ? mocks.stored : "",
+ (value: string) => void mocks.saved.push(value)
+ ]
+}));
+
+vi.mock("./components/OptionsPageHeader", () => ({ default: () => <div className="header-stub" /> }));
+
+import TranscriptionSettings from "./transcription";
+
+let host: HTMLElement;
+
+beforeEach(() => {
+ mocks.electron = true;
+ mocks.stored = "parakeet-fp32";
+ mocks.saved = [];
+ host = document.body.appendChild(document.createElement("div"));
+});
+
+afterEach(() => {
+ render(null, host);
+ document.body.innerHTML = "";
+});
+
+function open() {
+ act(() => {
+ render(<TranscriptionSettings />, host);
+ });
+ return host.querySelector<HTMLSelectElement>("select");
+}
+
+describe("TranscriptionSettings", () => {
+ it("selects the model a recording runs on, and saves another choice", () => {
+ const select = open();
+ if (!select) {
+ throw new Error("The model selector is missing.");
+ }
+ expect([ ...select.options ].map((option) => [ option.value, option.textContent ])).toEqual([
+ [ "parakeet-int8", "transcription_options.model_standard" ],
+ [ "parakeet-fp32", "transcription_options.model_high_accuracy" ]
+ ]);
+ expect(select.value).toBe("parakeet-fp32");
+
+ act(() => {
+ select.value = "parakeet-int8";
+ select.dispatchEvent(new Event("change", { bubbles: true }));
+ });
+ expect(mocks.saved).toEqual([ "parakeet-int8" ]);
+ });
+
+ it("shows the default model for a value it does not know", () => {
+ mocks.stored = "whisper";
+ expect(open()?.value).toBe("parakeet-int8");
+ });
+
+ it("explains that the page applies to desktop builds only, elsewhere", () => {
+ mocks.electron = false;
+ expect(open()).toBeNull();
+ expect(host.textContent).toContain("desktop.web_placeholder");
+ });
+});
diff --git a/apps/client/src/widgets/type_widgets/options/transcription.tsx b/apps/client/src/widgets/type_widgets/options/transcription.tsx
index 41aa6f9c60..b78fa0f5e5 100644
--- a/apps/client/src/widgets/type_widgets/options/transcription.tsx
+++ b/apps/client/src/widgets/type_widgets/options/transcription.tsx
@@ -1,16 +1,54 @@
+import { getTranscriptionModel, TRANSCRIPTION_MODELS, type TranscriptionModelId } from "@triliumnext/commons";
+
import { t } from "../../../services/i18n";
import { isElectron } from "../../../services/utils";
+import { Card, OptionCardSection } from "../../react/Card";
+import FormSelect from "../../react/FormSelect";
+import { useTriliumOption } from "../../react/hooks";
import NoItems from "../../react/NoItems";
import OptionsPageHeader from "./components/OptionsPageHeader";
export default function TranscriptionSettings() {
+ if (!isElectron()) {
+ return (
+ <>
+ <OptionsPageHeader />
+ <NoItems icon="bx bx-microphone" text={t("desktop.web_placeholder")} />
+ </>
+ );
+ }
+
return (
<>
<OptionsPageHeader />
- <NoItems
- icon="bx bx-microphone"
- text={isElectron() ? t("transcription_options.placeholder") : t("desktop.web_placeholder")}
- />
+ <RecognitionSettings />
</>
);
}
+
+function RecognitionSettings() {
+ const [ model, setModel ] = useTriliumOption("transcriptionModel");
+ const labels: Record<TranscriptionModelId, string> = {
+ "parakeet-int8": t("transcription_options.model_standard"),
+ "parakeet-fp32": t("transcription_options.model_high_accuracy")
+ };
+
+ return (
+ <Card heading={t("transcription_options.recognition_title")}>
+ <OptionCardSection
+ name="transcription-model"
+ label={t("transcription_options.european_languages")}
+ description={t("transcription_options.european_languages_description")}
+ >
+ <FormSelect
+ name="transcription-model"
+ currentValue={getTranscriptionModel(model).id}
+ onChange={setModel}
+ keyProperty="id"
+ titleProperty="label"
+ values={Object.values(TRANSCRIPTION_MODELS).map(({ id }) => ({ id, label: labels[id] }))}
+ />
+ </OptionCardSection>
+ </Card>
+ );
+}
diff --git a/apps/desktop/src/services/transcription/engine.spec.ts b/apps/desktop/src/services/transcription/engine.spec.ts
index 59e9bc733f..d37e34868d 100644
--- a/apps/desktop/src/services/transcription/engine.spec.ts
+++ b/apps/desktop/src/services/transcription/engine.spec.ts
@@ -103,7 +103,7 @@ vi.mock("fs", () => ({
const MODEL_DIR = "/models/parakeet";
const MODELS = {
- recognizer: MODEL_DIR,
+ recognizer: { directory: MODEL_DIR, precision: "int8" as const },
vad: "/models/silero/silero_vad.onnx",
speaker: "/models/titanet/nemo_en_titanet_small.onnx"
};
@@ -119,6 +119,8 @@ beforeEach(() => {
describe("loadSherpaEngine", () => {
it("loads Parakeet and Silero from their paths and reports their settings", async () => {
+ // The precision comes from the paths, whatever else the directory holds.
+ h.installed = [ "encoder.onnx" ];
const engine = await loadSherpaEngine(MODELS);
expect(h.recognizerConfig).toMatchObject({
@@ -152,9 +154,10 @@ describe("loadSherpaEngine", () => {
expect(h.decoded).toEqual([ samples ]);
});
- it("prefers the full-precision model, decoding it without a blank penalty", async () => {
- h.installed = [ "encoder.onnx" ];
- const engine = await loadSherpaEngine(MODELS);
+ it("loads the full-precision model when given, decoding it without a blank penalty", async () => {
+ const engine = await loadSherpaEngine({
+ ...MODELS, recognizer: { directory: MODEL_DIR, precision: "fp32" }
+ });
expect(h.recognizerConfig).toMatchObject({
modelConfig: {
diff --git a/apps/desktop/src/services/transcription/engine.ts b/apps/desktop/src/services/transcription/engine.ts
index 425fcd5119..e70c22d559 100644
--- a/apps/desktop/src/services/transcription/engine.ts
+++ b/apps/desktop/src/services/transcription/engine.ts
@@ -1,3 +1,4 @@
+import type { TranscriptionPrecision } from "@triliumnext/commons";
import fs from "fs";
import path from "path";
import type { SpeakerEmbeddingExtractor, Vad } from "sherpa-onnx-node";
@@ -62,15 +63,12 @@ const RECOGNIZER_SETTINGS = {
provider: "cpu"
};
-/** `fp32` is the full-precision export (2.4 GB), `int8` the quantized one (641 MB). */
-type ModelPrecision = "fp32" | "int8";
-
/**
* The int8 model emits nothing over some stretches of speech and drops whole clauses from segments
* with pauses; `blankPenalty` counters that. The full-precision model drops none, and the penalty
* only adds stray words to its transcript.
*/
-const DECODING_SETTINGS: Record<ModelPrecision, Record<string, unknown>> = {
+const DECODING_SETTINGS: Record<TranscriptionPrecision, Record<string, unknown>> = {
fp32: { decodingMethod: "greedy_search" },
int8: { decodingMethod: "greedy_search", blankPenalty: 0.5 }
};
@@ -107,16 +105,13 @@ const VAD_SETTINGS = {
/**
* Loads the local engine from `models`: Silero VAD for segmentation and Parakeet TDT 0.6B v3 for
* recognition, both through sherpa-onnx, and the speaker model when it is given and installed.
- * Recognition uses the full-precision model (`encoder.onnx` with `encoder.weights`) when the
- * recognizer directory holds it, else the int8 one (`encoder.int8.onnx`). Recognition runs on the
- * libuv thread pool; the VAD and the speaker embeddings run on the calling thread.
+ * The recognizer's precision picks its files: `encoder.onnx` (with `encoder.weights`) for `fp32`,
+ * `encoder.int8.onnx` for `int8`. Recognition runs on the libuv thread pool; the VAD and the
+ * speaker embeddings run on the calling thread.
*/
export async function loadSherpaEngine(models: ModelPaths): Promise<SpeechEngine> {
- const modelDir = models.recognizer;
+ const { directory: modelDir, precision } = models.recognizer;
const speakerModel = models.speaker && fs.existsSync(models.speaker) ? models.speaker : null;
- const precision: ModelPrecision = fs.existsSync(path.join(modelDir, "encoder.onnx"))
- ? "fp32"
- : "int8";
const suffix = precision === "fp32" ? ".onnx" : ".int8.onnx";
const decodingSettings = DECODING_SETTINGS[precision];
const mod = await import("sherpa-onnx-node");
diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.ts
index e8751f6539..7ad6223b15 100644
--- a/apps/desktop/src/services/transcription/index.spec.ts
+++ b/apps/desktop/src/services/transcription/index.spec.ts
@@ -16,6 +16,7 @@ const h = vi.hoisted(() => ({
coreListeners: new Map<string, () => void>(),
isDev: false,
installedResources: [] as string[],
+ transcriptionModel: null as string | null,
registerResource: vi.fn(),
log: { info: vi.fn(), error: vi.fn() },
listeners: null as WorkerListeners | null,
@@ -61,6 +62,9 @@ vi.mock("@triliumnext/core", () => ({
subscribe: (event: string, listener: () => void) => h.coreListeners.set(event, listener)
},
getLog: () => h.log,
+ options: {
+ getOptionOrNull: (name: string) => name === "transcriptionModel" ? h.transcriptionModel : null
+ },
protected_session: { isProtectedSessionAvailable: () => h.isProtectedSessionAvailable },
utils: { isDev: () => h.isDev }
}));
@@ -129,6 +133,7 @@ beforeEach(() => {
h.isProtectedSessionAvailable = false;
h.broadcast.length = 0;
h.isDev = false;
+ h.transcriptionModel = null;
h.installedResources = [
"transcription.parakeet", "transcription.silero-vad", "transcription.titanet"
];
@@ -183,7 +188,7 @@ describe("transcription IPC", () => {
expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({
noteId: "meeting",
models: {
- recognizer: "/data/resources/transcription.parakeet",
+ recognizer: { directory: "/data/resources/transcription.parakeet", precision: "int8" },
vad: path.join("/data/resources/transcription.silero-vad", "silero_vad.onnx"),
speaker: path.join("/data/resources/transcription.titanet", "nemo_en_titanet_small.onnx")
},
@@ -336,16 +341,31 @@ describe("transcription IPC", () => {
expect(h.broadcast).toEqual([]);
});
- it("registers the three models, and records without the speaker model", async () => {
+ it("registers every model, and records without the speaker model", async () => {
const invoke = await loadService();
- expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id))
- .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);
+ expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id)).toEqual([
+ "transcription.parakeet", "transcription.parakeet-fp32", "transcription.silero-vad",
+ "transcription.titanet"
+ ]);
h.installedResources = [ "transcription.parakeet", "transcription.silero-vad" ];
expect(await invoke("transcription-start", "meeting")).toBe("");
expect(h.worker.start.mock.calls[0][0].models.speaker).toBeNull();
});
+ it("records with the model the option selects, refusing while it is not downloaded", async () => {
+ const invoke = await loadService();
+ h.transcriptionModel = "parakeet-fp32";
+ expect(await invoke("transcription-start", "meeting"))
+ .toBe("transcription.model-not-downloaded");
+
+ h.installedResources.push("transcription.parakeet-fp32");
+ expect(await invoke("transcription-start", "meeting")).toBe("");
+ expect(h.worker.start.mock.calls[0][0].models.recognizer).toEqual({
+ directory: "/data/resources/transcription.parakeet-fp32", precision: "fp32"
+ });
+ });
+
it("explains why it could not start, ends the process and returns to idle", async () => {
const invoke = await loadService();
diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.ts
index 1418ad5048..8afa740192 100644
--- a/apps/desktop/src/services/transcription/index.ts
+++ b/apps/desktop/src/services/transcription/index.ts
@@ -1,5 +1,5 @@
import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";
-import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";
+import { becca, events, getLog, options, protected_session, utils as coreUtils } from "@triliumnext/core";
import dataDirs from "@triliumnext/server/src/services/data_dir.js";
import resources from "@triliumnext/server/src/services/resources.js";
import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";
@@ -62,7 +62,7 @@ export async function startRecording(noteId: unknown): Promise<string> {
if (typeof noteId !== "string" || !becca.getNote(noteId)) {
return t("transcription.note-not-found");
}
- const models = getModelPaths();
+ const models = getModelPaths(options.getOptionOrNull("transcriptionModel"));
if (!models) {
return t("transcription.model-not-downloaded");
}
diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.ts
index 37baa849e3..74160c944f 100644
--- a/apps/desktop/src/services/transcription/model.ts
+++ b/apps/desktop/src/services/transcription/model.ts
@@ -1,6 +1,10 @@
-import { TRANSCRIPTION_RESOURCES } from "@triliumnext/commons";
+import {
+ getTranscriptionModel, TRANSCRIPTION_MODELS as MODEL_CHOICES, TRANSCRIPTION_RESOURCES
+} from "@triliumnext/commons";
import dataDirs from "@triliumnext/server/src/services/data_dir.js";
-import resources, { type ResourceDefinition } from "@triliumnext/server/src/services/resources.js";
+import resources, {
+ type ResourceDefinition, type ResourceFile
+} from "@triliumnext/server/src/services/resources.js";
import fs from "fs";
import { t } from "i18next";
import path from "path";
@@ -8,8 +12,8 @@ import path from "path";
import type { ModelPaths } from "./protocol";
/**
- * A model directory installed by hand, which takes precedence over the downloaded models: it can
- * hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.
+ * A model directory installed by hand. It counts as each resource whose files it holds, in place of
+ * that resource's download.
*/
export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");
@@ -17,43 +21,77 @@ const VAD_FILE = "silero_vad.onnx";
const SPEAKER_FILE = "nemo_en_titanet_small.onnx";
const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";
+const PARAKEET_FP32 = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3/resolve/1a468a35cbba69418f126de829e75261dea4a4e4";
const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";
-/** Parakeet TDT 0.6B v3, int8: the files `loadSherpaEngine()` reads from its recognizer directory. */
-export const RECOGNIZER_MODEL: ResourceDefinition = {
- id: TRANSCRIPTION_RESOURCES.recognizer,
- title: () => t("transcription.recognizer-title"),
- description: () => t("transcription.recognizer-description"),
- icon: "bx bx-captions",
- license: "CC-BY-4.0",
- findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,
- files: [
- {
- path: "encoder.int8.onnx",
- url: `${PARAKEET}/encoder.int8.onnx`,
- size: 652184281,
- sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"
- },
- {
- path: "decoder.int8.onnx",
- url: `${PARAKEET}/decoder.int8.onnx`,
- size: 11845275,
- sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"
- },
- {
- path: "joiner.int8.onnx",
- url: `${PARAKEET}/joiner.int8.onnx`,
- size: 6355277,
- sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"
- },
- {
- path: "tokens.txt",
- url: `${PARAKEET}/tokens.txt`,
- size: 93939,
- sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"
- }
- ]
-};
+const PARAKEET_INT8_FILES: ResourceFile[] = [
+ {
+ path: "encoder.int8.onnx",
+ url: `${PARAKEET}/encoder.int8.onnx`,
+ size: 652184281,
+ sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"
+ },
+ {
+ path: "decoder.int8.onnx",
+ url: `${PARAKEET}/decoder.int8.onnx`,
+ size: 11845275,
+ sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"
+ },
+ {
+ path: "joiner.int8.onnx",
+ url: `${PARAKEET}/joiner.int8.onnx`,
+ size: 6355277,
+ sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"
+ },
+ {
+ path: "tokens.txt",
+ url: `${PARAKEET}/tokens.txt`,
+ size: 93939,
+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"
+ }
+];
+
+/** The full-precision export: `encoder.onnx` keeps its weights in `encoder.weights`. */
+const PARAKEET_FP32_FILES: ResourceFile[] = [
+ {
+ path: "encoder.onnx",
+ url: `${PARAKEET_FP32}/encoder.onnx`,
+ size: 41766257,
+ sha256: "3eed7ce424bf8339ad09233533c687e2dbd07e74ccf5027b5e7344019ea373b0"
+ },
+ {
+ path: "encoder.weights",
+ url: `${PARAKEET_FP32}/encoder.weights`,
+ size: 2435420160,
+ sha256: "3af3f51af5f2d01dbbf5af47d42c7962a2c205f11004254bb4f2b979862f39a8"
+ },
+ {
+ path: "decoder.onnx",
+ url: `${PARAKEET_FP32}/decoder.onnx`,
+ size: 47233743,
+ sha256: "d593cdb0e571f5a457ec2219af9968cbf6b0e8198e8f7839b40a8754593bf68c"
+ },
+ {
+ path: "joiner.onnx",
+ url: `${PARAKEET_FP32}/joiner.onnx`,
+ size: 25286330,
+ sha256: "b9b0bcf88ac571902e69a6536223ed2d94885e981b85045410f1403d53121a63"
+ },
+ {
+ path: "tokens.txt",
+ url: `${PARAKEET_FP32}/tokens.txt`,
+ size: 93939,
+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"
+ }
+];
+
+/** Parakeet TDT 0.6B v3 at each precision: the files `loadSherpaEngine()` reads as its recognizer. */
+export const RECOGNIZER_MODELS: ResourceDefinition[] = [
+ parakeet(MODEL_CHOICES["parakeet-int8"].resourceId, PARAKEET_INT8_FILES,
+ () => t("transcription.recognizer-int8-title")),
+ parakeet(MODEL_CHOICES["parakeet-fp32"].resourceId, PARAKEET_FP32_FILES,
+ () => t("transcription.recognizer-fp32-title"))
+];
export const VAD_MODEL: ResourceDefinition = {
id: TRANSCRIPTION_RESOURCES.vad,
@@ -88,22 +126,38 @@ export const SPEAKER_MODEL: ResourceDefinition = {
} ]
};
-export const TRANSCRIPTION_MODELS = [ RECOGNIZER_MODEL, VAD_MODEL, SPEAKER_MODEL ];
+export const TRANSCRIPTION_MODELS = [ ...RECOGNIZER_MODELS, VAD_MODEL, SPEAKER_MODEL ];
/**
- * Returns where the installed models are, or `null` while the recognizer or the VAD is missing.
+ * Returns where the installed models are for a recording with the model `modelId` (a key of
+ * `TRANSCRIPTION_MODELS` in commons), or `null` while its recognizer or the VAD is missing.
* Without the speaker model, the transcript labels no speakers.
*/
-export function getModelPaths(): ModelPaths | null {
- const recognizer = resources.getPath(RECOGNIZER_MODEL.id);
+export function getModelPaths(modelId: string | null): ModelPaths | null {
+ const model = getTranscriptionModel(modelId);
+ const recognizer = resources.getPath(model.resourceId);
const vad = resources.getPath(VAD_MODEL.id);
const speaker = resources.getPath(SPEAKER_MODEL.id);
if (!recognizer || !vad) {
return null;
}
return {
- recognizer,
+ recognizer: { directory: recognizer, precision: model.precision },
vad: path.join(vad, VAD_FILE),
speaker: speaker ? path.join(speaker, SPEAKER_FILE) : null
};
}
+
+function parakeet(id: string, files: ResourceFile[], title: () => string): ResourceDefinition {
+ return {
+ id,
+ title,
+ description: () => t("transcription.recognizer-description"),
+ icon: "bx bx-captions",
+ license: "CC-BY-4.0",
+ findLocalCopy: () => files.every((file) => fs.existsSync(path.join(LOCAL_MODEL_DIR, file.path)))
+ ? LOCAL_MODEL_DIR
+ : null,
+ files
+ };
+}
diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.ts
index 5023d4c25a..ffcc7e108e 100644
--- a/apps/desktop/src/services/transcription/protocol.ts
+++ b/apps/desktop/src/services/transcription/protocol.ts
@@ -1,4 +1,4 @@
-import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";
+import type { TranscriptionPrecision, TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";
/** Every source and engine works on mono 16 kHz samples. */
export const SAMPLE_RATE = 16000;
@@ -11,8 +11,8 @@ export interface Logger {
/** Where the engine's models are. */
export interface ModelPaths {
- /** The directory holding Parakeet: `tokens.txt` and the encoder, decoder and joiner. */
- recognizer: string;
+ /** Parakeet: the directory holding `tokens.txt` and the encoder, decoder and joiner. */
+ recognizer: { directory: string; precision: TranscriptionPrecision };
/** Silero VAD's `.onnx` file. */
vad: string;
/** The speaker embedding model's `.onnx` file, or `null` to label no speakers. */
diff --git a/apps/desktop/src/services/transcription/transcription_worker.spec.ts b/apps/desktop/src/services/transcription/transcription_worker.spec.ts
index ae95ffb834..56a6514d22 100644
--- a/apps/desktop/src/services/transcription/transcription_worker.spec.ts
+++ b/apps/desktop/src/services/transcription/transcription_worker.spec.ts
@@ -71,7 +71,11 @@ function capturePort() {
return Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });
}
-const MODELS = { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null };
+const MODELS = {
+ recognizer: { directory: "/models", precision: "int8" as const },
+ vad: "/models/silero_vad.onnx",
+ speaker: null
+};
const START: WorkerRequest = {
type: "start", noteId: "meeting", models: MODELS, recordingDirectory: null
diff --git a/apps/desktop/src/services/transcription/worker_client.spec.ts b/apps/desktop/src/services/transcription/worker_client.spec.ts
index 31f780caee..fc406fd0e8 100644
--- a/apps/desktop/src/services/transcription/worker_client.spec.ts
+++ b/apps/desktop/src/services/transcription/worker_client.spec.ts
@@ -55,7 +55,11 @@ afterEach(() => {
const REQUEST = {
noteId: "meeting",
- models: { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null },
+ models: {
+ recognizer: { directory: "/models", precision: "int8" as const },
+ vad: "/models/silero_vad.onnx",
+ speaker: null
+ },
recordingDirectory: null
};
diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.json
index d0ba962c0b..2670b1a5ea 100644
--- a/apps/server/src/assets/translations/en/server.json
+++ b/apps/server/src/assets/translations/en/server.json
@@ -573,7 +573,8 @@
"capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.",
"attachment-title": "Transcript",
"model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it.",
- "recognizer-title": "Parakeet TDT 0.6B v3",
+ "recognizer-int8-title": "Parakeet TDT 0.6B v3",
+ "recognizer-fp32-title": "Parakeet TDT 0.6B v3, high accuracy",
"recognizer-description": "A speech recognition model by NVIDIA that supports 25 European languages. Recognition runs entirely on this device, so your recordings never leave it.",
"vad-title": "Silero VAD",
"vad-description": "Used to detect speech in the audio, so that only the parts with speech are transcribed.",
diff --git a/packages/commons/src/lib/options_interface.ts b/packages/commons/src/lib/options_interface.ts
index fa2351da52..aa533a61d6 100644
--- a/packages/commons/src/lib/options_interface.ts
+++ b/packages/commons/src/lib/options_interface.ts
@@ -204,6 +204,11 @@ export interface OptionDefinitions extends KeyboardShortcutsOptions<KeyboardActi
codeLineWrapEnabled: boolean;
/** Whether a meeting transcript is shown without filler words and stutters. */
transcriptCleanupEnabled: boolean;
+ /**
+ * The speech recognition model a recording runs on, a key of `TRANSCRIPTION_MODELS`. Not synced:
+ * each device downloads and runs its own.
+ */
+ transcriptionModel: string;
leftPaneVisible: boolean;
rightPaneVisible: boolean;
nativeTitleBarVisible: boolean;
diff --git a/packages/commons/src/lib/transcript.spec.ts b/packages/commons/src/lib/transcript.spec.ts
index e48e727717..58f99e2a66 100644
--- a/packages/commons/src/lib/transcript.spec.ts
+++ b/packages/commons/src/lib/transcript.spec.ts
@@ -1,7 +1,8 @@
import { describe, expect, it } from "vitest";
import {
- cleanUtteranceText, getSpeakerKey, MAX_SPEAKER_NAME_LENGTH, parseTranscript, setSpeakerName,
+ cleanUtteranceText, getSpeakerKey, getTranscriptionModel, getTranscriptionResources,
+ MAX_SPEAKER_NAME_LENGTH, parseTranscript, setSpeakerName,
type Transcript
} from "./transcript.js";
@@ -82,3 +83,22 @@ describe("transcript", () => {
expect(performance.now() - startedAt).toBeLessThan(250);
});
});
+
+describe("transcription models", () => {
+ it("resolves the selected model, and an unknown or missing one to the int8 model", () => {
+ expect(getTranscriptionModel("parakeet-fp32"))
+ .toEqual({ id: "parakeet-fp32", resourceId: "transcription.parakeet-fp32", precision: "fp32" });
+ expect(getTranscriptionModel("parakeet-int8"))
+ .toEqual({ id: "parakeet-int8", resourceId: "transcription.parakeet", precision: "int8" });
+ expect(getTranscriptionModel("whisper").id).toBe("parakeet-int8");
+ expect(getTranscriptionModel(null).id).toBe("parakeet-int8");
+ expect(getTranscriptionModel("toString").id).toBe("parakeet-int8");
+ });
+
+ it("needs the selected recognizer, the VAD and the speaker model to record", () => {
+ expect(getTranscriptionResources("parakeet-fp32"))
+ .toEqual([ "transcription.parakeet-fp32", "transcription.silero-vad", "transcription.titanet" ]);
+ expect(getTranscriptionResources(undefined))
+ .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);
+ });
+});
diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.ts
index a7da875aba..60db8b3e1a 100644
--- a/packages/commons/src/lib/transcript.ts
+++ b/packages/commons/src/lib/transcript.ts
@@ -40,13 +40,45 @@ export const TRANSCRIPT_MIME = "application/json";
/** The longest speaker name a transcript accepts. */
export const MAX_SPEAKER_NAME_LENGTH = 100;
-/** The resources holding the models transcription runs on, defined in the desktop app. */
+/** The resources every recording needs besides its recognizer, defined in the desktop app. */
export const TRANSCRIPTION_RESOURCES = {
- recognizer: "transcription.parakeet",
vad: "transcription.silero-vad",
speaker: "transcription.titanet"
} as const;
+/** `fp32` is a full-precision model, `int8` a quantized one: smaller, faster and less accurate. */
+export type TranscriptionPrecision = "fp32" | "int8";
+
+/** A speech recognition model the `transcriptionModel` option can select. */
+export interface TranscriptionModel {
+ id: string;
+ /** The resource holding the model's files. */
+ resourceId: string;
+ precision: TranscriptionPrecision;
+}
+
+/** The selectable models, by ID: both are Parakeet TDT 0.6B v3, for 25 European languages. */
+export const TRANSCRIPTION_MODELS = {
+ "parakeet-int8": { id: "parakeet-int8", resourceId: "transcription.parakeet", precision: "int8" },
+ "parakeet-fp32": { id: "parakeet-fp32", resourceId: "transcription.parakeet-fp32", precision: "fp32" }
+} as const satisfies Record<string, TranscriptionModel>;
+
+export type TranscriptionModelId = keyof typeof TRANSCRIPTION_MODELS;
+
+export const DEFAULT_TRANSCRIPTION_MODEL: TranscriptionModelId = "parakeet-int8";
+
+/** Returns the model `id` names, or the default model when `id` names none. */
+export function getTranscriptionModel(id: string | null | undefined): TranscriptionModel {
+ return Object.hasOwn(TRANSCRIPTION_MODELS, id ?? "")
+ ? TRANSCRIPTION_MODELS[id as TranscriptionModelId]
+ : TRANSCRIPTION_MODELS[DEFAULT_TRANSCRIPTION_MODEL];
+}
+
+/** Returns the resources a recording with the model `modelId` needs. */
+export function getTranscriptionResources(modelId: string | null | undefined) {
+ return [ getTranscriptionModel(modelId).resourceId, ...Object.values(TRANSCRIPTION_RESOURCES) ];
+}
+
/** Reads a transcript attachment's content, or returns `null` for content of another shape. */
export function parseTranscript(content: string | null | undefined): Transcript | null {
if (!content) {
diff --git a/packages/trilium-core/src/routes/api/options.ts b/packages/trilium-core/src/routes/api/options.ts
index 64b7b3524e..93ca6adaeb 100644
--- a/packages/trilium-core/src/routes/api/options.ts
+++ b/packages/trilium-core/src/routes/api/options.ts
@@ -64,6 +64,7 @@ const ALLOWED_OPTIONS = new Set<OptionNames>([
"vimKeymapEnabled",
"codeLineWrapEnabled",
"transcriptCleanupEnabled",
+ "transcriptionModel",
"codeNotesMimeTypes",
"contentManagerSortOrder",
"contentManagerViewMode",
diff --git a/packages/trilium-core/src/services/options_init.ts b/packages/trilium-core/src/services/options_init.ts
index 5054dd9f6f..ffc9ed8830 100644
--- a/packages/trilium-core/src/services/options_init.ts
+++ b/packages/trilium-core/src/services/options_init.ts
@@ -1,4 +1,4 @@
-import { type KeyboardShortcutWithRequiredActionName, type OptionDefinitions, type OptionMap, type OptionNames, SANITIZER_DEFAULT_ALLOWED_TAGS } from "@triliumnext/commons";
+import { DEFAULT_TRANSCRIPTION_MODEL, type KeyboardShortcutWithRequiredActionName, type OptionDefinitions, type OptionMap, type OptionNames, SANITIZER_DEFAULT_ALLOWED_TAGS } from "@triliumnext/commons";
import appInfo from "./app_info.js";
import { getPlatform } from "./platform.js";
@@ -184,6 +184,7 @@ const defaultOptions: DefaultOption[] = [
{ name: "vimKeymapEnabled", value: "false", isSynced: false },
{ name: "codeLineWrapEnabled", value: "true", isSynced: false },
{ name: "transcriptCleanupEnabled", value: "true", isSynced: true },
+ { name: "transcriptionModel", value: DEFAULT_TRANSCRIPTION_MODEL, isSynced: false },
{ name: "codeNoteTabWidth", value: "4", isSynced: true },
{ name: "codeNoteIndentWithTabs", value: "false", isSynced: true },
{