Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(transcription): choose the speech recognition model in the settings
Elian Doranf6408f8f11df
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex bfc07627e7..c5738c77bc 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -55,9 +55,9 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri## The model-Users download the models from the app: the launcher calls `requireResources(Object.values(TRANSCRIPTION_RESOURCES))` before it creates the meeting note, which shows the download dialog for those missing. They are three resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>-<hash>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). The hash in the directory name covers every file's path and SHA-256, so changing a file of one model downloads that model alone again, into a new directory. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.+Users choose the recognizer under Options → Transcription: the non-synced option `transcriptionModel` holds a key of `TRANSCRIPTION_MODELS` (commons `transcript.ts`), `parakeet-int8` (the default, "Standard") or `parakeet-fp32` ("High accuracy"), each naming its resource and precision; an unknown value reads as the default (`getTranscriptionModel()`). The launcher calls `requireResources(getTranscriptionResources(<option>))` before it creates the meeting note, which shows the download dialog for those missing, and `startRecording()` resolves the same option through `getModelPaths()`. The models are resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>-<hash>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.parakeet-fp32` (2.5 GB, pinned to a Hugging Face revision), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). The hash in the directory name covers every file's path and SHA-256, so changing a file of one model downloads that model alone again, into a new directory. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.-A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself (Silero when `silero_vad.onnx` is there, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. Parakeet comes in two precisions, and `loadSherpaEngine()` takes the full-precision one when `encoder.onnx` is present, else the int8 one; it records which as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").+A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself, when it holds that model's files (each Parakeet precision when every one of its files is there, Silero when `silero_vad.onnx` is, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. `ModelPaths.recognizer` carries the precision with the directory, and `loadSherpaEngine()` loads that precision's files; it records it as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjsindex 7a1ff357b4..1198556ef9 100644--- a/.claude/skills/working-with-transcription/worker-test.mjs+++ b/.claude/skills/working-with-transcription/worker-test.mjs@@ -10,7 +10,8 @@// --source <name> which WAV of the recording to play, default system// --seconds <n> how much of it, default 120// --speed <n> how much faster than real time to post it, default 4-// --model <dir> the model directory, default apps/desktop/data/transcription-model+// --model <dir> the model directory, default apps/desktop/data/transcription-model; its+// full-precision Parakeet when it holds encoder.weights, else the int8 one// --out <file> writes the utterances there, in the transcript format//// Like `pnpm desktop:start`, it loads the TypeScript through NODE_OPTIONS="--import tsx", which@@ -125,7 +126,10 @@ async function runInElectron(config) {});const startedAt = performance.now();const models = {- recognizer: config.model,+ recognizer: {+ directory: config.model,+ precision: fs.existsSync(path.join(config.model, "encoder.weights")) ? "fp32" : "int8"+ },vad: path.join(config.model, "silero_vad.onnx"),speaker: path.join(config.model, "speaker", "nemo_en_titanet_small.onnx")};diff --git a/apps/client/src/translations/en/translation.json b/apps/client/src/translations/en/translation.jsonindex 62c579372c..bc90595c6d 100644--- a/apps/client/src/translations/en/translation.json+++ b/apps/client/src/translations/en/translation.json@@ -4625,6 +4625,10 @@"license": "{{license}} license"},"transcription_options": {- "placeholder": "Settings for transcribing meetings will appear here."+ "recognition_title": "Speech recognition",+ "european_languages": "European languages",+ "european_languages_description": "Parakeet TDT 0.6B v3 by NVIDIA recognizes 25 European languages and works out which one is spoken. High accuracy makes fewer mistakes, but it is a 2.4 GB download instead of 0.6 GB and needs about 1.6 GB more memory while recording. A model that is not downloaded yet is downloaded when you start a recording.",+ "model_standard": "Standard",+ "model_high_accuracy": "High accuracy"}}diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsxindex fb7434a93b..0ea8060998 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.spec.tsx@@ -9,7 +9,8 @@ const mocks = vi.hoisted(() => ({getInboxNote: vi.fn(),createNote: vi.fn(),showError: vi.fn(),- requireResources: vi.fn()+ requireResources: vi.fn(),+ transcriptionModel: null as string | null}));vi.mock("../../services/date_notes", () => ({@@ -21,6 +22,9 @@ vi.mock("../../services/note_create", () => ({vi.mock("../../services/dialog", () => ({requireResources: mocks.requireResources}));+vi.mock("../../services/options", () => ({+ default: { get: (name: string) => name === "transcriptionModel" ? mocks.transcriptionModel : null }+}));vi.mock("../../services/toast", () => ({default: { showError: mocks.showError }}));@@ -38,6 +42,7 @@ import RecordMeetingButton, { createMeetingNote, toggleRecording } from "./RecorbeforeEach(() => {vi.clearAllMocks();+ mocks.transcriptionModel = null;vi.useFakeTimers({ now: new Date(2026, 9, 13, 14, 30), toFake: [ "Date" ] });});@@ -110,6 +115,13 @@ describe("toggleRecording", () => {expect(transcription.start).toHaveBeenCalledOnce();});+ it("asks for the model the settings select", async () => {+ mocks.transcriptionModel = "parakeet-fp32";+ await toggleRecording("idle");+ expect(mocks.requireResources).toHaveBeenCalledExactlyOnceWith(+ [ "transcription.parakeet-fp32", "transcription.silero-vad", "transcription.titanet" ]);+ });+it("starts nothing when no meeting note could be created", async () => {mocks.getInboxNote.mockResolvedValue(null);vi.spyOn(console, "warn").mockImplementation(() => {});diff --git a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsxindex 44dd00127e..0d4d435739 100644--- a/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx+++ b/apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx@@ -1,7 +1,7 @@import "./RecordMeetingButton.css";import {- dayjs, MEETING_RECORDING_LABEL, TRANSCRIPTION_RESOURCES, type TranscriptionState,+ dayjs, getTranscriptionResources, MEETING_RECORDING_LABEL, type TranscriptionState,type TranscriptionStatus} from "@triliumnext/commons";import { useCallback, useEffect, useState } from "preact/hooks";@@ -10,6 +10,7 @@ import dateNoteService from "../../services/date_notes";import { requireResources } from "../../services/dialog";import { t } from "../../services/i18n";import noteCreateService from "../../services/note_create";+import options from "../../services/options";import toast from "../../services/toast";import {LaunchBarActionButton,@@ -104,7 +105,7 @@ export async function toggleRecording(status: TranscriptionStatus) {if (status !== "idle") {return;}- if (!await requireResources(Object.values(TRANSCRIPTION_RESOURCES))) {+ if (!await requireResources(getTranscriptionResources(options.get("transcriptionModel")))) {return;}diff --git a/apps/client/src/widgets/type_widgets/options/transcription.spec.tsx b/apps/client/src/widgets/type_widgets/options/transcription.spec.tsxnew file mode 100644index 0000000000..769bfd43ba--- /dev/null+++ b/apps/client/src/widgets/type_widgets/options/transcription.spec.tsx@@ -0,0 +1,80 @@+import { render } from "preact";+import { act } from "preact/test-utils";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";++const mocks = vi.hoisted(() => ({+ electron: true,+ stored: "" as string,+ saved: [] as string[]+}));++vi.mock("../../../services/utils", async (importOriginal) => ({+ ...(await importOriginal<typeof import("../../../services/utils")>()),+ isElectron: () => mocks.electron+}));++vi.mock("../../../services/i18n", () => ({ t: (key: string) => key }));++vi.mock("../../react/hooks", async (importOriginal) => ({+ ...(await importOriginal<typeof import("../../react/hooks")>()),+ useTriliumOption: (name: string) => [+ name === "transcriptionModel" ? mocks.stored : "",+ (value: string) => void mocks.saved.push(value)+ ]+}));++vi.mock("./components/OptionsPageHeader", () => ({ default: () => <div className="header-stub" /> }));++import TranscriptionSettings from "./transcription";++let host: HTMLElement;++beforeEach(() => {+ mocks.electron = true;+ mocks.stored = "parakeet-fp32";+ mocks.saved = [];+ host = document.body.appendChild(document.createElement("div"));+});++afterEach(() => {+ render(null, host);+ document.body.innerHTML = "";+});++function open() {+ act(() => {+ render(<TranscriptionSettings />, host);+ });+ return host.querySelector<HTMLSelectElement>("select");+}++describe("TranscriptionSettings", () => {+ it("selects the model a recording runs on, and saves another choice", () => {+ const select = open();+ if (!select) {+ throw new Error("The model selector is missing.");+ }+ expect([ ...select.options ].map((option) => [ option.value, option.textContent ])).toEqual([+ [ "parakeet-int8", "transcription_options.model_standard" ],+ [ "parakeet-fp32", "transcription_options.model_high_accuracy" ]+ ]);+ expect(select.value).toBe("parakeet-fp32");++ act(() => {+ select.value = "parakeet-int8";+ select.dispatchEvent(new Event("change", { bubbles: true }));+ });+ expect(mocks.saved).toEqual([ "parakeet-int8" ]);+ });++ it("shows the default model for a value it does not know", () => {+ mocks.stored = "whisper";+ expect(open()?.value).toBe("parakeet-int8");+ });++ it("explains that the page applies to desktop builds only, elsewhere", () => {+ mocks.electron = false;+ expect(open()).toBeNull();+ expect(host.textContent).toContain("desktop.web_placeholder");+ });+});diff --git a/apps/client/src/widgets/type_widgets/options/transcription.tsx b/apps/client/src/widgets/type_widgets/options/transcription.tsxindex 41aa6f9c60..b78fa0f5e5 100644--- a/apps/client/src/widgets/type_widgets/options/transcription.tsx+++ b/apps/client/src/widgets/type_widgets/options/transcription.tsx@@ -1,16 +1,54 @@+import { getTranscriptionModel, TRANSCRIPTION_MODELS, type TranscriptionModelId } from "@triliumnext/commons";+import { t } from "../../../services/i18n";import { isElectron } from "../../../services/utils";+import { Card, OptionCardSection } from "../../react/Card";+import FormSelect from "../../react/FormSelect";+import { useTriliumOption } from "../../react/hooks";import NoItems from "../../react/NoItems";import OptionsPageHeader from "./components/OptionsPageHeader";export default function TranscriptionSettings() {+ if (!isElectron()) {+ return (+ <>+ <OptionsPageHeader />+ <NoItems icon="bx bx-microphone" text={t("desktop.web_placeholder")} />+ </>+ );+ }+return (<><OptionsPageHeader />- <NoItems- icon="bx bx-microphone"- text={isElectron() ? t("transcription_options.placeholder") : t("desktop.web_placeholder")}- />+ <RecognitionSettings /></>);}++function RecognitionSettings() {+ const [ model, setModel ] = useTriliumOption("transcriptionModel");+ const labels: Record<TranscriptionModelId, string> = {+ "parakeet-int8": t("transcription_options.model_standard"),+ "parakeet-fp32": t("transcription_options.model_high_accuracy")+ };++ return (+ <Card heading={t("transcription_options.recognition_title")}>+ <OptionCardSection+ name="transcription-model"+ label={t("transcription_options.european_languages")}+ description={t("transcription_options.european_languages_description")}+ >+ <FormSelect+ name="transcription-model"+ currentValue={getTranscriptionModel(model).id}+ onChange={setModel}+ keyProperty="id"+ titleProperty="label"+ values={Object.values(TRANSCRIPTION_MODELS).map(({ id }) => ({ id, label: labels[id] }))}+ />+ </OptionCardSection>+ </Card>+ );+}diff --git a/apps/desktop/src/services/transcription/engine.spec.ts b/apps/desktop/src/services/transcription/engine.spec.tsindex 59e9bc733f..d37e34868d 100644--- a/apps/desktop/src/services/transcription/engine.spec.ts+++ b/apps/desktop/src/services/transcription/engine.spec.ts@@ -103,7 +103,7 @@ vi.mock("fs", () => ({const MODEL_DIR = "/models/parakeet";const MODELS = {- recognizer: MODEL_DIR,+ recognizer: { directory: MODEL_DIR, precision: "int8" as const },vad: "/models/silero/silero_vad.onnx",speaker: "/models/titanet/nemo_en_titanet_small.onnx"};@@ -119,6 +119,8 @@ beforeEach(() => {describe("loadSherpaEngine", () => {it("loads Parakeet and Silero from their paths and reports their settings", async () => {+ // The precision comes from the paths, whatever else the directory holds.+ h.installed = [ "encoder.onnx" ];const engine = await loadSherpaEngine(MODELS);expect(h.recognizerConfig).toMatchObject({@@ -152,9 +154,10 @@ describe("loadSherpaEngine", () => {expect(h.decoded).toEqual([ samples ]);});- it("prefers the full-precision model, decoding it without a blank penalty", async () => {- h.installed = [ "encoder.onnx" ];- const engine = await loadSherpaEngine(MODELS);+ it("loads the full-precision model when given, decoding it without a blank penalty", async () => {+ const engine = await loadSherpaEngine({+ ...MODELS, recognizer: { directory: MODEL_DIR, precision: "fp32" }+ });expect(h.recognizerConfig).toMatchObject({modelConfig: {diff --git a/apps/desktop/src/services/transcription/engine.ts b/apps/desktop/src/services/transcription/engine.tsindex 425fcd5119..e70c22d559 100644--- a/apps/desktop/src/services/transcription/engine.ts+++ b/apps/desktop/src/services/transcription/engine.ts@@ -1,3 +1,4 @@+import type { TranscriptionPrecision } from "@triliumnext/commons";import fs from "fs";import path from "path";import type { SpeakerEmbeddingExtractor, Vad } from "sherpa-onnx-node";@@ -62,15 +63,12 @@ const RECOGNIZER_SETTINGS = {provider: "cpu"};-/** `fp32` is the full-precision export (2.4 GB), `int8` the quantized one (641 MB). */-type ModelPrecision = "fp32" | "int8";-/*** The int8 model emits nothing over some stretches of speech and drops whole clauses from segments* with pauses; `blankPenalty` counters that. The full-precision model drops none, and the penalty* only adds stray words to its transcript.*/-const DECODING_SETTINGS: Record<ModelPrecision, Record<string, unknown>> = {+const DECODING_SETTINGS: Record<TranscriptionPrecision, Record<string, unknown>> = {fp32: { decodingMethod: "greedy_search" },int8: { decodingMethod: "greedy_search", blankPenalty: 0.5 }};@@ -107,16 +105,13 @@ const VAD_SETTINGS = {/*** Loads the local engine from `models`: Silero VAD for segmentation and Parakeet TDT 0.6B v3 for* recognition, both through sherpa-onnx, and the speaker model when it is given and installed.- * Recognition uses the full-precision model (`encoder.onnx` with `encoder.weights`) when the- * recognizer directory holds it, else the int8 one (`encoder.int8.onnx`). Recognition runs on the- * libuv thread pool; the VAD and the speaker embeddings run on the calling thread.+ * The recognizer's precision picks its files: `encoder.onnx` (with `encoder.weights`) for `fp32`,+ * `encoder.int8.onnx` for `int8`. Recognition runs on the libuv thread pool; the VAD and the+ * speaker embeddings run on the calling thread.*/export async function loadSherpaEngine(models: ModelPaths): Promise<SpeechEngine> {- const modelDir = models.recognizer;+ const { directory: modelDir, precision } = models.recognizer;const speakerModel = models.speaker && fs.existsSync(models.speaker) ? models.speaker : null;- const precision: ModelPrecision = fs.existsSync(path.join(modelDir, "encoder.onnx"))- ? "fp32"- : "int8";const suffix = precision === "fp32" ? ".onnx" : ".int8.onnx";const decodingSettings = DECODING_SETTINGS[precision];const mod = await import("sherpa-onnx-node");diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.tsindex e8751f6539..7ad6223b15 100644--- a/apps/desktop/src/services/transcription/index.spec.ts+++ b/apps/desktop/src/services/transcription/index.spec.ts@@ -16,6 +16,7 @@ const h = vi.hoisted(() => ({coreListeners: new Map<string, () => void>(),isDev: false,installedResources: [] as string[],+ transcriptionModel: null as string | null,registerResource: vi.fn(),log: { info: vi.fn(), error: vi.fn() },listeners: null as WorkerListeners | null,@@ -61,6 +62,9 @@ vi.mock("@triliumnext/core", () => ({subscribe: (event: string, listener: () => void) => h.coreListeners.set(event, listener)},getLog: () => h.log,+ options: {+ getOptionOrNull: (name: string) => name === "transcriptionModel" ? h.transcriptionModel : null+ },protected_session: { isProtectedSessionAvailable: () => h.isProtectedSessionAvailable },utils: { isDev: () => h.isDev }}));@@ -129,6 +133,7 @@ beforeEach(() => {h.isProtectedSessionAvailable = false;h.broadcast.length = 0;h.isDev = false;+ h.transcriptionModel = null;h.installedResources = ["transcription.parakeet", "transcription.silero-vad", "transcription.titanet"];@@ -183,7 +188,7 @@ describe("transcription IPC", () => {expect(h.worker.start).toHaveBeenCalledExactlyOnceWith({noteId: "meeting",models: {- recognizer: "/data/resources/transcription.parakeet",+ recognizer: { directory: "/data/resources/transcription.parakeet", precision: "int8" },vad: path.join("/data/resources/transcription.silero-vad", "silero_vad.onnx"),speaker: path.join("/data/resources/transcription.titanet", "nemo_en_titanet_small.onnx")},@@ -336,16 +341,31 @@ describe("transcription IPC", () => {expect(h.broadcast).toEqual([]);});- it("registers the three models, and records without the speaker model", async () => {+ it("registers every model, and records without the speaker model", async () => {const invoke = await loadService();- expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id))- .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);+ expect(h.registerResource.mock.calls[0].map(({ id }: { id: string }) => id)).toEqual([+ "transcription.parakeet", "transcription.parakeet-fp32", "transcription.silero-vad",+ "transcription.titanet"+ ]);h.installedResources = [ "transcription.parakeet", "transcription.silero-vad" ];expect(await invoke("transcription-start", "meeting")).toBe("");expect(h.worker.start.mock.calls[0][0].models.speaker).toBeNull();});+ it("records with the model the option selects, refusing while it is not downloaded", async () => {+ const invoke = await loadService();+ h.transcriptionModel = "parakeet-fp32";+ expect(await invoke("transcription-start", "meeting"))+ .toBe("transcription.model-not-downloaded");++ h.installedResources.push("transcription.parakeet-fp32");+ expect(await invoke("transcription-start", "meeting")).toBe("");+ expect(h.worker.start.mock.calls[0][0].models.recognizer).toEqual({+ directory: "/data/resources/transcription.parakeet-fp32", precision: "fp32"+ });+ });+it("explains why it could not start, ends the process and returns to idle", async () => {const invoke = await loadService();diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.tsindex 1418ad5048..8afa740192 100644--- a/apps/desktop/src/services/transcription/index.ts+++ b/apps/desktop/src/services/transcription/index.ts@@ -1,5 +1,5 @@import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";-import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";+import { becca, events, getLog, options, protected_session, utils as coreUtils } from "@triliumnext/core";import dataDirs from "@triliumnext/server/src/services/data_dir.js";import resources from "@triliumnext/server/src/services/resources.js";import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";@@ -62,7 +62,7 @@ export async function startRecording(noteId: unknown): Promise<string> {if (typeof noteId !== "string" || !becca.getNote(noteId)) {return t("transcription.note-not-found");}- const models = getModelPaths();+ const models = getModelPaths(options.getOptionOrNull("transcriptionModel"));if (!models) {return t("transcription.model-not-downloaded");}diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.tsindex 37baa849e3..74160c944f 100644--- a/apps/desktop/src/services/transcription/model.ts+++ b/apps/desktop/src/services/transcription/model.ts@@ -1,6 +1,10 @@-import { TRANSCRIPTION_RESOURCES } from "@triliumnext/commons";+import {+ getTranscriptionModel, TRANSCRIPTION_MODELS as MODEL_CHOICES, TRANSCRIPTION_RESOURCES+} from "@triliumnext/commons";import dataDirs from "@triliumnext/server/src/services/data_dir.js";-import resources, { type ResourceDefinition } from "@triliumnext/server/src/services/resources.js";+import resources, {+ type ResourceDefinition, type ResourceFile+} from "@triliumnext/server/src/services/resources.js";import fs from "fs";import { t } from "i18next";import path from "path";@@ -8,8 +12,8 @@ import path from "path";import type { ModelPaths } from "./protocol";/**- * A model directory installed by hand, which takes precedence over the downloaded models: it can- * hold the full-precision Parakeet model, which `loadSherpaEngine()` prefers.+ * A model directory installed by hand. It counts as each resource whose files it holds, in place of+ * that resource's download.*/export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");@@ -17,43 +21,77 @@ const VAD_FILE = "silero_vad.onnx";const SPEAKER_FILE = "nemo_en_titanet_small.onnx";const PARAKEET = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/resolve/2bda32ec70b097a55adaa07d9a7173915b43cc78";+const PARAKEET_FP32 = "https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3/resolve/1a468a35cbba69418f126de829e75261dea4a4e4";const SHERPA_RELEASES = "https://github.com/k2-fsa/sherpa-onnx/releases/download";-/** Parakeet TDT 0.6B v3, int8: the files `loadSherpaEngine()` reads from its recognizer directory. */-export const RECOGNIZER_MODEL: ResourceDefinition = {- id: TRANSCRIPTION_RESOURCES.recognizer,- title: () => t("transcription.recognizer-title"),- description: () => t("transcription.recognizer-description"),- icon: "bx bx-captions",- license: "CC-BY-4.0",- findLocalCopy: () => fs.existsSync(LOCAL_MODEL_DIR) ? LOCAL_MODEL_DIR : null,- files: [- {- path: "encoder.int8.onnx",- url: `${PARAKEET}/encoder.int8.onnx`,- size: 652184281,- sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"- },- {- path: "decoder.int8.onnx",- url: `${PARAKEET}/decoder.int8.onnx`,- size: 11845275,- sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"- },- {- path: "joiner.int8.onnx",- url: `${PARAKEET}/joiner.int8.onnx`,- size: 6355277,- sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"- },- {- path: "tokens.txt",- url: `${PARAKEET}/tokens.txt`,- size: 93939,- sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"- }- ]-};+const PARAKEET_INT8_FILES: ResourceFile[] = [+ {+ path: "encoder.int8.onnx",+ url: `${PARAKEET}/encoder.int8.onnx`,+ size: 652184281,+ sha256: "acfc2b4456377e15d04f0243af540b7fe7c992f8d898d751cf134c3a55fd2247"+ },+ {+ path: "decoder.int8.onnx",+ url: `${PARAKEET}/decoder.int8.onnx`,+ size: 11845275,+ sha256: "179e50c43d1a9de79c8a24149a2f9bac6eb5981823f2a2ed88d655b24248db4e"+ },+ {+ path: "joiner.int8.onnx",+ url: `${PARAKEET}/joiner.int8.onnx`,+ size: 6355277,+ sha256: "3164c13fc2821009440d20fcb5fdc78bff28b4db2f8d0f0b329101719c0948b3"+ },+ {+ path: "tokens.txt",+ url: `${PARAKEET}/tokens.txt`,+ size: 93939,+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"+ }+];++/** The full-precision export: `encoder.onnx` keeps its weights in `encoder.weights`. */+const PARAKEET_FP32_FILES: ResourceFile[] = [+ {+ path: "encoder.onnx",+ url: `${PARAKEET_FP32}/encoder.onnx`,+ size: 41766257,+ sha256: "3eed7ce424bf8339ad09233533c687e2dbd07e74ccf5027b5e7344019ea373b0"+ },+ {+ path: "encoder.weights",+ url: `${PARAKEET_FP32}/encoder.weights`,+ size: 2435420160,+ sha256: "3af3f51af5f2d01dbbf5af47d42c7962a2c205f11004254bb4f2b979862f39a8"+ },+ {+ path: "decoder.onnx",+ url: `${PARAKEET_FP32}/decoder.onnx`,+ size: 47233743,+ sha256: "d593cdb0e571f5a457ec2219af9968cbf6b0e8198e8f7839b40a8754593bf68c"+ },+ {+ path: "joiner.onnx",+ url: `${PARAKEET_FP32}/joiner.onnx`,+ size: 25286330,+ sha256: "b9b0bcf88ac571902e69a6536223ed2d94885e981b85045410f1403d53121a63"+ },+ {+ path: "tokens.txt",+ url: `${PARAKEET_FP32}/tokens.txt`,+ size: 93939,+ sha256: "d58544679ea4bc6ac563d1f545eb7d474bd6cfa467f0a6e2c1dc1c7d37e3c35d"+ }+];++/** Parakeet TDT 0.6B v3 at each precision: the files `loadSherpaEngine()` reads as its recognizer. */+export const RECOGNIZER_MODELS: ResourceDefinition[] = [+ parakeet(MODEL_CHOICES["parakeet-int8"].resourceId, PARAKEET_INT8_FILES,+ () => t("transcription.recognizer-int8-title")),+ parakeet(MODEL_CHOICES["parakeet-fp32"].resourceId, PARAKEET_FP32_FILES,+ () => t("transcription.recognizer-fp32-title"))+];export const VAD_MODEL: ResourceDefinition = {id: TRANSCRIPTION_RESOURCES.vad,@@ -88,22 +126,38 @@ export const SPEAKER_MODEL: ResourceDefinition = {} ]};-export const TRANSCRIPTION_MODELS = [ RECOGNIZER_MODEL, VAD_MODEL, SPEAKER_MODEL ];+export const TRANSCRIPTION_MODELS = [ ...RECOGNIZER_MODELS, VAD_MODEL, SPEAKER_MODEL ];/**- * Returns where the installed models are, or `null` while the recognizer or the VAD is missing.+ * Returns where the installed models are for a recording with the model `modelId` (a key of+ * `TRANSCRIPTION_MODELS` in commons), or `null` while its recognizer or the VAD is missing.* Without the speaker model, the transcript labels no speakers.*/-export function getModelPaths(): ModelPaths | null {- const recognizer = resources.getPath(RECOGNIZER_MODEL.id);+export function getModelPaths(modelId: string | null): ModelPaths | null {+ const model = getTranscriptionModel(modelId);+ const recognizer = resources.getPath(model.resourceId);const vad = resources.getPath(VAD_MODEL.id);const speaker = resources.getPath(SPEAKER_MODEL.id);if (!recognizer || !vad) {return null;}return {- recognizer,+ recognizer: { directory: recognizer, precision: model.precision },vad: path.join(vad, VAD_FILE),speaker: speaker ? path.join(speaker, SPEAKER_FILE) : null};}++function parakeet(id: string, files: ResourceFile[], title: () => string): ResourceDefinition {+ return {+ id,+ title,+ description: () => t("transcription.recognizer-description"),+ icon: "bx bx-captions",+ license: "CC-BY-4.0",+ findLocalCopy: () => files.every((file) => fs.existsSync(path.join(LOCAL_MODEL_DIR, file.path)))+ ? LOCAL_MODEL_DIR+ : null,+ files+ };+}diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.tsindex 5023d4c25a..ffcc7e108e 100644--- a/apps/desktop/src/services/transcription/protocol.ts+++ b/apps/desktop/src/services/transcription/protocol.ts@@ -1,4 +1,4 @@-import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";+import type { TranscriptionPrecision, TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";/** Every source and engine works on mono 16 kHz samples. */export const SAMPLE_RATE = 16000;@@ -11,8 +11,8 @@ export interface Logger {/** Where the engine's models are. */export interface ModelPaths {- /** The directory holding Parakeet: `tokens.txt` and the encoder, decoder and joiner. */- recognizer: string;+ /** Parakeet: the directory holding `tokens.txt` and the encoder, decoder and joiner. */+ recognizer: { directory: string; precision: TranscriptionPrecision };/** Silero VAD's `.onnx` file. */vad: string;/** The speaker embedding model's `.onnx` file, or `null` to label no speakers. */diff --git a/apps/desktop/src/services/transcription/transcription_worker.spec.ts b/apps/desktop/src/services/transcription/transcription_worker.spec.tsindex ae95ffb834..56a6514d22 100644--- a/apps/desktop/src/services/transcription/transcription_worker.spec.ts+++ b/apps/desktop/src/services/transcription/transcription_worker.spec.ts@@ -71,7 +71,11 @@ function capturePort() {return Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });}-const MODELS = { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null };+const MODELS = {+ recognizer: { directory: "/models", precision: "int8" as const },+ vad: "/models/silero_vad.onnx",+ speaker: null+};const START: WorkerRequest = {type: "start", noteId: "meeting", models: MODELS, recordingDirectory: nulldiff --git a/apps/desktop/src/services/transcription/worker_client.spec.ts b/apps/desktop/src/services/transcription/worker_client.spec.tsindex 31f780caee..fc406fd0e8 100644--- a/apps/desktop/src/services/transcription/worker_client.spec.ts+++ b/apps/desktop/src/services/transcription/worker_client.spec.ts@@ -55,7 +55,11 @@ afterEach(() => {const REQUEST = {noteId: "meeting",- models: { recognizer: "/models", vad: "/models/silero_vad.onnx", speaker: null },+ models: {+ recognizer: { directory: "/models", precision: "int8" as const },+ vad: "/models/silero_vad.onnx",+ speaker: null+ },recordingDirectory: null};diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.jsonindex d0ba962c0b..2670b1a5ea 100644--- a/apps/server/src/assets/translations/en/server.json+++ b/apps/server/src/assets/translations/en/server.json@@ -573,7 +573,8 @@"capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.","attachment-title": "Transcript","model-not-downloaded": "The speech recognition model is not downloaded yet. Start a recording from the launcher to download it.",- "recognizer-title": "Parakeet TDT 0.6B v3",+ "recognizer-int8-title": "Parakeet TDT 0.6B v3",+ "recognizer-fp32-title": "Parakeet TDT 0.6B v3, high accuracy","recognizer-description": "A speech recognition model by NVIDIA that supports 25 European languages. Recognition runs entirely on this device, so your recordings never leave it.","vad-title": "Silero VAD","vad-description": "Used to detect speech in the audio, so that only the parts with speech are transcribed.",diff --git a/packages/commons/src/lib/options_interface.ts b/packages/commons/src/lib/options_interface.tsindex fa2351da52..aa533a61d6 100644--- a/packages/commons/src/lib/options_interface.ts+++ b/packages/commons/src/lib/options_interface.ts@@ -204,6 +204,11 @@ export interface OptionDefinitions extends KeyboardShortcutsOptions<KeyboardActicodeLineWrapEnabled: boolean;/** Whether a meeting transcript is shown without filler words and stutters. */transcriptCleanupEnabled: boolean;+ /**+ * The speech recognition model a recording runs on, a key of `TRANSCRIPTION_MODELS`. Not synced:+ * each device downloads and runs its own.+ */+ transcriptionModel: string;leftPaneVisible: boolean;rightPaneVisible: boolean;nativeTitleBarVisible: boolean;diff --git a/packages/commons/src/lib/transcript.spec.ts b/packages/commons/src/lib/transcript.spec.tsindex e48e727717..58f99e2a66 100644--- a/packages/commons/src/lib/transcript.spec.ts+++ b/packages/commons/src/lib/transcript.spec.ts@@ -1,7 +1,8 @@import { describe, expect, it } from "vitest";import {- cleanUtteranceText, getSpeakerKey, MAX_SPEAKER_NAME_LENGTH, parseTranscript, setSpeakerName,+ cleanUtteranceText, getSpeakerKey, getTranscriptionModel, getTranscriptionResources,+ MAX_SPEAKER_NAME_LENGTH, parseTranscript, setSpeakerName,type Transcript} from "./transcript.js";@@ -82,3 +83,22 @@ describe("transcript", () => {expect(performance.now() - startedAt).toBeLessThan(250);});});++describe("transcription models", () => {+ it("resolves the selected model, and an unknown or missing one to the int8 model", () => {+ expect(getTranscriptionModel("parakeet-fp32"))+ .toEqual({ id: "parakeet-fp32", resourceId: "transcription.parakeet-fp32", precision: "fp32" });+ expect(getTranscriptionModel("parakeet-int8"))+ .toEqual({ id: "parakeet-int8", resourceId: "transcription.parakeet", precision: "int8" });+ expect(getTranscriptionModel("whisper").id).toBe("parakeet-int8");+ expect(getTranscriptionModel(null).id).toBe("parakeet-int8");+ expect(getTranscriptionModel("toString").id).toBe("parakeet-int8");+ });++ it("needs the selected recognizer, the VAD and the speaker model to record", () => {+ expect(getTranscriptionResources("parakeet-fp32"))+ .toEqual([ "transcription.parakeet-fp32", "transcription.silero-vad", "transcription.titanet" ]);+ expect(getTranscriptionResources(undefined))+ .toEqual([ "transcription.parakeet", "transcription.silero-vad", "transcription.titanet" ]);+ });+});diff --git a/packages/commons/src/lib/transcript.ts b/packages/commons/src/lib/transcript.tsindex a7da875aba..60db8b3e1a 100644--- a/packages/commons/src/lib/transcript.ts+++ b/packages/commons/src/lib/transcript.ts@@ -40,13 +40,45 @@ export const TRANSCRIPT_MIME = "application/json";/** The longest speaker name a transcript accepts. */export const MAX_SPEAKER_NAME_LENGTH = 100;-/** The resources holding the models transcription runs on, defined in the desktop app. */+/** The resources every recording needs besides its recognizer, defined in the desktop app. */export const TRANSCRIPTION_RESOURCES = {- recognizer: "transcription.parakeet",vad: "transcription.silero-vad",speaker: "transcription.titanet"} as const;+/** `fp32` is a full-precision model, `int8` a quantized one: smaller, faster and less accurate. */+export type TranscriptionPrecision = "fp32" | "int8";++/** A speech recognition model the `transcriptionModel` option can select. */+export interface TranscriptionModel {+ id: string;+ /** The resource holding the model's files. */+ resourceId: string;+ precision: TranscriptionPrecision;+}++/** The selectable models, by ID: both are Parakeet TDT 0.6B v3, for 25 European languages. */+export const TRANSCRIPTION_MODELS = {+ "parakeet-int8": { id: "parakeet-int8", resourceId: "transcription.parakeet", precision: "int8" },+ "parakeet-fp32": { id: "parakeet-fp32", resourceId: "transcription.parakeet-fp32", precision: "fp32" }+} as const satisfies Record<string, TranscriptionModel>;++export type TranscriptionModelId = keyof typeof TRANSCRIPTION_MODELS;++export const DEFAULT_TRANSCRIPTION_MODEL: TranscriptionModelId = "parakeet-int8";++/** Returns the model `id` names, or the default model when `id` names none. */+export function getTranscriptionModel(id: string | null | undefined): TranscriptionModel {+ return Object.hasOwn(TRANSCRIPTION_MODELS, id ?? "")+ ? TRANSCRIPTION_MODELS[id as TranscriptionModelId]+ : TRANSCRIPTION_MODELS[DEFAULT_TRANSCRIPTION_MODEL];+}++/** Returns the resources a recording with the model `modelId` needs. */+export function getTranscriptionResources(modelId: string | null | undefined) {+ return [ getTranscriptionModel(modelId).resourceId, ...Object.values(TRANSCRIPTION_RESOURCES) ];+}+/** Reads a transcript attachment's content, or returns `null` for content of another shape. */export function parseTranscript(content: string | null | undefined): Transcript | null {if (!content) {diff --git a/packages/trilium-core/src/routes/api/options.ts b/packages/trilium-core/src/routes/api/options.tsindex 64b7b3524e..93ca6adaeb 100644--- a/packages/trilium-core/src/routes/api/options.ts+++ b/packages/trilium-core/src/routes/api/options.ts@@ -64,6 +64,7 @@ const ALLOWED_OPTIONS = new Set<OptionNames>(["vimKeymapEnabled","codeLineWrapEnabled","transcriptCleanupEnabled",+ "transcriptionModel","codeNotesMimeTypes","contentManagerSortOrder","contentManagerViewMode",diff --git a/packages/trilium-core/src/services/options_init.ts b/packages/trilium-core/src/services/options_init.tsindex 5054dd9f6f..ffc9ed8830 100644--- a/packages/trilium-core/src/services/options_init.ts+++ b/packages/trilium-core/src/services/options_init.ts@@ -1,4 +1,4 @@-import { type KeyboardShortcutWithRequiredActionName, type OptionDefinitions, type OptionMap, type OptionNames, SANITIZER_DEFAULT_ALLOWED_TAGS } from "@triliumnext/commons";+import { DEFAULT_TRANSCRIPTION_MODEL, type KeyboardShortcutWithRequiredActionName, type OptionDefinitions, type OptionMap, type OptionNames, SANITIZER_DEFAULT_ALLOWED_TAGS } from "@triliumnext/commons";import appInfo from "./app_info.js";import { getPlatform } from "./platform.js";@@ -184,6 +184,7 @@ const defaultOptions: DefaultOption[] = [{ name: "vimKeymapEnabled", value: "false", isSynced: false },{ name: "codeLineWrapEnabled", value: "true", isSynced: false },{ name: "transcriptCleanupEnabled", value: "true", isSynced: true },+ { name: "transcriptionModel", value: DEFAULT_TRANSCRIPTION_MODEL, isSynced: false },{ name: "codeNoteTabWidth", value: "4", isSynced: true },{ name: "codeNoteIndentWithTabs", value: "false", isSynced: true },{