Skip to content

Build your personal knowledge base with Trilium Notes

refactor(transcription): load the models only from their downloads

Elian Dorane5cbd03787f1
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index 84de52c1d1..85dccaf895 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -37,7 +37,7 @@ The main process opens the capture window, starts the transcription process and
|---|---|
| `apps/desktop/src/services/transcription/capture.ts` | Main process: `CaptureWindow`, the hidden capture window and its page; microphone and loopback system audio; `CAPTURE_PROCESSOR`, the worklet that cuts each source into chunks (`capture-test.mjs` reads it from this file) |
| `…/index.ts` | Main process: IPC handlers, the state broadcast, `getRecordingDirectory()`; registers the model resource, starts a recording's process and capture window and connects them |
-| `…/model.ts` | Main process: the three models as downloadable resources (`RECOGNIZER_MODEL`, `VAD_MODEL`, `SPEAKER_MODEL`: URLs pinned to a revision, sizes, SHA-256), `LOCAL_MODEL_DIR`, and `getModelPaths()`, which resolves the `ModelPaths` a recording starts with |
+| `…/model.ts` | Main process: the three models as downloadable resources (`RECOGNIZER_MODELS`, one per precision, `VAD_MODEL`, `SPEAKER_MODEL`: URLs pinned to a revision, sizes, SHA-256) and `getModelPaths()`, which resolves the `ModelPaths` a recording starts with |
| `…/worker_client.ts` | Main process: `TranscriptionWorker`, which forks the transcription process, relays its events and stops or kills it |
| `…/attachment_sink.ts` | Main process: the `transcript` attachment, rewritten after every utterance |
| `…/protocol.ts` | Both: the messages between the processes, `SAMPLE_RATE`, `Logger` |
@@ -57,22 +57,11 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri
Users choose the recognizer under Options → Transcription: the non-synced option `transcriptionModel` holds a key of `TRANSCRIPTION_MODELS` (commons `transcript.ts`), `parakeet-int8` (the default, "Standard") or `parakeet-fp32` ("High accuracy"), each naming its resource and precision; an unknown value reads as the default (`getTranscriptionModel()`). The synced option `transcriptionSpeakersEnabled` (the "Identify speakers" switch, default on) decides whether a recording labels speakers. The launcher calls `requireResources(getTranscriptionResources(<model>, <speakers>))` before it creates the meeting note, which shows the download dialog for those missing (TitaNet only while the switch is on), and `startRecording()` resolves the same two options through `getModelPaths()`, which passes `speaker: null` while the switch is off. The models are resources of the resource manager (`apps/server/src/services/resources.ts`), each in `<data dir>/resources/<id>-<hash>/` and checked against the SHA-256 in `model.ts`: `transcription.parakeet` (int8, 670 MB), `transcription.parakeet-fp32` (2.5 GB, pinned to a Hugging Face revision), `transcription.silero-vad` (0.6 MB) and `transcription.titanet` (40 MB). The hash in the directory name covers every file's path and SHA-256, so changing a file of one model downloads that model alone again, into a new directory. A recording needs Parakeet and Silero; without TitaNet it labels no speakers.
-A directory installed by hand, `<data dir>/transcription-model` (`LOCAL_MODEL_DIR`), takes precedence over the downloads, each model for itself, when it holds that model's files (each Parakeet precision when every one of its files is there, Silero when `silero_vad.onnx` is, TitaNet when `speaker/nemo_en_titanet_small.onnx` is); for `pnpm desktop:start` that is `apps/desktop/data/transcription-model`, which the scripts below read. `ModelPaths.recognizer` carries the precision with the directory, and `loadSherpaEngine()` loads that precision's files; it records it as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far").
+`ModelPaths.recognizer` carries the precision with the directory, and `loadSherpaEngine()` loads that precision's files; it records it as `precision` in the engine settings and applies `blankPenalty` to int8 only (see "Findings so far"). Full precision loads in ~2 s to ~2.3 GB RSS.
-- **Full precision** (recommended): `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3` on Hugging Face, not in the sherpa releases — `encoder.onnx` + `encoder.weights`, `decoder.onnx`, `joiner.onnx`, `tokens.txt` (2.4 GB). Loads in ~2 s to ~2.3 GB RSS.
-- **int8**: the `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` archive from the `asr-models` release of `k2-fsa/sherpa-onnx` — `encoder.int8.onnx`, `decoder.int8.onnx`, `joiner.int8.onnx`, `tokens.txt` (641 MB).
-- `silero_vad.onnx` from the `asr-models` release.
+The scripts below read the downloads of `pnpm desktop:start`, in `apps/desktop/data/resources` (`findResource()` in `recordings.mjs`, which takes the most recently installed directory of a resource); download a model by starting a recording there with it selected, or point `--model` at another Parakeet directory. Files copied into a resource's directory by hand are adopted on the next download once their SHA-256 matches, so a copy from another machine needs no new download.
-```bash
-cd apps/desktop/data && mkdir -p transcription-model/speaker && cd transcription-model
-H=https://huggingface.co/csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3/resolve/main
-R=https://github.com/k2-fsa/sherpa-onnx/releases/download
-for f in tokens.txt decoder.onnx joiner.onnx encoder.onnx encoder.weights; do curl -fL -o $f "$H/$f"; done
-curl -fL -o silero_vad.onnx "$R/asr-models/silero_vad.onnx"
-curl -fL -o speaker/nemo_en_titanet_small.onnx "$R/speaker-recongition-models/nemo_en_titanet_small.onnx"
-```
-
-Without either directory, starting fails with `transcription.model-not-downloaded`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.
+A recording started without its models fails with `transcription.model-not-downloaded`. `replay.mjs` takes the precision `recording.json` names (int8 for recordings made before it was recorded) and falls back to the other.
## Recordings for troubleshooting
@@ -126,7 +115,7 @@ The export has errors of its own (it wrote "MicroEdge" and "SortMap", and split
## Speaker detection
-The app labels the system source's utterances live when `speaker/nemo_en_titanet_small.onnx` is installed: `EmbeddingSpeakerTracker` in `engine.ts` embeds each transcribed utterance and `SpeakerClusterer` (`speakers.ts`, settings `SPEAKER_SETTINGS`) assigns it, giving the utterance a `speaker` number that `TranscriptView` shows as "Speaker n". The microphone is never labelled. It runs synchronously in the transcription process, holding back the next chunks but nothing else. `replay.mjs` does not label speakers; use `diarize.mjs`.
+The app labels the system source's utterances live while TitaNet (`transcription.titanet`) is installed and "Identify speakers" is on: `EmbeddingSpeakerTracker` in `engine.ts` embeds each transcribed utterance and `SpeakerClusterer` (`speakers.ts`, settings `SPEAKER_SETTINGS`) assigns it, giving the utterance a `speaker` number that `TranscriptView` shows as "Speaker n". The microphone is never labelled. It runs synchronously in the transcription process, holding back the next chunks but nothing else. `replay.mjs` does not label speakers; use `diarize.mjs`.
**Turns within an utterance** (`TURN_SETTINGS`): a VAD segment holds both sides of a quick exchange, since a reply needs no 0.75 s pause. Once a speaker is known, an utterance of 4 s or more is embedded again in 2 s windows 1 s apart, each matched against the known speakers (`SpeakerClusterer.match()`, which moves no centroid); where another label holds for two windows (`findSpeakerChanges()`), the utterance is cut half a second into the new run, moved to the nearest word start within 1 s, preferring one after a sentence end (`snapToWordBoundary()`, on the token times Parakeet returns with `transcribe()`). `RecordingSession` writes each turn as an utterance of its own, its text the words starting in it. That costs one embedding per second of a long utterance, ~0.1–0.3 s for a 20 s one.
@@ -143,9 +132,9 @@ node $S/diarize.mjs latest --mode offline --clusters 2 # when the speaker
node $S/diarize.mjs latest --embedding campplus --threads 2
```
-Models, in `<data dir>/transcription-model/speaker/`, from `k2-fsa/sherpa-onnx` releases:
+TitaNet is the app's download (`transcription.titanet`). The models only `diarize.mjs` uses go in `apps/desktop/data/diarization-models/` (or `--models <dir>`), from `k2-fsa/sherpa-onnx` releases:
-- `nemo_en_titanet_small.onnx` (40 MB) and `3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx` (30 MB) from `speaker-recongition-models` (the tag is misspelled upstream);
+- `3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx` (30 MB) from `speaker-recongition-models` (the tag is misspelled upstream);
- `segmentation.onnx` (6 MB): `model.onnx` of `sherpa-onnx-pyannote-segmentation-3-0.tar.bz2` from `speaker-segmentation-models`, renamed.
`SpeakerEmbeddingExtractor.compute()` and `OfflineSpeakerDiarization.process()` are **synchronous**, with no async variant. Live labelling therefore runs in the transcription process, never in the main process, which also serves the backend. The offline method, ~20 s per two minutes, would stall even that process's capture for its duration.
@@ -175,8 +164,8 @@ Two distinct voices on a clean recording are the easy case; similar voices, more
Without a BPE vocabulary, sherpa looks a hotword up as a single entry of `tokens.txt`, so an ordinary word fails ("Cannot find ID for token androgynous") and **one failing term drops the whole list** for that utterance. The Parakeet archives ship no vocabulary; `hotwords.mjs` builds one:
```bash
-node $S/hotwords.mjs vocab # <model dir>/bpe.vocab from NVIDIA's tokenizer.json
-node $S/hotwords.mjs vocab apps/desktop/data/transcription-model-fp32
+node $S/hotwords.mjs vocab # bpe.vocab beside the downloaded Parakeet
+node $S/hotwords.mjs vocab <model dir> # or beside another one
node $S/hotwords.mjs check "heap/STM32/Iconiq" # how each term splits, and which are risky
node $S/replay.mjs <rec> --model <dir> --label hot --hotwords "heap/STM32/Stéphane" \
--set recognizer.decodingMethod=modified_beam_search --set recognizer.maxActivePaths=4 \
@@ -220,7 +209,7 @@ node $S/worker-test.mjs latest # 120 s of system.wav at
node $S/worker-test.mjs latest --source microphone --seconds 60 --speed 8
```
-`worker-test.mjs` runs the real `TranscriptionWorker` in Electron: it starts the transcription process on `apps/desktop/data/transcription-model` (or `--model`, writing the utterances to `--out`), and a hidden page posts a recording's WAV through a `MessageChannelMain` port as the capture page does. It prints the engine's load time, the utterances (with speakers), how long `stop` took and whether the process exited, and the main process's event-loop lag throughout. On two minutes of a meeting's system track: ready after 1.3 s, lag p50 0.1 ms, p99 1.1 ms, max 25 ms. Before the split, speaker labelling alone held the main thread 15–100 ms per utterance. On Windows, Electron launched by it prints nothing and runs into the timeout (unresolved); there, `RecordingSession` and `loadSherpaEngine()` can be driven in plain Node with `node --import tsx` instead, feeding the WAV in 4096-sample chunks, which tests everything but the process boundary.
+`worker-test.mjs` runs the real `TranscriptionWorker` in Electron: it starts the transcription process on the app's downloads in `apps/desktop/data/resources` (or the Parakeet of `--model`; `--out` writes the utterances), and a hidden page posts a recording's WAV through a `MessageChannelMain` port as the capture page does. It prints the engine's load time, the utterances (with speakers), how long `stop` took and whether the process exited, and the main process's event-loop lag throughout. On two minutes of a meeting's system track: ready after 1.3 s, lag p50 0.1 ms, p99 1.1 ms, max 25 ms. Before the split, speaker labelling alone held the main thread 15–100 ms per utterance. On Windows, Electron launched by it prints nothing and runs into the timeout (unresolved); there, `RecordingSession` and `loadSherpaEngine()` can be driven in plain Node with `node --import tsx` instead, feeding the WAV in 4096-sample chunks, which tests everything but the process boundary.
In development the process loads `transcription_worker.ts` through the `NODE_OPTIONS="--import tsx"` it inherits from `scripts/electron-start.mts`; built, `worker_client.ts` finds `transcription_worker.cjs` beside `main.mjs`. Packaged builds do not ship `sherpa-onnx-node` yet (it is external to every bundle, but not copied or unpacked from the asar).
diff --git a/.claude/skills/working-with-transcription/diarize.mjs b/.claude/skills/working-with-transcription/diarize.mjs
index 7ce8da2478..851a9a04f0 100644
--- a/.claude/skills/working-with-transcription/diarize.mjs
+++ b/.claude/skills/working-with-transcription/diarize.mjs
@@ -20,20 +20,23 @@
// --clusters <n> offline: the number of speakers, when known
// --cluster-threshold <n> offline: clustering threshold otherwise (default 0.5)
// --threads <n> threads per model (default 1)
+// --models <dir> where CAM++ and segmentation.onnx are
+// (default apps/desktop/data/diarization-models)
//
-// Models: <data dir>/transcription-model/speaker/ holds the embedding models and
-// segmentation.onnx; see SKILL.md, "Speaker detection".
+// Models: TitaNet is the app's download, in apps/desktop/data/resources; CAM++ and
+// segmentation.onnx, which only this script uses, go in --models. See SKILL.md, "Speaker
+// detection".
import fs from "node:fs";
import path from "node:path";
-import { listTranscripts, parseArgs, readTranscript, resolveRecording } from "./recordings.mjs";
+import {
+ listTranscripts, MODEL_RESOURCES, parseArgs, readTranscript, requireResource, resolveRecording
+} from "./recordings.mjs";
import { loadSherpa, SAMPLE_RATE } from "./sherpa.mjs";
-const EMBEDDING_MODELS = {
- titanet: "nemo_en_titanet_small.onnx",
- campplus: "3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx"
-};
+const DIARIZATION_MODELS_DIR = path.resolve(
+ import.meta.dirname, "../../../apps/desktop/data/diarization-models");
/** An utterance whose second speaker talks this long within it counts as a speaker change. */
const CHANGE_SECONDS = 1;
@@ -48,11 +51,8 @@ const transcriptLabel = options.transcript
?? (listTranscripts(recording.directory).has("greedy") ? "greedy" : "live");
const mode = options.mode ?? "both";
const threads = Number(options.threads ?? 1);
-const speakerDir = path.join(
- path.dirname(path.dirname(recording.directory)), "transcription-model", "speaker");
-const embeddingModel = EMBEDDING_MODELS[options.embedding ?? "titanet"]
- ? path.join(speakerDir, EMBEDDING_MODELS[options.embedding ?? "titanet"])
- : options.embedding;
+const modelsDir = options.models ?? DIARIZATION_MODELS_DIR;
+const embeddingModel = getEmbeddingModel(options.embedding ?? "titanet");
if (!fs.existsSync(embeddingModel)) {
fail(`No embedding model at ${embeddingModel}.`);
}
@@ -170,7 +170,7 @@ function labelOffline() {
const loadStarted = performance.now();
const diarization = new sherpa.OfflineSpeakerDiarization({
segmentation: {
- pyannote: { model: path.join(speakerDir, "segmentation.onnx") },
+ pyannote: { model: path.join(modelsDir, "segmentation.onnx") },
numThreads: threads
},
embedding: { model: embeddingModel, numThreads: threads },
@@ -286,3 +286,14 @@ function fail(message) {
console.error(message);
process.exit(1);
}
+
+/** The embedding model file `name` stands for: titanet, campplus, or a path. */
+function getEmbeddingModel(name) {
+ if (name === "titanet") {
+ return path.join(requireResource(MODEL_RESOURCES.speaker), "nemo_en_titanet_small.onnx");
+ }
+ if (name === "campplus") {
+ return path.join(modelsDir, "3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx");
+ }
+ return name;
+}
diff --git a/.claude/skills/working-with-transcription/hotwords.mjs b/.claude/skills/working-with-transcription/hotwords.mjs
index f8e2fe26fc..a1a5287c1a 100644
--- a/.claude/skills/working-with-transcription/hotwords.mjs
+++ b/.claude/skills/working-with-transcription/hotwords.mjs
@@ -7,8 +7,9 @@
// sherpa-onnx encodes a hotword into tokens with a BPE vocabulary (`modelingUnit: "bpe"`,
// `bpeVocab`), which the Parakeet archives do not ship. `vocab` builds one from NVIDIA's
// tokenizer.json, scoring each piece by the rank of the merge that forms it, so that hotwords are
-// split as the model itself splits words. The model directory defaults to
-// <data dir>/transcription-model; tokens.txt there must match the tokenizer.
+// split as the model itself splits words. The model directory defaults to the app's download of
+// Parakeet, full precision first; tokens.txt there must match the tokenizer. The app ignores
+// files it did not download, so bpe.vocab can stay beside them.
//
// `check` prints how each hotword is split and flags the ones whose first piece is a common word
// on its own: sherpa's NeMo beam search keeps the bonus of a hotword's first piece even when the
@@ -18,9 +19,9 @@
import fs from "node:fs";
import path from "node:path";
+import { findResource, MODEL_RESOURCES, requireResource } from "./recordings.mjs";
+
const TOKENIZER_URL = "https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/resolve/main/tokenizer.json";
-const REPO = path.resolve(import.meta.dirname, "../../..");
-const DEFAULT_MODEL_DIR = path.join(REPO, "apps/desktop/data/transcription-model");
const UNMERGED_SCORE = -100000;
/** Words a first piece is risky as: frequent enough that boosting them changes ordinary speech. */
const COMMON_WORDS = new Set([
@@ -31,9 +32,9 @@ const COMMON_WORDS = new Set([
const [ command, ...args ] = process.argv.slice(2);
if (command === "vocab") {
- await writeVocab(args[0] ?? DEFAULT_MODEL_DIR);
+ await writeVocab(args[0] ?? getDefaultModelDir());
} else if (command === "check" && args[0]) {
- check(args[0], args[1] ?? DEFAULT_MODEL_DIR);
+ check(args[0], args[1] ?? getDefaultModelDir());
} else {
console.error("Usage: node hotwords.mjs vocab [model dir] | check \"a/b/c\" [model dir]");
process.exit(1);
@@ -112,3 +113,8 @@ function readTokens(modelDir) {
return { piece: line.slice(0, separator), id: Number(line.slice(separator + 1)) };
});
}
+
+/** The app's download of Parakeet, full precision when it is there. */
+function getDefaultModelDir() {
+ return findResource(MODEL_RESOURCES.fp32) ?? requireResource(MODEL_RESOURCES.int8);
+}
diff --git a/.claude/skills/working-with-transcription/recordings.mjs b/.claude/skills/working-with-transcription/recordings.mjs
index 7271974556..e73e0b0261 100644
--- a/.claude/skills/working-with-transcription/recordings.mjs
+++ b/.claude/skills/working-with-transcription/recordings.mjs
@@ -1,4 +1,4 @@
-// Shared by replay.mjs and compare.mjs: finding a development recording and its transcripts.
+// Shared by the scripts: finding a development recording, its transcripts, and the models.
import fs from "node:fs";
import path from "node:path";
@@ -8,6 +8,42 @@ export const TRANSCRIPT_VERSION = 1;
/** Where `pnpm desktop:start` keeps recordings (its TRILIUM_DATA_DIR is apps/desktop/data). */
const DEV_RECORDINGS_DIR = path.resolve(
import.meta.dirname, "../../../apps/desktop/data/transcription-recordings");
+/** Where `pnpm desktop:start` downloads resources, one `<id>-<hash>` directory each. */
+const DEV_RESOURCES_DIR = path.resolve(import.meta.dirname, "../../../apps/desktop/data/resources");
+
+/** The resource IDs of the models, as `apps/desktop/src/services/transcription/model.ts` registers them. */
+export const MODEL_RESOURCES = {
+ int8: "transcription.parakeet",
+ fp32: "transcription.parakeet-fp32",
+ vad: "transcription.silero-vad",
+ speaker: "transcription.titanet"
+};
+
+/**
+ * Returns the directory the app downloaded the resource `id` into, the most recently installed
+ * when several versions are there, or `null` when none finished downloading.
+ */
+export function findResource(id) {
+ if (!fs.existsSync(DEV_RESOURCES_DIR)) {
+ return null;
+ }
+ const installed = fs.readdirSync(DEV_RESOURCES_DIR)
+ .filter((name) => name.slice(0, name.lastIndexOf("-")) === id)
+ .map((name) => path.join(DEV_RESOURCES_DIR, name, ".verified.json"))
+ .filter((record) => fs.existsSync(record))
+ .sort((a, b) => fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs);
+ return installed.length ? path.dirname(installed[0]) : null;
+}
+
+/** Like {@link findResource}, but fails with a hint to download the resource from the app. */
+export function requireResource(id) {
+ const directory = findResource(id);
+ if (!directory) {
+ throw new Error(`${id} is not downloaded to ${DEV_RESOURCES_DIR}. `
+ + "Start a recording in `pnpm desktop:start` to download it.");
+ }
+ return directory;
+}
/**
* Splits argv into positional arguments and `--name value` options. Names in `multiple` collect
diff --git a/.claude/skills/working-with-transcription/replay.mjs b/.claude/skills/working-with-transcription/replay.mjs
index e39bd33774..28c72102a6 100644
--- a/.claude/skills/working-with-transcription/replay.mjs
+++ b/.claude/skills/working-with-transcription/replay.mjs
@@ -9,9 +9,12 @@
// --set recognizer.maxActivePaths=8 --set vad.threshold=0.4
// (the value is parsed as JSON when it parses, else kept as a string)
// --hotwords <a/b/c> per-utterance hotwords, separated by "/" (needs modified_beam_search)
-// --model <dir> model directory (default: <data dir>/transcription-model); takes
-// <part>.onnx for a full-precision recording, else <part>.int8.onnx,
-// falling back to the other, for encoder, decoder and joiner
+// --model <dir> Parakeet directory (default: the app's download of the precision the
+// recording used, else of the other); takes <part>.onnx for a
+// full-precision recording, else <part>.int8.onnx, falling back to the
+// other, for encoder, decoder and joiner
+//
+// Silero VAD is read from the app's download, in apps/desktop/data/resources.
// --source <name> replay only microphone or system
// --pad <seconds> extend each segment by this much audio on both sides before decoding
// --force overwrite an existing transcript.<label>.json
@@ -22,7 +25,8 @@ import fs from "node:fs";
import path from "node:path";
import {
- parseArgs, resolveRecording, TRANSCRIPT_VERSION, transcriptPath
+ findResource, MODEL_RESOURCES, parseArgs, requireResource, resolveRecording,
+ TRANSCRIPT_VERSION, transcriptPath
} from "./recordings.mjs";
import { loadSherpa, SAMPLE_RATE, segmentSource } from "./sherpa.mjs";
@@ -49,8 +53,8 @@ const engine = structuredClone(recording.manifest.engine ?? {});
for (const assignment of options.set ?? []) {
applySetting(engine, assignment);
}
-const modelDirectory = options.model
- ?? path.join(path.dirname(path.dirname(recording.directory)), "transcription-model");
+const modelDirectory = options.model ?? findRecognizer(recording.manifest.engine?.precision);
+const vadModel = path.join(requireResource(MODEL_RESOURCES.vad), "silero_vad.onnx");
const hotwords = options.hotwords;
const padSamples = Math.round(Number(options.pad ?? 0) * SAMPLE_RATE);
if (hotwords && engine.recognizer?.decodingMethod !== "modified_beam_search") {
@@ -78,7 +82,7 @@ for (const [ source, { file, firstSampleOffsetMs } ] of Object.entries(recording
let decodeMs = 0;
let segmentCount = 0;
- for (const segment of segmentSource(sherpa, wave.samples, engine.vad ?? {}, modelDirectory)) {
+ for (const segment of segmentSource(sherpa, wave.samples, engine.vad ?? {}, vadModel)) {
const stream = hotwords ? recognizer.createStream(hotwords) : recognizer.createStream();
const samples = padSamples
? wave.samples.subarray(Math.max(0, segment.start - padSamples),
@@ -140,6 +144,14 @@ function buildRecognizerConfig(settings, modelDir) {
return config;
}
+/** The app's download of Parakeet at `precision` (int8 when unknown), else at the other. */
+function findRecognizer(precision) {
+ const [ preferred, fallback ] = precision === "fp32"
+ ? [ MODEL_RESOURCES.fp32, MODEL_RESOURCES.int8 ]
+ : [ MODEL_RESOURCES.int8, MODEL_RESOURCES.fp32 ];
+ return findResource(preferred) ?? requireResource(fallback);
+}
+
/**
* The file of the precision the recording used (`engine.precision`, int8 for recordings made before
* the app recorded it), else the other one.
diff --git a/.claude/skills/working-with-transcription/sherpa.mjs b/.claude/skills/working-with-transcription/sherpa.mjs
index 36c124d07e..f1e526d1b9 100644
--- a/.claude/skills/working-with-transcription/sherpa.mjs
+++ b/.claude/skills/working-with-transcription/sherpa.mjs
@@ -16,10 +16,10 @@ export function loadSherpa() {
* Cuts one source into utterances with Silero VAD, as the live `VadSegmenter` does: 512-sample
* windows, then a flush at the end.
*/
-export function* segmentSource(sherpa, samples, vadSettings, modelDir) {
+export function* segmentSource(sherpa, samples, vadSettings, vadModel) {
const windowSize = vadSettings.windowSize ?? 512;
const vad = new sherpa.Vad({
- sileroVad: { model: path.join(modelDir, "silero_vad.onnx"), ...vadSettings, windowSize },
+ sileroVad: { model: vadModel, ...vadSettings, windowSize },
sampleRate: SAMPLE_RATE,
numThreads: 1,
provider: "cpu"
diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjs
index 1198556ef9..5da3e4a9ed 100644
--- a/.claude/skills/working-with-transcription/worker-test.mjs
+++ b/.claude/skills/working-with-transcription/worker-test.mjs
@@ -10,8 +10,10 @@
// --source <name> which WAV of the recording to play, default system
// --seconds <n> how much of it, default 120
// --speed <n> how much faster than real time to post it, default 4
-// --model <dir> the model directory, default apps/desktop/data/transcription-model; its
-// full-precision Parakeet when it holds encoder.weights, else the int8 one
+// --model <dir> the Parakeet directory, default the app's download of the full-precision
+// model, else of the int8 one; full precision when it holds encoder.weights
+//
+// Silero VAD and TitaNet are read from the app's downloads, in apps/desktop/data/resources.
// --out <file> writes the utterances there, in the transcript format
//
// Like `pnpm desktop:start`, it loads the TypeScript through NODE_OPTIONS="--import tsx", which
@@ -24,11 +26,12 @@ import os from "node:os";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
-import { parseArgs, resolveRecording } from "./recordings.mjs";
+import {
+ findResource, MODEL_RESOURCES, parseArgs, requireResource, resolveRecording
+} from "./recordings.mjs";
const REPO = path.resolve(import.meta.dirname, "../../..");
const DESKTOP = path.join(REPO, "apps/desktop");
-const MODEL_DIR = path.join(DESKTOP, "data/transcription-model");
const CHUNK_SAMPLES = 4096;
const RATE = 16000;
const SELF = fileURLToPath(import.meta.url);
@@ -55,7 +58,10 @@ async function launch() {
source,
seconds: Number(options.seconds ?? 120),
speed: Number(options.speed ?? 4),
- model: path.resolve(options.model ?? MODEL_DIR),
+ model: path.resolve(options.model
+ ?? findResource(MODEL_RESOURCES.fp32) ?? requireResource(MODEL_RESOURCES.int8)),
+ vad: path.join(requireResource(MODEL_RESOURCES.vad), "silero_vad.onnx"),
+ speaker: findResource(MODEL_RESOURCES.speaker),
out: options.out ? path.resolve(options.out) : null
};
const { getElectronPath, getNixLdLibraryPath, isNixOS } = await import(pathToFileURL(path.join(REPO, "scripts/utils.mts")).href);
@@ -130,8 +136,8 @@ async function runInElectron(config) {
directory: config.model,
precision: fs.existsSync(path.join(config.model, "encoder.weights")) ? "fp32" : "int8"
},
- vad: path.join(config.model, "silero_vad.onnx"),
- speaker: path.join(config.model, "speaker", "nemo_en_titanet_small.onnx")
+ vad: config.vad,
+ speaker: config.speaker && path.join(config.speaker, "nemo_en_titanet_small.onnx")
};
await worker.start({ noteId: "worker-test", models, recordingDirectory: null });
const readyMs = Math.round(performance.now() - startedAt);
diff --git a/apps/client/src/widgets/type_widgets/options/resources.spec.tsx b/apps/client/src/widgets/type_widgets/options/resources.spec.tsx
index 3478933d31..f0ad33ae90 100644
--- a/apps/client/src/widgets/type_widgets/options/resources.spec.tsx
+++ b/apps/client/src/widgets/type_widgets/options/resources.spec.tsx
@@ -52,8 +52,7 @@ beforeEach(() => {
mocks.resources = [
resource({ id: "test.installed", installed: true, downloadedSize: 10 * MIB, directory: "/data/resources/test.installed-abc" }),
resource({ id: "test.partial", downloadedSize: 4 * MIB }),
- resource({ id: "test.missing" }),
- resource({ id: "test.by-hand", installed: true, directory: "/data/transcription-model" })
+ resource({ id: "test.missing" })
];
mocks.requests = 0;
mocks.handlers = [];
@@ -77,14 +76,13 @@ describe("ResourcesSettings", () => {
const rows = [ ...host.querySelectorAll(".resources-item") ];
expect(rows.map((row) => row.querySelector(".tn-card-option-title")?.textContent))
- .toEqual([ "test.installed", "test.partial", "test.by-hand" ]);
+ .toEqual([ "test.installed", "test.partial" ]);
expect(rows.map((row) => row.querySelector(".resources-status small")?.textContent)).toEqual([
"resources_options.installed",
- `resources_options.partially_downloaded {"downloaded":"4.00 MiB","size":"10 MiB"}`,
- "resources_options.installed"
+ `resources_options.partially_downloaded {"downloaded":"4.00 MiB","size":"10 MiB"}`
]);
expect(rows.map((row) => row.querySelector(".directory-stub")?.textContent ?? null))
- .toEqual([ "/data/resources/test.installed-abc", null, "/data/transcription-model" ]);
+ .toEqual([ "/data/resources/test.installed-abc", null ]);
expect(host.querySelector(".resources-total .resources-size")?.textContent).toBe("14 MiB");
});
diff --git a/apps/client/src/widgets/type_widgets/options/resources.tsx b/apps/client/src/widgets/type_widgets/options/resources.tsx
index cc45536eb1..aa6e9b7eb0 100644
--- a/apps/client/src/widgets/type_widgets/options/resources.tsx
+++ b/apps/client/src/widgets/type_widgets/options/resources.tsx
@@ -39,10 +39,7 @@ export default function ResourcesSettings() {
);
}
-/**
- * The resources installed or partly downloaded, including copies installed by hand, which have no
- * downloaded bytes. Fetched again whenever a download ends.
- */
+/** The resources installed or partly downloaded, fetched again whenever a download ends. */
function useResources() {
const [ resources, setResources ] = useState<ResourceInfo[]>();
@@ -50,7 +47,7 @@ function useResources() {
const refresh = () => {
void server.get<ResourcesStatusResponse>("resources")
.then((status) => setResources(status.resources.filter(
- ({ installed, downloadedSize }) => installed || downloadedSize > 0)));
+ ({ downloadedSize }) => downloadedSize > 0)));
};
const handler = (message: WebSocketMessage) => {
if (message.type === "resource-download" && message.download.status !== "downloading") {
diff --git a/apps/desktop/src/services/transcription/model.ts b/apps/desktop/src/services/transcription/model.ts
index c01506d4af..fe44583d64 100644
--- a/apps/desktop/src/services/transcription/model.ts
+++ b/apps/desktop/src/services/transcription/model.ts
@@ -1,22 +1,14 @@
import {
getTranscriptionModel, TRANSCRIPTION_MODELS as MODEL_CHOICES, TRANSCRIPTION_RESOURCES
} from "@triliumnext/commons";
-import dataDirs from "@triliumnext/server/src/services/data_dir.js";
import resources, {
type ResourceDefinition, type ResourceFile
} from "@triliumnext/server/src/services/resources.js";
-import fs from "fs";
import { t } from "i18next";
import path from "path";
import type { ModelPaths } from "./protocol";
-/**
- * A model directory installed by hand. It counts as each resource whose files it holds, in place of
- * that resource's download.
- */
-export const LOCAL_MODEL_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcription-model");
-
const VAD_FILE = "silero_vad.onnx";
const SPEAKER_FILE = "nemo_en_titanet_small.onnx";
@@ -99,7 +91,6 @@ export const VAD_MODEL: ResourceDefinition = {
description: () => t("transcription.vad-description"),
icon: "bx bx-pulse",
license: "MIT",
- findLocalCopy: () => fs.existsSync(path.join(LOCAL_MODEL_DIR, VAD_FILE)) ? LOCAL_MODEL_DIR : null,
files: [ {
path: VAD_FILE,
url: `${SHERPA_RELEASES}/asr-models/${VAD_FILE}`,
@@ -114,10 +105,6 @@ export const SPEAKER_MODEL: ResourceDefinition = {
description: () => t("transcription.speaker-description"),
icon: "bx bx-user-voice",
license: "Apache-2.0",
- findLocalCopy: () => {
- const directory = path.join(LOCAL_MODEL_DIR, "speaker");
- return fs.existsSync(path.join(directory, SPEAKER_FILE)) ? directory : null;
- },
files: [ {
path: SPEAKER_FILE,
url: `${SHERPA_RELEASES}/speaker-recongition-models/${SPEAKER_FILE}`,
@@ -156,9 +143,6 @@ function parakeet(id: string, files: ResourceFile[], title: () => string): Resou
description: () => t("transcription.recognizer-description"),
icon: "bx bx-captions",
license: "CC-BY-4.0",
- findLocalCopy: () => files.every((file) => fs.existsSync(path.join(LOCAL_MODEL_DIR, file.path)))
- ? LOCAL_MODEL_DIR
- : null,
files
};
}
diff --git a/apps/server/src/services/resources.spec.ts b/apps/server/src/services/resources.spec.ts
index d6acc2b65d..de6db87b14 100644
--- a/apps/server/src/services/resources.spec.ts
+++ b/apps/server/src/services/resources.spec.ts
@@ -300,10 +300,8 @@ describe("ResourceManager", () => {
]);
});
- it("prefers a local copy, and refuses invalid or unknown resources", () => {
- manager.register({ ...defineModel(), findLocalCopy: () => "/hand/installed" });
- expect(manager.getPath("test.model")).toBe("/hand/installed");
-
+ it("refuses invalid or unknown resources", () => {
+ manager.register(defineModel());
expect(() => manager.register(defineModel())).toThrow(/already registered/);
expect(() => manager.register(defineModel({ id: "Test Model" }))).toThrow(/invalid ID/);
expect(() => manager.register({ ...defineModel({ id: "test.escape" }), files: [ file("../outside", MODEL) ] }))
diff --git a/apps/server/src/services/resources.ts b/apps/server/src/services/resources.ts
index debca32302..20f5373a97 100644
--- a/apps/server/src/services/resources.ts
+++ b/apps/server/src/services/resources.ts
@@ -30,11 +30,6 @@ export interface ResourceDefinition {
/** The SPDX identifier of the license the files are under, such as `CC-BY-4.0`. */
license: string;
files: ResourceFile[];
- /**
- * Returns a directory that holds the files outside the manager, such as a model installed by
- * hand. While it returns one, the resource counts as installed there.
- */
- findLocalCopy?(): string | null;
}
interface Download {
@@ -86,10 +81,6 @@ export class ResourceManager {
/** Returns the directory holding the resource `id`, or `null` while it is not installed. */
getPath(id: string) {
const definition = this.getDefinition(id);
- const localCopy = definition.findLocalCopy?.();
- if (localCopy) {
- return localCopy;
- }
const directory = this.getDirectory(definition);
const verified = readVerifiedFiles(directory);
const isInstalled = definition.files