Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
fix(desktop): keep meeting transcripts whole on stop, quit and failure
Elian Doran1469217b508c
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex b7b6e666d0..b83f65d809 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -26,7 +26,11 @@ main process: TranscriptionWorker → AttachmentSink (transcript attachment)state broadcast "transcription-state" (status, noteId, pendingSources, error)```-The main process opens the capture window, starts the transcription process and connects the two with a `MessageChannelMain`; per chunk it does nothing, per utterance it writes the attachment. Each recording gets a process of its own, which loads the model (~1.3 s) and exits after `stop`. If it dies mid-recording, the main process closes the capture window, keeps what was written, and broadcasts an `idle` state with an `error`, which the launcher shows as a toast.+The main process opens the capture window, starts the transcription process and connects the two with a `MessageChannelMain`; per chunk it does nothing, per utterance it writes the attachment. Each recording gets a process of its own, which loads the model (~1.3 s) and exits after `stop`.++- **Stopping** runs in order: `CaptureWindow.stop()` calls the page's `finishCapture()`, which flushes each worklet's partial chunk (padded with silence to whole 512-sample VAD windows) and posts `{ end: true }`; then the process, told to stop, waits for that marker (it travels on another port than the stop request, up to 2 s) before flushing the VADs and the decode queue.+- **Quitting** during a recording is held back (`before-quit`) until that stop finishes, at most 15 s.+- **Failures** reach every window as a state with an `error`, which the launcher shows as a toast; `getState()` never returns it. The process dying ends the recording, keeping what was written; the capture page crashing or every source ending (`render-process-gone`, a track's `ended`) stops it normally, transcribing what was captured; a transcript that cannot be saved (a protected note while the protected session is locked) is reported once and kept in memory, and `AttachmentSink.save()` writes it on the next utterance or on `ENTER_PROTECTED_SESSION`.| File | Owns ||---|---|@@ -43,7 +47,7 @@ The main process opens the capture window, starts the transcription process andNothing the transcription process imports may reach `@triliumnext/core` or `@triliumnext/server` (the data directory module creates directories when it loads): it gets its paths and a `Logger` from the main process.| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `speakers`, `utterances`), `parseTranscript()`, speaker keys and `setSpeakerName()`; `cleanUtteranceText()`, the filler and stutter cleanup || Cleanup | The stored transcript stays verbatim. `TranscriptView` shows each utterance through `cleanUtteranceText()` while the synced option `transcriptCleanupEnabled` (default on, the "Hide filler words" switch above the transcript) holds, and leaves out utterances it empties. The rules are English-only for stutters (a function-word list) and keep numbers, comma-separated repeats and "that that"/"it it" |-| Speaker names | `TranscriptView` reads the stored transcript, applies `setSpeakerName()` and saves it whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename during a recording survives, and an utterance a rename overwrote comes back with the next write |+| Speaker names | While its note is being transcribed, `TranscriptView` renames through `electronApi.transcription.renameSpeaker()`: `AttachmentSink`, the only writer then, applies `setSpeakerName()` to the stored names in one transaction, so no utterance is overwritten. Otherwise it saves the stored transcript whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`; attachments get no route of their own. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename from another window or device survives the next utterance || `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in || `apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx` | The launcher and `useTranscriptionState()` |diff --git a/.claude/skills/working-with-transcription/capture-test.mjs b/.claude/skills/working-with-transcription/capture-test.mjsindex d545c3abbe..1a5856ed0f 100644--- a/.claude/skills/working-with-transcription/capture-test.mjs+++ b/.claude/skills/working-with-transcription/capture-test.mjs@@ -178,7 +178,7 @@ function buildPage(config, processor) {numberOfOutputs: 0,channelCount: 1,channelCountMode: "explicit",- processorOptions: { chunkSamples: CHUNK_SAMPLES }+ processorOptions: { chunkSamples: CHUNK_SAMPLES, windowSamples: 512 }});node.port.onmessage = (e) => ipcRenderer.send("samples", e.data);source.connect(node);diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjsindex 4b11940612..6e74c85644 100644--- a/.claude/skills/working-with-transcription/worker-test.mjs+++ b/.claude/skills/working-with-transcription/worker-test.mjs@@ -135,8 +135,6 @@ async function runInElectron(config) {worker.connectCapture(port2);window.webContents.postMessage("transcription-capture-port", null, [ port1 ]);const chunks = await posted;- // Lets the last chunks reach the process before it is told to stop.- await new Promise((resolve) => setTimeout(resolve, 500));const stopAt = performance.now();await worker.stop();@@ -186,6 +184,7 @@ function buildPage(config) {chunks++;await new Promise((resolve) => setTimeout(resolve, CHUNK_SAMPLES / RATE * 1000 / config.speed));}+ port.postMessage({ end: true });console.log(`posted ${chunks}`);});};diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsxindex 07daa4dd35..9246025c09 100644--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx@@ -86,6 +86,21 @@ describe("transcript helpers", () => {]);});+ it("orders turns by when they were spoken, not by when they were transcribed", () => {+ // The microphone's short reply ends, and is stored, before the long system utterance+ // that started first.+ const turns = groupIntoTurns([+ { source: "microphone", offsetMs: 6000, durationMs: 1000, text: "Yes." },+ { source: "system", offsetMs: 0, durationMs: 15_000, text: "Can you hear me?" },+ { source: "system", offsetMs: 16_000, durationMs: 1000, text: "Good." }+ ]);+ expect(turns.map(({ source, offsetMs }) => ({ source, offsetMs }))).toEqual([+ { source: "system", offsetMs: 0 },+ { source: "microphone", offsetMs: 6000 },+ { source: "system", offsetMs: 16_000 }+ ]);+ });+it("starts a new paragraph within a turn after a long pause", () => {const turns = groupIntoTurns([{ source: "system", offsetMs: 0, durationMs: 8000, text: "First point," },@@ -284,6 +299,48 @@ describe("TranscriptView", () => {expect(h.post).toHaveBeenCalledTimes(2);});+ it("renames a speaker through the recording that writes the transcript, while it runs", async () => {+ const renameSpeaker = vi.fn(async () => true);+ window.electronApi = {+ transcription: {+ getState: async () => ({ status: "recording", noteId: "meeting", pendingSources: [] }),+ onStateChanged: () => () => {},+ renameSpeaker+ }+ } as unknown as typeof window.electronApi;+ const getBlob = vi.fn(async () => ({+ content: transcript([ { source: "system", offsetMs: 0, speaker: 1, text: "Hi." } ])+ }));+ const attachment = { attachmentId: "transcript1", ownerId: "meeting", getBlob };+ const container = renderInto(+ <ParentComponent.Provider value={new Component()}>+ <TranscriptView attachment={attachment as unknown as FAttachment} />+ </ParentComponent.Provider>+ );+ const speaker = () => container.querySelector<HTMLElement>(".transcript-speaker");+ await vi.waitFor(() => expect(speaker()).not.toBeNull());+ const posted = h.post.mock.calls.length;++ await act(async () => speaker()?.click());+ const input = container.querySelector<HTMLInputElement>(".transcript-speaker-editor");+ expect(input).not.toBeNull();+ act(() => {+ if (input) {+ input.value = "Alexandra";+ input.dispatchEvent(new Event("input", { bubbles: true }));+ }+ });+ act(() => {+ input?.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true }));+ });+ await vi.waitFor(() => expect(renameSpeaker)+ .toHaveBeenCalledExactlyOnceWith("meeting", "system:1", "Alexandra"));+ await new Promise((resolve) => setTimeout(resolve));+ expect(h.post).toHaveBeenCalledTimes(posted);+ expect(getBlob).toHaveBeenCalledOnce();+ window.electronApi = undefined;+ });+it("shows who is speaking in the recorded meeting, before the words arrive", async () => {let onStateChanged: (state: TranscriptionState) => void = () => {};const recording = (noteId: string, pendingSources: TranscriptSource[]) =>diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsxindex 876e3e2118..6729551328 100644--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx@@ -231,16 +231,19 @@ function SpeakerName({ speakerKey, name, defaultName, onRename }: {}/**- * Names a speaker in the transcript as it is stored now, rather than as last shown: a running- * recording can have written utterances since. One written in the moment between this read and- * the save is restored by the recording's next write, which rewrites every utterance it holds.+ * Names a speaker. While this note is being transcribed, the recording applies the name, since it+ * rewrites the transcript after every utterance and a whole-transcript save here could overwrite+ * one. Otherwise the transcript as stored now is saved with the name.*/async function renameSpeaker(attachment: FAttachment, key: string, name: string) {+ const { attachmentId, ownerId, role, mime, title } = attachment;+ if (await window.electronApi?.transcription?.renameSpeaker(ownerId, key, name)) {+ return;+ }const latest = parseTranscript((await attachment.getBlob())?.content);if (!latest) {return;}- const { attachmentId, ownerId, role, mime, title } = attachment;await server.post(`notes/${ownerId}/attachments?matchBy=attachmentId`, {attachmentId, role, mime, title,content: JSON.stringify(setSpeakerName(latest, key, name), null, 2)@@ -282,6 +285,8 @@ export interface TranscriptTurn {/*** Joins each run of utterances from one source and speaker into a turn, timed by its first* utterance, and starts a new paragraph after a pause of {@link PARAGRAPH_PAUSE_MS} or more.+ * Utterances are taken in the order they began: a recording stores them as they finish, so a+ * short reply can precede the longer utterance it answers.* Speakers given the same name count as one. An utterance without a speaker continues the turn of* its source, and an utterance without a `durationMs` never starts a paragraph, since the pause* before the next is unknown.@@ -295,7 +300,7 @@ export function groupIntoTurns(};const turns: TranscriptTurn[] = [];let previousEndMs: number | null = null;- for (const utterance of utterances) {+ for (const utterance of utterances.toSorted((a, b) => a.offsetMs - b.offsetMs)) {const { source, speaker, offsetMs, durationMs } = utterance;const last = turns.at(-1);const isSameSpeaker = speaker === undefined || last?.speaker === undefineddiff --git a/apps/desktop/src/preload.spec.ts b/apps/desktop/src/preload.spec.tsindex 48c161f7ac..307dbb9ac4 100644--- a/apps/desktop/src/preload.spec.ts+++ b/apps/desktop/src/preload.spec.ts@@ -724,14 +724,19 @@ describe("preload script", () => {describe("transcription", () => {const transcription = () => getGroup("transcription");- it("start, stop and getState invoke their IPC channels", async () => {+ it("start, stop, getState and renameSpeaker invoke their IPC channels", async () => {await transcription().start("meeting");await transcription().stop();await transcription().getState();+ await transcription().renameSpeaker("meeting", "system:1", "Alexandra");expect(ipcRendererInvoked).toEqual([{ channel: "transcription-start", args: [ "meeting" ] },{ channel: "transcription-stop", args: [] },- { channel: "transcription-get-state", args: [] }+ { channel: "transcription-get-state", args: [] },+ {+ channel: "transcription-rename-speaker",+ args: [ "meeting", "system:1", "Alexandra" ]+ }]);});diff --git a/apps/desktop/src/preload.ts b/apps/desktop/src/preload.tsindex 7d13bb03bf..dc35e0e7f0 100644--- a/apps/desktop/src/preload.ts+++ b/apps/desktop/src/preload.ts@@ -264,6 +264,9 @@ contextBridge.exposeInMainWorld("electronApi", {getState() {return ipcRenderer.invoke("transcription-get-state");},+ renameSpeaker(noteId: string, key: string, name: string) {+ return ipcRenderer.invoke("transcription-rename-speaker", noteId, key, name);+ },onStateChanged(callback: (state: TranscriptionState) => void) {const listener = (_event: unknown, state: TranscriptionState) => callback(state);ipcRenderer.on("transcription-state", listener);diff --git a/apps/desktop/src/services/transcription/attachment_sink.spec.ts b/apps/desktop/src/services/transcription/attachment_sink.spec.tsindex 476334961d..7484e317e4 100644--- a/apps/desktop/src/services/transcription/attachment_sink.spec.ts+++ b/apps/desktop/src/services/transcription/attachment_sink.spec.ts@@ -84,6 +84,40 @@ describe("AttachmentSink", () => {expect(written.utterances).toHaveLength(1);});+ it("keeps an utterance it could not save, and saves it with the next write or save()", () => {+ const sink = new AttachmentSink("meeting");+ sink.open(STARTED_AT);+ h.setContent.mockImplementationOnce(() => {+ throw new Error("protected session is not available");+ });+ expect(() => sink.write({ source: "system", offsetMs: 0, text: "First." }))+ .toThrow("protected session is not available");+ sink.write({ source: "system", offsetMs: 1000, text: "Second." });+ const saved = JSON.parse(h.setContent.mock.calls[1][0]) as Transcript;+ expect(saved.utterances.map((utterance) => utterance.text)).toEqual([ "First.", "Second." ]);++ sink.save();+ expect(h.setContent).toHaveBeenCalledTimes(3);+ expect(h.setContent.mock.calls[2][0]).toBe(h.setContent.mock.calls[1][0]);+ });++ it("renames a speaker in the names as stored, keeping every utterance, in one transaction", () => {+ const sink = new AttachmentSink("meeting");+ sink.open(STARTED_AT);+ sink.write({ source: "system", offsetMs: 0, speaker: 1, text: "Hello." });+ h.getContent.mockReturnValue(JSON.stringify({+ version: 1, startedAt: "2026-10-13T14:30:00.000Z",+ speakers: { microphone: "Elian" }, utterances: []+ }));+ const transactions = h.transactions;++ sink.renameSpeaker("system:1", " Alexandra ");+ expect(h.transactions).toBe(transactions + 1);+ const renamed = JSON.parse(h.setContent.mock.calls.at(-1)?.[0]) as Transcript;+ expect(renamed.speakers).toEqual({ microphone: "Elian", "system:1": "Alexandra" });+ expect(renamed.utterances).toHaveLength(1);+ });+it("refuses to write before it is open", () => {const sink = new AttachmentSink("meeting");expect(() => sink.write({ source: "system", offsetMs: 0, text: "Hi" }))diff --git a/apps/desktop/src/services/transcription/attachment_sink.ts b/apps/desktop/src/services/transcription/attachment_sink.tsindex bb60dd41ef..dc4dcbe149 100644--- a/apps/desktop/src/services/transcription/attachment_sink.ts+++ b/apps/desktop/src/services/transcription/attachment_sink.ts@@ -1,5 +1,5 @@import {- parseTranscript, type Transcript, TRANSCRIPT_MIME, type TranscriptUtterance+ parseTranscript, setSpeakerName, type Transcript, TRANSCRIPT_MIME, type TranscriptUtterance} from "@triliumnext/commons";import { becca, binary_utils, cls, getSql } from "@triliumnext/core";import { t } from "i18next";@@ -9,7 +9,9 @@ import type { TranscriptSink } from "./session";/*** Keeps the transcript in a `transcript` attachment of the meeting note, rewritten after every* utterance. Each write commits its own transaction, which sends the change to every open window,- * and keeps the speaker names stored in the attachment, which the user can change meanwhile.+ * and keeps the speaker names stored in the attachment, which the user can change meanwhile. An+ * utterance that cannot be saved (a protected note while the protected session is locked) is kept+ * and saved with the next write or {@link save}.*/export class AttachmentSink implements TranscriptSink {private transcript: Transcript | null = null;@@ -39,15 +41,42 @@ export class AttachmentSink implements TranscriptSink {}transcript.utterances.push(utterance);+ this.save();+ }++ /** Rewrites the attachment with every utterance written so far. */+ save() {+ this.rewrite((speakers) => speakers);+ }++ /**+ * Names a speaker in the names as stored, which another window can have changed, or restores+ * its default name when `name` is blank.+ */+ renameSpeaker(key: string, name: string) {+ this.rewrite((speakers, transcript) =>+ setSpeakerName({ ...transcript, speakers }, key, name).speakers);+ }++ /** Rewrites the attachment in one transaction, the stored names passed through `rename`. */+ private rewrite(rename: (speakers: SpeakerNames | undefined, transcript: Transcript) =>+ SpeakerNames | undefined) {+ const { transcript, attachmentId } = this;+ if (!transcript || !attachmentId) {+ throw new Error("The transcript attachment was not created.");+ }+inTransaction(() => {const attachment = becca.getAttachmentOrThrow(attachmentId);const stored = parseTranscript(binary_utils.decodeUtf8(attachment.getContent()));- transcript.speakers = stored?.speakers;+ transcript.speakers = rename(stored?.speakers, transcript);attachment.setContent(serialize(transcript));});}}+type SpeakerNames = Transcript["speakers"];+function serialize(transcript: Transcript) {return JSON.stringify(transcript, null, 2);}diff --git a/apps/desktop/src/services/transcription/capture.spec.ts b/apps/desktop/src/services/transcription/capture.spec.tsnew file mode 100644index 0000000000..683521482a--- /dev/null+++ b/apps/desktop/src/services/transcription/capture.spec.ts@@ -0,0 +1,136 @@+import { EventEmitter } from "events";+import { describe, expect, it, vi } from "vitest";++const h = vi.hoisted(() => ({+ windows: [] as FakeWindow[],+ log: { info: vi.fn(), error: vi.fn() }+}));++/** The parts of a `BrowserWindow` the capture window uses. */+class FakeWindow {+ destroyed = false;+ webContents = Object.assign(new EventEmitter(), {+ ipc: new EventEmitter(),+ postMessage: vi.fn(),+ executeJavaScript: vi.fn(async (call: string) => call)+ });++ constructor() {+ h.windows.push(this);+ }++ loadFile = vi.fn(async () => {});+ isDestroyed = () => this.destroyed;+ destroy = vi.fn(() => {+ this.destroyed = true;+ });+}++vi.mock("electron", () => ({+ app: { getPath: () => "/tmp" },+ BrowserWindow: FakeWindow,+ session: {+ fromPartition: () => ({+ setPermissionRequestHandler: vi.fn(),+ setPermissionCheckHandler: vi.fn(),+ setDisplayMediaRequestHandler: vi.fn()+ })+ }+}));+vi.mock("fs", () => ({ default: { writeFileSync: vi.fn() } }));+vi.mock("@triliumnext/core", () => ({ getLog: () => h.log }));++const { CAPTURE_PROCESSOR, CaptureWindow } = await import("./capture");++interface FakeProcessor {+ port: {+ postMessage: ReturnType<typeof vi.fn>;+ onmessage: ((event: { data: unknown }) => void) | null;+ };+ process(inputs: Float32Array[][]): boolean;+}++/** Runs the worklet's source with fake `AudioWorkletProcessor` and `registerProcessor` globals. */+function createProcessor(chunkSamples: number, windowSamples: number): FakeProcessor {+ class AudioWorkletProcessor {+ port = { postMessage: vi.fn(), onmessage: null };+ }+ let Processor: (new (options: object) => FakeProcessor) | null = null;+ const registerProcessor = (_name: string, processor: typeof Processor) => {+ Processor = processor;+ };+ new Function("AudioWorkletProcessor", "registerProcessor", CAPTURE_PROCESSOR)(+ AudioWorkletProcessor, registerProcessor);+ if (!Processor) {+ throw new Error("The worklet registered no processor.");+ }+ return new (Processor as new (options: object) => FakeProcessor)({+ processorOptions: { chunkSamples, windowSamples }+ });+}++/** One 128-frame render quantum whose samples all hold `value`. */+function quantum(value: number) {+ return [ [ new Float32Array(128).fill(value) ] ];+}++describe("CAPTURE_PROCESSOR", () => {+ it("posts whole chunks, then on a flush what it holds, padded to whole windows", () => {+ const processor = createProcessor(512, 256);+ for (let i = 1; i <= 5; i++) {+ processor.process(quantum(i));+ }+ expect(processor.port.postMessage).toHaveBeenCalledOnce();+ expect([ ...processor.port.postMessage.mock.calls[0][0] as Float32Array ])+ .toEqual([ 1, 2, 3, 4 ].flatMap((value) => new Array(128).fill(value)));++ processor.port.onmessage?.({ data: "flush" });+ const [ , [ partial ], [ done ] ] = processor.port.postMessage.mock.calls;+ expect([ ...partial as Float32Array ])+ .toEqual([ ...new Array(128).fill(5), ...new Array(128).fill(0) ]);+ expect(done).toBe("flushed");++ processor.port.onmessage?.({ data: "flush" });+ expect(processor.port.postMessage.mock.calls.at(-1)?.[0]).toBe("flushed");+ expect(processor.port.postMessage).toHaveBeenCalledTimes(4);+ });+});++describe("CaptureWindow", () => {+ async function startCapture() {+ h.windows.length = 0;+ const onLost = vi.fn();+ const capture = new CaptureWindow({ onLost });+ expect(await capture.start("port" as never)).toEqual([ "microphone", "system" ]);+ const window = h.windows[0];+ expect(window.webContents.postMessage)+ .toHaveBeenCalledExactlyOnceWith("transcription-capture-port", null, [ "port" ]);+ return { capture, onLost, window };+ }++ it("reports the capture lost when its page crashes", async () => {+ const { onLost, window } = await startCapture();+ window.webContents.emit("render-process-gone", {}, { reason: "oom" });+ window.webContents.emit("render-process-gone", {}, { reason: "oom" });+ expect(onLost).toHaveBeenCalledExactlyOnceWith("the capture page ended (oom)");+ });++ it("reports the capture lost once every source has ended", async () => {+ const { onLost, window } = await startCapture();+ window.webContents.ipc.emit("transcription-source-ended", {}, "microphone");+ expect(onLost).not.toHaveBeenCalled();+ expect(h.log.error).toHaveBeenCalledWith("Transcription: the microphone source ended.");+ window.webContents.ipc.emit("transcription-source-ended", {}, "system");+ expect(onLost).toHaveBeenCalledExactlyOnceWith("every audio source ended");+ });++ it("finishes the page before closing it, and reports nothing once stopped", async () => {+ const { capture, onLost, window } = await startCapture();+ await capture.stop();+ expect(window.webContents.executeJavaScript).toHaveBeenLastCalledWith("finishCapture()");+ expect(window.destroy).toHaveBeenCalledOnce();++ window.webContents.emit("render-process-gone", {}, { reason: "killed" });+ expect(onLost).not.toHaveBeenCalled();+ });+});diff --git a/apps/desktop/src/services/transcription/capture.ts b/apps/desktop/src/services/transcription/capture.tsindex b7442c2c34..61f205bdbe 100644--- a/apps/desktop/src/services/transcription/capture.ts+++ b/apps/desktop/src/services/transcription/capture.ts@@ -7,10 +7,26 @@ import path from "path";import { SAMPLE_RATE } from "./protocol";const PORT_CHANNEL = "transcription-capture-port";+const SOURCE_ENDED_CHANNEL = "transcription-source-ended";-/** Samples per message. `VadSegmenter` drops what does not fill a 512-sample window. */+export interface CaptureWindowOptions {+ /**+ * Called once when capturing stops on its own: the page crashed, or every source ended (a+ * device was unplugged, the system audio went away). Not called after+ * {@link CaptureWindow.stop}.+ */+ onLost(reason: string): void;+}++/** Samples per message. A multiple of {@link VAD_WINDOW_SAMPLES}. */const CHUNK_SAMPLES = 4096;+/** `VadSegmenter` drops what does not fill a window, so the last chunk is padded to whole ones. */+const VAD_WINDOW_SAMPLES = 512;++/** How long {@link CaptureWindow.stop} waits for the page to post what it still holds. */+const FINISH_TIMEOUT_MS = 1000;+/** The page function that starts each source. */const SOURCE_STARTERS = [[ "microphone", "captureMicrophone()" ],@@ -24,6 +40,10 @@ const SOURCE_STARTERS = [*/export class CaptureWindow {private window: BrowserWindow | null = null;+ private readonly running = new Set<TranscriptSource>();+ private isLost = false;++ constructor(private readonly options: CaptureWindowOptions) {}/*** Starts every source it can, posting to `port`, and returns the ones that run.@@ -33,6 +53,15 @@ export class CaptureWindow {async start(port: MessagePortMain) {const window = await createCaptureWindow();this.window = window;+ window.webContents.on("render-process-gone", (_event, { reason }) =>+ this.lose(`the capture page ended (${reason})`));+ window.webContents.ipc.on(SOURCE_ENDED_CHANNEL, (_event, source: TranscriptSource) => {+ getLog().error(`Transcription: the ${source} source ended.`);+ this.running.delete(source);+ if (!this.running.size) {+ this.lose("every audio source ended");+ }+ });window.webContents.postMessage(PORT_CHANNEL, null, [ port ]);const sources: TranscriptSource[] = [];@@ -43,6 +72,7 @@ export class CaptureWindow {const tracks = await window.webContents.executeJavaScript(call, true);getLog().info(`Transcription: capturing ${source} from ${tracks}.`);sources.push(source);+ this.running.add(source);} catch (e) {const message = (e as Error)?.message ?? e;getLog().error(`Transcription: ${source} unavailable: ${message}`);@@ -56,11 +86,38 @@ export class CaptureWindow {return sources;}- stop() {- if (this.window && !this.window.isDestroyed()) {- this.window.destroy();- }+ /**+ * Posts what the worklets still hold, then an end marker the transcription process waits for,+ * and closes the window. Gives up on the page after {@link FINISH_TIMEOUT_MS}.+ */+ async stop() {+ const window = this.window;this.window = null;+ this.running.clear();+ if (!window || window.isDestroyed()) {+ return;+ }+ try {+ await Promise.race([+ window.webContents.executeJavaScript("finishCapture()"),+ new Promise((_resolve, reject) => setTimeout(+ () => reject(new Error("timed out")), FINISH_TIMEOUT_MS))+ ]);+ } catch (e) {+ const message = (e as Error)?.message ?? e;+ getLog().error(`Transcription: finishing the capture failed: ${message}`);+ } finally {+ if (!window.isDestroyed()) {+ window.destroy();+ }+ }+ }++ private lose(reason: string) {+ if (this.window && !this.isLost) {+ this.isLost = true;+ this.options.onLost(reason);+ }}}@@ -109,15 +166,29 @@ async function createCaptureWindow() {/*** The audio worklet that collects a source into chunks of `processorOptions.chunkSamples`. It runs* on the audio rendering thread, where a busy page cannot delay it and corrupt the start of a- * chunk. `.claude/skills/working-with-transcription/capture-test.mjs` reads it from this file.+ * chunk. A `"flush"` message posts the partial chunk, padded with silence to whole+ * `processorOptions.windowSamples`, then `"flushed"`.+ * `.claude/skills/working-with-transcription/capture-test.mjs` reads it from this file.*/-const CAPTURE_PROCESSOR = `+export const CAPTURE_PROCESSOR = `class ChunkCollector extends AudioWorkletProcessor {constructor({ processorOptions }) {super();this.chunkSamples = processorOptions.chunkSamples;+ this.windowSamples = processorOptions.windowSamples;this.chunk = new Float32Array(this.chunkSamples);this.filled = 0;+ this.port.onmessage = () => this.flush();+ }++ flush() {+ const length = Math.ceil(this.filled / this.windowSamples) * this.windowSamples;+ if (length) {+ this.port.postMessage(this.chunk.slice(0, length));+ this.chunk = new Float32Array(this.chunkSamples);+ this.filled = 0;+ }+ this.port.postMessage("flushed");}process(inputs) {@@ -149,6 +220,8 @@ const context = new AudioContext({ sampleRate: ${SAMPLE_RATE} });const processorLoaded = context.audioWorklet.addModule(URL.createObjectURL(new Blob([ ${JSON.stringify(CAPTURE_PROCESSOR)} ], { type: "text/javascript" })));+const flushes = [];+async function pipe(source, media) {const [ port ] = await Promise.all([ portReceived, processorLoaded ]);const collector = new AudioWorkletNode(context, "chunk-collector", {@@ -156,22 +229,47 @@ async function pipe(source, media) {numberOfOutputs: 0,channelCount: 1,channelCountMode: "explicit",- processorOptions: { chunkSamples: ${CHUNK_SAMPLES} }+ processorOptions: { chunkSamples: ${CHUNK_SAMPLES}, windowSamples: ${VAD_WINDOW_SAMPLES} }});+ let onFlushed = () => {};// Copied, not transferred: a MessagePortMain receives a message with a transferred buffer// as null.- collector.port.onmessage = (e) => port.postMessage({ source, samples: e.data });+ collector.port.onmessage = (e) => e.data === "flushed"+ ? onFlushed()+ : port.postMessage({ source, samples: e.data });+ flushes.push(() => new Promise((resolve) => {+ onFlushed = resolve;+ collector.port.postMessage("flush");+ }));context.createMediaStreamSource(media).connect(collector);}+async function finishCapture() {+ await Promise.all(flushes.map((flush) => flush()));+ (await portReceived).postMessage({ end: true });+}+function describe(tracks) {return tracks.map((track) => track.label).join(", ");}+function reportEnd(source, tracks) {+ let isReported = false;+ for (const track of tracks) {+ track.addEventListener("ended", () => {+ if (!isReported) {+ isReported = true;+ ipcRenderer.send("${SOURCE_ENDED_CHANNEL}", source);+ }+ });+ }+}+async function captureMicrophone() {const media = await navigator.mediaDevices.getUserMedia({audio: { channelCount: 1, echoCancellation: true, noiseSuppression: false }});+ reportEnd("microphone", media.getAudioTracks());await pipe("microphone", media);return describe(media.getAudioTracks());}@@ -194,6 +292,7 @@ async function captureSystemAudio() {console.log("System track muted=" + track.muted + " settings="+ JSON.stringify(track.getSettings()));}+ reportEnd("system", audioTracks);await pipe("system", new MediaStream(audioTracks));return describe(audioTracks);}diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.tsindex adb5e5ed3f..cbf2b95f80 100644--- a/apps/desktop/src/services/transcription/index.spec.ts+++ b/apps/desktop/src/services/transcription/index.spec.ts@@ -1,22 +1,34 @@import type { TranscriptionState } from "@triliumnext/commons";import { beforeEach, describe, expect, it, vi } from "vitest";+import type { CaptureWindowOptions } from "./capture";import type { WorkerListeners } from "./worker_client";const h = vi.hoisted(() => ({handlers: new Map<string, (...args: unknown[]) => unknown>(),+ appListeners: new Map<string, (...args: unknown[]) => unknown>(),+ quit: vi.fn(),broadcast: [] as TranscriptionState[],notes: new Set<string>([ "meeting" ]),+ isProtected: false,+ isProtectedSessionAvailable: false,+ coreListeners: new Map<string, () => void>(),isDev: false,isModelInstalled: true,log: { info: vi.fn(), error: vi.fn() },listeners: null as WorkerListeners | null,+ captureOptions: null as CaptureWindowOptions | null,worker: { start: vi.fn(), connectCapture: vi.fn(), stop: vi.fn(), kill: vi.fn() },capture: { start: vi.fn(), stop: vi.fn() },- sink: { open: vi.fn(), write: vi.fn() }+ sink: { open: vi.fn(), write: vi.fn(), save: vi.fn(), renameSpeaker: vi.fn() }}));vi.mock("electron", () => ({+ app: {+ on: (event: string, listener: (...args: unknown[]) => unknown) =>+ h.appListeners.set(event, listener),+ quit: h.quit+ },ipcMain: {handle: (channel: string, handler: (...args: unknown[]) => unknown) =>h.handlers.set(channel, handler)@@ -39,8 +51,16 @@ vi.mock("electron", () => ({}));vi.mock("fs", () => ({ default: { existsSync: () => h.isModelInstalled } }));vi.mock("@triliumnext/core", () => ({- becca: { getNote: (noteId: string) => h.notes.has(noteId) ? { noteId } : null },+ becca: {+ getNote: (noteId: string) =>+ h.notes.has(noteId) ? { noteId, isProtected: h.isProtected } : null+ },+ events: {+ ENTER_PROTECTED_SESSION: "ENTER_PROTECTED_SESSION",+ subscribe: (event: string, listener: () => void) => h.coreListeners.set(event, listener)+ },getLog: () => h.log,+ protected_session: { isProtectedSessionAvailable: () => h.isProtectedSessionAvailable },utils: { isDev: () => h.isDev }}));vi.mock("i18next", () => ({@@ -53,12 +73,18 @@ vi.mock("./capture", () => ({CaptureWindow: class {start = h.capture.start;stop = h.capture.stop;++ constructor(options: CaptureWindowOptions) {+ h.captureOptions = options;+ }}}));vi.mock("./attachment_sink", () => ({AttachmentSink: class {open = h.sink.open;write = h.sink.write;+ save = h.sink.save;+ renameSpeaker = h.sink.renameSpeaker;constructor(readonly noteId: string) {}}@@ -89,12 +115,18 @@ async function loadService() {beforeEach(() => {vi.clearAllMocks();h.handlers.clear();+ h.appListeners.clear();+ h.coreListeners.clear();+ h.isProtected = false;+ h.isProtectedSessionAvailable = false;h.broadcast.length = 0;h.isDev = false;h.isModelInstalled = true;h.listeners = null;h.worker.start.mockResolvedValue({ type: "ready", startedAt: STARTED_AT, settings: {} });h.capture.start.mockResolvedValue([ "microphone", "system" ]);+ h.worker.stop.mockResolvedValue(undefined);+ h.capture.stop.mockResolvedValue(undefined);});describe("transcription IPC", () => {@@ -117,6 +149,23 @@ describe("transcription IPC", () => {expect(await invoke("transcription-get-state")).toEqual(IDLE);});+ it("stops the engine only once the capture window has posted what it still held", async () => {+ const invoke = await loadService();+ await invoke("transcription-start", "meeting");+ let finishCapture = () => {};+ h.capture.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {+ finishCapture = resolve;+ }));++ const stopped = invoke("transcription-stop");+ await Promise.resolve();+ expect(h.capture.stop).toHaveBeenCalledOnce();+ expect(h.worker.stop).not.toHaveBeenCalled();+ finishCapture();+ await stopped;+ expect(h.worker.stop).toHaveBeenCalledOnce();+ });+it("connects the capture window to the transcription process once that is ready", async () => {const invoke = await loadService();await invoke("transcription-start", "meeting");@@ -159,6 +208,56 @@ describe("transcription IPC", () => {expect(h.broadcast.at(-1)).toEqual(IDLE);});+ it("says once that a locked protected session holds back the transcript, and saves it on unlocking", async () => {+ h.isProtected = true;+ const invoke = await loadService();+ await invoke("transcription-start", "meeting");+ h.sink.write.mockImplementation(() => {+ throw new Error("Cannot update content of blob since protected session is not available.");+ });+ const utterance = { source: "system" as const, offsetMs: 0, text: "Hi" };++ h.listeners?.onUtterance(utterance);+ h.listeners?.onUtterance(utterance);+ const errors = h.broadcast.filter((state) => state.error);+ expect(errors).toEqual([ 1, 2 ].map(() => ({+ status: "recording", noteId: "meeting", pendingSources: [],+ error: "transcription.save-locked"+ })));+ expect(await invoke("transcription-get-state")).not.toHaveProperty("error");++ await invoke("transcription-stop");+ h.coreListeners.get("ENTER_PROTECTED_SESSION")?.();+ expect(h.sink.save).toHaveBeenCalledOnce();+ h.coreListeners.get("ENTER_PROTECTED_SESSION")?.();+ expect(h.sink.save).toHaveBeenCalledOnce();+ h.sink.write.mockReset();+ });++ it("renames a speaker through the recording that writes the transcript, until it stops", async () => {+ const invoke = await loadService();+ const rename = (noteId: unknown, key: unknown, name: unknown) =>+ invoke("transcription-rename-speaker", noteId, key, name);+ expect(await rename("meeting", "system:1", "Alexandra")).toBe(false);++ await invoke("transcription-start", "meeting");+ expect(await rename("meeting", "system:1", "Alexandra")).toBe(true);+ expect(h.sink.renameSpeaker).toHaveBeenCalledExactlyOnceWith("system:1", "Alexandra");+ expect(await rename("other", "system:1", "Bob")).toBe(false);+ expect(await rename("meeting", "system:1", 42)).toBe(false);++ let finishStopping = () => {};+ h.worker.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {+ finishStopping = resolve;+ }));+ const stopped = invoke("transcription-stop");+ expect(await rename("meeting", "microphone", "Elian")).toBe(true);+ finishStopping();+ await stopped;+ expect(await rename("meeting", "microphone", "Elian")).toBe(false);+ expect(h.sink.renameSpeaker).toHaveBeenCalledTimes(2);+ });+it("keeps the audio and the transcript as produced only in development builds", async () => {vi.useFakeTimers({ now: STARTED_AT, toFake: [ "Date" ] });const invoke = await loadService();@@ -206,6 +305,21 @@ describe("transcription IPC", () => {expect(h.worker.stop).not.toHaveBeenCalled();});+ it("stops a recording whose capture was lost, keeping its queued speech, and says why", async () => {+ const invoke = await loadService();+ await invoke("transcription-start", "meeting");++ h.captureOptions?.onLost("every audio source ended");+ await vi.waitFor(() => expect(h.broadcast.at(-1))+ .toEqual({ ...IDLE, error: "transcription.capture-lost" }));+ expect(h.capture.stop).toHaveBeenCalledOnce();+ expect(h.worker.stop).toHaveBeenCalledOnce();+ expect(h.worker.kill).not.toHaveBeenCalled();+ expect(await invoke("transcription-get-state")).toEqual(IDLE);+ expect(h.log.error).toHaveBeenCalledWith(+ "Transcription: the capture stopped: every audio source ended.");+ });+it("ends a recording whose transcription process died, and says why once", async () => {const invoke = await loadService();await invoke("transcription-start", "meeting");@@ -222,4 +336,41 @@ describe("transcription IPC", () => {expect(h.worker.stop).not.toHaveBeenCalled();expect(await invoke("transcription-start", "meeting")).toBe("");});++ it("finishes the transcription before the application quits", async () => {+ const invoke = await loadService();+ const quit = () => {+ const event = { preventDefault: vi.fn() };+ h.appListeners.get("before-quit")?.(event);+ return event;+ };+ expect(quit().preventDefault).not.toHaveBeenCalled();++ await invoke("transcription-start", "meeting");+ let finishStopping = () => {};+ h.worker.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {+ finishStopping = resolve;+ }));+ expect(quit().preventDefault).toHaveBeenCalledOnce();+ await vi.waitFor(() => expect(h.worker.stop).toHaveBeenCalledOnce());+ expect(quit().preventDefault).toHaveBeenCalledOnce();+ expect(h.quit).not.toHaveBeenCalled();++ finishStopping();+ await vi.waitFor(() => expect(h.quit).toHaveBeenCalledOnce());+ expect(await invoke("transcription-get-state")).toEqual(IDLE);+ expect(quit().preventDefault).not.toHaveBeenCalled();+ });++ it("quits regardless once the transcription takes too long to finish", async () => {+ const invoke = await loadService();+ await invoke("transcription-start", "meeting");+ h.worker.stop.mockImplementationOnce(() => new Promise<void>(() => {}));+ vi.useFakeTimers();++ h.appListeners.get("before-quit")?.({ preventDefault: vi.fn() });+ await vi.advanceTimersByTimeAsync(15_000);+ expect(h.quit).toHaveBeenCalledOnce();+ vi.useRealTimers();+ });});diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.tsindex 85dac744be..366b3a9a5a 100644--- a/apps/desktop/src/services/transcription/index.ts+++ b/apps/desktop/src/services/transcription/index.ts@@ -1,7 +1,7 @@import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";-import { becca, getLog, utils as coreUtils } from "@triliumnext/core";+import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";import dataDirs from "@triliumnext/server/src/services/data_dir.js";-import { BrowserWindow, ipcMain, MessageChannelMain } from "electron";+import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";import fs from "fs";import { t } from "i18next";import path from "path";@@ -18,6 +18,9 @@ export const RECORDINGS_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcriptioconst IDLE: TranscriptionState = { status: "idle", noteId: null, pendingSources: [] };+/** How long quitting waits for a running transcription to finish. */+const QUIT_TIMEOUT_MS = 15_000;+interface Recording {worker: TranscriptionWorker;capture: CaptureWindow;@@ -25,11 +28,26 @@ interface Recording {let state = IDLE;let activeRecording: Recording | null = null;+/** The transcript the running recording writes, until its stop has written the last utterance. */+let writer: { noteId: string; sink: AttachmentSink } | null = null;+/** The running stop, while the speech already captured is still being transcribed. */+let stopping: Promise<void> | null = null;+/**+ * Transcripts whose last utterances could not be saved, kept until a save succeeds: on the next+ * utterance, or once the protected session is entered.+ */+const unsavedSinks = new Set<AttachmentSink>();+/** `waiting` while a quit waits for the transcription, `done` once it may go ahead. */+let quitting: "no" | "waiting" | "done" = "no";export function setupTranscriptionHandlers() {ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));ipcMain.handle("transcription-stop", () => stopRecording());ipcMain.handle("transcription-get-state", () => getState());+ ipcMain.handle("transcription-rename-speaker",+ (_event, noteId: unknown, key: unknown, name: unknown) => renameSpeaker(noteId, key, name));+ app.on("before-quit", (event) => finishBeforeQuit(event));+ events.subscribe(events.ENTER_PROTECTED_SESSION, () => saveUnsaved());}/**@@ -52,9 +70,11 @@ export async function startRecording(noteId: unknown): Promise<string> {setState({ status: "starting", noteId, pendingSources: [] });const sink = new AttachmentSink(noteId);const recording: Recording = {- capture: new CaptureWindow(),+ capture: new CaptureWindow({+ onLost: (reason) => handleCaptureLost(recording, reason)+ }),worker: new TranscriptionWorker({- onUtterance: (utterance) => writeUtterance(sink, utterance),+ onUtterance: (utterance) => writeUtterance(sink, noteId, utterance),onPendingChanged: (pendingSources) => setState({ ...state, pendingSources }),onUnexpectedExit: (code) => handleUnexpectedExit(recording, code),log: getLog()@@ -68,6 +88,7 @@ export async function startRecording(noteId: unknown): Promise<string> {recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null});sink.open(startedAt);+ writer = { noteId, sink };const { port1, port2 } = new MessageChannelMain();recording.worker.connectCapture(port2);const sources = await recording.capture.start(port1);@@ -76,7 +97,8 @@ export async function startRecording(noteId: unknown): Promise<string> {return "";} catch (e) {activeRecording = null;- recording.capture.stop();+ writer = null;+ void recording.capture.stop();recording.worker.kill();setState(IDLE);getLog().error(`Transcription did not start: ${(e as Error)?.stack ?? e}`);@@ -84,39 +106,127 @@ export async function startRecording(noteId: unknown): Promise<string> {}}-export async function stopRecording() {+/**+ * Stops the recording, resolving once its speech is transcribed; joins a stop that runs. An+ * `error` is reported to every window with the idle state that follows.+ */+export function stopRecording(error?: string): Promise<void> {const recording = activeRecording;if (state.status !== "recording" || !recording) {- return;+ return stopping ?? Promise.resolve();}setState({ ...state, status: "stopping" });activeRecording = null;- recording.capture.stop();- try {- await recording.worker.stop();- getLog().info(`Transcription: recording for note ${state.noteId} stopped.`);- } finally {- setState(IDLE);- }+ const { noteId } = state;+ stopping = recording.capture.stop()+ .then(() => recording.worker.stop())+ .then(() => getLog().info(`Transcription: recording for note ${noteId} stopped.`))+ .finally(() => {+ stopping = null;+ writer = null;+ state = IDLE;+ broadcast(error ? { ...IDLE, error } : IDLE);+ });+ return stopping;}export function getState() {return state;}+/**+ * Names a speaker of the transcript the recording of `noteId` writes, through the sink, which is+ * its only writer while it runs.+ *+ * @returns `false` when no recording of `noteId` runs, or the arguments are not strings.+ */+export function renameSpeaker(noteId: unknown, key: unknown, name: unknown) {+ const isValid = typeof key === "string" && typeof name === "string";+ if (!writer || writer.noteId !== noteId || !isValid) {+ return false;+ }+ writer.sink.renameSpeaker(key, name);+ return true;+}+/** The directory under {@link RECORDINGS_DIR} for a recording of `noteId` begun at `createdAt`. */export function getRecordingDirectory(createdAt: number, noteId: string) {const timestamp = new Date(createdAt).toISOString().replace(/[:.]/g, "-");return path.join(RECORDINGS_DIR, `${timestamp}_${noteId}`);}-function writeUtterance(sink: AttachmentSink, utterance: TranscriptUtterance) {+/**+ * Holds back a quit while a transcription runs, until the speech already captured is transcribed+ * or {@link QUIT_TIMEOUT_MS} passes, then quits again.+ */+function finishBeforeQuit(event: Electron.Event) {+ if (quitting === "done") {+ return;+ }+ if (quitting === "waiting") {+ event.preventDefault();+ return;+ }+ const finished = state.status === "recording" ? stopRecording() : stopping;+ if (!finished) {+ return;+ }++ event.preventDefault();+ quitting = "waiting";+ const timeout = new Promise((resolve) => setTimeout(resolve, QUIT_TIMEOUT_MS));+ void Promise.race([ finished, timeout ]).finally(() => {+ quitting = "done";+ app.quit();+ });+}++/**+ * Saves an utterance. When that fails, the sink keeps it and every window is told once, until a+ * later save succeeds.+ */+function writeUtterance(sink: AttachmentSink, noteId: string, utterance: TranscriptUtterance) {try {sink.write(utterance);+ unsavedSinks.delete(sink);} catch (e) {getLog().error(`Transcription: saving an utterance failed: ${e}`);+ if (unsavedSinks.has(sink)) {+ return;+ }+ unsavedSinks.add(sink);+ const isLocked = becca.getNote(noteId)?.isProtected+ && !protected_session.isProtectedSessionAvailable();+ broadcast({+ ...state,+ error: isLocked+ ? t("transcription.save-locked")+ : t("transcription.save-failed", { message: (e as Error)?.message ?? String(e) })+ });+ }+}++/** Saves the transcripts that failed to save before, as while the protected session was locked. */+function saveUnsaved() {+ for (const sink of unsavedSinks) {+ try {+ sink.save();+ unsavedSinks.delete(sink);+ getLog().info("Transcription: saved a transcript that could not be saved before.");+ } catch (e) {+ getLog().error(`Transcription: saving a transcript still failed: ${e}`);+ }+ }+}++/** Stops a recording whose capture stopped on its own, transcribing what it captured. */+function handleCaptureLost(recording: Recording, reason: string) {+ if (activeRecording !== recording || state.status !== "recording") {+ return;}+ getLog().error(`Transcription: the capture stopped: ${reason}.`);+ void stopRecording(t("transcription.capture-lost"));}/**@@ -128,7 +238,8 @@ function handleUnexpectedExit(recording: Recording, code: number) {return;}activeRecording = null;- recording.capture.stop();+ writer = null;+ void recording.capture.stop();getLog().error(`Transcription: the transcription process exited with code ${code}.`);state = IDLE;broadcast({ ...IDLE, error: t("transcription.stopped-unexpectedly") });diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.tsindex dfbb8e1d42..653bf9343c 100644--- a/apps/desktop/src/services/transcription/protocol.ts+++ b/apps/desktop/src/services/transcription/protocol.ts@@ -38,3 +38,6 @@ export interface CaptureChunk {source: TranscriptSource;samples: Float32Array;}++/** What the capture page posts: chunks, then once stopping, an end marker after the last one. */+export type CaptureMessage = CaptureChunk | { end: true };diff --git a/apps/desktop/src/services/transcription/session.spec.ts b/apps/desktop/src/services/transcription/session.spec.tsindex 0dc605d1d8..14125d320d 100644--- a/apps/desktop/src/services/transcription/session.spec.ts+++ b/apps/desktop/src/services/transcription/session.spec.ts@@ -1,9 +1,10 @@import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";+import { EventEmitter } from "events";import { beforeEach, describe, expect, it, vi } from "vitest";import type { SpeakerTracker, SpeechSegment } from "./engine";import {- type AudioTap, RecordingSession, type SamplesListener, type TranscriptSink+ type AudioTap, PortCapture, RecordingSession, type SamplesListener, type TranscriptSink} from "./session";const log = { info: vi.fn(), error: vi.fn() };@@ -245,3 +246,30 @@ describe("RecordingSession", () => {expect(log.error).toHaveBeenCalledWith("Transcription: system failed: VAD model missing");});});++describe("PortCapture", () => {+ it("delivers the page's chunks, and settles once the page has posted its last", async () => {+ const port = Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });+ const onSamples = vi.fn();+ const capture = new PortCapture();+ capture.start(onSamples);+ capture.connect(port as never);+ let hasEnded = false;+ void capture.ended.then(() => {+ hasEnded = true;+ });++ const samples = new Float32Array(512);+ port.emit("message", { data: { source: "system", samples } });+ port.emit("message", { data: { source: "speakers", samples } });+ port.emit("message", { data: null });+ expect(onSamples).toHaveBeenCalledExactlyOnceWith("system", samples);+ await Promise.resolve();+ expect(hasEnded).toBe(false);++ port.emit("message", { data: { end: true } });+ await capture.ended;+ capture.stop();+ expect(port.close).toHaveBeenCalledOnce();+ });+});diff --git a/apps/desktop/src/services/transcription/session.ts b/apps/desktop/src/services/transcription/session.tsindex 2019e0a87d..ac436ae5a8 100644--- a/apps/desktop/src/services/transcription/session.ts+++ b/apps/desktop/src/services/transcription/session.ts@@ -1,7 +1,8 @@import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";+import type { MessagePortMain } from "electron";import type { SpeakerTracker, SpeechEngine, SpeechSegment, SpeechSegmenter } from "./engine";-import { type Logger, SAMPLE_RATE } from "./protocol";+import { type CaptureMessage, type Logger, SAMPLE_RATE } from "./protocol";/** The sources whose speakers are told apart; the microphone carries only the local speaker. */const DIARIZED_SOURCES: readonly TranscriptSource[] = [ "system" ];@@ -254,3 +255,52 @@ class SourcePipeline {}}}++/** The samples the capture page posts to the port the main process connects. */+export class PortCapture implements AudioCapture {+ /**+ * Settles once the page has posted its end marker or the port closed: every sample the page+ * captured has then been delivered.+ */+ readonly ended: Promise<void>;+ private port: MessagePortMain | null = null;+ private onSamples: SamplesListener | null = null;+ private markEnded = () => {};++ constructor() {+ this.ended = new Promise((resolve) => {+ this.markEnded = resolve;+ });+ }++ start(onSamples: SamplesListener) {+ this.onSamples = onSamples;+ }++ connect(port: MessagePortMain) {+ this.port = port;+ port.on("message", ({ data }) => {+ const message = data as CaptureMessage | null;+ if (!message) {+ return;+ }+ if ("end" in message) {+ this.markEnded();+ } else if (isSource(message.source) && message.samples instanceof Float32Array) {+ this.onSamples?.(message.source, message.samples);+ }+ });+ port.on("close", () => this.markEnded());+ port.start();+ }++ stop() {+ this.onSamples = null;+ this.port?.close();+ this.port = null;+ }+}++function isSource(source: unknown): source is TranscriptSource {+ return source === "microphone" || source === "system";+}diff --git a/apps/desktop/src/services/transcription/transcription_worker.ts b/apps/desktop/src/services/transcription/transcription_worker.tsindex abe6bfbea5..a646115632 100644--- a/apps/desktop/src/services/transcription/transcription_worker.ts+++ b/apps/desktop/src/services/transcription/transcription_worker.ts@@ -4,15 +4,11 @@* posts its samples here directly; the main process only receives the utterances. Started by* `TranscriptionWorker` in `worker_client.ts`, which also ends it.*/-import type { TranscriptSource } from "@triliumnext/commons";-import type { MessagePortMain } from "electron";import { TranscriptFileSink, WavRecorder } from "./audio_recorder";import { loadSherpaEngine } from "./engine";-import type { CaptureChunk, Logger, WorkerEvent, WorkerRequest } from "./protocol";-import {- type AudioCapture, type AudioTap, RecordingSession, type SamplesListener, type TranscriptSink-} from "./session";+import type { Logger, WorkerEvent, WorkerRequest } from "./protocol";+import { type AudioTap, PortCapture, RecordingSession, type TranscriptSink } from "./session";const log: Logger = {info: (message) => post({ type: "log", level: "info", message }),@@ -21,6 +17,9 @@ const log: Logger = {type StartRequest = Extract<WorkerRequest, { type: "start" }>;+/** How long stopping waits for the capture page's end marker. */+const CAPTURE_END_TIMEOUT_MS = 2000;+let session: RecordingSession | null = null;let capture: PortCapture | null = null;@@ -68,7 +67,15 @@ async function start({ noteId, modelDirectory, recordingDirectory }: StartRequespost({ type: "ready", startedAt: newSession.startedAt, settings: engine.settings });}+/**+ * Waits for the capture page's last chunks, which travel on another port than the stop request,+ * then transcribes what the session still holds.+ */async function stop() {+ if (capture) {+ const timeout = new Promise((resolve) => setTimeout(resolve, CAPTURE_END_TIMEOUT_MS));+ await Promise.race([ capture.ended, timeout ]);+ }try {await session?.stop();} catch (e) {@@ -80,34 +87,3 @@ async function stop() {function post(event: WorkerEvent) {process.parentPort.postMessage(event);}--/** The samples the capture page posts to the port the main process connects. */-class PortCapture implements AudioCapture {- private port: MessagePortMain | null = null;- private onSamples: SamplesListener | null = null;-- start(onSamples: SamplesListener) {- this.onSamples = onSamples;- }-- connect(port: MessagePortMain) {- this.port = port;- port.on("message", ({ data }) => {- const chunk = data as CaptureChunk | null;- if (chunk && isSource(chunk.source) && chunk.samples instanceof Float32Array) {- this.onSamples?.(chunk.source, chunk.samples);- }- });- port.start();- }-- stop() {- this.onSamples = null;- this.port?.close();- this.port = null;- }-}--function isSource(source: unknown): source is TranscriptSource {- return source === "microphone" || source === "system";-}diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.jsonindex 0bfb215580..c880021410 100644--- a/apps/server/src/assets/translations/en/server.json+++ b/apps/server/src/assets/translations/en/server.json@@ -568,6 +568,9 @@"model-missing": "No speech recognition model was found in {{- directory}}.","start-failed": "The transcription could not start: {{- message}}","stopped-unexpectedly": "The transcription stopped unexpectedly. What was transcribed until then is kept.",+ "save-locked": "The transcript cannot be saved while the protected session is locked. The transcription keeps it and saves it once you enter the protected session, as long as Trilium stays open.",+ "save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.",+ "capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.","attachment-title": "Transcript"}}diff --git a/packages/commons/src/lib/electron_api_interface.ts b/packages/commons/src/lib/electron_api_interface.tsindex 7b762fa23b..3d89d189ee 100644--- a/packages/commons/src/lib/electron_api_interface.ts+++ b/packages/commons/src/lib/electron_api_interface.ts@@ -725,8 +725,8 @@ export interface TranscriptionState {/** The sources heard speaking, or whose speech is still being transcribed. */pendingSources: TranscriptSource[];/**- * Why the recording ended without being stopped, sent once with the `idle` state that- * follows; `getState()` never returns it.+ * A problem to tell the user about, sent once with the state it concerns: the recording ended+ * without being stopped, or its transcript cannot be saved. `getState()` never returns it.*/error?: string;}@@ -748,6 +748,15 @@ export interface ElectronTranscriptionApi {getState(): Promise<TranscriptionState>;+ /**+ * Names a speaker of the transcript the recording of `noteId` writes, or restores the default+ * name when `name` is blank. The recording applies it, so no utterance it writes meanwhile is+ * lost.+ *+ * @returns `false` when no recording of `noteId` runs; the transcript can then be saved whole.+ */+ renameSpeaker(noteId: string, key: string, name: string): Promise<boolean>;+/*** Calls `callback` every time the state changes, in any window.*