Skip to content

Build your personal knowledge base with Trilium Notes

fix(desktop): keep meeting transcripts whole on stop, quit and failure

Elian Doran1469217b508c
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.md
index b7b6e666d0..b83f65d809 100644
--- a/.claude/skills/working-with-transcription/SKILL.md
+++ b/.claude/skills/working-with-transcription/SKILL.md
@@ -26,7 +26,11 @@ main process: TranscriptionWorker → AttachmentSink (transcript attachment)
state broadcast "transcription-state" (status, noteId, pendingSources, error)
```
-The main process opens the capture window, starts the transcription process and connects the two with a `MessageChannelMain`; per chunk it does nothing, per utterance it writes the attachment. Each recording gets a process of its own, which loads the model (~1.3 s) and exits after `stop`. If it dies mid-recording, the main process closes the capture window, keeps what was written, and broadcasts an `idle` state with an `error`, which the launcher shows as a toast.
+The main process opens the capture window, starts the transcription process and connects the two with a `MessageChannelMain`; per chunk it does nothing, per utterance it writes the attachment. Each recording gets a process of its own, which loads the model (~1.3 s) and exits after `stop`.
+
+- **Stopping** runs in order: `CaptureWindow.stop()` calls the page's `finishCapture()`, which flushes each worklet's partial chunk (padded with silence to whole 512-sample VAD windows) and posts `{ end: true }`; then the process, told to stop, waits for that marker (it travels on another port than the stop request, up to 2 s) before flushing the VADs and the decode queue.
+- **Quitting** during a recording is held back (`before-quit`) until that stop finishes, at most 15 s.
+- **Failures** reach every window as a state with an `error`, which the launcher shows as a toast; `getState()` never returns it. The process dying ends the recording, keeping what was written; the capture page crashing or every source ending (`render-process-gone`, a track's `ended`) stops it normally, transcribing what was captured; a transcript that cannot be saved (a protected note while the protected session is locked) is reported once and kept in memory, and `AttachmentSink.save()` writes it on the next utterance or on `ENTER_PROTECTED_SESSION`.
| File | Owns |
|---|---|
@@ -43,7 +47,7 @@ The main process opens the capture window, starts the transcription process and
Nothing the transcription process imports may reach `@triliumnext/core` or `@triliumnext/server` (the data directory module creates directories when it loads): it gets its paths and a `Logger` from the main process.
| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `speakers`, `utterances`), `parseTranscript()`, speaker keys and `setSpeakerName()`; `cleanUtteranceText()`, the filler and stutter cleanup |
| Cleanup | The stored transcript stays verbatim. `TranscriptView` shows each utterance through `cleanUtteranceText()` while the synced option `transcriptCleanupEnabled` (default on, the "Hide filler words" switch above the transcript) holds, and leaves out utterances it empties. The rules are English-only for stutters (a function-word list) and keep numbers, comma-separated repeats and "that that"/"it it" |
-| Speaker names | `TranscriptView` reads the stored transcript, applies `setSpeakerName()` and saves it whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename during a recording survives, and an utterance a rename overwrote comes back with the next write |
+| Speaker names | While its note is being transcribed, `TranscriptView` renames through `electronApi.transcription.renameSpeaker()`: `AttachmentSink`, the only writer then, applies `setSpeakerName()` to the stored names in one transaction, so no utterance is overwritten. Otherwise it saves the stored transcript whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`; attachments get no route of their own. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename from another window or device survives the next utterance |
| `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in |
| `apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx` | The launcher and `useTranscriptionState()` |
diff --git a/.claude/skills/working-with-transcription/capture-test.mjs b/.claude/skills/working-with-transcription/capture-test.mjs
index d545c3abbe..1a5856ed0f 100644
--- a/.claude/skills/working-with-transcription/capture-test.mjs
+++ b/.claude/skills/working-with-transcription/capture-test.mjs
@@ -178,7 +178,7 @@ function buildPage(config, processor) {
numberOfOutputs: 0,
channelCount: 1,
channelCountMode: "explicit",
- processorOptions: { chunkSamples: CHUNK_SAMPLES }
+ processorOptions: { chunkSamples: CHUNK_SAMPLES, windowSamples: 512 }
});
node.port.onmessage = (e) => ipcRenderer.send("samples", e.data);
source.connect(node);
diff --git a/.claude/skills/working-with-transcription/worker-test.mjs b/.claude/skills/working-with-transcription/worker-test.mjs
index 4b11940612..6e74c85644 100644
--- a/.claude/skills/working-with-transcription/worker-test.mjs
+++ b/.claude/skills/working-with-transcription/worker-test.mjs
@@ -135,8 +135,6 @@ async function runInElectron(config) {
worker.connectCapture(port2);
window.webContents.postMessage("transcription-capture-port", null, [ port1 ]);
const chunks = await posted;
- // Lets the last chunks reach the process before it is told to stop.
- await new Promise((resolve) => setTimeout(resolve, 500));
const stopAt = performance.now();
await worker.stop();
@@ -186,6 +184,7 @@ function buildPage(config) {
chunks++;
await new Promise((resolve) => setTimeout(resolve, CHUNK_SAMPLES / RATE * 1000 / config.speed));
}
+ port.postMessage({ end: true });
console.log(`posted ${chunks}`);
});
};
diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx
index 07daa4dd35..9246025c09 100644
--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx
+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx
@@ -86,6 +86,21 @@ describe("transcript helpers", () => {
]);
});
+ it("orders turns by when they were spoken, not by when they were transcribed", () => {
+ // The microphone's short reply ends, and is stored, before the long system utterance
+ // that started first.
+ const turns = groupIntoTurns([
+ { source: "microphone", offsetMs: 6000, durationMs: 1000, text: "Yes." },
+ { source: "system", offsetMs: 0, durationMs: 15_000, text: "Can you hear me?" },
+ { source: "system", offsetMs: 16_000, durationMs: 1000, text: "Good." }
+ ]);
+ expect(turns.map(({ source, offsetMs }) => ({ source, offsetMs }))).toEqual([
+ { source: "system", offsetMs: 0 },
+ { source: "microphone", offsetMs: 6000 },
+ { source: "system", offsetMs: 16_000 }
+ ]);
+ });
+
it("starts a new paragraph within a turn after a long pause", () => {
const turns = groupIntoTurns([
{ source: "system", offsetMs: 0, durationMs: 8000, text: "First point," },
@@ -284,6 +299,48 @@ describe("TranscriptView", () => {
expect(h.post).toHaveBeenCalledTimes(2);
});
+ it("renames a speaker through the recording that writes the transcript, while it runs", async () => {
+ const renameSpeaker = vi.fn(async () => true);
+ window.electronApi = {
+ transcription: {
+ getState: async () => ({ status: "recording", noteId: "meeting", pendingSources: [] }),
+ onStateChanged: () => () => {},
+ renameSpeaker
+ }
+ } as unknown as typeof window.electronApi;
+ const getBlob = vi.fn(async () => ({
+ content: transcript([ { source: "system", offsetMs: 0, speaker: 1, text: "Hi." } ])
+ }));
+ const attachment = { attachmentId: "transcript1", ownerId: "meeting", getBlob };
+ const container = renderInto(
+ <ParentComponent.Provider value={new Component()}>
+ <TranscriptView attachment={attachment as unknown as FAttachment} />
+ </ParentComponent.Provider>
+ );
+ const speaker = () => container.querySelector<HTMLElement>(".transcript-speaker");
+ await vi.waitFor(() => expect(speaker()).not.toBeNull());
+ const posted = h.post.mock.calls.length;
+
+ await act(async () => speaker()?.click());
+ const input = container.querySelector<HTMLInputElement>(".transcript-speaker-editor");
+ expect(input).not.toBeNull();
+ act(() => {
+ if (input) {
+ input.value = "Alexandra";
+ input.dispatchEvent(new Event("input", { bubbles: true }));
+ }
+ });
+ act(() => {
+ input?.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true }));
+ });
+ await vi.waitFor(() => expect(renameSpeaker)
+ .toHaveBeenCalledExactlyOnceWith("meeting", "system:1", "Alexandra"));
+ await new Promise((resolve) => setTimeout(resolve));
+ expect(h.post).toHaveBeenCalledTimes(posted);
+ expect(getBlob).toHaveBeenCalledOnce();
+ window.electronApi = undefined;
+ });
+
it("shows who is speaking in the recorded meeting, before the words arrive", async () => {
let onStateChanged: (state: TranscriptionState) => void = () => {};
const recording = (noteId: string, pendingSources: TranscriptSource[]) =>
diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx
index 876e3e2118..6729551328 100644
--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx
+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx
@@ -231,16 +231,19 @@ function SpeakerName({ speakerKey, name, defaultName, onRename }: {
}
/**
- * Names a speaker in the transcript as it is stored now, rather than as last shown: a running
- * recording can have written utterances since. One written in the moment between this read and
- * the save is restored by the recording's next write, which rewrites every utterance it holds.
+ * Names a speaker. While this note is being transcribed, the recording applies the name, since it
+ * rewrites the transcript after every utterance and a whole-transcript save here could overwrite
+ * one. Otherwise the transcript as stored now is saved with the name.
*/
async function renameSpeaker(attachment: FAttachment, key: string, name: string) {
+ const { attachmentId, ownerId, role, mime, title } = attachment;
+ if (await window.electronApi?.transcription?.renameSpeaker(ownerId, key, name)) {
+ return;
+ }
const latest = parseTranscript((await attachment.getBlob())?.content);
if (!latest) {
return;
}
- const { attachmentId, ownerId, role, mime, title } = attachment;
await server.post(`notes/${ownerId}/attachments?matchBy=attachmentId`, {
attachmentId, role, mime, title,
content: JSON.stringify(setSpeakerName(latest, key, name), null, 2)
@@ -282,6 +285,8 @@ export interface TranscriptTurn {
/**
* Joins each run of utterances from one source and speaker into a turn, timed by its first
* utterance, and starts a new paragraph after a pause of {@link PARAGRAPH_PAUSE_MS} or more.
+ * Utterances are taken in the order they began: a recording stores them as they finish, so a
+ * short reply can precede the longer utterance it answers.
* Speakers given the same name count as one. An utterance without a speaker continues the turn of
* its source, and an utterance without a `durationMs` never starts a paragraph, since the pause
* before the next is unknown.
@@ -295,7 +300,7 @@ export function groupIntoTurns(
};
const turns: TranscriptTurn[] = [];
let previousEndMs: number | null = null;
- for (const utterance of utterances) {
+ for (const utterance of utterances.toSorted((a, b) => a.offsetMs - b.offsetMs)) {
const { source, speaker, offsetMs, durationMs } = utterance;
const last = turns.at(-1);
const isSameSpeaker = speaker === undefined || last?.speaker === undefined
diff --git a/apps/desktop/src/preload.spec.ts b/apps/desktop/src/preload.spec.ts
index 48c161f7ac..307dbb9ac4 100644
--- a/apps/desktop/src/preload.spec.ts
+++ b/apps/desktop/src/preload.spec.ts
@@ -724,14 +724,19 @@ describe("preload script", () => {
describe("transcription", () => {
const transcription = () => getGroup("transcription");
- it("start, stop and getState invoke their IPC channels", async () => {
+ it("start, stop, getState and renameSpeaker invoke their IPC channels", async () => {
await transcription().start("meeting");
await transcription().stop();
await transcription().getState();
+ await transcription().renameSpeaker("meeting", "system:1", "Alexandra");
expect(ipcRendererInvoked).toEqual([
{ channel: "transcription-start", args: [ "meeting" ] },
{ channel: "transcription-stop", args: [] },
- { channel: "transcription-get-state", args: [] }
+ { channel: "transcription-get-state", args: [] },
+ {
+ channel: "transcription-rename-speaker",
+ args: [ "meeting", "system:1", "Alexandra" ]
+ }
]);
});
diff --git a/apps/desktop/src/preload.ts b/apps/desktop/src/preload.ts
index 7d13bb03bf..dc35e0e7f0 100644
--- a/apps/desktop/src/preload.ts
+++ b/apps/desktop/src/preload.ts
@@ -264,6 +264,9 @@ contextBridge.exposeInMainWorld("electronApi", {
getState() {
return ipcRenderer.invoke("transcription-get-state");
},
+ renameSpeaker(noteId: string, key: string, name: string) {
+ return ipcRenderer.invoke("transcription-rename-speaker", noteId, key, name);
+ },
onStateChanged(callback: (state: TranscriptionState) => void) {
const listener = (_event: unknown, state: TranscriptionState) => callback(state);
ipcRenderer.on("transcription-state", listener);
diff --git a/apps/desktop/src/services/transcription/attachment_sink.spec.ts b/apps/desktop/src/services/transcription/attachment_sink.spec.ts
index 476334961d..7484e317e4 100644
--- a/apps/desktop/src/services/transcription/attachment_sink.spec.ts
+++ b/apps/desktop/src/services/transcription/attachment_sink.spec.ts
@@ -84,6 +84,40 @@ describe("AttachmentSink", () => {
expect(written.utterances).toHaveLength(1);
});
+ it("keeps an utterance it could not save, and saves it with the next write or save()", () => {
+ const sink = new AttachmentSink("meeting");
+ sink.open(STARTED_AT);
+ h.setContent.mockImplementationOnce(() => {
+ throw new Error("protected session is not available");
+ });
+ expect(() => sink.write({ source: "system", offsetMs: 0, text: "First." }))
+ .toThrow("protected session is not available");
+ sink.write({ source: "system", offsetMs: 1000, text: "Second." });
+ const saved = JSON.parse(h.setContent.mock.calls[1][0]) as Transcript;
+ expect(saved.utterances.map((utterance) => utterance.text)).toEqual([ "First.", "Second." ]);
+
+ sink.save();
+ expect(h.setContent).toHaveBeenCalledTimes(3);
+ expect(h.setContent.mock.calls[2][0]).toBe(h.setContent.mock.calls[1][0]);
+ });
+
+ it("renames a speaker in the names as stored, keeping every utterance, in one transaction", () => {
+ const sink = new AttachmentSink("meeting");
+ sink.open(STARTED_AT);
+ sink.write({ source: "system", offsetMs: 0, speaker: 1, text: "Hello." });
+ h.getContent.mockReturnValue(JSON.stringify({
+ version: 1, startedAt: "2026-10-13T14:30:00.000Z",
+ speakers: { microphone: "Elian" }, utterances: []
+ }));
+ const transactions = h.transactions;
+
+ sink.renameSpeaker("system:1", " Alexandra ");
+ expect(h.transactions).toBe(transactions + 1);
+ const renamed = JSON.parse(h.setContent.mock.calls.at(-1)?.[0]) as Transcript;
+ expect(renamed.speakers).toEqual({ microphone: "Elian", "system:1": "Alexandra" });
+ expect(renamed.utterances).toHaveLength(1);
+ });
+
it("refuses to write before it is open", () => {
const sink = new AttachmentSink("meeting");
expect(() => sink.write({ source: "system", offsetMs: 0, text: "Hi" }))
diff --git a/apps/desktop/src/services/transcription/attachment_sink.ts b/apps/desktop/src/services/transcription/attachment_sink.ts
index bb60dd41ef..dc4dcbe149 100644
--- a/apps/desktop/src/services/transcription/attachment_sink.ts
+++ b/apps/desktop/src/services/transcription/attachment_sink.ts
@@ -1,5 +1,5 @@
import {
- parseTranscript, type Transcript, TRANSCRIPT_MIME, type TranscriptUtterance
+ parseTranscript, setSpeakerName, type Transcript, TRANSCRIPT_MIME, type TranscriptUtterance
} from "@triliumnext/commons";
import { becca, binary_utils, cls, getSql } from "@triliumnext/core";
import { t } from "i18next";
@@ -9,7 +9,9 @@ import type { TranscriptSink } from "./session";
/**
* Keeps the transcript in a `transcript` attachment of the meeting note, rewritten after every
* utterance. Each write commits its own transaction, which sends the change to every open window,
- * and keeps the speaker names stored in the attachment, which the user can change meanwhile.
+ * and keeps the speaker names stored in the attachment, which the user can change meanwhile. An
+ * utterance that cannot be saved (a protected note while the protected session is locked) is kept
+ * and saved with the next write or {@link save}.
*/
export class AttachmentSink implements TranscriptSink {
private transcript: Transcript | null = null;
@@ -39,15 +41,42 @@ export class AttachmentSink implements TranscriptSink {
}
transcript.utterances.push(utterance);
+ this.save();
+ }
+
+ /** Rewrites the attachment with every utterance written so far. */
+ save() {
+ this.rewrite((speakers) => speakers);
+ }
+
+ /**
+ * Names a speaker in the names as stored, which another window can have changed, or restores
+ * its default name when `name` is blank.
+ */
+ renameSpeaker(key: string, name: string) {
+ this.rewrite((speakers, transcript) =>
+ setSpeakerName({ ...transcript, speakers }, key, name).speakers);
+ }
+
+ /** Rewrites the attachment in one transaction, the stored names passed through `rename`. */
+ private rewrite(rename: (speakers: SpeakerNames | undefined, transcript: Transcript) =>
+ SpeakerNames | undefined) {
+ const { transcript, attachmentId } = this;
+ if (!transcript || !attachmentId) {
+ throw new Error("The transcript attachment was not created.");
+ }
+
inTransaction(() => {
const attachment = becca.getAttachmentOrThrow(attachmentId);
const stored = parseTranscript(binary_utils.decodeUtf8(attachment.getContent()));
- transcript.speakers = stored?.speakers;
+ transcript.speakers = rename(stored?.speakers, transcript);
attachment.setContent(serialize(transcript));
});
}
}
+type SpeakerNames = Transcript["speakers"];
+
function serialize(transcript: Transcript) {
return JSON.stringify(transcript, null, 2);
}
diff --git a/apps/desktop/src/services/transcription/capture.spec.ts b/apps/desktop/src/services/transcription/capture.spec.ts
new file mode 100644
index 0000000000..683521482a
--- /dev/null
+++ b/apps/desktop/src/services/transcription/capture.spec.ts
@@ -0,0 +1,136 @@
+import { EventEmitter } from "events";
+import { describe, expect, it, vi } from "vitest";
+
+const h = vi.hoisted(() => ({
+ windows: [] as FakeWindow[],
+ log: { info: vi.fn(), error: vi.fn() }
+}));
+
+/** The parts of a `BrowserWindow` the capture window uses. */
+class FakeWindow {
+ destroyed = false;
+ webContents = Object.assign(new EventEmitter(), {
+ ipc: new EventEmitter(),
+ postMessage: vi.fn(),
+ executeJavaScript: vi.fn(async (call: string) => call)
+ });
+
+ constructor() {
+ h.windows.push(this);
+ }
+
+ loadFile = vi.fn(async () => {});
+ isDestroyed = () => this.destroyed;
+ destroy = vi.fn(() => {
+ this.destroyed = true;
+ });
+}
+
+vi.mock("electron", () => ({
+ app: { getPath: () => "/tmp" },
+ BrowserWindow: FakeWindow,
+ session: {
+ fromPartition: () => ({
+ setPermissionRequestHandler: vi.fn(),
+ setPermissionCheckHandler: vi.fn(),
+ setDisplayMediaRequestHandler: vi.fn()
+ })
+ }
+}));
+vi.mock("fs", () => ({ default: { writeFileSync: vi.fn() } }));
+vi.mock("@triliumnext/core", () => ({ getLog: () => h.log }));
+
+const { CAPTURE_PROCESSOR, CaptureWindow } = await import("./capture");
+
+interface FakeProcessor {
+ port: {
+ postMessage: ReturnType<typeof vi.fn>;
+ onmessage: ((event: { data: unknown }) => void) | null;
+ };
+ process(inputs: Float32Array[][]): boolean;
+}
+
+/** Runs the worklet's source with fake `AudioWorkletProcessor` and `registerProcessor` globals. */
+function createProcessor(chunkSamples: number, windowSamples: number): FakeProcessor {
+ class AudioWorkletProcessor {
+ port = { postMessage: vi.fn(), onmessage: null };
+ }
+ let Processor: (new (options: object) => FakeProcessor) | null = null;
+ const registerProcessor = (_name: string, processor: typeof Processor) => {
+ Processor = processor;
+ };
+ new Function("AudioWorkletProcessor", "registerProcessor", CAPTURE_PROCESSOR)(
+ AudioWorkletProcessor, registerProcessor);
+ if (!Processor) {
+ throw new Error("The worklet registered no processor.");
+ }
+ return new (Processor as new (options: object) => FakeProcessor)({
+ processorOptions: { chunkSamples, windowSamples }
+ });
+}
+
+/** One 128-frame render quantum whose samples all hold `value`. */
+function quantum(value: number) {
+ return [ [ new Float32Array(128).fill(value) ] ];
+}
+
+describe("CAPTURE_PROCESSOR", () => {
+ it("posts whole chunks, then on a flush what it holds, padded to whole windows", () => {
+ const processor = createProcessor(512, 256);
+ for (let i = 1; i <= 5; i++) {
+ processor.process(quantum(i));
+ }
+ expect(processor.port.postMessage).toHaveBeenCalledOnce();
+ expect([ ...processor.port.postMessage.mock.calls[0][0] as Float32Array ])
+ .toEqual([ 1, 2, 3, 4 ].flatMap((value) => new Array(128).fill(value)));
+
+ processor.port.onmessage?.({ data: "flush" });
+ const [ , [ partial ], [ done ] ] = processor.port.postMessage.mock.calls;
+ expect([ ...partial as Float32Array ])
+ .toEqual([ ...new Array(128).fill(5), ...new Array(128).fill(0) ]);
+ expect(done).toBe("flushed");
+
+ processor.port.onmessage?.({ data: "flush" });
+ expect(processor.port.postMessage.mock.calls.at(-1)?.[0]).toBe("flushed");
+ expect(processor.port.postMessage).toHaveBeenCalledTimes(4);
+ });
+});
+
+describe("CaptureWindow", () => {
+ async function startCapture() {
+ h.windows.length = 0;
+ const onLost = vi.fn();
+ const capture = new CaptureWindow({ onLost });
+ expect(await capture.start("port" as never)).toEqual([ "microphone", "system" ]);
+ const window = h.windows[0];
+ expect(window.webContents.postMessage)
+ .toHaveBeenCalledExactlyOnceWith("transcription-capture-port", null, [ "port" ]);
+ return { capture, onLost, window };
+ }
+
+ it("reports the capture lost when its page crashes", async () => {
+ const { onLost, window } = await startCapture();
+ window.webContents.emit("render-process-gone", {}, { reason: "oom" });
+ window.webContents.emit("render-process-gone", {}, { reason: "oom" });
+ expect(onLost).toHaveBeenCalledExactlyOnceWith("the capture page ended (oom)");
+ });
+
+ it("reports the capture lost once every source has ended", async () => {
+ const { onLost, window } = await startCapture();
+ window.webContents.ipc.emit("transcription-source-ended", {}, "microphone");
+ expect(onLost).not.toHaveBeenCalled();
+ expect(h.log.error).toHaveBeenCalledWith("Transcription: the microphone source ended.");
+ window.webContents.ipc.emit("transcription-source-ended", {}, "system");
+ expect(onLost).toHaveBeenCalledExactlyOnceWith("every audio source ended");
+ });
+
+ it("finishes the page before closing it, and reports nothing once stopped", async () => {
+ const { capture, onLost, window } = await startCapture();
+ await capture.stop();
+ expect(window.webContents.executeJavaScript).toHaveBeenLastCalledWith("finishCapture()");
+ expect(window.destroy).toHaveBeenCalledOnce();
+
+ window.webContents.emit("render-process-gone", {}, { reason: "killed" });
+ expect(onLost).not.toHaveBeenCalled();
+ });
+});
diff --git a/apps/desktop/src/services/transcription/capture.ts b/apps/desktop/src/services/transcription/capture.ts
index b7442c2c34..61f205bdbe 100644
--- a/apps/desktop/src/services/transcription/capture.ts
+++ b/apps/desktop/src/services/transcription/capture.ts
@@ -7,10 +7,26 @@ import path from "path";
import { SAMPLE_RATE } from "./protocol";
const PORT_CHANNEL = "transcription-capture-port";
+const SOURCE_ENDED_CHANNEL = "transcription-source-ended";
-/** Samples per message. `VadSegmenter` drops what does not fill a 512-sample window. */
+export interface CaptureWindowOptions {
+ /**
+ * Called once when capturing stops on its own: the page crashed, or every source ended (a
+ * device was unplugged, the system audio went away). Not called after
+ * {@link CaptureWindow.stop}.
+ */
+ onLost(reason: string): void;
+}
+
+/** Samples per message. A multiple of {@link VAD_WINDOW_SAMPLES}. */
const CHUNK_SAMPLES = 4096;
+/** `VadSegmenter` drops what does not fill a window, so the last chunk is padded to whole ones. */
+const VAD_WINDOW_SAMPLES = 512;
+
+/** How long {@link CaptureWindow.stop} waits for the page to post what it still holds. */
+const FINISH_TIMEOUT_MS = 1000;
+
/** The page function that starts each source. */
const SOURCE_STARTERS = [
[ "microphone", "captureMicrophone()" ],
@@ -24,6 +40,10 @@ const SOURCE_STARTERS = [
*/
export class CaptureWindow {
private window: BrowserWindow | null = null;
+ private readonly running = new Set<TranscriptSource>();
+ private isLost = false;
+
+ constructor(private readonly options: CaptureWindowOptions) {}
/**
* Starts every source it can, posting to `port`, and returns the ones that run.
@@ -33,6 +53,15 @@ export class CaptureWindow {
async start(port: MessagePortMain) {
const window = await createCaptureWindow();
this.window = window;
+ window.webContents.on("render-process-gone", (_event, { reason }) =>
+ this.lose(`the capture page ended (${reason})`));
+ window.webContents.ipc.on(SOURCE_ENDED_CHANNEL, (_event, source: TranscriptSource) => {
+ getLog().error(`Transcription: the ${source} source ended.`);
+ this.running.delete(source);
+ if (!this.running.size) {
+ this.lose("every audio source ended");
+ }
+ });
window.webContents.postMessage(PORT_CHANNEL, null, [ port ]);
const sources: TranscriptSource[] = [];
@@ -43,6 +72,7 @@ export class CaptureWindow {
const tracks = await window.webContents.executeJavaScript(call, true);
getLog().info(`Transcription: capturing ${source} from ${tracks}.`);
sources.push(source);
+ this.running.add(source);
} catch (e) {
const message = (e as Error)?.message ?? e;
getLog().error(`Transcription: ${source} unavailable: ${message}`);
@@ -56,11 +86,38 @@ export class CaptureWindow {
return sources;
}
- stop() {
- if (this.window && !this.window.isDestroyed()) {
- this.window.destroy();
- }
+ /**
+ * Posts what the worklets still hold, then an end marker the transcription process waits for,
+ * and closes the window. Gives up on the page after {@link FINISH_TIMEOUT_MS}.
+ */
+ async stop() {
+ const window = this.window;
this.window = null;
+ this.running.clear();
+ if (!window || window.isDestroyed()) {
+ return;
+ }
+ try {
+ await Promise.race([
+ window.webContents.executeJavaScript("finishCapture()"),
+ new Promise((_resolve, reject) => setTimeout(
+ () => reject(new Error("timed out")), FINISH_TIMEOUT_MS))
+ ]);
+ } catch (e) {
+ const message = (e as Error)?.message ?? e;
+ getLog().error(`Transcription: finishing the capture failed: ${message}`);
+ } finally {
+ if (!window.isDestroyed()) {
+ window.destroy();
+ }
+ }
+ }
+
+ private lose(reason: string) {
+ if (this.window && !this.isLost) {
+ this.isLost = true;
+ this.options.onLost(reason);
+ }
}
}
@@ -109,15 +166,29 @@ async function createCaptureWindow() {
/**
* The audio worklet that collects a source into chunks of `processorOptions.chunkSamples`. It runs
* on the audio rendering thread, where a busy page cannot delay it and corrupt the start of a
- * chunk. `.claude/skills/working-with-transcription/capture-test.mjs` reads it from this file.
+ * chunk. A `"flush"` message posts the partial chunk, padded with silence to whole
+ * `processorOptions.windowSamples`, then `"flushed"`.
+ * `.claude/skills/working-with-transcription/capture-test.mjs` reads it from this file.
*/
-const CAPTURE_PROCESSOR = `
+export const CAPTURE_PROCESSOR = `
class ChunkCollector extends AudioWorkletProcessor {
constructor({ processorOptions }) {
super();
this.chunkSamples = processorOptions.chunkSamples;
+ this.windowSamples = processorOptions.windowSamples;
this.chunk = new Float32Array(this.chunkSamples);
this.filled = 0;
+ this.port.onmessage = () => this.flush();
+ }
+
+ flush() {
+ const length = Math.ceil(this.filled / this.windowSamples) * this.windowSamples;
+ if (length) {
+ this.port.postMessage(this.chunk.slice(0, length));
+ this.chunk = new Float32Array(this.chunkSamples);
+ this.filled = 0;
+ }
+ this.port.postMessage("flushed");
}
process(inputs) {
@@ -149,6 +220,8 @@ const context = new AudioContext({ sampleRate: ${SAMPLE_RATE} });
const processorLoaded = context.audioWorklet.addModule(URL.createObjectURL(
new Blob([ ${JSON.stringify(CAPTURE_PROCESSOR)} ], { type: "text/javascript" })));
+const flushes = [];
+
async function pipe(source, media) {
const [ port ] = await Promise.all([ portReceived, processorLoaded ]);
const collector = new AudioWorkletNode(context, "chunk-collector", {
@@ -156,22 +229,47 @@ async function pipe(source, media) {
numberOfOutputs: 0,
channelCount: 1,
channelCountMode: "explicit",
- processorOptions: { chunkSamples: ${CHUNK_SAMPLES} }
+ processorOptions: { chunkSamples: ${CHUNK_SAMPLES}, windowSamples: ${VAD_WINDOW_SAMPLES} }
});
+ let onFlushed = () => {};
// Copied, not transferred: a MessagePortMain receives a message with a transferred buffer
// as null.
- collector.port.onmessage = (e) => port.postMessage({ source, samples: e.data });
+ collector.port.onmessage = (e) => e.data === "flushed"
+ ? onFlushed()
+ : port.postMessage({ source, samples: e.data });
+ flushes.push(() => new Promise((resolve) => {
+ onFlushed = resolve;
+ collector.port.postMessage("flush");
+ }));
context.createMediaStreamSource(media).connect(collector);
}
+async function finishCapture() {
+ await Promise.all(flushes.map((flush) => flush()));
+ (await portReceived).postMessage({ end: true });
+}
+
function describe(tracks) {
return tracks.map((track) => track.label).join(", ");
}
+function reportEnd(source, tracks) {
+ let isReported = false;
+ for (const track of tracks) {
+ track.addEventListener("ended", () => {
+ if (!isReported) {
+ isReported = true;
+ ipcRenderer.send("${SOURCE_ENDED_CHANNEL}", source);
+ }
+ });
+ }
+}
+
async function captureMicrophone() {
const media = await navigator.mediaDevices.getUserMedia({
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: false }
});
+ reportEnd("microphone", media.getAudioTracks());
await pipe("microphone", media);
return describe(media.getAudioTracks());
}
@@ -194,6 +292,7 @@ async function captureSystemAudio() {
console.log("System track muted=" + track.muted + " settings="
+ JSON.stringify(track.getSettings()));
}
+ reportEnd("system", audioTracks);
await pipe("system", new MediaStream(audioTracks));
return describe(audioTracks);
}
diff --git a/apps/desktop/src/services/transcription/index.spec.ts b/apps/desktop/src/services/transcription/index.spec.ts
index adb5e5ed3f..cbf2b95f80 100644
--- a/apps/desktop/src/services/transcription/index.spec.ts
+++ b/apps/desktop/src/services/transcription/index.spec.ts
@@ -1,22 +1,34 @@
import type { TranscriptionState } from "@triliumnext/commons";
import { beforeEach, describe, expect, it, vi } from "vitest";
+import type { CaptureWindowOptions } from "./capture";
import type { WorkerListeners } from "./worker_client";
const h = vi.hoisted(() => ({
handlers: new Map<string, (...args: unknown[]) => unknown>(),
+ appListeners: new Map<string, (...args: unknown[]) => unknown>(),
+ quit: vi.fn(),
broadcast: [] as TranscriptionState[],
notes: new Set<string>([ "meeting" ]),
+ isProtected: false,
+ isProtectedSessionAvailable: false,
+ coreListeners: new Map<string, () => void>(),
isDev: false,
isModelInstalled: true,
log: { info: vi.fn(), error: vi.fn() },
listeners: null as WorkerListeners | null,
+ captureOptions: null as CaptureWindowOptions | null,
worker: { start: vi.fn(), connectCapture: vi.fn(), stop: vi.fn(), kill: vi.fn() },
capture: { start: vi.fn(), stop: vi.fn() },
- sink: { open: vi.fn(), write: vi.fn() }
+ sink: { open: vi.fn(), write: vi.fn(), save: vi.fn(), renameSpeaker: vi.fn() }
}));
vi.mock("electron", () => ({
+ app: {
+ on: (event: string, listener: (...args: unknown[]) => unknown) =>
+ h.appListeners.set(event, listener),
+ quit: h.quit
+ },
ipcMain: {
handle: (channel: string, handler: (...args: unknown[]) => unknown) =>
h.handlers.set(channel, handler)
@@ -39,8 +51,16 @@ vi.mock("electron", () => ({
}));
vi.mock("fs", () => ({ default: { existsSync: () => h.isModelInstalled } }));
vi.mock("@triliumnext/core", () => ({
- becca: { getNote: (noteId: string) => h.notes.has(noteId) ? { noteId } : null },
+ becca: {
+ getNote: (noteId: string) =>
+ h.notes.has(noteId) ? { noteId, isProtected: h.isProtected } : null
+ },
+ events: {
+ ENTER_PROTECTED_SESSION: "ENTER_PROTECTED_SESSION",
+ subscribe: (event: string, listener: () => void) => h.coreListeners.set(event, listener)
+ },
getLog: () => h.log,
+ protected_session: { isProtectedSessionAvailable: () => h.isProtectedSessionAvailable },
utils: { isDev: () => h.isDev }
}));
vi.mock("i18next", () => ({
@@ -53,12 +73,18 @@ vi.mock("./capture", () => ({
CaptureWindow: class {
start = h.capture.start;
stop = h.capture.stop;
+
+ constructor(options: CaptureWindowOptions) {
+ h.captureOptions = options;
+ }
}
}));
vi.mock("./attachment_sink", () => ({
AttachmentSink: class {
open = h.sink.open;
write = h.sink.write;
+ save = h.sink.save;
+ renameSpeaker = h.sink.renameSpeaker;
constructor(readonly noteId: string) {}
}
@@ -89,12 +115,18 @@ async function loadService() {
beforeEach(() => {
vi.clearAllMocks();
h.handlers.clear();
+ h.appListeners.clear();
+ h.coreListeners.clear();
+ h.isProtected = false;
+ h.isProtectedSessionAvailable = false;
h.broadcast.length = 0;
h.isDev = false;
h.isModelInstalled = true;
h.listeners = null;
h.worker.start.mockResolvedValue({ type: "ready", startedAt: STARTED_AT, settings: {} });
h.capture.start.mockResolvedValue([ "microphone", "system" ]);
+ h.worker.stop.mockResolvedValue(undefined);
+ h.capture.stop.mockResolvedValue(undefined);
});
describe("transcription IPC", () => {
@@ -117,6 +149,23 @@ describe("transcription IPC", () => {
expect(await invoke("transcription-get-state")).toEqual(IDLE);
});
+ it("stops the engine only once the capture window has posted what it still held", async () => {
+ const invoke = await loadService();
+ await invoke("transcription-start", "meeting");
+ let finishCapture = () => {};
+ h.capture.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {
+ finishCapture = resolve;
+ }));
+
+ const stopped = invoke("transcription-stop");
+ await Promise.resolve();
+ expect(h.capture.stop).toHaveBeenCalledOnce();
+ expect(h.worker.stop).not.toHaveBeenCalled();
+ finishCapture();
+ await stopped;
+ expect(h.worker.stop).toHaveBeenCalledOnce();
+ });
+
it("connects the capture window to the transcription process once that is ready", async () => {
const invoke = await loadService();
await invoke("transcription-start", "meeting");
@@ -159,6 +208,56 @@ describe("transcription IPC", () => {
expect(h.broadcast.at(-1)).toEqual(IDLE);
});
+ it("says once that a locked protected session holds back the transcript, and saves it on unlocking", async () => {
+ h.isProtected = true;
+ const invoke = await loadService();
+ await invoke("transcription-start", "meeting");
+ h.sink.write.mockImplementation(() => {
+ throw new Error("Cannot update content of blob since protected session is not available.");
+ });
+ const utterance = { source: "system" as const, offsetMs: 0, text: "Hi" };
+
+ h.listeners?.onUtterance(utterance);
+ h.listeners?.onUtterance(utterance);
+ const errors = h.broadcast.filter((state) => state.error);
+ expect(errors).toEqual([ 1, 2 ].map(() => ({
+ status: "recording", noteId: "meeting", pendingSources: [],
+ error: "transcription.save-locked"
+ })));
+ expect(await invoke("transcription-get-state")).not.toHaveProperty("error");
+
+ await invoke("transcription-stop");
+ h.coreListeners.get("ENTER_PROTECTED_SESSION")?.();
+ expect(h.sink.save).toHaveBeenCalledOnce();
+ h.coreListeners.get("ENTER_PROTECTED_SESSION")?.();
+ expect(h.sink.save).toHaveBeenCalledOnce();
+ h.sink.write.mockReset();
+ });
+
+ it("renames a speaker through the recording that writes the transcript, until it stops", async () => {
+ const invoke = await loadService();
+ const rename = (noteId: unknown, key: unknown, name: unknown) =>
+ invoke("transcription-rename-speaker", noteId, key, name);
+ expect(await rename("meeting", "system:1", "Alexandra")).toBe(false);
+
+ await invoke("transcription-start", "meeting");
+ expect(await rename("meeting", "system:1", "Alexandra")).toBe(true);
+ expect(h.sink.renameSpeaker).toHaveBeenCalledExactlyOnceWith("system:1", "Alexandra");
+ expect(await rename("other", "system:1", "Bob")).toBe(false);
+ expect(await rename("meeting", "system:1", 42)).toBe(false);
+
+ let finishStopping = () => {};
+ h.worker.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {
+ finishStopping = resolve;
+ }));
+ const stopped = invoke("transcription-stop");
+ expect(await rename("meeting", "microphone", "Elian")).toBe(true);
+ finishStopping();
+ await stopped;
+ expect(await rename("meeting", "microphone", "Elian")).toBe(false);
+ expect(h.sink.renameSpeaker).toHaveBeenCalledTimes(2);
+ });
+
it("keeps the audio and the transcript as produced only in development builds", async () => {
vi.useFakeTimers({ now: STARTED_AT, toFake: [ "Date" ] });
const invoke = await loadService();
@@ -206,6 +305,21 @@ describe("transcription IPC", () => {
expect(h.worker.stop).not.toHaveBeenCalled();
});
+ it("stops a recording whose capture was lost, keeping its queued speech, and says why", async () => {
+ const invoke = await loadService();
+ await invoke("transcription-start", "meeting");
+
+ h.captureOptions?.onLost("every audio source ended");
+ await vi.waitFor(() => expect(h.broadcast.at(-1))
+ .toEqual({ ...IDLE, error: "transcription.capture-lost" }));
+ expect(h.capture.stop).toHaveBeenCalledOnce();
+ expect(h.worker.stop).toHaveBeenCalledOnce();
+ expect(h.worker.kill).not.toHaveBeenCalled();
+ expect(await invoke("transcription-get-state")).toEqual(IDLE);
+ expect(h.log.error).toHaveBeenCalledWith(
+ "Transcription: the capture stopped: every audio source ended.");
+ });
+
it("ends a recording whose transcription process died, and says why once", async () => {
const invoke = await loadService();
await invoke("transcription-start", "meeting");
@@ -222,4 +336,41 @@ describe("transcription IPC", () => {
expect(h.worker.stop).not.toHaveBeenCalled();
expect(await invoke("transcription-start", "meeting")).toBe("");
});
+
+ it("finishes the transcription before the application quits", async () => {
+ const invoke = await loadService();
+ const quit = () => {
+ const event = { preventDefault: vi.fn() };
+ h.appListeners.get("before-quit")?.(event);
+ return event;
+ };
+ expect(quit().preventDefault).not.toHaveBeenCalled();
+
+ await invoke("transcription-start", "meeting");
+ let finishStopping = () => {};
+ h.worker.stop.mockImplementationOnce(() => new Promise<void>((resolve) => {
+ finishStopping = resolve;
+ }));
+ expect(quit().preventDefault).toHaveBeenCalledOnce();
+ await vi.waitFor(() => expect(h.worker.stop).toHaveBeenCalledOnce());
+ expect(quit().preventDefault).toHaveBeenCalledOnce();
+ expect(h.quit).not.toHaveBeenCalled();
+
+ finishStopping();
+ await vi.waitFor(() => expect(h.quit).toHaveBeenCalledOnce());
+ expect(await invoke("transcription-get-state")).toEqual(IDLE);
+ expect(quit().preventDefault).not.toHaveBeenCalled();
+ });
+
+ it("quits regardless once the transcription takes too long to finish", async () => {
+ const invoke = await loadService();
+ await invoke("transcription-start", "meeting");
+ h.worker.stop.mockImplementationOnce(() => new Promise<void>(() => {}));
+ vi.useFakeTimers();
+
+ h.appListeners.get("before-quit")?.({ preventDefault: vi.fn() });
+ await vi.advanceTimersByTimeAsync(15_000);
+ expect(h.quit).toHaveBeenCalledOnce();
+ vi.useRealTimers();
+ });
});
diff --git a/apps/desktop/src/services/transcription/index.ts b/apps/desktop/src/services/transcription/index.ts
index 85dac744be..366b3a9a5a 100644
--- a/apps/desktop/src/services/transcription/index.ts
+++ b/apps/desktop/src/services/transcription/index.ts
@@ -1,7 +1,7 @@
import type { TranscriptionState, TranscriptUtterance } from "@triliumnext/commons";
-import { becca, getLog, utils as coreUtils } from "@triliumnext/core";
+import { becca, events, getLog, protected_session, utils as coreUtils } from "@triliumnext/core";
import dataDirs from "@triliumnext/server/src/services/data_dir.js";
-import { BrowserWindow, ipcMain, MessageChannelMain } from "electron";
+import { app, BrowserWindow, ipcMain, MessageChannelMain } from "electron";
import fs from "fs";
import { t } from "i18next";
import path from "path";
@@ -18,6 +18,9 @@ export const RECORDINGS_DIR = path.join(dataDirs.TRILIUM_DATA_DIR, "transcriptio
const IDLE: TranscriptionState = { status: "idle", noteId: null, pendingSources: [] };
+/** How long quitting waits for a running transcription to finish. */
+const QUIT_TIMEOUT_MS = 15_000;
+
interface Recording {
worker: TranscriptionWorker;
capture: CaptureWindow;
@@ -25,11 +28,26 @@ interface Recording {
let state = IDLE;
let activeRecording: Recording | null = null;
+/** The transcript the running recording writes, until its stop has written the last utterance. */
+let writer: { noteId: string; sink: AttachmentSink } | null = null;
+/** The running stop, while the speech already captured is still being transcribed. */
+let stopping: Promise<void> | null = null;
+/**
+ * Transcripts whose last utterances could not be saved, kept until a save succeeds: on the next
+ * utterance, or once the protected session is entered.
+ */
+const unsavedSinks = new Set<AttachmentSink>();
+/** `waiting` while a quit waits for the transcription, `done` once it may go ahead. */
+let quitting: "no" | "waiting" | "done" = "no";
export function setupTranscriptionHandlers() {
ipcMain.handle("transcription-start", (_event, noteId: unknown) => startRecording(noteId));
ipcMain.handle("transcription-stop", () => stopRecording());
ipcMain.handle("transcription-get-state", () => getState());
+ ipcMain.handle("transcription-rename-speaker",
+ (_event, noteId: unknown, key: unknown, name: unknown) => renameSpeaker(noteId, key, name));
+ app.on("before-quit", (event) => finishBeforeQuit(event));
+ events.subscribe(events.ENTER_PROTECTED_SESSION, () => saveUnsaved());
}
/**
@@ -52,9 +70,11 @@ export async function startRecording(noteId: unknown): Promise<string> {
setState({ status: "starting", noteId, pendingSources: [] });
const sink = new AttachmentSink(noteId);
const recording: Recording = {
- capture: new CaptureWindow(),
+ capture: new CaptureWindow({
+ onLost: (reason) => handleCaptureLost(recording, reason)
+ }),
worker: new TranscriptionWorker({
- onUtterance: (utterance) => writeUtterance(sink, utterance),
+ onUtterance: (utterance) => writeUtterance(sink, noteId, utterance),
onPendingChanged: (pendingSources) => setState({ ...state, pendingSources }),
onUnexpectedExit: (code) => handleUnexpectedExit(recording, code),
log: getLog()
@@ -68,6 +88,7 @@ export async function startRecording(noteId: unknown): Promise<string> {
recordingDirectory: coreUtils.isDev() ? getRecordingDirectory(Date.now(), noteId) : null
});
sink.open(startedAt);
+ writer = { noteId, sink };
const { port1, port2 } = new MessageChannelMain();
recording.worker.connectCapture(port2);
const sources = await recording.capture.start(port1);
@@ -76,7 +97,8 @@ export async function startRecording(noteId: unknown): Promise<string> {
return "";
} catch (e) {
activeRecording = null;
- recording.capture.stop();
+ writer = null;
+ void recording.capture.stop();
recording.worker.kill();
setState(IDLE);
getLog().error(`Transcription did not start: ${(e as Error)?.stack ?? e}`);
@@ -84,39 +106,127 @@ export async function startRecording(noteId: unknown): Promise<string> {
}
}
-export async function stopRecording() {
+/**
+ * Stops the recording, resolving once its speech is transcribed; joins a stop that runs. An
+ * `error` is reported to every window with the idle state that follows.
+ */
+export function stopRecording(error?: string): Promise<void> {
const recording = activeRecording;
if (state.status !== "recording" || !recording) {
- return;
+ return stopping ?? Promise.resolve();
}
setState({ ...state, status: "stopping" });
activeRecording = null;
- recording.capture.stop();
- try {
- await recording.worker.stop();
- getLog().info(`Transcription: recording for note ${state.noteId} stopped.`);
- } finally {
- setState(IDLE);
- }
+ const { noteId } = state;
+ stopping = recording.capture.stop()
+ .then(() => recording.worker.stop())
+ .then(() => getLog().info(`Transcription: recording for note ${noteId} stopped.`))
+ .finally(() => {
+ stopping = null;
+ writer = null;
+ state = IDLE;
+ broadcast(error ? { ...IDLE, error } : IDLE);
+ });
+ return stopping;
}
export function getState() {
return state;
}
+/**
+ * Names a speaker of the transcript the recording of `noteId` writes, through the sink, which is
+ * its only writer while it runs.
+ *
+ * @returns `false` when no recording of `noteId` runs, or the arguments are not strings.
+ */
+export function renameSpeaker(noteId: unknown, key: unknown, name: unknown) {
+ const isValid = typeof key === "string" && typeof name === "string";
+ if (!writer || writer.noteId !== noteId || !isValid) {
+ return false;
+ }
+ writer.sink.renameSpeaker(key, name);
+ return true;
+}
+
/** The directory under {@link RECORDINGS_DIR} for a recording of `noteId` begun at `createdAt`. */
export function getRecordingDirectory(createdAt: number, noteId: string) {
const timestamp = new Date(createdAt).toISOString().replace(/[:.]/g, "-");
return path.join(RECORDINGS_DIR, `${timestamp}_${noteId}`);
}
-function writeUtterance(sink: AttachmentSink, utterance: TranscriptUtterance) {
+/**
+ * Holds back a quit while a transcription runs, until the speech already captured is transcribed
+ * or {@link QUIT_TIMEOUT_MS} passes, then quits again.
+ */
+function finishBeforeQuit(event: Electron.Event) {
+ if (quitting === "done") {
+ return;
+ }
+ if (quitting === "waiting") {
+ event.preventDefault();
+ return;
+ }
+ const finished = state.status === "recording" ? stopRecording() : stopping;
+ if (!finished) {
+ return;
+ }
+
+ event.preventDefault();
+ quitting = "waiting";
+ const timeout = new Promise((resolve) => setTimeout(resolve, QUIT_TIMEOUT_MS));
+ void Promise.race([ finished, timeout ]).finally(() => {
+ quitting = "done";
+ app.quit();
+ });
+}
+
+/**
+ * Saves an utterance. When that fails, the sink keeps it and every window is told once, until a
+ * later save succeeds.
+ */
+function writeUtterance(sink: AttachmentSink, noteId: string, utterance: TranscriptUtterance) {
try {
sink.write(utterance);
+ unsavedSinks.delete(sink);
} catch (e) {
getLog().error(`Transcription: saving an utterance failed: ${e}`);
+ if (unsavedSinks.has(sink)) {
+ return;
+ }
+ unsavedSinks.add(sink);
+ const isLocked = becca.getNote(noteId)?.isProtected
+ && !protected_session.isProtectedSessionAvailable();
+ broadcast({
+ ...state,
+ error: isLocked
+ ? t("transcription.save-locked")
+ : t("transcription.save-failed", { message: (e as Error)?.message ?? String(e) })
+ });
+ }
+}
+
+/** Saves the transcripts that failed to save before, as while the protected session was locked. */
+function saveUnsaved() {
+ for (const sink of unsavedSinks) {
+ try {
+ sink.save();
+ unsavedSinks.delete(sink);
+ getLog().info("Transcription: saved a transcript that could not be saved before.");
+ } catch (e) {
+ getLog().error(`Transcription: saving a transcript still failed: ${e}`);
+ }
+ }
+}
+
+/** Stops a recording whose capture stopped on its own, transcribing what it captured. */
+function handleCaptureLost(recording: Recording, reason: string) {
+ if (activeRecording !== recording || state.status !== "recording") {
+ return;
}
+ getLog().error(`Transcription: the capture stopped: ${reason}.`);
+ void stopRecording(t("transcription.capture-lost"));
}
/**
@@ -128,7 +238,8 @@ function handleUnexpectedExit(recording: Recording, code: number) {
return;
}
activeRecording = null;
- recording.capture.stop();
+ writer = null;
+ void recording.capture.stop();
getLog().error(`Transcription: the transcription process exited with code ${code}.`);
state = IDLE;
broadcast({ ...IDLE, error: t("transcription.stopped-unexpectedly") });
diff --git a/apps/desktop/src/services/transcription/protocol.ts b/apps/desktop/src/services/transcription/protocol.ts
index dfbb8e1d42..653bf9343c 100644
--- a/apps/desktop/src/services/transcription/protocol.ts
+++ b/apps/desktop/src/services/transcription/protocol.ts
@@ -38,3 +38,6 @@ export interface CaptureChunk {
source: TranscriptSource;
samples: Float32Array;
}
+
+/** What the capture page posts: chunks, then once stopping, an end marker after the last one. */
+export type CaptureMessage = CaptureChunk | { end: true };
diff --git a/apps/desktop/src/services/transcription/session.spec.ts b/apps/desktop/src/services/transcription/session.spec.ts
index 0dc605d1d8..14125d320d 100644
--- a/apps/desktop/src/services/transcription/session.spec.ts
+++ b/apps/desktop/src/services/transcription/session.spec.ts
@@ -1,9 +1,10 @@
import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";
+import { EventEmitter } from "events";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { SpeakerTracker, SpeechSegment } from "./engine";
import {
- type AudioTap, RecordingSession, type SamplesListener, type TranscriptSink
+ type AudioTap, PortCapture, RecordingSession, type SamplesListener, type TranscriptSink
} from "./session";
const log = { info: vi.fn(), error: vi.fn() };
@@ -245,3 +246,30 @@ describe("RecordingSession", () => {
expect(log.error).toHaveBeenCalledWith("Transcription: system failed: VAD model missing");
});
});
+
+describe("PortCapture", () => {
+ it("delivers the page's chunks, and settles once the page has posted its last", async () => {
+ const port = Object.assign(new EventEmitter(), { start: vi.fn(), close: vi.fn() });
+ const onSamples = vi.fn();
+ const capture = new PortCapture();
+ capture.start(onSamples);
+ capture.connect(port as never);
+ let hasEnded = false;
+ void capture.ended.then(() => {
+ hasEnded = true;
+ });
+
+ const samples = new Float32Array(512);
+ port.emit("message", { data: { source: "system", samples } });
+ port.emit("message", { data: { source: "speakers", samples } });
+ port.emit("message", { data: null });
+ expect(onSamples).toHaveBeenCalledExactlyOnceWith("system", samples);
+ await Promise.resolve();
+ expect(hasEnded).toBe(false);
+
+ port.emit("message", { data: { end: true } });
+ await capture.ended;
+ capture.stop();
+ expect(port.close).toHaveBeenCalledOnce();
+ });
+});
diff --git a/apps/desktop/src/services/transcription/session.ts b/apps/desktop/src/services/transcription/session.ts
index 2019e0a87d..ac436ae5a8 100644
--- a/apps/desktop/src/services/transcription/session.ts
+++ b/apps/desktop/src/services/transcription/session.ts
@@ -1,7 +1,8 @@
import type { TranscriptSource, TranscriptUtterance } from "@triliumnext/commons";
+import type { MessagePortMain } from "electron";
import type { SpeakerTracker, SpeechEngine, SpeechSegment, SpeechSegmenter } from "./engine";
-import { type Logger, SAMPLE_RATE } from "./protocol";
+import { type CaptureMessage, type Logger, SAMPLE_RATE } from "./protocol";
/** The sources whose speakers are told apart; the microphone carries only the local speaker. */
const DIARIZED_SOURCES: readonly TranscriptSource[] = [ "system" ];
@@ -254,3 +255,52 @@ class SourcePipeline {
}
}
}
+
+/** The samples the capture page posts to the port the main process connects. */
+export class PortCapture implements AudioCapture {
+ /**
+ * Settles once the page has posted its end marker or the port closed: every sample the page
+ * captured has then been delivered.
+ */
+ readonly ended: Promise<void>;
+ private port: MessagePortMain | null = null;
+ private onSamples: SamplesListener | null = null;
+ private markEnded = () => {};
+
+ constructor() {
+ this.ended = new Promise((resolve) => {
+ this.markEnded = resolve;
+ });
+ }
+
+ start(onSamples: SamplesListener) {
+ this.onSamples = onSamples;
+ }
+
+ connect(port: MessagePortMain) {
+ this.port = port;
+ port.on("message", ({ data }) => {
+ const message = data as CaptureMessage | null;
+ if (!message) {
+ return;
+ }
+ if ("end" in message) {
+ this.markEnded();
+ } else if (isSource(message.source) && message.samples instanceof Float32Array) {
+ this.onSamples?.(message.source, message.samples);
+ }
+ });
+ port.on("close", () => this.markEnded());
+ port.start();
+ }
+
+ stop() {
+ this.onSamples = null;
+ this.port?.close();
+ this.port = null;
+ }
+}
+
+function isSource(source: unknown): source is TranscriptSource {
+ return source === "microphone" || source === "system";
+}
diff --git a/apps/desktop/src/services/transcription/transcription_worker.ts b/apps/desktop/src/services/transcription/transcription_worker.ts
index abe6bfbea5..a646115632 100644
--- a/apps/desktop/src/services/transcription/transcription_worker.ts
+++ b/apps/desktop/src/services/transcription/transcription_worker.ts
@@ -4,15 +4,11 @@
* posts its samples here directly; the main process only receives the utterances. Started by
* `TranscriptionWorker` in `worker_client.ts`, which also ends it.
*/
-import type { TranscriptSource } from "@triliumnext/commons";
-import type { MessagePortMain } from "electron";
import { TranscriptFileSink, WavRecorder } from "./audio_recorder";
import { loadSherpaEngine } from "./engine";
-import type { CaptureChunk, Logger, WorkerEvent, WorkerRequest } from "./protocol";
-import {
- type AudioCapture, type AudioTap, RecordingSession, type SamplesListener, type TranscriptSink
-} from "./session";
+import type { Logger, WorkerEvent, WorkerRequest } from "./protocol";
+import { type AudioTap, PortCapture, RecordingSession, type TranscriptSink } from "./session";
const log: Logger = {
info: (message) => post({ type: "log", level: "info", message }),
@@ -21,6 +17,9 @@ const log: Logger = {
type StartRequest = Extract<WorkerRequest, { type: "start" }>;
+/** How long stopping waits for the capture page's end marker. */
+const CAPTURE_END_TIMEOUT_MS = 2000;
+
let session: RecordingSession | null = null;
let capture: PortCapture | null = null;
@@ -68,7 +67,15 @@ async function start({ noteId, modelDirectory, recordingDirectory }: StartReques
post({ type: "ready", startedAt: newSession.startedAt, settings: engine.settings });
}
+/**
+ * Waits for the capture page's last chunks, which travel on another port than the stop request,
+ * then transcribes what the session still holds.
+ */
async function stop() {
+ if (capture) {
+ const timeout = new Promise((resolve) => setTimeout(resolve, CAPTURE_END_TIMEOUT_MS));
+ await Promise.race([ capture.ended, timeout ]);
+ }
try {
await session?.stop();
} catch (e) {
@@ -80,34 +87,3 @@ async function stop() {
function post(event: WorkerEvent) {
process.parentPort.postMessage(event);
}
-
-/** The samples the capture page posts to the port the main process connects. */
-class PortCapture implements AudioCapture {
- private port: MessagePortMain | null = null;
- private onSamples: SamplesListener | null = null;
-
- start(onSamples: SamplesListener) {
- this.onSamples = onSamples;
- }
-
- connect(port: MessagePortMain) {
- this.port = port;
- port.on("message", ({ data }) => {
- const chunk = data as CaptureChunk | null;
- if (chunk && isSource(chunk.source) && chunk.samples instanceof Float32Array) {
- this.onSamples?.(chunk.source, chunk.samples);
- }
- });
- port.start();
- }
-
- stop() {
- this.onSamples = null;
- this.port?.close();
- this.port = null;
- }
-}
-
-function isSource(source: unknown): source is TranscriptSource {
- return source === "microphone" || source === "system";
-}
diff --git a/apps/server/src/assets/translations/en/server.json b/apps/server/src/assets/translations/en/server.json
index 0bfb215580..c880021410 100644
--- a/apps/server/src/assets/translations/en/server.json
+++ b/apps/server/src/assets/translations/en/server.json
@@ -568,6 +568,9 @@
"model-missing": "No speech recognition model was found in {{- directory}}.",
"start-failed": "The transcription could not start: {{- message}}",
"stopped-unexpectedly": "The transcription stopped unexpectedly. What was transcribed until then is kept.",
+ "save-locked": "The transcript cannot be saved while the protected session is locked. The transcription keeps it and saves it once you enter the protected session, as long as Trilium stays open.",
+ "save-failed": "The transcript could not be saved: {{- message}} The transcription keeps it and tries again with the next sentence.",
+ "capture-lost": "The transcription stopped because the audio could no longer be captured. What was transcribed until then is kept.",
"attachment-title": "Transcript"
}
}
diff --git a/packages/commons/src/lib/electron_api_interface.ts b/packages/commons/src/lib/electron_api_interface.ts
index 7b762fa23b..3d89d189ee 100644
--- a/packages/commons/src/lib/electron_api_interface.ts
+++ b/packages/commons/src/lib/electron_api_interface.ts
@@ -725,8 +725,8 @@ export interface TranscriptionState {
/** The sources heard speaking, or whose speech is still being transcribed. */
pendingSources: TranscriptSource[];
/**
- * Why the recording ended without being stopped, sent once with the `idle` state that
- * follows; `getState()` never returns it.
+ * A problem to tell the user about, sent once with the state it concerns: the recording ended
+ * without being stopped, or its transcript cannot be saved. `getState()` never returns it.
*/
error?: string;
}
@@ -748,6 +748,15 @@ export interface ElectronTranscriptionApi {
getState(): Promise<TranscriptionState>;
+ /**
+ * Names a speaker of the transcript the recording of `noteId` writes, or restores the default
+ * name when `name` is blank. The recording applies it, so no utterance it writes meanwhile is
+ * lost.
+ *
+ * @returns `false` when no recording of `noteId` runs; the transcript can then be saved whole.
+ */
+ renameSpeaker(noteId: string, key: string, name: string): Promise<boolean>;
+
/**
* Calls `callback` every time the state changes, in any window.
*