Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(transcription): show the level meters in the transcript's header
Elian Doran36473567fbb7
diff --git a/.claude/skills/working-with-transcription/SKILL.md b/.claude/skills/working-with-transcription/SKILL.mdindex 87c9c4af0a..e491b54bc5 100644--- a/.claude/skills/working-with-transcription/SKILL.md+++ b/.claude/skills/working-with-transcription/SKILL.md@@ -31,7 +31,7 @@ main process: TranscriptionWorker → AttachmentSink (transcript attachment)The main process opens the capture window, starts the transcription process and connects the two with a `MessageChannelMain`; per chunk it does nothing, per utterance it writes the attachment. Each recording gets a process of its own, which loads the model (~1.3 s) and exits after `stop`.- **Stopping** runs in order: `CaptureWindow.stop()` calls the page's `finishCapture()`, which flushes each worklet's partial chunk (padded with silence to whole 512-sample VAD windows) and posts `{ end: true }`; then the process, told to stop, waits for that marker (it travels on another port than the stop request, up to 2 s) before flushing the VADs and the decode queue.-- **A stalled source**: the main process marks a source in `stalledSources` once it has delivered no chunk for 10 s (`NO_AUDIO_MS`, checked every second) and clears it on the next chunk; a muted source still delivers silence. `TranscriptView` marks that source's level meter (fed by `onLevel()`, shown above the turns while its meeting records) _No audio_; the launcher turns its time badge red and shows an error once.+- **A stalled source**: the main process marks a source in `stalledSources` once it has delivered no chunk for 10 s (`NO_AUDIO_MS`, checked every second) and clears it on the next chunk; a muted source still delivers silence. `TranscriptRecordingLevels` (in `TranscriptView.tsx`, fed by `onLevel()`, shown while its meeting records in the header of the pinned section through `Collapsible`'s `headerActions`) marks that source _No audio_; the launcher turns its time badge red and shows an error once.- **Quitting** during a recording is held back (`before-quit`) until that stop finishes, at most 15 s.- **Failures** reach every window as a state with an `error`, which the launcher shows as a toast; `getState()` never returns it. The process dying ends the recording, keeping what was written; the capture page crashing or every source ending (`render-process-gone`, a track's `ended`) stops it normally, transcribing what was captured; a transcript that cannot be saved (a protected note while the protected session is locked) is reported once and kept in memory, and `AttachmentSink.save()` writes it on the next utterance or on `ENTER_PROTECTED_SESSION`.@@ -52,7 +52,7 @@ Nothing the transcription process imports may reach `@triliumnext/core` or `@tri| `packages/commons/src/lib/transcript.ts` | The `Transcript` format (`version: 1`, `startedAt`, `speakers`, `utterances`), `parseTranscript()`, speaker keys and `setSpeakerName()`; `cleanUtteranceText()`, the filler and stutter cleanup || Cleanup | The stored transcript stays verbatim. `TranscriptView` shows each utterance through `cleanUtteranceText()` while the synced option `transcriptCleanupEnabled` (default on, the "Hide filler words" switch above the transcript) holds, and leaves out utterances it empties. The rules are English-only for stutters (a function-word list) and keep numbers, comma-separated repeats and "that that"/"it it" || Speaker names | While a recording writes the transcript, `TranscriptView` renames through `electronApi.transcription.renameSpeaker(attachmentId, …)` (by attachment, since a note can hold an older transcript beside the running one): `AttachmentSink`, the only writer then, applies `setSpeakerName()` to the stored names in one transaction, so no utterance is overwritten. Otherwise it saves the stored transcript whole through the generic `POST notes/:noteId/attachments?matchBy=attachmentId`; attachments get no route of their own. `AttachmentSink` re-reads `speakers` before each rewrite, so a rename from another window or device survives the next utterance |-| `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in, the level meters |+| `apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx` | Turns, clock times, the "Speaking…" line, fade-in, the level meters (`TranscriptRecordingLevels`) || `apps/client/src/widgets/launch_bar/RecordMeetingButton.tsx` | The launcher and `useTranscriptionState()` |## The modeldiff --git a/apps/client/src/widgets/layout/PinnedAttachments.spec.tsx b/apps/client/src/widgets/layout/PinnedAttachments.spec.tsxindex 7cec96bd74..6de9638447 100644--- a/apps/client/src/widgets/layout/PinnedAttachments.spec.tsx+++ b/apps/client/src/widgets/layout/PinnedAttachments.spec.tsx@@ -8,7 +8,9 @@ import PinnedAttachments from "./PinnedAttachments";vi.mock("../type_widgets/transcript/TranscriptView", () => ({default: ({ attachment }: { attachment: FAttachment }) =>- <div className="transcript-stub">{attachment.attachmentId}</div>+ <div className="transcript-stub">{attachment.attachmentId}</div>,+ TranscriptRecordingLevels: ({ attachment }: { attachment: FAttachment }) =>+ <span className="levels-stub">{attachment.attachmentId}</span>}));function attachment(attachmentId: string, role: string, title: string) {@@ -43,6 +45,20 @@ describe("PinnedAttachments", () => {expect(section.querySelector(".transcript-stub")?.textContent).toBe("transcript1");});+ it("shows a transcript's level meters in its header, also while collapsed", async () => {+ const container = await renderWith([+ attachment("transcript1", "transcript", "Transcript")+ ]);+ const levels = () => container.querySelector(".collapsible-header .levels-stub");+ const title = container.querySelector<HTMLButtonElement>(".collapsible-title");++ expect(levels()?.textContent).toBe("transcript1");+ expect(container.querySelector(".collapsible-body .levels-stub")).toBeNull();+ act(() => title?.click());+ expect(title?.getAttribute("aria-expanded")).toBe("false");+ expect(levels()).not.toBeNull();+ });+it("renders nothing for a note without one", async () => {const container = await renderWith([ attachment("image1", "image", "photo.png") ]);expect(container.querySelector(".pinned-attachment")).toBeNull();diff --git a/apps/client/src/widgets/layout/PinnedAttachments.tsx b/apps/client/src/widgets/layout/PinnedAttachments.tsxindex 5d6af7d341..b6cdd65d2f 100644--- a/apps/client/src/widgets/layout/PinnedAttachments.tsx+++ b/apps/client/src/widgets/layout/PinnedAttachments.tsx@@ -6,11 +6,23 @@ import type FAttachment from "../../entities/fattachment";import type FNote from "../../entities/fnote";import Collapsible from "../react/Collapsible";import { useAttachments } from "../react/hooks";-import TranscriptView from "../type_widgets/transcript/TranscriptView";+import TranscriptView, {+ TranscriptRecordingLevels+} from "../type_widgets/transcript/TranscriptView";-/** The attachment roles shown at the top of their note, each with the view that renders it. */-const PINNED_ROLE_VIEWS: Record<string, ComponentType<{ attachment: FAttachment }>> = {- transcript: TranscriptView+type AttachmentComponent = ComponentType<{ attachment: FAttachment }>;++interface PinnedRole {+ View: AttachmentComponent;+ HeaderActions?: AttachmentComponent;+}++/**+ * The attachment roles shown at the top of their note, each with the view that renders it and+ * what its section's header shows beside the title.+ */+const PINNED_ROLES: Record<string, PinnedRole> = {+ transcript: { View: TranscriptView, HeaderActions: TranscriptRecordingLevels }};/** Shows the note's attachments of a pinned role above its content, one section each. */@@ -20,14 +32,16 @@ export default function PinnedAttachments({ note }: { note: FNote }) {return (<>{attachments.map((attachment) => {- const View = Object.hasOwn(PINNED_ROLE_VIEWS, attachment.role)- ? PINNED_ROLE_VIEWS[attachment.role]- : null;- return View && (+ if (!Object.hasOwn(PINNED_ROLES, attachment.role)) {+ return null;+ }+ const { View, HeaderActions } = PINNED_ROLES[attachment.role];+ return (<Collapsiblekey={attachment.attachmentId}className="pinned-attachment"title={attachment.title}+ headerActions={HeaderActions && <HeaderActions attachment={attachment} />}initiallyExpanded><View attachment={attachment} />diff --git a/apps/client/src/widgets/react/Collapsible.css b/apps/client/src/widgets/react/Collapsible.cssindex 00aeb2bf7a..ade23d1ab8 100644--- a/apps/client/src/widgets/react/Collapsible.css+++ b/apps/client/src/widgets/react/Collapsible.css@@ -1,4 +1,11 @@.collapsible {+ .collapsible-header {+ display: flex;+ flex-wrap: wrap;+ align-items: center;+ gap: 0.25rem 1rem;+ }+.collapsible-title {line-height: 1em;display: flex;diff --git a/apps/client/src/widgets/react/Collapsible.tsx b/apps/client/src/widgets/react/Collapsible.tsxindex cfd8ccf9b4..90950d8475 100644--- a/apps/client/src/widgets/react/Collapsible.tsx+++ b/apps/client/src/widgets/react/Collapsible.tsx@@ -11,6 +11,8 @@ interface CollapsibleProps extends Pick<HTMLAttributes<HTMLDivElement>, "classNatitle: string;children: ComponentChildren;initiallyExpanded?: boolean;+ /** Shown beside the title, also while collapsed; not part of the button that toggles. */+ headerActions?: ComponentChildren;}export default function Collapsible({ initiallyExpanded, ...restProps }: CollapsibleProps) {@@ -18,7 +20,9 @@ export default function Collapsible({ initiallyExpanded, ...restProps }: Collapsreturn <ExternallyControlledCollapsible {...restProps} expanded={expanded} setExpanded={setExpanded} />;}-export function ExternallyControlledCollapsible({ title, children, className, expanded, setExpanded }: Omit<CollapsibleProps, "initiallyExpanded"> & {+export function ExternallyControlledCollapsible({+ title, children, className, expanded, setExpanded, headerActions+}: Omit<CollapsibleProps, "initiallyExpanded"> & {expanded: boolean | undefined;setExpanded: (expanded: boolean) => void}) {@@ -51,23 +55,32 @@ export function ExternallyControlledCollapsible({ title, children, className, ex}}, [expanded, transitionEnabled]);+ const titleButton = (+ <button+ className="collapsible-title tn-low-profile"+ onClick={(e) => {+ e.preventDefault();+ setExpanded(!expanded);+ }}+ aria-expanded={expanded}+ aria-controls={contentId}+ >+ <Icon className="arrow" icon="bx bx-chevron-right" /> + {title}+ </button>+ );+return (<div className={clsx("collapsible", className, {expanded,"with-transition": transitionEnabled})}>- <button- className="collapsible-title tn-low-profile"- onClick={(e) => {- e.preventDefault();- setExpanded(!expanded);- }}- aria-expanded={expanded}- aria-controls={contentId}- >- <Icon className="arrow" icon="bx bx-chevron-right" /> - {title}- </button>+ {headerActions ? (+ <div className="collapsible-header">+ {titleButton}+ <div className="collapsible-header-actions">{headerActions}</div>+ </div>+ ) : titleButton}<divid={contentId}diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.css b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.cssindex 99411610be..4418f4f7f8 100644--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.css+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.css@@ -14,52 +14,6 @@font-size: 0.9em;}- .transcript-levels {- display: flex;- flex-wrap: wrap;- align-items: center;- gap: 0.25rem 1rem;- margin-inline-end: auto;- user-select: none;- }-- .transcript-level {- display: flex;- align-items: center;- gap: 0.35rem;-- .transcript-level-bar {- position: relative;- width: 4rem;- height: 0.35rem;- border-radius: 0.2rem;- background: color-mix(in srgb, var(--main-text-color) 15%, transparent);- overflow: hidden;-- &::after {- content: "";- position: absolute;- inset-block: 0;- inset-inline-start: 0;- width: calc(var(--level, 0) * 100%);- background: var(--main-text-color);- transition: width 120ms linear;- }- }-- &.stalled {- color: var(--danger-color, #dc3545);-- .transcript-level-bar {- background: var(--danger-color, #dc3545);-- &::after {- display: none;- }- }- }- }-.transcript-turn {max-width: var(--preferred-max-content-width, 900px);margin-bottom: 1rem;@@ -153,13 +107,60 @@}}+.transcript-levels {+ display: flex;+ flex-wrap: wrap;+ align-items: center;+ gap: 0.25rem 1rem;+ color: var(--muted-text-color);+ font-size: 0.9em;+ user-select: none;+}++.transcript-level {+ display: flex;+ align-items: center;+ gap: 0.35rem;++ .transcript-level-bar {+ position: relative;+ width: 4rem;+ height: 0.35rem;+ border-radius: 0.2rem;+ background: color-mix(in srgb, var(--main-text-color) 15%, transparent);+ overflow: hidden;++ &::after {+ content: "";+ position: absolute;+ inset-block: 0;+ inset-inline-start: 0;+ width: calc(var(--level, 0) * 100%);+ background: var(--main-text-color);+ transition: width 120ms linear;+ }+ }++ &.stalled {+ color: var(--danger-color, #dc3545);++ .transcript-level-bar {+ background: var(--danger-color, #dc3545);++ &::after {+ display: none;+ }+ }+ }+}+@media (prefers-reduced-motion: reduce) {.transcript .transcript-speaking .bx,.transcript .transcript-new {animation: none;}- .transcript .transcript-level .transcript-level-bar::after {+ .transcript-level .transcript-level-bar::after {transition: none;}}diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsxindex 504a38b83f..ae5ed2d74c 100644--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.spec.tsx@@ -12,7 +12,7 @@ import options from "../../../services/options";import { renderInto } from "../../../test/render";import { ParentComponent } from "../../react/react_utils";import TranscriptView, {- formatOffset, groupIntoTurns, levelToMeter, type TranscriptTurn+ formatOffset, groupIntoTurns, levelToMeter, TranscriptRecordingLevels, type TranscriptTurn} from "./TranscriptView";const h = vi.hoisted(() => ({ post: vi.fn(), put: vi.fn() }));@@ -475,7 +475,7 @@ describe("TranscriptView", () => {window.electronApi = undefined;});- it("shows each source's level while its meeting records, and which is silent", async () => {+ it("shows each source's level while its meeting records, apart from the turns", async () => {let onStateChanged: (state: TranscriptionState) => void = () => {};let onLevel: (source: TranscriptSource, level: number) => void = () => {};const unsubscribeLevels = vi.fn();@@ -501,21 +501,24 @@ describe("TranscriptView", () => {attachmentId: "transcript1", ownerId: "meeting",getBlob: async () => ({ content: transcript([]) })};- const container = renderInto(+ const view = renderInto(<ParentComponent.Provider value={new Component()}><TranscriptView attachment={attachment as unknown as FAttachment} /></ParentComponent.Provider>);+ const container = renderInto(+ <TranscriptRecordingLevels attachment={attachment as unknown as FAttachment} />+ );const meter = (source: string) => container.querySelector<HTMLElement>(`.transcript-level[data-source="${source}"]`);const fill = (source: string) =>Number(meter(source)?.style.getPropertyValue("--level"));await vi.waitFor(() => expect(meter("microphone")).not.toBeNull());- expect(container.textContent).toContain("transcript.empty");+ await vi.waitFor(() => expect(view.textContent).toContain("transcript.empty"));+ expect(view.querySelector(".transcript-level, .transcript-options")).toBeNull();expect(meter("microphone")?.textContent).toContain("transcript.speaker_me");expect(meter("system")?.textContent).toContain("transcript.speaker_others");- expect(container.querySelector(".transcript-options input")).toBeNull();act(() => onLevel("microphone", 10 ** (-35 / 20)));expect(fill("microphone")).toBeCloseTo(0.5);expect(fill("system")).toBe(0);diff --git a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsxindex 1eeea630ae..5712a3fb02 100644--- a/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx+++ b/apps/client/src/widgets/type_widgets/transcript/TranscriptView.tsx@@ -32,17 +32,14 @@ const PARAGRAPH_PAUSE_MS = 2000;* Shows a `transcript` attachment as the conversation it records, one paragraph per speaker turn.* Follows the attachment as a running recording rewrites it, shows which speakers the recording* hears before their words are transcribed, and keeps a reader who is at the end there as new- * turns arrive. While its meeting records, shows each source's level above the turns, and which- * source delivers no audio. With `transcriptCleanupEnabled`, shows each utterance through+ * turns arrive. With `transcriptCleanupEnabled`, shows each utterance through* `cleanUtteranceText()` and leaves out the ones it empties; the stored transcript stays verbatim.*/export default function TranscriptView({ attachment }: { attachment: FAttachment }) {const [ content, setContent ] = useState<string | null>();const [ isCleanedUp, setCleanedUp ] = useTriliumOptionBool("transcriptCleanupEnabled");const recording = useTranscriptionState();- const isOwnRecording = recording.noteId === attachment.ownerId;- const pending = isOwnRecording ? recording.pendingSources : [];- const isRecording = isOwnRecording && recording.status === "recording";+ const pending = recording.noteId === attachment.ownerId ? recording.pendingSources : [];const scrollerRef = useRef<HTMLDivElement>(null);/** The scroll height at the previous render, or `null` before the transcript first shows. */const scrollHeightRef = useRef<number | null>(null);@@ -90,25 +87,8 @@ export default function TranscriptView({ attachment }: { attachment: FAttachmentif (!transcript) {return <NoItems icon="bx bx-error" text={t("transcript.unreadable")} />;}- const hasUtterances = transcript.utterances.length > 0;- const options = (hasUtterances || isRecording) && (- <div className="transcript-options">- {isRecording && <TranscriptLevels stalledSources={recording.stalledSources ?? []} />}- {hasUtterances && (- <FormToggle- switchOnName={t("transcript.cleanup")}- switchOffName={t("transcript.cleanup")}- switchOnTooltip={t("transcript.cleanup_tooltip")}- switchOffTooltip={t("transcript.cleanup_tooltip")}- currentValue={isCleanedUp}- onChange={setCleanedUp}- />- )}- </div>- );- if (!hasUtterances && !pending.length) {- const empty = <NoItems icon="bx bx-microphone" text={t("transcript.empty")} />;- return options ? <div className="transcript">{options}{empty}</div> : empty;+ if (!transcript.utterances.length && !pending.length) {+ return <NoItems icon="bx bx-microphone" text={t("transcript.empty")} />;}initialCountRef.current ??= transcript.utterances.length;@@ -131,7 +111,18 @@ export default function TranscriptView({ attachment }: { attachment: FAttachment);return (<div ref={scrollerRef} className="transcript">- {options}+ {transcript.utterances.length > 0 && (+ <div className="transcript-options">+ <FormToggle+ switchOnName={t("transcript.cleanup")}+ switchOffName={t("transcript.cleanup")}+ switchOnTooltip={t("transcript.cleanup_tooltip")}+ switchOffTooltip={t("transcript.cleanup_tooltip")}+ currentValue={isCleanedUp}+ onChange={setCleanedUp}+ />+ </div>+ )}{turns.map((turn) => (<div key={turn.offsetMs} className={`transcript-turn transcript-${turn.source}`}><div className="transcript-turn-header">@@ -182,9 +173,18 @@ const LEVEL_ICONS: Record<TranscriptSource, string> = {const LEVEL_SOURCES = Object.keys(LEVEL_ICONS) as TranscriptSource[];/**- * One meter per source, named as the transcript names it, filled to its latest level, and saying- * so when the source delivers no audio.+ * While the meeting of `attachment` records, one meter per source, named as the transcript names+ * it, filled to its latest level, and saying so when the source delivers no audio. Shown in the+ * header of the transcript's section, so it stays in view while the turns scroll or are collapsed.*/+export function TranscriptRecordingLevels({ attachment }: { attachment: FAttachment }) {+ const { status, noteId, stalledSources } = useTranscriptionState();+ if (status !== "recording" || noteId !== attachment.ownerId) {+ return null;+ }+ return <TranscriptLevels stalledSources={stalledSources ?? []} />;+}+function TranscriptLevels({ stalledSources }: { stalledSources: TranscriptSource[] }) {const levels = useTranscriptionLevels();return (diff --git a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Meeting transcription.html b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Meeting transcription.htmlindex 3cb8fad848..cb5b21e379 100644--- a/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Meeting transcription.html+++ b/apps/server/src/assets/doc_notes/en/User Guide/User Guide/Meeting transcription.html@@ -59,12 +59,13 @@the meeting note and is shown in a collapsible <em>Transcript</em> sectionabove the note's content. It updates as the meeting goes on, one paragraphper speaker turn, each with the time it started. While the meeting is being- transcribed, a meter above the transcript shows the level of your microphone- (<em>Me</em>) and of the meeting's audio (<em>Others</em>), so you can- tell that both are being heard; a source that delivers no audio is marked <em>No audio</em>.- While someone is talking, <em>Speaking…</em> shows under their name until- their words are transcribed. If you are reading the end of the transcript,- it scrolls to follow new text.</p>+ transcribed, the header of the <em>Transcript</em> section shows the level+ of your microphone (<em>Me</em>) and of the meeting's audio (<em>Others</em>),+ so you can tell that both are being heard, even while the section is collapsed;+ a source that delivers no audio is marked <em>No audio</em>. While someone+ is talking, <em>Speaking…</em> shows under their name until their words are+ transcribed. If you are reading the end of the transcript, it scrolls to+ follow new text.</p><ul><li>What you say into your microphone is labeled <em>Me</em>.</li><li>What the other participants say is labeled <em>Speaker 1</em>, <em>Speaker 2</em> anddiff --git a/docs/User Guide/User Guide/Meeting transcription.md b/docs/User Guide/User Guide/Meeting transcription.mdindex f1263014cf..191b90a752 100644--- a/docs/User Guide/User Guide/Meeting transcription.md+++ b/docs/User Guide/User Guide/Meeting transcription.md@@ -31,7 +31,7 @@ If the transcription stops on its own, for example because the audio could no lo## The transcript-The transcript is stored as an [attachment](Basic%20Concepts%20and%20Features/Notes/Attachments.md) of the meeting note and is shown in a collapsible _Transcript_ section above the note's content. It updates as the meeting goes on, one paragraph per speaker turn, each with the time it started. While the meeting is being transcribed, a meter above the transcript shows the level of your microphone (_Me_) and of the meeting's audio (_Others_), so you can tell that both are being heard; a source that delivers no audio is marked _No audio_. While someone is talking, _Speaking…_ shows under their name until their words are transcribed. If you are reading the end of the transcript, it scrolls to follow new text.+The transcript is stored as an [attachment](Basic%20Concepts%20and%20Features/Notes/Attachments.md) of the meeting note and is shown in a collapsible _Transcript_ section above the note's content. It updates as the meeting goes on, one paragraph per speaker turn, each with the time it started. While the meeting is being transcribed, the header of the _Transcript_ section shows the level of your microphone (_Me_) and of the meeting's audio (_Others_), so you can tell that both are being heard, even while the section is collapsed; a source that delivers no audio is marked _No audio_. While someone is talking, _Speaking…_ shows under their name until their words are transcribed. If you are reading the end of the transcript, it scrolls to follow new text.* What you say into your microphone is labeled _Me_.* What the other participants say is labeled _Speaker 1_, _Speaker 2_ and so on, while _Identify speakers_ is on; otherwise it is labeled _Others_.