Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
fix(stt): drop the renderer audio-decode fallback and say ffmpeg is missing
EtienneLescot485eee374538
diff --git a/electron/stt/extractAudio.test.ts b/electron/stt/extractAudio.test.tsindex d87217668..4bfe6a8b3 100644--- a/electron/stt/extractAudio.test.ts+++ b/electron/stt/extractAudio.test.ts@@ -14,9 +14,7 @@ vi.mock("../media/audioPeaks", () => ({const { extractMono16kPcm, FfmpegUnavailableError, MediaUnreadableError, NoAudioTrackError } =await import("./extractAudio");-const { STT_MEDIA_UNREADABLE, STT_NATIVE_EXTRACTION_UNAVAILABLE } = await import(- "./transcriptionContract"-);+const { STT_MEDIA_UNREADABLE } = await import("./transcriptionContract");/** A stand-in for the ffmpeg child: two pipes and a close event, nothing more. */function fakeChild() {@@ -179,13 +177,12 @@ describe("extractMono16kPcm", () => {expect(Array.from(await promise)).toEqual([expect.closeTo(0.1, 6), expect.closeTo(0.2, 6)]);});- it("refuses with the marker the renderer falls back on when ffmpeg is missing", async () => {- // The string is the contract across the IPC boundary, which drops the class.+ it("says ffmpeg is missing when none resolves", async () => {+ // The message is all that crosses the IPC boundary, and nothing falls back on it:+ // it is what the editor, the CLI and MCP show.resolveFfmpegMock.mockReturnValue(null);await expect(extractMono16kPcm("/tmp/a.mp3")).rejects.toBeInstanceOf(FfmpegUnavailableError);- await expect(extractMono16kPcm("/tmp/a.mp3")).rejects.toThrow(- STT_NATIVE_EXTRACTION_UNAVAILABLE,- );+ await expect(extractMono16kPcm("/tmp/a.mp3")).rejects.toThrow(/^ffmpeg is missing/);expect(spawnMock).not.toHaveBeenCalled();});diff --git a/electron/stt/extractAudio.ts b/electron/stt/extractAudio.tsindex 2ffebf559..f4308c2e9 100644--- a/electron/stt/extractAudio.ts+++ b/electron/stt/extractAudio.ts@@ -21,7 +21,7 @@import { spawn } from "node:child_process";import { mediaClockAudioArgs, resolveFfmpeg } from "../media/audioPeaks";-import { STT_MEDIA_UNREADABLE, STT_NATIVE_EXTRACTION_UNAVAILABLE } from "./transcriptionContract";+import { STT_MEDIA_UNREADABLE } from "./transcriptionContract";/** What whisper.cpp wants, and what `decodePeaks` already asks ffmpeg for. */const SAMPLE_RATE = 16_000;@@ -30,14 +30,17 @@ const SAMPLE_RATE = 16_000;const EXTRACT_TIMEOUT_MS = 60_000;/**- * Thrown when no ffmpeg can be resolved, so the caller can fall back to the- * renderer pipeline rather than failing the transcription outright. A distinct type- * because "there is no ffmpeg here" and "this file has no audio" want opposite- * responses: fall back, versus report a permanent failure for this asset.+ * Thrown when no ffmpeg can be resolved. There is no fallback behind it: the renderer+ * decode that used to catch this reads the whole file on the UI thread, and the hidden+ * CLI runner could not read the file at all. Every package ships ffmpeg, so the message+ * names what is missing and how to put it back, for the editor, the CLI and MCP alike.*/export class FfmpegUnavailableError extends Error {constructor() {- super(`${STT_NATIVE_EXTRACTION_UNAVAILABLE}: no ffmpeg binary for native audio extraction`);+ super(+ "ffmpeg is missing: transcription needs it to read the audio, and this install has none. " ++ "Reinstall OpenScreen, or set OPENSCREEN_FFMPEG_PATH to an ffmpeg executable.",+ );this.name = "FfmpegUnavailableError";}}diff --git a/electron/stt/index.ts b/electron/stt/index.tsindex 7a536409b..1272b99fa 100644--- a/electron/stt/index.ts+++ b/electron/stt/index.ts@@ -53,7 +53,7 @@ import { WhisperServerManager } from "./whisperServer";* worth revisiting only if a much smaller model ever becomes the default.*/-/** The renderer always sends mono 16 kHz (see `extractMono16kFromVideoUrl`). */+/** What `extractMono16kPcm` decodes to (see `extractAudio.ts`). */const SAMPLE_RATE = 16_000;/** Attempts per chunk before the whole transcription fails. */diff --git a/electron/stt/transcriptionContract.ts b/electron/stt/transcriptionContract.tsindex d30181487..ce82aff5d 100644--- a/electron/stt/transcriptionContract.ts+++ b/electron/stt/transcriptionContract.ts@@ -107,10 +107,9 @@ export interface SttStatusEvent {/** IPC request: renderer → main. */export interface SttTranscribeRequest {/**- * Mono-16k samples the CALLER decoded. Optional since native extraction landed:- * pass `sourcePath` instead and the main process decodes with ffmpeg, off the UI- * thread and without the renderer ever holding the audio. Kept for the caption- * path, which already has samples in hand and has no file to point at.+ * Mono-16k samples the CALLER decoded. No renderer sends these any more: it passes+ * `sourcePath` and the main process decodes with ffmpeg, off the UI thread and+ * without the renderer ever holding the audio. Kept as the manager's own seam.** Exactly one of `samples` / `sourcePath` is required.*/@@ -120,8 +119,7 @@ export interface SttTranscribeRequest {* Preferred: the renderer's own pipeline read the whole file, copied it twice and* resampled it on the UI thread, which is what froze the editor at open.*- * The caller falls back to its own decode when this cannot be honoured — see- * `FfmpegUnavailableError`.+ * Fails with `FfmpegUnavailableError` when no ffmpeg can be resolved.*/sourcePath?: string;/**@@ -136,17 +134,6 @@ export interface SttTranscribeRequest {prompt?: string;}-/**- * Marker carried in the error message when the main process cannot decode a- * `sourcePath` because no ffmpeg is resolvable on this install.- *- * A string rather than an error class because this crosses `ipcRenderer.invoke`,- * which reconstructs a plain `Error` from the message and drops the prototype and- * the `name`. Exported so neither side spells it out by hand — a fallback keyed on- * a literal typed twice is a fallback that silently stops working.- */-export const STT_NATIVE_EXTRACTION_UNAVAILABLE = "stt:native-extraction-unavailable";-/*** Marker carried in the error message when audio extraction could not read the* `sourcePath` itself: the OS refused the read, or ffmpeg hung opening the file@@ -154,7 +141,8 @@ export const STT_NATIVE_EXTRACTION_UNAVAILABLE = "stt:native-extraction-unavaila** Only `extractAudio.ts` sets it, so the renderer can tell "this file's folder is* not readable" from an engine failure that happens to mention a permission (model- * loading, the whisper helper). Same IPC reason as above for being a string.+ * loading, the whisper helper). A string rather than an error class because this+ * crosses `ipcRenderer.invoke`, which rebuilds a plain `Error` from the message.*/export const STT_MEDIA_UNREADABLE = "stt:media-unreadable";diff --git a/scripts/before-pack.cjs b/scripts/before-pack.cjsindex d4ecb5d48..cdb8b05c6 100644--- a/scripts/before-pack.cjs+++ b/scripts/before-pack.cjs@@ -134,8 +134,7 @@ const MAC_REQUIRED = [match: (name) => name === "ffmpeg",what: "the LGPL ffmpeg CLI (spawned for waveform peaks and STT audio extraction)",breaks:- "transcription falls back to the renderer decode or fails outright on machines with no\n" +- 'system ffmpeg, shown to the user only as "Failed to fetch" (#616)',+ 'transcription fails in the editor, over MCP and in the CLI with "ffmpeg is missing" (#616)',fix: "Build it with:\n\n npm run build:native:compositor:mac\n\nwhich stages the SDK's ffmpeg beside the vendored dylibs.",},];@@ -194,9 +193,7 @@ const LINUX_REQUIRED = [{match: (name) => name === "ffmpeg",what: "the LGPL ffmpeg CLI (spawned for waveform peaks and STT audio extraction)",- breaks:- "transcription fails over MCP and the CLI with stt:native-extraction-unavailable,\n" +- "and the editor falls back to a slow renderer decode",+ breaks: 'transcription fails in the editor, over MCP and in the CLI with "ffmpeg is missing"',fix: `${FIX_LINUX_HELPER}\n\nwhich stages the SDK's shared ffmpeg beside helper-ffmpeg/.`,},{diff --git a/scripts/before-pack.test.mjs b/scripts/before-pack.test.mjsindex b7df9a3bd..027f847b6 100644--- a/scripts/before-pack.test.mjs+++ b/scripts/before-pack.test.mjs@@ -359,7 +359,7 @@ describe("MAC_REQUIRED", () => {});// The Linux twin of #616: packages shipped the libav shared objects but no ffmpeg CLI, so-// transcription over MCP and the CLI failed with stt:native-extraction-unavailable.+// transcription over MCP and the CLI failed for want of ffmpeg.describe("LINUX_REQUIRED", () => {it("refuses a payload that has everything except the ffmpeg binary", () => {const { LINUX_REQUIRED, checkNativePayload } = testing();diff --git a/src/hooks/streamingAudioPeaks.ts b/src/hooks/streamingAudioPeaks.tsindex 8e906ad21..ed9f314af 100644--- a/src/hooks/streamingAudioPeaks.ts+++ b/src/hooks/streamingAudioPeaks.ts@@ -1,5 +1,4 @@import { WebDemuxer } from "web-demuxer";-import { audioDataFrameToMono } from "@/lib/captioning/extractMono16kWebDemuxer";/*** Streaming trim-waveform peaks for recordings too large to load into memory.@@ -17,6 +16,39 @@ import { audioDataFrameToMono } from "@/lib/captioning/extractMono16kWebDemuxer"* min/max starting from 0 (silence baseline) and channels averaged per sample.*/+/** Mixes one WebCodecs AudioData frame down to mono (averaged across channels). */+function audioDataFrameToMono(frame: AudioData): Float32Array {+ const frames = frame.numberOfFrames;+ const ch = frame.numberOfChannels;+ const out = new Float32Array(frames);+ const fmt = frame.format || "";+ const planar = fmt.includes("planar");++ if (planar) {+ const plane = new Float32Array(frames);+ for (let c = 0; c < ch; c++) {+ frame.copyTo(plane, { planeIndex: c });+ for (let i = 0; i < frames; i++) {+ out[i] += plane[i];+ }+ }+ for (let i = 0; i < frames; i++) {+ out[i] /= ch;+ }+ } else {+ const interleaved = new Float32Array(frames * ch);+ frame.copyTo(interleaved, { planeIndex: 0 });+ for (let i = 0; i < frames; i++) {+ let sum = 0;+ for (let c = 0; c < ch; c++) {+ sum += interleaved[i * ch + c];+ }+ out[i] = sum / ch;+ }+ }+ return out;+}+const DECODE_QUEUE_BACKPRESSURE = 20;const LOAD_TIMEOUT_MS = 60_000;const READ_END_PADDING_SEC = 0.5;diff --git a/src/lib/ai-edition/document/transcribe.test.ts b/src/lib/ai-edition/document/transcribe.test.tsindex a7c2d194b..b94ddf454 100644--- a/src/lib/ai-edition/document/transcribe.test.ts+++ b/src/lib/ai-edition/document/transcribe.test.ts@@ -1,29 +1,15 @@import { beforeEach, describe, expect, it, vi } from "vitest";-import { STT_NATIVE_EXTRACTION_UNAVAILABLE } from "../../../../electron/stt/transcriptionContract";import { type AxcutDocument, axcutSchemaVersion, transcriptSchema } from "../schema";+import { classifyTranscriptionError } from "../transcription/status";import { transcribeAsset } from "./transcribe";-vi.mock("@/components/video-editor/projectPersistence", () => ({- toFileUrl: (path: string) => `file://${path}`,-}));-vi.mock("@/lib/captioning", () => ({- extractMono16kFromVideoUrl: vi.fn(async () => ({- samples: new Float32Array([0, 0, 0]),- sampleRate: 16_000,- })),- transcribeMono16kToSegments: vi.fn(),transcribeSourceFileToSegments: vi.fn(),}));-const { extractMono16kFromVideoUrl, transcribeMono16kToSegments, transcribeSourceFileToSegments } =- await import("@/lib/captioning");-// `transcribeAsset` sends the PATH now and lets the main process decode; the samples-// entry point is only reached when no ffmpeg can be resolved. The assertions below-// therefore target the native call, and the fallback has tests of its own at the end.+const { transcribeSourceFileToSegments } = await import("@/lib/captioning");+// `transcribeAsset` sends the PATH and lets the main process decode.const transcribeMock = vi.mocked(transcribeSourceFileToSegments);-const rendererMock = vi.mocked(transcribeMono16kToSegments);-const extractMock = vi.mocked(extractMono16kFromVideoUrl);function makeDoc(): AxcutDocument {return {@@ -183,10 +169,7 @@ describe("transcribeAsset native extraction", () => {vi.clearAllMocks();});- it("hands the main process a path instead of decoding in the renderer", async () => {- // The point of the change: the renderer must not touch the audio at all on the- // happy path. `extractMono16kFromVideoUrl` reads the whole file, copies it twice- // and resamples on the UI thread — that is the freeze this avoids.+ it("hands the main process a path", async () => {transcribeMock.mockResolvedValueOnce({segments: [{ startSec: 0, endSec: 1, text: "hi" }],granularity: "word",@@ -194,51 +177,38 @@ describe("transcribeAsset native extraction", () => {});await transcribeAsset(makeDoc(), "asset_1");expect(transcribeMock).toHaveBeenCalledWith("/tmp/demo.mp4", expect.anything());- expect(extractMock).not.toHaveBeenCalled();- expect(rendererMock).not.toHaveBeenCalled();});- it("falls back to the renderer decode when the install has no ffmpeg", async () => {- // A dev checkout that never fetched ffmpeg, or a platform build missing it, must- // still transcribe rather than lose the feature.+ it("says ffmpeg is missing instead of falling back to a renderer decode", async () => {+ // The fallback used to catch this and decode in the renderer, which the CLI's+ // hidden runner was not allowed to read: "File path is not approved or is not a+ // supported video file". Now the reason reaches the caller as the IPC delivers it.+ const missing =+ "ffmpeg is missing: transcription needs it to read the audio, and this install has none.";transcribeMock.mockRejectedValueOnce(- new Error(`${STT_NATIVE_EXTRACTION_UNAVAILABLE}: no ffmpeg binary`),+ new Error(+ `Error invoking remote method 'stt:transcribe': FfmpegUnavailableError: ${missing}`,+ ),);- rendererMock.mockResolvedValueOnce({- segments: [{ startSec: 0, endSec: 1, text: "hi" }],- granularity: "word",- detectedLanguage: "en",- });- const transcript = await transcribeAsset(makeDoc(), "asset_1");- expect(extractMock).toHaveBeenCalled();- expect(rendererMock).toHaveBeenCalled();- expect(transcript.segments.length).toBeGreaterThan(0);+ const error = await transcribeAsset(makeDoc(), "asset_1").catch((e: unknown) => e);+ expect(transcribeMock).toHaveBeenCalledTimes(1);+ // What the CLI prints after "Transcription failed: ", and the editor shows.+ expect(classifyTranscriptionError(error)).toEqual({ kind: "error", message: missing });});- it("passes the project's names and terms on both paths", async () => {+ it("passes the project's names and terms", async () => {const doc = makeDoc();doc.project.vocabulary = "OpenScreen, Kubernetes";- transcribeMock.mockRejectedValueOnce(- new Error(`${STT_NATIVE_EXTRACTION_UNAVAILABLE}: no ffmpeg binary`),- );- rendererMock.mockResolvedValueOnce({+ transcribeMock.mockResolvedValueOnce({segments: [],granularity: "word",detectedLanguage: "en",});await transcribeAsset(doc, "asset_1");- const withPrompt = expect.objectContaining({ prompt: "OpenScreen, Kubernetes" });- expect(transcribeMock).toHaveBeenCalledWith("/tmp/demo.mp4", withPrompt);- expect(rendererMock).toHaveBeenCalledWith(expect.anything(), withPrompt);- });-- it("does NOT fall back on any other failure", async () => {- // "This file has no audio" is a verdict. Re-deriving it in the renderer would buy- // the same answer for the price of the decode this change exists to avoid.- transcribeMock.mockRejectedValueOnce(new Error("No decodable audio in /tmp/demo.mp4"));- await expect(transcribeAsset(makeDoc(), "asset_1")).rejects.toThrow("No decodable audio");- expect(extractMock).not.toHaveBeenCalled();- expect(rendererMock).not.toHaveBeenCalled();+ expect(transcribeMock).toHaveBeenCalledWith(+ "/tmp/demo.mp4",+ expect.objectContaining({ prompt: "OpenScreen, Kubernetes" }),+ );});});diff --git a/src/lib/ai-edition/document/transcribe.ts b/src/lib/ai-edition/document/transcribe.tsindex 4f3f1c701..4253eaf22 100644--- a/src/lib/ai-edition/document/transcribe.ts+++ b/src/lib/ai-edition/document/transcribe.ts@@ -2,17 +2,11 @@// src/lib/captioning/) as a transcribeAsset function that returns an// AxcutTranscript and persists it into the document.//-// ponytail: reuses extractMono16kFromVideoUrl + transcribeMono16kToSegments-// verbatim. No Python, no faster-whisper, no network calls. Privacy-safe.+// ponytail: reuses transcribeSourceFileToSegments verbatim. No Python, no+// faster-whisper, no network calls. Privacy-safe.-import { toFileUrl } from "@/components/video-editor/projectPersistence";-import {- extractMono16kFromVideoUrl,- transcribeMono16kToSegments,- transcribeSourceFileToSegments,-} from "@/lib/captioning";+import { transcribeSourceFileToSegments } from "@/lib/captioning";import type { SttRendererStatus } from "@/lib/captioning/transcribe";-import { STT_NATIVE_EXTRACTION_UNAVAILABLE } from "../../../../electron/stt/transcriptionContract";import type { AxcutDocument, AxcutTranscript } from "../schema";import { transcriptFromSegments } from "../transcription/fromStt";@@ -51,8 +45,6 @@ export async function transcribeAsset(throw new Error(`Asset ${assetId} not found in document.`);}- const videoUrl = toFileUrl(asset.originalPath);-options.onStatus?.({ phase: "extracting-audio" });// Stay on loading-model until the main process reports inference chunks.@@ -86,38 +78,17 @@ export async function transcribeAsset(...(status.totalBytes !== undefined ? { totalBytes: status.totalBytes } : {}),});- // Native first. `extractMono16kFromVideoUrl` runs in the RENDERER: it reads the- // whole media into memory, copies it twice and resamples on the UI thread, which- // is what froze the editor at project open on a long import (measured on a- // four-minute bed: ~86 MB of decoded float32 there against 15.7 MB in the main- // process). Handing the path over keeps every byte on the other side of the IPC,- // and the whisper helper it feeds was already a separate process.- //- // The fallback is not decoration: an install with no resolvable ffmpeg — a dev- // checkout that never fetched it, a platform build missing the binary — must still- // transcribe rather than lose the feature. Only THAT case falls back. "This file- // has no audio" is a verdict, and re-deriving it in the renderer would buy the same- // answer for the price of the decode this exists to avoid.+ // The main process decodes the file with ffmpeg; the renderer never holds the audio.+ // No renderer decode behind it: that path read the whole file on the UI thread,+ // and the CLI's hidden runner was not even allowed to read it. Every package ships+ // ffmpeg, so a missing one surfaces as `FfmpegUnavailableError`'s message, which+ // says so, in the editor, the CLI and MCP alike.const result = await transcribeSourceFileToSegments(asset.originalPath, {trimRegions: [],signal: options.signal,language: forcedLanguage,prompt,onStatus: forwardStatus,- }).catch(async (error: unknown) => {- const message = error instanceof Error ? error.message : String(error);- if (!message.includes(STT_NATIVE_EXTRACTION_UNAVAILABLE)) throw error;- const audioResult = await extractMono16kFromVideoUrl(videoUrl, {- signal: options.signal,- });- options.onStatus?.({ phase: "transcribing" });- return transcribeMono16kToSegments(audioResult.samples, {- trimRegions: [],- signal: options.signal,- language: forcedLanguage,- prompt,- onStatus: forwardStatus,- });});// Prefer the model-reported language (covers both forced picks and auto-detect),diff --git a/src/lib/ai-edition/transcription/status.ts b/src/lib/ai-edition/transcription/status.tsindex ea317e60e..081766695 100644--- a/src/lib/ai-edition/transcription/status.ts+++ b/src/lib/ai-edition/transcription/status.ts@@ -149,10 +149,9 @@ export function withTranscriptionFailure(/*** Map an exception out of `transcribeAsset` onto a failure the UI can explain.- * The deterministic cases come from whichever layer decoded the audio, and there- * are two of them: `extractMono16kWebDemuxer` in the renderer, and — since- * native extraction landed — `extractAudio.ts`'s `NoAudioTrackError` in the main- * process, whose wording ("No decodable audio in …") is its own.+ * The deterministic cases come from `extractAudio.ts`'s `NoAudioTrackError` in the+ * main process ("No decodable audio in …"). The other phrasings are the renderer+ * decode's, gone since the main process decodes every file.** Matched on the message rather than on the class because `ipcRenderer.invoke`* rebuilds a plain `Error` and drops both the prototype and the `name`. Missingdiff --git a/src/lib/captioning/captionConstants.ts b/src/lib/captioning/captionConstants.tsdeleted file mode 100644index 1bacb7cc7..000000000--- a/src/lib/captioning/captionConstants.ts+++ /dev/null@@ -1,2 +0,0 @@-/** Max audio length for auto-captions (decode + transcribe); keep demuxer read aligned with this. */-export const MAX_CAPTION_AUDIO_SEC = 4 * 60 * 60;diff --git a/src/lib/captioning/extractMono16k.ts b/src/lib/captioning/extractMono16k.tsdeleted file mode 100644index 1116bb9a2..000000000--- a/src/lib/captioning/extractMono16k.ts+++ /dev/null@@ -1,197 +0,0 @@-import { materializeLocalSourceFile, releaseLocalSourceFile } from "@/lib/exporter/localSourceFile";-import { MAX_IN_MEMORY_SOURCE_BYTES } from "@/lib/exporter/sourceFileLimits";-import { MAX_CAPTION_AUDIO_SEC } from "./captionConstants";-import { extractMonoPcmViaWebDemuxer } from "./extractMono16kWebDemuxer";--export { MAX_CAPTION_AUDIO_SEC };--const FETCH_TIMEOUT_MS = 120_000;-// The demuxer caption path holds every decoded AudioData frame plus full-rate-// merge buffers in memory (~50 MB per minute of 48 kHz audio all-in), so very-// long recordings would exhaust the renderer heap well before the 4 h caption-// ceiling. For sources too large to load in memory anyway, cap the decoded-// audio; captions come back truncated instead of crashing the renderer.-const LARGE_FILE_CAPTION_SEC = 30 * 60;--async function fetchWithTimeout(url: string, signal?: AbortSignal): Promise<Response> {- const ctrl = new AbortController();- const timer = window.setTimeout(() => ctrl.abort(), FETCH_TIMEOUT_MS);- const onAbort = () => ctrl.abort();- if (signal) {- if (signal.aborted) ctrl.abort();- else signal.addEventListener("abort", onAbort, { once: true });- }- try {- return await fetch(url, { signal: ctrl.signal });- } finally {- window.clearTimeout(timer);- if (signal) signal.removeEventListener("abort", onAbort);- }-}--/**- * Load the editor video like `StreamingVideoDecoder`: Electron `readBinaryFile`- * for local paths (fetch(file://) is unreliable in the renderer), otherwise- * HTTP/blob/data URLs via fetch.- */-async function loadSourceVideoFile(videoUrl: string, signal?: AbortSignal): Promise<File> {- const isRemoteUrl = /^(https?:|blob:|data:)/i.test(videoUrl);-- if (!isRemoteUrl && window.electronAPI) {- // Streams large recordings through OPFS instead of reading them whole, so- // captions work for multi-GB files just like export does. The signal also- // aborts the copy itself when the caption pass is cancelled.- const filename = (videoUrl.split(/[\\/]/).pop() || "video").replace(/^file:/, "");- return materializeLocalSourceFile(videoUrl, filename, { signal });- }-- const response = await fetchWithTimeout(videoUrl, signal);- if (!response.ok) {- throw new Error(`Failed to load video for captions: ${response.status} ${response.statusText}`);- }- const blob = await response.blob();- if (signal?.aborted) throw new DOMException("Aborted", "AbortError");- const filename = videoUrl.split("/").pop() || "video";- return new File([blob], filename, { type: blob.type || "video/webm" });-}--function mixToMono(audioBuffer: AudioBuffer): Float32Array {- const { length, numberOfChannels } = audioBuffer;- if (numberOfChannels === 0) return new Float32Array(length);- // `getChannelData` is a WebIDL call, so calling it INSIDE the sample loop cost- // one call per sample per channel — ~57 M of them for a ten-minute stereo- // recording, seconds of blocked main thread. That was survivable while- // transcription only ran when the user asked for it; it now runs by itself when- // a project opens, and a frozen window (spinners included) is exactly what the- // automatic pass must not look like. Hoisting the channel arrays out of the loop- // leaves plain typed-array indexing.- const channels: Float32Array[] = [];- for (let c = 0; c < numberOfChannels; c++) channels.push(audioBuffer.getChannelData(c));- // Mono source: the mixdown is a copy. `slice` keeps the caller's contract of- // owning its buffer (the AudioBuffer's own array is reused by the context).- if (numberOfChannels === 1) return channels[0].slice();- const out = new Float32Array(length);- for (let i = 0; i < length; i++) {- let sum = 0;- for (let c = 0; c < numberOfChannels; c++) sum += channels[c][i];- out[i] = sum / numberOfChannels;- }- return out;-}--async function resampleMono(- mono: Float32Array,- fromRate: number,- toRate: number,- signal?: AbortSignal,-): Promise<Float32Array> {- if (signal?.aborted) throw new DOMException("Aborted", "AbortError");- if (fromRate === toRate) return mono;- const durationSec = mono.length / fromRate;- const outLength = Math.max(1, Math.ceil(durationSec * toRate));- const offline = new OfflineAudioContext(1, outLength, toRate);- const buf = offline.createBuffer(1, mono.length, fromRate);- buf.copyToChannel(Float32Array.from(mono), 0);- const src = offline.createBufferSource();- src.buffer = buf;- src.connect(offline.destination);- src.start(0);- const rendered = await offline.startRendering();- if (signal?.aborted) throw new DOMException("Aborted", "AbortError");- return rendered.getChannelData(0).slice();-}--async function truncateAndResampleTo16k(- mono: Float32Array,- fromRate: number,- durationSec: number,- signal?: AbortSignal,-): Promise<{ samples: Float32Array; truncated: boolean; durationSec: number }> {- let truncated = false;- let work = mono;- if (durationSec > MAX_CAPTION_AUDIO_SEC) {- const maxSamples = Math.floor(MAX_CAPTION_AUDIO_SEC * fromRate);- work = mono.subarray(0, Math.min(mono.length, maxSamples));- truncated = true;- }-- const samples = await resampleMono(work, fromRate, 16_000, signal);- return { samples, truncated, durationSec: samples.length / 16_000 };-}--/**- * Decode the video's audio track to mono 16 kHz float samples (Whisper input).- * Prefers `decodeAudioData` when the container is supported, else the same- * web-demuxer + AudioDecoder path as export.- */-export async function extractMono16kFromVideoUrl(- videoUrl: string,- options?: { signal?: AbortSignal },-): Promise<{ samples: Float32Array; truncated: boolean; durationSec: number }> {- const file = await loadSourceVideoFile(videoUrl, options?.signal);-- /** When this returns null, use web-demuxer + AudioDecoder (same as export). */- const tryDecodeAudioDataPath = async (): Promise<{- samples: Float32Array;- truncated: boolean;- durationSec: number;- } | null> => {- const audioContext = new AudioContext();- try {- const ab = await file.arrayBuffer();- if (options?.signal?.aborted) throw new DOMException("Aborted", "AbortError");- const audioBuffer = await audioContext.decodeAudioData(ab.slice(0));- if (- audioBuffer.numberOfChannels === 0 ||- audioBuffer.length === 0 ||- !Number.isFinite(audioBuffer.duration) ||- audioBuffer.duration <= 0- ) {- return null;- }- const durationSec = audioBuffer.duration;- const mono = mixToMono(audioBuffer);- const fromRate = audioBuffer.sampleRate;- const out = await truncateAndResampleTo16k(mono, fromRate, durationSec, options?.signal);- // decodeAudioData can resolve for some WebM/Matroska inputs yet yield almost no usable- // PCM, and captions only fall back to the demuxer path on throw, so return null to recover.- if (out.samples.length < 800) {- return null;- }- return out;- } catch {- return null;- } finally {- await audioContext.close().catch(() => undefined);- }- };-- try {- // Large recordings skip the in-memory decodeAudioData path (it would load- // the whole file) and go straight to the streaming web-demuxer path below.- const isLargeFile = file.size > MAX_IN_MEMORY_SOURCE_BYTES;- const primary = isLargeFile ? null : await tryDecodeAudioDataPath();- if (primary) {- return primary;- }-- // For oversized sources, also cap how much audio the demuxer path decodes- // — its frame/merge buffers are in-memory and scale with duration.- const pcm = await extractMonoPcmViaWebDemuxer(- file,- options?.signal,- isLargeFile ? LARGE_FILE_CAPTION_SEC : undefined,- );- const out = await truncateAndResampleTo16k(- pcm.mono,- pcm.sampleRate,- pcm.durationSec,- options?.signal,- );- return { ...out, truncated: out.truncated || pcm.capped };- } finally {- // Release the OPFS cache reference taken when streaming a large source.- // The File name is the cache-entry key (no-op for small/remote sources).- releaseLocalSourceFile(file.name);- }-}diff --git a/src/lib/captioning/extractMono16kWebDemuxer.ts b/src/lib/captioning/extractMono16kWebDemuxer.tsdeleted file mode 100644index 0d1a2ba78..000000000--- a/src/lib/captioning/extractMono16kWebDemuxer.ts+++ /dev/null@@ -1,198 +0,0 @@-import { WebDemuxer } from "web-demuxer";--import { MAX_CAPTION_AUDIO_SEC } from "./captionConstants";--const DECODE_QUEUE_BACKPRESSURE = 20;-const SOURCE_LOAD_TIMEOUT_MS = 60_000;-const READ_END_PADDING_SEC = 0.5;--function webDemuxerWasmUrl(): string {- return new URL("./wasm/web-demuxer.wasm", window.location.href).href;-}--/** Mixes one WebCodecs AudioData frame down to mono (averaged across channels). */-export function audioDataFrameToMono(frame: AudioData): Float32Array {- const frames = frame.numberOfFrames;- const ch = frame.numberOfChannels;- const out = new Float32Array(frames);- const fmt = frame.format || "";- const planar = fmt.includes("planar");-- if (planar) {- const plane = new Float32Array(frames);- for (let c = 0; c < ch; c++) {- frame.copyTo(plane, { planeIndex: c });- for (let i = 0; i < frames; i++) {- out[i] += plane[i];- }- }- for (let i = 0; i < frames; i++) {- out[i] /= ch;- }- } else {- const interleaved = new Float32Array(frames * ch);- frame.copyTo(interleaved, { planeIndex: 0 });- for (let i = 0; i < frames; i++) {- let sum = 0;- for (let c = 0; c < ch; c++) {- sum += interleaved[i * ch + c];- }- out[i] = sum / ch;- }- }- return out;-}--function mergeAndConsumeDecodedAudioToMonoLinear(- frames: AudioData[],- sampleRate: number,- durationSec: number,-): Float32Array {- const sorted = [...frames].sort((a, b) => a.timestamp - b.timestamp);- const totalSamples = Math.max(1, Math.ceil(durationSec * sampleRate));- const acc = new Float32Array(totalSamples);- const weight = new Float32Array(totalSamples);-- for (const frame of sorted) {- const startSample = Math.round((frame.timestamp / 1e6) * sampleRate);- const slice = audioDataFrameToMono(frame);- for (let i = 0; i < slice.length; i++) {- const pos = startSample + i;- if (pos >= 0 && pos < totalSamples) {- acc[pos] += slice[i];- weight[pos] += 1;- }- }- frame.close();- }-- for (let i = 0; i < totalSamples; i++) {- if (weight[i] > 0) {- acc[i] /= weight[i];- }- }- return acc;-}--function withTimeout<T>(promise: Promise<T>, ms: number, message: string): Promise<T> {- return new Promise<T>((resolve, reject) => {- const id = window.setTimeout(() => reject(new Error(message)), ms);- promise- .then((v) => {- window.clearTimeout(id);- resolve(v);- })- .catch((e) => {- window.clearTimeout(id);- reject(e instanceof Error ? e : new Error(String(e)));- });- });-}--/**- * Demux + WebCodecs audio decode (same stack as export). Use when `decodeAudioData`- * can't handle the container (e.g. WebM with video).- *- * @param maxReadSec Optional cap on how much audio to demux/decode. The decoded- * frames and merge buffers are held in memory, so very long recordings must be- * capped below MAX_CAPTION_AUDIO_SEC to avoid exhausting the renderer heap.- * `capped` reports whether the cap actually cut the track short.- */-export async function extractMonoPcmViaWebDemuxer(- file: File,- signal?: AbortSignal,- maxReadSec?: number,-): Promise<{ mono: Float32Array; sampleRate: number; durationSec: number; capped: boolean }> {- const demuxer = new WebDemuxer({ wasmFilePath: webDemuxerWasmUrl() });- await withTimeout(- demuxer.load(file),- SOURCE_LOAD_TIMEOUT_MS,- "Timed out while parsing the source video for captions.",- );-- if (signal?.aborted) throw new DOMException("Aborted", "AbortError");-- const mediaInfo = await withTimeout(- demuxer.getMediaInfo(),- SOURCE_LOAD_TIMEOUT_MS,- "Timed out while reading media info for captions.",- );-- const reportedDurationSec =- Number.isFinite(mediaInfo.duration) && mediaInfo.duration > 0 ? mediaInfo.duration : 0;-- let audioConfig: AudioDecoderConfig;- try {- audioConfig = await demuxer.getDecoderConfig("audio");- } catch {- throw new Error("No audio track found in this video.");- }-- const codecCheck = await AudioDecoder.isConfigSupported(audioConfig);- if (!codecCheck.supported) {- throw new Error(`Audio codec not supported for captions: ${audioConfig.codec}`);- }-- const sampleRate = audioConfig.sampleRate || 48_000;-- // Many WebM/Matroska files report a too-short duration, so capping read at reported time stops- // demux early and clips everything past that. Read to the caption-decode ceiling instead; the- // demuxer stops when the track ends.- const readCapSec = Math.min(maxReadSec ?? MAX_CAPTION_AUDIO_SEC, MAX_CAPTION_AUDIO_SEC);- const readEndSec = readCapSec + READ_END_PADDING_SEC;- const decodedFrames: AudioData[] = [];-- const decoder = new AudioDecoder({- output: (data: AudioData) => decodedFrames.push(data),- error: (e: DOMException) => console.error("[captioning] AudioDecoder error:", e),- });- decoder.configure(audioConfig);-- const reader = demuxer.read("audio", 0, readEndSec).getReader();- try {- while (!signal?.aborted) {- const { done, value: chunk } = await reader.read();- if (done || !chunk) break;- decoder.decode(chunk);- while (decoder.decodeQueueSize > DECODE_QUEUE_BACKPRESSURE && !signal?.aborted) {- await new Promise((r) => setTimeout(r, 1));- }- }- } finally {- try {- await reader.cancel();- } catch {- /* already closed */- }- }-- if (decoder.state === "configured") {- await decoder.flush();- decoder.close();- }-- if (signal?.aborted) {- for (const f of decodedFrames) f.close();- throw new DOMException("Aborted", "AbortError");- }-- if (decodedFrames.length === 0) {- throw new Error("Decoded zero audio frames from this video.");- }-- let maxEndUs = 0;- for (const f of decodedFrames) {- const end = f.timestamp + (f.duration ?? 0);- if (end > maxEndUs) maxEndUs = end;- }- const inferredDurationSec = maxEndUs / 1e6;- // Prefer extent implied by decoded frames (fixes bad container duration); fall back to reported- // metadata when frames lack duration.- const durationSec = inferredDurationSec > 0.02 ? inferredDurationSec : reportedDurationSec;-- // The cap cut the track short when the reported extent exceeds what we read.- const capped = reportedDurationSec > readCapSec + READ_END_PADDING_SEC;-- const mono = mergeAndConsumeDecodedAudioToMonoLinear(decodedFrames, sampleRate, durationSec);- return { mono, sampleRate, durationSec, capped };-}diff --git a/src/lib/captioning/index.ts b/src/lib/captioning/index.tsindex a5ad5dabd..339782578 100644--- a/src/lib/captioning/index.ts+++ b/src/lib/captioning/index.ts@@ -6,10 +6,9 @@ export {mergeAdjacentCaptionSegments,splitMergedCaptionsByWordBounds,} from "./annotationsFromCaptions";-export { extractMono16kFromVideoUrl, MAX_CAPTION_AUDIO_SEC } from "./extractMono16k";export type {CaptionSegment,CaptionTimestampGranularity,TranscribeMono16kResult,} from "./transcribe";-export { transcribeMono16kToSegments, transcribeSourceFileToSegments } from "./transcribe";+export { transcribeSourceFileToSegments } from "./transcribe";diff --git a/src/lib/captioning/transcribe.test.ts b/src/lib/captioning/transcribe.test.tsindex 505fc59b0..9ef1d94b3 100644--- a/src/lib/captioning/transcribe.test.ts+++ b/src/lib/captioning/transcribe.test.ts@@ -1,10 +1,10 @@// @vitest-environment jsdomimport { afterEach, beforeEach, describe, expect, it, type Mock, vi } from "vitest";-import { transcribeMono16kToSegments } from "./transcribe";+import { transcribeSourceFileToSegments } from "./transcribe";/**- * The renderer-side `transcribeMono16kToSegments` is a thin adapter on top of+ * The renderer-side `transcribeSourceFileToSegments` is a thin adapter on top of* `electronAPI.stt.transcribe`. It must:* - emit each word from `wordSegments` as a single-token `CaptionSegment`,* - tag the run `granularity: "word"`,@@ -75,7 +75,7 @@ const removeMockApi = () => {lastStatusCb = null;};-describe("transcribeMono16kToSegments", () => {+describe("transcribeSourceFileToSegments", () => {beforeEach(() => {installMockApi();});@@ -95,8 +95,7 @@ describe("transcribeMono16kToSegments", () => {backend: "whisper-cpu",});- const samples = new Float32Array(1600);- const result = await transcribeMono16kToSegments(samples);+ const result = await transcribeSourceFileToSegments("/tmp/a.mp4");expect(result.granularity).toBe("word");expect(result.segments).toEqual([@@ -104,7 +103,7 @@ describe("transcribeMono16kToSegments", () => {{ text: "world", startSec: 0.31, endSec: 0.65 },]);expect(mockApi.transcribe).toHaveBeenCalledWith({- samples,+ sourcePath: "/tmp/a.mp4",language: undefined,});});@@ -117,7 +116,7 @@ describe("transcribeMono16kToSegments", () => {backend: "whisper-cpu",});- const result = await transcribeMono16kToSegments(new Float32Array(1600));+ const result = await transcribeSourceFileToSegments("/tmp/a.mp4");expect(result.granularity).toBe("phrase");expect(result.segments).toEqual([{ text: "hello world", startSec: 0, endSec: 0.65 }]);});@@ -136,7 +135,7 @@ describe("transcribeMono16kToSegments", () => {};});- await transcribeMono16kToSegments(new Float32Array(1600), { onStatus });+ await transcribeSourceFileToSegments("/tmp/a.mp4", { onStatus });expect(onStatus).toHaveBeenCalledWith({ phase: "model" });// The chunk progress must survive the hop, not just the phase — it is what// drives the progress bar.@@ -151,7 +150,7 @@ describe("transcribeMono16kToSegments", () => {it("returns empty results when running outside Electron (browser tests)", async () => {removeMockApi();- const result = await transcribeMono16kToSegments(new Float32Array(1600));+ const result = await transcribeSourceFileToSegments("/tmp/a.mp4");expect(result.granularity).toBe("word");expect(result.segments).toEqual([]);});@@ -160,7 +159,7 @@ describe("transcribeMono16kToSegments", () => {const controller = new AbortController();controller.abort();await expect(- transcribeMono16kToSegments(new Float32Array(1600), { signal: controller.signal }),+ transcribeSourceFileToSegments("/tmp/a.mp4", { signal: controller.signal }),).rejects.toMatchObject({ name: "AbortError" });expect(mockApi.transcribe).not.toHaveBeenCalled();});diff --git a/src/lib/captioning/transcribe.ts b/src/lib/captioning/transcribe.tsindex 806b66914..0d4cf997e 100644--- a/src/lib/captioning/transcribe.ts+++ b/src/lib/captioning/transcribe.ts@@ -10,7 +10,7 @@ export interface CaptionSegment {/*** How caption layout should interpret `CaptionSegment` times from- * `transcribeMono16kToSegments`. The native pipeline gets per-word timestamps+ * `transcribeSourceFileToSegments`. The native pipeline gets per-word timestamps* from whisper.cpp's own output; `"phrase"` is a fallback for the rare case* where whisper reports zero words for a segment.*/@@ -53,7 +53,7 @@ export interface SttRendererStatus {}interface RendererSttApi {- transcribe: (request: { samples: Float32Array; language?: string; prompt?: string }) => Promise<{+ transcribe: (request: { sourcePath: string; language?: string; prompt?: string }) => Promise<{segments: CaptionSegment[];wordSegments: SttWordSegment[];detectedLanguage: string;@@ -63,42 +63,6 @@ interface RendererSttApi {onStatus?: (callback: (event: SttRendererStatus) => void) => () => void;}-/**- * Transcribes mono 16 kHz audio into per-word timed caption segments. The- * renderer is a thin IPC adapter: it forwards the audio to the Electron main- * process where `whisper-server` runs recognition and emits its own- * per-word timestamps in the same pass.- *- * The previous in-Web-Worker implementation (Transformers.js + ORT-WASM) was- * 0.5× realtime on tiny models and had no word-level accuracy under 50 ms;- * see `technical-documentation/architecture/transcription-and-captions.md` § History for context.- */-export function transcribeMono16kToSegments(- samples: Float32Array,- options?: TranscribeOptions,-): Promise<TranscribeMono16kResult> {- return runTranscription({ samples }, options);-}--/**- * Same recognition, from a FILE the main process decodes itself.- *- * Preferred over `transcribeMono16kToSegments` wherever there is a path to point at.- * The samples entry point decodes in the renderer — whole file into memory, an- * `arrayBuffer()` copy, a `slice(0)` copy, then a resample loop on the UI thread —- * which is what froze the editor at open on a long import. Here the renderer sends a- * string and gets segments back.- *- * Rejects with a message carrying `STT_NATIVE_EXTRACTION_UNAVAILABLE` when the- * install has no ffmpeg, so the caller can fall back rather than lose the transcript.- */-export function transcribeSourceFileToSegments(- sourcePath: string,- options?: TranscribeOptions,-): Promise<TranscribeMono16kResult> {- return runTranscription({ sourcePath }, options);-}-export interface TranscribeOptions {trimRegions?: TrimRegion[];onStatus?: (status: SttRendererStatus) => void;@@ -108,8 +72,22 @@ export interface TranscribeOptions {prompt?: string;}-function runTranscription(- payload: { samples: Float32Array } | { sourcePath: string },+/**+ * Transcribes a media FILE into per-word timed caption segments. The renderer is a+ * thin IPC adapter: it sends the path, the main process decodes it with ffmpeg and+ * `whisper-server` runs recognition, emitting per-word timestamps in the same pass.+ * The renderer never holds the audio: decoding it here read the whole file, copied it+ * twice and resampled on the UI thread, which is what froze the editor at open.+ *+ * Rejects with `FfmpegUnavailableError`'s message when the install has no ffmpeg.+ * There is no renderer decode to fall back on any more.+ *+ * The previous in-Web-Worker implementation (Transformers.js + ORT-WASM) was+ * 0.5× realtime on tiny models and had no word-level accuracy under 50 ms;+ * see `technical-documentation/architecture/transcription-and-captions.md` § History for context.+ */+export function transcribeSourceFileToSegments(+ sourcePath: string,options?: TranscribeOptions,): Promise<TranscribeMono16kResult> {if (options?.signal?.aborted) {@@ -138,7 +116,7 @@ function runTranscription(// trimmed leading silence with a peak detector and got false positives on// quiet music intros / room tone. VAD or nothing.return api- .transcribe({ ...payload, language: forcedLanguage, prompt: options?.prompt })+ .transcribe({ sourcePath, language: forcedLanguage, prompt: options?.prompt }).then((result) => ({ ...segmentsFromStt(result), detectedLanguage: result.detectedLanguage })).catch((error: unknown) => {// A run the caller cancelled surfaces as an abort, not as an enginediff --git a/technical-documentation/architecture/transcription-and-captions.md b/technical-documentation/architecture/transcription-and-captions.mdindex a39171c6d..f71ff8b33 100644--- a/technical-documentation/architecture/transcription-and-captions.md+++ b/technical-documentation/architecture/transcription-and-captions.md@@ -15,8 +15,8 @@ the transcript).flowchart LRZ["Asset added<br/>(import or recording)"] -- "sync()" --> Y["transcriptionStore queue<br/>(src/lib/ai-edition/store/transcriptionStore.ts)"]Y -- "one job at a time" --> A["Recorded audio"]- A -- "extract mono 16 kHz" --> B["transcribeAsset<br/>(src/lib/ai-edition/document/transcribe.ts)"]- B -- "IPC: Float32Array + language" --> C["SttManager<br/>(electron/stt/index.ts)"]+ A -- "file path" --> B["transcribeAsset<br/>(src/lib/ai-edition/document/transcribe.ts)"]+ B -- "IPC: source path + language" --> C["SttManager<br/>(electron/stt/index.ts)<br/>ffmpeg: mono 16 kHz"]C -- "POST /inference (WAV)" --> D["whisper-stt-server<br/>(electron/native/whisper-stt/)"]D -- "VAD cut + whisper_full() + DTW" --> E["SttTranscribeResponse<br/>(segments + wordSegments)"]E --> C@@ -27,10 +27,11 @@ flowchart LRH -- "annotation path<br/>(scene description)" --> I["Native compositor<br/>(preview AND export)"]```-The renderer pieces — `transcribeMono16kToSegments`+The renderer pieces — `transcribeSourceFileToSegments`(`src/lib/captioning/transcribe.ts`) called from `transcribeAsset`(`src/lib/ai-edition/document/transcribe.ts`) — are a thin IPC adapter: the-audio crosses to the main process, the helper does the recognition, and the+file path crosses to the main process, which decodes it with ffmpeg+(`electron/stt/extractAudio.ts`), the helper does the recognition, and theresult is mapped back onto the `AxcutTranscript` shape the rest of the editorreads. From the moment that transcript is persisted, captions and the wordsthey show cannot drift — captions are a derived view, not a parallel store.@@ -71,8 +72,7 @@ What the store owns and what it does not:the panes' one button: every timeline asset that still lacks a transcript,skipping the ones already known to be silent and waiting on (rather thanduplicating) a run the background pass already has in flight.-- **One run at a time.** whisper-server is a single process and the audio- extraction path holds decoded frames in renderer memory, so the pump is a+- **One run at a time.** whisper-server is a single process, so the pump is asequential loop, not a fan-out.- **The auto pass never loops.** `sync` enqueues an asset only when it has notranscript, no job entry (queued / running / failed alike) and no persisted@@ -132,12 +132,12 @@ Three edges make that promise hold, and each is load-bearing:clip's word stream swallow Backspace and hover-bin clicks in silence for thewhole background pass.-Still synchronous, and still on the main thread: `mixToMono`-([`src/lib/captioning/extractMono16k.ts`](../../src/lib/captioning/extractMono16k.ts))-now hoists its channel arrays out of the sample loop (it was making one WebIDL-call per sample per channel), but a long recording will still block the renderer-for a moment during "extracting-audio". Moving the mixdown to a worker is the-next step if it shows up in practice.+Audio extraction never runs in the renderer. There is no renderer decode to fall+back on: it read the whole file on the UI thread, froze the editor at open, and+the CLI's hidden runner was not even allowed to read the file. An install with+no ffmpeg fails with "ffmpeg is missing", in the editor, the CLI and MCP alike;+every package ships one, and `before-pack` refuses a macOS or Linux payload+without it.## The STT engine