Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.
JavaScript aianthropiccross-platformgeminigroqlinuxmacosnvidiaopen-sourceopenaiparakeetspeech-to-texttranscribewhisperwindows
feat(transcription): Gemini Live streaming for dictation (#2062)
Gabriel Stein9c41a8705ee7
diff --git a/docs/network-allowlist.md b/docs/network-allowlist.mdindex 9f86d83b0..94a601bf0 100644--- a/docs/network-allowlist.md+++ b/docs/network-allowlist.md@@ -18,14 +18,15 @@ onboarding).## Required for streaming transcription-OpenWhispr Cloud routes streaming sessions through one of three providers.-Allowlist all three unless a specific provider is pinned in configuration.+OpenWhispr Cloud routes streaming sessions through one of four providers.+Allowlist all four unless a specific provider is pinned in configuration.-| Host | Protocol | Port | Purpose |-| -------------------------- | ---------- | ---- | --------------------------------------------------------------------------------- |-| `api.deepgram.com` | WSS | 443 | Deepgram streaming transcription. |-| `api.openai.com` | WSS, HTTPS | 443 | OpenAI Realtime streaming transcription. |-| `streaming.assemblyai.com` | WSS, HTTPS | 443 | AssemblyAI streaming transcription. Token endpoint is HTTPS; live session is WSS. |+| Host | Protocol | Port | Purpose |+| ----------------------------------- | ---------- | ---- | --------------------------------------------------------------------------------------- |+| `api.deepgram.com` | WSS | 443 | Deepgram streaming transcription. |+| `api.openai.com` | WSS, HTTPS | 443 | OpenAI Realtime streaming transcription. |+| `streaming.assemblyai.com` | WSS, HTTPS | 443 | AssemblyAI streaming transcription. Token endpoint is HTTPS; live session is WSS. |+| `generativelanguage.googleapis.com` | WSS, HTTPS | 443 | Gemini Live streaming transcription. Live session is WSS; batch transcription is HTTPS. |## Required for local model downloads@@ -55,11 +56,11 @@ Contacted only when a user pastes a URL into the Upload view to download andtranscribe its audio. Downloads are HTTPS-only and hosts resolving toprivate/internal addresses are rejected.-| Host | Protocol | Port | Purpose |-| ----------------------------------- | -------- | ---- | -------------------------------------------------------------------------- |-| `www.youtube.com`, `youtube.com`, `youtu.be`, `m.youtube.com`, `music.youtube.com` | HTTPS | 443 | YouTube page/metadata fetch for pasted YouTube links (bundled yt-dlp). |-| `*.googlevideo.com` | HTTPS | 443 | YouTube media CDN — the actual audio stream download. |-| _User-pasted hosts_ | HTTPS | 443 | Direct audio/video URL imports contact whatever public host the user pastes. |+| Host | Protocol | Port | Purpose |+| ---------------------------------------------------------------------------------- | -------- | ---- | ---------------------------------------------------------------------------- |+| `www.youtube.com`, `youtube.com`, `youtu.be`, `m.youtube.com`, `music.youtube.com` | HTTPS | 443 | YouTube page/metadata fetch for pasted YouTube links (bundled yt-dlp). |+| `*.googlevideo.com` | HTTPS | 443 | YouTube media CDN — the actual audio stream download. |+| _User-pasted hosts_ | HTTPS | 443 | Direct audio/video URL imports contact whatever public host the user pastes. |## BYOK provider hosts (only if configured)@@ -71,7 +72,7 @@ provider. Skip any provider not in use.| `api.openai.com` | HTTPS | 443 | OpenAI API key configured (transcription or reasoning). || `*.cognitiveservices.azure.com`, `*.openai.azure.com`, `*.services.ai.azure.com` | HTTPS | 443 | Azure AI Foundry / Azure OpenAI speech-to-text configured (custom transcription provider pointed at your own Azure resource endpoint). || `api.anthropic.com` | HTTPS | 443 | Anthropic API key configured. |-| `generativelanguage.googleapis.com` | HTTPS | 443 | Gemini API key configured. |+| `generativelanguage.googleapis.com` | WSS, HTTPS | 443 | Gemini API key configured. Batch transcription is HTTPS; the Live streaming model opens a WSS session. || `api.groq.com` | HTTPS | 443 | Groq API key configured. || `atc.tinfoil.sh`, `*.tinfoil.sh` | WSS, HTTPS | 443 | Tinfoil API key configured. `atc.tinfoil.sh` serves the enclave attestation bundle (verified locally against an embedded sigstore root). Inference and realtime transcription connect to an enclave host assigned dynamically at runtime (e.g. `inference.tinfoil.sh`, `router.infN.tinfoil.sh`), so allowlist `*.tinfoil.sh` rather than pinning hosts. || `api.mistral.ai` | HTTPS | 443 | Mistral API key configured. |@@ -105,6 +106,7 @@ curl -v https://api.openwhispr.com/api/healthcurl -v https://api.deepgram.com/v1/projectscurl -v https://api.openai.com/v1/modelscurl -v https://streaming.assemblyai.com/v3/token+curl -v https://generativelanguage.googleapis.com/v1beta/models# Model downloads (only if local mode is in use)curl -v -I https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.bindiff --git a/preload.js b/preload.jsindex b20035329..1619de8f5 100644--- a/preload.js+++ b/preload.js@@ -837,6 +837,27 @@ contextBridge.exposeInMainWorld("electronAPI", {(callback) => (_event, data) => callback(data)),+ // Gemini Live Streaming+ geminiStreamingWarmup: (options) => ipcRenderer.invoke("gemini-streaming-warmup", options),+ geminiStreamingStart: (options) => ipcRenderer.invoke("gemini-streaming-start", options),+ geminiStreamingSend: (audioBuffer) => ipcRenderer.send("gemini-streaming-send", audioBuffer),+ geminiStreamingFinalize: () => ipcRenderer.send("gemini-streaming-finalize"),+ geminiStreamingStop: () => ipcRenderer.invoke("gemini-streaming-stop"),+ geminiStreamingStatus: () => ipcRenderer.invoke("gemini-streaming-status"),+ onGeminiPartialTranscript: registerListener(+ "gemini-partial-transcript",+ (callback) => (_event, text) => callback(text)+ ),+ onGeminiFinalTranscript: registerListener(+ "gemini-final-transcript",+ (callback) => (_event, text) => callback(text)+ ),+ onGeminiError: registerListener("gemini-error", (callback) => (_event, error) => callback(error)),+ onGeminiSessionEnd: registerListener(+ "gemini-session-end",+ (callback) => (_event, data) => callback(data)+ ),+// Corti streaming (BYOK)cortiStreamingWarmup: (options) => ipcRenderer.invoke("corti-streaming-warmup", options),cortiStreamingStart: (options) => ipcRenderer.invoke("corti-streaming-start", options),diff --git a/scripts/stt-canary.mjs b/scripts/stt-canary.mjsindex 7fc34094a..37a2c4524 100644--- a/scripts/stt-canary.mjs+++ b/scripts/stt-canary.mjs@@ -5,9 +5,9 @@* endpoint the app dials — so a provider changing its auth, endpoint, or* handshake surfaces here on a schedule instead of in a customer report* (#1624 sat in shipped builds for three days as a swallowed warmup warning).- * Batch providers whose request shape is theirs alone rather than the shared- * OpenAI-compatible multipart (Gemini) send a real transcription through the- * shipped module for the same reason.+ * Providers whose request shape is theirs alone rather than the shared+ * OpenAI-compatible multipart (Gemini, batch and Live) send a real+ * transcription through the shipped module for the same reason.** Run: node scripts/stt-canary.mjs* Keys come from STT_CANARY_<PROVIDER>_KEY env vars; providers without a key@@ -25,10 +25,12 @@ import ffmpegPath from "ffmpeg-static";import tokenProviders from "../src/helpers/realtimeTokenProviders.js";import audioUtils from "../src/utils/audioUtils.js";import geminiTranscription from "../src/helpers/geminiTranscription.js";+import geminiLive from "../src/helpers/geminiLiveStreaming.js";const { fetchRealtimeTokenForProvider } = tokenProviders;const { pcm16ToWav } = audioUtils;const { transcribeWithGemini } = geminiTranscription;+const { GeminiLiveStreaming } = geminiLive;const HANDSHAKE_TIMEOUT_MS = 15000;// Half a second of 16 kHz mono silence: enough for a provider to accept and@@ -110,12 +112,89 @@ async function probeGeminiBatch(key, audio, contentType) {return { ok: true };}+// A completed Live turn proves the raw-key handshake, the setup message, the+// audio frame shape and the transcript events — the socket opening proves none+// of it, and server-side VAD suppresses silence, so the probe has to speak.+const CANARY_PHRASE = "The quick brown fox. OpenWhispr transcription test.";+const LIVE_FRAME_MS = 50;+const LIVE_FRAME_BYTES = 1600; // one 50ms dictation worklet frame at 16kHz s16le++// Gemini's own TTS keeps the fixture out of the repo and needs no second key.+async function synthesizeCanarySpeech(key) {+ const response = await fetch(+ "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-preview-tts:generateContent",+ {+ method: "POST",+ headers: { "Content-Type": "application/json", "x-goog-api-key": key },+ body: JSON.stringify({+ contents: [{ parts: [{ text: `Say clearly: ${CANARY_PHRASE}` }] }],+ generationConfig: { responseModalities: ["AUDIO"] },+ }),+ }+ );+ if (!response.ok) throw new Error(`speech fixture request failed: HTTP ${response.status}`);+ const data = await response.json();+ const audio = data?.candidates?.[0]?.content?.parts?.[0]?.inlineData?.data;+ if (!audio) throw new Error("speech fixture returned no audio");+ // TTS answers with 24kHz L16; dictation captures 16kHz.+ const resampled = spawnSync(+ ffmpegPath,+ [+ "-f",+ "s16le",+ "-ar",+ "24000",+ "-ac",+ "1",+ "-i",+ "pipe:0",+ "-ar",+ "16000",+ "-f",+ "s16le",+ "pipe:1",+ ],+ { input: Buffer.from(audio, "base64"), maxBuffer: 64 * 1024 * 1024 }+ );+ if (resampled.status !== 0) throw new Error("could not resample the speech fixture to 16kHz");+ return resampled.stdout;+}++async function probeGeminiLive(key) {+ const token = await fetchRealtimeTokenForProvider("gemini-realtime", tokenDeps(key), {+ mode: "byok",+ });+ const pcm = await synthesizeCanarySpeech(key);+ const streaming = new GeminiLiveStreaming();+ let partials = 0;+ streaming.onPartialTranscript = () => {+ partials += 1;+ };+ try {+ await streaming.connect({ token, mode: "byok", keyterms: ["OpenWhispr"] });+ // Paced like the mic worklet: the server finalizes relative to real-time+ // ingest, so a burst followed by audioStreamEnd starves the final.+ for (let offset = 0; offset < pcm.length; offset += LIVE_FRAME_BYTES) {+ streaming.sendAudio(pcm.subarray(offset, offset + LIVE_FRAME_BYTES));+ await new Promise((resolve) => setTimeout(resolve, LIVE_FRAME_MS));+ }+ const { text } = await streaming.disconnect(true);+ if (!/quick brown fox/i.test(text)) {+ return { ok: false, detail: `no usable transcript: ${JSON.stringify(text)}` };+ }+ return { ok: true, note: `${partials} partials, final "${text}"` };+ } finally {+ streaming.cleanup();+ }+}+const tokenDeps = (key) => ({environmentManager: {getOpenAIKey: () => key,getTinfoilKey: () => key,getDeepgramKey: () => key,getAssemblyAIKey: () => key,+ getGeminiKey: () => key,},proxyFetch: fetch,});@@ -173,6 +252,11 @@ const PROBES = [: { ok: false, detail: "empty token" };},},+ {+ id: "gemini-live-streaming",+ keyEnv: "STT_CANARY_GEMINI_KEY",+ run: (key) => probeGeminiLive(key),+ },{id: "gemini-batch-wav",keyEnv: "STT_CANARY_GEMINI_KEY",diff --git a/src/components/TranscriptionModelPicker.tsx b/src/components/TranscriptionModelPicker.tsxindex 226f0f915..a989a8860 100644--- a/src/components/TranscriptionModelPicker.tsx+++ b/src/components/TranscriptionModelPicker.tsx@@ -14,7 +14,7 @@ import { useDialogs } from "../hooks/useDialogs";import { useModelDownload, type DownloadProgress } from "../hooks/useModelDownload";import {getTranscriptionProviders,- getStreamingTranscriptionProviders,+ getMeetingStreamingTranscriptionProviders,TranscriptionProviderData,WHISPER_MODEL_INFO,PARAKEET_MODEL_INFO,@@ -463,8 +463,11 @@ export default function TranscriptionModelPicker({(providerId: string) => isProviderAllowedByPolicy(policyState, "transcription", providerId),[policyState]);+ // streamingOnly is Note Recording's picker, so it offers the streaming+ // providers note recording can actually run — not every streaming provider.const availableCloudProviders = useMemo(- () => (streamingOnly ? getStreamingTranscriptionProviders() : getTranscriptionProviders()),+ () =>+ streamingOnly ? getMeetingStreamingTranscriptionProviders() : getTranscriptionProviders(),[streamingOnly]);const cloudProviders = useMemo(diff --git a/src/components/settings/MeetingSettings.tsx b/src/components/settings/MeetingSettings.tsxindex c6ae5d6ed..9b4e2eeb8 100644--- a/src/components/settings/MeetingSettings.tsx+++ b/src/components/settings/MeetingSettings.tsx@@ -10,9 +10,9 @@ import { Toggle } from "../ui/toggle";import TranscriptionModelPicker from "../TranscriptionModelPicker";import type { InferenceMode } from "../../types/electron";import { useStartOnboarding } from "../../hooks/useStartOnboarding";-import { getStreamingTranscriptionProviders } from "../../models/ModelRegistry";+import { getMeetingStreamingTranscriptionProviders } from "../../models/ModelRegistry";-const MEETING_BYOK_PROVIDER_IDS = getStreamingTranscriptionProviders().map(+const MEETING_BYOK_PROVIDER_IDS = getMeetingStreamingTranscriptionProviders().map((provider) => provider.id);diff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.jsindex ac4feaf16..055fb8e54 100644--- a/src/helpers/audioManager.js+++ b/src/helpers/audioManager.js@@ -380,6 +380,24 @@ const STREAMING_PROVIDERS = {onSessionEnd: (cb) => window.electronAPI.onAssemblyAiSessionEnd(cb),},"openai-realtime": makeDictationRealtimeProvider("openai-realtime"),+ gemini: {+ // The final transcript lands ~500ms after audioStreamEnd (which finalize+ // sends), ~2s at the p95 tail, so the stop sequence waits for it under a+ // wider ceiling. geminiLiveStreaming.js measures the same 3s budget from+ // audioStreamEnd before its own disconnect gives up.+ awaitsFinalTranscript: true,+ finalCeilingMs: 3000,+ warmup: (opts) => window.electronAPI.geminiStreamingWarmup(opts),+ start: (opts) => window.electronAPI.geminiStreamingStart(opts),+ send: (buf) => window.electronAPI.geminiStreamingSend(buf),+ finalize: () => window.electronAPI.geminiStreamingFinalize(),+ stop: () => window.electronAPI.geminiStreamingStop(),+ status: () => window.electronAPI.geminiStreamingStatus(),+ onPartial: (cb) => window.electronAPI.onGeminiPartialTranscript(cb),+ onFinal: (cb) => window.electronAPI.onGeminiFinalTranscript(cb),+ onError: (cb) => window.electronAPI.onGeminiError(cb),+ onSessionEnd: (cb) => window.electronAPI.onGeminiSessionEnd(cb),+ },corti: {warmup: (opts) => window.electronAPI.cortiStreamingWarmup(opts),start: (opts) => window.electronAPI.cortiStreamingStart(opts),@@ -3780,10 +3798,11 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);const selfHostedModel = resolveSelfHostedTranscriptionModel(s);if (selfHostedModel) return selfHostedModel;const provider = s.cloudTranscriptionProvider || "openai";- // Tinfoil pins its batch model in the registry rather than in settings.- if (provider === "tinfoil") {- return getBatchTranscriptionModel("tinfoil");- }+ // Tinfoil and Gemini pin their batch model in the registry rather than in+ // settings: their streaming model has no batch endpoint, so a streaming+ // fallback that reused the selected model would POST an unusable id.+ const batchModel = getBatchTranscriptionModel(provider);+ if (batchModel) return batchModel;return resolveByokModel(provider, s.cloudTranscriptionModel);} catch (error) {return "gpt-4o-mini-transcribe";@@ -4036,6 +4055,16 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);return !!model?.streaming && !!s.tinfoilApiKey;}+ // Gemini Live streams over its own WSS on either credential; the batch+ // Gemini model on the same provider stays on HTTP.+ if (s.cloudTranscriptionProvider === "gemini") {+ const provider = getTranscriptionProvider("gemini");+ const model = provider?.models.find((m) => m.id === s.cloudTranscriptionModel);+ if (!model?.streaming) return false;+ if (s.cloudTranscriptionMode === "byok") return !!s.geminiApiKey;+ return !!(isSignedInOverride ?? s.isSignedIn);+ }+// The managed-cloud bootstrap only controls OpenWhispr Cloud. A user's// BYOK realtime model must not be downgraded because managed dictation is// configured for batch processing.@@ -4571,7 +4600,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);// Resolves once the transcript stops moving. An outstanding partial proves its// final is still in flight, so only the ceiling ends the wait until it lands —// a plain debounce would expire on the very tail this exists to catch.- awaitStreamingTextSettled() {+ awaitStreamingTextSettled(ceilingMs = STREAMING_FINAL_CEILING_MS) {return new Promise((resolve) => {const settle = () => {clearTimeout(this.streamingTextDebounce);@@ -4580,7 +4609,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);this.streamingTextDebounce = null;resolve();};- const ceiling = setTimeout(settle, STREAMING_FINAL_CEILING_MS);+ const ceiling = setTimeout(settle, ceilingMs);const arm = () => {clearTimeout(this.streamingTextDebounce);if (this.streamingPartialText) return;@@ -4823,7 +4852,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);const provider = this.getStreamingProvider();provider.finalize?.();if (provider.awaitsFinalTranscript) {- await this.awaitStreamingTextSettled();+ await this.awaitStreamingTextSettled(provider.finalCeilingMs);} else {await new Promise((resolve) => setTimeout(resolve, 300));}diff --git a/src/helpers/dictationStreamingRouting.js b/src/helpers/dictationStreamingRouting.jsindex 90fff48c0..be156522c 100644--- a/src/helpers/dictationStreamingRouting.js+++ b/src/helpers/dictationStreamingRouting.js@@ -7,6 +7,11 @@export const REALTIME_MODELS = new Set(["gpt-4o-mini-transcribe", "gpt-4o-transcribe"]);+// REALTIME_MODELS is the OpenAI-only shortcut (it forces "openai-realtime"), so+// Gemini's live model routes on its own. Keying on the model id and not the+// provider is required: the batch model on the same provider is HTTP-only.+export const GEMINI_LIVE_MODEL = "gemini-3.5-transcribe-live";+export function defaultStreamingProviderName(context) {return context === "notes" ? "deepgram" : "openai-realtime";}@@ -21,6 +26,12 @@ export function resolveStreamingProviderName({ settings, context, sttConfig }) {) {return "corti";}+ if (+ settings.cloudTranscriptionProvider === "gemini" &&+ settings.cloudTranscriptionModel === GEMINI_LIVE_MODEL+ ) {+ return "gemini";+ }if (REALTIME_MODELS.has(settings.cloudTranscriptionModel)) {return "openai-realtime";}diff --git a/src/helpers/geminiLiveStreaming.js b/src/helpers/geminiLiveStreaming.jsnew file mode 100644index 000000000..f659bf4b5--- /dev/null+++ b/src/helpers/geminiLiveStreaming.js@@ -0,0 +1,478 @@+const WebSocket = require("ws");+const debugLogger = require("./debugLogger");+const { computePcm16Rms } = require("../utils/audioUtils");++const SAMPLE_RATE = 16000;+const WEBSOCKET_TIMEOUT_MS = 15000;+// Budget for the final transcript, measured from audioStreamEnd (it lands+// ~500ms after, ~2s at the p95 tail). The renderer's gemini settle ceiling in+// audioManager.js is the same 3s; disconnect only spends what is left of it.+const DISCONNECT_TIMEOUT_MS = 3000;+const KEEPALIVE_INTERVAL_MS = 15000;+const COLD_START_BUFFER_MAX = 3 * SAMPLE_RATE * 2; // 3 seconds of 16-bit PCM+// Google documents a 1000-phrase ceiling but only promises quality up to 100;+// the batch dictionary paths truncate the same array at 100.+const MAX_CUSTOM_VOCABULARY = 100;+// Trailing room noise in dictation recordings measures 0.003-0.010 RMS while+// speech frames sit above ~0.0105, so only a frame at this level is a new+// utterance rather than the tail of a turn the server already closed.+const TURN_REOPEN_RMS = 0.012;+const GEMINI_LIVE_MODEL = "gemini-3.5-transcribe-live";++// In managed mode the model comes from the BYOK-shaped settings, which still+// hold another provider's id, so anything but a Gemini id is ignored.+const resolveLiveModel = (model) =>+ model && model.startsWith("gemini-") ? model : GEMINI_LIVE_MODEL;++const WS_BASE = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage";++// The two Live methods are auth-disjoint: BidiGenerateContent accepts only+// `key`, BidiGenerateContentConstrained only `access_token` (an auth_tokens+// resource name). Crossing them closes the socket with 1007 right after a+// successful HTTP 101, so the credential kind decides the method, not the URL.+function buildGeminiLiveUrl({ mode, token }) {+ if (mode === "byok") {+ return `${WS_BASE}.v1beta.GenerativeService.BidiGenerateContent?key=${token}`;+ }+ return `${WS_BASE}.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=${token}`;+}++// Every Live failure arrives as a close frame after the upgrade succeeded (the+// socket's `error` event never carries them), and the server truncates close+// reasons to the 123-byte WebSocket limit — classify on the code alone; the+// reason is only logged by the close handler.+function closeFrameError(code) {+ if (code === 1011) {+ // "Token has been used too many times" / "Token has expired". The managed+ // tokens are single-use, so this is the one retryable auth failure; byok+ // re-handshakes with the same raw key and never sees it.+ return Object.assign(new Error("Gemini Live session token expired"), {+ code: "AUTH_EXPIRED",+ });+ }+ if (code === 1007 || code === 1008) {+ // 1007 is a malformed credential, 1008 one Google no longer knows (revoked+ // or deleted); neither is retryable, both are fixed in Settings.+ return Object.assign(new Error("Invalid Gemini API key. Check your key in Settings."), {+ code: "INVALID_KEY",+ });+ }+ return new Error(`Gemini Live closed before ready (code: ${code})`);+}++class GeminiLiveStreaming {+ constructor() {+ this.ws = null;+ this.isConnected = false;+ this.isConnecting = false;+ this.isDisconnecting = false;+ this.completedSegments = [];+ this.onPartialTranscript = null;+ this.onFinalTranscript = null;+ this.onError = null;+ this.onSessionEnd = null;+ this.onConnectionLost = null;+ this.pendingResolve = null;+ this.pendingReject = null;+ this.connectionTimeout = null;+ this.keepAliveInterval = null;+ this.coldStartBuffer = [];+ this.coldStartBufferSize = 0;+ this.bufferingAudio = false;+ this.audioBytesSent = 0;+ this.currentModel = GEMINI_LIVE_MODEL;+ this._connectionLossNotified = false;+ this._audioStreamEndSentAt = null;+ this._turnEnded = false;+ this._turnEndResolve = null;+ }++ // Starts buffering audio before the socket exists, covering the token fetch+ // and the 350ms-1.4s handshake so sendAudio() doesn't drop the first words.+ beginConnecting() {+ this.bufferingAudio = true;+ this.coldStartBuffer = [];+ this.coldStartBufferSize = 0;+ }++ getFullTranscript() {+ return this.completedSegments.join(" ");+ }++ // Test seam: overridden to point at a loopback server.+ buildWebSocketUrl(options) {+ return buildGeminiLiveUrl(options);+ }++ buildSetupMessage({ model, language, keyterms } = {}) {+ const customVocabulary = (keyterms || [])+ .map((term) => String(term).trim())+ .filter(Boolean)+ .slice(0, MAX_CUSTOM_VOCABULARY);+ const inputAudioTranscription = {};+ // `languageCodes` on its own costs punctuation and sentence casing, and an+ // invalid locale is accepted silently — only ever send a language the user+ // picked explicitly ("auto" is stripped upstream).+ if (language) inputAudioTranscription.languageCodes = [language];+ if (customVocabulary.length) inputAudioTranscription.customVocabulary = customVocabulary;++ return {+ setup: {+ model: `models/${resolveLiveModel(model)}`,+ generationConfig: { responseModalities: ["TEXT"] },+ inputAudioTranscription,+ },+ };+ }++ async connect(options = {}) {+ const { token, mode, refreshToken } = options;+ if (!token) {+ throw Object.assign(new Error("Gemini API key not configured. Add your key in Settings."), {+ code: "API_KEY_MISSING",+ });+ }++ if (this.isConnected || this.isConnecting) {+ debugLogger.debug("Gemini Live already connected/connecting");+ return;+ }++ if (!this.bufferingAudio) this.beginConnecting();+ this.completedSegments = [];+ this.audioBytesSent = 0;+ this.currentModel = resolveLiveModel(options.model);+ this._connectionLossNotified = false;+ this._audioStreamEndSentAt = null;+ this._turnEnded = false;++ try {+ await this._openSocket(options);+ } catch (err) {+ if (err.code !== "AUTH_EXPIRED" || mode === "byok" || !refreshToken) throw err;+ debugLogger.debug("Gemini Live re-minting single-use token after 1011 close");+ this.bufferingAudio = true;+ await this._openSocket({ ...options, token: await refreshToken() });+ }+ }++ _openSocket(options) {+ const url = this.buildWebSocketUrl(options);+ debugLogger.debug("Gemini Live connecting", {+ mode: options.mode === "byok" ? "byok" : "openwhispr",+ model: this.currentModel,+ });++ return new Promise((resolve, reject) => {+ this.pendingResolve = resolve;+ this.pendingReject = reject;++ this.isConnecting = true;+ this.connectionTimeout = setTimeout(() => {+ this.cleanup();+ reject(new Error("Gemini Live connection timeout"));+ }, WEBSOCKET_TIMEOUT_MS);++ this.ws = new WebSocket(url);++ this.ws.on("open", () => {+ // Audio sent before setupComplete is discarded by the server, so the+ // setup message is the only thing that goes out on open.+ this.ws.send(JSON.stringify(this.buildSetupMessage(options)));+ });++ this.ws.on("message", (data) => {+ this.handleMessage(data);+ });++ this.ws.on("error", (error) => {+ const wasActive = this.isConnected;+ debugLogger.error("Gemini Live WebSocket error", { error: error.message });+ this.cleanup();+ this._rejectPending(error);+ if (wasActive && !this.isDisconnecting) {+ this._notifyConnectionLost(error);+ } else if (!this.isDisconnecting) {+ this.onError?.(error);+ }+ });++ this.ws.on("close", (code, reason) => {+ const wasActive = this.isConnected;+ debugLogger.debug("Gemini Live WebSocket closed", {+ code,+ reason: reason?.toString() || "",+ wasActive,+ });+ this._resolveTurnEnd();+ this.cleanup();+ if (!wasActive) {+ this._rejectPending(closeFrameError(code));+ } else if (!this.isDisconnecting) {+ // A session killed mid-dictation (the ~10 minute cap, a dead network)+ // still hands over what it transcribed; there is no resumption handle+ // to reconnect with, so the caller finalizes with this text.+ this.onSessionEnd?.({ text: this.getFullTranscript() });+ this._notifyConnectionLost(new Error(`Connection lost (code: ${code})`));+ }+ });+ });+ }++ // The complete server vocabulary is `setupComplete` plus+ // `serverContent.{speechState, interimInputTranscription, inputTranscription,+ // generationComplete}`. There is no `turnComplete`, no `usageMetadata` and no+ // `sessionResumptionUpdate` on this model, so end-of-turn is the final+ // transcript arriving with `generationComplete` in the same millisecond.+ handleMessage(data) {+ try {+ const message = JSON.parse(data.toString());++ if (message.setupComplete) this._markConnected();++ const serverContent = message.serverContent;+ if (!serverContent) return;++ // Partials are revised, not appended to ("The quick brown" becomes+ // "the quick brown fox"), so consumers must replace the whole string.+ const partial = serverContent.interimInputTranscription?.text;+ if (partial) {+ this._turnEnded = false;+ this.onPartialTranscript?.(partial);+ }++ const final = serverContent.inputTranscription?.text?.trim();+ if (final) {+ this.completedSegments.push(final);+ const fullText = this.getFullTranscript();+ this.onFinalTranscript?.(fullText, Date.now());+ debugLogger.debug("Gemini Live turn completed", {+ turnText: final.slice(0, 100),+ totalLength: fullText.length,+ segments: this.completedSegments.length,+ });+ }++ // The server closes a turn on trailing silence, often before the hotkey is+ // released; that generationComplete already answers the final. Speech+ // (a loud frame or a new partial) reopens a turn, so a later stop waits again.+ if (serverContent.generationComplete) this._resolveTurnEnd();+ } catch (err) {+ debugLogger.error("Gemini Live message parse error", { error: err.message });+ }+ }++ _markConnected() {+ if (this.isConnected) return;+ this.isConnected = true;+ this.isConnecting = false;+ clearTimeout(this.connectionTimeout);+ this.connectionTimeout = null;+ this.startKeepAlive();+ debugLogger.debug("Gemini Live setup complete", { model: this.currentModel });+ // A dictation shorter than the handshake leaves every frame in the buffer,+ // and sendAudio never runs again to drain it.+ this._flushColdStartBuffer();+ if (this.pendingResolve) {+ this.pendingResolve();+ this.pendingResolve = null;+ this.pendingReject = null;+ }+ }++ _rejectPending(error) {+ if (!this.pendingReject) return;+ this.pendingReject(error);+ this.pendingReject = null;+ this.pendingResolve = null;+ }++ _notifyConnectionLost(error) {+ if (this._connectionLossNotified) return;+ this._connectionLossNotified = true;+ if (this.onConnectionLost) {+ this.onConnectionLost(error);+ } else {+ this.onError?.(error);+ }+ }++ _resolveTurnEnd() {+ this._turnEnded = true;+ this._turnEndResolve?.();+ this._turnEndResolve = null;+ }++ // A warm socket sits idle between dictations with no other liveness check; a+ // network path that dies silently leaves isConnected stuck true and the next+ // recording gets sent into a dead socket.+ startKeepAlive() {+ this.stopKeepAlive();+ const socket = this.ws;+ if (!socket) return;++ socket.isAlive = true;+ socket.on("pong", () => {+ socket.isAlive = true;+ });++ this.keepAliveInterval = setInterval(() => {+ if (socket !== this.ws || socket.readyState !== WebSocket.OPEN) {+ this.stopKeepAlive();+ return;+ }+ if (socket.isAlive === false) {+ debugLogger.debug("Gemini Live keep-alive missed pong, terminating stale connection");+ socket.terminate();+ return;+ }+ socket.isAlive = false;+ try {+ socket.ping();+ } catch (err) {+ debugLogger.debug("Gemini Live keep-alive ping failed", { error: err.message });+ socket.terminate();+ }+ }, KEEPALIVE_INTERVAL_MS);+ }++ stopKeepAlive() {+ if (this.keepAliveInterval) {+ clearInterval(this.keepAliveInterval);+ this.keepAliveInterval = null;+ }+ }++ _sendAudioFrame(pcmBuffer) {+ this.ws.send(+ JSON.stringify({+ realtimeInput: {+ audio: { data: pcmBuffer.toString("base64"), mimeType: `audio/pcm;rate=${SAMPLE_RATE}` },+ },+ })+ );+ this.audioBytesSent += pcmBuffer.length;+ if (this._turnEnded && computePcm16Rms(pcmBuffer) >= TURN_REOPEN_RMS) this._turnEnded = false;+ }++ sendAudio(pcmBuffer) {+ // The capture worklet ends its stream with a "flushed" string sentinel;+ // only s16 PCM (even byte length) is audio, and only audioStreamEnd closes+ // a Gemini turn.+ if (!pcmBuffer || pcmBuffer.length % 2 !== 0) return false;++ if (this.ws?.readyState !== WebSocket.OPEN || !this.isConnected) {+ if (this.bufferingAudio && this.coldStartBufferSize < COLD_START_BUFFER_MAX) {+ const copy = Buffer.from(pcmBuffer);+ this.coldStartBuffer.push(copy);+ this.coldStartBufferSize += copy.length;+ }+ return false;+ }++ this._flushColdStartBuffer();+ this._sendAudioFrame(pcmBuffer);+ return true;+ }++ _flushColdStartBuffer() {+ if (this.coldStartBuffer.length === 0) return;+ debugLogger.debug("Gemini Live flushing cold-start buffer", {+ chunks: this.coldStartBuffer.length,+ bytes: this.coldStartBufferSize,+ });+ for (const buffered of this.coldStartBuffer) {+ this._sendAudioFrame(buffered);+ }+ this.coldStartBuffer = [];+ this.coldStartBufferSize = 0;+ }++ // Gemini has no mid-stream finalize: audioStreamEnd ends the turn, and the+ // server answers with the final transcript. Sending it from here (the stop+ // sequence calls finalize() before it waits for text) buys back the wait.+ finalize() {+ if (this.ws?.readyState !== WebSocket.OPEN || this._audioStreamEndSentAt) return false;+ this._audioStreamEndSentAt = Date.now();+ this.ws.send(JSON.stringify({ realtimeInput: { audioStreamEnd: true } }));+ debugLogger.debug("Gemini Live audioStreamEnd sent", { audioBytesSent: this.audioBytesSent });+ return true;+ }++ async disconnect(closeStream = true) {+ debugLogger.debug("Gemini Live disconnect", {+ audioBytesSent: this.audioBytesSent,+ segments: this.completedSegments.length,+ textLength: this.getFullTranscript().length,+ readyState: this.ws?.readyState,+ });++ if (!this.ws) return this._takeTranscript();++ this.isDisconnecting = true;++ if (closeStream && this.ws.readyState === WebSocket.OPEN && this.isConnected) {+ this._flushColdStartBuffer();+ }++ if (closeStream && this.ws.readyState === WebSocket.OPEN && this.audioBytesSent > 0) {+ this.finalize();+ if (!this._turnEnded) await this._awaitTurnEnd();+ }++ const result = this._takeTranscript();+ this.cleanup();+ this.isDisconnecting = false;+ return result;+ }++ // The stop sequence finalizes first and waits on the renderer side before it+ // gets here, so the budget runs from audioStreamEnd, not from this call.+ async _awaitTurnEnd() {+ const remaining = DISCONNECT_TIMEOUT_MS - (Date.now() - this._audioStreamEndSentAt);+ if (remaining <= 0) return;+ let timeoutId;+ await Promise.race([+ new Promise((resolve) => {+ this._turnEndResolve = resolve;+ }),+ new Promise((resolve) => {+ timeoutId = setTimeout(() => {+ debugLogger.debug("Gemini Live final transcript timeout, using accumulated text");+ resolve();+ }, remaining);+ }),+ ]);+ clearTimeout(timeoutId);+ this._turnEndResolve = null;+ }++ _takeTranscript() {+ const result = { text: this.getFullTranscript() };+ this.completedSegments = [];+ return result;+ }++ cleanup() {+ clearTimeout(this.connectionTimeout);+ this.connectionTimeout = null;+ this.stopKeepAlive();++ if (this.ws) {+ try {+ this.ws.close();+ } catch {}+ this.ws = null;+ }++ this.isConnected = false;+ this.isConnecting = false;+ this.bufferingAudio = false;+ }++ getStatus() {+ return { isConnected: this.isConnected, isConnecting: this.isConnecting };+ }+}++module.exports = { GeminiLiveStreaming, buildGeminiLiveUrl, GEMINI_LIVE_MODEL };diff --git a/src/helpers/ipcHandlers.js b/src/helpers/ipcHandlers.jsindex db760014f..4c12f4a83 100644--- a/src/helpers/ipcHandlers.js+++ b/src/helpers/ipcHandlers.js@@ -82,6 +82,7 @@ const HyprlandShortcutManager = require("./hyprlandShortcut");const AssemblyAiStreaming = require("./assemblyAiStreaming");const { i18nMain, changeLanguage } = require("./i18nMain");const DeepgramStreaming = require("./deepgramStreaming");+const { GeminiLiveStreaming, GEMINI_LIVE_MODEL } = require("./geminiLiveStreaming");const CortiStreaming = require("./cortiStreaming");const OpenAIRealtimeStreaming = require("./openaiRealtimeStreaming");const { getCortiToken } = require("./cortiAuth");@@ -630,6 +631,7 @@ class IPCHandlers {this._micHoldSenders = new Map();this.assemblyAiStreaming = null;this.deepgramStreaming = null;+ this.geminiStreaming = null;this.cortiStreaming = null;this._dictationStreaming = null;this._dictationConnectPromise = null;@@ -10648,6 +10650,155 @@ class IPCHandlers {return this.deepgramStreaming.getStatus();});+ let geminiStreamingStartInProgress = false;+ let geminiSendDropCount = 0;+ // One handshake at a time. A start that raced an in-flight warmup used to+ // clear the cold-start buffer, spend a second single-use managed token and+ // report success while the warmup's handshake (and its failure) were still+ // pending on a promise nobody read.+ let geminiConnectInFlight = null;++ // Re-bound on every warmup/start so a warm socket promoted by a different+ // window can never emit into the window that opened it.+ const ensureGeminiStreaming = (event) => {+ if (!this.geminiStreaming) {+ this.geminiStreaming = new GeminiLiveStreaming();+ }+ const win = BrowserWindow.fromWebContents(event.sender);+ const emit = (channel, payload) => {+ if (win && !win.isDestroyed()) win.webContents.send(channel, payload);+ };+ const streaming = this.geminiStreaming;+ streaming.onPartialTranscript = (text) => emit("gemini-partial-transcript", text);+ streaming.onFinalTranscript = (text) => emit("gemini-final-transcript", text);+ streaming.onError = (error) => emit("gemini-error", error.message);+ streaming.onSessionEnd = (data) => emit("gemini-session-end", data);+ return streaming;+ };++ const connectGeminiStreaming = (event, options) => {+ if (geminiConnectInFlight) return geminiConnectInFlight;+ geminiConnectInFlight = (async () => {+ const streaming = ensureGeminiStreaming(event);+ // byok resolves to the raw API key, managed to a single-use ephemeral+ // token; the client picks its Live method from `mode` accordingly.+ const tokenOptions = { mode: options.mode, provider: "gemini-realtime" };+ // Buffer before the token fetch (a real network round trip) so+ // gemini-streaming-send has somewhere to put the first frames.+ streaming.beginConnecting();+ const token = await fetchRealtimeToken(event, tokenOptions);+ await streaming.connect({+ ...options,+ token,+ refreshToken: () => fetchRealtimeToken(event, tokenOptions),+ });+ })().finally(() => {+ geminiConnectInFlight = null;+ });+ return geminiConnectInFlight;+ };++ ipcMain.handle("gemini-streaming-warmup", async (event, options = {}) => {+ try {+ if (this.geminiStreaming?.isConnected) {+ ensureGeminiStreaming(event);+ debugLogger.debug("Gemini Live connection already warm", {}, "streaming");+ return { success: true, alreadyWarm: true };+ }+ await connectGeminiStreaming(event, options);+ return { success: true };+ } catch (error) {+ debugLogger.error("Gemini streaming warmup error", { error: error.message });+ return toPolicyFailure(error);+ }+ });++ ipcMain.handle("gemini-streaming-start", async (event, options = {}) => {+ if (geminiStreamingStartInProgress) {+ debugLogger.debug("Gemini streaming start already in progress, ignoring", {}, "streaming");+ return { success: false, error: "Operation in progress" };+ }++ geminiStreamingStartInProgress = true;+ try {+ const streaming = ensureGeminiStreaming(event);+ if (geminiConnectInFlight) await geminiConnectInFlight;+ const usedWarmConnection = streaming.isConnected && !options.forceNew;+ if (!usedWarmConnection) {+ if (streaming.isConnected) await streaming.disconnect(false);+ await connectGeminiStreaming(event, options);+ }+ geminiSendDropCount = 0;+ debugLogger.debug("Gemini streaming started", { usedWarmConnection }, "streaming");+ return { success: true, usedWarmConnection };+ } catch (error) {+ debugLogger.error("Gemini streaming start error", { error: error.message });+ if (error.code === "AUTH_EXPIRED") {+ return { success: false, error: "Session expired", code: "AUTH_EXPIRED" };+ }+ return streamingStartFailure(error);+ } finally {+ geminiStreamingStartInProgress = false;+ }+ });++ ipcMain.on("gemini-streaming-send", (_event, audioBuffer) => {+ try {+ if (!this.geminiStreaming) return;+ const sent = this.geminiStreaming.sendAudio(Buffer.from(audioBuffer));+ if (!sent) {+ geminiSendDropCount++;+ if (geminiSendDropCount <= 3 || geminiSendDropCount % 50 === 0) {+ debugLogger.warn(+ "Gemini audio send dropped",+ {+ dropCount: geminiSendDropCount,+ isConnected: this.geminiStreaming.isConnected,+ wsReadyState: this.geminiStreaming.ws?.readyState,+ },+ "streaming"+ );+ }+ } else if (geminiSendDropCount > 0) {+ debugLogger.debug(+ "Gemini audio send resumed after drops",+ { previousDrops: geminiSendDropCount },+ "streaming"+ );+ geminiSendDropCount = 0;+ }+ } catch (error) {+ debugLogger.error("Gemini streaming send error", { error: error.message });+ }+ });++ ipcMain.on("gemini-streaming-finalize", () => {+ this.geminiStreaming?.finalize();+ });++ ipcMain.handle("gemini-streaming-stop", async () => {+ try {+ const model = this.geminiStreaming?.currentModel || GEMINI_LIVE_MODEL;+ const audioBytesSent = this.geminiStreaming?.audioBytesSent || 0;+ let result = { text: "" };+ if (this.geminiStreaming) {+ result = await this.geminiStreaming.disconnect(true);+ }++ return { success: true, text: result?.text || "", model, audioBytesSent };+ } catch (error) {+ debugLogger.error("Gemini streaming stop error", { error: error.message });+ return { success: false, error: error.message };+ }+ });++ ipcMain.handle("gemini-streaming-status", async () => {+ if (!this.geminiStreaming) {+ return { isConnected: false, isConnecting: false };+ }+ return this.geminiStreaming.getStatus();+ });+ipcMain.handle("corti-streaming-warmup", async (_event, options = {}) => {try {if (!this.cortiStreaming) {diff --git a/src/helpers/meetingTranscriptionRouting.js b/src/helpers/meetingTranscriptionRouting.jsindex 306127f4e..1b56bee47 100644--- a/src/helpers/meetingTranscriptionRouting.js+++ b/src/helpers/meetingTranscriptionRouting.js@@ -1,3 +1,22 @@+// Note recording only offers providers the main process will actually run:+// meeting prepare/start check ALLOWED_MEETING_PROVIDERS (derived from the+// streaming client table in meetingStreamingProviders.js) and reject anything+// else with no user-visible message. Without this intersection, every registry+// model marked `streaming: true` reaches the notes picker — including+// dictation-only ones like Gemini Live — and fails silently there. The closure+// is pinned by test/helpers/meetingStreamingProviders.test.js.+export const MEETING_STREAMING_PROVIDER_IDS = [+ "openai",+ "assemblyai",+ "deepgram",+ "corti",+ "tinfoil",+];++export function filterMeetingStreamingProviders(providers) {+ return providers.filter((provider) => MEETING_STREAMING_PROVIDER_IDS.includes(provider.id));+}+const DEFAULT_MANAGED_PROVIDER = {id: "openai",models: [{ id: "gpt-4o-mini-transcribe", default: true }],diff --git a/src/helpers/realtimeTokenProviders.js b/src/helpers/realtimeTokenProviders.jsindex 5951056cd..1bac8a58c 100644--- a/src/helpers/realtimeTokenProviders.js+++ b/src/helpers/realtimeTokenProviders.js@@ -56,6 +56,24 @@ const REALTIME_TOKEN_PROVIDERS = {});},+ "gemini-realtime": async ({ environmentManager, postServerToken }, options, streams) => {+ if (options.mode === "byok") {+ const apiKey = environmentManager.getGeminiKey();+ if (!apiKey) {+ throw new Error("No Gemini API key configured. Add your key in Settings.");+ }+ // The raw key opens the Live socket directly (BidiGenerateContent?key=)+ // and is not consumed by a handshake, so both streams can share it.+ return duplicate(streams, apiKey);+ }+ // Managed tokens are minted with uses:1, so N sockets need N mints.+ return dual(streams, async () => {+ const data = await postServerToken("/api/gemini-live-token");+ if (!data.token) throw new Error("No Gemini token received");+ return data.token;+ });+ },+"corti-realtime": async ({ mintCortiToken }, options, streams) => {// One token covers both meeting streams; it's only used at the WSS handshake.const { token } = await mintCortiToken(options);diff --git a/src/locales/de/translation.json b/src/locales/de/translation.jsonindex b3bfb4a12..8e0276c4f 100644--- a/src/locales/de/translation.json+++ b/src/locales/de/translation.json@@ -2625,6 +2625,7 @@"groq_whisper_large_v3_turbo": "216x Echtzeitgeschwindigkeit","mistral_voxtral_mini_latest": "Schnelle mehrsprachige Transkription","gemini_3_5_transcribe": "Googles Gemini-Spracherkennung",+ "gemini_3_5_transcribe_live": "Spracherkennung per Echtzeit-Streaming","corti_transcribe": "Medizinische Transkription in klinischer Qualität","xai_grok_stt": "Hochpräzise Spracherkennung von xAI","tinfoil_voxtral": "Nachweisbar private Transkription"diff --git a/src/locales/en/translation.json b/src/locales/en/translation.jsonindex 7f32134ef..0df61238d 100644--- a/src/locales/en/translation.json+++ b/src/locales/en/translation.json@@ -2625,6 +2625,7 @@"groq_whisper_large_v3_turbo": "216x real-time speed","mistral_voxtral_mini_latest": "Fast multilingual transcription","gemini_3_5_transcribe": "Google's Gemini speech-to-text",+ "gemini_3_5_transcribe_live": "Realtime streaming speech-to-text","corti_transcribe": "Clinical-grade medical transcription","xai_grok_stt": "High-accuracy speech recognition by xAI","tinfoil_voxtral": "Verifiably private transcription"diff --git a/src/locales/es/translation.json b/src/locales/es/translation.jsonindex 0d2a960d5..4bd80ff3c 100644--- a/src/locales/es/translation.json+++ b/src/locales/es/translation.json@@ -2580,6 +2580,7 @@"groq_whisper_large_v3_turbo": "Velocidad 216x en tiempo real","mistral_voxtral_mini_latest": "Transcripción multilingüe rápida","gemini_3_5_transcribe": "Voz a texto con Gemini de Google",+ "gemini_3_5_transcribe_live": "Voz a texto en streaming en tiempo real","corti_transcribe": "Transcripción médica de nivel clínico","xai_grok_stt": "Reconocimiento de voz de alta precisión por xAI","tinfoil_voxtral": "Transcripción verificablemente privada"diff --git a/src/locales/fr/translation.json b/src/locales/fr/translation.jsonindex 3f05df531..2d816a927 100644--- a/src/locales/fr/translation.json+++ b/src/locales/fr/translation.json@@ -2625,6 +2625,7 @@"groq_whisper_large_v3_turbo": "Vitesse 216x en temps réel","mistral_voxtral_mini_latest": "Transcription multilingue rapide","gemini_3_5_transcribe": "Reconnaissance vocale Gemini de Google",+ "gemini_3_5_transcribe_live": "Reconnaissance vocale en streaming temps réel","corti_transcribe": "Transcription médicale de qualité clinique","xai_grok_stt": "Reconnaissance vocale haute précision par xAI","tinfoil_voxtral": "Transcription vérifiablement privée"diff --git a/src/locales/it/translation.json b/src/locales/it/translation.jsonindex 504cadc6b..a4583499a 100644--- a/src/locales/it/translation.json+++ b/src/locales/it/translation.json@@ -2532,6 +2532,7 @@"groq_whisper_large_v3": "Modello Large v3, veloce e preciso","mistral_voxtral_mini_latest": "Trascrizione multilingue veloce","gemini_3_5_transcribe": "Riconoscimento vocale Gemini di Google",+ "gemini_3_5_transcribe_live": "Riconoscimento vocale in streaming in tempo reale","corti_transcribe": "Trascrizione medica di livello clinico","xai_grok_stt": "Riconoscimento vocale ad alta precisione di xAI","tinfoil_voxtral": "Trascrizione verificabilmente privata"diff --git a/src/locales/ja/translation.json b/src/locales/ja/translation.jsonindex f1d18bbe2..68ed615dc 100644--- a/src/locales/ja/translation.json+++ b/src/locales/ja/translation.json@@ -2532,6 +2532,7 @@"groq_whisper_large_v3": "Large v3 モデル、高速かつ高精度","mistral_voxtral_mini_latest": "高速多言語文字起こし","gemini_3_5_transcribe": "GoogleのGemini音声文字起こし",+ "gemini_3_5_transcribe_live": "リアルタイムストリーミング文字起こし","corti_transcribe": "臨床グレードの医療文字起こし","xai_grok_stt": "xAIによる高精度音声認識","tinfoil_voxtral": "検証可能なプライベート文字起こし"diff --git a/src/locales/pt/translation.json b/src/locales/pt/translation.jsonindex 62feb6b0b..4ad5644ff 100644--- a/src/locales/pt/translation.json+++ b/src/locales/pt/translation.json@@ -2511,6 +2511,7 @@"groq_whisper_large_v3": "Modelo Large v3, rápido e preciso","mistral_voxtral_mini_latest": "Transcrição multilíngue rápida","gemini_3_5_transcribe": "Fala para texto Gemini do Google",+ "gemini_3_5_transcribe_live": "Fala para texto em streaming em tempo real","corti_transcribe": "Transcrição médica de nível clínico","xai_grok_stt": "Reconhecimento de voz de alta precisão pela xAI","tinfoil_voxtral": "Transcrição verificavelmente privada"diff --git a/src/locales/ru/translation.json b/src/locales/ru/translation.jsonindex a02ef1370..46bfe4e75 100644--- a/src/locales/ru/translation.json+++ b/src/locales/ru/translation.json@@ -2548,6 +2548,7 @@"groq_whisper_large_v3": "Модель Large v3, быстрая и точная","mistral_voxtral_mini_latest": "Быстрая многоязычная транскрипция","gemini_3_5_transcribe": "Распознавание речи Gemini от Google",+ "gemini_3_5_transcribe_live": "Распознавание речи в реальном времени","corti_transcribe": "Медицинская транскрипция клинического уровня","xai_grok_stt": "Высокоточное распознавание речи от xAI","tinfoil_voxtral": "Проверяемо приватная транскрипция"diff --git a/src/locales/zh-CN/translation.json b/src/locales/zh-CN/translation.jsonindex 90d1ed054..8b0c9dc03 100644--- a/src/locales/zh-CN/translation.json+++ b/src/locales/zh-CN/translation.json@@ -2532,6 +2532,7 @@"groq_whisper_large_v3": "Large v3 模型,快速且精准","mistral_voxtral_mini_latest": "快速多语言转录","gemini_3_5_transcribe": "Google Gemini 语音转文字",+ "gemini_3_5_transcribe_live": "实时流式语音转文字","corti_transcribe": "临床级医疗转录","xai_grok_stt": "xAI 高精度语音识别","tinfoil_voxtral": "可验证的私密转录"diff --git a/src/locales/zh-TW/translation.json b/src/locales/zh-TW/translation.jsonindex 0b79050b6..056778382 100644--- a/src/locales/zh-TW/translation.json+++ b/src/locales/zh-TW/translation.json@@ -2532,6 +2532,7 @@"groq_whisper_large_v3": "Large v3 模型,快速且精準","mistral_voxtral_mini_latest": "快速多語言轉錄","gemini_3_5_transcribe": "Google Gemini 語音轉文字",+ "gemini_3_5_transcribe_live": "即時串流語音轉文字","corti_transcribe": "臨床級醫療轉錄","xai_grok_stt": "xAI 高精度語音辨識","tinfoil_voxtral": "可驗證的私密轉錄"diff --git a/src/models/ModelRegistry.ts b/src/models/ModelRegistry.tsindex d9168a27c..876939898 100644--- a/src/models/ModelRegistry.ts+++ b/src/models/ModelRegistry.ts@@ -1,6 +1,7 @@import modelDataRaw from "./modelRegistryData.json";import { isCloudCleanupMode, getSettings } from "../stores/settingsStore";import { readCachedTinfoilModels } from "./tinfoilModelCache";+import { filterMeetingStreamingProviders } from "../helpers/meetingTranscriptionRouting";import type { InferenceMode } from "../types/electron";export interface ModelDefinition {@@ -432,6 +433,12 @@ export function getStreamingTranscriptionProviders(): TranscriptionProviderData[.filter((p) => p.models.length > 0);}+// Streaming providers note recording can actually run (see+// meetingTranscriptionRouting.MEETING_STREAMING_PROVIDER_IDS).+export function getMeetingStreamingTranscriptionProviders(): TranscriptionProviderData[] {+ return filterMeetingStreamingProviders(getStreamingTranscriptionProviders());+}+export function getTranscriptionProvider(providerId: string): TranscriptionProviderData | undefined {diff --git a/src/models/modelRegistryData.json b/src/models/modelRegistryData.jsonindex fa0c4824f..1029caa62 100644--- a/src/models/modelRegistryData.json+++ b/src/models/modelRegistryData.json@@ -277,12 +277,20 @@"id": "gemini","name": "Gemini","baseUrl": "https://generativelanguage.googleapis.com/v1beta",+ "batchModel": "gemini-3.5-transcribe","models": [{"id": "gemini-3.5-transcribe","name": "Gemini 3.5 Transcribe","description": "Google's Gemini speech-to-text","descriptionKey": "models.descriptions.transcription.gemini_3_5_transcribe"+ },+ {+ "id": "gemini-3.5-transcribe-live",+ "name": "Gemini 3.5 Transcribe Live",+ "description": "Realtime streaming speech-to-text",+ "descriptionKey": "models.descriptions.transcription.gemini_3_5_transcribe_live",+ "streaming": true}]},diff --git a/src/stores/meetingRecordingStore.ts b/src/stores/meetingRecordingStore.tsindex 01feda941..5984d1553 100644--- a/src/stores/meetingRecordingStore.ts+++ b/src/stores/meetingRecordingStore.ts@@ -1,7 +1,7 @@import { create } from "zustand";import { getSettings, selectResolvedMeetingTranscription } from "./settingsStore";import { useStreamingProvidersStore } from "./streamingProvidersStore";-import { getStreamingTranscriptionProviders } from "../models/ModelRegistry";+import { getMeetingStreamingTranscriptionProviders } from "../models/ModelRegistry";import { resolveMeetingTranscriptionOptions } from "../helpers/meetingTranscriptionRouting";import { followsSystemDefaultMic } from "../helpers/micSelectionRecovery";import { resolvePreferredMicrophone } from "../helpers/microphoneSelection";@@ -163,7 +163,7 @@ const getMeetingTranscriptionOptions = () => {cohereModel: resolved.cohereModel,selectedProvider: resolved.cloudTranscriptionProvider,selectedModel: resolved.cloudTranscriptionModel,- byokProviders: getStreamingTranscriptionProviders(),+ byokProviders: getMeetingStreamingTranscriptionProviders(),managedProviders: useStreamingProvidersStore.getState().providers,cortiEnvironment: state.cortiEnvironment,cortiTenant: state.cortiTenant,diff --git a/src/stores/settingsStore.ts b/src/stores/settingsStore.tsindex 593ebba04..d5d11bf88 100644--- a/src/stores/settingsStore.ts+++ b/src/stores/settingsStore.ts@@ -35,6 +35,7 @@ import { pickDefaultModelId } from "../models/providerDefaultModel";// the switch store only zustand, so neither reopens the ModelRegistry cycle.import { readCachedTinfoilModels } from "../models/tinfoilModelCache";import { recordTinfoilModelSwitch } from "./tinfoilModelSwitchStore";+import { MEETING_STREAMING_PROVIDER_IDS } from "../helpers/meetingTranscriptionRouting";import {getTranscriptionSelection,isScreenContextAllowed,@@ -92,7 +93,11 @@ const MEETING_TRANSCRIPTION_POLICY_CATALOG = {// Self-hosted realtime is not implemented for Note Recording.modes: ["openwhispr", "providers", "local"] as const,byokProviders: modelRegistryData.transcriptionProviders- .filter((provider) => provider.models.some((model) => model.streaming))+ .filter(+ (provider) =>+ MEETING_STREAMING_PROVIDER_IDS.includes(provider.id) &&+ provider.models.some((model) => model.streaming)+ ).map((provider) => provider.id),};diff --git a/src/types/electron.ts b/src/types/electron.tsindex 149113518..9db8e295a 100644--- a/src/types/electron.ts+++ b/src/types/electron.ts@@ -38,7 +38,8 @@ export interface NoteRecordingProvider {models: NoteRecordingProviderModel[];}-// Session options for the shared dictation-realtime-* channels. `provider` is+// Session options every dictation streaming channel takes — the shared+// dictation-realtime-* set and the per-provider ones. `provider` is// what fetchRealtimeToken's allowlist keys on — the renderer must always send// it (built by dictationStreamingRouting.buildStreamingSessionOptions); the// main process defaults a missing value to "openai-realtime" for pre-1.8.4@@ -2706,6 +2707,32 @@ declare global {callback: (data: { audioDuration?: number; text?: string }) => void) => () => void;+ // Gemini Live Streaming+ geminiStreamingWarmup?: (+ options?: DictationRealtimeSessionOptions+ ) => Promise<+ { success: boolean; alreadyWarm?: boolean; error?: string } & PolicyFailureMetadata+ >;+ geminiStreamingStart?: (+ options?: DictationRealtimeSessionOptions & { forceNew?: boolean }+ ) => Promise<+ { success: boolean; usedWarmConnection?: boolean; error?: string } & PolicyFailureMetadata+ >;+ geminiStreamingSend?: (audioBuffer: ArrayBuffer) => void;+ geminiStreamingFinalize?: () => void;+ geminiStreamingStop?: () => Promise<{+ success: boolean;+ text?: string;+ model?: string;+ audioBytesSent?: number;+ error?: string;+ }>;+ geminiStreamingStatus?: () => Promise<{ isConnected: boolean; isConnecting: boolean }>;+ onGeminiPartialTranscript?: (callback: (text: string) => void) => () => void;+ onGeminiFinalTranscript?: (callback: (text: string) => void) => () => void;+ onGeminiError?: (callback: (error: string) => void) => () => void;+ onGeminiSessionEnd?: (callback: (data: { text?: string }) => void) => () => void;+// Corti streaming (BYOK)cortiStreamingWarmup?: (options?: {environment?: string;diff --git a/test/helpers/audioManagerStreamingRouting.test.js b/test/helpers/audioManagerStreamingRouting.test.jsindex ce7641dff..983cf0a62 100644--- a/test/helpers/audioManagerStreamingRouting.test.js+++ b/test/helpers/audioManagerStreamingRouting.test.js@@ -59,6 +59,67 @@ test("OpenAI dictation realtime requests identify the token provider", async (t)]);});+test("Gemini's live model routes onto the gemini-streaming-* channels", async (t) => {+ const manager = await loadManager(t);+ setSettings({+ cloudTranscriptionProvider: "gemini",+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",+ geminiApiKey: "gm-test",+ });+ const calls = [];+ globalThis.window.electronAPI.geminiStreamingWarmup = async (options) => {+ calls.push(["warmup", options]);+ return { success: true };+ };+ globalThis.window.electronAPI.geminiStreamingStart = async (options) => {+ calls.push(["start", options]);+ return { success: true };+ };++ assert.equal(manager.getStreamingProviderName(), "gemini");+ const provider = manager.getStreamingProvider();+ assert.equal(provider.awaitsFinalTranscript, true);+ // Pinned to geminiLiveStreaming.js's DISCONNECT_TIMEOUT_MS: both sides+ // measure the same 3s from audioStreamEnd.+ assert.equal(provider.finalCeilingMs, 3000);++ const options = { provider: "gemini", model: "gemini-3.5-transcribe-live", mode: "byok" };+ await provider.warmup(options);+ await provider.start(options);++ assert.deepEqual(calls, [+ ["warmup", options],+ ["start", options],+ ]);+});++test("Gemini streaming needs a streaming model, plus a key (byok) or an account (managed)", async (t) => {+ const manager = await loadManager(t);+ manager.sttConfig = { dictation: { mode: "streaming" } };+ const gemini = (overrides) =>+ setSettings({+ cloudTranscriptionProvider: "gemini",+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",+ ...overrides,+ });++ gemini({ geminiApiKey: "gm-test" });+ assert.equal(manager.shouldUseStreaming(), true);++ gemini({ geminiApiKey: "" });+ assert.equal(manager.shouldUseStreaming(), false);++ gemini({ cloudTranscriptionMode: "openwhispr", isSignedIn: true });+ assert.equal(manager.shouldUseStreaming(), true);++ gemini({ cloudTranscriptionMode: "openwhispr", isSignedIn: false });+ assert.equal(manager.shouldUseStreaming(), false);++ // The batch Gemini model on the same provider stays on HTTP.+ gemini({ cloudTranscriptionModel: "gemini-3.5-transcribe", geminiApiKey: "gm-test" });+ assert.equal(manager.shouldUseStreaming(), false);+});+test("managed OpenWhispr Cloud still respects its batch configuration", async (t) => {const manager = await loadManager(t);manager.sttConfig = { dictation: { mode: "batch" } };diff --git a/test/helpers/audioManagerStreamingSettle.test.js b/test/helpers/audioManagerStreamingSettle.test.jsindex 09746e6a6..cf3c58ec4 100644--- a/test/helpers/audioManagerStreamingSettle.test.js+++ b/test/helpers/audioManagerStreamingSettle.test.js@@ -72,6 +72,17 @@ test("the ceiling bounds a final that never lands", async (t) => {assert.ok(ms < CEILING_MS + 300, `overran the ceiling: ${ms}ms`);});+test("a provider can widen the ceiling for a slower final", async (t) => {+ const manager = await loadManager(t);+ const providerCeilingMs = 600;+ manager.streamingPartialText = "never finalized";++ const ms = await elapsed(() => manager.awaitStreamingTextSettled(providerCeilingMs));++ assert.ok(ms >= providerCeilingMs - TIMER_SLACK_MS, `ignored the provider ceiling: ${ms}ms`);+ assert.ok(ms < CEILING_MS, `fell back to the default ceiling: ${ms}ms`);+});+test("clears its timers so a settled wait leaves no handles behind", async (t) => {const manager = await loadManager(t);diff --git a/test/helpers/dictationStreamingMatrix.test.js b/test/helpers/dictationStreamingMatrix.test.jsindex f74955753..9eb576941 100644--- a/test/helpers/dictationStreamingMatrix.test.js+++ b/test/helpers/dictationStreamingMatrix.test.js@@ -20,6 +20,7 @@ const RENDERER_STREAMING_PROVIDERS = ["deepgram","assemblyai","openai-realtime",+ "gemini","corti","tinfoil-realtime",];@@ -66,6 +67,26 @@ const RESOLUTION_MATRIX = [{ settings: settingsWith({ cloudTranscriptionProvider: "corti" }) },"openai-realtime",],+ [+ "gemini's live model routes to its own channels",+ {+ settings: settingsWith({+ cloudTranscriptionProvider: "gemini",+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",+ }),+ },+ "gemini",+ ],+ [+ "gemini's batch model keeps the dictation default (HTTP, not streaming)",+ {+ settings: settingsWith({+ cloudTranscriptionProvider: "gemini",+ cloudTranscriptionModel: "gemini-3.5-transcribe",+ }),+ },+ "openai-realtime",+ ],["batch model in notes context defaults to deepgram",{ settings: settingsWith({ cloudTranscriptionModel: "whisper-1" }), context: "notes" },diff --git a/test/helpers/geminiLiveStreaming.test.js b/test/helpers/geminiLiveStreaming.test.jsnew file mode 100644index 000000000..a790ec5f8--- /dev/null+++ b/test/helpers/geminiLiveStreaming.test.js@@ -0,0 +1,530 @@+const test = require("node:test");+const assert = require("node:assert/strict");+const { WebSocketServer } = require("ws");++const {+ GeminiLiveStreaming,+ buildGeminiLiveUrl,+} = require("../../src/helpers/geminiLiveStreaming");++const FRAME = Buffer.alloc(1600); // one 50ms worklet frame at 16kHz s16le (silence)+const LOUD_FRAME = Buffer.alloc(1600, 0x20); // same frame at speech level (~0.25 RMS)++// Loopback Live server. `script(socket, message)` reacts to each client+// message; the default answers `setup` with setupComplete, like the real one.+async function withServer(run, script) {+ const server = new WebSocketServer({ host: "127.0.0.1", port: 0 });+ await new Promise((resolve) => server.once("listening", resolve));+ const received = [];+ const urls = [];+ const reply = script || ((socket) => socket.send(JSON.stringify({ setupComplete: {} })));++ server.on("connection", (socket, request) => {+ urls.push(request.url);+ socket.on("message", (data) => {+ const message = JSON.parse(data.toString());+ received.push(message);+ reply(socket, message, urls.length);+ });+ });++ const streaming = new GeminiLiveStreaming();+ streaming.buildWebSocketUrl = ({ token }) =>+ `ws://127.0.0.1:${server.address().port}/?t=${token}`;++ try {+ await run({ streaming, received, urls });+ } finally {+ streaming.cleanup();+ await new Promise((resolve) => server.close(resolve));+ }+}++const audioFrames = (received) => received.filter((message) => message.realtimeInput?.audio);++test("url: the two Live methods are auth-disjoint and pinned per mode", () => {+ assert.equal(+ buildGeminiLiveUrl({ mode: "byok", token: "AI-KEY" }),+ "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=AI-KEY"+ );+ assert.equal(+ buildGeminiLiveUrl({ mode: "openwhispr", token: "auth_tokens/abc" }),+ "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=auth_tokens/abc"+ );+});++test("setup: the custom dictionary always ships; a language only when chosen", () => {+ const streaming = new GeminiLiveStreaming();++ assert.deepEqual(streaming.buildSetupMessage({ keyterms: ["OpenWhispr", " ", ""] }), {+ setup: {+ model: "models/gemini-3.5-transcribe-live",+ generationConfig: { responseModalities: ["TEXT"] },+ inputAudioTranscription: { customVocabulary: ["OpenWhispr"] },+ },+ });++ assert.deepEqual(+ streaming.buildSetupMessage({ model: "gemini-next", language: "en", keyterms: [] }),+ {+ setup: {+ model: "models/gemini-next",+ generationConfig: { responseModalities: ["TEXT"] },+ inputAudioTranscription: { languageCodes: ["en"] },+ },+ }+ );++ const many = streaming.buildSetupMessage({+ keyterms: Array.from({ length: 150 }, (_, i) => `term-${i}`),+ });+ assert.equal(many.setup.inputAudioTranscription.customVocabulary.length, 100);+});++test("connect resolves on setupComplete, not on socket open", async () => {+ await withServer(+ async ({ streaming, received }) => {+ streaming.beginConnecting();+ // Frames from the token-fetch / handshake window are buffered, not sent.+ assert.equal(streaming.sendAudio(FRAME), false);++ const connected = streaming.connect({ token: "t", mode: "byok", keyterms: ["OpenWhispr"] });+ await new Promise((resolve) => setTimeout(resolve, 30));+ assert.equal(streaming.isConnected, false, "open alone must not mark the session ready");+ assert.equal(streaming.sendAudio(FRAME), false);++ await connected;+ assert.equal(streaming.isConnected, true);+ assert.deepEqual(received[0].setup.inputAudioTranscription, {+ customVocabulary: ["OpenWhispr"],+ });+ },+ (socket, message) => {+ if (!message.setup) return;+ setTimeout(() => socket.send(JSON.stringify({ setupComplete: {} })), 60);+ }+ );+});++test("connect rejects when the socket drops before setupComplete", async () => {+ await withServer(+ async ({ streaming }) => {+ await assert.rejects(+ streaming.connect({ token: "t", mode: "byok" }),+ /closed before ready \(code: 1006\)/+ );+ assert.equal(streaming.isConnected, false);+ },+ (socket) => socket.terminate()+ );+});++test("audio: buffered frames flush in order, then stream as base64 PCM frames", async () => {+ await withServer(async ({ streaming, received }) => {+ streaming.beginConnecting();+ const first = Buffer.alloc(1600, 1);+ const second = Buffer.alloc(1600, 2);+ streaming.sendAudio(first);+ streaming.sendAudio(second);++ await streaming.connect({ token: "t", mode: "byok" });+ const live = Buffer.alloc(1600, 3);+ assert.equal(streaming.sendAudio(live), true);+ await new Promise((resolve) => setTimeout(resolve, 30));++ const frames = audioFrames(received);+ assert.deepEqual(+ frames.map((frame) => frame.realtimeInput.audio.mimeType),+ ["audio/pcm;rate=16000", "audio/pcm;rate=16000", "audio/pcm;rate=16000"]+ );+ assert.deepEqual(+ frames.map((frame) => Buffer.from(frame.realtimeInput.audio.data, "base64")[0]),+ [1, 2, 3]+ );+ assert.equal(streaming.audioBytesSent, 4800);+ });+});++test("audio: a dictation shorter than the handshake still reaches the wire", async () => {+ await withServer(async ({ streaming, received }) => {+ // Every frame arrives before setupComplete and no frame follows it, so the+ // only drain opportunity is readiness itself.+ streaming.beginConnecting();+ streaming.sendAudio(Buffer.alloc(1600, 7));+ streaming.sendAudio(Buffer.alloc(1600, 8));++ await streaming.connect({ token: "t", mode: "byok" });+ await new Promise((resolve) => setTimeout(resolve, 30));++ assert.deepEqual(+ audioFrames(received).map(+ (frame) => Buffer.from(frame.realtimeInput.audio.data, "base64")[0]+ ),+ [7, 8]+ );+ assert.equal(streaming.audioBytesSent, 3200);++ await streaming.disconnect();+ assert.equal(+ received.filter((message) => message.realtimeInput?.audioStreamEnd).length,+ 1,+ "the turn must still be closed"+ );+ });+});++test("setup: a non-Gemini model id never reaches the wire", async () => {+ const streaming = new GeminiLiveStreaming();+ // Managed dictation carries the BYOK default, which is an OpenAI id.+ assert.equal(+ streaming.buildSetupMessage({ model: "gpt-4o-mini-transcribe" }).setup.model,+ "models/gemini-3.5-transcribe-live"+ );+ assert.equal(+ streaming.buildSetupMessage({ model: "gemini-3.5-transcribe-live" }).setup.model,+ "models/gemini-3.5-transcribe-live"+ );+});++test("audio: cold-start buffering is bounded at three seconds of PCM", async () => {+ const streaming = new GeminiLiveStreaming();+ streaming.beginConnecting();+ for (let i = 0; i < 100; i++) streaming.sendAudio(Buffer.alloc(1600));+ assert.equal(streaming.coldStartBufferSize, 96000);+});++test("characterization: the worklet's flush sentinel is never sent as audio", async () => {+ await withServer(async ({ streaming, received }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ assert.equal(streaming.sendAudio(Buffer.from("flushed")), false);+ await new Promise((resolve) => setTimeout(resolve, 30));+ assert.equal(audioFrames(received).length, 0);+ assert.equal(streaming.audioBytesSent, 0);+ });+});++test("transcripts: partials replace, finals accumulate", async () => {+ await withServer(+ async ({ streaming }) => {+ const partials = [];+ const finals = [];+ streaming.onPartialTranscript = (text) => partials.push(text);+ streaming.onFinalTranscript = (text) => finals.push(text);++ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);+ await new Promise((resolve) => setTimeout(resolve, 60));++ // Partials are revised, not appended to — casing included.+ assert.deepEqual(partials, ["The quick brown", "the quick brown fox"]);+ assert.deepEqual(finals, [+ "The quick brown fox.",+ "The quick brown fox. OpenWhispr transcription test.",+ ]);+ assert.equal(+ streaming.getFullTranscript(),+ "The quick brown fox. OpenWhispr transcription test."+ );+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (!message.realtimeInput?.audio) return;+ for (const text of ["The quick brown", "the quick brown fox"]) {+ socket.send(JSON.stringify({ serverContent: { interimInputTranscription: { text } } }));+ }+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "The quick brown fox." } } })+ );+ socket.send(+ JSON.stringify({+ serverContent: { inputTranscription: { text: "OpenWhispr transcription test." } },+ })+ );+ }+ );+});++test("disconnect sends audioStreamEnd once and waits for the end of the turn", async () => {+ await withServer(+ async ({ streaming, received }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);++ // The stop sequence calls finalize(), waits for the text to settle, then+ // stops — so disconnect must not send audioStreamEnd a second time.+ assert.equal(streaming.finalize(), true);+ assert.equal(streaming.finalize(), false, "audioStreamEnd must not be sent twice");+ await new Promise((resolve) => setTimeout(resolve, 40));++ const result = await streaming.disconnect(true);+ assert.equal(result.text, "The quick brown fox.");+ assert.equal(received.filter((m) => m.realtimeInput?.audioStreamEnd).length, 1);+ // The transcript is handed over, not kept for the next session.+ assert.equal(streaming.getFullTranscript(), "");+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (!message.realtimeInput?.audioStreamEnd) return;+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "The quick brown fox." } } })+ );+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));+ }+ );+});++test("disconnect still waits for the final when finalize() already closed the turn", async () => {+ await withServer(+ async ({ streaming, received }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);++ // The stop sequence finalizes, then stops as soon as its own wait gives+ // up — disconnect must not treat the sent audioStreamEnd as "answered".+ assert.equal(streaming.finalize(), true);+ const result = await streaming.disconnect(true);++ assert.equal(result.text, "the slow final");+ assert.equal(received.filter((m) => m.realtimeInput?.audioStreamEnd).length, 1);+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (!message.realtimeInput?.audioStreamEnd) return;+ setTimeout(() => {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "the slow final" } } })+ );+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));+ }, 120);+ }+ );+});++test("disconnect spends only what is left of the budget since audioStreamEnd", async () => {+ await withServer(+ async ({ streaming }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);+ await new Promise((resolve) => setTimeout(resolve, 30));++ assert.equal(streaming.finalize(), true);+ // The renderer already waited out the whole budget on its side.+ streaming._audioStreamEndSentAt -= 10000;+ const start = Date.now();+ const result = await streaming.disconnect(true);++ assert.equal(result.text, "already here");+ assert.ok(Date.now() - start < 500, "must not re-spend the budget");+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (message.realtimeInput?.audio) {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "already here" } } })+ );+ }+ }+ );+});++test("a turn the server closed before the hotkey release answers the stop immediately", async () => {+ await withServer(+ async ({ streaming, received }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);+ await new Promise((resolve) => setTimeout(resolve, 30));+ // Trailing silence let the server finalize on its own; nothing more will come.+ assert.equal(streaming._turnEnded, true);+ // Room-noise frames keep arriving until the hotkey is released; they are not speech.+ streaming.sendAudio(FRAME);+ assert.equal(streaming._turnEnded, true);++ const startedAt = Date.now();+ assert.deepEqual(await streaming.disconnect(true), { text: "already final" });+ assert.ok(Date.now() - startedAt < 500, "must not wait out the budget for a second turn end");+ await new Promise((resolve) => setTimeout(resolve, 30));+ assert.equal(+ received.filter((message) => message.realtimeInput?.audioStreamEnd).length,+ 1,+ "the stream is still closed cleanly"+ );+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (message.realtimeInput?.audio) {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "already final" } } })+ );+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));+ }+ }+ );+});++test("speech after a closed turn reopens it, so the stop waits for the last turn", async () => {+ await withServer(+ async ({ streaming }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(Buffer.alloc(1600, 1));+ await new Promise((resolve) => setTimeout(resolve, 30));+ assert.equal(streaming._turnEnded, true);++ // The user keeps talking: a speech-level frame reopens the turn and the stop must wait.+ streaming.sendAudio(LOUD_FRAME);+ assert.equal(streaming._turnEnded, false);++ assert.deepEqual(await streaming.disconnect(true), { text: "first segment last words" });+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ const audio = message.realtimeInput?.audio;+ if (audio && Buffer.from(audio.data, "base64")[0] !== LOUD_FRAME[0]) {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "first segment" } } })+ );+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));+ }+ if (message.realtimeInput?.audioStreamEnd) {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "last words" } } })+ );+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));+ }+ }+ );+});++test("disconnect returns the accumulated text when the turn never ends cleanly", async () => {+ await withServer(+ async ({ streaming }) => {+ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);+ await new Promise((resolve) => setTimeout(resolve, 30));++ // No generationComplete: the socket dies instead, which must release the+ // finalization wait rather than burn its whole budget.+ assert.deepEqual(await streaming.disconnect(true), { text: "partial turn" });+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (message.realtimeInput?.audio) {+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "partial turn" } } })+ );+ }+ if (message.realtimeInput?.audioStreamEnd) socket.close(1000, "");+ }+ );+});++// 1007 is a malformed credential, 1008 a revoked or deleted one. The reason is+// truncated by the server and is never part of the user-facing message.+for (const [code, reason] of [+ [1007, "API key not valid. Please pass a valid API key."],+ [1008, "Requested entity was not found."],+]) {+ test(`errors: a ${code} close frame becomes the coded, fixable key error`, async () => {+ await withServer(+ async ({ streaming }) => {+ await assert.rejects(+ streaming.connect({ token: "nope", mode: "byok" }),+ (err) =>+ err.code === "INVALID_KEY" &&+ /Check your key in Settings/.test(err.message) &&+ !err.message.includes(reason)+ );+ },+ (socket) => socket.close(code, reason)+ );+ });+}++test("errors: a spent managed token is re-minted once and the session survives", async () => {+ await withServer(+ async ({ streaming, urls }) => {+ await streaming.connect({+ token: "auth_tokens/spent",+ mode: "openwhispr",+ refreshToken: async () => "auth_tokens/fresh",+ });+ assert.equal(streaming.isConnected, true);+ assert.deepEqual(urls, ["/?t=auth_tokens/spent", "/?t=auth_tokens/fresh"]);+ },+ (socket, _message, connectionCount) => {+ if (connectionCount === 1) {+ socket.close(1011, "Token has been used too many times");+ return;+ }+ socket.send(JSON.stringify({ setupComplete: {} }));+ }+ );+});++test("errors: byok never retries a 1011 close (there is no token to re-mint)", async () => {+ await withServer(+ async ({ streaming, urls }) => {+ await assert.rejects(+ streaming.connect({+ token: "key",+ mode: "byok",+ refreshToken: async () => "unused",+ }),+ (err) => err.code === "AUTH_EXPIRED"+ );+ assert.equal(urls.length, 1);+ },+ (socket) => socket.close(1011, "Token has expired")+ );+});++test("an unexpected close hands the transcript over and reports the loss once", async () => {+ await withServer(+ async ({ streaming }) => {+ const errors = [];+ const sessionEnds = [];+ streaming.onError = (err) => errors.push(err.message);+ streaming.onSessionEnd = (data) => sessionEnds.push(data);++ await streaming.connect({ token: "t", mode: "byok" });+ streaming.sendAudio(FRAME);+ await new Promise((resolve) => setTimeout(resolve, 60));++ assert.deepEqual(sessionEnds, [{ text: "half a sentence" }]);+ // Distinct from the pre-ready failure, and without the raw server reason.+ assert.deepEqual(errors, ["Connection lost (code: 1011)"]);+ assert.equal(streaming.isConnected, false);+ },+ (socket, message) => {+ if (message.setup) {+ socket.send(JSON.stringify({ setupComplete: {} }));+ return;+ }+ if (!message.realtimeInput?.audio) return;+ socket.send(+ JSON.stringify({ serverContent: { inputTranscription: { text: "half a sentence" } } })+ );+ setTimeout(() => socket.close(1011, "Token has expired"), 10);+ }+ );+});diff --git a/test/helpers/meetingStreamingProviders.test.js b/test/helpers/meetingStreamingProviders.test.jsindex 1734905c9..0c82945ab 100644--- a/test/helpers/meetingStreamingProviders.test.js+++ b/test/helpers/meetingStreamingProviders.test.js@@ -34,6 +34,34 @@ test("allow-list accepts tinfoil-realtime and local", async () => {assert.equal(ALLOWED_MEETING_PROVIDERS.has("local"), true);});+test("every provider note recording offers is one the main process accepts", async () => {+ const { ALLOWED_MEETING_PROVIDERS } = await load();+ const { MEETING_STREAMING_PROVIDER_IDS } =+ await import("../../src/helpers/meetingTranscriptionRouting.js");++ // Note recording derives its streaming provider id by concatenation+ // (`${provider.id}-realtime`), so an offered catalog id with no client class+ // is rejected at meeting-transcription-prepare with no user-visible message.+ for (const id of MEETING_STREAMING_PROVIDER_IDS) {+ assert.equal(+ ALLOWED_MEETING_PROVIDERS.has(`${id}-realtime`),+ true,+ `${id} is offered for note recording but ${id}-realtime has no streaming client`+ );+ }+});++test("a dictation-only streaming provider is not offered for note recording", async () => {+ const { getStreamingTranscriptionProviders, getMeetingStreamingTranscriptionProviders } =+ await import("../../src/models/ModelRegistry.ts");++ const streamingIds = getStreamingTranscriptionProviders().map((provider) => provider.id);+ const meetingIds = getMeetingStreamingTranscriptionProviders().map((provider) => provider.id);++ assert.ok(streamingIds.includes("gemini"), "gemini ships a streaming dictation model");+ assert.equal(meetingIds.includes("gemini"), false);+});+test("allow-list rejects unknown and batch-only providers", async () => {const { ALLOWED_MEETING_PROVIDERS } = await load();diff --git a/test/helpers/realtimeTokenProviders.test.js b/test/helpers/realtimeTokenProviders.test.jsindex 86bd9beee..55ece63ae 100644--- a/test/helpers/realtimeTokenProviders.test.js+++ b/test/helpers/realtimeTokenProviders.test.js@@ -15,6 +15,7 @@ const deps = (overrides = {}) => ({getTinfoilKey: () => "tk-tinfoil",getDeepgramKey: () => "dg-key",getAssemblyAIKey: () => "aai-key",+ getGeminiKey: () => "gm-key",...overrides.environmentManager,},proxyFetch: overrides.proxyFetch || (async () => jsonResponse(200, { token: "aai-token" })),@@ -135,6 +136,40 @@ test("deepgram byok duplicates the key; cloud mints per stream", async () => {);});+test("gemini byok duplicates the raw key; cloud mints one token per stream", async () => {+ const { fetchRealtimeTokenForProvider } = await load();+ // The raw key opens the Live socket itself and survives any number of+ // handshakes, so both streams share it.+ assert.deepEqual(+ await fetchRealtimeTokenForProvider(+ "gemini-realtime",+ deps(),+ { mode: "byok" },+ { streams: 2 }+ ),+ ["gm-key", "gm-key"]+ );+ // Managed tokens are minted with uses:1 — a shared token would be rejected+ // at the second handshake with close code 1011.+ let mints = 0;+ const cloudDeps = deps({ postServerToken: async () => ({ token: `gm-${++mints}` }) });+ assert.deepEqual(+ await fetchRealtimeTokenForProvider(+ "gemini-realtime",+ cloudDeps,+ { mode: "openwhispr" },+ { streams: 2 }+ ),+ ["gm-1", "gm-2"]+ );+ await assert.rejects(+ fetchRealtimeTokenForProvider("gemini-realtime", deps({ postServerToken: async () => ({}) }), {+ mode: "openwhispr",+ }),+ /No Gemini token received/+ );+});+test("corti mints one token and shares it across both streams", async () => {const { fetchRealtimeTokenForProvider } = await load();let mints = 0;@@ -148,8 +183,18 @@ test("corti mints one token and shares it across both streams", async () => {test("missing byok keys throw configuration errors, not token errors", async () => {const { fetchRealtimeTokenForProvider } = await load();- const empty = { getOpenAIKey: () => "", getDeepgramKey: () => "", getAssemblyAIKey: () => "" };- for (const provider of ["openai-realtime", "deepgram-realtime", "assemblyai-realtime"]) {+ const empty = {+ getOpenAIKey: () => "",+ getDeepgramKey: () => "",+ getAssemblyAIKey: () => "",+ getGeminiKey: () => "",+ };+ for (const provider of [+ "openai-realtime",+ "deepgram-realtime",+ "assemblyai-realtime",+ "gemini-realtime",+ ]) {await assert.rejects(fetchRealtimeTokenForProvider(provider, deps({ environmentManager: empty }), {mode: "byok",@@ -163,10 +208,12 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+lconst { fetchRealtimeTokenForProvider } = await load();const wire = [];const cloudDeps = deps({- postServerToken: async (path, body) => {+ // `body = {}` mirrors ipcHandlers' postServerToken default, so an entry that+ // posts no body at all is recorded as the `{}` it really sends.+ postServerToken: async (path, body = {}) => {// JSON.stringify drops undefined keys exactly like the real fetch does.wire.push({ path, json: JSON.stringify(body) });- return { clientSecret: "cs-single", clientSecrets: ["cs-a", "cs-b"] };+ return { clientSecret: "cs-single", clientSecrets: ["cs-a", "cs-b"], token: "gm-token" };},});@@ -188,6 +235,9 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+l{ streams: 1 });+ // Gemini's endpoint reads nothing from the request; it must post a bare body.+ await fetchRealtimeTokenForProvider("gemini-realtime", cloudDeps, { mode: "openwhispr" });+assert.deepEqual(wire, [{ path: "/api/openai-realtime-token", json: '{"streams":1}' },{@@ -195,5 +245,6 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+ljson: '{"model":"gpt-4o-mini-transcribe","language":"en","streams":2}',},{ path: "/api/openai-realtime-token", json: '{"model":"gpt-4o-mini-transcribe","streams":1}' },+ { path: "/api/gemini-live-token", json: "{}" },]);});