Skip to content

Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.

feat(transcription): Gemini Live streaming for dictation (#2062)

Gabriel Stein9c41a8705ee7
diff --git a/docs/network-allowlist.md b/docs/network-allowlist.md
index 9f86d83b0..94a601bf0 100644
--- a/docs/network-allowlist.md
+++ b/docs/network-allowlist.md
@@ -18,14 +18,15 @@ onboarding).
## Required for streaming transcription
-OpenWhispr Cloud routes streaming sessions through one of three providers.
-Allowlist all three unless a specific provider is pinned in configuration.
+OpenWhispr Cloud routes streaming sessions through one of four providers.
+Allowlist all four unless a specific provider is pinned in configuration.
-| Host | Protocol | Port | Purpose |
-| -------------------------- | ---------- | ---- | --------------------------------------------------------------------------------- |
-| `api.deepgram.com` | WSS | 443 | Deepgram streaming transcription. |
-| `api.openai.com` | WSS, HTTPS | 443 | OpenAI Realtime streaming transcription. |
-| `streaming.assemblyai.com` | WSS, HTTPS | 443 | AssemblyAI streaming transcription. Token endpoint is HTTPS; live session is WSS. |
+| Host | Protocol | Port | Purpose |
+| ----------------------------------- | ---------- | ---- | --------------------------------------------------------------------------------------- |
+| `api.deepgram.com` | WSS | 443 | Deepgram streaming transcription. |
+| `api.openai.com` | WSS, HTTPS | 443 | OpenAI Realtime streaming transcription. |
+| `streaming.assemblyai.com` | WSS, HTTPS | 443 | AssemblyAI streaming transcription. Token endpoint is HTTPS; live session is WSS. |
+| `generativelanguage.googleapis.com` | WSS, HTTPS | 443 | Gemini Live streaming transcription. Live session is WSS; batch transcription is HTTPS. |
## Required for local model downloads
@@ -55,11 +56,11 @@ Contacted only when a user pastes a URL into the Upload view to download and
transcribe its audio. Downloads are HTTPS-only and hosts resolving to
private/internal addresses are rejected.
-| Host | Protocol | Port | Purpose |
-| ----------------------------------- | -------- | ---- | -------------------------------------------------------------------------- |
-| `www.youtube.com`, `youtube.com`, `youtu.be`, `m.youtube.com`, `music.youtube.com` | HTTPS | 443 | YouTube page/metadata fetch for pasted YouTube links (bundled yt-dlp). |
-| `*.googlevideo.com` | HTTPS | 443 | YouTube media CDN — the actual audio stream download. |
-| _User-pasted hosts_ | HTTPS | 443 | Direct audio/video URL imports contact whatever public host the user pastes. |
+| Host | Protocol | Port | Purpose |
+| ---------------------------------------------------------------------------------- | -------- | ---- | ---------------------------------------------------------------------------- |
+| `www.youtube.com`, `youtube.com`, `youtu.be`, `m.youtube.com`, `music.youtube.com` | HTTPS | 443 | YouTube page/metadata fetch for pasted YouTube links (bundled yt-dlp). |
+| `*.googlevideo.com` | HTTPS | 443 | YouTube media CDN — the actual audio stream download. |
+| _User-pasted hosts_ | HTTPS | 443 | Direct audio/video URL imports contact whatever public host the user pastes. |
## BYOK provider hosts (only if configured)
@@ -71,7 +72,7 @@ provider. Skip any provider not in use.
| `api.openai.com` | HTTPS | 443 | OpenAI API key configured (transcription or reasoning). |
| `*.cognitiveservices.azure.com`, `*.openai.azure.com`, `*.services.ai.azure.com` | HTTPS | 443 | Azure AI Foundry / Azure OpenAI speech-to-text configured (custom transcription provider pointed at your own Azure resource endpoint). |
| `api.anthropic.com` | HTTPS | 443 | Anthropic API key configured. |
-| `generativelanguage.googleapis.com` | HTTPS | 443 | Gemini API key configured. |
+| `generativelanguage.googleapis.com` | WSS, HTTPS | 443 | Gemini API key configured. Batch transcription is HTTPS; the Live streaming model opens a WSS session. |
| `api.groq.com` | HTTPS | 443 | Groq API key configured. |
| `atc.tinfoil.sh`, `*.tinfoil.sh` | WSS, HTTPS | 443 | Tinfoil API key configured. `atc.tinfoil.sh` serves the enclave attestation bundle (verified locally against an embedded sigstore root). Inference and realtime transcription connect to an enclave host assigned dynamically at runtime (e.g. `inference.tinfoil.sh`, `router.infN.tinfoil.sh`), so allowlist `*.tinfoil.sh` rather than pinning hosts. |
| `api.mistral.ai` | HTTPS | 443 | Mistral API key configured. |
@@ -105,6 +106,7 @@ curl -v https://api.openwhispr.com/api/health
curl -v https://api.deepgram.com/v1/projects
curl -v https://api.openai.com/v1/models
curl -v https://streaming.assemblyai.com/v3/token
+curl -v https://generativelanguage.googleapis.com/v1beta/models
# Model downloads (only if local mode is in use)
curl -v -I https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.bin
diff --git a/preload.js b/preload.js
index b20035329..1619de8f5 100644
--- a/preload.js
+++ b/preload.js
@@ -837,6 +837,27 @@ contextBridge.exposeInMainWorld("electronAPI", {
(callback) => (_event, data) => callback(data)
),
+ // Gemini Live Streaming
+ geminiStreamingWarmup: (options) => ipcRenderer.invoke("gemini-streaming-warmup", options),
+ geminiStreamingStart: (options) => ipcRenderer.invoke("gemini-streaming-start", options),
+ geminiStreamingSend: (audioBuffer) => ipcRenderer.send("gemini-streaming-send", audioBuffer),
+ geminiStreamingFinalize: () => ipcRenderer.send("gemini-streaming-finalize"),
+ geminiStreamingStop: () => ipcRenderer.invoke("gemini-streaming-stop"),
+ geminiStreamingStatus: () => ipcRenderer.invoke("gemini-streaming-status"),
+ onGeminiPartialTranscript: registerListener(
+ "gemini-partial-transcript",
+ (callback) => (_event, text) => callback(text)
+ ),
+ onGeminiFinalTranscript: registerListener(
+ "gemini-final-transcript",
+ (callback) => (_event, text) => callback(text)
+ ),
+ onGeminiError: registerListener("gemini-error", (callback) => (_event, error) => callback(error)),
+ onGeminiSessionEnd: registerListener(
+ "gemini-session-end",
+ (callback) => (_event, data) => callback(data)
+ ),
+
// Corti streaming (BYOK)
cortiStreamingWarmup: (options) => ipcRenderer.invoke("corti-streaming-warmup", options),
cortiStreamingStart: (options) => ipcRenderer.invoke("corti-streaming-start", options),
diff --git a/scripts/stt-canary.mjs b/scripts/stt-canary.mjs
index 7fc34094a..37a2c4524 100644
--- a/scripts/stt-canary.mjs
+++ b/scripts/stt-canary.mjs
@@ -5,9 +5,9 @@
* endpoint the app dials — so a provider changing its auth, endpoint, or
* handshake surfaces here on a schedule instead of in a customer report
* (#1624 sat in shipped builds for three days as a swallowed warmup warning).
- * Batch providers whose request shape is theirs alone rather than the shared
- * OpenAI-compatible multipart (Gemini) send a real transcription through the
- * shipped module for the same reason.
+ * Providers whose request shape is theirs alone rather than the shared
+ * OpenAI-compatible multipart (Gemini, batch and Live) send a real
+ * transcription through the shipped module for the same reason.
*
* Run: node scripts/stt-canary.mjs
* Keys come from STT_CANARY_<PROVIDER>_KEY env vars; providers without a key
@@ -25,10 +25,12 @@ import ffmpegPath from "ffmpeg-static";
import tokenProviders from "../src/helpers/realtimeTokenProviders.js";
import audioUtils from "../src/utils/audioUtils.js";
import geminiTranscription from "../src/helpers/geminiTranscription.js";
+import geminiLive from "../src/helpers/geminiLiveStreaming.js";
const { fetchRealtimeTokenForProvider } = tokenProviders;
const { pcm16ToWav } = audioUtils;
const { transcribeWithGemini } = geminiTranscription;
+const { GeminiLiveStreaming } = geminiLive;
const HANDSHAKE_TIMEOUT_MS = 15000;
// Half a second of 16 kHz mono silence: enough for a provider to accept and
@@ -110,12 +112,89 @@ async function probeGeminiBatch(key, audio, contentType) {
return { ok: true };
}
+// A completed Live turn proves the raw-key handshake, the setup message, the
+// audio frame shape and the transcript events — the socket opening proves none
+// of it, and server-side VAD suppresses silence, so the probe has to speak.
+const CANARY_PHRASE = "The quick brown fox. OpenWhispr transcription test.";
+const LIVE_FRAME_MS = 50;
+const LIVE_FRAME_BYTES = 1600; // one 50ms dictation worklet frame at 16kHz s16le
+
+// Gemini's own TTS keeps the fixture out of the repo and needs no second key.
+async function synthesizeCanarySpeech(key) {
+ const response = await fetch(
+ "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-preview-tts:generateContent",
+ {
+ method: "POST",
+ headers: { "Content-Type": "application/json", "x-goog-api-key": key },
+ body: JSON.stringify({
+ contents: [{ parts: [{ text: `Say clearly: ${CANARY_PHRASE}` }] }],
+ generationConfig: { responseModalities: ["AUDIO"] },
+ }),
+ }
+ );
+ if (!response.ok) throw new Error(`speech fixture request failed: HTTP ${response.status}`);
+ const data = await response.json();
+ const audio = data?.candidates?.[0]?.content?.parts?.[0]?.inlineData?.data;
+ if (!audio) throw new Error("speech fixture returned no audio");
+ // TTS answers with 24kHz L16; dictation captures 16kHz.
+ const resampled = spawnSync(
+ ffmpegPath,
+ [
+ "-f",
+ "s16le",
+ "-ar",
+ "24000",
+ "-ac",
+ "1",
+ "-i",
+ "pipe:0",
+ "-ar",
+ "16000",
+ "-f",
+ "s16le",
+ "pipe:1",
+ ],
+ { input: Buffer.from(audio, "base64"), maxBuffer: 64 * 1024 * 1024 }
+ );
+ if (resampled.status !== 0) throw new Error("could not resample the speech fixture to 16kHz");
+ return resampled.stdout;
+}
+
+async function probeGeminiLive(key) {
+ const token = await fetchRealtimeTokenForProvider("gemini-realtime", tokenDeps(key), {
+ mode: "byok",
+ });
+ const pcm = await synthesizeCanarySpeech(key);
+ const streaming = new GeminiLiveStreaming();
+ let partials = 0;
+ streaming.onPartialTranscript = () => {
+ partials += 1;
+ };
+ try {
+ await streaming.connect({ token, mode: "byok", keyterms: ["OpenWhispr"] });
+ // Paced like the mic worklet: the server finalizes relative to real-time
+ // ingest, so a burst followed by audioStreamEnd starves the final.
+ for (let offset = 0; offset < pcm.length; offset += LIVE_FRAME_BYTES) {
+ streaming.sendAudio(pcm.subarray(offset, offset + LIVE_FRAME_BYTES));
+ await new Promise((resolve) => setTimeout(resolve, LIVE_FRAME_MS));
+ }
+ const { text } = await streaming.disconnect(true);
+ if (!/quick brown fox/i.test(text)) {
+ return { ok: false, detail: `no usable transcript: ${JSON.stringify(text)}` };
+ }
+ return { ok: true, note: `${partials} partials, final "${text}"` };
+ } finally {
+ streaming.cleanup();
+ }
+}
+
const tokenDeps = (key) => ({
environmentManager: {
getOpenAIKey: () => key,
getTinfoilKey: () => key,
getDeepgramKey: () => key,
getAssemblyAIKey: () => key,
+ getGeminiKey: () => key,
},
proxyFetch: fetch,
});
@@ -173,6 +252,11 @@ const PROBES = [
: { ok: false, detail: "empty token" };
},
},
+ {
+ id: "gemini-live-streaming",
+ keyEnv: "STT_CANARY_GEMINI_KEY",
+ run: (key) => probeGeminiLive(key),
+ },
{
id: "gemini-batch-wav",
keyEnv: "STT_CANARY_GEMINI_KEY",
diff --git a/src/components/TranscriptionModelPicker.tsx b/src/components/TranscriptionModelPicker.tsx
index 226f0f915..a989a8860 100644
--- a/src/components/TranscriptionModelPicker.tsx
+++ b/src/components/TranscriptionModelPicker.tsx
@@ -14,7 +14,7 @@ import { useDialogs } from "../hooks/useDialogs";
import { useModelDownload, type DownloadProgress } from "../hooks/useModelDownload";
import {
getTranscriptionProviders,
- getStreamingTranscriptionProviders,
+ getMeetingStreamingTranscriptionProviders,
TranscriptionProviderData,
WHISPER_MODEL_INFO,
PARAKEET_MODEL_INFO,
@@ -463,8 +463,11 @@ export default function TranscriptionModelPicker({
(providerId: string) => isProviderAllowedByPolicy(policyState, "transcription", providerId),
[policyState]
);
+ // streamingOnly is Note Recording's picker, so it offers the streaming
+ // providers note recording can actually run — not every streaming provider.
const availableCloudProviders = useMemo(
- () => (streamingOnly ? getStreamingTranscriptionProviders() : getTranscriptionProviders()),
+ () =>
+ streamingOnly ? getMeetingStreamingTranscriptionProviders() : getTranscriptionProviders(),
[streamingOnly]
);
const cloudProviders = useMemo(
diff --git a/src/components/settings/MeetingSettings.tsx b/src/components/settings/MeetingSettings.tsx
index c6ae5d6ed..9b4e2eeb8 100644
--- a/src/components/settings/MeetingSettings.tsx
+++ b/src/components/settings/MeetingSettings.tsx
@@ -10,9 +10,9 @@ import { Toggle } from "../ui/toggle";
import TranscriptionModelPicker from "../TranscriptionModelPicker";
import type { InferenceMode } from "../../types/electron";
import { useStartOnboarding } from "../../hooks/useStartOnboarding";
-import { getStreamingTranscriptionProviders } from "../../models/ModelRegistry";
+import { getMeetingStreamingTranscriptionProviders } from "../../models/ModelRegistry";
-const MEETING_BYOK_PROVIDER_IDS = getStreamingTranscriptionProviders().map(
+const MEETING_BYOK_PROVIDER_IDS = getMeetingStreamingTranscriptionProviders().map(
(provider) => provider.id
);
diff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.js
index ac4feaf16..055fb8e54 100644
--- a/src/helpers/audioManager.js
+++ b/src/helpers/audioManager.js
@@ -380,6 +380,24 @@ const STREAMING_PROVIDERS = {
onSessionEnd: (cb) => window.electronAPI.onAssemblyAiSessionEnd(cb),
},
"openai-realtime": makeDictationRealtimeProvider("openai-realtime"),
+ gemini: {
+ // The final transcript lands ~500ms after audioStreamEnd (which finalize
+ // sends), ~2s at the p95 tail, so the stop sequence waits for it under a
+ // wider ceiling. geminiLiveStreaming.js measures the same 3s budget from
+ // audioStreamEnd before its own disconnect gives up.
+ awaitsFinalTranscript: true,
+ finalCeilingMs: 3000,
+ warmup: (opts) => window.electronAPI.geminiStreamingWarmup(opts),
+ start: (opts) => window.electronAPI.geminiStreamingStart(opts),
+ send: (buf) => window.electronAPI.geminiStreamingSend(buf),
+ finalize: () => window.electronAPI.geminiStreamingFinalize(),
+ stop: () => window.electronAPI.geminiStreamingStop(),
+ status: () => window.electronAPI.geminiStreamingStatus(),
+ onPartial: (cb) => window.electronAPI.onGeminiPartialTranscript(cb),
+ onFinal: (cb) => window.electronAPI.onGeminiFinalTranscript(cb),
+ onError: (cb) => window.electronAPI.onGeminiError(cb),
+ onSessionEnd: (cb) => window.electronAPI.onGeminiSessionEnd(cb),
+ },
corti: {
warmup: (opts) => window.electronAPI.cortiStreamingWarmup(opts),
start: (opts) => window.electronAPI.cortiStreamingStart(opts),
@@ -3780,10 +3798,11 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
const selfHostedModel = resolveSelfHostedTranscriptionModel(s);
if (selfHostedModel) return selfHostedModel;
const provider = s.cloudTranscriptionProvider || "openai";
- // Tinfoil pins its batch model in the registry rather than in settings.
- if (provider === "tinfoil") {
- return getBatchTranscriptionModel("tinfoil");
- }
+ // Tinfoil and Gemini pin their batch model in the registry rather than in
+ // settings: their streaming model has no batch endpoint, so a streaming
+ // fallback that reused the selected model would POST an unusable id.
+ const batchModel = getBatchTranscriptionModel(provider);
+ if (batchModel) return batchModel;
return resolveByokModel(provider, s.cloudTranscriptionModel);
} catch (error) {
return "gpt-4o-mini-transcribe";
@@ -4036,6 +4055,16 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
return !!model?.streaming && !!s.tinfoilApiKey;
}
+ // Gemini Live streams over its own WSS on either credential; the batch
+ // Gemini model on the same provider stays on HTTP.
+ if (s.cloudTranscriptionProvider === "gemini") {
+ const provider = getTranscriptionProvider("gemini");
+ const model = provider?.models.find((m) => m.id === s.cloudTranscriptionModel);
+ if (!model?.streaming) return false;
+ if (s.cloudTranscriptionMode === "byok") return !!s.geminiApiKey;
+ return !!(isSignedInOverride ?? s.isSignedIn);
+ }
+
// The managed-cloud bootstrap only controls OpenWhispr Cloud. A user's
// BYOK realtime model must not be downgraded because managed dictation is
// configured for batch processing.
@@ -4571,7 +4600,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
// Resolves once the transcript stops moving. An outstanding partial proves its
// final is still in flight, so only the ceiling ends the wait until it lands —
// a plain debounce would expire on the very tail this exists to catch.
- awaitStreamingTextSettled() {
+ awaitStreamingTextSettled(ceilingMs = STREAMING_FINAL_CEILING_MS) {
return new Promise((resolve) => {
const settle = () => {
clearTimeout(this.streamingTextDebounce);
@@ -4580,7 +4609,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
this.streamingTextDebounce = null;
resolve();
};
- const ceiling = setTimeout(settle, STREAMING_FINAL_CEILING_MS);
+ const ceiling = setTimeout(settle, ceilingMs);
const arm = () => {
clearTimeout(this.streamingTextDebounce);
if (this.streamingPartialText) return;
@@ -4823,7 +4852,7 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
const provider = this.getStreamingProvider();
provider.finalize?.();
if (provider.awaitsFinalTranscript) {
- await this.awaitStreamingTextSettled();
+ await this.awaitStreamingTextSettled(provider.finalCeilingMs);
} else {
await new Promise((resolve) => setTimeout(resolve, 300));
}
diff --git a/src/helpers/dictationStreamingRouting.js b/src/helpers/dictationStreamingRouting.js
index 90fff48c0..be156522c 100644
--- a/src/helpers/dictationStreamingRouting.js
+++ b/src/helpers/dictationStreamingRouting.js
@@ -7,6 +7,11 @@
export const REALTIME_MODELS = new Set(["gpt-4o-mini-transcribe", "gpt-4o-transcribe"]);
+// REALTIME_MODELS is the OpenAI-only shortcut (it forces "openai-realtime"), so
+// Gemini's live model routes on its own. Keying on the model id and not the
+// provider is required: the batch model on the same provider is HTTP-only.
+export const GEMINI_LIVE_MODEL = "gemini-3.5-transcribe-live";
+
export function defaultStreamingProviderName(context) {
return context === "notes" ? "deepgram" : "openai-realtime";
}
@@ -21,6 +26,12 @@ export function resolveStreamingProviderName({ settings, context, sttConfig }) {
) {
return "corti";
}
+ if (
+ settings.cloudTranscriptionProvider === "gemini" &&
+ settings.cloudTranscriptionModel === GEMINI_LIVE_MODEL
+ ) {
+ return "gemini";
+ }
if (REALTIME_MODELS.has(settings.cloudTranscriptionModel)) {
return "openai-realtime";
}
diff --git a/src/helpers/geminiLiveStreaming.js b/src/helpers/geminiLiveStreaming.js
new file mode 100644
index 000000000..f659bf4b5
--- /dev/null
+++ b/src/helpers/geminiLiveStreaming.js
@@ -0,0 +1,478 @@
+const WebSocket = require("ws");
+const debugLogger = require("./debugLogger");
+const { computePcm16Rms } = require("../utils/audioUtils");
+
+const SAMPLE_RATE = 16000;
+const WEBSOCKET_TIMEOUT_MS = 15000;
+// Budget for the final transcript, measured from audioStreamEnd (it lands
+// ~500ms after, ~2s at the p95 tail). The renderer's gemini settle ceiling in
+// audioManager.js is the same 3s; disconnect only spends what is left of it.
+const DISCONNECT_TIMEOUT_MS = 3000;
+const KEEPALIVE_INTERVAL_MS = 15000;
+const COLD_START_BUFFER_MAX = 3 * SAMPLE_RATE * 2; // 3 seconds of 16-bit PCM
+// Google documents a 1000-phrase ceiling but only promises quality up to 100;
+// the batch dictionary paths truncate the same array at 100.
+const MAX_CUSTOM_VOCABULARY = 100;
+// Trailing room noise in dictation recordings measures 0.003-0.010 RMS while
+// speech frames sit above ~0.0105, so only a frame at this level is a new
+// utterance rather than the tail of a turn the server already closed.
+const TURN_REOPEN_RMS = 0.012;
+const GEMINI_LIVE_MODEL = "gemini-3.5-transcribe-live";
+
+// In managed mode the model comes from the BYOK-shaped settings, which still
+// hold another provider's id, so anything but a Gemini id is ignored.
+const resolveLiveModel = (model) =>
+ model && model.startsWith("gemini-") ? model : GEMINI_LIVE_MODEL;
+
+const WS_BASE = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage";
+
+// The two Live methods are auth-disjoint: BidiGenerateContent accepts only
+// `key`, BidiGenerateContentConstrained only `access_token` (an auth_tokens
+// resource name). Crossing them closes the socket with 1007 right after a
+// successful HTTP 101, so the credential kind decides the method, not the URL.
+function buildGeminiLiveUrl({ mode, token }) {
+ if (mode === "byok") {
+ return `${WS_BASE}.v1beta.GenerativeService.BidiGenerateContent?key=${token}`;
+ }
+ return `${WS_BASE}.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=${token}`;
+}
+
+// Every Live failure arrives as a close frame after the upgrade succeeded (the
+// socket's `error` event never carries them), and the server truncates close
+// reasons to the 123-byte WebSocket limit — classify on the code alone; the
+// reason is only logged by the close handler.
+function closeFrameError(code) {
+ if (code === 1011) {
+ // "Token has been used too many times" / "Token has expired". The managed
+ // tokens are single-use, so this is the one retryable auth failure; byok
+ // re-handshakes with the same raw key and never sees it.
+ return Object.assign(new Error("Gemini Live session token expired"), {
+ code: "AUTH_EXPIRED",
+ });
+ }
+ if (code === 1007 || code === 1008) {
+ // 1007 is a malformed credential, 1008 one Google no longer knows (revoked
+ // or deleted); neither is retryable, both are fixed in Settings.
+ return Object.assign(new Error("Invalid Gemini API key. Check your key in Settings."), {
+ code: "INVALID_KEY",
+ });
+ }
+ return new Error(`Gemini Live closed before ready (code: ${code})`);
+}
+
+class GeminiLiveStreaming {
+ constructor() {
+ this.ws = null;
+ this.isConnected = false;
+ this.isConnecting = false;
+ this.isDisconnecting = false;
+ this.completedSegments = [];
+ this.onPartialTranscript = null;
+ this.onFinalTranscript = null;
+ this.onError = null;
+ this.onSessionEnd = null;
+ this.onConnectionLost = null;
+ this.pendingResolve = null;
+ this.pendingReject = null;
+ this.connectionTimeout = null;
+ this.keepAliveInterval = null;
+ this.coldStartBuffer = [];
+ this.coldStartBufferSize = 0;
+ this.bufferingAudio = false;
+ this.audioBytesSent = 0;
+ this.currentModel = GEMINI_LIVE_MODEL;
+ this._connectionLossNotified = false;
+ this._audioStreamEndSentAt = null;
+ this._turnEnded = false;
+ this._turnEndResolve = null;
+ }
+
+ // Starts buffering audio before the socket exists, covering the token fetch
+ // and the 350ms-1.4s handshake so sendAudio() doesn't drop the first words.
+ beginConnecting() {
+ this.bufferingAudio = true;
+ this.coldStartBuffer = [];
+ this.coldStartBufferSize = 0;
+ }
+
+ getFullTranscript() {
+ return this.completedSegments.join(" ");
+ }
+
+ // Test seam: overridden to point at a loopback server.
+ buildWebSocketUrl(options) {
+ return buildGeminiLiveUrl(options);
+ }
+
+ buildSetupMessage({ model, language, keyterms } = {}) {
+ const customVocabulary = (keyterms || [])
+ .map((term) => String(term).trim())
+ .filter(Boolean)
+ .slice(0, MAX_CUSTOM_VOCABULARY);
+ const inputAudioTranscription = {};
+ // `languageCodes` on its own costs punctuation and sentence casing, and an
+ // invalid locale is accepted silently — only ever send a language the user
+ // picked explicitly ("auto" is stripped upstream).
+ if (language) inputAudioTranscription.languageCodes = [language];
+ if (customVocabulary.length) inputAudioTranscription.customVocabulary = customVocabulary;
+
+ return {
+ setup: {
+ model: `models/${resolveLiveModel(model)}`,
+ generationConfig: { responseModalities: ["TEXT"] },
+ inputAudioTranscription,
+ },
+ };
+ }
+
+ async connect(options = {}) {
+ const { token, mode, refreshToken } = options;
+ if (!token) {
+ throw Object.assign(new Error("Gemini API key not configured. Add your key in Settings."), {
+ code: "API_KEY_MISSING",
+ });
+ }
+
+ if (this.isConnected || this.isConnecting) {
+ debugLogger.debug("Gemini Live already connected/connecting");
+ return;
+ }
+
+ if (!this.bufferingAudio) this.beginConnecting();
+ this.completedSegments = [];
+ this.audioBytesSent = 0;
+ this.currentModel = resolveLiveModel(options.model);
+ this._connectionLossNotified = false;
+ this._audioStreamEndSentAt = null;
+ this._turnEnded = false;
+
+ try {
+ await this._openSocket(options);
+ } catch (err) {
+ if (err.code !== "AUTH_EXPIRED" || mode === "byok" || !refreshToken) throw err;
+ debugLogger.debug("Gemini Live re-minting single-use token after 1011 close");
+ this.bufferingAudio = true;
+ await this._openSocket({ ...options, token: await refreshToken() });
+ }
+ }
+
+ _openSocket(options) {
+ const url = this.buildWebSocketUrl(options);
+ debugLogger.debug("Gemini Live connecting", {
+ mode: options.mode === "byok" ? "byok" : "openwhispr",
+ model: this.currentModel,
+ });
+
+ return new Promise((resolve, reject) => {
+ this.pendingResolve = resolve;
+ this.pendingReject = reject;
+
+ this.isConnecting = true;
+ this.connectionTimeout = setTimeout(() => {
+ this.cleanup();
+ reject(new Error("Gemini Live connection timeout"));
+ }, WEBSOCKET_TIMEOUT_MS);
+
+ this.ws = new WebSocket(url);
+
+ this.ws.on("open", () => {
+ // Audio sent before setupComplete is discarded by the server, so the
+ // setup message is the only thing that goes out on open.
+ this.ws.send(JSON.stringify(this.buildSetupMessage(options)));
+ });
+
+ this.ws.on("message", (data) => {
+ this.handleMessage(data);
+ });
+
+ this.ws.on("error", (error) => {
+ const wasActive = this.isConnected;
+ debugLogger.error("Gemini Live WebSocket error", { error: error.message });
+ this.cleanup();
+ this._rejectPending(error);
+ if (wasActive && !this.isDisconnecting) {
+ this._notifyConnectionLost(error);
+ } else if (!this.isDisconnecting) {
+ this.onError?.(error);
+ }
+ });
+
+ this.ws.on("close", (code, reason) => {
+ const wasActive = this.isConnected;
+ debugLogger.debug("Gemini Live WebSocket closed", {
+ code,
+ reason: reason?.toString() || "",
+ wasActive,
+ });
+ this._resolveTurnEnd();
+ this.cleanup();
+ if (!wasActive) {
+ this._rejectPending(closeFrameError(code));
+ } else if (!this.isDisconnecting) {
+ // A session killed mid-dictation (the ~10 minute cap, a dead network)
+ // still hands over what it transcribed; there is no resumption handle
+ // to reconnect with, so the caller finalizes with this text.
+ this.onSessionEnd?.({ text: this.getFullTranscript() });
+ this._notifyConnectionLost(new Error(`Connection lost (code: ${code})`));
+ }
+ });
+ });
+ }
+
+ // The complete server vocabulary is `setupComplete` plus
+ // `serverContent.{speechState, interimInputTranscription, inputTranscription,
+ // generationComplete}`. There is no `turnComplete`, no `usageMetadata` and no
+ // `sessionResumptionUpdate` on this model, so end-of-turn is the final
+ // transcript arriving with `generationComplete` in the same millisecond.
+ handleMessage(data) {
+ try {
+ const message = JSON.parse(data.toString());
+
+ if (message.setupComplete) this._markConnected();
+
+ const serverContent = message.serverContent;
+ if (!serverContent) return;
+
+ // Partials are revised, not appended to ("The quick brown" becomes
+ // "the quick brown fox"), so consumers must replace the whole string.
+ const partial = serverContent.interimInputTranscription?.text;
+ if (partial) {
+ this._turnEnded = false;
+ this.onPartialTranscript?.(partial);
+ }
+
+ const final = serverContent.inputTranscription?.text?.trim();
+ if (final) {
+ this.completedSegments.push(final);
+ const fullText = this.getFullTranscript();
+ this.onFinalTranscript?.(fullText, Date.now());
+ debugLogger.debug("Gemini Live turn completed", {
+ turnText: final.slice(0, 100),
+ totalLength: fullText.length,
+ segments: this.completedSegments.length,
+ });
+ }
+
+ // The server closes a turn on trailing silence, often before the hotkey is
+ // released; that generationComplete already answers the final. Speech
+ // (a loud frame or a new partial) reopens a turn, so a later stop waits again.
+ if (serverContent.generationComplete) this._resolveTurnEnd();
+ } catch (err) {
+ debugLogger.error("Gemini Live message parse error", { error: err.message });
+ }
+ }
+
+ _markConnected() {
+ if (this.isConnected) return;
+ this.isConnected = true;
+ this.isConnecting = false;
+ clearTimeout(this.connectionTimeout);
+ this.connectionTimeout = null;
+ this.startKeepAlive();
+ debugLogger.debug("Gemini Live setup complete", { model: this.currentModel });
+ // A dictation shorter than the handshake leaves every frame in the buffer,
+ // and sendAudio never runs again to drain it.
+ this._flushColdStartBuffer();
+ if (this.pendingResolve) {
+ this.pendingResolve();
+ this.pendingResolve = null;
+ this.pendingReject = null;
+ }
+ }
+
+ _rejectPending(error) {
+ if (!this.pendingReject) return;
+ this.pendingReject(error);
+ this.pendingReject = null;
+ this.pendingResolve = null;
+ }
+
+ _notifyConnectionLost(error) {
+ if (this._connectionLossNotified) return;
+ this._connectionLossNotified = true;
+ if (this.onConnectionLost) {
+ this.onConnectionLost(error);
+ } else {
+ this.onError?.(error);
+ }
+ }
+
+ _resolveTurnEnd() {
+ this._turnEnded = true;
+ this._turnEndResolve?.();
+ this._turnEndResolve = null;
+ }
+
+ // A warm socket sits idle between dictations with no other liveness check; a
+ // network path that dies silently leaves isConnected stuck true and the next
+ // recording gets sent into a dead socket.
+ startKeepAlive() {
+ this.stopKeepAlive();
+ const socket = this.ws;
+ if (!socket) return;
+
+ socket.isAlive = true;
+ socket.on("pong", () => {
+ socket.isAlive = true;
+ });
+
+ this.keepAliveInterval = setInterval(() => {
+ if (socket !== this.ws || socket.readyState !== WebSocket.OPEN) {
+ this.stopKeepAlive();
+ return;
+ }
+ if (socket.isAlive === false) {
+ debugLogger.debug("Gemini Live keep-alive missed pong, terminating stale connection");
+ socket.terminate();
+ return;
+ }
+ socket.isAlive = false;
+ try {
+ socket.ping();
+ } catch (err) {
+ debugLogger.debug("Gemini Live keep-alive ping failed", { error: err.message });
+ socket.terminate();
+ }
+ }, KEEPALIVE_INTERVAL_MS);
+ }
+
+ stopKeepAlive() {
+ if (this.keepAliveInterval) {
+ clearInterval(this.keepAliveInterval);
+ this.keepAliveInterval = null;
+ }
+ }
+
+ _sendAudioFrame(pcmBuffer) {
+ this.ws.send(
+ JSON.stringify({
+ realtimeInput: {
+ audio: { data: pcmBuffer.toString("base64"), mimeType: `audio/pcm;rate=${SAMPLE_RATE}` },
+ },
+ })
+ );
+ this.audioBytesSent += pcmBuffer.length;
+ if (this._turnEnded && computePcm16Rms(pcmBuffer) >= TURN_REOPEN_RMS) this._turnEnded = false;
+ }
+
+ sendAudio(pcmBuffer) {
+ // The capture worklet ends its stream with a "flushed" string sentinel;
+ // only s16 PCM (even byte length) is audio, and only audioStreamEnd closes
+ // a Gemini turn.
+ if (!pcmBuffer || pcmBuffer.length % 2 !== 0) return false;
+
+ if (this.ws?.readyState !== WebSocket.OPEN || !this.isConnected) {
+ if (this.bufferingAudio && this.coldStartBufferSize < COLD_START_BUFFER_MAX) {
+ const copy = Buffer.from(pcmBuffer);
+ this.coldStartBuffer.push(copy);
+ this.coldStartBufferSize += copy.length;
+ }
+ return false;
+ }
+
+ this._flushColdStartBuffer();
+ this._sendAudioFrame(pcmBuffer);
+ return true;
+ }
+
+ _flushColdStartBuffer() {
+ if (this.coldStartBuffer.length === 0) return;
+ debugLogger.debug("Gemini Live flushing cold-start buffer", {
+ chunks: this.coldStartBuffer.length,
+ bytes: this.coldStartBufferSize,
+ });
+ for (const buffered of this.coldStartBuffer) {
+ this._sendAudioFrame(buffered);
+ }
+ this.coldStartBuffer = [];
+ this.coldStartBufferSize = 0;
+ }
+
+ // Gemini has no mid-stream finalize: audioStreamEnd ends the turn, and the
+ // server answers with the final transcript. Sending it from here (the stop
+ // sequence calls finalize() before it waits for text) buys back the wait.
+ finalize() {
+ if (this.ws?.readyState !== WebSocket.OPEN || this._audioStreamEndSentAt) return false;
+ this._audioStreamEndSentAt = Date.now();
+ this.ws.send(JSON.stringify({ realtimeInput: { audioStreamEnd: true } }));
+ debugLogger.debug("Gemini Live audioStreamEnd sent", { audioBytesSent: this.audioBytesSent });
+ return true;
+ }
+
+ async disconnect(closeStream = true) {
+ debugLogger.debug("Gemini Live disconnect", {
+ audioBytesSent: this.audioBytesSent,
+ segments: this.completedSegments.length,
+ textLength: this.getFullTranscript().length,
+ readyState: this.ws?.readyState,
+ });
+
+ if (!this.ws) return this._takeTranscript();
+
+ this.isDisconnecting = true;
+
+ if (closeStream && this.ws.readyState === WebSocket.OPEN && this.isConnected) {
+ this._flushColdStartBuffer();
+ }
+
+ if (closeStream && this.ws.readyState === WebSocket.OPEN && this.audioBytesSent > 0) {
+ this.finalize();
+ if (!this._turnEnded) await this._awaitTurnEnd();
+ }
+
+ const result = this._takeTranscript();
+ this.cleanup();
+ this.isDisconnecting = false;
+ return result;
+ }
+
+ // The stop sequence finalizes first and waits on the renderer side before it
+ // gets here, so the budget runs from audioStreamEnd, not from this call.
+ async _awaitTurnEnd() {
+ const remaining = DISCONNECT_TIMEOUT_MS - (Date.now() - this._audioStreamEndSentAt);
+ if (remaining <= 0) return;
+ let timeoutId;
+ await Promise.race([
+ new Promise((resolve) => {
+ this._turnEndResolve = resolve;
+ }),
+ new Promise((resolve) => {
+ timeoutId = setTimeout(() => {
+ debugLogger.debug("Gemini Live final transcript timeout, using accumulated text");
+ resolve();
+ }, remaining);
+ }),
+ ]);
+ clearTimeout(timeoutId);
+ this._turnEndResolve = null;
+ }
+
+ _takeTranscript() {
+ const result = { text: this.getFullTranscript() };
+ this.completedSegments = [];
+ return result;
+ }
+
+ cleanup() {
+ clearTimeout(this.connectionTimeout);
+ this.connectionTimeout = null;
+ this.stopKeepAlive();
+
+ if (this.ws) {
+ try {
+ this.ws.close();
+ } catch {}
+ this.ws = null;
+ }
+
+ this.isConnected = false;
+ this.isConnecting = false;
+ this.bufferingAudio = false;
+ }
+
+ getStatus() {
+ return { isConnected: this.isConnected, isConnecting: this.isConnecting };
+ }
+}
+
+module.exports = { GeminiLiveStreaming, buildGeminiLiveUrl, GEMINI_LIVE_MODEL };
diff --git a/src/helpers/ipcHandlers.js b/src/helpers/ipcHandlers.js
index db760014f..4c12f4a83 100644
--- a/src/helpers/ipcHandlers.js
+++ b/src/helpers/ipcHandlers.js
@@ -82,6 +82,7 @@ const HyprlandShortcutManager = require("./hyprlandShortcut");
const AssemblyAiStreaming = require("./assemblyAiStreaming");
const { i18nMain, changeLanguage } = require("./i18nMain");
const DeepgramStreaming = require("./deepgramStreaming");
+const { GeminiLiveStreaming, GEMINI_LIVE_MODEL } = require("./geminiLiveStreaming");
const CortiStreaming = require("./cortiStreaming");
const OpenAIRealtimeStreaming = require("./openaiRealtimeStreaming");
const { getCortiToken } = require("./cortiAuth");
@@ -630,6 +631,7 @@ class IPCHandlers {
this._micHoldSenders = new Map();
this.assemblyAiStreaming = null;
this.deepgramStreaming = null;
+ this.geminiStreaming = null;
this.cortiStreaming = null;
this._dictationStreaming = null;
this._dictationConnectPromise = null;
@@ -10648,6 +10650,155 @@ class IPCHandlers {
return this.deepgramStreaming.getStatus();
});
+ let geminiStreamingStartInProgress = false;
+ let geminiSendDropCount = 0;
+ // One handshake at a time. A start that raced an in-flight warmup used to
+ // clear the cold-start buffer, spend a second single-use managed token and
+ // report success while the warmup's handshake (and its failure) were still
+ // pending on a promise nobody read.
+ let geminiConnectInFlight = null;
+
+ // Re-bound on every warmup/start so a warm socket promoted by a different
+ // window can never emit into the window that opened it.
+ const ensureGeminiStreaming = (event) => {
+ if (!this.geminiStreaming) {
+ this.geminiStreaming = new GeminiLiveStreaming();
+ }
+ const win = BrowserWindow.fromWebContents(event.sender);
+ const emit = (channel, payload) => {
+ if (win && !win.isDestroyed()) win.webContents.send(channel, payload);
+ };
+ const streaming = this.geminiStreaming;
+ streaming.onPartialTranscript = (text) => emit("gemini-partial-transcript", text);
+ streaming.onFinalTranscript = (text) => emit("gemini-final-transcript", text);
+ streaming.onError = (error) => emit("gemini-error", error.message);
+ streaming.onSessionEnd = (data) => emit("gemini-session-end", data);
+ return streaming;
+ };
+
+ const connectGeminiStreaming = (event, options) => {
+ if (geminiConnectInFlight) return geminiConnectInFlight;
+ geminiConnectInFlight = (async () => {
+ const streaming = ensureGeminiStreaming(event);
+ // byok resolves to the raw API key, managed to a single-use ephemeral
+ // token; the client picks its Live method from `mode` accordingly.
+ const tokenOptions = { mode: options.mode, provider: "gemini-realtime" };
+ // Buffer before the token fetch (a real network round trip) so
+ // gemini-streaming-send has somewhere to put the first frames.
+ streaming.beginConnecting();
+ const token = await fetchRealtimeToken(event, tokenOptions);
+ await streaming.connect({
+ ...options,
+ token,
+ refreshToken: () => fetchRealtimeToken(event, tokenOptions),
+ });
+ })().finally(() => {
+ geminiConnectInFlight = null;
+ });
+ return geminiConnectInFlight;
+ };
+
+ ipcMain.handle("gemini-streaming-warmup", async (event, options = {}) => {
+ try {
+ if (this.geminiStreaming?.isConnected) {
+ ensureGeminiStreaming(event);
+ debugLogger.debug("Gemini Live connection already warm", {}, "streaming");
+ return { success: true, alreadyWarm: true };
+ }
+ await connectGeminiStreaming(event, options);
+ return { success: true };
+ } catch (error) {
+ debugLogger.error("Gemini streaming warmup error", { error: error.message });
+ return toPolicyFailure(error);
+ }
+ });
+
+ ipcMain.handle("gemini-streaming-start", async (event, options = {}) => {
+ if (geminiStreamingStartInProgress) {
+ debugLogger.debug("Gemini streaming start already in progress, ignoring", {}, "streaming");
+ return { success: false, error: "Operation in progress" };
+ }
+
+ geminiStreamingStartInProgress = true;
+ try {
+ const streaming = ensureGeminiStreaming(event);
+ if (geminiConnectInFlight) await geminiConnectInFlight;
+ const usedWarmConnection = streaming.isConnected && !options.forceNew;
+ if (!usedWarmConnection) {
+ if (streaming.isConnected) await streaming.disconnect(false);
+ await connectGeminiStreaming(event, options);
+ }
+ geminiSendDropCount = 0;
+ debugLogger.debug("Gemini streaming started", { usedWarmConnection }, "streaming");
+ return { success: true, usedWarmConnection };
+ } catch (error) {
+ debugLogger.error("Gemini streaming start error", { error: error.message });
+ if (error.code === "AUTH_EXPIRED") {
+ return { success: false, error: "Session expired", code: "AUTH_EXPIRED" };
+ }
+ return streamingStartFailure(error);
+ } finally {
+ geminiStreamingStartInProgress = false;
+ }
+ });
+
+ ipcMain.on("gemini-streaming-send", (_event, audioBuffer) => {
+ try {
+ if (!this.geminiStreaming) return;
+ const sent = this.geminiStreaming.sendAudio(Buffer.from(audioBuffer));
+ if (!sent) {
+ geminiSendDropCount++;
+ if (geminiSendDropCount <= 3 || geminiSendDropCount % 50 === 0) {
+ debugLogger.warn(
+ "Gemini audio send dropped",
+ {
+ dropCount: geminiSendDropCount,
+ isConnected: this.geminiStreaming.isConnected,
+ wsReadyState: this.geminiStreaming.ws?.readyState,
+ },
+ "streaming"
+ );
+ }
+ } else if (geminiSendDropCount > 0) {
+ debugLogger.debug(
+ "Gemini audio send resumed after drops",
+ { previousDrops: geminiSendDropCount },
+ "streaming"
+ );
+ geminiSendDropCount = 0;
+ }
+ } catch (error) {
+ debugLogger.error("Gemini streaming send error", { error: error.message });
+ }
+ });
+
+ ipcMain.on("gemini-streaming-finalize", () => {
+ this.geminiStreaming?.finalize();
+ });
+
+ ipcMain.handle("gemini-streaming-stop", async () => {
+ try {
+ const model = this.geminiStreaming?.currentModel || GEMINI_LIVE_MODEL;
+ const audioBytesSent = this.geminiStreaming?.audioBytesSent || 0;
+ let result = { text: "" };
+ if (this.geminiStreaming) {
+ result = await this.geminiStreaming.disconnect(true);
+ }
+
+ return { success: true, text: result?.text || "", model, audioBytesSent };
+ } catch (error) {
+ debugLogger.error("Gemini streaming stop error", { error: error.message });
+ return { success: false, error: error.message };
+ }
+ });
+
+ ipcMain.handle("gemini-streaming-status", async () => {
+ if (!this.geminiStreaming) {
+ return { isConnected: false, isConnecting: false };
+ }
+ return this.geminiStreaming.getStatus();
+ });
+
ipcMain.handle("corti-streaming-warmup", async (_event, options = {}) => {
try {
if (!this.cortiStreaming) {
diff --git a/src/helpers/meetingTranscriptionRouting.js b/src/helpers/meetingTranscriptionRouting.js
index 306127f4e..1b56bee47 100644
--- a/src/helpers/meetingTranscriptionRouting.js
+++ b/src/helpers/meetingTranscriptionRouting.js
@@ -1,3 +1,22 @@
+// Note recording only offers providers the main process will actually run:
+// meeting prepare/start check ALLOWED_MEETING_PROVIDERS (derived from the
+// streaming client table in meetingStreamingProviders.js) and reject anything
+// else with no user-visible message. Without this intersection, every registry
+// model marked `streaming: true` reaches the notes picker — including
+// dictation-only ones like Gemini Live — and fails silently there. The closure
+// is pinned by test/helpers/meetingStreamingProviders.test.js.
+export const MEETING_STREAMING_PROVIDER_IDS = [
+ "openai",
+ "assemblyai",
+ "deepgram",
+ "corti",
+ "tinfoil",
+];
+
+export function filterMeetingStreamingProviders(providers) {
+ return providers.filter((provider) => MEETING_STREAMING_PROVIDER_IDS.includes(provider.id));
+}
+
const DEFAULT_MANAGED_PROVIDER = {
id: "openai",
models: [{ id: "gpt-4o-mini-transcribe", default: true }],
diff --git a/src/helpers/realtimeTokenProviders.js b/src/helpers/realtimeTokenProviders.js
index 5951056cd..1bac8a58c 100644
--- a/src/helpers/realtimeTokenProviders.js
+++ b/src/helpers/realtimeTokenProviders.js
@@ -56,6 +56,24 @@ const REALTIME_TOKEN_PROVIDERS = {
});
},
+ "gemini-realtime": async ({ environmentManager, postServerToken }, options, streams) => {
+ if (options.mode === "byok") {
+ const apiKey = environmentManager.getGeminiKey();
+ if (!apiKey) {
+ throw new Error("No Gemini API key configured. Add your key in Settings.");
+ }
+ // The raw key opens the Live socket directly (BidiGenerateContent?key=)
+ // and is not consumed by a handshake, so both streams can share it.
+ return duplicate(streams, apiKey);
+ }
+ // Managed tokens are minted with uses:1, so N sockets need N mints.
+ return dual(streams, async () => {
+ const data = await postServerToken("/api/gemini-live-token");
+ if (!data.token) throw new Error("No Gemini token received");
+ return data.token;
+ });
+ },
+
"corti-realtime": async ({ mintCortiToken }, options, streams) => {
// One token covers both meeting streams; it's only used at the WSS handshake.
const { token } = await mintCortiToken(options);
diff --git a/src/locales/de/translation.json b/src/locales/de/translation.json
index b3bfb4a12..8e0276c4f 100644
--- a/src/locales/de/translation.json
+++ b/src/locales/de/translation.json
@@ -2625,6 +2625,7 @@
"groq_whisper_large_v3_turbo": "216x Echtzeitgeschwindigkeit",
"mistral_voxtral_mini_latest": "Schnelle mehrsprachige Transkription",
"gemini_3_5_transcribe": "Googles Gemini-Spracherkennung",
+ "gemini_3_5_transcribe_live": "Spracherkennung per Echtzeit-Streaming",
"corti_transcribe": "Medizinische Transkription in klinischer Qualität",
"xai_grok_stt": "Hochpräzise Spracherkennung von xAI",
"tinfoil_voxtral": "Nachweisbar private Transkription"
diff --git a/src/locales/en/translation.json b/src/locales/en/translation.json
index 7f32134ef..0df61238d 100644
--- a/src/locales/en/translation.json
+++ b/src/locales/en/translation.json
@@ -2625,6 +2625,7 @@
"groq_whisper_large_v3_turbo": "216x real-time speed",
"mistral_voxtral_mini_latest": "Fast multilingual transcription",
"gemini_3_5_transcribe": "Google's Gemini speech-to-text",
+ "gemini_3_5_transcribe_live": "Realtime streaming speech-to-text",
"corti_transcribe": "Clinical-grade medical transcription",
"xai_grok_stt": "High-accuracy speech recognition by xAI",
"tinfoil_voxtral": "Verifiably private transcription"
diff --git a/src/locales/es/translation.json b/src/locales/es/translation.json
index 0d2a960d5..4bd80ff3c 100644
--- a/src/locales/es/translation.json
+++ b/src/locales/es/translation.json
@@ -2580,6 +2580,7 @@
"groq_whisper_large_v3_turbo": "Velocidad 216x en tiempo real",
"mistral_voxtral_mini_latest": "Transcripción multilingüe rápida",
"gemini_3_5_transcribe": "Voz a texto con Gemini de Google",
+ "gemini_3_5_transcribe_live": "Voz a texto en streaming en tiempo real",
"corti_transcribe": "Transcripción médica de nivel clínico",
"xai_grok_stt": "Reconocimiento de voz de alta precisión por xAI",
"tinfoil_voxtral": "Transcripción verificablemente privada"
diff --git a/src/locales/fr/translation.json b/src/locales/fr/translation.json
index 3f05df531..2d816a927 100644
--- a/src/locales/fr/translation.json
+++ b/src/locales/fr/translation.json
@@ -2625,6 +2625,7 @@
"groq_whisper_large_v3_turbo": "Vitesse 216x en temps réel",
"mistral_voxtral_mini_latest": "Transcription multilingue rapide",
"gemini_3_5_transcribe": "Reconnaissance vocale Gemini de Google",
+ "gemini_3_5_transcribe_live": "Reconnaissance vocale en streaming temps réel",
"corti_transcribe": "Transcription médicale de qualité clinique",
"xai_grok_stt": "Reconnaissance vocale haute précision par xAI",
"tinfoil_voxtral": "Transcription vérifiablement privée"
diff --git a/src/locales/it/translation.json b/src/locales/it/translation.json
index 504cadc6b..a4583499a 100644
--- a/src/locales/it/translation.json
+++ b/src/locales/it/translation.json
@@ -2532,6 +2532,7 @@
"groq_whisper_large_v3": "Modello Large v3, veloce e preciso",
"mistral_voxtral_mini_latest": "Trascrizione multilingue veloce",
"gemini_3_5_transcribe": "Riconoscimento vocale Gemini di Google",
+ "gemini_3_5_transcribe_live": "Riconoscimento vocale in streaming in tempo reale",
"corti_transcribe": "Trascrizione medica di livello clinico",
"xai_grok_stt": "Riconoscimento vocale ad alta precisione di xAI",
"tinfoil_voxtral": "Trascrizione verificabilmente privata"
diff --git a/src/locales/ja/translation.json b/src/locales/ja/translation.json
index f1d18bbe2..68ed615dc 100644
--- a/src/locales/ja/translation.json
+++ b/src/locales/ja/translation.json
@@ -2532,6 +2532,7 @@
"groq_whisper_large_v3": "Large v3 モデル、高速かつ高精度",
"mistral_voxtral_mini_latest": "高速多言語文字起こし",
"gemini_3_5_transcribe": "GoogleのGemini音声文字起こし",
+ "gemini_3_5_transcribe_live": "リアルタイムストリーミング文字起こし",
"corti_transcribe": "臨床グレードの医療文字起こし",
"xai_grok_stt": "xAIによる高精度音声認識",
"tinfoil_voxtral": "検証可能なプライベート文字起こし"
diff --git a/src/locales/pt/translation.json b/src/locales/pt/translation.json
index 62feb6b0b..4ad5644ff 100644
--- a/src/locales/pt/translation.json
+++ b/src/locales/pt/translation.json
@@ -2511,6 +2511,7 @@
"groq_whisper_large_v3": "Modelo Large v3, rápido e preciso",
"mistral_voxtral_mini_latest": "Transcrição multilíngue rápida",
"gemini_3_5_transcribe": "Fala para texto Gemini do Google",
+ "gemini_3_5_transcribe_live": "Fala para texto em streaming em tempo real",
"corti_transcribe": "Transcrição médica de nível clínico",
"xai_grok_stt": "Reconhecimento de voz de alta precisão pela xAI",
"tinfoil_voxtral": "Transcrição verificavelmente privada"
diff --git a/src/locales/ru/translation.json b/src/locales/ru/translation.json
index a02ef1370..46bfe4e75 100644
--- a/src/locales/ru/translation.json
+++ b/src/locales/ru/translation.json
@@ -2548,6 +2548,7 @@
"groq_whisper_large_v3": "Модель Large v3, быстрая и точная",
"mistral_voxtral_mini_latest": "Быстрая многоязычная транскрипция",
"gemini_3_5_transcribe": "Распознавание речи Gemini от Google",
+ "gemini_3_5_transcribe_live": "Распознавание речи в реальном времени",
"corti_transcribe": "Медицинская транскрипция клинического уровня",
"xai_grok_stt": "Высокоточное распознавание речи от xAI",
"tinfoil_voxtral": "Проверяемо приватная транскрипция"
diff --git a/src/locales/zh-CN/translation.json b/src/locales/zh-CN/translation.json
index 90d1ed054..8b0c9dc03 100644
--- a/src/locales/zh-CN/translation.json
+++ b/src/locales/zh-CN/translation.json
@@ -2532,6 +2532,7 @@
"groq_whisper_large_v3": "Large v3 模型,快速且精准",
"mistral_voxtral_mini_latest": "快速多语言转录",
"gemini_3_5_transcribe": "Google Gemini 语音转文字",
+ "gemini_3_5_transcribe_live": "实时流式语音转文字",
"corti_transcribe": "临床级医疗转录",
"xai_grok_stt": "xAI 高精度语音识别",
"tinfoil_voxtral": "可验证的私密转录"
diff --git a/src/locales/zh-TW/translation.json b/src/locales/zh-TW/translation.json
index 0b79050b6..056778382 100644
--- a/src/locales/zh-TW/translation.json
+++ b/src/locales/zh-TW/translation.json
@@ -2532,6 +2532,7 @@
"groq_whisper_large_v3": "Large v3 模型,快速且精準",
"mistral_voxtral_mini_latest": "快速多語言轉錄",
"gemini_3_5_transcribe": "Google Gemini 語音轉文字",
+ "gemini_3_5_transcribe_live": "即時串流語音轉文字",
"corti_transcribe": "臨床級醫療轉錄",
"xai_grok_stt": "xAI 高精度語音辨識",
"tinfoil_voxtral": "可驗證的私密轉錄"
diff --git a/src/models/ModelRegistry.ts b/src/models/ModelRegistry.ts
index d9168a27c..876939898 100644
--- a/src/models/ModelRegistry.ts
+++ b/src/models/ModelRegistry.ts
@@ -1,6 +1,7 @@
import modelDataRaw from "./modelRegistryData.json";
import { isCloudCleanupMode, getSettings } from "../stores/settingsStore";
import { readCachedTinfoilModels } from "./tinfoilModelCache";
+import { filterMeetingStreamingProviders } from "../helpers/meetingTranscriptionRouting";
import type { InferenceMode } from "../types/electron";
export interface ModelDefinition {
@@ -432,6 +433,12 @@ export function getStreamingTranscriptionProviders(): TranscriptionProviderData[
.filter((p) => p.models.length > 0);
}
+// Streaming providers note recording can actually run (see
+// meetingTranscriptionRouting.MEETING_STREAMING_PROVIDER_IDS).
+export function getMeetingStreamingTranscriptionProviders(): TranscriptionProviderData[] {
+ return filterMeetingStreamingProviders(getStreamingTranscriptionProviders());
+}
+
export function getTranscriptionProvider(
providerId: string
): TranscriptionProviderData | undefined {
diff --git a/src/models/modelRegistryData.json b/src/models/modelRegistryData.json
index fa0c4824f..1029caa62 100644
--- a/src/models/modelRegistryData.json
+++ b/src/models/modelRegistryData.json
@@ -277,12 +277,20 @@
"id": "gemini",
"name": "Gemini",
"baseUrl": "https://generativelanguage.googleapis.com/v1beta",
+ "batchModel": "gemini-3.5-transcribe",
"models": [
{
"id": "gemini-3.5-transcribe",
"name": "Gemini 3.5 Transcribe",
"description": "Google's Gemini speech-to-text",
"descriptionKey": "models.descriptions.transcription.gemini_3_5_transcribe"
+ },
+ {
+ "id": "gemini-3.5-transcribe-live",
+ "name": "Gemini 3.5 Transcribe Live",
+ "description": "Realtime streaming speech-to-text",
+ "descriptionKey": "models.descriptions.transcription.gemini_3_5_transcribe_live",
+ "streaming": true
}
]
},
diff --git a/src/stores/meetingRecordingStore.ts b/src/stores/meetingRecordingStore.ts
index 01feda941..5984d1553 100644
--- a/src/stores/meetingRecordingStore.ts
+++ b/src/stores/meetingRecordingStore.ts
@@ -1,7 +1,7 @@
import { create } from "zustand";
import { getSettings, selectResolvedMeetingTranscription } from "./settingsStore";
import { useStreamingProvidersStore } from "./streamingProvidersStore";
-import { getStreamingTranscriptionProviders } from "../models/ModelRegistry";
+import { getMeetingStreamingTranscriptionProviders } from "../models/ModelRegistry";
import { resolveMeetingTranscriptionOptions } from "../helpers/meetingTranscriptionRouting";
import { followsSystemDefaultMic } from "../helpers/micSelectionRecovery";
import { resolvePreferredMicrophone } from "../helpers/microphoneSelection";
@@ -163,7 +163,7 @@ const getMeetingTranscriptionOptions = () => {
cohereModel: resolved.cohereModel,
selectedProvider: resolved.cloudTranscriptionProvider,
selectedModel: resolved.cloudTranscriptionModel,
- byokProviders: getStreamingTranscriptionProviders(),
+ byokProviders: getMeetingStreamingTranscriptionProviders(),
managedProviders: useStreamingProvidersStore.getState().providers,
cortiEnvironment: state.cortiEnvironment,
cortiTenant: state.cortiTenant,
diff --git a/src/stores/settingsStore.ts b/src/stores/settingsStore.ts
index 593ebba04..d5d11bf88 100644
--- a/src/stores/settingsStore.ts
+++ b/src/stores/settingsStore.ts
@@ -35,6 +35,7 @@ import { pickDefaultModelId } from "../models/providerDefaultModel";
// the switch store only zustand, so neither reopens the ModelRegistry cycle.
import { readCachedTinfoilModels } from "../models/tinfoilModelCache";
import { recordTinfoilModelSwitch } from "./tinfoilModelSwitchStore";
+import { MEETING_STREAMING_PROVIDER_IDS } from "../helpers/meetingTranscriptionRouting";
import {
getTranscriptionSelection,
isScreenContextAllowed,
@@ -92,7 +93,11 @@ const MEETING_TRANSCRIPTION_POLICY_CATALOG = {
// Self-hosted realtime is not implemented for Note Recording.
modes: ["openwhispr", "providers", "local"] as const,
byokProviders: modelRegistryData.transcriptionProviders
- .filter((provider) => provider.models.some((model) => model.streaming))
+ .filter(
+ (provider) =>
+ MEETING_STREAMING_PROVIDER_IDS.includes(provider.id) &&
+ provider.models.some((model) => model.streaming)
+ )
.map((provider) => provider.id),
};
diff --git a/src/types/electron.ts b/src/types/electron.ts
index 149113518..9db8e295a 100644
--- a/src/types/electron.ts
+++ b/src/types/electron.ts
@@ -38,7 +38,8 @@ export interface NoteRecordingProvider {
models: NoteRecordingProviderModel[];
}
-// Session options for the shared dictation-realtime-* channels. `provider` is
+// Session options every dictation streaming channel takes — the shared
+// dictation-realtime-* set and the per-provider ones. `provider` is
// what fetchRealtimeToken's allowlist keys on — the renderer must always send
// it (built by dictationStreamingRouting.buildStreamingSessionOptions); the
// main process defaults a missing value to "openai-realtime" for pre-1.8.4
@@ -2706,6 +2707,32 @@ declare global {
callback: (data: { audioDuration?: number; text?: string }) => void
) => () => void;
+ // Gemini Live Streaming
+ geminiStreamingWarmup?: (
+ options?: DictationRealtimeSessionOptions
+ ) => Promise<
+ { success: boolean; alreadyWarm?: boolean; error?: string } & PolicyFailureMetadata
+ >;
+ geminiStreamingStart?: (
+ options?: DictationRealtimeSessionOptions & { forceNew?: boolean }
+ ) => Promise<
+ { success: boolean; usedWarmConnection?: boolean; error?: string } & PolicyFailureMetadata
+ >;
+ geminiStreamingSend?: (audioBuffer: ArrayBuffer) => void;
+ geminiStreamingFinalize?: () => void;
+ geminiStreamingStop?: () => Promise<{
+ success: boolean;
+ text?: string;
+ model?: string;
+ audioBytesSent?: number;
+ error?: string;
+ }>;
+ geminiStreamingStatus?: () => Promise<{ isConnected: boolean; isConnecting: boolean }>;
+ onGeminiPartialTranscript?: (callback: (text: string) => void) => () => void;
+ onGeminiFinalTranscript?: (callback: (text: string) => void) => () => void;
+ onGeminiError?: (callback: (error: string) => void) => () => void;
+ onGeminiSessionEnd?: (callback: (data: { text?: string }) => void) => () => void;
+
// Corti streaming (BYOK)
cortiStreamingWarmup?: (options?: {
environment?: string;
diff --git a/test/helpers/audioManagerStreamingRouting.test.js b/test/helpers/audioManagerStreamingRouting.test.js
index ce7641dff..983cf0a62 100644
--- a/test/helpers/audioManagerStreamingRouting.test.js
+++ b/test/helpers/audioManagerStreamingRouting.test.js
@@ -59,6 +59,67 @@ test("OpenAI dictation realtime requests identify the token provider", async (t)
]);
});
+test("Gemini's live model routes onto the gemini-streaming-* channels", async (t) => {
+ const manager = await loadManager(t);
+ setSettings({
+ cloudTranscriptionProvider: "gemini",
+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",
+ geminiApiKey: "gm-test",
+ });
+ const calls = [];
+ globalThis.window.electronAPI.geminiStreamingWarmup = async (options) => {
+ calls.push(["warmup", options]);
+ return { success: true };
+ };
+ globalThis.window.electronAPI.geminiStreamingStart = async (options) => {
+ calls.push(["start", options]);
+ return { success: true };
+ };
+
+ assert.equal(manager.getStreamingProviderName(), "gemini");
+ const provider = manager.getStreamingProvider();
+ assert.equal(provider.awaitsFinalTranscript, true);
+ // Pinned to geminiLiveStreaming.js's DISCONNECT_TIMEOUT_MS: both sides
+ // measure the same 3s from audioStreamEnd.
+ assert.equal(provider.finalCeilingMs, 3000);
+
+ const options = { provider: "gemini", model: "gemini-3.5-transcribe-live", mode: "byok" };
+ await provider.warmup(options);
+ await provider.start(options);
+
+ assert.deepEqual(calls, [
+ ["warmup", options],
+ ["start", options],
+ ]);
+});
+
+test("Gemini streaming needs a streaming model, plus a key (byok) or an account (managed)", async (t) => {
+ const manager = await loadManager(t);
+ manager.sttConfig = { dictation: { mode: "streaming" } };
+ const gemini = (overrides) =>
+ setSettings({
+ cloudTranscriptionProvider: "gemini",
+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",
+ ...overrides,
+ });
+
+ gemini({ geminiApiKey: "gm-test" });
+ assert.equal(manager.shouldUseStreaming(), true);
+
+ gemini({ geminiApiKey: "" });
+ assert.equal(manager.shouldUseStreaming(), false);
+
+ gemini({ cloudTranscriptionMode: "openwhispr", isSignedIn: true });
+ assert.equal(manager.shouldUseStreaming(), true);
+
+ gemini({ cloudTranscriptionMode: "openwhispr", isSignedIn: false });
+ assert.equal(manager.shouldUseStreaming(), false);
+
+ // The batch Gemini model on the same provider stays on HTTP.
+ gemini({ cloudTranscriptionModel: "gemini-3.5-transcribe", geminiApiKey: "gm-test" });
+ assert.equal(manager.shouldUseStreaming(), false);
+});
+
test("managed OpenWhispr Cloud still respects its batch configuration", async (t) => {
const manager = await loadManager(t);
manager.sttConfig = { dictation: { mode: "batch" } };
diff --git a/test/helpers/audioManagerStreamingSettle.test.js b/test/helpers/audioManagerStreamingSettle.test.js
index 09746e6a6..cf3c58ec4 100644
--- a/test/helpers/audioManagerStreamingSettle.test.js
+++ b/test/helpers/audioManagerStreamingSettle.test.js
@@ -72,6 +72,17 @@ test("the ceiling bounds a final that never lands", async (t) => {
assert.ok(ms < CEILING_MS + 300, `overran the ceiling: ${ms}ms`);
});
+test("a provider can widen the ceiling for a slower final", async (t) => {
+ const manager = await loadManager(t);
+ const providerCeilingMs = 600;
+ manager.streamingPartialText = "never finalized";
+
+ const ms = await elapsed(() => manager.awaitStreamingTextSettled(providerCeilingMs));
+
+ assert.ok(ms >= providerCeilingMs - TIMER_SLACK_MS, `ignored the provider ceiling: ${ms}ms`);
+ assert.ok(ms < CEILING_MS, `fell back to the default ceiling: ${ms}ms`);
+});
+
test("clears its timers so a settled wait leaves no handles behind", async (t) => {
const manager = await loadManager(t);
diff --git a/test/helpers/dictationStreamingMatrix.test.js b/test/helpers/dictationStreamingMatrix.test.js
index f74955753..9eb576941 100644
--- a/test/helpers/dictationStreamingMatrix.test.js
+++ b/test/helpers/dictationStreamingMatrix.test.js
@@ -20,6 +20,7 @@ const RENDERER_STREAMING_PROVIDERS = [
"deepgram",
"assemblyai",
"openai-realtime",
+ "gemini",
"corti",
"tinfoil-realtime",
];
@@ -66,6 +67,26 @@ const RESOLUTION_MATRIX = [
{ settings: settingsWith({ cloudTranscriptionProvider: "corti" }) },
"openai-realtime",
],
+ [
+ "gemini's live model routes to its own channels",
+ {
+ settings: settingsWith({
+ cloudTranscriptionProvider: "gemini",
+ cloudTranscriptionModel: "gemini-3.5-transcribe-live",
+ }),
+ },
+ "gemini",
+ ],
+ [
+ "gemini's batch model keeps the dictation default (HTTP, not streaming)",
+ {
+ settings: settingsWith({
+ cloudTranscriptionProvider: "gemini",
+ cloudTranscriptionModel: "gemini-3.5-transcribe",
+ }),
+ },
+ "openai-realtime",
+ ],
[
"batch model in notes context defaults to deepgram",
{ settings: settingsWith({ cloudTranscriptionModel: "whisper-1" }), context: "notes" },
diff --git a/test/helpers/geminiLiveStreaming.test.js b/test/helpers/geminiLiveStreaming.test.js
new file mode 100644
index 000000000..a790ec5f8
--- /dev/null
+++ b/test/helpers/geminiLiveStreaming.test.js
@@ -0,0 +1,530 @@
+const test = require("node:test");
+const assert = require("node:assert/strict");
+const { WebSocketServer } = require("ws");
+
+const {
+ GeminiLiveStreaming,
+ buildGeminiLiveUrl,
+} = require("../../src/helpers/geminiLiveStreaming");
+
+const FRAME = Buffer.alloc(1600); // one 50ms worklet frame at 16kHz s16le (silence)
+const LOUD_FRAME = Buffer.alloc(1600, 0x20); // same frame at speech level (~0.25 RMS)
+
+// Loopback Live server. `script(socket, message)` reacts to each client
+// message; the default answers `setup` with setupComplete, like the real one.
+async function withServer(run, script) {
+ const server = new WebSocketServer({ host: "127.0.0.1", port: 0 });
+ await new Promise((resolve) => server.once("listening", resolve));
+ const received = [];
+ const urls = [];
+ const reply = script || ((socket) => socket.send(JSON.stringify({ setupComplete: {} })));
+
+ server.on("connection", (socket, request) => {
+ urls.push(request.url);
+ socket.on("message", (data) => {
+ const message = JSON.parse(data.toString());
+ received.push(message);
+ reply(socket, message, urls.length);
+ });
+ });
+
+ const streaming = new GeminiLiveStreaming();
+ streaming.buildWebSocketUrl = ({ token }) =>
+ `ws://127.0.0.1:${server.address().port}/?t=${token}`;
+
+ try {
+ await run({ streaming, received, urls });
+ } finally {
+ streaming.cleanup();
+ await new Promise((resolve) => server.close(resolve));
+ }
+}
+
+const audioFrames = (received) => received.filter((message) => message.realtimeInput?.audio);
+
+test("url: the two Live methods are auth-disjoint and pinned per mode", () => {
+ assert.equal(
+ buildGeminiLiveUrl({ mode: "byok", token: "AI-KEY" }),
+ "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=AI-KEY"
+ );
+ assert.equal(
+ buildGeminiLiveUrl({ mode: "openwhispr", token: "auth_tokens/abc" }),
+ "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=auth_tokens/abc"
+ );
+});
+
+test("setup: the custom dictionary always ships; a language only when chosen", () => {
+ const streaming = new GeminiLiveStreaming();
+
+ assert.deepEqual(streaming.buildSetupMessage({ keyterms: ["OpenWhispr", " ", ""] }), {
+ setup: {
+ model: "models/gemini-3.5-transcribe-live",
+ generationConfig: { responseModalities: ["TEXT"] },
+ inputAudioTranscription: { customVocabulary: ["OpenWhispr"] },
+ },
+ });
+
+ assert.deepEqual(
+ streaming.buildSetupMessage({ model: "gemini-next", language: "en", keyterms: [] }),
+ {
+ setup: {
+ model: "models/gemini-next",
+ generationConfig: { responseModalities: ["TEXT"] },
+ inputAudioTranscription: { languageCodes: ["en"] },
+ },
+ }
+ );
+
+ const many = streaming.buildSetupMessage({
+ keyterms: Array.from({ length: 150 }, (_, i) => `term-${i}`),
+ });
+ assert.equal(many.setup.inputAudioTranscription.customVocabulary.length, 100);
+});
+
+test("connect resolves on setupComplete, not on socket open", async () => {
+ await withServer(
+ async ({ streaming, received }) => {
+ streaming.beginConnecting();
+ // Frames from the token-fetch / handshake window are buffered, not sent.
+ assert.equal(streaming.sendAudio(FRAME), false);
+
+ const connected = streaming.connect({ token: "t", mode: "byok", keyterms: ["OpenWhispr"] });
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ assert.equal(streaming.isConnected, false, "open alone must not mark the session ready");
+ assert.equal(streaming.sendAudio(FRAME), false);
+
+ await connected;
+ assert.equal(streaming.isConnected, true);
+ assert.deepEqual(received[0].setup.inputAudioTranscription, {
+ customVocabulary: ["OpenWhispr"],
+ });
+ },
+ (socket, message) => {
+ if (!message.setup) return;
+ setTimeout(() => socket.send(JSON.stringify({ setupComplete: {} })), 60);
+ }
+ );
+});
+
+test("connect rejects when the socket drops before setupComplete", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ await assert.rejects(
+ streaming.connect({ token: "t", mode: "byok" }),
+ /closed before ready \(code: 1006\)/
+ );
+ assert.equal(streaming.isConnected, false);
+ },
+ (socket) => socket.terminate()
+ );
+});
+
+test("audio: buffered frames flush in order, then stream as base64 PCM frames", async () => {
+ await withServer(async ({ streaming, received }) => {
+ streaming.beginConnecting();
+ const first = Buffer.alloc(1600, 1);
+ const second = Buffer.alloc(1600, 2);
+ streaming.sendAudio(first);
+ streaming.sendAudio(second);
+
+ await streaming.connect({ token: "t", mode: "byok" });
+ const live = Buffer.alloc(1600, 3);
+ assert.equal(streaming.sendAudio(live), true);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+
+ const frames = audioFrames(received);
+ assert.deepEqual(
+ frames.map((frame) => frame.realtimeInput.audio.mimeType),
+ ["audio/pcm;rate=16000", "audio/pcm;rate=16000", "audio/pcm;rate=16000"]
+ );
+ assert.deepEqual(
+ frames.map((frame) => Buffer.from(frame.realtimeInput.audio.data, "base64")[0]),
+ [1, 2, 3]
+ );
+ assert.equal(streaming.audioBytesSent, 4800);
+ });
+});
+
+test("audio: a dictation shorter than the handshake still reaches the wire", async () => {
+ await withServer(async ({ streaming, received }) => {
+ // Every frame arrives before setupComplete and no frame follows it, so the
+ // only drain opportunity is readiness itself.
+ streaming.beginConnecting();
+ streaming.sendAudio(Buffer.alloc(1600, 7));
+ streaming.sendAudio(Buffer.alloc(1600, 8));
+
+ await streaming.connect({ token: "t", mode: "byok" });
+ await new Promise((resolve) => setTimeout(resolve, 30));
+
+ assert.deepEqual(
+ audioFrames(received).map(
+ (frame) => Buffer.from(frame.realtimeInput.audio.data, "base64")[0]
+ ),
+ [7, 8]
+ );
+ assert.equal(streaming.audioBytesSent, 3200);
+
+ await streaming.disconnect();
+ assert.equal(
+ received.filter((message) => message.realtimeInput?.audioStreamEnd).length,
+ 1,
+ "the turn must still be closed"
+ );
+ });
+});
+
+test("setup: a non-Gemini model id never reaches the wire", async () => {
+ const streaming = new GeminiLiveStreaming();
+ // Managed dictation carries the BYOK default, which is an OpenAI id.
+ assert.equal(
+ streaming.buildSetupMessage({ model: "gpt-4o-mini-transcribe" }).setup.model,
+ "models/gemini-3.5-transcribe-live"
+ );
+ assert.equal(
+ streaming.buildSetupMessage({ model: "gemini-3.5-transcribe-live" }).setup.model,
+ "models/gemini-3.5-transcribe-live"
+ );
+});
+
+test("audio: cold-start buffering is bounded at three seconds of PCM", async () => {
+ const streaming = new GeminiLiveStreaming();
+ streaming.beginConnecting();
+ for (let i = 0; i < 100; i++) streaming.sendAudio(Buffer.alloc(1600));
+ assert.equal(streaming.coldStartBufferSize, 96000);
+});
+
+test("characterization: the worklet's flush sentinel is never sent as audio", async () => {
+ await withServer(async ({ streaming, received }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ assert.equal(streaming.sendAudio(Buffer.from("flushed")), false);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ assert.equal(audioFrames(received).length, 0);
+ assert.equal(streaming.audioBytesSent, 0);
+ });
+});
+
+test("transcripts: partials replace, finals accumulate", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ const partials = [];
+ const finals = [];
+ streaming.onPartialTranscript = (text) => partials.push(text);
+ streaming.onFinalTranscript = (text) => finals.push(text);
+
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+ await new Promise((resolve) => setTimeout(resolve, 60));
+
+ // Partials are revised, not appended to — casing included.
+ assert.deepEqual(partials, ["The quick brown", "the quick brown fox"]);
+ assert.deepEqual(finals, [
+ "The quick brown fox.",
+ "The quick brown fox. OpenWhispr transcription test.",
+ ]);
+ assert.equal(
+ streaming.getFullTranscript(),
+ "The quick brown fox. OpenWhispr transcription test."
+ );
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (!message.realtimeInput?.audio) return;
+ for (const text of ["The quick brown", "the quick brown fox"]) {
+ socket.send(JSON.stringify({ serverContent: { interimInputTranscription: { text } } }));
+ }
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "The quick brown fox." } } })
+ );
+ socket.send(
+ JSON.stringify({
+ serverContent: { inputTranscription: { text: "OpenWhispr transcription test." } },
+ })
+ );
+ }
+ );
+});
+
+test("disconnect sends audioStreamEnd once and waits for the end of the turn", async () => {
+ await withServer(
+ async ({ streaming, received }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+
+ // The stop sequence calls finalize(), waits for the text to settle, then
+ // stops — so disconnect must not send audioStreamEnd a second time.
+ assert.equal(streaming.finalize(), true);
+ assert.equal(streaming.finalize(), false, "audioStreamEnd must not be sent twice");
+ await new Promise((resolve) => setTimeout(resolve, 40));
+
+ const result = await streaming.disconnect(true);
+ assert.equal(result.text, "The quick brown fox.");
+ assert.equal(received.filter((m) => m.realtimeInput?.audioStreamEnd).length, 1);
+ // The transcript is handed over, not kept for the next session.
+ assert.equal(streaming.getFullTranscript(), "");
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (!message.realtimeInput?.audioStreamEnd) return;
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "The quick brown fox." } } })
+ );
+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));
+ }
+ );
+});
+
+test("disconnect still waits for the final when finalize() already closed the turn", async () => {
+ await withServer(
+ async ({ streaming, received }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+
+ // The stop sequence finalizes, then stops as soon as its own wait gives
+ // up — disconnect must not treat the sent audioStreamEnd as "answered".
+ assert.equal(streaming.finalize(), true);
+ const result = await streaming.disconnect(true);
+
+ assert.equal(result.text, "the slow final");
+ assert.equal(received.filter((m) => m.realtimeInput?.audioStreamEnd).length, 1);
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (!message.realtimeInput?.audioStreamEnd) return;
+ setTimeout(() => {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "the slow final" } } })
+ );
+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));
+ }, 120);
+ }
+ );
+});
+
+test("disconnect spends only what is left of the budget since audioStreamEnd", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+
+ assert.equal(streaming.finalize(), true);
+ // The renderer already waited out the whole budget on its side.
+ streaming._audioStreamEndSentAt -= 10000;
+ const start = Date.now();
+ const result = await streaming.disconnect(true);
+
+ assert.equal(result.text, "already here");
+ assert.ok(Date.now() - start < 500, "must not re-spend the budget");
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (message.realtimeInput?.audio) {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "already here" } } })
+ );
+ }
+ }
+ );
+});
+
+test("a turn the server closed before the hotkey release answers the stop immediately", async () => {
+ await withServer(
+ async ({ streaming, received }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ // Trailing silence let the server finalize on its own; nothing more will come.
+ assert.equal(streaming._turnEnded, true);
+ // Room-noise frames keep arriving until the hotkey is released; they are not speech.
+ streaming.sendAudio(FRAME);
+ assert.equal(streaming._turnEnded, true);
+
+ const startedAt = Date.now();
+ assert.deepEqual(await streaming.disconnect(true), { text: "already final" });
+ assert.ok(Date.now() - startedAt < 500, "must not wait out the budget for a second turn end");
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ assert.equal(
+ received.filter((message) => message.realtimeInput?.audioStreamEnd).length,
+ 1,
+ "the stream is still closed cleanly"
+ );
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (message.realtimeInput?.audio) {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "already final" } } })
+ );
+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));
+ }
+ }
+ );
+});
+
+test("speech after a closed turn reopens it, so the stop waits for the last turn", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(Buffer.alloc(1600, 1));
+ await new Promise((resolve) => setTimeout(resolve, 30));
+ assert.equal(streaming._turnEnded, true);
+
+ // The user keeps talking: a speech-level frame reopens the turn and the stop must wait.
+ streaming.sendAudio(LOUD_FRAME);
+ assert.equal(streaming._turnEnded, false);
+
+ assert.deepEqual(await streaming.disconnect(true), { text: "first segment last words" });
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ const audio = message.realtimeInput?.audio;
+ if (audio && Buffer.from(audio.data, "base64")[0] !== LOUD_FRAME[0]) {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "first segment" } } })
+ );
+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));
+ }
+ if (message.realtimeInput?.audioStreamEnd) {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "last words" } } })
+ );
+ socket.send(JSON.stringify({ serverContent: { generationComplete: true } }));
+ }
+ }
+ );
+});
+
+test("disconnect returns the accumulated text when the turn never ends cleanly", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+ await new Promise((resolve) => setTimeout(resolve, 30));
+
+ // No generationComplete: the socket dies instead, which must release the
+ // finalization wait rather than burn its whole budget.
+ assert.deepEqual(await streaming.disconnect(true), { text: "partial turn" });
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (message.realtimeInput?.audio) {
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "partial turn" } } })
+ );
+ }
+ if (message.realtimeInput?.audioStreamEnd) socket.close(1000, "");
+ }
+ );
+});
+
+// 1007 is a malformed credential, 1008 a revoked or deleted one. The reason is
+// truncated by the server and is never part of the user-facing message.
+for (const [code, reason] of [
+ [1007, "API key not valid. Please pass a valid API key."],
+ [1008, "Requested entity was not found."],
+]) {
+ test(`errors: a ${code} close frame becomes the coded, fixable key error`, async () => {
+ await withServer(
+ async ({ streaming }) => {
+ await assert.rejects(
+ streaming.connect({ token: "nope", mode: "byok" }),
+ (err) =>
+ err.code === "INVALID_KEY" &&
+ /Check your key in Settings/.test(err.message) &&
+ !err.message.includes(reason)
+ );
+ },
+ (socket) => socket.close(code, reason)
+ );
+ });
+}
+
+test("errors: a spent managed token is re-minted once and the session survives", async () => {
+ await withServer(
+ async ({ streaming, urls }) => {
+ await streaming.connect({
+ token: "auth_tokens/spent",
+ mode: "openwhispr",
+ refreshToken: async () => "auth_tokens/fresh",
+ });
+ assert.equal(streaming.isConnected, true);
+ assert.deepEqual(urls, ["/?t=auth_tokens/spent", "/?t=auth_tokens/fresh"]);
+ },
+ (socket, _message, connectionCount) => {
+ if (connectionCount === 1) {
+ socket.close(1011, "Token has been used too many times");
+ return;
+ }
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ }
+ );
+});
+
+test("errors: byok never retries a 1011 close (there is no token to re-mint)", async () => {
+ await withServer(
+ async ({ streaming, urls }) => {
+ await assert.rejects(
+ streaming.connect({
+ token: "key",
+ mode: "byok",
+ refreshToken: async () => "unused",
+ }),
+ (err) => err.code === "AUTH_EXPIRED"
+ );
+ assert.equal(urls.length, 1);
+ },
+ (socket) => socket.close(1011, "Token has expired")
+ );
+});
+
+test("an unexpected close hands the transcript over and reports the loss once", async () => {
+ await withServer(
+ async ({ streaming }) => {
+ const errors = [];
+ const sessionEnds = [];
+ streaming.onError = (err) => errors.push(err.message);
+ streaming.onSessionEnd = (data) => sessionEnds.push(data);
+
+ await streaming.connect({ token: "t", mode: "byok" });
+ streaming.sendAudio(FRAME);
+ await new Promise((resolve) => setTimeout(resolve, 60));
+
+ assert.deepEqual(sessionEnds, [{ text: "half a sentence" }]);
+ // Distinct from the pre-ready failure, and without the raw server reason.
+ assert.deepEqual(errors, ["Connection lost (code: 1011)"]);
+ assert.equal(streaming.isConnected, false);
+ },
+ (socket, message) => {
+ if (message.setup) {
+ socket.send(JSON.stringify({ setupComplete: {} }));
+ return;
+ }
+ if (!message.realtimeInput?.audio) return;
+ socket.send(
+ JSON.stringify({ serverContent: { inputTranscription: { text: "half a sentence" } } })
+ );
+ setTimeout(() => socket.close(1011, "Token has expired"), 10);
+ }
+ );
+});
diff --git a/test/helpers/meetingStreamingProviders.test.js b/test/helpers/meetingStreamingProviders.test.js
index 1734905c9..0c82945ab 100644
--- a/test/helpers/meetingStreamingProviders.test.js
+++ b/test/helpers/meetingStreamingProviders.test.js
@@ -34,6 +34,34 @@ test("allow-list accepts tinfoil-realtime and local", async () => {
assert.equal(ALLOWED_MEETING_PROVIDERS.has("local"), true);
});
+test("every provider note recording offers is one the main process accepts", async () => {
+ const { ALLOWED_MEETING_PROVIDERS } = await load();
+ const { MEETING_STREAMING_PROVIDER_IDS } =
+ await import("../../src/helpers/meetingTranscriptionRouting.js");
+
+ // Note recording derives its streaming provider id by concatenation
+ // (`${provider.id}-realtime`), so an offered catalog id with no client class
+ // is rejected at meeting-transcription-prepare with no user-visible message.
+ for (const id of MEETING_STREAMING_PROVIDER_IDS) {
+ assert.equal(
+ ALLOWED_MEETING_PROVIDERS.has(`${id}-realtime`),
+ true,
+ `${id} is offered for note recording but ${id}-realtime has no streaming client`
+ );
+ }
+});
+
+test("a dictation-only streaming provider is not offered for note recording", async () => {
+ const { getStreamingTranscriptionProviders, getMeetingStreamingTranscriptionProviders } =
+ await import("../../src/models/ModelRegistry.ts");
+
+ const streamingIds = getStreamingTranscriptionProviders().map((provider) => provider.id);
+ const meetingIds = getMeetingStreamingTranscriptionProviders().map((provider) => provider.id);
+
+ assert.ok(streamingIds.includes("gemini"), "gemini ships a streaming dictation model");
+ assert.equal(meetingIds.includes("gemini"), false);
+});
+
test("allow-list rejects unknown and batch-only providers", async () => {
const { ALLOWED_MEETING_PROVIDERS } = await load();
diff --git a/test/helpers/realtimeTokenProviders.test.js b/test/helpers/realtimeTokenProviders.test.js
index 86bd9beee..55ece63ae 100644
--- a/test/helpers/realtimeTokenProviders.test.js
+++ b/test/helpers/realtimeTokenProviders.test.js
@@ -15,6 +15,7 @@ const deps = (overrides = {}) => ({
getTinfoilKey: () => "tk-tinfoil",
getDeepgramKey: () => "dg-key",
getAssemblyAIKey: () => "aai-key",
+ getGeminiKey: () => "gm-key",
...overrides.environmentManager,
},
proxyFetch: overrides.proxyFetch || (async () => jsonResponse(200, { token: "aai-token" })),
@@ -135,6 +136,40 @@ test("deepgram byok duplicates the key; cloud mints per stream", async () => {
);
});
+test("gemini byok duplicates the raw key; cloud mints one token per stream", async () => {
+ const { fetchRealtimeTokenForProvider } = await load();
+ // The raw key opens the Live socket itself and survives any number of
+ // handshakes, so both streams share it.
+ assert.deepEqual(
+ await fetchRealtimeTokenForProvider(
+ "gemini-realtime",
+ deps(),
+ { mode: "byok" },
+ { streams: 2 }
+ ),
+ ["gm-key", "gm-key"]
+ );
+ // Managed tokens are minted with uses:1 — a shared token would be rejected
+ // at the second handshake with close code 1011.
+ let mints = 0;
+ const cloudDeps = deps({ postServerToken: async () => ({ token: `gm-${++mints}` }) });
+ assert.deepEqual(
+ await fetchRealtimeTokenForProvider(
+ "gemini-realtime",
+ cloudDeps,
+ { mode: "openwhispr" },
+ { streams: 2 }
+ ),
+ ["gm-1", "gm-2"]
+ );
+ await assert.rejects(
+ fetchRealtimeTokenForProvider("gemini-realtime", deps({ postServerToken: async () => ({}) }), {
+ mode: "openwhispr",
+ }),
+ /No Gemini token received/
+ );
+});
+
test("corti mints one token and shares it across both streams", async () => {
const { fetchRealtimeTokenForProvider } = await load();
let mints = 0;
@@ -148,8 +183,18 @@ test("corti mints one token and shares it across both streams", async () => {
test("missing byok keys throw configuration errors, not token errors", async () => {
const { fetchRealtimeTokenForProvider } = await load();
- const empty = { getOpenAIKey: () => "", getDeepgramKey: () => "", getAssemblyAIKey: () => "" };
- for (const provider of ["openai-realtime", "deepgram-realtime", "assemblyai-realtime"]) {
+ const empty = {
+ getOpenAIKey: () => "",
+ getDeepgramKey: () => "",
+ getAssemblyAIKey: () => "",
+ getGeminiKey: () => "",
+ };
+ for (const provider of [
+ "openai-realtime",
+ "deepgram-realtime",
+ "assemblyai-realtime",
+ "gemini-realtime",
+ ]) {
await assert.rejects(
fetchRealtimeTokenForProvider(provider, deps({ environmentManager: empty }), {
mode: "byok",
@@ -163,10 +208,12 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+l
const { fetchRealtimeTokenForProvider } = await load();
const wire = [];
const cloudDeps = deps({
- postServerToken: async (path, body) => {
+ // `body = {}` mirrors ipcHandlers' postServerToken default, so an entry that
+ // posts no body at all is recorded as the `{}` it really sends.
+ postServerToken: async (path, body = {}) => {
// JSON.stringify drops undefined keys exactly like the real fetch does.
wire.push({ path, json: JSON.stringify(body) });
- return { clientSecret: "cs-single", clientSecrets: ["cs-a", "cs-b"] };
+ return { clientSecret: "cs-single", clientSecrets: ["cs-a", "cs-b"], token: "gm-token" };
},
});
@@ -188,6 +235,9 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+l
{ streams: 1 }
);
+ // Gemini's endpoint reads nothing from the request; it must post a bare body.
+ await fetchRealtimeTokenForProvider("gemini-realtime", cloudDeps, { mode: "openwhispr" });
+
assert.deepEqual(wire, [
{ path: "/api/openai-realtime-token", json: '{"streams":1}' },
{
@@ -195,5 +245,6 @@ test('wire bodies: dictation posts the bare {"streams":1}; meetings post model+l
json: '{"model":"gpt-4o-mini-transcribe","language":"en","streams":2}',
},
{ path: "/api/openai-realtime-token", json: '{"model":"gpt-4o-mini-transcribe","streams":1}' },
+ { path: "/api/gemini-live-token", json: "{}" },
]);
});