Skip to content

Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.

fix(voice-agent): keep screen context within budget and never fail on it

Gabriel Stein34b9250db508
diff --git a/CLAUDE.md b/CLAUDE.md
index 828cb21d8..c47be439e 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -629,15 +629,16 @@ A dedicated global hotkey that starts a dictation whose transcript is sent strai
When "Share screen context" is enabled (Settings → AI Models → Voice Agent → Screen Context, store key `voiceAgentScreenContext`, default off), each voice-agent recording start captures the display the cursor is on and attaches it to the agent request as a base64 JPEG (long edge ≤ 1568 px).
- Capture: `src/helpers/screenContextCapture.js` (main process; `screen.getCursorScreenPoint` + `desktopCapturer`). macOS requires the Screen Recording TCC permission (`useScreenRecordingPermission` hook + `PermissionCard` in `DictationAgentSettings`); Windows/Linux X11 need no permission; Linux Wayland is unsupported (capture silently skipped)
+- Encoding: `encodeWithinBudget()` walks a JPEG quality ladder (82 → 70 → 55), then falls back to a 1024 px resize, keeping the payload under `MAX_ENCODED_BYTES` (1.5 MB ≈ 2.05M base64 chars, inside the API's 2.8M cap). Each rung re-encodes the original bitmap, so quality drops never compound. A screen that still won't fit is dropped
- Trigger: `useAudioRecording.js` calls `audioManager.beginScreenContextCapture()` at voice-agent start (fire-and-forget); `consumeScreenContext()` (3s guard) is awaited when the reasoning route is built. The dictation overlay gets `setContentProtection` while the setting is on so the pill never appears in captures
- Routing: `resolveAgentImageTarget()` (`src/helpers/dictationRouting.js`) decides attach/drop. An optional vision override (`dictationAgentVision` inference scope, gated by `useDictationAgentVisionModel`, cloud/BYOK modes only) routes screenshot-carrying requests to a dedicated model; otherwise the base model gets the image only when its provider client is image-wired (`supportsImages` on the `InferenceProvider`) and the model has `supportsVision` in `modelRegistryData.json` (cloud mode defers to the server). Dropping the image never fails the dictation
- Prompt: `appendScreenContextSuffix()` adds the `screenContextSuffix` prompts key only when an image is attached
- Image-wired clients: `openai` (Responses `input_image` / Chat `image_url`, also `custom`/`openrouter`), `anthropic` (base64 content block via `process-anthropic-reasoning`), `gemini` (`inlineData`), `openwhispr` (forwards `screenContext` + `promptMode: "agent"` to `/api/reason`; the server routes to its `REASONING_VISION_*` model chain)
- IPC: `capture-screen-context`, `check-screen-recording-access`, `request-screen-recording-access`, `open-screen-recording-settings`, `screen-context-set-enabled`
- Privacy: screenshots live only in renderer memory for one request — never written to disk, stored in history, or logged (loggers emit `hasScreenContext` booleans only)
-- Failure UX: any agent-route reasoning failure now surfaces an "Agent Unavailable" toast (`AGENT_REASONING_FAILED`) while still pasting the raw transcript
+- Failure UX: a rejected screenshot is retried once text-only from `processWithReasoningModel()` (rebuilding the prompt without the screen-context suffix) and reported via a non-destructive `SCREEN_CONTEXT_SKIPPED` toast, so an image problem never costs the user their command. Only if that retry also fails does the "Agent Unavailable" toast (`AGENT_REASONING_FAILED`) fire and the raw transcript get pasted
-**Tests**: `test/helpers/dictationRouting.test.js` (run with `node --test`)
+**Tests**: `test/helpers/dictationRouting.test.js`, `test/helpers/screenContextCapture.test.js` (run with `node --test`)
## Development Guidelines
diff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.js
index e25789c62..543ff2496 100644
--- a/src/helpers/audioManager.js
+++ b/src/helpers/audioManager.js
@@ -114,6 +114,17 @@ function getEffectiveRetentionPreferences() {
const providerSupportsImages = (providerId) =>
!!(providerId && PROVIDER_REGISTRY[providerId]?.supportsImages);
+// Shared by the agent route and its text-only retry, which needs the prompt
+// without the screen-context suffix.
+function dictationAgentPrompt(settings, agentName) {
+ return resolvePrompt("dictationAgent", {
+ agentName,
+ language: settings.preferredLanguage,
+ customDictionary: getDictionaryHintWords(settings),
+ uiLanguage: settings.uiLanguage,
+ });
+}
+
function dictationAgentReachable(settings) {
return resolveDictationAgentInference(settings, { isCloudAgent: isCloudDictationAgentMode() })
.reachable;
@@ -193,20 +204,16 @@ function resolveReasoningRoute(
});
const target = useVisionOverride ? vision : agent;
- let systemPrompt = resolvePrompt("dictationAgent", {
- agentName,
- language: settings.preferredLanguage,
- customDictionary: getDictionaryHintWords(settings),
- uiLanguage: settings.uiLanguage,
- });
- if (attach) systemPrompt = appendScreenContextSuffix(systemPrompt, settings.uiLanguage);
+ const systemPrompt = dictationAgentPrompt(settings, agentName);
return {
kind: "agent",
model: target.model,
config: {
...target.config,
- systemPrompt,
+ systemPrompt: attach
+ ? appendScreenContextSuffix(systemPrompt, settings.uiLanguage)
+ : systemPrompt,
...(attach ? { screenContext } : {}),
},
};
@@ -671,6 +678,16 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
});
}
+ // The command still ran, so this is a downgrade notice rather than a failure.
+ _notifyScreenContextSkipped() {
+ this.onError?.({
+ code: "SCREEN_CONTEXT_SKIPPED",
+ title: "Screen Context Skipped",
+ messageKey: "hooks.audioRecording.errorDescriptions.screenContextSkipped",
+ variant: "default",
+ });
+ }
+
setContext(context) {
this.context = context;
}
@@ -2098,6 +2115,20 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);
stack: error.stack,
});
+ // A screenshot the model or transport rejects must not cost the user
+ // their command — rerun it text-only. Only the agent route attaches one,
+ // so rebuilding its prompt also drops the screen-context instructions
+ // that would otherwise reference an image we are no longer sending.
+ if (config?.screenContext) {
+ const { screenContext, ...textOnlyConfig } = config;
+ const result = await ReasoningService.processText(text, model, agentName, {
+ ...textOnlyConfig,
+ systemPrompt: dictationAgentPrompt(getSettings(), agentName),
+ });
+ this._notifyScreenContextSkipped();
+ return result;
+ }
+
throw error;
}
}
diff --git a/src/helpers/screenContextCapture.js b/src/helpers/screenContextCapture.js
index a71822054..41729cdea 100644
--- a/src/helpers/screenContextCapture.js
+++ b/src/helpers/screenContextCapture.js
@@ -4,7 +4,14 @@ const debugLogger = require("./debugLogger");
// Vision models downsample images past ~1.5k px on the long edge; capturing
// larger only inflates the payload without adding model-visible detail.
const MAX_EDGE_PX = 1568;
-const JPEG_QUALITY = 75;
+// A 1568px screenshot lands well under the cap at this quality, so spend the
+// headroom on legible on-screen text; the ladder below covers the rare
+// near-incompressible screen (video, noise) that doesn't fit.
+const QUALITY_LADDER = [82, 70, 55];
+// Keeps the base64 payload (~1.37x) inside the API's 2.8M character limit
+// with room for the transcript, prompt and dictionary.
+const MAX_ENCODED_BYTES = 1_500_000;
+const FALLBACK_EDGE_PX = 1024;
// Wayland routes desktopCapturer through the xdg-desktop-portal picker (a
// dialog per capture) and cursor coordinates are unreliable there, so screen
@@ -25,6 +32,19 @@ function getAccessStatus() {
return "granted";
}
+// Encodes from the same source bitmap at each step, so lowering quality never
+// compounds artifacts from an earlier pass.
+function encodeWithinBudget(image) {
+ for (const quality of QUALITY_LADDER) {
+ const encoded = image.toJPEG(quality);
+ if (encoded.length <= MAX_ENCODED_BYTES) return encoded;
+ }
+
+ const resized = image.resize({ width: FALLBACK_EDGE_PX, quality: "good" });
+ const smallest = resized.toJPEG(QUALITY_LADDER.at(-1));
+ return smallest.length <= MAX_ENCODED_BYTES ? smallest : null;
+}
+
// Returns null on any failure — a screenshot must never break the dictation
// it accompanies.
async function captureCursorDisplay() {
@@ -44,10 +64,13 @@ async function captureCursorDisplay() {
const source = sources.find((s) => s.display_id === String(display.id)) || sources[0];
if (!source || source.thumbnail.isEmpty()) return null;
- return {
- mediaType: "image/jpeg",
- data: source.thumbnail.toJPEG(JPEG_QUALITY).toString("base64"),
- };
+ const encoded = encodeWithinBudget(source.thumbnail);
+ if (!encoded) {
+ debugLogger.warn("Screen context too large to send", {}, "screenContext");
+ return null;
+ }
+
+ return { mediaType: "image/jpeg", data: encoded.toString("base64") };
} catch (error) {
debugLogger.warn("Screen context capture failed", { error: error.message }, "screenContext");
return null;
diff --git a/src/hooks/useAudioRecording.js b/src/hooks/useAudioRecording.js
index 87825c095..74f19e185 100644
--- a/src/hooks/useAudioRecording.js
+++ b/src/hooks/useAudioRecording.js
@@ -199,7 +199,7 @@ export const useAudioRecording = (toast, options = {}) => {
toast({
title,
description,
- variant: "destructive",
+ variant: error.variant || "destructive",
duration: error.code === "AUTH_EXPIRED" ? 8000 : undefined,
});
if (getSettings().pauseMediaOnDictation) {
diff --git a/src/locales/de/translation.json b/src/locales/de/translation.json
index 212a8c50b..261497325 100644
--- a/src/locales/de/translation.json
+++ b/src/locales/de/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Sitzung abgelaufen",
"offline": "Sie sind offline",
"agentUnavailable": "Agent nicht verfügbar",
+ "screenContextSkipped": "Bildschirmkontext übersprungen",
"dailyLimitReached": "Tägliches Limit erreicht",
"providerRateLimited": "Anbieter-Ratenlimit erreicht"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "Du hast dein heutiges Transkriptionslimit erreicht. Versuch es morgen erneut oder upgrade deinen Plan.",
"providerRateLimited": "Dein Transkriptionsanbieter hat die Anfrage begrenzt. Warte kurz und versuch es erneut.",
"agentReasoningFailed": "Ihr Sprachagent konnte den Befehl nicht verarbeiten – stattdessen wurde das unbearbeitete Transkript eingefügt.",
- "providerRateLimited": "Dein Transkriptionsanbieter hat die Anfrage begrenzt. Warte kurz und versuch es erneut."
+ "screenContextSkipped": "Der Screenshot konnte nicht gesendet werden, daher lief Ihr Befehl ohne Bildschirmkontext."
},
"translationFallback": {
"unreachableTitle": "Übersetzung nicht verfügbar",
@@ -3089,7 +3090,7 @@
},
"dictationAgent": {
"enabled": "Voice Agent aktivieren",
- "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen. Unten konfigurierbar.",
+ "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Markierter Text wird an das konfigurierte Modell gesendet und direkt bearbeitet. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen.",
"screenContext": {
"title": "Bildschirmkontext",
"description": "Hängt einen Screenshot Ihres aktiven Bildschirms an Sprachagent-Befehle an.",
@@ -3101,7 +3102,6 @@
"permissionTitle": "Bildschirmaufnahme",
"permissionDescription": "macOS benötigt die Berechtigung zur Bildschirmaufnahme, um Ihren Bildschirm zu erfassen."
},
- "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Markierter Text wird an das konfigurierte Modell gesendet und direkt bearbeitet. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen.",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Den von OpenWhispr verwalteten Agenten nutzen, kein API-Schlüssel nötig.",
diff --git a/src/locales/en/translation.json b/src/locales/en/translation.json
index a59a03469..989c39c9e 100644
--- a/src/locales/en/translation.json
+++ b/src/locales/en/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Session Expired",
"offline": "You're Offline",
"agentUnavailable": "Agent Unavailable",
+ "screenContextSkipped": "Screen Context Skipped",
"dailyLimitReached": "Daily Limit Reached",
"providerRateLimited": "Provider Rate Limited"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "You've reached today's transcription limit. Try again tomorrow or upgrade your plan.",
"providerRateLimited": "Your transcription provider rate-limited the request. Wait a moment and try again.",
"agentReasoningFailed": "Your voice agent couldn't process the command — the raw transcript was inserted instead.",
- "providerRateLimited": "Your transcription provider rate-limited the request. Wait a moment and try again."
+ "screenContextSkipped": "The screenshot couldn't be sent, so your command ran without screen context."
},
"translationFallback": {
"unreachableTitle": "Translation unavailable",
@@ -3089,7 +3090,7 @@
},
"dictationAgent": {
"enabled": "Enable voice agent",
- "enabledDescription": "Dictate instructions to your agent. Activate by saying \"{{agentName}}\". Configurable below.",
+ "enabledDescription": "Dictate instructions to your agent. Highlighted text is sent to the configured model and edited in place. Activate by saying \"{{agentName}}\".",
"screenContext": {
"title": "Screen Context",
"description": "Attach a screenshot of your active screen to voice agent commands.",
@@ -3101,7 +3102,6 @@
"permissionTitle": "Screen Recording",
"permissionDescription": "macOS requires Screen Recording permission to capture your screen."
},
- "enabledDescription": "Dictate instructions to your agent. Highlighted text is sent to the configured model and edited in place. Activate by saying \"{{agentName}}\".",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Use the OpenWhispr-managed agent — no API key needed.",
diff --git a/src/locales/es/translation.json b/src/locales/es/translation.json
index 8eec93370..26861bb4b 100644
--- a/src/locales/es/translation.json
+++ b/src/locales/es/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Sesión expirada",
"offline": "Sin conexión",
"agentUnavailable": "Agente no disponible",
+ "screenContextSkipped": "Contexto de pantalla omitido",
"dailyLimitReached": "Límite diario alcanzado",
"providerRateLimited": "Límite de tasa del proveedor"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "Has alcanzado el límite de transcripción de hoy. Inténtalo mañana o mejora tu plan.",
"providerRateLimited": "Tu proveedor de transcripción limitó la solicitud. Espera un momento e inténtalo de nuevo.",
"agentReasoningFailed": "Tu agente de voz no pudo procesar el comando; se insertó la transcripción sin procesar.",
- "providerRateLimited": "Tu proveedor de transcripción limitó la solicitud. Espera un momento e inténtalo de nuevo."
+ "screenContextSkipped": "No se pudo enviar la captura, así que tu comando se ejecutó sin contexto de pantalla."
},
"translationFallback": {
"unreachableTitle": "Traducción no disponible",
@@ -2989,7 +2990,7 @@
},
"dictationAgent": {
"enabled": "Activar agente de voz",
- "enabledDescription": "Dicta instrucciones a tu agente. Actívalo diciendo \"{{agentName}}\". Configurable más abajo.",
+ "enabledDescription": "Dicta instrucciones a tu agente. El texto resaltado se envía al modelo configurado y se edita en su lugar. Actívalo diciendo \"{{agentName}}\".",
"screenContext": {
"title": "Contexto de pantalla",
"description": "Adjunta una captura de tu pantalla activa a los comandos del agente de voz.",
@@ -3001,7 +3002,6 @@
"permissionTitle": "Grabación de pantalla",
"permissionDescription": "macOS requiere permiso de grabación de pantalla para capturar tu pantalla."
},
- "enabledDescription": "Dicta instrucciones a tu agente. El texto resaltado se envía al modelo configurado y se edita en su lugar. Actívalo diciendo \"{{agentName}}\".",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Usa el agente gestionado por OpenWhispr, sin necesidad de clave API.",
diff --git a/src/locales/fr/translation.json b/src/locales/fr/translation.json
index 6deec7bd8..498fc53ae 100644
--- a/src/locales/fr/translation.json
+++ b/src/locales/fr/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Session expirée",
"offline": "Vous êtes hors ligne",
"agentUnavailable": "Agent indisponible",
+ "screenContextSkipped": "Contexte d'écran ignoré",
"dailyLimitReached": "Limite quotidienne atteinte",
"providerRateLimited": "Limite de débit du fournisseur"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "Vous avez atteint votre limite de transcription du jour. Réessayez demain ou améliorez votre forfait.",
"providerRateLimited": "Votre fournisseur de transcription a limité la requête. Patientez un instant et réessayez.",
"agentReasoningFailed": "Votre agent vocal n'a pas pu traiter la commande ; la transcription brute a été insérée à la place.",
- "providerRateLimited": "Votre fournisseur de transcription a limité la requête. Patientez un instant et réessayez."
+ "screenContextSkipped": "La capture n'a pas pu être envoyée ; votre commande a été exécutée sans contexte d'écran."
},
"translationFallback": {
"unreachableTitle": "Traduction indisponible",
@@ -3089,7 +3090,7 @@
},
"dictationAgent": {
"enabled": "Activer l'agent vocal",
- "enabledDescription": "Dictez des instructions à votre agent. Activez-le en disant « \"{{agentName}}\" ». Configurable ci-dessous.",
+ "enabledDescription": "Dictez des instructions à votre agent. Le texte surligné est envoyé au modèle configuré et modifié sur place. Activez-le en disant « \"{{agentName}}\" ».",
"screenContext": {
"title": "Contexte d'écran",
"description": "Joint une capture de votre écran actif aux commandes de l'agent vocal.",
@@ -3101,7 +3102,6 @@
"permissionTitle": "Enregistrement de l'écran",
"permissionDescription": "macOS exige l'autorisation d'enregistrement de l'écran pour capturer votre écran."
},
- "enabledDescription": "Dictez des instructions à votre agent. Le texte surligné est envoyé au modèle configuré et modifié sur place. Activez-le en disant « \"{{agentName}}\" ».",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Utilisez l'agent géré par OpenWhispr, sans clé API.",
diff --git a/src/locales/it/translation.json b/src/locales/it/translation.json
index ef8aa91fe..e35283feb 100644
--- a/src/locales/it/translation.json
+++ b/src/locales/it/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Sessione scaduta",
"offline": "Sei offline",
"agentUnavailable": "Agente non disponibile",
+ "screenContextSkipped": "Contesto dello schermo ignorato",
"dailyLimitReached": "Limite giornaliero raggiunto",
"providerRateLimited": "Limite di velocità del provider"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "Hai raggiunto il limite di trascrizioni di oggi. Riprova domani o aggiorna il tuo piano.",
"providerRateLimited": "Il tuo provider di trascrizione ha limitato la richiesta. Attendi un momento e riprova.",
"agentReasoningFailed": "L'agente vocale non è riuscito a elaborare il comando; è stata inserita la trascrizione grezza.",
- "providerRateLimited": "Il tuo provider di trascrizione ha limitato la richiesta. Attendi un momento e riprova."
+ "screenContextSkipped": "Non è stato possibile inviare lo screenshot, quindi il comando è stato eseguito senza contesto dello schermo."
},
"translationFallback": {
"unreachableTitle": "Traduzione non disponibile",
@@ -2941,7 +2942,7 @@
},
"dictationAgent": {
"enabled": "Attiva agente vocale",
- "enabledDescription": "Detta istruzioni al tuo agente. Attivalo dicendo \"{{agentName}}\". Configurabile qui sotto.",
+ "enabledDescription": "Detta istruzioni al tuo agente. Il testo evidenziato viene inviato al modello configurato e modificato sul posto. Attivalo dicendo \"{{agentName}}\".",
"screenContext": {
"title": "Contesto dello schermo",
"description": "Allega uno screenshot dello schermo attivo ai comandi dell'agente vocale.",
@@ -2953,7 +2954,6 @@
"permissionTitle": "Registrazione schermo",
"permissionDescription": "macOS richiede l'autorizzazione alla registrazione dello schermo per catturare lo schermo."
},
- "enabledDescription": "Detta istruzioni al tuo agente. Il testo evidenziato viene inviato al modello configurato e modificato sul posto. Attivalo dicendo \"{{agentName}}\".",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Usa l'agente gestito da OpenWhispr, senza bisogno di chiave API.",
diff --git a/src/locales/ja/translation.json b/src/locales/ja/translation.json
index d7a3c654b..be2d39ced 100644
--- a/src/locales/ja/translation.json
+++ b/src/locales/ja/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "セッション期限切れ",
"offline": "オフラインです",
"agentUnavailable": "エージェントを利用できません",
+ "screenContextSkipped": "画面コンテキストをスキップしました",
"dailyLimitReached": "1 日の上限に達しました",
"providerRateLimited": "プロバイダーのレート制限"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "本日の文字起こし上限に達しました。明日もう一度お試しいただくか、プランをアップグレードしてください。",
"providerRateLimited": "文字起こしプロバイダーがリクエストを制限しました。しばらく待ってからもう一度お試しください。",
"agentReasoningFailed": "ボイスエージェントがコマンドを処理できなかったため、未処理の文字起こしを挿入しました。",
- "providerRateLimited": "文字起こしプロバイダーがリクエストを制限しました。しばらく待ってからもう一度お試しください。"
+ "screenContextSkipped": "スクリーンショットを送信できなかったため、画面コンテキストなしでコマンドを実行しました。"
},
"translationFallback": {
"unreachableTitle": "翻訳を利用できません",
@@ -2941,7 +2942,7 @@
},
"dictationAgent": {
"enabled": "音声エージェントを有効にする",
- "enabledDescription": "エージェントに指示を口述できます。「{{agentName}}」と話しかけて起動します。設定は以下で行えます。",
+ "enabledDescription": "エージェントに指示を口述できます。選択中のテキストは設定済みのモデルに送信され、その場で編集されます。「{{agentName}}」と話しかけて起動します。",
"screenContext": {
"title": "画面コンテキスト",
"description": "ボイスエージェントのコマンドに、作業中の画面のスクリーンショットを添付します。",
@@ -2953,7 +2954,6 @@
"permissionTitle": "画面収録",
"permissionDescription": "画面をキャプチャするには macOS の画面収録の許可が必要です。"
},
- "enabledDescription": "エージェントに指示を口述できます。選択中のテキストは設定済みのモデルに送信され、その場で編集されます。「{{agentName}}」と話しかけて起動します。",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "OpenWhispr が管理するエージェントを使用。API キーは不要です。",
diff --git a/src/locales/pt/translation.json b/src/locales/pt/translation.json
index d4167a2a2..345e65fe4 100644
--- a/src/locales/pt/translation.json
+++ b/src/locales/pt/translation.json
@@ -550,6 +550,7 @@
"sessionExpired": "Sessão expirada",
"offline": "Sem conexão",
"agentUnavailable": "Agente indisponível",
+ "screenContextSkipped": "Contexto de tela ignorado",
"dailyLimitReached": "Limite diário atingido",
"providerRateLimited": "Limite de taxa do provedor"
},
@@ -580,7 +581,7 @@
"dailyLimitReached": "Você atingiu o limite de transcrições de hoje. Tente novamente amanhã ou faça upgrade do seu plano.",
"providerRateLimited": "Seu provedor de transcrição limitou a solicitação. Aguarde um momento e tente novamente.",
"agentReasoningFailed": "Seu agente de voz não conseguiu processar o comando; a transcrição bruta foi inserida no lugar.",
- "providerRateLimited": "Seu provedor de transcrição limitou a solicitação. Aguarde um momento e tente novamente."
+ "screenContextSkipped": "Não foi possível enviar a captura, então seu comando foi executado sem contexto de tela."
},
"translationFallback": {
"unreachableTitle": "Tradução indisponível",
@@ -2941,7 +2942,7 @@
},
"dictationAgent": {
"enabled": "Ativar agente de voz",
- "enabledDescription": "Dite instruções para o seu agente. Ative dizendo \"{{agentName}}\". Configurável abaixo.",
+ "enabledDescription": "Dite instruções para o seu agente. O texto destacado é enviado ao modelo configurado e editado no lugar. Ative dizendo \"{{agentName}}\".",
"screenContext": {
"title": "Contexto de tela",
"description": "Anexa uma captura da tela ativa aos comandos do agente de voz.",
@@ -2953,7 +2954,6 @@
"permissionTitle": "Gravação de tela",
"permissionDescription": "O macOS exige permissão de gravação de tela para capturar sua tela."
},
- "enabledDescription": "Dite instruções para o seu agente. O texto destacado é enviado ao modelo configurado e editado no lugar. Ative dizendo \"{{agentName}}\".",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Use o agente gerenciado pelo OpenWhispr, sem precisar de chave API.",
diff --git a/src/locales/ru/translation.json b/src/locales/ru/translation.json
index d2b15d004..db0d8579a 100644
--- a/src/locales/ru/translation.json
+++ b/src/locales/ru/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "Сессия истекла",
"offline": "Нет подключения к интернету",
"agentUnavailable": "Агент недоступен",
+ "screenContextSkipped": "Контекст экрана пропущен",
"dailyLimitReached": "Дневной лимит достигнут",
"providerRateLimited": "Ограничение провайдера"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "Достигнут дневной лимит транскрипций. Попробуйте завтра или обновите тариф.",
"providerRateLimited": "Ваш провайдер транскрипции ограничил запрос. Подождите немного и попробуйте снова.",
"agentReasoningFailed": "Голосовой агент не смог обработать команду — вместо этого вставлена необработанная расшифровка.",
- "providerRateLimited": "Ваш провайдер транскрипции ограничил запрос. Подождите немного и попробуйте снова."
+ "screenContextSkipped": "Снимок экрана не удалось отправить, поэтому команда выполнена без контекста экрана."
},
"translationFallback": {
"unreachableTitle": "Перевод недоступен",
@@ -2971,7 +2972,7 @@
},
"dictationAgent": {
"enabled": "Включить голосового агента",
- "enabledDescription": "Диктуйте инструкции своему агенту. Активируется словом \"{{agentName}}\". Настраивается ниже.",
+ "enabledDescription": "Диктуйте инструкции своему агенту. Выделенный текст отправляется в настроенную модель и редактируется на месте. Активируется словом \"{{agentName}}\".",
"screenContext": {
"title": "Контекст экрана",
"description": "Прикрепляет снимок активного экрана к командам голосового агента.",
@@ -2983,7 +2984,6 @@
"permissionTitle": "Запись экрана",
"permissionDescription": "macOS требует разрешение на запись экрана для создания снимков."
},
- "enabledDescription": "Диктуйте инструкции своему агенту. Выделенный текст отправляется в настроенную модель и редактируется на месте. Активируется словом \"{{agentName}}\".",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "Используйте агента под управлением OpenWhispr, без API-ключа.",
diff --git a/src/locales/zh-CN/translation.json b/src/locales/zh-CN/translation.json
index 8f3d9de5b..0f01c5d7c 100644
--- a/src/locales/zh-CN/translation.json
+++ b/src/locales/zh-CN/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "会话已过期",
"offline": "你已离线",
"agentUnavailable": "助手不可用",
+ "screenContextSkipped": "已跳过屏幕上下文",
"dailyLimitReached": "已达到每日上限",
"providerRateLimited": "服务商速率限制"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "已达今日转录上限。请明天再试或升级你的套餐。",
"providerRateLimited": "你的转录服务商限制了请求频率。请稍候再试。",
"agentReasoningFailed": "语音助手无法处理该命令,已改为插入原始转录文本。",
- "providerRateLimited": "你的转录服务商限制了请求频率。请稍候再试。"
+ "screenContextSkipped": "截图无法发送,因此已在没有屏幕上下文的情况下执行你的命令。"
},
"translationFallback": {
"unreachableTitle": "翻译不可用",
@@ -2941,7 +2942,7 @@
},
"dictationAgent": {
"enabled": "启用语音助手",
- "enabledDescription": "向助手口述指令。说出“{{agentName}}”即可激活,可在下方配置。",
+ "enabledDescription": "向助手口述指令。选中的文本会发送到所配置的模型并就地编辑。说出“{{agentName}}”即可激活。",
"screenContext": {
"title": "屏幕上下文",
"description": "在语音助手命令中附带当前屏幕的截图。",
@@ -2953,7 +2954,6 @@
"permissionTitle": "屏幕录制",
"permissionDescription": "macOS 需要屏幕录制权限才能截取屏幕。"
},
- "enabledDescription": "向助手口述指令。选中的文本会发送到所配置的模型并就地编辑。说出“{{agentName}}”即可激活。",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "使用 OpenWhispr 托管的助手,无需 API Key。",
diff --git a/src/locales/zh-TW/translation.json b/src/locales/zh-TW/translation.json
index ec1953007..a742f51fa 100644
--- a/src/locales/zh-TW/translation.json
+++ b/src/locales/zh-TW/translation.json
@@ -571,6 +571,7 @@
"sessionExpired": "工作階段已過期",
"offline": "你目前離線",
"agentUnavailable": "助理無法使用",
+ "screenContextSkipped": "已略過螢幕內容",
"dailyLimitReached": "已達每日上限",
"providerRateLimited": "服務商速率限制"
},
@@ -601,7 +602,7 @@
"dailyLimitReached": "已達今日的轉錄上限。請明天再試或升級你的方案。",
"providerRateLimited": "你的轉錄服務商限制了請求頻率。請稍候再試。",
"agentReasoningFailed": "語音助理無法處理該指令,已改為插入原始逐字稿。",
- "providerRateLimited": "你的轉錄服務商限制了請求頻率。請稍候再試。"
+ "screenContextSkipped": "截圖無法傳送,因此已在沒有螢幕內容的情況下執行你的指令。"
},
"translationFallback": {
"unreachableTitle": "翻譯無法使用",
@@ -2941,7 +2942,7 @@
},
"dictationAgent": {
"enabled": "啟用語音代理",
- "enabledDescription": "向你的代理口述指令。說出 \"{{agentName}}\" 即可啟用。可於下方設定。",
+ "enabledDescription": "向你的代理口述指令。選取的文字會傳送到所設定的模型並就地編輯。說出 \"{{agentName}}\" 即可啟用。",
"screenContext": {
"title": "螢幕內容",
"description": "在語音助理指令中附上目前螢幕的截圖。",
@@ -2953,7 +2954,6 @@
"permissionTitle": "螢幕錄製",
"permissionDescription": "macOS 需要螢幕錄製權限才能擷取螢幕。"
},
- "enabledDescription": "向你的代理口述指令。選取的文字會傳送到所設定的模型並就地編輯。說出 \"{{agentName}}\" 即可啟用。",
"modes": {
"openwhispr": "OpenWhispr Cloud",
"openwhisprDesc": "使用 OpenWhispr 託管的代理,無需 API Key。",
diff --git a/src/utils/recordingErrors.ts b/src/utils/recordingErrors.ts
index 47f7054df..f19e2a613 100644
--- a/src/utils/recordingErrors.ts
+++ b/src/utils/recordingErrors.ts
@@ -5,6 +5,8 @@ type RecordingError = {
title: string;
description?: string;
messageKey?: string;
+ /** Toast variant; defaults to destructive for genuine failures. */
+ variant?: "default" | "destructive";
};
export function getRecordingErrorTitle(error: RecordingError, t: TFunction): string {
@@ -19,6 +21,9 @@ export function getRecordingErrorTitle(error: RecordingError, t: TFunction): str
if (error.code === "AGENT_REASONING_FAILED") {
return t("hooks.audioRecording.errorTitles.agentUnavailable");
}
+ if (error.code === "SCREEN_CONTEXT_SKIPPED") {
+ return t("hooks.audioRecording.errorTitles.screenContextSkipped");
+ }
if (error.code === "LIMIT_REACHED")
return t("hooks.audioRecording.errorTitles.dailyLimitReached");
if (error.code === "PROVIDER_RATE_LIMITED")
diff --git a/test/helpers/screenContextCapture.test.js b/test/helpers/screenContextCapture.test.js
new file mode 100644
index 000000000..8809155ec
--- /dev/null
+++ b/test/helpers/screenContextCapture.test.js
@@ -0,0 +1,120 @@
+const test = require("node:test");
+const assert = require("node:assert/strict");
+const Module = require("node:module");
+
+const modulePath = require.resolve("../../src/helpers/screenContextCapture");
+const originalLoad = Module._load;
+
+// Stands in for Electron's NativeImage: `bytesAtQuality` maps a JPEG quality to
+// the encoded size that quality would produce for a given screen's content.
+// The encoded stub reports that size and tags itself with the quality used, so
+// tests can assert which rung of the ladder produced the payload.
+function fakeImage(bytesAtQuality, { edge = 1568, resizes = [] } = {}) {
+ return {
+ isEmpty: () => false,
+ toJPEG(quality) {
+ return {
+ length: bytesAtQuality(quality, edge),
+ toString: () => Buffer.from(`jpeg-${quality}`).toString("base64"),
+ };
+ },
+ resize({ width }) {
+ resizes.push(width);
+ return fakeImage(bytesAtQuality, { edge: width, resizes });
+ },
+ };
+}
+
+function loadCapture({ image, platform = "darwin", accessStatus = "granted" }) {
+ delete require.cache[modulePath];
+ const calls = { thumbnailSizes: [] };
+
+ Module._load = function loadWithMocks(request, parent, isMain) {
+ if (request === "electron") {
+ return {
+ screen: {
+ getCursorScreenPoint: () => ({ x: 0, y: 0 }),
+ getDisplayNearestPoint: () => ({ id: 1, size: { width: 2560, height: 1440 } }),
+ },
+ desktopCapturer: {
+ getSources: async ({ thumbnailSize }) => {
+ calls.thumbnailSizes.push(thumbnailSize);
+ return [{ display_id: "1", thumbnail: image }];
+ },
+ },
+ systemPreferences: { getMediaAccessStatus: () => accessStatus },
+ };
+ }
+ return originalLoad.call(this, request, parent, isMain);
+ };
+
+ const originalPlatform = process.platform;
+ Object.defineProperty(process, "platform", { value: platform, configurable: true });
+ try {
+ return { capture: require(modulePath), calls, originalPlatform };
+ } finally {
+ Module._load = originalLoad;
+ Object.defineProperty(process, "platform", {
+ value: originalPlatform,
+ configurable: true,
+ });
+ }
+}
+
+test("a typical screenshot is sent at the highest quality", async () => {
+ const { capture } = loadCapture({ image: fakeImage(() => 300_000) });
+
+ const result = await capture.captureCursorDisplay();
+
+ assert.equal(result.mediaType, "image/jpeg");
+ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-82");
+});
+
+test("an oversized screenshot steps down the quality ladder until it fits", async () => {
+ // Only the lowest quality gets under the budget.
+ const { capture } = loadCapture({
+ image: fakeImage((quality) => (quality === 55 ? 900_000 : 2_000_000)),
+ });
+
+ const result = await capture.captureCursorDisplay();
+
+ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-55");
+});
+
+test("a screenshot that no quality fits is resized before a final attempt", async () => {
+ const resizes = [];
+ const { capture } = loadCapture({
+ // Incompressible at full size; fits once the long edge shrinks.
+ image: fakeImage((_quality, edge) => (edge < 1568 ? 800_000 : 4_000_000), { resizes }),
+ });
+
+ const result = await capture.captureCursorDisplay();
+
+ assert.deepEqual(resizes, [1024]);
+ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-55");
+});
+
+test("a screenshot that still cannot fit is dropped rather than sent oversized", async () => {
+ const { capture } = loadCapture({ image: fakeImage(() => 9_000_000) });
+
+ assert.equal(await capture.captureCursorDisplay(), null);
+});
+
+test("capture is skipped when screen recording access is not granted", async () => {
+ const { capture, calls } = loadCapture({
+ image: fakeImage(() => 300_000),
+ accessStatus: "denied",
+ });
+
+ assert.equal(await capture.captureCursorDisplay(), null);
+ assert.equal(calls.thumbnailSizes.length, 0);
+});
+
+test("the captured thumbnail is bounded to the vision-model edge", async () => {
+ const { capture, calls } = loadCapture({ image: fakeImage(() => 300_000) });
+
+ await capture.captureCursorDisplay();
+
+ const [size] = calls.thumbnailSizes;
+ assert.equal(Math.max(size.width, size.height), 1568);
+});