Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.
JavaScript aianthropiccross-platformgeminigroqlinuxmacosnvidiaopen-sourceopenaiparakeetspeech-to-texttranscribewhisperwindows
fix(voice-agent): keep screen context within budget and never fail on it
Gabriel Stein34b9250db508
diff --git a/CLAUDE.md b/CLAUDE.mdindex 828cb21d8..c47be439e 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -629,15 +629,16 @@ A dedicated global hotkey that starts a dictation whose transcript is sent straiWhen "Share screen context" is enabled (Settings → AI Models → Voice Agent → Screen Context, store key `voiceAgentScreenContext`, default off), each voice-agent recording start captures the display the cursor is on and attaches it to the agent request as a base64 JPEG (long edge ≤ 1568 px).- Capture: `src/helpers/screenContextCapture.js` (main process; `screen.getCursorScreenPoint` + `desktopCapturer`). macOS requires the Screen Recording TCC permission (`useScreenRecordingPermission` hook + `PermissionCard` in `DictationAgentSettings`); Windows/Linux X11 need no permission; Linux Wayland is unsupported (capture silently skipped)+- Encoding: `encodeWithinBudget()` walks a JPEG quality ladder (82 → 70 → 55), then falls back to a 1024 px resize, keeping the payload under `MAX_ENCODED_BYTES` (1.5 MB ≈ 2.05M base64 chars, inside the API's 2.8M cap). Each rung re-encodes the original bitmap, so quality drops never compound. A screen that still won't fit is dropped- Trigger: `useAudioRecording.js` calls `audioManager.beginScreenContextCapture()` at voice-agent start (fire-and-forget); `consumeScreenContext()` (3s guard) is awaited when the reasoning route is built. The dictation overlay gets `setContentProtection` while the setting is on so the pill never appears in captures- Routing: `resolveAgentImageTarget()` (`src/helpers/dictationRouting.js`) decides attach/drop. An optional vision override (`dictationAgentVision` inference scope, gated by `useDictationAgentVisionModel`, cloud/BYOK modes only) routes screenshot-carrying requests to a dedicated model; otherwise the base model gets the image only when its provider client is image-wired (`supportsImages` on the `InferenceProvider`) and the model has `supportsVision` in `modelRegistryData.json` (cloud mode defers to the server). Dropping the image never fails the dictation- Prompt: `appendScreenContextSuffix()` adds the `screenContextSuffix` prompts key only when an image is attached- Image-wired clients: `openai` (Responses `input_image` / Chat `image_url`, also `custom`/`openrouter`), `anthropic` (base64 content block via `process-anthropic-reasoning`), `gemini` (`inlineData`), `openwhispr` (forwards `screenContext` + `promptMode: "agent"` to `/api/reason`; the server routes to its `REASONING_VISION_*` model chain)- IPC: `capture-screen-context`, `check-screen-recording-access`, `request-screen-recording-access`, `open-screen-recording-settings`, `screen-context-set-enabled`- Privacy: screenshots live only in renderer memory for one request — never written to disk, stored in history, or logged (loggers emit `hasScreenContext` booleans only)-- Failure UX: any agent-route reasoning failure now surfaces an "Agent Unavailable" toast (`AGENT_REASONING_FAILED`) while still pasting the raw transcript+- Failure UX: a rejected screenshot is retried once text-only from `processWithReasoningModel()` (rebuilding the prompt without the screen-context suffix) and reported via a non-destructive `SCREEN_CONTEXT_SKIPPED` toast, so an image problem never costs the user their command. Only if that retry also fails does the "Agent Unavailable" toast (`AGENT_REASONING_FAILED`) fire and the raw transcript get pasted-**Tests**: `test/helpers/dictationRouting.test.js` (run with `node --test`)+**Tests**: `test/helpers/dictationRouting.test.js`, `test/helpers/screenContextCapture.test.js` (run with `node --test`)## Development Guidelinesdiff --git a/src/helpers/audioManager.js b/src/helpers/audioManager.jsindex e25789c62..543ff2496 100644--- a/src/helpers/audioManager.js+++ b/src/helpers/audioManager.js@@ -114,6 +114,17 @@ function getEffectiveRetentionPreferences() {const providerSupportsImages = (providerId) =>!!(providerId && PROVIDER_REGISTRY[providerId]?.supportsImages);+// Shared by the agent route and its text-only retry, which needs the prompt+// without the screen-context suffix.+function dictationAgentPrompt(settings, agentName) {+ return resolvePrompt("dictationAgent", {+ agentName,+ language: settings.preferredLanguage,+ customDictionary: getDictionaryHintWords(settings),+ uiLanguage: settings.uiLanguage,+ });+}+function dictationAgentReachable(settings) {return resolveDictationAgentInference(settings, { isCloudAgent: isCloudDictationAgentMode() }).reachable;@@ -193,20 +204,16 @@ function resolveReasoningRoute(});const target = useVisionOverride ? vision : agent;- let systemPrompt = resolvePrompt("dictationAgent", {- agentName,- language: settings.preferredLanguage,- customDictionary: getDictionaryHintWords(settings),- uiLanguage: settings.uiLanguage,- });- if (attach) systemPrompt = appendScreenContextSuffix(systemPrompt, settings.uiLanguage);+ const systemPrompt = dictationAgentPrompt(settings, agentName);return {kind: "agent",model: target.model,config: {...target.config,- systemPrompt,+ systemPrompt: attach+ ? appendScreenContextSuffix(systemPrompt, settings.uiLanguage)+ : systemPrompt,...(attach ? { screenContext } : {}),},};@@ -671,6 +678,16 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);});}+ // The command still ran, so this is a downgrade notice rather than a failure.+ _notifyScreenContextSkipped() {+ this.onError?.({+ code: "SCREEN_CONTEXT_SKIPPED",+ title: "Screen Context Skipped",+ messageKey: "hooks.audioRecording.errorDescriptions.screenContextSkipped",+ variant: "default",+ });+ }+setContext(context) {this.context = context;}@@ -2098,6 +2115,20 @@ registerProcessor("pcm-streaming-processor", PCMStreamingProcessor);stack: error.stack,});+ // A screenshot the model or transport rejects must not cost the user+ // their command — rerun it text-only. Only the agent route attaches one,+ // so rebuilding its prompt also drops the screen-context instructions+ // that would otherwise reference an image we are no longer sending.+ if (config?.screenContext) {+ const { screenContext, ...textOnlyConfig } = config;+ const result = await ReasoningService.processText(text, model, agentName, {+ ...textOnlyConfig,+ systemPrompt: dictationAgentPrompt(getSettings(), agentName),+ });+ this._notifyScreenContextSkipped();+ return result;+ }+throw error;}}diff --git a/src/helpers/screenContextCapture.js b/src/helpers/screenContextCapture.jsindex a71822054..41729cdea 100644--- a/src/helpers/screenContextCapture.js+++ b/src/helpers/screenContextCapture.js@@ -4,7 +4,14 @@ const debugLogger = require("./debugLogger");// Vision models downsample images past ~1.5k px on the long edge; capturing// larger only inflates the payload without adding model-visible detail.const MAX_EDGE_PX = 1568;-const JPEG_QUALITY = 75;+// A 1568px screenshot lands well under the cap at this quality, so spend the+// headroom on legible on-screen text; the ladder below covers the rare+// near-incompressible screen (video, noise) that doesn't fit.+const QUALITY_LADDER = [82, 70, 55];+// Keeps the base64 payload (~1.37x) inside the API's 2.8M character limit+// with room for the transcript, prompt and dictionary.+const MAX_ENCODED_BYTES = 1_500_000;+const FALLBACK_EDGE_PX = 1024;// Wayland routes desktopCapturer through the xdg-desktop-portal picker (a// dialog per capture) and cursor coordinates are unreliable there, so screen@@ -25,6 +32,19 @@ function getAccessStatus() {return "granted";}+// Encodes from the same source bitmap at each step, so lowering quality never+// compounds artifacts from an earlier pass.+function encodeWithinBudget(image) {+ for (const quality of QUALITY_LADDER) {+ const encoded = image.toJPEG(quality);+ if (encoded.length <= MAX_ENCODED_BYTES) return encoded;+ }++ const resized = image.resize({ width: FALLBACK_EDGE_PX, quality: "good" });+ const smallest = resized.toJPEG(QUALITY_LADDER.at(-1));+ return smallest.length <= MAX_ENCODED_BYTES ? smallest : null;+}+// Returns null on any failure — a screenshot must never break the dictation// it accompanies.async function captureCursorDisplay() {@@ -44,10 +64,13 @@ async function captureCursorDisplay() {const source = sources.find((s) => s.display_id === String(display.id)) || sources[0];if (!source || source.thumbnail.isEmpty()) return null;- return {- mediaType: "image/jpeg",- data: source.thumbnail.toJPEG(JPEG_QUALITY).toString("base64"),- };+ const encoded = encodeWithinBudget(source.thumbnail);+ if (!encoded) {+ debugLogger.warn("Screen context too large to send", {}, "screenContext");+ return null;+ }++ return { mediaType: "image/jpeg", data: encoded.toString("base64") };} catch (error) {debugLogger.warn("Screen context capture failed", { error: error.message }, "screenContext");return null;diff --git a/src/hooks/useAudioRecording.js b/src/hooks/useAudioRecording.jsindex 87825c095..74f19e185 100644--- a/src/hooks/useAudioRecording.js+++ b/src/hooks/useAudioRecording.js@@ -199,7 +199,7 @@ export const useAudioRecording = (toast, options = {}) => {toast({title,description,- variant: "destructive",+ variant: error.variant || "destructive",duration: error.code === "AUTH_EXPIRED" ? 8000 : undefined,});if (getSettings().pauseMediaOnDictation) {diff --git a/src/locales/de/translation.json b/src/locales/de/translation.jsonindex 212a8c50b..261497325 100644--- a/src/locales/de/translation.json+++ b/src/locales/de/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Sitzung abgelaufen","offline": "Sie sind offline","agentUnavailable": "Agent nicht verfügbar",+ "screenContextSkipped": "Bildschirmkontext übersprungen","dailyLimitReached": "Tägliches Limit erreicht","providerRateLimited": "Anbieter-Ratenlimit erreicht"},@@ -601,7 +602,7 @@"dailyLimitReached": "Du hast dein heutiges Transkriptionslimit erreicht. Versuch es morgen erneut oder upgrade deinen Plan.","providerRateLimited": "Dein Transkriptionsanbieter hat die Anfrage begrenzt. Warte kurz und versuch es erneut.","agentReasoningFailed": "Ihr Sprachagent konnte den Befehl nicht verarbeiten – stattdessen wurde das unbearbeitete Transkript eingefügt.",- "providerRateLimited": "Dein Transkriptionsanbieter hat die Anfrage begrenzt. Warte kurz und versuch es erneut."+ "screenContextSkipped": "Der Screenshot konnte nicht gesendet werden, daher lief Ihr Befehl ohne Bildschirmkontext."},"translationFallback": {"unreachableTitle": "Übersetzung nicht verfügbar",@@ -3089,7 +3090,7 @@},"dictationAgent": {"enabled": "Voice Agent aktivieren",- "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen. Unten konfigurierbar.",+ "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Markierter Text wird an das konfigurierte Modell gesendet und direkt bearbeitet. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen.","screenContext": {"title": "Bildschirmkontext","description": "Hängt einen Screenshot Ihres aktiven Bildschirms an Sprachagent-Befehle an.",@@ -3101,7 +3102,6 @@"permissionTitle": "Bildschirmaufnahme","permissionDescription": "macOS benötigt die Berechtigung zur Bildschirmaufnahme, um Ihren Bildschirm zu erfassen."},- "enabledDescription": "Diktieren Sie Anweisungen an Ihren Agenten. Markierter Text wird an das konfigurierte Modell gesendet und direkt bearbeitet. Aktivieren Sie ihn, indem Sie \"{{agentName}}\" sagen.","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Den von OpenWhispr verwalteten Agenten nutzen, kein API-Schlüssel nötig.",diff --git a/src/locales/en/translation.json b/src/locales/en/translation.jsonindex a59a03469..989c39c9e 100644--- a/src/locales/en/translation.json+++ b/src/locales/en/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Session Expired","offline": "You're Offline","agentUnavailable": "Agent Unavailable",+ "screenContextSkipped": "Screen Context Skipped","dailyLimitReached": "Daily Limit Reached","providerRateLimited": "Provider Rate Limited"},@@ -601,7 +602,7 @@"dailyLimitReached": "You've reached today's transcription limit. Try again tomorrow or upgrade your plan.","providerRateLimited": "Your transcription provider rate-limited the request. Wait a moment and try again.","agentReasoningFailed": "Your voice agent couldn't process the command — the raw transcript was inserted instead.",- "providerRateLimited": "Your transcription provider rate-limited the request. Wait a moment and try again."+ "screenContextSkipped": "The screenshot couldn't be sent, so your command ran without screen context."},"translationFallback": {"unreachableTitle": "Translation unavailable",@@ -3089,7 +3090,7 @@},"dictationAgent": {"enabled": "Enable voice agent",- "enabledDescription": "Dictate instructions to your agent. Activate by saying \"{{agentName}}\". Configurable below.",+ "enabledDescription": "Dictate instructions to your agent. Highlighted text is sent to the configured model and edited in place. Activate by saying \"{{agentName}}\".","screenContext": {"title": "Screen Context","description": "Attach a screenshot of your active screen to voice agent commands.",@@ -3101,7 +3102,6 @@"permissionTitle": "Screen Recording","permissionDescription": "macOS requires Screen Recording permission to capture your screen."},- "enabledDescription": "Dictate instructions to your agent. Highlighted text is sent to the configured model and edited in place. Activate by saying \"{{agentName}}\".","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Use the OpenWhispr-managed agent — no API key needed.",diff --git a/src/locales/es/translation.json b/src/locales/es/translation.jsonindex 8eec93370..26861bb4b 100644--- a/src/locales/es/translation.json+++ b/src/locales/es/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Sesión expirada","offline": "Sin conexión","agentUnavailable": "Agente no disponible",+ "screenContextSkipped": "Contexto de pantalla omitido","dailyLimitReached": "Límite diario alcanzado","providerRateLimited": "Límite de tasa del proveedor"},@@ -601,7 +602,7 @@"dailyLimitReached": "Has alcanzado el límite de transcripción de hoy. Inténtalo mañana o mejora tu plan.","providerRateLimited": "Tu proveedor de transcripción limitó la solicitud. Espera un momento e inténtalo de nuevo.","agentReasoningFailed": "Tu agente de voz no pudo procesar el comando; se insertó la transcripción sin procesar.",- "providerRateLimited": "Tu proveedor de transcripción limitó la solicitud. Espera un momento e inténtalo de nuevo."+ "screenContextSkipped": "No se pudo enviar la captura, así que tu comando se ejecutó sin contexto de pantalla."},"translationFallback": {"unreachableTitle": "Traducción no disponible",@@ -2989,7 +2990,7 @@},"dictationAgent": {"enabled": "Activar agente de voz",- "enabledDescription": "Dicta instrucciones a tu agente. Actívalo diciendo \"{{agentName}}\". Configurable más abajo.",+ "enabledDescription": "Dicta instrucciones a tu agente. El texto resaltado se envía al modelo configurado y se edita en su lugar. Actívalo diciendo \"{{agentName}}\".","screenContext": {"title": "Contexto de pantalla","description": "Adjunta una captura de tu pantalla activa a los comandos del agente de voz.",@@ -3001,7 +3002,6 @@"permissionTitle": "Grabación de pantalla","permissionDescription": "macOS requiere permiso de grabación de pantalla para capturar tu pantalla."},- "enabledDescription": "Dicta instrucciones a tu agente. El texto resaltado se envía al modelo configurado y se edita en su lugar. Actívalo diciendo \"{{agentName}}\".","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Usa el agente gestionado por OpenWhispr, sin necesidad de clave API.",diff --git a/src/locales/fr/translation.json b/src/locales/fr/translation.jsonindex 6deec7bd8..498fc53ae 100644--- a/src/locales/fr/translation.json+++ b/src/locales/fr/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Session expirée","offline": "Vous êtes hors ligne","agentUnavailable": "Agent indisponible",+ "screenContextSkipped": "Contexte d'écran ignoré","dailyLimitReached": "Limite quotidienne atteinte","providerRateLimited": "Limite de débit du fournisseur"},@@ -601,7 +602,7 @@"dailyLimitReached": "Vous avez atteint votre limite de transcription du jour. Réessayez demain ou améliorez votre forfait.","providerRateLimited": "Votre fournisseur de transcription a limité la requête. Patientez un instant et réessayez.","agentReasoningFailed": "Votre agent vocal n'a pas pu traiter la commande ; la transcription brute a été insérée à la place.",- "providerRateLimited": "Votre fournisseur de transcription a limité la requête. Patientez un instant et réessayez."+ "screenContextSkipped": "La capture n'a pas pu être envoyée ; votre commande a été exécutée sans contexte d'écran."},"translationFallback": {"unreachableTitle": "Traduction indisponible",@@ -3089,7 +3090,7 @@},"dictationAgent": {"enabled": "Activer l'agent vocal",- "enabledDescription": "Dictez des instructions à votre agent. Activez-le en disant « \"{{agentName}}\" ». Configurable ci-dessous.",+ "enabledDescription": "Dictez des instructions à votre agent. Le texte surligné est envoyé au modèle configuré et modifié sur place. Activez-le en disant « \"{{agentName}}\" ».","screenContext": {"title": "Contexte d'écran","description": "Joint une capture de votre écran actif aux commandes de l'agent vocal.",@@ -3101,7 +3102,6 @@"permissionTitle": "Enregistrement de l'écran","permissionDescription": "macOS exige l'autorisation d'enregistrement de l'écran pour capturer votre écran."},- "enabledDescription": "Dictez des instructions à votre agent. Le texte surligné est envoyé au modèle configuré et modifié sur place. Activez-le en disant « \"{{agentName}}\" ».","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Utilisez l'agent géré par OpenWhispr, sans clé API.",diff --git a/src/locales/it/translation.json b/src/locales/it/translation.jsonindex ef8aa91fe..e35283feb 100644--- a/src/locales/it/translation.json+++ b/src/locales/it/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Sessione scaduta","offline": "Sei offline","agentUnavailable": "Agente non disponibile",+ "screenContextSkipped": "Contesto dello schermo ignorato","dailyLimitReached": "Limite giornaliero raggiunto","providerRateLimited": "Limite di velocità del provider"},@@ -601,7 +602,7 @@"dailyLimitReached": "Hai raggiunto il limite di trascrizioni di oggi. Riprova domani o aggiorna il tuo piano.","providerRateLimited": "Il tuo provider di trascrizione ha limitato la richiesta. Attendi un momento e riprova.","agentReasoningFailed": "L'agente vocale non è riuscito a elaborare il comando; è stata inserita la trascrizione grezza.",- "providerRateLimited": "Il tuo provider di trascrizione ha limitato la richiesta. Attendi un momento e riprova."+ "screenContextSkipped": "Non è stato possibile inviare lo screenshot, quindi il comando è stato eseguito senza contesto dello schermo."},"translationFallback": {"unreachableTitle": "Traduzione non disponibile",@@ -2941,7 +2942,7 @@},"dictationAgent": {"enabled": "Attiva agente vocale",- "enabledDescription": "Detta istruzioni al tuo agente. Attivalo dicendo \"{{agentName}}\". Configurabile qui sotto.",+ "enabledDescription": "Detta istruzioni al tuo agente. Il testo evidenziato viene inviato al modello configurato e modificato sul posto. Attivalo dicendo \"{{agentName}}\".","screenContext": {"title": "Contesto dello schermo","description": "Allega uno screenshot dello schermo attivo ai comandi dell'agente vocale.",@@ -2953,7 +2954,6 @@"permissionTitle": "Registrazione schermo","permissionDescription": "macOS richiede l'autorizzazione alla registrazione dello schermo per catturare lo schermo."},- "enabledDescription": "Detta istruzioni al tuo agente. Il testo evidenziato viene inviato al modello configurato e modificato sul posto. Attivalo dicendo \"{{agentName}}\".","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Usa l'agente gestito da OpenWhispr, senza bisogno di chiave API.",diff --git a/src/locales/ja/translation.json b/src/locales/ja/translation.jsonindex d7a3c654b..be2d39ced 100644--- a/src/locales/ja/translation.json+++ b/src/locales/ja/translation.json@@ -571,6 +571,7 @@"sessionExpired": "セッション期限切れ","offline": "オフラインです","agentUnavailable": "エージェントを利用できません",+ "screenContextSkipped": "画面コンテキストをスキップしました","dailyLimitReached": "1 日の上限に達しました","providerRateLimited": "プロバイダーのレート制限"},@@ -601,7 +602,7 @@"dailyLimitReached": "本日の文字起こし上限に達しました。明日もう一度お試しいただくか、プランをアップグレードしてください。","providerRateLimited": "文字起こしプロバイダーがリクエストを制限しました。しばらく待ってからもう一度お試しください。","agentReasoningFailed": "ボイスエージェントがコマンドを処理できなかったため、未処理の文字起こしを挿入しました。",- "providerRateLimited": "文字起こしプロバイダーがリクエストを制限しました。しばらく待ってからもう一度お試しください。"+ "screenContextSkipped": "スクリーンショットを送信できなかったため、画面コンテキストなしでコマンドを実行しました。"},"translationFallback": {"unreachableTitle": "翻訳を利用できません",@@ -2941,7 +2942,7 @@},"dictationAgent": {"enabled": "音声エージェントを有効にする",- "enabledDescription": "エージェントに指示を口述できます。「{{agentName}}」と話しかけて起動します。設定は以下で行えます。",+ "enabledDescription": "エージェントに指示を口述できます。選択中のテキストは設定済みのモデルに送信され、その場で編集されます。「{{agentName}}」と話しかけて起動します。","screenContext": {"title": "画面コンテキスト","description": "ボイスエージェントのコマンドに、作業中の画面のスクリーンショットを添付します。",@@ -2953,7 +2954,6 @@"permissionTitle": "画面収録","permissionDescription": "画面をキャプチャするには macOS の画面収録の許可が必要です。"},- "enabledDescription": "エージェントに指示を口述できます。選択中のテキストは設定済みのモデルに送信され、その場で編集されます。「{{agentName}}」と話しかけて起動します。","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "OpenWhispr が管理するエージェントを使用。API キーは不要です。",diff --git a/src/locales/pt/translation.json b/src/locales/pt/translation.jsonindex d4167a2a2..345e65fe4 100644--- a/src/locales/pt/translation.json+++ b/src/locales/pt/translation.json@@ -550,6 +550,7 @@"sessionExpired": "Sessão expirada","offline": "Sem conexão","agentUnavailable": "Agente indisponível",+ "screenContextSkipped": "Contexto de tela ignorado","dailyLimitReached": "Limite diário atingido","providerRateLimited": "Limite de taxa do provedor"},@@ -580,7 +581,7 @@"dailyLimitReached": "Você atingiu o limite de transcrições de hoje. Tente novamente amanhã ou faça upgrade do seu plano.","providerRateLimited": "Seu provedor de transcrição limitou a solicitação. Aguarde um momento e tente novamente.","agentReasoningFailed": "Seu agente de voz não conseguiu processar o comando; a transcrição bruta foi inserida no lugar.",- "providerRateLimited": "Seu provedor de transcrição limitou a solicitação. Aguarde um momento e tente novamente."+ "screenContextSkipped": "Não foi possível enviar a captura, então seu comando foi executado sem contexto de tela."},"translationFallback": {"unreachableTitle": "Tradução indisponível",@@ -2941,7 +2942,7 @@},"dictationAgent": {"enabled": "Ativar agente de voz",- "enabledDescription": "Dite instruções para o seu agente. Ative dizendo \"{{agentName}}\". Configurável abaixo.",+ "enabledDescription": "Dite instruções para o seu agente. O texto destacado é enviado ao modelo configurado e editado no lugar. Ative dizendo \"{{agentName}}\".","screenContext": {"title": "Contexto de tela","description": "Anexa uma captura da tela ativa aos comandos do agente de voz.",@@ -2953,7 +2954,6 @@"permissionTitle": "Gravação de tela","permissionDescription": "O macOS exige permissão de gravação de tela para capturar sua tela."},- "enabledDescription": "Dite instruções para o seu agente. O texto destacado é enviado ao modelo configurado e editado no lugar. Ative dizendo \"{{agentName}}\".","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Use o agente gerenciado pelo OpenWhispr, sem precisar de chave API.",diff --git a/src/locales/ru/translation.json b/src/locales/ru/translation.jsonindex d2b15d004..db0d8579a 100644--- a/src/locales/ru/translation.json+++ b/src/locales/ru/translation.json@@ -571,6 +571,7 @@"sessionExpired": "Сессия истекла","offline": "Нет подключения к интернету","agentUnavailable": "Агент недоступен",+ "screenContextSkipped": "Контекст экрана пропущен","dailyLimitReached": "Дневной лимит достигнут","providerRateLimited": "Ограничение провайдера"},@@ -601,7 +602,7 @@"dailyLimitReached": "Достигнут дневной лимит транскрипций. Попробуйте завтра или обновите тариф.","providerRateLimited": "Ваш провайдер транскрипции ограничил запрос. Подождите немного и попробуйте снова.","agentReasoningFailed": "Голосовой агент не смог обработать команду — вместо этого вставлена необработанная расшифровка.",- "providerRateLimited": "Ваш провайдер транскрипции ограничил запрос. Подождите немного и попробуйте снова."+ "screenContextSkipped": "Снимок экрана не удалось отправить, поэтому команда выполнена без контекста экрана."},"translationFallback": {"unreachableTitle": "Перевод недоступен",@@ -2971,7 +2972,7 @@},"dictationAgent": {"enabled": "Включить голосового агента",- "enabledDescription": "Диктуйте инструкции своему агенту. Активируется словом \"{{agentName}}\". Настраивается ниже.",+ "enabledDescription": "Диктуйте инструкции своему агенту. Выделенный текст отправляется в настроенную модель и редактируется на месте. Активируется словом \"{{agentName}}\".","screenContext": {"title": "Контекст экрана","description": "Прикрепляет снимок активного экрана к командам голосового агента.",@@ -2983,7 +2984,6 @@"permissionTitle": "Запись экрана","permissionDescription": "macOS требует разрешение на запись экрана для создания снимков."},- "enabledDescription": "Диктуйте инструкции своему агенту. Выделенный текст отправляется в настроенную модель и редактируется на месте. Активируется словом \"{{agentName}}\".","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "Используйте агента под управлением OpenWhispr, без API-ключа.",diff --git a/src/locales/zh-CN/translation.json b/src/locales/zh-CN/translation.jsonindex 8f3d9de5b..0f01c5d7c 100644--- a/src/locales/zh-CN/translation.json+++ b/src/locales/zh-CN/translation.json@@ -571,6 +571,7 @@"sessionExpired": "会话已过期","offline": "你已离线","agentUnavailable": "助手不可用",+ "screenContextSkipped": "已跳过屏幕上下文","dailyLimitReached": "已达到每日上限","providerRateLimited": "服务商速率限制"},@@ -601,7 +602,7 @@"dailyLimitReached": "已达今日转录上限。请明天再试或升级你的套餐。","providerRateLimited": "你的转录服务商限制了请求频率。请稍候再试。","agentReasoningFailed": "语音助手无法处理该命令,已改为插入原始转录文本。",- "providerRateLimited": "你的转录服务商限制了请求频率。请稍候再试。"+ "screenContextSkipped": "截图无法发送,因此已在没有屏幕上下文的情况下执行你的命令。"},"translationFallback": {"unreachableTitle": "翻译不可用",@@ -2941,7 +2942,7 @@},"dictationAgent": {"enabled": "启用语音助手",- "enabledDescription": "向助手口述指令。说出“{{agentName}}”即可激活,可在下方配置。",+ "enabledDescription": "向助手口述指令。选中的文本会发送到所配置的模型并就地编辑。说出“{{agentName}}”即可激活。","screenContext": {"title": "屏幕上下文","description": "在语音助手命令中附带当前屏幕的截图。",@@ -2953,7 +2954,6 @@"permissionTitle": "屏幕录制","permissionDescription": "macOS 需要屏幕录制权限才能截取屏幕。"},- "enabledDescription": "向助手口述指令。选中的文本会发送到所配置的模型并就地编辑。说出“{{agentName}}”即可激活。","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "使用 OpenWhispr 托管的助手,无需 API Key。",diff --git a/src/locales/zh-TW/translation.json b/src/locales/zh-TW/translation.jsonindex ec1953007..a742f51fa 100644--- a/src/locales/zh-TW/translation.json+++ b/src/locales/zh-TW/translation.json@@ -571,6 +571,7 @@"sessionExpired": "工作階段已過期","offline": "你目前離線","agentUnavailable": "助理無法使用",+ "screenContextSkipped": "已略過螢幕內容","dailyLimitReached": "已達每日上限","providerRateLimited": "服務商速率限制"},@@ -601,7 +602,7 @@"dailyLimitReached": "已達今日的轉錄上限。請明天再試或升級你的方案。","providerRateLimited": "你的轉錄服務商限制了請求頻率。請稍候再試。","agentReasoningFailed": "語音助理無法處理該指令,已改為插入原始逐字稿。",- "providerRateLimited": "你的轉錄服務商限制了請求頻率。請稍候再試。"+ "screenContextSkipped": "截圖無法傳送,因此已在沒有螢幕內容的情況下執行你的指令。"},"translationFallback": {"unreachableTitle": "翻譯無法使用",@@ -2941,7 +2942,7 @@},"dictationAgent": {"enabled": "啟用語音代理",- "enabledDescription": "向你的代理口述指令。說出 \"{{agentName}}\" 即可啟用。可於下方設定。",+ "enabledDescription": "向你的代理口述指令。選取的文字會傳送到所設定的模型並就地編輯。說出 \"{{agentName}}\" 即可啟用。","screenContext": {"title": "螢幕內容","description": "在語音助理指令中附上目前螢幕的截圖。",@@ -2953,7 +2954,6 @@"permissionTitle": "螢幕錄製","permissionDescription": "macOS 需要螢幕錄製權限才能擷取螢幕。"},- "enabledDescription": "向你的代理口述指令。選取的文字會傳送到所設定的模型並就地編輯。說出 \"{{agentName}}\" 即可啟用。","modes": {"openwhispr": "OpenWhispr Cloud","openwhisprDesc": "使用 OpenWhispr 託管的代理,無需 API Key。",diff --git a/src/utils/recordingErrors.ts b/src/utils/recordingErrors.tsindex 47f7054df..f19e2a613 100644--- a/src/utils/recordingErrors.ts+++ b/src/utils/recordingErrors.ts@@ -5,6 +5,8 @@ type RecordingError = {title: string;description?: string;messageKey?: string;+ /** Toast variant; defaults to destructive for genuine failures. */+ variant?: "default" | "destructive";};export function getRecordingErrorTitle(error: RecordingError, t: TFunction): string {@@ -19,6 +21,9 @@ export function getRecordingErrorTitle(error: RecordingError, t: TFunction): strif (error.code === "AGENT_REASONING_FAILED") {return t("hooks.audioRecording.errorTitles.agentUnavailable");}+ if (error.code === "SCREEN_CONTEXT_SKIPPED") {+ return t("hooks.audioRecording.errorTitles.screenContextSkipped");+ }if (error.code === "LIMIT_REACHED")return t("hooks.audioRecording.errorTitles.dailyLimitReached");if (error.code === "PROVIDER_RATE_LIMITED")diff --git a/test/helpers/screenContextCapture.test.js b/test/helpers/screenContextCapture.test.jsnew file mode 100644index 000000000..8809155ec--- /dev/null+++ b/test/helpers/screenContextCapture.test.js@@ -0,0 +1,120 @@+const test = require("node:test");+const assert = require("node:assert/strict");+const Module = require("node:module");++const modulePath = require.resolve("../../src/helpers/screenContextCapture");+const originalLoad = Module._load;++// Stands in for Electron's NativeImage: `bytesAtQuality` maps a JPEG quality to+// the encoded size that quality would produce for a given screen's content.+// The encoded stub reports that size and tags itself with the quality used, so+// tests can assert which rung of the ladder produced the payload.+function fakeImage(bytesAtQuality, { edge = 1568, resizes = [] } = {}) {+ return {+ isEmpty: () => false,+ toJPEG(quality) {+ return {+ length: bytesAtQuality(quality, edge),+ toString: () => Buffer.from(`jpeg-${quality}`).toString("base64"),+ };+ },+ resize({ width }) {+ resizes.push(width);+ return fakeImage(bytesAtQuality, { edge: width, resizes });+ },+ };+}++function loadCapture({ image, platform = "darwin", accessStatus = "granted" }) {+ delete require.cache[modulePath];+ const calls = { thumbnailSizes: [] };++ Module._load = function loadWithMocks(request, parent, isMain) {+ if (request === "electron") {+ return {+ screen: {+ getCursorScreenPoint: () => ({ x: 0, y: 0 }),+ getDisplayNearestPoint: () => ({ id: 1, size: { width: 2560, height: 1440 } }),+ },+ desktopCapturer: {+ getSources: async ({ thumbnailSize }) => {+ calls.thumbnailSizes.push(thumbnailSize);+ return [{ display_id: "1", thumbnail: image }];+ },+ },+ systemPreferences: { getMediaAccessStatus: () => accessStatus },+ };+ }+ return originalLoad.call(this, request, parent, isMain);+ };++ const originalPlatform = process.platform;+ Object.defineProperty(process, "platform", { value: platform, configurable: true });+ try {+ return { capture: require(modulePath), calls, originalPlatform };+ } finally {+ Module._load = originalLoad;+ Object.defineProperty(process, "platform", {+ value: originalPlatform,+ configurable: true,+ });+ }+}++test("a typical screenshot is sent at the highest quality", async () => {+ const { capture } = loadCapture({ image: fakeImage(() => 300_000) });++ const result = await capture.captureCursorDisplay();++ assert.equal(result.mediaType, "image/jpeg");+ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-82");+});++test("an oversized screenshot steps down the quality ladder until it fits", async () => {+ // Only the lowest quality gets under the budget.+ const { capture } = loadCapture({+ image: fakeImage((quality) => (quality === 55 ? 900_000 : 2_000_000)),+ });++ const result = await capture.captureCursorDisplay();++ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-55");+});++test("a screenshot that no quality fits is resized before a final attempt", async () => {+ const resizes = [];+ const { capture } = loadCapture({+ // Incompressible at full size; fits once the long edge shrinks.+ image: fakeImage((_quality, edge) => (edge < 1568 ? 800_000 : 4_000_000), { resizes }),+ });++ const result = await capture.captureCursorDisplay();++ assert.deepEqual(resizes, [1024]);+ assert.equal(Buffer.from(result.data, "base64").toString(), "jpeg-55");+});++test("a screenshot that still cannot fit is dropped rather than sent oversized", async () => {+ const { capture } = loadCapture({ image: fakeImage(() => 9_000_000) });++ assert.equal(await capture.captureCursorDisplay(), null);+});++test("capture is skipped when screen recording access is not granted", async () => {+ const { capture, calls } = loadCapture({+ image: fakeImage(() => 300_000),+ accessStatus: "denied",+ });++ assert.equal(await capture.captureCursorDisplay(), null);+ assert.equal(calls.thumbnailSizes.length, 0);+});++test("the captured thumbnail is bounded to the vision-model edge", async () => {+ const { capture, calls } = loadCapture({ image: fakeImage(() => 300_000) });++ await capture.captureCursorDisplay();++ const [size] = calls.thumbnailSizes;+ assert.equal(Math.max(size.width, size.height), 1568);+});