Skip to content

Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.

135 KB
Raw Blame

OpenWhispr Technical Reference for AI Assistants

This document provides comprehensive technical details about the OpenWhispr project architecture for AI assistants working on the codebase.

Response Style

Keep responses focused, brief, and concise. Keep disclaimers and caveats short, and spend most of the response on the main answer. When asked to explain something, give a high-level summary unless an in-depth explanation is specifically requested.

Project Overview

OpenWhispr is an Electron-based desktop dictation application that uses whisper.cpp for speech-to-text transcription. It supports both local (privacy-focused) and cloud (OpenAI API) processing modes.

Architecture Overview

Core Technologies

  • Frontend: React 19, TypeScript, Tailwind CSS v4, Vite
  • Desktop Framework: Electron 41 with context isolation
  • Database: better-sqlite3 for local transcription history
  • UI Components: shadcn/ui with Radix primitives
  • Speech Processing: whisper.cpp + NVIDIA Parakeet (via sherpa-onnx) + OpenAI API
  • Audio Processing: FFmpeg (bundled via ffmpeg-static)
  • Node.js: 24 (pinned in .nvmrc — CI uses Node 24, do NOT regenerate package-lock.json with a different major version)

Key Architectural Decisions

  1. Dual Window Architecture:

    • Main Window: Minimal overlay for dictation (draggable, always on top)
    • Control Panel: Full settings interface (normal window)
    • Both use same React codebase with URL-based routing
  2. Process Separation:

    • Main Process: Electron main, IPC handlers, database operations
    • Renderer Process: React app with context isolation
    • Preload Script: Secure bridge between processes
    • ONNX Utility Process: hosts all onnxruntime-node inference (text embeddings, speaker embeddings, fbank). Lazy-spawned on first use via src/helpers/onnxWorkerClient.js → src/workers/onnxWorker.js. Native crashes (e.g., ORT bad_alloc) confine to the worker; main process rejects in-flight requests, waits out a backoff, and respawns the worker on the next request (never eagerly, so a crash can't leave an empty worker running). A request that times out kills the worker so it respawns the same way; the embedding clients reload their sessions on the new worker. Exits once idle with no session loaded (releaseIfIdle, called after semantic search releases its model and after speakerEmbeddings.js unloads the speaker model 5 minutes after its last extract). Each session's calls are serialized in the worker, so an unload never releases a session mid-inference. Stopped in will-quit.
  3. Audio Pipeline:

    • MediaRecorder API → Blob → ArrayBuffer → IPC → File → whisper.cpp
    • Automatic cleanup of temporary files after processing

File Structure and Responsibilities

Main Process Files

  • main.js: Application entry point, initializes all managers
  • preload.js: Exposes safe IPC methods to renderer via window.api

Native Resources (resources/)

  • windows-key-listener.c: C source for Windows low-level keyboard hook (Push-to-Talk)
  • windows-mic-listener.c: C source for WASAPI mic session monitor (event-driven mic detection)
  • windows-system-audio-helper.c: C source for WASAPI process-loopback system audio capture (meeting transcription). Excludes OpenWhispr's own process tree, so it hears every app on every output device. Requires Windows 10 2004+; falls back to Chromium display-media loopback when unavailable. Mid-session, the helper emits a capture_silent warning when its own stream is silent while a render endpoint is metering output, checked every second for the whole session because Windows keeps some apps' streams out of process loopback (Teams call audio, #1265). Main then starts renderer loopback beside the helper, and meetingSystemAudioHandover.js gives loopback the system channel only after it hears audio the helper misses, so a false alarm never drops a working capture. Outputs 24 kHz mono s16le PCM on stdout, line-delimited JSON events on stderr (same protocol as linux-system-audio-helper)
  • macos-mic-listener.swift: Swift source for the CoreAudio process-object microphone listener (event-driven mic detection); falls back to aggregate device activity and retries PID monitoring from its heartbeat
  • globe-listener.swift: Swift source for macOS Globe/Fn key detection
  • macos-window-bounds.swift: Swift source for the System Settings window reporter used by the onboarding permission guide. Prints the dialog's bounds, whether a sheet or authorization prompt is in front, which app owns the front window (settings / self / other, attributed by process because window owner names are localized) and whether System Settings is running. CoreGraphics window list only, so it needs neither Accessibility nor Screen Recording
  • bin/: Directory for compiled native binaries (whisper-cpp, nircmd, key/mic listeners)

Helper Modules (src/helpers/)

  • audioManager.js: Handles audio device management
  • clipboard.js: Cross-platform clipboard operations
    • macOS: AppleScript-based paste with accessibility permission check
    • Windows: PowerShell SendKeys with nircmd.exe fallback
    • Linux: compositor-aware Wayland paste (Hyprland sendshortcut, wlroots wtype, GNOME/KDE portal keysyms) with native uinput/XTest and system-tool fallbacks. KDE's portal paste sends Shift+Insert to every window but Warp, which binds paste to Ctrl+Shift+V only: KWin resolves keysyms in the active layout only, so a Ctrl+V on a non-Latin layout arrives as Ctrl alone (#2055). The app runs on XWayland, which the compositor won't let read a native Wayland app's clipboard while a Wayland window is focused, so _saveClipboard falls back to wl-paste --type text when Electron reads nothing; otherwise the restore would wipe the user's clipboard. On KDE it also takes wl-paste's UTF-8 plain text whenever it differs from Electron's, since KWin can leave a stale X11 owner in place after a Wayland app copies, so a rich or image restore first writes the original's text (empty for an image) to Wayland, or the app's own transcript or selection sentinel left there would read as newer. A Wayland-only copy with no plain text (a screenshot copied in Spectacle) isn't saved, so a dictation replaces it with X11's older text
    • Linux: wlroots, Hyprland and niri (xwayland-satellite) hand XWayland's clipboard to a native Wayland app only while an X11 window has focus; Mutter, COSMIC and KWin 6.0.3+ do it themselves (KDE also writes the transcript and primary selection with wl-copy, and Settings asks for wl-clipboard there too, since _saveClipboard reads the Wayland clipboard there with wl-paste). There (requiresWlCopy in linuxSession.js; a leftover SWAYSOCK or Hyprland signature under another desktop doesn't count) the transcript reaches Wayland apps only through wl-copy, so _pasteText pastes only when wl-copy wrote both the Wayland clipboard (_writeClipboardWayland) and the primary selection that terminals paste on Shift+Insert (_writePrimarySelection). wl-copy --primary goes through data-control, which still takes a primary selection under niri disable-primary (checked on niri 26.04) and Hyprland middle_click_paste = false (its data-control code never reads that option). Otherwise it injects nothing and returns reason: "wl-copy-missing" (not installed) or "wl-copy-failed" (e.g. killed by its 200 ms timeout): the renderer shows the dictation-error pill (Retry writes and pastes the kept text again, never records), a selection edit reports code: "wl_copy_missing"/"wl_copy_failed", and the paste-tools card and Settings → Wayland Paste Setup ask for wl-clipboard (#2621). There, a missing wl-copy is probed again rather than read from the 30 s command cache, so one installed after a held-back paste counts at once
    • Linux COSMIC: paste reads the focused app from cosmic-comp's zcosmic_toplevel_info_v1 (cosmicToplevel.js, a minimal Wayland wire client, bound at version 1, which still sends app_id and state), never from xdotool, whose XWayland answer goes stale while a native Wayland window has focus. A known terminal is pasted through wtype (Ctrl+Shift+V; Shift+Insert for Konsole); everything else keeps uinput Shift+Insert, since COSMIC gives no PID to spot Electron apps hosting TUIs. Short terminal names (st) match whole words only, or every com.system76.* app id would read as a terminal
    • Linux: before any paste or selection-copy chord, linux-fast-paste --capabilities --await-modifier-release <ms> waits for physically held modifiers to be released (XKB state on X11, EVIOCGKEY on Wayland; unknown without /dev/input access pastes as before). A chord injected into held keys arrives as a different shortcut, and a virtual device cannot release another device's key (#2113). Still held after 1.5 s → not pasted, reason: "modifiers-held", and the renderer shows the dictation-error pill with the transcript on the clipboard; a selection edit blocked the same way reports code: "modifiers_held" and gets the same pill (the assistant caret paste reports the same code, for logs only). ydotoold's virtual keyboard is ignored, since an interrupted chord can leave its modifier down, and an unreadable key state is logged once per session. The wait runs before pasteLinux detects the target window, and a selection copy that had to wait looks the target up again, since focus can move while a key is held: a fresh capture then runs the command on its own (focus_moved), and a revalidated session declines as target_changed. Retry on a modifiers-held pill pastes the kept transcript again rather than recording again (the macOS/Windows push force-stop pill keeps its record-again Retry, since neither platform waits for held keys), ignores clicks while its paste is pending, and never dismisses a newer pill. useAudioRecording keeps the pill in processing until the paste attempt settles (the audio manager settles before the paste starts), but never reports that to main, which drops dictation hotkeys while processing; a paste held back after the next dictation started leaves the transcript on the clipboard without a pill. The helper measures its wait on CLOCK_MONOTONIC, since the JS watchdog kills it at timeout + 1 s
  • database.js: SQLite operations for transcription history
  • noteUndo.js: Undo for assistant update_note edits. The tool passes { undoable, expected, turn } to db-update-note; inside one transaction main refuses the edit if a guarded field changed since the tool read the note, writes it, and journals the previous values in assistant_note_undo (one row per note, bound to the account and kept 10 minutes). An AFTER UPDATE trigger retires the row when any guarded field changes again (manual save, sync pull, delete, identity or folder change), treating '' and NULL as equal because a pull stores an empty field as NULL; sync acknowledgements don't retire it. Edits of one note in the same chat turn (turn = the assistant message id) share one Undo; update_note runs one note's edits in turn, since the AI SDK runs a step's tool calls concurrently. AssistantNoteUndoListener (control panel and dictation window) offers each edit once, in the first focused window (db-claim-note-undo); typing in the open note retires it (discardNoteUndo). The trigger is recreated and a journal of another shape is dropped at launch; if setup fails, edits save without Undo
  • debugLogger.js: Debug logging system with file output
  • providerHttpErrors.js: Pure ESM classifier for BYOK provider failures (transcription and AI models), shared by main (require(esm)) and renderer. providerHttpError({ provider, status, body, headers, model, surface, selfHosted }) builds an Error carrying code (PROVIDER_AUTH_FAILED, PROVIDER_QUOTA_EXHAUSTED, …), messageKey (providerErrors.*), messageParams, settingsTarget (speechToText/llms, fixable errors only) and redacted technicalDetails; asProviderError classifies AI SDK, timeout and network errors and leaves already-keyed errors alone. Raw provider bodies never reach UI text. Rendered by src/utils/describeProviderError.ts
  • devServerManager.js: Vite dev server integration
  • dockManager.js: Single owner of the macOS Dock icon
    • The icon follows the control panel: it appears when the panel opens and goes away when the panel closes to the tray, so no other caller (in particular the dictation panel's hide path) can resurrect it
    • Every path that surfaces or hides the control panel reports that state explicitly: openers (tray, app menu, deep links, activate, second instance, first show) go through windowManager.showControlPanel(), hiding through hideControlPanelToTray()
    • showControlPanel() reports the Dock before restoring, showing and focusing the window: on an active app dock.show() activates the Dock and then re-activates the app, which macOS does not honour, so a window surfaced first is left behind the previously frontmost app
    • showControlPanel() calls show() even on an open panel, since on macOS that activates the app and focus() does not; openers that send the user to another app (meeting-prompt Join, the meeting hotkey, Upcoming Meetings' Join & transcribe) pass { activate: false }, so an open panel updates behind the meeting instead of covering it. A hidden panel is still shown
    • app.dock.show()/hide() run only when the requested state differs from app.dock.isVisible(), so reopening an already open panel never triggers that Dock hand-off
    • Never derive that state from the window's show/hide events: on macOS those are occlusion events, so they also fire when the panel is merely covered by another window, minimized, or on another Space, and the icon flickers as the user switches windows
    • Decision logic lives in dockPolicy.js (pure, unit-tested in test/helpers/dockPolicy.test.js)
  • dragManager.js: Window dragging functionality
  • environment.js: Environment variable and OpenAI API management
  • hotkeyManager.js: Global hotkey registration and management
    • Named hotkey slots: dictation, voiceAgent (voice assistant — dictation routed to the assistant panel), translation, meeting
    • Handles platform-specific defaults (GLOBE on macOS, Control+Super on Windows/Linux)
    • Auto-fallback to F8/F9 if default hotkey is unavailable
    • Notifies renderer via IPC when hotkey registration fails
    • Integrates with GnomeShortcutManager for GNOME Wayland support
    • Integrates with HyprlandShortcutManager for Hyprland Wayland support
    • Integrates with KDEShortcutManager for KDE Wayland support
    • Each press and release of a bound hotkey writes a debug-level Hotkey event line (logHotkeyEvent: source, action, key, owning slots, activation mode; key names only). Unbound keys and hotkey capture are skipped, since the macOS listener reports every right modifier and side mouse button. Every globalShortcut registration goes through registerGlobalShortcut, which logs a held key once and leaves keys the Windows/Linux listener watches to that listener; GNOME, KDE and Hyprland report through their onHotkeyEvent hook, and handleMacPushModifierUp logs the release of a macOS compound hold
  • gnomeShortcut.js: GNOME Wayland global shortcut integration
    • Uses D-Bus service to receive hotkey toggle commands
    • Registers shortcuts via gsettings (visible in GNOME Settings → Keyboard → Shortcuts)
    • Converts Electron hotkey format to GNOME keysym format
    • Only active on Linux + Wayland + GNOME desktop
    • D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
  • hyprlandShortcut.js: Hyprland Wayland global shortcut integration
    • Uses D-Bus service to receive hotkey toggle commands (same com.openwhispr.App service)
    • Registers shortcuts via hyprctl keyword bind (runtime keybinding)
    • Converts Electron hotkey format to Hyprland bind format (MODS, key)
    • Only active on Linux + Wayland + Hyprland (detected via HYPRLAND_INSTANCE_SIGNATURE)
    • D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
  • kdeShortcut.js: KDE Wayland global shortcut integration
    • Uses D-Bus to communicate with KGlobalAccel for global hotkey registration
    • Registers hotkeys via setShortcut/doRegister D-Bus calls on the KGlobalAccel interface
    • Listens for globalShortcutPressed signals to trigger callbacks
    • Converts Electron hotkey format to Qt key codes
    • Only active on Linux + KDE desktop (detected via XDG_CURRENT_DESKTOP)
    • D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)
  • autoStart.js: Single entry point for launch at login, used by ipcHandlers.js and main.js
    • Dispatches to setLoginItemSettings() on macOS/Windows and to linuxAutostart.js on Linux
    • getAutoStartState() returns { enabled, requiresApproval }; requiresApproval is macOS-only and means SMAppService registered the item but the user has not allowed it under System Settings → General → Login Items yet
    • wasLaunchedAtLoginHidden() decides whether this launch should go straight to the tray
    • syncAutoStartEntry() runs from initializeCoreManagers() and repairs entries written by older builds
    • Decision logic lives in autoStartPolicy.js (pure, unit-tested in test/helpers/autoStartPolicy.test.js)
  • autoStartPolicy.js: Electron-free launch-at-login and relaunch decisions
    • HIDDEN_LAUNCH_FLAG (--hidden) is how a login launch tells the app to start in the tray. Windows has no native equivalent (openAsHidden is macOS-only and a no-op on macOS 13+), so the flag rides on the login item's args; Linux puts it on the autostart entry's Exec; macOS uses wasOpenedAtLogin instead
    • On Windows, read the state from executableWillLaunchAtLogin, never from openAtLogin: openAtLogin only compares the Run value against the current executable and args and ignores the StartupApproved key that Task Manager and Settings write when a user disables a startup app
    • Reads and writes must pass identical args, or openAtLogin always reports false
    • getRelaunchOptions() and getRelaunchWaiter() shape the relaunch-app IPC that follows cleanup-app (Reset app data; Delete account with device erase): the relaunch drops --hidden and any cold-start deep link, and an AppImage is started again from its on-disk file ($APPIMAGE) by a detached waiter, since it runs from a FUSE mount that disappears when the app exits. The handler in ipcHandlers.js only quits under npm run dev and, on macOS with an update Squirrel already holds, hands the restart to the updater
  • linuxAutostart.js: Launch-at-login on Linux via an XDG autostart entry
    • app.setLoginItemSettings() is a no-op on Linux, so the entry is written directly to $XDG_CONFIG_HOME/autostart/open-whispr.desktop, matching the executable name electron-builder packages under
    • Exec resolves from $APPIMAGE first: process.execPath is the ephemeral AppImage FUSE mount
    • isAutostartEnabled() honors X-GNOME-Autostart-enabled=false and Hidden=true, which GNOME Tweaks and KDE's autostart editor write in place instead of deleting the file
    • Exec carries --hidden (see autoStartPolicy.js), and syncAutostartEntry() compares against the full value including that flag — comparing against the bare path would make every launch look stale
    • syncAutostartEntry() runs from autoStart.syncAutoStartEntry() in initializeCoreManagers() and re-points a stale Exec after the executable moves (renamed or auto-updated AppImage); it never re-enables an entry the user disabled, and no-ops in development
    • Unit-tested in test/helpers/linuxAutostart.test.js
  • ipcHandlers.js: Centralized IPC handler registration
  • windowsKeyManager.js: Windows Push-to-Talk support with native key listener
    • Spawns native windows-key-listener.exe binary for low-level keyboard hooks
    • Supports compound hotkeys (e.g., Ctrl+Shift+F11, CommandOrControl+Space)
    • Emits key-down and key-up events for push-to-talk functionality
    • Graceful fallback if binary unavailable
  • meetingDetectionEngine.js: Orchestrates meeting detection from all sources
    • Gates notifications during recording (tap-to-talk and push-to-talk)
    • Post-recording cooldown (2.5s) before showing queued notifications
    • Priority-based coalescing (process > audio) — one notification, not three
    • Both detectors start off and only run once the renderer has synced its saved notification preferences (sync-notification-preferences); they stop when meeting prompts are disabled and restart when re-enabled. The derivation lives in meetingDetectionPreferencePolicy.js (pure, unit-tested in test/helpers/meetingDetectionPreferencePolicy.test.js); ipcHandlers.js is a thin adapter over it
  • meetingProcessDetector.js: Detects running meeting apps
    • macOS: Event-driven via systemPreferences.subscribeWorkspaceNotification (zero CPU)
    • Windows/Linux: Shared processListCache polling (30s interval)
    • Scans are start-generation guarded: a process-list read that completes after stop() or a newer start() is discarded, since the detector is now stopped and started at runtime (notification toggles, auto-end sessions)
  • audioActivityDetector.js: Detects microphone usage for unscheduled meetings
    • macOS: Event-driven via macos-mic-listener binary (CoreAudio process objects; aggregate device activity prompts only while a known meeting app is running)
    • Windows: Event-driven via windows-mic-listener.exe (WASAPI sessions, self-PID exclusion)
    • Linux: Event-driven via pactl subscribe (PulseAudio source-output events)
    • Windows/Linux: Graceful fallback to polling if the native approach fails; macOS has no safe polling signal and respawns the listener with backoff instead
  • processListCache.js: Shared singleton process list cache (5s TTL, ps-list npm)
  • meetingEchoLeakDetector.js: Audio-layer echo analysis for meeting recordings — correlates each mic chunk against the recent system-audio tap (lag search 0–500 ms in 5 ms steps) and classifies it clean_local / suspected_render_bleed / double_talk; drives chunk muting and per-segment suppression flags. PCM-driven tests in test/helpers/meetingEchoLeakDetector.test.js
  • meetingMicGate.js: Pure RMS/peak chunk stats + the meeting mic gate verdict (send / zero for streaming, send / skip for local) with the exported silence and bleed thresholds; ipcHandlers.js (dispatchMeetingAudioBuffer, transcribeLocalMeetingChunk) only applies the verdict. Unit-tested in test/helpers/meetingMicGate.test.js
  • meetingMicHoldback.js: Pure holdback/retract policy for risky mic finals — pending-final partition, retract window, risky-profile classifier, text-layer duplicate check, racing-retract candidate selection, pending-overlap partition. ipcHandlers.js keeps thin adapters over its closure state (meetingDiarizationSegments, meetingPendingMicFinals, hasNearbyTranscriptMatch). Unit-tested in test/helpers/meetingMicHoldback.test.js
  • googleCalendarManager.js: Google Calendar sync (REST, OAuth via googleCalendarOAuth.js)
    • 10s socket timeout on API requests
    • Incremental sync via syncToken; full re-sync on 410 prunes stale events (note-linked rows retained)
    • Sync tokens pin the timeMin/timeMax window of the full sync that created them (incremental syncs never roll it forward), so tokens are discarded after 1 day to keep the 33-day lookahead covering the availability tool's 31-day horizon
  • microsoftCalendarManager.js: Microsoft Calendar sync via Graph API (OAuth via microsoftCalendarOAuth.js)
    • calendarView/delta incremental sync over a 38-day window; delta token discarded after 7 days (Graph delta links never roll their window forward)
    • Delta can return recurring-series occurrences as bare stubs (no subject/attendees/meeting link); they're backfilled from their series master, one GET /me/calendars/{calendarId}/events/{id} per series (calendar-scoped: /me/events/{id} 404s for shared calendars). A failed backfill shortens the delta token TTL to 10 min so an early full sync retries instead of leaving untitled blocks
    • Full re-sync (410 or expired token) prunes stale events like Google
  • appleCalendarManager.js: Apple Calendar (EventKit) via the macos-calendar-listener Swift helper — macOS only, snapshot-push over stdout, no tokens ("connected" = apple_calendars has rows). The helper takes --lookback-seconds (an old binary ignores it and keeps a ~2-hour lookback; rebuild with npm run compile:calendar-listener)
  • calendarCacheWindow.js: CALENDAR_LOOKBACK_DAYS (3) — every provider keeps this much past history (Google timeMin, Microsoft startDateTime, the Apple helper flag) so notes can link recent meetings offline. Widening it needs a user_version token reset in database.js (v5 did it for 3 days)
  • calendarReminderScheduler.js: Provider-agnostic meeting reminder scheduling over the shared calendar_events table (provider-scoped reset keys, so one provider's disconnect doesn't re-fire another's reminders)
  • calendarSyncInterval.js: Shared interval runner for the REST providers — exponential backoff (2min → 4min → 8min → cap 30min on consecutive failures, reset on success) and 30s focus-sync throttle
  • oauthLoopbackFlow.js: Shared PKCE auth-code flow over an ephemeral 127.0.0.1 server, used by both calendar OAuth helpers
  • Events from all providers land in the shared calendar_events table with a provider column; queries suppress the Apple copy of a meeting when a REST row occupies the same time slot + title (Calendar.app mirrors the same accounts); all-day rows pair by local start and end days + title, since REST stores plain dates and the Apple helper local instants
  • The note header's date/attendees dropdown (NoteParticipants.tsx + NoteEventPicker.tsx) links a note to a stored event (getLinkableCalendarEvents, local only): linking merges the event's guests into participants in the same write, leaving out rooms and declined guests and leaving attendees already on the note as they are, and tags the ones it adds addedByEvent: <event id> (meeting detection tags the attendees it copies onto new notes the same way, via eventAttendees.js); the user's own self row (every calendar sync stores self as the user, Google's shared calendars included) is kept whatever they answered and never tagged, since they recorded the note whichever event it links. Removing or switching the event drops exactly the tagged ones, and speaker counts skip resource attendees; Join only opens the link; web_link (Google htmlLink, Outlook webLink, none for Apple) backs "Open in calendar". An unlink is stored as calendar_event_id = "", never NULL (updateNote converts an explicit NULL on a linked note), because the cloud and upsertNoteFromCloud COALESCE a NULL as "unchanged"
  • menuManager.js: Application menu management
  • toneAppCategories.js: Tone & Style's app classifier (pure, test/helpers/toneAppCategories.test.js): an app identity, window title and page host map to personal, work, email or other (see Tone & Style)
  • tray.js: System tray icon and menu
  • whisper.js: Local whisper.cpp integration and model management
  • parakeet.js: NVIDIA Parakeet model management via sherpa-onnx
  • parakeetServer.js: sherpa-onnx CLI wrapper for transcription
  • qdrantManager.js: Qdrant vector DB sidecar process lifecycle (spawn, health check, shutdown)
  • semanticSearchLifecycle.js: Single owner of the semantic search resources — starts Qdrant and the embedding model on demand, drains the SQLite change journal, and releases both after 5 minutes idle
  • localEmbeddings.js: Local text embedding via ONNX Runtime + all-MiniLM-L6-v2 (384-dim vectors)
  • vectorIndex.js: Qdrant collection management — upsert, delete, search
  • windowConfig.js: Centralized window configuration
  • plasmaWorkAreas.js: Per-monitor work areas on KDE Plasma. On X11 and XWayland, Chromium clips only the primary display with the single root _NET_WORKAREA, which KWin builds from every monitor's panels at once (#1110). This asks plasmashell over D-Bus (evaluateScript) for each screen's always-visible panels and rebuilds each display's workArea for the dictation pill, its companion windows and dragging. On X11, Qt reports a screen's origin in device pixels and only its size scaled, so those origins are divided by the display scale before matching. It refreshes on display events and each time a dictation shows the pill, retrying after 1, 5 and 15 s when plasmashell isn't ready; a display event starts those retries over. An auto-placed pill re-anchors when the result changes; a dragged one stays put. With a single display, or outside Linux + KDE, Chromium's work area is used as is
  • windowManager.js: Window creation and lifecycle management
  • cliBridge.js: Loopback HTTP server on ports 8200–8219, bearer-token auth (token at ~/.openwhispr/cli-bridge.json), 127.0.0.1-only. Used by the unified CLI to talk to a running desktop app. POST /v1/transcribe takes a file path (never audio) and runs the user's downloaded local model through IPCHandlers.transcribeLocalFile, approving the path with approveAudioPath first; GET /v1/transcribe/models lists local models with download state and the app's default (localTranscriptionModels.js, read from the .env pre-warm values).
  • connectors/: Agent connectors
    • connectorManager.js owns every outside action: prepare → ApprovalCard → commit for anything other people see, runDirect for private drafts. The model can never reach commit
    • Three action kinds per action, fixed in code: approval (prepare → card → commit), direct (runDirect) and query (a read for the model, e.g. an issue search). connectorManager.query runs prepare's gates (policy, action kind, account, the bound login), also refuses a login whose ownerAccountId is another account (account_changed), and writes no receipt; queryResult.js caps what reaches the model (20 flat items, 16 fields, strings cut at 1,000 characters, truncated on any cut) and strips characters that hide text (controls, DEL and C1, bidi overrides, zero-width spaces, tag characters; ZWNJ and ZWJ stay). runQueryAction marks the items untrusted, with guidance never to follow instructions in them, and takes the attendee fence's tag name out of them (withoutAttendeesFence); the chat keeps no copy of a search's items in the saved (and synced) conversation (isQueryResultData)
    • Registration is by list: createConnectors.js (CONNECTOR_FACTORIES, one build<Name>Connector(deps) per provider module) in main, connectorToolModules.ts (CONNECTOR_TOOL_MODULES, registered by readyConnectorIds()) for tools, and connectorRows.tsx (CONNECTOR_ROWS) for Settings rows. Each tool carries its own promptInstruction and connectorId; the connector rules join the prompt when any connector tool is offered
    • runApprovalAction caps a turn at 5 approval cards across connectors (approval_card turn slot, given back when no card appears) and holds every approval action's turn off the caret (onHoldDelivery), whatever the outcome; a turn's scope passes one hold per tool call on to the chat surface, plus a later one that newly keeps the clipboard. Cards lay out fields by verbKey: email, issue (title and description) and comment; a sent commit may name what it created (resultLabel, shown as "Created ENG-124")
    • pendingActions.js: pending → committing → sent/failed/unknown; actions are bound to the connection (account, workspace, generation) they were prepared on and to the OpenWhispr account that prepared them (another account's commit is withdrawn as account_changed)
    • connector_actions (SQLite) is the durable receipt: destination labels, states and links only, never content. Each row is stamped with the account bound to the credential the action ran under (accountScopeBinding, read before and after the policy wait; with no account, prepare and runDirect refuse as receipt_unavailable; a commit under a changed or missing account is withdrawn as not_sent/account_changed), so the connector-recent-actions history IPC lists only that account's rows and account deletion removes them. Interrupted rows are reconciled on launch (committing → unknown; rows with no account are deleted), expired pending actions are recorded as expired, a direct run the connector stopped (not_sent) is cancelled, and a direct run that throws or returns a malformed result is unknown (it may have happened), never failed. Every connector result is whitelisted and type-checked per state before it is recorded or reaches the renderer. A cancel (Esc) reaches a direct run until its side effect through runtime.signal, checked right before the compose window opens
    • Policy fails closed (connectorPolicy.js's connectorPolicyState classifies the snapshot; connectorIpc.js's createConnectorPolicyResolver enforces the 1.5s deadline): only a successful, well-formed snapshot allows; unresolvable, any error code, or >1.5s with no verdict already held for the account (workspacePolicyManager.peekPolicy) → unavailable; org switch features.connectorsEnabled, and agentEnabled: false turns connectors off too
    • emailConnector.js opens Gmail/Outlook/mailto compose windows (emailCompose.js, shared ESM); a body that won't fit the compose URL (2,000 chars; 6,000 for Gmail on macOS/Linux, since Windows caps opened URLs near 2,081) goes to the clipboard after the OS accepts the link (on Linux a mailto is refused first when xdg-mime reports no default — empty output with exit 0, or exit 4 on KDE 5 — except under Flatpak, where the portal picks the handler), then the subject too if it's still too long; a recipient list that alone doesn't fit is refused. email_draft opens at most 3 drafts per turn, and only one of them may use the clipboard (ToolExecutionContext.claimTurnSlot; a draft that didn't open gives its slots back, and the compose preview is built before any slot is claimed). Subject and body go through toWellFormed(), the renderer NFC-normalises addresses (bareEmailAddress), and a recipient with a non-ASCII domain is shown with its punycode form (recipientLabel) to the model, in the tool step and in receipts. Automatic picks Send from chat (Gmail) when the Gmail connector is connected (including one that needs reconnecting), then the Gmail compose link when a Google calendar is connected, otherwise the Microsoft tenant rule as today: Outlook work vs Outlook.com from the Microsoft account's tenant id (tid, consumer tenant 9188040d-…), falling back to the email domain for accounts connected before it was stored (resolveEmailDraftTarget in src/utils/emailDraftTarget.ts)
    • Gmail send (gmailMime.js, gmailApi.js, gmailAuth.js, gmailConnector.js) is an approval connector: with Gmail connected (Settings → Integrations → Connectors), email_draft shows an editable card and commit sends through the Gmail API. The only Gmail scope is gmail.send (plus openid email); a grant without it, or without a verified Google address (email_verified), is revoked, and a refresh is bound to the card's login (single-flight per OpenWhispr account and login). A login records the OAuth client that issued it (clientId). A refresh Google refuses is never reported as a network error: invalid_grant, another OAuth refusal (not invalid_client/unauthorized_client/deleted_client, build faults that are gmail_unavailable), or a login issued to a client this build no longer uses flags the login for reconnecting, and a Workspace admin block (admin_policy_enforced, org_internal) is domain_policy. Google's /revoke ends the user's whole grant to a Cloud project, so Disconnect (and a refused grant) skips it when a connected Google Calendar account with the same address is on the same project as the login's own client (sharesCalendarGrant, by the client id's project number) and only deletes the local tokens; the disconnect result then carries grantKept and the Settings row says Google still lists the app until the calendar is disconnected too. The shared-grant check reads the calendar accounts only once the projects match. A revoke is skipped the same way when another OpenWhispr account on the device has a Gmail login for the same Google address on the same project (readAllAccounts), since it holds the same grant. Connecting a different Google (or Slack) account over an existing login revokes the replaced one (loginKey); reconnecting the same account doesn't, since Google's revoke would end the new login's grant too. The Google client is GMAIL_CLIENT_ID/GMAIL_CLIENT_SECRET when both are set, else the calendar pair (GOOGLE_CALENDAR_CLIENT_ID/_SECRET); with neither, Gmail reports configured: false and its row hides. The card blocks Send on an empty To, a bad address, more than 50 recipients, a subject over 250 characters or a body over 512 KB as sent (UTF-8, CRLF; MAX_EMAIL_BODY_BYTES, emailBodyBytes), a size that with the largest headers the other limits allow stays under the 1 MB raw cap (src/utils/emailApprovalFields.ts, limits shared from emailCompose.js; To/Cc accept , ; and the Arabic and CJK commas, and a pasted Name <address> — including Outlook's Last, First <address> — becomes its bare address, with the resolved addresses shown under the field); main re-validates the same limits at Send (addresses as dot-atoms of at most 64 bytes without =?, ≤ 50 recipients, subject ≤ 250 characters without line breaks, body ≤ 512 KB, raw message ≤ 1 MB — all checked before any token refresh) and refuses CR/LF in every header. 5xx, a timeout after writing, and a 200 without a message id are unknown (link to the Sent folder; no retry); a 401 is refreshed and retried once under the same login; a 429 asking for ≤ 5 s is retried once; a 413 is too_long. A connector card whose preview shows a field the action doesn't declare editable is refused as malformed, so a typo in a declaration can't silently drop the user's edit. Release rule: a release build falls back to the calendar's Google client, so gmail.send must pass Google verification before a release ships it; develop against a separate Testing project with only gmail.send (GMAIL_CLIENT_ID/_SECRET), never the production calendar project
    • Approval actions must declare their editable fields (actions.<action>.editable: addresses, line, text; a missing declaration or an unknown type throws when the manager is created); a preview field must be declared and hold its declared type, or there's no card; sanitizeEdits keeps exactly those, and a declared field of the wrong type withdraws the card as not_sent/invalid_edit rather than sending the prepared value. Previews may carry fields, which the approval store edits and the tool result reports as final (finalText for a card without fields) on a sent, failed or unknown Send alike. connectorErrorText() words card and tool-step errors per connector (connectors.<scope>.errors.<connectorId>.<code> before the shared key)
    • Reset app data calls connectorManager.revokeAllStored() before deleting userData/connectors: it revokes every login on the device, for every account, even with no one signed in (Settings signs out before cleanup-app), and Gmail even on a grant shared with the calendar. Deleting an account uses disconnectAll() for the signed-in account's logins; a Gmail grant shared with a calendar is kept, unless the user also erases the device (delete-account-data's { erasingDevice }), since cleanup-app then finds no Gmail login left to revoke. Signing out keeps logins (every action is refused while signed out)
    • contactSearch.js (find_contact) searches the meetings nearest to now on selected calendars (SELECTED_CALENDAR_EVENT_FILTER; for Google only primary calendars, since a colleague's shared calendar isn't the user's meetings, which also leaves out the user's own secondary Google calendars; shared Microsoft or Apple-mirrored calendars still count; calendar_events only spans the sync window; cancelled and declined meetings don't count) plus the contacts table, most recently synced first, behind the same org policy check as the actions. Excluded: the user's calendar accounts and their Microsoft addresses (/me mail, userPrincipalName and proxyAddresses, stored in microsoft_calendar_tokens.own_addresses by fetchCalendars, which also purges them from contacts), self attendees, an Apple organizer who is the current user, any address flagged resource in any stored event (rows with unreadable attendee JSON are skipped) (rooms, flagged by all three calendar managers) and *.calendar.google.com addresses; the calendar managers also keep the user and rooms out of contacts and purge those addresses when they see them again. Sources are many-to-many (contact_sources: google:<account>, microsoft:<account>, apple, manual): disconnecting an account removes its source rows and deletes only contacts no other source still has; the lookup reads contacts with a manual or connected-account source (rows from before sources existed are left out; note-participant autocomplete still reads every row). Picking an existing contact in a note never adds manual, so it can't make a synced contact permanent
    • Slack (slackConnector.js): PKCE user token through oauthLoopbackFlow.js. Slack only distributes apps with HTTPS redirect URLs, so it redirects to the openwhispr.com relay (openwhispr-website, app/auth/slack/callback/route.ts), which forwards to the loopback server by the port in state (v1.<port>.<nonce>). The loopback server shows a local result page, and the flow waits 5 minutes. SLACK_OAUTH_REDIRECT_URI points a dev build at a website preview. Rotating tokens refresh single-flight, only for the login a pending action is bound to, and are saved before use; transient refresh errors never disconnect (slackAuth.js). Channels the user is in and people are cached 10 min; a list cut off at the page cap resolves only an exact #channel or an email (slackDirectory.js, targetResolution.js). conversations.open runs only at Send. Disconnect revokes the refresh and access tokens. Failures follow SLACK_PRE_SEND_REJECTIONS (slackApi.js); everything else is unknown
    • Linear (linearApi.js, linearAuth.js, linearTeams.js, linearConnector.js, tools in linearTools.ts): PKCE login with no client secret (LINEAR_CLIENT_ID), scopes read, issues:create, comments:create. Linear matches a registered redirect exactly, port included, so it redirects to the openwhispr.com relay (openwhispr-website, app/auth/linear/callback/route.ts), which forwards to the loopback server by the port in state; LINEAR_OAUTH_REDIRECT_URI points a dev build at a website preview (https only). linear_search_issues is a query action (≤ 10 compact results, untrusted); linear_create_issue and linear_comment go through the issue/comment cards. Teams and projects are cached 10 minutes per login (linearTeams.js); a name that matches nothing in a list older than 30 s reads it once more, and a project the team doesn't have is a question, never dropped. Lengths count characters, as the card does. Links in results and receipts keep no title slug (linear.app/<urlKey>/issue/<KEY>). A create and a comment each carry a client UUID id, made at prepare, so an uncertain one is looked up once: found is sent; a comment the lookup misses stays unknown. Connecting another Linear user or workspace over a login revokes the replaced one (loginKey). A GraphQL error inside a 200 is failed only for a listed rejection code (LINEAR_PRE_SEND_REJECTIONS) with no data beside it; Linear reports a missing entity as INPUT_ERROR "Entity not found", and a repeated create id ("conflict on insert") is not a refusal
    • GitHub (githubApi.js, githubDeviceFlow.js, githubAuth.js, githubInstallations.js, githubConnector.js, tools in githubTools.ts): a GitHub App with the device flow and no client secret (GITHUB_APP_CLIENT_ID, GITHUB_APP_SLUG; release builds write them to .env from the GH_APP_CLIENT_ID/GH_APP_SLUG secrets, since secret names can't start with GITHUB_; the row hides unless both are set); permissions Issues RW, Pull requests RW, Metadata R, so nothing can read code. The 8-hour token refreshes without a secret and the rotated refresh token is saved before use. Everything is limited to the repositories the App is installed on (githubInstallations.js, 60 s cache per login, one read at a time shared by everyone asking, and a read in flight when the list is cleared is never cached; a repo missing from a list older than 5 s is looked for in a fresh one, so installing the App from a "not installed" answer works at once). A bare repo name matches exactly first, then loosely through resolveTarget ("open whispr" for open-whispr), but only in a complete list; a miss in a list cut at 1,000 repositories per installation is repo_unlisted (the App may be installed after all), not not_installed; the row's Choose/Manage repositories opens the install page (manageUrl). The status never waits on GitHub: it reports the last repository count read and reads again in the background (reusing a list younger than 5 s), announcing a changed count, the first read for a login even when it failed, or a login the read found needing a reconnect, through notifyStatusChanged; until that first read finishes (after a connect, and at launch) the status carries workspaceLabelPending: true, a field no other connector sends (the manager asks every connector at once, so a GitHub read would hold up the others). Search result titles and snippets are redacted of email addresses over a bounded window only, since the redaction pattern is quadratic on long runs of word characters. github_search_issues is a query action; a GitHub App user token must search issues and pull requests apart (is:issue/is:pull-request, or a 422), so type: "any" runs both and merges them newest first. The user's own repo:/org:/user: qualifiers are removed (in groups too, with any operator they leave behind), and so are their own is:/type: issue or pull-request qualifiers, since the tool's type wins; a quoted phrase is left as written, and a quote left open or a parenthesis without its pair is dropped, since either would take in the qualifiers added after the words; a state the query names itself (is:closed, …) replaces the default state:open, and an invalid state or type is refused rather than defaulted. GitHub's 256-character limit counts only the words, so the repo: list is bounded by the size of q as sent in the query string (MAX_SEARCH_QUERY_LENGTH). GitHub answers 422 for a repo: the user can no longer search (unticked, deleted, access lost), so a 422 re-reads the installed repos fresh and searches once more. Suspended installations are skipped. github_create_issue and github_comment go through the issue/comment cards, with a live "This will notify @…" note while the card is pending (githubMentions.ts, liveCardNotes.ts), and a sent result tells the model who was notified (notified, from the card's final fields when edited, else the call's own). github_comment takes owner/repo#N, a github.com link, or repo#N/#N, whose repo is found among the installed ones (a question when ambiguous). A new issue needs a repo with issues on (archived and issues-off repos aren't offered as candidates), and more than 10 labels is refused rather than trimmed. A 401 on a token another request already refreshed reuses that token (refreshRejected), since every GitHub refresh ends the previous one. Connect shows the device code in the row (sent over the connector-connect-progress broadcast) with its own Cancel (connectInRow); connector-cancel-connect stops it, even while it still waits on the policy check (pendingConnects in connectorIpc.js), as do leaving Settings, the org turning connectors off, an account switch or a sign-out in main (connectorManager.accountChanged). A device code that can't be asked for is network (no answer, a timeout, a 408 or 5xx) or rate_limited (a 429), not token_exchange_failed, since the user never saw a code. The device poll keeps going through network failures until the code expires and treats a bare 429 like slow_down; the /user read after approval is retried once. A refresh answered with 408 or temporarily_unavailable/server_error keeps the login and is tried once more; a 429 keeps it without an immediate retry (rate_limited), and a refusal of the OAuth client itself (incorrect_client_credentials, device_flow_disabled, unauthorized_client, invalid_client: a build with a wrong client id) is not_configured and never flags the login. An archived repo takes no new issue or comment, and a locked conversation only takes a comment from someone with push access. Writes have no idempotency key, so anything uncertain after the send is final unknown. Only Reset app data and account deletion (the manager's removingAll) end the tokens, through GitHub's unauthenticated POST /credentials/revoke (an authenticated call is refused). That endpoint is meant for exposed credentials and emails the user, so Disconnect, a reconnect and a connect that can't be saved only delete the login locally; the App's authorization stays on github.com until the user removes it there
    • Token handling shared by the OAuth connectors (boundLogin.js): reads, the reconnect flag, saving a refreshed token and one refresh at a time, only ever for the login an action is bound to. Gmail's and Linear's clients share one timed POST and its body helpers (providerHttp.js); GitHub's uses the body helpers
    • Logins: one encrypted file per OpenWhispr account and connector under userData/connectors/ (atomic writes, 0o700, connectorCredentials.js). Every write names the account and the generation it started from, so an OAuth round trip, refresh or disconnect that outlives a reconnect or account switch writes nothing. Bindings carry ownerAccountId; receipts carry account_id. Signed out → every connector action is refused (signed_out). Pending actions also expire in main (sweepExpired, every minute)
  • postMigrationDetector.js: Detects users returning from the pre-Gizmo bundle ID via a .bundle-migrated sentinel in userData; consumed by ipcHandlers.js to drive the PostMigrationOnboarding modal
  • permissionGuideManager.js: macOS onboarding permission overlay — a frameless always-on-top window anchored to the bottom of the System Settings window (placement in permissionGuidePlacement.js, pure), polled one read at a time via settingsWindowState.js (exec of macos-window-bounds). Steps aside for authorization prompts, other apps and a dialog on another Space; closes when the dialog closes; never subscribes to the control panel's hide (occlusion fires it), so hideControlPanelToTray() closes it explicitly. Renderer side: permissionGuideController.ts (pure state machine, unit-tested) + usePermissionGuide.ts
  • templatePrompts.js: Pure compiler from a note template or action to the system prompt the model receives (see Note Templates and Actions). Shared by actionProcessingStore.ts, the note chat, and the live canary's note probe

React Components (src/components/)

  • App.jsx: Main dictation interface with recording states
  • ControlPanel.tsx: Settings, history, model management UI
  • OnboardingFlow.tsx: First-time setup wizard over a versioned, route-based step machine (src/components/onboarding/flow.ts; session persisted in onboardingSessionV2). Routes vary by auth path (account vs guest), policy (agent allowed), and setup mode (cloud / BYOK / local / enterprise); step components live in src/components/onboarding/
  • PostMigrationOnboarding.tsx: One-time modal for users returning from the pre-Gizmo bundle ID; reuses PermissionsSection to walk through re-granting Microphone, Accessibility, and System Audio. Triggered by postMigrationDetector.js (see Helper Modules)
  • SettingsPage.tsx: Comprehensive settings interface
  • WhisperModelPicker.tsx: Model selection and download UI
  • ConnectorsSection.tsx: Integrations → Connectors: the "Draft emails in" picker (Automatic shows what it resolved to; Send from chat (Gmail) says to connect Gmail until it can send, and the card's description follows where drafts actually go), a paid-plan upsell, "turned off by your organization" for a resolved org block and "not available right now" while the policy is loading, failed or requires a newer app
  • ui/: Reusable UI components (buttons, cards, inputs, etc.)
  • ui/RichTextEditor.tsx: Tiptap note editor. Note bodies (content, enhanced_content) are stored as Markdown via tiptap-markdown (html: false). NoteBottomBar publishes its height in --bottom-overlay-height; the editor reads it for its bottom scroll margin and threshold, its bottom padding and the line menu's flip padding, so the caret and the menu stay clear of the bar
    • RichTextEditorExtensions.ts holds the extension list (RichTextEditor adds the mention extension in front of it); order matters: at equal priority, later extensions' keymaps and clipboard props run first
    • RichTextEditorTable.ts keeps every table to what a GFM pipe table can store: header first row, one single-line paragraph per cell, no merged cells, no alignment or column widths. Pasted cells are flattened while parsing, a normalizer fixes what commands and pastes leave behind, and the table serializer escapes | in cells. Tests: test/components/richTextEditor.test.js (happy-dom)
    • Floating menus: RichTextEditorFormatMenu.tsx shows one formatting toolbar (marks, text style, lists, quote, insert table) above selected text (Tiptap BubbleMenu) and below the caret on an empty top-level line, flipping above when there's no room (FloatingMenu); RichTextEditorTableMenu.tsx is the table's ⋯ menu. They share RichTextEditorMenus.ts: a menu hides when focus leaves it, so dropdowns portal into the menu element itself, never into the editor's scroller (EditorContent moves the scroller's children when a note closes); useHideOnFocusLeave hides a menu when focus leaves from inside it or after a click on it, which Tiptap misses. The menus detach their element without unmounting it, so buttons use a native title rather than <Tooltip>, and a dropdown inside a menu is controlled and closed from the menu's onHide — otherwise it stays open over a scroll-locked page when the menu hides. Keep menu props stable, or each render dispatches an updateOptions transaction. The line menu is placed again in the frame after a transaction that scrolled the caret into view, since ProseMirror scrolls after the menu measured

React Hooks (src/hooks/)

  • useAudioRecording.js: MediaRecorder API wrapper with error handling
  • useAssistantPanel.js: Assistant panel lifecycle (open/close choreography, thinking flourish, footer phases, pending voice commands, conversation session boundaries)
  • useLiveTranscriptPanel.js: Live transcript panel lifecycle (entrance choreography, buffered text scheduler, measure-then-reveal resize pipeline, preview IPC wiring)
  • useMainWindowSizeOwner.js: Single owner of the main window size ladder (panel > menu > toast > pill > base) and the dictation-error pill handoff
  • useWindowResizeCompensation.js: Masks split native setBounds frames via CSS-variable counter-translation
  • useMainProcessNotifications.tsx: Main-process notifications (hotkey fallback, GPU fallback, learned corrections) as toasts
  • useClipboard.ts: Clipboard operations hook
  • useDialogs.ts: Electron dialog integration
  • useHotkey.js: Hotkey state management
  • useLocalStorage.ts: Type-safe localStorage wrapper
  • usePermissions.ts: System permission checks and settings access
    • openMicPrivacySettings(): Opens OS microphone privacy settings
    • openSoundInputSettings(): Opens OS sound input device settings
    • openAccessibilitySettings(): Opens OS accessibility settings (macOS only)
  • useSettings.ts: Application settings management
  • useWhisper.ts: Whisper binary availability check

Services

  • ReasoningService.ts: AI processing for agent-addressed commands
    • Detects when user addresses their named agent and removes the agent name from final output
    • Provider implementations live in a registry at src/services/ai/inferenceProviders/index.ts covering 8 providers (anthropic, enterprise, gemini, groq, lan, local, openai, openwhispr), each implementing the InferenceProvider interface from types.ts
    • Per-scope LLM config: 4 scopes (dictationCleanup, dictationAgent, noteFormatting, chatIntelligence) defined in src/config/inferenceScopes.ts
    • selectResolvedLLMConfig(state, scope) in settingsStore.ts resolves provider/model per scope with fallback chains

whisper.cpp Integration

  • whisper.js: Native binary wrapper for local transcription
    • Bundled binaries in resources/bin/whisper-cpp-{platform}-{arch}
    • Falls back to system installation (brew install whisper-cpp)
    • GGML model downloads from HuggingFace
    • Models stored in ~/.cache/openwhispr/whisper-models/

NVIDIA Parakeet Integration (via sherpa-onnx)

  • parakeet.js: Model management for NVIDIA Parakeet ASR models

    • Uses sherpa-onnx runtime for cross-platform ONNX inference
    • Bundled binaries in resources/bin/sherpa-onnx-{platform}-{arch}
    • Windows: the bundled ONNX Runtime ships as ow-onnxrt.dll, not onnxruntime.dll. scripts/download-sherpa-onnx.js renames it and rewrites the import tables of every sherpa image (scripts/lib/pe-imports.js) because Windows 11 ships an older onnxruntime.dll in System32 and some loader configurations resolve the bare name to that copy (#2054). afterPack.js fails the Windows build if onnxruntime.dll is present or ow-onnxrt.dll is missing
    • Uses the upstream -no-tts archives and bundles only the three executables plus ONNX Runtime. The TTS builds statically link espeak-ng (GPL-3.0) into sherpa-onnx-c-api, and nothing in the app loads the sherpa C/C++ API libraries, so they are not shipped
    • INT8 quantized models for efficient CPU inference
    • Models stored in ~/.cache/openwhispr/parakeet-models/
    • Server pre-warming on startup when LOCAL_TRANSCRIPTION_PROVIDER=nvidia is set
    • Provider preference persisted to .env via saveAllKeysToEnvFile() on server start/stop
  • Available Models:

    • parakeet-tdt-0.6b-v3: Multilingual (25 languages), ~680MB
    • parakeet-unified-en-0.6b: English-only, ~631MB, state-of-the-art EN accuracy (5.91% avg WER on Open ASR Leaderboard)
    • nemotron-speech-streaming-en-0.6b: English-only, ~632MB, cache-aware streaming FastConformer ("runtime": "online" in the registry)
    • nemotron-3.5-asr-streaming-0.6b: Multilingual (15 transcription-ready languages, auto detection), ~650MB, cache-aware streaming FastConformer ("runtime": "online")
    • cohere-transcribe-03-2026: Multilingual (14 languages, NO auto detection — one language per server start), ~1.7GB download / ~2.7GB on disk, Conformer encoder-decoder ("modelType": "cohere-transcribe"), most accurate local model (5.42% avg WER). Exposed as the "cohere" local provider in settings/UI but served by the same ParakeetManager/sherpa-onnx offline WS server; the dictation language is part of the server identity, so changing it restarts the server. Segments long audio at 30s (model max ~35s)
  • Runtimes: Models are offline (default) or online per their registry runtime field. Offline models use the bundled sherpa-onnx-ws-{platform}-{arch} (offline websocket server); online models use sherpa-onnx-online-ws-{platform}-{arch} (online websocket server). Both are downloaded by scripts/download-sherpa-onnx.js. Partial/final JSON results are merged by parakeetWsResult.js.

  • Streaming Commit (online models): For online-runtime models, dictation streams worklet PCM to a persistent websocket stream during capture (regardless of the preview toggle) and commits the flushed text at stop as the final transcript — no second decode of the recording. The stop flush is truncation-aware (finish() extends its deadline while results keep arriving and flags truncated); anything but a clean flush falls back to the record-then-transcribe path (transcribe-local-parakeet), which for online models decodes the whole recording over a single stream (no 15s segmentation; segments only bound memory for very long files).

  • Live Transcription Preview: When the preview toggle is on and an online-runtime model is selected, the preview shares the streaming-commit websocket: partial results update the preview window live (replacing text via showTranscriptionPreview). Offline models keep the 1.5s buffered-chunk path (appending via appendTranscriptionPreview). If the stream can't start or dies mid-dictation, the preview falls back to the chunked path and the final transcript falls back to the full decode. Tests: test/helpers/parakeetOnlineStream.test.js (mock websocket server).

  • Download URLs: Models from sherpa-onnx ASR models release on GitHub

Local Semantic Search (Qdrant + MiniLM)

Offline semantic search that finds notes by meaning, not just keywords. Used by the AI agent's search_notes tool. Its resources are lazy (#2143): Qdrant and the embedding model start on the first semantic search (or the first vector write while the index is active), the embedding model downloads on that first activation if missing, and everything is released after 5 minutes idle. While asleep, keyword FTS5 serves searches and note changes are journaled in SQLite.

Architecture:

  • Qdrant sidecar: Rust binary spawned as child process (qdrantManager.js), port 6333–6350
  • Embedding model: all-MiniLM-L6-v2 via ONNX Runtime (localEmbeddings.js), 384-dim vectors
  • Vector index: Qdrant collection management (vectorIndex.js), cosine distance
  • Lifecycle owner: semanticSearchLifecycle.js — the only caller that starts or stops Qdrant, the embedding model and the collection
  • Hybrid search: FTS5 + Qdrant in parallel → Reciprocal Rank Fusion (K=60) with 0.3 cosine score threshold

Pipeline:

  1. App launches → nothing starts. The first db-semantic-search-notes call activates the lifecycle: embedding model downloaded if missing (~22MB) → Qdrant binary starts → collection ensured → the journal is drained. A cold search does not wait: it answers with FTS5 results while activation runs in the background. An existing collection serves searches while the journal drains; a newly created one waits until that activation's drain finishes. A journal row whose upsert keeps failing is parked after three failed attempts, so one bad note never blocks the rest of the index
  2. Note create/update/delete → SQLite write → triggers journal the note id in pending_vector_changes → IPCHandlers.notifyVectorChanges() wakes an already-active index to drain the journal; an idle index drains it on its next activation
  3. Agent searches → db-semantic-search-notes IPC → parallel FTS5 + vector search → RRF merge → ranked results
  4. 5 minutes without a search or write → Qdrant is stopped and the embedding session released; FTS5 keeps serving

Search fallback chain (in searchNotesTool.ts): cloud search → local semantic → FTS5 keyword

Storage:

  • Qdrant data: ~/.cache/openwhispr/qdrant-data/ (qdrant-data-dev/ in development)
  • Qdrant binary: resources/bin/qdrant-{platform}-{arch} (bundled — downloaded during prebuild / predev:main)
  • Embedding model: ~/.cache/openwhispr/embedding-models/all-MiniLM-L6-v2/ (downloaded on the first semantic search)
  • Change journal: pending_vector_changes table in the notes SQLite database, maintained by triggers

Dependencies: @qdrant/js-client-rest, onnxruntime-node

Dev setup: The Qdrant binary downloads automatically via predev/prestart. The embedding model downloads on the first semantic search. To manually download: npm run download:qdrant and npm run download:embedding-model.

Build Scripts (scripts/)

  • download-whisper-cpp.js: Downloads whisper.cpp binaries from GitHub releases
  • download-llama-server.js: Downloads llama.cpp server for local LLM inference
  • download-nircmd.js: Downloads nircmd.exe for Windows clipboard operations
  • download-windows-key-listener.js: Downloads prebuilt Windows key listener binary
  • download-windows-mic-listener.js: Downloads prebuilt Windows mic listener binary
  • download-sherpa-onnx.js: Downloads sherpa-onnx binaries for Parakeet support
  • download-qdrant.js: Downloads Qdrant vector DB binary for local semantic search
  • download-minilm.js: Downloads all-MiniLM-L6-v2 ONNX model + tokenizer for local embeddings
  • download-brand-fonts.js: Fetches the licensed Yowza font files from the private OpenWhispr/brand-assets release into the git-ignored src/assets/fonts/yowza/. Runs in predev:main and every prebuild* chain; skips (Noto Sans fallback) without repo access, fails only when BRAND_FONTS_REQUIRED=1 (release CI)
  • sync-nucleo-icons.js: Regenerates src/components/icons/ from nucleo-map.json using the local Nucleo install (~/.nucleo/skills); only the icons the app uses are vendored
  • build-globe-listener.js: Compiles macOS Globe key listener from Swift source
  • build-macos-mic-listener.js: Compiles macOS mic listener from Swift source
  • build-macos-window-bounds.js: Compiles the System Settings window reporter from Swift source
  • build-windows-key-listener.js: Compiles the Windows key listener (MSVC, MinGW or clang) whenever compile:native runs on a Windows host, downloading the prebuilt binary when no compiler works; skips a binary newer than its source
  • run-electron.js: Development script to launch Electron with proper environment
  • lib/download-utils.js: Shared utilities for downloading and extracting files
    • fetchLatestRelease(repo, options): Fetches latest release from GitHub API
    • downloadFile(url, dest): Downloads file with progress and retry logic
    • extractZip(zipPath, destDir): Cross-platform zip extraction
    • parseArgs(): Parses CLI arguments for platform/arch targeting
    • Supports GITHUB_TOKEN for authenticated requests (higher rate limits)

Key Implementation Details

1. FFmpeg Integration

FFmpeg is bundled with the app and doesn't require system installation:

// FFmpeg is unpacked from ASAR to app.asar.unpacked/node_modules/ffmpeg-static/

2. Audio Recording Flow

  1. User presses hotkey → MediaRecorder starts
  2. Audio chunks collected in array
  3. User presses hotkey again → Recording stops
  4. Blob created from chunks → Converted to ArrayBuffer
  5. Sent via IPC
  6. Main process writes to temporary file
  7. whisper.cpp processes file → Result sent back
  8. Temporary file deleted

For offline local engines (whisper.cpp, Parakeet/Orukeet, Cohere) the renderer also keeps a 16 kHz mono PCM copy of the same stream from the moment the mic opens (pcmTap.js, pre-roll included, capped at 4 minutes) and sends that WAV instead of the WebM, so no FFmpeg decode runs after stop; the WebM still goes to history and remains the fallback for cloud providers and long recordings.

3. Local Whisper Models (GGML format)

Models stored in ~/.cache/openwhispr/whisper-models/:

  • tiny: ~75MB (fastest, lowest quality)
  • base: ~142MB (recommended balance)
  • small: ~466MB (better quality)
  • medium: ~1.5GB (high quality)
  • large: ~3GB (best quality)
  • turbo: ~1.6GB (fast with good quality)

4. Database Schema

CREATE TABLE transcriptions (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
  original_text TEXT NOT NULL,
  processed_text TEXT,
  is_processed BOOLEAN DEFAULT 0,
  processing_method TEXT DEFAULT 'none',
  agent_name TEXT,
  error TEXT
);

5. Settings Storage

Settings stored in localStorage with these keys:

  • whisperModel: Selected Whisper model
  • useLocalWhisper: Boolean for local vs cloud
  • language: Selected language code
  • agentName: User's custom agent name
  • reasoningModel: Selected AI model for processing
  • reasoningProvider: AI provider (openai/anthropic/gemini/local)
  • hotkey: Custom hotkey configuration
  • hasCompletedOnboarding: Onboarding completion flag
  • customDictionary: JSON array of words/phrases for improved transcription accuracy
  • keepLocalModelLoaded: Keeps the shared llama-server loaded instead of unloading it after 5 idle minutes (#1207; default off). Synced to main with sync-startup-preferences, outside its policy gate; LlamaServerManager.setKeepResident acts only on a change, since every window resyncs on load
  • toneStyleEnabled, toneStyles: whether Tone & Style is on (the setup dialog's Finish, or Settings' Tones & voices switch) and the tone picked per kind of app (JSON, normalized by normalizeToneStyles)
  • emailDraftTarget: Where email drafts open (auto, gmail, gmailSend, outlookWork, outlookPersonal, mailto); gmailSend sends through the Gmail connector's approval card ("Send from chat (Gmail)"); an unknown value reads as auto

Secret env vars (12 total: 7 BYOK API keys + 5 enterprise cloud creds — see SECRET_KEYS in environment.js) are encrypted at rest via Electron safeStorage and stored as per-key files under userData/secure-keys/. They are loaded into process.env at startup by EnvironmentManager.init(). Renderer reads them via IPC (get-*-key) and writes via debounced IPC (save-*-key). On Linux without a keyring, secrets fall back to plaintext.

Non-secret env vars persisted to .env (via saveAllKeysToEnvFile()):

  • LOCAL_TRANSCRIPTION_PROVIDER: Transcription engine (nvidia for Parakeet)
  • PARAKEET_MODEL: Selected Parakeet model name (e.g., parakeet-tdt-0.6b-v3)
  • OPENWHISPR_LOG_LEVEL: Log level written by the Settings debug toggle (set-debug-logging): debug when on, info when off

6. Language Support

60 languages supported, defined in src/config/languageRegistry.json and read through src/utils/languageSupport.ts:

  • Each entry has a two-letter code, label, flag, and per-engine support flags (whisper, parakeet, assemblyai) — 59 Whisper, 26 Parakeet, 7 AssemblyAI
  • "auto" is a 61st entry for automatic detection
  • Passed to whisper.cpp via -l parameter

7. Agent Naming System

  • User names their agent during onboarding (step 6/8)
  • Name stored in localStorage and database
  • ReasoningService detects "Hey [AgentName]" patterns
  • Standalone wake-word commands stream into the assistant panel (the address is stripped first, stripAgentAddress); a highlighted selection is edited in place by the dictation agent
  • Supports multiple AI providers (all models defined in src/models/modelRegistryData.json):
    • OpenAI (Responses API):
      • GPT-6 Astra (gpt-6-astra) - Most capable OpenAI model for coding, computer use, and research, 1M context
      • GPT-5.6 Series (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna) - Flagship, balanced, and fastest tiers
      • GPT-5.5 (gpt-5.5) - Frontier model for complex reasoning, 1M context
      • GPT-5.2 (gpt-5.2) - Strong reasoning model
      • GPT-5 Mini (gpt-5-mini) - Fast and cost-efficient
      • GPT-5 Nano (gpt-5-nano) - Ultra-fast, low latency
      • GPT-4.1 Series (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano) - Strong baseline with 1M context
      • GPT-5 and newer reject temperature, so their registry entries carry supportsTemperature: false
    • Anthropic (Via IPC bridge to avoid CORS):
      • Claude Fable 5.1 (claude-fable-5-1) - Most capable Claude model, Mythos-class, 1M context
      • Claude Fable 5 (claude-fable-5) - Previous Mythos-class flagship, 1M context
      • Claude Opus 5.5 (claude-opus-5-5) - Latest Opus model for complex reasoning, 1M context
      • Claude Sonnet 5.5 (claude-sonnet-5-5) - Latest Sonnet: fast and capable for everyday work, 1M context
      • Claude Haiku 5.5 (claude-haiku-5-5) - Fastest, lowest-cost Claude model, 1M context
      • Claude Sonnet 5 (claude-sonnet-5) - Fast, capable agentic model at lower cost
      • Claude Sonnet 4.6 (claude-sonnet-4-6) - Balanced performance
      • Claude Haiku 4.5 (claude-haiku-4-5) - Fast with near-frontier intelligence
      • Claude Opus 5 (claude-opus-5) - Previous Opus flagship, 1M context
      • Claude Opus 4.8 (claude-opus-4-8) - Powerful Opus model tuned for honesty and reliability, 1M context
      • Claude Opus 4.7 (claude-opus-4-7) - Powerful Opus model, 1M context
      • Claude Opus 4.6 (claude-opus-4-6) - Previous Opus generation, 1M context
      • Claude Sonnet 4.5 (claude-sonnet-4-5) - Previous Sonnet generation
      • Claude Opus 4.5 (claude-opus-4-5) - Earlier Opus model
      • Models from Opus 4.7 onward (including the 5.5 generation) reject temperature (supportsTemperature: false); the Anthropic IPC bridge sends no thinking parameters, so no Anthropic entry carries supportsThinking
    • Google Gemini (Direct API integration):
      • Gemini 3.5 Flash (gemini-3.5-flash) - Latest fast, high-capability Gemini model
      • Gemini 3.5 Flash Lite (gemini-3.5-flash-lite) - Fastest, most cost-effective 3.5 model
      • Gemini 3.1 Pro (gemini-3.1-pro-preview) - Most capable Gemini model
      • Gemini 3.1 Flash Lite (gemini-3.1-flash-lite) - Frontier-class performance at low cost. It thinks, and requests that aren't streamed pin it to minimal unless a reasoning effort is picked; it has no supportsThinking flag, so Settings shows the effort picker rather than the thinking toggle
      • Gemini 3 Flash (gemini-3-flash-preview) - Ultra-fast, high-capability next-gen model
      • Gemini 2.5 Flash Lite (gemini-2.5-flash-lite) - Lowest latency and cost (retired for new API keys; kept for existing ones)
      • Gemma 4 (gemma-4-31b-it, gemma-4-26b-a4b-it) - Google's open Gemma 4 models served through the Gemini API
    • Local: GGUF models via llama.cpp (Qwen, Llama, Mistral, GPT-OSS)

8. Model Registry Architecture

All desktop AI model definitions are centralized in src/models/modelRegistryData.json as the single source of truth. The mobile app keeps its own hand-updated copies (see below):

{
  "cloudProviders": [...],   // OpenAI, Anthropic, Gemini API models
  "localProviders": [...]    // GGUF models with download URLs
}

Key files:

  • src/models/modelRegistryData.json - Single source of truth for all desktop models
  • openwhispr-mobile/src/config/providerCatalog.json - Mobile's trimmed copy of the OpenAI and Groq entries. It, mobile's endpoint rules and its cleanup prompt (openwhispr-mobile/src/config/prompts/defaultPrompts.json) are copies, not shared code, so update them by hand when desktop's change (the catalog is covered in openwhispr-mobile/CONTRIBUTING.md)
  • src/models/ModelRegistry.ts - TypeScript wrapper with helper methods; also derives REASONING_PROVIDERS (buildReasoningProviders()), consumed by the model pickers
  • src/models/providerDefaultModel.ts - pickProviderDefaultModel(); with no defaultModel on the provider, the first model in the array is the default
  • src/config/retiredCloudModels.ts - Remaps selections pinned to a retired model
  • src/helpers/modelManagerBridge.js - Handles local model downloads

Local model features:

  • Each model has hfRepo for direct HuggingFace download URLs
  • Chat formatting comes from the GGUF's embedded template (llama-server runs with --jinja); the registry carries no prompt templates
  • Download URLs constructed as: {baseUrl}/{hfRepo}/resolve/main/{fileName}

9. API Integrations and Updates

OpenAI Responses API (September 2025):

  • Migrated from Chat Completions to new Responses API
  • Endpoint: https://api.openai.com/v1/responses
  • Simplified request format with input array instead of messages
  • New response format with output array containing typed items
  • Automatic handling of GPT-5 and o-series model requirements
  • No temperature parameter for newer models (GPT-5, o-series)

Anthropic Integration:

  • Routes through IPC handler to avoid CORS issues in renderer process
  • Uses main process for API calls with proper error handling
  • Model IDs use alias format (e.g., claude-sonnet-4-6 not date-suffixed versions)

Gemini Integration:

  • Direct API calls from renderer process
  • Increased token limits for Gemini 3.1 Pro (2000 minimum)
  • Proper handling of thinking process in responses
  • Error handling for MAX_TOKENS finish reason

API Key Persistence:

  • All API keys now properly persist to .env file
  • Keys stored in environment variables and reloaded on app start
  • Centralized saveAllKeysToEnvFile() method ensures consistency

10. System Settings Integration

The app can open OS-level settings for microphone permissions, sound input selection, and accessibility:

IPC Handlers (in ipcHandlers.js):

  • open-microphone-settings: Opens microphone privacy settings
  • open-sound-input-settings: Opens sound/audio input device settings
  • open-accessibility-settings: Opens accessibility privacy settings (macOS only)
  • open-login-items-settings: Opens the login/startup items pane (macOS Login Items, Windows Startup Apps)

Platform-specific URLs:

Platform Microphone Privacy Sound Input Accessibility Login Items
macOS x-apple.systempreferences:com.apple.preference.security?Privacy_Microphone x-apple.systempreferences:com.apple.preference.sound?input x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility x-apple.systempreferences:com.apple.LoginItems-Settings.extension
Windows ms-settings:privacy-microphone ms-settings:sound N/A ms-settings:startupapps
Linux Manual (no URL scheme) Manual (e.g., pavucontrol) N/A N/A (XDG autostart entry, see linuxAutostart.js)

UI Component (MicPermissionWarning.tsx):

  • Shows platform-appropriate buttons and messages
  • Linux only shows "Open Sound Settings" (no separate privacy settings)
  • macOS/Windows show both sound and privacy buttons

11. Debug Mode

Enable with --log-level=debug or OPENWHISPR_LOG_LEVEL=debug (can be set in .env):

  • Logs saved to platform-specific app data directory
  • Comprehensive logging of audio pipeline
  • FFmpeg path resolution details
  • Audio level analysis
  • Complete reasoning pipeline debugging with stage-by-stage logging

Packaged Windows builds keep logger output off stdout/stderr by default. Launch with --console-logs to opt into terminal output independently of the configured log level. Default INFO entries are not persisted in this mode; enabling debug logging retains them in the existing log file without enabling terminal output.

12. Windows Push-to-Talk

Native Windows support for true push-to-talk functionality using low-level keyboard hooks:

Architecture:

  • resources/windows-key-listener.c: Native C program using Windows SetWindowsHookEx for keyboard hooks
  • src/helpers/windowsKeyManager.js: Node.js wrapper that spawns and manages the native binary
  • Binary outputs KEY_DOWN and KEY_UP to stdout when target key is pressed/released

Compound Hotkey Support:

  • Parses hotkey strings like CommandOrControl+Shift+F11
  • Maps modifiers: CommandOrControl/Ctrl → VK_CONTROL, Alt/Option → VK_MENU, Shift → VK_SHIFT
  • Verifies all required modifiers are held before emitting key events

Binary Distribution:

  • Compiled from source by scripts/build-windows-key-listener.js, part of compile:native, so every prebuild*, predev:main and prestart chain on a Windows host builds it; neither script runs on other hosts
  • Fallback: prebuilt binary from GitHub releases (windows-key-listener-v* tags) via scripts/download-windows-key-listener.js
  • Release CI (release.yml, build-and-notarize.yml) compiles it with MSVC during build:win
  • CI workflow: .github/workflows/build-windows-key-listener.yml rebuilds the prebuilt binary when resources/windows-key-listener.c changes on main
  • afterPack.js fails a Windows package without the binary. A dev run without it starts with Hold unavailable (a saved Hold is reset to Tap at startup), modifier-only hotkeys refused, and F8 as the default hotkey (hotkeyManager.js)

IPC Events:

  • windows-key-listener:key-down: Fired when hotkey pressed (start recording)
  • windows-key-listener:key-up: Fired when hotkey released (stop recording)

13. Custom Dictionary

Improve transcription accuracy for specific words, names, or technical terms:

How it works:

  • User adds words/phrases through Settings → Custom Dictionary
  • Words stored as JSON array in localStorage (customDictionary key)
  • On transcription, words are joined and passed as prompt parameter to Whisper
  • Works with both local whisper.cpp and cloud OpenAI Whisper API

Implementation:

  • src/hooks/useSettings.ts: Manages customDictionary state
  • src/components/SettingsPage.tsx: UI for adding/removing dictionary words
  • src/helpers/audioManager.js: Reads dictionary and adds to transcription options
  • src/helpers/whisperServer.js: Includes dictionary as prompt in API request

Whisper Prompt Parameter:

  • Whisper uses the prompt as context/hints for transcription
  • Words in the prompt are more likely to be recognized correctly
  • Useful for: uncommon names, technical jargon, brand names, domain-specific terms

14. GNOME Wayland Global Hotkeys

On GNOME Wayland, Electron's globalShortcut API doesn't work due to Wayland's security model. OpenWhispr uses native GNOME shortcuts:

Architecture:

  1. main.js enables GlobalShortcutsPortal feature flag for Wayland
  2. hotkeyManager.js detects GNOME + Wayland and initializes GnomeShortcutManager
  3. gnomeShortcut.js creates D-Bus service at com.openwhispr.App
  4. Shortcuts registered via gsettings as custom GNOME keybindings
  5. GNOME triggers dbus-send command which calls the D-Bus Toggle() method

Key Constants:

  • D-Bus service: com.openwhispr.App
  • D-Bus path: /com/openwhispr/App
  • gsettings path: /org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/openwhispr/

IPC Integration:

  • get-hotkey-mode-info: Returns { isUsingGnome, isUsingHyprland, isUsingNativeShortcut } to renderer
  • UI hides activation mode selector when isUsingNativeShortcut is true
  • Forces tap-to-talk mode (push-to-talk not supported)

Hotkey Format Conversion:

  • Electron format: F8, CommandOrControl+Shift+Space
  • GNOME format: F8, <Control><Shift>space
  • Backtick () → grave` in GNOME keysym format
  • F13-F24 bind the keysym the active keymap gives that key (xkbcomp -xkb $DISPLAY -, xkb keys <FK13>-<FK24>), for gsettings and the Hold portal alike: stock xkeyboard-config sends XF86Launch7 for F16 and has no F16 keysym, so GNOME can't grab F16 (#1716). A key with no keysym (keysym_missing, e.g. stock F19/F24) or one gsd-media-keys already binds (keysym_reserved, e.g. F13 → XF86Tools opens Settings) is refused with a message. An unreadable keymap keeps the F-key name

15. Hyprland Wayland Global Hotkeys

On Hyprland (wlroots Wayland compositor), Electron's globalShortcut API and the GlobalShortcutsPortal feature don't work reliably. OpenWhispr uses native Hyprland keybindings:

Architecture:

  1. main.js enables GlobalShortcutsPortal feature flag for Wayland (fallback)
  2. hotkeyManager.js detects Hyprland + Wayland and initializes HyprlandShortcutManager
  3. hyprlandShortcut.js creates D-Bus service at com.openwhispr.App (same as GNOME)
  4. Shortcuts registered via hyprctl keyword bind (runtime keybinding)
  5. Hyprland triggers dbus-send command which calls the D-Bus Toggle() method

Detection:

  • Primary: HYPRLAND_INSTANCE_SIGNATURE environment variable (set by Hyprland)
  • Fallback: XDG_CURRENT_DESKTOP contains "hyprland"

Hotkey Format Conversion:

  • Electron format: Control+Super, CommandOrControl+Shift+Space
  • Hyprland format: CTRL, Super_L, CTRL SHIFT, space
  • Modifier-only combos (e.g., Control+Super) → CTRL, Super_L

Bind/Unbind Commands:

  • Register: hyprctl keyword bind "ALT, R, exec, dbus-send --session ..."
  • Unregister: hyprctl keyword unbind "ALT, R"
  • Bindings are ephemeral (don't survive Hyprland restart) but re-registered on app startup

Limitations:

  • Push-to-talk not supported (Hyprland bind fires a single exec, not key-down/key-up)
  • Requires hyprctl on PATH (ships with Hyprland)

16. Meeting Detection (Event-Driven)

Detects meetings via three independent sources, orchestrated by MeetingDetectionEngine:

Architecture:

  • MeetingDetectionEngine listens to events from MeetingProcessDetector and AudioActivityDetector
  • CalendarReminderScheduler provides calendar context (imminent events, active meetings) from the shared calendar_events table, fed by the Google/Microsoft/Apple calendar managers
  • All three sources feed into a unified notification pipeline

Process Detection (known meeting apps — Zoom, Teams, Webex, FaceTime):

  • macOS: systemPreferences.subscribeWorkspaceNotification — zero CPU, instant detection
  • Windows/Linux: processListCache shared polling (30s interval, ps-list npm)
  • Context-only: a running meeting app never prompts by itself (FaceTime idles in the background). It corroborates unattributed macOS device activity (below) and drives the auto-end process-exit fast path

Microphone Detection (unscheduled/browser meetings like Google Meet):

  • macOS: macos-mic-listener binary — CoreAudio process-object input monitoring, excluding the background com.apple.CoreSpeech service (it runs input during ordinary playback). CAPABILITY PID transitions prompt on their own. CAPABILITY AGGREGATE device activity (no process objects — roughly macOS < 14.2 — or a transient snapshot failure) includes playback on combined input/output devices, so it prompts only while a known meeting app is running; the helper retries PID monitoring every 30s from its heartbeat and the detector accepts the later CAPABILITY PID. Capability is logged at info level; sustained-audio-detected carries attributed
  • Windows: windows-mic-listener.exe — WASAPI IAudioSessionManager2 session monitoring; self-mic exclusion happens in JavaScript using current OpenWhispr and capture-helper PIDs
  • Linux: pactl subscribe — PulseAudio source-output events. Production captures (media.role=production, e.g. OBS) and sink-monitor captures (stream.capture.sink=true) still hold the mic for auto-end but don't prompt; the unfiltered fallbacks (a failed or JSON-less pactl listing, a failed excluded-pid lookup while polling, pw-cli) can't tell them apart
  • Windows/Linux: Fall back to polling if the native binary/command is unavailable. macOS has no safe polling signal: a missing binary pauses audio prompts, a crashed listener is respawned with exponential backoff (5s → 60s), and meanwhile only calendar reminders prompt while auto-end uses its existing silence fallback

Calendar Reminders (scheduled meetings):

  • CalendarReminderScheduler fires meetingDetectionEngine.handleCalendarReminder(event) 1 minute before the scheduled start (MEETING_REMINDER_LEAD_MS) — no native OS notifications; all meeting prompts use the in-app overlay so they survive Focus/DND and screen-share notification muting
  • Calendar-sourced prompts show a Join primary action when the event has a meeting link (getMeetingJoinUrl in src/helpers/meetingJoinUrl.js, shared with the renderer's Upcoming Meetings join button) — Join opens the link and starts the note

UX Rules:

  • All prompts render in one always-on-top overlay window (MeetingNotificationCard), content-protected so it never appears in screen shares
  • Prompt copy is derived in the renderer from { variant, event, joinUrl } (meetingNotification.* i18n keys); variants: detected (mic evidence), starting (calendar event not yet started), underway (event in progress)
  • Per-source notification prefs: notifyCalendarReminders gates calendar prompts, notifyMeetingDetection gates mic/process prompts and, with notificationsEnabled, whether the mic and process detectors run at all (meetingProcessDetection additionally gates the process detector); nothing runs until the renderer has synced the saved snapshot
  • During recording (tap-to-talk or push-to-talk): ALL notifications suppressed
  • After recording: 2.5s cooldown before showing queued notifications
  • Multiple signals coalesced: one overlay at a time; a newer prompt replaces the current one
  • Calendar-aware: if an ongoing or imminent calendar event exists, the prompt shows the event name and links the note to the event
  • Active meeting recording (meeting mode): all detections suppressed

Binary Distribution:

  • macOS: Compiled from Swift source via scripts/build-macos-mic-listener.js during compile:native
  • Windows: Prebuilt binary downloaded via scripts/download-windows-mic-listener.js during prebuild:win
  • CI workflow: .github/workflows/build-windows-mic-listener.yml auto-builds on push to main

Calendar Sync Resilience:

  • 10s socket timeout on all Google Calendar API requests
  • Exponential backoff on consecutive failures: 2min → 4min → 8min → cap 30min
  • Reset to normal 2min interval on any successful sync

17. Voice Assistant Hotkey

A dedicated global hotkey that starts a dictation whose transcript is sent straight to the voice assistant as a command — no wake word ("Hey [AgentName]") needed — and that always bypasses the cleanup model. With auto-paste enabled, a caret verified as writable at capture time (an opaque caret session in selectionManager.js, revalidated before pasting) receives the completed answer and the pill returns to idle. A dormant macOS accessibility tree reports "unknown": synthetic copy can still recover selected text, but cannot establish a safe caret destination. Unknown targets therefore stay on the panel route, since they can also be integrated terminals, pages without inputs, or fields with selections hidden by clipboard capture. A verified caret whose accessibility becomes unknown before delivery falls back the same way. Terminal and secure-field checks still apply to verified targets. On Linux the caret is verified by linux-text-monitor --probe-editable <pid>, which answers only for the focused field in that process's active window; the pid comes from the target (its AT-SPI pid, xdotool/kdotool getwindowpid, or hyprctl activewindow), and a target whose pid can't be read stays on the panel route. Otherwise the answer streams into a floating assistant panel attached to the dictation pill (there is no separate assistant window) and, with auto-paste enabled, is also copied to the clipboard for manual paste (the Copy button confirms for six seconds). The pill window is content-protected while the panel is open (the panel never appears in screen shares).

Flow:

  1. Hotkey pressed → voiceAgent slot callback in main.js → windowManager.sendToggleVoiceAgent() → toggle-voice-agent IPC to the main window → recording capsule appears
  2. useAudioRecording.js starts a recording with audioManager.setVoiceAgentRequested(true) (any other start resets it to false)
  3. On transcription, resolveReasoningRoute consults resolveDictationRouteKind() (src/helpers/dictationRouting.js): a voice assistant recording always takes the agent route and never falls back to cleanup. The dictation agent's reachability only gates selection edits — a selection with the dictation agent unconfigured routes to the panel with the selected text quoted instead of editing in place
  4. Standalone commands (no text selected) run through the chat pipeline (src/components/dictation/AssistantPanel.tsx): chat tools (notes search/create/update, calendar, web search, clipboard, get_snippet — triggers listed in the tool description, body fetched on demand — and update_dictionary / update_snippets, which write through the settings store so the change syncs like a UI edit), RAG memory, and the custom dictionary plus snippet triggers injected into the system prompt. Conversations persist in the agent_conversations table and are browsable from the ControlPanel chat. The system prompt (getAgentSystemPrompt, appended after the user-editable base prompt, so it reaches custom prompts too) lists the offered tools grouped by capability, forbids claiming an ability a listed tool covers (web search is for public information the context doesn't already answer, never people the user knows), and names what is unavailable and how to enable it — for an unavailable connector the model still writes the email or message in its reply and ends with one line on how to have it sent (resolveUnavailableCapabilities in src/config/agentCapabilities.ts: signed out, org policy, paid plan, a Slack/Linear/GitHub connector not connected or needing a reconnect, no calendar, or a local model too small for tools; connectors are named only on surfaces that offer them, and the onboarding demo names nothing, nameUnavailableCapabilities: false). History (toHistoryMessages in src/components/chat/historyMessages.ts) prefixes earlier assistant turns with [Tools used: …], showing only allowlisted search queries, contact names and note titles — never drafted content or results. A connector action adds its outcome from the saved result (sent, draft opened, may have been sent, cancelled by the user, not sent, failed, unavailable, needed details), and a sent one its reference and an https link (github_create_issue (sent: o/r#12 https://…)), so a follow-up turn ("it's #engineering", "retry") never redoes what already went out; destinations, card edits and error text stay out. A call cut off before its result reads "(outcome not recorded)", since a send can still commit after Esc. A reply that opens with such a note itself is shown, saved and delivered without it (withoutEchoedToolTrace)
  5. Response delivery: a capture with status: "editable" (a focused writable non-terminal field with no selection) plus auto-paste banks a deliverySessionId; the completed answer is pasted via paste-at-captured-target, which revalidates the target and fails closed to the panel + clipboard on any change (assistantResponseDelivery.ts, pasteAtCapturedTarget in selectionManager.js). A follow-up spoken while the panel is already open stays panel-first. Cancelled or empty responses never paste and never touch the clipboard
  6. Selection edits: highlighted text goes through the dictationAgent scope and is safely replaced in place. When the editor judges the command a question about the selection ("explain this") rather than a change, it answers SELECTION_QUESTION_MARKER (selectionEditing.js, both the cloud and local prompts) and the command goes to the panel with the selection quoted, leaving the selection untouched
  7. Connector tools (email_draft, find_contact, slack_send_message, the Linear tools, the GitHub tools; paid, policy-gated) run through src/helpers/connectors. Typed chat (ChatView), this panel and every note's chat (useEmbeddedChat) opt in (allowConnectors); container chat and the onboarding demo don't get them. Note chat also gets a "Meeting attendees" block: the note chat sends its meeting as one request (NoteAttendeesRequest: note id, parsed participants, calendar event id, the user's OpenWhispr address), and main adds the speakers identified in the note (speaker profiles with an address) and the linked calendar event's organizer (Outlook and Apple leave them out of the attendees), keeps only valid addresses, strips control and invisible format characters, angle brackets and their look-alikes, and address-like words from names, and filters with find_contact's exclusions plus the user's OpenWhispr address and the Gmail login's address (connector-note-attendees, createNoteAttendeesLookup), organizer included. The block is fenced in <meeting_attendees> tags and marked as data, and is added only when connector tools are registered; the note's title and text and the search_notes and get_note results lose the tag name (withoutAttendeesFence), so an invite's title can't fake a second list. A participant flagged self also excludes that address when it comes back as an identified speaker or the organizer. A participant's self flag marks whoever recorded the meeting, so the renderer honours it only on the user's own note (ownsNote, attendeesForUser; unknown ownership counts as someone else's). The note chat passes its note as openNote, which getAgentSystemPrompt puts under its own rule: for questions about what was said, decided or written, answer from this note, and when it doesn't cover the question, say so and offer to search the user's other notes; search_notes runs only when the user asks to look beyond the note, and the answer names the note used. Unlike the other chats it runs no up-front library search (RAG): other notes reach it only through search_notes and get_note, so a model too small for tools answers from the open note alone. The open note carries its transcript whole, as Speaker: text lines labelled exactly as get_note labels them (noteChatTranscript: saved speaker names, Note taker for whoever recorded it, numbered in-person speakers; after a line saying whether the note taker is the user, named on their own note), unless the model has get_note and the transcript is longer than it takes (FULL_TRANSCRIPT_MAX_TOKENS in transcriptEvidence.ts, counted with estimateNoteTokens since CJK text is about a token per character: 64,000 hosted, 4,000 on a local or LAN model); then it carries the first 500 characters (transcript_preview) and a rule to read further with get_note before saying the note doesn't cover something. Whole-note actions (requestText) always get the whole transcript. get_note returns the start of the transcript beside the editable fields; transcript_query (literal and case-insensitive, speaker names included; the person taking the notes is Note taker) or a transcript_offset past 0 returns one 500-character passage with the times it spans, paging metadata (offset 0 is an ordinary read, since models fill in defaults) and a revision naming the text read so far, so a recording that keeps appending can still be paged. Both tool loops keep the two newest passages, and every passage of the latest step even when it read more in parallel (compactTranscriptHistory); a query that matched nothing is never compacted. Its OpenWhispr Cloud requests are ordinary /api/agent/stream requests: the API keeps no assistant memory and adds nothing to the prompt, so the client's own system prompt is what the model sees (the noteChat flag went once openwhispr-api#243 removed memory). Like ChatView, New chat and switching conversations cancel the running turn, so a pending card can't hold the send lock or save into another conversation. An approval card, any email_draft (its compose window takes focus), any find_contact (a lookup is almost always followed by a question for the user), and any slack_send_message, linear_search_issues, linear_create_issue, linear_comment or GitHub tool (holds on every outcome — a card, a question back, results or a receipt) hold the turn in the panel (onHoldDelivery) instead of pasting at the caret. A held answer is still auto-copied, except after an email_draft that reserved the clipboard for overflow text (onHoldDelivery({ preserveClipboard: true }), sticky for the turn) and on a caret turn whose delivery restores the clipboard (Keep Transcription in Clipboard off)

Storage & IPC:

  • Env var: VOICE_AGENT_KEY (persisted via environment.js), store key: voiceAgentKey (no default — user opt-in)
  • IPC handlers: update-voice-agent-hotkey, get-voice-agent-key
  • Hotkey slot: voiceAgent (tap-to-toggle; GNOME-native slot via ToggleVoiceAgent D-Bus method, KDE via KGlobalAccel, otherwise globalShortcut)

Panel:

  • Esc collapses the panel; a Copy button copies the answer; a follow-up input takes typed questions. Esc while the command is still thinking cancels it; a command whose stream ends without content settles as agentMode.chat.emptyResponse
  • Pressing the hotkey again while the panel is open records a follow-up into the same conversation

UI:

  • Settings → Hotkeys → "Voice Assistant Hotkey" (with cross-slot conflict validation)
  • Onboarding: optional pair of steps (hotkey, then demo) after the Notes step, so the demo can suggest meeting times from a calendar connected there. The demo card is a mail thread; the reply is answered headlessly in the dictation window by useOnboardingAssistantDemo (the panel's streaming pipeline, tools and screenshot included) and streamed back over onboarding-demo-event as processing (transcript) → replying (partial reply, tool while one runs) → success
  • Panel conversations and in-place selection edits both run on the Voice Assistant scope (Settings → AI Models → Voice Assistant; dictationAgent/dictationAgentVision), resolved by resolveChatStreamingInference; the panel falls back to the chatIntelligence scope while the Voice Assistant scope is unreachable. Typed chat (Control Panel, note and container chat) stays on chatIntelligence

Screen Context (opt-in):

When "Share screen context" is enabled (Settings → AI Models → Voice Assistant → Screen Context, store key voiceAgentScreenContext, default off), each voice-assistant recording start captures the display the cursor is on and attaches it to the request as a base64 JPEG (long edge ≤ 1568 px).

  • Capture: src/helpers/screenContextCapture.js (main process; screen.getCursorScreenPoint + desktopCapturer). macOS requires the Screen Recording TCC permission (useScreenRecordingPermission hook + PermissionCard in DictationAgentSettings); Windows/Linux X11 need no permission; Linux Wayland is unsupported (capture silently skipped)
  • Encoding: encodeWithinBudget() walks a JPEG quality ladder (82 → 70 → 55), then falls back to a 1024 px resize, keeping the payload under MAX_ENCODED_BYTES (1.5 MB ≈ 2.05M base64 chars, inside the API's 2.8M cap). Each rung re-encodes the original bitmap, so quality drops never compound. A screen that still won't fit is dropped
  • Trigger: useAudioRecording.js calls audioManager.beginScreenContextCapture() at recording start (fire-and-forget); consumeScreenContext() (3s guard) is awaited when the request is built. The dictation overlay gets setContentProtection while the setting is on so the pill never appears in captures
  • Routing: the press-time screenshot is re-decided against the model that will actually answer. On BYOK, it attaches only when the chat model's provider client is image-wired (supportsImages on the InferenceProvider) and the model has supportsVision in modelRegistryData.json; on OpenWhispr Cloud, attachment is pending API vision routing for the chat path. Dropping the image never fails the command
  • Privacy: screenshots live only in renderer memory for one request — never written to disk, stored in history, or logged (loggers emit hasScreenContext booleans only)
  • IPC: capture-screen-context, check-screen-recording-access, request-screen-recording-access, open-screen-recording-settings, screen-context-set-enabled

Tests: test/helpers/dictationRouting.test.js, test/helpers/screenContextCapture.test.js (run with node --test)

18. Meeting Transcription: Echo, Duplicate, and Segment Pipeline

Live meeting transcription runs two streams (mic + system-audio tap) and must keep the remote party's voice, as heard through the local speakers, out of the mic transcript. The policy lives in pure, unit-tested seams so it can be tuned without touching the IPC plumbing:

  • Audio layer: meetingEchoLeakDetector.js correlates mic chunks against recent system audio; meetingMicGate.js turns chunk RMS/peak (+ a "system speaking" lookback) into a send / zero / skip verdict that ipcHandlers.js applies in dispatchMeetingAudioBuffer (streaming) and transcribeLocalMeetingChunk (local)
  • Echo cancellation: the WebRTC AEC3 helper (meetingAecManager.js, native/meeting-aec-helper/) replaces the mic channel only when Settings → Speech-to-Text → Note Recording → Echo cancellation for meetings is on (meetingAecEnabled, default off, sent as the aecEnabled start option). With headphones there is no echo, and AEC3 still cuts the user's own speech, worst in the first seconds (#1025)
  • Text layer: meetingMicHoldback.js decides whether a mic final is risky (held back), a duplicate of recent system text (dropped), or racing an arriving system final (retracted); the ipcHandlers.js adapters (shouldSkipDuplicateMicSegment, hasRiskyMicDuplicateProfile, removeRacingMicEntriesFor, removePendingMicFinalsFor) apply the results to closure state and emit meeting-transcription-segment events (partial / final / retract)
  • Renderer: src/stores/meetingSegmentReducer.ts is the pure (state, event, deps) → reduction transition for those events (timestamp-sorted insert, retract by exact match, per-source partial slots); meetingRecordingStore.ts supplies mintSegmentId / decorateFinal (speaker identifications, provisional speaker, locks — must not write to the store) and applies the reduction. Tests: test/stores/meetingSegmentReducer.test.js, test/stores/meetingRecordingStoreImports.test.js
  • Regression pins: BYOK session.update payload and the disconnect-commit callback (test/helpers/openaiRealtimeStreaming.test.js, tinfoilRealtimeStreaming.test.js), token-endpoint wire bodies (test/helpers/realtimeTokenProviders.test.js). Tests named characterization: … pin known oddities on purpose — flip them deliberately when changing policy
  • Fixtures: test/helpers/harness/pcmFixtures.js — deterministic 24 kHz PCM generators (makeSine, makeSeededNoise, mix, delayBy, toInt16Buffer, chunkBuffer, …) used by the gate and echo-detector tests

19. Note Templates and Actions

Rows in the actions table (built-ins seeded from src/helpers/builtinActions.js) are one of two kinds:

  • Templates (kind: 'template') write a note's AI summary (enhanced_content) from its notes, meeting context, and transcript, never from the summary they replace. A template is a context (prompt) plus ordered sections ({heading, instruction}); compileTemplatePrompt wraps sections in the shared notes rules. A template without sections keeps the request it always had after the SOURCE_FIDELITY_RULE prefix (the generic system prompt, or the material preamble for a standalone key). The note records the template in enhancement_template_id (its client_id), which syncs with the note, and enhancement_prompt the compiled system prompt
  • Actions (kind: 'action') work from the AI summary when the note has one, and from its notes and transcript otherwise (buildNoteRunInput). Only the built-in Make notes shorter (shorten) and Make notes longer (lengthen) actions edit that summary, selected by fixed built-in identity (toActionItem), never an editable name or stored output. A summary edit requires complete output, never splits into parts, and writes only enhanced_content; with no summary yet it writes one from the transcript and applies the action to it, run like a template (split into parts on a local model, the material hash recorded); every other built-in and custom action, including Write TL;DR (addTldr), runs in the note chat via SendToAIOptions.requestText (the chat shows a built-in's call to action where it has one, otherwise the action's name, and the preamble points the model at the summary or the transcript and has it answer in the chat as text in the note's language, using a tool to draft, send or post only when the action's instructions explicitly ask to send, post or file something, and leaving the note unchanged, with an asked-for edit written in the chat instead; its turn is never offered update_note (keepNotesUnchanged), so it can't change a note whatever the model decides; the note chat's other tools and attendee block otherwise apply as usual, and its transcript context names speakers only on notes known to be the user's own (ownsNote), never listing the unfiltered invitees)
  • The material picks the wrapper (runNoteAction): speaker segments make a meeting (isMeetingNote: MEETING_SYSTEM_PROMPT or MEETING_INPUT_PREAMBLE, the ## Meeting Context block and Speaker: text lines); a transcript in plain text (a recording that captured no segments, one made before 1.6.6, or an upload whose provider named no speakers: its text is stored as segments with no speaker or source (speakerlessTranscriptText; one untimed segment when it returned no segments), or as the note's content on uploads saved before the transcript held it) is its own case (hasPlainTranscript: PLAIN_TRANSCRIPT_SYSTEM_PROMPT or PLAIN_TRANSCRIPT_INPUT_PREAMBLE, #2666), with no Meeting Context since nothing shows who is speaking, a part prompt without speaker labels when a local model splits it, and action items as - [ ] Action — Owner lines with an owner or deadline only when the transcript or the user's notes state it, never a placeholder, and never a table unless the instructions ask for one; no transcript is a personal note (BASE_SYSTEM_PROMPT or NOTE_INPUT_PREAMBLE). Both transcript cases get the custom dictionary, and their known people (the user, the invitees, and a meeting's named speakers) turn matching owners into mentions (tagActionItemOwners)
  • The default template is the built-in AI Summary (the Detailed Notes sections, stored under the old Generate Notes key generateNotes), listed first (launch re-applies the built-in order). Launch retires a stock Detailed Notes row (LEGACY_DETAILED_NOTES) unless AI Summary was customized; an edited one is kept and listed second, and saved template ids are never rewritten. Invariant: AI Summary, compiled from its sections, equals the flat 1.10.2 Detailed Notes prompt it replaced after the SOURCE_FIDELITY_RULE prefix used by templates and summary actions (sourceFidelity.js), but for the closing rule on material with nothing to summarize (test/helpers/templatePrompts.test.js); a built-in's previousPrompts only upgrades rows that are still flat. Chat actions get the rule once, from the chat system prompt (getAgentSystemPrompt), and the Cloud request leaves out the leading system message that repeats systemPrompt, which the API prepends itself
  • For material with nothing substantive (only greetings, filler or recording checks), sectioned templates, AI Summary and a summary action writing a first summary ask the model to answer NOTHING_TO_SUMMARIZE; custom context-only templates and flat built-ins edited before this rule keep their own rules. A reply that is the marker (bare, fenced, emphasised or after a short sentence) outside a thinking block writes nothing; a summary that merely contains it is saved, and the runner shows notes.actions.errors.nothingToSummarize as a neutral notice (notice on the error event) instead of saving a placeholder summary
  • UI: TemplatePicker opens from the chevron in the AI Summary tab, which names the note's template; the "Generate AI Summary" callout (runs the note's template, else the default), ActionPicker in the collapsed ask bar (runs the last action picked from it, in place of the send button), ActionChips once the in-view chat opens and in the docked chat (the first five actions in docked chat; the centered row omits Make notes longer without backfilling another chip; each has its action's icon, and All actions keeps every action; hovering a chip shows its icon, title and description without a destination badge; the in-view chat pins All actions at the end, and the docked chat shows them as filled pills in one tray with its composer: All actions, then the actions, scrolling together), a / menu in the note composers (slashCommands on ChatInput, each row with its action's icon, or a "/" for an icon this build doesn't know; it fills the chat above the composer until a command runs, the "/" is deleted, nothing matches or the composer loses focus), and ActionManagerDialog (Templates | Actions). The notes list and the docked chat widen or narrow from their edge facing the note (useResizableWidth, PanelResizeHandle), and keep their width across launches (localStorage notesSidebarWidth, noteChatWidth). Anything the ask bar sends (a message, or a chat action from the picker, a chip or the / menu) starts a new conversation in the docked chat, which holds the conversation and its history; the in-view chat is only the chips and the composer. Destinations are fixed and described in the editor; saving never asks a model to choose one. Legacy custom summary destinations read as chat (toActionItem), keeping the existing IPC fields, IDs and prompts. Summary generation stays in the AI Summary tab and callout. A run that writes the summary is undoable for 6 s instead of asking first: the runner queues the fields it overwrote (appliedEvents) and BackgroundActionToastListener shows the Undo toast

20. Tone & Style

Dictation cleanup writes in the tone the user picked for the kind of app they are dictating into: personal messages (formal, casual, very casual), work messages, email and everything else (formal, casual, excited). Off by default (cleanup keeps the speaker's own tone, the standard tone); turned on by finishing the setup dialog or the Tones & voices switch in Settings > Tone & Style. The assistant and chats also write drafts (messages, emails, replies, posts) in the tone of the kind of text the request is for. Translation keeps its source's tone.

  • Detection (main, at dictation start): capture-dictation-app-category reads the captured target through selectionManager.getTargetAppContext() and classifies it with toneAppCategories.js. macOS: macos-text-monitor --app-context <pid> prints the app name, bundle id, focused window title (accessibility, no Screen Recording needed) and, for a browser page, the URL host. Windows: windows-fast-paste --detect-only also prints WINDOW_TITLE (an older helper without it classifies by exe only). Linux: the window title is read on demand (xdotool/kdotool getwindowname) or taken from Hyprland's active window; an AT-SPI target's comes from linux-fast-paste --atspi-target (WINDOW_TITLE_B64, the active window's accessible name; an older helper prints none). Titles are kept beside the captured target (_targetTitles), never on it, since capture results reach the renderer. A browser (and a Windows UWP frame) is judged by its page or app title (its host, else the site name the title ends with; on a host not listed, only a site name the host carries, so a Notion page titled "Hey" stays other); native apps by whole-token app identity. Titles and hosts never leave the main process or reach the logs; only the category does
  • Cleanup: useAudioRecording starts audioManager.beginToneStyleCapture() for every non-translation recording while Tone & Style is on; the cleanup route's consumeToneStyle() (1.5 s guard, unknown app reads as other) puts toneStyle on the reasoning config. Cloud cleanup sends tone + appCategory on /api/reason, and in the combined Orukeet cleanup's pipelineOptions: its prompt is fixed when the session is minted, so startStreamingRecording waits for the classification until 250 ms after the capture began (overlapping the mic opening, so the start cue isn't held back after it) before minting, returns without minting if the start was cancelled meanwhile, and holds the audio captured meanwhile until the session has started. An app still unknown then (a Wayland AT-SPI target takes about a second) gets no combined cleanup, and /api/reason cleans up in its tone at stop. Warmup mints for the app the last start learned in time (lastToneAppCategory), and without combined cleanup while none is known; BYOK and local append appendToneStyleSuffix (src/config/prompts/toneStyle.ts) after the dictionary suffix. Its sentences are verbatim copies of openwhispr-api's lib/prompts.ts (English in every locale, as on the server); the mobile app's composerPrompt.ts copies the tone and priority sentences too, so change all three together
  • Assistant and chat: getAgentSystemPrompt's writingStyle option adds a WRITING STYLE block (buildWritingStyleBlock) to every chat surface's prompt while Tone & Style is on: the model works out from the request, the recipient and the tool (an email_draft body is an email) which kind of text it writes, and uses the user's tone for it; its own answers keep its voice, a tone the user asks for wins, and code is never restyled. A voice-assistant answer pasted at a verified caret writes its reply for that app (text written through a tool, such as an email_draft body, still takes its own kind's style): processAgentCommand consumes the capture and banks deliveryAppCategory, which reaches the send as destinationAppCategory through the paste delivery. Before pasting, deliverAssistantResponse replaces the answer's em dashes, except in ranges (which take an en dash), code, links and non-Latin scripts. Selection edits get one sentence (selectionEditToneSentence) that applies the selection's app style only to a prose rewrite asked for without a tone. The desktop builds every agent prompt itself, so the API needs nothing here
  • UI: Home banner (ToneStyleBanner, cleanup-off copy turns cleanup on) and the setup dialog (ToneStyleSetupDialog: intro plus one step per category), offered once the user has dictated TONE_STYLE_PROMPT_MIN_WORDS (70, src/config/toneStyle.ts) words of local history; the dialog opens by itself once, the banner stays until set up or "Not now" (useToneStylePrompt). Settings > Tone & Style (settings/ToneStyleSection.tsx) has a Tones & voices switch (toneStyleEnabled); while it's off, the picks and Save Changes are grayed out. App badges live in src/assets/icons/apps/

Development Guidelines

Internationalization (i18n) — REQUIRED

All user-facing strings must use the i18n system. Never hardcode UI text in components.

Setup: react-i18next (v15) with i18next (v25). Translation files in src/locales/{lang}/translation.json.

Supported languages: en, es, fr, de, pt, it, ru, zh-CN, zh-TW

How to use:

import { useTranslation } from "react-i18next";

const { t } = useTranslation();
// Simple: t("notes.list.title")
// With interpolation: t("notes.upload.using", { model: "Whisper" })

Rules:

  1. Every new UI string must have a translation key in en/translation.json and all other language files
  2. Use useTranslation() hook in components and hooks
  3. Keep {{variable}} interpolation syntax for dynamic values
  4. Do NOT translate: brand names (OpenWhispr, Pro), technical terms (Markdown, Signal ID), format names (MP3, WAV), or the shared assistant fullPrompt in prompts.json. The other prompt keys (cleanupPrompt, translatePrompt, dictionarySuffix, screenContextSuffix) are translated per locale, and any change to a default prompt must update CURRENT_DEFAULT_PROMPT_HASHES in src/config/retiredPrompts.js
  5. Group keys by feature area (e.g., notes.editor.*, referral.toasts.*)

Image and Icon Assets — REQUIRED

Raster UI assets live in src/assets/ (onboarding ones are named onboarding-*). Vector provider/brand marks live in src/assets/icons/.

UI icons come from src/components/icons/ (vendored Nucleo core outline components behind lucide-style names, e.g. import { Check, Loader2 } from "../icons"). To add one, map a name to a Nucleo label in src/components/icons/nucleo-map.json and run node scripts/sync-nucleo-icons.js; never import from lucide-react or a machine-local Nucleo path. Icons outside that set — the status marks, the note toolbar's formatting glyphs, the note chat's rounded copy and "/" glyphs, and the design's action icons (ListEnd, FolderRounded) — are drawn by hand in src/components/icons/primitives.tsx, which the generated index.ts re-exports.

Typography: --font-family-sans is Yowza (brand, Latin only) falling back to the bundled Noto Sans; --font-family-display is Yowza Soft for headings. The font files are licensed and never committed — src/brandFonts.ts registers whatever scripts/download-brand-fonts.js fetched at build time, and a build without them silently uses Noto Sans.

Rules:

  1. Always import the asset; never write a literal path like /assets/foo.webp. Packaged builds load the renderer from a file:// origin, so root-relative paths resolve to nothing. Importing lets Vite fingerprint the file and rewrite the URL:

    import microphoneIcon from "@/assets/onboarding-permission-microphone.webp";
    <img src={microphoneIcon} ... />;
    
  2. WebP, lossless for UI art with hard edges or alpha — cwebp -lossless -z 9 -alpha_q 100 in.png -o out.webp (roughly 40–55% smaller than PNG). Reserve lossy for photographic art.

  3. Author at 2x the CSS slot and no larger (a 44px slot gets an 88px asset). Retina-sharp without paying for pixels that get downscaled away.

  4. Bake rounded corners into the artwork as transparency rather than adding a CSS rounded-* on the <img> — CSS rounding on top of already-rounded art clips the corners twice.

  5. Decorative images take alt="" plus aria-hidden="true". Most icons sit beside a visible label that already names them (e.g. the permission rows), so a descriptive alt makes screen readers announce the same thing twice. Only write real alt text when the image is the only source of that information.

  6. Always set explicit width/height matching the CSS size, to reserve layout space before decode. Add decoding="async" and draggable={false} (in Electron a draggable image can be dragged out of the window).

  7. Assets under 4 KB are inlined by Vite as base64 data URIs, so they will not appear in dist/assets/. Grep the JS chunks for data:image/webp;base64, before concluding an asset went missing.

Adding New Features

  1. New IPC Channel: Add to both ipcHandlers.js and preload.js
  2. New Setting: Update useSettings.ts and SettingsPage.tsx
  3. New UI Component: Follow shadcn/ui patterns in src/components/ui
  4. New Manager: Create in src/helpers/, initialize in main.js
  5. New UI Strings: Add translation keys to all 10 language files (see i18n section above)
  6. New Sidecar Binary: Add download script in scripts/, add to prebuild* scripts in package.json, add manager in src/helpers/, initialize in main.js. Spawn the child with detached: process.platform !== "win32" so it has its own process group on Unix. Right after spawn call sidecarPidFile.write(name, child.pid) and on close call sidecarPidFile.clear(name). Add the binary fragment to EXPECTED_BINARY_FRAGMENTS in sidecarReaper.js. Register a stop function via sidecarRegistry.register(name, () => manager.stop()) in registerSidecars() — that single registration replaces the old will-quit line.

Testing Checklist

  • Test both local and cloud processing modes
  • Verify hotkey works globally
  • Check clipboard pasting on all platforms
  • Test with different audio input devices
  • Verify whisper.cpp binary detection
  • Test all Whisper models
  • Check agent naming functionality
  • Test custom dictionary with uncommon words
  • Verify Windows Push-to-Talk with compound hotkeys
  • Test GNOME Wayland hotkeys (if on GNOME + Wayland)
  • Test Hyprland Wayland hotkeys (if on Hyprland + Wayland)
  • Verify activation mode selector is hidden on GNOME Wayland and Hyprland Wayland
  • Verify meeting detection works with event-driven mode (check debug logs for "event-driven")
  • Test meeting notification suppression during recording
  • Test post-recording cooldown (notifications shouldn't flash immediately)
  • Create a note about "quarterly revenue projections", search via agent for "financial forecast" — should match semantically
  • Verify Qdrant stays down at launch and starts on the first agent search (check debug logs for "qdrant started successfully"), then stops after 5 idle minutes
  • Kill Qdrant process manually — verify FTS5 keyword search still works as fallback

Common Issues and Solutions

  1. No Audio Detected:

    • Check FFmpeg path resolution
    • Verify microphone permissions
    • Check audio levels in debug logs
  2. Transcription Fails:

    • Ensure whisper.cpp binary is available
    • Check model is downloaded
    • Check temporary file creation
    • Verify FFmpeg is executable
  3. Clipboard Not Working:

    • macOS: Check accessibility permissions (required for AppleScript paste)
    • Linux: Native linux-fast-paste binary (XTest) is tried first, works for X11 and XWayland apps
      • X11: xdotool fallback if native binary unavailable
      • Hyprland Wayland: wtype → sendshortcut → uinput/ydotool
      • Sway/wlroots Wayland: wtype → uinput/ydotool
      • COSMIC Wayland: wtype for a detected terminal → uinput/ydotool
      • GNOME/KDE Wayland: portal keysyms (Shift+Insert on KDE, except Warp) → uinput/ydotool
      • Physical Wayland fallbacks use Shift+Insert to avoid layout-sensitive KEY_V
    • Windows: PowerShell SendKeys (built-in) or nircmd.exe (bundled)
  4. Build Issues:

    • Use npm run pack for unsigned builds (CSC_IDENTITY_AUTO_DISCOVERY=false)
    • Signing requires Apple Developer account
    • ASAR unpacking needed for FFmpeg
    • Run npm run download:whisper-cpp before packaging (current platform)
    • Use npm run download:whisper-cpp:all for multi-platform packaging
    • afterSign.js automatically skips signing when CSC_IDENTITY_AUTO_DISCOVERY=false
    • Lockfile: Always use Node 24 when running npm install (matches CI). If your local Node version differs, use nvm exec 24 npm install. Running npm install with a different major version will produce an incompatible package-lock.json that breaks npm ci in CI.
  5. Windows Push-to-Talk Binary:

    • Compiled from resources/windows-key-listener.c whenever compile:native runs on a Windows host (prebuild:win, predev:main, …); the prebuilt binary download from GitHub releases is the fallback
    • If neither compiling nor the download produces it, afterPack.js fails the Windows build; an unpackaged dev run starts without it, with Hold and modifier-only hotkeys unavailable and F8 as the default hotkey
    • To compile locally: install Visual Studio Build Tools or MinGW-w64
    • CI workflow (.github/workflows/build-windows-key-listener.yml) rebuilds the prebuilt binary when resources/windows-key-listener.c changes on main
  6. Meeting Detection Not Working:

    • Check debug logs for "event-driven" vs "polling" mode; macOS also logs macOS microphone detection capability as PID or AGGREGATE
    • macOS: Verify macos-mic-listener binary exists in resources/bin/ (compiled during npm run compile:native)
    • Windows: Verify windows-mic-listener.exe exists in resources/bin/ (downloaded during prebuild:win)
    • Linux: Verify pactl is installed (pulseaudio-utils or pipewire-pulse package)
    • If the event-driven binary is missing, Windows/Linux fall back to polling; macOS pauses audio prompts (and respawns a crashed listener with backoff)
  7. Local Semantic Search Not Working:

    • Qdrant binary should be in resources/bin/qdrant-{platform}-{arch} (auto-downloaded during predev/prebuild)
    • Embedding model should be in ~/.cache/openwhispr/embedding-models/all-MiniLM-L6-v2/model.onnx (downloaded on the first semantic search)
    • Run npm run download:qdrant and npm run download:embedding-model manually if missing
    • Check debug logs for "qdrant" entries (port, health check, errors)
    • If Qdrant fails to start, search still works via FTS5 keyword fallback
    • Semantic search is only available through the AI agent's search_notes tool, not the manual search UI

Platform-Specific Notes

macOS:

  • Requires accessibility permissions for clipboard (auto-paste)
  • Requires microphone permission (prompted by system)
  • Uses AppleScript for reliable pasting
  • Notarization needed for distribution
  • Shows in dock with indicator dot when running (LSUIElement: false)
  • whisper.cpp bundled for both arm64 and x64
  • System settings accessible via x-apple.systempreferences: URL scheme
  • Launch at login: setLoginItemSettings(), which routes through SMAppService on macOS 13+. openAsHidden is deprecated and does nothing, so wasOpenedAtLogin is what sends a login launch to the tray. An item can register and still report status: "requires-approval" until the user allows it under System Settings → General → Login Items

Windows:

  • No special accessibility permissions needed
  • Microphone privacy settings at ms-settings:privacy-microphone
  • Sound settings at ms-settings:sound
  • NSIS installer for distribution
  • whisper.cpp bundled for x64
  • Launch at login: HKCU\...\Run entry written by Electron, named after the AppUserModelId, carrying --hidden so a login launch goes to the tray
    • Read the state from executableWillLaunchAtLogin; openAtLogin misses a startup app disabled from Task Manager or Settings
    • resources/nsis/installer.nsh removes the Run and StartupApproved\Run values on uninstall (but not on update), which Electron itself never cleans up
  • Tray identity: signed production builds pass a permanent GUID to new Tray() (tray.js) so Windows keeps the user's tray placement across updates. Never change the GUID
    • Windows binds an unsigned executable's GUID to its path, so the GUID is gated on the windowsTrayIdentity marker that electron-builder.json injects through extraMetadata
    • electron-builder.json forces Windows code signing; unsigned Windows builds (PR CI, local) must use electron-builder.unsigned-win.json, which clears the marker. Never run the win-unpacked a failed signed build leaves behind: it carries the marker and may be unsigned
  • Push-to-Talk: Native key listener binary (windows-key-listener.exe) enables true push-to-talk
    • Uses Windows Low-Level Keyboard Hook (WH_KEYBOARD_LL)
    • Supports compound hotkeys (e.g., Ctrl+Shift+F11)
    • Compiled from source during the build, with the prebuilt GitHub release binary as fallback (see section 12, Binary Distribution)
    • afterPack.js fails a Windows package without it

Linux:

  • Multiple package manager support
  • Standard XDG directories
  • AppImage for distribution
  • whisper.cpp bundled for x64
  • No standardized URL scheme for system settings (user must open manually)
  • Privacy settings button hidden in UI (not applicable on Linux)
  • Recommend pavucontrol for audio device management
  • GPU compositing: on, as on Windows, except whenever an NVIDIA kernel module (proprietary or open, any version) is loaded (linuxGpuCompositing.js, /proc/driver/nvidia/version): it flickered transparent windows in #203, Chromium's blocklist doesn't catch it, and explicit sync (driver 555+, Xwayland 24.1+, compositor support) can't be checked before app ready. Anyone else can turn it off with --disable-gpu-compositing in open-whispr-flags.conf, read by the packaged launcher (scripts/lib/linux-launcher.js). The mode is logged at startup, which only reaches the log file at --log-level=debug
  • Backdrop blurs: Linux can still composite on the CPU, where a large blur is redrawn on every repaint above it (#2298):
    • On Linux (ui/overlayBlur.ts), the full-window dialog overlays (shared Dialog, Settings, Ctrl+K, referral) skip their default backdrop blur; a blur passed through overlayClassName still applies
    • On Linux, the note action overlay (ActionProcessingOverlay), whose scanner animation never stops, uses a denser scrim and an opaque card, so the scanner line passes behind the label
    • Settings cards (SettingsPanel), the Settings selects and the LanguageSelector trigger sit on an opaque pane, so they carry no blur on any platform
  • Launch at login: XDG autostart entry at ~/.config/autostart/open-whispr.desktop (see linuxAutostart.js), since Electron's setLoginItemSettings() does nothing on Linux
    • Disabling it from GNOME Tweaks or KDE's autostart editor is reflected in the Settings toggle
    • "Start minimized" is handled app-side by the startMinimized setting, not by the desktop entry
  • Clipboard paste tools (at least one required for auto-paste):
    • X11: xdotool (recommended)
    • Hyprland Wayland: wtype, then hyprctl sendshortcut (avoids the sendshortcut stuck-modifier bug when wtype is installed)
    • Sway/wlroots Wayland: wtype (requires the virtual keyboard protocol)
    • COSMIC Wayland: wtype for terminals (Warp has no Shift+Insert paste); other windows paste with uinput Shift+Insert. Settings and onboarding ask for wtype there once something else pastes (needsWtype in src/utils/linuxPasteTools.ts), but checkPasteTools never reports it as the method
    • GNOME/KDE Wayland: RemoteDesktop portal keysyms (Shift+Insert on KDE, except Warp), then uinput/ydotool
    • Wayland physical fallback: Shift+Insert avoids layout-sensitive KEY_V; ydotool requires the ydotoold daemon
    • Terminal detection: Auto-detects terminal emulators and uses Ctrl+Shift+V
    • Fallback: Text copied to clipboard with manual paste instructions
  • GNOME Wayland global hotkeys:
    • Uses native GNOME shortcuts via D-Bus and gsettings (no special permissions needed)
    • Hotkeys visible in GNOME Settings → Keyboard → Shortcuts → Custom
    • Default fallback: F8 when Control+Super cannot be registered
    • Push-to-talk unavailable (GNOME shortcuts only fire single toggle event)
    • Falls back to X11/globalShortcut if GNOME integration fails
    • D-Bus transport: @homebridge/dbus-native (pure JavaScript, no native addons)

Code Style and Conventions

  • Use TypeScript for new React components
  • Follow existing patterns in helpers/
  • Descriptive error messages for users
  • Comprehensive debug logging
  • Clean up resources (files, listeners)
  • Handle edge cases gracefully

Performance Considerations

  • Whisper model size vs speed tradeoff
  • Audio blob size limits for IPC (10MB)
  • Temporary file cleanup
  • Memory usage with large models
  • Process timeout protection (5 minutes)
  • Meeting detection uses event-driven OS APIs (near-zero CPU) with polling fallback on Windows/Linux
  • Process list cache shared between detectors to avoid duplicate tasklist/pgrep calls
  • Calendar sync (Google/Microsoft) uses exponential backoff to avoid hammering APIs on network failures

Security Considerations

  • API keys and enterprise cloud creds (12 secrets total) encrypted at rest via Electron safeStorage → OS keychain (Keychain / DPAPI / libsecret), stored as per-key files in userData/secure-keys/. Linux without a keyring falls back to plaintext (Electron default). Closed in #629.
  • Context isolation enabled
  • No remote code execution
  • Sanitized file paths
  • Limited IPC surface area

Future Enhancements to Consider

  • Streaming transcription support
  • Custom wake word detection
  • Multi-language UI (implemented — 9 languages via react-i18next)
  • Cloud model selection
  • Batch transcription
  • Export formats beyond clipboard