Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.
JavaScript aianthropiccross-platformgeminigroqlinuxmacosnvidiaopen-sourceopenaiparakeetspeech-to-texttranscribewhisperwindows
feat: added in agent capabilities
Gabriel Stein49b98c4da525
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c2f078fe6..161ec67f2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- **Agent Naming System**: Personalize your AI assistant with a custom name for more natural interactions+ - Name your agent during onboarding (step 6 of 8)+ - Address your agent directly: "Hey [AgentName], make this more professional"+ - Update agent name anytime through settings+ - Smart AI processing distinguishes between commands and regular dictation+ - Clean output automatically removes agent name references- **Draggable Interface**: Click and drag the dictation panel to any position on screen- **Dynamic Hotkey Display**: Tooltip shows your actual hotkey setting instead of generic text- **Flexible Hotkey System**: Fixed hardcoded hotkey limitation - now fully respects user settings@@ -24,6 +30,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- **User Experience**: Eliminated confusion between drag and click actions### Technical Details+- **Agent Naming Implementation**:+ - Added centralized agent name utility (`src/utils/agentName.ts`)+ - Enhanced onboarding flow with agent naming step+ - Updated ReasoningService with context-aware AI processing+ - Added agent name settings section with comprehensive UI+ - Implemented smart prompt generation for agent-addressed vs regular text- Added IPC handlers for dynamic hotkey updates (`update-hotkey`)- Implemented window-level dragging using screen cursor tracking- Added real-time hotkey loading from localStorage in main dictation componentdiff --git a/README.md b/README.mdindex 3628591e8..a5ba96f83 100644--- a/README.md+++ b/README.md@@ -6,6 +6,7 @@ An open source desktop dictation application that converts speech to text using- π€ **Global Hotkey**: Customizable hotkey to start/stop dictation from anywhere (default: backtick `)- π€ **Dual AI Processing**: Choose between local Whisper models (private) or OpenAI API (fast)+- π― **Agent Naming**: Personalize your AI assistant with a custom name for natural interactions- π **Privacy-First**: Local processing keeps your voice data completely private- π¨ **Modern UI**: Built with React 19, TypeScript, and Tailwind CSS v4- π **Fast**: Optimized with Vite and modern tooling@@ -59,7 +60,12 @@ An open source desktop dictation application that converts speech to text using- **Microphone Access**: Required for voice recording- **Accessibility Permissions**: Required for automatic text pasting (macOS)-3. **Configure Global Hotkey**: Default is backtick (`) but can be customized+3. **Name Your Agent**: Give your AI assistant a personal name (e.g., "Assistant", "Jarvis", "Alex")+ - Makes interactions feel more natural and conversational+ - Helps distinguish between giving commands and regular dictation+ - Can be changed anytime in settings++4. **Configure Global Hotkey**: Default is backtick (`) but can be customized## Usage@@ -77,6 +83,22 @@ An open source desktop dictation application that converts speech to text using- **Models**: Download and manage local Whisper models- **Settings**: Configure API keys, customize hotkeys, and manage permissions+### Agent Naming+Once you've named your agent during setup, you can interact with it in two ways:++**π― Agent Commands** (for AI assistance):+- "Hey [AgentName], make this more professional"+- "Hey [AgentName], format this as a list"+- "Hey [AgentName], write a thank you email"+- "Hey [AgentName], convert this to bullet points"++**π Regular Dictation** (for normal text):+- "This is just normal text I want transcribed"+- "Meeting notes: John mentioned the quarterly report"+- "Dear Sarah, thank you for your help"++The AI automatically detects when you're giving it commands versus dictating regular text, and removes agent name references from the final output.+### Processing Options- **Local Processing**:- Install Whisper automatically through the Control Panel@@ -106,6 +128,8 @@ open-wispr/β βββ vite.config.js # Vite configurationβ βββ components/β β βββ ControlPanel.tsx # Settings and history UI+β β βββ OnboardingFlow.tsx # First-time setup wizard+β β βββ SettingsPage.tsx # Settings interfaceβ β βββ ui/ # shadcn/ui componentsβ β β βββ button.tsxβ β β βββ card.tsx@@ -117,6 +141,10 @@ open-wispr/β β β βββ tooltip.tsxβ β βββ lib/β β βββ utils.ts # Utility functions+β βββ services/+β β βββ ReasoningService.js # AI processing and agent name detection+β βββ utils/+β β βββ agentName.ts # Agent name management utilityβ βββ components.json # shadcn/ui configurationβββ assets/ # App icons and resources```diff --git a/src/components/OnboardingFlow.tsx b/src/components/OnboardingFlow.tsxindex 2b23adff8..7fbfce469 100644--- a/src/components/OnboardingFlow.tsx+++ b/src/components/OnboardingFlow.tsx@@ -17,6 +17,7 @@ import {Sparkles,Lock,X,+ User,} from "lucide-react";import TitleBar from "./TitleBar";import WhisperModelPicker from "./WhisperModelPicker";@@ -33,6 +34,7 @@ import { useClipboard } from "../hooks/useClipboard";import { useSettings } from "../hooks/useSettings";import { getLanguageLabel, getReasoningModelLabel } from "../utils/languages";import LanguageSelector from "./ui/LanguageSelector";+import { setAgentName as saveAgentName } from "../utils/agentName";interface OnboardingFlowProps {onComplete: () => void;@@ -68,6 +70,7 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {const [apiKey, setApiKey] = useState(openaiApiKey);const [hotkey, setHotkey] = useState(dictationKey || "`");+ const [agentName, setAgentName] = useState("Agent");const { alertDialog, showAlertDialog, hideAlertDialog } = useDialogs();const practiceTextareaRef = useRef<HTMLInputElement>(null);@@ -82,6 +85,7 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {{ title: "Permissions", icon: Shield },{ title: "Hotkey", icon: Keyboard },{ title: "Test", icon: TestTube },+ { title: "Agent Name", icon: User },{ title: "Finish", icon: Check },];@@ -104,6 +108,7 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {const saveSettings = useCallback(async () => {updateTranscriptionSettings({ whisperModel, preferredLanguage });setDictationKey(hotkey);+ saveAgentName(agentName);localStorage.setItem("micPermissionGranted",@@ -123,6 +128,7 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {whisperModel,hotkey,preferredLanguage,+ agentName,permissionsHook.micPermissionGranted,permissionsHook.accessibilityPermissionGranted,useLocalWhisper,@@ -597,7 +603,53 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {</div>);- case 6: // Complete+ case 6: // Agent Name+ return (+ <div className="space-y-6">+ <div className="text-center">+ <h2 className="text-2xl font-bold text-stone-900 mb-2">+ Name Your Agent+ </h2>+ <p className="text-stone-600">+ Give your agent a name so you can address it specifically when+ giving instructions.+ </p>+ </div>++ <div className="space-y-4 p-4 bg-gradient-to-r from-purple-50 to-blue-50 border border-purple-200 rounded-xl">+ <h4 className="font-medium text-purple-900 mb-3">+ π‘ How this helps:+ </h4>+ <ul className="text-sm text-purple-800 space-y-1">+ <li>+ β’ Say "Hey {agentName || "Agent"}, write a formal email" for+ specific instructions+ </li>+ <li>+ β’ Use the name to distinguish between dictation and commands+ </li>+ <li>β’ Makes interactions feel more natural and personal</li>+ </ul>+ </div>++ <div className="space-y-4">+ <label className="block text-sm font-medium text-gray-700 mb-2">+ Agent Name+ </label>+ <Input+ placeholder="e.g., Assistant, Jarvis, Alex..."+ value={agentName}+ onChange={(e) => setAgentName(e.target.value)}+ className="text-center text-lg font-mono"+ />+ <p className="text-xs text-gray-500 mt-2">+ You can change this anytime in settings+ </p>+ </div>+ </div>+ );++ case 7: // Completereturn (<div className="text-center space-y-6"><div className="w-16 h-16 mx-auto bg-green-100 rounded-full flex items-center justify-center">@@ -637,6 +689,10 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {{getLanguageLabel(preferredLanguage)}</span></div>+ <div className="flex justify-between">+ <span>Agent Name:</span>+ <span className="font-medium">{agentName}</span>+ </div><div className="flex justify-between"><span>Permissions:</span><span className="font-medium text-green-600">@@ -685,6 +741,8 @@ export default function OnboardingFlow({ onComplete }: OnboardingFlowProps) {case 5:return true; // Practice step is always ready to proceedcase 6:+ return agentName.trim() !== ""; // Agent name step+ case 7:return true;default:return false;diff --git a/src/components/SettingsPage.tsx b/src/components/SettingsPage.tsxindex 2ab6d645e..aa1a56d34 100644--- a/src/components/SettingsPage.tsx+++ b/src/components/SettingsPage.tsx@@ -13,6 +13,7 @@ import {Brain,Shield,ChevronRight,+ User,} from "lucide-react";import TitleBar from "./TitleBar";import WhisperModelPicker from "./WhisperModelPicker";@@ -21,6 +22,7 @@ import ApiKeyInput from "./ui/ApiKeyInput";import { ConfirmDialog, AlertDialog } from "./ui/dialog";import { useSettings } from "../hooks/useSettings";import { useDialogs } from "../hooks/useDialogs";+import { useAgentName } from "../utils/agentName";import { useWhisper } from "../hooks/useWhisper";import { usePermissions } from "../hooks/usePermissions";import { useClipboard } from "../hooks/useClipboard";@@ -35,6 +37,7 @@ type SettingsSectionType =| "transcription"| "reasoning"| "hotkey"+ | "agentName"| "updates"| "about";@@ -101,6 +104,7 @@ export default function SettingsPage({ onBack }: SettingsPageProps) {const whisperHook = useWhisper(showAlertDialog);const permissionsHook = usePermissions(showAlertDialog);const { pasteFromClipboardWithFallback } = useClipboard(showAlertDialog);+ const { agentName, setAgentName } = useAgentName();useEffect(() => {whisperHook.checkWhisperInstallation();@@ -270,6 +274,11 @@ export default function SettingsPage({ onBack }: SettingsPageProps) {label: "Hotkey Setup",icon: Keyboard,},+ {+ id: "agentName" as SettingsSectionType,+ label: "Agent Name",+ icon: User,+ },{id: "updates" as SettingsSectionType,label: "App Updates",@@ -556,6 +565,93 @@ export default function SettingsPage({ onBack }: SettingsPageProps) {</div>);+ case "agentName":+ return (+ <div className="space-y-6">+ <div>+ <h3 className="text-lg font-semibold text-gray-900 mb-2">+ Agent Name Configuration+ </h3>+ <p className="text-sm text-gray-600 mb-6">+ Customize your agent's name to make interactions feel more+ personal and natural.+ </p>+ </div>++ <div className="space-y-4 p-4 bg-gradient-to-r from-purple-50 to-blue-50 border border-purple-200 rounded-xl">+ <h4 className="font-medium text-purple-900 mb-3">+ π‘ How to use agent names:+ </h4>+ <ul className="text-sm text-purple-800 space-y-2">+ <li>+ β’ Say "Hey {agentName}, write a formal email" for specific+ instructions+ </li>+ <li>+ β’ Use "Hey {agentName}, format this as a list" for text+ enhancement commands+ </li>+ <li>+ β’ The agent will recognize when you're addressing it directly+ vs. dictating content+ </li>+ <li>+ β’ Makes conversations feel more natural and helps distinguish+ commands from dictation+ </li>+ </ul>+ </div>++ <div className="space-y-4 p-4 bg-gray-50 border border-gray-200 rounded-xl">+ <h4 className="font-medium text-gray-900">Current Agent Name</h4>+ <div className="flex gap-3">+ <Input+ placeholder="e.g., Assistant, Jarvis, Alex..."+ value={agentName}+ onChange={(e) => setAgentName(e.target.value)}+ className="flex-1 text-center text-lg font-mono"+ />+ <Button+ onClick={() => {+ setAgentName(agentName.trim());+ showAlertDialog({+ title: "Agent Name Updated",+ description: `Your agent is now named "${agentName.trim()}". You can address it by saying "Hey ${agentName.trim()}" followed by your instructions.`,+ });+ }}+ disabled={!agentName.trim()}+ >+ Save+ </Button>+ </div>+ <p className="text-xs text-gray-600 mt-2">+ Choose a name that feels natural to say and remember+ </p>+ </div>++ <div className="bg-blue-50 p-4 rounded-lg">+ <h4 className="font-medium text-blue-900 mb-2">+ π― Example Usage:+ </h4>+ <div className="text-sm text-blue-800 space-y-1">+ <p>+ β’ "Hey {agentName}, write an email to my team about the+ meeting"+ </p>+ <p>+ β’ "Hey {agentName}, make this more professional" (after+ dictating text)+ </p>+ <p>β’ "Hey {agentName}, convert this to bullet points"</p>+ <p>+ β’ Regular dictation: "This is just normal text" (no agent name+ needed)+ </p>+ </div>+ </div>+ </div>+ );+case "updates":return (<div className="space-y-6">diff --git a/src/services/ReasoningService.js b/src/services/ReasoningService.jsindex 506219aa0..045b7c443 100644--- a/src/services/ReasoningService.js+++ b/src/services/ReasoningService.js@@ -1,6 +1,33 @@import { getModelProvider } from "../utils/languages.ts";+import { getAgentName } from "../utils/agentName.ts";++const getReasoningPrompt = (text, agentName) => {+ const hasAgentReference =+ agentName &&+ (text.toLowerCase().includes(`hey ${agentName.toLowerCase()}`) ||+ text.toLowerCase().includes(agentName.toLowerCase()));++ if (hasAgentReference) {+ return `You are ${agentName}, an AI text formatting assistant. The user has addressed you by name and is giving you specific instructions to process their text.++Your role:+1. When addressed as "${agentName}" or "Hey ${agentName}", you are being given instructions about how to format or process text+2. Look for commands like:+ - "${agentName}, make this more professional"+ - "${agentName}, format this as a list"+ - "${agentName}, write an email about..."+ - "${agentName}, convert this to bullet points"+3. Follow the user's formatting instructions while preserving their intent+4. If asked to write content, create it based on their request+5. Remove the agent name references from the final output (don't include "Hey ${agentName}" in your response)+6. For editing commands, apply the requested changes to the text that follows++Transcript with instructions:+"{{text}}"-const REASONING_PROMPT = `You are a text formatting assistant. Your job is to clean up and format voice-to-text transcriptions while preserving the speaker's natural tone and intent.+Processed text:`;+ } else {+ return `You are a text formatting assistant. Your job is to clean up and format voice-to-text transcriptions while preserving the speaker's natural tone and intent.Rules:1. If the speaker gives instructions like "scratch that", "ignore that", "delete the previous part", "never mind", or similar - follow them and remove the referenced content@@ -14,6 +41,8 @@ Transcript to format:"{{text}}"Formatted text:`;+ }+};class ReasoningService {constructor() {@@ -40,7 +69,11 @@ class ReasoningService {async processWithOpenAI(text, model) {const apiKey = await this.getAPIKey("openai");- const prompt = REASONING_PROMPT.replace("{{text}}", text);+ const agentName = getAgentName();+ const prompt = getReasoningPrompt(text, agentName).replace(+ "{{text}}",+ text+ );const maxTokens = Math.max(100, text.length * 2);const response = await fetch("https://api.openai.com/v1/chat/completions", {@@ -74,7 +107,11 @@ class ReasoningService {async processWithAnthropic(text, model) {const apiKey = await this.getAPIKey("anthropic");- const prompt = REASONING_PROMPT.replace("{{text}}", text);+ const agentName = getAgentName();+ const prompt = getReasoningPrompt(text, agentName).replace(+ "{{text}}",+ text+ );const maxTokens = Math.max(100, text.length * 2);const response = await fetch("https://api.anthropic.com/v1/messages", {diff --git a/src/utils/agentName.ts b/src/utils/agentName.tsnew file mode 100644index 000000000..dc9eaeb9d--- /dev/null+++ b/src/utils/agentName.ts@@ -0,0 +1,26 @@+import { useState } from "react";++const AGENT_NAME_KEY = "agentName";++export const getAgentName = (): string => {+ return localStorage.getItem(AGENT_NAME_KEY) || "Agent";+};++export const setAgentName = (name: string): void => {+ localStorage.setItem(AGENT_NAME_KEY, name);+};++export const clearAgentName = (): void => {+ localStorage.removeItem(AGENT_NAME_KEY);+};++export const useAgentName = () => {+ const [agentName, setAgentNameState] = useState<string>(getAgentName());++ const updateAgentName = (name: string) => {+ setAgentName(name);+ setAgentNameState(name);+ };++ return { agentName, setAgentName: updateAgentName };+};