Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform.
JavaScript aianthropiccross-platformgeminigroqlinuxmacosnvidiaopen-sourceopenaiparakeetspeech-to-texttranscribewhisperwindows
feat: updated docs
Gabriel Stein6da919f4fc10
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 161ec67f2..706cfa745 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added+- **Automatic Python Installation**: The app now detects and offers to install Python automatically+ - macOS: Uses Homebrew if available, falls back to official installer+ - Windows: Downloads and installs official Python with proper PATH configuration+ - Linux: Uses system package manager (apt, yum, or pacman)+- **Enhanced Developer Experience**:+ - Added MIT LICENSE file+ - Improved documentation for personal vs distribution builds+ - Added FAQ section to README+ - Added security information section+ - Clearer prerequisites and setup instructions+### Added- **Agent Naming System**: Personalize your AI assistant with a custom name for more natural interactions- Name your agent during onboarding (step 6 of 8)diff --git a/CLAUDE.md b/CLAUDE.mdnew file mode 100644index 000000000..faa1fdee0--- /dev/null+++ b/CLAUDE.md@@ -0,0 +1,267 @@+# OpenWispr Technical Reference for AI Assistants++This document provides comprehensive technical details about the OpenWispr project architecture for AI assistants working on the codebase.++## Project Overview++OpenWispr is an Electron-based desktop dictation application that uses OpenAI Whisper for speech-to-text transcription. It supports both local (privacy-focused) and cloud (OpenAI API) processing modes.++## Architecture Overview++### Core Technologies+- **Frontend**: React 19, TypeScript, Tailwind CSS v4, Vite+- **Desktop Framework**: Electron 36 with context isolation+- **Database**: better-sqlite3 for local transcription history+- **UI Components**: shadcn/ui with Radix primitives+- **Speech Processing**: OpenAI Whisper (local Python bridge + API)+- **Audio Processing**: FFmpeg (bundled via ffmpeg-static)++### Key Architectural Decisions++1. **Dual Window Architecture**:+ - Main Window: Minimal overlay for dictation (draggable, always on top)+ - Control Panel: Full settings interface (normal window)+ - Both use same React codebase with URL-based routing++2. **Process Separation**:+ - Main Process: Electron main, IPC handlers, database operations+ - Renderer Process: React app with context isolation+ - Preload Script: Secure bridge between processes++3. **Audio Pipeline**:+ - MediaRecorder API → Blob → ArrayBuffer → Base64 → IPC → File → FFmpeg → Whisper+ - Automatic cleanup of temporary files after processing++## File Structure and Responsibilities++### Main Process Files++- **main.js**: Application entry point, initializes all managers+- **preload.js**: Exposes safe IPC methods to renderer via window.api++### Helper Modules (src/helpers/)++- **audioManager.js**: Handles audio device management+- **clipboard.js**: Cross-platform clipboard operations with AppleScript fallback+- **database.js**: SQLite operations for transcription history+- **debugLogger.js**: Debug logging system with file output+- **devServerManager.js**: Vite dev server integration+- **dragManager.js**: Window dragging functionality+- **environment.js**: Environment variable and OpenAI API management+- **hotkeyManager.js**: Global hotkey registration and management+- **ipcHandlers.js**: Centralized IPC handler registration+- **menuManager.js**: Application menu management+- **pythonInstaller.js**: Automatic Python installation for all platforms+- **tray.js**: System tray icon and menu+- **whisper.js**: Local Whisper integration and Python bridge+- **windowConfig.js**: Centralized window configuration+- **windowManager.js**: Window creation and lifecycle management++### React Components (src/components/)++- **App.jsx**: Main dictation interface with recording states+- **ControlPanel.tsx**: Settings, history, model management UI+- **OnboardingFlow.tsx**: 8-step first-time setup wizard+- **SettingsPage.tsx**: Comprehensive settings interface+- **WhisperModelPicker.tsx**: Model selection and download UI+- **ui/**: Reusable UI components (buttons, cards, inputs, etc.)++### React Hooks (src/hooks/)++- **useAudioRecording.js**: MediaRecorder API wrapper with error handling+- **useClipboard.ts**: Clipboard operations hook+- **useDialogs.ts**: Electron dialog integration+- **useHotkey.js**: Hotkey state management+- **useLocalStorage.ts**: Type-safe localStorage wrapper+- **usePermissions.ts**: System permission checks+- **usePython.ts**: Python installation state+- **useSettings.ts**: Application settings management+- **useWhisper.ts**: Whisper model management++### Services++- **ReasoningService.js**: AI processing for agent-addressed commands+ - Detects when user addresses their named agent+ - Routes to appropriate AI provider (OpenAI/Anthropic)+ - Removes agent name from final output++### Python Bridge++- **whisper_bridge.py**: Standalone Python script for local Whisper+ - Accepts audio file path and model selection+ - Returns JSON with transcription result+ - Handles FFmpeg path resolution for bundled executable+ - 30-second timeout protection++## Key Implementation Details++### 1. FFmpeg Integration++FFmpeg is bundled with the app and doesn't require system installation:+```javascript+// FFmpeg is unpacked from ASAR to app.asar.unpacked/node_modules/ffmpeg-static/+// Python bridge receives FFmpeg path via environment variables:+// FFMPEG_PATH, FFMPEG_EXECUTABLE, FFMPEG_BINARY+```++### 2. Audio Recording Flow++1. User presses hotkey → MediaRecorder starts+2. Audio chunks collected in array+3. User presses hotkey again → Recording stops+4. Blob created from chunks → Converted to ArrayBuffer+5. Sent via IPC as Base64 string (size limits)+6. Main process writes to temporary file+7. Whisper processes file → Result sent back+8. Temporary file deleted++### 3. Local Whisper Models++Models stored in `~/.cache/whisper/`:+- tiny: 39MB (fastest, lowest quality)+- base: 74MB (recommended balance)+- small: 244MB (better quality)+- medium: 769MB (high quality)+- large: 1.5GB (best quality)+- turbo: 809MB (fast with good quality)++### 4. Database Schema++```sql+CREATE TABLE transcriptions (+ id INTEGER PRIMARY KEY AUTOINCREMENT,+ timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,+ original_text TEXT NOT NULL,+ processed_text TEXT,+ is_processed BOOLEAN DEFAULT 0,+ processing_method TEXT DEFAULT 'none',+ agent_name TEXT,+ error TEXT+);+```++### 5. Settings Storage++Settings stored in localStorage with these keys:+- `whisperModel`: Selected Whisper model+- `useLocalWhisper`: Boolean for local vs cloud+- `openaiApiKey`: Encrypted API key+- `anthropicApiKey`: Encrypted API key+- `language`: Selected language code+- `agentName`: User's custom agent name+- `reasoningModel`: Selected AI model for processing+- `hotkey`: Custom hotkey configuration+- `hasCompletedOnboarding`: Onboarding completion flag++### 6. Language Support++58 languages supported (see src/utils/languages.ts):+- Each language has a two-letter code and label+- "auto" for automatic detection+- Passed to Whisper via --language parameter++### 7. Agent Naming System++- User names their agent during onboarding (step 6/8)+- Name stored in localStorage and database+- ReasoningService detects "Hey [AgentName]" patterns+- AI processes command and removes agent reference from output++### 8. Debug Mode++Enable with `--debug` flag or `OPENWISPR_DEBUG=true`:+- Logs saved to platform-specific app data directory+- Comprehensive logging of audio pipeline+- FFmpeg path resolution details+- Audio level analysis++## Development Guidelines++### Adding New Features++1. **New IPC Channel**: Add to both ipcHandlers.js and preload.js+2. **New Setting**: Update useSettings.ts and SettingsPage.tsx+3. **New UI Component**: Follow shadcn/ui patterns in src/components/ui+4. **New Manager**: Create in src/helpers/, initialize in main.js++### Testing Checklist++- [ ] Test both local and cloud processing modes+- [ ] Verify hotkey works globally+- [ ] Check clipboard pasting on all platforms+- [ ] Test with different audio input devices+- [ ] Verify Python auto-installation+- [ ] Test all Whisper models+- [ ] Check agent naming functionality++### Common Issues and Solutions++1. **No Audio Detected**:+ - Check FFmpeg path resolution+ - Verify microphone permissions+ - Check audio levels in debug logs++2. **Transcription Fails**:+ - Ensure Python/Whisper installed+ - Check temporary file creation+ - Verify FFmpeg is executable++3. **Clipboard Not Working**:+ - macOS: Check accessibility permissions+ - Use AppleScript fallback on macOS++4. **Build Issues**:+ - Use `npm run pack` for unsigned builds+ - Signing requires Apple Developer account+ - ASAR unpacking needed for FFmpeg/Python bridge++### Platform-Specific Notes++**macOS**:+- Requires accessibility permissions for clipboard+- Uses AppleScript for reliable pasting+- Notarization needed for distribution++**Windows**:+- Python installer handles PATH automatically+- No special permissions needed+- NSIS installer for distribution++**Linux**:+- Multiple package manager support+- Standard XDG directories+- AppImage for distribution++## Code Style and Conventions++- Use TypeScript for new React components+- Follow existing patterns in helpers/+- Descriptive error messages for users+- Comprehensive debug logging+- Clean up resources (files, listeners)+- Handle edge cases gracefully++## Performance Considerations++- Whisper model size vs speed tradeoff+- Audio blob size limits for IPC (10MB)+- Temporary file cleanup+- Memory usage with large models+- Process timeout protection (30s)++## Security Considerations++- API keys stored in system keychain when possible+- Context isolation enabled+- No remote code execution+- Sanitized file paths+- Limited IPC surface area++## Future Enhancements to Consider++- Streaming transcription support+- Custom wake word detection+- Multi-language UI+- Cloud model selection+- Batch transcription+- Export formats beyond clipboard\ No newline at end of filediff --git a/LICENSE b/LICENSEnew file mode 100644index 000000000..382baee3c--- /dev/null+++ b/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2024 OpenWispr Team++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.\ No newline at end of filediff --git a/README.md b/README.mdindex a5ba96f83..d7886a427 100644--- a/README.md+++ b/README.md@@ -2,6 +2,10 @@An open source desktop dictation application that converts speech to text using OpenAI Whisper. Features both local and cloud processing options for maximum flexibility and privacy.+## License++This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. This means you can freely use, modify, and distribute this software for personal or commercial purposes.+## Features- 🎤 **Global Hotkey**: Customizable hotkey to start/stop dictation from anywhere (default: backtick `)@@ -17,38 +21,66 @@ An open source desktop dictation application that converts speech to text using- ⚡ **Automatic Pasting**: Transcribed text automatically pastes at your cursor location- 🖱️ **Draggable Interface**: Move the dictation panel anywhere on your screen+## Prerequisites++- **Node.js 18+** and npm (Download from [nodejs.org](https://nodejs.org/))+- **macOS 10.15+**, **Windows 10+**, or **Linux**+- **Python 3.7+** (Optional - the app can install it automatically for local Whisper processing)+## Quick Start-1. **Install dependencies**:+### For Personal Use (Recommended)++1. **Clone the repository**:+ ```bash+ git clone https://github.com/HeroTools/open-wispr.git+ cd open-wispr+ ```++2. **Install dependencies**:```bashnpm install```-2. **Optional: Set up OpenAI API key** (only needed for cloud processing):+3. **Optional: Set up OpenAI API key** (only needed for cloud processing):- **Method A - .env file**:+ **Method A - Environment file**:```bashcp env.example .env# Edit .env and add your OpenAI API key```- **Method B - Control Panel**:- ```bash- # Run the app and configure the API key through the Control Panel- ```+ **Method B - In-app configuration**:+ - Run the app and configure the API key through the Control Panel-3. **Run in development**:+4. **Run the application**:```bash- npm run dev+ npm run dev # Development mode with hot reload+ # OR+ npm start # Production mode```-4. **Build for production**:- ```bash- # Build for your platform- npm run build:mac # macOS- npm run build:win # Windows- npm run build:linux # Linux- ```+### Building for Personal Use (Optional)++If you want to build a standalone app for personal use:++```bash+# Build without code signing (personal use only)+npm run pack++# The built app will be in the 'dist' folder+```++### Building for Distribution++For maintainers who need to distribute signed builds:++```bash+# Requires code signing certificates and notarization setup+npm run build:mac # macOS (requires Apple Developer account)+npm run build:win # Windows (requires code signing cert)+npm run build:linux # Linux+```### First Time Setup@@ -236,14 +268,23 @@ DEBUG=false### Local Whisper Setup-For local processing, OpenWispr will automatically:-1. Install Python Whisper package via pip-2. Download your chosen model (tiny, base, small, medium, large)-3. Handle all transcription locally+For local processing, OpenWispr offers automated setup:++1. **Automatic Python Installation** (if needed):+ - The app will detect if Python is missing+ - Offers to install Python 3.11 automatically+ - macOS: Uses Homebrew if available, otherwise official installer+ - Windows: Downloads and installs official Python+ - Linux: Uses system package manager (apt, yum, or pacman)++2. **Automatic Whisper Setup**:+ - Installs OpenAI Whisper package via pip+ - Downloads your chosen model on first use+ - Handles all transcription locally**Requirements**:-- Python 3.7+ installed on your system- Sufficient disk space for models (39MB - 1.5GB depending on model)+- Admin/sudo access may be required for Python installation### Customization@@ -257,11 +298,30 @@ For local processing, OpenWispr will automatically:## Contributing+We welcome contributions! Please follow these steps:+1. Fork the repository-2. Create a feature branch-3. Make your changes-4. Run tests and lint-5. Submit a pull request+2. Create a feature branch (`git checkout -b feature/amazing-feature`)+3. Commit your changes (`git commit -m 'Add amazing feature'`)+4. Push to the branch (`git push origin feature/amazing-feature`)+5. Open a Pull Request++### Development Guidelines++- Run `npm run lint` before committing+- Follow the existing code style+- Update documentation as needed+- Test on your target platform before submitting+## Security++OpenWispr is designed with privacy and security in mind:++- **Local Processing Option**: Keep your voice data completely private+- **No Analytics**: We don't collect any usage data or telemetry+- **Open Source**: All code is available for review+- **Secure Storage**: API keys are stored securely in your system's keychain/credential manager+- **Minimal Permissions**: Only requests necessary permissions (microphone, accessibility)+## Troubleshooting### Common Issues@@ -296,3 +356,35 @@ For local processing, OpenWispr will automatically:- **Cloud Processing**: Generally faster but requires internet connection- **Model Selection**: tiny (fastest) → base (recommended) → small → medium → large (best quality)- **Permissions**: Ensure all required permissions are granted for smooth operation++## FAQ++**Q: Is OpenWispr really free?**+A: Yes! OpenWispr is open source and free to use. You only pay for OpenAI API usage if you choose cloud processing.++**Q: Which processing method should I use?**+A: Use local processing for privacy and offline use. Use cloud processing for speed and convenience.++**Q: Can I use this commercially?**+A: Yes! The MIT license allows commercial use.++**Q: How do I change the hotkey?**+A: Open the Control Panel (right-click tray icon) and go to Settings. You can set any key as your hotkey.++**Q: Is my data secure?**+A: With local processing, your audio never leaves your device. With cloud processing, audio is sent to OpenAI's servers (see their privacy policy).++**Q: What languages are supported?**+A: OpenWispr supports 58 languages including English, Spanish, French, German, Chinese, Japanese, and more. Set your preferred language in the .env file or use auto-detect.++## Project Status++OpenWispr is actively maintained and ready for production use. Current version: 1.0.1++- ✅ Core functionality complete+- ✅ Cross-platform support+- ✅ Local and cloud processing+- ✅ Automatic Python/Whisper installation+- ✅ Agent naming system+- ✅ Draggable interface+- 🚧 Continuous improvements and bug fixes