The open-source AI voice studio. Clone, dictate, create.
Revamp README.md for improved clarity and presentation
Jamie Pinec14fb937ef35
diff --git a/.github/assets/icon-dark.webp b/.github/assets/icon-dark.webpnew file mode 100644index 00000000..0e0c5830Binary files /dev/null and b/.github/assets/icon-dark.webp differdiff --git a/.github/assets/screenshot.webp b/.github/assets/screenshot.webpnew file mode 100644index 00000000..4fe03fceBinary files /dev/null and b/.github/assets/screenshot.webp differdiff --git a/README.md b/README.mdindex eae6b2f0..81468f92 100644--- a/README.md+++ b/README.md@@ -1,409 +1,234 @@-# voicebox+<p align="center">+ <img src=".github/assets/icon-dark.webp" alt="Voicebox" width="120" height="120" />+</p>-A production-quality desktop app for Qwen3-TTS voice cloning and generation.+<h1 align="center">Voicebox</h1>-**Domain:** voicebox.sh+<p align="center">+ <strong>The open-source voice synthesis studio.</strong><br/>+ Clone voices. Generate speech. Build voice-powered apps.<br/>+ All running locally on your machine.+</p>++<p align="center">+ <a href="https://voicebox.sh">Website</a> •+ <a href="#download">Download</a> •+ <a href="#features">Features</a> •+ <a href="#api">API</a> •+ <a href="#roadmap">Roadmap</a>+</p>++<br/>++<p align="center">+ <img src=".github/assets/screenshot.webp" alt="Voicebox App Screenshot" width="800" />+</p>++<br/>++## Why Voicebox?++Voice AI is exploding, but most tools are either cloud-locked, expensive, or a nightmare to set up. Voicebox is different:++- **100% Local** — Your voice data never leaves your machine+- **Lightweight** — ~50MB app, runs on modest hardware+- **Fast** — Sub-2-second generation with caching+- **Flexible** — Use the app, integrate the API, or both+- **Open Source** — No subscriptions, no limits, no lock-in++Built with **Tauri** (Rust), **TypeScript**, **React**, and **Python**. Native performance meets modern DX.++---++## Download++Voicebox is available now for macOS, Windows, and Linux.++| Platform | Download |+|----------|----------|+| macOS (Apple Silicon) | [voicebox-macos-arm64.dmg](https://github.com/voicebox-sh/voicebox/releases/latest) |+| macOS (Intel) | [voicebox-macos-x64.dmg](https://github.com/voicebox-sh/voicebox/releases/latest) |+| Windows | [voicebox-windows-x64.exe](https://github.com/voicebox-sh/voicebox/releases/latest) |+| Linux | [voicebox-linux-x64.AppImage](https://github.com/voicebox-sh/voicebox/releases/latest) |++---++## Features++### Voice Cloning with Qwen3-TTS++Powered by Alibaba's **Qwen3-TTS** — a breakthrough model that achieves near-perfect voice cloning from just a few seconds of audio.++- **Instant cloning** — Upload a sample, get a voice profile+- **High fidelity** — Natural prosody, emotion, and cadence+- **Multi-language** — English, Chinese, and more coming++### Voice Profile Management++- **Create profiles** from audio files or record directly in-app+- **Import/Export** profiles to share or backup+- **Organize** with descriptions and language tags++### Speech Generation++- **Text-to-speech** with any cloned voice+- **Batch generation** for long-form content+- **Smart caching** — regenerate instantly with voice prompt caching++### Recording & Transcription++- **In-app recording** with waveform visualization+- **Automatic transcription** powered by Whisper+- **Export recordings** in multiple formats++### Generation History++- **Full history** of all generated audio+- **Search & filter** by voice, text, or date+- **Re-generate** any past generation with one click++### Flexible Deployment++- **Local mode** — Everything runs on your machine+- **Remote mode** — Connect to a GPU server on your network+- **One-click server** — Turn any machine into a Voicebox server----## Vision--Qwen3-TTS is a breakthrough model from Alibaba that achieves near-perfect voice cloning. The existing implementations (Voice-Clone-Studio, mimic, etc.) are either feature-rich but architecturally messy, or well-structured but limited in scope.--voicebox aims to build the definitive Qwen3-TTS application by combining the best patterns from existing projects while avoiding their architectural mistakes.--## Design Principles--1. **Clean architecture from day one** - No monolithic files, proper separation of concerns-2. **Desktop-first experience** - Native feel via Tauri, not a web app in disguise-3. **Production code quality** - Type safety, modularity, maintainability-4. **Performance and UX** - Smart caching, async operations, responsive UI-5. **Extensible design** - Easy to add new models, effects, and features-6. **Flexible deployment** - Run backend locally or connect to remote GPU machine with one click--## Technology Stack--### Backend (Python)-- **FastAPI** - Async REST API-- **SQLAlchemy** - Database ORM with migrations-- **Pydantic** - Request/response validation-- **Qwen3-TTS** - Voice cloning model-- **Whisper** - Speech-to-text transcription-- **librosa + soundfile** - Audio processing--### Frontend (Tauri + TypeScript)-- **Tauri** - Native desktop framework-- **React** - UI framework-- **TypeScript** - Type safety throughout-- **Bun** - Fast package manager and JavaScript runtime-- **React Query** - Server state management and API calls-- **OpenAPI (generated)** - Type-safe API client from FastAPI schema-- **Tailwind CSS** - Styling-- **Zustand** - Client-side state management-- **WaveSurfer.js** - Audio visualization--### Database-- **SQLite** - Local storage-- **Alembic** - Schema migrations--## Server/Client Mode--voicebox supports flexible deployment for users with multiple machines:--### Local Mode (Default)-- Backend runs locally alongside the Tauri app-- Best for users with GPU on their primary machine--### Remote Mode (One-Click Setup)-- **Use case:** Your laptop doesn't have a GPU, but your desktop does-- **Server:** Run voicebox on GPU machine, click "Start Server"- - Starts FastAPI backend on local network- - Shows connection URL (e.g., `http://192.168.1.100:8000`)-- **Client:** Run voicebox on laptop, enter server URL- - Connects to remote backend- - Full UI functionality, inference happens on GPU machine-- **Security:** Local network only for now (no internet exposure)--### How It Works+## API++Voicebox exposes a full REST API, so you can integrate voice synthesis into your own apps.++```bash+# Generate speech+curl -X POST http://localhost:8000/api/generate \+ -H "Content-Type: application/json" \+ -d '{"text": "Hello world", "profile_id": "abc123"}'++# List voice profiles+curl http://localhost:8000/api/profiles++# Create a profile from audio+curl -X POST http://localhost:8000/api/profiles \+ -F "[email protected]" \+ -F "name=My Voice"```-┌─────────────────┐ ┌─────────────────┐-│ Laptop │ │ Desktop │-│ (Client) │ │ (Server) │-│ │ │ │-│ Tauri App ────────────────▶ FastAPI │-│ React UI │ HTTP │ Qwen3-TTS │-│ │ │ SQLite │-│ │ │ CUDA/GPU │-└─────────────────┘ └─────────────────┘++**Use cases:**++- Game dialogue systems+- Podcast/video production pipelines+- Accessibility tools+- Voice assistants+- Content creation automation++Full API documentation available at `http://localhost:8000/docs` when running.++---++## Tech Stack++| Layer | Technology |+|-------|------------|+| Desktop App | Tauri (Rust) |+| Frontend | React, TypeScript, Tailwind CSS |+| State | Zustand, React Query |+| Backend | FastAPI (Python) |+| Voice Model | Qwen3-TTS |+| Transcription | Whisper |+| Database | SQLite |+| Audio | WaveSurfer.js, librosa |++**Why this stack?**++- **Tauri over Electron** — 10x smaller bundle, native performance, lower memory+- **FastAPI** — Async Python with automatic OpenAPI schema generation+- **Type-safe end-to-end** — Generated TypeScript client from OpenAPI spec++---++## Roadmap++Voicebox is the beginning of something bigger. Here's what's coming:++### Coming Soon++| Feature | Description |+|---------|-------------|+| **Real-time Synthesis** | Stream audio as it generates, word by word |+| **Conversation Mode** | Multi-speaker dialogues with automatic turn-taking |+| **Voice Effects** | Pitch shift, reverb, M3GAN-style effects |+| **Timeline Editor** | Audio studio with word-level precision editing |+| **More Models** | XTTS, Bark, and other open-source voice models |++### Future Vision++- **Voice Design** — Create new voices from text descriptions+- **Project System** — Save and load complex multi-voice sessions+- **Plugin Architecture** — Extend with custom models and effects+- **Mobile Companion** — Control Voicebox from your phone++Voicebox aims to be the **one-stop shop for everything voice** — cloning, synthesis, editing, effects, and beyond.++---++## Development++### Prerequisites++- [Bun](https://bun.sh) (package manager)+- [Rust](https://rustup.rs) (for Tauri)+- [Python 3.11+](https://python.org) (for backend)+- CUDA-capable GPU recommended (CPU inference supported but slower)++### Setup++```bash+# Clone the repo+git clone https://github.com/voicebox-sh/voicebox.git+cd voicebox++# Install dependencies+bun install++# Install Python dependencies+cd backend && pip install -r requirements.txt && cd ..++# Start development+bun run dev```-**Benefits:**-- Use powerful GPU machine from lightweight laptop-- No complex setup - just click "Start Server"-- All data (history, profiles) lives on server-- Client is just a UI - no local storage needed in remote mode--## Core Features--### Phase 1 (MVP)-- Voice profile management-- Single-reference voice cloning-- Generation history with search-- Basic audio playback and preview-- Server/client mode (local network)-- One-click server startup--### Phase 2-- Multi-reference voice combination-- Batch variation generation-- Advanced audio normalization-- Export options and formats--### Phase 3-- Audio studio with timeline editing-- Word-level timestamps-- Project system (save/load sessions)-- Export options--### Phase 4-- Voice design (text-to-voice)-- Preset voices with style control-- Conversation mode (multi-speaker)-- Custom audio effects--## Key Differentiators--What makes voicebox better than existing implementations:--1. **Clean codebase** - Modular architecture, no 2,000+ line files-2. **Type safety end-to-end** - OpenAPI-generated TypeScript client, Pydantic backend, React Query-3. **Smart caching** - Voice prompt caching for instant re-generation-4. **Desktop UX** - Native performance, keyboard shortcuts, native dialogs-5. **Server/client mode** - One-click remote GPU access from any device-6. **Multi-reference** - Combine voice samples for higher quality-7. **Audio studio** - Timeline-based editing with word-level precision-8. **Production patterns** - Cross-platform, graceful degradation, error recovery-9. **Database-backed** - Searchable history, project persistence-10. **Extensible** - Clean plugin system for models and features--## Architecture Overview+### Project Structure```voicebox/-├── app/ # Shared React frontend (used by web & desktop)-│ ├── src/-│ │ ├── components/ # React components-│ │ │ ├── VoiceProfiles/-│ │ │ ├── Generation/-│ │ │ ├── AudioStudio/-│ │ │ ├── History/-│ │ │ └── ServerSettings/-│ │ ├── lib/-│ │ │ ├── api/ # Generated OpenAPI client-│ │ │ ├── hooks/ # React Query hooks-│ │ │ └── utils/-│ │ ├── types/-│ │ └── App.tsx-│ ├── package.json-│ └── vite.config.ts-│-├── tauri/ # Tauri desktop app (thin wrapper)-│ ├── src/-│ │ └── main.tsx # Entry point, imports from ../app-│ ├── src-tauri/ # Rust backend-│ │ ├── src/-│ │ │ └── main.rs # Sidecar management, IPC-│ │ ├── binaries/ # Bundled Python server-│ │ │ └── voicebox-server-{platform}-│ │ ├── Cargo.toml-│ │ └── tauri.conf.json-│ └── package.json-│-├── web/ # Web deployment (thin wrapper)-│ ├── src/-│ │ └── main.tsx # Entry point, imports from ../app-│ ├── package.json-│ └── vite.config.ts-│-├── backend/ # Python FastAPI server-│ ├── main.py # FastAPI app + server mode-│ ├── models.py # Pydantic models-│ ├── tts.py # TTS inference-│ ├── transcribe.py # Whisper ASR-│ ├── profiles.py # Voice profiles-│ ├── history.py # Generation history-│ ├── studio.py # Audio editing-│ ├── database.py # SQLite ORM-│ ├── utils/-│ │ ├── audio.py # Audio processing-│ │ ├── cache.py # Prompt caching-│ │ └── validation.py-│ ├── requirements.txt-│ └── build_binary.py # PyInstaller build script-│-├── scripts/-│ ├── build-server.sh # Build Python binary for all platforms-│ └── generate-api.sh # Generate OpenAPI client-│-├── data/ # User data-│ ├── profiles/-│ ├── generations/-│ ├── projects/-│ └── voicebox.db-│-├── package.json # Root workspace config-└── docs/- ├── ANALYSIS.md # Analysis of existing projects- ├── TAURI_PLAN.md # Tauri app structure and bundling strategy- └── ARCHITECTURE.md # Detailed architecture docs+├── app/ # Shared React frontend+├── tauri/ # Desktop app (Tauri + Rust)+├── web/ # Web deployment+├── backend/ # Python FastAPI server+├── landing/ # Marketing website+└── scripts/ # Build & release scripts```-**Key architectural decisions:**-- **Shared frontend** - `app/` contains all React code, used by both desktop and web-- **Thin wrappers** - `tauri/` and `web/` just configure build tools and entry points-- **Bundled backend** - Python server packaged as sidecar binary with PyInstaller-- **Type-safe API** - OpenAPI schema generated from FastAPI, TypeScript client auto-generated--See [TAURI_PLAN.md](./docs/TAURI_PLAN.md) for detailed bundling strategy.--## Lessons from Existing Projects--voicebox learns from five existing Qwen3-TTS implementations:--### voice (Rust CLI)-- ✅ Clean Rust/Python IPC pattern-- ✅ M3GAN voice effect-- ✅ Voice profile abstraction-- ❌ No concurrent requests-- ❌ No generation history--### Voice-Clone-Studio-- ✅ Brilliant voice prompt caching-- ✅ Feature-rich (voice design, presets, conversations)-- ✅ VRAM-efficient model management-- ❌ 2,815-line single file-- ❌ Global state everywhere--### Qwen3-TTS_server-- ✅ Clean modular structure-- ✅ FastAPI REST API design-- ✅ Health endpoint for monitoring-- ❌ No authentication or rate limiting-- ❌ No caching or streaming-- ❌ No OpenAPI client generation--### mimic-- ✅ Excellent backend architecture (async, modular)-- ✅ Audio studio with timeline-- ✅ Database-backed history-- ✅ Multi-sample voice profiles-- ❌ 2,794-line app.js frontend-- ❌ Global state in UI--### qwen3-tts-enhanced-- ✅ Multi-reference combination-- ✅ Cross-platform graceful degradation-- ✅ Audio validation-- ✅ Production error handling-- ❌ Still monolithic (1,892 lines)-- ❌ No API layer--See [ANALYSIS.md](./docs/ANALYSIS.md) for detailed breakdown of each project.--## Development Roadmap--### Week 1: Foundation-- Project structure setup-- Backend skeleton (FastAPI + SQLite)-- OpenAPI schema generation-- Frontend skeleton (Tauri + React)-- TypeScript client generation from OpenAPI-- React Query setup-- Basic voice profile CRUD-- Server mode implementation-- Client connection UI--### Week 2: Core Features-- TTS integration-- Voice cloning pipeline-- Voice prompt caching-- Generation history--### Week 3: UX Polish-- Audio playback and preview-- Profile management UI-- History search and filters-- Error handling and validation--### Week 4: Advanced Features-- Multi-reference combination-- Batch generation-- Audio normalization-- M3GAN effect--### Week 5+: Studio Features-- Timeline editor-- Word-level timestamps-- Project system-- Export pipeline--## Technical Decisions--### Why Tauri over Electron?-- Smaller bundle size (Rust vs. Node.js)-- Better performance (native vs. V8)-- Lower memory usage-- Rust for system-level operations--### Why FastAPI over Flask?-- Native async/await support-- Automatic OpenAPI schema generation-- Pydantic validation built-in-- Better performance--### Why OpenAPI + React Query?-- **Type safety end-to-end** - FastAPI generates OpenAPI schema, we generate TypeScript client-- **No manual API code** - Client generated from `openapi.json` using openapi-typescript-codegen-- **Automatic caching** - React Query handles request deduplication and background refetching-- **Optimistic updates** - Update UI immediately, rollback on error-- **DevX** - Full autocomplete and type checking for all API calls--**Example workflow:**-```bash-# Backend generates OpenAPI schema-python backend/main.py --openapi > openapi.json+----# Frontend generates TypeScript client-bun run generate-client+## Contributing-# Use type-safe hooks in React-import { useQuery } from '@tanstack/react-query';-import { ProfilesService } from '@/lib/api';+Contributions welcome! Whether it's bug fixes, new features, or documentation improvements.-const { data: profiles } = useQuery({- queryKey: ['profiles'],- queryFn: () => ProfilesService.listProfiles()-});-```+1. Fork the repo+2. Create a feature branch+3. Make your changes+4. Submit a PR-### Why Bun over npm/yarn/pnpm?-- **Speed** - 20-30x faster than npm for install operations-- **Drop-in replacement** - Compatible with npm ecosystem, no migration needed-- **Built-in tooling** - Bundler, test runner, and package manager in one-- **Performance** - Faster script execution than Node.js-- **Developer experience** - Better error messages, workspaces support--### Why SQLite over file-based storage?-- Full-text search-- Transactions and integrity-- Migrations via Alembic-- Easy to backup/restore--### Why React over Vue/Svelte?-- Larger ecosystem-- Better TypeScript support-- Familiar to most developers-- Mature tooling--### Why bundle Python server with PyInstaller?-- **No Python installation required** - Users don't need Python on their system-- **Consistent environment** - Exact dependencies bundled, no version conflicts-- **Single-click install** - One installer includes everything-- **Tauri sidecar pattern** - Rust spawns/manages Python process lifecycle-- **Platform-specific binaries** - PyInstaller creates native executables for each platform--**Tradeoffs:**-- Larger bundle size (~500MB with models vs ~50MB without backend)-- Need separate build for each platform (macOS Intel/ARM, Windows, Linux)-- First launch slower (model loading time)--**Alternative considered:** Require users to install Python and run `pip install` - rejected for poor UX--### Why no Docker initially?-- Desktop app, not server deployment-- Users install locally-- Can add later for server mode--## Performance Targets--- **First generation:** < 10 seconds (cold start)-- **Cached generation:** < 2 seconds (warm start)-- **UI responsiveness:** 60 FPS at all times-- **Memory usage:** < 4GB VRAM for small models-- **Startup time:** < 3 seconds to UI-- **Database queries:** < 100ms for history search--## Quality Standards--- **No files over 500 lines** (except auto-generated)-- **Type hints on all Python functions**-- **TypeScript strict mode enabled**-- **OpenAPI client auto-generated from schema**-- **ESLint + Prettier for frontend**-- **Black + isort for backend**-- **All user-facing errors have context**-- **No global mutable state**-- **React Query for all server state**--## Project Status--**Current phase:** Planning and analysis--**Documentation:**-- [ANALYSIS.md](./docs/ANALYSIS.md) - Comprehensive analysis of existing implementations-- [TAURI_PLAN.md](./docs/TAURI_PLAN.md) - Tauri app architecture and Python server bundling strategy+---## License-TBD+MIT License — use it however you want.-## Credits--Built by analyzing and learning from:-- voice (Rust CLI)-- Voice-Clone-Studio-- Qwen3-TTS_server-- mimic-- qwen3-tts-enhanced+----Powered by Alibaba's Qwen3-TTS model.+<p align="center">+ <a href="https://voicebox.sh">voicebox.sh</a>+</p>