The open-source AI voice studio. Clone, dictate, create.
Update TTS Provider Architecture status to v0.1.13
Jamie Pinecb541521d2a0
diff --git a/docs/plans/TTS_PROVIDER_ARCHITECTURE.md b/docs/plans/TTS_PROVIDER_ARCHITECTURE.mdindex 37ad40f7..417c840e 100644--- a/docs/plans/TTS_PROVIDER_ARCHITECTURE.md+++ b/docs/plans/TTS_PROVIDER_ARCHITECTURE.md@@ -1,6 +1,6 @@# TTS Provider Architecture-**Status:** Planned for v0.2.0+**Status:** Planned for v0.1.13**Created:** 2025-01-31**Problem:** GitHub 2GB release limit + poor UX for frequent updates requiring 2.4GB re-downloads@@ -14,6 +14,7 @@ Split the monolithic backend into modular components:2. **TTS Providers** (downloadable plugins): Separate executables for model inferenceThis architecture solves:+- ✅ GitHub 2GB release artifact limit- ✅ Frequent app updates without re-downloading large ML models- ✅ User choice of compute backend (CPU/GPU/Cloud)@@ -68,6 +69,7 @@ This architecture solves:### Current Architecture Issues**Monolithic Binary:**+- CPU version: ~295MB- CUDA version: ~2.37GB- GitHub releases: 2GB file size limit (BLOCKED)@@ -75,6 +77,7 @@ This architecture solves:- Poor UX: update app → restart → download CUDA update → restart again**User Pain Points:**+1. Cannot release CUDA version on GitHub (over 2GB)2. Every app update forces 2.4GB re-download for GPU users3. No flexibility (can't use OpenAI, remote servers, etc.)@@ -91,6 +94,7 @@ This architecture solves:**Size:** ~150-200MB**Includes:**+- Tauri runtime + React UI- FastAPI backend (pure Python, no PyTorch)- Whisper model (tiny, ~50MB)@@ -99,6 +103,7 @@ This architecture solves:- Provider management system**Does NOT include:**+- PyTorch (CPU or CUDA)- TTS models (Qwen3-TTS)- Heavy ML dependencies@@ -113,6 +118,7 @@ This architecture solves:**Size:** ~300MB**Includes:**+- PyTorch CPU build- Qwen3-TTS package- Transformers@@ -129,6 +135,7 @@ This architecture solves:**Size:** ~2.4GB**Includes:**+- PyTorch CUDA build (cu121)- Qwen3-TTS package- CUDA runtime, cuDNN, cuBLAS@@ -146,6 +153,7 @@ This architecture solves:**Size:** ~800MB**Includes:**+- MLX framework- MLX-optimized Qwen3-TTS- Metal acceleration@@ -161,11 +169,13 @@ This architecture solves:**Size:** 0MB**How it works:**+- User provides URL to their own TTS server- Backend proxies requests to that server- Implements API spec from `EXTERNAL_PROVIDERS.md`**Use cases:**+- AMD GPU users running their own server- Team deployments with shared GPU server- Cloud hosting (Modal, RunPod, Replicate)@@ -178,11 +188,13 @@ This architecture solves:**Size:** 0MB**How it works:**+- User provides OpenAI API key- Backend wraps OpenAI Audio API- Voice profiles map to OpenAI voices**Benefits:**+- Zero local compute- Pay-per-use- Instant setup@@ -200,22 +212,26 @@ All TTS providers must implement these endpoints:Generate speech from text.**Request:**+```json{- "text": "Hello world!",- "voice_prompt": { /* voice prompt object */ },- "language": "en",- "seed": 12345,- "model_size": "1.7B"+ "text": "Hello world!",+ "voice_prompt": {+ /* voice prompt object */+ },+ "language": "en",+ "seed": 12345,+ "model_size": "1.7B"}```**Response:**+```json{- "audio": "base64-encoded-audio",- "sample_rate": 24000,- "duration": 2.5+ "audio": "base64-encoded-audio",+ "sample_rate": 24000,+ "duration": 2.5}```@@ -224,13 +240,17 @@ Generate speech from text.Create voice prompt from reference audio.**Request:** (multipart/form-data)+- `audio`: Audio file- `reference_text`: Transcript**Response:**+```json{- "voice_prompt": { /* serialized prompt */ }+ "voice_prompt": {+ /* serialized prompt */+ }}```@@ -239,13 +259,14 @@ Create voice prompt from reference audio.Health check.**Response:**+```json{- "status": "healthy",- "provider": "pytorch-cuda",- "version": "1.0.0",- "model": "Qwen3-TTS-12Hz-1.7B-Base",- "device": "cuda:0"+ "status": "healthy",+ "provider": "pytorch-cuda",+ "version": "1.0.0",+ "model": "Qwen3-TTS-12Hz-1.7B-Base",+ "device": "cuda:0"}```@@ -254,13 +275,14 @@ Health check.Model status.**Response:**+```json{- "model_loaded": true,- "model_size": "1.7B",- "available_sizes": ["0.6B", "1.7B"],- "gpu_available": true,- "vram_used_mb": 1234+ "model_loaded": true,+ "model_size": "1.7B",+ "available_sizes": ["0.6B", "1.7B"],+ "gpu_available": true,+ "vram_used_mb": 1234}```@@ -434,6 +456,7 @@ class ProviderInstaller:```**Provider Storage Location:**+- Windows: `%APPDATA%/voicebox/providers/`- macOS: `~/Library/Application Support/voicebox/providers/`- Linux: `~/.local/share/voicebox/providers/`@@ -448,133 +471,133 @@ class ProviderInstaller:```tsxexport function ProviderSettings() {- const [selectedProvider, setSelectedProvider] = useState<ProviderType>('auto');- const { data: installedProviders } = useQuery({- queryKey: ['providers', 'installed'],- queryFn: () => apiClient.getInstalledProviders()- });-- return (- <Card>- <CardHeader>- <CardTitle>TTS Provider</CardTitle>- <CardDescription>- Choose how Voicebox generates speech- </CardDescription>- </CardHeader>- <CardContent>- <RadioGroup value={selectedProvider} onValueChange={setSelectedProvider}>-- {/* Auto-detect */}- <div className="flex items-center space-x-2">- <RadioGroupItem value="auto" id="auto" />- <Label htmlFor="auto">- <div className="font-medium">Auto-detect (Recommended)</div>- <div className="text-sm text-muted-foreground">- Automatically choose the best available provider- </div>- </Label>- </div>-- {/* PyTorch CUDA */}- <div className="flex items-center justify-between">- <div className="flex items-center space-x-2">- <RadioGroupItem value="pytorch-cuda" id="cuda" disabled={!gpuAvailable} />- <Label htmlFor="cuda">- <div className="font-medium">PyTorch CUDA (NVIDIA GPU)</div>- <div className="text-sm text-muted-foreground">- 4-5x faster inference on NVIDIA GPUs- </div>- </Label>- </div>- {!installedProviders?.includes('pytorch-cuda') && gpuAvailable && (- <Button onClick={() => downloadProvider('pytorch-cuda')} size="sm">- Download (2.4GB)- </Button>- )}- </div>-- {/* PyTorch CPU */}- <div className="flex items-center justify-between">- <div className="flex items-center space-x-2">- <RadioGroupItem value="pytorch-cpu" id="cpu" />- <Label htmlFor="cpu">- <div className="font-medium">PyTorch CPU</div>- <div className="text-sm text-muted-foreground">- Works on any system, slower inference- </div>- </Label>- </div>- {!installedProviders?.includes('pytorch-cpu') && (- <Button onClick={() => downloadProvider('pytorch-cpu')} size="sm">- Download (300MB)- </Button>- )}- </div>-- {/* MLX (macOS only) */}- {isMacOS && (- <div className="flex items-center justify-between">- <div className="flex items-center space-x-2">- <RadioGroupItem value="mlx" id="mlx" />- <Label htmlFor="mlx">- <div className="font-medium">MLX (Apple Silicon)</div>- <div className="text-sm text-muted-foreground">- Optimized for M1/M2/M3 chips- </div>- </Label>- </div>- {!installedProviders?.includes('mlx') && (- <Button onClick={() => downloadProvider('mlx')} size="sm">- Download (800MB)- </Button>- )}- </div>- )}-- {/* Remote */}- <div className="space-y-2">- <div className="flex items-center space-x-2">- <RadioGroupItem value="remote" id="remote" />- <Label htmlFor="remote">- <div className="font-medium">Remote Server</div>- <div className="text-sm text-muted-foreground">- Connect to your own TTS server- </div>- </Label>- </div>- {selectedProvider === 'remote' && (- <Input- placeholder="http://your-server:8000"- className="ml-6"- />- )}- </div>-- {/* OpenAI */}- <div className="space-y-2">- <div className="flex items-center space-x-2">- <RadioGroupItem value="openai" id="openai" />- <Label htmlFor="openai">- <div className="font-medium">OpenAI API</div>- <div className="text-sm text-muted-foreground">- Use OpenAI's TTS API (requires API key)- </div>- </Label>- </div>- {selectedProvider === 'openai' && (- <Input- type="password"- placeholder="sk-..."- className="ml-6"- />- )}- </div>-- </RadioGroup>- </CardContent>- </Card>- );+ const [selectedProvider, setSelectedProvider] =+ useState<ProviderType>("auto");+ const {data: installedProviders} = useQuery({+ queryKey: ["providers", "installed"],+ queryFn: () => apiClient.getInstalledProviders(),+ });++ return (+ <Card>+ <CardHeader>+ <CardTitle>TTS Provider</CardTitle>+ <CardDescription>Choose how Voicebox generates speech</CardDescription>+ </CardHeader>+ <CardContent>+ <RadioGroup+ value={selectedProvider}+ onValueChange={setSelectedProvider}+ >+ {/* Auto-detect */}+ <div className="flex items-center space-x-2">+ <RadioGroupItem value="auto" id="auto" />+ <Label htmlFor="auto">+ <div className="font-medium">Auto-detect (Recommended)</div>+ <div className="text-sm text-muted-foreground">+ Automatically choose the best available provider+ </div>+ </Label>+ </div>++ {/* PyTorch CUDA */}+ <div className="flex items-center justify-between">+ <div className="flex items-center space-x-2">+ <RadioGroupItem+ value="pytorch-cuda"+ id="cuda"+ disabled={!gpuAvailable}+ />+ <Label htmlFor="cuda">+ <div className="font-medium">PyTorch CUDA (NVIDIA GPU)</div>+ <div className="text-sm text-muted-foreground">+ 4-5x faster inference on NVIDIA GPUs+ </div>+ </Label>+ </div>+ {!installedProviders?.includes("pytorch-cuda") && gpuAvailable && (+ <Button+ onClick={() => downloadProvider("pytorch-cuda")}+ size="sm"+ >+ Download (2.4GB)+ </Button>+ )}+ </div>++ {/* PyTorch CPU */}+ <div className="flex items-center justify-between">+ <div className="flex items-center space-x-2">+ <RadioGroupItem value="pytorch-cpu" id="cpu" />+ <Label htmlFor="cpu">+ <div className="font-medium">PyTorch CPU</div>+ <div className="text-sm text-muted-foreground">+ Works on any system, slower inference+ </div>+ </Label>+ </div>+ {!installedProviders?.includes("pytorch-cpu") && (+ <Button onClick={() => downloadProvider("pytorch-cpu")} size="sm">+ Download (300MB)+ </Button>+ )}+ </div>++ {/* MLX (macOS only) */}+ {isMacOS && (+ <div className="flex items-center justify-between">+ <div className="flex items-center space-x-2">+ <RadioGroupItem value="mlx" id="mlx" />+ <Label htmlFor="mlx">+ <div className="font-medium">MLX (Apple Silicon)</div>+ <div className="text-sm text-muted-foreground">+ Optimized for M1/M2/M3 chips+ </div>+ </Label>+ </div>+ {!installedProviders?.includes("mlx") && (+ <Button onClick={() => downloadProvider("mlx")} size="sm">+ Download (800MB)+ </Button>+ )}+ </div>+ )}++ {/* Remote */}+ <div className="space-y-2">+ <div className="flex items-center space-x-2">+ <RadioGroupItem value="remote" id="remote" />+ <Label htmlFor="remote">+ <div className="font-medium">Remote Server</div>+ <div className="text-sm text-muted-foreground">+ Connect to your own TTS server+ </div>+ </Label>+ </div>+ {selectedProvider === "remote" && (+ <Input placeholder="http://your-server:8000" className="ml-6" />+ )}+ </div>++ {/* OpenAI */}+ <div className="space-y-2">+ <div className="flex items-center space-x-2">+ <RadioGroupItem value="openai" id="openai" />+ <Label htmlFor="openai">+ <div className="font-medium">OpenAI API</div>+ <div className="text-sm text-muted-foreground">+ Use OpenAI's TTS API (requires API key)+ </div>+ </Label>+ </div>+ {selectedProvider === "openai" && (+ <Input type="password" placeholder="sk-..." className="ml-6" />+ )}+ </div>+ </RadioGroup>+ </CardContent>+ </Card>+ );}```@@ -718,11 +741,11 @@ Providers have their own version numbers, independent of the main app:**Example:**| App Version | Min Provider Version | Max Provider Version |-|-------------|---------------------|---------------------|-| v0.2.0 | v1.0.0 | v1.x.x |-| v0.3.0 | v1.0.0 | v1.x.x |-| v0.4.0 | v1.2.0 | v1.x.x |-| v1.0.0 | v2.0.0 | v2.x.x |+| ----------- | -------------------- | -------------------- |+| v0.2.0 | v1.0.0 | v1.x.x |+| v0.3.0 | v1.0.0 | v1.x.x |+| v0.4.0 | v1.2.0 | v1.x.x |+| v1.0.0 | v2.0.0 | v2.x.x |**Backend checks compatibility:**@@ -735,6 +758,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:```**UI shows warning if incompatible:**+```⚠️ Provider version 0.9.0 is outdated. Update to v1.0.0+```@@ -748,6 +772,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:1. User downloads and installs Voicebox (~150MB)2. App launches → detects no TTS provider installed3. Shows setup wizard:+```Choose your TTS provider:@@ -769,6 +794,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:[ ] OpenAI APIAPI Key: ________________```+4. User selects provider → downloads with progress bar5. Provider installs to AppData/Application Support6. App starts provider → ready to use@@ -818,16 +844,16 @@ async def check_provider_compatibility(provider_version: str) -> bool:## Benefits-| Benefit | Details |-|---------|---------|-| **GitHub Releases Work** | Main app ~150MB << 2GB limit |-| **Fast Updates** | UI/feature updates don't require re-downloading providers |-| **User Choice** | CPU, CUDA, MLX, OpenAI, remote server |-| **External Provider Support** | Users can run their own TTS servers |-| **Bandwidth Savings** | Only download provider once, app updates are small |-| **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) |-| **Team Deployments** | Multiple users share one remote provider |-| **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. |+| Benefit | Details |+| ----------------------------- | --------------------------------------------------------- |+| **GitHub Releases Work** | Main app ~150MB << 2GB limit |+| **Fast Updates** | UI/feature updates don't require re-downloading providers |+| **User Choice** | CPU, CUDA, MLX, OpenAI, remote server |+| **External Provider Support** | Users can run their own TTS servers |+| **Bandwidth Savings** | Only download provider once, app updates are small |+| **Future-Proof** | Easy to add new providers (ElevenLabs, custom models) |+| **Team Deployments** | Multiple users share one remote provider |+| **Cloud-Ready** | Works with Modal, Replicate, RunPod, etc. |---@@ -838,6 +864,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:**Question:** Should providers have independent versions or match app version?**Options:**+- A. Independent (providers: v1.x, app: v0.2.x)- B. Matched (both use v0.2.x)@@ -850,6 +877,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:**Question:** Should providers auto-update separately from app?**Options:**+- A. Manual updates only (user clicks "Update Provider")- B. Optional auto-update (user can enable)- C. Always auto-update@@ -863,6 +891,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:**Question:** How does app find installed providers?**Options:**+- A. Check standard paths in AppData/Application Support- B. Registry (Windows) / plist (macOS)- C. Config file with provider locations@@ -876,6 +905,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:**Question:** What if no provider is installed?**Options:**+- A. Show setup wizard on first launch- B. Block app until provider installed- C. Allow app to run in "demo mode" (transcription only)@@ -889,6 +919,7 @@ async def check_provider_compatibility(provider_version: str) -> bool:**Question:** Should provider start automatically with app?**Options:**+- A. Always start selected provider on app launch- B. Start on-demand (when user generates speech)- C. User preference@@ -928,5 +959,6 @@ If you want to build a custom TTS provider:4. Share in GitHub Discussions**Questions?**+- GitHub Issues: [voicebox/issues](https://github.com/jamiepine/voicebox/issues)- Discord: Coming soon