Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
Add documentation for Mistral and VibeVoice transcription connectors
Murtaza Nasirfb3045afd130
diff --git a/README.md b/README.mdindex c609e08c..ae76522b 100644--- a/README.md+++ b/README.md@@ -133,6 +133,8 @@ Speakr uses a **connector-based architecture** that auto-detects your transcript|--------|-------|---------------------|----------------|| **OpenAI Transcribe** | Just API key | ✅ `gpt-4o-transcribe-diarize` | ❌ || **WhisperX ASR** | GPU container | ✅ Best quality | ✅ |+| **Mistral Voxtral** | Just API key | ✅ Built-in | ❌ |+| **VibeVoice ASR** | Self-hosted (vLLM) | ✅ Built-in | ❌ || **Legacy Whisper** | Just API key | ❌ | ❌ |**Simplest setup (OpenAI with diarization):**@@ -148,6 +150,21 @@ ASR_RETURN_SPEAKER_EMBEDDINGS=true # Enable voice profiles```Requires [WhisperX ASR Service](https://github.com/murtaza-nasir/whisperx-asr-service) container with GPU.+**Mistral Voxtral (cloud diarization):**+```bash+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-mistral-key+TRANSCRIPTION_MODEL=voxtral-mini-latest+```++**VibeVoice ASR (self-hosted, no cloud dependency):**+```bash+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+TRANSCRIPTION_MODEL=vibevoice+```+Requires [VibeVoice](https://huggingface.co/microsoft/VibeVoice-ASR) served via vLLM with GPU.+> **⚠️ PyTorch 2.6 Users:** If you encounter a "Weights only load failed" error with WhisperX, add `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=true` to your ASR container. See [troubleshooting](https://murtaza-nasir.github.io/speakr/troubleshooting#pytorch-26-weights-loading-error-whisperx-asr-service) for details.**[View Full Installation Guide →](https://murtaza-nasir.github.io/speakr/getting-started/installation)**diff --git a/config/env.transcription.example b/config/env.transcription.exampleindex abb58581..5d91e020 100644--- a/config/env.transcription.example+++ b/config/env.transcription.example@@ -50,7 +50,7 @@ GPT5_VERBOSITY=medium# =============================================================================# CONNECTOR SELECTION (Auto-detected if not set)# =============================================================================-# Options: openai_whisper, openai_transcribe, asr_endpoint+# Options: openai_whisper, openai_transcribe, asr_endpoint, mistral, vibevoice# Leave empty to auto-detect based on other settings# TRANSCRIPTION_CONNECTOR=@@ -110,18 +110,51 @@ TRANSCRIPTION_MODEL=gpt-4o-transcribe-diarize# Enables automatic speaker matching across recordings# ASR_RETURN_SPEAKER_EMBEDDINGS=false+# =============================================================================+# MISTRAL / VOXTRAL CONFIGURATION+# =============================================================================+# Mistral's Voxtral model provides cloud-based transcription with built-in+# speaker diarization and language detection. Requires a Mistral API key.+#+# TRANSCRIPTION_CONNECTOR=mistral+# TRANSCRIPTION_API_KEY=your_mistral_api_key+# TRANSCRIPTION_MODEL=voxtral-mini-latest+#+# To use a custom Mistral-compatible endpoint:+# TRANSCRIPTION_BASE_URL=https://api.mistral.ai++# =============================================================================+# VIBEVOICE CONFIGURATION (Self-hosted via vLLM)+# =============================================================================+# Microsoft's VibeVoice ASR model provides transcription with speaker+# diarization, timestamps, and language detection for 50+ languages. It runs+# on your own hardware via vLLM and handles up to 60 minutes per request.+# Longer files are automatically chunked by the app.+#+# Requirements:+# - vLLM server with the VibeVoice model loaded+# - GPU(s) with enough VRAM (bf16 needs ~18GB, fits on 2x consumer GPUs)+#+# TRANSCRIPTION_CONNECTOR=vibevoice+# TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+# TRANSCRIPTION_MODEL=vibevoice+#+# If your vLLM server requires authentication:+# TRANSCRIPTION_API_KEY=your_api_key+# =============================================================================# CHUNKING CONFIGURATION (For large files)# =============================================================================# Chunking is now connector-aware with this priority:-# 1. Connector handles internally (openai_transcribe, asr_endpoint) → No app chunking+# 1. Connector handles internally (openai_transcribe, asr_endpoint, mistral) → No app chunking# 2. ENABLE_CHUNKING=false → Disable chunking (only affects openai_whisper)# 3. CHUNK_LIMIT set → Use your settings# 4. Connector defaults → Use connector's recommended limits# 5. App default → 20MB size-based#-# For openai_transcribe/asr_endpoint: These settings are IGNORED (connector handles it)+# For openai_transcribe/asr_endpoint/mistral: These settings are IGNORED (connector handles it)# For openai_whisper: These settings control chunking behavior+# For vibevoice: App chunks files >58 min into ~50 min pieces automatically# ENABLE_CHUNKING=false # Uncomment to disable chunking for openai_whisper@@ -155,6 +188,16 @@ CHUNK_OVERLAP_SECONDS=3# TRANSCRIPTION_API_KEY=not-needed# TRANSCRIPTION_BASE_URL=http://localhost:8080/v1# TRANSCRIPTION_MODEL=Systran/faster-distil-whisper-large-v3+#+# --- Mistral Voxtral (cloud diarization) ---+# TRANSCRIPTION_CONNECTOR=mistral+# TRANSCRIPTION_API_KEY=your-mistral-key+# TRANSCRIPTION_MODEL=voxtral-mini-latest+#+# --- VibeVoice ASR (self-hosted diarization) ---+# TRANSCRIPTION_CONNECTOR=vibevoice+# TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+# TRANSCRIPTION_MODEL=vibevoice# =============================================================================# APPLICATION SETTINGS@@ -277,9 +320,12 @@ SQLALCHEMY_DATABASE_URI=sqlite:////data/instance/transcriptions.dbUPLOAD_FOLDER=/data/uploads# =============================================================================-# FUTURE: Additional Provider Notes+# ADDITIONAL PROVIDER NOTES# =============================================================================# The connector architecture is designed to support additional providers.+# Currently available: openai_whisper, openai_transcribe, asr_endpoint,+# azure_openai_transcribe, mistral, vibevoice+## Future connectors may include:## - Deepgram: Known for excellent diarization and real-time transcriptiondiff --git a/docs/admin-guide/migration-guide.md b/docs/admin-guide/migration-guide.mdindex 6e9ec130..50af4c6a 100644--- a/docs/admin-guide/migration-guide.md+++ b/docs/admin-guide/migration-guide.md@@ -89,6 +89,28 @@ TRANSCRIPTION_MODEL=gpt-4o-transcribe-diarizeThis uses OpenAI's built-in diarization. The connector is auto-detected from the model name.+### Using Mistral Voxtral++Mistral's Voxtral provides cloud-based transcription with diarization:++```bash+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-mistral-key+TRANSCRIPTION_MODEL=voxtral-mini-latest+```++### Using VibeVoice ASR (Self-Hosted)++VibeVoice runs on your own hardware via vLLM, with no cloud dependency:++```bash+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+TRANSCRIPTION_MODEL=vibevoice+```++Both connectors support speaker diarization, timestamps, and automatic language detection.+## Chunking Behavior ChangesThe new architecture makes chunking **connector-aware**:@@ -97,6 +119,8 @@ The new architecture makes chunking **connector-aware**:|-----------|-------------------|| **ASR Endpoint** | Handled internally—your `CHUNK_*` settings are ignored || **OpenAI Transcribe** | Handled internally via `chunking_strategy=auto`—your settings are ignored |+| **Mistral** | Handled internally—your `CHUNK_*` settings are ignored |+| **VibeVoice** | App chunks files over ~58 minutes into ~50 minute pieces automatically || **OpenAI Whisper** | Uses your `CHUNK_LIMIT` and `CHUNK_OVERLAP_SECONDS` settings |If you were manually configuring chunking for ASR endpoints, you can remove those settings as they no longer have any effect.@@ -137,6 +161,34 @@ After updating your configuration:## Recommended Configuration+### For Mistral Voxtral (Cloud Diarization)++```bash+# Transcription+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-mistral-key+TRANSCRIPTION_MODEL=voxtral-mini-latest++# Text generation+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1+TEXT_MODEL_API_KEY=sk-or-v1-xxx+TEXT_MODEL_NAME=openai/gpt-4o-mini+```++### For VibeVoice ASR (Self-Hosted, No Cloud)++```bash+# Transcription+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+TRANSCRIPTION_MODEL=vibevoice++# Text generation+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1+TEXT_MODEL_API_KEY=sk-or-v1-xxx+TEXT_MODEL_NAME=openai/gpt-4o-mini+```+### For Self-Hosted (Best Quality)Using WhisperX ASR Service for superior transcription and diarization:diff --git a/docs/advanced/custom-connectors.md b/docs/advanced/custom-connectors.mdindex fa5153fd..9ec5145c 100644--- a/docs/advanced/custom-connectors.md+++ b/docs/advanced/custom-connectors.md@@ -545,3 +545,5 @@ Popular providers we'd love to see connectors for:- Google Cloud Speech-to-Text- Amazon Transcribe- ~~Azure Speech Services~~ → **Azure OpenAI connector now available (experimental, v0.8.6+)**+- ~~Mistral Voxtral~~ → **Mistral connector now available**+- ~~VibeVoice ASR~~ → **VibeVoice connector now available (self-hosted via vLLM)**diff --git a/docs/faq.md b/docs/faq.mdindex 36cb89ec..f917cf54 100644--- a/docs/faq.md+++ b/docs/faq.md@@ -91,6 +91,8 @@ Whisper API provides basic transcription - converting speech to text without speOpenAI's `gpt-4o-transcribe-diarize` model also offers cloud-based diarization. For longer files (over ~23 minutes), speaker tracking supports up to 4 speakers - recordings with more speakers may have inconsistent labels across sections.+Mistral's Voxtral model is another cloud option with built-in diarization and language detection. If you prefer self-hosted, VibeVoice ASR runs on your own GPU(s) via vLLM and provides diarization with timestamps for 50+ languages.+**Note on ASR engines**: For speaker diarization to work properly with the ASR webservice, you must use `ASR_ENGINE=whisperx`, not `faster_whisper`. While faster_whisper provides transcription, it doesn't support speaker identification.### Can Speakr transcribe languages other than English?diff --git a/docs/features.md b/docs/features.mdindex 6b7475b7..39b47944 100644--- a/docs/features.md+++ b/docs/features.md@@ -6,7 +6,7 @@ Speakr combines powerful transcription capabilities with intelligent AI features### Multi-Engine Support-Speakr supports multiple transcription engines to match your needs and budget. Use [OpenAI's Whisper API](getting-started.md#option-a-openai-whisper-configuration) for quick, cloud-based transcription with excellent accuracy. Deploy the [recommended ASR container](getting-started.md#option-b-custom-asr-endpoint-configuration) for advanced features like speaker diarization and local processing. See the [installation guide](getting-started/installation.md) for detailed setup instructions. The system automatically handles different audio formats, converting them as needed for optimal transcription quality.+Speakr supports multiple transcription engines to match your needs and budget. Use [OpenAI's Whisper API](getting-started.md#option-a-openai-whisper-configuration) for quick, cloud-based transcription with excellent accuracy. Deploy the [recommended ASR container](getting-started.md#option-b-custom-asr-endpoint-configuration) for advanced features like speaker diarization and local processing. Use [Mistral Voxtral](getting-started/installation.md#for-mistral-voxtral-cloud-based-diarization) for cloud-based diarization via the Mistral API, or run [VibeVoice ASR](getting-started/installation.md#for-vibevoice-asr-self-hosted-via-vllm) on your own hardware for fully self-hosted transcription with diarization. See the [installation guide](getting-started/installation.md) for detailed setup instructions. The system automatically handles different audio formats, converting them as needed for optimal transcription quality.### Speaker Diarization@@ -14,6 +14,10 @@ When using the [ASR endpoint](getting-started.md#option-b-custom-asr-endpoint-co**Cloud Diarization with OpenAI**: When using OpenAI's `gpt-4o-transcribe-diarize` model, speaker diarization is also available with speakers labeled as A, B, C, etc. For longer files (over ~23 minutes) that require chunking, the system maintains speaker identity across chunks using audio reference samples. This technique supports **up to 4 speakers** - recordings with more speakers may have inconsistent labels across different sections of the transcript.+**Cloud Diarization with Mistral**: Mistral's Voxtral model provides built-in speaker diarization and automatic language detection. It handles chunking internally, so long files work without extra configuration.++**Self-Hosted Diarization with VibeVoice**: Microsoft's VibeVoice ASR model runs on your own hardware via vLLM and provides speaker diarization with timestamps and language detection for 50+ languages. It's a good option if you want the quality of a dedicated ASR model without any cloud dependency.+### Interactive Audio SynchronizationSpeakr provides seamless bidirectional synchronization between audio playback and transcript text for recordings with speaker diarization. Click any part of the transcript to jump directly to that moment in the audio, making it easy to review specific sections. As the audio plays, the system automatically highlights the currently spoken text in real-time with smooth visual feedback. Enable auto-scroll follow mode to keep the active segment centered in view, allowing you to follow along effortlessly during playback.diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.mdindex 04479ddc..ba35c78d 100644--- a/docs/getting-started/installation.md+++ b/docs/getting-started/installation.md@@ -6,11 +6,13 @@ This comprehensive guide covers deploying Speakr for production use, including dBefore diving into installation, it's helpful to understand how Speakr works. The application integrates with external APIs for two main purposes: transcription services that convert your audio to text, and text generation services that power features like summaries, titles, and interactive chat. Speakr is designed to be flexible, supporting both cloud-based services like OpenAI and self-hosted solutions running on your own infrastructure.-Speakr uses a **connector-based architecture** for transcription services, providing a unified interface regardless of which provider you choose. Three connectors are currently available:+Speakr uses a **connector-based architecture** for transcription services, providing a unified interface regardless of which provider you choose. The following connectors are currently available:1. **ASR Endpoint** (Recommended for best quality) - For self-hosted solutions like WhisperX that offer GPU-accelerated transcription with superior accuracy, voice profiles, and speaker diarization2. **OpenAI Transcribe** - Uses `gpt-4o-transcribe-diarize` for cloud-based speaker diarization without requiring additional containers-3. **OpenAI Whisper** - Legacy whisper-1 model for basic transcription using any legacy OpenAI compatible whisper API+3. **Mistral Voxtral** - Cloud-based transcription with built-in diarization and language detection via the Mistral API+4. **VibeVoice ASR** - Microsoft's self-hosted multimodal ASR model with diarization, timestamps, and 50+ language support, served via vLLM+5. **OpenAI Whisper** - Legacy whisper-1 model for basic transcription using any legacy OpenAI compatible whisper APIThe connector can be specified or inferred based on your configuration. For text generation, Speakr uses the OpenAI Chat Completions API format with the `/chat/completions` endpoint, which is widely supported across different AI providers.@@ -196,6 +198,32 @@ ASR_RETURN_SPEAKER_EMBEDDINGS=trueThis setting is disabled by default because it's only supported by WhisperX. If you're using the basic OpenAI Whisper ASR Webservice, leave this setting disabled or omit it entirely to avoid errors.+#### For Mistral Voxtral (Cloud-Based Diarization)++Mistral's Voxtral model provides cloud-based transcription with built-in speaker diarization and automatic language detection. It requires a Mistral API key and no additional infrastructure. Voxtral handles chunking internally, so large files work without any extra configuration.++```bash+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-mistral-api-key+TRANSCRIPTION_MODEL=voxtral-mini-latest+```++Configure the text generation model as described above for summaries and chat features.++#### For VibeVoice ASR (Self-Hosted via vLLM)++Microsoft's VibeVoice ASR model provides transcription with speaker diarization, timestamps, and language detection for 50+ languages. It runs entirely on your own hardware via vLLM, so there's no cloud dependency. The model handles up to 60 minutes of audio per request, and longer files are automatically chunked by the app.++You'll need a vLLM server with the VibeVoice model loaded. The bf16 model requires roughly 18GB of VRAM, which fits on two consumer GPUs (like 2x RTX 3090) using tensor parallelism. See the [VibeVoice model page](https://huggingface.co/microsoft/VibeVoice-ASR) for vLLM setup instructions.++```bash+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+TRANSCRIPTION_MODEL=vibevoice+```++If your vLLM server requires authentication, also set `TRANSCRIPTION_API_KEY`. Configure the text generation model as described above for summaries and chat features.+### Step 4: Configure System SettingsOne of Speakr's conveniences is automatic admin account creation. Instead of going through a registration process, you define the admin credentials in your environment file, and Speakr creates the account automatically on first startup. This ensures you can log in immediately without any additional setup steps:diff --git a/docs/testing/connector-testing-checklist.md b/docs/testing/connector-testing-checklist.mdindex 406959ad..394a4074 100644--- a/docs/testing/connector-testing-checklist.md+++ b/docs/testing/connector-testing-checklist.md@@ -116,6 +116,47 @@ ASR_BASE_URL=http://whisperx-asr:9000 # Should be ignored- [ ] Speaker identification button NOT visible- [ ] Bubble view toggle NOT available+### Test 2.4: Mistral Voxtral Diarization+**Config:**+```bash+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-key+TRANSCRIPTION_MODEL=voxtral-mini-latest+```++**Steps:**++1. Upload audio with 2+ speakers+2. Wait for transcription to complete++**Expected:**++- [ ] Transcription shows speaker labels (speaker_0, speaker_1...)+- [ ] Speaker identification button appears in UI+- [ ] Bubble view toggle available+- [ ] Can rename speakers++### Test 2.5: VibeVoice ASR Diarization+**Config:**+```bash+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+TRANSCRIPTION_MODEL=vibevoice+```++**Steps:**++1. Upload audio with 2+ speakers+2. Wait for transcription to complete++**Expected:**++- [ ] Transcription shows speaker labels (SPEAKER_00, SPEAKER_01...)+- [ ] Speaker identification button appears in UI+- [ ] Bubble view toggle available+- [ ] Non-speech segments ([Silence], [Human Sounds]) are attributed to the previous speaker+- [ ] Can rename speakers+---## 3. UI Feature Visibility@@ -179,7 +220,41 @@ CHUNK_LIMIT=10MB- [ ] Logs show chunk processing- [ ] Transcription merges correctly-### Test 4.3: Chunking Disabled+### Test 4.3: Mistral (Internal Chunking)+**Config:**+```bash+TRANSCRIPTION_CONNECTOR=mistral+TRANSCRIPTION_API_KEY=your-key+```++**Steps:**++1. Upload file >10 minutes++**Expected:**++- [ ] File uploads without app-level chunking+- [ ] Mistral handles chunking internally+- [ ] Transcription completes successfully++### Test 4.4: VibeVoice (App Chunking for Long Files)+**Config:**+```bash+TRANSCRIPTION_CONNECTOR=vibevoice+TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000+```++**Steps:**++1. Upload file >60 minutes++**Expected:**++- [ ] App splits into ~50 minute chunks+- [ ] Each chunk transcribes successfully+- [ ] Segments merge correctly across chunk boundaries++### Test 4.5: Chunking Disabled**Config:**```bashTRANSCRIPTION_MODEL=whisper-1