Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Connector Architecture Testing Checklist
This checklist covers manual testing scenarios for the transcription connector architecture. Use this alongside the automated tests (tests/test_connector_architecture.py) for comprehensive coverage.
Prerequisites
- Speakr running in Docker
- Access to at least one of:
- OpenAI API key (for OpenAI connectors)
- Self-hosted ASR service (for ASR endpoint connector)
- Test audio files:
- Short file (<1 minute, single speaker)
- Longer file (>5 minutes, multiple speakers)
- Opus format file (for codec conversion testing)
1. Connector Auto-Detection
Test 1.1: OpenAI Transcribe Auto-Detection
Config:
TRANSCRIPTION_API_KEY=sk-xxx
TRANSCRIPTION_MODEL=gpt-4o-transcribe-diarize
Expected:
- Logs show "Auto-detected OpenAI Transcribe"
-
/api/system/infoshowsconnector: "openai_transcribe" - Transcription works with speaker labels (A, B, C...)
Test 1.2: ASR Endpoint Auto-Detection
Config:
ASR_BASE_URL=http://whisperx-asr:9000
Expected:
- Logs show "Auto-detected ASR endpoint"
-
/api/system/infoshowsconnector: "asr_endpoint" - No deprecation warning (USE_ASR_ENDPOINT not set)
Test 1.3: Legacy USE_ASR_ENDPOINT Backwards Compatibility
Config:
USE_ASR_ENDPOINT=true
ASR_BASE_URL=http://whisperx-asr:9000
Expected:
- Logs show deprecation warning for USE_ASR_ENDPOINT
- Connector still works correctly
-
/api/system/infoshowsconnector: "asr_endpoint"
Test 1.4: Explicit Connector Selection
Config:
TRANSCRIPTION_CONNECTOR=openai_whisper
TRANSCRIPTION_API_KEY=sk-xxx
ASR_BASE_URL=http://whisperx-asr:9000 # Should be ignored
Expected:
- Uses openai_whisper despite ASR_BASE_URL being set
-
/api/system/infoshowsconnector: "openai_whisper"
2. Speaker Diarization
Test 2.1: OpenAI Transcribe Diarization
Config: Use TRANSCRIPTION_MODEL=gpt-4o-transcribe-diarize
Steps:
- Upload audio with 2+ speakers
- Wait for transcription to complete
Expected:
- Transcription shows speaker labels (A, B, C...)
- Speaker identification button appears in UI
- Bubble view toggle available
- Can rename speakers
Test 2.2: ASR Endpoint Diarization
Config: Use ASR endpoint with ASR_DIARIZE=true
Steps:
- Upload audio with 2+ speakers
- Wait for transcription to complete
Expected:
- Transcription shows speaker labels (SPEAKER_00, SPEAKER_01...)
- Speaker identification button appears in UI
- Min/max speakers options visible in upload form
- Min/max speakers options visible in reprocess modal
Test 2.3: OpenAI Whisper (No Diarization)
Config: Use TRANSCRIPTION_MODEL=whisper-1
Steps:
- Upload audio with 2+ speakers
- Wait for transcription to complete
Expected:
- Transcription is plain text (no speaker labels)
- Speaker identification button NOT visible
- Bubble view toggle NOT available
Test 2.4: Mistral Voxtral Diarization
Config:
TRANSCRIPTION_CONNECTOR=mistral
TRANSCRIPTION_API_KEY=your-key
TRANSCRIPTION_MODEL=voxtral-mini-latest
Steps:
- Upload audio with 2+ speakers
- Wait for transcription to complete
Expected:
- Transcription shows speaker labels (speaker_0, speaker_1...)
- Speaker identification button appears in UI
- Bubble view toggle available
- Can rename speakers
Test 2.5: VibeVoice ASR Diarization
Config:
TRANSCRIPTION_CONNECTOR=vibevoice
TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000
TRANSCRIPTION_MODEL=vibevoice
Steps:
- Upload audio with 2+ speakers
- Wait for transcription to complete
Expected:
- Transcription shows speaker labels (SPEAKER_00, SPEAKER_01...)
- Speaker identification button appears in UI
- Bubble view toggle available
- Non-speech segments ([Silence], [Human Sounds]) are attributed to the previous speaker
- Can rename speakers
3. UI Feature Visibility
Test 3.1: Speaker Count Controls (ASR Only)
ASR Config:
- Min speakers field visible in upload form
- Max speakers field visible in upload form
- Min/max speakers visible in reprocess modal
OpenAI Config:
- Min speakers field NOT visible
- Max speakers field NOT visible
Test 3.2: Speaker Identification Based on Data
Upload the same file with different connectors and verify:
- Speaker ID button shows when transcription HAS diarization
- Speaker ID button hidden when transcription has NO diarization
- Behavior is the same regardless of current connector config
4. Chunking Behavior
Test 4.1: ASR Endpoint (Internal Chunking)
Config:
ASR_BASE_URL=http://whisperx-asr:9000
ENABLE_CHUNKING=true
CHUNK_LIMIT=10MB
Steps:
- Upload file >10MB
Expected:
- File uploads without app-level chunking
- Logs show "Connector handles chunking internally"
- No chunk progress in UI
Test 4.2: OpenAI Whisper (App Chunking)
Config:
TRANSCRIPTION_MODEL=whisper-1
ENABLE_CHUNKING=true
CHUNK_LIMIT=10MB
Steps:
- Upload file >10MB
Expected:
- File is chunked by the app
- Logs show chunk processing
- Transcription merges correctly
Test 4.3: Mistral (Internal Chunking)
Config:
TRANSCRIPTION_CONNECTOR=mistral
TRANSCRIPTION_API_KEY=your-key
Steps:
- Upload file >10 minutes
Expected:
- File uploads without app-level chunking
- Mistral handles chunking internally
- Transcription completes successfully
Test 4.4: VibeVoice (App Chunking for Long Files)
Config:
TRANSCRIPTION_CONNECTOR=vibevoice
TRANSCRIPTION_BASE_URL=http://your-vllm-server:8000
Steps:
- Upload file >60 minutes
Expected:
- App splits into ~50 minute chunks
- Each chunk transcribes successfully
- Segments merge correctly across chunk boundaries
Test 4.5: Chunking Disabled
Config:
TRANSCRIPTION_MODEL=whisper-1
ENABLE_CHUNKING=false
Steps:
- Upload file >25MB
Expected:
- Upload fails with file size error (OpenAI 25MB limit)
- Error message is clear
5. Codec Handling
Test 5.1: Opus File with OpenAI Connector
Config: Any OpenAI connector
Steps:
- Upload an opus format audio file
- Check logs
Expected:
- File is converted before transcription
- Logs show codec conversion
- Transcription succeeds
Test 5.2: AUDIO_UNSUPPORTED_CODECS Override
Config:
AUDIO_UNSUPPORTED_CODECS=aac
Steps:
- Upload an AAC format audio file
- Check logs
Expected:
- File is converted before transcription
- Logs show "Excluding codecs from supported list"
Test 5.3: Supported Format (No Conversion)
Steps:
- Upload an MP3 file
Expected:
- No conversion occurs
- Transcription proceeds directly
6. Error Handling
Test 6.1: Invalid API Key
Config:
TRANSCRIPTION_API_KEY=invalid-key
Expected:
- Clear error message about authentication
- Job fails gracefully
- UI shows error state
Test 6.2: Unreachable ASR Endpoint
Config:
ASR_BASE_URL=http://nonexistent:9000
Expected:
- Connection error logged
- Job fails with timeout/connection error
- UI shows error state
Test 6.3: Invalid Connector Name
Config:
TRANSCRIPTION_CONNECTOR=nonexistent
Expected:
- App fails to start (or logs critical error)
- Error message lists available connectors
7. Voice Profiles (ASR Only)
Test 7.1: Speaker Embeddings
Config:
ASR_BASE_URL=http://whisperx-asr:9000
ASR_RETURN_SPEAKER_EMBEDDINGS=true
Steps:
- Upload audio with known speaker
- Create voice profile from transcription
- Upload another audio with same speaker
Expected:
- Speaker embeddings returned in response
- Voice profile creation works
- Automatic speaker matching works
Test 7.2: Embeddings Disabled
Config:
ASR_RETURN_SPEAKER_EMBEDDINGS=false
Expected:
- Voice profile features not available
- No embeddings in response
8. System Info API
Test 8.1: Connector Capabilities Exposed
Endpoint: GET /api/system/info
Expected Response Structure:
{
"transcription": {
"connector": "asr_endpoint",
"supports_diarization": true,
"supports_speaker_embeddings": true,
"supports_speaker_count": true
}
}
- connector field matches active connector
- supports_diarization accurate
- supports_speaker_embeddings accurate
- supports_speaker_count accurate
9. Reprocessing
Test 9.1: Reprocess with Different Language
Steps:
- Transcribe file (auto-detect language)
- Reprocess with specific language
Expected:
- Reprocess uses selected language
- Transcription updates correctly
Test 9.2: Reprocess with Speaker Count (ASR)
Steps:
- Transcribe file with auto speaker detection
- Reprocess with min_speakers=2, max_speakers=3
Expected:
- Min/max speakers respected
- Diarization quality may improve
10. Hotwords & Initial Prompt
Test 10.1: ASR Endpoint with Hotwords
Config: ASR endpoint connector
Steps:
- Upload audio containing domain-specific terms
- Set hotwords in Advanced ASR Options (e.g., "Speakr, CTranslate2, PyAnnote")
Expected:
- Hotwords passed as
hotwordsquery parameter to ASR - Transcription accuracy improves for specified terms
- Server logs show hotwords in request
Test 10.2: ASR Endpoint with Initial Prompt
Config: ASR endpoint connector
Steps:
- Upload audio file
- Set initial prompt in Advanced ASR Options
Expected:
- Initial prompt passed as
initial_promptquery parameter to ASR - Server logs show initial_prompt in request
Test 10.3: OpenAI Connector with Hotwords + Prompt
Config: OpenAI Whisper or OpenAI Transcribe connector
Steps:
- Upload audio with both hotwords and initial prompt set
Expected:
- Hotwords and initial prompt combined into single
promptparameter - Combined with
.separator
Test 10.4: Precedence Hierarchy
Steps:
- Set user-level hotwords in Account Settings
- Create a tag with different hotwords
- Upload with tag applied but no form-level hotwords
Expected:
- Tag hotwords used (overrides user defaults)
- Upload form values override tag when explicitly set
11. Performance & Edge Cases
Test 11.1: Very Long Audio
Upload audio >1 hour
Expected:
- Processing completes (may take time)
- No timeout errors
- Transcription is complete
Test 11.2: Empty/Silent Audio
Upload silent audio file
Expected:
- Transcription returns empty or minimal text
- No errors
Test 11.3: Corrupt Audio File
Upload corrupt/invalid audio file
Expected:
- Clear error message
- Job fails gracefully
Notes
- Run automated tests first:
docker exec speakr-dev python /app/tests/test_connector_architecture.py - After significant changes, test with all three connectors
- Voice profile tests only work with WhisperX ASR Service