Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
feat: request and store per-speaker embeddings from OpenASR so voice profiles work through that connector (#380)
Murtaza Nasir88e65e7b6247
diff --git a/config/env.transcription.example b/config/env.transcription.exampleindex de445623..90daa186 100644--- a/config/env.transcription.example+++ b/config/env.transcription.example@@ -131,7 +131,7 @@ TRANSCRIPTION_MODEL=gpt-4o-transcribe-diarize# ASR_MIN_SPEAKERS=1# ASR_MAX_SPEAKERS=5-# Return speaker embeddings for speaker identification (WhisperX only)+# Return speaker embeddings for speaker identification (WhisperX and OpenASR)# Enables automatic speaker matching across recordings# ASR_RETURN_SPEAKER_EMBEDDINGS=false@@ -655,6 +655,11 @@ FILE_STORAGE_KEY_PREFIX=recordings# ASR_DIARIZE=true# ASR_TIMEOUT=1800#+# Voice profiles: OpenASR servers from openasr#379 onward can return one+# embedding per speaker alongside diarization. Same flag as WhisperX; leave+# it off on older servers, which reject the parameter.+# ASR_RETURN_SPEAKER_EMBEDDINGS=true+## Per-user hotwords and initial prompts are combined into the standard# OpenAI-compatible 'prompt' parameter. The server handles long files itself,# so the app does not chunk for it.diff --git a/docs/admin-guide/whisperx-setup.md b/docs/admin-guide/whisperx-setup.mdindex 5ff85e36..3b5d118d 100644--- a/docs/admin-guide/whisperx-setup.md+++ b/docs/admin-guide/whisperx-setup.md@@ -161,7 +161,7 @@ ASR_BASE_URL=http://whisperx-asr-api:9000ASR_RETURN_SPEAKER_EMBEDDINGS=true```-> **Important:** The `ASR_RETURN_SPEAKER_EMBEDDINGS=true` setting is required to enable voice profile features. This setting is only supported by WhisperX and should not be enabled when using the basic OpenAI Whisper ASR Webservice.+> **Important:** The `ASR_RETURN_SPEAKER_EMBEDDINGS=true` setting is required to enable voice profile features. This setting is supported by WhisperX and by OpenASR servers from openasr#379 onward, and should not be enabled when using the basic OpenAI Whisper ASR Webservice.Restart Speakr:diff --git a/docs/features.md b/docs/features.mdindex 5cc564cf..3629034b 100644--- a/docs/features.md+++ b/docs/features.md@@ -36,6 +36,8 @@ When using the [ASR endpoint](getting-started.md#option-b-custom-asr-endpoint-coYou can guide diarization with a speaker count. The upload, recording, and reprocess dialogs offer a Range / Exact toggle (remembered per user): Range sends a minimum and maximum, while Exact sends one number, and providers that only accept an exact count (such as OpenASR) show the single field automatically. Tag and folder defaults work with either form.+**Self-Hosted Diarization with OpenASR**: OpenASR runs locally with an OpenAI-compatible API, diarizes with an exact speaker count, and, on servers from openasr#379 onward, returns speaker embeddings when `ASR_RETURN_SPEAKER_EMBEDDINGS=true`, so voice profiles work with it the same way they do with WhisperX.+**Cloud Diarization with OpenAI**: When using OpenAI's `gpt-4o-transcribe-diarize` model, speaker diarization is also available with speakers labeled as A, B, C, etc. For longer files (over ~23 minutes) that require chunking, the system maintains speaker identity across chunks using audio reference samples. This technique supports **up to 4 speakers** - recordings with more speakers may have inconsistent labels across different sections of the transcript.**Cloud Diarization with Mistral**: Mistral's Voxtral model provides built-in speaker diarization and automatic language detection. It handles chunking internally, so long files work without extra configuration.diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.mdindex db6d00d8..992797e8 100644--- a/docs/getting-started/installation.md+++ b/docs/getting-started/installation.md@@ -142,7 +142,7 @@ For the best transcription and diarization quality, self-hosting an ASR service- Repository: [murtaza-nasir/whisperx-asr-service](https://github.com/murtaza-nasir/whisperx-asr-service)- Uses `pyannote/speaker-diarization-community-1` model with exclusive diarization- **Superior transcription and diarization quality** with large models (large-v3, distil-large-v3)- - Supports 256-dimensional speaker embeddings for voice profile identification+ - Returns speaker embeddings for voice profile identification- Better timestamp alignment between speakers and words- **Required for:** Voice profiles, automatic speaker recognition, speaker embeddings- **Environment file:** `config/env.whisperx.example`@@ -157,6 +157,12 @@ For the best transcription and diarization quality, self-hosting an ASR service- **Environment file:** `config/env.asr.example`- **Note:** Do not set `ASR_RETURN_SPEAKER_EMBEDDINGS=true` with this service as it will cause errors+3. **OpenASR** - Local server with diarization and, from openasr#379 onward, speaker embeddings+ - Website: [openasr.org](https://openasr.org)+ - **Supports:** Speaker diarization with an exact speaker count, voice profiles when embeddings are enabled+ - **Environment file:** `config/env.transcription.example` (OpenASR section)+ - **Setting:** `ASR_RETURN_SPEAKER_EMBEDDINGS=true` to enable voice profile features on a server that supports it+> **Important:** Before proceeding with this configuration, you'll need to set up one of the ASR service containers. See [Running ASR Service for Speaker Diarization](#running-asr-service-for-speaker-diarization) for complete instructions on deploying both containers together or separately.Download the appropriate ASR configuration template:diff --git a/src/services/transcription/connectors/openasr.py b/src/services/transcription/connectors/openasr.pyindex 7398ab6e..5b94ff9d 100644--- a/src/services/transcription/connectors/openasr.py+++ b/src/services/transcription/connectors/openasr.py@@ -53,8 +53,16 @@ class OpenASRTranscriptionConnector(BaseTranscriptionConnector):self.api_key = config.get('api_key', '')self.model = config.get('model', '')self.default_diarize = config.get('diarize', False)+ # Per-speaker voice embeddings (#380). OpenASR mirrors the WhisperX+ # contract: `return_speaker_embeddings=true` on the request, a+ # `speaker_embeddings` map keyed by speaker label in the verbose_json+ # response. Off by default so servers older than openasr#379, which+ # reject unknown fields, keep working.+ self.return_embeddings = bool(config.get('return_speaker_embeddings', False))self._config_timeout = config.get('timeout', 1800)super().__init__(config)+ if self.return_embeddings:+ self.CAPABILITIES = self.CAPABILITIES | {TranscriptionCapability.SPEAKER_EMBEDDINGS}def _validate_config(self) -> None:if not self.config.get('base_url'):@@ -85,6 +93,8 @@ class OpenASRTranscriptionConnector(BaseTranscriptionConnector):exact_speakers = request.max_speakers or request.min_speakersif exact_speakers:files['speakers'] = (None, str(int(exact_speakers)))+ if self.return_embeddings:+ files['return_speaker_embeddings'] = (None, 'true')if request.language:files['language'] = (None, request.language)@@ -162,10 +172,17 @@ class OpenASRTranscriptionConnector(BaseTranscriptionConnector):head, _, tail = name.rpartition('_')return (head, int(tail)) if tail.isdigit() else (name, -1)+ # Only meaningful alongside diarization; a server that did not+ # diarize has no speaker labels for the map to key on.+ speaker_embeddings = data.get('speaker_embeddings') if diarize else None+ if speaker_embeddings:+ logger.info(f"Received speaker embeddings for speakers: {sorted(speaker_embeddings)}")+return TranscriptionResponse(text=full_text,segments=segments or None,speakers=sorted(speakers, key=_speaker_key) if speakers else None,+ speaker_embeddings=speaker_embeddings or None,language=data.get('language'),provider=self.PROVIDER_NAME,model=model,diff --git a/src/services/transcription/registry.py b/src/services/transcription/registry.pyindex 5911c1ea..c423f4a2 100644--- a/src/services/transcription/registry.py+++ b/src/services/transcription/registry.py@@ -384,6 +384,9 @@ class ConnectorRegistry:'model': model,'diarize': diarize,'timeout': timeout,+ # Same flag as the WhisperX connector: OpenASR >= openasr#379+ # returns per-speaker embeddings on request (#380).+ 'return_speaker_embeddings': (os.environ.get('ASR_RETURN_SPEAKER_EMBEDDINGS') or 'false').lower() == 'true',}else: # openai_whisper (default)diff --git a/tests/test_openasr_connector.py b/tests/test_openasr_connector.pyindex 1bb74079..adc3c3af 100644--- a/tests/test_openasr_connector.py+++ b/tests/test_openasr_connector.py@@ -220,3 +220,84 @@ def test_no_speakers_param_without_diarize():_connector().transcribe(_request(min_speakers=3, max_speakers=3))_, kwargs = client.calls[0]assert 'speakers' not in kwargs['files']+++# --- Speaker embeddings (#380) ---------------------------------------------+#+# OpenASR (openasr#379) mirrors the WhisperX contract: request with+# return_speaker_embeddings=true, receive a speaker_embeddings map keyed by+# speaker label. Off by default because older servers reject unknown fields.++EMBEDDINGS = {'SPEAKER_2': [0.1] * 192, 'SPEAKER_10': [0.2] * 192}+++def test_embeddings_are_not_requested_by_default():+ connector = _connector(diarize=True)+ client = _Client(response=_Response(payload=VERBOSE_JSON))+ with patch('httpx.Client', return_value=client):+ response = connector.transcribe(_request(diarize=True))+ _, kwargs = client.calls[0]+ assert 'return_speaker_embeddings' not in kwargs['files']+ assert response.speaker_embeddings is None+ assert TranscriptionCapability.SPEAKER_EMBEDDINGS not in connector.CAPABILITIES+++def test_enabling_embeddings_declares_the_capability():+ assert TranscriptionCapability.SPEAKER_EMBEDDINGS in _connector(return_speaker_embeddings=True).CAPABILITIES+ # The class attribute must not be mutated by one enabled instance.+ assert TranscriptionCapability.SPEAKER_EMBEDDINGS not in OpenASRTranscriptionConnector.CAPABILITIES+++def test_embeddings_requested_and_parsed_when_enabled_with_diarization():+ connector = _connector(diarize=True, return_speaker_embeddings=True)+ payload = dict(VERBOSE_JSON, speaker_embeddings=EMBEDDINGS)+ client = _Client(response=_Response(payload=payload))+ with patch('httpx.Client', return_value=client):+ response = connector.transcribe(_request(diarize=True))+ _, kwargs = client.calls[0]+ assert kwargs['files']['return_speaker_embeddings'] == (None, 'true')+ assert response.speaker_embeddings == EMBEDDINGS+ assert len(response.speaker_embeddings['SPEAKER_2']) == 192+ # The rest of the response is unaffected.+ assert response.speakers == ['SPEAKER_2', 'SPEAKER_10']+++def test_embeddings_not_requested_without_diarization():+ connector = _connector(diarize=False, return_speaker_embeddings=True)+ payload = dict(VERBOSE_JSON, speaker_embeddings=EMBEDDINGS)+ client = _Client(response=_Response(payload=payload))+ with patch('httpx.Client', return_value=client):+ response = connector.transcribe(_request(diarize=False))+ _, kwargs = client.calls[0]+ assert 'return_speaker_embeddings' not in kwargs['files']+ # A map without speaker labels to key on is dropped rather than stored.+ assert response.speaker_embeddings is None+++def test_server_without_embedding_support_still_transcribes():+ connector = _connector(diarize=True, return_speaker_embeddings=True)+ client = _Client(response=_Response(payload=VERBOSE_JSON))+ with patch('httpx.Client', return_value=client):+ response = connector.transcribe(_request(diarize=True))+ assert response.speaker_embeddings is None+ assert response.text == 'hello world'+++def test_empty_embedding_map_is_stored_as_none():+ connector = _connector(diarize=True, return_speaker_embeddings=True)+ client = _Client(response=_Response(payload=dict(VERBOSE_JSON, speaker_embeddings={})))+ with patch('httpx.Client', return_value=client):+ response = connector.transcribe(_request(diarize=True))+ assert response.speaker_embeddings is None+++def test_registry_reads_the_shared_env_flag(monkeypatch):+ from src.services.transcription.registry import ConnectorRegistry+ registry = ConnectorRegistry()+ monkeypatch.setenv('TRANSCRIPTION_BASE_URL', 'http://127.0.0.1:8080')+ monkeypatch.setenv('ASR_RETURN_SPEAKER_EMBEDDINGS', 'true')+ assert registry._build_config_from_env('openasr')['return_speaker_embeddings'] is True+ monkeypatch.setenv('ASR_RETURN_SPEAKER_EMBEDDINGS', 'false')+ assert registry._build_config_from_env('openasr')['return_speaker_embeddings'] is False+ monkeypatch.delenv('ASR_RETURN_SPEAKER_EMBEDDINGS')+ assert registry._build_config_from_env('openasr')['return_speaker_embeddings'] is False