Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

Add OpenAI-compatible API mode for embeddings via EMBEDDING_BASE_URL

Murtaza Nasirecc50d44175f
diff --git a/docs/admin-guide/model-configuration.md b/docs/admin-guide/model-configuration.md
index 3f140444..ee7eb5bb 100644
--- a/docs/admin-guide/model-configuration.md
+++ b/docs/admin-guide/model-configuration.md
@@ -416,19 +416,49 @@ The override is propagated to the connector via the `model` field on `Transcript
## Configurable Embedding Model
-Speakr's Inquire mode (semantic search) uses [sentence-transformers](https://www.sbert.net/) to embed transcript chunks. The default model is `all-MiniLM-L6-v2` (384-dim vectors) which is fast, small, and good enough for most use cases.
+Speakr's Inquire mode (semantic search) uses [sentence-transformers](https://www.sbert.net/) to embed transcript chunks locally by default. The default model is `all-MiniLM-L6-v2` (384-dim vectors), which is fast, small, and sufficient for most use cases.
-If you'd prefer a more accurate model, set:
+To use a different local model, set:
```bash
-EMBEDDING_MODEL=all-mpnet-base-v2 # 768-dim, higher quality
+EMBEDDING_MODEL=all-mpnet-base-v2 # 768-dim, higher quality
# or
EMBEDDING_MODEL=multi-qa-MiniLM-L6-cos-v1 # tuned for question-answering
```
Any sentence-transformers compatible model name works.
-**Important compatibility note**: the model name is recorded in `system_setting` on first startup. If you change `EMBEDDING_MODEL` later, Speakr logs a warning at startup and Inquire mode will return wrong results until you reprocess affected recordings (chunks embedded with the old model won't match the new model's vector space). To rebuild embeddings, reprocess each recording.
+### Remote (API-based) Embeddings
+
+If you would rather offload embeddings to an OpenAI-compatible HTTP endpoint (vLLM, OpenRouter, OpenAI, Together, or any other compatible provider), set `EMBEDDING_BASE_URL` and Speakr switches to API mode. The same `EMBEDDING_MODEL` env var becomes the model identifier sent in each request.
+
+```bash
+# OpenAI directly
+EMBEDDING_BASE_URL=https://api.openai.com/v1
+EMBEDDING_API_KEY=sk-...
+EMBEDDING_MODEL=text-embedding-3-small
+EMBEDDING_DIMENSIONS=768 # optional, only honoured by providers that support output-dim trimming
+```
+
+```bash
+# Self-hosted vLLM serving an embedding model
+EMBEDDING_BASE_URL=http://vllm-host:8000/v1
+EMBEDDING_API_KEY=not-needed
+EMBEDDING_MODEL=BAAI/bge-large-en-v1.5
+```
+
+```bash
+# OpenRouter
+EMBEDDING_BASE_URL=https://openrouter.ai/api/v1
+EMBEDDING_API_KEY=sk-or-...
+EMBEDDING_MODEL=openai/text-embedding-3-large
+```
+
+API mode does not require sentence-transformers to be installed, which makes it the natural fit for the lightweight Docker image (`learnedmachine/speakr:lite`). Semantic search still requires `scikit-learn` for cosine similarity; the lite image includes it.
+
+### Compatibility Note
+
+The active embedding identifier (provider plus model) is recorded in `system_setting` on first startup. If you later change `EMBEDDING_MODEL` or `EMBEDDING_BASE_URL`, Speakr logs a warning at startup and Inquire mode will return wrong results until you reprocess affected recordings, because chunks embedded with the previous configuration occupy a different vector space. To rebuild embeddings after a change, reprocess each recording.
## Mistral Voxtral Chunking
diff --git a/src/init_db.py b/src/init_db.py
index bf579bdf..2d488404 100644
--- a/src/init_db.py
+++ b/src/init_db.py
@@ -657,35 +657,41 @@ def initialize_database(app):
)
app.logger.info("Initialized enable_folders setting")
- # Track the embedding model name in system_setting so we can warn when
- # it changes between restarts. Issue #262 — old vectors won't match a
- # new model's output dimensionality and inquire mode will silently
- # return zero results until users reprocess.
+ # Track the embedding identifier (provider + model) in system_setting
+ # so we can warn when either changes between restarts. Issue #262 —
+ # old vectors will not match a new model's output dimensionality or
+ # semantic space, and inquire mode will silently return wrong results
+ # until users reprocess.
try:
- current_model = os.environ.get('EMBEDDING_MODEL', 'all-MiniLM-L6-v2').strip() or 'all-MiniLM-L6-v2'
- stored_model = SystemSetting.get_setting('embedding_model_name', None)
+ from src.services.embeddings import EMBEDDING_IDENTIFIER
+ current_identifier = EMBEDDING_IDENTIFIER
+ # Read the legacy key first so existing instances upgrade smoothly.
+ stored_identifier = (
+ SystemSetting.get_setting('embedding_identifier', None)
+ or SystemSetting.get_setting('embedding_model_name', None)
+ )
chunk_count = TranscriptChunk.query.filter(TranscriptChunk.embedding.isnot(None)).count()
- if stored_model is None:
+ if stored_identifier is None:
SystemSetting.set_setting(
- key='embedding_model_name',
- value=current_model,
- description='Sentence-transformers model that produced the stored chunk embeddings. Used to detect dimensionality mismatches at startup.',
+ key='embedding_identifier',
+ value=current_identifier,
+ description='Identifier of the embedding provider and model that produced the stored chunk vectors. Used to detect dimensionality and semantic-space mismatches at startup.',
setting_type='string',
)
if chunk_count:
- app.logger.info(f"Recorded embedding_model_name={current_model} (existing {chunk_count} chunks assumed to match)")
- elif stored_model != current_model:
+ app.logger.info(f"Recorded embedding_identifier={current_identifier} (existing {chunk_count} chunks assumed to match)")
+ elif stored_identifier != current_identifier:
if chunk_count:
app.logger.warning(
- f"EMBEDDING_MODEL changed from {stored_model!r} to {current_model!r} but {chunk_count} "
- "chunks were embedded with the previous model. Inquire mode will return wrong results "
- "until you reprocess affected recordings."
+ f"Embedding identifier changed from {stored_identifier!r} to {current_identifier!r} "
+ f"but {chunk_count} chunks were embedded with the previous configuration. "
+ "Inquire mode will return wrong results until you reprocess affected recordings."
)
- SystemSetting.set_setting(key='embedding_model_name', value=current_model)
- app.logger.info(f"Updated embedding_model_name in system_setting: {stored_model!r} -> {current_model!r}")
+ SystemSetting.set_setting(key='embedding_identifier', value=current_identifier)
+ app.logger.info(f"Updated embedding_identifier in system_setting: {stored_identifier!r} -> {current_identifier!r}")
except Exception as e:
db.session.rollback()
- app.logger.warning(f"embedding_model_name compatibility check skipped: {e}")
+ app.logger.warning(f"embedding_identifier compatibility check skipped: {e}")
# One-shot migration: clean up legacy User.transcription_language values
# that were stored as display names ("Français", "English") before the
diff --git a/src/services/embeddings.py b/src/services/embeddings.py
index 369aa4cc..dd427a0b 100644
--- a/src/services/embeddings.py
+++ b/src/services/embeddings.py
@@ -10,33 +10,69 @@ from sqlalchemy.orm import joinedload
try:
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
- EMBEDDINGS_AVAILABLE = True
+ LOCAL_EMBEDDINGS_AVAILABLE = True
except ImportError:
- EMBEDDINGS_AVAILABLE = False
+ LOCAL_EMBEDDINGS_AVAILABLE = False
+ SentenceTransformer = None # type: ignore
cosine_similarity = None
+# sklearn's cosine_similarity may be available even when sentence-transformers
+# is not (e.g., the lite image with API-mode embeddings). Try to import it
+# independently so semantic search can still run on API-generated vectors.
+if cosine_similarity is None:
+ try:
+ from sklearn.metrics.pairwise import cosine_similarity # type: ignore
+ except ImportError:
+ cosine_similarity = None
+
from src.database import db
from src.models import Recording, TranscriptChunk, InternalShare, RecordingTag
ENABLE_INTERNAL_SHARING = os.environ.get('ENABLE_INTERNAL_SHARING', 'false').lower() == 'true'
-# Sentence-transformers model used to generate semantic-search vectors. The
-# default — all-MiniLM-L6-v2 — produces 384-dim vectors and is what older
-# Speakr instances embedded their chunks with. Changing this env var on an
-# existing instance will cause a dimensionality mismatch with stored vectors;
-# see startup warning in src/init_db.py and the docs.
+# Embedding model used to generate semantic-search vectors. By default Speakr
+# loads sentence-transformers locally; setting EMBEDDING_BASE_URL switches to
+# an OpenAI-compatible HTTP endpoint (vLLM, OpenRouter, OpenAI directly, etc.)
+# and the same EMBEDDING_MODEL value is sent as the model name in requests.
+# Changing the model or the endpoint after chunks are already embedded will
+# produce a dimensionality / semantic-space mismatch; see the startup warning
+# in src/init_db.py and the docs for guidance.
EMBEDDING_MODEL = os.environ.get('EMBEDDING_MODEL', 'all-MiniLM-L6-v2').strip() or 'all-MiniLM-L6-v2'
+EMBEDDING_BASE_URL = os.environ.get('EMBEDDING_BASE_URL', '').strip()
+EMBEDDING_API_KEY = os.environ.get('EMBEDDING_API_KEY', '').strip()
+_dim_raw = os.environ.get('EMBEDDING_DIMENSIONS', '').strip()
+try:
+ EMBEDDING_DIMENSIONS = int(_dim_raw) if _dim_raw else None
+except (TypeError, ValueError):
+ EMBEDDING_DIMENSIONS = None
+
+# When EMBEDDING_BASE_URL is set, embeddings are produced via an OpenAI-
+# compatible /v1/embeddings call rather than locally.
+USE_API_EMBEDDINGS = bool(EMBEDDING_BASE_URL)
+
+# Identifier persisted to system_setting so the dimensionality compatibility
+# check covers both "model changed" and "provider changed" scenarios.
+EMBEDDING_IDENTIFIER = f"{EMBEDDING_BASE_URL or 'local'}::{EMBEDDING_MODEL}"
+
+# True when at least one embedding path can produce vectors. The legacy name
+# EMBEDDINGS_AVAILABLE is preserved for callers that import it elsewhere.
+EMBEDDINGS_AVAILABLE = LOCAL_EMBEDDINGS_AVAILABLE or (USE_API_EMBEDDINGS and cosine_similarity is not None)
# Initialize embedding model (lazy loading)
_embedding_model = None
+_embedding_api_client = None
def get_embedding_model():
- """Get or initialize the sentence transformer model."""
+ """Get or initialize the local sentence transformer model.
+
+ Returns None when running in API mode or when sentence-transformers is
+ not installed.
+ """
global _embedding_model
- if not EMBEDDINGS_AVAILABLE:
+ if USE_API_EMBEDDINGS or not LOCAL_EMBEDDINGS_AVAILABLE:
return None
if _embedding_model is None:
@@ -49,6 +85,52 @@ def get_embedding_model():
return _embedding_model
+def get_embedding_api_client():
+ """Get or initialize the OpenAI-compatible embeddings API client.
+
+ Returns None when API mode is disabled or the openai package is missing.
+ """
+ global _embedding_api_client
+
+ if not USE_API_EMBEDDINGS:
+ return None
+
+ if _embedding_api_client is None:
+ try:
+ from openai import OpenAI
+ from src.services.llm import llm_timeout, LLM_MAX_RETRIES, http_client_no_proxy
+ _embedding_api_client = OpenAI(
+ api_key=EMBEDDING_API_KEY or "not-needed",
+ base_url=EMBEDDING_BASE_URL,
+ http_client=http_client_no_proxy,
+ timeout=llm_timeout,
+ max_retries=LLM_MAX_RETRIES,
+ )
+ current_app.logger.info(
+ f"Embedding API client initialized: base_url={EMBEDDING_BASE_URL}, model={EMBEDDING_MODEL}"
+ )
+ except Exception as e:
+ current_app.logger.error(f"Failed to initialize embedding API client: {e}")
+ return None
+ return _embedding_api_client
+
+
+def _api_embed(texts):
+ """Call the OpenAI-compatible embeddings endpoint and return numpy vectors."""
+ client = get_embedding_api_client()
+ if client is None or not texts:
+ return []
+ try:
+ kwargs = {'input': texts, 'model': EMBEDDING_MODEL}
+ if EMBEDDING_DIMENSIONS is not None:
+ kwargs['dimensions'] = EMBEDDING_DIMENSIONS
+ response = client.embeddings.create(**kwargs)
+ return [np.array(d.embedding, dtype=np.float32) for d in response.data]
+ except Exception as e:
+ current_app.logger.error(f"Embedding API call failed: {e}")
+ return []
+
+
def chunk_transcription(transcription, max_chunk_length=500, overlap=50):
"""
@@ -102,21 +184,31 @@ def chunk_transcription(transcription, max_chunk_length=500, overlap=50):
def generate_embeddings(texts):
"""
Generate embeddings for a list of texts.
-
+
+ Routes through the OpenAI-compatible API when EMBEDDING_BASE_URL is set;
+ otherwise uses the locally loaded sentence-transformers model.
+
Args:
texts (list): List of text strings
-
+
Returns:
- list: List of embedding vectors as numpy arrays, or empty list if embeddings unavailable
+ list: List of embedding vectors as numpy arrays, or empty list if no
+ embedding path is available.
"""
- if not EMBEDDINGS_AVAILABLE:
+ if not texts:
+ return []
+
+ if USE_API_EMBEDDINGS:
+ return _api_embed(texts)
+
+ if not LOCAL_EMBEDDINGS_AVAILABLE:
current_app.logger.warning("Embeddings not available - skipping embedding generation")
return []
-
+
model = get_embedding_model()
- if not model or not texts:
+ if not model:
return []
-
+
try:
embeddings = model.encode(texts)
return [embedding.astype(np.float32) for embedding in embeddings]
@@ -341,12 +433,21 @@ def semantic_search_chunks(user_id, query, filters=None, top_k=5):
current_app.logger.info("Embeddings not available - using basic text search as fallback")
return basic_text_search_chunks(user_id, query, filters, top_k)
- # Generate embedding for the query
- model = get_embedding_model()
- if not model:
+ if cosine_similarity is None:
+ current_app.logger.info("scikit-learn not installed - using basic text search as fallback")
return basic_text_search_chunks(user_id, query, filters, top_k)
- query_embedding = model.encode([query])[0]
+ # Generate embedding for the query (via API or local model)
+ if USE_API_EMBEDDINGS:
+ api_vectors = _api_embed([query])
+ if not api_vectors:
+ return basic_text_search_chunks(user_id, query, filters, top_k)
+ query_embedding = api_vectors[0]
+ else:
+ model = get_embedding_model()
+ if not model:
+ return basic_text_search_chunks(user_id, query, filters, top_k)
+ query_embedding = model.encode([query])[0]
# Get all accessible recording IDs (own + shared)
accessible_recording_ids = get_accessible_recording_ids(user_id)
diff --git a/tests/test_embedding_api_mode.py b/tests/test_embedding_api_mode.py
new file mode 100644
index 00000000..97fcd929
--- /dev/null
+++ b/tests/test_embedding_api_mode.py
@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""
+Unit tests for issue #262 follow-up: API-mode embeddings.
+
+Verifies that:
+- USE_API_EMBEDDINGS reflects the EMBEDDING_BASE_URL env var.
+- EMBEDDING_IDENTIFIER composes the provider + model name.
+- generate_embeddings() routes through the API client when API mode is on.
+- The local SentenceTransformer path is not initialised in API mode.
+
+The OpenAI client is patched out; the test does not make real network calls.
+
+Run with: docker exec speakr-dev python /app/tests/test_embedding_api_mode.py
+"""
+
+import importlib
+import os
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+
+PASSED = 0
+FAILED = 0
+
+
+def run(name, func):
+ global PASSED, FAILED
+ try:
+ func()
+ print(f" ✓ {name}")
+ PASSED += 1
+ except AssertionError as e:
+ print(f" ✗ {name}: {e}")
+ FAILED += 1
+ except Exception as e:
+ print(f" ✗ {name}: EXCEPTION - {e}")
+ FAILED += 1
+
+
+def reload_with_env(**env):
+ """Reload the embeddings module with the requested env overrides."""
+ keys = list(env.keys()) + [
+ "EMBEDDING_MODEL", "EMBEDDING_BASE_URL", "EMBEDDING_API_KEY", "EMBEDDING_DIMENSIONS",
+ ]
+ saved = {k: os.environ.get(k) for k in keys}
+ try:
+ for k in keys:
+ os.environ.pop(k, None)
+ for k, v in env.items():
+ if v is not None:
+ os.environ[k] = v
+ import src.services.embeddings as emb
+ importlib.reload(emb)
+ return emb
+ finally:
+ # Restore env so later tests see a clean slate, then reload again.
+ for k in keys:
+ os.environ.pop(k, None)
+ if saved[k] is not None:
+ os.environ[k] = saved[k]
+
+
+def test_default_is_local_mode():
+ emb = reload_with_env()
+ assert emb.USE_API_EMBEDDINGS is False, f"got USE_API_EMBEDDINGS={emb.USE_API_EMBEDDINGS}"
+ assert emb.EMBEDDING_MODEL == "all-MiniLM-L6-v2"
+ assert emb.EMBEDDING_IDENTIFIER == "local::all-MiniLM-L6-v2", emb.EMBEDDING_IDENTIFIER
+
+
+def test_base_url_switches_to_api_mode():
+ emb = reload_with_env(
+ EMBEDDING_BASE_URL="https://api.openai.com/v1",
+ EMBEDDING_API_KEY="sk-test",
+ EMBEDDING_MODEL="text-embedding-3-small",
+ )
+ assert emb.USE_API_EMBEDDINGS is True
+ assert emb.EMBEDDING_BASE_URL == "https://api.openai.com/v1"
+ assert emb.EMBEDDING_MODEL == "text-embedding-3-small"
+ assert emb.EMBEDDING_IDENTIFIER == "https://api.openai.com/v1::text-embedding-3-small"
+
+
+def test_dimensions_parsed():
+ emb = reload_with_env(
+ EMBEDDING_BASE_URL="https://api.openai.com/v1",
+ EMBEDDING_MODEL="text-embedding-3-large",
+ EMBEDDING_DIMENSIONS="1024",
+ )
+ assert emb.EMBEDDING_DIMENSIONS == 1024
+
+
+def test_dimensions_invalid_falls_back_to_none():
+ emb = reload_with_env(
+ EMBEDDING_BASE_URL="https://api.openai.com/v1",
+ EMBEDDING_DIMENSIONS="not-a-number",
+ )
+ assert emb.EMBEDDING_DIMENSIONS is None
+
+
+def test_generate_embeddings_uses_api_when_active():
+ emb = reload_with_env(
+ EMBEDDING_BASE_URL="http://localhost:9999/v1",
+ EMBEDDING_API_KEY="not-needed",
+ EMBEDDING_MODEL="bge-base",
+ )
+ fake_response = MagicMock()
+ fake_response.data = [
+ MagicMock(embedding=[0.1, 0.2, 0.3]),
+ MagicMock(embedding=[0.4, 0.5, 0.6]),
+ ]
+ fake_client = MagicMock()
+ fake_client.embeddings.create.return_value = fake_response
+
+ from flask import Flask
+ test_app = Flask(__name__)
+ with test_app.app_context():
+ with patch.object(emb, "get_embedding_api_client", return_value=fake_client):
+ vectors = emb.generate_embeddings(["hello", "world"])
+
+ assert len(vectors) == 2, f"expected 2 vectors, got {len(vectors)}"
+ assert all(v.dtype.name == "float32" for v in vectors)
+ fake_client.embeddings.create.assert_called_once()
+ call_kwargs = fake_client.embeddings.create.call_args.kwargs
+ assert call_kwargs["model"] == "bge-base"
+ assert call_kwargs["input"] == ["hello", "world"]
+ # No explicit dimensions, so kwarg should be absent.
+ assert "dimensions" not in call_kwargs
+
+
+def test_generate_embeddings_passes_dimensions_when_set():
+ emb = reload_with_env(
+ EMBEDDING_BASE_URL="http://localhost:9999/v1",
+ EMBEDDING_MODEL="text-embedding-3-large",
+ EMBEDDING_DIMENSIONS="512",
+ )
+ fake_response = MagicMock()
+ fake_response.data = [MagicMock(embedding=[0.0] * 512)]
+ fake_client = MagicMock()
+ fake_client.embeddings.create.return_value = fake_response
+
+ from flask import Flask
+ test_app = Flask(__name__)
+ with test_app.app_context():
+ with patch.object(emb, "get_embedding_api_client", return_value=fake_client):
+ emb.generate_embeddings(["hello"])
+
+ call_kwargs = fake_client.embeddings.create.call_args.kwargs
+ assert call_kwargs["dimensions"] == 512
+
+
+def test_get_embedding_model_returns_none_in_api_mode():
+ emb = reload_with_env(EMBEDDING_BASE_URL="http://localhost:9999/v1")
+ from flask import Flask
+ test_app = Flask(__name__)
+ with test_app.app_context():
+ assert emb.get_embedding_model() is None
+
+
+def main():
+ print("=== Issue #262 follow-up: API-mode embeddings ===\n")
+ run("default config is local mode", test_default_is_local_mode)
+ run("EMBEDDING_BASE_URL switches to API mode", test_base_url_switches_to_api_mode)
+ run("EMBEDDING_DIMENSIONS parses to int", test_dimensions_parsed)
+ run("invalid EMBEDDING_DIMENSIONS falls back to None", test_dimensions_invalid_falls_back_to_none)
+ run("generate_embeddings uses API client when active", test_generate_embeddings_uses_api_when_active)
+ run("generate_embeddings forwards dimensions when set", test_generate_embeddings_passes_dimensions_when_set)
+ run("get_embedding_model returns None in API mode", test_get_embedding_model_returns_none_in_api_mode)
+
+ # Reset the module to its default state for any subsequent test runs.
+ reload_with_env()
+
+ print(f"\nResults: {PASSED} passed, {FAILED} failed")
+ return 0 if FAILED == 0 else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())