Speakr is a personal, self-hosted web application designed for transcribing audio recordings
Python
Add separate chat model configuration for different service tiers (#143)
Murtaza Nasiraaf22909f343
diff --git a/config/env.asr.example b/config/env.asr.exampleindex 00f15f0a..989e0131 100644--- a/config/env.asr.example+++ b/config/env.asr.example@@ -33,6 +33,26 @@ GPT5_REASONING_EFFORT=medium# - high: Detailed explanationsGPT5_VERBOSITY=medium+# --- Chat Model Configuration (Optional) ---+# Configure a separate model for real-time chat interactions.+# If not set, chat will use the TEXT_MODEL_* settings above.+#+# Use cases:+# - Use a faster model for chat while using a more capable model for summarization+# - Use a cheaper model for interactive chat to reduce costs+# - Use different service tiers for different operations+#+# CHAT_MODEL_API_KEY=your_chat_api_key+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1+# CHAT_MODEL_NAME=openai/gpt-4o++# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---+# These settings allow independent control of GPT-5 parameters for chat.+# If not set, falls back to the main GPT5_* settings above.+#+# CHAT_GPT5_REASONING_EFFORT=medium+# CHAT_GPT5_VERBOSITY=medium+# --- Transcription Service (ASR Endpoint) ---# Setting this to true enables all ASR features including speaker diarization# Note: File chunking is NOT supported with ASR endpointsdiff --git a/config/env.whisper.example b/config/env.whisper.exampleindex db1d1dfb..3d357388 100644--- a/config/env.whisper.example+++ b/config/env.whisper.example@@ -33,6 +33,26 @@ GPT5_REASONING_EFFORT=medium# - high: Detailed explanationsGPT5_VERBOSITY=medium+# --- Chat Model Configuration (Optional) ---+# Configure a separate model for real-time chat interactions.+# If not set, chat will use the TEXT_MODEL_* settings above.+#+# Use cases:+# - Use a faster model for chat while using a more capable model for summarization+# - Use a cheaper model for interactive chat to reduce costs+# - Use different service tiers for different operations+#+# CHAT_MODEL_API_KEY=your_chat_api_key+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1+# CHAT_MODEL_NAME=openai/gpt-4o++# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---+# These settings allow independent control of GPT-5 parameters for chat.+# If not set, falls back to the main GPT5_* settings above.+#+# CHAT_GPT5_REASONING_EFFORT=medium+# CHAT_GPT5_VERBOSITY=medium+# --- Transcription Service (OpenAI Whisper API) ---TRANSCRIPTION_BASE_URL=https://api.openai.com/v1TRANSCRIPTION_API_KEY=your_openai_api_keydiff --git a/config/env.whisperx.example b/config/env.whisperx.exampleindex af830353..28837f55 100644--- a/config/env.whisperx.example+++ b/config/env.whisperx.example@@ -43,6 +43,26 @@ GPT5_REASONING_EFFORT=medium# - high: Detailed explanationsGPT5_VERBOSITY=medium+# --- Chat Model Configuration (Optional) ---+# Configure a separate model for real-time chat interactions.+# If not set, chat will use the TEXT_MODEL_* settings above.+#+# Use cases:+# - Use a faster model for chat while using a more capable model for summarization+# - Use a cheaper model for interactive chat to reduce costs+# - Use different service tiers for different operations+#+# CHAT_MODEL_API_KEY=your_chat_api_key+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1+# CHAT_MODEL_NAME=openai/gpt-4o++# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---+# These settings allow independent control of GPT-5 parameters for chat.+# If not set, falls back to the main GPT5_* settings above.+#+# CHAT_GPT5_REASONING_EFFORT=medium+# CHAT_GPT5_VERBOSITY=medium+# --- Transcription Service (WhisperX ASR Endpoint) ---# Setting this to true enables all ASR features including speaker diarization and voice profiles# Note: File chunking is NOT supported with ASR endpointsdiff --git a/docs/admin-guide/model-configuration.md b/docs/admin-guide/model-configuration.mdindex 0ac608a9..c4326874 100644--- a/docs/admin-guide/model-configuration.md+++ b/docs/admin-guide/model-configuration.md@@ -209,6 +209,101 @@ For most use cases, we recommend:This provides a good balance of cost, speed, and quality.+## Separate Chat Model Configuration++Speakr allows you to configure a separate model specifically for real-time chat interactions, while using a different model for background tasks like summarization and title generation. This enables you to:++- **Use different service tiers**: Configure a faster, more expensive model for interactive chat while using a cheaper model for background processing+- **Optimize costs**: Use a budget-friendly model for summarization while keeping a high-quality model for chat+- **Balance speed and quality**: Prioritize low latency for chat while accepting slower processing for summaries++### Configuration++Add these optional environment variables to your `.env` file:++```bash+# Chat Model Configuration (Optional)+# If not set, chat will use TEXT_MODEL_* settings+CHAT_MODEL_API_KEY=your_chat_api_key+CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1+CHAT_MODEL_NAME=openai/gpt-4o+```++### Fallback Behavior++| Configuration | Behavior |+|--------------|----------|+| No `CHAT_MODEL_*` variables set | Chat uses `TEXT_MODEL_*` settings (default) |+| Only `CHAT_MODEL_NAME` set | Falls back to `TEXT_MODEL_*` (API key required) |+| Only `CHAT_MODEL_API_KEY` set | Falls back to `TEXT_MODEL_*` (model name required) |+| `CHAT_MODEL_API_KEY` + `CHAT_MODEL_NAME` set | Uses chat config with `TEXT_MODEL_BASE_URL` |+| All `CHAT_MODEL_*` variables set | Uses fully dedicated chat configuration |++### GPT-5 Settings for Chat++If you use GPT-5 models for chat, you can configure separate GPT-5 parameters:++```bash+# Chat-specific GPT-5 settings (optional)+# Falls back to GPT5_* settings if not specified+CHAT_GPT5_REASONING_EFFORT=medium+CHAT_GPT5_VERBOSITY=medium+```++### Example Configurations++**Cheap Summarization + Premium Chat**:+```bash+# Background tasks: Use budget model+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1+TEXT_MODEL_API_KEY=your_openrouter_key+TEXT_MODEL_NAME=openai/gpt-4o-mini++# Interactive chat: Use premium model+CHAT_MODEL_API_KEY=your_openai_key+CHAT_MODEL_BASE_URL=https://api.openai.com/v1+CHAT_MODEL_NAME=gpt-5-mini+CHAT_GPT5_REASONING_EFFORT=low+CHAT_GPT5_VERBOSITY=medium+```++**Same Provider, Different Models**:+```bash+# Background tasks: Smaller model+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1+TEXT_MODEL_API_KEY=your_api_key+TEXT_MODEL_NAME=google/gemini-2.5-flash-lite++# Interactive chat: Larger model (same provider)+CHAT_MODEL_NAME=openai/gpt-4o+# Note: CHAT_MODEL_API_KEY not needed if using same key+# Note: CHAT_MODEL_BASE_URL not needed if using same endpoint+```++**Different Service Tiers (OpenAI)**:+```bash+# Background tasks: Standard tier+TEXT_MODEL_BASE_URL=https://api.openai.com/v1+TEXT_MODEL_API_KEY=your_standard_tier_key+TEXT_MODEL_NAME=gpt-4o-mini++# Interactive chat: Priority tier for faster responses+CHAT_MODEL_API_KEY=your_priority_tier_key+CHAT_MODEL_NAME=gpt-4o+```++### When to Use Separate Chat Models++**Recommended for**:+- High-volume deployments where chat responsiveness is critical+- Users who need different service tiers for different operations+- Cost optimization when chat usage is significantly higher than summarization++**Not needed for**:+- Small deployments with low usage+- When using the same model for all operations is acceptable+- Simple setups where configuration simplicity is preferred+## Model Selection Guidelines### For Summaries@@ -268,6 +363,15 @@ CHAT_MAX_TOKENS=2000# GPT-5 specific (only used with GPT-5 models and OpenAI API)GPT5_REASONING_EFFORT=medium # minimal, low, medium, highGPT5_VERBOSITY=medium # low, medium, high++# Chat model configuration (optional - falls back to TEXT_MODEL_* if not set)+CHAT_MODEL_API_KEY=your_chat_api_key+CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1+CHAT_MODEL_NAME=openai/gpt-4o++# Chat-specific GPT-5 settings (optional - falls back to GPT5_* if not set)+CHAT_GPT5_REASONING_EFFORT=medium # minimal, low, medium, high+CHAT_GPT5_VERBOSITY=medium # low, medium, high```## Troubleshootingdiff --git a/docs/faq.md b/docs/faq.mdindex 85e3503f..c03bc9a4 100644--- a/docs/faq.md+++ b/docs/faq.md@@ -104,6 +104,16 @@ There's no hard limit on recording length, but practical considerations apply. VSummary generation uses the language model configured in your [environment file](getting-started.md#step-3-configure-your-transcription-service). Customize summaries with [AI prompts](admin-guide/prompts.md) - through a local LLM endpoint or a Cloud provider like OpenAI or OpenRouter. The model choice affects [summary quality](features.md#automatic-summarization), cost, and processing speed. Monitor performance in [system statistics](admin-guide/statistics.md).+### Can I use different models for chat and summaries?++Yes! You can configure a separate model specifically for real-time chat interactions while using a different model for background tasks like summarization. This is useful for:++- **Different service tiers**: Use a faster, premium model for interactive chat while using a budget model for background processing+- **Cost optimization**: Use cheaper models for summarization while keeping high-quality responses for chat+- **Speed optimization**: Prioritize low latency for chat while accepting slower processing for summaries++Configure this using the optional `CHAT_MODEL_*` environment variables. See the [Model Configuration](admin-guide/model-configuration.md#separate-chat-model-configuration) guide for details.+## Privacy and Security### Is my data really private?diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.mdindex 2dc010a3..5b070bca 100644--- a/docs/getting-started/installation.md+++ b/docs/getting-started/installation.md@@ -78,6 +78,8 @@ TEXT_MODEL_NAME=openai/gpt-4o-miniIf you prefer to use OpenAI directly for text generation, simply change the base URL to `https://api.openai.com/v1` and use your OpenAI API key. You can also use local models through Ollama or LM Studio by pointing to `http://localhost:11434/v1` or similar.+> **Tip:** For advanced model configuration options—including GPT-5 support, separate chat model settings for different service tiers, and cost optimization strategies—see the [Model Configuration](../admin-guide/model-configuration.md) guide.+Next, configure the transcription service. This is what converts your audio files into text:```bashdiff --git a/src/api/inquire.py b/src/api/inquire.pyindex cb6b53fb..643f0a25 100644--- a/src/api/inquire.py+++ b/src/api/inquire.py@@ -17,7 +17,7 @@ from src.database import dbfrom src.models import *from src.utils import *from src.services.embeddings import get_accessible_recording_ids, semantic_search_chunks-from src.services.llm import call_llm_completion, process_streaming_with_thinking, client+from src.services.llm import call_llm_completion, call_chat_completion, process_streaming_with_thinking, client, chat_client# Create blueprintinquire_bp = Blueprint('inquire', __name__)@@ -631,16 +631,16 @@ Order your response with notes from the most recent meetings first. Always use pmessages.append({"role": "user", "content": user_message})# Enable streaming- stream = call_llm_completion(+ stream = call_chat_completion(messages=messages,temperature=0.7,max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),stream=True)-+# Buffer content to detect full transcript requestsresponse_buffer = ""-+# Buffer content to detect full transcript requestsresponse_buffer = ""content_buffer = ""@@ -692,8 +692,8 @@ Order your response with notes from the most recent meetings first. Always use p# Generate new response with full contextyield create_status_response('responding', 'Analyzing full transcript...')-- new_stream = call_llm_completion(++ new_stream = call_chat_completion(messages=updated_messages,temperature=0.7,max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),diff --git a/src/api/recordings.py b/src/api/recordings.pyindex 4954ac24..78952994 100644--- a/src/api/recordings.py+++ b/src/api/recordings.py@@ -28,7 +28,7 @@ from src.services.speaker import update_speaker_usage, identify_unidentified_spefrom src.services.speaker_embedding_matcher import update_speaker_embeddingfrom src.services.speaker_snippets import create_speaker_snippetsfrom src.services.document import process_markdown_to_docx-from src.services.llm import client, call_llm_completion, process_streaming_with_thinking+from src.services.llm import client, chat_client, call_llm_completion, call_chat_completion, process_streaming_with_thinkingfrom src.services.embeddings import process_recording_chunksfrom src.file_exporter import export_recording, mark_export_as_deleted@@ -2630,9 +2630,9 @@ def chat_with_transcription():if not has_recording_access(recording, current_user):return jsonify({'error': 'You do not have permission to chat with this recording'}), 403- # Check if OpenRouter client is available- if client is None:- return jsonify({'error': 'Chat service is not available (OpenRouter client not configured)'}), 503+ # Check if chat client is available+ if chat_client is None:+ return jsonify({'error': 'Chat service is not available (chat client not configured)'}), 503# Prepare the system prompt with the transcriptionuser_chat_output_language = current_user.output_language if current_user.is_authenticated else None@@ -2678,7 +2678,7 @@ Additional context and notes about the meeting:def generate():try:# Enable streaming- stream = call_llm_completion(+ stream = call_chat_completion(messages=messages,temperature=0.7,max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),diff --git a/src/services/__init__.py b/src/services/__init__.pyindex cd5ad305..4aea0def 100644--- a/src/services/__init__.py+++ b/src/services/__init__.py@@ -22,6 +22,8 @@ __all__ = ['is_gpt5_model','is_using_openai_api','call_llm_completion',+ 'call_chat_completion',+ 'chat_client','format_api_error_message',# Document services'process_markdown_to_docx',diff --git a/src/services/llm.py b/src/services/llm.pyindex 2225eec5..d654c08c 100644--- a/src/services/llm.py+++ b/src/services/llm.py@@ -21,6 +21,41 @@ if TEXT_MODEL_BASE_URL:TEXT_MODEL_BASE_URL = TEXT_MODEL_BASE_URL.split('#')[0].strip()TEXT_MODEL_NAME = os.environ.get("TEXT_MODEL_NAME", "openai/gpt-3.5-turbo")+# Chat model configuration (optional - falls back to TEXT_MODEL_* if not set)+CHAT_MODEL_API_KEY = os.environ.get("CHAT_MODEL_API_KEY")+CHAT_MODEL_BASE_URL = os.environ.get("CHAT_MODEL_BASE_URL")+if CHAT_MODEL_BASE_URL:+ CHAT_MODEL_BASE_URL = CHAT_MODEL_BASE_URL.split('#')[0].strip()+CHAT_MODEL_NAME = os.environ.get("CHAT_MODEL_NAME")++# Chat-specific GPT-5 settings (optional - falls back to main GPT5_* settings)+CHAT_GPT5_REASONING_EFFORT = os.environ.get("CHAT_GPT5_REASONING_EFFORT")+CHAT_GPT5_VERBOSITY = os.environ.get("CHAT_GPT5_VERBOSITY")+++def get_chat_config():+ """+ Get chat model configuration, falling back to TEXT_MODEL if not set.++ Returns a dict with api_key, base_url, model_name, and GPT-5 settings.+ """+ if CHAT_MODEL_API_KEY and CHAT_MODEL_NAME:+ return {+ 'api_key': CHAT_MODEL_API_KEY,+ 'base_url': CHAT_MODEL_BASE_URL or TEXT_MODEL_BASE_URL,+ 'model_name': CHAT_MODEL_NAME,+ 'gpt5_reasoning_effort': CHAT_GPT5_REASONING_EFFORT or os.environ.get("GPT5_REASONING_EFFORT", "medium"),+ 'gpt5_verbosity': CHAT_GPT5_VERBOSITY or os.environ.get("GPT5_VERBOSITY", "medium")+ }+ return {+ 'api_key': TEXT_MODEL_API_KEY,+ 'base_url': TEXT_MODEL_BASE_URL,+ 'model_name': TEXT_MODEL_NAME,+ 'gpt5_reasoning_effort': os.environ.get("GPT5_REASONING_EFFORT", "medium"),+ 'gpt5_verbosity': os.environ.get("GPT5_VERBOSITY", "medium")+ }++# Set up HTTP client with custom headers for OpenRouter app identificationapp_headers = {"HTTP-Referer": "https://github.com/murtaza-nasir/speakr",@@ -44,6 +79,25 @@ try:except Exception as client_init_e:client = None+# Create chat client (may be same as main client if no separate config)+chat_client = None+try:+ chat_config = get_chat_config()+ if chat_config['api_key']:+ if CHAT_MODEL_API_KEY and CHAT_MODEL_API_KEY != TEXT_MODEL_API_KEY:+ # Separate chat configuration - create dedicated client+ chat_client = OpenAI(+ api_key=chat_config['api_key'],+ base_url=chat_config['base_url'],+ http_client=http_client_no_proxy+ )+ logger.info(f"Separate chat client initialized: {chat_config['base_url']} / {chat_config['model_name']}")+ else:+ # Use same client as main LLM+ chat_client = client+except Exception as chat_client_init_e:+ logger.warning(f"Failed to initialize chat client, falling back to main client: {chat_client_init_e}")+ chat_client = clientdef is_gpt5_model(model_name):@@ -149,7 +203,73 @@ def call_llm_completion(messages, temperature=0.7, response_format=None, stream=logger.error(f"LLM API call failed: {e}")raise-# Store details for the transcription client (potentially different)++def call_chat_completion(messages, temperature=0.7, response_format=None, stream=False, max_tokens=None):+ """+ Chat-specific LLM completion function. Uses dedicated chat model if configured,+ otherwise falls back to standard TEXT_MODEL configuration.++ Args:+ messages: List of message dicts with 'role' and 'content'+ temperature: Sampling temperature (0-1) - ignored for GPT-5 models+ response_format: Optional response format dict (e.g., {"type": "json_object"})+ stream: Whether to stream the response+ max_tokens: Optional maximum tokens to generate++ Returns:+ OpenAI completion object or generator (if streaming)+ """+ effective_client = chat_client if chat_client else client+ chat_config = get_chat_config()++ if not effective_client:+ raise ValueError("Chat LLM client not initialized")++ if not chat_config['api_key']:+ raise ValueError("Chat model API key not configured")++ try:+ model_name = chat_config['model_name']+ base_url = chat_config['base_url'] or ''++ # Check if we're using GPT-5 with OpenAI API+ using_gpt5 = is_gpt5_model(model_name) and 'api.openai.com' in base_url++ completion_args = {+ "model": model_name,+ "messages": messages,+ "stream": stream+ }++ if using_gpt5:+ logger.debug(f"Using GPT-5 chat model: {model_name}")+ # Use chat-specific GPT-5 settings from config+ completion_args["reasoning_effort"] = chat_config['gpt5_reasoning_effort']+ completion_args["verbosity"] = chat_config['gpt5_verbosity']++ if max_tokens:+ completion_args["max_completion_tokens"] = max_tokens+ else:+ completion_args["temperature"] = temperature+ if max_tokens:+ completion_args["max_tokens"] = max_tokens++ if response_format:+ completion_args["response_format"] = response_format++ response = effective_client.chat.completions.create(**completion_args)++ # Debug log for empty responses+ if not stream and response.choices:+ content = response.choices[0].message.content+ if not content:+ logger.warning(f"Chat LLM returned empty content. Model: {model_name}, finish_reason: {response.choices[0].finish_reason}")++ return response++ except Exception as e:+ logger.error(f"Chat LLM API call failed: {e}")+ raisedef format_api_error_message(error_str):