Skip to content

Speakr is a personal, self-hosted web application designed for transcribing audio recordings

Python

Add separate chat model configuration for different service tiers (#143)

Murtaza Nasiraaf22909f343
diff --git a/config/env.asr.example b/config/env.asr.example
index 00f15f0a..989e0131 100644
--- a/config/env.asr.example
+++ b/config/env.asr.example
@@ -33,6 +33,26 @@ GPT5_REASONING_EFFORT=medium
# - high: Detailed explanations
GPT5_VERBOSITY=medium
+# --- Chat Model Configuration (Optional) ---
+# Configure a separate model for real-time chat interactions.
+# If not set, chat will use the TEXT_MODEL_* settings above.
+#
+# Use cases:
+# - Use a faster model for chat while using a more capable model for summarization
+# - Use a cheaper model for interactive chat to reduce costs
+# - Use different service tiers for different operations
+#
+# CHAT_MODEL_API_KEY=your_chat_api_key
+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+# CHAT_MODEL_NAME=openai/gpt-4o
+
+# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---
+# These settings allow independent control of GPT-5 parameters for chat.
+# If not set, falls back to the main GPT5_* settings above.
+#
+# CHAT_GPT5_REASONING_EFFORT=medium
+# CHAT_GPT5_VERBOSITY=medium
+
# --- Transcription Service (ASR Endpoint) ---
# Setting this to true enables all ASR features including speaker diarization
# Note: File chunking is NOT supported with ASR endpoints
diff --git a/config/env.whisper.example b/config/env.whisper.example
index db1d1dfb..3d357388 100644
--- a/config/env.whisper.example
+++ b/config/env.whisper.example
@@ -33,6 +33,26 @@ GPT5_REASONING_EFFORT=medium
# - high: Detailed explanations
GPT5_VERBOSITY=medium
+# --- Chat Model Configuration (Optional) ---
+# Configure a separate model for real-time chat interactions.
+# If not set, chat will use the TEXT_MODEL_* settings above.
+#
+# Use cases:
+# - Use a faster model for chat while using a more capable model for summarization
+# - Use a cheaper model for interactive chat to reduce costs
+# - Use different service tiers for different operations
+#
+# CHAT_MODEL_API_KEY=your_chat_api_key
+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+# CHAT_MODEL_NAME=openai/gpt-4o
+
+# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---
+# These settings allow independent control of GPT-5 parameters for chat.
+# If not set, falls back to the main GPT5_* settings above.
+#
+# CHAT_GPT5_REASONING_EFFORT=medium
+# CHAT_GPT5_VERBOSITY=medium
+
# --- Transcription Service (OpenAI Whisper API) ---
TRANSCRIPTION_BASE_URL=https://api.openai.com/v1
TRANSCRIPTION_API_KEY=your_openai_api_key
diff --git a/config/env.whisperx.example b/config/env.whisperx.example
index af830353..28837f55 100644
--- a/config/env.whisperx.example
+++ b/config/env.whisperx.example
@@ -43,6 +43,26 @@ GPT5_REASONING_EFFORT=medium
# - high: Detailed explanations
GPT5_VERBOSITY=medium
+# --- Chat Model Configuration (Optional) ---
+# Configure a separate model for real-time chat interactions.
+# If not set, chat will use the TEXT_MODEL_* settings above.
+#
+# Use cases:
+# - Use a faster model for chat while using a more capable model for summarization
+# - Use a cheaper model for interactive chat to reduce costs
+# - Use different service tiers for different operations
+#
+# CHAT_MODEL_API_KEY=your_chat_api_key
+# CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+# CHAT_MODEL_NAME=openai/gpt-4o
+
+# --- Chat GPT-5 Settings (only used with OpenAI API and GPT-5 chat models) ---
+# These settings allow independent control of GPT-5 parameters for chat.
+# If not set, falls back to the main GPT5_* settings above.
+#
+# CHAT_GPT5_REASONING_EFFORT=medium
+# CHAT_GPT5_VERBOSITY=medium
+
# --- Transcription Service (WhisperX ASR Endpoint) ---
# Setting this to true enables all ASR features including speaker diarization and voice profiles
# Note: File chunking is NOT supported with ASR endpoints
diff --git a/docs/admin-guide/model-configuration.md b/docs/admin-guide/model-configuration.md
index 0ac608a9..c4326874 100644
--- a/docs/admin-guide/model-configuration.md
+++ b/docs/admin-guide/model-configuration.md
@@ -209,6 +209,101 @@ For most use cases, we recommend:
This provides a good balance of cost, speed, and quality.
+## Separate Chat Model Configuration
+
+Speakr allows you to configure a separate model specifically for real-time chat interactions, while using a different model for background tasks like summarization and title generation. This enables you to:
+
+- **Use different service tiers**: Configure a faster, more expensive model for interactive chat while using a cheaper model for background processing
+- **Optimize costs**: Use a budget-friendly model for summarization while keeping a high-quality model for chat
+- **Balance speed and quality**: Prioritize low latency for chat while accepting slower processing for summaries
+
+### Configuration
+
+Add these optional environment variables to your `.env` file:
+
+```bash
+# Chat Model Configuration (Optional)
+# If not set, chat will use TEXT_MODEL_* settings
+CHAT_MODEL_API_KEY=your_chat_api_key
+CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+CHAT_MODEL_NAME=openai/gpt-4o
+```
+
+### Fallback Behavior
+
+| Configuration | Behavior |
+|--------------|----------|
+| No `CHAT_MODEL_*` variables set | Chat uses `TEXT_MODEL_*` settings (default) |
+| Only `CHAT_MODEL_NAME` set | Falls back to `TEXT_MODEL_*` (API key required) |
+| Only `CHAT_MODEL_API_KEY` set | Falls back to `TEXT_MODEL_*` (model name required) |
+| `CHAT_MODEL_API_KEY` + `CHAT_MODEL_NAME` set | Uses chat config with `TEXT_MODEL_BASE_URL` |
+| All `CHAT_MODEL_*` variables set | Uses fully dedicated chat configuration |
+
+### GPT-5 Settings for Chat
+
+If you use GPT-5 models for chat, you can configure separate GPT-5 parameters:
+
+```bash
+# Chat-specific GPT-5 settings (optional)
+# Falls back to GPT5_* settings if not specified
+CHAT_GPT5_REASONING_EFFORT=medium
+CHAT_GPT5_VERBOSITY=medium
+```
+
+### Example Configurations
+
+**Cheap Summarization + Premium Chat**:
+```bash
+# Background tasks: Use budget model
+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+TEXT_MODEL_API_KEY=your_openrouter_key
+TEXT_MODEL_NAME=openai/gpt-4o-mini
+
+# Interactive chat: Use premium model
+CHAT_MODEL_API_KEY=your_openai_key
+CHAT_MODEL_BASE_URL=https://api.openai.com/v1
+CHAT_MODEL_NAME=gpt-5-mini
+CHAT_GPT5_REASONING_EFFORT=low
+CHAT_GPT5_VERBOSITY=medium
+```
+
+**Same Provider, Different Models**:
+```bash
+# Background tasks: Smaller model
+TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+TEXT_MODEL_API_KEY=your_api_key
+TEXT_MODEL_NAME=google/gemini-2.5-flash-lite
+
+# Interactive chat: Larger model (same provider)
+CHAT_MODEL_NAME=openai/gpt-4o
+# Note: CHAT_MODEL_API_KEY not needed if using same key
+# Note: CHAT_MODEL_BASE_URL not needed if using same endpoint
+```
+
+**Different Service Tiers (OpenAI)**:
+```bash
+# Background tasks: Standard tier
+TEXT_MODEL_BASE_URL=https://api.openai.com/v1
+TEXT_MODEL_API_KEY=your_standard_tier_key
+TEXT_MODEL_NAME=gpt-4o-mini
+
+# Interactive chat: Priority tier for faster responses
+CHAT_MODEL_API_KEY=your_priority_tier_key
+CHAT_MODEL_NAME=gpt-4o
+```
+
+### When to Use Separate Chat Models
+
+**Recommended for**:
+- High-volume deployments where chat responsiveness is critical
+- Users who need different service tiers for different operations
+- Cost optimization when chat usage is significantly higher than summarization
+
+**Not needed for**:
+- Small deployments with low usage
+- When using the same model for all operations is acceptable
+- Simple setups where configuration simplicity is preferred
+
## Model Selection Guidelines
### For Summaries
@@ -268,6 +363,15 @@ CHAT_MAX_TOKENS=2000
# GPT-5 specific (only used with GPT-5 models and OpenAI API)
GPT5_REASONING_EFFORT=medium # minimal, low, medium, high
GPT5_VERBOSITY=medium # low, medium, high
+
+# Chat model configuration (optional - falls back to TEXT_MODEL_* if not set)
+CHAT_MODEL_API_KEY=your_chat_api_key
+CHAT_MODEL_BASE_URL=https://openrouter.ai/api/v1
+CHAT_MODEL_NAME=openai/gpt-4o
+
+# Chat-specific GPT-5 settings (optional - falls back to GPT5_* if not set)
+CHAT_GPT5_REASONING_EFFORT=medium # minimal, low, medium, high
+CHAT_GPT5_VERBOSITY=medium # low, medium, high
```
## Troubleshooting
diff --git a/docs/faq.md b/docs/faq.md
index 85e3503f..c03bc9a4 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -104,6 +104,16 @@ There's no hard limit on recording length, but practical considerations apply. V
Summary generation uses the language model configured in your [environment file](getting-started.md#step-3-configure-your-transcription-service). Customize summaries with [AI prompts](admin-guide/prompts.md) - through a local LLM endpoint or a Cloud provider like OpenAI or OpenRouter. The model choice affects [summary quality](features.md#automatic-summarization), cost, and processing speed. Monitor performance in [system statistics](admin-guide/statistics.md).
+### Can I use different models for chat and summaries?
+
+Yes! You can configure a separate model specifically for real-time chat interactions while using a different model for background tasks like summarization. This is useful for:
+
+- **Different service tiers**: Use a faster, premium model for interactive chat while using a budget model for background processing
+- **Cost optimization**: Use cheaper models for summarization while keeping high-quality responses for chat
+- **Speed optimization**: Prioritize low latency for chat while accepting slower processing for summaries
+
+Configure this using the optional `CHAT_MODEL_*` environment variables. See the [Model Configuration](admin-guide/model-configuration.md#separate-chat-model-configuration) guide for details.
+
## Privacy and Security
### Is my data really private?
diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md
index 2dc010a3..5b070bca 100644
--- a/docs/getting-started/installation.md
+++ b/docs/getting-started/installation.md
@@ -78,6 +78,8 @@ TEXT_MODEL_NAME=openai/gpt-4o-mini
If you prefer to use OpenAI directly for text generation, simply change the base URL to `https://api.openai.com/v1` and use your OpenAI API key. You can also use local models through Ollama or LM Studio by pointing to `http://localhost:11434/v1` or similar.
+> **Tip:** For advanced model configuration options—including GPT-5 support, separate chat model settings for different service tiers, and cost optimization strategies—see the [Model Configuration](../admin-guide/model-configuration.md) guide.
+
Next, configure the transcription service. This is what converts your audio files into text:
```bash
diff --git a/src/api/inquire.py b/src/api/inquire.py
index cb6b53fb..643f0a25 100644
--- a/src/api/inquire.py
+++ b/src/api/inquire.py
@@ -17,7 +17,7 @@ from src.database import db
from src.models import *
from src.utils import *
from src.services.embeddings import get_accessible_recording_ids, semantic_search_chunks
-from src.services.llm import call_llm_completion, process_streaming_with_thinking, client
+from src.services.llm import call_llm_completion, call_chat_completion, process_streaming_with_thinking, client, chat_client
# Create blueprint
inquire_bp = Blueprint('inquire', __name__)
@@ -631,16 +631,16 @@ Order your response with notes from the most recent meetings first. Always use p
messages.append({"role": "user", "content": user_message})
# Enable streaming
- stream = call_llm_completion(
+ stream = call_chat_completion(
messages=messages,
temperature=0.7,
max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),
stream=True
)
-
+
# Buffer content to detect full transcript requests
response_buffer = ""
-
+
# Buffer content to detect full transcript requests
response_buffer = ""
content_buffer = ""
@@ -692,8 +692,8 @@ Order your response with notes from the most recent meetings first. Always use p
# Generate new response with full context
yield create_status_response('responding', 'Analyzing full transcript...')
-
- new_stream = call_llm_completion(
+
+ new_stream = call_chat_completion(
messages=updated_messages,
temperature=0.7,
max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),
diff --git a/src/api/recordings.py b/src/api/recordings.py
index 4954ac24..78952994 100644
--- a/src/api/recordings.py
+++ b/src/api/recordings.py
@@ -28,7 +28,7 @@ from src.services.speaker import update_speaker_usage, identify_unidentified_spe
from src.services.speaker_embedding_matcher import update_speaker_embedding
from src.services.speaker_snippets import create_speaker_snippets
from src.services.document import process_markdown_to_docx
-from src.services.llm import client, call_llm_completion, process_streaming_with_thinking
+from src.services.llm import client, chat_client, call_llm_completion, call_chat_completion, process_streaming_with_thinking
from src.services.embeddings import process_recording_chunks
from src.file_exporter import export_recording, mark_export_as_deleted
@@ -2630,9 +2630,9 @@ def chat_with_transcription():
if not has_recording_access(recording, current_user):
return jsonify({'error': 'You do not have permission to chat with this recording'}), 403
- # Check if OpenRouter client is available
- if client is None:
- return jsonify({'error': 'Chat service is not available (OpenRouter client not configured)'}), 503
+ # Check if chat client is available
+ if chat_client is None:
+ return jsonify({'error': 'Chat service is not available (chat client not configured)'}), 503
# Prepare the system prompt with the transcription
user_chat_output_language = current_user.output_language if current_user.is_authenticated else None
@@ -2678,7 +2678,7 @@ Additional context and notes about the meeting:
def generate():
try:
# Enable streaming
- stream = call_llm_completion(
+ stream = call_chat_completion(
messages=messages,
temperature=0.7,
max_tokens=int(os.environ.get("CHAT_MAX_TOKENS", "2000")),
diff --git a/src/services/__init__.py b/src/services/__init__.py
index cd5ad305..4aea0def 100644
--- a/src/services/__init__.py
+++ b/src/services/__init__.py
@@ -22,6 +22,8 @@ __all__ = [
'is_gpt5_model',
'is_using_openai_api',
'call_llm_completion',
+ 'call_chat_completion',
+ 'chat_client',
'format_api_error_message',
# Document services
'process_markdown_to_docx',
diff --git a/src/services/llm.py b/src/services/llm.py
index 2225eec5..d654c08c 100644
--- a/src/services/llm.py
+++ b/src/services/llm.py
@@ -21,6 +21,41 @@ if TEXT_MODEL_BASE_URL:
TEXT_MODEL_BASE_URL = TEXT_MODEL_BASE_URL.split('#')[0].strip()
TEXT_MODEL_NAME = os.environ.get("TEXT_MODEL_NAME", "openai/gpt-3.5-turbo")
+# Chat model configuration (optional - falls back to TEXT_MODEL_* if not set)
+CHAT_MODEL_API_KEY = os.environ.get("CHAT_MODEL_API_KEY")
+CHAT_MODEL_BASE_URL = os.environ.get("CHAT_MODEL_BASE_URL")
+if CHAT_MODEL_BASE_URL:
+ CHAT_MODEL_BASE_URL = CHAT_MODEL_BASE_URL.split('#')[0].strip()
+CHAT_MODEL_NAME = os.environ.get("CHAT_MODEL_NAME")
+
+# Chat-specific GPT-5 settings (optional - falls back to main GPT5_* settings)
+CHAT_GPT5_REASONING_EFFORT = os.environ.get("CHAT_GPT5_REASONING_EFFORT")
+CHAT_GPT5_VERBOSITY = os.environ.get("CHAT_GPT5_VERBOSITY")
+
+
+def get_chat_config():
+ """
+ Get chat model configuration, falling back to TEXT_MODEL if not set.
+
+ Returns a dict with api_key, base_url, model_name, and GPT-5 settings.
+ """
+ if CHAT_MODEL_API_KEY and CHAT_MODEL_NAME:
+ return {
+ 'api_key': CHAT_MODEL_API_KEY,
+ 'base_url': CHAT_MODEL_BASE_URL or TEXT_MODEL_BASE_URL,
+ 'model_name': CHAT_MODEL_NAME,
+ 'gpt5_reasoning_effort': CHAT_GPT5_REASONING_EFFORT or os.environ.get("GPT5_REASONING_EFFORT", "medium"),
+ 'gpt5_verbosity': CHAT_GPT5_VERBOSITY or os.environ.get("GPT5_VERBOSITY", "medium")
+ }
+ return {
+ 'api_key': TEXT_MODEL_API_KEY,
+ 'base_url': TEXT_MODEL_BASE_URL,
+ 'model_name': TEXT_MODEL_NAME,
+ 'gpt5_reasoning_effort': os.environ.get("GPT5_REASONING_EFFORT", "medium"),
+ 'gpt5_verbosity': os.environ.get("GPT5_VERBOSITY", "medium")
+ }
+
+
# Set up HTTP client with custom headers for OpenRouter app identification
app_headers = {
"HTTP-Referer": "https://github.com/murtaza-nasir/speakr",
@@ -44,6 +79,25 @@ try:
except Exception as client_init_e:
client = None
+# Create chat client (may be same as main client if no separate config)
+chat_client = None
+try:
+ chat_config = get_chat_config()
+ if chat_config['api_key']:
+ if CHAT_MODEL_API_KEY and CHAT_MODEL_API_KEY != TEXT_MODEL_API_KEY:
+ # Separate chat configuration - create dedicated client
+ chat_client = OpenAI(
+ api_key=chat_config['api_key'],
+ base_url=chat_config['base_url'],
+ http_client=http_client_no_proxy
+ )
+ logger.info(f"Separate chat client initialized: {chat_config['base_url']} / {chat_config['model_name']}")
+ else:
+ # Use same client as main LLM
+ chat_client = client
+except Exception as chat_client_init_e:
+ logger.warning(f"Failed to initialize chat client, falling back to main client: {chat_client_init_e}")
+ chat_client = client
def is_gpt5_model(model_name):
@@ -149,7 +203,73 @@ def call_llm_completion(messages, temperature=0.7, response_format=None, stream=
logger.error(f"LLM API call failed: {e}")
raise
-# Store details for the transcription client (potentially different)
+
+def call_chat_completion(messages, temperature=0.7, response_format=None, stream=False, max_tokens=None):
+ """
+ Chat-specific LLM completion function. Uses dedicated chat model if configured,
+ otherwise falls back to standard TEXT_MODEL configuration.
+
+ Args:
+ messages: List of message dicts with 'role' and 'content'
+ temperature: Sampling temperature (0-1) - ignored for GPT-5 models
+ response_format: Optional response format dict (e.g., {"type": "json_object"})
+ stream: Whether to stream the response
+ max_tokens: Optional maximum tokens to generate
+
+ Returns:
+ OpenAI completion object or generator (if streaming)
+ """
+ effective_client = chat_client if chat_client else client
+ chat_config = get_chat_config()
+
+ if not effective_client:
+ raise ValueError("Chat LLM client not initialized")
+
+ if not chat_config['api_key']:
+ raise ValueError("Chat model API key not configured")
+
+ try:
+ model_name = chat_config['model_name']
+ base_url = chat_config['base_url'] or ''
+
+ # Check if we're using GPT-5 with OpenAI API
+ using_gpt5 = is_gpt5_model(model_name) and 'api.openai.com' in base_url
+
+ completion_args = {
+ "model": model_name,
+ "messages": messages,
+ "stream": stream
+ }
+
+ if using_gpt5:
+ logger.debug(f"Using GPT-5 chat model: {model_name}")
+ # Use chat-specific GPT-5 settings from config
+ completion_args["reasoning_effort"] = chat_config['gpt5_reasoning_effort']
+ completion_args["verbosity"] = chat_config['gpt5_verbosity']
+
+ if max_tokens:
+ completion_args["max_completion_tokens"] = max_tokens
+ else:
+ completion_args["temperature"] = temperature
+ if max_tokens:
+ completion_args["max_tokens"] = max_tokens
+
+ if response_format:
+ completion_args["response_format"] = response_format
+
+ response = effective_client.chat.completions.create(**completion_args)
+
+ # Debug log for empty responses
+ if not stream and response.choices:
+ content = response.choices[0].message.content
+ if not content:
+ logger.warning(f"Chat LLM returned empty content. Model: {model_name}, finish_reason: {response.choices[0].finish_reason}")
+
+ return response
+
+ except Exception as e:
+ logger.error(f"Chat LLM API call failed: {e}")
+ raise
def format_api_error_message(error_str):