~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Implementation Guide for Custom Context Window Size
This document outlines the steps to implement the custom context window size feature requested in issue #241.
Background
Users have reported issues with models that have smaller context windows (like some Llama.cpp and LM Studio models) failing because the default token limit (30,000) exceeds their capabilities.
Issue: Custom context size #241
Implementation Steps
1. Add New Setting
Add a new setting in the default settings JSON file:
"llm.context_window_size": {
"category": "llm_parameters",
"description": "Maximum context window size in tokens for the LLM",
"editable": true,
"max_value": 128000.0,
"min_value": 512.0,
"name": "Context Window Size",
"options": null,
"step": null,
"type": "LLM",
"ui_element": "number",
"value": 32000,
"visible": true
}
2. Modify the LLM Configuration
Update the get_llm function in src/local_deep_research/config/llm_config.py:
- Add context window size to common parameters:
# Common parameters for all models
common_params = {
"temperature": temperature,
}
# Get context window size from settings
context_window_size = get_db_setting("llm.context_window_size", 32000)
# Ensure max_tokens doesn't exceed context window size
if get_db_setting("llm.supports_max_tokens", True):
# Use 80% of context window to leave room for prompts
max_tokens = min(get_db_setting("llm.max_tokens", 30000), int(context_window_size * 0.8))
common_params["max_tokens"] = max_tokens
- Add specific handling for models that need context window size parameter:
For example, in the LlamaCpp section:
# For LlamaCpp, also set context size directly if supported
if hasattr(LlamaCpp, "n_ctx"): # Check if the parameter exists
llm = LlamaCpp(
model_path=model_path,
temperature=temperature,
max_tokens=max_tokens,
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
f16_kv=f16_kv,
n_ctx=context_window_size, # Set context window size
verbose=True,
)
3. Update UI to Make Context Size Visible
The UI should already display the setting once it's added to the default settings JSON.
4. Update Documentation
Add documentation explaining:
- What the context window size setting does
- How it affects different model providers
- Recommended values for common models
Testing Approach
- Test with different providers (especially Llama.cpp and LM Studio)
- Verify models with smaller context windows work correctly
- Ensure max_tokens is calculated correctly based on context window size
- Check for any regressions in models that worked previously
Impact Assessment
This change:
- Allows users to customize context window size for their specific models
- Helps prevent errors when using models with smaller context windows
- Makes the software more adaptable to different LLM capabilities
- Is backwards compatible with existing configurations
Example Usage
Example user workflow:
- User sets up a Llama.cpp model with 4K context window size
- User changes the context window size setting to 4096
- System automatically adjusts max_tokens to 3277 (80% of 4096)
- Model works properly without "context too long" errors
Potential Future Enhancements
- Auto-detection of model context window size
- Provider-specific context window settings
- Dynamic adjustment based on model characteristics