Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

3.5 KB
Raw Blame

Implementation Guide for Custom Context Window Size

This document outlines the steps to implement the custom context window size feature requested in issue #241.

Background

Users have reported issues with models that have smaller context windows (like some Llama.cpp and LM Studio models) failing because the default token limit (30,000) exceeds their capabilities.

Issue: Custom context size #241

Implementation Steps

1. Add New Setting

Add a new setting in the default settings JSON file:

"llm.context_window_size": {
    "category": "llm_parameters",
    "description": "Maximum context window size in tokens for the LLM",
    "editable": true,
    "max_value": 128000.0,
    "min_value": 512.0,
    "name": "Context Window Size",
    "options": null,
    "step": null,
    "type": "LLM",
    "ui_element": "number",
    "value": 32000,
    "visible": true
}

2. Modify the LLM Configuration

Update the get_llm function in src/local_deep_research/config/llm_config.py:

  1. Add context window size to common parameters:
# Common parameters for all models
common_params = {
    "temperature": temperature,
}

# Get context window size from settings
context_window_size = get_db_setting("llm.context_window_size", 32000)

# Ensure max_tokens doesn't exceed context window size
if get_db_setting("llm.supports_max_tokens", True):
    # Use 80% of context window to leave room for prompts
    max_tokens = min(get_db_setting("llm.max_tokens", 30000), int(context_window_size * 0.8))
    common_params["max_tokens"] = max_tokens
  1. Add specific handling for models that need context window size parameter:

For example, in the LlamaCpp section:

# For LlamaCpp, also set context size directly if supported
if hasattr(LlamaCpp, "n_ctx"):  # Check if the parameter exists
    llm = LlamaCpp(
        model_path=model_path,
        temperature=temperature,
        max_tokens=max_tokens,
        n_gpu_layers=n_gpu_layers,
        n_batch=n_batch,
        f16_kv=f16_kv,
        n_ctx=context_window_size,  # Set context window size
        verbose=True,
    )

3. Update UI to Make Context Size Visible

The UI should already display the setting once it's added to the default settings JSON.

4. Update Documentation

Add documentation explaining:

  1. What the context window size setting does
  2. How it affects different model providers
  3. Recommended values for common models

Testing Approach

  1. Test with different providers (especially Llama.cpp and LM Studio)
  2. Verify models with smaller context windows work correctly
  3. Ensure max_tokens is calculated correctly based on context window size
  4. Check for any regressions in models that worked previously

Impact Assessment

This change:

  • Allows users to customize context window size for their specific models
  • Helps prevent errors when using models with smaller context windows
  • Makes the software more adaptable to different LLM capabilities
  • Is backwards compatible with existing configurations

Example Usage

Example user workflow:

  1. User sets up a Llama.cpp model with 4K context window size
  2. User changes the context window size setting to 4096
  3. System automatically adjusts max_tokens to 3277 (80% of 4096)
  4. Model works properly without "context too long" errors

Potential Future Enhancements

  1. Auto-detection of model context window size
  2. Provider-specific context window settings
  3. Dynamic adjustment based on model characteristics