Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

4.3 KB
Raw Blame

LDR Benchmarking Framework

The Local Deep Research (LDR) Benchmarking Framework allows you to evaluate and compare LDR's performance on standardized benchmarks.

Features

  • Run benchmarks on SimpleQA and BrowseComp datasets
  • Configure search parameters (iterations, questions per iteration, search tool)
  • Use automated evaluation with Claude 3.7 Sonnet (default) or other models
  • Generate detailed reports with metrics and examples
  • Compare multiple configurations to find optimal settings
  • API, CLI, and web interface integration

Benchmark Datasets

SimpleQA

A straightforward question-answering benchmark with factual questions. This benchmark tests LDR's ability to find and synthesize factual information.

BrowseComp

A web browsing comprehension benchmark with more complex questions requiring synthesis across multiple sources. This benchmark tests LDR's ability to understand and navigate complex information needs.

Usage

Programmatic API

from local_deep_research.api.benchmark_functions import evaluate_simpleqa

# Run SimpleQA benchmark with 20 examples
result = evaluate_simpleqa(
    num_examples=20,
    search_iterations=3,
    questions_per_iteration=3,
    search_tool="searxng"
)

# Print accuracy
print(f"Accuracy: {result['metrics']['accuracy']:.3f}")

Command Line Interface

# Run SimpleQA benchmark
python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa --examples 20 --iterations 3

# Run BrowseComp benchmark
python -m local_deep_research.benchmarks.cli.benchmark_commands browsecomp --examples 10 --search-tool wikipedia

# Compare configurations
python -m local_deep_research.benchmarks.cli.benchmark_commands compare --dataset simpleqa --examples 5

Web Interface

The benchmark dashboard is available at /benchmark in the LDR web interface. You can:

  1. Select a benchmark to run
  2. Configure parameters
  3. Run the benchmark
  4. View results and reports

Evaluation

By default, benchmarks are evaluated using Claude 3.7 Sonnet via OpenRouter. You can customize the evaluation model:

# Use a different model for evaluation
result = evaluate_simpleqa(
    num_examples=10,
    evaluation_model="gpt-4o",
    evaluation_provider="openai"
)

You can also use human evaluation:

# Use human evaluation
result = evaluate_simpleqa(
    num_examples=5,
    human_evaluation=True
)

Configuration Comparison

Compare multiple configurations to find optimal settings:

from local_deep_research.api.benchmark_functions import compare_configurations

# Define configurations to compare
configurations = [
    {
        "name": "Base Config",
        "search_tool": "searxng",
        "iterations": 1,
        "questions_per_iteration": 3
    },
    {
        "name": "More Iterations",
        "search_tool": "searxng",
        "iterations": 3,
        "questions_per_iteration": 3
    },
    {
        "name": "Different Search Engine",
        "search_tool": "wikipedia",
        "iterations": 1,
        "questions_per_iteration": 3
    }
]

# Run comparison
result = compare_configurations(
    dataset_type="simpleqa",
    num_examples=10,
    configurations=configurations
)

Output Format

Benchmark results include:

  • metrics: Accuracy, processing time, confidence scores
  • report_path: Path to generated report
  • results_path: Path to raw results file
  • total_examples: Number of examples processed
  • status: Completion status

Example Reports

Reports include:

  • Overall accuracy and metrics
  • Configuration details
  • Example correct and incorrect answers
  • Time and date information

Integration with LDR Web App

The benchmarking framework is fully integrated with the LDR web interface. You can run benchmarks and view results directly in the web app.

Adding Custom Benchmarks

To add a custom benchmark:

  1. Create a dataset loader in datasets.py
  2. Add evaluation templates in templates.py
  3. Create benchmark runners in runners.py
  4. Expose the benchmark through the API in api/benchmark_functions.py

Performance Considerations

  • Running benchmarks can be resource-intensive
  • Start with a small number of examples for testing
  • Full benchmarks with 100+ examples may take several hours to complete
  • Consider using a more powerful machine for large benchmarks