~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
LDR Benchmarking Framework
The Local Deep Research (LDR) Benchmarking Framework allows you to evaluate and compare LDR's performance on standardized benchmarks.
Features
- Run benchmarks on SimpleQA and BrowseComp datasets
- Configure search parameters (iterations, questions per iteration, search tool)
- Use automated evaluation with Claude 3.7 Sonnet (default) or other models
- Generate detailed reports with metrics and examples
- Compare multiple configurations to find optimal settings
- API, CLI, and web interface integration
Benchmark Datasets
SimpleQA
A straightforward question-answering benchmark with factual questions. This benchmark tests LDR's ability to find and synthesize factual information.
BrowseComp
A web browsing comprehension benchmark with more complex questions requiring synthesis across multiple sources. This benchmark tests LDR's ability to understand and navigate complex information needs.
Usage
Programmatic API
from local_deep_research.api.benchmark_functions import evaluate_simpleqa
# Run SimpleQA benchmark with 20 examples
result = evaluate_simpleqa(
num_examples=20,
search_iterations=3,
questions_per_iteration=3,
search_tool="searxng"
)
# Print accuracy
print(f"Accuracy: {result['metrics']['accuracy']:.3f}")
Command Line Interface
# Run SimpleQA benchmark
python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa --examples 20 --iterations 3
# Run BrowseComp benchmark
python -m local_deep_research.benchmarks.cli.benchmark_commands browsecomp --examples 10 --search-tool wikipedia
# Compare configurations
python -m local_deep_research.benchmarks.cli.benchmark_commands compare --dataset simpleqa --examples 5
Web Interface
The benchmark dashboard is available at /benchmark in the LDR web interface. You can:
- Select a benchmark to run
- Configure parameters
- Run the benchmark
- View results and reports
Evaluation
By default, benchmarks are evaluated using Claude 3.7 Sonnet via OpenRouter. You can customize the evaluation model:
# Use a different model for evaluation
result = evaluate_simpleqa(
num_examples=10,
evaluation_model="gpt-4o",
evaluation_provider="openai"
)
You can also use human evaluation:
# Use human evaluation
result = evaluate_simpleqa(
num_examples=5,
human_evaluation=True
)
Configuration Comparison
Compare multiple configurations to find optimal settings:
from local_deep_research.api.benchmark_functions import compare_configurations
# Define configurations to compare
configurations = [
{
"name": "Base Config",
"search_tool": "searxng",
"iterations": 1,
"questions_per_iteration": 3
},
{
"name": "More Iterations",
"search_tool": "searxng",
"iterations": 3,
"questions_per_iteration": 3
},
{
"name": "Different Search Engine",
"search_tool": "wikipedia",
"iterations": 1,
"questions_per_iteration": 3
}
]
# Run comparison
result = compare_configurations(
dataset_type="simpleqa",
num_examples=10,
configurations=configurations
)
Output Format
Benchmark results include:
- metrics: Accuracy, processing time, confidence scores
- report_path: Path to generated report
- results_path: Path to raw results file
- total_examples: Number of examples processed
- status: Completion status
Example Reports
Reports include:
- Overall accuracy and metrics
- Configuration details
- Example correct and incorrect answers
- Time and date information
Integration with LDR Web App
The benchmarking framework is fully integrated with the LDR web interface. You can run benchmarks and view results directly in the web app.
Adding Custom Benchmarks
To add a custom benchmark:
- Create a dataset loader in
datasets.py - Add evaluation templates in
templates.py - Create benchmark runners in
runners.py - Expose the benchmark through the API in
api/benchmark_functions.py
Performance Considerations
- Running benchmarks can be resource-intensive
- Start with a small number of examples for testing
- Full benchmarks with 100+ examples may take several hours to complete
- Consider using a more powerful machine for large benchmarks