Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

4.4 KB
Raw Blame

Benchmarks for Local Deep Research

This directory contains scripts for running benchmarks to evaluate Local Deep Research's performance.

Available Benchmarks

SimpleQA

The SimpleQA benchmark evaluates factual question answering capabilities.

python run_simpleqa.py --examples 10 --iterations 3 --questions 3

Options:

  • --examples: Number of examples to run (default: 10)
  • --iterations: Number of search iterations (default: 3)
  • --questions: Questions per iteration (default: 3)
  • --search-tool: Search tool to use (default: "searxng")
  • --output-dir: Directory to save results (default: "benchmark_results", relative to the LDR data directory; see Where results are written)
  • --no-eval: Skip evaluation
  • --human-eval: Use human evaluation
  • --eval-model: Model to use for evaluation
  • --eval-provider: Provider to use for evaluation

BrowseComp

The BrowseComp benchmark evaluates web browsing comprehension and complex question answering.

python run_browsecomp.py --examples 5 --iterations 3 --questions 3

Options:

  • --examples: Number of examples to run (default: 2)
  • --iterations: Number of search iterations (default: 1)
  • --questions: Questions per iteration (default: 1)
  • --search-tool: Search tool to use (default: "searxng")
  • --output-dir: Directory to save results (default: "browsecomp_results", relative to the current directory; this script writes its files itself)

See browsecomp_benchmark_readme.md for more information on how BrowseComp works.

Running All Benchmarks

To run both benchmarks and compare results:

# Run SimpleQA with default settings
python run_simpleqa.py

# Run BrowseComp with increased iterations and questions
python run_browsecomp.py --iterations 3 --questions 3

Where results are written

Scripts that hand an output directory to the library's benchmark runner (evaluate_simpleqa, evaluate_browsecomp, run_simpleqa_benchmark, run_browsecomp_benchmark or compare_configurations) get those benchmark files below the LDR data directory (LDR_DATA_DIR, or the platform default). A relative directory is placed relative to the data directory, not the current directory. A directory outside the data directory, such as the project-root benchmark_results/<timestamp> paths some scripts build, is replaced by benchmark_results there, with a warning in the log. If that default is not usable either, for example because it is a symlink to another disk, the run stops with an error before it starts.

This applies to run_simpleqa.py, run_gemini_benchmark.py and gemini/run_gemini_benchmark_fixed.py, which call evaluate_simpleqa and evaluate_browsecomp, and to ../run_benchmark.py, ../optimization/run_gemini_benchmark.py and ../optimization/run_parallel_benchmark.py. The report and results paths these functions return are the real locations. A script's own summary JSON and its "Results saved to" line still use the directory the script created, which may then hold only that summary.

claude_grading/benchmark.py, scripts/run_benchmark_with_claude_grading.py and scripts/run_focused_benchmark_fixed.py call the benchmark evaluators directly, on their own and through CompositeBenchmarkEvaluator, so their output is split. The SimpleQA evaluator's default path writes its results, evaluation and report files itself, into a simpleqa subdirectory of the directory it is given, so they stay in the script's own directory tree with its summary JSON (for example ./benchmark_results/claude_grading_<timestamp>/simpleqa/simpleqa). The BrowseComp evaluator creates its browsecomp subdirectory there too, but run_browsecomp_benchmark writes its benchmark files below the data directory, so that subdirectory stays empty. Parameter optimizer runs (../optimization/) split the same way and keep the optimizer's own study files in the directory they are given.

run_browsecomp.py, run_resumable_parallel_benchmark.py, browsecomp/run_browsecomp_fixed_v2.py and scripts/run_grader_only.py call quick_summary, grade_results or generate_report themselves and write every file to the paths they are given.

Evaluating Results

Results are saved in the output directories described above and include:

  • Raw results (JSONL format)
  • Evaluation results (JSONL format)
  • Summary reports (Markdown format)

The scripts will also print a summary of the results to the console, including accuracy metrics.