~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Benchmarks for Local Deep Research
This directory contains scripts for running benchmarks to evaluate Local Deep Research's performance.
Available Benchmarks
SimpleQA
The SimpleQA benchmark evaluates factual question answering capabilities.
python run_simpleqa.py --examples 10 --iterations 3 --questions 3
Options:
--examples: Number of examples to run (default: 10)--iterations: Number of search iterations (default: 3)--questions: Questions per iteration (default: 3)--search-tool: Search tool to use (default: "searxng")--output-dir: Directory to save results (default: "benchmark_results", relative to the LDR data directory; see Where results are written)--no-eval: Skip evaluation--human-eval: Use human evaluation--eval-model: Model to use for evaluation--eval-provider: Provider to use for evaluation
BrowseComp
The BrowseComp benchmark evaluates web browsing comprehension and complex question answering.
python run_browsecomp.py --examples 5 --iterations 3 --questions 3
Options:
--examples: Number of examples to run (default: 2)--iterations: Number of search iterations (default: 1)--questions: Questions per iteration (default: 1)--search-tool: Search tool to use (default: "searxng")--output-dir: Directory to save results (default: "browsecomp_results", relative to the current directory; this script writes its files itself)
See browsecomp_benchmark_readme.md for more information on how BrowseComp works.
Running All Benchmarks
To run both benchmarks and compare results:
# Run SimpleQA with default settings
python run_simpleqa.py
# Run BrowseComp with increased iterations and questions
python run_browsecomp.py --iterations 3 --questions 3
Where results are written
Scripts that hand an output directory to the library's benchmark runner
(evaluate_simpleqa, evaluate_browsecomp, run_simpleqa_benchmark,
run_browsecomp_benchmark or compare_configurations) get those benchmark
files below the LDR data directory (LDR_DATA_DIR, or the platform
default). A relative directory is placed relative to the data directory,
not the current directory. A directory outside the data directory, such as
the project-root benchmark_results/<timestamp> paths some scripts build,
is replaced by benchmark_results there, with a warning in the log. If that
default is not usable either, for example because it is a symlink to another
disk, the run stops with an error before it starts.
This applies to run_simpleqa.py, run_gemini_benchmark.py and
gemini/run_gemini_benchmark_fixed.py, which call evaluate_simpleqa and
evaluate_browsecomp, and to ../run_benchmark.py,
../optimization/run_gemini_benchmark.py and
../optimization/run_parallel_benchmark.py. The report and results paths
these functions return are the real locations. A script's own summary JSON
and its "Results saved to" line still use the directory the script created,
which may then hold only that summary.
claude_grading/benchmark.py, scripts/run_benchmark_with_claude_grading.py
and scripts/run_focused_benchmark_fixed.py call the benchmark evaluators
directly, on their own and through CompositeBenchmarkEvaluator, so their
output is split. The SimpleQA evaluator's default path writes its results,
evaluation and report files itself, into a simpleqa subdirectory of the
directory it is given, so they stay in the script's own directory tree with
its summary JSON (for example
./benchmark_results/claude_grading_<timestamp>/simpleqa/simpleqa). The
BrowseComp evaluator creates its browsecomp subdirectory there too, but
run_browsecomp_benchmark writes its benchmark files below the data
directory, so that subdirectory stays empty. Parameter optimizer runs
(../optimization/) split the same way and keep the optimizer's own study
files in the directory they are given.
run_browsecomp.py, run_resumable_parallel_benchmark.py,
browsecomp/run_browsecomp_fixed_v2.py and scripts/run_grader_only.py
call quick_summary, grade_results or generate_report themselves and
write every file to the paths they are given.
Evaluating Results
Results are saved in the output directories described above and include:
- Raw results (JSONL format)
- Evaluation results (JSONL format)
- Summary reports (Markdown format)
The scripts will also print a summary of the results to the console, including accuracy metrics.