~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
Gemini Benchmark Examples
This directory contains example scripts for running benchmarks with Gemini models via OpenRouter.
Scripts Included
run_gemini_benchmark_fixed.py
A comprehensive benchmark script that runs both SimpleQA and BrowseComp evaluations using Google's Gemini 2.0 Flash model via the OpenRouter API.
Key features:
- Patches the LLM configuration to use Gemini for all evaluations
- Supports both SimpleQA and BrowseComp benchmarks
- Properly handles result collection and reporting
Usage
To run the benchmark with Gemini:
# Run with default settings (1 example)
python run_gemini_benchmark_fixed.py
# Run with custom number of examples
python run_gemini_benchmark_fixed.py --examples 5
Notes
These scripts assume you have:
- An OpenRouter API key configured in your LDR database
- The correct access permissions for the Gemini model