Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

889 B
Raw Blame

Gemini Benchmark Examples

This directory contains example scripts for running benchmarks with Gemini models via OpenRouter.

Scripts Included

run_gemini_benchmark_fixed.py

A comprehensive benchmark script that runs both SimpleQA and BrowseComp evaluations using Google's Gemini 2.0 Flash model via the OpenRouter API.

Key features:

  • Patches the LLM configuration to use Gemini for all evaluations
  • Supports both SimpleQA and BrowseComp benchmarks
  • Properly handles result collection and reporting

Usage

To run the benchmark with Gemini:

# Run with default settings (1 example)
python run_gemini_benchmark_fixed.py

# Run with custom number of examples
python run_gemini_benchmark_fixed.py --examples 5

Notes

These scripts assume you have:

  1. An OpenRouter API key configured in your LDR database
  2. The correct access permissions for the Gemini model