Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

1.4 KB
Raw Blame

Claude API Grading Benchmark

This benchmark integrates Claude 3 Sonnet for grading benchmark results with proper API access through the local database.

Features

  • Uses Claude 3 Sonnet for grading benchmark results
  • Accesses API keys from the local database
  • Supports SimpleQA and BrowseComp benchmarks
  • Provides composite scoring with customizable weights
  • Comprehensive metrics and accuracy reports

Usage

From the project root directory:

# Run with default settings (source_based strategy, 1 iteration, 5 examples)
./examples/benchmarks/claude_grading/run_benchmark.sh

# Run with custom parameters
./examples/benchmarks/claude_grading/run_benchmark.sh --strategy source_based --iterations 2 --examples 200

How It Works

The benchmark integrates with the evaluation system by patching the grading module to use the local get_llm function, which properly retrieves API keys from the database and configures the Claude model for grading.

This approach ensures accurate grading of benchmark results and enables comparison between different strategies and configurations.

Requirements

  • Valid Claude API key stored in the local database
  • SearXNG search engine running locally
  • Python dependencies installed

Output

Results are saved in the benchmark_results directory with comprehensive metrics:

  • Accuracy scores
  • Processing times
  • Grading confidence
  • Detailed evaluation reports