Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

3.5 KB
Raw Blame

tests/performance — live-service tests & eval harnesses

Code here hits real external services (arXiv, OpenAlex, Ollama, etc.) and measures end-to-end behaviour. Everything here is excluded from CI (.github/workflows/docker-tests.yml runs -m 'not integration' for pytest, and the non-test_*.py eval scripts aren't collected by pytest at all).

Layout

tests/performance/
├── _shared/                   — general pipeline harnesses, not tied to one subsystem
│   ├── run_full_search.py     — thin CLI around quick_summary(); --engine / --model flags
│   └── build_eval_dataset.py  — cross-product runner (query × engine × model) over run_full_search.py
├── relevance_filter/          — LLM-as-judge relevance filter (live arXiv + Ollama)
│   ├── test_live.py           — pytest live arXiv + Ollama test (integration + requires_llm + slow)
│   ├── eval_prompt.py         — human-judgment harness: vary prompt, same judge, same arXiv
│   └── eval_models.py         — human-judgment harness: vary judge, same prompt, same arXiv
├── strategies/                — decomposition / iterative-reasoning strategies (live LLM + meta_search)
│   └── compare_strategies_visual.py — human-judgment harness: cross-strategy timeline plots
├── content_fetcher/           — HTML extraction quality across 200+ real URLs (live network)
├── search_engines/            — new-adapter integration against live APIs (Open Library, Zenodo, etc.)
├── mcp/                       — MCP client concurrency (real subprocess echo server)
├── database/                  — encrypted-DB backwards-compat (installs previous PyPI release)
└── api_auth/                  — authenticated research API validation (requires running server + Puppeteer)

When you add performance tests / harnesses for a new subsystem (e.g. embeddings, rate limiting, search-engine latency), create a new sibling folder alongside relevance_filter/. Put anything that's genuinely subsystem-agnostic in _shared/.

Running the pytest tests

LDR_TESTING_WITH_MOCKS=false \
LDR_TEST_OLLAMA_BASE_URL=http://localhost:11434 \
LDR_TEST_OLLAMA_MODEL=qwen3.5:9b \
pdm run pytest tests/performance/ -v -s -m integration

LDR_TESTING_WITH_MOCKS=false is required — tests/conftest.py defaults it to true which auto-skips anything tagged requires_llm.

Pytest tests should:

  • carry @pytest.mark.integration (and usually @pytest.mark.requires_llm and @pytest.mark.slow) so they stay out of CI
  • skip cleanly when the required external service is unreachable rather than fail
  • print their KEPT/REMOVED decisions (or equivalent detail) with print() — run with -s to see them — so humans reviewing output can judge quality, not just pass/fail

Running the eval harnesses

The eval_*.py, run_*.py, build_*.py scripts are human-judgment tools — success signal is a human reading the output, not a pass/fail assertion. They're runnable as plain Python:

# Swap between prompt variants against the same live arXiv results
pdm run python tests/performance/relevance_filter/eval_prompt.py

# Swap between judge LLMs, same prompt
pdm run python tests/performance/relevance_filter/eval_models.py

# One-shot end-to-end report
pdm run python tests/performance/_shared/run_full_search.py \
    --query "LLM interpretability latest research" --engine arxiv

# Cross-product dataset build
pdm run python tests/performance/_shared/build_eval_dataset.py