~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
tests/performance — live-service tests & eval harnesses
Code here hits real external services (arXiv, OpenAlex, Ollama, etc.)
and measures end-to-end behaviour. Everything here is excluded from CI
(.github/workflows/docker-tests.yml runs -m 'not integration' for
pytest, and the non-test_*.py eval scripts aren't collected by
pytest at all).
Layout
tests/performance/
├── _shared/ — general pipeline harnesses, not tied to one subsystem
│ ├── run_full_search.py — thin CLI around quick_summary(); --engine / --model flags
│ └── build_eval_dataset.py — cross-product runner (query × engine × model) over run_full_search.py
├── relevance_filter/ — LLM-as-judge relevance filter (live arXiv + Ollama)
│ ├── test_live.py — pytest live arXiv + Ollama test (integration + requires_llm + slow)
│ ├── eval_prompt.py — human-judgment harness: vary prompt, same judge, same arXiv
│ └── eval_models.py — human-judgment harness: vary judge, same prompt, same arXiv
├── strategies/ — decomposition / iterative-reasoning strategies (live LLM + meta_search)
│ └── compare_strategies_visual.py — human-judgment harness: cross-strategy timeline plots
├── content_fetcher/ — HTML extraction quality across 200+ real URLs (live network)
├── search_engines/ — new-adapter integration against live APIs (Open Library, Zenodo, etc.)
├── mcp/ — MCP client concurrency (real subprocess echo server)
├── database/ — encrypted-DB backwards-compat (installs previous PyPI release)
└── api_auth/ — authenticated research API validation (requires running server + Puppeteer)
When you add performance tests / harnesses for a new subsystem (e.g.
embeddings, rate limiting, search-engine latency), create a new sibling
folder alongside relevance_filter/. Put anything that's genuinely
subsystem-agnostic in _shared/.
Running the pytest tests
LDR_TESTING_WITH_MOCKS=false \
LDR_TEST_OLLAMA_BASE_URL=http://localhost:11434 \
LDR_TEST_OLLAMA_MODEL=qwen3.5:9b \
pdm run pytest tests/performance/ -v -s -m integration
LDR_TESTING_WITH_MOCKS=false is required — tests/conftest.py defaults
it to true which auto-skips anything tagged requires_llm.
Pytest tests should:
- carry
@pytest.mark.integration(and usually@pytest.mark.requires_llmand@pytest.mark.slow) so they stay out of CI - skip cleanly when the required external service is unreachable rather than fail
- print their KEPT/REMOVED decisions (or equivalent detail) with
print()— run with-sto see them — so humans reviewing output can judge quality, not just pass/fail
Running the eval harnesses
The eval_*.py, run_*.py, build_*.py scripts are human-judgment
tools — success signal is a human reading the output, not a pass/fail
assertion. They're runnable as plain Python:
# Swap between prompt variants against the same live arXiv results
pdm run python tests/performance/relevance_filter/eval_prompt.py
# Swap between judge LLMs, same prompt
pdm run python tests/performance/relevance_filter/eval_models.py
# One-shot end-to-end report
pdm run python tests/performance/_shared/run_full_search.py \
--query "LLM interpretability latest research" --engine arxiv
# Cross-product dataset build
pdm run python tests/performance/_shared/build_eval_dataset.py