Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

2.2 KB
Raw Blame

Quarto export measurements

Measure the public document conversion and full in-memory ZIP export while varying bibliography size and report text length independently. The ZIP timing includes compression and both archive writes. Corpus construction, exporter construction, archive decompression and correctness checks are outside timing; the export logger's sink is disabled. No Quarto/Pandoc process, network or model is used. This measures serialization, not rendering a document to HTML or PDF.

pdm run pytest tests/performance/quarto_export -n 0 -q \
  --quarto-profile local --quarto-report /tmp/quarto-export.json

CI uses sizes 4/8, two measured samples and one warmup. Local uses 128/256, seven samples and two warmups. --quarto-{profile,size,iterations,warmups,report, max-p95-ms} overrides LDR_PERF_QUARTO_*; LDR_PERF_PROFILE is the shared profile fallback. Timing ceilings are opt-in; CI always asserts correctness and work bounds. Native thread counts should remain one for comparisons.

Every iteration compares the complete expected document, bibliography, archive membership, filename and MIME type. The expected text is composed independently from the corpus parameters, except the from: reader line, which is the exporter's own reader format (its behaviour is pinned by the citation formatter contract tests); UTC date is checked separately. Both grouped and individual references must resolve while unlisted numbers remain prose. Negative controls alter document references, bibliography URLs or archive membership. Reused and concurrently shared exporters must observe each input without changing earlier results.

A wrapper around the real bibliography-entry parser counts calls and characters presented to it, adding small instrumentation overhead. The baseline bounds are two parses per document and three per ZIP, allowing future reductions. This is a count of parser invocations, not a claim that the parser touches each character once. Raw samples, corpus hashes and both loaded implementation hashes are retained by the shared measurement helper and automatic CI runner. Per-character p95 divides a whole-export statistic; it is not a character latency percentile.