AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fts: WAND fast path for should-bool term disjunctions — 2.0x on the history vote, byte-identical results
xerj-orgcf6a998912bf
diff --git a/ROADMAP.md b/ROADMAP.mdindex e9861550f..e31f0e69d 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -106,7 +106,7 @@ The 1.0 bar: **every public claim verified against the release binary, and every**Zero-token** is the name for work XERJ does locally, inside the engine, so that a person or an agent spends no model tokens *finding*, *judging*, *watching for* or *handing over* an answer. Search already works that way — no model runs to answer a query. This section is the rest of that idea, as product work (long form: [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)), with each item's status checked against `main` on 2026-09-18 (`4d8dadbf`, 32 commits after `v1.0.0-rc.74`) by reading the code named beside it. "In flight" means a pushed branch exists and nothing of it is on `main`; "planned" means no code exists. Tracking issue: [#941](https://github.com/xerj-org/xerj/issues/941).-**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` now):+**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` as of v1.0.0-rc.75):- **Judged search — a rerank stage.** Retrieve with BM25 or `hybrid`, then have a second stage re-judge the top *N* before the page is returned. *On `main` since rc.75:* the opt-in `rerank` block on `_search`, inert until an operator configures a key; a LOCAL cross-encoder provider was measured and did NOT beat the shipped hybrid (SciFact 0.7021 hybrid vs 0.7186 local-base, interval spanning zero; NFCorpus 0.3445 vs 0.3597 local-small), so it stays opt-in and unmerged. *Before rc.75:* the ES `rescore` block (query rescorer) and the `hybrid` query type with `rrf|linear`; there is no rerank stage. *In flight:* branch `feat/rerank-stage` — a `rerank` block on `_search` that hands the top hits to an external relevance judge, opt-in per request and inert until an operator configures a key. It is the only *search-time* feature that would send document text off the machine — proxy embeddings (`[embedding] default_endpoint`) send it at write time and the WAL tap replays writes to an external `_bulk` endpoint, both operator-configured and off by default ([every outbound connection](./docs/RERANK.md#every-way-data-leaves-a-xerj-node)) — which is exactly why a *local* judge is on this list. *The bar for a local model is a measured one.* On the BEIR test splits with the opt-in neural embedder (`--embed-mode neural`, all-MiniLM-L6-v2, CPU) the shipped `hybrid` RRF query is already the best arm we have — nDCG@10 **0.699–0.704 on SciFact and 0.345 on NFCorpus** over three runs, against 0.657 / 0.302 for BM25 alone, 0.676 / 0.329 for vectors alone, and 0.686 / 0.332 for "BM25 top-30 re-ordered by the same bi-encoder", which is *worse* than hybrid on both. So re-ordering with the embedder we already ship is not worth building, and **a local cross-encoder ships only if it beats 0.699 / 0.345 on the same harness, by more than the run-to-run spread** ([`benchmarks/neural-path-triage/`](./benchmarks/neural-path-triage/)). Those figures are with the neural embedder; the default embedder is lexical feature hashing and was not part of this run. The hybrid figure is the only arm that does not reproduce — 0.6993, 0.7023 and 0.7044 on SciFact across three runs of unchanged indices — because tied RRF scores are ordered by a per-process hash seed ([#940](https://github.com/xerj-org/xerj/issues/940)); a gain inside that spread is not a gain.- **Share links and a guest reading room.** Hand one person a link to one corpus or one document, read-only, without creating them an account. *On `main` since rc.75:* `xerj share` with scoped, expiring links, a guest reading room, and a `--tunnel` mode that names Cloudflare as a reader of the traffic. *Before rc.75:* nothing — there was no share or guest code in the tree, and reading a corpus needs an account session or an API key. *In flight:* branches `feat/share-links` (scoped, expiring links; a share can never name a system index) and `feat/console-corpus-reader` (the corpus browser and the reader the link opens). Both branches carry commits their authors marked unverified; they are not claims yet.diff --git a/engine/crates/xerj-fts/src/search.rs b/engine/crates/xerj-fts/src/search.rsindex aae6b60ec..7f4b24491 100644--- a/engine/crates/xerj-fts/src/search.rs+++ b/engine/crates/xerj-fts/src/search.rs@@ -128,6 +128,19 @@ impl TopN {self.total}+ /// The worst hit currently RETAINED (the heap's max by `Ord` = the+ /// next eviction victim), or `None` while the heap is below `cap` (or+ /// `cap == 0`). Read by the WAND walk's pruning check: a candidate+ /// whose score upper bound is below this cannot be admitted, so it+ /// never needs scoring at all.+ pub fn worst(&self) -> Option<&ScoredHit> {+ if self.cap == 0 || self.heap.len() < self.cap {+ None+ } else {+ self.heap.peek()+ }+ }+/// Consume the collector: `(best-first sorted hits, exact total)`.pub fn finish(self) -> (Vec<ScoredHit>, u64) {let total = self.total;@@ -592,11 +605,249 @@ impl FtsSearcher {cap: usize,explain: bool,) -> Result<(Vec<ScoredHit>, u64)> {+ // WAND fast path for the shape a `match`/`multi_match` query becomes+ // after planning (`bool{should: [Term..]}`, nothing else). Only when+ // nothing downstream needs the generic walk: `explain` needs every+ // retained hit's breakdown, and `cap == usize::MAX` callers want the+ // complete match set.+ if !explain && cap != usize::MAX {+ if let Some(res) = self.wand_should_bool(query, cap)? {+ return Ok(res);+ }+ }// The no-op observer monomorphises away, so this stays byte-identical to// the pre-#179 bounded search on the hot (no-delete) path.self.search_bounded_observed(query, cap, explain, |_| {})}+ /// Doc-at-a-time top-`cap` evaluation for the exact shape a `match` /+ /// `multi_match` query becomes after planning: a `bool` with ONLY+ /// `should` clauses, every clause a plain [`TermQuery`], effective+ /// `min_should_match == 1`. Returns `None` for any other shape so the+ /// caller falls back to the generic path unchanged.+ ///+ /// Why this exists: the generic path materialises a `Vec<ScoredHit>` per+ /// clause, folds them through a `HashMap<u32, u32>` match counter plus a+ /// `HashMap<u32, f32>` score map, collects the union, and sorts it — fine+ /// for rare terms, quadratic-feeling for stopwords. A three-stopword+ /// `match` over a 120k-doc index (~180k postings, ~100k distinct docs)+ /// spent ~28 ms there, almost all of it per-posting work that could not+ /// change the answer: BM25 evaluation + a `field_length` lookup for docs+ /// that never had a chance of entering a size-10 page.+ ///+ /// Design: WAND-style upper-bound pruning (Broder et al., *Faster Top-k+ /// Document Retrieval Using Block-Max Indexes*; the same dynamic-pruning+ /// idea as tantivy's `block_wand`, MIT — quickwit-oss/tantivy+ /// `src/query/boolean_query/block_wand_union.rs`, whose block-max half+ /// needs impact-ordered postings and skip tables this postings format+ /// does not carry). The walk still visits EVERY matching doc — postings+ /// have no skip lists, and the caller needs an EXACT total — but a doc+ /// is scored only when the sum of its matched terms' score upper bounds+ /// (`idf × (k1+1) × boost`, the supremum of `score_term` over any tf/dl)+ /// can still enter the heap. For stopword-only docs that bound sum sits+ // below the 10th-best score almost immediately, so the entire scoring+ /// cost for them disappears; what remains per skipped doc is one+ /// cursor compare and one `next()`.+ ///+ /// Output identity with the generic path, by construction:+ /// - **Scores are bit-identical**: the matched terms are summed in+ /// CLAUSE order — the same order `execute_bool`'s `score_map`+ /// accumulates in — then multiplied by `bq.boost`, so every f32+ /// addition happens in the same sequence.+ /// - **Ordering is identical**: hits pass through the same [`TopN`] /+ /// [`ScoredHit`] `Ord`. A doc is skipped only when its upper bound is+ /// STRICTLY below the worst retained score, so a tie-score doc with a+ /// smaller `doc_id` (which `Ord` ranks better and `TopN` would admit)+ /// is never pruned.+ /// - **The total is identical**: every matching doc is visited and+ /// counted, `[TopN::total]` or not.+ ///+ /// Deadline: the walk polls every 4 096 docs; stopping early yields a+ /// subset of the true hits and an undercount — the same partial-implies-+ /// narrower contract `execute_bool`'s `run_clauses` already upholds for+ /// abandoned `should` clauses, surfaced as `timed_out: true` by+ /// [`Self::deadline_tripped`].+ fn wand_should_bool(+ &self,+ query: &Query,+ cap: usize,+ ) -> Result<Option<(Vec<ScoredHit>, u64)>> {+ let Query::Bool(bq) = query else {+ return Ok(None);+ };+ if bq.must.is_empty()+ && bq.filter.is_empty()+ && bq.must_not.is_empty()+ && !bq.should.is_empty()+ && bq.min_should_match.unwrap_or(1) == 1+ {+ // All clauses must be plain terms (the common `match` shape);+ // anything richer — phrases, nested bools, wildcards — keeps the+ // generic path.+ for q in &bq.should {+ if !matches!(q, Query::Term(_)) {+ return Ok(None);+ }+ }+ } else {+ return Ok(None);+ }++ // One cursor per clause, mirroring `scan_term`'s term-lookup and+ // reader construction (positions decoded per the field's real+ // layout and simply never read — the format is positions-dependent,+ // so a docs-only reader would misparse the stream).+ struct WandTerm<'a, 'b> {+ /// Clause index — the f32 summation order (see above) and the+ /// cursor-order tiebreak.+ ord: usize,+ tq: &'b TermQuery,+ bm25: crate::bm25::Bm25Scorer,+ /// The `doc_freq` the generic path would score with+ /// (`scoring_df`, i.e. index-wide when collection stats are on).+ score_df: u64,+ reader: PostingsReader<'a>,+ doc: u32,+ tf: u32,+ /// Supremum of this term's contribution over any (tf, dl):+ /// `idf × (k1+1) × boost`. `tf_norm < k1+1` always, so this+ /// never underestimates `score_term`.+ ubound: f32,+ }+ let mut scorers: Vec<WandTerm> = Vec::with_capacity(bq.should.len());+ for (ord, q) in bq.should.iter().enumerate() {+ // Cooperative deadline on the clause fan-out axis, mirroring+ // `run_clauses` (poll every 64 clauses): a projected disjunction+ // can carry tens of thousands of clauses, and dropping a should+ // clause only narrows the answer — the partial-implies-narrower+ // contract `execute_bool` already upholds.+ if ord & 63 == 0 && self.deadline_hit() {+ break;+ }+ let Query::Term(tq) = q else { unreachable!("checked above") };+ let Some(tp) = self.reader.lookup_term(&tq.field, &tq.term) else {+ // Term absent from this segment: the clause matches nothing+ // and contributes no score — exactly the generic path's+ // empty per-clause `Vec`.+ continue;+ };+ let Some(post_data) = self.reader.postings_data(&tq.field, &tp) else {+ continue;+ };+ // The postings byte format is positions-dependent: a docs-only+ // reader would synthesise tf=1 and misparse a positions-bearing+ // stream. Read the field's real layout, exactly like+ // `scan_term`; the decoded positions are simply never read.+ let has_positions = self.reader.field_has_positions(&tq.field);+ let mut reader =+ PostingsReader::new_with_positions(post_data, tp.doc_frequency, has_positions);+ let Some(first) = reader.next() else { continue };+ let bm25 = self.make_scorer(&tq.field);+ let score_df = self.scoring_df(&tq.field, &tq.term, tp.doc_frequency as u64);+ let ubound = bm25.idf(score_df) * (bm25.k1 + 1.0) * tq.boost;+ scorers.push(WandTerm {+ ord,+ tq,+ bm25,+ score_df,+ reader,+ doc: first.doc_id,+ tf: first.term_freq,+ ubound,+ });+ }+ // Cursor order: (doc, ord). Deterministic, and keeps the scorers+ // sitting on the same doc contiguous.+ scorers.sort_by_key(|s| (s.doc, s.ord));+ if scorers.is_empty() {+ return Ok(Some((Vec::new(), 0)));+ }++ let mut top = TopN::new(cap);+ let mut total: u64 = 0;+ let mut since_poll: u32 = 0;+ loop {+ let d = scorers[0].doc;+ let k = scorers.iter().take_while(|s| s.doc == d).count();++ // Skip-or-score. While the heap is below `cap` there is no bar+ // to clear (and `cap == 0` retains nothing, so never score).+ // Once full, prune only STRICTLY below the worst retained score:+ // at equality the candidate could tie the worst score and win+ // the `doc_id` tiebreak, which `TopN` would admit — so equality+ // must still be scored.+ let score_it = match top.worst() {+ None => cap > 0,+ Some(worst) => {+ let bound: f32 = scorers[..k].iter().map(|s| s.ubound).sum();+ bound >= worst.score+ }+ };+ if score_it {+ // Sum the matched terms in CLAUSE order (bit-identical to+ // `execute_bool`'s `score_map` accumulation), then apply the+ // bool boost exactly where the generic path applies it.+ // `scorers[..k]` is reordered by `ord` IN PLACE — ords are+ // unique, so the unstable sort is deterministic, and no+ // per-doc buffer is allocated (a heap alloc per scored doc+ // was measurable at 120k-doc disjunctions).+ scorers[..k].sort_unstable_by_key(|s| s.ord);+ let first_field = scorers[0].tq.field.as_str();+ let first_len = self.reader.field_length(first_field, d).unwrap_or(1) as u32;+ let mut sum = 0.0f32;+ for s in &scorers[..k] {+ // Clauses usually share one field (`match`) — look the+ // length up once; only multi-field queries pay per field.+ let doc_len = if s.tq.field == first_field {+ first_len+ } else {+ self.reader.field_length(&s.tq.field, d).unwrap_or(1) as u32+ };+ sum += s.bm25.score_term(s.score_df, s.tf, doc_len) * s.tq.boost;+ }+ top.push(ScoredHit {+ doc_id: d,+ score: sum * bq.boost,+ explanation: None,+ });+ }++ total += 1;++ // Advance every scorer sitting on `d` — exactly `scorers[..k]`+ // (the sort invariant makes the at-d cursors a prefix).+ // Exhausted cursors are marked `u32::MAX`, which sorts last, and+ // pop off after the re-sort. `sort_unstable` never allocates the+ // scratch buffer a stable sort spills to on >20-element slices —+ // this re-sort runs once per DOC, so that alloc was per-doc too.+ for i in 0..k {+ match scorers[i].reader.next() {+ Some(next) => {+ scorers[i].doc = next.doc_id;+ scorers[i].tf = next.term_freq;+ }+ None => scorers[i].doc = u32::MAX,+ }+ }+ scorers.sort_unstable_by_key(|s| (s.doc, s.ord));+ while scorers.last().is_some_and(|s| s.doc == u32::MAX) {+ scorers.pop();+ }+ if scorers.is_empty() {+ break;+ }++ // Cooperative deadline, polled like the expansion loops: partial+ // = subset of hits + undercount, latched for `timed_out: true`.+ since_poll += 1;+ if since_poll & 4095 == 0 && self.deadline_hit() {+ break;+ }+ }+ let (hits, _) = top.finish();+ Ok(Some((hits, total)))+ }+/// Like [`Self::search_bounded`], but invokes `observe(doc_id)` once for/// EVERY match in the full match set — before the `cap` is applied — so a/// caller can derive an exact per-match statistic without materialising the@@ -3064,4 +3315,303 @@ mod tests {}}}++ /// Corpus sized to make the WAND bound actually bite: one term in every+ /// document (low idf → a tiny score upper bound) beside one term in three+ /// documents (high idf → the only scores that can fill a page).+ ///+ /// Returns the searcher.+ fn wand_fixture(dir: &std::path::Path) -> FtsSearcher {+ let registry = Arc::new(AnalyzerRegistry::default());+ let mut writer = FtsIndexWriter::new(dir, "seg0", Arc::clone(®istry));+ writer.configure_field(+ "body",+ FieldIndexConfig {+ analyzer: "whitespace".to_owned(),+ ..Default::default()+ },+ );+ for i in 0..300u32 {+ // Docs 0..3 carry "needle"; every doc carries "the". Lengths vary+ // so `field_length` lookups differ across the walk.+ let text = if i < 3 {+ format!("the needle pad{i}")+ } else {+ format!("the pad{i} filler{}", i % 7)+ };+ let fields: HashMap<String, FieldValues> =+ [("body".to_owned(), FieldValues::from(text))].into_iter().collect();+ writer.add_document(i, &fields);+ }+ writer.finish().unwrap();+ let reader = Arc::new(FtsIndexReader::open(dir, "seg0", &["body"]).unwrap());+ FtsSearcher::new(reader, registry)+ }++ /// The WAND path (`search_bounded` on a should-only bool of terms) must be+ /// indistinguishable from the generic path — same hits, same order,+ /// **bit-identical scores**, same exact total — on a corpus where the+ /// pruning branch genuinely fires, and it must fire by construction, not+ /// by luck.+ ///+ /// The pruning precondition is ASSERTED, not assumed (the lesson of+ /// `bool_deadline_never_broadens_a_must_not`): once the page holds the+ /// three needle docs, a "the"-only doc can enter only if+ /// `idf(the)·(k1+1) ≥ worst-retained-score` — and this test fails loudly+ /// if that ever stops holding, because then the 297 pruning decisions it+ /// relies on would not have happened and the equivalence above would be+ /// passing vacuously over the same docs the legacy test already covers.+ #[test]+ fn wand_should_bool_matches_generic_path_when_pruning_fires() {+ let dir = TempDir::new().unwrap();+ let searcher = wand_fixture(dir.path());++ let q = Query::Bool(Box::new(+ BoolQuery::new()+ .should(Query::Term(TermQuery::new("body", "the")))+ .should(Query::Term(TermQuery::new("body", "needle"))),+ ));++ // Reference: usize::MAX never takes the WAND path, so this is the+ // generic walk's answer in full.+ let full = searcher.search(&q, usize::MAX, false).unwrap();+ assert_eq!(full.len(), 300, "every doc matches 'the'");+ for w in full.windows(2) {+ assert!(w[0].score >= w[1].score);+ }++ // --- the non-vacuity precondition ---+ // The three needle docs must be the page at cap=3, and the common+ // term's score supremum must sit strictly below that page's worst —+ // exactly the comparison the walk makes for each of the 297+ // "the"-only docs.+ assert_eq!(+ full[..3].iter().map(|h| h.doc_id).collect::<Vec<_>>(),+ vec![0, 1, 2],+ "the three needle docs must lead the ranking"+ );+ let bm25 = searcher.make_scorer("body");+ let the_df = searcher.scoring_df("body", "the", 300);+ let the_ubound = bm25.idf(the_df) * (bm25.k1 + 1.0);+ assert!(+ the_ubound < full[2].score,+ "precondition: 'the'-only docs (bound {the_ubound:.4}) must be prunable \+ against the cap-3 page (worst {:.4}) — otherwise this test exercises \+ no pruning",+ full[2].score+ );++ for cap in [0usize, 1, 2, 3, 5, 300, 305] {+ let (bounded, total) = searcher.search_bounded(&q, cap, false).unwrap();+ assert_eq!(total, 300, "exact total must survive pruning at cap={cap}");+ let expected: Vec<(u32, u32)> = full+ .iter()+ .take(cap)+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ let got: Vec<(u32, u32)> = bounded+ .iter()+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ assert_eq!(+ got, expected,+ "WAND top-{cap} must be byte-identical to the generic path"+ );+ }+ }++ /// Clause shapes that must NOT take the WAND path (`min_should_match > 1`,+ /// `must` beside `should`, a non-`Term` should clause) still get correct+ /// bounded answers — the shape guard falls them through to the generic+ /// walk, and this pins that the fallthrough, not just the guard, is right.+ #[test]+ fn wand_should_bool_falls_back_for_richer_shapes() {+ let dir = TempDir::new().unwrap();+ let searcher = wand_fixture(dir.path());++ let shapes: Vec<(&str, Query)> = vec![+ (+ "min_should_match=2",+ Query::Bool(Box::new(+ BoolQuery::new()+ .should(Query::Term(TermQuery::new("body", "the")))+ .should(Query::Term(TermQuery::new("body", "needle")))+ .min_should_match(2),+ )),+ ),+ (+ "must + should",+ Query::Bool(Box::new(+ BoolQuery::new()+ .must(Query::Term(TermQuery::new("body", "needle")))+ .should(Query::Term(TermQuery::new("body", "the"))),+ )),+ ),+ (+ "non-Term should clause",+ Query::Bool(Box::new(+ BoolQuery::new()+ .should(Query::Term(TermQuery::new("body", "the")))+ .should(Query::Prefix(PrefixQuery::new("body", "needl"))),+ )),+ ),+ ];++ for (name, q) in &shapes {+ let full = searcher.search(q, usize::MAX, false).unwrap();+ assert!(!full.is_empty(), "{name}: fixture must match something");+ for cap in [0usize, 1, 2, 3, 10, full.len(), full.len() + 5] {+ let (bounded, total) = searcher.search_bounded(q, cap, false).unwrap();+ assert_eq!(+ total,+ full.len() as u64,+ "{name}: total at cap={cap}"+ );+ let expected: Vec<(u32, u32)> = full+ .iter()+ .take(cap)+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ let got: Vec<(u32, u32)> = bounded+ .iter()+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ assert_eq!(got, expected, "{name}: top-{cap}");+ }+ }+ }++ /// The WAND path's own clause bookkeeping on the awkward-but-legal edges:+ /// a DUPLICATE term (two cursors over the same postings, both must land in+ /// the clause-order sum), a boosted term (the bound must carry the boost+ /// or pruning would over-prune), an absent term (dropped like the generic+ /// path's empty per-clause vec), and a bool-level boost (applied exactly+ /// where the generic path applies it).+ #[test]+ fn wand_should_bool_duplicate_boosted_absent_and_bool_boost() {+ let dir = TempDir::new().unwrap();+ let searcher = wand_fixture(dir.path());++ let mut tq_dup = TermQuery::new("body", "needle");+ tq_dup.boost = 2.0;+ let q = Query::Bool(Box::new(+ BoolQuery::new()+ .should(Query::Term(tq_dup))+ // duplicate clause over the same postings, default boost+ .should(Query::Term(TermQuery::new("body", "needle")))+ .should(Query::Term(TermQuery::new("body", "the")))+ // absent from every doc+ .should(Query::Term(TermQuery::new("body", "absentterm")))+ .boost(1.5),+ ));++ let full = searcher.search(&q, usize::MAX, false).unwrap();+ assert_eq!(full.len(), 300, "the still matches everything");+ // The duplicate must actually double-count: each needle doc's score+ // here exceeds the same doc's single-needle score by roughly the+ // single contribution (boost 2.0 + 1.0 = 3× vs 1×).+ let single = Query::Bool(Box::new(+ BoolQuery::new().should(Query::Term(TermQuery::new("body", "needle"))),+ ));+ let single_hits = searcher.search(&single, usize::MAX, false).unwrap();+ let s0 = single_hits.iter().find(|h| h.doc_id == 0).unwrap().score;+ let d0 = full.iter().find(|h| h.doc_id == 0).unwrap().score;+ assert!(+ d0 > s0 * 4.0,+ "duplicate + 2.0 boost + 1.5 bool boost must compound: {d0:.4} vs {s0:.4}"+ );++ for cap in [0usize, 1, 2, 3, 7, 300] {+ let (bounded, total) = searcher.search_bounded(&q, cap, false).unwrap();+ assert_eq!(total, 300, "total at cap={cap}");+ let expected: Vec<(u32, u32)> = full+ .iter()+ .take(cap)+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ let got: Vec<(u32, u32)> = bounded+ .iter()+ .map(|h| (h.doc_id, h.score.to_bits()))+ .collect();+ assert_eq!(got, expected, "top-{cap} with duplicate/boosted/absent clauses");+ }+ }++ /// Micro-bench (ignored by default; run with `--ignored`): the WAND path+ /// vs the generic bounded walk on a stopword-heavy disjunction over a+ /// 200k-doc synthetic index — the shape that motivated the fast path.+ /// `WAND_BENCH="clauses,cap"` (default "3,10") selects the clause count+ /// and page size; `WAND_BENCH="1,0"` measures the pure walk (nothing ever+ /// scored). Asserts result equivalence once so the timing cannot silently+ /// drift onto unequal answers.+ #[test]+ #[ignore]+ fn bench_wand_vs_generic_stopword_disjunction() {+ let cfg = std::env::var("WAND_BENCH").unwrap_or_else(|_| "3,10".to_owned());+ let (nclauses, cap): (usize, usize) = {+ let mut it = cfg.split(',');+ (+ it.next().unwrap_or("3").parse().unwrap(),+ it.next().unwrap_or("10").parse().unwrap(),+ )+ };+ let dir = tempfile::TempDir::new().unwrap();+ let registry = Arc::new(AnalyzerRegistry::default());+ let mut writer = FtsIndexWriter::new(dir.path(), "seg0", Arc::clone(®istry));+ writer.configure_field(+ "body",+ FieldIndexConfig {+ analyzer: "whitespace".to_owned(),+ ..Default::default()+ },+ );+ const N: u32 = 200_000;+ let stops = ["the", "of", "and", "to", "in", "is"];+ for i in 0..N {+ // Terms of full df (the "stopwords"), a graded mid-df term, and+ // unique padding for realistic lengths.+ let mid = if i % 4 == 0 { "mid" } else { "pad" };+ let text = format!("{} {mid} u{i}", stops[..nclauses.min(stops.len())].join(" "));+ let fields: HashMap<String, FieldValues> =+ [("body".to_owned(), FieldValues::from(text))].into_iter().collect();+ writer.add_document(i, &fields);+ }+ writer.finish().unwrap();+ let reader = Arc::new(FtsIndexReader::open(dir.path(), "seg0", &["body"]).unwrap());+ let searcher = FtsSearcher::new(reader, registry);++ let mut bq = BoolQuery::new();+ for t in &stops[..nclauses.min(stops.len())] {+ bq = bq.should(Query::Term(TermQuery::new("body", *t)));+ }+ let q = Query::Bool(Box::new(bq));+ const REPS: u32 = 20;+ let _ = searcher.search_bounded(&q, cap, false).unwrap(); // warm+ let mut wand_us = 0f64;+ for _ in 0..REPS {+ let t0 = std::time::Instant::now();+ let _ = searcher.search_bounded(&q, cap, false).unwrap();+ wand_us += t0.elapsed().as_secs_f64() * 1e6;+ }+ let (g_hits, g_total) = searcher.search_bounded_observed(&q, cap, false, |_| {}).unwrap();+ let mut gen_us = 0f64;+ for _ in 0..REPS {+ let t0 = std::time::Instant::now();+ let _ = searcher+ .search_bounded_observed(&q, cap, false, |_| {})+ .unwrap();+ gen_us += t0.elapsed().as_secs_f64() * 1e6;+ }+ let (w_hits, w_total) = searcher.search_bounded(&q, cap, false).unwrap();+ assert_eq!(w_total, g_total);+ assert_eq!(+ w_hits.iter().map(|h| (h.doc_id, h.score.to_bits())).collect::<Vec<_>>(),+ g_hits.iter().map(|h| (h.doc_id, h.score.to_bits())).collect::<Vec<_>>(),+ );+ println!(+ "{nclauses}-stopword disjunction over {N} docs (df={N} each), cap={cap}: \+ generic {gen_us:.0} µs vs WAND {wand_us:.0} µs ({:.1}×), total={w_total}",+ gen_us / wand_us+ );+ }}