AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): split the BM25 stats probe budget by cost class — long queries stop falling back to per-segment df/avgdl (#1186)
xerj-orgc3795f7f4e79
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 04b8b13c9..636df6251 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **Long BM25 queries silently fell back to per-segment IDF/avgdl**+ (issue [#1186](https://github.com/xerj-org/xerj/issues/1186)) — the+ index-wide statistics pre-pass (#188) budgeted `(fields + terms) ×+ segments ≤ 4096`, so a many-term query on a multi-segment index crossed+ the cap, the pre-pass silently declined, and every segment scored+ against its own `df`/`avgdl`: padding a query with 1,500 terms that+ match NO document reordered the whole top-5 (reported `1-1:1.6797521`+ → `8-4:1.6904112`; after `_forcemerge?max_num_segments=1` both queries+ agreed — the merge had nothing to do with it, it just brought the+ probe count back under the cap). The budget is now split by cost+ class: field probes (one mmap'd FST per segment × field — the wide+ `query_string` explosion the cap existed for) stay at 4096, term+ probes (one FST seek per segment × term — the same order of cost+ execution itself pays for postings) get their own 1,048,576 budget, and+ every decline now logs a warning instead of changing the ranking+ silently. The reporter's literal repro (1,502 terms over a 32-core+ box's ~128 flush segments, ≈192k term probes) and a 4-shard/32-segment+ variant both now score identically to their unpadded forms,+ bit-for-bit.+- **`xerj code --mode hybrid` dropped lexical-only indices from the BM25leg** (issue [#1146](https://github.com/xerj-org/xerj/issues/1146)) — ona corpus where only some indices map `body` as `semantic_text`, the fuseddiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex bf9062be6..467d6679e 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -27566,13 +27566,39 @@ impl Index {// silently reverting every segment to per-segment BM25 statistics.pinned: Option<PinnedIds<'_>>,) -> Option<xerj_fts::CollectionStats> {- // Widest (field × term) pre-pass we will pay for. A field-less- // `query_string` over a wide mapping can project hundreds of leaves;- // past this point the FST seeks stop being cheap relative to the- // search itself, and falling back to per-arm statistics is the- // conservative choice (same "decline rather than over-serve"- // convention as `MAX_QS_CROSS_PRODUCT`).- const MAX_STATS_PROBES: usize = 4096;+ // Widest pre-pass we will pay for, budgeted per COST CLASS (#1186):+ //+ // * FIELD probes — `open_stats_only` mmaps one FST per (segment,+ // field) and `field_stats` walks its meta. A field-less+ // `query_string` over a wide mapping can project hundreds of+ // leaves, and that product (fields × segments) is the explosion+ // the original 4096 combined cap existed to stop. Unchanged.+ //+ // * TERM probes — one FST seek per (segment, term). These were+ // lumped into the same 4096 cap, which had a correctness cost the+ // field case does not: a long `match` query (an agent pasting a+ // blob, a 1502-term repro) crossed the cap on any multi-segment+ // index, the pre-pass silently declined, and every segment fell+ // back to its own df/avgdl — padding a query with terms that+ // match NOTHING reordered the whole top-5 (#1186). Execution+ // itself already pays ~one postings seek per (segment, term), so+ // the pre-pass at most doubles a cost the query is already+ // incurring: Lucene collects `TermStates` for every clause it+ // executes and never falls back to per-leaf statistics (its+ // bound is the clause count, which refuses the query instead of+ // re-scoring it). The term budget below only exists so a+ // pathological million-term query cannot make the pre-pass+ // dominate a search that execution would itself refuse. The bar+ // is set at 2^20 probes (~1–2 s of FST seeks) — a query must+ // exceed a MILLION (term × segment) products to hit it, i.e. a+ // 1,500-term query on a ~700-segment index; anything the engine+ // answers in sane time keeps index-wide statistics, and when the+ // budget DOES decline it says so on the log instead of silently+ // changing the ranking. (The reporter's literal repro — 1,502+ // terms on a 32-core box's ~128 flush segments, ≈192k probes —+ // sits comfortably inside.)+ const MAX_STATS_FIELD_PROBES: usize = 4096;+ const MAX_STATS_TERM_PROBES: usize = 1_048_576;let fq = query_node_to_fts_projected(query,@@ -27611,7 +27637,27 @@ impl Index {}}- if (fields.len() + pairs.len()) * snap.segments.len().max(1) > MAX_STATS_PROBES {+ let segs = snap.segments.len().max(1);+ if fields.len() * segs > MAX_STATS_FIELD_PROBES {+ tracing::warn!(+ fields = fields.len(),+ segments = segs,+ "search: BM25 collection statistics declined ({}) field probes > \+ {MAX_STATS_FIELD_PROBES} — segments will score against their own \+ df/avgdl and scores will not be comparable across segments",+ fields.len() * segs+ );+ return None;+ }+ if pairs.len() * segs > MAX_STATS_TERM_PROBES {+ tracing::warn!(+ terms = pairs.len(),+ segments = segs,+ "search: BM25 collection statistics declined ({}) term probes > \+ {MAX_STATS_TERM_PROBES} — segments will score against their own \+ df/avgdl and scores will not be comparable across segments",+ pairs.len() * segs+ );return None;}@@ -27623,7 +27669,15 @@ impl Index {else {// A segment whose statistics we cannot read would skew the// union low and silently inflate every other arm's IDF.- // Fall back to per-arm statistics rather than half a union.+ // Fall back to per-arm statistics rather than half a union —+ // and SAY so (#1186: every decline from index-wide stats is+ // a silent ranking change, never a quiet local decision).+ tracing::warn!(+ segment = %meta.id,+ "search: BM25 collection statistics declined — segment stats \+ unreadable; segments will score against their own df/avgdl and \+ scores will not be comparable across segments"+ );return None;};for f in &fields {diff --git a/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs b/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rsindex e02195781..ed4b76f3f 100644--- a/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs+++ b/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs@@ -51,6 +51,17 @@ fn make_engine(dir: &TempDir, ingest_shards: usize) -> Engine {Engine::new(config).expect("engine::new")}+/// [`make_engine`] with tiny-segment merging pinned off (#1186's `nomerge`+/// config): the tiered merge otherwise collapses the many small segments the+/// probe-budget test needs, mid-test and timing-dependent.+fn make_engine_nomerge(dir: &TempDir, ingest_shards: usize) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ config.engine.ingest_shards = ingest_shards;+ config.merge.min_merge_count = 100_000;+ Engine::new(config).expect("engine::new")+}+/// One buried occurrence of the term in a LONG field that grows with `i` —/// low BM25, strictly decreasing.fn weak_body(i: usize) -> String {@@ -287,3 +298,90 @@ async fn deletes_do_not_break_the_union() {);}}++/// #1186 — padding a query with terms that match NOTHING must return the+/// same hits with the same scores as the unpadded query.+///+/// A term with df=0 contributes 0 to every document's BM25 score, so padding+/// is score-neutral — PROVIDED both queries are scored against the same+/// index-wide statistics. It stopped being neutral the moment the padded+/// query crossed the stats pre-pass probe budget+/// `(fields + pairs) * segments > 4096`: the pre-pass silently returned+/// `None`, every segment fell back to its OWN df/avgdl, and the whole top-5+/// reordered (reported: top hit `1-1:1.6797521` → `8-4:1.6904112`; after+/// `_forcemerge?max_num_segments=1` both queries agreed again — proof the+/// reorder was the stats fallback, not the extra terms).+///+/// The corpus mirrors the reporter's: 8 flush cycles of 20 docs each on a+/// 4-shard engine (a segment per non-empty shard per flush) with merging+/// pinned off. The padded query projects 1 field + 1502 terms over ~32+/// segments ≈ 48k probes — far past the old 4096 combined cap (where the+/// ranking changed) and well inside the split per-term budget (where both+/// queries are scored against the same union).+#[tokio::test]+async fn padding_a_query_with_terms_that_match_nothing_never_reorders_results() {+ let dir = TempDir::new().unwrap();+ let engine = make_engine_nomerge(&dir, 4);+ engine.create_index("t", Schema::empty()).unwrap();+ let idx = engine.get_index("t").unwrap();++ // Per-batch term frequencies differ (batch b carries b `alpha`s and b+ // `beta`s), so per-segment statistics genuinely differ from index-wide+ // ones — a silent fallback MOVES scores, it does not just rescale them.+ for b in 1..=8 {+ for i in 1..=20 {+ let alphas = "alpha ".repeat(b);+ let beta = if i <= b { " beta" } else { "" };+ idx.index_document(+ Some(format!("{b}-{i}")),+ json!({"body": format!("doc {i} batch {b} gamma{alphas}{beta}")}),+ )+ .await+ .unwrap();+ }+ idx.flush().await.unwrap();+ }++ let segments = idx.stats().await.segment_count;+ assert!(+ segments >= 3,+ "sanity: need ≥3 segments for the padded query ({pairs} pairs × \+ {segments}) to cross the old 4096-probe cap; got {segments}",+ pairs = 1 + 1502+ );++ let unpadded = parse_request(&json!({+ "size": 5, "_source": false,+ "query": {"match": {"body": "alpha beta"}}+ }))+ .expect("parse_request");+ let pads = (1..=1500)+ .map(|i| format!("zqx{i}"))+ .collect::<Vec<_>>()+ .join(" ");+ let padded = parse_request(&json!({+ "size": 5, "_source": false,+ "query": {"match": {"body": format!("alpha beta {pads}")}}+ }))+ .expect("parse_request");++ let plain = idx.search(&unpadded).await.unwrap();+ let padded_r = idx.search(&padded).await.unwrap();++ assert_eq!(+ plain.total.value, padded_r.total.value,+ "terms that match nothing must not change hits.total"+ );+ let top = |hits: &[xerj_query::Hit]| {+ hits.iter()+ .map(|h| (h.id.clone(), h.score.to_bits()))+ .collect::<Vec<_>>()+ };+ assert_eq!(+ top(&plain.hits),+ top(&padded_r.hits),+ "padding the query with 1500 terms that match no document reordered \+ the top-5 or changed a score — the stats pre-pass declined and every \+ segment silently scored against its own df/avgdl (#1186)"+ );+}