Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): split the BM25 stats probe budget by cost class — long queries stop falling back to per-segment df/avgdl (#1186)

xerj-orgc3795f7f4e79
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 04b8b13c9..636df6251 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **Long BM25 queries silently fell back to per-segment IDF/avgdl**
+ (issue [#1186](https://github.com/xerj-org/xerj/issues/1186)) — the
+ index-wide statistics pre-pass (#188) budgeted `(fields + terms) ×
+ segments ≤ 4096`, so a many-term query on a multi-segment index crossed
+ the cap, the pre-pass silently declined, and every segment scored
+ against its own `df`/`avgdl`: padding a query with 1,500 terms that
+ match NO document reordered the whole top-5 (reported `1-1:1.6797521`
+ → `8-4:1.6904112`; after `_forcemerge?max_num_segments=1` both queries
+ agreed — the merge had nothing to do with it, it just brought the
+ probe count back under the cap). The budget is now split by cost
+ class: field probes (one mmap'd FST per segment × field — the wide
+ `query_string` explosion the cap existed for) stay at 4096, term
+ probes (one FST seek per segment × term — the same order of cost
+ execution itself pays for postings) get their own 1,048,576 budget, and
+ every decline now logs a warning instead of changing the ranking
+ silently. The reporter's literal repro (1,502 terms over a 32-core
+ box's ~128 flush segments, ≈192k term probes) and a 4-shard/32-segment
+ variant both now score identically to their unpadded forms,
+ bit-for-bit.
+
- **`xerj code --mode hybrid` dropped lexical-only indices from the BM25
leg** (issue [#1146](https://github.com/xerj-org/xerj/issues/1146)) — on
a corpus where only some indices map `body` as `semantic_text`, the fused
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index bf9062be6..467d6679e 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -27566,13 +27566,39 @@ impl Index {
// silently reverting every segment to per-segment BM25 statistics.
pinned: Option<PinnedIds<'_>>,
) -> Option<xerj_fts::CollectionStats> {
- // Widest (field × term) pre-pass we will pay for. A field-less
- // `query_string` over a wide mapping can project hundreds of leaves;
- // past this point the FST seeks stop being cheap relative to the
- // search itself, and falling back to per-arm statistics is the
- // conservative choice (same "decline rather than over-serve"
- // convention as `MAX_QS_CROSS_PRODUCT`).
- const MAX_STATS_PROBES: usize = 4096;
+ // Widest pre-pass we will pay for, budgeted per COST CLASS (#1186):
+ //
+ // * FIELD probes — `open_stats_only` mmaps one FST per (segment,
+ // field) and `field_stats` walks its meta. A field-less
+ // `query_string` over a wide mapping can project hundreds of
+ // leaves, and that product (fields × segments) is the explosion
+ // the original 4096 combined cap existed to stop. Unchanged.
+ //
+ // * TERM probes — one FST seek per (segment, term). These were
+ // lumped into the same 4096 cap, which had a correctness cost the
+ // field case does not: a long `match` query (an agent pasting a
+ // blob, a 1502-term repro) crossed the cap on any multi-segment
+ // index, the pre-pass silently declined, and every segment fell
+ // back to its own df/avgdl — padding a query with terms that
+ // match NOTHING reordered the whole top-5 (#1186). Execution
+ // itself already pays ~one postings seek per (segment, term), so
+ // the pre-pass at most doubles a cost the query is already
+ // incurring: Lucene collects `TermStates` for every clause it
+ // executes and never falls back to per-leaf statistics (its
+ // bound is the clause count, which refuses the query instead of
+ // re-scoring it). The term budget below only exists so a
+ // pathological million-term query cannot make the pre-pass
+ // dominate a search that execution would itself refuse. The bar
+ // is set at 2^20 probes (~1–2 s of FST seeks) — a query must
+ // exceed a MILLION (term × segment) products to hit it, i.e. a
+ // 1,500-term query on a ~700-segment index; anything the engine
+ // answers in sane time keeps index-wide statistics, and when the
+ // budget DOES decline it says so on the log instead of silently
+ // changing the ranking. (The reporter's literal repro — 1,502
+ // terms on a 32-core box's ~128 flush segments, ≈192k probes —
+ // sits comfortably inside.)
+ const MAX_STATS_FIELD_PROBES: usize = 4096;
+ const MAX_STATS_TERM_PROBES: usize = 1_048_576;
let fq = query_node_to_fts_projected(
query,
@@ -27611,7 +27637,27 @@ impl Index {
}
}
- if (fields.len() + pairs.len()) * snap.segments.len().max(1) > MAX_STATS_PROBES {
+ let segs = snap.segments.len().max(1);
+ if fields.len() * segs > MAX_STATS_FIELD_PROBES {
+ tracing::warn!(
+ fields = fields.len(),
+ segments = segs,
+ "search: BM25 collection statistics declined ({}) field probes > \
+ {MAX_STATS_FIELD_PROBES} — segments will score against their own \
+ df/avgdl and scores will not be comparable across segments",
+ fields.len() * segs
+ );
+ return None;
+ }
+ if pairs.len() * segs > MAX_STATS_TERM_PROBES {
+ tracing::warn!(
+ terms = pairs.len(),
+ segments = segs,
+ "search: BM25 collection statistics declined ({}) term probes > \
+ {MAX_STATS_TERM_PROBES} — segments will score against their own \
+ df/avgdl and scores will not be comparable across segments",
+ pairs.len() * segs
+ );
return None;
}
@@ -27623,7 +27669,15 @@ impl Index {
else {
// A segment whose statistics we cannot read would skew the
// union low and silently inflate every other arm's IDF.
- // Fall back to per-arm statistics rather than half a union.
+ // Fall back to per-arm statistics rather than half a union —
+ // and SAY so (#1186: every decline from index-wide stats is
+ // a silent ranking change, never a quiet local decision).
+ tracing::warn!(
+ segment = %meta.id,
+ "search: BM25 collection statistics declined — segment stats \
+ unreadable; segments will score against their own df/avgdl and \
+ scores will not be comparable across segments"
+ );
return None;
};
for f in &fields {
diff --git a/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs b/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs
index e02195781..ed4b76f3f 100644
--- a/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs
+++ b/engine/crates/xerj-engine/tests/bm25_stats_are_index_wide.rs
@@ -51,6 +51,17 @@ fn make_engine(dir: &TempDir, ingest_shards: usize) -> Engine {
Engine::new(config).expect("engine::new")
}
+/// [`make_engine`] with tiny-segment merging pinned off (#1186's `nomerge`
+/// config): the tiered merge otherwise collapses the many small segments the
+/// probe-budget test needs, mid-test and timing-dependent.
+fn make_engine_nomerge(dir: &TempDir, ingest_shards: usize) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ config.engine.ingest_shards = ingest_shards;
+ config.merge.min_merge_count = 100_000;
+ Engine::new(config).expect("engine::new")
+}
+
/// One buried occurrence of the term in a LONG field that grows with `i` —
/// low BM25, strictly decreasing.
fn weak_body(i: usize) -> String {
@@ -287,3 +298,90 @@ async fn deletes_do_not_break_the_union() {
);
}
}
+
+/// #1186 — padding a query with terms that match NOTHING must return the
+/// same hits with the same scores as the unpadded query.
+///
+/// A term with df=0 contributes 0 to every document's BM25 score, so padding
+/// is score-neutral — PROVIDED both queries are scored against the same
+/// index-wide statistics. It stopped being neutral the moment the padded
+/// query crossed the stats pre-pass probe budget
+/// `(fields + pairs) * segments > 4096`: the pre-pass silently returned
+/// `None`, every segment fell back to its OWN df/avgdl, and the whole top-5
+/// reordered (reported: top hit `1-1:1.6797521` → `8-4:1.6904112`; after
+/// `_forcemerge?max_num_segments=1` both queries agreed again — proof the
+/// reorder was the stats fallback, not the extra terms).
+///
+/// The corpus mirrors the reporter's: 8 flush cycles of 20 docs each on a
+/// 4-shard engine (a segment per non-empty shard per flush) with merging
+/// pinned off. The padded query projects 1 field + 1502 terms over ~32
+/// segments ≈ 48k probes — far past the old 4096 combined cap (where the
+/// ranking changed) and well inside the split per-term budget (where both
+/// queries are scored against the same union).
+#[tokio::test]
+async fn padding_a_query_with_terms_that_match_nothing_never_reorders_results() {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine_nomerge(&dir, 4);
+ engine.create_index("t", Schema::empty()).unwrap();
+ let idx = engine.get_index("t").unwrap();
+
+ // Per-batch term frequencies differ (batch b carries b `alpha`s and b
+ // `beta`s), so per-segment statistics genuinely differ from index-wide
+ // ones — a silent fallback MOVES scores, it does not just rescale them.
+ for b in 1..=8 {
+ for i in 1..=20 {
+ let alphas = "alpha ".repeat(b);
+ let beta = if i <= b { " beta" } else { "" };
+ idx.index_document(
+ Some(format!("{b}-{i}")),
+ json!({"body": format!("doc {i} batch {b} gamma{alphas}{beta}")}),
+ )
+ .await
+ .unwrap();
+ }
+ idx.flush().await.unwrap();
+ }
+
+ let segments = idx.stats().await.segment_count;
+ assert!(
+ segments >= 3,
+ "sanity: need ≥3 segments for the padded query ({pairs} pairs × \
+ {segments}) to cross the old 4096-probe cap; got {segments}",
+ pairs = 1 + 1502
+ );
+
+ let unpadded = parse_request(&json!({
+ "size": 5, "_source": false,
+ "query": {"match": {"body": "alpha beta"}}
+ }))
+ .expect("parse_request");
+ let pads = (1..=1500)
+ .map(|i| format!("zqx{i}"))
+ .collect::<Vec<_>>()
+ .join(" ");
+ let padded = parse_request(&json!({
+ "size": 5, "_source": false,
+ "query": {"match": {"body": format!("alpha beta {pads}")}}
+ }))
+ .expect("parse_request");
+
+ let plain = idx.search(&unpadded).await.unwrap();
+ let padded_r = idx.search(&padded).await.unwrap();
+
+ assert_eq!(
+ plain.total.value, padded_r.total.value,
+ "terms that match nothing must not change hits.total"
+ );
+ let top = |hits: &[xerj_query::Hit]| {
+ hits.iter()
+ .map(|h| (h.id.clone(), h.score.to_bits()))
+ .collect::<Vec<_>>()
+ };
+ assert_eq!(
+ top(&plain.hits),
+ top(&padded_r.hits),
+ "padding the query with 1500 terms that match no document reordered \
+ the top-5 or changed a score — the stats pre-pass declined and every \
+ segment silently scored against its own df/avgdl (#1186)"
+ );
+}