Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(search): regexp hits.total leaked segments×256 under the implicit index sort — gate the prefilter cap on sort_topk

xerj-org65096ba334eb
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 99f1a8993..679e080c2 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -4630,11 +4630,28 @@ impl Index {
// Dictionary-expansion pre-filter (mirrors the
// `try_shortcut_count` Regexp arm). When the
// exact total comes from the shortcut
- // (count_authoritative) the scan early-breaks at
+ // (count_authoritative) AND the hits collector is
+ // the arrival-order bounded `all_hits` (no field
+ // sort), the scan early-breaks at
// `materialisation_limit` hits, so cap the
- // position set there; otherwise the scan tallies
- // the total itself and needs every position.
- let cap = if count_authoritative {
+ // position set there; otherwise the scan must
+ // visit EVERY match — to tally the exact total
+ // and/or to offer every match to the field-sort
+ // top-N heap — so only a COMPLETE position set is
+ // a valid filter.
+ //
+ // The `sort_topk.is_none()` gate is load-bearing:
+ // with a field sort active (incl. the implicit
+ // `@timestamp desc` index sort injected by the ES
+ // layer for time-series mappings) hits bypass
+ // `all_hits`, so the caller's early-break and the
+ // post-loop `scan_hit_cap` shortcut-count
+ // overwrite NEVER fire — a capped prefilter would
+ // leak `segments × materialisation_limit` into
+ // `hits.total` (the 512-vs-847306 bench
+ // regression) and starve the top-N heap of
+ // candidates.
+ let cap = if count_authoritative && sort_topk.is_none() {
Some(materialisation_limit)
} else {
None