Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): a one-clause bool is a no-op — stop it changing _score and rank (#399)

xerj-orgbf7aff16891f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e46790f3c..6b333a04a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **Wrapping a query in a one-clause `bool` no longer changes its `_score` or
+ its ranking** ([#399](https://github.com/xerj-org/xerj/issues/399)).
+ `{"bool":{"must":[X]}}` and bare `X` are the same query — Lucene's
+ `BooleanQuery.rewrite` erases a one-clause boolean before any `Weight` is
+ built (`BooleanQuery.java:279-298`) — but XERJ steers on the *shape* of the
+ query tree, and `is_doc_scan_query` matches every `bool`. On an index whose
+ documents were still in the memtable, the wrapped form was therefore scored
+ per-document by the IDF-less brute scorer (`1 + ln(1 + tf)`) while the
+ identical bare `match` took the memtable BM25 arm. Measured on a
+ 120-document fixture with one `text` field, same clause throughout: the
+ wrapped query scored `d004` at `2.7917595` where the bare clause scored it
+ `0.0068451734`, and `size:1` returned a different document for each. The
+ flushed (segment FTS) population already agreed and still does.
+
+ A one-clause `bool` is now erased before any path decision reads the tree —
+ a lone `must`, or a lone `should` with `minimum_should_match` absent, `0`
+ or `1`, exactly the cases Lucene rewrites to the clause itself. A lone
+ `filter` (`BoostQuery(ConstantScoreQuery(q), 0)`, score 0) and a lone
+ `must_not` (pure-negative) are *not* their child and are left alone. The
+ two page-local score-rewrite gates further down now read the same
+ normalised tree the executor was steered by, rather than the raw request,
+ so the divergence cannot reappear one gate later.
+
+ Known and unchanged: on an unflushed index a `filter`-only `bool` still
+ scores `1.0` instead of `0.0`. That is the separately-tracked filter-context
+ defect in the memtable's brute scorer, not this one, and this change neither
+ fixes nor worsens it.
+
- **A file excluded by a widened ignore/hidden rule was reported as a user
deletion, so the graph resume path refused the rerun and its recovery
contract told operators to "restore the removed file(s)" — a file still on
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index f7c7fb97b..036ab91a6 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -14265,11 +14265,17 @@ impl Index {
// the lowering the missing postings hand the clause to the
// stored-doc scan. See `lower_lexically_typeless_clauses`.
let typeless = crate::memtable::lexically_typeless_fields(&schema.schema);
- if typeless.is_empty() {
+ let lowered = if typeless.is_empty() {
resolved
} else {
lower_lexically_typeless_clauses(&resolved, &typeless)
- }
+ };
+ // #399 — and it belongs HERE for exactly the reason above: every
+ // path decision below is made on the SHAPE of this tree, so a
+ // `bool` wrapper that ES erases before it builds a `Weight` has to
+ // be erased before `is_doc_scan_query` or `query_node_to_fts` can
+ // read it. See `unwrap_single_clause_bool`.
+ unwrap_single_clause_bool(lowered)
};
let query = &resolved_query;
// #574: resolve each rescore stage's secondary query the same way the
@@ -17892,9 +17898,17 @@ impl Index {
// Top-level constant_score: scores are the wrapper's constant —
// the hit-set idf heuristic must not rewrite them.
&& !matches!(query, QueryNode::Constant { .. })
- && query_uses_bool_text(&request.query)
+ // `query`, not `request.query` (#399): these gates decide whether
+ // to REWRITE what the executor just scored, so they have to read
+ // the same tree the executor was steered by — alias-resolved,
+ // typeless-lowered, one-clause-bools erased. Reading the raw
+ // request re-opens the defect from the other side: after the
+ // unwrap `bool{must:[X]}` and bare `X` take the same path and
+ // score the same, and then a gate that still sees a `Bool` would
+ // rewrite one page and not the other.
+ && query_uses_bool_text(query)
{
- let field_term_pairs = collect_match_field_terms(&request.query);
+ let field_term_pairs = collect_match_field_terms(query);
if !field_term_pairs.is_empty() {
let n_docs = final_hits.len() as f32;
// For each (field, term) clause, compute its doc frequency.
@@ -17954,7 +17968,13 @@ impl Index {
// TF-IDF fallback: when all BM25 scores are negligibly small, recompute
// using a simpler term-frequency heuristic for more useful ranking.
if max_score > 0.0 && max_score < 0.001 {
- let query_terms = extract_query_text(&request.query)
+ // `query`, not `request.query` — same argument as the rescore
+ // gate above (#399). `extract_query_text` answers `None` for a
+ // `Bool`, so with the raw request a very common term (BM25
+ // below 0.001) would be TF-IDF-rewritten as a bare `match` and
+ // left alone as `bool{must:[match]}` — the identical query
+ // shape divergence, one gate further down.
+ let query_terms = extract_query_text(query)
.map(|t| {
t.split_whitespace()
.map(|s| s.to_lowercase())
@@ -34413,6 +34433,94 @@ fn extract_query_text(q: &QueryNode) -> Option<String> {
}
}
+/// Erase `bool` wrappers whose only clause is a single SCORING clause.
+///
+/// `{"bool":{"must":[X]}}` and bare `X` are the same query and ES gives them
+/// the same `_score`: `BooleanQuery.rewrite` replaces a one-clause boolean
+/// with the clause's own query *before* any `Weight` exists — a lone `MUST`,
+/// or a lone `SHOULD` with `minimumNumberShouldMatch` 0 or 1 (BooleanQuery.java:279-298,
+/// "optimize 1-clause queries") — so the wrapper is structurally incapable of
+/// contributing anything.
+///
+/// XERJ steers on the SHAPE of the query tree, which is what made the wrapper
+/// anything but neutral (#399). `is_doc_scan_query` matches every
+/// `QueryNode::Bool`, so on an unflushed index `bool{must:[match]}` was scored
+/// per-document by the IDF-less brute scorer (`score_query_against_doc`:
+/// `1 + ln(1 + tf)`) while the identical bare `Match` took the memtable BM25
+/// arm through `extract_query_text`. Measured on 120 memtable-resident docs,
+/// one `text` field, same clause: `2.7917595` against `0.0068451734`, and a
+/// different top hit.
+///
+/// Only the two shapes Lucene rewrites to the clause itself are erased:
+///
+/// * a lone `filter` rewrites to `BoostQuery(ConstantScoreQuery(q), 0)` —
+/// score 0, NOT the clause's score (BooleanQuery.java:290-292);
+/// * a lone `must_not` is a pure-negative bool (BooleanQuery.java:274-277);
+///
+/// neither is its child, so both keep the handling they already have, and
+/// `must_not`/`filter` children are left untouched for the same reason — they
+/// are non-scoring, so this stays confined to the scoring path.
+fn unwrap_single_clause_bool(q: QueryNode) -> QueryNode {
+ match q {
+ QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match,
+ } => {
+ // Recurse into the scoring children first, so nested wrappers
+ // collapse in one walk — Lucene's `rewrite` is recursive over its
+ // clauses for the same reason (BooleanQuery.java:300-342).
+ let mut must: Vec<QueryNode> =
+ must.into_iter().map(unwrap_single_clause_bool).collect();
+ let mut should: Vec<QueryNode> =
+ should.into_iter().map(unwrap_single_clause_bool).collect();
+ let only_must = should.is_empty() && must_not.is_empty() && filter.is_empty();
+ if must.len() == 1
+ && only_must
+ && matches!(minimum_should_match, None | Some(MinShouldMatch::Fixed(0)))
+ {
+ return must.remove(0);
+ }
+ let only_should = must.is_empty() && must_not.is_empty() && filter.is_empty();
+ if should.len() == 1
+ && only_should
+ && matches!(
+ minimum_should_match,
+ None | Some(MinShouldMatch::Fixed(0)) | Some(MinShouldMatch::Fixed(1))
+ )
+ {
+ return should.remove(0);
+ }
+ QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match,
+ }
+ }
+ // The wrappers a `bool` can sit under without changing what its
+ // clauses mean. A `boost` on a `bool` is parsed into `Boosted`
+ // (`parse_bool`), so unwrapping underneath it is the same composition
+ // Lucene's `BoostQuery.rewrite` performs (BoostQuery.java:75-102).
+ QueryNode::Boosted { boost, query } => QueryNode::Boosted {
+ boost,
+ query: Box::new(unwrap_single_clause_bool(*query)),
+ },
+ QueryNode::Constant { score, query } => QueryNode::Constant {
+ score,
+ query: Box::new(unwrap_single_clause_bool(*query)),
+ },
+ QueryNode::Named { name, query } => QueryNode::Named {
+ name,
+ query: Box::new(unwrap_single_clause_bool(*query)),
+ },
+ other => other,
+ }
+}
+
/// Returns true for query types that should be evaluated by scanning stored documents.
fn is_doc_scan_query(q: &QueryNode) -> bool {
// Wildcard field match queries must go through doc scanning.
diff --git a/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rs b/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rs
new file mode 100644
index 000000000..e75ea7ade
--- /dev/null
+++ b/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rs
@@ -0,0 +1,323 @@
+//! #399 — a one-clause `bool` is a no-op: it must not change `_score` or rank.
+//!
+//! `{"bool":{"must":[X]}}` and bare `X` are the same query. On an UNFLUSHED
+//! index they were not: `Index::search` steers on the *shape* of the tree, and
+//! a `Bool` node is matched by `is_doc_scan_query`, so the memtable arm scored
+//! it with the IDF-less brute scorer (`score_query_against_doc`, `1 + ln(1 +
+//! tf)`) while the identical bare `Match` took the memtable BM25 arm
+//! (`extract_query_text` → `search_text_boosted_with_total_using`, index-wide
+//! stats). Same 120 documents, same clause, ~400× apart and in a different
+//! order.
+//!
+//! #361/#387 fixed the *flushed* half of this family (`bool.filter` summing
+//! into `_score`, and the page-local IDF rescore overwriting exact BM25); the
+//! flushed assertions below are the regression guard for that. The memtable
+//! half is what #399 reports and what the fix normalises away.
+//!
+//! Reference (apache/lucene, Apache-2.0, adapted not copied):
+//! * `BooleanQuery.rewrite` — "optimize 1-clause queries": with
+//! `minimumNumberShouldMatch == 0` a lone `SHOULD` or `MUST` clause
+//! rewrites to the clause's own query, and with
+//! `minimumNumberShouldMatch == 1` a lone `SHOULD` does too
+//! (BooleanQuery.java:279-298). The wrapper is erased before a `Weight`
+//! is ever built, so it cannot contribute a score.
+//! * A lone `FILTER` clause instead rewrites to
+//! `new BoostQuery(new ConstantScoreQuery(query), 0)` — score 0, NOT the
+//! clause's score (BooleanQuery.java:290-292); `filter_only_bool_*` below
+//! pins that asymmetry.
+
+use std::collections::HashMap;
+
+use serde_json::{json, Value};
+use tempfile::TempDir;
+use xerj_common::config::Config;
+use xerj_common::types::{FieldConfig, FieldType, Schema};
+use xerj_engine::{Engine, Index};
+use xerj_query::executor::SearchResult;
+use xerj_query::parse_request;
+
+fn make_engine(dir: &TempDir) -> Engine {
+ let mut config = Config::default();
+ config.server.data_dir = dir.path().to_str().unwrap().to_string();
+ Engine::new(config).expect("engine::new")
+}
+
+fn req(query: Value, size: usize) -> xerj_query::ast::SearchRequest {
+ parse_request(&json!({"query": query, "size": size, "track_total_hits": true}))
+ .expect("parse_request")
+}
+
+/// The issue's corpus: 120 docs, one `text` field, one `keyword` field.
+///
+/// `alpha` is in every document with tf 1..=5, so exact BM25 gives it a tiny
+/// IDF (`ln(1 + 0.5/120.5)`) and the brute scorer gives `1 + ln(1 + tf)` —
+/// two scales that cannot be confused for each other.
+async fn seed(name: &str, flush: bool) -> (TempDir, std::sync::Arc<Index>) {
+ let dir = TempDir::new().unwrap();
+ let engine = make_engine(&dir);
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ schema
+ .fields
+ .push(FieldConfig::new("repo", FieldType::Keyword));
+ engine.create_index(name, schema).unwrap();
+ let idx = engine.get_index(name).unwrap();
+
+ for i in 0..120usize {
+ let alphas = "alpha ".repeat(1 + i % 5);
+ let betas = if i < 60 {
+ "beta ".repeat(1 + i % 3)
+ } else {
+ String::new()
+ };
+ let filler = "gamma ".repeat(1 + i % 7);
+ idx.index_document(
+ Some(format!("d{i:03}")),
+ json!({
+ "body": format!("{alphas}{betas}{filler}"),
+ "repo": if i % 2 == 0 { "lucene" } else { "usearch" },
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ if flush {
+ idx.flush().await.unwrap();
+ }
+
+ // The two populations are the whole point of the test — assert which one
+ // this fixture actually exercises rather than trusting the flush call.
+ let stats = idx.stats().await;
+ if flush {
+ assert_eq!(
+ stats.memtable_doc_count, 0,
+ "{name}: flush did not drain the memtable"
+ );
+ } else {
+ assert_eq!(
+ stats.memtable_doc_count, 120,
+ "{name}: fixture must stay memtable-resident"
+ );
+ }
+ (dir, idx)
+}
+
+fn score_map(result: &SearchResult) -> HashMap<String, f32> {
+ result
+ .hits
+ .iter()
+ .map(|h| (h.id.clone(), h.score))
+ .collect()
+}
+
+fn ids(result: &SearchResult) -> Vec<String> {
+ result.hits.iter().map(|h| h.id.clone()).collect()
+}
+
+/// Every hit's `_score` must equal the bare clause's score for the same
+/// `_id`, and the order must be identical.
+fn assert_same_ranking(label: &str, reference: &SearchResult, actual: &SearchResult) {
+ assert!(
+ !actual.hits.is_empty(),
+ "{label}: returned no hits — the comparison would be vacuous"
+ );
+ assert_eq!(
+ reference.total.value, actual.total.value,
+ "{label}: matched a different number of documents"
+ );
+ let reference_scores = score_map(reference);
+ for hit in &actual.hits {
+ let expected = reference_scores
+ .get(&hit.id)
+ .unwrap_or_else(|| panic!("{label}: {} is not in the bare-clause result", hit.id));
+ assert!(
+ (hit.score - expected).abs() < 1e-4,
+ "{label}: _score for {} is {} but the bare clause scores it {expected}",
+ hit.id,
+ hit.score
+ );
+ }
+ assert_eq!(
+ ids(actual),
+ ids(reference),
+ "{label}: hit order diverged from the bare clause"
+ );
+}
+
+const TEXT: &str = "alpha";
+
+fn text_query() -> Value {
+ json!({"match": {"body": TEXT}})
+}
+
+/// The four wrappers Lucene's 1-clause rewrite erases, each of which must
+/// leave `_score` and rank byte-for-byte where the bare clause put them.
+fn neutral_wrappers() -> Vec<(&'static str, Value)> {
+ vec![
+ ("bool.must[1]", json!({"bool": {"must": [text_query()]}})),
+ (
+ "bool.should[1]",
+ json!({"bool": {"should": [text_query()]}}),
+ ),
+ (
+ // `minimum_should_match: 1` on a lone should is the same query —
+ // BooleanQuery.java:283-284.
+ "bool.should[1]+msm1",
+ json!({"bool": {"should": [text_query()], "minimum_should_match": 1}}),
+ ),
+ (
+ // Nested wrappers collapse all the way down.
+ "bool.must[bool.must[1]]",
+ json!({"bool": {"must": [{"bool": {"must": [text_query()]}}]}}),
+ ),
+ ]
+}
+
+/// The headline of #399, on the population that showed it: an UNFLUSHED index.
+#[tokio::test]
+async fn single_clause_bool_is_score_neutral_on_memtable() {
+ let (_dir, idx) = seed("bool-one-clause-memtable", false).await;
+
+ let reference = idx.search(&req(text_query(), 50)).await.unwrap();
+ assert_eq!(reference.total.value, 120, "fixture: every doc has `alpha`");
+ let distinct: std::collections::HashSet<u32> = reference
+ .hits
+ .iter()
+ .map(|h| (h.score * 1e6) as u32)
+ .collect();
+ assert!(
+ distinct.len() > 1,
+ "fixture must produce distinct scores, got {:?}",
+ reference.hits.iter().map(|h| h.score).collect::<Vec<_>>()
+ );
+
+ for (label, wrapped) in neutral_wrappers() {
+ let actual = idx.search(&req(wrapped, 50)).await.unwrap();
+ assert_same_ranking(label, &reference, &actual);
+ }
+}
+
+/// Same assertions on the flushed (segment FTS) population — the half #387
+/// fixed. Guards against a fix for the memtable half re-introducing the
+/// divergence here.
+#[tokio::test]
+async fn single_clause_bool_is_score_neutral_on_segments() {
+ let (_dir, idx) = seed("bool-one-clause-segments", true).await;
+
+ let reference = idx.search(&req(text_query(), 50)).await.unwrap();
+ assert_eq!(reference.total.value, 120);
+
+ for (label, wrapped) in neutral_wrappers() {
+ let actual = idx.search(&req(wrapped, 50)).await.unwrap();
+ assert_same_ranking(label, &reference, &actual);
+ }
+}
+
+/// `_score` must not depend on `size` for either shape, and the two shapes
+/// must agree at every page size — the property a client doing RRF/threshold
+/// fusion actually relies on.
+#[tokio::test]
+async fn wrapped_and_bare_agree_at_every_page_size() {
+ for flush in [false, true] {
+ let (_dir, idx) = seed(&format!("bool-one-clause-size-{flush}"), flush).await;
+ for size in [1usize, 10, 50] {
+ let bare = idx.search(&req(text_query(), size)).await.unwrap();
+ let wrapped = idx
+ .search(&req(json!({"bool": {"must": [text_query()]}}), size))
+ .await
+ .unwrap();
+ assert_same_ranking(&format!("flush={flush} size={size}"), &bare, &wrapped);
+ }
+ }
+}
+
+/// The same neutrality for a `term` on a `keyword` field.
+///
+/// This one is the anti-regression half: three places in the engine ALREADY
+/// treat a single-`must` bool as the identity — `scored_fast_plan` wraps a
+/// bare scoring leaf "as a single-must bool (Σ over one clause is the
+/// identity)", `mem_bool_preds` accepts a bare `Term`/`Range` as its own
+/// one-predicate fused walk, and `peel_knn_query` peels `bool{must:[knn]}` to
+/// the same `PeeledKnn` as a bare `knn`. Erasing the wrapper must not push
+/// this shape off any of them onto a different scorer; green before and after.
+#[tokio::test]
+async fn single_clause_bool_is_score_neutral_for_term_on_keyword() {
+ for flush in [false, true] {
+ let (_dir, idx) = seed(&format!("bool-one-clause-term-{flush}"), flush).await;
+ let bare = idx
+ .search(&req(json!({"term": {"repo": "lucene"}}), 50))
+ .await
+ .unwrap();
+ assert_eq!(bare.total.value, 60, "flush={flush}");
+ let wrapped = idx
+ .search(&req(
+ json!({"bool": {"must": [{"term": {"repo": "lucene"}}]}}),
+ 50,
+ ))
+ .await
+ .unwrap();
+ assert_same_ranking(&format!("bool.must[term] flush={flush}"), &bare, &wrapped);
+ }
+}
+
+/// A lone `filter` clause is NOT the neutral case: Lucene rewrites it to
+/// `BoostQuery(ConstantScoreQuery(query), 0)` (BooleanQuery.java:290-292), so
+/// the score is 0, not the clause's own score. This is the guard that the
+/// unwrap above cannot quietly grow a `filter` arm.
+///
+/// FLUSHED ONLY, deliberately. On an unflushed index the same query scores
+/// **1.0**, because the memtable's brute scorer falls back to 1.0 for a
+/// zero-sum bool — a pre-existing, separately-tracked filter-context defect
+/// that #399 does not touch and this change does not alter (see the `if score
+/// == 0.0` arm of `score_query_against_doc`'s `Bool` case in
+/// `xerj-engine/src/index.rs`, whose own comment scopes it out). Asserting
+/// 1.0 here would pin a wrong number as correct, so the case is stated and
+/// left to its own fix instead.
+#[tokio::test]
+async fn filter_only_bool_still_scores_zero_when_flushed() {
+ let (_dir, idx) = seed("bool-one-clause-filter", true).await;
+ let result = idx
+ .search(&req(
+ json!({"bool": {"filter": [{"term": {"repo": "lucene"}}]}}),
+ 50,
+ ))
+ .await
+ .unwrap();
+ assert_eq!(result.total.value, 60);
+ assert_eq!(result.hits.len(), 50);
+ for hit in &result.hits {
+ assert_eq!(
+ hit.score, 0.0,
+ "{} scored {} on a filter-only bool",
+ hit.id, hit.score
+ );
+ }
+}
+
+/// A lone `must_not` is a pure-negative bool — it has no positive clause, so
+/// it must not be unwrapped into its (negated) child. Green before the fix;
+/// it must stay green. (`clauses.size() == clauseSets.get(MUST_NOT).size()`
+/// → `MatchNoDocsQuery("pure negative BooleanQuery")`, BooleanQuery.java:274-277
+/// — Lucene returns no matches; XERJ answers it as an exclusion over all docs,
+/// which is the pre-existing behaviour this test pins, not a claim about ES.)
+#[tokio::test]
+async fn must_not_only_bool_is_not_unwrapped() {
+ let (_dir, idx) = seed("bool-one-clause-must-not", false).await;
+ let result = idx
+ .search(&req(
+ json!({"bool": {"must_not": [{"term": {"repo": "lucene"}}]}}),
+ 50,
+ ))
+ .await
+ .unwrap();
+ for hit in &result.hits {
+ assert_eq!(
+ hit.source.get("repo").and_then(Value::as_str),
+ Some("usearch"),
+ "{} survived a must_not on repo:lucene",
+ hit.id
+ );
+ }
+}