AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): a one-clause bool is a no-op — stop it changing _score and rank (#399)
xerj-orgbf7aff16891f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e46790f3c..6b333a04a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **Wrapping a query in a one-clause `bool` no longer changes its `_score` or+ its ranking** ([#399](https://github.com/xerj-org/xerj/issues/399)).+ `{"bool":{"must":[X]}}` and bare `X` are the same query — Lucene's+ `BooleanQuery.rewrite` erases a one-clause boolean before any `Weight` is+ built (`BooleanQuery.java:279-298`) — but XERJ steers on the *shape* of the+ query tree, and `is_doc_scan_query` matches every `bool`. On an index whose+ documents were still in the memtable, the wrapped form was therefore scored+ per-document by the IDF-less brute scorer (`1 + ln(1 + tf)`) while the+ identical bare `match` took the memtable BM25 arm. Measured on a+ 120-document fixture with one `text` field, same clause throughout: the+ wrapped query scored `d004` at `2.7917595` where the bare clause scored it+ `0.0068451734`, and `size:1` returned a different document for each. The+ flushed (segment FTS) population already agreed and still does.++ A one-clause `bool` is now erased before any path decision reads the tree —+ a lone `must`, or a lone `should` with `minimum_should_match` absent, `0`+ or `1`, exactly the cases Lucene rewrites to the clause itself. A lone+ `filter` (`BoostQuery(ConstantScoreQuery(q), 0)`, score 0) and a lone+ `must_not` (pure-negative) are *not* their child and are left alone. The+ two page-local score-rewrite gates further down now read the same+ normalised tree the executor was steered by, rather than the raw request,+ so the divergence cannot reappear one gate later.++ Known and unchanged: on an unflushed index a `filter`-only `bool` still+ scores `1.0` instead of `0.0`. That is the separately-tracked filter-context+ defect in the memtable's brute scorer, not this one, and this change neither+ fixes nor worsens it.+- **A file excluded by a widened ignore/hidden rule was reported as a userdeletion, so the graph resume path refused the rerun and its recoverycontract told operators to "restore the removed file(s)" — a file still ondiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex f7c7fb97b..036ab91a6 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -14265,11 +14265,17 @@ impl Index {// the lowering the missing postings hand the clause to the// stored-doc scan. See `lower_lexically_typeless_clauses`.let typeless = crate::memtable::lexically_typeless_fields(&schema.schema);- if typeless.is_empty() {+ let lowered = if typeless.is_empty() {resolved} else {lower_lexically_typeless_clauses(&resolved, &typeless)- }+ };+ // #399 — and it belongs HERE for exactly the reason above: every+ // path decision below is made on the SHAPE of this tree, so a+ // `bool` wrapper that ES erases before it builds a `Weight` has to+ // be erased before `is_doc_scan_query` or `query_node_to_fts` can+ // read it. See `unwrap_single_clause_bool`.+ unwrap_single_clause_bool(lowered)};let query = &resolved_query;// #574: resolve each rescore stage's secondary query the same way the@@ -17892,9 +17898,17 @@ impl Index {// Top-level constant_score: scores are the wrapper's constant —// the hit-set idf heuristic must not rewrite them.&& !matches!(query, QueryNode::Constant { .. })- && query_uses_bool_text(&request.query)+ // `query`, not `request.query` (#399): these gates decide whether+ // to REWRITE what the executor just scored, so they have to read+ // the same tree the executor was steered by — alias-resolved,+ // typeless-lowered, one-clause-bools erased. Reading the raw+ // request re-opens the defect from the other side: after the+ // unwrap `bool{must:[X]}` and bare `X` take the same path and+ // score the same, and then a gate that still sees a `Bool` would+ // rewrite one page and not the other.+ && query_uses_bool_text(query){- let field_term_pairs = collect_match_field_terms(&request.query);+ let field_term_pairs = collect_match_field_terms(query);if !field_term_pairs.is_empty() {let n_docs = final_hits.len() as f32;// For each (field, term) clause, compute its doc frequency.@@ -17954,7 +17968,13 @@ impl Index {// TF-IDF fallback: when all BM25 scores are negligibly small, recompute// using a simpler term-frequency heuristic for more useful ranking.if max_score > 0.0 && max_score < 0.001 {- let query_terms = extract_query_text(&request.query)+ // `query`, not `request.query` — same argument as the rescore+ // gate above (#399). `extract_query_text` answers `None` for a+ // `Bool`, so with the raw request a very common term (BM25+ // below 0.001) would be TF-IDF-rewritten as a bare `match` and+ // left alone as `bool{must:[match]}` — the identical query+ // shape divergence, one gate further down.+ let query_terms = extract_query_text(query).map(|t| {t.split_whitespace().map(|s| s.to_lowercase())@@ -34413,6 +34433,94 @@ fn extract_query_text(q: &QueryNode) -> Option<String> {}}+/// Erase `bool` wrappers whose only clause is a single SCORING clause.+///+/// `{"bool":{"must":[X]}}` and bare `X` are the same query and ES gives them+/// the same `_score`: `BooleanQuery.rewrite` replaces a one-clause boolean+/// with the clause's own query *before* any `Weight` exists — a lone `MUST`,+/// or a lone `SHOULD` with `minimumNumberShouldMatch` 0 or 1 (BooleanQuery.java:279-298,+/// "optimize 1-clause queries") — so the wrapper is structurally incapable of+/// contributing anything.+///+/// XERJ steers on the SHAPE of the query tree, which is what made the wrapper+/// anything but neutral (#399). `is_doc_scan_query` matches every+/// `QueryNode::Bool`, so on an unflushed index `bool{must:[match]}` was scored+/// per-document by the IDF-less brute scorer (`score_query_against_doc`:+/// `1 + ln(1 + tf)`) while the identical bare `Match` took the memtable BM25+/// arm through `extract_query_text`. Measured on 120 memtable-resident docs,+/// one `text` field, same clause: `2.7917595` against `0.0068451734`, and a+/// different top hit.+///+/// Only the two shapes Lucene rewrites to the clause itself are erased:+///+/// * a lone `filter` rewrites to `BoostQuery(ConstantScoreQuery(q), 0)` —+/// score 0, NOT the clause's score (BooleanQuery.java:290-292);+/// * a lone `must_not` is a pure-negative bool (BooleanQuery.java:274-277);+///+/// neither is its child, so both keep the handling they already have, and+/// `must_not`/`filter` children are left untouched for the same reason — they+/// are non-scoring, so this stays confined to the scoring path.+fn unwrap_single_clause_bool(q: QueryNode) -> QueryNode {+ match q {+ QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match,+ } => {+ // Recurse into the scoring children first, so nested wrappers+ // collapse in one walk — Lucene's `rewrite` is recursive over its+ // clauses for the same reason (BooleanQuery.java:300-342).+ let mut must: Vec<QueryNode> =+ must.into_iter().map(unwrap_single_clause_bool).collect();+ let mut should: Vec<QueryNode> =+ should.into_iter().map(unwrap_single_clause_bool).collect();+ let only_must = should.is_empty() && must_not.is_empty() && filter.is_empty();+ if must.len() == 1+ && only_must+ && matches!(minimum_should_match, None | Some(MinShouldMatch::Fixed(0)))+ {+ return must.remove(0);+ }+ let only_should = must.is_empty() && must_not.is_empty() && filter.is_empty();+ if should.len() == 1+ && only_should+ && matches!(+ minimum_should_match,+ None | Some(MinShouldMatch::Fixed(0)) | Some(MinShouldMatch::Fixed(1))+ )+ {+ return should.remove(0);+ }+ QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match,+ }+ }+ // The wrappers a `bool` can sit under without changing what its+ // clauses mean. A `boost` on a `bool` is parsed into `Boosted`+ // (`parse_bool`), so unwrapping underneath it is the same composition+ // Lucene's `BoostQuery.rewrite` performs (BoostQuery.java:75-102).+ QueryNode::Boosted { boost, query } => QueryNode::Boosted {+ boost,+ query: Box::new(unwrap_single_clause_bool(*query)),+ },+ QueryNode::Constant { score, query } => QueryNode::Constant {+ score,+ query: Box::new(unwrap_single_clause_bool(*query)),+ },+ QueryNode::Named { name, query } => QueryNode::Named {+ name,+ query: Box::new(unwrap_single_clause_bool(*query)),+ },+ other => other,+ }+}+/// Returns true for query types that should be evaluated by scanning stored documents.fn is_doc_scan_query(q: &QueryNode) -> bool {// Wildcard field match queries must go through doc scanning.diff --git a/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rs b/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rsnew file mode 100644index 000000000..e75ea7ade--- /dev/null+++ b/engine/crates/xerj-engine/tests/bool_single_clause_is_score_neutral.rs@@ -0,0 +1,323 @@+//! #399 — a one-clause `bool` is a no-op: it must not change `_score` or rank.+//!+//! `{"bool":{"must":[X]}}` and bare `X` are the same query. On an UNFLUSHED+//! index they were not: `Index::search` steers on the *shape* of the tree, and+//! a `Bool` node is matched by `is_doc_scan_query`, so the memtable arm scored+//! it with the IDF-less brute scorer (`score_query_against_doc`, `1 + ln(1 ++//! tf)`) while the identical bare `Match` took the memtable BM25 arm+//! (`extract_query_text` → `search_text_boosted_with_total_using`, index-wide+//! stats). Same 120 documents, same clause, ~400× apart and in a different+//! order.+//!+//! #361/#387 fixed the *flushed* half of this family (`bool.filter` summing+//! into `_score`, and the page-local IDF rescore overwriting exact BM25); the+//! flushed assertions below are the regression guard for that. The memtable+//! half is what #399 reports and what the fix normalises away.+//!+//! Reference (apache/lucene, Apache-2.0, adapted not copied):+//! * `BooleanQuery.rewrite` — "optimize 1-clause queries": with+//! `minimumNumberShouldMatch == 0` a lone `SHOULD` or `MUST` clause+//! rewrites to the clause's own query, and with+//! `minimumNumberShouldMatch == 1` a lone `SHOULD` does too+//! (BooleanQuery.java:279-298). The wrapper is erased before a `Weight`+//! is ever built, so it cannot contribute a score.+//! * A lone `FILTER` clause instead rewrites to+//! `new BoostQuery(new ConstantScoreQuery(query), 0)` — score 0, NOT the+//! clause's score (BooleanQuery.java:290-292); `filter_only_bool_*` below+//! pins that asymmetry.++use std::collections::HashMap;++use serde_json::{json, Value};+use tempfile::TempDir;+use xerj_common::config::Config;+use xerj_common::types::{FieldConfig, FieldType, Schema};+use xerj_engine::{Engine, Index};+use xerj_query::executor::SearchResult;+use xerj_query::parse_request;++fn make_engine(dir: &TempDir) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ Engine::new(config).expect("engine::new")+}++fn req(query: Value, size: usize) -> xerj_query::ast::SearchRequest {+ parse_request(&json!({"query": query, "size": size, "track_total_hits": true}))+ .expect("parse_request")+}++/// The issue's corpus: 120 docs, one `text` field, one `keyword` field.+///+/// `alpha` is in every document with tf 1..=5, so exact BM25 gives it a tiny+/// IDF (`ln(1 + 0.5/120.5)`) and the brute scorer gives `1 + ln(1 + tf)` —+/// two scales that cannot be confused for each other.+async fn seed(name: &str, flush: bool) -> (TempDir, std::sync::Arc<Index>) {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ schema+ .fields+ .push(FieldConfig::new("repo", FieldType::Keyword));+ engine.create_index(name, schema).unwrap();+ let idx = engine.get_index(name).unwrap();++ for i in 0..120usize {+ let alphas = "alpha ".repeat(1 + i % 5);+ let betas = if i < 60 {+ "beta ".repeat(1 + i % 3)+ } else {+ String::new()+ };+ let filler = "gamma ".repeat(1 + i % 7);+ idx.index_document(+ Some(format!("d{i:03}")),+ json!({+ "body": format!("{alphas}{betas}{filler}"),+ "repo": if i % 2 == 0 { "lucene" } else { "usearch" },+ }),+ )+ .await+ .unwrap();+ }+ if flush {+ idx.flush().await.unwrap();+ }++ // The two populations are the whole point of the test — assert which one+ // this fixture actually exercises rather than trusting the flush call.+ let stats = idx.stats().await;+ if flush {+ assert_eq!(+ stats.memtable_doc_count, 0,+ "{name}: flush did not drain the memtable"+ );+ } else {+ assert_eq!(+ stats.memtable_doc_count, 120,+ "{name}: fixture must stay memtable-resident"+ );+ }+ (dir, idx)+}++fn score_map(result: &SearchResult) -> HashMap<String, f32> {+ result+ .hits+ .iter()+ .map(|h| (h.id.clone(), h.score))+ .collect()+}++fn ids(result: &SearchResult) -> Vec<String> {+ result.hits.iter().map(|h| h.id.clone()).collect()+}++/// Every hit's `_score` must equal the bare clause's score for the same+/// `_id`, and the order must be identical.+fn assert_same_ranking(label: &str, reference: &SearchResult, actual: &SearchResult) {+ assert!(+ !actual.hits.is_empty(),+ "{label}: returned no hits — the comparison would be vacuous"+ );+ assert_eq!(+ reference.total.value, actual.total.value,+ "{label}: matched a different number of documents"+ );+ let reference_scores = score_map(reference);+ for hit in &actual.hits {+ let expected = reference_scores+ .get(&hit.id)+ .unwrap_or_else(|| panic!("{label}: {} is not in the bare-clause result", hit.id));+ assert!(+ (hit.score - expected).abs() < 1e-4,+ "{label}: _score for {} is {} but the bare clause scores it {expected}",+ hit.id,+ hit.score+ );+ }+ assert_eq!(+ ids(actual),+ ids(reference),+ "{label}: hit order diverged from the bare clause"+ );+}++const TEXT: &str = "alpha";++fn text_query() -> Value {+ json!({"match": {"body": TEXT}})+}++/// The four wrappers Lucene's 1-clause rewrite erases, each of which must+/// leave `_score` and rank byte-for-byte where the bare clause put them.+fn neutral_wrappers() -> Vec<(&'static str, Value)> {+ vec![+ ("bool.must[1]", json!({"bool": {"must": [text_query()]}})),+ (+ "bool.should[1]",+ json!({"bool": {"should": [text_query()]}}),+ ),+ (+ // `minimum_should_match: 1` on a lone should is the same query —+ // BooleanQuery.java:283-284.+ "bool.should[1]+msm1",+ json!({"bool": {"should": [text_query()], "minimum_should_match": 1}}),+ ),+ (+ // Nested wrappers collapse all the way down.+ "bool.must[bool.must[1]]",+ json!({"bool": {"must": [{"bool": {"must": [text_query()]}}]}}),+ ),+ ]+}++/// The headline of #399, on the population that showed it: an UNFLUSHED index.+#[tokio::test]+async fn single_clause_bool_is_score_neutral_on_memtable() {+ let (_dir, idx) = seed("bool-one-clause-memtable", false).await;++ let reference = idx.search(&req(text_query(), 50)).await.unwrap();+ assert_eq!(reference.total.value, 120, "fixture: every doc has `alpha`");+ let distinct: std::collections::HashSet<u32> = reference+ .hits+ .iter()+ .map(|h| (h.score * 1e6) as u32)+ .collect();+ assert!(+ distinct.len() > 1,+ "fixture must produce distinct scores, got {:?}",+ reference.hits.iter().map(|h| h.score).collect::<Vec<_>>()+ );++ for (label, wrapped) in neutral_wrappers() {+ let actual = idx.search(&req(wrapped, 50)).await.unwrap();+ assert_same_ranking(label, &reference, &actual);+ }+}++/// Same assertions on the flushed (segment FTS) population — the half #387+/// fixed. Guards against a fix for the memtable half re-introducing the+/// divergence here.+#[tokio::test]+async fn single_clause_bool_is_score_neutral_on_segments() {+ let (_dir, idx) = seed("bool-one-clause-segments", true).await;++ let reference = idx.search(&req(text_query(), 50)).await.unwrap();+ assert_eq!(reference.total.value, 120);++ for (label, wrapped) in neutral_wrappers() {+ let actual = idx.search(&req(wrapped, 50)).await.unwrap();+ assert_same_ranking(label, &reference, &actual);+ }+}++/// `_score` must not depend on `size` for either shape, and the two shapes+/// must agree at every page size — the property a client doing RRF/threshold+/// fusion actually relies on.+#[tokio::test]+async fn wrapped_and_bare_agree_at_every_page_size() {+ for flush in [false, true] {+ let (_dir, idx) = seed(&format!("bool-one-clause-size-{flush}"), flush).await;+ for size in [1usize, 10, 50] {+ let bare = idx.search(&req(text_query(), size)).await.unwrap();+ let wrapped = idx+ .search(&req(json!({"bool": {"must": [text_query()]}}), size))+ .await+ .unwrap();+ assert_same_ranking(&format!("flush={flush} size={size}"), &bare, &wrapped);+ }+ }+}++/// The same neutrality for a `term` on a `keyword` field.+///+/// This one is the anti-regression half: three places in the engine ALREADY+/// treat a single-`must` bool as the identity — `scored_fast_plan` wraps a+/// bare scoring leaf "as a single-must bool (Σ over one clause is the+/// identity)", `mem_bool_preds` accepts a bare `Term`/`Range` as its own+/// one-predicate fused walk, and `peel_knn_query` peels `bool{must:[knn]}` to+/// the same `PeeledKnn` as a bare `knn`. Erasing the wrapper must not push+/// this shape off any of them onto a different scorer; green before and after.+#[tokio::test]+async fn single_clause_bool_is_score_neutral_for_term_on_keyword() {+ for flush in [false, true] {+ let (_dir, idx) = seed(&format!("bool-one-clause-term-{flush}"), flush).await;+ let bare = idx+ .search(&req(json!({"term": {"repo": "lucene"}}), 50))+ .await+ .unwrap();+ assert_eq!(bare.total.value, 60, "flush={flush}");+ let wrapped = idx+ .search(&req(+ json!({"bool": {"must": [{"term": {"repo": "lucene"}}]}}),+ 50,+ ))+ .await+ .unwrap();+ assert_same_ranking(&format!("bool.must[term] flush={flush}"), &bare, &wrapped);+ }+}++/// A lone `filter` clause is NOT the neutral case: Lucene rewrites it to+/// `BoostQuery(ConstantScoreQuery(query), 0)` (BooleanQuery.java:290-292), so+/// the score is 0, not the clause's own score. This is the guard that the+/// unwrap above cannot quietly grow a `filter` arm.+///+/// FLUSHED ONLY, deliberately. On an unflushed index the same query scores+/// **1.0**, because the memtable's brute scorer falls back to 1.0 for a+/// zero-sum bool — a pre-existing, separately-tracked filter-context defect+/// that #399 does not touch and this change does not alter (see the `if score+/// == 0.0` arm of `score_query_against_doc`'s `Bool` case in+/// `xerj-engine/src/index.rs`, whose own comment scopes it out). Asserting+/// 1.0 here would pin a wrong number as correct, so the case is stated and+/// left to its own fix instead.+#[tokio::test]+async fn filter_only_bool_still_scores_zero_when_flushed() {+ let (_dir, idx) = seed("bool-one-clause-filter", true).await;+ let result = idx+ .search(&req(+ json!({"bool": {"filter": [{"term": {"repo": "lucene"}}]}}),+ 50,+ ))+ .await+ .unwrap();+ assert_eq!(result.total.value, 60);+ assert_eq!(result.hits.len(), 50);+ for hit in &result.hits {+ assert_eq!(+ hit.score, 0.0,+ "{} scored {} on a filter-only bool",+ hit.id, hit.score+ );+ }+}++/// A lone `must_not` is a pure-negative bool — it has no positive clause, so+/// it must not be unwrapped into its (negated) child. Green before the fix;+/// it must stay green. (`clauses.size() == clauseSets.get(MUST_NOT).size()`+/// → `MatchNoDocsQuery("pure negative BooleanQuery")`, BooleanQuery.java:274-277+/// — Lucene returns no matches; XERJ answers it as an exclusion over all docs,+/// which is the pre-existing behaviour this test pins, not a claim about ES.)+#[tokio::test]+async fn must_not_only_bool_is_not_unwrapped() {+ let (_dir, idx) = seed("bool-one-clause-must-not", false).await;+ let result = idx+ .search(&req(+ json!({"bool": {"must_not": [{"term": {"repo": "lucene"}}]}}),+ 50,+ ))+ .await+ .unwrap();+ for hit in &result.hits {+ assert_eq!(+ hit.source.get("repo").and_then(Value::as_str),+ Some("usearch"),+ "{} survived a must_not on repo:lucene",+ hit.id+ );+ }+}