AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): bail to brute for uncolumned nested-object and suppressed-array fields (#139)
xerj-orgc502c0dbdaf3
diff --git a/engine/crates/xerj-engine/src/fast_aggs.rs b/engine/crates/xerj-engine/src/fast_aggs.rsindex 3247b31f2..d4d2cf822 100644--- a/engine/crates/xerj-engine/src/fast_aggs.rs+++ b/engine/crates/xerj-engine/src/fast_aggs.rs@@ -253,19 +253,17 @@ impl SegEntry {/// only the top-level keys of `_source`, so a scalar leaf nested under an/// object has no column of its own either.///-/// The consequence, stated honestly because an earlier draft of this comment-/// got it wrong: this predicate closes the MULTI-FIELD-SUFFIX divergence,-/// where the parent that owns a column is a real top-level field-/// (`extension.raw.raw` -> `extension`). It does NOT close the general-/// nested-OBJECT divergence: `geo.city`, a genuine nested value that brute's-/// `get_field_value` resolves by recursing into the object, has no ancestor-/// column at any depth, so this returns `false` and the fast path answers it-/// as empty when brute would not. That is a separate, pre-existing divergence-/// (it is identical before and after this change — measured `geo.city`-/// predicate: fast 300 vs brute 11 300 on both), and it is tracked apart from-/// the suffix problem this function is scoped to. A genuinely absent sub-path-/// (`geo.missing`) is indistinguishable from a present one here for the same-/// reason.+/// The scope, stated honestly because an earlier draft got it wrong: this+/// predicate closes only the MULTI-FIELD-SUFFIX divergence, where the parent+/// that owns a column is a real top-level field (`extension.raw.raw` ->+/// `extension`). It CANNOT see the nested-OBJECT divergence (`geo.city`, which+/// brute resolves by recursing into the object) or the suppressed-array one+/// (`tags`, whose column is dropped for being multi-valued): neither has an+/// ancestor column at any depth, so this returns `false` for both, and a+/// genuinely absent sub-path (`geo.missing`) is indistinguishable from a+/// present one here. Those two are closed instead by the schema-keyed root+/// check in [`field_needs_brute_fallback`] (#128), which is why the predicate+/// path routes its column misses through that function rather than this one.fn dotted_suffix_diverges_from_brute(cols: &super::DocValueMap, field: &str) -> bool {if dv_col(cols, field).is_some() {return false;@@ -280,34 +278,38 @@ fn dotted_suffix_diverges_from_brute(cols: &super::DocValueMap, field: &str) ->false}-/// True when `field` is a dotted JSON path (`geo.dest`, `machine.ram`,-/// `machine.os.keyword`) that fast_aggs cannot resolve to any column —-/// neither the exact name nor its `.keyword`-stripped form (see-/// `SegEntry::col`) — AND its top-level root is mapped `object`/`nested` in-/// the schema. That combination means the field is a genuine nested-/// sub-path fast_aggs' flat columnar storage structurally cannot reach, not-/// a field that's simply absent from the index — an absent field's root-/// isn't schema-mapped at all, and IS a valid empty/null result worth-/// returning. A free function (not a `FastCtx` method) so it's testable+/// True when `field` is a name fast_aggs cannot resolve to any column at any+/// depth — neither the exact name nor its `.keyword`-stripped form (see+/// `SegEntry::col`) in any segment — yet brute's `get_field_value` still+/// resolves it, so answering it columnarly would report empty where brute+/// reports data. A free function (not a `FastCtx` method) so it's testable/// without constructing a full `FastCtx`, which needs a real `&Index`.///-/// `object_fields` holds the names of every schema field mapped `object` or-/// `nested` — see `FastCtx::object_fields` for why a schema-based check is-/// required here instead of a physical-column one (segments never carry a-/// column for an object/nested field itself, only for its scalar leaves, so-/// "does the dotted path's root resolve as a column" is always false and-/// can't distinguish a genuinely nested path from a genuinely absent one).+/// Three ways brute out-resolves the columnar path, all bailed here:///-/// It also covers the second, schema-independent case: a multi-field suffix-/// (of any depth) other than `.keyword` that a brute resolver resolves by-/// stripping and the columnar path does not — see-/// [`dotted_suffix_diverges_from_brute`]. That check is per-segment and-/// runs FIRST, because a name that resolves exactly in one segment but only-/// via an ancestor in another still diverges on the second segment's rows.+/// * a multi-field suffix (of any depth) other than `.keyword` that a brute+/// resolver strips and the columnar path does not — see+/// [`dotted_suffix_diverges_from_brute`]. That check is per-segment and+/// runs FIRST, because a name that resolves exactly in one segment but only+/// via an ancestor in another still diverges on the second segment's rows;+/// * a scalar leaf nested under an object (`geo.city`, `machine.os.keyword`):+/// its root (`geo` / `machine`) is a mapped field, but no segment ever+/// carries a column for an object field itself, so brute recurses into the+/// `_source` object while the columnar path has nothing;+/// * a TOP-LEVEL field whose column was SUPPRESSED because it is multi-valued+/// (`tags` / `tags.raw`): `build_doc_value_columns` ships NO column for an+/// array-poisoned field, so brute reads the `_source` array while the+/// columnar path sees no column — indistinguishable at the column level+/// from a genuine absence, hence the schema check below.+///+/// `mapped_fields` — see `FastCtx::mapped_fields` — is the schema signal that+/// separates those last two (present-but-uncolumned) from a genuine absence:+/// a field whose name AND top-level root are both unmapped was never indexed,+/// and IS a valid empty/null result worth returning on the fast path.fn field_needs_brute_fallback(field: &str,segs: &[SegEntry],- object_fields: &std::collections::HashSet<String>,+ mapped_fields: &std::collections::HashSet<String>,) -> bool {if segs.iter()@@ -318,10 +320,14 @@ fn field_needs_brute_fallback(if segs.iter().any(|s| s.col(field).is_some()) {return false;}- let Some(dot) = field.find('.') else {- return false;- };- object_fields.contains(&field[..dot])+ // No column at any depth in any segment. Bail iff brute would still+ // resolve it — i.e. the exact name (a suppressed top-level array such as+ // `tags`, or a flat dotted field name) or, for a dotted path, its+ // top-level root (`geo` for `geo.city`, `tags` for `tags.raw`) is a+ // field the schema maps. A name whose root is unmapped is genuinely+ // absent and stays on the fast path as a real empty result.+ let root = field.split_once('.').map_or(field, |(r, _)| r);+ mapped_fields.contains(field) || mapped_fields.contains(root)}/// Memtable docs for the fast path.@@ -398,13 +404,19 @@ pub(super) struct FastCtx<'a> {/// the brute path renders them as "false"/"true" term keys — so the/// terms executor needs the mapping to reproduce that.bool_fields: &'a std::collections::HashSet<String>,- /// Schema fields mapped `object`/`nested`. No segment ever carries a- /// column for an object field itself (only for its scalar leaves, and- /// only when those leaves happen to be top-level columns — which nested- /// JSON leaves never are), so a dotted field whose root is in this set- /// is a genuinely mapped path the columnar path structurally can't- /// serve, not a genuinely absent field. See `field_needs_brute_fallback`.- object_fields: &'a std::collections::HashSet<String>,+ /// Every field name the schema maps. Dynamic mapping only ever maps+ /// TOP-LEVEL keys, so a nested leaf (`geo.city`) is reached through its+ /// root (`geo`), which IS in this set. A field with no doc-value column+ /// at any depth in any segment, whose name OR top-level root is in this+ /// set, is one brute's `get_field_value` still resolves — an object's+ /// nested leaf (`geo.city`, root `geo` mapped object), or a top-level+ /// field whose column was SUPPRESSED because it is multi-valued (`tags`+ /// / `tags.raw`, mapped keyword, array-poisoned in+ /// `build_doc_value_columns`) — so the columnar path must bail rather+ /// than answer it empty. A name whose root is absent from this set is+ /// genuinely absent and stays on the fast path. See+ /// `field_needs_brute_fallback`.+ mapped_fields: &'a std::collections::HashSet<String>,/// Top-level query filter (from `{size:0, query:Q, aggs:…}`), compiled to/// a columnar predicate. `None` == match_all (the whole corpus). When/// present, EVERY executor restricts its columnar reduction to matching@@ -642,7 +654,7 @@ impl Index {snap: &xerj_storage::index_store::IndexSnapshot,segments_dir: &std::path::Path,bool_fields: &std::collections::HashSet<String>,- object_fields: &std::collections::HashSet<String>,+ mapped_fields: &std::collections::HashSet<String>,) -> Option<(Value, Option<u64>)> {if fast_aggs_disabled() {return None;@@ -735,7 +747,7 @@ impl Index {mem_docs: std::sync::OnceLock::new(),needs_owned_mem,bool_fields,- object_fields,+ mapped_fields,top_filter,top_filter_query,top_filter_mem_preds,@@ -754,7 +766,7 @@ impl Index {Some(pred) => {let mut total: u64 = 0;for seg in &ctx.segs {- let sp = resolve_pred(&seg.cols, pred)?;+ let sp = resolve_pred(&seg.cols, pred, &ctx.segs, ctx.mapped_fields)?;total += seg_pred_count(&sp, seg.docs);}if let Some(q) = &ctx.top_filter_query {@@ -1199,7 +1211,7 @@ impl<'a> FastCtx<'a> {mem_docs: std::sync::OnceLock::new(),needs_owned_mem: self.needs_owned_mem,bool_fields: self.bool_fields,- object_fields: self.object_fields,+ mapped_fields: self.mapped_fields,top_filter: None,top_filter_query: None,top_filter_mem_preds: None,@@ -1227,7 +1239,7 @@ impl<'a> FastCtx<'a> {}fn field_is_structurally_unsupported(&self, field: &str) -> bool {- field_needs_brute_fallback(field, &self.segs, self.object_fields)+ field_needs_brute_fallback(field, &self.segs, self.mapped_fields)}fn exec_agg(&self, agg_type: &str, params: &Value, sub: Option<&Value>) -> Option<Value> {@@ -1589,7 +1601,7 @@ impl<'a> FastCtx<'a> {// `percentile_ranks` / `median_absolute_deviation` onto the O(N)// `_source` scan.let seg_pred = match &self.top_filter {- Some(p) => Some(resolve_pred(&s.cols, p)?),+ Some(p) => Some(resolve_pred(&s.cols, p, &self.segs, self.mapped_fields)?),None => None,};match s.col(field) {@@ -1676,7 +1688,7 @@ impl<'a> FastCtx<'a> {// under a filter would have reported whole-index moments for a// filtered query (silently wrong, not merely slow).let seg_pred = match &self.top_filter {- Some(p) => Some(resolve_pred(&s.cols, p)?),+ Some(p) => Some(resolve_pred(&s.cols, p, &self.segs, self.mapped_fields)?),None => None,};match s.col(field) {@@ -2201,7 +2213,12 @@ impl<'a> FastCtx<'a> {// every fused-pass consumer (terms, filtered metric) reduces over// exactly the matching doc set.let top_sp: Option<SegPred> = match &self.top_filter {- Some(pred) => Some(resolve_pred(&seg.cols, pred)?),+ Some(pred) => Some(resolve_pred(+ &seg.cols,+ pred,+ &self.segs,+ self.mapped_fields,+ )?),None => None,};// Resolve metric columns once.@@ -3593,7 +3610,7 @@ impl<'a> FastCtx<'a> {let mut accs: Vec<MetricAcc> = vec![MetricAcc::default(); plan.metrics.len()];for si in 0..self.segs.len() {let seg = &self.segs[si];- let sp = resolve_pred(&seg.cols, &pred)?;+ let sp = resolve_pred(&seg.cols, &pred, &self.segs, self.mapped_fields)?;if plan.metrics.is_empty() {count += seg_pred_count(&sp, seg.docs);} else {@@ -3685,7 +3702,7 @@ impl<'a> FastCtx<'a> {let mut accs: Vec<MetricAcc> = vec![MetricAcc::default(); plan.metrics.len()];for si in 0..self.segs.len() {let seg = &self.segs[si];- let sp = resolve_pred(&seg.cols, &pred)?;+ let sp = resolve_pred(&seg.cols, &pred, &self.segs, self.mapped_fields)?;if plan.metrics.is_empty() {count += seg_pred_count(&sp, seg.docs);} else {@@ -3749,7 +3766,7 @@ impl<'a> FastCtx<'a> {for seg in &self.segs {let sps: Vec<SegPred<'_>> = preds.iter()- .map(|p| resolve_pred(&seg.cols, p))+ .map(|p| resolve_pred(&seg.cols, p, &self.segs, self.mapped_fields)).collect::<Option<Vec<_>>>()?;for row in 0..seg.docs {let mut mask: u32 = 0;@@ -5244,15 +5261,24 @@ fn compile_pred(filter: &Value) -> Option<Pred> {////// `Some(SegPred::Never)` is an ASSERTION that the segment genuinely holds/// no matching row. That assertion only holds when EVERY brute resolver-/// would also find nothing under this name. For a multi-field suffix one-/// of them resolves by stripping it is false — and the widest of them,+/// would also find nothing under this name. It is false for a multi-field+/// suffix one of them resolves by stripping (the widest,/// `index.rs::get_field_value` behind a top-level query filter, strips-/// trailing segments recursively, so `<field>.raw.raw` counts too.-/// Returning `Never` there is how a `filters`/`filter`/top-level-query+/// trailing segments recursively, so `<field>.raw.raw` counts too); for a+/// scalar leaf nested under an object (`geo.city`, resolved by recursing+/// into the `_source` object); and for a top-level field whose column was+/// suppressed because it is multi-valued (`tags`, read straight from the+/// `_source` array). All three are exactly what+/// [`field_needs_brute_fallback`] detects, so this defers to it: returning+/// `Never` for any of them is how a `filters`/`filter`/top-level-query/// predicate reported a small plausible count instead of the real one./// `None` bails the whole request to the exact brute path.-fn miss_is_genuinely_empty<'a>(cols: &super::DocValueMap, field: &str) -> Option<SegPred<'a>> {- if dotted_suffix_diverges_from_brute(cols, field) {+fn miss_is_genuinely_empty<'a>(+ field: &str,+ segs: &[SegEntry],+ mapped_fields: &std::collections::HashSet<String>,+) -> Option<SegPred<'a>> {+ if field_needs_brute_fallback(field, segs, mapped_fields) {None} else {Some(SegPred::Never)@@ -5283,8 +5309,15 @@ enum SegPred<'a> {/// turned a `<field>.keyword` filter into a silent segment-wide `Never`.////// A miss on a name the brute reference WOULD resolve is therefore not a-/// `Never` at all — see [`miss_is_genuinely_empty`].-fn resolve_pred<'a>(cols: &'a super::DocValueMap, pred: &Pred) -> Option<SegPred<'a>> {+/// `Never` at all — see [`miss_is_genuinely_empty`], which needs the whole+/// segment set and the schema's mapped-field names to tell a genuine absence+/// from a nested/suppressed field the columns simply cannot represent.+fn resolve_pred<'a>(+ cols: &'a super::DocValueMap,+ pred: &Pred,+ segs: &[SegEntry],+ mapped_fields: &std::collections::HashSet<String>,+) -> Option<SegPred<'a>> {Some(match pred {Pred::MatchAll => SegPred::Always,Pred::TermKw { field, value } => match dv_col(cols, field) {@@ -5293,7 +5326,7 @@ fn resolve_pred<'a>(cols: &'a super::DocValueMap, pred: &Pred) -> Option<SegPredNone => SegPred::Never,},Some(Column::Numeric(_)) => return None,- None => miss_is_genuinely_empty(cols, field)?,+ None => miss_is_genuinely_empty(field, segs, mapped_fields)?,},Pred::TermsKw { field, values } => match dv_col(cols, field) {Some(Column::Keyword(k)) => {@@ -5305,7 +5338,7 @@ fn resolve_pred<'a>(cols: &'a super::DocValueMap, pred: &Pred) -> Option<SegPred}}Some(Column::Numeric(_)) => return None,- None => miss_is_genuinely_empty(cols, field)?,+ None => miss_is_genuinely_empty(field, segs, mapped_fields)?,},Pred::RangeKw {field,@@ -5365,7 +5398,7 @@ fn resolve_pred<'a>(cols: &'a super::DocValueMap, pred: &Pred) -> Option<SegPred}}Some(Column::Numeric(_)) => return None,- None => miss_is_genuinely_empty(cols, field)?,+ None => miss_is_genuinely_empty(field, segs, mapped_fields)?,},Pred::RangeNum {field,@@ -5378,12 +5411,12 @@ fn resolve_pred<'a>(cols: &'a super::DocValueMap, pred: &Pred) -> Option<SegPredSegPred::Num(n, *lo, *lo_incl, *hi, *hi_incl, n.null_bitmap.is_empty())}Some(Column::Keyword(_)) => return None,- None => miss_is_genuinely_empty(cols, field)?,+ None => miss_is_genuinely_empty(field, segs, mapped_fields)?,},Pred::And(subs) => {let mut resolved: Vec<SegPred<'a>> = Vec::with_capacity(subs.len());for s in subs {- let sp = resolve_pred(cols, s)?;+ let sp = resolve_pred(cols, s, segs, mapped_fields)?;if matches!(sp, SegPred::Never) {return Some(SegPred::Never); // short-circuit: whole AND is empty}@@ -5836,8 +5869,13 @@ mod range_kw_tests {fn suffixed_predicate_bails_instead_of_matching_nothing() {let k = KeywordColumn::from_iter(vec![Some("css".to_string()), Some("gz".to_string())]).expect("build column");- let mut m = std::collections::BTreeMap::new();- m.insert("extension".to_string(), Column::Keyword(k));+ let mut cols = std::collections::BTreeMap::new();+ cols.insert("extension".to_string(), Column::Keyword(k));+ // `resolve_pred` now routes a column miss through+ // `field_needs_brute_fallback`, which scans the whole segment set for+ // the ancestor column, so the fixture must be a real segment.+ let segs = vec![seg_with_cols(cols)];+ let mapped = mapped_fields(&["extension"]);for field in ["extension.raw","extension.raw.raw",@@ -5850,7 +5888,7 @@ mod range_kw_tests {value: "css".into(),};assert!(- resolve_pred(&m, &pred).is_none(),+ resolve_pred(&segs[0].cols, &pred, &segs, &mapped).is_none(),"`{field}` must bail the request to brute, not resolve to Never");}@@ -5860,7 +5898,10 @@ mod range_kw_tests {value: "css".into(),};assert!(- matches!(resolve_pred(&m, &absent), Some(SegPred::Never)),+ matches!(+ resolve_pred(&segs[0].cols, &absent, &segs, &mapped),+ Some(SegPred::Never)+ ),"a genuinely absent dotted field must stay on the fast path as Never");}@@ -5886,14 +5927,14 @@ mod range_kw_tests {"extension.raw.raw.raw",] {assert!(- field_needs_brute_fallback(field, &[build()], &object_fields(&[])),+ field_needs_brute_fallback(field, &[build()], &mapped_fields(&[])),"a `terms`/metric target on `{field}` must bail to brute — the \executors `continue` past a column-less segment, so it would \otherwise be built from the memtable alone");}assert!(- !field_needs_brute_fallback("nosuchfield.raw", &[build()], &object_fields(&[])),+ !field_needs_brute_fallback("nosuchfield.raw", &[build()], &mapped_fields(&[])),"a dotted field with no ancestor column anywhere is a genuine \absence and must stay on the fast path");@@ -5907,7 +5948,7 @@ mod range_kw_tests {}}- fn object_fields(names: &[&str]) -> std::collections::HashSet<String> {+ fn mapped_fields(names: &[&str]) -> std::collections::HashSet<String> {names.iter().map(|s| s.to_string()).collect()}@@ -5930,7 +5971,7 @@ mod range_kw_tests {let cols = std::collections::BTreeMap::new();let seg = seg_with_cols(cols);assert!(- field_needs_brute_fallback("geo.dest", &[seg], &object_fields(&["geo"])),+ field_needs_brute_fallback("geo.dest", &[seg], &mapped_fields(&["geo"])),"geo.dest has no column of its own and geo is mapped object — must bail to brute force");}@@ -5944,7 +5985,7 @@ mod range_kw_tests {let cols = std::collections::BTreeMap::new();let seg = seg_with_cols(cols);assert!(- field_needs_brute_fallback("machine.os.keyword", &[seg], &object_fields(&["machine"])),+ field_needs_brute_fallback("machine.os.keyword", &[seg], &mapped_fields(&["machine"])),"machine.os.keyword's root (machine) is mapped object — must bail to brute force");}@@ -5959,30 +6000,59 @@ mod range_kw_tests {let cols = std::collections::BTreeMap::new();let seg = seg_with_cols(cols);assert!(- !field_needs_brute_fallback("nonexistent.field", &[seg], &object_fields(&["geo"])),+ !field_needs_brute_fallback("nonexistent.field", &[seg], &mapped_fields(&["geo"])),"the field's root isn't mapped object at all — a real absence, not a nested-path gap");}- /// A field with no dot at all is never flagged, regardless of whether- /// it resolves — this guard only concerns dotted nested paths.+ /// A flat field that the schema does NOT map is a genuine absence — no+ /// column, no `_source` value for brute to find either — so it is never+ /// flagged and stays on the fast path as a real empty result.#[test]- fn flat_field_never_needs_fallback() {+ fn unmapped_flat_field_does_not_need_fallback() {let cols = std::collections::BTreeMap::new();let seg = seg_with_cols(cols);assert!(!field_needs_brute_fallback("extension",&[seg],- &object_fields(&[])+ &mapped_fields(&[])+ ));+ }++ /// #128: a TOP-LEVEL field the schema maps but whose column was SUPPRESSED+ /// in every segment (an array-poisoned multi-valued field like `tags`, for+ /// which `build_doc_value_columns` ships no column) must bail — brute's+ /// `get_field_value` reads the `_source` array, so a columnar `Never`+ /// would report empty where brute reports data. Both the flat name and its+ /// `.raw` multi-field are covered, because the schema check keys off the+ /// top-level root.+ #[test]+ fn suppressed_top_level_array_field_needs_fallback() {+ let seg = seg_with_cols(std::collections::BTreeMap::new());+ for field in ["tags", "tags.raw"] {+ assert!(+ field_needs_brute_fallback(+ field,+ &[seg_with_cols(std::collections::BTreeMap::new())],+ &mapped_fields(&["tags"])+ ),+ "`{field}` is a suppressed multi-valued field — brute resolves \+ it, so the columnar path must bail"+ );+ }+ // Control: a genuinely unmapped sibling stays fast.+ assert!(!field_needs_brute_fallback(+ "nosuch.raw",+ &[seg],+ &mapped_fields(&["tags"])));}/// The already-fixed `.keyword` multi-field case (`extension.keyword`)/// must NOT be flagged — `SegEntry::col` already resolves it via the- /// `.keyword`-stripped fallback, so the fast path stays fast for this- /// shape exactly as before this guard was added. `extension` is a- /// scalar keyword field, not an object, so it wouldn't appear in- /// `object_fields` in real use either.+ /// `.keyword`-stripped fallback (the physical-column check short-circuits+ /// before the schema probe), so the fast path stays fast for this shape+ /// exactly as before this guard was added.#[test]fn keyword_multi_field_does_not_need_fallback() {let mut cols = std::collections::BTreeMap::new();@@ -5994,7 +6064,7 @@ mod range_kw_tests {assert!(!field_needs_brute_fallback("extension.keyword",&[seg],- &object_fields(&[])+ &mapped_fields(&[])));}@@ -6014,12 +6084,16 @@ mod range_kw_tests {assert!(!field_needs_brute_fallback("geo.keyword",&[seg],- &object_fields(&["geo"])+ &mapped_fields(&["geo"])));}fn count(pred: &Pred, cols: &std::collections::BTreeMap<String, Column>, docs: u32) -> u64 {- let sp = resolve_pred(cols, pred).expect("resolvable");+ // These fixtures all resolve (or bail inside `resolve_pred` on a+ // type/width gate), never through the column-miss path, so the+ // segment set and schema map are unused here.+ let sp =+ resolve_pred(cols, pred, &[], &std::collections::HashSet::new()).expect("resolvable");let by_count = seg_pred_count(&sp, docs);let by_scan = (0..docs).filter(|r| seg_pred_matches(&sp, *r)).count() as u64;assert_eq!(by_count, by_scan, "seg_pred_count disagrees with row scan");@@ -6101,7 +6175,7 @@ mod range_kw_tests {m.insert("ts".to_string(), Column::Keyword(k));let p = rng(Some(&ts(2, 0)), true, Some(&ts(3, 0)), false);assert!(- resolve_pred(&m, &p).is_none(),+ resolve_pred(&m, &p, &[], &std::collections::HashSet::new()).is_none(),"ragged dictionary must bail");}@@ -6111,7 +6185,10 @@ mod range_kw_tests {fn unnormalised_bound_bails_to_brute() {let c = cols();let p = rng(Some("2026-06-02"), true, Some("2026-06-03"), false);- assert!(resolve_pred(&c, &p).is_none(), "short bound must bail");+ assert!(+ resolve_pred(&c, &p, &[], &std::collections::HashSet::new()).is_none(),+ "short bound must bail"+ );}/// A numeric column under a lexicographic range must bail (the date/keyword@@ -6123,7 +6200,7 @@ mod range_kw_tests {let n = NumericColumn::from_iter((0..10).map(|i| Some((i as f64).to_bits() as i64)));m.insert("ts".to_string(), Column::Numeric(n));let p = rng(Some(&ts(2, 0)), true, None, true);- assert!(resolve_pred(&m, &p).is_none());+ assert!(resolve_pred(&m, &p, &[], &std::collections::HashSet::new()).is_none());}/// Regression: the *predicate* resolver must apply the same `.keyword`@@ -6252,7 +6329,7 @@ mod range_kw_tests {value: "x".into(),};assert!(matches!(- resolve_pred(&c, &p).expect("resolvable"),+ resolve_pred(&c, &p, &[], &std::collections::HashSet::new()).expect("resolvable"),SegPred::Never));}diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 56a073d7f..7b158b543 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -20008,32 +20008,33 @@ impl Index {// false/true", so snapshot the boolean field names here (async// lock) and hand them to the sync executor.//- // Object/nested-mapped fields never get a `.dv` column of their own- // (only their scalar leaves would, and nested leaves never become- // top-level columns at all) — a dotted field whose root is in this- // set (e.g. `geo.dest` under object `geo`) is a genuinely mapped- // path the columnar executors structurally can't reach, not a- // genuinely absent field, so it must fall back to the brute path- // instead of silently returning empty/null.- let (bool_fields, object_fields): (+ // `mapped_fields` is every field name the schema knows (dynamic+ // mapping only ever maps top-level keys). A field that has no `.dv`+ // column at any depth in any segment yet whose name — or, for a+ // dotted path, top-level root — is in this set is one brute's+ // `get_field_value` still resolves, and the columnar path structurally+ // cannot: a scalar leaf nested under an object (`geo.dest` under+ // object `geo`, resolved by recursing into `_source`), or a top-level+ // field whose column was SUPPRESSED because it is multi-valued (`tags`+ // / `tags.raw`, read straight from the `_source` array). Both must+ // fall back to the brute path instead of silently returning+ // empty/null; a name whose root is unmapped is genuinely absent and a+ // valid empty result. Keyed off the schema, not a column probe, since+ // by construction neither of those fields carries a column to probe.+ let (bool_fields, mapped_fields): (std::collections::HashSet<String>,std::collections::HashSet<String>,) = {let guard = self.schema.read().await;let mut bool_fields = std::collections::HashSet::new();- let mut object_fields = std::collections::HashSet::new();+ let mut mapped_fields = std::collections::HashSet::new();for f in &guard.schema.fields {- match f.field_type {- xerj_common::FieldType::Boolean => {- bool_fields.insert(f.name.clone());- }- xerj_common::FieldType::Object | xerj_common::FieldType::Nested => {- object_fields.insert(f.name.clone());- }- _ => {}+ if f.field_type == xerj_common::FieldType::Boolean {+ bool_fields.insert(f.name.clone());}+ mapped_fields.insert(f.name.clone());}- (bool_fields, object_fields)+ (bool_fields, mapped_fields)};self.try_fast_aggs(aggs_def,@@ -20041,7 +20042,7 @@ impl Index {snap,segments_dir,&bool_fields,- &object_fields,+ &mapped_fields,)}}diff --git a/engine/crates/xerj-engine/tests/fast_aggs_nested_object.rs b/engine/crates/xerj-engine/tests/fast_aggs_nested_object.rsnew file mode 100644index 000000000..f763a8342--- /dev/null+++ b/engine/crates/xerj-engine/tests/fast_aggs_nested_object.rs@@ -0,0 +1,279 @@+//! Regression (#128): the columnar fast path answered a predicate or agg+//! TARGET on a field it has no doc-value column for AT ANY DEPTH as EMPTY,+//! while brute's `get_field_value` still resolves it — so fast and brute+//! disagreed. Two distinct shapes, one root cause (`build_doc_value_columns`+//! builds columns only for top-level SCALAR keys):+//!+//! * a scalar leaf nested under an OBJECT (`geo.city`, `geo` mapped object):+//! no column for `geo` itself, none for `geo.city`; brute recurses into the+//! `_source` object. The #120 `dotted_suffix_diverges_from_brute` bail only+//! fires when an ANCESTOR owns a column, which a nested object never does,+//! so the predicate resolved to `SegPred::Never` (segment matches nothing);+//!+//! * a TOP-LEVEL field whose column was SUPPRESSED because it is multi-valued+//! (`tags` / `tags.raw`): `build_doc_value_columns` ships NO column for an+//! array-poisoned field, so the columnar path sees no column and again+//! answered `Never`, while brute reads the `_source` array.+//!+//! Both were identical before and after #120. The 300-vs-11,300 figures in+//! issue #128 came from a 12,000-flushed + 200-memtable corpus; the tests+//! below use a smaller fixture and assert fast/brute AGREEMENT rather than+//! those exact counts, so the numbers are the motivating measurement, not this+//! file's.+//!+//! The fix keeps columnar resolution narrow — there is no single brute+//! behaviour to widen TO — and instead BAILS to the brute path whenever a+//! field has no column at any depth in any segment yet its name (or, for a+//! dotted path, its top-level root) is one the schema maps, i.e. one brute's+//! `get_field_value` would still resolve. After a bail the request is answered+//! by the very resolver it would have been measured against.+//!+//! The invariant under test is fast-path/brute-path agreement; the absolute+//! assertions exist only so a change that makes BOTH paths return the same+//! wrong (empty) answer cannot pass. The divergence tests (nested-object and+//! suppressed-array) FAIL on the pre-fix source, where the fast leg keeps the+//! request and answers it from the memtable alone while the brute leg reads+//! the full corpus. `genuinely_absent_field_is_empty_on_both_paths` is a+//! control, not a divergence test: it passes on the pre-fix source too, and+//! exists to prove the bail does not fire for a field that is genuinely+//! absent everywhere.++use serde_json::{json, Value};+use tempfile::TempDir;+use xerj_common::config::Config;+use xerj_common::types::Schema;+use xerj_engine::Engine;+use xerj_query::parse_request;++const CITIES: [&str; 4] = ["paris", "london", "berlin", "tokyo"];++fn make_engine(dir: &TempDir) -> Engine {+ let mut config = Config::default();+ config.server.data_dir = dir.path().to_str().unwrap().to_string();+ Engine::new(config).expect("engine::new")+}++/// 12 000 flushed docs + 200 still in the memtable — past `FAST_AGG_MIN_DOCS`+/// (10 000), so the columnar path is the one under test, and split across both+/// arms so a segment-only defect cannot hide behind a correct memtable answer.+///+/// Flushed: `geo.city` cycles the four `CITIES` (3 000 each). `tags` is the+/// two-element array `[city, "all"]` — a genuine multi-valued field, so its+/// `.dv` column is suppressed in the flushed segment. `v` is 1.0 everywhere,+/// so a filtered `sum` equals the match count. Memtable: 200 more `paris`+/// docs, so `paris` totals 3 200 while the other cities stay at 3 000, and a+/// segment leg that silently contributes zero is visible against the+/// non-zero memtable leg.+async fn seed(idx: &std::sync::Arc<xerj_engine::Index>) {+ for i in 0..12_000u32 {+ let city = CITIES[(i % 4) as usize];+ idx.index_document(+ Some(i.to_string()),+ json!({+ "geo": { "city": city },+ "tags": [city, "all"],+ "v": 1.0,+ "i": i,+ }),+ )+ .await+ .unwrap();+ }+ idx.flush().await.unwrap();+ for i in 12_000..12_200u32 {+ idx.index_document(+ Some(i.to_string()),+ json!({+ "geo": { "city": "paris" },+ "tags": ["paris", "all"],+ "v": 1.0,+ "i": i,+ }),+ )+ .await+ .unwrap();+ }+}++/// Force every subsequent agg request onto the brute `_source` path WITHOUT+/// changing a single live value: re-index one existing doc with a+/// byte-identical body. That records an overwrite in the version map, and+/// `ghost_events() > 0` is exactly the gate `try_fast_aggs` bails on. The live+/// corpus is unchanged, so the brute answer is the reference answer for the+/// same data the fast path just aggregated.+async fn force_brute(idx: &std::sync::Arc<xerj_engine::Index>) {+ idx.index_document(+ Some("0".to_string()),+ json!({+ "geo": { "city": CITIES[0] },+ "tags": [CITIES[0], "all"],+ "v": 1.0,+ "i": 0,+ }),+ )+ .await+ .unwrap();+}++async fn run(idx: &std::sync::Arc<xerj_engine::Index>, body: Value) -> Value {+ let req = parse_request(&body).unwrap();+ let res = idx.search(&req).await.unwrap();+ json!({+ "total": res.total.value,+ "aggs": res.aggs.clone().unwrap_or(json!(null)),+ })+}++/// Run `body` twice against an identical corpus — once with the columnar path+/// live, once with it disabled — and return the brute answer after asserting+/// the two agree.+async fn agree(name: &str, body: fn() -> Value, why: &str) -> Value {+ let dir = TempDir::new().unwrap();+ let engine = make_engine(&dir);+ engine.create_index(name, Schema::empty()).unwrap();+ let idx = engine.get_index(name).unwrap();+ seed(&idx).await;++ let fast = run(&idx, body()).await;+ force_brute(&idx).await;+ let brute = run(&idx, body()).await;++ assert_eq!(fast, brute, "{why}");+ brute+}++// ── Predicates (top-level query filter — moves `hits.total`) ─────────────────++fn geo_city_query() -> Value {+ json!({+ "query": { "term": { "geo.city": "paris" } },+ "size": 0,+ "aggs": { "sum_v": { "sum": { "field": "v" } } }+ })+}++fn tags_query() -> Value {+ json!({+ "query": { "term": { "tags": "paris" } },+ "size": 0,+ "aggs": { "sum_v": { "sum": { "field": "v" } } }+ })+}++fn tags_raw_query() -> Value {+ json!({+ "query": { "term": { "tags.raw": "paris" } },+ "size": 0,+ "aggs": { "sum_v": { "sum": { "field": "v" } } }+ })+}++/// Predicate on a scalar leaf nested under an object. `geo` is mapped object,+/// so no column exists for it at any depth; brute recurses into `_source`.+#[tokio::test]+async fn fast_and_brute_agree_on_nested_object_predicate() {+ let brute = agree(+ "geocitypred",+ geo_city_query,+ "columnar fast path disagrees with the brute reference on a top-level \+ `term` query against a nested-object field `geo.city`",+ )+ .await;+ // 3 000 flushed `paris` + 200 memtable, each with v == 1.0.+ assert_eq!(brute["total"], 3_200);+ assert_eq!(brute["aggs"]["sum_v"]["value"], 3_200.0);+}++/// Predicate on a top-level array field whose `.dv` column is suppressed+/// because it is multi-valued.+#[tokio::test]+async fn fast_and_brute_agree_on_suppressed_array_predicate() {+ let brute = agree(+ "tagspred",+ tags_query,+ "columnar fast path disagrees with the brute reference on a top-level \+ `term` query against a suppressed multi-valued field `tags`",+ )+ .await;+ assert_eq!(brute["total"], 3_200);+ assert_eq!(brute["aggs"]["sum_v"]["value"], 3_200.0);+}++/// The `.raw` multi-field of the same suppressed array: brute's top-level+/// query resolver strips the suffix and reads `tags`, so the columnar path+/// must reach the same fallback via the field's root.+#[tokio::test]+async fn fast_and_brute_agree_on_suppressed_array_raw_suffix_predicate() {+ let brute = agree(+ "tagsrawpred",+ tags_raw_query,+ "columnar fast path disagrees with the brute reference on a top-level \+ `term` query against `tags.raw` (suppressed array, `.raw` multi-field)",+ )+ .await;+ assert_eq!(brute["total"], 3_200);+ assert_eq!(brute["aggs"]["sum_v"]["value"], 3_200.0);+}++// ── Aggregation TARGET on the suppressed array ───────────────────────────────++/// A `terms` aggregation TARGETING the suppressed array. Broader than the+/// predicate case: the executors `continue` past a column-less segment instead+/// of bailing, so pre-fix the buckets were built from the memtable alone.+#[tokio::test]+async fn fast_and_brute_agree_on_suppressed_array_terms_target() {+ let brute = agree(+ "tagsterms",+ || {+ json!({+ "query": { "match_all": {} },+ "size": 0,+ "aggs": { "by_tag": { "terms": { "field": "tags", "size": 10 } } }+ })+ },+ "columnar fast path disagrees with the brute reference on a `terms` \+ aggregation TARGETING the suppressed multi-valued field `tags`",+ )+ .await;+ let buckets = brute["aggs"]["by_tag"]["buckets"]+ .as_array()+ .unwrap()+ .clone();+ // Multi-valued: every one of the 12 200 docs carries `"all"`, and each of+ // the four cities is carried by its own docs (`paris` 3 200, the rest+ // 3 000). A memtable-only answer could never produce these counts.+ let all = buckets+ .iter()+ .find(|b| b["key"] == "all")+ .expect("all bucket");+ assert_eq!(all["doc_count"], 12_200);+ let paris = buckets+ .iter()+ .find(|b| b["key"] == "paris")+ .expect("paris bucket");+ assert_eq!(paris["doc_count"], 3_200);+ assert_eq!(buckets.len(), 5);+}++// ── The narrow-resolution half of the contract ───────────────────────────────++/// A genuinely absent field (root unmapped) must read empty on BOTH paths+/// WITHOUT the fast path bailing — this is what stops the fix from degenerating+/// into "every uncolumned field goes brute". `nosuchfield.city` is never+/// indexed, so its root isn't in the schema, and empty is the correct answer.+#[tokio::test]+async fn genuinely_absent_field_is_empty_on_both_paths() {+ let brute = agree(+ "absentnested",+ || {+ json!({+ "query": { "term": { "nosuchfield.city": "paris" } },+ "size": 0,+ "aggs": { "sum_v": { "sum": { "field": "v" } } }+ })+ },+ "a field whose root the schema never mapped must read empty on both paths",+ )+ .await;+ assert_eq!(brute["total"], 0);+}