AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
perf(engine): drop the term dictionary for dense_vector, and the doc-scan fallback with it (#328)
xerj-orgb699657e17e3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c46c5dcfb..a688ec81d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -153,6 +153,132 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0from the moment it is allowlisted. Seed the target with snapshot/restorefirst if it needs the existing documents.+### Performance++- **A field mapped `dense_vector` no longer gets a full-text term dictionary**+ ([#328](https://github.com/xerj-org/xerj/issues/328)). The field was fed+ through the lexical indexing path and given an FST + postings that no query+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from+ `_source`/doc values, and `_field_caps` and highlighting never open the+ field's term dictionary. `extract_field_text`'s array arm joined the+ components with spaces, so a 128-dim vector became one enormous+ decimal-string term per document.++ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long ++ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing+ every file in the data directory except the write-ahead log:+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are+ byte-identical on both sides.++ `.wal` is excluded from that total because its tail is reclaimed+ asynchronously and it is not reproducible: on the 300-doc fixture the+ committed regression test uses, eight runs put the *whole directory* anywhere+ between 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. The+ test prints both and asserts on the durable one. It reports+ **3,249,323 B → 857,843–858,071 B** for that fixture (`main` side measured by+ reverting `index.rs` and `memtable.rs` to `main` on the same fixture), which+ is the same −73.6% at 1/17th the corpus.++ A `dense_vector` nested under an object mapping is covered too — its+ components were landing in the *parent's* term dictionary, because the segment+ builder flattens the whole object into one text field. On the nested fixture+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`+ sub-mapping under a `passages` object).++ The exclusion covers the whole family a vector field generates, not just the+ base name: `<field>_chunks` (the per-document multi-vector) and+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides+ which fields get an HNSW graph. The companion is the bigger half —+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change+ index — so a base-name-only exclusion would leave more behind than it removes.++ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now+ lowered to `match_none` at plan time rather than falling through to the+ stored-doc scan. That is the correctness half of the change, not only the+ speed half: with the postings removed and no lowering,+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**+ instead of none, because the scan renders the float array to text and every+ component contains a `0`. Measured on the corpus above, first call for each+ shape (`main` → postings-gone-without-lowering → this release):++ | query | `main` | no lowering | this release |+ |---|---|---|---|+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |++ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`+ list keeps its lexical members, and a `nested` query is deliberately not+ rewritten (its field names resolve against the element, not the root).++ **Three behaviour changes to know about:**++ - **A `dense_vector` (and its `_chunks` companion) now has no lexical surface+ at all: every lexical leaf naming one answers 0 hits.** Before this release+ such a clause fell through to the stored-doc scan, which renders the float+ array back to decimal text and matches against *that*, so whether a shape+ answered 0 or answered most of the corpus depended entirely on whether the+ rendered decimals happened to contain the probe. This is a CLASS of changed+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,+ `main` → this release:++ | query | before | after |+ |---|---:|---:|+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |+ | `prefix {emb: "0"}` | 146 | 0 |+ | `prefix {emb_chunks: "0"}` | 146 | 0 |+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |+ | `term {emb: <exact component>}` numeric | 1 | 0 |+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |++ Shapes whose probe text never appeared in a rendered float (`term` with a+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,+ `query_string`) answered 0 on both sides. ES rejects these queries on a+ `dense_vector` outright and Lucene gives a field with no terms an empty+ scorer, so 0 is the answer this release adopts for all of them. The+ regression test pins the whole class rather than sampling it.+ - **A `bool` that mentions a `dense_vector` can change `_score` and+ `max_score` without changing its hit count.** Dropping the dead clause makes+ the surviving bool projectable onto the inverted index, so BM25 scores it+ where the stored-doc scan used to. Measured on the same fixture:++ | query | hits | `_score` | `max_score` |+ |---|---|---|---|+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |++ Both moves are toward consistency — on `main` these queries reported a+ `max_score` that **no returned hit carried**, and the first now scores+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces+ to. Absolute scores are not part of XERJ's compatibility surface, but+ anything comparing scores across a version boundary or asserting on+ `max_score` for a bool that touches a vector field will see this. Plain+ lexical queries are unaffected; the blast radius is bools with a+ `dense_vector` clause in them.+ - **Existing indices keep their bloat until a merge rewrites them.** This is a+ write-side rule. Segments written by an earlier release keep their `.emb.*`+ / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads+ whichever shape it finds. Run an explicit+ `POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an+ index that already exists.++ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,+ where XERJ answers `200`. This release removes bytes and adopts Lucene's+ zero-hit answer; it does not add the rejection.+## [1.0.0-rc.16] - 2026-08-13Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 1afc4b49a..0c71c71d8 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -13672,7 +13672,20 @@ impl Index {),)));}- resolved+ // #328's query half, and it rides the same guard for the same+ // reason: a `dense_vector` has no term dictionary to consult, so a+ // lexical clause naming one is lowered to `match_none` HERE,+ // before `is_doc_scan_query` or `query_node_to_fts` ever see the+ // tree. `unsearchable_query_field` cannot cover it — a vector+ // field is `indexed`, so it is not "unsearchable" — and without+ // the lowering the missing postings hand the clause to the+ // stored-doc scan. See `lower_lexically_typeless_clauses`.+ let typeless = crate::memtable::lexically_typeless_fields(&schema.schema);+ if typeless.is_empty() {+ resolved+ } else {+ lower_lexically_typeless_clauses(&resolved, &typeless)+ }};let query = &resolved_query;@@ -14055,6 +14068,22 @@ impl Index {if !f.options.indexed && !f.options.doc_values {continue;}+ // #328 — a `dense_vector` has no lexical representation at+ // all, so it is not a field a `*` expansion may project onto.+ // A vector field is declared with vector attributes and+ // nothing else, so it never acquires index options and can+ // never own terms (`lucene/core/src/java/org/apache/lucene/+ // document/KnnFloatVectorField.java:70`, and `FieldInfo.java:565`+ // for what `IndexOptions.NONE` means). Leaving it in would push+ // a clause onto a field whose term dictionary this release+ // stops building (`memtable::fts_excluded_fields`), and the+ // per-segment `fts_has_field` gate would then hand the whole+ // query to the stored-doc scan. A clause that NAMES the field+ // rather than expanding onto it is handled separately, by+ // `lower_lexically_typeless_clauses`.+ if matches!(f.field_type, FieldType::Vector) {+ continue;+ }if matches!(f.field_type, FieldType::Text) {tf.push(f.name.clone());} else {@@ -19159,7 +19188,18 @@ fn extract_fts_fields_excluding(if excluded.contains(key) {continue;}- let text = extract_field_text(val);+ // This is the MERGE path, and a merge must not resurrect postings+ // that flush correctly skipped (#328). A root key is caught above;+ // an object-valued key is flattened whole by `extract_field_text`,+ // so a `dense_vector` nested under it (`passages.vec`) would come+ // back here even though flush dropped it. Prune it out of the+ // flattened blob exactly the way `collect_text_fields` does.+ let text = if val.is_object() && crate::memtable::has_excluded_descendant(key, excluded)+ {+ crate::memtable::extract_text_value_excluding(val, key, excluded)+ } else {+ extract_field_text(val)+ };if !text.is_empty() {fields.insert(key.clone(), text);}@@ -29374,7 +29414,12 @@ fn extract_numeric_vector(source: &Value, field: &str) -> Option<Vec<f32>> {/// Collect every dense_vector-mapped field in schema order (dotted paths/// for fields nested under object mappings).-fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {+///+/// Also the single source of truth for `memtable::lexically_typeless_fields`+/// (#328) — the set of fields that get an HNSW graph and the set that get no+/// term dictionary must be the same set, or a field ends up with both or+/// neither.+pub(crate) fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {fn walk(fields: &[FieldConfig], prefix: &str, out: &mut Vec<String>) {for fc in fields {let path = if prefix.is_empty() {@@ -30069,6 +30114,982 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {}}+/// Lower every LEXICAL clause that names a lexically typeless field to+/// `QueryNode::MatchNone` — the query half of #328.+///+/// `typeless` is `memtable::lexically_typeless_fields`: the `dense_vector`+/// fields and their `_chunks` / passage-metadata companions, i.e. exactly the+/// fields this release stops giving a term dictionary. Dropping the postings+/// alone is not enough, and this is the part the write-side change cannot do by+/// itself. Every OTHER member of `fts_excluded_fields` is protected from the+/// stored-doc scan by `unsearchable_query_field`, which 400s the query before+/// execution; a `dense_vector` is not, because it IS indexed — kNN needs the+/// HNSW graph. So without this pass a lexical clause on a vector field falls+/// through the per-segment `fts_has_field` gate onto the stored-doc scan, whose+/// `Term` arm matches ANY ELEMENT of a JSON array and whose `match` arm splits+/// on non-alphanumerics. Measured post-`force_merge` on a 5,000-doc × 128-dim+/// corpus with the postings removed but this pass absent, against `main`+/// (first call for each shape, before the result cache sees it):+/// `{"term":{"emb":"<component>"}}` 89.2 ms → 186.8 ms → **0.061 ms** with+/// this pass, and `{"multi_match":{"fields":["emb"],"query":"0"}}`+/// **0 hits → all 5,000 hits** → 0 hits, because every rendered float component+/// contains a `0`. The `multi_match` row is a correctness regression, not a+/// slow path, and it is the reason this pass exists.+///+/// Lucene's answer to "this field has no terms" is the same one, in two places:+/// `Terms.getTerms` hands back `Terms.EMPTY` rather than null so the scorer is+/// empty instead of absent (`lucene/core/src/java/org/apache/lucene/index/+/// Terms.java:40`), and a query that cannot match is REWRITTEN to+/// `MatchNoDocsQuery` (`lucene/core/src/java/org/apache/lucene/search/+/// MatchNoDocsQuery.java:23`) — as `AbstractKnnVectorQuery` does on an empty+/// index (`lucene/core/src/test/org/apache/lucene/search/+/// BaseKnnVectorQueryTestCase.java:164`). Neither ever falls back to walking+/// the corpus: `TestTermAutomatonQuery.testFieldMissing` asserts a plain 0 hits+/// for a query on `bogusfield` (`lucene/sandbox/src/test/org/apache/lucene/+/// sandbox/search/TestTermAutomatonQuery.java:1051`). The empty FIELD LIST case+/// below is `BooleanQuery.rewrite`'s first line, verbatim in spirit —+/// `if (clauses.size() == 0) return new MatchNoDocsQuery("empty BooleanQuery")`+/// (`lucene/core/src/java/org/apache/lucene/search/BooleanQuery.java:271`), and+/// the per-occur rules in the `Bool` arm are the same method's, clause for+/// clause (`:323` SHOULD/MUST_NOT ignored, `:327` MUST/FILTER returned).+///+/// This is deliberately NOT a rejection: XERJ keeps answering 200 where ES 400s+/// `term`/`match`/`range` on a `dense_vector`. Adding the rejection is a+/// separate, breaking change.+///+/// It is not answer-preserving, and the exception is a CLASS rather than a row.+/// A `dense_vector` and its `_chunks` companion end up with no lexical surface+/// at all, so every lexical leaf naming one answers 0 where the stored-doc scan+/// used to match the rendered decimal text of the float array. On the 300-doc+/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`+/// 300 hits (EVERY document) → 0, `prefix {emb:"0"}` 146 → 0,+/// `match_phrase_prefix {emb:"0"}` 50 → 0, numeric `term`/`terms` 1 → 0, and the+/// same for the `_chunks` companion. Note where the boundary is: `prefix`,+/// `wildcard` and `fuzzy` lose their hits to the WRITE-side exclusion (no term+/// dictionary left to enumerate), while `multi_match`/`simple_query_string` need+/// THIS pass — with the postings gone and no lowering they answer every+/// document. The full three-state table is on `memtable::fts_excluded_fields`.+///+/// The Bool fold below additionally moves `_score` for bools whose hit count+/// does not change, because dropping a dead `should` makes the bool projectable+/// onto the inverted index and BM25 replaces the stored-doc scorer. Measured and+/// documented there and in the CHANGELOG; pinned in the (d2c) block of+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary`.+///+/// Three classes of clause are deliberately left alone, because none of them+/// ever read the term dictionary and all of them are measured unchanged:+///+/// * `Exists` — answered from `_source` / doc values (5,000/5,000 docs on+/// `main` and here), and ES agrees it is not an error on a field with no+/// terms;+/// * `Knn` / `SemanticSearch` on the field itself — the whole reason the field+/// exists. Their `filter` sub-query IS walked;+/// * the geo leaves — a `geo_distance` on a `dense_vector` is nonsense either+/// way, but it resolves from `_source` and this change does not touch it.+///+/// `Nested` is not walked either, and that is a scope decision rather than an+/// oversight: field names inside a `nested` query resolve against the ELEMENT,+/// not the root (see `strip_nested_path_in_query`), so a root-level `emb` in+/// the set is not the same `emb` the inner clause names. Rewriting through it+/// could turn a legitimate inner clause into a silent zero. The inner clause+/// keeps exactly today's behaviour.+fn lower_lexically_typeless_clauses(+ q: &QueryNode,+ typeless: &std::collections::HashSet<String>,+) -> QueryNode {+ // One entry of a `fields: [...]` list. `"emb^3"` carries a boost; a+ // pattern (`"*"`, `"body.*"`) is an expansion, and the expansion sets in+ // `search_inner` already drop vector fields, so it must stay.+ let is_typeless_spec = |spec: &String| -> bool {+ let (name, _boost) = parse_field_boost(spec);+ !name.contains('*') && typeless.contains(name)+ };+ // A `fields: [...]` list with every vector entry removed. `None` means+ // "leave this node alone": either nothing was dropped, or the list was+ // already empty, which means "all fields" rather than "no fields".+ let prune_specs = |fields: &Vec<String>| -> Option<Vec<String>> {+ if fields.is_empty() || !fields.iter().any(is_typeless_spec) {+ return None;+ }+ Some(+ fields+ .iter()+ .filter(|spec| !is_typeless_spec(spec))+ .cloned()+ .collect(),+ )+ };+ let recurse = |c: &QueryNode| lower_lexically_typeless_clauses(c, typeless);+ match q {+ // ── Leaves that name exactly one field ────────────────────────────+ QueryNode::Term { field, .. }+ | QueryNode::Terms { field, .. }+ | QueryNode::Range { field, .. }+ | QueryNode::Prefix { field, .. }+ | QueryNode::Wildcard { field, .. }+ | QueryNode::Match { field, .. }+ | QueryNode::MatchPhrase { field, .. }+ | QueryNode::MatchPhrasePrefix { field, .. }+ | QueryNode::Fuzzy { field, .. }+ | QueryNode::Regexp { field, .. }+ | QueryNode::Intervals { field, .. }+ | QueryNode::SpanTerm { field, .. } => {+ if typeless.contains(field) {+ QueryNode::MatchNone+ } else {+ q.clone()+ }+ }++ // ── Multi-field forms: prune the list, empty list → MatchNone ─────+ // The empty case is the specific hole. `query_node_to_fts` returns+ // `None` for a field spec it cannot project, and `MultiMatch` is+ // unconditionally in `is_doc_scan_query`, so a list that named ONLY+ // vector fields would otherwise reach the stored-doc scan and match+ // every document instead of none.+ QueryNode::MultiMatch {+ fields,+ query,+ match_type,+ operator,+ analyzer,+ boost,+ slop,+ max_expansions,+ } => match prune_specs(fields) {+ None => q.clone(),+ Some(kept) if kept.is_empty() => QueryNode::MatchNone,+ Some(kept) => QueryNode::MultiMatch {+ fields: kept,+ query: query.clone(),+ match_type: *match_type,+ operator: *operator,+ analyzer: analyzer.clone(),+ boost: *boost,+ slop: *slop,+ max_expansions: *max_expansions,+ },+ },+ QueryNode::SimpleQueryString { query, fields } => match prune_specs(fields) {+ None => q.clone(),+ Some(kept) if kept.is_empty() => QueryNode::MatchNone,+ Some(kept) => QueryNode::SimpleQueryString {+ query: query.clone(),+ fields: kept,+ },+ },+ QueryNode::QueryString {+ default_field: Some(f),+ ..+ } if is_typeless_spec(f) => QueryNode::MatchNone,++ // ── Composites: recurse, then fold the MatchNone we just created ──+ QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match,+ } => {+ let was_empty =+ must.is_empty() && should.is_empty() && must_not.is_empty() && filter.is_empty();+ let must: Vec<QueryNode> = must.iter().map(recurse).collect();+ let mut should: Vec<QueryNode> = should.iter().map(recurse).collect();+ let mut must_not: Vec<QueryNode> = must_not.iter().map(recurse).collect();+ let filter: Vec<QueryNode> = filter.iter().map(recurse).collect();+ // The fold is DELIBERATELY minimal — only the shapes this pass can+ // itself produce, and only where the answer is unambiguous. It is+ // not `xerj_query::rewriter::rewrite`, which is not on this path+ // and whose "empty bool → MatchAll" rule fires AFTER it has+ // stripped `MatchNone` out of `should`, i.e. turns an unsatisfiable+ // bool into a match-everything one.+ //+ // Without a fold the lowering is CORRECT but not fast: a+ // `simple_query_string {"fields":["emb"]}` lowers to+ // `Bool{should:[Match{emb}]}` (`parser::make_simple_query_node`),+ // which becomes `Bool{should:[MatchNone]}` — still a `Bool`, still+ // in `is_doc_scan_query`, so still a full stored-doc scan that+ // evaluates to nothing.+ //+ // The per-occur rule below is `BooleanQuery.rewrite`'s, clause for+ // clause: a `MatchNoDocsQuery` in SHOULD or MUST_NOT "can be safely+ // ignored" and one in MUST or FILTER is returned as the whole query+ // (`lucene/core/src/java/org/apache/lucene/search/BooleanQuery.java:323`+ // and `:327`). Lucene's neighbouring "pure negative BooleanQuery →+ // MatchNoDocsQuery" rule (`:276`) is deliberately NOT adopted: that+ // is Lucene's bool, not ES's. XERJ follows ES, where a bool with+ // only `must_not` matches every non-excluded document — exactly what+ // `doc_matches_query`'s Bool arm implements.++ // ── MUST / FILTER: an unsatisfiable REQUIRED clause sinks the bool.+ if must.iter().any(QueryNode::is_match_none)+ || filter.iter().any(QueryNode::is_match_none)+ {+ return QueryNode::MatchNone;+ }++ // ── MUST_NOT: a clause that matches nothing EXCLUDES nothing, so+ // dropping it is exact for every shape — `doc_matches_query` only+ // ever asks `must_not.iter().any(...)`, and `MatchNone` is never+ // `any`. Dropping it is also what closes the doc-scan fallback this+ // pass would otherwise open: `query_node_to_fts`'s Bool arm `?`s on+ // EVERY `must_not` child and its `MatchNone` arm is `None`, so one+ // surviving `MatchNone` un-projects the whole bool and hands a+ // query that used to run on the inverted index to the stored-doc+ // scan. `{"bool":{"must_not":[{"term":{"emb":…}}]}}` is the shape.+ must_not.retain(|c| !c.is_match_none());++ // ── SHOULD. Both folds below turn on whether `should` is REQUIRED,+ // and the rule has to be the one `doc_matches_query` actually+ // implements, not the one ES documents in prose:+ //+ // * `None` → required iff `must` and `filter` are both empty.+ // `must_not` does NOT relax it.+ // * `Fixed(0)` → optional.+ // * `Fixed(n)` → required.+ // * `Percentage(p)` → required for EVERY p including 0, because+ // the count is `floor(len × p/100).max(1)`. The+ // previous revision of this pass treated+ // `Percentage(0)` as "optional"; that is only ever+ // a missed fold, never a wrong answer, but it is+ // still not what the evaluator does.+ // * `Field` / `Script` (terms_set) → the required count is read+ // PER DOCUMENT and 0 is a legal value, so neither+ // fold is sound. Leave the node exactly as recursed.+ let should_required = match minimum_should_match {+ None => must.is_empty() && filter.is_empty(),+ Some(MinShouldMatch::Fixed(0)) => false,+ Some(MinShouldMatch::Fixed(_)) | Some(MinShouldMatch::Percentage(_)) => true,+ Some(MinShouldMatch::Field(_)) | Some(MinShouldMatch::Script { .. }) => {+ return QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match: minimum_should_match.clone(),+ };+ }+ };+ // Every optional clause unsatisfiable, and at least one required →+ // the bool matches nothing, whatever `must_not` says. `must_not`+ // is NOT part of this test: it can only remove further documents,+ // never add one back. (The previous revision required+ // `must_not.is_empty()` here and then fell through to the retain+ // below, which turned `{"bool":{"should":[<vector clause>],+ // "must_not":[X]}}` from "0 hits" into "every document except X".)+ if should_required && !should.is_empty() && should.iter().all(QueryNode::is_match_none)+ {+ return QueryNode::MatchNone;+ }+ // Otherwise drop the dead optional clauses — but only when the+ // required count does not move with the clause COUNT. It does not+ // for `None` (1 or 0) and for `Fixed(n)` (n), which is why Lucene+ // can drop unconditionally at `BooleanQuery.java:323`. It DOES for+ // `Percentage(p)`, where shrinking `should` shrinks the threshold+ // and can make an unsatisfiable bool satisfiable.+ //+ // THIS `retain` IS THE ONE WIRE-VISIBLE SCORE CHANGE in the pass.+ // It does not move a hit COUNT — `doc_matches_query` never counts a+ // `MatchNone` as a satisfied `should` — but the shrunken bool is+ // projectable through `query_node_to_fts_with_keyword_fields` where+ // the original was not, so BM25 scores it instead of+ // `score_query_against_doc`. On the 300-doc fixture+ // `bool{should:[term emb, term cat:"even"]}` keeps its 150 hits and+ // goes `_score` 0.008402659 → 0.6931471 (exactly what+ // `{"term":{"cat":"even"}}` scores on its own, which is what the+ // bool now reduces to) and `max_score` 1.6931472 → 0.6931471. Both+ // are in the CHANGELOG and pinned in the (d2c) block of+ // `user_mapped_dense_vector_builds_no_fts_term_dictionary`.+ let count_free_msm =+ matches!(minimum_should_match, None | Some(MinShouldMatch::Fixed(_)));+ if count_free_msm && should.iter().any(QueryNode::is_match_none) {+ should.retain(|c| !c.is_match_none());+ }+ // Dropping the last `must_not` (or the last `should`) can empty a+ // bool that arrived with clauses. An all-empty bool matches every+ // document and scores 1.0 in `doc_matches_query` /+ // `score_query_against_doc` — bit-identical to `MatchAll`, which+ // takes the match-all fast path instead of `is_doc_scan_query`'s+ // `Bool` arm. Lucene folds the same shape, to its own bool+ // semantics, at `BooleanQuery.java:271`.+ if !was_empty+ && must.is_empty()+ && should.is_empty()+ && must_not.is_empty()+ && filter.is_empty()+ {+ return QueryNode::MatchAll;+ }+ QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match: minimum_should_match.clone(),+ }+ }+ // `constant_score` and a top-level `boost` wrapper both BUBBLE UP an+ // unsatisfiable inner query rather than wrapping it — Lucene does the+ // same, in the same words, at+ // `lucene/core/src/java/org/apache/lucene/search/ConstantScoreQuery.java:56`+ // ("bubble up MatchNoDocsQuery") and `BoostQuery.java:87`. It is exact+ // — neither wrapper can make a non-matching inner query match — and it+ // matters here because `Constant` is the ONLY wrapper with an arm in+ // `query_node_to_fts_with_keyword_fields`: left as+ // `Constant{MatchNone}` it projects to `None`, and `Constant` is in+ // `is_doc_scan_query`, so `{"constant_score":{"filter":{"term":+ // {"emb":…}}}}` would trade an inverted-index lookup for a full+ // stored-doc scan.+ QueryNode::Constant { score, query } => match recurse(query) {+ QueryNode::MatchNone => QueryNode::MatchNone,+ lowered => QueryNode::Constant {+ score: *score,+ query: Box::new(lowered),+ },+ },+ QueryNode::Boosted { boost, query } => match recurse(query) {+ QueryNode::MatchNone => QueryNode::MatchNone,+ lowered => QueryNode::Boosted {+ boost: *boost,+ query: Box::new(lowered),+ },+ },+ QueryNode::Named { name, query } => QueryNode::Named {+ name: name.clone(),+ query: Box::new(recurse(query)),+ },+ QueryNode::FunctionScore {+ query,+ functions,+ score_mode,+ boost_mode,+ max_boost,+ } => QueryNode::FunctionScore {+ query: Box::new(recurse(query)),+ functions: functions.clone(),+ score_mode: *score_mode,+ boost_mode: *boost_mode,+ max_boost: *max_boost,+ },+ QueryNode::Pinned { ids, organic } => QueryNode::Pinned {+ ids: ids.clone(),+ organic: Box::new(recurse(organic)),+ },+ QueryNode::Boosting {+ positive,+ negative,+ negative_boost,+ } => QueryNode::Boosting {+ positive: Box::new(recurse(positive)),+ negative: Box::new(recurse(negative)),+ negative_boost: *negative_boost,+ },+ QueryNode::DisMax {+ queries,+ tie_breaker,+ } => QueryNode::DisMax {+ queries: queries.iter().map(recurse).collect(),+ tie_breaker: *tie_breaker,+ },+ QueryNode::Hybrid { queries, fusion } => QueryNode::Hybrid {+ queries: queries+ .iter()+ .map(|wq| xerj_query::ast::WeightedQuery {+ query: recurse(&wq.query),+ weight: wq.weight,+ })+ .collect(),+ fusion: fusion.clone(),+ },+ QueryNode::Knn {+ field,+ vector,+ k,+ num_candidates,+ filter,+ boost,+ similarity,+ } => QueryNode::Knn {+ field: field.clone(),+ vector: vector.clone(),+ k: *k,+ num_candidates: *num_candidates,+ filter: filter.as_ref().map(|f| Box::new(recurse(f))),+ boost: *boost,+ similarity: *similarity,+ },+ QueryNode::SemanticSearch {+ field,+ text,+ k,+ filter,+ boost,+ } => QueryNode::SemanticSearch {+ field: field.clone(),+ text: text.clone(),+ k: *k,+ filter: filter.as_ref().map(|f| Box::new(recurse(f))),+ boost: *boost,+ },+ QueryNode::SpanNear {+ clauses,+ slop,+ in_order,+ } => QueryNode::SpanNear {+ clauses: clauses.iter().map(recurse).collect(),+ slop: *slop,+ in_order: *in_order,+ },+ QueryNode::SpanOr { clauses } => QueryNode::SpanOr {+ clauses: clauses.iter().map(recurse).collect(),+ },+ QueryNode::SpanNot { include, exclude } => QueryNode::SpanNot {+ include: Box::new(recurse(include)),+ exclude: Box::new(recurse(exclude)),+ },+ QueryNode::SpanFirst { match_query, end } => QueryNode::SpanFirst {+ match_query: Box::new(recurse(match_query)),+ end: *end,+ },+ QueryNode::SpanContaining { little, big } => QueryNode::SpanContaining {+ little: Box::new(recurse(little)),+ big: Box::new(recurse(big)),+ },+ QueryNode::SpanWithin { little, big } => QueryNode::SpanWithin {+ little: Box::new(recurse(little)),+ big: Box::new(recurse(big)),+ },++ // Everything else names no lexical target on a vector field:+ // MatchAll/MatchNone/Ids/Script, `Exists`, the geo leaves, a+ // field-less `query_string`, `percolate` (the field holds stored+ // queries, not data), `more_like_this` (with `fields` the parser+ // lowers it to a `bool.should` of `match`, which the leaf arms above+ // catch), and `Nested` (see the doc comment).+ other => other.clone(),+ }+}++/// Unit coverage for [`lower_lexically_typeless_clauses`] — #328.+///+/// The integration tests in `tests/integration.rs` pin the ANSWERS this+/// rewriter has to produce end to end; these pin the SHAPE it produces, which+/// is what decides whether the query then runs on the inverted index or on the+/// stored-doc scan. The distinction is not cosmetic: `Bool{must_not:[MatchNone]}`+/// and `Bool{}` give the same hits, but only the second projects through+/// `query_node_to_fts_with_keyword_fields` — the first `?`s out on its+/// `must_not` child and falls to a full corpus walk.+#[cfg(test)]+mod lexically_typeless_lowering_tests {+ use super::*;++ fn typeless(names: &[&str]) -> std::collections::HashSet<String> {+ names.iter().map(|s| s.to_string()).collect()+ }++ fn term(field: &str, value: &str) -> QueryNode {+ QueryNode::Term {+ field: field.into(),+ value: Value::String(value.into()),+ boost: None,+ }+ }++ fn bool_of(+ must: Vec<QueryNode>,+ should: Vec<QueryNode>,+ must_not: Vec<QueryNode>,+ filter: Vec<QueryNode>,+ msm: Option<MinShouldMatch>,+ ) -> QueryNode {+ QueryNode::Bool {+ must,+ should,+ must_not,+ filter,+ minimum_should_match: msm,+ }+ }++ fn lower(q: &QueryNode) -> QueryNode {+ lower_lexically_typeless_clauses(q, &typeless(&["emb", "emb_chunks"]))+ }++ // ── Leaves ───────────────────────────────────────────────────────────++ #[test]+ fn single_field_leaf_on_a_vector_field_lowers_to_match_none() {+ for leaf in [+ term("emb", "0.5"),+ QueryNode::Terms {+ field: "emb".into(),+ values: vec![Value::String("0.5".into())],+ boost: None,+ },+ QueryNode::Range {+ field: "emb".into(),+ gte: Some(Value::from(-2.0)),+ gt: None,+ lte: Some(Value::from(2.0)),+ lt: None,+ boost: None,+ },+ QueryNode::Prefix {+ field: "emb".into(),+ value: "0".into(),+ boost: None,+ constant_score: true,+ },+ QueryNode::Wildcard {+ field: "emb".into(),+ value: "0*".into(),+ boost: None,+ constant_score: true,+ },+ QueryNode::Match {+ field: "emb".into(),+ query: "0".into(),+ operator: xerj_query::ast::BoolOperator::Or,+ analyzer: None,+ boost: None,+ minimum_should_match: None,+ },+ term("emb_chunks", "0.5"),+ ] {+ assert!(+ lower(&leaf).is_match_none(),+ "leaf naming a lexically typeless field must lower: {leaf:?}"+ );+ }+ }++ #[test]+ fn leaf_on_a_lexical_field_is_untouched() {+ let q = term("cat", "even");+ assert_eq!(lower(&q), q, "a normal field keeps its clause");+ }++ /// `Exists` is answered from `_source` / doc values, never from the term+ /// dictionary, so it must survive — this is the clause that keeps+ /// `{"exists":{"field":"emb"}}` answering 5,000/5,000 after the postings+ /// are gone.+ #[test]+ fn exists_and_knn_on_a_vector_field_survive() {+ let exists = QueryNode::Exists {+ field: "emb".into(),+ };+ assert_eq!(lower(&exists), exists);++ let knn = QueryNode::Knn {+ field: "emb".into(),+ vector: vec![0.0, 1.0],+ k: 3,+ num_candidates: None,+ filter: Some(Box::new(term("cat", "even"))),+ boost: None,+ similarity: None,+ };+ match lower(&knn) {+ QueryNode::Knn { field, filter, .. } => {+ assert_eq!(field, "emb", "the kNN target is the whole point");+ assert_eq!(+ *filter.expect("filter kept"),+ term("cat", "even"),+ "a lexical filter under kNN is walked but not lowered"+ );+ }+ other => panic!("kNN must not be lowered: {other:?}"),+ }+ }++ /// A kNN filter that names the vector field itself IS lowered — the filter+ /// is an ordinary lexical sub-query, unlike the `field` it hangs off.+ #[test]+ fn knn_filter_naming_the_vector_field_is_lowered() {+ let knn = QueryNode::Knn {+ field: "emb".into(),+ vector: vec![0.0, 1.0],+ k: 3,+ num_candidates: None,+ filter: Some(Box::new(term("emb", "0.5"))),+ boost: None,+ similarity: None,+ };+ match lower(&knn) {+ QueryNode::Knn { filter, .. } => {+ assert!(filter.expect("filter kept").is_match_none());+ }+ other => panic!("expected knn, got {other:?}"),+ }+ }++ // ── Multi-field forms ────────────────────────────────────────────────++ #[test]+ fn multi_match_keeps_its_lexical_fields_and_drops_the_vector_ones() {+ let q = QueryNode::MultiMatch {+ fields: vec!["emb".into(), "body^3".into(), "emb_chunks".into()],+ query: "0".into(),+ match_type: xerj_query::ast::MultiMatchType::BestFields,+ operator: Some(xerj_query::ast::BoolOperator::Or),+ analyzer: None,+ boost: None,+ slop: 0,+ max_expansions: 50,+ };+ match lower(&q) {+ QueryNode::MultiMatch { fields, .. } => {+ assert_eq!(fields, vec!["body^3".to_string()], "boost suffix preserved");+ }+ other => panic!("expected a pruned multi_match, got {other:?}"),+ }+ }++ /// The regression the PR body originally mis-reported as "byte-identical":+ /// a `fields` list naming ONLY vector fields must become `MatchNone`, not+ /// an empty-`fields` multi_match (which means "all fields" downstream) and+ /// not a stored-doc scan (which answers every document, because the scan+ /// renders the float array to text and every component contains a `0`).+ #[test]+ fn multi_match_over_only_vector_fields_becomes_match_none() {+ let q = QueryNode::MultiMatch {+ fields: vec!["emb".into(), "emb_chunks^2".into()],+ query: "0".into(),+ match_type: xerj_query::ast::MultiMatchType::BestFields,+ operator: Some(xerj_query::ast::BoolOperator::Or),+ analyzer: None,+ boost: None,+ slop: 0,+ max_expansions: 50,+ };+ assert!(lower(&q).is_match_none());+ }++ /// An EMPTY `fields` list means "every field", not "no field" — pruning+ /// cannot empty it, so the node must be left exactly as it arrived.+ #[test]+ fn multi_match_with_no_fields_is_left_alone() {+ let q = QueryNode::MultiMatch {+ fields: vec![],+ query: "0".into(),+ match_type: xerj_query::ast::MultiMatchType::BestFields,+ operator: Some(xerj_query::ast::BoolOperator::Or),+ analyzer: None,+ boost: None,+ slop: 0,+ max_expansions: 50,+ };+ assert_eq!(lower(&q), q);+ }++ /// A pattern entry is an EXPANSION, and the expansion sets in+ /// `search_inner` already drop vector fields, so it must not be pruned+ /// here — pruning `"*"` would silently turn "search everything" into+ /// "search nothing".+ #[test]+ fn a_wildcard_field_spec_is_not_pruned() {+ let q = QueryNode::SimpleQueryString {+ query: "0".into(),+ fields: vec!["*".into(), "emb".into()],+ };+ match lower(&q) {+ QueryNode::SimpleQueryString { fields, .. } => {+ assert_eq!(fields, vec!["*".to_string()]);+ }+ other => panic!("expected simple_query_string, got {other:?}"),+ }+ }++ #[test]+ fn query_string_with_a_vector_default_field_becomes_match_none() {+ let q = QueryNode::QueryString {+ query: "0".into(),+ default_field: Some("emb".into()),+ default_operator: None,+ boost: None,+ };+ assert!(lower(&q).is_match_none());+ }++ // ── Bool folding ─────────────────────────────────────────────────────++ #[test]+ fn match_none_in_must_or_filter_sinks_the_whole_bool() {+ assert!(lower(&bool_of(+ vec![term("emb", "0.5"), term("cat", "even")],+ vec![],+ vec![],+ vec![],+ None+ ))+ .is_match_none());+ assert!(lower(&bool_of(+ vec![],+ vec![],+ vec![],+ vec![term("emb", "0.5")],+ None+ ))+ .is_match_none());+ }++ /// `must_not` is the arm the first cut of this pass left open. A clause+ /// that matches nothing excludes nothing, so it must be DROPPED — leaving+ /// a `MatchNone` there keeps the bool un-projectable and sends a query+ /// that used to run on the inverted index to the stored-doc scan.+ #[test]+ fn match_none_in_must_not_is_dropped_not_kept() {+ match lower(&bool_of(+ vec![term("cat", "even")],+ vec![],+ vec![term("emb", "0.5")],+ vec![],+ None,+ )) {+ QueryNode::Bool { must, must_not, .. } => {+ assert_eq!(must, vec![term("cat", "even")]);+ assert!(must_not.is_empty(), "a dead exclusion is not an exclusion");+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ }++ /// The bool empties out completely: `{"bool":{"must_not":[{"term":+ /// {"emb":…}}]}}` excludes nothing and therefore matches everything.+ /// `MatchAll` and an all-empty `Bool` agree on hits AND on `_score` (1.0),+ /// but only `MatchAll` skips `is_doc_scan_query`.+ #[test]+ fn a_bool_emptied_by_the_fold_becomes_match_all() {+ assert_eq!(+ lower(&bool_of(+ vec![],+ vec![],+ vec![term("emb", "0.5")],+ vec![],+ None+ )),+ QueryNode::MatchAll+ );+ }++ /// …but a bool that arrived empty is NOT rewritten: this pass only ever+ /// reacts to something it itself lowered.+ #[test]+ fn a_bool_that_arrived_empty_is_untouched() {+ let q = bool_of(vec![], vec![], vec![], vec![], None);+ assert_eq!(lower(&q), q);+ }++ /// THE CORRECTNESS CASE. `should` is required whenever `must` and `filter`+ /// are empty — `must_not` does not relax it (`doc_matches_query`'s Bool arm+ /// tests `must.is_empty() && filter.is_empty()`). The first cut of this+ /// pass required `must_not.is_empty()` before folding and then fell through+ /// to dropping the dead `should` clause, which turned this query from+ /// "0 hits" into "every document except the excluded ones".+ #[test]+ fn all_should_unsatisfiable_sinks_the_bool_even_with_a_must_not() {+ assert!(lower(&bool_of(+ vec![],+ vec![term("emb", "0.5")],+ vec![term("cat", "even")],+ vec![],+ None,+ ))+ .is_match_none());+ }++ #[test]+ fn all_should_unsatisfiable_sinks_a_should_only_bool() {+ assert!(lower(&bool_of(+ vec![],+ vec![term("emb", "0.5")],+ vec![],+ vec![],+ None+ ))+ .is_match_none());+ }++ /// With a `must` present, `should` is scoring-only (`min = 0`), so a dead+ /// optional clause is dropped and the bool survives.+ #[test]+ fn a_dead_should_beside_a_must_is_dropped_and_the_bool_survives() {+ match lower(&bool_of(+ vec![term("cat", "even")],+ vec![term("emb", "0.5"), term("body", "liquidity")],+ vec![],+ vec![],+ None,+ )) {+ QueryNode::Bool { must, should, .. } => {+ assert_eq!(must, vec![term("cat", "even")]);+ assert_eq!(should, vec![term("body", "liquidity")]);+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ }++ /// An explicit numeric `minimum_should_match` does not move with the+ /// clause count, so the dead clause is dropped — and the survivor count+ /// can now be below the threshold, which is exactly the original meaning.+ #[test]+ fn fixed_minimum_should_match_still_drops_the_dead_clause() {+ match lower(&bool_of(+ vec![],+ vec![term("emb", "0.5"), term("body", "liquidity")],+ vec![],+ vec![],+ Some(MinShouldMatch::Fixed(2)),+ )) {+ QueryNode::Bool {+ should,+ minimum_should_match,+ ..+ } => {+ assert_eq!(should, vec![term("body", "liquidity")]);+ assert_eq!(minimum_should_match, Some(MinShouldMatch::Fixed(2)));+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ }++ /// `Fixed(0)` makes `should` optional, so an all-dead `should` list does+ /// NOT sink the bool.+ #[test]+ fn fixed_zero_minimum_should_match_keeps_the_bool_alive() {+ match lower(&bool_of(+ vec![],+ vec![term("emb", "0.5")],+ vec![term("cat", "even")],+ vec![],+ Some(MinShouldMatch::Fixed(0)),+ )) {+ QueryNode::Bool {+ should, must_not, ..+ } => {+ assert!(should.is_empty(), "dead optional clause dropped");+ assert_eq!(must_not, vec![term("cat", "even")]);+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ }++ /// A PERCENTAGE threshold moves with `should.len()`+ /// (`floor(len × pct/100).max(1)`), so the list must not be shrunk. It is+ /// still required at every percentage — including 0, because of the+ /// `.max(1)` — so an all-dead list sinks the bool.+ #[test]+ fn percentage_minimum_should_match_does_not_shrink_the_clause_list() {+ match lower(&bool_of(+ vec![],+ vec![term("emb", "0.5"), term("body", "a"), term("body", "b")],+ vec![],+ vec![],+ Some(MinShouldMatch::Percentage(67)),+ )) {+ QueryNode::Bool { should, .. } => {+ assert_eq!(should.len(), 3, "threshold rides on the clause COUNT");+ assert!(should[0].is_match_none());+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ assert!(lower(&bool_of(+ vec![],+ vec![term("emb", "0.5")],+ vec![],+ vec![],+ Some(MinShouldMatch::Percentage(0)),+ ))+ .is_match_none());+ }++ /// `terms_set`: the required count is read PER DOCUMENT and 0 is legal, so+ /// neither fold is sound and the node is left exactly as recursed.+ #[test]+ fn per_document_minimum_should_match_blocks_both_folds() {+ match lower(&bool_of(+ vec![],+ vec![term("emb", "0.5")],+ vec![],+ vec![],+ Some(MinShouldMatch::Field("required".into())),+ )) {+ QueryNode::Bool {+ should,+ minimum_should_match,+ ..+ } => {+ assert_eq!(should.len(), 1, "clause count preserved");+ assert!(should[0].is_match_none(), "leaf still lowered");+ assert_eq!(+ minimum_should_match,+ Some(MinShouldMatch::Field("required".into()))+ );+ }+ other => panic!("expected a bool, got {other:?}"),+ }+ }++ // ── Wrappers ─────────────────────────────────────────────────────────++ /// `Constant` is the only wrapper with an arm in+ /// `query_node_to_fts_with_keyword_fields`, so it is the only one where+ /// failing to bubble up costs an inverted-index lookup.+ #[test]+ fn constant_score_and_boost_bubble_up_match_none() {+ assert!(lower(&QueryNode::Constant {+ score: 2.0,+ query: Box::new(term("emb", "0.5")),+ })+ .is_match_none());+ assert!(lower(&QueryNode::Boosted {+ boost: 2.0,+ query: Box::new(term("emb", "0.5")),+ })+ .is_match_none());+ }++ #[test]+ fn constant_score_over_a_lexical_clause_is_untouched() {+ let q = QueryNode::Constant {+ score: 2.0,+ query: Box::new(term("cat", "even")),+ };+ assert_eq!(lower(&q), q);+ }++ /// Field names inside a `nested` query resolve against the ELEMENT, not+ /// the root, so a root-level `emb` in the set is not the inner `emb`.+ /// Deliberately out of scope — pinned so the decision is a decision.+ #[test]+ fn nested_is_not_walked() {+ let q = QueryNode::Nested {+ path: "chunks".into(),+ query: Box::new(term("emb", "0.5")),+ score_mode: None,+ };+ assert_eq!(lower(&q), q);+ }++ /// The whole pass is a no-op on a schema with no vector fields — the+ /// caller skips it entirely on an empty set, and this pins that the+ /// function agrees.+ #[test]+ fn an_empty_typeless_set_changes_nothing() {+ let q = bool_of(+ vec![term("emb", "0.5")],+ vec![term("body", "x")],+ vec![term("cat", "even")],+ vec![],+ None,+ );+ assert_eq!(+ lower_lexically_typeless_clauses(&q, &std::collections::HashSet::new()),+ q+ );+ }+}+/// Rewrite a query by resolving any field aliases to their canonical field names.////// Traverses the query tree, replacing alias field names with their target paths.diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rsindex c1b06d301..c44936c8f 100644--- a/engine/crates/xerj-engine/src/memtable.rs+++ b/engine/crates/xerj-engine/src/memtable.rs@@ -3679,20 +3679,38 @@ fn collect_text_fields() {if let Value::Object(obj) = v {for (k, val) in obj {- if prefix.is_empty() && excluded.contains(k) {- continue;- }let path = if prefix.is_empty() {k.clone()} else {format!("{}.{}", prefix, k)};+ // Matched on the FULL dotted path, not just the root key — #328.+ // Every pre-#328 member of the set is a root-level name, for which+ // `path == k` and this is the check that was here before; a+ // `dense_vector` nested under an object mapping is the first member+ // that can arrive as `passages.vec`.+ if excluded.contains(&path) {+ continue;+ }match val {Value::Object(_) => {// Root-level JSON-blob for flattened-style// whole-object queries.+ //+ // Built from a PRUNED copy of the object (#328): the blob+ // is where a nested `dense_vector` actually landed. The+ // recursion below skips `passages.vec`, but this line+ // serialised the whole `passages` object — vector included+ // — into one text field, so excluding the leaf alone moved+ // no bytes at all. Pinned by+ // `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`,+ // which reports `<seg>.passages.fst` and requires it under+ // 4 KiB — the fixture's 19,200 vector components cannot fit+ // in that. Both mapping shapes are covered there: the+ // dotted top-level name `"passages.vec"` and a `vec`+ // sub-mapping under a `passages` object.if prefix.is_empty() {- let t = extract_text_value(val);+ let t = extract_text_value_excluding(val, &path, excluded);if !t.is_empty() && !out.contains_key(&path) {out.insert(path.clone(), t);}@@ -3806,11 +3824,13 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash/// Every field that must be kept OUT of the full-text inverted index.///-/// Two disjoint reasons, unioned into one set so flush, merge, the memtable+/// Three disjoint reasons, unioned into one set so flush, merge, the memtable/// analyser and `build_fts_field_configs` cannot disagree:////// * semantic-derived vector fields (see above) — engine-internal payloads/// that were never user text;+/// * fields the user MAPPED as `dense_vector` (`FieldType::Vector`) — #328.+/// See "Why a declared vector has no lexical form" below;/// * fields whose mapping declares `"index": false` — #204's last open/// accepted-and-ignored instance. The option was echoed by `GET _mapping`/// and then ignored, so the field kept a full inverted index and stayed@@ -3844,8 +3864,166 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash/// `unsearchable_query_field` in `index.rs` rejects every query naming it/// before execution — so no fallback of any strength ever runs, and the/// postings are pure waste whatever the type.+///+/// ## Why a declared vector has no lexical form (#328)+///+/// `#12` removed the term dictionary for the vectors XERJ *generates* from a+/// semantic mapping. A field the user maps as `dense_vector` took a different+/// route through the same code and kept its own: kNN needs the HNSW graph, so+/// the field is `"index": true`, so the `index: false` arm above never fires.+/// The field then flowed through `extract_fts_fields_excluding` into+/// `extract_field_text`, whose `Value::Array` arm joins the elements with+/// spaces — turning a 128-dim vector into one enormous decimal-string term per+/// document. kNN is served by `hnsw/graph.bin`; `exists` is resolved from+/// `_source` / doc values; `_field_caps` and highlighting never open the+/// field's FST — so every one of those bytes is unreadable by construction.+///+/// Lucene draws the type line in exactly the same place and gets the same+/// consequence for free: `KnnFloatVectorField.createFieldType` sets vector+/// attributes and then `freeze()`s, and sets nothing else+/// (`lucene/core/src/java/org/apache/lucene/document/KnnFloatVectorField.java:70`;+/// `KnnByteVectorField.java:82` is identical), so `IndexOptions` stays `NONE`+/// (`lucene/core/src/java/org/apache/lucene/index/FieldInfo.java:565`) and a+/// term dictionary for the field cannot exist. `ColumnValidation` enumerates+/// "vectors" as an indexing feature DISJOINT from "index options"+/// (`lucene/core/src/java/org/apache/lucene/document/column/ColumnValidation.java:104`)+/// — the same type-level separation this is.+///+/// Measured — one harness, one corpus, built twice and diffed file by file+/// after `force_merge(1)`. 5,000 docs × 128 dim: `text` + `keyword` + `long` ++/// a `dense_vector` + its `_chunks` companion (the corpus carries the companion+/// because that is the shape RFC #148 reports). Every byte below is a+/// `std::fs::metadata().len()` sum over the data dir, not an estimate:+///+/// | file, 5,000 × 128 | `main` | this branch |+/// |---|---:|---:|+/// | `<seg>.emb.fst` | 13,256,659 B | 0 |+/// | `<seg>.emb.post` | 10,834 B | 0 |+/// | `<seg>.emb.norms` | 48 B | 0 |+/// | `<seg>.emb_chunks.fst` | 26,518,005 B | 0 |+/// | `<seg>.emb_chunks.post` | 10,834 B | 0 |+/// | `<seg>.body.fst` | 278 B | 278 B |+/// | `<seg>.cat.fst` | 53 B | 53 B |+/// | `<seg>.n.fst` | 209 B | 209 B |+/// | **durable index** (all files bar `.wal`) | **54,068,549 B** | **14,251,975 B** (−73.6%, 3.79×) |+///+/// Two observations the table is here to make. First, the term dictionary of a+/// vector-carrying index essentially *is* the vector field: the three lexical+/// `.fst` files total 540 B against 39.8 MB of vector postings. Second — and+/// this is why the exclusion cannot stop at the base name — the `_chunks`+/// companion is the BIGGER half. At 26,518,005 B it is twice `emb.fst` and 49%+/// of `main`'s whole index on its own, so a base-name-only exclusion would+/// leave more behind than it removes.+///+/// The row is the DURABLE index — every file in the data directory except+/// `.wal` — because the write-ahead log's tail is reclaimed asynchronously and+/// the whole-directory total is therefore not reproducible: eight runs of the+/// 300-doc regression test put it anywhere between 858,147 B and 1,105,001 B+/// while the durable bytes moved by 230 B. `.seg` (the stored documents, 11.2 MB+/// here) is counted on both sides and is why the ratio is 3.79× rather than the+/// 39.8 MB the term dictionaries cost outright.+///+/// The 300-doc × 128-dim fixture the regression tests actually run is the same+/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,+/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,+/// durable index 3,249,323 B → 857,843–858,071 B over eight runs. That test+/// prints both totals and asserts a ceiling on the durable one, so this figure+/// is enforced rather than merely published.+///+/// The exclusion is *by declared type*, not by value shape: an unmapped+/// numeric array stays lexical exactly as before (the same line HNSW draws in+/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a+/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT excluded:+/// its declared payload is chunk *text* plus a vector, and only the vector+/// half is droppable — that needs a value-shape-aware rule, not a type rule.+///+/// Removing the postings is only half of it, and the other half is NOT the+/// `*`-expansion filter. Every other member of this set is protected from the+/// stored-doc scan by `unsearchable_query_field`, which rejects the query+/// before execution; a `dense_vector` is not, because it IS indexed. So two+/// things happen on the query side:+///+/// * `text_fields` / `exact_fields` in `search_inner` drop vector fields, so a+/// `*` expansion stops projecting a clause onto them;+/// * and `index::lower_lexically_typeless_clauses` lowers any clause that+/// NAMES one to `match_none` at plan time. The expansion filter alone does+/// nothing for `{"term":{"emb":…}}` or `{"multi_match":{"fields":["emb"]}}`,+/// which name the field outright — those fall through the per-segment+/// `fts_has_field` gate onto the stored-doc scan, whose `Term` arm matches+/// ANY ELEMENT of a JSON array (`json_values_equal`) and whose `match` arm+/// splits on non-alphanumerics.+///+/// Measured on the corpus above, post-`force_merge`, three source states —+/// `main` / write-side exclusion only (postings gone, no plan-time lowering) /+/// this branch. Timings are the FIRST call for each shape, before the result+/// cache has seen it; a repeat of any row here is ~0.002 ms on all three and+/// measures the cache, not the plan:+///+/// | query | `main` | postings gone, no lowering | this branch |+/// |---|---|---|---|+/// | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |+/// | `term {emb: <component>}` numeric | **1 hit**, 87.6 ms | **1 hit**, 187.6 ms | **0 hits**, 0.011 ms |+/// | `terms {emb: [<component>]}` numeric | **1 hit**, 85.6 ms | **1 hit**, 112.0 ms | **0 hits**, 0.012 ms |+/// | `prefix {emb: "0"}` | **2,500 hits**, 231.8 ms | 0 hits, 95.0 ms | **0 hits**, 0.008 ms |+/// | `wildcard {emb: "0*"}` | **5,000 hits**, 316.6 ms | 0 hits, 95.1 ms | **0 hits**, 0.008 ms |+/// | `fuzzy {emb: "0"~2}` | **5,000 hits**, 798.5 ms | 0 hits, 97.7 ms | **0 hits**, 0.011 ms |+/// | `match_phrase_prefix {emb: "0"}` | **50 hits**, 84.6 ms | 0 hits, 89.3 ms | **0 hits**, 0.009 ms |+/// | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |+/// | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |+/// | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |+/// | `term {emb_chunks: "<component>"}` | 0 hits, 86.0 ms | 0 hits, 116.6 ms | 0 hits, 0.008 ms |+/// | `wildcard {emb_chunks: "0*"}` | **5,000 hits**, 626.9 ms | 0 hits, 89.9 ms | **0 hits**, 0.007 ms |+/// | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |+/// | `bool{should:[term emb], must_not:[term cat]}` | 0 hits, 81.4 ms | 0 hits, 92.4 ms | 0 hits, 0.011 ms |+/// | `bool{must_not:[term emb]}` | 5,000 hits, 84.9 ms | 5,000 hits, 98.9 ms | 5,000 hits, 2.31 ms |+/// | `constant_score{filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |+/// | `exists {field: emb}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |+/// | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |+/// | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |+///+/// The two `multi_match` rows are the ones that are not merely slow. Without the+/// lowering they answer EVERY document, because the scan renders the float array+/// to text and every component contains a `0` — a 0-hit query silently becomes+/// a match-all. Those rows are the reason the write-side change cannot ship on+/// its own.+///+/// A WHOLE CLASS OF ANSWERS CHANGES, and it is stated rather than buried. SEVEN+/// rows above differ between `main` and this branch (nine shapes on the 300-doc+/// regression fixture, which probes both companions), and they are all the same+/// thing: a `dense_vector` and its `_chunks` companion no longer have any+/// lexical surface, so every lexical leaf naming one answers 0. On `main` those+/// clauses reached the stored-doc scan, which renders the float array back to+/// decimal text and matches against THAT — its `Term` arm matches any ELEMENT of+/// a JSON array (`json_values_equal`) and its `match` arm splits on+/// non-alphanumerics — so a shape answered 0 or answered most of the corpus+/// purely according to whether the rendered decimals happened to contain the+/// probe. `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}` matched EVERY document.+/// None of that is a retrieval semantic anyone asked for: ES rejects these+/// queries on a `dense_vector` outright (`DenseVectorFieldType` has no term+/// query) and Lucene gives a field with no terms an empty scorer+/// (`lucene/core/src/java/org/apache/lucene/index/Terms.java:40`). 0 is the+/// Lucene answer. It is still a behaviour change and the CHANGELOG describes the+/// class; `user_mapped_dense_vector_builds_no_fts_term_dictionary`'s (d) loop+/// pins it shape by shape.+///+/// `_score` MOVES TOO, for bools whose hit count does not. Dropping the dead+/// clause out of `should` makes the surviving bool projectable through+/// `query_node_to_fts_with_keyword_fields`, so BM25 scores it where the+/// stored-doc scan used to. On the 300-doc fixture,+/// `bool{should:[term emb, term cat:"even"]}` keeps its 150 hits but goes+/// `_score` 0.008402659 → 0.6931471 and `max_score` 1.6931472 → 0.6931471, and+/// `bool{must:[match body], must_not:[term emb]}` keeps its 300 hits and its+/// `_score` but goes `max_score` 1.6931472 → 0.008402659. Both moves are toward+/// consistency: on `main` these reported a `max_score` no returned hit carried.+/// Isolated to the plan-time fold — with the write-side exclusion alone and the+/// lowering disabled, both are byte-identical to `main`. Pinned in that test's+/// (d2c) block.+///+/// Not claimed and not changed: XERJ still answers 200 rather than 400 for+/// these shapes. Adding ES's rejection is a separate, breaking change.pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String> {let mut excluded = semantic_derived_vector_fields(schema);+ excluded.extend(lexically_typeless_fields(schema));for field in &schema.fields {if field.options.indexed {continue;@@ -3859,6 +4037,83 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>excluded}+/// Fields whose DECLARED type has no lexical representation at all — #328.+///+/// Today that is exactly `FieldType::Vector` (`dense_vector`), so the base list+/// is deliberately the SAME list, from the same walk, that decides which fields+/// get an HNSW graph: [`crate::index::collect_dense_vector_fields`]. Two+/// independent notions of "which fields are vectors" is how you end up with a+/// field that has both a graph and a term dictionary, or neither — so there is+/// one walk and this is a view of it.+///+/// Each vector contributes THREE names, exactly the shape+/// [`semantic_derived_vector_fields`] already emits per target, because a+/// `dense_vector` never arrives alone: `passage_scored_vector_fields` unions+/// user-mapped vectors with semantic targets and both grow a `<field>_chunks`+/// companion (the per-document MULTI-vector) and a+/// `__xerj_passage_meta__<field>` sidecar. Excluding only the base name leaves+/// the companion behind, and the companion is the BIGGER half: on the 5,000-doc+/// × 128-dim corpus measured in [`fts_excluded_fields`], `<seg>.emb_chunks.fst`+/// is 26,518,005 B against `<seg>.emb.fst`'s 13,256,659 B — twice the size, and+/// 49% of `main`'s whole 54,068,549 B durable index on its own.+///+/// Lucene draws the same line and never lets a multi-vector reach an analyzer:+/// its per-document float[][] shape is `LateInteractionField`, which `extends+/// BinaryDocValuesField`+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`)+/// and hands its constructor straight to `super(name, encode(value))` (`:44`)+/// — doc values only, no index options, so no term dictionary can exist for it+/// any more than for the single-vector `KnnFloatVectorField`+/// (`KnnFloatVectorField.java:70`: vector attributes, then `freeze()`).+///+/// NAME SYNTHESIS, stated rather than discovered: a user field literally named+/// `emb_chunks` alongside a `dense_vector` `emb` loses its term dictionary to+/// this collision. That is the same collision `semantic_derived_vector_fields`+/// has always accepted for `<target>_chunks`, and the same names+/// `observe_passage_chunks_in_source` reserves, so the ambiguity is the+/// engine's convention rather than something introduced here.+///+/// ALLOCATION: this runs once per SEARCH (`index::search_inner` calls it to+/// decide whether the plan-time lowering has anything to do), and the vast+/// majority of indices have no vector field at all. `collect_dense_vector_fields`+/// clones the name of EVERY field it walks past, not just the vector ones, so+/// the no-vector case is short-circuited by the allocation-free predicate+/// below and returns an empty `HashSet`, which does not allocate either. The+/// two must agree by construction, and+/// `the_fast_path_predicate_agrees_with_the_walk` pins that they do.+pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<String> {+ let mut excluded = std::collections::HashSet::new();+ if !has_vector_field(&schema.fields) {+ return excluded;+ }+ for field in crate::index::collect_dense_vector_fields(schema) {+ excluded.insert(format!("{field}_chunks"));+ excluded.insert(format!(+ "{}{}",+ xerj_query::executor::PASSAGE_METADATA_PREFIX,+ field+ ));+ excluded.insert(field);+ }+ excluded+}++/// Does this field list contain a `dense_vector` anywhere? Allocation-free.+///+/// The recursion condition is `collect_dense_vector_fields`'s, deliberately: it+/// stops at a `Vector` rather than descending into its sub-fields, so the two+/// walks visit exactly the same nodes and cannot disagree about whether the set+/// would come back empty.+fn has_vector_field(fields: &[xerj_common::types::FieldConfig]) -> bool {+ fields.iter().any(|fc| {+ if matches!(fc.field_type, FieldType::Vector) {+ true+ } else {+ !fc.fields.is_empty() && has_vector_field(&fc.fields)+ }+ })+}+#[cfg(test)]mod semantic_derived_vector_exclusion_tests {use super::*;@@ -4039,6 +4294,257 @@ fn extract_text_value(val: &Value) -> String {}}+/// Does `excluded` hold anything strictly BELOW `path`?+///+/// Allocation-free on the hot path: the set is schema-derived and normally+/// holds zero dotted entries, so this is a handful of length compares per+/// object-valued field per document.+pub(crate) fn has_excluded_descendant(+ path: &str,+ excluded: &std::collections::HashSet<String>,+) -> bool {+ excluded.iter().any(|entry| {+ entry.len() > path.len() && entry.as_bytes()[path.len()] == b'.' && entry.starts_with(path)+ })+}++/// [`extract_text_value`], with every descendant named in `excluded` removed+/// from the flattened object first — #328.+///+/// `path` is the dotted path of `val` itself. Only objects can have+/// descendants, so every other shape short-circuits to `extract_text_value`,+/// and an object with no excluded descendant does too — the clone is paid only+/// by the documents that actually carry an excluded sub-field.+pub(crate) fn extract_text_value_excluding(+ val: &Value,+ path: &str,+ excluded: &std::collections::HashSet<String>,+) -> String {+ fn prune(val: &Value, path: &str, excluded: &std::collections::HashSet<String>) -> Value {+ match val {+ Value::Object(map) => {+ let mut kept = serde_json::Map::with_capacity(map.len());+ for (key, child) in map {+ let child_path = if path.is_empty() {+ key.clone()+ } else {+ format!("{path}.{key}")+ };+ if excluded.contains(&child_path) {+ continue;+ }+ kept.insert(key.clone(), prune(child, &child_path, excluded));+ }+ Value::Object(kept)+ }+ other => other.clone(),+ }+ }+ if !matches!(val, Value::Object(_)) || !has_excluded_descendant(path, excluded) {+ return extract_text_value(val);+ }+ extract_text_value(&prune(val, path, excluded))+}++/// Unit coverage for the three helpers #328 adds — the widest new surface in+/// this file is [`lexically_typeless_fields`], which is `pub`.+#[cfg(test)]+mod lexically_typeless_fields_tests {+ use super::*;+ use serde_json::json;+ use xerj_common::types::{FieldConfig, FieldType, Schema};++ fn vector(name: &str, dims: usize) -> FieldConfig {+ let mut f = FieldConfig::new(name, FieldType::Vector);+ f.options.dimensions = Some(dims);+ f.options.similarity = Some("cosine".to_string());+ f+ }++ /// Every `dense_vector` contributes THREE names, because a vector never+ /// arrives alone: the base field, the `_chunks` multi-vector companion and+ /// the passage-metadata sidecar. Excluding only the base name is what left+ /// `<seg>.emb_chunks.fst` — the bigger of the two artifacts — behind.+ #[test]+ fn every_vector_contributes_its_whole_family() {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ schema.add_field(vector("emb", 8)).unwrap();++ let typeless = lexically_typeless_fields(&schema);+ assert_eq!(+ typeless,+ std::collections::HashSet::from([+ "emb".to_string(),+ "emb_chunks".to_string(),+ "__xerj_passage_meta__emb".to_string(),+ ])+ );+ assert!(!typeless.contains("body"), "a text field is not typeless");+ }++ /// The exclusion is by DECLARED type, not by value shape: a numeric array+ /// under a non-vector mapping stays lexical exactly as before.+ #[test]+ fn a_non_vector_field_is_never_typeless() {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ schema+ .add_field(FieldConfig::new("cat", FieldType::Keyword))+ .unwrap();+ schema+ .add_field(FieldConfig::new("n", FieldType::Long))+ .unwrap();+ assert!(lexically_typeless_fields(&schema).is_empty());+ }++ /// A vector nested under an object mapping arrives as a DOTTED path — the+ /// first member of this set that is not a root-level name, and the reason+ /// `collect_text_fields` had to start matching the full path.+ #[test]+ fn a_nested_vector_is_collected_by_its_dotted_path() {+ let mut schema = Schema::empty();+ let mut parent = FieldConfig::new("passages", FieldType::Object);+ parent.fields.push(vector("vec", 4));+ schema.add_field(parent).unwrap();++ let typeless = lexically_typeless_fields(&schema);+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");+ assert!(typeless.contains("passages.vec_chunks"));+ assert!(+ !typeless.contains("vec"),+ "the leaf name alone must not match"+ );+ }++ /// `fts_excluded_fields` is the union, so the vector family must appear+ /// there too — that union is what flush, merge and `build_fts_field_configs`+ /// all read.+ #[test]+ fn fts_excluded_fields_unions_the_vector_family_in() {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ schema.add_field(vector("emb", 8)).unwrap();++ let excluded = fts_excluded_fields(&schema);+ for name in ["emb", "emb_chunks", "__xerj_passage_meta__emb"] {+ assert!(excluded.contains(name), "{name} missing from {excluded:?}");+ }+ assert!(!excluded.contains("body"));+ }++ /// The allocation-free short-circuit and the allocating walk must agree on+ /// "is this set empty", for every shape either of them can see — otherwise+ /// the fast path would silently disable the whole #328 query half on some+ /// schema.+ #[test]+ fn the_fast_path_predicate_agrees_with_the_walk() {+ let mut nested = FieldConfig::new("passages", FieldType::Object);+ nested.fields.push(vector("vec", 4));+ let mut deep = FieldConfig::new("a", FieldType::Object);+ let mut mid = FieldConfig::new("b", FieldType::Object);+ mid.fields.push(vector("c", 2));+ deep.fields.push(mid);+ let mut object_without_a_vector = FieldConfig::new("meta", FieldType::Object);+ object_without_a_vector+ .fields+ .push(FieldConfig::new("note", FieldType::Text));++ for (label, fields) in [+ ("empty", vec![]),+ ("text only", vec![FieldConfig::new("body", FieldType::Text)]),+ ("object with no vector", vec![object_without_a_vector]),+ ("root vector", vec![vector("emb", 8)]),+ ("nested vector", vec![nested]),+ ("two levels deep", vec![deep]),+ (+ "vector beside text",+ vec![FieldConfig::new("body", FieldType::Text), vector("emb", 8)],+ ),+ ] {+ let mut schema = Schema::empty();+ for f in fields {+ schema.add_field(f).unwrap();+ }+ assert_eq!(+ has_vector_field(&schema.fields),+ !crate::index::collect_dense_vector_fields(&schema).is_empty(),+ "predicate and walk disagree on `{label}`"+ );+ assert_eq!(+ has_vector_field(&schema.fields),+ !lexically_typeless_fields(&schema).is_empty(),+ "short-circuit changed the answer on `{label}`"+ );+ }+ }++ #[test]+ fn has_excluded_descendant_matches_only_strictly_below_the_path() {+ let set = std::collections::HashSet::from(["passages.vec".to_string(), "emb".to_string()]);+ assert!(has_excluded_descendant("passages", &set));+ // The path ITSELF is not a descendant of itself — the caller checks+ // `excluded.contains(&path)` separately, and double-counting it here+ // would make the prune fire for a leaf that was already skipped.+ assert!(!has_excluded_descendant("passages.vec", &set));+ assert!(!has_excluded_descendant("emb", &set));+ // A shared PREFIX is not a path prefix: `passages_extra` must not match+ // `passages.vec`, which is why the check tests for the `.` separator+ // rather than calling `starts_with` alone.+ assert!(!has_excluded_descendant("passages_extra", &set));+ assert!(!has_excluded_descendant("other", &set));+ }++ /// The flattened root-level blob is where a nested vector actually lands,+ /// so the prune has to happen there and not only in the recursion.+ #[test]+ fn extract_text_value_excluding_drops_the_nested_vector_from_the_blob() {+ let set = std::collections::HashSet::from(["passages.vec".to_string()]);+ let val = json!({+ "text": "quarterly liquidity",+ "vec": [0.125, 0.25, 0.5],+ "nested": {"keep": "yes"}+ });+ let pruned = extract_text_value_excluding(&val, "passages", &set);+ assert!(+ pruned.contains("quarterly liquidity"),+ "text kept: {pruned}"+ );+ assert!(+ pruned.contains("yes"),+ "unrelated sub-object kept: {pruned}"+ );+ assert!(+ !pruned.contains("0.125") && !pruned.contains("0.25"),+ "vector components must be gone: {pruned}"+ );+ }++ /// No excluded descendant → identical to `extract_text_value`, and no clone+ /// is paid. This is the path every document without a nested vector takes.+ #[test]+ fn extract_text_value_excluding_is_a_no_op_without_a_descendant() {+ let set = std::collections::HashSet::from(["other.vec".to_string()]);+ let val = json!({"text": "quarterly liquidity", "n": 7});+ assert_eq!(+ extract_text_value_excluding(&val, "passages", &set),+ extract_text_value(&val)+ );+ // A non-object short-circuits regardless of what the set holds.+ let scalar = json!("plain string");+ assert_eq!(+ extract_text_value_excluding(&scalar, "passages", &set),+ extract_text_value(&scalar)+ );+ }+}+#[cfg(test)]mod bounded_delta_counts_tests {use super::*;diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rsindex a1ee0150a..995a8cb38 100644--- a/engine/crates/xerj-engine/tests/integration.rs+++ b/engine/crates/xerj-engine/tests/integration.rs@@ -127,6 +127,766 @@ async fn test_semantic_vectors_stay_stored_but_not_fts_indexed_after_merge_and_rassert_queries(&idx).await;}+/// #328 — a USER-MAPPED `dense_vector` must contribute no lexical artifacts.+///+/// The `#12` fix above covers the vectors XERJ *generates* from a semantic+/// mapping. A field the user maps as `dense_vector` took a different route:+/// it is `indexed == true` (kNN needs the HNSW graph), so the `index: false`+/// arm of `memtable::fts_excluded_fields` never fired and the field's 128+/// floats were flattened to decimal strings and tokenised into a term+/// dictionary that no query path reads.+///+/// This pins all four halves of the fix:+/// * no `<seg>.emb.{fst,post,norms}` is written, at flush or after merge —+/// AND none for the `<field>_chunks` companion either. A `dense_vector`+/// never arrives alone: `passage_scored_vector_fields` gives every one of+/// them a `_chunks` multi-vector and a `__xerj_passage_meta__` sidecar, and+/// the companion is the bigger artifact of the two on a chunked corpus;+/// * kNN, `exists` and the lexical fields keep answering exactly as before;+/// * EVERY lexical leaf on the vector field answers 0, which is a BEHAVIOUR+/// CHANGE for a whole class of shapes rather than for one row. `main` sends+/// these to the stored-doc scan, which renders the float array back to+/// decimal text and matches against it, so `wildcard {emb:"0*"}` and+/// `fuzzy {emb:"0"~2}` answered ALL 300 documents, `prefix {emb:"0"}` 146,+/// `match_phrase_prefix {emb:"0"}` 50 and numeric `term`/`terms` 1 each.+/// None of that is a retrieval semantic anyone asked for and all of it is+/// now 0 — see the table in the (d) loop for the measured before/after;+/// * and neither does an EXPLICITLY-FIELDED lexical query. This is the one+/// the writer-side change cannot cover on its own and the one that fails+/// loudly: with the postings gone but no plan-time lowering,+/// `{"multi_match":{"query":"0","fields":["emb"]}}` returns EVERY document+/// instead of none, because the scan renders the float array to text and+/// every component contains a `0`. Measured on a 5,000-doc × 128-dim+/// corpus, post-`force_merge`, three source states (first call each, before+/// the result cache): `main` 0 hits / 0.409 ms, postings-removed-only+/// **5,000 hits** / 123.5 ms, this branch 0 hits / 0.012 ms. The same run+/// puts `{"term":{"emb":"<component>"}}` at 89.2 → 186.8 → 0.061 ms, all+/// three answering 0.+#[tokio::test]+async fn user_mapped_dense_vector_builds_no_fts_term_dictionary() {+ fn segment_files(dir: &std::path::Path) -> Vec<(String, u64)> {+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {+ for entry in std::fs::read_dir(dir).unwrap() {+ let entry = entry.unwrap();+ if entry.file_type().unwrap().is_dir() {+ walk(&entry.path(), out);+ } else {+ out.push((+ entry.file_name().to_string_lossy().into_owned(),+ entry.metadata().unwrap().len(),+ ));+ }+ }+ }+ let mut out = Vec::new();+ walk(dir, &mut out);+ out+ }++ const DIMS: usize = 128;+ const DOCS: usize = 300;++ let dir = TempDir::new().unwrap();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ schema+ .fields+ .push(FieldConfig::new("cat", FieldType::Keyword));+ schema.fields.push(FieldConfig::new("n", FieldType::Long));+ let mut emb = FieldConfig::new("emb", FieldType::Vector);+ emb.options.dimensions = Some(DIMS);+ emb.options.similarity = Some("cosine".to_string());+ schema.fields.push(emb);++ let engine = make_engine(&dir);+ engine.create_index("duprobe", schema).unwrap();+ let idx = engine.get_index("duprobe").unwrap();++ // Deterministic pseudo-random components with enough decimal digits that+ // each one is a distinct term — the shape that made the FST large.+ let component = |doc: usize, dim: usize| -> f64 {+ let h = (doc as u64)+ .wrapping_mul(6_364_136_223_846_793_005)+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0+ };+ let mut first_vector = Vec::new();+ for d in 0..DOCS {+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();+ if d == 0 {+ first_vector = v.clone();+ }+ idx.index_document(+ Some(format!("d{d}")),+ json!({+ "body": format!("quarterly liquidity evidence document number {d}"),+ "cat": if d % 2 == 0 { "even" } else { "odd" },+ "n": d,+ "emb": v,+ // The `_chunks` companion, i.e. the per-document MULTI-vector.+ // Present in the fixture because a chunked corpus is the shape+ // RFC #148 reports, and excluding only `emb` leaves this one+ // fully tokenised.+ "emb_chunks": [v, v],+ }),+ )+ .await+ .unwrap();+ }+ idx.refresh().await.unwrap();+ idx.force_merge(1).await.unwrap();++ let files = segment_files(dir.path());+ let bytes_of = |suffix: &str| -> u64 {+ files+ .iter()+ .filter(|(name, _)| name.ends_with(suffix))+ .map(|(_, len)| *len)+ .sum()+ };+ // Reported so the saving is a measured number in the log, not a claim.+ //+ // TWO totals, and only the second one is quotable. The whole data dir+ // includes the write-ahead log, whose tail is reclaimed asynchronously, so+ // it is NOT reproducible: eight runs of this test on one machine printed a+ // whole-dir total between 858,147 B and 1,105,001 B, a 29% spread, entirely+ // because the surviving `.wal` files ranged from 256 B to 6 MB while the+ // segment bytes did not move at all. `durable-index` excludes `.wal` and is+ // reproducible to ~230 B (the run-to-run jitter in the HNSW `graph.bin`):+ // the same eight runs printed 857,843 – 858,071 B. That is the number the+ // CHANGELOG quotes and the one the ceiling below pins.+ let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();+ let durable: u64 = files+ .iter()+ .filter(|(name, _)| !name.ends_with(".wal"))+ .map(|(_, len)| *len)+ .sum();+ eprintln!(+ "#328 lexical bytes: emb.fst={} emb.post={} emb.norms={} emb_chunks.fst={} | \+ body.fst={} body.post={} cat.fst={} n.fst={} | durable-index={durable} \+ whole-dir={whole_dir}",+ bytes_of(".emb.fst"),+ bytes_of(".emb.post"),+ bytes_of(".emb.norms"),+ bytes_of(".emb_chunks.fst"),+ bytes_of(".body.fst"),+ bytes_of(".body.post"),+ bytes_of(".cat.fst"),+ bytes_of(".n.fst"),+ );+ // The published saving, enforced. `origin/main` writes 3,249,323 B of+ // durable index for this fixture and this branch writes ~858 KB; the ceiling+ // sits three orders of magnitude clear of the jitter and still 2.2 MB below+ // `main`, so it is a real fail-before and not a golden-file trap. Watched+ // fail at exactly 3,249,323 with both source files reverted to `origin/main`.+ assert!(+ durable < 1_500_000,+ "durable index must lose the vector term dictionaries; got {durable} B \+ (`origin/main` writes 3,249,323 B for this fixture, this branch ~858 KB)"+ );++ for suffix in [+ ".emb.fst",+ ".emb.post",+ ".emb.norms",+ // The companion, and it is the BIGGER half. On this 300-doc fixture,+ // `main` writes `.emb.fst` = 796,274 B and `.emb_chunks.fst` =+ // 1,592,118 B; at 5,000 × 128 it is 13,256,659 B against 26,518,005 B.+ // Excluding the base name alone would leave more behind than it+ // removes.+ ".emb_chunks.fst",+ ".emb_chunks.post",+ ".emb_chunks.norms",+ ] {+ assert!(+ !files.iter().any(|(name, _)| name.ends_with(suffix)),+ "a dense_vector field must contribute no `{suffix}` artifact; got {:?}",+ files+ .iter()+ .filter(|(name, _)| name.contains(".emb"))+ .collect::<Vec<_>>()+ );+ }+ // The lexical fields are untouched — this is a per-type exclusion, not a+ // blanket one.+ for suffix in [".body.fst", ".cat.fst", ".n.fst"] {+ assert!(+ files.iter().any(|(name, _)| name.ends_with(suffix)),+ "lexical fields must keep their term dictionaries; missing {suffix}"+ );+ }++ // (a) kNN still answers from the HNSW graph.+ let knn = idx+ .search(+ &parse_request(&json!({+ "query": {"knn": {"field": "emb", "query_vector": first_vector, "k": 3}},+ "size": 10+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(knn.hits.len(), 3, "kNN must still retrieve neighbours");+ assert_eq!(+ knn.hits[0].id, "d0",+ "nearest neighbour must be the probe doc"+ );++ // (b) `exists` is answered from `_source` / doc values, never the postings.+ let exists = idx+ .search(&make_search(json!({"exists": {"field": "emb"}})))+ .await+ .unwrap();+ assert_eq!(exists.total.value, DOCS as u64);++ // (c) the lexical fields still match.+ let lexical = idx+ .search(&make_search(json!({"match": {"body": "liquidity"}})))+ .await+ .unwrap();+ assert_eq!(lexical.total.value, DOCS as u64);+ let kw = idx+ .search(&make_search(json!({"term": {"cat": "even"}})))+ .await+ .unwrap();+ assert_eq!(kw.total.value, (DOCS / 2) as u64);++ // (d) EVERY lexical leaf that names the vector field answers 0.+ //+ // This is a CLASS, not a list of individually interesting queries, and the+ // class is the behaviour change this fix ships. On `main` a lexical clause+ // on `emb` falls through the missing/oversized term dictionary onto the+ // stored-doc scan, which renders the float array back to decimal text and+ // matches against that text — so whether a shape answered 0 or answered+ // most of the corpus on `main` depended entirely on whether the rendered+ // decimals happened to contain the probe. Measured on THIS fixture with+ // both source files reverted to `origin/main` (300 docs × 128 dim, after+ // `force_merge(1)`), `main` → this branch:+ //+ // term {emb: <component>} numeric .. 1 → 0+ // terms {emb: [<component>]} numeric .. 1 → 0+ // prefix {emb: "0"} .. 146 → 0+ // wildcard{emb: "0*"} .. 300 → 0 (every document)+ // fuzzy {emb: {value:"0", fuzziness:2}} 300 → 0 (every document)+ // match_phrase_prefix {emb: "0"} .. 50 → 0+ // prefix {emb_chunks: "0"} .. 146 → 0+ // wildcard{emb_chunks: "0*"} .. 300 → 0 (every document)+ // fuzzy {emb_chunks: {value:"0", …}} .. 300 → 0 (every document)+ //+ // …and the shapes whose probe text simply never appeared in a rendered+ // float answered 0 on both sides. All of them are pinned below at 0, so+ // the class is closed rather than sampled: a `dense_vector` and its+ // `_chunks` companion have no lexical surface at all.+ //+ // The two halves of the fix cover DIFFERENT rows of that table, which is why+ // both are needed. Running the same loop with the write-side exclusion alone+ // (the plan-time lowering disabled in `search_inner`) leaves exactly four+ // shapes non-zero — numeric `term`/`terms` at 1 each and the two+ // `multi_match`es at 300 — while `prefix`/`wildcard`/`fuzzy`/+ // `match_phrase_prefix` are already 0, because a multi-term query has no+ // term dictionary left to enumerate. Reverting both files to `origin/main`+ // leaves nine non-zero. Both counts were watched.+ //+ // A POSITIVE component: a leading `-` is `query_string`'s NOT operator,+ // which would make the probe measure the parser rather than the field.+ let probe_num = first_vector+ .iter()+ .copied()+ .find(|v| *v > 0.0)+ .expect("fixture vector has a positive component");+ let probe = format!("{probe_num}");+ let mut lexical_leaks: Vec<String> = Vec::new();+ for q in [+ json!({"term": {"emb": probe}}),+ json!({"match": {"emb": probe}}),+ json!({"match": {"emb": "0.5"}}),+ json!({"query_string": {"query": probe}}),+ json!({"multi_match": {"query": probe, "fields": ["*"]}}),+ // Numeric `term` / `terms` / `range`.+ //+ // The first two answered 1 on `main`: the stored-doc scan's `Term` arm+ // matches any ELEMENT of a JSON array — "this document's vector happens+ // to contain this float". ES rejects `term` on a `dense_vector`+ // outright and Lucene gives a field with no terms an empty scorer, so 0+ // is the answer this pins.+ json!({"term": {"emb": probe_num}}),+ json!({"terms": {"emb": [probe_num]}}),+ json!({"terms": {"emb": [probe]}}),+ json!({"range": {"emb": {"gte": -2.0, "lte": 2.0}}}),+ // The MULTI-TERM leaves, and the loudest half of the class. A rendered+ // float almost always contains a `0` somewhere, so on `main` these+ // matched a large fraction of the corpus — `wildcard` and `fuzzy`+ // matched ALL 300 documents — purely as an artifact of decimal+ // rendering. Every one of them is a `field`-naming leaf in+ // `lower_lexically_typeless_clauses`'s first arm.+ json!({"prefix": {"emb": "0"}}),+ json!({"wildcard": {"emb": "0*"}}),+ json!({"fuzzy": {"emb": {"value": "0", "fuzziness": 2}}}),+ json!({"regexp": {"emb": "0.*"}}),+ json!({"match_phrase": {"emb": probe}}),+ json!({"match_phrase_prefix": {"emb": "0"}}),+ // EXPLICITLY-FIELDED forms — the half the writer-side change cannot+ // reach. Each of these names `emb` outright rather than expanding onto+ // it, so no `*`-expansion filter helps; without a plan-time lowering+ // they land on the stored-doc scan. The `multi_match` one is the+ // correctness failure, not merely a slow one: postings-removed-only,+ // it answers DOCS instead of 0.+ json!({"multi_match": {"query": "0", "fields": ["emb"]}}),+ json!({"simple_query_string": {"query": "0", "fields": ["emb"]}}),+ json!({"query_string": {"query": "0", "default_field": "emb"}}),+ // …and the `_chunks` companion, which is in the exclusion set for the+ // same reason and therefore needs the same lowering — including the+ // multi-term leaves, which matched 146 and 300 documents on `main`.+ json!({"term": {"emb_chunks": probe}}),+ json!({"prefix": {"emb_chunks": "0"}}),+ json!({"wildcard": {"emb_chunks": "0*"}}),+ json!({"fuzzy": {"emb_chunks": {"value": "0", "fuzziness": 2}}}),+ json!({"multi_match": {"query": "0", "fields": ["emb_chunks"]}}),+ ] {+ let hits = idx.search(&make_search(q.clone())).await.unwrap();+ // Accumulated rather than asserted per row, deliberately: this loop is+ // pinning a CLASS, so a regression report that names every shape that+ // came back non-zero is worth far more than one that stops at the first.+ if hits.total.value != 0 {+ lexical_leaks.push(format!("{q} → {} hits", hits.total.value));+ }+ }+ assert!(+ lexical_leaks.is_empty(),+ "a lexical query on a dense_vector (or its `_chunks` companion) must \+ match nothing; {} of the shapes above did not:\n {}",+ lexical_leaks.len(),+ lexical_leaks.join("\n ")+ );++ // (d2) a MIXED field list keeps its lexical members. Only the vector entry+ // is dropped, so this is `{"multi_match":{"query":"liquidity","fields":+ // ["body"]}}` in effect — the whole corpus, not nothing. The empty-list+ // lowering must not swallow a list that still has a real field in it.+ let mixed = idx+ .search(&make_search(+ json!({"multi_match": {"query": "liquidity", "fields": ["emb", "body"]}}),+ ))+ .await+ .unwrap();+ assert_eq!(+ mixed.total.value, DOCS as u64,+ "dropping the vector entry must leave the rest of a `fields` list intact"+ );++ // (d2b) COMPOSITES. Lowering a leaf to `match_none` is only half a rewrite:+ // the enclosing `bool` then has to fold it the way the EVALUATOR reads it,+ // and `doc_matches_query`'s Bool arm makes `should` required whenever `must`+ // and `filter` are empty — `must_not` does NOT relax it. The first cut of+ // this fix skipped the "every optional clause is dead" fold whenever a+ // `must_not` was present and then dropped the dead clause anyway, which+ // turned the first row below from 0 hits into 150.+ //+ // Every expected value here is `main`'s answer for the same request: a+ // lexical clause on `emb` matched nothing before the postings went away and+ // must match nothing after.+ for (q, want) in [+ // The regression. `should` is required, its only clause is dead.+ (+ json!({"bool": {+ "should": [{"term": {"emb": probe}}],+ "must_not": [{"term": {"cat": "odd"}}]+ }}),+ 0u64,+ ),+ // Should-only, no `must_not` — the shape a `simple_query_string` lowers+ // to.+ (json!({"bool": {"should": [{"term": {"emb": probe}}]}}), 0),+ // A live `should` beside the dead one still matches on its own.+ (+ json!({"bool": {"should": [+ {"term": {"emb": probe}},+ {"term": {"cat": "even"}}+ ]}}),+ (DOCS / 2) as u64,+ ),+ // `must` / `filter`: an unsatisfiable REQUIRED clause sinks the bool.+ (+ json!({"bool": {+ "must": [{"term": {"emb": probe}}, {"match": {"body": "liquidity"}}]+ }}),+ 0,+ ),+ (json!({"bool": {"filter": [{"term": {"emb": probe}}]}}), 0),+ // `must_not`: a clause that matches nothing EXCLUDES nothing. Same+ // answer as `main`; what changes is the route — a `match_none` left+ // sitting in `must_not` un-projects the whole bool in+ // `query_node_to_fts_with_keyword_fields` and sends a query that used+ // to run on the inverted index to the stored-doc scan.+ (+ json!({"bool": {"must_not": [{"term": {"emb": probe}}]}}),+ DOCS as u64,+ ),+ (+ json!({"bool": {+ "must": [{"match": {"body": "liquidity"}}],+ "must_not": [{"term": {"emb": probe}}]+ }}),+ DOCS as u64,+ ),+ // `constant_score` is the only wrapper with an arm in the FTS+ // projection, so it is the only one where failing to bubble up the+ // `match_none` costs an inverted-index lookup.+ (+ json!({"constant_score": {"filter": {"term": {"emb": probe}}}}),+ 0,+ ),+ ] {+ let hits = idx.search(&make_search(q.clone())).await.unwrap();+ assert_eq!(+ hits.total.value, want,+ "composite over a dense_vector clause: expected {want}, got {} for {q}",+ hits.total.value+ );+ }++ // (d2c) …and the composites that keep their HIT COUNT still change their+ // `_score`, which is wire-visible and therefore has to be pinned here+ // rather than discovered by a user.+ //+ // Cause: `lower_lexically_typeless_clauses` drops the dead clause out of+ // `should`, which makes the surviving bool PROJECTABLE through+ // `query_node_to_fts_with_keyword_fields`. On `main` the same bool contains+ // a clause that cannot be projected, so the whole query runs on the+ // stored-doc scan and is scored by `score_query_against_doc` instead of+ // BM25. Isolated to the fold: with the write-side exclusion alone and the+ // lowering disabled, both scores below are byte-identical to `main`.+ //+ // Measured on this fixture, `origin/main` → this branch:+ //+ // bool{should:[term emb, term cat:even]}+ // hits 150 → 150 (unchanged)+ // _score 0.008402659 → 0.6931471+ // max_score 1.6931472 → 0.6931471+ // bool{must:[match body], must_not:[term emb]}+ // hits 300 → 300 (unchanged)+ // _score 0.008402659 → 0.008402659 (unchanged)+ // max_score 1.6931472 → 0.008402659+ //+ // The assertions are properties rather than golden floats, so they survive+ // a BM25 tuning change but still fail on `main`:+ //+ // 1. `max_score` equals the top hit's `_score`. On `main` it does NOT for+ // either shape — the scan path reports a `max_score` of 1.6931472 that+ // no returned hit carries. That inconsistency is what goes away.+ // 2. a bool whose only surviving `should` is `{"term":{"cat":"even"}}`+ // scores exactly like that term query on its own. On `main` it scores+ // 0.008402659 against the term query's 0.6931471.+ let cat_alone = idx+ .search(&make_search(json!({"term": {"cat": "even"}})))+ .await+ .unwrap();+ let should_mixed = idx+ .search(&make_search(json!({"bool": {"should": [+ {"term": {"emb": probe}},+ {"term": {"cat": "even"}}+ ]}})))+ .await+ .unwrap();+ let mustnot_bool = idx+ .search(&make_search(json!({"bool": {+ "must": [{"match": {"body": "liquidity"}}],+ "must_not": [{"term": {"emb": probe}}]+ }})))+ .await+ .unwrap();+ // Hit counts first, so a regression that empties one of these reports+ // "expected 150 hits" rather than panicking on `hits[0]` below.+ for (label, res, want) in [+ ("term {cat:even}", &cat_alone, (DOCS / 2) as u64),+ (+ "bool{should:[dead, term cat]}",+ &should_mixed,+ (DOCS / 2) as u64,+ ),+ (+ "bool{must:[match], must_not:[dead]}",+ &mustnot_bool,+ DOCS as u64,+ ),+ ] {+ assert_eq!(+ res.total.value, want,+ "{label}: the score probes must keep `main`'s hit count"+ );+ assert!(!res.hits.is_empty(), "{label}: expected a scored top hit");+ }+ eprintln!(+ "#328 scores: term{{cat}}={:?}/{:?} should-mixed={:?}/{:?} must-not={:?}/{:?} \+ (top/max)",+ cat_alone.hits[0].score,+ cat_alone.max_score,+ should_mixed.hits[0].score,+ should_mixed.max_score,+ mustnot_bool.hits[0].score,+ mustnot_bool.max_score,+ );+ for (label, res) in [+ ("bool{should:[dead, term cat]}", &should_mixed),+ ("bool{must:[match], must_not:[dead]}", &mustnot_bool),+ ] {+ assert_eq!(+ res.max_score,+ Some(res.hits[0].score),+ "{label}: max_score must be the top hit's _score, got max={:?} top={:?}",+ res.max_score,+ res.hits[0].score+ );+ }+ assert_eq!(+ should_mixed.hits[0].score, cat_alone.hits[0].score,+ "a bool whose only live `should` is `term cat:even` must score exactly \+ like that term query alone; got {:?} against {:?}",+ should_mixed.hits[0].score, cat_alone.hits[0].score+ );++ // (d3) the mapping still describes the field — the exclusion is about+ // lexical BYTES, not about the field's existence. This is the leaf of the+ // `_field_caps` guarantee reachable from the engine crate: `_field_caps`+ // is served by the API layer from exactly this schema and never opens a+ // term dictionary, so a field that reports `dense_vector` here reports+ // `dense_vector` there.+ let mapped = idx.schema().await;+ let emb_field = mapped+ .fields+ .iter()+ .find(|f| f.name == "emb")+ .expect("`emb` must still be a mapped field");+ assert!(matches!(emb_field.field_type, FieldType::Vector));+ assert_eq!(emb_field.options.dimensions, Some(DIMS));++ // (d4) highlighting a vector field yields no fragments and no panic, while+ // a real text field in the same request still highlights. Highlighting+ // resolves against the stored document and never opens the field's FST,+ // so removing the FST cannot change it.+ let hl = idx+ .search(+ &parse_request(&json!({+ "query": {"match": {"body": "liquidity"}},+ "size": 1,+ "highlight": {"fields": {"body": {}, "emb": {}}}+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ let frags = hl.hits[0]+ .highlight+ .as_ref()+ .expect("highlight must be present");+ assert!(+ frags.get("body").is_some_and(|f| !f.is_empty()),+ "a text field must still highlight"+ );+ assert!(+ frags.get("emb").is_none_or(|f| f.is_empty()),+ "a dense_vector must produce no highlight fragments, got {:?}",+ frags.get("emb")+ );++ // (e) the index reopens and searches after a restart. The exclusion is a+ // WRITE-side rule and no reader requires the field's sidecar to exist, so+ // a segment written without one loads exactly like one written with it.+ // (This does not, and cannot from inside one test binary, exercise a+ // segment produced by the previous release: those keep their `.emb.*`+ // files until a merge rewrites them, and the per-segment `fts_has_field`+ // gate reads whichever shape it finds.)+ drop(idx);+ drop(engine); // release the data dir's `node.lock` before reopening it+ let reopened = make_engine(&dir);+ let idx = reopened.get_index("duprobe").unwrap();+ let after = idx+ .search(&make_search(json!({"match": {"body": "liquidity"}})))+ .await+ .unwrap();+ assert_eq!(after.total.value, DOCS as u64);+}++/// #328, nested half — a `dense_vector` under an OBJECT mapping.+///+/// Excluding the leaf path alone moves no bytes here, and that is the trap the+/// first cut of this fix fell into: the segment builder never writes a+/// `passages.vec` field at all. It flattens the whole `passages` object into+/// ONE text field, so the vector's 64 decimal components land in+/// `<seg>.passages.fst` under the parent's name. The size of that file is+/// reported below rather than asserted to a golden number, and the ceiling is+/// deliberately generous: the point is that the fixture's 19,200 vector+/// components are gone, and they cannot fit in 4 KiB.+///+/// Both mapping shapes that reach `FieldType::Vector` are pinned, because they+/// take different routes into the schema and only one of them was covered by+/// the walk: a dotted top-level name (`"passages.vec"`, what `put_mapping`+/// produces) and a `vec` sub-mapping under a `passages` object (what+/// `es_properties_to_fields` produces from nested `properties`).+#[tokio::test]+async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary() {+ const DIMS: usize = 64;+ const DOCS: usize = 300;++ let component = |doc: usize, dim: usize| -> f64 {+ let h = (doc as u64)+ .wrapping_mul(6_364_136_223_846_793_005)+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0+ };++ for shape in ["dotted-name", "sub-mapping"] {+ let dir = TempDir::new().unwrap();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ let mut vec_field = FieldConfig::new(+ if shape == "dotted-name" {+ "passages.vec"+ } else {+ "vec"+ },+ FieldType::Vector,+ );+ vec_field.options.dimensions = Some(DIMS);+ vec_field.options.similarity = Some("cosine".to_string());+ if shape == "dotted-name" {+ schema.fields.push(vec_field);+ } else {+ let mut parent = FieldConfig::new("passages", FieldType::Object);+ parent.fields.push(vec_field);+ schema.fields.push(parent);+ }++ let engine = make_engine(&dir);+ engine.create_index("nested-vec", schema).unwrap();+ let idx = engine.get_index("nested-vec").unwrap();++ let mut first_vector = Vec::new();+ for d in 0..DOCS {+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();+ if d == 0 {+ first_vector = v.clone();+ }+ idx.index_document(+ Some(format!("d{d}")),+ json!({+ "body": format!("quarterly liquidity evidence {d}"),+ "passages": {"vec": v},+ }),+ )+ .await+ .unwrap();+ }+ idx.refresh().await.unwrap();+ idx.force_merge(1).await.unwrap();++ let mut files: Vec<(String, u64)> = Vec::new();+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {+ for entry in std::fs::read_dir(dir).unwrap() {+ let entry = entry.unwrap();+ if entry.file_type().unwrap().is_dir() {+ walk(&entry.path(), out);+ } else {+ out.push((+ entry.file_name().to_string_lossy().into_owned(),+ entry.metadata().unwrap().len(),+ ));+ }+ }+ }+ walk(dir.path(), &mut files);++ let parent_fst: u64 = files+ .iter()+ .filter(|(name, _)| name.ends_with(".passages.fst"))+ .map(|(_, len)| *len)+ .sum();+ // Reported so the saving is a measured number in the log, not a claim.+ // `passages.fst` is the load-bearing figure and it is exact — 398,558 B+ // on `origin/main`, 45 B here, for both mapping shapes. `durable-index`+ // is diagnostic only and `whole-dir` is not reproducible at all: it+ // carries the write-ahead log, whose tail is reclaimed asynchronously.+ // Nothing quotes either of them; see the sibling test for the numbers+ // the CHANGELOG does quote.+ let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();+ let durable: u64 = files+ .iter()+ .filter(|(name, _)| !name.ends_with(".wal"))+ .map(|(_, len)| *len)+ .sum();+ eprintln!(+ "#328 nested [{shape}]: passages.fst={parent_fst} \+ durable-index={durable} whole-dir={whole_dir}"+ );+ // A generous ceiling, not a golden number: the point is that the+ // 19,200 vector components are gone, and they cannot fit in 4 KiB.+ assert!(+ parent_fst < 4096,+ "[{shape}] the parent object's term dictionary must not carry the \+ nested vector's components; `.passages.fst` = {parent_fst} B"+ );+ assert!(+ !files+ .iter()+ .any(|(name, _)| name.ends_with(".passages.vec.fst")),+ "[{shape}] the nested vector leaf must have no term dictionary either"+ );++ // The parent is still a real field, just without the vector in it.+ let lexical = idx+ .search(&make_search(json!({"match": {"body": "liquidity"}})))+ .await+ .unwrap();+ assert_eq!(lexical.total.value, DOCS as u64, "[{shape}]");++ // A component of the vector must not be findable through the parent.+ let probe = format!(+ "{}",+ first_vector+ .iter()+ .copied()+ .find(|v| *v > 0.0)+ .expect("fixture vector has a positive component")+ );+ for q in [+ json!({"match": {"passages": probe}}),+ json!({"term": {"passages": probe}}),+ ] {+ let hits = idx.search(&make_search(q.clone())).await.unwrap();+ assert_eq!(+ hits.total.value, 0,+ "[{shape}] a vector component must not be lexically findable \+ through its parent object, got {} for {q}",+ hits.total.value+ );+ }++ // kNN on the nested field still answers from the HNSW graph.+ let knn = idx+ .search(+ &parse_request(&json!({+ "query": {"knn": {+ "field": "passages.vec", "query_vector": first_vector, "k": 3+ }},+ "size": 10+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(knn.hits.len(), 3, "[{shape}] kNN must still retrieve");+ assert_eq!(+ knn.hits[0].id, "d0",+ "[{shape}] nearest must be the probe doc"+ );+ }+}+#[tokio::test]async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {let dir = TempDir::new().unwrap();