Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

perf(engine): drop the term dictionary for dense_vector, and the doc-scan fallback with it (#328)

xerj-orgb699657e17e3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c46c5dcfb..a688ec81d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -153,6 +153,132 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
from the moment it is allowlisted. Seed the target with snapshot/restore
first if it needs the existing documents.
+### Performance
+
+- **A field mapped `dense_vector` no longer gets a full-text term dictionary**
+ ([#328](https://github.com/xerj-org/xerj/issues/328)). The field was fed
+ through the lexical indexing path and given an FST + postings that no query
+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from
+ `_source`/doc values, and `_field_caps` and highlighting never open the
+ field's term dictionary. `extract_field_text`'s array arm joined the
+ components with spaces, so a 128-dim vector became one enormous
+ decimal-string term per document.
+
+ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long +
+ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing
+ every file in the data directory except the write-ahead log:
+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`
+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,
+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,
+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are
+ byte-identical on both sides.
+
+ `.wal` is excluded from that total because its tail is reclaimed
+ asynchronously and it is not reproducible: on the 300-doc fixture the
+ committed regression test uses, eight runs put the *whole directory* anywhere
+ between 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. The
+ test prints both and asserts on the durable one. It reports
+ **3,249,323 B → 857,843–858,071 B** for that fixture (`main` side measured by
+ reverting `index.rs` and `memtable.rs` to `main` on the same fixture), which
+ is the same −73.6% at 1/17th the corpus.
+
+ A `dense_vector` nested under an object mapping is covered too — its
+ components were landing in the *parent's* term dictionary, because the segment
+ builder flattens the whole object into one text field. On the nested fixture
+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach
+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`
+ sub-mapping under a `passages` object).
+
+ The exclusion covers the whole family a vector field generates, not just the
+ base name: `<field>_chunks` (the per-document multi-vector) and
+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides
+ which fields get an HNSW graph. The companion is the bigger half —
+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change
+ index — so a base-name-only exclusion would leave more behind than it removes.
+
+ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now
+ lowered to `match_none` at plan time rather than falling through to the
+ stored-doc scan. That is the correctness half of the change, not only the
+ speed half: with the postings removed and no lowering,
+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**
+ instead of none, because the scan renders the float array to text and every
+ component contains a `0`. Measured on the corpus above, first call for each
+ shape (`main` → postings-gone-without-lowering → this release):
+
+ | query | `main` | no lowering | this release |
+ |---|---|---|---|
+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |
+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |
+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |
+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |
+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |
+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |
+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |
+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |
+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |
+
+ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`
+ list keeps its lexical members, and a `nested` query is deliberately not
+ rewritten (its field names resolve against the element, not the root).
+
+ **Three behaviour changes to know about:**
+
+ - **A `dense_vector` (and its `_chunks` companion) now has no lexical surface
+ at all: every lexical leaf naming one answers 0 hits.** Before this release
+ such a clause fell through to the stored-doc scan, which renders the float
+ array back to decimal text and matches against *that*, so whether a shape
+ answered 0 or answered most of the corpus depended entirely on whether the
+ rendered decimals happened to contain the probe. This is a CLASS of changed
+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,
+ `main` → this release:
+
+ | query | before | after |
+ |---|---:|---:|
+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |
+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |
+ | `prefix {emb: "0"}` | 146 | 0 |
+ | `prefix {emb_chunks: "0"}` | 146 | 0 |
+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |
+ | `term {emb: <exact component>}` numeric | 1 | 0 |
+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |
+
+ Shapes whose probe text never appeared in a rendered float (`term` with a
+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,
+ `query_string`) answered 0 on both sides. ES rejects these queries on a
+ `dense_vector` outright and Lucene gives a field with no terms an empty
+ scorer, so 0 is the answer this release adopts for all of them. The
+ regression test pins the whole class rather than sampling it.
+ - **A `bool` that mentions a `dense_vector` can change `_score` and
+ `max_score` without changing its hit count.** Dropping the dead clause makes
+ the surviving bool projectable onto the inverted index, so BM25 scores it
+ where the stored-doc scan used to. Measured on the same fixture:
+
+ | query | hits | `_score` | `max_score` |
+ |---|---|---|---|
+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |
+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |
+
+ Both moves are toward consistency — on `main` these queries reported a
+ `max_score` that **no returned hit carried**, and the first now scores
+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces
+ to. Absolute scores are not part of XERJ's compatibility surface, but
+ anything comparing scores across a version boundary or asserting on
+ `max_score` for a bool that touches a vector field will see this. Plain
+ lexical queries are unaffected; the blast radius is bools with a
+ `dense_vector` clause in them.
+ - **Existing indices keep their bloat until a merge rewrites them.** This is a
+ write-side rule. Segments written by an earlier release keep their `.emb.*`
+ / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads
+ whichever shape it finds. Run an explicit
+ `POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an
+ index that already exists.
+
+ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,
+ where XERJ answers `200`. This release removes bytes and adopts Lucene's
+ zero-hit answer; it does not add the rejection.
+
## [1.0.0-rc.16] - 2026-08-13
Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 1afc4b49a..0c71c71d8 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -13672,7 +13672,20 @@ impl Index {
),
)));
}
- resolved
+ // #328's query half, and it rides the same guard for the same
+ // reason: a `dense_vector` has no term dictionary to consult, so a
+ // lexical clause naming one is lowered to `match_none` HERE,
+ // before `is_doc_scan_query` or `query_node_to_fts` ever see the
+ // tree. `unsearchable_query_field` cannot cover it — a vector
+ // field is `indexed`, so it is not "unsearchable" — and without
+ // the lowering the missing postings hand the clause to the
+ // stored-doc scan. See `lower_lexically_typeless_clauses`.
+ let typeless = crate::memtable::lexically_typeless_fields(&schema.schema);
+ if typeless.is_empty() {
+ resolved
+ } else {
+ lower_lexically_typeless_clauses(&resolved, &typeless)
+ }
};
let query = &resolved_query;
@@ -14055,6 +14068,22 @@ impl Index {
if !f.options.indexed && !f.options.doc_values {
continue;
}
+ // #328 — a `dense_vector` has no lexical representation at
+ // all, so it is not a field a `*` expansion may project onto.
+ // A vector field is declared with vector attributes and
+ // nothing else, so it never acquires index options and can
+ // never own terms (`lucene/core/src/java/org/apache/lucene/
+ // document/KnnFloatVectorField.java:70`, and `FieldInfo.java:565`
+ // for what `IndexOptions.NONE` means). Leaving it in would push
+ // a clause onto a field whose term dictionary this release
+ // stops building (`memtable::fts_excluded_fields`), and the
+ // per-segment `fts_has_field` gate would then hand the whole
+ // query to the stored-doc scan. A clause that NAMES the field
+ // rather than expanding onto it is handled separately, by
+ // `lower_lexically_typeless_clauses`.
+ if matches!(f.field_type, FieldType::Vector) {
+ continue;
+ }
if matches!(f.field_type, FieldType::Text) {
tf.push(f.name.clone());
} else {
@@ -19159,7 +19188,18 @@ fn extract_fts_fields_excluding(
if excluded.contains(key) {
continue;
}
- let text = extract_field_text(val);
+ // This is the MERGE path, and a merge must not resurrect postings
+ // that flush correctly skipped (#328). A root key is caught above;
+ // an object-valued key is flattened whole by `extract_field_text`,
+ // so a `dense_vector` nested under it (`passages.vec`) would come
+ // back here even though flush dropped it. Prune it out of the
+ // flattened blob exactly the way `collect_text_fields` does.
+ let text = if val.is_object() && crate::memtable::has_excluded_descendant(key, excluded)
+ {
+ crate::memtable::extract_text_value_excluding(val, key, excluded)
+ } else {
+ extract_field_text(val)
+ };
if !text.is_empty() {
fields.insert(key.clone(), text);
}
@@ -29374,7 +29414,12 @@ fn extract_numeric_vector(source: &Value, field: &str) -> Option<Vec<f32>> {
/// Collect every dense_vector-mapped field in schema order (dotted paths
/// for fields nested under object mappings).
-fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {
+///
+/// Also the single source of truth for `memtable::lexically_typeless_fields`
+/// (#328) — the set of fields that get an HNSW graph and the set that get no
+/// term dictionary must be the same set, or a field ends up with both or
+/// neither.
+pub(crate) fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {
fn walk(fields: &[FieldConfig], prefix: &str, out: &mut Vec<String>) {
for fc in fields {
let path = if prefix.is_empty() {
@@ -30069,6 +30114,982 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {
}
}
+/// Lower every LEXICAL clause that names a lexically typeless field to
+/// `QueryNode::MatchNone` — the query half of #328.
+///
+/// `typeless` is `memtable::lexically_typeless_fields`: the `dense_vector`
+/// fields and their `_chunks` / passage-metadata companions, i.e. exactly the
+/// fields this release stops giving a term dictionary. Dropping the postings
+/// alone is not enough, and this is the part the write-side change cannot do by
+/// itself. Every OTHER member of `fts_excluded_fields` is protected from the
+/// stored-doc scan by `unsearchable_query_field`, which 400s the query before
+/// execution; a `dense_vector` is not, because it IS indexed — kNN needs the
+/// HNSW graph. So without this pass a lexical clause on a vector field falls
+/// through the per-segment `fts_has_field` gate onto the stored-doc scan, whose
+/// `Term` arm matches ANY ELEMENT of a JSON array and whose `match` arm splits
+/// on non-alphanumerics. Measured post-`force_merge` on a 5,000-doc × 128-dim
+/// corpus with the postings removed but this pass absent, against `main`
+/// (first call for each shape, before the result cache sees it):
+/// `{"term":{"emb":"<component>"}}` 89.2 ms → 186.8 ms → **0.061 ms** with
+/// this pass, and `{"multi_match":{"fields":["emb"],"query":"0"}}`
+/// **0 hits → all 5,000 hits** → 0 hits, because every rendered float component
+/// contains a `0`. The `multi_match` row is a correctness regression, not a
+/// slow path, and it is the reason this pass exists.
+///
+/// Lucene's answer to "this field has no terms" is the same one, in two places:
+/// `Terms.getTerms` hands back `Terms.EMPTY` rather than null so the scorer is
+/// empty instead of absent (`lucene/core/src/java/org/apache/lucene/index/
+/// Terms.java:40`), and a query that cannot match is REWRITTEN to
+/// `MatchNoDocsQuery` (`lucene/core/src/java/org/apache/lucene/search/
+/// MatchNoDocsQuery.java:23`) — as `AbstractKnnVectorQuery` does on an empty
+/// index (`lucene/core/src/test/org/apache/lucene/search/
+/// BaseKnnVectorQueryTestCase.java:164`). Neither ever falls back to walking
+/// the corpus: `TestTermAutomatonQuery.testFieldMissing` asserts a plain 0 hits
+/// for a query on `bogusfield` (`lucene/sandbox/src/test/org/apache/lucene/
+/// sandbox/search/TestTermAutomatonQuery.java:1051`). The empty FIELD LIST case
+/// below is `BooleanQuery.rewrite`'s first line, verbatim in spirit —
+/// `if (clauses.size() == 0) return new MatchNoDocsQuery("empty BooleanQuery")`
+/// (`lucene/core/src/java/org/apache/lucene/search/BooleanQuery.java:271`), and
+/// the per-occur rules in the `Bool` arm are the same method's, clause for
+/// clause (`:323` SHOULD/MUST_NOT ignored, `:327` MUST/FILTER returned).
+///
+/// This is deliberately NOT a rejection: XERJ keeps answering 200 where ES 400s
+/// `term`/`match`/`range` on a `dense_vector`. Adding the rejection is a
+/// separate, breaking change.
+///
+/// It is not answer-preserving, and the exception is a CLASS rather than a row.
+/// A `dense_vector` and its `_chunks` companion end up with no lexical surface
+/// at all, so every lexical leaf naming one answers 0 where the stored-doc scan
+/// used to match the rendered decimal text of the float array. On the 300-doc
+/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`
+/// 300 hits (EVERY document) → 0, `prefix {emb:"0"}` 146 → 0,
+/// `match_phrase_prefix {emb:"0"}` 50 → 0, numeric `term`/`terms` 1 → 0, and the
+/// same for the `_chunks` companion. Note where the boundary is: `prefix`,
+/// `wildcard` and `fuzzy` lose their hits to the WRITE-side exclusion (no term
+/// dictionary left to enumerate), while `multi_match`/`simple_query_string` need
+/// THIS pass — with the postings gone and no lowering they answer every
+/// document. The full three-state table is on `memtable::fts_excluded_fields`.
+///
+/// The Bool fold below additionally moves `_score` for bools whose hit count
+/// does not change, because dropping a dead `should` makes the bool projectable
+/// onto the inverted index and BM25 replaces the stored-doc scorer. Measured and
+/// documented there and in the CHANGELOG; pinned in the (d2c) block of
+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary`.
+///
+/// Three classes of clause are deliberately left alone, because none of them
+/// ever read the term dictionary and all of them are measured unchanged:
+///
+/// * `Exists` — answered from `_source` / doc values (5,000/5,000 docs on
+/// `main` and here), and ES agrees it is not an error on a field with no
+/// terms;
+/// * `Knn` / `SemanticSearch` on the field itself — the whole reason the field
+/// exists. Their `filter` sub-query IS walked;
+/// * the geo leaves — a `geo_distance` on a `dense_vector` is nonsense either
+/// way, but it resolves from `_source` and this change does not touch it.
+///
+/// `Nested` is not walked either, and that is a scope decision rather than an
+/// oversight: field names inside a `nested` query resolve against the ELEMENT,
+/// not the root (see `strip_nested_path_in_query`), so a root-level `emb` in
+/// the set is not the same `emb` the inner clause names. Rewriting through it
+/// could turn a legitimate inner clause into a silent zero. The inner clause
+/// keeps exactly today's behaviour.
+fn lower_lexically_typeless_clauses(
+ q: &QueryNode,
+ typeless: &std::collections::HashSet<String>,
+) -> QueryNode {
+ // One entry of a `fields: [...]` list. `"emb^3"` carries a boost; a
+ // pattern (`"*"`, `"body.*"`) is an expansion, and the expansion sets in
+ // `search_inner` already drop vector fields, so it must stay.
+ let is_typeless_spec = |spec: &String| -> bool {
+ let (name, _boost) = parse_field_boost(spec);
+ !name.contains('*') && typeless.contains(name)
+ };
+ // A `fields: [...]` list with every vector entry removed. `None` means
+ // "leave this node alone": either nothing was dropped, or the list was
+ // already empty, which means "all fields" rather than "no fields".
+ let prune_specs = |fields: &Vec<String>| -> Option<Vec<String>> {
+ if fields.is_empty() || !fields.iter().any(is_typeless_spec) {
+ return None;
+ }
+ Some(
+ fields
+ .iter()
+ .filter(|spec| !is_typeless_spec(spec))
+ .cloned()
+ .collect(),
+ )
+ };
+ let recurse = |c: &QueryNode| lower_lexically_typeless_clauses(c, typeless);
+ match q {
+ // ── Leaves that name exactly one field ────────────────────────────
+ QueryNode::Term { field, .. }
+ | QueryNode::Terms { field, .. }
+ | QueryNode::Range { field, .. }
+ | QueryNode::Prefix { field, .. }
+ | QueryNode::Wildcard { field, .. }
+ | QueryNode::Match { field, .. }
+ | QueryNode::MatchPhrase { field, .. }
+ | QueryNode::MatchPhrasePrefix { field, .. }
+ | QueryNode::Fuzzy { field, .. }
+ | QueryNode::Regexp { field, .. }
+ | QueryNode::Intervals { field, .. }
+ | QueryNode::SpanTerm { field, .. } => {
+ if typeless.contains(field) {
+ QueryNode::MatchNone
+ } else {
+ q.clone()
+ }
+ }
+
+ // ── Multi-field forms: prune the list, empty list → MatchNone ─────
+ // The empty case is the specific hole. `query_node_to_fts` returns
+ // `None` for a field spec it cannot project, and `MultiMatch` is
+ // unconditionally in `is_doc_scan_query`, so a list that named ONLY
+ // vector fields would otherwise reach the stored-doc scan and match
+ // every document instead of none.
+ QueryNode::MultiMatch {
+ fields,
+ query,
+ match_type,
+ operator,
+ analyzer,
+ boost,
+ slop,
+ max_expansions,
+ } => match prune_specs(fields) {
+ None => q.clone(),
+ Some(kept) if kept.is_empty() => QueryNode::MatchNone,
+ Some(kept) => QueryNode::MultiMatch {
+ fields: kept,
+ query: query.clone(),
+ match_type: *match_type,
+ operator: *operator,
+ analyzer: analyzer.clone(),
+ boost: *boost,
+ slop: *slop,
+ max_expansions: *max_expansions,
+ },
+ },
+ QueryNode::SimpleQueryString { query, fields } => match prune_specs(fields) {
+ None => q.clone(),
+ Some(kept) if kept.is_empty() => QueryNode::MatchNone,
+ Some(kept) => QueryNode::SimpleQueryString {
+ query: query.clone(),
+ fields: kept,
+ },
+ },
+ QueryNode::QueryString {
+ default_field: Some(f),
+ ..
+ } if is_typeless_spec(f) => QueryNode::MatchNone,
+
+ // ── Composites: recurse, then fold the MatchNone we just created ──
+ QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match,
+ } => {
+ let was_empty =
+ must.is_empty() && should.is_empty() && must_not.is_empty() && filter.is_empty();
+ let must: Vec<QueryNode> = must.iter().map(recurse).collect();
+ let mut should: Vec<QueryNode> = should.iter().map(recurse).collect();
+ let mut must_not: Vec<QueryNode> = must_not.iter().map(recurse).collect();
+ let filter: Vec<QueryNode> = filter.iter().map(recurse).collect();
+ // The fold is DELIBERATELY minimal — only the shapes this pass can
+ // itself produce, and only where the answer is unambiguous. It is
+ // not `xerj_query::rewriter::rewrite`, which is not on this path
+ // and whose "empty bool → MatchAll" rule fires AFTER it has
+ // stripped `MatchNone` out of `should`, i.e. turns an unsatisfiable
+ // bool into a match-everything one.
+ //
+ // Without a fold the lowering is CORRECT but not fast: a
+ // `simple_query_string {"fields":["emb"]}` lowers to
+ // `Bool{should:[Match{emb}]}` (`parser::make_simple_query_node`),
+ // which becomes `Bool{should:[MatchNone]}` — still a `Bool`, still
+ // in `is_doc_scan_query`, so still a full stored-doc scan that
+ // evaluates to nothing.
+ //
+ // The per-occur rule below is `BooleanQuery.rewrite`'s, clause for
+ // clause: a `MatchNoDocsQuery` in SHOULD or MUST_NOT "can be safely
+ // ignored" and one in MUST or FILTER is returned as the whole query
+ // (`lucene/core/src/java/org/apache/lucene/search/BooleanQuery.java:323`
+ // and `:327`). Lucene's neighbouring "pure negative BooleanQuery →
+ // MatchNoDocsQuery" rule (`:276`) is deliberately NOT adopted: that
+ // is Lucene's bool, not ES's. XERJ follows ES, where a bool with
+ // only `must_not` matches every non-excluded document — exactly what
+ // `doc_matches_query`'s Bool arm implements.
+
+ // ── MUST / FILTER: an unsatisfiable REQUIRED clause sinks the bool.
+ if must.iter().any(QueryNode::is_match_none)
+ || filter.iter().any(QueryNode::is_match_none)
+ {
+ return QueryNode::MatchNone;
+ }
+
+ // ── MUST_NOT: a clause that matches nothing EXCLUDES nothing, so
+ // dropping it is exact for every shape — `doc_matches_query` only
+ // ever asks `must_not.iter().any(...)`, and `MatchNone` is never
+ // `any`. Dropping it is also what closes the doc-scan fallback this
+ // pass would otherwise open: `query_node_to_fts`'s Bool arm `?`s on
+ // EVERY `must_not` child and its `MatchNone` arm is `None`, so one
+ // surviving `MatchNone` un-projects the whole bool and hands a
+ // query that used to run on the inverted index to the stored-doc
+ // scan. `{"bool":{"must_not":[{"term":{"emb":…}}]}}` is the shape.
+ must_not.retain(|c| !c.is_match_none());
+
+ // ── SHOULD. Both folds below turn on whether `should` is REQUIRED,
+ // and the rule has to be the one `doc_matches_query` actually
+ // implements, not the one ES documents in prose:
+ //
+ // * `None` → required iff `must` and `filter` are both empty.
+ // `must_not` does NOT relax it.
+ // * `Fixed(0)` → optional.
+ // * `Fixed(n)` → required.
+ // * `Percentage(p)` → required for EVERY p including 0, because
+ // the count is `floor(len × p/100).max(1)`. The
+ // previous revision of this pass treated
+ // `Percentage(0)` as "optional"; that is only ever
+ // a missed fold, never a wrong answer, but it is
+ // still not what the evaluator does.
+ // * `Field` / `Script` (terms_set) → the required count is read
+ // PER DOCUMENT and 0 is a legal value, so neither
+ // fold is sound. Leave the node exactly as recursed.
+ let should_required = match minimum_should_match {
+ None => must.is_empty() && filter.is_empty(),
+ Some(MinShouldMatch::Fixed(0)) => false,
+ Some(MinShouldMatch::Fixed(_)) | Some(MinShouldMatch::Percentage(_)) => true,
+ Some(MinShouldMatch::Field(_)) | Some(MinShouldMatch::Script { .. }) => {
+ return QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match: minimum_should_match.clone(),
+ };
+ }
+ };
+ // Every optional clause unsatisfiable, and at least one required →
+ // the bool matches nothing, whatever `must_not` says. `must_not`
+ // is NOT part of this test: it can only remove further documents,
+ // never add one back. (The previous revision required
+ // `must_not.is_empty()` here and then fell through to the retain
+ // below, which turned `{"bool":{"should":[<vector clause>],
+ // "must_not":[X]}}` from "0 hits" into "every document except X".)
+ if should_required && !should.is_empty() && should.iter().all(QueryNode::is_match_none)
+ {
+ return QueryNode::MatchNone;
+ }
+ // Otherwise drop the dead optional clauses — but only when the
+ // required count does not move with the clause COUNT. It does not
+ // for `None` (1 or 0) and for `Fixed(n)` (n), which is why Lucene
+ // can drop unconditionally at `BooleanQuery.java:323`. It DOES for
+ // `Percentage(p)`, where shrinking `should` shrinks the threshold
+ // and can make an unsatisfiable bool satisfiable.
+ //
+ // THIS `retain` IS THE ONE WIRE-VISIBLE SCORE CHANGE in the pass.
+ // It does not move a hit COUNT — `doc_matches_query` never counts a
+ // `MatchNone` as a satisfied `should` — but the shrunken bool is
+ // projectable through `query_node_to_fts_with_keyword_fields` where
+ // the original was not, so BM25 scores it instead of
+ // `score_query_against_doc`. On the 300-doc fixture
+ // `bool{should:[term emb, term cat:"even"]}` keeps its 150 hits and
+ // goes `_score` 0.008402659 → 0.6931471 (exactly what
+ // `{"term":{"cat":"even"}}` scores on its own, which is what the
+ // bool now reduces to) and `max_score` 1.6931472 → 0.6931471. Both
+ // are in the CHANGELOG and pinned in the (d2c) block of
+ // `user_mapped_dense_vector_builds_no_fts_term_dictionary`.
+ let count_free_msm =
+ matches!(minimum_should_match, None | Some(MinShouldMatch::Fixed(_)));
+ if count_free_msm && should.iter().any(QueryNode::is_match_none) {
+ should.retain(|c| !c.is_match_none());
+ }
+ // Dropping the last `must_not` (or the last `should`) can empty a
+ // bool that arrived with clauses. An all-empty bool matches every
+ // document and scores 1.0 in `doc_matches_query` /
+ // `score_query_against_doc` — bit-identical to `MatchAll`, which
+ // takes the match-all fast path instead of `is_doc_scan_query`'s
+ // `Bool` arm. Lucene folds the same shape, to its own bool
+ // semantics, at `BooleanQuery.java:271`.
+ if !was_empty
+ && must.is_empty()
+ && should.is_empty()
+ && must_not.is_empty()
+ && filter.is_empty()
+ {
+ return QueryNode::MatchAll;
+ }
+ QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match: minimum_should_match.clone(),
+ }
+ }
+ // `constant_score` and a top-level `boost` wrapper both BUBBLE UP an
+ // unsatisfiable inner query rather than wrapping it — Lucene does the
+ // same, in the same words, at
+ // `lucene/core/src/java/org/apache/lucene/search/ConstantScoreQuery.java:56`
+ // ("bubble up MatchNoDocsQuery") and `BoostQuery.java:87`. It is exact
+ // — neither wrapper can make a non-matching inner query match — and it
+ // matters here because `Constant` is the ONLY wrapper with an arm in
+ // `query_node_to_fts_with_keyword_fields`: left as
+ // `Constant{MatchNone}` it projects to `None`, and `Constant` is in
+ // `is_doc_scan_query`, so `{"constant_score":{"filter":{"term":
+ // {"emb":…}}}}` would trade an inverted-index lookup for a full
+ // stored-doc scan.
+ QueryNode::Constant { score, query } => match recurse(query) {
+ QueryNode::MatchNone => QueryNode::MatchNone,
+ lowered => QueryNode::Constant {
+ score: *score,
+ query: Box::new(lowered),
+ },
+ },
+ QueryNode::Boosted { boost, query } => match recurse(query) {
+ QueryNode::MatchNone => QueryNode::MatchNone,
+ lowered => QueryNode::Boosted {
+ boost: *boost,
+ query: Box::new(lowered),
+ },
+ },
+ QueryNode::Named { name, query } => QueryNode::Named {
+ name: name.clone(),
+ query: Box::new(recurse(query)),
+ },
+ QueryNode::FunctionScore {
+ query,
+ functions,
+ score_mode,
+ boost_mode,
+ max_boost,
+ } => QueryNode::FunctionScore {
+ query: Box::new(recurse(query)),
+ functions: functions.clone(),
+ score_mode: *score_mode,
+ boost_mode: *boost_mode,
+ max_boost: *max_boost,
+ },
+ QueryNode::Pinned { ids, organic } => QueryNode::Pinned {
+ ids: ids.clone(),
+ organic: Box::new(recurse(organic)),
+ },
+ QueryNode::Boosting {
+ positive,
+ negative,
+ negative_boost,
+ } => QueryNode::Boosting {
+ positive: Box::new(recurse(positive)),
+ negative: Box::new(recurse(negative)),
+ negative_boost: *negative_boost,
+ },
+ QueryNode::DisMax {
+ queries,
+ tie_breaker,
+ } => QueryNode::DisMax {
+ queries: queries.iter().map(recurse).collect(),
+ tie_breaker: *tie_breaker,
+ },
+ QueryNode::Hybrid { queries, fusion } => QueryNode::Hybrid {
+ queries: queries
+ .iter()
+ .map(|wq| xerj_query::ast::WeightedQuery {
+ query: recurse(&wq.query),
+ weight: wq.weight,
+ })
+ .collect(),
+ fusion: fusion.clone(),
+ },
+ QueryNode::Knn {
+ field,
+ vector,
+ k,
+ num_candidates,
+ filter,
+ boost,
+ similarity,
+ } => QueryNode::Knn {
+ field: field.clone(),
+ vector: vector.clone(),
+ k: *k,
+ num_candidates: *num_candidates,
+ filter: filter.as_ref().map(|f| Box::new(recurse(f))),
+ boost: *boost,
+ similarity: *similarity,
+ },
+ QueryNode::SemanticSearch {
+ field,
+ text,
+ k,
+ filter,
+ boost,
+ } => QueryNode::SemanticSearch {
+ field: field.clone(),
+ text: text.clone(),
+ k: *k,
+ filter: filter.as_ref().map(|f| Box::new(recurse(f))),
+ boost: *boost,
+ },
+ QueryNode::SpanNear {
+ clauses,
+ slop,
+ in_order,
+ } => QueryNode::SpanNear {
+ clauses: clauses.iter().map(recurse).collect(),
+ slop: *slop,
+ in_order: *in_order,
+ },
+ QueryNode::SpanOr { clauses } => QueryNode::SpanOr {
+ clauses: clauses.iter().map(recurse).collect(),
+ },
+ QueryNode::SpanNot { include, exclude } => QueryNode::SpanNot {
+ include: Box::new(recurse(include)),
+ exclude: Box::new(recurse(exclude)),
+ },
+ QueryNode::SpanFirst { match_query, end } => QueryNode::SpanFirst {
+ match_query: Box::new(recurse(match_query)),
+ end: *end,
+ },
+ QueryNode::SpanContaining { little, big } => QueryNode::SpanContaining {
+ little: Box::new(recurse(little)),
+ big: Box::new(recurse(big)),
+ },
+ QueryNode::SpanWithin { little, big } => QueryNode::SpanWithin {
+ little: Box::new(recurse(little)),
+ big: Box::new(recurse(big)),
+ },
+
+ // Everything else names no lexical target on a vector field:
+ // MatchAll/MatchNone/Ids/Script, `Exists`, the geo leaves, a
+ // field-less `query_string`, `percolate` (the field holds stored
+ // queries, not data), `more_like_this` (with `fields` the parser
+ // lowers it to a `bool.should` of `match`, which the leaf arms above
+ // catch), and `Nested` (see the doc comment).
+ other => other.clone(),
+ }
+}
+
+/// Unit coverage for [`lower_lexically_typeless_clauses`] — #328.
+///
+/// The integration tests in `tests/integration.rs` pin the ANSWERS this
+/// rewriter has to produce end to end; these pin the SHAPE it produces, which
+/// is what decides whether the query then runs on the inverted index or on the
+/// stored-doc scan. The distinction is not cosmetic: `Bool{must_not:[MatchNone]}`
+/// and `Bool{}` give the same hits, but only the second projects through
+/// `query_node_to_fts_with_keyword_fields` — the first `?`s out on its
+/// `must_not` child and falls to a full corpus walk.
+#[cfg(test)]
+mod lexically_typeless_lowering_tests {
+ use super::*;
+
+ fn typeless(names: &[&str]) -> std::collections::HashSet<String> {
+ names.iter().map(|s| s.to_string()).collect()
+ }
+
+ fn term(field: &str, value: &str) -> QueryNode {
+ QueryNode::Term {
+ field: field.into(),
+ value: Value::String(value.into()),
+ boost: None,
+ }
+ }
+
+ fn bool_of(
+ must: Vec<QueryNode>,
+ should: Vec<QueryNode>,
+ must_not: Vec<QueryNode>,
+ filter: Vec<QueryNode>,
+ msm: Option<MinShouldMatch>,
+ ) -> QueryNode {
+ QueryNode::Bool {
+ must,
+ should,
+ must_not,
+ filter,
+ minimum_should_match: msm,
+ }
+ }
+
+ fn lower(q: &QueryNode) -> QueryNode {
+ lower_lexically_typeless_clauses(q, &typeless(&["emb", "emb_chunks"]))
+ }
+
+ // ── Leaves ───────────────────────────────────────────────────────────
+
+ #[test]
+ fn single_field_leaf_on_a_vector_field_lowers_to_match_none() {
+ for leaf in [
+ term("emb", "0.5"),
+ QueryNode::Terms {
+ field: "emb".into(),
+ values: vec![Value::String("0.5".into())],
+ boost: None,
+ },
+ QueryNode::Range {
+ field: "emb".into(),
+ gte: Some(Value::from(-2.0)),
+ gt: None,
+ lte: Some(Value::from(2.0)),
+ lt: None,
+ boost: None,
+ },
+ QueryNode::Prefix {
+ field: "emb".into(),
+ value: "0".into(),
+ boost: None,
+ constant_score: true,
+ },
+ QueryNode::Wildcard {
+ field: "emb".into(),
+ value: "0*".into(),
+ boost: None,
+ constant_score: true,
+ },
+ QueryNode::Match {
+ field: "emb".into(),
+ query: "0".into(),
+ operator: xerj_query::ast::BoolOperator::Or,
+ analyzer: None,
+ boost: None,
+ minimum_should_match: None,
+ },
+ term("emb_chunks", "0.5"),
+ ] {
+ assert!(
+ lower(&leaf).is_match_none(),
+ "leaf naming a lexically typeless field must lower: {leaf:?}"
+ );
+ }
+ }
+
+ #[test]
+ fn leaf_on_a_lexical_field_is_untouched() {
+ let q = term("cat", "even");
+ assert_eq!(lower(&q), q, "a normal field keeps its clause");
+ }
+
+ /// `Exists` is answered from `_source` / doc values, never from the term
+ /// dictionary, so it must survive — this is the clause that keeps
+ /// `{"exists":{"field":"emb"}}` answering 5,000/5,000 after the postings
+ /// are gone.
+ #[test]
+ fn exists_and_knn_on_a_vector_field_survive() {
+ let exists = QueryNode::Exists {
+ field: "emb".into(),
+ };
+ assert_eq!(lower(&exists), exists);
+
+ let knn = QueryNode::Knn {
+ field: "emb".into(),
+ vector: vec![0.0, 1.0],
+ k: 3,
+ num_candidates: None,
+ filter: Some(Box::new(term("cat", "even"))),
+ boost: None,
+ similarity: None,
+ };
+ match lower(&knn) {
+ QueryNode::Knn { field, filter, .. } => {
+ assert_eq!(field, "emb", "the kNN target is the whole point");
+ assert_eq!(
+ *filter.expect("filter kept"),
+ term("cat", "even"),
+ "a lexical filter under kNN is walked but not lowered"
+ );
+ }
+ other => panic!("kNN must not be lowered: {other:?}"),
+ }
+ }
+
+ /// A kNN filter that names the vector field itself IS lowered — the filter
+ /// is an ordinary lexical sub-query, unlike the `field` it hangs off.
+ #[test]
+ fn knn_filter_naming_the_vector_field_is_lowered() {
+ let knn = QueryNode::Knn {
+ field: "emb".into(),
+ vector: vec![0.0, 1.0],
+ k: 3,
+ num_candidates: None,
+ filter: Some(Box::new(term("emb", "0.5"))),
+ boost: None,
+ similarity: None,
+ };
+ match lower(&knn) {
+ QueryNode::Knn { filter, .. } => {
+ assert!(filter.expect("filter kept").is_match_none());
+ }
+ other => panic!("expected knn, got {other:?}"),
+ }
+ }
+
+ // ── Multi-field forms ────────────────────────────────────────────────
+
+ #[test]
+ fn multi_match_keeps_its_lexical_fields_and_drops_the_vector_ones() {
+ let q = QueryNode::MultiMatch {
+ fields: vec!["emb".into(), "body^3".into(), "emb_chunks".into()],
+ query: "0".into(),
+ match_type: xerj_query::ast::MultiMatchType::BestFields,
+ operator: Some(xerj_query::ast::BoolOperator::Or),
+ analyzer: None,
+ boost: None,
+ slop: 0,
+ max_expansions: 50,
+ };
+ match lower(&q) {
+ QueryNode::MultiMatch { fields, .. } => {
+ assert_eq!(fields, vec!["body^3".to_string()], "boost suffix preserved");
+ }
+ other => panic!("expected a pruned multi_match, got {other:?}"),
+ }
+ }
+
+ /// The regression the PR body originally mis-reported as "byte-identical":
+ /// a `fields` list naming ONLY vector fields must become `MatchNone`, not
+ /// an empty-`fields` multi_match (which means "all fields" downstream) and
+ /// not a stored-doc scan (which answers every document, because the scan
+ /// renders the float array to text and every component contains a `0`).
+ #[test]
+ fn multi_match_over_only_vector_fields_becomes_match_none() {
+ let q = QueryNode::MultiMatch {
+ fields: vec!["emb".into(), "emb_chunks^2".into()],
+ query: "0".into(),
+ match_type: xerj_query::ast::MultiMatchType::BestFields,
+ operator: Some(xerj_query::ast::BoolOperator::Or),
+ analyzer: None,
+ boost: None,
+ slop: 0,
+ max_expansions: 50,
+ };
+ assert!(lower(&q).is_match_none());
+ }
+
+ /// An EMPTY `fields` list means "every field", not "no field" — pruning
+ /// cannot empty it, so the node must be left exactly as it arrived.
+ #[test]
+ fn multi_match_with_no_fields_is_left_alone() {
+ let q = QueryNode::MultiMatch {
+ fields: vec![],
+ query: "0".into(),
+ match_type: xerj_query::ast::MultiMatchType::BestFields,
+ operator: Some(xerj_query::ast::BoolOperator::Or),
+ analyzer: None,
+ boost: None,
+ slop: 0,
+ max_expansions: 50,
+ };
+ assert_eq!(lower(&q), q);
+ }
+
+ /// A pattern entry is an EXPANSION, and the expansion sets in
+ /// `search_inner` already drop vector fields, so it must not be pruned
+ /// here — pruning `"*"` would silently turn "search everything" into
+ /// "search nothing".
+ #[test]
+ fn a_wildcard_field_spec_is_not_pruned() {
+ let q = QueryNode::SimpleQueryString {
+ query: "0".into(),
+ fields: vec!["*".into(), "emb".into()],
+ };
+ match lower(&q) {
+ QueryNode::SimpleQueryString { fields, .. } => {
+ assert_eq!(fields, vec!["*".to_string()]);
+ }
+ other => panic!("expected simple_query_string, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn query_string_with_a_vector_default_field_becomes_match_none() {
+ let q = QueryNode::QueryString {
+ query: "0".into(),
+ default_field: Some("emb".into()),
+ default_operator: None,
+ boost: None,
+ };
+ assert!(lower(&q).is_match_none());
+ }
+
+ // ── Bool folding ─────────────────────────────────────────────────────
+
+ #[test]
+ fn match_none_in_must_or_filter_sinks_the_whole_bool() {
+ assert!(lower(&bool_of(
+ vec![term("emb", "0.5"), term("cat", "even")],
+ vec![],
+ vec![],
+ vec![],
+ None
+ ))
+ .is_match_none());
+ assert!(lower(&bool_of(
+ vec![],
+ vec![],
+ vec![],
+ vec![term("emb", "0.5")],
+ None
+ ))
+ .is_match_none());
+ }
+
+ /// `must_not` is the arm the first cut of this pass left open. A clause
+ /// that matches nothing excludes nothing, so it must be DROPPED — leaving
+ /// a `MatchNone` there keeps the bool un-projectable and sends a query
+ /// that used to run on the inverted index to the stored-doc scan.
+ #[test]
+ fn match_none_in_must_not_is_dropped_not_kept() {
+ match lower(&bool_of(
+ vec![term("cat", "even")],
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![],
+ None,
+ )) {
+ QueryNode::Bool { must, must_not, .. } => {
+ assert_eq!(must, vec![term("cat", "even")]);
+ assert!(must_not.is_empty(), "a dead exclusion is not an exclusion");
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ }
+
+ /// The bool empties out completely: `{"bool":{"must_not":[{"term":
+ /// {"emb":…}}]}}` excludes nothing and therefore matches everything.
+ /// `MatchAll` and an all-empty `Bool` agree on hits AND on `_score` (1.0),
+ /// but only `MatchAll` skips `is_doc_scan_query`.
+ #[test]
+ fn a_bool_emptied_by_the_fold_becomes_match_all() {
+ assert_eq!(
+ lower(&bool_of(
+ vec![],
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![],
+ None
+ )),
+ QueryNode::MatchAll
+ );
+ }
+
+ /// …but a bool that arrived empty is NOT rewritten: this pass only ever
+ /// reacts to something it itself lowered.
+ #[test]
+ fn a_bool_that_arrived_empty_is_untouched() {
+ let q = bool_of(vec![], vec![], vec![], vec![], None);
+ assert_eq!(lower(&q), q);
+ }
+
+ /// THE CORRECTNESS CASE. `should` is required whenever `must` and `filter`
+ /// are empty — `must_not` does not relax it (`doc_matches_query`'s Bool arm
+ /// tests `must.is_empty() && filter.is_empty()`). The first cut of this
+ /// pass required `must_not.is_empty()` before folding and then fell through
+ /// to dropping the dead `should` clause, which turned this query from
+ /// "0 hits" into "every document except the excluded ones".
+ #[test]
+ fn all_should_unsatisfiable_sinks_the_bool_even_with_a_must_not() {
+ assert!(lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![term("cat", "even")],
+ vec![],
+ None,
+ ))
+ .is_match_none());
+ }
+
+ #[test]
+ fn all_should_unsatisfiable_sinks_a_should_only_bool() {
+ assert!(lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![],
+ vec![],
+ None
+ ))
+ .is_match_none());
+ }
+
+ /// With a `must` present, `should` is scoring-only (`min = 0`), so a dead
+ /// optional clause is dropped and the bool survives.
+ #[test]
+ fn a_dead_should_beside_a_must_is_dropped_and_the_bool_survives() {
+ match lower(&bool_of(
+ vec![term("cat", "even")],
+ vec![term("emb", "0.5"), term("body", "liquidity")],
+ vec![],
+ vec![],
+ None,
+ )) {
+ QueryNode::Bool { must, should, .. } => {
+ assert_eq!(must, vec![term("cat", "even")]);
+ assert_eq!(should, vec![term("body", "liquidity")]);
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ }
+
+ /// An explicit numeric `minimum_should_match` does not move with the
+ /// clause count, so the dead clause is dropped — and the survivor count
+ /// can now be below the threshold, which is exactly the original meaning.
+ #[test]
+ fn fixed_minimum_should_match_still_drops_the_dead_clause() {
+ match lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5"), term("body", "liquidity")],
+ vec![],
+ vec![],
+ Some(MinShouldMatch::Fixed(2)),
+ )) {
+ QueryNode::Bool {
+ should,
+ minimum_should_match,
+ ..
+ } => {
+ assert_eq!(should, vec![term("body", "liquidity")]);
+ assert_eq!(minimum_should_match, Some(MinShouldMatch::Fixed(2)));
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ }
+
+ /// `Fixed(0)` makes `should` optional, so an all-dead `should` list does
+ /// NOT sink the bool.
+ #[test]
+ fn fixed_zero_minimum_should_match_keeps_the_bool_alive() {
+ match lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![term("cat", "even")],
+ vec![],
+ Some(MinShouldMatch::Fixed(0)),
+ )) {
+ QueryNode::Bool {
+ should, must_not, ..
+ } => {
+ assert!(should.is_empty(), "dead optional clause dropped");
+ assert_eq!(must_not, vec![term("cat", "even")]);
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ }
+
+ /// A PERCENTAGE threshold moves with `should.len()`
+ /// (`floor(len × pct/100).max(1)`), so the list must not be shrunk. It is
+ /// still required at every percentage — including 0, because of the
+ /// `.max(1)` — so an all-dead list sinks the bool.
+ #[test]
+ fn percentage_minimum_should_match_does_not_shrink_the_clause_list() {
+ match lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5"), term("body", "a"), term("body", "b")],
+ vec![],
+ vec![],
+ Some(MinShouldMatch::Percentage(67)),
+ )) {
+ QueryNode::Bool { should, .. } => {
+ assert_eq!(should.len(), 3, "threshold rides on the clause COUNT");
+ assert!(should[0].is_match_none());
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ assert!(lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![],
+ vec![],
+ Some(MinShouldMatch::Percentage(0)),
+ ))
+ .is_match_none());
+ }
+
+ /// `terms_set`: the required count is read PER DOCUMENT and 0 is legal, so
+ /// neither fold is sound and the node is left exactly as recursed.
+ #[test]
+ fn per_document_minimum_should_match_blocks_both_folds() {
+ match lower(&bool_of(
+ vec![],
+ vec![term("emb", "0.5")],
+ vec![],
+ vec![],
+ Some(MinShouldMatch::Field("required".into())),
+ )) {
+ QueryNode::Bool {
+ should,
+ minimum_should_match,
+ ..
+ } => {
+ assert_eq!(should.len(), 1, "clause count preserved");
+ assert!(should[0].is_match_none(), "leaf still lowered");
+ assert_eq!(
+ minimum_should_match,
+ Some(MinShouldMatch::Field("required".into()))
+ );
+ }
+ other => panic!("expected a bool, got {other:?}"),
+ }
+ }
+
+ // ── Wrappers ─────────────────────────────────────────────────────────
+
+ /// `Constant` is the only wrapper with an arm in
+ /// `query_node_to_fts_with_keyword_fields`, so it is the only one where
+ /// failing to bubble up costs an inverted-index lookup.
+ #[test]
+ fn constant_score_and_boost_bubble_up_match_none() {
+ assert!(lower(&QueryNode::Constant {
+ score: 2.0,
+ query: Box::new(term("emb", "0.5")),
+ })
+ .is_match_none());
+ assert!(lower(&QueryNode::Boosted {
+ boost: 2.0,
+ query: Box::new(term("emb", "0.5")),
+ })
+ .is_match_none());
+ }
+
+ #[test]
+ fn constant_score_over_a_lexical_clause_is_untouched() {
+ let q = QueryNode::Constant {
+ score: 2.0,
+ query: Box::new(term("cat", "even")),
+ };
+ assert_eq!(lower(&q), q);
+ }
+
+ /// Field names inside a `nested` query resolve against the ELEMENT, not
+ /// the root, so a root-level `emb` in the set is not the inner `emb`.
+ /// Deliberately out of scope — pinned so the decision is a decision.
+ #[test]
+ fn nested_is_not_walked() {
+ let q = QueryNode::Nested {
+ path: "chunks".into(),
+ query: Box::new(term("emb", "0.5")),
+ score_mode: None,
+ };
+ assert_eq!(lower(&q), q);
+ }
+
+ /// The whole pass is a no-op on a schema with no vector fields — the
+ /// caller skips it entirely on an empty set, and this pins that the
+ /// function agrees.
+ #[test]
+ fn an_empty_typeless_set_changes_nothing() {
+ let q = bool_of(
+ vec![term("emb", "0.5")],
+ vec![term("body", "x")],
+ vec![term("cat", "even")],
+ vec![],
+ None,
+ );
+ assert_eq!(
+ lower_lexically_typeless_clauses(&q, &std::collections::HashSet::new()),
+ q
+ );
+ }
+}
+
/// Rewrite a query by resolving any field aliases to their canonical field names.
///
/// Traverses the query tree, replacing alias field names with their target paths.
diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rs
index c1b06d301..c44936c8f 100644
--- a/engine/crates/xerj-engine/src/memtable.rs
+++ b/engine/crates/xerj-engine/src/memtable.rs
@@ -3679,20 +3679,38 @@ fn collect_text_fields(
) {
if let Value::Object(obj) = v {
for (k, val) in obj {
- if prefix.is_empty() && excluded.contains(k) {
- continue;
- }
let path = if prefix.is_empty() {
k.clone()
} else {
format!("{}.{}", prefix, k)
};
+ // Matched on the FULL dotted path, not just the root key — #328.
+ // Every pre-#328 member of the set is a root-level name, for which
+ // `path == k` and this is the check that was here before; a
+ // `dense_vector` nested under an object mapping is the first member
+ // that can arrive as `passages.vec`.
+ if excluded.contains(&path) {
+ continue;
+ }
match val {
Value::Object(_) => {
// Root-level JSON-blob for flattened-style
// whole-object queries.
+ //
+ // Built from a PRUNED copy of the object (#328): the blob
+ // is where a nested `dense_vector` actually landed. The
+ // recursion below skips `passages.vec`, but this line
+ // serialised the whole `passages` object — vector included
+ // — into one text field, so excluding the leaf alone moved
+ // no bytes at all. Pinned by
+ // `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`,
+ // which reports `<seg>.passages.fst` and requires it under
+ // 4 KiB — the fixture's 19,200 vector components cannot fit
+ // in that. Both mapping shapes are covered there: the
+ // dotted top-level name `"passages.vec"` and a `vec`
+ // sub-mapping under a `passages` object.
if prefix.is_empty() {
- let t = extract_text_value(val);
+ let t = extract_text_value_excluding(val, &path, excluded);
if !t.is_empty() && !out.contains_key(&path) {
out.insert(path.clone(), t);
}
@@ -3806,11 +3824,13 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
/// Every field that must be kept OUT of the full-text inverted index.
///
-/// Two disjoint reasons, unioned into one set so flush, merge, the memtable
+/// Three disjoint reasons, unioned into one set so flush, merge, the memtable
/// analyser and `build_fts_field_configs` cannot disagree:
///
/// * semantic-derived vector fields (see above) — engine-internal payloads
/// that were never user text;
+/// * fields the user MAPPED as `dense_vector` (`FieldType::Vector`) — #328.
+/// See "Why a declared vector has no lexical form" below;
/// * fields whose mapping declares `"index": false` — #204's last open
/// accepted-and-ignored instance. The option was echoed by `GET _mapping`
/// and then ignored, so the field kept a full inverted index and stayed
@@ -3844,8 +3864,166 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
/// `unsearchable_query_field` in `index.rs` rejects every query naming it
/// before execution — so no fallback of any strength ever runs, and the
/// postings are pure waste whatever the type.
+///
+/// ## Why a declared vector has no lexical form (#328)
+///
+/// `#12` removed the term dictionary for the vectors XERJ *generates* from a
+/// semantic mapping. A field the user maps as `dense_vector` took a different
+/// route through the same code and kept its own: kNN needs the HNSW graph, so
+/// the field is `"index": true`, so the `index: false` arm above never fires.
+/// The field then flowed through `extract_fts_fields_excluding` into
+/// `extract_field_text`, whose `Value::Array` arm joins the elements with
+/// spaces — turning a 128-dim vector into one enormous decimal-string term per
+/// document. kNN is served by `hnsw/graph.bin`; `exists` is resolved from
+/// `_source` / doc values; `_field_caps` and highlighting never open the
+/// field's FST — so every one of those bytes is unreadable by construction.
+///
+/// Lucene draws the type line in exactly the same place and gets the same
+/// consequence for free: `KnnFloatVectorField.createFieldType` sets vector
+/// attributes and then `freeze()`s, and sets nothing else
+/// (`lucene/core/src/java/org/apache/lucene/document/KnnFloatVectorField.java:70`;
+/// `KnnByteVectorField.java:82` is identical), so `IndexOptions` stays `NONE`
+/// (`lucene/core/src/java/org/apache/lucene/index/FieldInfo.java:565`) and a
+/// term dictionary for the field cannot exist. `ColumnValidation` enumerates
+/// "vectors" as an indexing feature DISJOINT from "index options"
+/// (`lucene/core/src/java/org/apache/lucene/document/column/ColumnValidation.java:104`)
+/// — the same type-level separation this is.
+///
+/// Measured — one harness, one corpus, built twice and diffed file by file
+/// after `force_merge(1)`. 5,000 docs × 128 dim: `text` + `keyword` + `long` +
+/// a `dense_vector` + its `_chunks` companion (the corpus carries the companion
+/// because that is the shape RFC #148 reports). Every byte below is a
+/// `std::fs::metadata().len()` sum over the data dir, not an estimate:
+///
+/// | file, 5,000 × 128 | `main` | this branch |
+/// |---|---:|---:|
+/// | `<seg>.emb.fst` | 13,256,659 B | 0 |
+/// | `<seg>.emb.post` | 10,834 B | 0 |
+/// | `<seg>.emb.norms` | 48 B | 0 |
+/// | `<seg>.emb_chunks.fst` | 26,518,005 B | 0 |
+/// | `<seg>.emb_chunks.post` | 10,834 B | 0 |
+/// | `<seg>.body.fst` | 278 B | 278 B |
+/// | `<seg>.cat.fst` | 53 B | 53 B |
+/// | `<seg>.n.fst` | 209 B | 209 B |
+/// | **durable index** (all files bar `.wal`) | **54,068,549 B** | **14,251,975 B** (−73.6%, 3.79×) |
+///
+/// Two observations the table is here to make. First, the term dictionary of a
+/// vector-carrying index essentially *is* the vector field: the three lexical
+/// `.fst` files total 540 B against 39.8 MB of vector postings. Second — and
+/// this is why the exclusion cannot stop at the base name — the `_chunks`
+/// companion is the BIGGER half. At 26,518,005 B it is twice `emb.fst` and 49%
+/// of `main`'s whole index on its own, so a base-name-only exclusion would
+/// leave more behind than it removes.
+///
+/// The row is the DURABLE index — every file in the data directory except
+/// `.wal` — because the write-ahead log's tail is reclaimed asynchronously and
+/// the whole-directory total is therefore not reproducible: eight runs of the
+/// 300-doc regression test put it anywhere between 858,147 B and 1,105,001 B
+/// while the durable bytes moved by 230 B. `.seg` (the stored documents, 11.2 MB
+/// here) is counted on both sides and is why the ratio is 3.79× rather than the
+/// 39.8 MB the term dictionaries cost outright.
+///
+/// The 300-doc × 128-dim fixture the regression tests actually run is the same
+/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,
+/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,
+/// durable index 3,249,323 B → 857,843–858,071 B over eight runs. That test
+/// prints both totals and asserts a ceiling on the durable one, so this figure
+/// is enforced rather than merely published.
+///
+/// The exclusion is *by declared type*, not by value shape: an unmapped
+/// numeric array stays lexical exactly as before (the same line HNSW draws in
+/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a
+/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT excluded:
+/// its declared payload is chunk *text* plus a vector, and only the vector
+/// half is droppable — that needs a value-shape-aware rule, not a type rule.
+///
+/// Removing the postings is only half of it, and the other half is NOT the
+/// `*`-expansion filter. Every other member of this set is protected from the
+/// stored-doc scan by `unsearchable_query_field`, which rejects the query
+/// before execution; a `dense_vector` is not, because it IS indexed. So two
+/// things happen on the query side:
+///
+/// * `text_fields` / `exact_fields` in `search_inner` drop vector fields, so a
+/// `*` expansion stops projecting a clause onto them;
+/// * and `index::lower_lexically_typeless_clauses` lowers any clause that
+/// NAMES one to `match_none` at plan time. The expansion filter alone does
+/// nothing for `{"term":{"emb":…}}` or `{"multi_match":{"fields":["emb"]}}`,
+/// which name the field outright — those fall through the per-segment
+/// `fts_has_field` gate onto the stored-doc scan, whose `Term` arm matches
+/// ANY ELEMENT of a JSON array (`json_values_equal`) and whose `match` arm
+/// splits on non-alphanumerics.
+///
+/// Measured on the corpus above, post-`force_merge`, three source states —
+/// `main` / write-side exclusion only (postings gone, no plan-time lowering) /
+/// this branch. Timings are the FIRST call for each shape, before the result
+/// cache has seen it; a repeat of any row here is ~0.002 ms on all three and
+/// measures the cache, not the plan:
+///
+/// | query | `main` | postings gone, no lowering | this branch |
+/// |---|---|---|---|
+/// | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |
+/// | `term {emb: <component>}` numeric | **1 hit**, 87.6 ms | **1 hit**, 187.6 ms | **0 hits**, 0.011 ms |
+/// | `terms {emb: [<component>]}` numeric | **1 hit**, 85.6 ms | **1 hit**, 112.0 ms | **0 hits**, 0.012 ms |
+/// | `prefix {emb: "0"}` | **2,500 hits**, 231.8 ms | 0 hits, 95.0 ms | **0 hits**, 0.008 ms |
+/// | `wildcard {emb: "0*"}` | **5,000 hits**, 316.6 ms | 0 hits, 95.1 ms | **0 hits**, 0.008 ms |
+/// | `fuzzy {emb: "0"~2}` | **5,000 hits**, 798.5 ms | 0 hits, 97.7 ms | **0 hits**, 0.011 ms |
+/// | `match_phrase_prefix {emb: "0"}` | **50 hits**, 84.6 ms | 0 hits, 89.3 ms | **0 hits**, 0.009 ms |
+/// | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |
+/// | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |
+/// | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |
+/// | `term {emb_chunks: "<component>"}` | 0 hits, 86.0 ms | 0 hits, 116.6 ms | 0 hits, 0.008 ms |
+/// | `wildcard {emb_chunks: "0*"}` | **5,000 hits**, 626.9 ms | 0 hits, 89.9 ms | **0 hits**, 0.007 ms |
+/// | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |
+/// | `bool{should:[term emb], must_not:[term cat]}` | 0 hits, 81.4 ms | 0 hits, 92.4 ms | 0 hits, 0.011 ms |
+/// | `bool{must_not:[term emb]}` | 5,000 hits, 84.9 ms | 5,000 hits, 98.9 ms | 5,000 hits, 2.31 ms |
+/// | `constant_score{filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |
+/// | `exists {field: emb}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |
+/// | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |
+/// | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |
+///
+/// The two `multi_match` rows are the ones that are not merely slow. Without the
+/// lowering they answer EVERY document, because the scan renders the float array
+/// to text and every component contains a `0` — a 0-hit query silently becomes
+/// a match-all. Those rows are the reason the write-side change cannot ship on
+/// its own.
+///
+/// A WHOLE CLASS OF ANSWERS CHANGES, and it is stated rather than buried. SEVEN
+/// rows above differ between `main` and this branch (nine shapes on the 300-doc
+/// regression fixture, which probes both companions), and they are all the same
+/// thing: a `dense_vector` and its `_chunks` companion no longer have any
+/// lexical surface, so every lexical leaf naming one answers 0. On `main` those
+/// clauses reached the stored-doc scan, which renders the float array back to
+/// decimal text and matches against THAT — its `Term` arm matches any ELEMENT of
+/// a JSON array (`json_values_equal`) and its `match` arm splits on
+/// non-alphanumerics — so a shape answered 0 or answered most of the corpus
+/// purely according to whether the rendered decimals happened to contain the
+/// probe. `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}` matched EVERY document.
+/// None of that is a retrieval semantic anyone asked for: ES rejects these
+/// queries on a `dense_vector` outright (`DenseVectorFieldType` has no term
+/// query) and Lucene gives a field with no terms an empty scorer
+/// (`lucene/core/src/java/org/apache/lucene/index/Terms.java:40`). 0 is the
+/// Lucene answer. It is still a behaviour change and the CHANGELOG describes the
+/// class; `user_mapped_dense_vector_builds_no_fts_term_dictionary`'s (d) loop
+/// pins it shape by shape.
+///
+/// `_score` MOVES TOO, for bools whose hit count does not. Dropping the dead
+/// clause out of `should` makes the surviving bool projectable through
+/// `query_node_to_fts_with_keyword_fields`, so BM25 scores it where the
+/// stored-doc scan used to. On the 300-doc fixture,
+/// `bool{should:[term emb, term cat:"even"]}` keeps its 150 hits but goes
+/// `_score` 0.008402659 → 0.6931471 and `max_score` 1.6931472 → 0.6931471, and
+/// `bool{must:[match body], must_not:[term emb]}` keeps its 300 hits and its
+/// `_score` but goes `max_score` 1.6931472 → 0.008402659. Both moves are toward
+/// consistency: on `main` these reported a `max_score` no returned hit carried.
+/// Isolated to the plan-time fold — with the write-side exclusion alone and the
+/// lowering disabled, both are byte-identical to `main`. Pinned in that test's
+/// (d2c) block.
+///
+/// Not claimed and not changed: XERJ still answers 200 rather than 400 for
+/// these shapes. Adding ES's rejection is a separate, breaking change.
pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String> {
let mut excluded = semantic_derived_vector_fields(schema);
+ excluded.extend(lexically_typeless_fields(schema));
for field in &schema.fields {
if field.options.indexed {
continue;
@@ -3859,6 +4037,83 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>
excluded
}
+/// Fields whose DECLARED type has no lexical representation at all — #328.
+///
+/// Today that is exactly `FieldType::Vector` (`dense_vector`), so the base list
+/// is deliberately the SAME list, from the same walk, that decides which fields
+/// get an HNSW graph: [`crate::index::collect_dense_vector_fields`]. Two
+/// independent notions of "which fields are vectors" is how you end up with a
+/// field that has both a graph and a term dictionary, or neither — so there is
+/// one walk and this is a view of it.
+///
+/// Each vector contributes THREE names, exactly the shape
+/// [`semantic_derived_vector_fields`] already emits per target, because a
+/// `dense_vector` never arrives alone: `passage_scored_vector_fields` unions
+/// user-mapped vectors with semantic targets and both grow a `<field>_chunks`
+/// companion (the per-document MULTI-vector) and a
+/// `__xerj_passage_meta__<field>` sidecar. Excluding only the base name leaves
+/// the companion behind, and the companion is the BIGGER half: on the 5,000-doc
+/// × 128-dim corpus measured in [`fts_excluded_fields`], `<seg>.emb_chunks.fst`
+/// is 26,518,005 B against `<seg>.emb.fst`'s 13,256,659 B — twice the size, and
+/// 49% of `main`'s whole 54,068,549 B durable index on its own.
+///
+/// Lucene draws the same line and never lets a multi-vector reach an analyzer:
+/// its per-document float[][] shape is `LateInteractionField`, which `extends
+/// BinaryDocValuesField`
+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`)
+/// and hands its constructor straight to `super(name, encode(value))` (`:44`)
+/// — doc values only, no index options, so no term dictionary can exist for it
+/// any more than for the single-vector `KnnFloatVectorField`
+/// (`KnnFloatVectorField.java:70`: vector attributes, then `freeze()`).
+///
+/// NAME SYNTHESIS, stated rather than discovered: a user field literally named
+/// `emb_chunks` alongside a `dense_vector` `emb` loses its term dictionary to
+/// this collision. That is the same collision `semantic_derived_vector_fields`
+/// has always accepted for `<target>_chunks`, and the same names
+/// `observe_passage_chunks_in_source` reserves, so the ambiguity is the
+/// engine's convention rather than something introduced here.
+///
+/// ALLOCATION: this runs once per SEARCH (`index::search_inner` calls it to
+/// decide whether the plan-time lowering has anything to do), and the vast
+/// majority of indices have no vector field at all. `collect_dense_vector_fields`
+/// clones the name of EVERY field it walks past, not just the vector ones, so
+/// the no-vector case is short-circuited by the allocation-free predicate
+/// below and returns an empty `HashSet`, which does not allocate either. The
+/// two must agree by construction, and
+/// `the_fast_path_predicate_agrees_with_the_walk` pins that they do.
+pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<String> {
+ let mut excluded = std::collections::HashSet::new();
+ if !has_vector_field(&schema.fields) {
+ return excluded;
+ }
+ for field in crate::index::collect_dense_vector_fields(schema) {
+ excluded.insert(format!("{field}_chunks"));
+ excluded.insert(format!(
+ "{}{}",
+ xerj_query::executor::PASSAGE_METADATA_PREFIX,
+ field
+ ));
+ excluded.insert(field);
+ }
+ excluded
+}
+
+/// Does this field list contain a `dense_vector` anywhere? Allocation-free.
+///
+/// The recursion condition is `collect_dense_vector_fields`'s, deliberately: it
+/// stops at a `Vector` rather than descending into its sub-fields, so the two
+/// walks visit exactly the same nodes and cannot disagree about whether the set
+/// would come back empty.
+fn has_vector_field(fields: &[xerj_common::types::FieldConfig]) -> bool {
+ fields.iter().any(|fc| {
+ if matches!(fc.field_type, FieldType::Vector) {
+ true
+ } else {
+ !fc.fields.is_empty() && has_vector_field(&fc.fields)
+ }
+ })
+}
+
#[cfg(test)]
mod semantic_derived_vector_exclusion_tests {
use super::*;
@@ -4039,6 +4294,257 @@ fn extract_text_value(val: &Value) -> String {
}
}
+/// Does `excluded` hold anything strictly BELOW `path`?
+///
+/// Allocation-free on the hot path: the set is schema-derived and normally
+/// holds zero dotted entries, so this is a handful of length compares per
+/// object-valued field per document.
+pub(crate) fn has_excluded_descendant(
+ path: &str,
+ excluded: &std::collections::HashSet<String>,
+) -> bool {
+ excluded.iter().any(|entry| {
+ entry.len() > path.len() && entry.as_bytes()[path.len()] == b'.' && entry.starts_with(path)
+ })
+}
+
+/// [`extract_text_value`], with every descendant named in `excluded` removed
+/// from the flattened object first — #328.
+///
+/// `path` is the dotted path of `val` itself. Only objects can have
+/// descendants, so every other shape short-circuits to `extract_text_value`,
+/// and an object with no excluded descendant does too — the clone is paid only
+/// by the documents that actually carry an excluded sub-field.
+pub(crate) fn extract_text_value_excluding(
+ val: &Value,
+ path: &str,
+ excluded: &std::collections::HashSet<String>,
+) -> String {
+ fn prune(val: &Value, path: &str, excluded: &std::collections::HashSet<String>) -> Value {
+ match val {
+ Value::Object(map) => {
+ let mut kept = serde_json::Map::with_capacity(map.len());
+ for (key, child) in map {
+ let child_path = if path.is_empty() {
+ key.clone()
+ } else {
+ format!("{path}.{key}")
+ };
+ if excluded.contains(&child_path) {
+ continue;
+ }
+ kept.insert(key.clone(), prune(child, &child_path, excluded));
+ }
+ Value::Object(kept)
+ }
+ other => other.clone(),
+ }
+ }
+ if !matches!(val, Value::Object(_)) || !has_excluded_descendant(path, excluded) {
+ return extract_text_value(val);
+ }
+ extract_text_value(&prune(val, path, excluded))
+}
+
+/// Unit coverage for the three helpers #328 adds — the widest new surface in
+/// this file is [`lexically_typeless_fields`], which is `pub`.
+#[cfg(test)]
+mod lexically_typeless_fields_tests {
+ use super::*;
+ use serde_json::json;
+ use xerj_common::types::{FieldConfig, FieldType, Schema};
+
+ fn vector(name: &str, dims: usize) -> FieldConfig {
+ let mut f = FieldConfig::new(name, FieldType::Vector);
+ f.options.dimensions = Some(dims);
+ f.options.similarity = Some("cosine".to_string());
+ f
+ }
+
+ /// Every `dense_vector` contributes THREE names, because a vector never
+ /// arrives alone: the base field, the `_chunks` multi-vector companion and
+ /// the passage-metadata sidecar. Excluding only the base name is what left
+ /// `<seg>.emb_chunks.fst` — the bigger of the two artifacts — behind.
+ #[test]
+ fn every_vector_contributes_its_whole_family() {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+
+ let typeless = lexically_typeless_fields(&schema);
+ assert_eq!(
+ typeless,
+ std::collections::HashSet::from([
+ "emb".to_string(),
+ "emb_chunks".to_string(),
+ "__xerj_passage_meta__emb".to_string(),
+ ])
+ );
+ assert!(!typeless.contains("body"), "a text field is not typeless");
+ }
+
+ /// The exclusion is by DECLARED type, not by value shape: a numeric array
+ /// under a non-vector mapping stays lexical exactly as before.
+ #[test]
+ fn a_non_vector_field_is_never_typeless() {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ schema
+ .add_field(FieldConfig::new("cat", FieldType::Keyword))
+ .unwrap();
+ schema
+ .add_field(FieldConfig::new("n", FieldType::Long))
+ .unwrap();
+ assert!(lexically_typeless_fields(&schema).is_empty());
+ }
+
+ /// A vector nested under an object mapping arrives as a DOTTED path — the
+ /// first member of this set that is not a root-level name, and the reason
+ /// `collect_text_fields` had to start matching the full path.
+ #[test]
+ fn a_nested_vector_is_collected_by_its_dotted_path() {
+ let mut schema = Schema::empty();
+ let mut parent = FieldConfig::new("passages", FieldType::Object);
+ parent.fields.push(vector("vec", 4));
+ schema.add_field(parent).unwrap();
+
+ let typeless = lexically_typeless_fields(&schema);
+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");
+ assert!(typeless.contains("passages.vec_chunks"));
+ assert!(
+ !typeless.contains("vec"),
+ "the leaf name alone must not match"
+ );
+ }
+
+ /// `fts_excluded_fields` is the union, so the vector family must appear
+ /// there too — that union is what flush, merge and `build_fts_field_configs`
+ /// all read.
+ #[test]
+ fn fts_excluded_fields_unions_the_vector_family_in() {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+
+ let excluded = fts_excluded_fields(&schema);
+ for name in ["emb", "emb_chunks", "__xerj_passage_meta__emb"] {
+ assert!(excluded.contains(name), "{name} missing from {excluded:?}");
+ }
+ assert!(!excluded.contains("body"));
+ }
+
+ /// The allocation-free short-circuit and the allocating walk must agree on
+ /// "is this set empty", for every shape either of them can see — otherwise
+ /// the fast path would silently disable the whole #328 query half on some
+ /// schema.
+ #[test]
+ fn the_fast_path_predicate_agrees_with_the_walk() {
+ let mut nested = FieldConfig::new("passages", FieldType::Object);
+ nested.fields.push(vector("vec", 4));
+ let mut deep = FieldConfig::new("a", FieldType::Object);
+ let mut mid = FieldConfig::new("b", FieldType::Object);
+ mid.fields.push(vector("c", 2));
+ deep.fields.push(mid);
+ let mut object_without_a_vector = FieldConfig::new("meta", FieldType::Object);
+ object_without_a_vector
+ .fields
+ .push(FieldConfig::new("note", FieldType::Text));
+
+ for (label, fields) in [
+ ("empty", vec![]),
+ ("text only", vec![FieldConfig::new("body", FieldType::Text)]),
+ ("object with no vector", vec![object_without_a_vector]),
+ ("root vector", vec![vector("emb", 8)]),
+ ("nested vector", vec![nested]),
+ ("two levels deep", vec![deep]),
+ (
+ "vector beside text",
+ vec![FieldConfig::new("body", FieldType::Text), vector("emb", 8)],
+ ),
+ ] {
+ let mut schema = Schema::empty();
+ for f in fields {
+ schema.add_field(f).unwrap();
+ }
+ assert_eq!(
+ has_vector_field(&schema.fields),
+ !crate::index::collect_dense_vector_fields(&schema).is_empty(),
+ "predicate and walk disagree on `{label}`"
+ );
+ assert_eq!(
+ has_vector_field(&schema.fields),
+ !lexically_typeless_fields(&schema).is_empty(),
+ "short-circuit changed the answer on `{label}`"
+ );
+ }
+ }
+
+ #[test]
+ fn has_excluded_descendant_matches_only_strictly_below_the_path() {
+ let set = std::collections::HashSet::from(["passages.vec".to_string(), "emb".to_string()]);
+ assert!(has_excluded_descendant("passages", &set));
+ // The path ITSELF is not a descendant of itself — the caller checks
+ // `excluded.contains(&path)` separately, and double-counting it here
+ // would make the prune fire for a leaf that was already skipped.
+ assert!(!has_excluded_descendant("passages.vec", &set));
+ assert!(!has_excluded_descendant("emb", &set));
+ // A shared PREFIX is not a path prefix: `passages_extra` must not match
+ // `passages.vec`, which is why the check tests for the `.` separator
+ // rather than calling `starts_with` alone.
+ assert!(!has_excluded_descendant("passages_extra", &set));
+ assert!(!has_excluded_descendant("other", &set));
+ }
+
+ /// The flattened root-level blob is where a nested vector actually lands,
+ /// so the prune has to happen there and not only in the recursion.
+ #[test]
+ fn extract_text_value_excluding_drops_the_nested_vector_from_the_blob() {
+ let set = std::collections::HashSet::from(["passages.vec".to_string()]);
+ let val = json!({
+ "text": "quarterly liquidity",
+ "vec": [0.125, 0.25, 0.5],
+ "nested": {"keep": "yes"}
+ });
+ let pruned = extract_text_value_excluding(&val, "passages", &set);
+ assert!(
+ pruned.contains("quarterly liquidity"),
+ "text kept: {pruned}"
+ );
+ assert!(
+ pruned.contains("yes"),
+ "unrelated sub-object kept: {pruned}"
+ );
+ assert!(
+ !pruned.contains("0.125") && !pruned.contains("0.25"),
+ "vector components must be gone: {pruned}"
+ );
+ }
+
+ /// No excluded descendant → identical to `extract_text_value`, and no clone
+ /// is paid. This is the path every document without a nested vector takes.
+ #[test]
+ fn extract_text_value_excluding_is_a_no_op_without_a_descendant() {
+ let set = std::collections::HashSet::from(["other.vec".to_string()]);
+ let val = json!({"text": "quarterly liquidity", "n": 7});
+ assert_eq!(
+ extract_text_value_excluding(&val, "passages", &set),
+ extract_text_value(&val)
+ );
+ // A non-object short-circuits regardless of what the set holds.
+ let scalar = json!("plain string");
+ assert_eq!(
+ extract_text_value_excluding(&scalar, "passages", &set),
+ extract_text_value(&scalar)
+ );
+ }
+}
+
#[cfg(test)]
mod bounded_delta_counts_tests {
use super::*;
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index a1ee0150a..995a8cb38 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -127,6 +127,766 @@ async fn test_semantic_vectors_stay_stored_but_not_fts_indexed_after_merge_and_r
assert_queries(&idx).await;
}
+/// #328 — a USER-MAPPED `dense_vector` must contribute no lexical artifacts.
+///
+/// The `#12` fix above covers the vectors XERJ *generates* from a semantic
+/// mapping. A field the user maps as `dense_vector` took a different route:
+/// it is `indexed == true` (kNN needs the HNSW graph), so the `index: false`
+/// arm of `memtable::fts_excluded_fields` never fired and the field's 128
+/// floats were flattened to decimal strings and tokenised into a term
+/// dictionary that no query path reads.
+///
+/// This pins all four halves of the fix:
+/// * no `<seg>.emb.{fst,post,norms}` is written, at flush or after merge —
+/// AND none for the `<field>_chunks` companion either. A `dense_vector`
+/// never arrives alone: `passage_scored_vector_fields` gives every one of
+/// them a `_chunks` multi-vector and a `__xerj_passage_meta__` sidecar, and
+/// the companion is the bigger artifact of the two on a chunked corpus;
+/// * kNN, `exists` and the lexical fields keep answering exactly as before;
+/// * EVERY lexical leaf on the vector field answers 0, which is a BEHAVIOUR
+/// CHANGE for a whole class of shapes rather than for one row. `main` sends
+/// these to the stored-doc scan, which renders the float array back to
+/// decimal text and matches against it, so `wildcard {emb:"0*"}` and
+/// `fuzzy {emb:"0"~2}` answered ALL 300 documents, `prefix {emb:"0"}` 146,
+/// `match_phrase_prefix {emb:"0"}` 50 and numeric `term`/`terms` 1 each.
+/// None of that is a retrieval semantic anyone asked for and all of it is
+/// now 0 — see the table in the (d) loop for the measured before/after;
+/// * and neither does an EXPLICITLY-FIELDED lexical query. This is the one
+/// the writer-side change cannot cover on its own and the one that fails
+/// loudly: with the postings gone but no plan-time lowering,
+/// `{"multi_match":{"query":"0","fields":["emb"]}}` returns EVERY document
+/// instead of none, because the scan renders the float array to text and
+/// every component contains a `0`. Measured on a 5,000-doc × 128-dim
+/// corpus, post-`force_merge`, three source states (first call each, before
+/// the result cache): `main` 0 hits / 0.409 ms, postings-removed-only
+/// **5,000 hits** / 123.5 ms, this branch 0 hits / 0.012 ms. The same run
+/// puts `{"term":{"emb":"<component>"}}` at 89.2 → 186.8 → 0.061 ms, all
+/// three answering 0.
+#[tokio::test]
+async fn user_mapped_dense_vector_builds_no_fts_term_dictionary() {
+ fn segment_files(dir: &std::path::Path) -> Vec<(String, u64)> {
+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {
+ for entry in std::fs::read_dir(dir).unwrap() {
+ let entry = entry.unwrap();
+ if entry.file_type().unwrap().is_dir() {
+ walk(&entry.path(), out);
+ } else {
+ out.push((
+ entry.file_name().to_string_lossy().into_owned(),
+ entry.metadata().unwrap().len(),
+ ));
+ }
+ }
+ }
+ let mut out = Vec::new();
+ walk(dir, &mut out);
+ out
+ }
+
+ const DIMS: usize = 128;
+ const DOCS: usize = 300;
+
+ let dir = TempDir::new().unwrap();
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ schema
+ .fields
+ .push(FieldConfig::new("cat", FieldType::Keyword));
+ schema.fields.push(FieldConfig::new("n", FieldType::Long));
+ let mut emb = FieldConfig::new("emb", FieldType::Vector);
+ emb.options.dimensions = Some(DIMS);
+ emb.options.similarity = Some("cosine".to_string());
+ schema.fields.push(emb);
+
+ let engine = make_engine(&dir);
+ engine.create_index("duprobe", schema).unwrap();
+ let idx = engine.get_index("duprobe").unwrap();
+
+ // Deterministic pseudo-random components with enough decimal digits that
+ // each one is a distinct term — the shape that made the FST large.
+ let component = |doc: usize, dim: usize| -> f64 {
+ let h = (doc as u64)
+ .wrapping_mul(6_364_136_223_846_793_005)
+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));
+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0
+ };
+ let mut first_vector = Vec::new();
+ for d in 0..DOCS {
+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();
+ if d == 0 {
+ first_vector = v.clone();
+ }
+ idx.index_document(
+ Some(format!("d{d}")),
+ json!({
+ "body": format!("quarterly liquidity evidence document number {d}"),
+ "cat": if d % 2 == 0 { "even" } else { "odd" },
+ "n": d,
+ "emb": v,
+ // The `_chunks` companion, i.e. the per-document MULTI-vector.
+ // Present in the fixture because a chunked corpus is the shape
+ // RFC #148 reports, and excluding only `emb` leaves this one
+ // fully tokenised.
+ "emb_chunks": [v, v],
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ idx.refresh().await.unwrap();
+ idx.force_merge(1).await.unwrap();
+
+ let files = segment_files(dir.path());
+ let bytes_of = |suffix: &str| -> u64 {
+ files
+ .iter()
+ .filter(|(name, _)| name.ends_with(suffix))
+ .map(|(_, len)| *len)
+ .sum()
+ };
+ // Reported so the saving is a measured number in the log, not a claim.
+ //
+ // TWO totals, and only the second one is quotable. The whole data dir
+ // includes the write-ahead log, whose tail is reclaimed asynchronously, so
+ // it is NOT reproducible: eight runs of this test on one machine printed a
+ // whole-dir total between 858,147 B and 1,105,001 B, a 29% spread, entirely
+ // because the surviving `.wal` files ranged from 256 B to 6 MB while the
+ // segment bytes did not move at all. `durable-index` excludes `.wal` and is
+ // reproducible to ~230 B (the run-to-run jitter in the HNSW `graph.bin`):
+ // the same eight runs printed 857,843 – 858,071 B. That is the number the
+ // CHANGELOG quotes and the one the ceiling below pins.
+ let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();
+ let durable: u64 = files
+ .iter()
+ .filter(|(name, _)| !name.ends_with(".wal"))
+ .map(|(_, len)| *len)
+ .sum();
+ eprintln!(
+ "#328 lexical bytes: emb.fst={} emb.post={} emb.norms={} emb_chunks.fst={} | \
+ body.fst={} body.post={} cat.fst={} n.fst={} | durable-index={durable} \
+ whole-dir={whole_dir}",
+ bytes_of(".emb.fst"),
+ bytes_of(".emb.post"),
+ bytes_of(".emb.norms"),
+ bytes_of(".emb_chunks.fst"),
+ bytes_of(".body.fst"),
+ bytes_of(".body.post"),
+ bytes_of(".cat.fst"),
+ bytes_of(".n.fst"),
+ );
+ // The published saving, enforced. `origin/main` writes 3,249,323 B of
+ // durable index for this fixture and this branch writes ~858 KB; the ceiling
+ // sits three orders of magnitude clear of the jitter and still 2.2 MB below
+ // `main`, so it is a real fail-before and not a golden-file trap. Watched
+ // fail at exactly 3,249,323 with both source files reverted to `origin/main`.
+ assert!(
+ durable < 1_500_000,
+ "durable index must lose the vector term dictionaries; got {durable} B \
+ (`origin/main` writes 3,249,323 B for this fixture, this branch ~858 KB)"
+ );
+
+ for suffix in [
+ ".emb.fst",
+ ".emb.post",
+ ".emb.norms",
+ // The companion, and it is the BIGGER half. On this 300-doc fixture,
+ // `main` writes `.emb.fst` = 796,274 B and `.emb_chunks.fst` =
+ // 1,592,118 B; at 5,000 × 128 it is 13,256,659 B against 26,518,005 B.
+ // Excluding the base name alone would leave more behind than it
+ // removes.
+ ".emb_chunks.fst",
+ ".emb_chunks.post",
+ ".emb_chunks.norms",
+ ] {
+ assert!(
+ !files.iter().any(|(name, _)| name.ends_with(suffix)),
+ "a dense_vector field must contribute no `{suffix}` artifact; got {:?}",
+ files
+ .iter()
+ .filter(|(name, _)| name.contains(".emb"))
+ .collect::<Vec<_>>()
+ );
+ }
+ // The lexical fields are untouched — this is a per-type exclusion, not a
+ // blanket one.
+ for suffix in [".body.fst", ".cat.fst", ".n.fst"] {
+ assert!(
+ files.iter().any(|(name, _)| name.ends_with(suffix)),
+ "lexical fields must keep their term dictionaries; missing {suffix}"
+ );
+ }
+
+ // (a) kNN still answers from the HNSW graph.
+ let knn = idx
+ .search(
+ &parse_request(&json!({
+ "query": {"knn": {"field": "emb", "query_vector": first_vector, "k": 3}},
+ "size": 10
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(knn.hits.len(), 3, "kNN must still retrieve neighbours");
+ assert_eq!(
+ knn.hits[0].id, "d0",
+ "nearest neighbour must be the probe doc"
+ );
+
+ // (b) `exists` is answered from `_source` / doc values, never the postings.
+ let exists = idx
+ .search(&make_search(json!({"exists": {"field": "emb"}})))
+ .await
+ .unwrap();
+ assert_eq!(exists.total.value, DOCS as u64);
+
+ // (c) the lexical fields still match.
+ let lexical = idx
+ .search(&make_search(json!({"match": {"body": "liquidity"}})))
+ .await
+ .unwrap();
+ assert_eq!(lexical.total.value, DOCS as u64);
+ let kw = idx
+ .search(&make_search(json!({"term": {"cat": "even"}})))
+ .await
+ .unwrap();
+ assert_eq!(kw.total.value, (DOCS / 2) as u64);
+
+ // (d) EVERY lexical leaf that names the vector field answers 0.
+ //
+ // This is a CLASS, not a list of individually interesting queries, and the
+ // class is the behaviour change this fix ships. On `main` a lexical clause
+ // on `emb` falls through the missing/oversized term dictionary onto the
+ // stored-doc scan, which renders the float array back to decimal text and
+ // matches against that text — so whether a shape answered 0 or answered
+ // most of the corpus on `main` depended entirely on whether the rendered
+ // decimals happened to contain the probe. Measured on THIS fixture with
+ // both source files reverted to `origin/main` (300 docs × 128 dim, after
+ // `force_merge(1)`), `main` → this branch:
+ //
+ // term {emb: <component>} numeric .. 1 → 0
+ // terms {emb: [<component>]} numeric .. 1 → 0
+ // prefix {emb: "0"} .. 146 → 0
+ // wildcard{emb: "0*"} .. 300 → 0 (every document)
+ // fuzzy {emb: {value:"0", fuzziness:2}} 300 → 0 (every document)
+ // match_phrase_prefix {emb: "0"} .. 50 → 0
+ // prefix {emb_chunks: "0"} .. 146 → 0
+ // wildcard{emb_chunks: "0*"} .. 300 → 0 (every document)
+ // fuzzy {emb_chunks: {value:"0", …}} .. 300 → 0 (every document)
+ //
+ // …and the shapes whose probe text simply never appeared in a rendered
+ // float answered 0 on both sides. All of them are pinned below at 0, so
+ // the class is closed rather than sampled: a `dense_vector` and its
+ // `_chunks` companion have no lexical surface at all.
+ //
+ // The two halves of the fix cover DIFFERENT rows of that table, which is why
+ // both are needed. Running the same loop with the write-side exclusion alone
+ // (the plan-time lowering disabled in `search_inner`) leaves exactly four
+ // shapes non-zero — numeric `term`/`terms` at 1 each and the two
+ // `multi_match`es at 300 — while `prefix`/`wildcard`/`fuzzy`/
+ // `match_phrase_prefix` are already 0, because a multi-term query has no
+ // term dictionary left to enumerate. Reverting both files to `origin/main`
+ // leaves nine non-zero. Both counts were watched.
+ //
+ // A POSITIVE component: a leading `-` is `query_string`'s NOT operator,
+ // which would make the probe measure the parser rather than the field.
+ let probe_num = first_vector
+ .iter()
+ .copied()
+ .find(|v| *v > 0.0)
+ .expect("fixture vector has a positive component");
+ let probe = format!("{probe_num}");
+ let mut lexical_leaks: Vec<String> = Vec::new();
+ for q in [
+ json!({"term": {"emb": probe}}),
+ json!({"match": {"emb": probe}}),
+ json!({"match": {"emb": "0.5"}}),
+ json!({"query_string": {"query": probe}}),
+ json!({"multi_match": {"query": probe, "fields": ["*"]}}),
+ // Numeric `term` / `terms` / `range`.
+ //
+ // The first two answered 1 on `main`: the stored-doc scan's `Term` arm
+ // matches any ELEMENT of a JSON array — "this document's vector happens
+ // to contain this float". ES rejects `term` on a `dense_vector`
+ // outright and Lucene gives a field with no terms an empty scorer, so 0
+ // is the answer this pins.
+ json!({"term": {"emb": probe_num}}),
+ json!({"terms": {"emb": [probe_num]}}),
+ json!({"terms": {"emb": [probe]}}),
+ json!({"range": {"emb": {"gte": -2.0, "lte": 2.0}}}),
+ // The MULTI-TERM leaves, and the loudest half of the class. A rendered
+ // float almost always contains a `0` somewhere, so on `main` these
+ // matched a large fraction of the corpus — `wildcard` and `fuzzy`
+ // matched ALL 300 documents — purely as an artifact of decimal
+ // rendering. Every one of them is a `field`-naming leaf in
+ // `lower_lexically_typeless_clauses`'s first arm.
+ json!({"prefix": {"emb": "0"}}),
+ json!({"wildcard": {"emb": "0*"}}),
+ json!({"fuzzy": {"emb": {"value": "0", "fuzziness": 2}}}),
+ json!({"regexp": {"emb": "0.*"}}),
+ json!({"match_phrase": {"emb": probe}}),
+ json!({"match_phrase_prefix": {"emb": "0"}}),
+ // EXPLICITLY-FIELDED forms — the half the writer-side change cannot
+ // reach. Each of these names `emb` outright rather than expanding onto
+ // it, so no `*`-expansion filter helps; without a plan-time lowering
+ // they land on the stored-doc scan. The `multi_match` one is the
+ // correctness failure, not merely a slow one: postings-removed-only,
+ // it answers DOCS instead of 0.
+ json!({"multi_match": {"query": "0", "fields": ["emb"]}}),
+ json!({"simple_query_string": {"query": "0", "fields": ["emb"]}}),
+ json!({"query_string": {"query": "0", "default_field": "emb"}}),
+ // …and the `_chunks` companion, which is in the exclusion set for the
+ // same reason and therefore needs the same lowering — including the
+ // multi-term leaves, which matched 146 and 300 documents on `main`.
+ json!({"term": {"emb_chunks": probe}}),
+ json!({"prefix": {"emb_chunks": "0"}}),
+ json!({"wildcard": {"emb_chunks": "0*"}}),
+ json!({"fuzzy": {"emb_chunks": {"value": "0", "fuzziness": 2}}}),
+ json!({"multi_match": {"query": "0", "fields": ["emb_chunks"]}}),
+ ] {
+ let hits = idx.search(&make_search(q.clone())).await.unwrap();
+ // Accumulated rather than asserted per row, deliberately: this loop is
+ // pinning a CLASS, so a regression report that names every shape that
+ // came back non-zero is worth far more than one that stops at the first.
+ if hits.total.value != 0 {
+ lexical_leaks.push(format!("{q} → {} hits", hits.total.value));
+ }
+ }
+ assert!(
+ lexical_leaks.is_empty(),
+ "a lexical query on a dense_vector (or its `_chunks` companion) must \
+ match nothing; {} of the shapes above did not:\n {}",
+ lexical_leaks.len(),
+ lexical_leaks.join("\n ")
+ );
+
+ // (d2) a MIXED field list keeps its lexical members. Only the vector entry
+ // is dropped, so this is `{"multi_match":{"query":"liquidity","fields":
+ // ["body"]}}` in effect — the whole corpus, not nothing. The empty-list
+ // lowering must not swallow a list that still has a real field in it.
+ let mixed = idx
+ .search(&make_search(
+ json!({"multi_match": {"query": "liquidity", "fields": ["emb", "body"]}}),
+ ))
+ .await
+ .unwrap();
+ assert_eq!(
+ mixed.total.value, DOCS as u64,
+ "dropping the vector entry must leave the rest of a `fields` list intact"
+ );
+
+ // (d2b) COMPOSITES. Lowering a leaf to `match_none` is only half a rewrite:
+ // the enclosing `bool` then has to fold it the way the EVALUATOR reads it,
+ // and `doc_matches_query`'s Bool arm makes `should` required whenever `must`
+ // and `filter` are empty — `must_not` does NOT relax it. The first cut of
+ // this fix skipped the "every optional clause is dead" fold whenever a
+ // `must_not` was present and then dropped the dead clause anyway, which
+ // turned the first row below from 0 hits into 150.
+ //
+ // Every expected value here is `main`'s answer for the same request: a
+ // lexical clause on `emb` matched nothing before the postings went away and
+ // must match nothing after.
+ for (q, want) in [
+ // The regression. `should` is required, its only clause is dead.
+ (
+ json!({"bool": {
+ "should": [{"term": {"emb": probe}}],
+ "must_not": [{"term": {"cat": "odd"}}]
+ }}),
+ 0u64,
+ ),
+ // Should-only, no `must_not` — the shape a `simple_query_string` lowers
+ // to.
+ (json!({"bool": {"should": [{"term": {"emb": probe}}]}}), 0),
+ // A live `should` beside the dead one still matches on its own.
+ (
+ json!({"bool": {"should": [
+ {"term": {"emb": probe}},
+ {"term": {"cat": "even"}}
+ ]}}),
+ (DOCS / 2) as u64,
+ ),
+ // `must` / `filter`: an unsatisfiable REQUIRED clause sinks the bool.
+ (
+ json!({"bool": {
+ "must": [{"term": {"emb": probe}}, {"match": {"body": "liquidity"}}]
+ }}),
+ 0,
+ ),
+ (json!({"bool": {"filter": [{"term": {"emb": probe}}]}}), 0),
+ // `must_not`: a clause that matches nothing EXCLUDES nothing. Same
+ // answer as `main`; what changes is the route — a `match_none` left
+ // sitting in `must_not` un-projects the whole bool in
+ // `query_node_to_fts_with_keyword_fields` and sends a query that used
+ // to run on the inverted index to the stored-doc scan.
+ (
+ json!({"bool": {"must_not": [{"term": {"emb": probe}}]}}),
+ DOCS as u64,
+ ),
+ (
+ json!({"bool": {
+ "must": [{"match": {"body": "liquidity"}}],
+ "must_not": [{"term": {"emb": probe}}]
+ }}),
+ DOCS as u64,
+ ),
+ // `constant_score` is the only wrapper with an arm in the FTS
+ // projection, so it is the only one where failing to bubble up the
+ // `match_none` costs an inverted-index lookup.
+ (
+ json!({"constant_score": {"filter": {"term": {"emb": probe}}}}),
+ 0,
+ ),
+ ] {
+ let hits = idx.search(&make_search(q.clone())).await.unwrap();
+ assert_eq!(
+ hits.total.value, want,
+ "composite over a dense_vector clause: expected {want}, got {} for {q}",
+ hits.total.value
+ );
+ }
+
+ // (d2c) …and the composites that keep their HIT COUNT still change their
+ // `_score`, which is wire-visible and therefore has to be pinned here
+ // rather than discovered by a user.
+ //
+ // Cause: `lower_lexically_typeless_clauses` drops the dead clause out of
+ // `should`, which makes the surviving bool PROJECTABLE through
+ // `query_node_to_fts_with_keyword_fields`. On `main` the same bool contains
+ // a clause that cannot be projected, so the whole query runs on the
+ // stored-doc scan and is scored by `score_query_against_doc` instead of
+ // BM25. Isolated to the fold: with the write-side exclusion alone and the
+ // lowering disabled, both scores below are byte-identical to `main`.
+ //
+ // Measured on this fixture, `origin/main` → this branch:
+ //
+ // bool{should:[term emb, term cat:even]}
+ // hits 150 → 150 (unchanged)
+ // _score 0.008402659 → 0.6931471
+ // max_score 1.6931472 → 0.6931471
+ // bool{must:[match body], must_not:[term emb]}
+ // hits 300 → 300 (unchanged)
+ // _score 0.008402659 → 0.008402659 (unchanged)
+ // max_score 1.6931472 → 0.008402659
+ //
+ // The assertions are properties rather than golden floats, so they survive
+ // a BM25 tuning change but still fail on `main`:
+ //
+ // 1. `max_score` equals the top hit's `_score`. On `main` it does NOT for
+ // either shape — the scan path reports a `max_score` of 1.6931472 that
+ // no returned hit carries. That inconsistency is what goes away.
+ // 2. a bool whose only surviving `should` is `{"term":{"cat":"even"}}`
+ // scores exactly like that term query on its own. On `main` it scores
+ // 0.008402659 against the term query's 0.6931471.
+ let cat_alone = idx
+ .search(&make_search(json!({"term": {"cat": "even"}})))
+ .await
+ .unwrap();
+ let should_mixed = idx
+ .search(&make_search(json!({"bool": {"should": [
+ {"term": {"emb": probe}},
+ {"term": {"cat": "even"}}
+ ]}})))
+ .await
+ .unwrap();
+ let mustnot_bool = idx
+ .search(&make_search(json!({"bool": {
+ "must": [{"match": {"body": "liquidity"}}],
+ "must_not": [{"term": {"emb": probe}}]
+ }})))
+ .await
+ .unwrap();
+ // Hit counts first, so a regression that empties one of these reports
+ // "expected 150 hits" rather than panicking on `hits[0]` below.
+ for (label, res, want) in [
+ ("term {cat:even}", &cat_alone, (DOCS / 2) as u64),
+ (
+ "bool{should:[dead, term cat]}",
+ &should_mixed,
+ (DOCS / 2) as u64,
+ ),
+ (
+ "bool{must:[match], must_not:[dead]}",
+ &mustnot_bool,
+ DOCS as u64,
+ ),
+ ] {
+ assert_eq!(
+ res.total.value, want,
+ "{label}: the score probes must keep `main`'s hit count"
+ );
+ assert!(!res.hits.is_empty(), "{label}: expected a scored top hit");
+ }
+ eprintln!(
+ "#328 scores: term{{cat}}={:?}/{:?} should-mixed={:?}/{:?} must-not={:?}/{:?} \
+ (top/max)",
+ cat_alone.hits[0].score,
+ cat_alone.max_score,
+ should_mixed.hits[0].score,
+ should_mixed.max_score,
+ mustnot_bool.hits[0].score,
+ mustnot_bool.max_score,
+ );
+ for (label, res) in [
+ ("bool{should:[dead, term cat]}", &should_mixed),
+ ("bool{must:[match], must_not:[dead]}", &mustnot_bool),
+ ] {
+ assert_eq!(
+ res.max_score,
+ Some(res.hits[0].score),
+ "{label}: max_score must be the top hit's _score, got max={:?} top={:?}",
+ res.max_score,
+ res.hits[0].score
+ );
+ }
+ assert_eq!(
+ should_mixed.hits[0].score, cat_alone.hits[0].score,
+ "a bool whose only live `should` is `term cat:even` must score exactly \
+ like that term query alone; got {:?} against {:?}",
+ should_mixed.hits[0].score, cat_alone.hits[0].score
+ );
+
+ // (d3) the mapping still describes the field — the exclusion is about
+ // lexical BYTES, not about the field's existence. This is the leaf of the
+ // `_field_caps` guarantee reachable from the engine crate: `_field_caps`
+ // is served by the API layer from exactly this schema and never opens a
+ // term dictionary, so a field that reports `dense_vector` here reports
+ // `dense_vector` there.
+ let mapped = idx.schema().await;
+ let emb_field = mapped
+ .fields
+ .iter()
+ .find(|f| f.name == "emb")
+ .expect("`emb` must still be a mapped field");
+ assert!(matches!(emb_field.field_type, FieldType::Vector));
+ assert_eq!(emb_field.options.dimensions, Some(DIMS));
+
+ // (d4) highlighting a vector field yields no fragments and no panic, while
+ // a real text field in the same request still highlights. Highlighting
+ // resolves against the stored document and never opens the field's FST,
+ // so removing the FST cannot change it.
+ let hl = idx
+ .search(
+ &parse_request(&json!({
+ "query": {"match": {"body": "liquidity"}},
+ "size": 1,
+ "highlight": {"fields": {"body": {}, "emb": {}}}
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ let frags = hl.hits[0]
+ .highlight
+ .as_ref()
+ .expect("highlight must be present");
+ assert!(
+ frags.get("body").is_some_and(|f| !f.is_empty()),
+ "a text field must still highlight"
+ );
+ assert!(
+ frags.get("emb").is_none_or(|f| f.is_empty()),
+ "a dense_vector must produce no highlight fragments, got {:?}",
+ frags.get("emb")
+ );
+
+ // (e) the index reopens and searches after a restart. The exclusion is a
+ // WRITE-side rule and no reader requires the field's sidecar to exist, so
+ // a segment written without one loads exactly like one written with it.
+ // (This does not, and cannot from inside one test binary, exercise a
+ // segment produced by the previous release: those keep their `.emb.*`
+ // files until a merge rewrites them, and the per-segment `fts_has_field`
+ // gate reads whichever shape it finds.)
+ drop(idx);
+ drop(engine); // release the data dir's `node.lock` before reopening it
+ let reopened = make_engine(&dir);
+ let idx = reopened.get_index("duprobe").unwrap();
+ let after = idx
+ .search(&make_search(json!({"match": {"body": "liquidity"}})))
+ .await
+ .unwrap();
+ assert_eq!(after.total.value, DOCS as u64);
+}
+
+/// #328, nested half — a `dense_vector` under an OBJECT mapping.
+///
+/// Excluding the leaf path alone moves no bytes here, and that is the trap the
+/// first cut of this fix fell into: the segment builder never writes a
+/// `passages.vec` field at all. It flattens the whole `passages` object into
+/// ONE text field, so the vector's 64 decimal components land in
+/// `<seg>.passages.fst` under the parent's name. The size of that file is
+/// reported below rather than asserted to a golden number, and the ceiling is
+/// deliberately generous: the point is that the fixture's 19,200 vector
+/// components are gone, and they cannot fit in 4 KiB.
+///
+/// Both mapping shapes that reach `FieldType::Vector` are pinned, because they
+/// take different routes into the schema and only one of them was covered by
+/// the walk: a dotted top-level name (`"passages.vec"`, what `put_mapping`
+/// produces) and a `vec` sub-mapping under a `passages` object (what
+/// `es_properties_to_fields` produces from nested `properties`).
+#[tokio::test]
+async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary() {
+ const DIMS: usize = 64;
+ const DOCS: usize = 300;
+
+ let component = |doc: usize, dim: usize| -> f64 {
+ let h = (doc as u64)
+ .wrapping_mul(6_364_136_223_846_793_005)
+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));
+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0
+ };
+
+ for shape in ["dotted-name", "sub-mapping"] {
+ let dir = TempDir::new().unwrap();
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ let mut vec_field = FieldConfig::new(
+ if shape == "dotted-name" {
+ "passages.vec"
+ } else {
+ "vec"
+ },
+ FieldType::Vector,
+ );
+ vec_field.options.dimensions = Some(DIMS);
+ vec_field.options.similarity = Some("cosine".to_string());
+ if shape == "dotted-name" {
+ schema.fields.push(vec_field);
+ } else {
+ let mut parent = FieldConfig::new("passages", FieldType::Object);
+ parent.fields.push(vec_field);
+ schema.fields.push(parent);
+ }
+
+ let engine = make_engine(&dir);
+ engine.create_index("nested-vec", schema).unwrap();
+ let idx = engine.get_index("nested-vec").unwrap();
+
+ let mut first_vector = Vec::new();
+ for d in 0..DOCS {
+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();
+ if d == 0 {
+ first_vector = v.clone();
+ }
+ idx.index_document(
+ Some(format!("d{d}")),
+ json!({
+ "body": format!("quarterly liquidity evidence {d}"),
+ "passages": {"vec": v},
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ idx.refresh().await.unwrap();
+ idx.force_merge(1).await.unwrap();
+
+ let mut files: Vec<(String, u64)> = Vec::new();
+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {
+ for entry in std::fs::read_dir(dir).unwrap() {
+ let entry = entry.unwrap();
+ if entry.file_type().unwrap().is_dir() {
+ walk(&entry.path(), out);
+ } else {
+ out.push((
+ entry.file_name().to_string_lossy().into_owned(),
+ entry.metadata().unwrap().len(),
+ ));
+ }
+ }
+ }
+ walk(dir.path(), &mut files);
+
+ let parent_fst: u64 = files
+ .iter()
+ .filter(|(name, _)| name.ends_with(".passages.fst"))
+ .map(|(_, len)| *len)
+ .sum();
+ // Reported so the saving is a measured number in the log, not a claim.
+ // `passages.fst` is the load-bearing figure and it is exact — 398,558 B
+ // on `origin/main`, 45 B here, for both mapping shapes. `durable-index`
+ // is diagnostic only and `whole-dir` is not reproducible at all: it
+ // carries the write-ahead log, whose tail is reclaimed asynchronously.
+ // Nothing quotes either of them; see the sibling test for the numbers
+ // the CHANGELOG does quote.
+ let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();
+ let durable: u64 = files
+ .iter()
+ .filter(|(name, _)| !name.ends_with(".wal"))
+ .map(|(_, len)| *len)
+ .sum();
+ eprintln!(
+ "#328 nested [{shape}]: passages.fst={parent_fst} \
+ durable-index={durable} whole-dir={whole_dir}"
+ );
+ // A generous ceiling, not a golden number: the point is that the
+ // 19,200 vector components are gone, and they cannot fit in 4 KiB.
+ assert!(
+ parent_fst < 4096,
+ "[{shape}] the parent object's term dictionary must not carry the \
+ nested vector's components; `.passages.fst` = {parent_fst} B"
+ );
+ assert!(
+ !files
+ .iter()
+ .any(|(name, _)| name.ends_with(".passages.vec.fst")),
+ "[{shape}] the nested vector leaf must have no term dictionary either"
+ );
+
+ // The parent is still a real field, just without the vector in it.
+ let lexical = idx
+ .search(&make_search(json!({"match": {"body": "liquidity"}})))
+ .await
+ .unwrap();
+ assert_eq!(lexical.total.value, DOCS as u64, "[{shape}]");
+
+ // A component of the vector must not be findable through the parent.
+ let probe = format!(
+ "{}",
+ first_vector
+ .iter()
+ .copied()
+ .find(|v| *v > 0.0)
+ .expect("fixture vector has a positive component")
+ );
+ for q in [
+ json!({"match": {"passages": probe}}),
+ json!({"term": {"passages": probe}}),
+ ] {
+ let hits = idx.search(&make_search(q.clone())).await.unwrap();
+ assert_eq!(
+ hits.total.value, 0,
+ "[{shape}] a vector component must not be lexically findable \
+ through its parent object, got {} for {q}",
+ hits.total.value
+ );
+ }
+
+ // kNN on the nested field still answers from the HNSW graph.
+ let knn = idx
+ .search(
+ &parse_request(&json!({
+ "query": {"knn": {
+ "field": "passages.vec", "query_vector": first_vector, "k": 3
+ }},
+ "size": 10
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(knn.hits.len(), 3, "[{shape}] kNN must still retrieve");
+ assert_eq!(
+ knn.hits[0].id, "d0",
+ "[{shape}] nearest must be the probe doc"
+ );
+ }
+}
+
#[tokio::test]
async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {
let dir = TempDir::new().unwrap();