AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): stop the #328 companion rule from unmapping a user's own field
xerj-org862bcd75e30e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a688ec81d..ec10d219c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -155,8 +155,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Performance-- **A field mapped `dense_vector` no longer gets a full-text term dictionary**- ([#328](https://github.com/xerj-org/xerj/issues/328)). The field was fed+- **A field mapped `dense_vector` — and the undeclared `_chunks` companion the+ passage pipeline writes beside it — no longer gets a full-text term+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both+ halves are named in that headline because the companion is where most of the+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the+ mapping at all — it is a dynamic field matched by NAME, not a field anyone+ declared `dense_vector`. The field was fedthrough the lexical indexing path and given an FST + postings that no querypath can read: kNN is served by `hnsw/graph.bin`, `exists` from`_source`/doc values, and `_field_caps` and highlighting never open the@@ -178,9 +184,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0committed regression test uses, eight runs put the *whole directory* anywherebetween 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. Thetest prints both and asserts on the durable one. It reports- **3,249,323 B → 857,843–858,071 B** for that fixture (`main` side measured by- reverting `index.rs` and `memtable.rs` to `main` on the same fixture), which- is the same −73.6% at 1/17th the corpus.+ **3,249,323 B → 857,563–858,071 B** for that fixture (before-side measured by+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the+ committed test on this revision printed 857,563 / 857,623 / 857,643 /+ 857,659 / 857,787 / 857,827 / 857,859 / 857,955 B; the ceiling it asserts is+ 1,500,000 B, three orders of magnitude clear of that jitter and still 1.7 MB+ below the before-side.A `dense_vector` nested under an object mapping is covered too — itscomponents were landing in the *parent's* term dictionary, because the segment@@ -196,6 +206,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.026,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-changeindex — so a base-name-only exclusion would leave more behind than it removes.+ **Two rules, and they do not have the same strength.** The base field is+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two+ companions are excluded BY NAME, because the passage pipeline writes them into+ `_source` with no mapping entry behind them, so there is no declared type to+ read — and a name-based rule YIELDS to a declaration it could not have+ produced itself. **If you have mapped your own field called `<vector>_chunks`+ or `__xerj_passage_meta__<vector>` as `text`, `keyword`, `date`, `boolean` or+ `ip`, nothing about it changes**: it keeps its term dictionary, every query+ naming it keeps answering exactly what it answered before, and only the+ `dense_vector` beside it loses its postings. Measured on+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,+ `bool{must:[match body],filter:[term …]}` 25 → 25,+ `constant_score{filter:term}` 25 → 25,+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A dynamically mapped+ string field of that name is exempt on the same rule (dynamic mapping+ registers it as `text`) — measured 25 → 25.++ `long` and `double` are the exception, because they are what dynamic mapping+ infers from a real multi-vector and the rule cannot tell the two apart. See+ the fourth behaviour change below; that case is a hard incompatibility.+A lexical clause that NAMES a `dense_vector` (or one of its companions) is nowlowered to `match_none` at plan time rather than falling through to thestored-doc scan. That is the correctness half of the change, not only the@@ -221,10 +257,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0list keeps its lexical members, and a `nested` query is deliberately notrewritten (its field names resolve against the element, not the root).- **Three behaviour changes to know about:**-- - **A `dense_vector` (and its `_chunks` companion) now has no lexical surface- at all: every lexical leaf naming one answers 0 hits.** Before this release+ **Four behaviour changes to know about:**++ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no+ lexical surface at all: every lexical leaf naming one *outright* answers 0+ hits.** "Outright" excludes one shape, measured rather than assumed: a+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because+ resolving a pattern needs the expansion universe — dynamic fields included —+ and lowering one that would have expanded onto a real lexical field is the+ silent-zero failure this release goes out of its way to avoid. So+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls+ to the stored-doc scan, which resolves the pattern against the source's+ field names and matches the rendered decimals. Unchanged in both columns, so+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`+ all answer 0. Before this releasesuch a clause fell through to the stored-doc scan, which renders the floatarray back to decimal text and matches against *that*, so whether a shapeanswered 0 or answered most of the corpus depended entirely on whether the@@ -268,12 +315,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`max_score` for a bool that touches a vector field will see this. Plainlexical queries are unaffected; the blast radius is bools with a`dense_vector` clause in them.+ - **A `long` or `double` field named `<vector>_chunks` or+ `__xerj_passage_meta__<vector>` stops being lexically searchable.** This is+ the by-name half of the rule above, and it is the one case where it cannot+ yield, because `long`/`double` is precisely what dynamic mapping infers from+ a real multi-vector (an array of float arrays walks to its first scalar).+ Deferring to that declaration was measured and it puts+ `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture — i.e. it+ gives up the larger half of the saving above. So the name wins, and a user's+ own numeric field under that name loses its term dictionary. Measured on+ `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9+ (before → after):++ | query | before | after |+ |---|---:|---:|+ | `term {emb_chunks: 7}` | 25 | **0** |+ | `terms {emb_chunks: [7]}` | 25 | **0** |+ | `range {emb_chunks: {gte: 8}}` | 25 | **0** |+ | `bool{filter:[term emb_chunks 7]}` | 25 | **0** |+ | `bool{must:[match_all], must_not:[term emb_chunks 7]}` | 25 | **50** |+ | `terms` agg / `stats` on `emb_chunks` | 25/25, sum 400 | 25/25, sum 400 |++ The `must_not` row returns the documents the exclusion was written to+ remove, so treat this as a hard incompatibility rather than a footnote:+ **if you have your own numeric field named `<vector>_chunks`, rename it+ before upgrading.** Any other declared type (`text`, `keyword`, `date`,+ `boolean`, `ip`) is exempt and unchanged, as is a dynamically mapped string+ field of that name. Doc values are never affected on any of these paths —+ the `terms` and `stats` aggregations in the last row keep working on the+ residual case, which is how the rows above were confirmed to be a planning+ change and not data loss.- **Existing indices keep their bloat until a merge rewrites them.** This is awrite-side rule. Segments written by an earlier release keep their `.emb.*`/ `.emb_chunks.*` files, and the per-segment `fts_has_field` gate readswhichever shape it finds. Run an explicit`POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an- index that already exists.+ index that already exists. The same applies to the by-name rule above: an+ existing index with a `long`/`double` `<vector>_chunks` that is searchable+ today stops being searchable for those clauses as soon as its segments are+ rewritten — before any mapping change, and with no error.*Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,where XERJ answers `200`. This release removes bytes and adopts Lucene'sdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 0c71c71d8..e3a7ff0e7 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -30158,9 +30158,15 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {/// separate, breaking change.////// It is not answer-preserving, and the exception is a CLASS rather than a row.-/// A `dense_vector` and its `_chunks` companion end up with no lexical surface-/// at all, so every lexical leaf naming one answers 0 where the stored-doc scan-/// used to match the rendered decimal text of the float array. On the 300-doc+/// A `dense_vector` and its vector-shaped `_chunks` companion end up with no+/// lexical surface at all, so every lexical leaf naming one OUTRIGHT answers 0+/// where the stored-doc scan used to match the rendered decimal text of the+/// float array. "Outright" is load-bearing twice over: a `fields: ["emb.*"]`+/// PATTERN is not lowered (see `is_typeless_spec` below — measured 300 of 300+/// on the fixture, on the merge base and here alike), and a `<field>_chunks`+/// the user mapped as `text`/`keyword`/`date`/`boolean`/`ip` is not in+/// `typeless` at all, so nothing naming it is lowered either+/// (`memtable::lexically_typeless_fields`). On the 300-doc/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`/// 300 hits (EVERY document) → 0, `prefix {emb:"0"}` 146 → 0,/// `match_phrase_prefix {emb:"0"}` 50 → 0, numeric `term`/`terms` 1 → 0, and the@@ -30197,9 +30203,35 @@ fn lower_lexically_typeless_clauses(q: &QueryNode,typeless: &std::collections::HashSet<String>,) -> QueryNode {- // One entry of a `fields: [...]` list. `"emb^3"` carries a boost; a- // pattern (`"*"`, `"body.*"`) is an expansion, and the expansion sets in- // `search_inner` already drop vector fields, so it must stay.+ // One entry of a `fields: [...]` list. `"emb^3"` carries a boost.+ //+ // A PATTERN (`"*"`, `"body.*"`) is deliberately NOT lowered, and the honest+ // reason is that this pass cannot decide it, not that it is already+ // handled. Resolving a pattern needs the expansion universe — which+ // includes DYNAMIC fields that are in no schema — and lowering a pattern+ // that would have expanded onto a real lexical field is the same+ // silent-zero failure the declared-sibling rule in+ // `memtable::lexically_typeless_fields` exists to prevent. So the pattern+ // stays and the gap stays with it, MEASURED rather than assumed. On the+ // 300-doc × 128-dim fixture, identical on the `ca4d75a` merge base and+ // here:+ //+ // {"multi_match":{"query":"0","fields":["emb.*"]}} 300 of 300+ // {"multi_match":{"query":"liquidity","fields":["emb.*"]}} 0 of 300+ // {"multi_match":{"query":"0","fields":["emb_chunks.*"]}} 300 of 300+ // {"multi_match":{"query":"0","fields":["emb"]}} 0 of 300+ // {"multi_match":{"query":"0","fields":["emb*"]}} 0 of 300+ // {"multi_match":{"query":"0","fields":["*"]}} 0 of 300+ // {"simple_query_string":{"query":"0","fields":["emb.*"]}} 0 of 300+ //+ // `search_inner`'s expansion sets do drop vector fields, so the pattern+ // never PROJECTS onto one — but an unprojectable `MultiMatch` falls to the+ // stored-doc scan, and the scan resolves the pattern against the SOURCE's+ // field names instead, where `emb` is present and renders as decimal text.+ // That is why the first row is 300 and not 0: every rendered component+ // contains a `0`, while `liquidity` never appears in the array. Not a+ // regression (both columns agree) and not closed here; the CHANGELOG says+ // "names one outright" rather than "names one" for exactly this reason.let is_typeless_spec = |spec: &String| -> bool {let (name, _boost) = parse_field_boost(spec);!name.contains('*') && typeless.contains(name)diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rsindex c44936c8f..82d284fdf 100644--- a/engine/crates/xerj-engine/src/memtable.rs+++ b/engine/crates/xerj-engine/src/memtable.rs@@ -3926,16 +3926,39 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash/// The 300-doc × 128-dim fixture the regression tests actually run is the same/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,-/// durable index 3,249,323 B → 857,843–858,071 B over eight runs. That test+/// durable index 3,249,323 B → 857,563–858,071 B (eight runs on this revision+/// printed 857,563 / 857,623 / 857,643 / 857,659 / 857,787 / 857,827 /+/// 857,859 / 857,955). That test/// prints both totals and asserts a ceiling on the durable one, so this figure/// is enforced rather than merely published.///-/// The exclusion is *by declared type*, not by value shape: an unmapped-/// numeric array stays lexical exactly as before (the same line HNSW draws in-/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a-/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT excluded:-/// its declared payload is chunk *text* plus a vector, and only the vector-/// half is droppable — that needs a value-shape-aware rule, not a type rule.+/// TWO RULES DECIDE THIS SET, and they are stated separately because they do+/// not have the same strength. [`lexically_typeless_fields`] has the full+/// statement; the summary is:+///+/// * BY DECLARED TYPE, for the field itself: `FieldType::Vector` and nothing+/// else. Not by value shape — an unmapped numeric array stays lexical+/// exactly as before (the same line HNSW draws in+/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a+/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT+/// excluded: its declared payload is chunk *text* plus a vector, and only+/// the vector half is droppable — that needs a value-shape-aware rule, not+/// a type rule.+/// * BY NAME, for the two companions `<field>_chunks` and+/// `__xerj_passage_meta__<field>`, which the passage pipeline writes into+/// `_source` with no `FieldConfig` behind them, so there is no declared type+/// to read. This rule is the weaker of the two and YIELDS to a declaration+/// it could not have produced itself: a field mapped `text`, `keyword`,+/// `date`, `boolean`, `ip`, `chunk` or `object` under one of those names+/// keeps its full lexical surface. It does NOT yield to `long`/`double`,+/// which is exactly what dynamic mapping infers from a real multi-vector —+/// the residual, with its measured cost and its measured A/B, is on+/// [`lexically_typeless_fields`] and in `CHANGELOG.md`.+///+/// The bytes in the table above are almost all the second rule's: `emb_chunks`+/// in that fixture is a DYNAMIC field, absent from the user's mapping and+/// registered as `Double` by dynamic mapping, and it is 26,518,005 B of the+/// 39,796,380 B removed.////// Removing the postings is only half of it, and the other half is NOT the/// `*`-expansion filter. Every other member of this set is protected from the@@ -3990,8 +4013,15 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash/// A WHOLE CLASS OF ANSWERS CHANGES, and it is stated rather than buried. SEVEN/// rows above differ between `main` and this branch (nine shapes on the 300-doc/// regression fixture, which probes both companions), and they are all the same-/// thing: a `dense_vector` and its `_chunks` companion no longer have any-/// lexical surface, so every lexical leaf naming one answers 0. On `main` those+/// thing: a `dense_vector` and its UNDECLARED `_chunks` companion no longer+/// have any lexical surface, so every lexical leaf naming one OUTRIGHT answers+/// 0. Two exclusions from that sentence, both measured and both stated where+/// they live: a `fields: ["emb.*"]` PATTERN is not lowered and still answers+/// 300 of 300 for `"0"` on the fixture, exactly as on the merge base+/// (`index::lower_lexically_typeless_clauses`'s `is_typeless_spec`), and a+/// `<field>_chunks` the user mapped `text`/`keyword`/`date`/`boolean`/`ip` is+/// not in the set at all, so nothing naming it changes+/// ([`lexically_typeless_fields`]). On `main` those/// clauses reached the stored-doc scan, which renders the float array back to/// decimal text and matches against THAT — its `Term` arm matches any ELEMENT of/// a JSON array (`json_values_equal`) and its `match` arm splits on@@ -4061,17 +4091,136 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>/// its per-document float[][] shape is `LateInteractionField`, which `extends/// BinaryDocValuesField`/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`)-/// and hands its constructor straight to `super(name, encode(value))` (`:44`)+/// and hands its constructor straight to `super(name, encode(value))` (`:45`;+/// the constructor signature is `:44`)/// — doc values only, no index options, so no term dictionary can exist for it/// any more than for the single-vector `KnnFloatVectorField`/// (`KnnFloatVectorField.java:70`: vector attributes, then `freeze()`).///-/// NAME SYNTHESIS, stated rather than discovered: a user field literally named-/// `emb_chunks` alongside a `dense_vector` `emb` loses its term dictionary to-/// this collision. That is the same collision `semantic_derived_vector_fields`-/// has always accepted for `<target>_chunks`, and the same names-/// `observe_passage_chunks_in_source` reserves, so the ambiguity is the-/// engine's convention rather than something introduced here.+/// TWO RULES, and they are not the same rule. The first is BY DECLARED TYPE:+/// `FieldType::Vector` has no lexical form, so the base name is excluded+/// unconditionally. The second is BY NAME: the two companions are synthesised+/// from the base name because no USER mapping declares them — they arrive in+/// `_source` from the passage pipeline, and whatever `FieldConfig` they end up+/// with is one dynamic mapping inferred from their values. A name-based rule+/// can collide with a field the user actually mapped, so it yields to one —+/// but only where the declaration rules the collision out:+///+/// * if the schema declares `<field>_chunks` (or the sidecar) with a type a+/// MULTI-VECTOR could not have produced — `text`, `keyword`, `date`,+/// `boolean`, `ip`, `chunk`, `object` — that declaration wins and the field+/// keeps its term dictionary and its whole lexical surface. Lucene draws+/// exactly this line: whether a field gets a term dictionary is read off the+/// field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,+/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),+/// never off its name, and one name meaning two different things is an+/// outright error rather than a silent downgrade+/// (`IndexingChain.FieldSchema.setIndexOptions` → `assertSame` →+/// `raiseNotSame`, `IndexingChain.java:2213` and `:2193`);+/// * `double` / `long` / absent → the name rule applies. Not a shortcut: it is+/// what the companion ITSELF declares. `<field>_chunks` is user-supplied+/// `_source` (an array of float arrays), so dynamic mapping registers it+/// like any other unmapped key and `infer_field_type` lands on+/// `FieldType::Double`. Deferring to that declaration was measured and it+/// brings `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture+/// — the whole 26,518,005 B of the 39,796,380 B saving that lives in the+/// companion at 5,000 × 128. See [`declares_non_vector_shaped_field`] for+/// the residual this leaves and what it costs to close.+///+/// This is deliberately NOT the contract `semantic_derived_vector_fields`+/// carries: that one synthesises `<target>_chunks` unconditionally, and a+/// declared field of that name has always lost its postings to it. That+/// behaviour is untouched here — it is `main`'s, on a mapping shape this PR+/// does not introduce — but it is not a precedent this rule reuses, because it+/// fires only when an embedding config designates the target, whereas this rule+/// fires for every user-mapped `dense_vector`. Nor is+/// `observe_passage_chunks_in_source` a precedent: it reserves `<field>_chunks`+/// only when `_source` holds a NON-EMPTY ARRAY there+/// (`index.rs`'s `Some(Value::Array(chunks)) if !chunks.is_empty()`), so a+/// string or keyword value was never captured by it. Measured on+/// `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs × 16 dim, half+/// `tenant-a` — three source states, all three run in one worktree: the merge+/// base `ca4d75a` (`main` when this branch was cut), the unconditional+/// name-only rule this replaces (`e629a58`), and here.+///+/// | | `main` | name-only | here |+/// |---|---|---|---|+/// | `<seg>.emb_chunks.fst` | 54 B | **0** | 54 B |+/// | `<seg>.emb.fst` | 16,713 B | 0 | 0 |+/// | `term {emb_chunks:"tenant-a"}` | 25 | **0** | 25 |+/// | `terms {emb_chunks:["tenant-a"]}` | 25 | **0** | 25 |+/// | `bool{filter:[term emb_chunks]}` | 25 | **0** | 25 |+/// | `bool{must:[match body],filter:[term …]}` | 25 | **0** | 25 |+/// | `constant_score{filter:term emb_chunks}` | 25 | **0** | 25 |+/// | `bool{must_not:[term emb_chunks]}` | 25 | **50** | 25 |+/// | `bool{must:[match_all],must_not:[term …]}` | 25 | **50** | 25 |+/// | `terms` agg on `emb_chunks` | 25/25 | 25/25 | 25/25 |+///+/// The same schema with `emb_chunks` mapped `text` (values `tenant-a widgets` /+/// `tenant-b gadgets`, so one token is unique to each half):+///+/// | | `main` | name-only | here |+/// |---|---|---|---|+/// | `<seg>.emb_chunks.fst` | 71 B | **0** | 71 B |+/// | `match {emb_chunks:"widgets"}` | 25 | **0** | 25 |+/// | `match_phrase {emb_chunks:"tenant-a widgets"}` | 25 | **0** | 25 |+/// | `multi_match {fields:["emb_chunks"]}` | 25 | **0** | 25 |+/// | `simple_query_string {fields:["emb_chunks"]}` | 25 | **0** | 25 |+/// | `prefix {emb_chunks:"widget"}` | 25 | **0** | 25 |+/// | `bool{must:[match_all],must_not:[match …]}` | 25 | **50** | 25 |+/// | unfielded `query_string {"query":"widgets"}` | 25 | 25 | 25 |+///+/// Every ANSWER row is back to `main`'s value while `emb.fst` stays gone, which+/// is the whole claim: the vector loses its term dictionary, the declared field+/// beside it loses nothing.+///+/// The `must_not` rows are why this is a correctness rule and not a footprint+/// one: under the name-only rule the clause lowered to `match_none`, the Bool+/// fold dropped it as "excludes nothing", and the query RETURNED the 25+/// documents the exclusion was written to remove — with the aggregation proving+/// the data was present and correctly populated the whole time. The unfielded+/// `query_string` row is the self-inconsistency it produced: 25 through the+/// stored-doc scan while every fielded shape said 0.+///+/// One shape is NOT in these tables and deliberately not asserted anywhere:+/// bare `bool{must_not:[match …]}` with no `must`/`filter` answers 0 on+/// `ca4d75a` (and 50 under the name-only rule). That is a PRE-EXISTING+/// XERJ quirk with nothing to do with #328 — a control index with no vector+/// field at all, `{text body, text tags, keyword kw}`, answers+/// `bool{must_not:[match tags:"widgets"]}` = 0 and+/// `bool{must:[match_all],must_not:[match tags:"widgets"]}` = 25 there too.+/// The `match_all`-anchored form is the one the tests pin, because it is the+/// one that is correct on `main` to begin with.+///+/// THE RESIDUAL, stated in full because it is the same class and it is NOT+/// fixed: a field the user declares `long` or `double` under one of these two+/// names is still excluded by name, and the plan-time lowering still zeroes+/// every lexical clause on it. Measured on+/// `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9 —+/// merge base `ca4d75a` → here:+///+/// `term {emb_chunks:7}` 25 → **0**+/// `terms {emb_chunks:[7]}` 25 → **0**+/// `range {emb_chunks:{gte:8}}` 25 → **0**+/// `bool{filter:[term emb_chunks 7]}` 25 → **0**+/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50**+/// `terms` agg / `stats` on `emb_chunks` 25/25, sum 400 → 25/25, sum 400+///+/// It is narrower than what it replaces — the reported case, a `keyword`+/// tenant discriminator, is now byte-identical to the merge base — but it is+/// the same failure shape, and closing it needs a decision this function cannot+/// make on its own: either give the schema per-field PROVENANCE (explicit+/// mapping vs. dynamically inferred) so an explicit `double` can outrank the+/// name, or drop the by-name rule entirely and give up 67% of the saving. Both+/// are larger than #328. Documented as a behaviour change in `CHANGELOG.md`+/// and pinned by `a_numerically_declared_sibling_is_still_excluded_by_name`+/// here and the `[long]` arm of+/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in+/// `tests/integration.rs`, so it cannot drift silently.+///+/// Pinned by `a_declared_sibling_keeps_its_lexical_surface` here and by+/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in+/// `tests/integration.rs`.////// ALLOCATION: this runs once per SEARCH (`index::search_inner` calls it to/// decide whether the plan-time lowering has anything to do), and the vast@@ -4087,17 +4236,88 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<Sreturn excluded;}for field in crate::index::collect_dense_vector_fields(schema) {- excluded.insert(format!("{field}_chunks"));- excluded.insert(format!(- "{}{}",- xerj_query::executor::PASSAGE_METADATA_PREFIX,- field- ));+ // BY NAME, and only where no declaration contradicts it.+ for synthesised in [+ format!("{field}_chunks"),+ format!("{}{}", xerj_query::executor::PASSAGE_METADATA_PREFIX, field),+ ] {+ if !declares_non_vector_shaped_field(schema, &synthesised) {+ excluded.insert(synthesised);+ }+ }+ // BY DECLARED TYPE: unconditional.excluded.insert(field);}excluded}+/// Does `schema` declare `path` as a field whose type a MULTI-VECTOR could not+/// have produced — i.e. does the name-based companion rule have to yield to it?+///+/// `path` is dotted exactly as [`crate::index::collect_dense_vector_fields`]+/// emits it, so the walk descends `FieldConfig::fields` segment by segment+/// rather than using `Schema::has_field`, which only sees root names and would+/// answer `false` for every nested companion.+///+/// THE LINE IS DRAWN AT "COULD THIS DECLARATION HAVE COME FROM THE COMPANION+/// ITSELF", and it is drawn there because the companion is not always+/// undeclared. `<field>_chunks` is user-supplied `_source` (RFC #148's+/// per-document multi-vector: an array of float arrays), so DYNAMIC MAPPING+/// registers it like any other unmapped key — `infer_field_type` walks to the+/// first non-null element, twice, and lands on `FieldType::Double`+/// (`index.rs`'s `Value::Array` arm). Measured: with "any declaration wins",+/// `<seg>.emb_chunks.fst` comes back at 1,592,118 B on the 300-doc fixture and+/// the whole 67% of the saving that lives in the companion is lost, because+/// dynamic mapping had already declared it.+///+/// So:+///+/// * `Double` / `Long` — the shape `infer_field_type` produces from an array+/// of float (or integer) arrays. Indistinguishable from the companion, so+/// the name rule applies and this returns `false`. This is the residual+/// case: a user's OWN numeric field named `<vector>_chunks` loses its term+/// dictionary. It does not lose its data — doc values, `terms`/`range`+/// aggregations and sorts are untouched — and the measured cost of the+/// alternative is the entire companion saving.+/// * `Vector` — already in the set by declared type; `false` is a no-op.+/// * `Text` / `Keyword` / `Date` / `Boolean` / `Ip` / `Chunk` / `Object` — a+/// multi-vector cannot infer to any of these, so a declaration of one is+/// unambiguously the user's own field and the name rule yields. This is the+/// case that matters: a tenant or ACL discriminator named `emb_chunks` keeps+/// its whole lexical surface.+/// * absent — nothing to defer to; the name rule applies.+///+/// BOTH SPELLINGS of a nested name are resolved, because both reach the schema.+/// `put_mapping` stores a dotted path as ONE top-level `FieldConfig` named+/// `"passages.vec_chunks"`, while `es_properties_to_fields` stores it as a+/// `vec_chunks` child under a `passages` object — the same two shapes+/// `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`+/// exercises for the vector itself. The literal lookup runs first so the flat+/// spelling cannot fall through the segmented walk and be treated as undeclared.+fn declares_non_vector_shaped_field(schema: &Schema, path: &str) -> bool {+ let is_lexical = |fc: &xerj_common::types::FieldConfig| {+ !matches!(+ fc.field_type,+ FieldType::Vector | FieldType::Double | FieldType::Long+ )+ };+ if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {+ return is_lexical(field);+ }+ let mut fields: &[xerj_common::types::FieldConfig] = &schema.fields;+ let mut segments = path.split('.').peekable();+ while let Some(segment) = segments.next() {+ let Some(field) = fields.iter().find(|fc| fc.name == segment) else {+ return false;+ };+ if segments.peek().is_none() {+ return is_lexical(field);+ }+ fields = &field.fields;+ }+ false+}+/// Does this field list contain a `dense_vector` anywhere? Allocation-free.////// The recursion condition is `collect_dense_vector_fields`'s, deliberately: it@@ -4385,6 +4605,160 @@ mod lexically_typeless_fields_tests {assert!(!typeless.contains("body"), "a text field is not typeless");}+ /// The name-based half YIELDS to a declaration. A user who maps a field+ /// called `emb_chunks` beside a `dense_vector emb` gets a real keyword+ /// field, not a silently unqueryable one — the previous revision of this+ /// function excluded the name unconditionally, which took the postings away+ /// from a field the schema declares and (through the plan-time lowering)+ /// made `bool{must_not:[term emb_chunks]}` RETURN the documents it excludes.+ ///+ /// Lucene reads "does this field get a term dictionary" off the field's own+ /// `IndexableFieldType` and never off its name+ /// (`IndexingChain.java:1416`), and treats one name carrying two different+ /// index options as an error rather than a silent downgrade (`:2213`).+ #[test]+ fn a_declared_sibling_keeps_its_lexical_surface() {+ for declared in [+ FieldType::Keyword,+ FieldType::Text,+ FieldType::Date,+ FieldType::Boolean,+ FieldType::Ip,+ ] {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ schema+ .add_field(FieldConfig::new("emb_chunks", declared))+ .unwrap();+ schema+ .add_field(FieldConfig::new("__xerj_passage_meta__emb", declared))+ .unwrap();+ schema.add_field(vector("emb", 8)).unwrap();++ let typeless = lexically_typeless_fields(&schema);+ assert_eq!(+ typeless,+ std::collections::HashSet::from(["emb".to_string()]),+ "a declared `{declared:?}` sibling must keep its term dictionary"+ );+ // And the union the write path actually reads agrees.+ let excluded = fts_excluded_fields(&schema);+ assert!(excluded.contains("emb"));+ assert!(+ !excluded.contains("emb_chunks"),+ "declared sibling excluded from the inverted index: {excluded:?}"+ );+ }+ }++ /// THE RESIDUAL, pinned so it cannot drift into an accident. A `Double` or+ /// `Long` declaration is exactly what dynamic mapping produces from the+ /// real multi-vector (`infer_field_type` walks an array of float arrays to+ /// its first scalar), so the name rule cannot yield to it without losing+ /// the whole companion saving — measured at `<seg>.emb_chunks.fst`+ /// 1,592,118 B reappearing on the 300-doc fixture. Documented in the+ /// CHANGELOG as a behaviour change rather than fixed.+ #[test]+ fn a_numerically_declared_sibling_is_still_excluded_by_name() {+ for declared in [FieldType::Double, FieldType::Long] {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("emb_chunks", declared))+ .unwrap();+ schema.add_field(vector("emb", 8)).unwrap();+ assert!(+ lexically_typeless_fields(&schema).contains("emb_chunks"),+ "a `{declared:?}` declaration is the multi-vector's own inferred \+ shape and must not rescue the name"+ );+ assert!(!declares_non_vector_shaped_field(&schema, "emb_chunks"));+ }+ }++ /// A sibling declared as a `dense_vector` is excluded — by the TYPE rule,+ /// which is unconditional — so yielding to the declaration must not+ /// accidentally rescue it.+ #[test]+ fn a_sibling_declared_as_a_vector_is_still_typeless() {+ let mut schema = Schema::empty();+ schema.add_field(vector("emb_chunks", 8)).unwrap();+ schema.add_field(vector("emb", 8)).unwrap();++ let typeless = lexically_typeless_fields(&schema);+ for name in [+ "emb",+ "emb_chunks",+ "emb_chunks_chunks",+ "__xerj_passage_meta__emb",+ ] {+ assert!(typeless.contains(name), "{name} missing from {typeless:?}");+ }+ }++ /// The declaration check walks DOTTED paths, because a nested vector's+ /// companion is nested too. `Schema::has_field` only sees root names and+ /// would answer "not declared" for every one of these.+ #[test]+ fn a_declared_nested_sibling_keeps_its_lexical_surface() {+ let mut schema = Schema::empty();+ let mut parent = FieldConfig::new("passages", FieldType::Object);+ parent.fields.push(vector("vec", 4));+ parent+ .fields+ .push(FieldConfig::new("vec_chunks", FieldType::Keyword));+ schema.add_field(parent).unwrap();++ let typeless = lexically_typeless_fields(&schema);+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");+ assert!(+ !typeless.contains("passages.vec_chunks"),+ "declared nested sibling excluded: {typeless:?}"+ );+ assert!(declares_non_vector_shaped_field(+ &schema,+ "passages.vec_chunks"+ ));+ assert!(+ !declares_non_vector_shaped_field(&schema, "passages.vec"),+ "a declared vector is not a lexical declaration"+ );+ assert!(+ !declares_non_vector_shaped_field(&schema, "passages.absent"),+ "an undeclared name has no declaration to defer to"+ );+ assert!(+ !declares_non_vector_shaped_field(&schema, "passages.vec_chunks.deeper"),+ "walking past a leaf must not resolve"+ );+ }++ /// The FLAT spelling of a nested name — one top-level `FieldConfig` called+ /// `"passages.vec_chunks"`, which is what `put_mapping` produces — must+ /// resolve too. Without the literal lookup the segmented walk looks for a+ /// root `passages` object, does not find one, and treats a declared field+ /// as undeclared.+ #[test]+ fn the_flat_dotted_spelling_of_a_declared_sibling_resolves() {+ let mut schema = Schema::empty();+ schema.add_field(vector("passages.vec", 4)).unwrap();+ schema+ .add_field(FieldConfig::new("passages.vec_chunks", FieldType::Keyword))+ .unwrap();++ assert!(declares_non_vector_shaped_field(+ &schema,+ "passages.vec_chunks"+ ));+ let typeless = lexically_typeless_fields(&schema);+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");+ assert!(+ !typeless.contains("passages.vec_chunks"),+ "the flat spelling was treated as undeclared: {typeless:?}"+ );+ }+/// The exclusion is by DECLARED type, not by value shape: a numeric array/// under a non-vector mapping stays lexical exactly as before.#[test]diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rsindex 995a8cb38..e7ea4ee42 100644--- a/engine/crates/xerj-engine/tests/integration.rs+++ b/engine/crates/xerj-engine/tests/integration.rs@@ -887,6 +887,347 @@ async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary}}+/// #328, collision half — a field the user DECLARES under a companion's name+/// must keep its whole lexical surface.+///+/// `memtable::lexically_typeless_fields` synthesises `<field>_chunks` and+/// `__xerj_passage_meta__<field>` from the base name, because the passage+/// pipeline writes them into `_source` with no `FieldConfig` behind them. The+/// first revision of that rule fired UNCONDITIONALLY, so a user who mapped+/// `emb_chunks` beside a `dense_vector emb` lost the postings for a field the+/// schema declares — and then `lower_lexically_typeless_clauses` lowered every+/// lexical clause naming it to `match_none`. Measured on this fixture, merge+/// base `ca4d75a` → the name-only rule → here:+///+/// term {emb_chunks:"tenant-a"} 25 → 0 → 25+/// terms {emb_chunks:["tenant-a"]} 25 → 0 → 25+/// bool{filter:[term emb_chunks]} 25 → 0 → 25+/// bool{must:[match body],filter:[term …]} 25 → 0 → 25+/// bool{must_not:[term emb_chunks]} 25 → 50 → 25+/// terms agg on emb_chunks 25/25 → 25/25 → 25/25+///+/// The `must_not` row is the severe one and the reason this is a correctness+/// test rather than a footprint one: the query RETURNED the 25 documents the+/// exclusion was written to remove. The aggregation row proves the data was+/// present and correctly populated the whole time — the planner had declared a+/// live user field unqueryable, with no error and no mapping change. If+/// `<vector>_chunks` is a tenant or ACL discriminator that is a silent+/// data-exposure failure mode.+///+/// Lucene reads "does this field get a term dictionary" off the field's own+/// `IndexableFieldType` and never off its name+/// (`lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),+/// and one name carrying two different sets of index options is an outright+/// error rather than a silent downgrade (`IndexingChain.FieldSchema+/// .setIndexOptions` → `assertSame` → `raiseNotSame`, `:2213` / `:2193`).+///+/// THREE ARMS, and the third one asserts the OPPOSITE. `keyword` and `text`+/// yield: the sibling keeps its `.fst` and every answer above. `long` does NOT+/// yield, because `long`/`double` is exactly what dynamic mapping infers from+/// a real multi-vector (`infer_field_type` on an array of float arrays), so+/// deferring to it would put `<seg>.emb_chunks.fst` back at 1,592,118 B on the+/// 300-doc fixture and give up the larger half of #328's saving. That arm+/// therefore requires 0 hits and `must_not` = 50 — the documented residual,+/// asserted rather than left to chance, so it cannot silently widen. The+/// aggregation is checked on all three arms and buckets 25/25 on all three.+#[tokio::test]+async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one() {+ const DIMS: usize = 16;+ const DOCS: usize = 50;++ let component = |doc: usize, dim: usize| -> f64 {+ let h = (doc as u64)+ .wrapping_mul(6_364_136_223_846_793_005)+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0+ };++ for declared in [FieldType::Keyword, FieldType::Text, FieldType::Long] {+ let label = match declared {+ FieldType::Keyword => "keyword",+ FieldType::Text => "text",+ _ => "long",+ };+ // `long`/`double` is the RESIDUAL arm: it is what dynamic mapping+ // infers from a real multi-vector, so the name rule cannot yield to it+ // and this field IS still excluded. Pinned here, opposite in sign to+ // the other two arms, so the limitation cannot drift into an accident.+ let yields = !matches!(declared, FieldType::Long);+ let dir = TempDir::new().unwrap();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ // The collision: a user field named exactly like the companion the+ // vector below would otherwise synthesise.+ schema.fields.push(FieldConfig::new("emb_chunks", declared));+ let mut emb = FieldConfig::new("emb", FieldType::Vector);+ emb.options.dimensions = Some(DIMS);+ emb.options.similarity = Some("cosine".to_string());+ schema.fields.push(emb);++ let engine = make_engine(&dir);+ engine.create_index("collide", schema).unwrap();+ let idx = engine.get_index("collide").unwrap();++ let mut first_vector = Vec::new();+ for d in 0..DOCS {+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();+ if d == 0 {+ first_vector = v.clone();+ }+ // A keyword mapping gets the bare discriminator; a text mapping gets+ // it plus a token that is unique to the half, because `match` on+ // "tenant-a" analyses to [tenant, a] and would match both halves.+ let tenant = if d % 2 == 0 { "tenant-a" } else { "tenant-b" };+ let chunks_value: Value = match declared {+ FieldType::Keyword => json!(tenant),+ FieldType::Text => json!(format!(+ "{tenant} {}",+ if d % 2 == 0 { "widgets" } else { "gadgets" }+ )),+ // The residual arm: a scalar number, which is the shape+ // `infer_field_type` also produces from a real multi-vector.+ _ => json!(if d % 2 == 0 { 7 } else { 9 }),+ };+ idx.index_document(+ Some(format!("d{d}")),+ json!({+ "body": format!("quarterly liquidity evidence {d}"),+ "emb_chunks": chunks_value,+ "emb": v,+ }),+ )+ .await+ .unwrap();+ }+ idx.refresh().await.unwrap();+ idx.force_merge(1).await.unwrap();++ let mut files: Vec<(String, u64)> = Vec::new();+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {+ for entry in std::fs::read_dir(dir).unwrap() {+ let entry = entry.unwrap();+ if entry.file_type().unwrap().is_dir() {+ walk(&entry.path(), out);+ } else {+ out.push((+ entry.file_name().to_string_lossy().into_owned(),+ entry.metadata().unwrap().len(),+ ));+ }+ }+ }+ walk(dir.path(), &mut files);+ let bytes_of = |suffix: &str| -> u64 {+ files+ .iter()+ .filter(|(name, _)| name.ends_with(suffix))+ .map(|(_, len)| *len)+ .sum()+ };+ eprintln!(+ "#328 collision [{label}]: emb_chunks.fst={} emb.fst={} body.fst={}",+ bytes_of(".emb_chunks.fst"),+ bytes_of(".emb.fst"),+ bytes_of(".body.fst"),+ );+ // The DECLARED sibling keeps its term dictionary — except on the+ // residual arm, where the declaration is indistinguishable from the+ // multi-vector's own inferred type and the name rule wins.+ assert_eq!(+ files+ .iter()+ .any(|(name, _)| name.ends_with(".emb_chunks.fst")),+ yields,+ "[{label}] `.emb_chunks.fst` present should be {yields}; got {:?}",+ files+ .iter()+ .filter(|(name, _)| name.contains(".emb"))+ .collect::<Vec<_>>()+ );+ // … and the `dense_vector` beside it still loses its own.+ assert!(+ !files.iter().any(|(name, _)| name.ends_with(".emb.fst")),+ "[{label}] the dense_vector must still have no term dictionary"+ );++ // ── The answers. Every row here is 25 on the merge base `ca4d75a`; the+ // unconditional name-only rule answered 0 for all of them, and 50 for+ // the `must_not` shape below. On the two YIELDING arms this test+ // requires 25 — the merge base's own answer. On the residual `long`+ // arm it requires 0, which is what the name rule produces and what the+ // CHANGELOG documents as a behaviour change.+ let want = if yields { (DOCS / 2) as u64 } else { 0 };+ let probes: Vec<(&str, Value)> = if matches!(declared, FieldType::Keyword) {+ vec![+ ("term", json!({"term": {"emb_chunks": "tenant-a"}})),+ ("terms", json!({"terms": {"emb_chunks": ["tenant-a"]}})),+ (+ "bool-filter",+ json!({"bool": {"filter": [{"term": {"emb_chunks": "tenant-a"}}]}}),+ ),+ (+ "bool-must-plus-filter",+ json!({"bool": {+ "must": [{"match": {"body": "liquidity"}}],+ "filter": [{"term": {"emb_chunks": "tenant-a"}}]+ }}),+ ),+ (+ "constant-score",+ json!({"constant_score": {"filter": {"term": {"emb_chunks": "tenant-a"}}}}),+ ),+ ]+ } else if matches!(declared, FieldType::Long) {+ vec![+ ("term", json!({"term": {"emb_chunks": 7}})),+ ("terms", json!({"terms": {"emb_chunks": [7]}})),+ ("range", json!({"range": {"emb_chunks": {"gte": 8}}})),+ (+ "bool-filter",+ json!({"bool": {"filter": [{"term": {"emb_chunks": 7}}]}}),+ ),+ ]+ } else {+ vec![+ ("match", json!({"match": {"emb_chunks": "widgets"}})),+ (+ "match-phrase",+ json!({"match_phrase": {"emb_chunks": "tenant-a widgets"}}),+ ),+ (+ "multi-match",+ json!({"multi_match": {"query": "widgets", "fields": ["emb_chunks"]}}),+ ),+ (+ "simple-query-string",+ json!({"simple_query_string": {"query": "widgets", "fields": ["emb_chunks"]}}),+ ),+ ("prefix", json!({"prefix": {"emb_chunks": "widget"}})),+ ]+ };+ for (name, q) in &probes {+ let hits = idx.search(&make_search(q.clone())).await.unwrap();+ assert_eq!(+ hits.total.value, want,+ "[{label}] {name}: expected {want}, got {} for {q}",+ hits.total.value+ );+ }++ // The severe one, isolated: `must_not` on the declared sibling must+ // still EXCLUDE. Under the name-only rule the clause lowered to+ // `match_none`, the Bool fold dropped it as "excludes nothing", and the+ // query returned all 50 — the 25 it was written to remove included.+ //+ // The text half is anchored with an explicit `match_all` `must`,+ // because bare `bool{must_not:[match …]}` answers 0 on `origin/main`+ // too — a PRE-EXISTING quirk with nothing to do with #328 (a control+ // index carrying no vector field at all reproduces it: `{text body,+ // text tags}`, `bool{must_not:[match tags:"widgets"]}` = 0 while+ // `bool{must:[match_all],must_not:[match tags:"widgets"]}` = 25). This+ // test pins the shape that is correct on `main` to begin with, so a+ // failure here is this change's doing and nobody else's.+ let must_not_probe = if matches!(declared, FieldType::Keyword) {+ json!({"bool": {"must_not": [{"term": {"emb_chunks": "tenant-a"}}]}})+ } else if matches!(declared, FieldType::Long) {+ json!({"bool": {+ "must": [{"match_all": {}}],+ "must_not": [{"term": {"emb_chunks": 7}}]+ }})+ } else {+ json!({"bool": {+ "must": [{"match_all": {}}],+ "must_not": [{"match": {"emb_chunks": "widgets"}}]+ }})+ };+ let excluded = idx+ .search(&make_search(must_not_probe.clone()))+ .await+ .unwrap();+ // On the yielding arms the exclusion still works and returns exactly the+ // other half. On the residual arm it does NOT: the clause lowers to+ // `match_none`, the Bool fold drops it as "excludes nothing", and all+ // 50 come back — the 25 the exclusion was written to remove included.+ // That is the behaviour change the CHANGELOG documents, asserted here+ // so it cannot happen by accident on a shape nobody looked at.+ assert_eq!(+ excluded.total.value,+ if yields {+ (DOCS / 2) as u64+ } else {+ DOCS as u64+ },+ "[{label}] must_not: got {} for {must_not_probe}",+ excluded.total.value+ );+ if yields {+ for hit in &excluded.hits {+ let n: usize = hit.id.trim_start_matches('d').parse().unwrap();+ assert_eq!(+ n % 2,+ 1,+ "[{label}] must_not returned an excluded document: {}",+ hit.id+ );+ }+ }++ // Doc values were never in question — this row answers 25/25 on ALL+ // THREE arms, including the residual one where every lexical clause+ // says 0. That is what proves the data is present and correctly+ // populated, and that it is the planner, not the storage, that changed.+ let agg = idx+ .search(+ &parse_request(&json!({+ "size": 0,+ "aggs": {"by_tenant": {"terms": {"field": "emb_chunks", "size": 10}}}+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ let buckets = agg.aggs.as_ref().expect("aggs present")["by_tenant"]["buckets"]+ .as_array()+ .unwrap()+ .clone();+ assert!(+ buckets.iter().any(|b| b["doc_count"] == 25),+ "[{label}] terms agg on the declared sibling must still bucket it, \+ got {buckets:?}"+ );++ // The vector itself is unchanged by the yield: every lexical leaf+ // naming it still answers 0, and kNN still answers from the graph.+ for q in [+ json!({"term": {"emb": "0"}}),+ json!({"wildcard": {"emb": "0*"}}),+ json!({"multi_match": {"query": "0", "fields": ["emb"]}}),+ ] {+ let hits = idx.search(&make_search(q.clone())).await.unwrap();+ assert_eq!(+ hits.total.value, 0,+ "[{label}] a lexical clause on the dense_vector must answer 0, \+ got {} for {q}",+ hits.total.value+ );+ }+ let knn = idx+ .search(+ &parse_request(&json!({+ "query": {"knn": {"field": "emb", "query_vector": first_vector, "k": 3}},+ "size": 10+ }))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(knn.hits.len(), 3, "[{label}] kNN must still retrieve");+ assert_eq!(knn.hits[0].id, "d0", "[{label}] nearest must be the probe");+ }+}+#[tokio::test]async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {let dir = TempDir::new().unwrap();