Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): stop the #328 companion rule from unmapping a user's own field

xerj-org862bcd75e30e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a688ec81d..ec10d219c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -155,8 +155,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Performance
-- **A field mapped `dense_vector` no longer gets a full-text term dictionary**
- ([#328](https://github.com/xerj-org/xerj/issues/328)). The field was fed
+- **A field mapped `dense_vector` — and the undeclared `_chunks` companion the
+ passage pipeline writes beside it — no longer gets a full-text term
+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both
+ halves are named in that headline because the companion is where most of the
+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is
+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the
+ mapping at all — it is a dynamic field matched by NAME, not a field anyone
+ declared `dense_vector`. The field was fed
through the lexical indexing path and given an FST + postings that no query
path can read: kNN is served by `hnsw/graph.bin`, `exists` from
`_source`/doc values, and `_field_caps` and highlighting never open the
@@ -178,9 +184,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
committed regression test uses, eight runs put the *whole directory* anywhere
between 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. The
test prints both and asserts on the durable one. It reports
- **3,249,323 B → 857,843–858,071 B** for that fixture (`main` side measured by
- reverting `index.rs` and `memtable.rs` to `main` on the same fixture), which
- is the same −73.6% at 1/17th the corpus.
+ **3,249,323 B → 857,563–858,071 B** for that fixture (before-side measured by
+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same
+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the
+ committed test on this revision printed 857,563 / 857,623 / 857,643 /
+ 857,659 / 857,787 / 857,827 / 857,859 / 857,955 B; the ceiling it asserts is
+ 1,500,000 B, three orders of magnitude clear of that jitter and still 1.7 MB
+ below the before-side.
A `dense_vector` nested under an object mapping is covered too — its
components were landing in the *parent's* term dictionary, because the segment
@@ -196,6 +206,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change
index — so a base-name-only exclusion would leave more behind than it removes.
+ **Two rules, and they do not have the same strength.** The base field is
+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two
+ companions are excluded BY NAME, because the passage pipeline writes them into
+ `_source` with no mapping entry behind them, so there is no declared type to
+ read — and a name-based rule YIELDS to a declaration it could not have
+ produced itself. **If you have mapped your own field called `<vector>_chunks`
+ or `__xerj_passage_meta__<vector>` as `text`, `keyword`, `date`, `boolean` or
+ `ip`, nothing about it changes**: it keeps its term dictionary, every query
+ naming it keeps answering exactly what it answered before, and only the
+ `dense_vector` beside it loses its postings. Measured on
+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`
+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,
+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,
+ `bool{must:[match body],filter:[term …]}` 25 → 25,
+ `constant_score{filter:term}` 25 → 25,
+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`
+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:
+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`
+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A dynamically mapped
+ string field of that name is exempt on the same rule (dynamic mapping
+ registers it as `text`) — measured 25 → 25.
+
+ `long` and `double` are the exception, because they are what dynamic mapping
+ infers from a real multi-vector and the rule cannot tell the two apart. See
+ the fourth behaviour change below; that case is a hard incompatibility.
+
A lexical clause that NAMES a `dense_vector` (or one of its companions) is now
lowered to `match_none` at plan time rather than falling through to the
stored-doc scan. That is the correctness half of the change, not only the
@@ -221,10 +257,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
list keeps its lexical members, and a `nested` query is deliberately not
rewritten (its field names resolve against the element, not the root).
- **Three behaviour changes to know about:**
-
- - **A `dense_vector` (and its `_chunks` companion) now has no lexical surface
- at all: every lexical leaf naming one answers 0 hits.** Before this release
+ **Four behaviour changes to know about:**
+
+ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no
+ lexical surface at all: every lexical leaf naming one *outright* answers 0
+ hits.** "Outright" excludes one shape, measured rather than assumed: a
+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because
+ resolving a pattern needs the expansion universe — dynamic fields included —
+ and lowering one that would have expanded onto a real lexical field is the
+ silent-zero failure this release goes out of its way to avoid. So
+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300
+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls
+ to the stored-doc scan, which resolves the pattern against the source's
+ field names and matches the rendered decimals. Unchanged in both columns, so
+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`
+ all answer 0. Before this release
such a clause fell through to the stored-doc scan, which renders the float
array back to decimal text and matches against *that*, so whether a shape
answered 0 or answered most of the corpus depended entirely on whether the
@@ -268,12 +315,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`max_score` for a bool that touches a vector field will see this. Plain
lexical queries are unaffected; the blast radius is bools with a
`dense_vector` clause in them.
+ - **A `long` or `double` field named `<vector>_chunks` or
+ `__xerj_passage_meta__<vector>` stops being lexically searchable.** This is
+ the by-name half of the rule above, and it is the one case where it cannot
+ yield, because `long`/`double` is precisely what dynamic mapping infers from
+ a real multi-vector (an array of float arrays walks to its first scalar).
+ Deferring to that declaration was measured and it puts
+ `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture — i.e. it
+ gives up the larger half of the saving above. So the name wins, and a user's
+ own numeric field under that name loses its term dictionary. Measured on
+ `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9
+ (before → after):
+
+ | query | before | after |
+ |---|---:|---:|
+ | `term {emb_chunks: 7}` | 25 | **0** |
+ | `terms {emb_chunks: [7]}` | 25 | **0** |
+ | `range {emb_chunks: {gte: 8}}` | 25 | **0** |
+ | `bool{filter:[term emb_chunks 7]}` | 25 | **0** |
+ | `bool{must:[match_all], must_not:[term emb_chunks 7]}` | 25 | **50** |
+ | `terms` agg / `stats` on `emb_chunks` | 25/25, sum 400 | 25/25, sum 400 |
+
+ The `must_not` row returns the documents the exclusion was written to
+ remove, so treat this as a hard incompatibility rather than a footnote:
+ **if you have your own numeric field named `<vector>_chunks`, rename it
+ before upgrading.** Any other declared type (`text`, `keyword`, `date`,
+ `boolean`, `ip`) is exempt and unchanged, as is a dynamically mapped string
+ field of that name. Doc values are never affected on any of these paths —
+ the `terms` and `stats` aggregations in the last row keep working on the
+ residual case, which is how the rows above were confirmed to be a planning
+ change and not data loss.
- **Existing indices keep their bloat until a merge rewrites them.** This is a
write-side rule. Segments written by an earlier release keep their `.emb.*`
/ `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads
whichever shape it finds. Run an explicit
`POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an
- index that already exists.
+ index that already exists. The same applies to the by-name rule above: an
+ existing index with a `long`/`double` `<vector>_chunks` that is searchable
+ today stops being searchable for those clauses as soon as its segments are
+ rewritten — before any mapping change, and with no error.
*Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,
where XERJ answers `200`. This release removes bytes and adopts Lucene's
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 0c71c71d8..e3a7ff0e7 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -30158,9 +30158,15 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {
/// separate, breaking change.
///
/// It is not answer-preserving, and the exception is a CLASS rather than a row.
-/// A `dense_vector` and its `_chunks` companion end up with no lexical surface
-/// at all, so every lexical leaf naming one answers 0 where the stored-doc scan
-/// used to match the rendered decimal text of the float array. On the 300-doc
+/// A `dense_vector` and its vector-shaped `_chunks` companion end up with no
+/// lexical surface at all, so every lexical leaf naming one OUTRIGHT answers 0
+/// where the stored-doc scan used to match the rendered decimal text of the
+/// float array. "Outright" is load-bearing twice over: a `fields: ["emb.*"]`
+/// PATTERN is not lowered (see `is_typeless_spec` below — measured 300 of 300
+/// on the fixture, on the merge base and here alike), and a `<field>_chunks`
+/// the user mapped as `text`/`keyword`/`date`/`boolean`/`ip` is not in
+/// `typeless` at all, so nothing naming it is lowered either
+/// (`memtable::lexically_typeless_fields`). On the 300-doc
/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`
/// 300 hits (EVERY document) → 0, `prefix {emb:"0"}` 146 → 0,
/// `match_phrase_prefix {emb:"0"}` 50 → 0, numeric `term`/`terms` 1 → 0, and the
@@ -30197,9 +30203,35 @@ fn lower_lexically_typeless_clauses(
q: &QueryNode,
typeless: &std::collections::HashSet<String>,
) -> QueryNode {
- // One entry of a `fields: [...]` list. `"emb^3"` carries a boost; a
- // pattern (`"*"`, `"body.*"`) is an expansion, and the expansion sets in
- // `search_inner` already drop vector fields, so it must stay.
+ // One entry of a `fields: [...]` list. `"emb^3"` carries a boost.
+ //
+ // A PATTERN (`"*"`, `"body.*"`) is deliberately NOT lowered, and the honest
+ // reason is that this pass cannot decide it, not that it is already
+ // handled. Resolving a pattern needs the expansion universe — which
+ // includes DYNAMIC fields that are in no schema — and lowering a pattern
+ // that would have expanded onto a real lexical field is the same
+ // silent-zero failure the declared-sibling rule in
+ // `memtable::lexically_typeless_fields` exists to prevent. So the pattern
+ // stays and the gap stays with it, MEASURED rather than assumed. On the
+ // 300-doc × 128-dim fixture, identical on the `ca4d75a` merge base and
+ // here:
+ //
+ // {"multi_match":{"query":"0","fields":["emb.*"]}} 300 of 300
+ // {"multi_match":{"query":"liquidity","fields":["emb.*"]}} 0 of 300
+ // {"multi_match":{"query":"0","fields":["emb_chunks.*"]}} 300 of 300
+ // {"multi_match":{"query":"0","fields":["emb"]}} 0 of 300
+ // {"multi_match":{"query":"0","fields":["emb*"]}} 0 of 300
+ // {"multi_match":{"query":"0","fields":["*"]}} 0 of 300
+ // {"simple_query_string":{"query":"0","fields":["emb.*"]}} 0 of 300
+ //
+ // `search_inner`'s expansion sets do drop vector fields, so the pattern
+ // never PROJECTS onto one — but an unprojectable `MultiMatch` falls to the
+ // stored-doc scan, and the scan resolves the pattern against the SOURCE's
+ // field names instead, where `emb` is present and renders as decimal text.
+ // That is why the first row is 300 and not 0: every rendered component
+ // contains a `0`, while `liquidity` never appears in the array. Not a
+ // regression (both columns agree) and not closed here; the CHANGELOG says
+ // "names one outright" rather than "names one" for exactly this reason.
let is_typeless_spec = |spec: &String| -> bool {
let (name, _boost) = parse_field_boost(spec);
!name.contains('*') && typeless.contains(name)
diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rs
index c44936c8f..82d284fdf 100644
--- a/engine/crates/xerj-engine/src/memtable.rs
+++ b/engine/crates/xerj-engine/src/memtable.rs
@@ -3926,16 +3926,39 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
/// The 300-doc × 128-dim fixture the regression tests actually run is the same
/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,
/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,
-/// durable index 3,249,323 B → 857,843–858,071 B over eight runs. That test
+/// durable index 3,249,323 B → 857,563–858,071 B (eight runs on this revision
+/// printed 857,563 / 857,623 / 857,643 / 857,659 / 857,787 / 857,827 /
+/// 857,859 / 857,955). That test
/// prints both totals and asserts a ceiling on the durable one, so this figure
/// is enforced rather than merely published.
///
-/// The exclusion is *by declared type*, not by value shape: an unmapped
-/// numeric array stays lexical exactly as before (the same line HNSW draws in
-/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a
-/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT excluded:
-/// its declared payload is chunk *text* plus a vector, and only the vector
-/// half is droppable — that needs a value-shape-aware rule, not a type rule.
+/// TWO RULES DECIDE THIS SET, and they are stated separately because they do
+/// not have the same strength. [`lexically_typeless_fields`] has the full
+/// statement; the summary is:
+///
+/// * BY DECLARED TYPE, for the field itself: `FieldType::Vector` and nothing
+/// else. Not by value shape — an unmapped numeric array stays lexical
+/// exactly as before (the same line HNSW draws in
+/// `load_hnsw_artifacts_sync`, which refuses a graph pinned to a
+/// non-`dense_vector` field). `FieldType::Chunk` is deliberately NOT
+/// excluded: its declared payload is chunk *text* plus a vector, and only
+/// the vector half is droppable — that needs a value-shape-aware rule, not
+/// a type rule.
+/// * BY NAME, for the two companions `<field>_chunks` and
+/// `__xerj_passage_meta__<field>`, which the passage pipeline writes into
+/// `_source` with no `FieldConfig` behind them, so there is no declared type
+/// to read. This rule is the weaker of the two and YIELDS to a declaration
+/// it could not have produced itself: a field mapped `text`, `keyword`,
+/// `date`, `boolean`, `ip`, `chunk` or `object` under one of those names
+/// keeps its full lexical surface. It does NOT yield to `long`/`double`,
+/// which is exactly what dynamic mapping infers from a real multi-vector —
+/// the residual, with its measured cost and its measured A/B, is on
+/// [`lexically_typeless_fields`] and in `CHANGELOG.md`.
+///
+/// The bytes in the table above are almost all the second rule's: `emb_chunks`
+/// in that fixture is a DYNAMIC field, absent from the user's mapping and
+/// registered as `Double` by dynamic mapping, and it is 26,518,005 B of the
+/// 39,796,380 B removed.
///
/// Removing the postings is only half of it, and the other half is NOT the
/// `*`-expansion filter. Every other member of this set is protected from the
@@ -3990,8 +4013,15 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
/// A WHOLE CLASS OF ANSWERS CHANGES, and it is stated rather than buried. SEVEN
/// rows above differ between `main` and this branch (nine shapes on the 300-doc
/// regression fixture, which probes both companions), and they are all the same
-/// thing: a `dense_vector` and its `_chunks` companion no longer have any
-/// lexical surface, so every lexical leaf naming one answers 0. On `main` those
+/// thing: a `dense_vector` and its UNDECLARED `_chunks` companion no longer
+/// have any lexical surface, so every lexical leaf naming one OUTRIGHT answers
+/// 0. Two exclusions from that sentence, both measured and both stated where
+/// they live: a `fields: ["emb.*"]` PATTERN is not lowered and still answers
+/// 300 of 300 for `"0"` on the fixture, exactly as on the merge base
+/// (`index::lower_lexically_typeless_clauses`'s `is_typeless_spec`), and a
+/// `<field>_chunks` the user mapped `text`/`keyword`/`date`/`boolean`/`ip` is
+/// not in the set at all, so nothing naming it changes
+/// ([`lexically_typeless_fields`]). On `main` those
/// clauses reached the stored-doc scan, which renders the float array back to
/// decimal text and matches against THAT — its `Term` arm matches any ELEMENT of
/// a JSON array (`json_values_equal`) and its `match` arm splits on
@@ -4061,17 +4091,136 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>
/// its per-document float[][] shape is `LateInteractionField`, which `extends
/// BinaryDocValuesField`
/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`)
-/// and hands its constructor straight to `super(name, encode(value))` (`:44`)
+/// and hands its constructor straight to `super(name, encode(value))` (`:45`;
+/// the constructor signature is `:44`)
/// — doc values only, no index options, so no term dictionary can exist for it
/// any more than for the single-vector `KnnFloatVectorField`
/// (`KnnFloatVectorField.java:70`: vector attributes, then `freeze()`).
///
-/// NAME SYNTHESIS, stated rather than discovered: a user field literally named
-/// `emb_chunks` alongside a `dense_vector` `emb` loses its term dictionary to
-/// this collision. That is the same collision `semantic_derived_vector_fields`
-/// has always accepted for `<target>_chunks`, and the same names
-/// `observe_passage_chunks_in_source` reserves, so the ambiguity is the
-/// engine's convention rather than something introduced here.
+/// TWO RULES, and they are not the same rule. The first is BY DECLARED TYPE:
+/// `FieldType::Vector` has no lexical form, so the base name is excluded
+/// unconditionally. The second is BY NAME: the two companions are synthesised
+/// from the base name because no USER mapping declares them — they arrive in
+/// `_source` from the passage pipeline, and whatever `FieldConfig` they end up
+/// with is one dynamic mapping inferred from their values. A name-based rule
+/// can collide with a field the user actually mapped, so it yields to one —
+/// but only where the declaration rules the collision out:
+///
+/// * if the schema declares `<field>_chunks` (or the sidecar) with a type a
+/// MULTI-VECTOR could not have produced — `text`, `keyword`, `date`,
+/// `boolean`, `ip`, `chunk`, `object` — that declaration wins and the field
+/// keeps its term dictionary and its whole lexical surface. Lucene draws
+/// exactly this line: whether a field gets a term dictionary is read off the
+/// field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,
+/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),
+/// never off its name, and one name meaning two different things is an
+/// outright error rather than a silent downgrade
+/// (`IndexingChain.FieldSchema.setIndexOptions` → `assertSame` →
+/// `raiseNotSame`, `IndexingChain.java:2213` and `:2193`);
+/// * `double` / `long` / absent → the name rule applies. Not a shortcut: it is
+/// what the companion ITSELF declares. `<field>_chunks` is user-supplied
+/// `_source` (an array of float arrays), so dynamic mapping registers it
+/// like any other unmapped key and `infer_field_type` lands on
+/// `FieldType::Double`. Deferring to that declaration was measured and it
+/// brings `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture
+/// — the whole 26,518,005 B of the 39,796,380 B saving that lives in the
+/// companion at 5,000 × 128. See [`declares_non_vector_shaped_field`] for
+/// the residual this leaves and what it costs to close.
+///
+/// This is deliberately NOT the contract `semantic_derived_vector_fields`
+/// carries: that one synthesises `<target>_chunks` unconditionally, and a
+/// declared field of that name has always lost its postings to it. That
+/// behaviour is untouched here — it is `main`'s, on a mapping shape this PR
+/// does not introduce — but it is not a precedent this rule reuses, because it
+/// fires only when an embedding config designates the target, whereas this rule
+/// fires for every user-mapped `dense_vector`. Nor is
+/// `observe_passage_chunks_in_source` a precedent: it reserves `<field>_chunks`
+/// only when `_source` holds a NON-EMPTY ARRAY there
+/// (`index.rs`'s `Some(Value::Array(chunks)) if !chunks.is_empty()`), so a
+/// string or keyword value was never captured by it. Measured on
+/// `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs × 16 dim, half
+/// `tenant-a` — three source states, all three run in one worktree: the merge
+/// base `ca4d75a` (`main` when this branch was cut), the unconditional
+/// name-only rule this replaces (`e629a58`), and here.
+///
+/// | | `main` | name-only | here |
+/// |---|---|---|---|
+/// | `<seg>.emb_chunks.fst` | 54 B | **0** | 54 B |
+/// | `<seg>.emb.fst` | 16,713 B | 0 | 0 |
+/// | `term {emb_chunks:"tenant-a"}` | 25 | **0** | 25 |
+/// | `terms {emb_chunks:["tenant-a"]}` | 25 | **0** | 25 |
+/// | `bool{filter:[term emb_chunks]}` | 25 | **0** | 25 |
+/// | `bool{must:[match body],filter:[term …]}` | 25 | **0** | 25 |
+/// | `constant_score{filter:term emb_chunks}` | 25 | **0** | 25 |
+/// | `bool{must_not:[term emb_chunks]}` | 25 | **50** | 25 |
+/// | `bool{must:[match_all],must_not:[term …]}` | 25 | **50** | 25 |
+/// | `terms` agg on `emb_chunks` | 25/25 | 25/25 | 25/25 |
+///
+/// The same schema with `emb_chunks` mapped `text` (values `tenant-a widgets` /
+/// `tenant-b gadgets`, so one token is unique to each half):
+///
+/// | | `main` | name-only | here |
+/// |---|---|---|---|
+/// | `<seg>.emb_chunks.fst` | 71 B | **0** | 71 B |
+/// | `match {emb_chunks:"widgets"}` | 25 | **0** | 25 |
+/// | `match_phrase {emb_chunks:"tenant-a widgets"}` | 25 | **0** | 25 |
+/// | `multi_match {fields:["emb_chunks"]}` | 25 | **0** | 25 |
+/// | `simple_query_string {fields:["emb_chunks"]}` | 25 | **0** | 25 |
+/// | `prefix {emb_chunks:"widget"}` | 25 | **0** | 25 |
+/// | `bool{must:[match_all],must_not:[match …]}` | 25 | **50** | 25 |
+/// | unfielded `query_string {"query":"widgets"}` | 25 | 25 | 25 |
+///
+/// Every ANSWER row is back to `main`'s value while `emb.fst` stays gone, which
+/// is the whole claim: the vector loses its term dictionary, the declared field
+/// beside it loses nothing.
+///
+/// The `must_not` rows are why this is a correctness rule and not a footprint
+/// one: under the name-only rule the clause lowered to `match_none`, the Bool
+/// fold dropped it as "excludes nothing", and the query RETURNED the 25
+/// documents the exclusion was written to remove — with the aggregation proving
+/// the data was present and correctly populated the whole time. The unfielded
+/// `query_string` row is the self-inconsistency it produced: 25 through the
+/// stored-doc scan while every fielded shape said 0.
+///
+/// One shape is NOT in these tables and deliberately not asserted anywhere:
+/// bare `bool{must_not:[match …]}` with no `must`/`filter` answers 0 on
+/// `ca4d75a` (and 50 under the name-only rule). That is a PRE-EXISTING
+/// XERJ quirk with nothing to do with #328 — a control index with no vector
+/// field at all, `{text body, text tags, keyword kw}`, answers
+/// `bool{must_not:[match tags:"widgets"]}` = 0 and
+/// `bool{must:[match_all],must_not:[match tags:"widgets"]}` = 25 there too.
+/// The `match_all`-anchored form is the one the tests pin, because it is the
+/// one that is correct on `main` to begin with.
+///
+/// THE RESIDUAL, stated in full because it is the same class and it is NOT
+/// fixed: a field the user declares `long` or `double` under one of these two
+/// names is still excluded by name, and the plan-time lowering still zeroes
+/// every lexical clause on it. Measured on
+/// `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9 —
+/// merge base `ca4d75a` → here:
+///
+/// `term {emb_chunks:7}` 25 → **0**
+/// `terms {emb_chunks:[7]}` 25 → **0**
+/// `range {emb_chunks:{gte:8}}` 25 → **0**
+/// `bool{filter:[term emb_chunks 7]}` 25 → **0**
+/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50**
+/// `terms` agg / `stats` on `emb_chunks` 25/25, sum 400 → 25/25, sum 400
+///
+/// It is narrower than what it replaces — the reported case, a `keyword`
+/// tenant discriminator, is now byte-identical to the merge base — but it is
+/// the same failure shape, and closing it needs a decision this function cannot
+/// make on its own: either give the schema per-field PROVENANCE (explicit
+/// mapping vs. dynamically inferred) so an explicit `double` can outrank the
+/// name, or drop the by-name rule entirely and give up 67% of the saving. Both
+/// are larger than #328. Documented as a behaviour change in `CHANGELOG.md`
+/// and pinned by `a_numerically_declared_sibling_is_still_excluded_by_name`
+/// here and the `[long]` arm of
+/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in
+/// `tests/integration.rs`, so it cannot drift silently.
+///
+/// Pinned by `a_declared_sibling_keeps_its_lexical_surface` here and by
+/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in
+/// `tests/integration.rs`.
///
/// ALLOCATION: this runs once per SEARCH (`index::search_inner` calls it to
/// decide whether the plan-time lowering has anything to do), and the vast
@@ -4087,17 +4236,88 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S
return excluded;
}
for field in crate::index::collect_dense_vector_fields(schema) {
- excluded.insert(format!("{field}_chunks"));
- excluded.insert(format!(
- "{}{}",
- xerj_query::executor::PASSAGE_METADATA_PREFIX,
- field
- ));
+ // BY NAME, and only where no declaration contradicts it.
+ for synthesised in [
+ format!("{field}_chunks"),
+ format!("{}{}", xerj_query::executor::PASSAGE_METADATA_PREFIX, field),
+ ] {
+ if !declares_non_vector_shaped_field(schema, &synthesised) {
+ excluded.insert(synthesised);
+ }
+ }
+ // BY DECLARED TYPE: unconditional.
excluded.insert(field);
}
excluded
}
+/// Does `schema` declare `path` as a field whose type a MULTI-VECTOR could not
+/// have produced — i.e. does the name-based companion rule have to yield to it?
+///
+/// `path` is dotted exactly as [`crate::index::collect_dense_vector_fields`]
+/// emits it, so the walk descends `FieldConfig::fields` segment by segment
+/// rather than using `Schema::has_field`, which only sees root names and would
+/// answer `false` for every nested companion.
+///
+/// THE LINE IS DRAWN AT "COULD THIS DECLARATION HAVE COME FROM THE COMPANION
+/// ITSELF", and it is drawn there because the companion is not always
+/// undeclared. `<field>_chunks` is user-supplied `_source` (RFC #148's
+/// per-document multi-vector: an array of float arrays), so DYNAMIC MAPPING
+/// registers it like any other unmapped key — `infer_field_type` walks to the
+/// first non-null element, twice, and lands on `FieldType::Double`
+/// (`index.rs`'s `Value::Array` arm). Measured: with "any declaration wins",
+/// `<seg>.emb_chunks.fst` comes back at 1,592,118 B on the 300-doc fixture and
+/// the whole 67% of the saving that lives in the companion is lost, because
+/// dynamic mapping had already declared it.
+///
+/// So:
+///
+/// * `Double` / `Long` — the shape `infer_field_type` produces from an array
+/// of float (or integer) arrays. Indistinguishable from the companion, so
+/// the name rule applies and this returns `false`. This is the residual
+/// case: a user's OWN numeric field named `<vector>_chunks` loses its term
+/// dictionary. It does not lose its data — doc values, `terms`/`range`
+/// aggregations and sorts are untouched — and the measured cost of the
+/// alternative is the entire companion saving.
+/// * `Vector` — already in the set by declared type; `false` is a no-op.
+/// * `Text` / `Keyword` / `Date` / `Boolean` / `Ip` / `Chunk` / `Object` — a
+/// multi-vector cannot infer to any of these, so a declaration of one is
+/// unambiguously the user's own field and the name rule yields. This is the
+/// case that matters: a tenant or ACL discriminator named `emb_chunks` keeps
+/// its whole lexical surface.
+/// * absent — nothing to defer to; the name rule applies.
+///
+/// BOTH SPELLINGS of a nested name are resolved, because both reach the schema.
+/// `put_mapping` stores a dotted path as ONE top-level `FieldConfig` named
+/// `"passages.vec_chunks"`, while `es_properties_to_fields` stores it as a
+/// `vec_chunks` child under a `passages` object — the same two shapes
+/// `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`
+/// exercises for the vector itself. The literal lookup runs first so the flat
+/// spelling cannot fall through the segmented walk and be treated as undeclared.
+fn declares_non_vector_shaped_field(schema: &Schema, path: &str) -> bool {
+ let is_lexical = |fc: &xerj_common::types::FieldConfig| {
+ !matches!(
+ fc.field_type,
+ FieldType::Vector | FieldType::Double | FieldType::Long
+ )
+ };
+ if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {
+ return is_lexical(field);
+ }
+ let mut fields: &[xerj_common::types::FieldConfig] = &schema.fields;
+ let mut segments = path.split('.').peekable();
+ while let Some(segment) = segments.next() {
+ let Some(field) = fields.iter().find(|fc| fc.name == segment) else {
+ return false;
+ };
+ if segments.peek().is_none() {
+ return is_lexical(field);
+ }
+ fields = &field.fields;
+ }
+ false
+}
+
/// Does this field list contain a `dense_vector` anywhere? Allocation-free.
///
/// The recursion condition is `collect_dense_vector_fields`'s, deliberately: it
@@ -4385,6 +4605,160 @@ mod lexically_typeless_fields_tests {
assert!(!typeless.contains("body"), "a text field is not typeless");
}
+ /// The name-based half YIELDS to a declaration. A user who maps a field
+ /// called `emb_chunks` beside a `dense_vector emb` gets a real keyword
+ /// field, not a silently unqueryable one — the previous revision of this
+ /// function excluded the name unconditionally, which took the postings away
+ /// from a field the schema declares and (through the plan-time lowering)
+ /// made `bool{must_not:[term emb_chunks]}` RETURN the documents it excludes.
+ ///
+ /// Lucene reads "does this field get a term dictionary" off the field's own
+ /// `IndexableFieldType` and never off its name
+ /// (`IndexingChain.java:1416`), and treats one name carrying two different
+ /// index options as an error rather than a silent downgrade (`:2213`).
+ #[test]
+ fn a_declared_sibling_keeps_its_lexical_surface() {
+ for declared in [
+ FieldType::Keyword,
+ FieldType::Text,
+ FieldType::Date,
+ FieldType::Boolean,
+ FieldType::Ip,
+ ] {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ schema
+ .add_field(FieldConfig::new("emb_chunks", declared))
+ .unwrap();
+ schema
+ .add_field(FieldConfig::new("__xerj_passage_meta__emb", declared))
+ .unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+
+ let typeless = lexically_typeless_fields(&schema);
+ assert_eq!(
+ typeless,
+ std::collections::HashSet::from(["emb".to_string()]),
+ "a declared `{declared:?}` sibling must keep its term dictionary"
+ );
+ // And the union the write path actually reads agrees.
+ let excluded = fts_excluded_fields(&schema);
+ assert!(excluded.contains("emb"));
+ assert!(
+ !excluded.contains("emb_chunks"),
+ "declared sibling excluded from the inverted index: {excluded:?}"
+ );
+ }
+ }
+
+ /// THE RESIDUAL, pinned so it cannot drift into an accident. A `Double` or
+ /// `Long` declaration is exactly what dynamic mapping produces from the
+ /// real multi-vector (`infer_field_type` walks an array of float arrays to
+ /// its first scalar), so the name rule cannot yield to it without losing
+ /// the whole companion saving — measured at `<seg>.emb_chunks.fst`
+ /// 1,592,118 B reappearing on the 300-doc fixture. Documented in the
+ /// CHANGELOG as a behaviour change rather than fixed.
+ #[test]
+ fn a_numerically_declared_sibling_is_still_excluded_by_name() {
+ for declared in [FieldType::Double, FieldType::Long] {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("emb_chunks", declared))
+ .unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+ assert!(
+ lexically_typeless_fields(&schema).contains("emb_chunks"),
+ "a `{declared:?}` declaration is the multi-vector's own inferred \
+ shape and must not rescue the name"
+ );
+ assert!(!declares_non_vector_shaped_field(&schema, "emb_chunks"));
+ }
+ }
+
+ /// A sibling declared as a `dense_vector` is excluded — by the TYPE rule,
+ /// which is unconditional — so yielding to the declaration must not
+ /// accidentally rescue it.
+ #[test]
+ fn a_sibling_declared_as_a_vector_is_still_typeless() {
+ let mut schema = Schema::empty();
+ schema.add_field(vector("emb_chunks", 8)).unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+
+ let typeless = lexically_typeless_fields(&schema);
+ for name in [
+ "emb",
+ "emb_chunks",
+ "emb_chunks_chunks",
+ "__xerj_passage_meta__emb",
+ ] {
+ assert!(typeless.contains(name), "{name} missing from {typeless:?}");
+ }
+ }
+
+ /// The declaration check walks DOTTED paths, because a nested vector's
+ /// companion is nested too. `Schema::has_field` only sees root names and
+ /// would answer "not declared" for every one of these.
+ #[test]
+ fn a_declared_nested_sibling_keeps_its_lexical_surface() {
+ let mut schema = Schema::empty();
+ let mut parent = FieldConfig::new("passages", FieldType::Object);
+ parent.fields.push(vector("vec", 4));
+ parent
+ .fields
+ .push(FieldConfig::new("vec_chunks", FieldType::Keyword));
+ schema.add_field(parent).unwrap();
+
+ let typeless = lexically_typeless_fields(&schema);
+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");
+ assert!(
+ !typeless.contains("passages.vec_chunks"),
+ "declared nested sibling excluded: {typeless:?}"
+ );
+ assert!(declares_non_vector_shaped_field(
+ &schema,
+ "passages.vec_chunks"
+ ));
+ assert!(
+ !declares_non_vector_shaped_field(&schema, "passages.vec"),
+ "a declared vector is not a lexical declaration"
+ );
+ assert!(
+ !declares_non_vector_shaped_field(&schema, "passages.absent"),
+ "an undeclared name has no declaration to defer to"
+ );
+ assert!(
+ !declares_non_vector_shaped_field(&schema, "passages.vec_chunks.deeper"),
+ "walking past a leaf must not resolve"
+ );
+ }
+
+ /// The FLAT spelling of a nested name — one top-level `FieldConfig` called
+ /// `"passages.vec_chunks"`, which is what `put_mapping` produces — must
+ /// resolve too. Without the literal lookup the segmented walk looks for a
+ /// root `passages` object, does not find one, and treats a declared field
+ /// as undeclared.
+ #[test]
+ fn the_flat_dotted_spelling_of_a_declared_sibling_resolves() {
+ let mut schema = Schema::empty();
+ schema.add_field(vector("passages.vec", 4)).unwrap();
+ schema
+ .add_field(FieldConfig::new("passages.vec_chunks", FieldType::Keyword))
+ .unwrap();
+
+ assert!(declares_non_vector_shaped_field(
+ &schema,
+ "passages.vec_chunks"
+ ));
+ let typeless = lexically_typeless_fields(&schema);
+ assert!(typeless.contains("passages.vec"), "got {typeless:?}");
+ assert!(
+ !typeless.contains("passages.vec_chunks"),
+ "the flat spelling was treated as undeclared: {typeless:?}"
+ );
+ }
+
/// The exclusion is by DECLARED type, not by value shape: a numeric array
/// under a non-vector mapping stays lexical exactly as before.
#[test]
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index 995a8cb38..e7ea4ee42 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -887,6 +887,347 @@ async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary
}
}
+/// #328, collision half — a field the user DECLARES under a companion's name
+/// must keep its whole lexical surface.
+///
+/// `memtable::lexically_typeless_fields` synthesises `<field>_chunks` and
+/// `__xerj_passage_meta__<field>` from the base name, because the passage
+/// pipeline writes them into `_source` with no `FieldConfig` behind them. The
+/// first revision of that rule fired UNCONDITIONALLY, so a user who mapped
+/// `emb_chunks` beside a `dense_vector emb` lost the postings for a field the
+/// schema declares — and then `lower_lexically_typeless_clauses` lowered every
+/// lexical clause naming it to `match_none`. Measured on this fixture, merge
+/// base `ca4d75a` → the name-only rule → here:
+///
+/// term {emb_chunks:"tenant-a"} 25 → 0 → 25
+/// terms {emb_chunks:["tenant-a"]} 25 → 0 → 25
+/// bool{filter:[term emb_chunks]} 25 → 0 → 25
+/// bool{must:[match body],filter:[term …]} 25 → 0 → 25
+/// bool{must_not:[term emb_chunks]} 25 → 50 → 25
+/// terms agg on emb_chunks 25/25 → 25/25 → 25/25
+///
+/// The `must_not` row is the severe one and the reason this is a correctness
+/// test rather than a footprint one: the query RETURNED the 25 documents the
+/// exclusion was written to remove. The aggregation row proves the data was
+/// present and correctly populated the whole time — the planner had declared a
+/// live user field unqueryable, with no error and no mapping change. If
+/// `<vector>_chunks` is a tenant or ACL discriminator that is a silent
+/// data-exposure failure mode.
+///
+/// Lucene reads "does this field get a term dictionary" off the field's own
+/// `IndexableFieldType` and never off its name
+/// (`lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),
+/// and one name carrying two different sets of index options is an outright
+/// error rather than a silent downgrade (`IndexingChain.FieldSchema
+/// .setIndexOptions` → `assertSame` → `raiseNotSame`, `:2213` / `:2193`).
+///
+/// THREE ARMS, and the third one asserts the OPPOSITE. `keyword` and `text`
+/// yield: the sibling keeps its `.fst` and every answer above. `long` does NOT
+/// yield, because `long`/`double` is exactly what dynamic mapping infers from
+/// a real multi-vector (`infer_field_type` on an array of float arrays), so
+/// deferring to it would put `<seg>.emb_chunks.fst` back at 1,592,118 B on the
+/// 300-doc fixture and give up the larger half of #328's saving. That arm
+/// therefore requires 0 hits and `must_not` = 50 — the documented residual,
+/// asserted rather than left to chance, so it cannot silently widen. The
+/// aggregation is checked on all three arms and buckets 25/25 on all three.
+#[tokio::test]
+async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one() {
+ const DIMS: usize = 16;
+ const DOCS: usize = 50;
+
+ let component = |doc: usize, dim: usize| -> f64 {
+ let h = (doc as u64)
+ .wrapping_mul(6_364_136_223_846_793_005)
+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));
+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0
+ };
+
+ for declared in [FieldType::Keyword, FieldType::Text, FieldType::Long] {
+ let label = match declared {
+ FieldType::Keyword => "keyword",
+ FieldType::Text => "text",
+ _ => "long",
+ };
+ // `long`/`double` is the RESIDUAL arm: it is what dynamic mapping
+ // infers from a real multi-vector, so the name rule cannot yield to it
+ // and this field IS still excluded. Pinned here, opposite in sign to
+ // the other two arms, so the limitation cannot drift into an accident.
+ let yields = !matches!(declared, FieldType::Long);
+ let dir = TempDir::new().unwrap();
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ // The collision: a user field named exactly like the companion the
+ // vector below would otherwise synthesise.
+ schema.fields.push(FieldConfig::new("emb_chunks", declared));
+ let mut emb = FieldConfig::new("emb", FieldType::Vector);
+ emb.options.dimensions = Some(DIMS);
+ emb.options.similarity = Some("cosine".to_string());
+ schema.fields.push(emb);
+
+ let engine = make_engine(&dir);
+ engine.create_index("collide", schema).unwrap();
+ let idx = engine.get_index("collide").unwrap();
+
+ let mut first_vector = Vec::new();
+ for d in 0..DOCS {
+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();
+ if d == 0 {
+ first_vector = v.clone();
+ }
+ // A keyword mapping gets the bare discriminator; a text mapping gets
+ // it plus a token that is unique to the half, because `match` on
+ // "tenant-a" analyses to [tenant, a] and would match both halves.
+ let tenant = if d % 2 == 0 { "tenant-a" } else { "tenant-b" };
+ let chunks_value: Value = match declared {
+ FieldType::Keyword => json!(tenant),
+ FieldType::Text => json!(format!(
+ "{tenant} {}",
+ if d % 2 == 0 { "widgets" } else { "gadgets" }
+ )),
+ // The residual arm: a scalar number, which is the shape
+ // `infer_field_type` also produces from a real multi-vector.
+ _ => json!(if d % 2 == 0 { 7 } else { 9 }),
+ };
+ idx.index_document(
+ Some(format!("d{d}")),
+ json!({
+ "body": format!("quarterly liquidity evidence {d}"),
+ "emb_chunks": chunks_value,
+ "emb": v,
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ idx.refresh().await.unwrap();
+ idx.force_merge(1).await.unwrap();
+
+ let mut files: Vec<(String, u64)> = Vec::new();
+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {
+ for entry in std::fs::read_dir(dir).unwrap() {
+ let entry = entry.unwrap();
+ if entry.file_type().unwrap().is_dir() {
+ walk(&entry.path(), out);
+ } else {
+ out.push((
+ entry.file_name().to_string_lossy().into_owned(),
+ entry.metadata().unwrap().len(),
+ ));
+ }
+ }
+ }
+ walk(dir.path(), &mut files);
+ let bytes_of = |suffix: &str| -> u64 {
+ files
+ .iter()
+ .filter(|(name, _)| name.ends_with(suffix))
+ .map(|(_, len)| *len)
+ .sum()
+ };
+ eprintln!(
+ "#328 collision [{label}]: emb_chunks.fst={} emb.fst={} body.fst={}",
+ bytes_of(".emb_chunks.fst"),
+ bytes_of(".emb.fst"),
+ bytes_of(".body.fst"),
+ );
+ // The DECLARED sibling keeps its term dictionary — except on the
+ // residual arm, where the declaration is indistinguishable from the
+ // multi-vector's own inferred type and the name rule wins.
+ assert_eq!(
+ files
+ .iter()
+ .any(|(name, _)| name.ends_with(".emb_chunks.fst")),
+ yields,
+ "[{label}] `.emb_chunks.fst` present should be {yields}; got {:?}",
+ files
+ .iter()
+ .filter(|(name, _)| name.contains(".emb"))
+ .collect::<Vec<_>>()
+ );
+ // … and the `dense_vector` beside it still loses its own.
+ assert!(
+ !files.iter().any(|(name, _)| name.ends_with(".emb.fst")),
+ "[{label}] the dense_vector must still have no term dictionary"
+ );
+
+ // ── The answers. Every row here is 25 on the merge base `ca4d75a`; the
+ // unconditional name-only rule answered 0 for all of them, and 50 for
+ // the `must_not` shape below. On the two YIELDING arms this test
+ // requires 25 — the merge base's own answer. On the residual `long`
+ // arm it requires 0, which is what the name rule produces and what the
+ // CHANGELOG documents as a behaviour change.
+ let want = if yields { (DOCS / 2) as u64 } else { 0 };
+ let probes: Vec<(&str, Value)> = if matches!(declared, FieldType::Keyword) {
+ vec![
+ ("term", json!({"term": {"emb_chunks": "tenant-a"}})),
+ ("terms", json!({"terms": {"emb_chunks": ["tenant-a"]}})),
+ (
+ "bool-filter",
+ json!({"bool": {"filter": [{"term": {"emb_chunks": "tenant-a"}}]}}),
+ ),
+ (
+ "bool-must-plus-filter",
+ json!({"bool": {
+ "must": [{"match": {"body": "liquidity"}}],
+ "filter": [{"term": {"emb_chunks": "tenant-a"}}]
+ }}),
+ ),
+ (
+ "constant-score",
+ json!({"constant_score": {"filter": {"term": {"emb_chunks": "tenant-a"}}}}),
+ ),
+ ]
+ } else if matches!(declared, FieldType::Long) {
+ vec![
+ ("term", json!({"term": {"emb_chunks": 7}})),
+ ("terms", json!({"terms": {"emb_chunks": [7]}})),
+ ("range", json!({"range": {"emb_chunks": {"gte": 8}}})),
+ (
+ "bool-filter",
+ json!({"bool": {"filter": [{"term": {"emb_chunks": 7}}]}}),
+ ),
+ ]
+ } else {
+ vec![
+ ("match", json!({"match": {"emb_chunks": "widgets"}})),
+ (
+ "match-phrase",
+ json!({"match_phrase": {"emb_chunks": "tenant-a widgets"}}),
+ ),
+ (
+ "multi-match",
+ json!({"multi_match": {"query": "widgets", "fields": ["emb_chunks"]}}),
+ ),
+ (
+ "simple-query-string",
+ json!({"simple_query_string": {"query": "widgets", "fields": ["emb_chunks"]}}),
+ ),
+ ("prefix", json!({"prefix": {"emb_chunks": "widget"}})),
+ ]
+ };
+ for (name, q) in &probes {
+ let hits = idx.search(&make_search(q.clone())).await.unwrap();
+ assert_eq!(
+ hits.total.value, want,
+ "[{label}] {name}: expected {want}, got {} for {q}",
+ hits.total.value
+ );
+ }
+
+ // The severe one, isolated: `must_not` on the declared sibling must
+ // still EXCLUDE. Under the name-only rule the clause lowered to
+ // `match_none`, the Bool fold dropped it as "excludes nothing", and the
+ // query returned all 50 — the 25 it was written to remove included.
+ //
+ // The text half is anchored with an explicit `match_all` `must`,
+ // because bare `bool{must_not:[match …]}` answers 0 on `origin/main`
+ // too — a PRE-EXISTING quirk with nothing to do with #328 (a control
+ // index carrying no vector field at all reproduces it: `{text body,
+ // text tags}`, `bool{must_not:[match tags:"widgets"]}` = 0 while
+ // `bool{must:[match_all],must_not:[match tags:"widgets"]}` = 25). This
+ // test pins the shape that is correct on `main` to begin with, so a
+ // failure here is this change's doing and nobody else's.
+ let must_not_probe = if matches!(declared, FieldType::Keyword) {
+ json!({"bool": {"must_not": [{"term": {"emb_chunks": "tenant-a"}}]}})
+ } else if matches!(declared, FieldType::Long) {
+ json!({"bool": {
+ "must": [{"match_all": {}}],
+ "must_not": [{"term": {"emb_chunks": 7}}]
+ }})
+ } else {
+ json!({"bool": {
+ "must": [{"match_all": {}}],
+ "must_not": [{"match": {"emb_chunks": "widgets"}}]
+ }})
+ };
+ let excluded = idx
+ .search(&make_search(must_not_probe.clone()))
+ .await
+ .unwrap();
+ // On the yielding arms the exclusion still works and returns exactly the
+ // other half. On the residual arm it does NOT: the clause lowers to
+ // `match_none`, the Bool fold drops it as "excludes nothing", and all
+ // 50 come back — the 25 the exclusion was written to remove included.
+ // That is the behaviour change the CHANGELOG documents, asserted here
+ // so it cannot happen by accident on a shape nobody looked at.
+ assert_eq!(
+ excluded.total.value,
+ if yields {
+ (DOCS / 2) as u64
+ } else {
+ DOCS as u64
+ },
+ "[{label}] must_not: got {} for {must_not_probe}",
+ excluded.total.value
+ );
+ if yields {
+ for hit in &excluded.hits {
+ let n: usize = hit.id.trim_start_matches('d').parse().unwrap();
+ assert_eq!(
+ n % 2,
+ 1,
+ "[{label}] must_not returned an excluded document: {}",
+ hit.id
+ );
+ }
+ }
+
+ // Doc values were never in question — this row answers 25/25 on ALL
+ // THREE arms, including the residual one where every lexical clause
+ // says 0. That is what proves the data is present and correctly
+ // populated, and that it is the planner, not the storage, that changed.
+ let agg = idx
+ .search(
+ &parse_request(&json!({
+ "size": 0,
+ "aggs": {"by_tenant": {"terms": {"field": "emb_chunks", "size": 10}}}
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ let buckets = agg.aggs.as_ref().expect("aggs present")["by_tenant"]["buckets"]
+ .as_array()
+ .unwrap()
+ .clone();
+ assert!(
+ buckets.iter().any(|b| b["doc_count"] == 25),
+ "[{label}] terms agg on the declared sibling must still bucket it, \
+ got {buckets:?}"
+ );
+
+ // The vector itself is unchanged by the yield: every lexical leaf
+ // naming it still answers 0, and kNN still answers from the graph.
+ for q in [
+ json!({"term": {"emb": "0"}}),
+ json!({"wildcard": {"emb": "0*"}}),
+ json!({"multi_match": {"query": "0", "fields": ["emb"]}}),
+ ] {
+ let hits = idx.search(&make_search(q.clone())).await.unwrap();
+ assert_eq!(
+ hits.total.value, 0,
+ "[{label}] a lexical clause on the dense_vector must answer 0, \
+ got {} for {q}",
+ hits.total.value
+ );
+ }
+ let knn = idx
+ .search(
+ &parse_request(&json!({
+ "query": {"knn": {"field": "emb", "query_vector": first_vector, "k": 3}},
+ "size": 10
+ }))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(knn.hits.len(), 3, "[{label}] kNN must still retrieve");
+ assert_eq!(knn.hits[0].id, "d0", "[{label}] nearest must be the probe");
+ }
+}
+
#[tokio::test]
async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {
let dir = TempDir::new().unwrap();