AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(engine): close the #328 companion residual rather than document it (#328)
xerj-orgcf1904891624
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ec10d219c..1b314655c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -161,8 +161,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0halves are named in that headline because the companion is where most of thebytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is`<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the- mapping at all — it is a dynamic field matched by NAME, not a field anyone- declared `dense_vector`. The field was fed+ mapping at all — it is matched by NAME, not by anyone declaring it+ `dense_vector`. The field was fedthrough the lexical indexing path and given an FST + postings that no querypath can read: kNN is served by `hnsw/graph.bin`, `exists` from`_source`/doc values, and `_field_caps` and highlighting never open the@@ -181,16 +181,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`.wal` is excluded from that total because its tail is reclaimedasynchronously and it is not reproducible: on the 300-doc fixture the- committed regression test uses, eight runs put the *whole directory* anywhere- between 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. The+ committed regression test uses, runs put the *whole directory* anywhere+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. Thetest prints both and asserts on the durable one. It reports- **3,249,323 B → 857,563–858,071 B** for that fixture (before-side measured by+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured byreverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the samefixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the- committed test on this revision printed 857,563 / 857,623 / 857,643 /- 857,659 / 857,787 / 857,827 / 857,859 / 857,955 B; the ceiling it asserts is- 1,500,000 B, three orders of magnitude clear of that jitter and still 1.7 MB- below the before-side.+ committed test on this revision printed 857,265 / 857,529 / 857,533 /+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and+ still 1.7 MB below the before-side.A `dense_vector` nested under an object mapping is covered too — itscomponents were landing in the *parent's* term dictionary, because the segment@@ -208,14 +208,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0**Two rules, and they do not have the same strength.** The base field isexcluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two- companions are excluded BY NAME, because the passage pipeline writes them into- `_source` with no mapping entry behind them, so there is no declared type to- read — and a name-based rule YIELDS to a declaration it could not have- produced itself. **If you have mapped your own field called `<vector>_chunks`- or `__xerj_passage_meta__<vector>` as `text`, `keyword`, `date`, `boolean` or- `ip`, nothing about it changes**: it keeps its term dictionary, every query- naming it keeps answering exactly what it answered before, and only the- `dense_vector` beside it loses its postings. Measured on+ companions are excluded BY NAME, because they arrive in `_source` with no+ mapping entry behind them, so there is no declared type to read — and a+ name-based rule YIELDS to any declaration it finds. **If you have mapped your+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,+ `object`, `nested`, or a type string XERJ does not recognise — nothing about+ it changes**: it keeps its term dictionary, every query naming it keeps+ answering exactly what it answered before, and only the `dense_vector` beside+ it loses its postings. The same holds for a field of that name XERJ mapped for+ you dynamically, whatever type it inferred, as long as the value is not itself+ a multi-vector. Measured on`{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`(before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,`terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,@@ -224,13 +229,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:`match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`- all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A dynamically mapped- string field of that name is exempt on the same rule (dynamic mapping- registers it as `text`) — measured 25 → 25.-- `long` and `double` are the exception, because they are what dynamic mapping- infers from a real multi-vector and the rule cannot tell the two apart. See- the fourth behaviour change below; that case is a hard incompatibility.+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name+ XERJ mapped for you *dynamically* is exempt on the same rule — measured+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one+ (44 B `.fst`).++ **What makes "yields to any declaration" affordable** is that the companion+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied+ `_source` — an array of float arrays — so dynamic mapping used to register it+ like any other unmapped key and infer `double` from it, producing a mapping+ entry indistinguishable from one you wrote. Dynamic mapping now declines to+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is+ already mapped *and* the value is a rectangular, non-empty array of non-empty+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,+ a string, a flat array or a ragged one under that name is your field and is+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is+ mapped `long`, keeps a 44 B `.fst` and answers 25.++ Both halves were measured on their own, by disabling each in the tree and+ re-running the committed tests:++ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |+ |---|---:|---:|---|+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |++ The middle row is the behaviour rc.16 would have shipped; the bottom row is+ the 67% of the saving that an unconditional "any declaration wins" gives up.+ Neither cost is paid here.A lexical clause that NAMES a `dense_vector` (or one of its companions) is nowlowered to `match_none` at plan time rather than falling through to the@@ -315,45 +347,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0`max_score` for a bool that touches a vector field will see this. Plainlexical queries are unaffected; the blast radius is bools with a`dense_vector` clause in them.- - **A `long` or `double` field named `<vector>_chunks` or- `__xerj_passage_meta__<vector>` stops being lexically searchable.** This is- the by-name half of the rule above, and it is the one case where it cannot- yield, because `long`/`double` is precisely what dynamic mapping infers from- a real multi-vector (an array of float arrays walks to its first scalar).- Deferring to that declaration was measured and it puts- `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture — i.e. it- gives up the larger half of the saving above. So the name wins, and a user's- own numeric field under that name loses its term dictionary. Measured on- `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9- (before → after):-- | query | before | after |- |---|---:|---:|- | `term {emb_chunks: 7}` | 25 | **0** |- | `terms {emb_chunks: [7]}` | 25 | **0** |- | `range {emb_chunks: {gte: 8}}` | 25 | **0** |- | `bool{filter:[term emb_chunks 7]}` | 25 | **0** |- | `bool{must:[match_all], must_not:[term emb_chunks 7]}` | 25 | **50** |- | `terms` agg / `stats` on `emb_chunks` | 25/25, sum 400 | 25/25, sum 400 |-- The `must_not` row returns the documents the exclusion was written to- remove, so treat this as a hard incompatibility rather than a footnote:- **if you have your own numeric field named `<vector>_chunks`, rename it- before upgrading.** Any other declared type (`text`, `keyword`, `date`,- `boolean`, `ip`) is exempt and unchanged, as is a dynamically mapped string- field of that name. Doc values are never affected on any of these paths —- the `terms` and `stats` aggregations in the last row keep working on the- residual case, which is how the rows above were confirmed to be a planning- change and not data loss.+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that+ XERJ mapped for you.** The multi-vector companion beside a declared+ `dense_vector` used to be registered by dynamic mapping as `double` (an+ array of float arrays walks to its first scalar). It is not registered at+ all now, so it disappears from `_mapping` and `_field_caps` — the same+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads+ `_source` changes; the values are stored and returned exactly as before, and+ `knn` and `semantic` on the vector itself are untouched. A field of that+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector+ value, is unaffected and still listed.++ One consequence worth knowing: aggregating or sorting on that companion+ stops working, because it no longer has a mapping entry to aggregate+ through. On a real multi-vector this was never a meaningful operation, but+ it was a `200` before and is not now.- **Existing indices keep their bloat until a merge rewrites them.** This is awrite-side rule. Segments written by an earlier release keep their `.emb.*`/ `.emb_chunks.*` files, and the per-segment `fts_has_field` gate readswhichever shape it finds. Run an explicit`POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an- index that already exists. The same applies to the by-name rule above: an- existing index with a `long`/`double` `<vector>_chunks` that is searchable- today stops being searchable for those clauses as soon as its segments are- rewritten — before any mapping change, and with no error.+ index that already exists.++ The companion rule has an ordering limit of the same kind, and it fails in+ the safe direction: XERJ can only decline to map a companion it can+ recognise, and recognising one needs the `dense_vector` to be mapped+ *before* the documents arrive. Index first and add the mapping afterwards+ and `<vector>_chunks` already carries a dynamic `double` entry, which the+ by-name rule then yields to — so that index keeps the companion's term+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of+ reclaiming it. It costs bytes, not answers.*Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,where XERJ answers `200`. This release removes bytes and adopts Lucene'sdiff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex e3a7ff0e7..639115330 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -18653,7 +18653,9 @@ impl Index {continue;};for (key, val) in obj {- if key.starts_with(PASSAGE_METADATA_PREFIX) {+ if key.starts_with(PASSAGE_METADATA_PREFIX)+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)+ {continue;}match schema.schema.field(key) {@@ -18782,7 +18784,9 @@ impl Index {return;}obj.iter().any(|(key, val)| {- if key.starts_with(PASSAGE_METADATA_PREFIX) {+ if key.starts_with(PASSAGE_METADATA_PREFIX)+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)+ {return false;}match schema.schema.field(key) {@@ -18829,7 +18833,9 @@ impl Index {let mut schema_changed = false;let mut field_count = schema.schema.field_count() as u32;for (key, val) in obj {- if key.starts_with(PASSAGE_METADATA_PREFIX) {+ if key.starts_with(PASSAGE_METADATA_PREFIX)+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)+ {continue;}if let Some(existing) = schema.schema.field_mut(key) {@@ -30164,7 +30170,8 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {/// float array. "Outright" is load-bearing twice over: a `fields: ["emb.*"]`/// PATTERN is not lowered (see `is_typeless_spec` below — measured 300 of 300/// on the fixture, on the merge base and here alike), and a `<field>_chunks`-/// the user mapped as `text`/`keyword`/`date`/`boolean`/`ip` is not in+/// the user mapped as ANYTHING other than `dense_vector` — or that dynamic+/// mapping registered from a value that is not a multi-vector — is not in/// `typeless` at all, so nothing naming it is lowered either/// (`memtable::lexically_typeless_fields`). On the 300-doc/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`@@ -31786,6 +31793,89 @@ fn dynamic_field_config(key: &str, val: &Value, date_detection: bool) -> FieldCodynamic_field_config_at_depth(key, val, date_detection, 1)}+/// Is `key` the per-document MULTI-VECTOR companion of a `dense_vector` that+/// is already declared, carrying the value shape RFC #148 gives one?+///+/// Dynamic mapping must NOT register such a key. `<vector>_chunks` arrives in+/// `_source` as an array of float arrays, and `infer_field_type` walks it to+/// its first scalar and lands on `FieldType::Double` — a `FieldConfig` that is+/// byte-identical to what a user mapping `"type": "double"` would have+/// produced. Once that config exists nothing downstream can tell the two+/// apart, and `memtable::declares_non_vector_shaped_field` is left choosing+/// between honouring a declaration nobody made (giving the companion back its+/// term dictionary: measured `<seg>.emb_chunks.fst` 0 B → 1,592,118 B on the+/// 300-doc `#328` fixture) and overriding a declaration the user DID make+/// (silently unmapping their own numeric field). Not inventing the config is+/// what removes the choice.+///+/// It is the same refusal the `__xerj_passage_meta__` sidecar already gets in+/// the same `||` at each of this predicate's call sites+/// (`key.starts_with(PASSAGE_METADATA_PREFIX)`), and the same modelling Lucene+/// uses: a per-document multi-vector is `LateInteractionField extends+/// BinaryDocValuesField`+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`,+/// `:44`, `:45`) — its own field type, never a numeric one arrived at by+/// inference.+///+/// THREE conditions, all required, so the blast radius is exactly the+/// companion:+///+/// * the name is `<base>_chunks`;+/// * `<base>` is ALREADY declared `dense_vector` in the live schema — a+/// `foo_chunks` with no `foo` vector beside it is an ordinary field and is+/// mapped as one;+/// * the value [`looks_like_multi_vector`] — a rectangular, non-empty array of+/// non-empty all-numeric arrays. A scalar, a string, a flat array or a+/// ragged one is somebody's own field and is mapped as one.+///+/// KNOWN LIMIT, and it fails SAFE: this only fires when the `dense_vector` is+/// declared BEFORE the documents arrive. Index first and map afterwards and+/// `<vector>_chunks` already holds a dynamic `double` config, which+/// `declares_non_vector_shaped_field` then (correctly, on the information it+/// has) yields to — so the companion keeps its term dictionary. That costs+/// bytes, not answers. Pinned by+/// `documents_before_the_mapping_leave_the_companion_mapped_and_lexical`.+/// ALLOCATION: this runs for every key of every ingested document that dynamic+/// mapping does not already know, on both the read-lock pre-screen and the+/// write-lock loop, so it allocates nothing and orders its three tests+/// cheapest-first: an `O(1)` suffix strip, then an allocation-free schema walk+/// ([`declares_dense_vector`], which is why `collect_dense_vector_fields`'+/// `Vec<String>` is not used here), then the value walk.+fn is_undeclared_multi_vector_companion(schema: &Schema, key: &str, val: &Value) -> bool {+ let Some(base) = key.strip_suffix(crate::memtable::MULTI_VECTOR_COMPANION_SUFFIX) else {+ return false;+ };+ !base.is_empty()+ && declares_dense_vector(schema, base)+ && crate::memtable::looks_like_multi_vector(val)+}++/// Is `path` declared `dense_vector`? Allocation-free, and resolves BOTH+/// spellings of a nested name for the same reason+/// `memtable::declares_non_vector_shaped_field` does: `put_mapping` stores a+/// dotted path as one top-level `FieldConfig` named `"passages.vec"`, while+/// `es_properties_to_fields` stores it as a `vec` child under a `passages`+/// object. The literal lookup runs first so the flat spelling cannot fall+/// through the segmented walk and be reported as undeclared.+fn declares_dense_vector(schema: &Schema, path: &str) -> bool {+ let is_vector = |fc: &FieldConfig| matches!(fc.field_type, FieldType::Vector);+ if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {+ return is_vector(field);+ }+ let mut fields: &[FieldConfig] = &schema.fields;+ let mut segments = path.split('.').peekable();+ while let Some(segment) = segments.next() {+ let Some(field) = fields.iter().find(|fc| fc.name == segment) else {+ return false;+ };+ if segments.peek().is_none() {+ return is_vector(field);+ }+ fields = &field.fields;+ }+ false+}+fn dynamic_field_config_at_depth(key: &str,val: &Value,@@ -31912,6 +32002,26 @@ fn merge_dynamic_children_into(};let mut changed = false;for (key, sub_val) in obj {+ // The nested form of `is_undeclared_multi_vector_companion`. It needs+ // no dotted path and no `Schema`, because a companion and its+ // `dense_vector` are SIBLINGS in the very object being merged: for a+ // `passages` mapping carrying a `vec` child of type `dense_vector`, the+ // companion is the `vec_chunks` child right beside it. Checking+ // `field.fields` locally keeps this function's signature (and its four+ // call sites) untouched.+ if key+ .strip_suffix(crate::memtable::MULTI_VECTOR_COMPANION_SUFFIX)+ .filter(|base| !base.is_empty())+ .is_some_and(|base| {+ field+ .fields+ .iter()+ .any(|f| f.name == base && matches!(f.field_type, FieldType::Vector))+ })+ && crate::memtable::looks_like_multi_vector(sub_val)+ {+ continue;+ }match field.fields.iter_mut().find(|f| f.name == *key) {Some(existing) => {if merge_dynamic_children_into(@@ -40486,6 +40596,187 @@ mod date_detection_tests {}}+/// Unit coverage for the #328 dynamic-mapping refusal — the half of the+/// companion rule that keeps `memtable::declares_non_vector_shaped_field` free+/// to yield to any declaration it finds.+#[cfg(test)]+mod multi_vector_companion_mapping_tests {+ use super::*;++ fn vector(name: &str, dims: usize) -> FieldConfig {+ let mut fc = FieldConfig::new(name, FieldType::Vector);+ fc.options.dimensions = Some(dims);+ fc+ }++ fn schema_with_vector() -> Schema {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("body", FieldType::Text))+ .unwrap();+ schema.add_field(vector("emb", 2)).unwrap();+ schema+ }++ /// ALL THREE conditions are required. Drop any one and dynamic mapping goes+ /// back to registering the key like any other.+ #[test]+ fn the_refusal_needs_the_name_the_declaration_and_the_shape() {+ let schema = schema_with_vector();+ let mv = serde_json::json!([[1.0, 2.0], [3.0, 4.0]]);++ // All three present.+ assert!(is_undeclared_multi_vector_companion(+ &schema,+ "emb_chunks",+ &mv+ ));++ // Wrong NAME: same value, no `_chunks` suffix, and a suffix with no+ // base in front of it.+ assert!(!is_undeclared_multi_vector_companion(&schema, "emb", &mv));+ assert!(!is_undeclared_multi_vector_companion(+ &schema, "matrix", &mv+ ));+ assert!(!is_undeclared_multi_vector_companion(+ &schema, "_chunks", &mv+ ));++ // No DECLARATION to be a companion of: `other_chunks` has no `other`+ // vector beside it, so it is an ordinary field even though its value is+ // rectangular numeric.+ assert!(!is_undeclared_multi_vector_companion(+ &schema,+ "other_chunks",+ &mv+ ));+ // …and a base declared as something that is not a vector does not+ // create a companion either.+ let mut lexical_base = Schema::empty();+ lexical_base+ .add_field(FieldConfig::new("emb", FieldType::Text))+ .unwrap();+ assert!(!is_undeclared_multi_vector_companion(+ &lexical_base,+ "emb_chunks",+ &mv+ ));++ // Wrong SHAPE: a user's own scalar, string, flat array or ragged array+ // under the companion's name is still mapped.+ for own in [+ serde_json::json!(7),+ serde_json::json!(7.5),+ serde_json::json!("tenant-a"),+ serde_json::json!([1.0, 2.0]),+ serde_json::json!([[1.0, 2.0], [3.0]]),+ ] {+ assert!(+ !is_undeclared_multi_vector_companion(&schema, "emb_chunks", &own),+ "{own} is somebody's own field and must be mapped"+ );+ }+ }++ /// Both spellings of a nested vector reach the schema, so both must be+ /// recognised — the same two shapes+ /// `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`+ /// exercises. The flat lookup runs first so it cannot fall through the+ /// segmented walk.+ #[test]+ fn declares_dense_vector_resolves_both_nested_spellings() {+ let mut dotted = Schema::empty();+ dotted.add_field(vector("passages.vec", 2)).unwrap();+ assert!(declares_dense_vector(&dotted, "passages.vec"));+ assert!(!declares_dense_vector(&dotted, "passages"));+ assert!(!declares_dense_vector(&dotted, "passages.other"));++ let mut sub = Schema::empty();+ let mut parent = FieldConfig::new("passages", FieldType::Object);+ parent.fields.push(vector("vec", 2));+ parent+ .fields+ .push(FieldConfig::new("title", FieldType::Text));+ sub.add_field(parent).unwrap();+ assert!(declares_dense_vector(&sub, "passages.vec"));+ assert!(!declares_dense_vector(&sub, "passages.title"));+ assert!(!declares_dense_vector(&sub, "passages"));+ // Does not walk past a leaf, and does not invent missing segments.+ assert!(!declares_dense_vector(&sub, "passages.vec.deeper"));+ assert!(!declares_dense_vector(&sub, "absent.vec"));+ }++ /// The nested form of the refusal, which lives in+ /// `merge_dynamic_children_into` and reads the sibling list directly rather+ /// than a dotted path. A `vec_chunks` beside a `dense_vector vec` is not+ /// registered; a `vec_chunks` with a scalar value, or one whose sibling is+ /// not a vector, is.+ #[test]+ fn a_nested_companion_is_not_merged_in_beside_its_vector() {+ let mut parent = FieldConfig::new("passages", FieldType::Object);+ parent.fields.push(vector("vec", 2));+ let mut count = 1u32;+ let changed = merge_dynamic_children_into(+ &mut parent,+ &serde_json::json!({"vec_chunks": [[1.0, 2.0], [3.0, 4.0]], "title": "q3"}),+ true,+ 1,+ &mut count,+ 100,+ );+ assert!(changed, "the ordinary sibling still lands");+ assert!(+ parent.fields.iter().any(|f| f.name == "title"),+ "{parent:?}"+ );+ assert!(+ !parent.fields.iter().any(|f| f.name == "vec_chunks"),+ "the nested multi-vector companion must not be registered: {parent:?}"+ );+ assert_eq!(count, 2, "exactly one field was added");++ // A scalar under the same name IS somebody's own field.+ let mut own = FieldConfig::new("passages", FieldType::Object);+ own.fields.push(vector("vec", 2));+ let mut own_count = 1u32;+ merge_dynamic_children_into(+ &mut own,+ &serde_json::json!({"vec_chunks": 7}),+ true,+ 1,+ &mut own_count,+ 100,+ );+ assert_eq!(+ own.fields+ .iter()+ .find(|f| f.name == "vec_chunks")+ .map(|f| f.field_type),+ Some(FieldType::Long),+ "{own:?}"+ );++ // …and so is a rectangular numeric array with no vector sibling.+ let mut no_vector = FieldConfig::new("passages", FieldType::Object);+ no_vector+ .fields+ .push(FieldConfig::new("vec", FieldType::Text));+ let mut nv_count = 1u32;+ merge_dynamic_children_into(+ &mut no_vector,+ &serde_json::json!({"vec_chunks": [[1.0, 2.0]]}),+ true,+ 1,+ &mut nv_count,+ 100,+ );+ assert!(+ no_vector.fields.iter().any(|f| f.name == "vec_chunks"),+ "no `dense_vector` sibling means no companion: {no_vector:?}"+ );+ }+}+#[cfg(test)]mod fts_projection_tests {use super::*;diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rsindex 82d284fdf..201eab085 100644--- a/engine/crates/xerj-engine/src/memtable.rs+++ b/engine/crates/xerj-engine/src/memtable.rs@@ -3917,20 +3917,23 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash////// The row is the DURABLE index — every file in the data directory except/// `.wal` — because the write-ahead log's tail is reclaimed asynchronously and-/// the whole-directory total is therefore not reproducible: eight runs of the-/// 300-doc regression test put it anywhere between 858,147 B and 1,105,001 B-/// while the durable bytes moved by 230 B. `.seg` (the stored documents, 11.2 MB+/// the whole-directory total is therefore not reproducible: runs of the 300-doc+/// regression test have put it anywhere between 858,147 B and 1,105,001 B while+/// the durable bytes moved by ~500 B. `.seg` (the stored documents, 11.2 MB/// here) is counted on both sides and is why the ratio is 3.79× rather than the/// 39.8 MB the term dictionaries cost outright.////// The 300-doc × 128-dim fixture the regression tests actually run is the same/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,-/// durable index 3,249,323 B → 857,563–858,071 B (eight runs on this revision-/// printed 857,563 / 857,623 / 857,643 / 857,659 / 857,787 / 857,827 /-/// 857,859 / 857,955). That test-/// prints both totals and asserts a ceiling on the durable one, so this figure-/// is enforced rather than merely published.+/// durable index 3,249,323 B → 857,265–857,769 B (eight runs on this revision+/// printed 857,265 / 857,529 / 857,533 / 857,545 / 857,549 / 857,657 /+/// 857,657 / 857,769, a 504 B spread). That test prints both totals and asserts+/// a ceiling on the durable one, so this figure is enforced rather than merely+/// published. The same eight runs are quoted verbatim in `CHANGELOG.md` and in+/// the comment above the `eprintln!` in+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary`; all three move+/// together or none of them do.////// TWO RULES DECIDE THIS SET, and they are stated separately because they do/// not have the same strength. [`lexically_typeless_fields`] has the full@@ -3945,19 +3948,21 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash/// the vector half is droppable — that needs a value-shape-aware rule, not/// a type rule./// * BY NAME, for the two companions `<field>_chunks` and-/// `__xerj_passage_meta__<field>`, which the passage pipeline writes into-/// `_source` with no `FieldConfig` behind them, so there is no declared type-/// to read. This rule is the weaker of the two and YIELDS to a declaration-/// it could not have produced itself: a field mapped `text`, `keyword`,-/// `date`, `boolean`, `ip`, `chunk` or `object` under one of those names-/// keeps its full lexical surface. It does NOT yield to `long`/`double`,-/// which is exactly what dynamic mapping infers from a real multi-vector —-/// the residual, with its measured cost and its measured A/B, is on-/// [`lexically_typeless_fields`] and in `CHANGELOG.md`.+/// `__xerj_passage_meta__<field>`, which arrive in `_source` with no+/// `FieldConfig` behind them, so there is no declared type to read. This+/// rule is the weaker of the two and YIELDS to ANY declaration: a field+/// mapped as anything other than `dense_vector` under one of those names —+/// `text`, `keyword`, `long`, `double`, `date`, `boolean`, `ip`,+/// `geo_point`, `binary`, `object`, `nested`, `chunk`, or an ES type string+/// `es_type_to_field_type` does not recognise — keeps its full lexical+/// surface. That is only sound because the companion is prevented from ever+/// ACQUIRING a declaration: `index::is_undeclared_multi_vector_companion`+/// stops dynamic mapping registering a `<vector>_chunks` key whose value is+/// a real multi-vector, so an "absent" answer here means absent.////// The bytes in the table above are almost all the second rule's: `emb_chunks`-/// in that fixture is a DYNAMIC field, absent from the user's mapping and-/// registered as `Double` by dynamic mapping, and it is 26,518,005 B of the+/// in that fixture is absent from the user's mapping — and, since this change,+/// absent from the evolved schema too — and it is 26,518,005 B of the/// 39,796,380 B removed.////// Removing the postings is only half of it, and the other half is NOT the@@ -4067,6 +4072,55 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>excluded}+pub const MULTI_VECTOR_COMPANION_SUFFIX: &str = "_chunks";++/// Is `val` the exact `_source` shape RFC #148 gives a per-document+/// MULTI-VECTOR — a non-empty array of equal-length, non-empty, all-numeric+/// arrays?+///+/// This is the value-shape half of the companion rule. The schema alone cannot+/// tell a multi-vector from a user's own numeric field (see+/// [`declares_non_vector_shaped_field`]); the VALUE can, and it is available at+/// exactly the moment that matters — when dynamic mapping is about to invent a+/// `FieldConfig` for an unmapped `<vector>_chunks` key+/// (`index::evolve_schema_from_doc`/`_docs`). Refusing to invent one there is+/// what keeps the schema honest, so the plan-time side of the rule can then+/// defer to ANY declaration it does find.+///+/// The equal-length requirement is not decoration: every chunk of one document+/// embeds with the same model, so a real multi-vector is rectangular. A ragged+/// array of numeric arrays is therefore NOT one, is left to dynamic mapping,+/// and keeps its term dictionary — the fail-safe direction.+///+/// Lucene models the same object as its own field type rather than inferring a+/// numeric one: `LateInteractionField extends BinaryDocValuesField`+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`),+/// constructed from a `float[][]` and encoded straight to a `BytesRef`+/// (`:44`, `:45`) — it never reaches an analyzer or a term dictionary at all.+pub(crate) fn looks_like_multi_vector(val: &serde_json::Value) -> bool {+ let Some(rows) = val.as_array() else {+ return false;+ };+ let mut dims: Option<usize> = None;+ if rows.is_empty() {+ return false;+ }+ for row in rows {+ let Some(components) = row.as_array() else {+ return false;+ };+ if components.is_empty() || !components.iter().all(serde_json::Value::is_number) {+ return false;+ }+ match dims {+ None => dims = Some(components.len()),+ Some(d) if d == components.len() => {}+ Some(_) => return false,+ }+ }+ true+}+/// Fields whose DECLARED type has no lexical representation at all — #328.////// Today that is exactly `FieldType::Vector` (`dense_vector`), so the base list@@ -4103,29 +4157,47 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>/// from the base name because no USER mapping declares them — they arrive in/// `_source` from the passage pipeline, and whatever `FieldConfig` they end up/// with is one dynamic mapping inferred from their values. A name-based rule-/// can collide with a field the user actually mapped, so it yields to one —-/// but only where the declaration rules the collision out:+/// can collide with a field the user actually mapped, so it yields to ANY+/// declaration it finds:///-/// * if the schema declares `<field>_chunks` (or the sidecar) with a type a-/// MULTI-VECTOR could not have produced — `text`, `keyword`, `date`,-/// `boolean`, `ip`, `chunk`, `object` — that declaration wins and the field-/// keeps its term dictionary and its whole lexical surface. Lucene draws-/// exactly this line: whether a field gets a term dictionary is read off the-/// field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,+/// * declared as anything other than `dense_vector` — `text`, `keyword`,+/// `long`, `double`, `date`, `boolean`, `ip`, `geo_point`, `binary`,+/// `object`, `nested`, `chunk`, or a mapping type+/// `es_type_to_field_type` did not recognise — the declaration wins and the+/// field keeps its term dictionary and its whole lexical surface. Lucene+/// draws exactly this line: whether a field gets a term dictionary is read+/// off the field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),/// never off its name, and one name meaning two different things is an/// outright error rather than a silent downgrade/// (`IndexingChain.FieldSchema.setIndexOptions` → `assertSame` →-/// `raiseNotSame`, `IndexingChain.java:2213` and `:2193`);-/// * `double` / `long` / absent → the name rule applies. Not a shortcut: it is-/// what the companion ITSELF declares. `<field>_chunks` is user-supplied-/// `_source` (an array of float arrays), so dynamic mapping registers it-/// like any other unmapped key and `infer_field_type` lands on-/// `FieldType::Double`. Deferring to that declaration was measured and it-/// brings `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture-/// — the whole 26,518,005 B of the 39,796,380 B saving that lives in the-/// companion at 5,000 × 128. See [`declares_non_vector_shaped_field`] for-/// the residual this leaves and what it costs to close.+/// `raiseNotSame`, `IndexingChain.java:2213`, `:2187` and `:2193`);+/// * absent → the name rule applies, and this is the companion's own case.+///+/// THAT SECOND BULLET IS ONLY SAFE BECAUSE THE COMPANION IS KEPT UNDECLARED.+/// `<field>_chunks` is user-supplied `_source` (an array of float arrays), so+/// dynamic mapping used to register it like any other unmapped key —+/// `infer_field_type` walks to the first scalar and lands on+/// `FieldType::Double`, a `FieldConfig` byte-identical to a user's own+/// `"type": "double"`. That is what forced an earlier revision to carve+/// `double`/`long` out of the yield set above, and with them a real user field+/// under that name. `index::is_undeclared_multi_vector_companion` removes the+/// ambiguity at its source instead: dynamic mapping refuses to invent the+/// config when the value [`looks_like_multi_vector`], so the only `double` or+/// `long` under this name is one somebody wrote.+///+/// BOTH HALVES ARE LOAD-BEARING, measured by disabling each on its own against+/// the committed 300-doc × 128-dim fixture and the 50-doc collision fixture:+///+/// | | `.emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |+/// |---|---|---|---|+/// | here | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |+/// | yield-to-numeric restored | 0 B | 857,445 B | **0 B `.fst`, answers 0** |+/// | refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |+///+/// The middle row is the residual this replaces; the bottom row is the 67% of+/// the saving an unconditional "any declaration wins" gives up. Neither is+/// paid for here.////// This is deliberately NOT the contract `semantic_derived_vector_fields`/// carries: that one synthesises `<target>_chunks` unconditionally, and a@@ -4192,31 +4264,38 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>/// The `match_all`-anchored form is the one the tests pin, because it is the/// one that is correct on `main` to begin with.///-/// THE RESIDUAL, stated in full because it is the same class and it is NOT-/// fixed: a field the user declares `long` or `double` under one of these two-/// names is still excluded by name, and the plan-time lowering still zeroes-/// every lexical clause on it. Measured on-/// `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9 —-/// merge base `ca4d75a` → here:+/// THE NUMERIC ARM, which an earlier revision of this branch shipped as an+/// accepted residual and which is now closed. On that revision a field the user+/// declared `long` or `double` under one of these two names was still excluded+/// by name and the plan-time lowering still zeroed every lexical clause on it.+/// Measured on `{text body, long emb_chunks, dense_vector emb}`, 50 docs,+/// values 7 and 9 — merge base `ca4d75a` → that revision → here:///-/// `term {emb_chunks:7}` 25 → **0**-/// `terms {emb_chunks:[7]}` 25 → **0**-/// `range {emb_chunks:{gte:8}}` 25 → **0**-/// `bool{filter:[term emb_chunks 7]}` 25 → **0**-/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50**-/// `terms` agg / `stats` on `emb_chunks` 25/25, sum 400 → 25/25, sum 400+/// `term {emb_chunks:7}` 25 → **0** → 25+/// `terms {emb_chunks:[7]}` 25 → **0** → 25+/// `range {emb_chunks:{gte:8}}` 25 → **0** → 25+/// `bool{filter:[term emb_chunks 7]}` 25 → **0** → 25+/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50** → 25+/// `terms` agg / `stats` on `emb_chunks` 25/25 on all three///-/// It is narrower than what it replaces — the reported case, a `keyword`-/// tenant discriminator, is now byte-identical to the merge base — but it is-/// the same failure shape, and closing it needs a decision this function cannot-/// make on its own: either give the schema per-field PROVENANCE (explicit-/// mapping vs. dynamically inferred) so an explicit `double` can outrank the-/// name, or drop the by-name rule entirely and give up 67% of the saving. Both-/// are larger than #328. Documented as a behaviour change in `CHANGELOG.md`-/// and pinned by `a_numerically_declared_sibling_is_still_excluded_by_name`-/// here and the `[long]` arm of-/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in-/// `tests/integration.rs`, so it cannot drift silently.+/// The `double` arm and the DYNAMICALLY mapped numeric case (a scalar `7`/`9`+/// under this name with no mapping at all, which dynamic mapping registers as+/// `long`) behave identically and are pinned alongside it. It was NOT closed by+/// per-field schema provenance and NOT by giving up the by-name rule — the two+/// options an earlier revision of this comment called exhaustive, wrongly. It+/// is closed by not manufacturing the ambiguous declaration in the first place+/// (`index::is_undeclared_multi_vector_companion`), which costs nothing: the+/// 300-doc fixture still reports `emb_chunks.fst=0` and durable 857,265–857,769 B.+///+/// THE LIMIT THAT REMAINS, and it fails safe: the refusal needs the+/// `dense_vector` declared BEFORE the documents arrive. Index first and map+/// afterwards and `emb_chunks` already carries a dynamic `double` config, which+/// this function then yields to — measured `.emb_chunks.fst` = 9,467 B on a+/// 50-doc × 16-dim fixture, with `exists {field: emb_chunks}` still answering+/// 50. That costs bytes, not answers, and is pinned by+/// `a_companion_mapped_before_its_vector_keeps_its_dictionary_and_its_answers`+/// in `tests/integration.rs`. The nested-under-an-object spelling has the same+/// shape of limit and the same direction.////// Pinned by `a_declared_sibling_keeps_its_lexical_surface` here and by/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in@@ -4238,7 +4317,7 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<Sfor field in crate::index::collect_dense_vector_fields(schema) {// BY NAME, and only where no declaration contradicts it.for synthesised in [- format!("{field}_chunks"),+ format!("{field}{MULTI_VECTOR_COMPANION_SUFFIX}"),format!("{}{}", xerj_query::executor::PASSAGE_METADATA_PREFIX, field),] {if !declares_non_vector_shaped_field(schema, &synthesised) {@@ -4259,33 +4338,41 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S/// rather than using `Schema::has_field`, which only sees root names and would/// answer `false` for every nested companion.///-/// THE LINE IS DRAWN AT "COULD THIS DECLARATION HAVE COME FROM THE COMPANION-/// ITSELF", and it is drawn there because the companion is not always-/// undeclared. `<field>_chunks` is user-supplied `_source` (RFC #148's-/// per-document multi-vector: an array of float arrays), so DYNAMIC MAPPING-/// registers it like any other unmapped key — `infer_field_type` walks to the-/// first non-null element, twice, and lands on `FieldType::Double`-/// (`index.rs`'s `Value::Array` arm). Measured: with "any declaration wins",-/// `<seg>.emb_chunks.fst` comes back at 1,592,118 B on the 300-doc fixture and-/// the whole 67% of the saving that lives in the companion is lost, because-/// dynamic mapping had already declared it.+/// THE RULE IS ITS COMPLEMENT, NOT A LIST: this yields for EVERY declared type+/// except `dense_vector`. It is stated that way on purpose — an earlier+/// revision enumerated `Text`/`Keyword`/`Date`/`Boolean`/`Ip`/`Chunk`/`Object`+/// and read as exhaustive when it was not (`GeoPoint`, `Binary` and `Nested`+/// also yielded, and `es_type_to_field_type` maps every UNRECOGNISED ES type+/// string to `Object`, so unknown mapping types yielded too). So:///-/// So:+/// * `Vector` — already in the set by declared type; `false` is a no-op, and+/// the companion of a companion is not a thing.+/// * anything else — a declaration is a declaration. The user mapped a field+/// under this name, so the name rule yields and the field keeps its term+/// dictionary and its whole lexical surface: `text`, `keyword`, `long`,+/// `double`, `date`, `boolean`, `ip`, `geo_point`, `binary`, `object`,+/// `nested`, `chunk`, and anything `es_type_to_field_type` did not+/// recognise.+/// * absent — nothing to defer to; the name rule applies. This is the+/// companion's own case, and it is the ONLY one, because+/// `index::evolve_schema_from_doc`/`_docs` refuse to invent a `FieldConfig`+/// for an unmapped `<vector>_chunks` key whose value+/// [`looks_like_multi_vector`]. Without that refusal this predicate would be+/// answering about a declaration DYNAMIC MAPPING wrote: `infer_field_type`+/// walks an array of float arrays to its first scalar and lands on+/// `FieldType::Double`, indistinguishable from a user's own `double`. That+/// is why an earlier revision had to carve `Double`/`Long` out of the yield+/// set, and why a user's own numeric field under this name silently lost its+/// lexical surface. The carve-out is gone; the ambiguity is prevented at the+/// only place it was ever manufactured.///-/// * `Double` / `Long` — the shape `infer_field_type` produces from an array-/// of float (or integer) arrays. Indistinguishable from the companion, so-/// the name rule applies and this returns `false`. This is the residual-/// case: a user's OWN numeric field named `<vector>_chunks` loses its term-/// dictionary. It does not lose its data — doc values, `terms`/`range`-/// aggregations and sorts are untouched — and the measured cost of the-/// alternative is the entire companion saving.-/// * `Vector` — already in the set by declared type; `false` is a no-op.-/// * `Text` / `Keyword` / `Date` / `Boolean` / `Ip` / `Chunk` / `Object` — a-/// multi-vector cannot infer to any of these, so a declaration of one is-/// unambiguously the user's own field and the name rule yields. This is the-/// case that matters: a tenant or ACL discriminator named `emb_chunks` keeps-/// its whole lexical surface.-/// * absent — nothing to defer to; the name rule applies.+/// Lucene draws exactly this line: whether a field gets a term dictionary is+/// read off the field's own `IndexableFieldType`+/// (`IndexingChain.invertAndStore`,+/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),+/// never off its name, and one name meaning two different things is an outright+/// error rather than a silent downgrade (`FieldSchema.setIndexOptions` →+/// `assertSame` → `raiseNotSame`, `IndexingChain.java:2213`, `:2187`, `:2193`).////// BOTH SPELLINGS of a nested name are resolved, because both reach the schema./// `put_mapping` stores a dotted path as ONE top-level `FieldConfig` named@@ -4295,12 +4382,8 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S/// exercises for the vector itself. The literal lookup runs first so the flat/// spelling cannot fall through the segmented walk and be treated as undeclared.fn declares_non_vector_shaped_field(schema: &Schema, path: &str) -> bool {- let is_lexical = |fc: &xerj_common::types::FieldConfig| {- !matches!(- fc.field_type,- FieldType::Vector | FieldType::Double | FieldType::Long- )- };+ let is_lexical =+ |fc: &xerj_common::types::FieldConfig| !matches!(fc.field_type, FieldType::Vector);if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {return is_lexical(field);}@@ -4653,15 +4736,16 @@ mod lexically_typeless_fields_tests {}}- /// THE RESIDUAL, pinned so it cannot drift into an accident. A `Double` or- /// `Long` declaration is exactly what dynamic mapping produces from the- /// real multi-vector (`infer_field_type` walks an array of float arrays to- /// its first scalar), so the name rule cannot yield to it without losing- /// the whole companion saving — measured at `<seg>.emb_chunks.fst`- /// 1,592,118 B reappearing on the 300-doc fixture. Documented in the- /// CHANGELOG as a behaviour change rather than fixed.+ /// THE NUMERIC ARM, which used to be the accepted residual: a `Double` or+ /// `Long` declaration is exactly what dynamic mapping USED to produce from+ /// a real multi-vector, so the name rule could not yield to it without+ /// losing the whole companion saving. It yields now, because+ /// `index::is_undeclared_multi_vector_companion` stops that config from+ /// being invented — so a `long`/`double` under this name is necessarily+ /// somebody's own field. This assertion is the inverse of the one it+ /// replaces; if the carve-out ever comes back it fails here first.#[test]- fn a_numerically_declared_sibling_is_still_excluded_by_name() {+ fn a_numerically_declared_sibling_keeps_its_lexical_surface() {for declared in [FieldType::Double, FieldType::Long] {let mut schema = Schema::empty();schema@@ -4669,11 +4753,92 @@ mod lexically_typeless_fields_tests {.unwrap();schema.add_field(vector("emb", 8)).unwrap();assert!(- lexically_typeless_fields(&schema).contains("emb_chunks"),- "a `{declared:?}` declaration is the multi-vector's own inferred \- shape and must not rescue the name"+ !lexically_typeless_fields(&schema).contains("emb_chunks"),+ "a `{declared:?}` declaration is somebody's own field and the \+ name rule must yield to it");- assert!(!declares_non_vector_shaped_field(&schema, "emb_chunks"));+ assert!(declares_non_vector_shaped_field(&schema, "emb_chunks"));+ }+ }++ /// The yield set is the COMPLEMENT of `Vector`, not the enumerated list an+ /// earlier doc comment gave. `GeoPoint`, `Binary` and `Nested` were never+ /// in that list and always yielded, and `es_type_to_field_type` maps every+ /// unrecognised ES type string to `Object`, so an unknown mapping type+ /// yields too. Pinned across every variant so the doc and the code cannot+ /// drift apart again.+ #[test]+ fn every_declared_type_except_dense_vector_keeps_its_lexical_surface() {+ for declared in [+ FieldType::Text,+ FieldType::Keyword,+ FieldType::Long,+ FieldType::Double,+ FieldType::Boolean,+ FieldType::Date,+ FieldType::Ip,+ FieldType::GeoPoint,+ FieldType::Binary,+ FieldType::Object,+ FieldType::Nested,+ ] {+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("emb_chunks", declared))+ .unwrap();+ schema.add_field(vector("emb", 8)).unwrap();+ assert!(+ declares_non_vector_shaped_field(&schema, "emb_chunks"),+ "`{declared:?}` is a declaration and must be yielded to"+ );+ let typeless = lexically_typeless_fields(&schema);+ assert!(+ !typeless.contains("emb_chunks"),+ "`{declared:?}` sibling excluded anyway: {typeless:?}"+ );+ // …while the vector itself is excluded on every one of them.+ assert!(typeless.contains("emb"));+ }+ }++ /// The value-shape half of the rule, exhaustively. Only a rectangular,+ /// non-empty array of non-empty all-numeric arrays is a multi-vector;+ /// everything else is somebody's field and keeps its mapping.+ #[test]+ fn looks_like_multi_vector_accepts_only_the_rfc_148_shape() {+ for yes in [+ serde_json::json!([[1.0, 2.0], [3.0, 4.0]]),+ serde_json::json!([[1, 2, 3]]),+ serde_json::json!([[0.5]]),+ ] {+ assert!(looks_like_multi_vector(&yes), "should accept {yes}");+ }+ for no in [+ // scalars and strings — a user's own field+ serde_json::json!(7),+ serde_json::json!(7.5),+ serde_json::json!("tenant-a"),+ serde_json::json!(true),+ serde_json::json!(null),+ serde_json::json!({"a": 1}),+ // a FLAT numeric array is a single vector, not a multi-vector, and+ // is not what this rule is about+ serde_json::json!([1.0, 2.0, 3.0]),+ // empty outer, empty inner+ serde_json::json!([]),+ serde_json::json!([[]]),+ // RAGGED — cannot have come from one embedding model, so it is left+ // to dynamic mapping (the fail-safe direction)+ serde_json::json!([[1.0, 2.0], [3.0]]),+ // non-numeric elements+ serde_json::json!([["a", "b"]]),+ serde_json::json!([[1.0, "b"]]),+ serde_json::json!([[1.0], null]),+ serde_json::json!([[1.0], 2.0]),+ // array of objects+ serde_json::json!([{"a": 1}]),+ ] {+ assert!(!looks_like_multi_vector(&no), "should reject {no}");}}diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rsindex e7ea4ee42..849902872 100644--- a/engine/crates/xerj-engine/tests/integration.rs+++ b/engine/crates/xerj-engine/tests/integration.rs@@ -250,13 +250,16 @@ async fn user_mapped_dense_vector_builds_no_fts_term_dictionary() {//// TWO totals, and only the second one is quotable. The whole data dir// includes the write-ahead log, whose tail is reclaimed asynchronously, so- // it is NOT reproducible: eight runs of this test on one machine printed a- // whole-dir total between 858,147 B and 1,105,001 B, a 29% spread, entirely- // because the surviving `.wal` files ranged from 256 B to 6 MB while the- // segment bytes did not move at all. `durable-index` excludes `.wal` and is- // reproducible to ~230 B (the run-to-run jitter in the HNSW `graph.bin`):- // the same eight runs printed 857,843 – 858,071 B. That is the number the- // CHANGELOG quotes and the one the ceiling below pins.+ // it is NOT reproducible: the eight runs below printed a whole-dir total+ // between 1,069,604 B and 1,104,599 B, and earlier revisions of this test+ // have printed as little as 858,147 B, entirely because the surviving+ // `.wal` files vary by megabytes while the segment bytes do not move at+ // all. `durable-index` excludes `.wal` and is reproducible to ~500 B (the+ // run-to-run jitter in the HNSW `graph.bin`): eight runs on THIS revision+ // printed 857,265 / 857,529 / 857,533 / 857,545 / 857,549 / 857,657 /+ // 857,657 / 857,769 B — a 504 B spread. That range is what the CHANGELOG+ // quotes and the one the ceiling below pins. Re-measure all three places+ // together (here, `memtable.rs`'s table, `CHANGELOG.md`) if it moves.let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();let durable: u64 = files.iter()@@ -921,15 +924,21 @@ async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary/// error rather than a silent downgrade (`IndexingChain.FieldSchema/// .setIndexOptions` → `assertSame` → `raiseNotSame`, `:2213` / `:2193`).///-/// THREE ARMS, and the third one asserts the OPPOSITE. `keyword` and `text`-/// yield: the sibling keeps its `.fst` and every answer above. `long` does NOT-/// yield, because `long`/`double` is exactly what dynamic mapping infers from-/// a real multi-vector (`infer_field_type` on an array of float arrays), so-/// deferring to it would put `<seg>.emb_chunks.fst` back at 1,592,118 B on the-/// 300-doc fixture and give up the larger half of #328's saving. That arm-/// therefore requires 0 hits and `must_not` = 50 — the documented residual,-/// asserted rather than left to chance, so it cannot silently widen. The-/// aggregation is checked on all three arms and buckets 25/25 on all three.+/// FOUR ARMS AND THEY ALL YIELD, including the two numeric ones. An earlier+/// revision of this test asserted the OPPOSITE for `long`: the name rule could+/// not yield to `long`/`double` because that is what `infer_field_type` derives+/// from an array of float arrays, so the declaration was indistinguishable from+/// the multi-vector's own. That residual is closed at its source —+/// `index::is_undeclared_multi_vector_companion` stops dynamic mapping+/// inventing the numeric `FieldConfig` in the first place, so the only `long`+/// or `double` under this name is one a user actually wrote, and+/// `declares_non_vector_shaped_field` now yields to every declared type except+/// `dense_vector`. Measured here: the `[long]` arm keeps `.emb_chunks.fst` at+/// 44 B and answers 25 on every row, while the 300-doc fixture in+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary` still reports+/// `emb_chunks.fst=0` — the saving is not paid for with the collision.+///+/// The aggregation is checked on all four arms and buckets 25/25 on all four.#[tokio::test]async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one() {const DIMS: usize = 16;@@ -942,17 +951,18 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0};- for declared in [FieldType::Keyword, FieldType::Text, FieldType::Long] {+ for declared in [+ FieldType::Keyword,+ FieldType::Text,+ FieldType::Long,+ FieldType::Double,+ ] {let label = match declared {FieldType::Keyword => "keyword",FieldType::Text => "text",- _ => "long",+ FieldType::Long => "long",+ _ => "double",};- // `long`/`double` is the RESIDUAL arm: it is what dynamic mapping- // infers from a real multi-vector, so the name rule cannot yield to it- // and this field IS still excluded. Pinned here, opposite in sign to- // the other two arms, so the limitation cannot drift into an accident.- let yields = !matches!(declared, FieldType::Long);let dir = TempDir::new().unwrap();let mut schema = Schema::empty();schema@@ -986,9 +996,13 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo"{tenant} {}",if d % 2 == 0 { "widgets" } else { "gadgets" })),- // The residual arm: a scalar number, which is the shape- // `infer_field_type` also produces from a real multi-vector.- _ => json!(if d % 2 == 0 { 7 } else { 9 }),+ // The two numeric arms. A SCALAR, which is what a user's own+ // `long`/`double` field looks like — and, decisively, not the+ // rectangular array-of-numeric-arrays `looks_like_multi_vector`+ // recognises, so nothing about this value is companion-shaped+ // either.+ FieldType::Long => json!(if d % 2 == 0 { 7 } else { 9 }),+ _ => json!(if d % 2 == 0 { 7.5 } else { 9.5 }),};idx.index_document(Some(format!("d{d}")),@@ -1032,15 +1046,14 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loobytes_of(".emb.fst"),bytes_of(".body.fst"),);- // The DECLARED sibling keeps its term dictionary — except on the- // residual arm, where the declaration is indistinguishable from the- // multi-vector's own inferred type and the name rule wins.- assert_eq!(+ // The DECLARED sibling keeps its term dictionary. On ALL FOUR arms —+ // this is the assertion that fails if the numeric carve-out ever comes+ // back.+ assert!(files.iter().any(|(name, _)| name.ends_with(".emb_chunks.fst")),- yields,- "[{label}] `.emb_chunks.fst` present should be {yields}; got {:?}",+ "[{label}] a declared sibling must keep `.emb_chunks.fst`; got {:?}",files.iter().filter(|(name, _)| name.contains(".emb"))@@ -1054,11 +1067,9 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo// ── The answers. Every row here is 25 on the merge base `ca4d75a`; the// unconditional name-only rule answered 0 for all of them, and 50 for- // the `must_not` shape below. On the two YIELDING arms this test- // requires 25 — the merge base's own answer. On the residual `long`- // arm it requires 0, which is what the name rule produces and what the- // CHANGELOG documents as a behaviour change.- let want = if yields { (DOCS / 2) as u64 } else { 0 };+ // the `must_not` shape below. All four arms require 25 — the merge+ // base's own answer.+ let want = (DOCS / 2) as u64;let probes: Vec<(&str, Value)> = if matches!(declared, FieldType::Keyword) {vec![("term", json!({"term": {"emb_chunks": "tenant-a"}})),@@ -1089,6 +1100,16 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loojson!({"bool": {"filter": [{"term": {"emb_chunks": 7}}]}}),),]+ } else if matches!(declared, FieldType::Double) {+ vec![+ ("term", json!({"term": {"emb_chunks": 7.5}})),+ ("terms", json!({"terms": {"emb_chunks": [7.5]}})),+ ("range", json!({"range": {"emb_chunks": {"gte": 8}}})),+ (+ "bool-filter",+ json!({"bool": {"filter": [{"term": {"emb_chunks": 7.5}}]}}),+ ),+ ]} else {vec![("match", json!({"match": {"emb_chunks": "widgets"}})),@@ -1136,6 +1157,11 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo"must": [{"match_all": {}}],"must_not": [{"term": {"emb_chunks": 7}}]}})+ } else if matches!(declared, FieldType::Double) {+ json!({"bool": {+ "must": [{"match_all": {}}],+ "must_not": [{"term": {"emb_chunks": 7.5}}]+ }})} else {json!({"bool": {"must": [{"match_all": {}}],@@ -1146,38 +1172,30 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo.search(&make_search(must_not_probe.clone())).await.unwrap();- // On the yielding arms the exclusion still works and returns exactly the- // other half. On the residual arm it does NOT: the clause lowers to- // `match_none`, the Bool fold drops it as "excludes nothing", and all- // 50 come back — the 25 the exclusion was written to remove included.- // That is the behaviour change the CHANGELOG documents, asserted here- // so it cannot happen by accident on a shape nobody looked at.+ // The exclusion returns exactly the other half on all four arms. Under+ // the name-only rule the numeric arms returned all 50 — the 25 the+ // exclusion was written to remove included; that is the shape this+ // assertion exists to keep out.assert_eq!(excluded.total.value,- if yields {- (DOCS / 2) as u64- } else {- DOCS as u64- },+ (DOCS / 2) as u64,"[{label}] must_not: got {} for {must_not_probe}",excluded.total.value);- if yields {- for hit in &excluded.hits {- let n: usize = hit.id.trim_start_matches('d').parse().unwrap();- assert_eq!(- n % 2,- 1,- "[{label}] must_not returned an excluded document: {}",- hit.id- );- }+ for hit in &excluded.hits {+ let n: usize = hit.id.trim_start_matches('d').parse().unwrap();+ assert_eq!(+ n % 2,+ 1,+ "[{label}] must_not returned an excluded document: {}",+ hit.id+ );}- // Doc values were never in question — this row answers 25/25 on ALL- // THREE arms, including the residual one where every lexical clause- // says 0. That is what proves the data is present and correctly- // populated, and that it is the planner, not the storage, that changed.+ // Doc values were never in question — this row answers 25/25 on all+ // four arms. It answered 25/25 under the name-only rule too, which is+ // what proved the data was present and correctly populated while every+ // lexical clause on it said 0: it was the planner, not the storage.let agg = idx.search(&parse_request(&json!({@@ -1228,6 +1246,281 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo}}+/// #328, the other half of the same rule: what dynamic mapping is allowed to do+/// with an UNDECLARED `<vector>_chunks` key, which is what decides whether the+/// collision test above can be true at the same time as the saving.+///+/// Three arms, one index each, all with `emb` declared `dense_vector` and+/// `emb_chunks` absent from the mapping. What differs is only the VALUE, and+/// the value is the whole rule+/// (`memtable::looks_like_multi_vector` →+/// `index::is_undeclared_multi_vector_companion`):+///+/// * `[[…],[…]]` rectangular numeric — a real RFC #148 multi-vector.+/// Dynamic mapping must not register it, it must acquire no `FieldConfig`,+/// it must have no `.emb_chunks.fst`, and lexical clauses on it answer 0.+/// This is where the 26,518,005 B / 67% of #328's saving comes from.+/// * a SCALAR string — somebody's own field that happens to sit under the+/// name. Registered as `text` and fully lexical. This one was already safe+/// before the fix.+/// * a SCALAR number — somebody's own field, registered as `long`, and fully+/// lexical. THIS is the arm that was silently broken: dynamic mapping wrote+/// a `long` config, `declares_non_vector_shaped_field` carved `long` out of+/// its yield set precisely because it could not tell that config from the+/// multi-vector's, and every lexical clause on a live user field lowered to+/// `match_none`. Nothing about the mapping changed and no error was raised.+///+/// The value-shape test is why all three can hold at once, and it is the same+/// place Lucene puts the distinction: a per-document multi-vector is+/// `LateInteractionField extends BinaryDocValuesField`+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`,+/// `:44`, `:45`) — its own field type, never a numeric one arrived at by+/// inference.+#[tokio::test]+async fn an_undeclared_chunks_key_is_unmapped_only_when_its_value_is_a_multi_vector() {+ const DIMS: usize = 16;+ const DOCS: usize = 50;++ let component = |doc: usize, dim: usize| -> f64 {+ let h = (doc as u64)+ .wrapping_mul(6_364_136_223_846_793_005)+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0+ };++ // (label, value builder, expected-to-stay-lexical, probe)+ #[allow(clippy::type_complexity)]+ let arms: Vec<(&str, Box<dyn Fn(usize, &[f64]) -> Value>, bool, Value)> = vec![+ (+ "multi-vector",+ Box::new(|_d, v: &[f64]| json!([v, v])),+ false,+ json!({"term": {"emb_chunks": "0"}}),+ ),+ (+ "scalar-string",+ Box::new(|d, _v: &[f64]| json!(if d % 2 == 0 { "tenant-a" } else { "tenant-b" })),+ true,+ json!({"term": {"emb_chunks": "tenant-a"}}),+ ),+ (+ "scalar-number",+ Box::new(|d, _v: &[f64]| json!(if d % 2 == 0 { 7 } else { 9 })),+ true,+ json!({"term": {"emb_chunks": 7}}),+ ),+ ];++ for (label, value_of, stays_lexical, probe) in arms {+ let dir = TempDir::new().unwrap();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ let mut emb = FieldConfig::new("emb", FieldType::Vector);+ emb.options.dimensions = Some(DIMS);+ emb.options.similarity = Some("cosine".to_string());+ schema.fields.push(emb);+ // NOTE: no `emb_chunks` here. Whatever config it ends up with is one+ // dynamic mapping decided, which is exactly what is under test.++ let engine = make_engine(&dir);+ engine.create_index("dyn", schema).unwrap();+ let idx = engine.get_index("dyn").unwrap();++ for d in 0..DOCS {+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();+ idx.index_document(+ Some(format!("d{d}")),+ json!({+ "body": format!("quarterly liquidity evidence {d}"),+ "emb_chunks": value_of(d, &v),+ "emb": v,+ }),+ )+ .await+ .unwrap();+ }+ idx.refresh().await.unwrap();+ idx.force_merge(1).await.unwrap();++ let live = idx.schema().await;+ let mapped = live.fields.iter().find(|f| f.name == "emb_chunks");++ let mut files: Vec<(String, u64)> = Vec::new();+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {+ for entry in std::fs::read_dir(dir).unwrap() {+ let entry = entry.unwrap();+ if entry.file_type().unwrap().is_dir() {+ walk(&entry.path(), out);+ } else {+ out.push((+ entry.file_name().to_string_lossy().into_owned(),+ entry.metadata().unwrap().len(),+ ));+ }+ }+ }+ walk(dir.path(), &mut files);+ let fst: u64 = files+ .iter()+ .filter(|(name, _)| name.ends_with(".emb_chunks.fst"))+ .map(|(_, len)| *len)+ .sum();+ let hits = idx.search(&make_search(probe.clone())).await.unwrap();+ eprintln!(+ "#328 dynamic [{label}]: mapped={:?} emb_chunks.fst={fst} hits={}",+ mapped.map(|f| f.field_type),+ hits.total.value+ );++ if stays_lexical {+ assert!(+ mapped.is_some(),+ "[{label}] dynamic mapping must still register an ordinary field \+ under this name"+ );+ assert!(+ fst > 0,+ "[{label}] a dynamically mapped field of the user's own must keep \+ its term dictionary"+ );+ assert_eq!(+ hits.total.value,+ (DOCS / 2) as u64,+ "[{label}] a dynamically mapped field of the user's own must stay \+ queryable, got {} for {probe}",+ hits.total.value+ );+ } else {+ assert!(+ mapped.is_none(),+ "[{label}] a real multi-vector must not acquire a FieldConfig; got {mapped:?}"+ );+ assert_eq!(fst, 0, "[{label}] a real multi-vector must have no `.fst`");+ assert_eq!(+ hits.total.value, 0,+ "[{label}] a lexical clause on a real multi-vector must answer 0"+ );+ }+ // The `dense_vector` itself is unaffected on every arm.+ assert!(+ !files.iter().any(|(name, _)| name.ends_with(".emb.fst")),+ "[{label}] the dense_vector must still have no term dictionary"+ );+ }+}++/// #328, the documented LIMIT of the rule above, asserted rather than assumed.+///+/// `is_undeclared_multi_vector_companion` can only refuse to map a companion it+/// can recognise, and recognising one needs the `dense_vector` to be declared+/// ALREADY. Index the documents first and map `emb` afterwards and dynamic+/// mapping has already written a `double` config for `emb_chunks` — which+/// `declares_non_vector_shaped_field` then yields to, exactly as it would to a+/// user's own declaration, because on the information in the schema the two are+/// the same thing.+///+/// The consequence is bytes, not answers: `<seg>.emb_chunks.fst` survives. That+/// is the safe direction of the trade and the reason this limit is acceptable,+/// so the test asserts BOTH halves — the dictionary is there AND the field+/// still answers — so that a future change cannot quietly convert it into the+/// silent-zero direction.+#[tokio::test]+async fn a_companion_mapped_before_its_vector_keeps_its_dictionary_and_its_answers() {+ const DIMS: usize = 16;+ const DOCS: usize = 50;++ let dir = TempDir::new().unwrap();+ let mut schema = Schema::empty();+ schema+ .fields+ .push(FieldConfig::new("body", FieldType::Text));+ let engine = make_engine(&dir);+ engine.create_index("late", schema).unwrap();+ let idx = engine.get_index("late").unwrap();++ // Documents FIRST — no `emb` in the mapping yet, so `emb_chunks` is just an+ // array of numeric arrays and dynamic mapping registers it as `double`.+ for d in 0..DOCS {+ let v: Vec<f64> = (0..DIMS).map(|dim| (d + dim) as f64 * 0.25).collect();+ idx.index_document(+ Some(format!("d{d}")),+ json!({+ "body": format!("quarterly liquidity evidence {d}"),+ "emb_chunks": [v.clone(), v.clone()],+ }),+ )+ .await+ .unwrap();+ }+ idx.refresh().await.unwrap();++ let before = idx.schema().await;+ let inferred = before+ .fields+ .iter()+ .find(|f| f.name == "emb_chunks")+ .map(|f| f.field_type);+ assert_eq!(+ inferred,+ Some(FieldType::Double),+ "with no `dense_vector` declared, the companion is an ordinary numeric \+ field and must be mapped like one"+ );++ // …and only NOW is the vector declared. From this point on+ // `lexically_typeless_fields` synthesises `emb_chunks` as a companion+ // candidate — and finds the `double` config dynamic mapping already wrote.+ let mut emb = FieldConfig::new("emb", FieldType::Vector);+ emb.options.dimensions = Some(DIMS);+ emb.options.similarity = Some("cosine".to_string());+ idx.add_field(emb).await.unwrap();+ idx.force_merge(1).await.unwrap();++ let mut files: Vec<(String, u64)> = Vec::new();+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {+ for entry in std::fs::read_dir(dir).unwrap() {+ let entry = entry.unwrap();+ if entry.file_type().unwrap().is_dir() {+ walk(&entry.path(), out);+ } else {+ out.push((+ entry.file_name().to_string_lossy().into_owned(),+ entry.metadata().unwrap().len(),+ ));+ }+ }+ }+ walk(dir.path(), &mut files);+ let fst: u64 = files+ .iter()+ .filter(|(name, _)| name.ends_with(".emb_chunks.fst"))+ .map(|(_, len)| *len)+ .sum();+ let hits = idx+ .search(&make_search(json!({"exists": {"field": "emb_chunks"}})))+ .await+ .unwrap();+ eprintln!(+ "#328 mapping-after-data: emb_chunks.fst={fst} exists={}",+ hits.total.value+ );++ // The cost of the limit: the companion's dictionary survives.+ assert!(+ fst > 0,+ "mapping the vector after the data leaves the companion's dictionary in \+ place — that is the documented cost, and it is bytes"+ );+ // The thing that must NOT happen: it is still a live, answering field.+ assert_eq!(+ hits.total.value, DOCS as u64,+ "the companion must stay queryable when the schema says it is an \+ ordinary numeric field"+ );+}+#[tokio::test]async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {let dir = TempDir::new().unwrap();