Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(engine): close the #328 companion residual rather than document it (#328)

xerj-orgcf1904891624
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ec10d219c..1b314655c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -161,8 +161,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
halves are named in that headline because the companion is where most of the
bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is
`<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the
- mapping at all — it is a dynamic field matched by NAME, not a field anyone
- declared `dense_vector`. The field was fed
+ mapping at all — it is matched by NAME, not by anyone declaring it
+ `dense_vector`. The field was fed
through the lexical indexing path and given an FST + postings that no query
path can read: kNN is served by `hnsw/graph.bin`, `exists` from
`_source`/doc values, and `_field_caps` and highlighting never open the
@@ -181,16 +181,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`.wal` is excluded from that total because its tail is reclaimed
asynchronously and it is not reproducible: on the 300-doc fixture the
- committed regression test uses, eight runs put the *whole directory* anywhere
- between 858,147 B and 1,105,001 B while the durable bytes moved by 230 B. The
+ committed regression test uses, runs put the *whole directory* anywhere
+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The
test prints both and asserts on the durable one. It reports
- **3,249,323 B → 857,563–858,071 B** for that fixture (before-side measured by
+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured by
reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same
fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the
- committed test on this revision printed 857,563 / 857,623 / 857,643 /
- 857,659 / 857,787 / 857,827 / 857,859 / 857,955 B; the ceiling it asserts is
- 1,500,000 B, three orders of magnitude clear of that jitter and still 1.7 MB
- below the before-side.
+ committed test on this revision printed 857,265 / 857,529 / 857,533 /
+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling
+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and
+ still 1.7 MB below the before-side.
A `dense_vector` nested under an object mapping is covered too — its
components were landing in the *parent's* term dictionary, because the segment
@@ -208,14 +208,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
**Two rules, and they do not have the same strength.** The base field is
excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two
- companions are excluded BY NAME, because the passage pipeline writes them into
- `_source` with no mapping entry behind them, so there is no declared type to
- read — and a name-based rule YIELDS to a declaration it could not have
- produced itself. **If you have mapped your own field called `<vector>_chunks`
- or `__xerj_passage_meta__<vector>` as `text`, `keyword`, `date`, `boolean` or
- `ip`, nothing about it changes**: it keeps its term dictionary, every query
- naming it keeps answering exactly what it answered before, and only the
- `dense_vector` beside it loses its postings. Measured on
+ companions are excluded BY NAME, because they arrive in `_source` with no
+ mapping entry behind them, so there is no declared type to read — and a
+ name-based rule YIELDS to any declaration it finds. **If you have mapped your
+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as
+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,
+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,
+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,
+ `object`, `nested`, or a type string XERJ does not recognise — nothing about
+ it changes**: it keeps its term dictionary, every query naming it keeps
+ answering exactly what it answered before, and only the `dense_vector` beside
+ it loses its postings. The same holds for a field of that name XERJ mapped for
+ you dynamically, whatever type it inferred, as long as the value is not itself
+ a multi-vector. Measured on
`{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`
(before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,
`terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,
@@ -224,13 +229,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`
54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:
`match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`
- all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A dynamically mapped
- string field of that name is exempt on the same rule (dynamic mapping
- registers it as `text`) — measured 25 → 25.
-
- `long` and `double` are the exception, because they are what dynamic mapping
- infers from a real multi-vector and the rule cannot tell the two apart. See
- the fourth behaviour change below; that case is a hard incompatibility.
+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a
+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,
+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name
+ XERJ mapped for you *dynamically* is exempt on the same rule — measured
+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one
+ (44 B `.fst`).
+
+ **What makes "yields to any declaration" affordable** is that the companion
+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied
+ `_source` — an array of float arrays — so dynamic mapping used to register it
+ like any other unmapped key and infer `double` from it, producing a mapping
+ entry indistinguishable from one you wrote. Dynamic mapping now declines to
+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is
+ already mapped *and* the value is a rectangular, non-empty array of non-empty
+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,
+ a string, a flat array or a ragged one under that name is your field and is
+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`
+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,
+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`
+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is
+ mapped `long`, keeps a 44 B `.fst` and answers 25.
+
+ Both halves were measured on their own, by disabling each in the tree and
+ re-running the committed tests:
+
+ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |
+ |---|---:|---:|---|
+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |
+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |
+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |
+
+ The middle row is the behaviour rc.16 would have shipped; the bottom row is
+ the 67% of the saving that an unconditional "any declaration wins" gives up.
+ Neither cost is paid here.
A lexical clause that NAMES a `dense_vector` (or one of its companions) is now
lowered to `match_none` at plan time rather than falling through to the
@@ -315,45 +347,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`max_score` for a bool that touches a vector field will see this. Plain
lexical queries are unaffected; the blast radius is bools with a
`dense_vector` clause in them.
- - **A `long` or `double` field named `<vector>_chunks` or
- `__xerj_passage_meta__<vector>` stops being lexically searchable.** This is
- the by-name half of the rule above, and it is the one case where it cannot
- yield, because `long`/`double` is precisely what dynamic mapping infers from
- a real multi-vector (an array of float arrays walks to its first scalar).
- Deferring to that declaration was measured and it puts
- `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture — i.e. it
- gives up the larger half of the saving above. So the name wins, and a user's
- own numeric field under that name loses its term dictionary. Measured on
- `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9
- (before → after):
-
- | query | before | after |
- |---|---:|---:|
- | `term {emb_chunks: 7}` | 25 | **0** |
- | `terms {emb_chunks: [7]}` | 25 | **0** |
- | `range {emb_chunks: {gte: 8}}` | 25 | **0** |
- | `bool{filter:[term emb_chunks 7]}` | 25 | **0** |
- | `bool{must:[match_all], must_not:[term emb_chunks 7]}` | 25 | **50** |
- | `terms` agg / `stats` on `emb_chunks` | 25/25, sum 400 | 25/25, sum 400 |
-
- The `must_not` row returns the documents the exclusion was written to
- remove, so treat this as a hard incompatibility rather than a footnote:
- **if you have your own numeric field named `<vector>_chunks`, rename it
- before upgrading.** Any other declared type (`text`, `keyword`, `date`,
- `boolean`, `ip`) is exempt and unchanged, as is a dynamically mapped string
- field of that name. Doc values are never affected on any of these paths —
- the `terms` and `stats` aggregations in the last row keep working on the
- residual case, which is how the rows above were confirmed to be a planning
- change and not data loss.
+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that
+ XERJ mapped for you.** The multi-vector companion beside a declared
+ `dense_vector` used to be registered by dynamic mapping as `double` (an
+ array of float arrays walks to its first scalar). It is not registered at
+ all now, so it disappears from `_mapping` and `_field_caps` — the same
+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads
+ `_source` changes; the values are stored and returned exactly as before, and
+ `knn` and `semantic` on the vector itself are untouched. A field of that
+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector
+ value, is unaffected and still listed.
+
+ One consequence worth knowing: aggregating or sorting on that companion
+ stops working, because it no longer has a mapping entry to aggregate
+ through. On a real multi-vector this was never a meaningful operation, but
+ it was a `200` before and is not now.
- **Existing indices keep their bloat until a merge rewrites them.** This is a
write-side rule. Segments written by an earlier release keep their `.emb.*`
/ `.emb_chunks.*` files, and the per-segment `fts_has_field` gate reads
whichever shape it finds. Run an explicit
`POST /<index>/_forcemerge?max_num_segments=1` to reclaim the bytes on an
- index that already exists. The same applies to the by-name rule above: an
- existing index with a `long`/`double` `<vector>_chunks` that is searchable
- today stops being searchable for those clauses as soon as its segments are
- rewritten — before any mapping change, and with no error.
+ index that already exists.
+
+ The companion rule has an ordering limit of the same kind, and it fails in
+ the safe direction: XERJ can only decline to map a companion it can
+ recognise, and recognising one needs the `dense_vector` to be mapped
+ *before* the documents arrive. Index first and add the mapping afterwards
+ and `<vector>_chunks` already carries a dynamic `double` entry, which the
+ by-name rule then yields to — so that index keeps the companion's term
+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of
+ reclaiming it. It costs bytes, not answers.
*Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,
where XERJ answers `200`. This release removes bytes and adopts Lucene's
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index e3a7ff0e7..639115330 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -18653,7 +18653,9 @@ impl Index {
continue;
};
for (key, val) in obj {
- if key.starts_with(PASSAGE_METADATA_PREFIX) {
+ if key.starts_with(PASSAGE_METADATA_PREFIX)
+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)
+ {
continue;
}
match schema.schema.field(key) {
@@ -18782,7 +18784,9 @@ impl Index {
return;
}
obj.iter().any(|(key, val)| {
- if key.starts_with(PASSAGE_METADATA_PREFIX) {
+ if key.starts_with(PASSAGE_METADATA_PREFIX)
+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)
+ {
return false;
}
match schema.schema.field(key) {
@@ -18829,7 +18833,9 @@ impl Index {
let mut schema_changed = false;
let mut field_count = schema.schema.field_count() as u32;
for (key, val) in obj {
- if key.starts_with(PASSAGE_METADATA_PREFIX) {
+ if key.starts_with(PASSAGE_METADATA_PREFIX)
+ || is_undeclared_multi_vector_companion(&schema.schema, key, val)
+ {
continue;
}
if let Some(existing) = schema.schema.field_mut(key) {
@@ -30164,7 +30170,8 @@ fn unsearchable_query_field(q: &QueryNode, schema: &Schema) -> Option<String> {
/// float array. "Outright" is load-bearing twice over: a `fields: ["emb.*"]`
/// PATTERN is not lowered (see `is_typeless_spec` below — measured 300 of 300
/// on the fixture, on the merge base and here alike), and a `<field>_chunks`
-/// the user mapped as `text`/`keyword`/`date`/`boolean`/`ip` is not in
+/// the user mapped as ANYTHING other than `dense_vector` — or that dynamic
+/// mapping registered from a value that is not a multi-vector — is not in
/// `typeless` at all, so nothing naming it is lowered either
/// (`memtable::lexically_typeless_fields`). On the 300-doc
/// regression fixture that is `wildcard {emb:"0*"}` and `fuzzy {emb:"0"~2}`
@@ -31786,6 +31793,89 @@ fn dynamic_field_config(key: &str, val: &Value, date_detection: bool) -> FieldCo
dynamic_field_config_at_depth(key, val, date_detection, 1)
}
+/// Is `key` the per-document MULTI-VECTOR companion of a `dense_vector` that
+/// is already declared, carrying the value shape RFC #148 gives one?
+///
+/// Dynamic mapping must NOT register such a key. `<vector>_chunks` arrives in
+/// `_source` as an array of float arrays, and `infer_field_type` walks it to
+/// its first scalar and lands on `FieldType::Double` — a `FieldConfig` that is
+/// byte-identical to what a user mapping `"type": "double"` would have
+/// produced. Once that config exists nothing downstream can tell the two
+/// apart, and `memtable::declares_non_vector_shaped_field` is left choosing
+/// between honouring a declaration nobody made (giving the companion back its
+/// term dictionary: measured `<seg>.emb_chunks.fst` 0 B → 1,592,118 B on the
+/// 300-doc `#328` fixture) and overriding a declaration the user DID make
+/// (silently unmapping their own numeric field). Not inventing the config is
+/// what removes the choice.
+///
+/// It is the same refusal the `__xerj_passage_meta__` sidecar already gets in
+/// the same `||` at each of this predicate's call sites
+/// (`key.starts_with(PASSAGE_METADATA_PREFIX)`), and the same modelling Lucene
+/// uses: a per-document multi-vector is `LateInteractionField extends
+/// BinaryDocValuesField`
+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`,
+/// `:44`, `:45`) — its own field type, never a numeric one arrived at by
+/// inference.
+///
+/// THREE conditions, all required, so the blast radius is exactly the
+/// companion:
+///
+/// * the name is `<base>_chunks`;
+/// * `<base>` is ALREADY declared `dense_vector` in the live schema — a
+/// `foo_chunks` with no `foo` vector beside it is an ordinary field and is
+/// mapped as one;
+/// * the value [`looks_like_multi_vector`] — a rectangular, non-empty array of
+/// non-empty all-numeric arrays. A scalar, a string, a flat array or a
+/// ragged one is somebody's own field and is mapped as one.
+///
+/// KNOWN LIMIT, and it fails SAFE: this only fires when the `dense_vector` is
+/// declared BEFORE the documents arrive. Index first and map afterwards and
+/// `<vector>_chunks` already holds a dynamic `double` config, which
+/// `declares_non_vector_shaped_field` then (correctly, on the information it
+/// has) yields to — so the companion keeps its term dictionary. That costs
+/// bytes, not answers. Pinned by
+/// `documents_before_the_mapping_leave_the_companion_mapped_and_lexical`.
+/// ALLOCATION: this runs for every key of every ingested document that dynamic
+/// mapping does not already know, on both the read-lock pre-screen and the
+/// write-lock loop, so it allocates nothing and orders its three tests
+/// cheapest-first: an `O(1)` suffix strip, then an allocation-free schema walk
+/// ([`declares_dense_vector`], which is why `collect_dense_vector_fields`'
+/// `Vec<String>` is not used here), then the value walk.
+fn is_undeclared_multi_vector_companion(schema: &Schema, key: &str, val: &Value) -> bool {
+ let Some(base) = key.strip_suffix(crate::memtable::MULTI_VECTOR_COMPANION_SUFFIX) else {
+ return false;
+ };
+ !base.is_empty()
+ && declares_dense_vector(schema, base)
+ && crate::memtable::looks_like_multi_vector(val)
+}
+
+/// Is `path` declared `dense_vector`? Allocation-free, and resolves BOTH
+/// spellings of a nested name for the same reason
+/// `memtable::declares_non_vector_shaped_field` does: `put_mapping` stores a
+/// dotted path as one top-level `FieldConfig` named `"passages.vec"`, while
+/// `es_properties_to_fields` stores it as a `vec` child under a `passages`
+/// object. The literal lookup runs first so the flat spelling cannot fall
+/// through the segmented walk and be reported as undeclared.
+fn declares_dense_vector(schema: &Schema, path: &str) -> bool {
+ let is_vector = |fc: &FieldConfig| matches!(fc.field_type, FieldType::Vector);
+ if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {
+ return is_vector(field);
+ }
+ let mut fields: &[FieldConfig] = &schema.fields;
+ let mut segments = path.split('.').peekable();
+ while let Some(segment) = segments.next() {
+ let Some(field) = fields.iter().find(|fc| fc.name == segment) else {
+ return false;
+ };
+ if segments.peek().is_none() {
+ return is_vector(field);
+ }
+ fields = &field.fields;
+ }
+ false
+}
+
fn dynamic_field_config_at_depth(
key: &str,
val: &Value,
@@ -31912,6 +32002,26 @@ fn merge_dynamic_children_into(
};
let mut changed = false;
for (key, sub_val) in obj {
+ // The nested form of `is_undeclared_multi_vector_companion`. It needs
+ // no dotted path and no `Schema`, because a companion and its
+ // `dense_vector` are SIBLINGS in the very object being merged: for a
+ // `passages` mapping carrying a `vec` child of type `dense_vector`, the
+ // companion is the `vec_chunks` child right beside it. Checking
+ // `field.fields` locally keeps this function's signature (and its four
+ // call sites) untouched.
+ if key
+ .strip_suffix(crate::memtable::MULTI_VECTOR_COMPANION_SUFFIX)
+ .filter(|base| !base.is_empty())
+ .is_some_and(|base| {
+ field
+ .fields
+ .iter()
+ .any(|f| f.name == base && matches!(f.field_type, FieldType::Vector))
+ })
+ && crate::memtable::looks_like_multi_vector(sub_val)
+ {
+ continue;
+ }
match field.fields.iter_mut().find(|f| f.name == *key) {
Some(existing) => {
if merge_dynamic_children_into(
@@ -40486,6 +40596,187 @@ mod date_detection_tests {
}
}
+/// Unit coverage for the #328 dynamic-mapping refusal — the half of the
+/// companion rule that keeps `memtable::declares_non_vector_shaped_field` free
+/// to yield to any declaration it finds.
+#[cfg(test)]
+mod multi_vector_companion_mapping_tests {
+ use super::*;
+
+ fn vector(name: &str, dims: usize) -> FieldConfig {
+ let mut fc = FieldConfig::new(name, FieldType::Vector);
+ fc.options.dimensions = Some(dims);
+ fc
+ }
+
+ fn schema_with_vector() -> Schema {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("body", FieldType::Text))
+ .unwrap();
+ schema.add_field(vector("emb", 2)).unwrap();
+ schema
+ }
+
+ /// ALL THREE conditions are required. Drop any one and dynamic mapping goes
+ /// back to registering the key like any other.
+ #[test]
+ fn the_refusal_needs_the_name_the_declaration_and_the_shape() {
+ let schema = schema_with_vector();
+ let mv = serde_json::json!([[1.0, 2.0], [3.0, 4.0]]);
+
+ // All three present.
+ assert!(is_undeclared_multi_vector_companion(
+ &schema,
+ "emb_chunks",
+ &mv
+ ));
+
+ // Wrong NAME: same value, no `_chunks` suffix, and a suffix with no
+ // base in front of it.
+ assert!(!is_undeclared_multi_vector_companion(&schema, "emb", &mv));
+ assert!(!is_undeclared_multi_vector_companion(
+ &schema, "matrix", &mv
+ ));
+ assert!(!is_undeclared_multi_vector_companion(
+ &schema, "_chunks", &mv
+ ));
+
+ // No DECLARATION to be a companion of: `other_chunks` has no `other`
+ // vector beside it, so it is an ordinary field even though its value is
+ // rectangular numeric.
+ assert!(!is_undeclared_multi_vector_companion(
+ &schema,
+ "other_chunks",
+ &mv
+ ));
+ // …and a base declared as something that is not a vector does not
+ // create a companion either.
+ let mut lexical_base = Schema::empty();
+ lexical_base
+ .add_field(FieldConfig::new("emb", FieldType::Text))
+ .unwrap();
+ assert!(!is_undeclared_multi_vector_companion(
+ &lexical_base,
+ "emb_chunks",
+ &mv
+ ));
+
+ // Wrong SHAPE: a user's own scalar, string, flat array or ragged array
+ // under the companion's name is still mapped.
+ for own in [
+ serde_json::json!(7),
+ serde_json::json!(7.5),
+ serde_json::json!("tenant-a"),
+ serde_json::json!([1.0, 2.0]),
+ serde_json::json!([[1.0, 2.0], [3.0]]),
+ ] {
+ assert!(
+ !is_undeclared_multi_vector_companion(&schema, "emb_chunks", &own),
+ "{own} is somebody's own field and must be mapped"
+ );
+ }
+ }
+
+ /// Both spellings of a nested vector reach the schema, so both must be
+ /// recognised — the same two shapes
+ /// `nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary`
+ /// exercises. The flat lookup runs first so it cannot fall through the
+ /// segmented walk.
+ #[test]
+ fn declares_dense_vector_resolves_both_nested_spellings() {
+ let mut dotted = Schema::empty();
+ dotted.add_field(vector("passages.vec", 2)).unwrap();
+ assert!(declares_dense_vector(&dotted, "passages.vec"));
+ assert!(!declares_dense_vector(&dotted, "passages"));
+ assert!(!declares_dense_vector(&dotted, "passages.other"));
+
+ let mut sub = Schema::empty();
+ let mut parent = FieldConfig::new("passages", FieldType::Object);
+ parent.fields.push(vector("vec", 2));
+ parent
+ .fields
+ .push(FieldConfig::new("title", FieldType::Text));
+ sub.add_field(parent).unwrap();
+ assert!(declares_dense_vector(&sub, "passages.vec"));
+ assert!(!declares_dense_vector(&sub, "passages.title"));
+ assert!(!declares_dense_vector(&sub, "passages"));
+ // Does not walk past a leaf, and does not invent missing segments.
+ assert!(!declares_dense_vector(&sub, "passages.vec.deeper"));
+ assert!(!declares_dense_vector(&sub, "absent.vec"));
+ }
+
+ /// The nested form of the refusal, which lives in
+ /// `merge_dynamic_children_into` and reads the sibling list directly rather
+ /// than a dotted path. A `vec_chunks` beside a `dense_vector vec` is not
+ /// registered; a `vec_chunks` with a scalar value, or one whose sibling is
+ /// not a vector, is.
+ #[test]
+ fn a_nested_companion_is_not_merged_in_beside_its_vector() {
+ let mut parent = FieldConfig::new("passages", FieldType::Object);
+ parent.fields.push(vector("vec", 2));
+ let mut count = 1u32;
+ let changed = merge_dynamic_children_into(
+ &mut parent,
+ &serde_json::json!({"vec_chunks": [[1.0, 2.0], [3.0, 4.0]], "title": "q3"}),
+ true,
+ 1,
+ &mut count,
+ 100,
+ );
+ assert!(changed, "the ordinary sibling still lands");
+ assert!(
+ parent.fields.iter().any(|f| f.name == "title"),
+ "{parent:?}"
+ );
+ assert!(
+ !parent.fields.iter().any(|f| f.name == "vec_chunks"),
+ "the nested multi-vector companion must not be registered: {parent:?}"
+ );
+ assert_eq!(count, 2, "exactly one field was added");
+
+ // A scalar under the same name IS somebody's own field.
+ let mut own = FieldConfig::new("passages", FieldType::Object);
+ own.fields.push(vector("vec", 2));
+ let mut own_count = 1u32;
+ merge_dynamic_children_into(
+ &mut own,
+ &serde_json::json!({"vec_chunks": 7}),
+ true,
+ 1,
+ &mut own_count,
+ 100,
+ );
+ assert_eq!(
+ own.fields
+ .iter()
+ .find(|f| f.name == "vec_chunks")
+ .map(|f| f.field_type),
+ Some(FieldType::Long),
+ "{own:?}"
+ );
+
+ // …and so is a rectangular numeric array with no vector sibling.
+ let mut no_vector = FieldConfig::new("passages", FieldType::Object);
+ no_vector
+ .fields
+ .push(FieldConfig::new("vec", FieldType::Text));
+ let mut nv_count = 1u32;
+ merge_dynamic_children_into(
+ &mut no_vector,
+ &serde_json::json!({"vec_chunks": [[1.0, 2.0]]}),
+ true,
+ 1,
+ &mut nv_count,
+ 100,
+ );
+ assert!(
+ no_vector.fields.iter().any(|f| f.name == "vec_chunks"),
+ "no `dense_vector` sibling means no companion: {no_vector:?}"
+ );
+ }
+}
+
#[cfg(test)]
mod fts_projection_tests {
use super::*;
diff --git a/engine/crates/xerj-engine/src/memtable.rs b/engine/crates/xerj-engine/src/memtable.rs
index 82d284fdf..201eab085 100644
--- a/engine/crates/xerj-engine/src/memtable.rs
+++ b/engine/crates/xerj-engine/src/memtable.rs
@@ -3917,20 +3917,23 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
///
/// The row is the DURABLE index — every file in the data directory except
/// `.wal` — because the write-ahead log's tail is reclaimed asynchronously and
-/// the whole-directory total is therefore not reproducible: eight runs of the
-/// 300-doc regression test put it anywhere between 858,147 B and 1,105,001 B
-/// while the durable bytes moved by 230 B. `.seg` (the stored documents, 11.2 MB
+/// the whole-directory total is therefore not reproducible: runs of the 300-doc
+/// regression test have put it anywhere between 858,147 B and 1,105,001 B while
+/// the durable bytes moved by ~500 B. `.seg` (the stored documents, 11.2 MB
/// here) is counted on both sides and is why the ratio is 3.79× rather than the
/// 39.8 MB the term dictionaries cost outright.
///
/// The 300-doc × 128-dim fixture the regression tests actually run is the same
/// shape one order of magnitude down: `emb.fst` 796,274 B, `emb.post` 591 B,
/// `emb.norms` 313 B, `emb_chunks.fst` 1,592,118 B, `emb_chunks.post` 591 B,
-/// durable index 3,249,323 B → 857,563–858,071 B (eight runs on this revision
-/// printed 857,563 / 857,623 / 857,643 / 857,659 / 857,787 / 857,827 /
-/// 857,859 / 857,955). That test
-/// prints both totals and asserts a ceiling on the durable one, so this figure
-/// is enforced rather than merely published.
+/// durable index 3,249,323 B → 857,265–857,769 B (eight runs on this revision
+/// printed 857,265 / 857,529 / 857,533 / 857,545 / 857,549 / 857,657 /
+/// 857,657 / 857,769, a 504 B spread). That test prints both totals and asserts
+/// a ceiling on the durable one, so this figure is enforced rather than merely
+/// published. The same eight runs are quoted verbatim in `CHANGELOG.md` and in
+/// the comment above the `eprintln!` in
+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary`; all three move
+/// together or none of them do.
///
/// TWO RULES DECIDE THIS SET, and they are stated separately because they do
/// not have the same strength. [`lexically_typeless_fields`] has the full
@@ -3945,19 +3948,21 @@ pub fn semantic_derived_vector_fields(schema: &Schema) -> std::collections::Hash
/// the vector half is droppable — that needs a value-shape-aware rule, not
/// a type rule.
/// * BY NAME, for the two companions `<field>_chunks` and
-/// `__xerj_passage_meta__<field>`, which the passage pipeline writes into
-/// `_source` with no `FieldConfig` behind them, so there is no declared type
-/// to read. This rule is the weaker of the two and YIELDS to a declaration
-/// it could not have produced itself: a field mapped `text`, `keyword`,
-/// `date`, `boolean`, `ip`, `chunk` or `object` under one of those names
-/// keeps its full lexical surface. It does NOT yield to `long`/`double`,
-/// which is exactly what dynamic mapping infers from a real multi-vector —
-/// the residual, with its measured cost and its measured A/B, is on
-/// [`lexically_typeless_fields`] and in `CHANGELOG.md`.
+/// `__xerj_passage_meta__<field>`, which arrive in `_source` with no
+/// `FieldConfig` behind them, so there is no declared type to read. This
+/// rule is the weaker of the two and YIELDS to ANY declaration: a field
+/// mapped as anything other than `dense_vector` under one of those names —
+/// `text`, `keyword`, `long`, `double`, `date`, `boolean`, `ip`,
+/// `geo_point`, `binary`, `object`, `nested`, `chunk`, or an ES type string
+/// `es_type_to_field_type` does not recognise — keeps its full lexical
+/// surface. That is only sound because the companion is prevented from ever
+/// ACQUIRING a declaration: `index::is_undeclared_multi_vector_companion`
+/// stops dynamic mapping registering a `<vector>_chunks` key whose value is
+/// a real multi-vector, so an "absent" answer here means absent.
///
/// The bytes in the table above are almost all the second rule's: `emb_chunks`
-/// in that fixture is a DYNAMIC field, absent from the user's mapping and
-/// registered as `Double` by dynamic mapping, and it is 26,518,005 B of the
+/// in that fixture is absent from the user's mapping — and, since this change,
+/// absent from the evolved schema too — and it is 26,518,005 B of the
/// 39,796,380 B removed.
///
/// Removing the postings is only half of it, and the other half is NOT the
@@ -4067,6 +4072,55 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>
excluded
}
+pub const MULTI_VECTOR_COMPANION_SUFFIX: &str = "_chunks";
+
+/// Is `val` the exact `_source` shape RFC #148 gives a per-document
+/// MULTI-VECTOR — a non-empty array of equal-length, non-empty, all-numeric
+/// arrays?
+///
+/// This is the value-shape half of the companion rule. The schema alone cannot
+/// tell a multi-vector from a user's own numeric field (see
+/// [`declares_non_vector_shaped_field`]); the VALUE can, and it is available at
+/// exactly the moment that matters — when dynamic mapping is about to invent a
+/// `FieldConfig` for an unmapped `<vector>_chunks` key
+/// (`index::evolve_schema_from_doc`/`_docs`). Refusing to invent one there is
+/// what keeps the schema honest, so the plan-time side of the rule can then
+/// defer to ANY declaration it does find.
+///
+/// The equal-length requirement is not decoration: every chunk of one document
+/// embeds with the same model, so a real multi-vector is rectangular. A ragged
+/// array of numeric arrays is therefore NOT one, is left to dynamic mapping,
+/// and keeps its term dictionary — the fail-safe direction.
+///
+/// Lucene models the same object as its own field type rather than inferring a
+/// numeric one: `LateInteractionField extends BinaryDocValuesField`
+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`),
+/// constructed from a `float[][]` and encoded straight to a `BytesRef`
+/// (`:44`, `:45`) — it never reaches an analyzer or a term dictionary at all.
+pub(crate) fn looks_like_multi_vector(val: &serde_json::Value) -> bool {
+ let Some(rows) = val.as_array() else {
+ return false;
+ };
+ let mut dims: Option<usize> = None;
+ if rows.is_empty() {
+ return false;
+ }
+ for row in rows {
+ let Some(components) = row.as_array() else {
+ return false;
+ };
+ if components.is_empty() || !components.iter().all(serde_json::Value::is_number) {
+ return false;
+ }
+ match dims {
+ None => dims = Some(components.len()),
+ Some(d) if d == components.len() => {}
+ Some(_) => return false,
+ }
+ }
+ true
+}
+
/// Fields whose DECLARED type has no lexical representation at all — #328.
///
/// Today that is exactly `FieldType::Vector` (`dense_vector`), so the base list
@@ -4103,29 +4157,47 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>
/// from the base name because no USER mapping declares them — they arrive in
/// `_source` from the passage pipeline, and whatever `FieldConfig` they end up
/// with is one dynamic mapping inferred from their values. A name-based rule
-/// can collide with a field the user actually mapped, so it yields to one —
-/// but only where the declaration rules the collision out:
+/// can collide with a field the user actually mapped, so it yields to ANY
+/// declaration it finds:
///
-/// * if the schema declares `<field>_chunks` (or the sidecar) with a type a
-/// MULTI-VECTOR could not have produced — `text`, `keyword`, `date`,
-/// `boolean`, `ip`, `chunk`, `object` — that declaration wins and the field
-/// keeps its term dictionary and its whole lexical surface. Lucene draws
-/// exactly this line: whether a field gets a term dictionary is read off the
-/// field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,
+/// * declared as anything other than `dense_vector` — `text`, `keyword`,
+/// `long`, `double`, `date`, `boolean`, `ip`, `geo_point`, `binary`,
+/// `object`, `nested`, `chunk`, or a mapping type
+/// `es_type_to_field_type` did not recognise — the declaration wins and the
+/// field keeps its term dictionary and its whole lexical surface. Lucene
+/// draws exactly this line: whether a field gets a term dictionary is read
+/// off the field's own `IndexableFieldType` (`IndexingChain.invertAndStore`,
/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),
/// never off its name, and one name meaning two different things is an
/// outright error rather than a silent downgrade
/// (`IndexingChain.FieldSchema.setIndexOptions` → `assertSame` →
-/// `raiseNotSame`, `IndexingChain.java:2213` and `:2193`);
-/// * `double` / `long` / absent → the name rule applies. Not a shortcut: it is
-/// what the companion ITSELF declares. `<field>_chunks` is user-supplied
-/// `_source` (an array of float arrays), so dynamic mapping registers it
-/// like any other unmapped key and `infer_field_type` lands on
-/// `FieldType::Double`. Deferring to that declaration was measured and it
-/// brings `<seg>.emb_chunks.fst` back at 1,592,118 B on the 300-doc fixture
-/// — the whole 26,518,005 B of the 39,796,380 B saving that lives in the
-/// companion at 5,000 × 128. See [`declares_non_vector_shaped_field`] for
-/// the residual this leaves and what it costs to close.
+/// `raiseNotSame`, `IndexingChain.java:2213`, `:2187` and `:2193`);
+/// * absent → the name rule applies, and this is the companion's own case.
+///
+/// THAT SECOND BULLET IS ONLY SAFE BECAUSE THE COMPANION IS KEPT UNDECLARED.
+/// `<field>_chunks` is user-supplied `_source` (an array of float arrays), so
+/// dynamic mapping used to register it like any other unmapped key —
+/// `infer_field_type` walks to the first scalar and lands on
+/// `FieldType::Double`, a `FieldConfig` byte-identical to a user's own
+/// `"type": "double"`. That is what forced an earlier revision to carve
+/// `double`/`long` out of the yield set above, and with them a real user field
+/// under that name. `index::is_undeclared_multi_vector_companion` removes the
+/// ambiguity at its source instead: dynamic mapping refuses to invent the
+/// config when the value [`looks_like_multi_vector`], so the only `double` or
+/// `long` under this name is one somebody wrote.
+///
+/// BOTH HALVES ARE LOAD-BEARING, measured by disabling each on its own against
+/// the committed 300-doc × 128-dim fixture and the 50-doc collision fixture:
+///
+/// | | `.emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |
+/// |---|---|---|---|
+/// | here | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |
+/// | yield-to-numeric restored | 0 B | 857,445 B | **0 B `.fst`, answers 0** |
+/// | refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |
+///
+/// The middle row is the residual this replaces; the bottom row is the 67% of
+/// the saving an unconditional "any declaration wins" gives up. Neither is
+/// paid for here.
///
/// This is deliberately NOT the contract `semantic_derived_vector_fields`
/// carries: that one synthesises `<target>_chunks` unconditionally, and a
@@ -4192,31 +4264,38 @@ pub fn fts_excluded_fields(schema: &Schema) -> std::collections::HashSet<String>
/// The `match_all`-anchored form is the one the tests pin, because it is the
/// one that is correct on `main` to begin with.
///
-/// THE RESIDUAL, stated in full because it is the same class and it is NOT
-/// fixed: a field the user declares `long` or `double` under one of these two
-/// names is still excluded by name, and the plan-time lowering still zeroes
-/// every lexical clause on it. Measured on
-/// `{text body, long emb_chunks, dense_vector emb}`, 50 docs, values 7 and 9 —
-/// merge base `ca4d75a` → here:
+/// THE NUMERIC ARM, which an earlier revision of this branch shipped as an
+/// accepted residual and which is now closed. On that revision a field the user
+/// declared `long` or `double` under one of these two names was still excluded
+/// by name and the plan-time lowering still zeroed every lexical clause on it.
+/// Measured on `{text body, long emb_chunks, dense_vector emb}`, 50 docs,
+/// values 7 and 9 — merge base `ca4d75a` → that revision → here:
///
-/// `term {emb_chunks:7}` 25 → **0**
-/// `terms {emb_chunks:[7]}` 25 → **0**
-/// `range {emb_chunks:{gte:8}}` 25 → **0**
-/// `bool{filter:[term emb_chunks 7]}` 25 → **0**
-/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50**
-/// `terms` agg / `stats` on `emb_chunks` 25/25, sum 400 → 25/25, sum 400
+/// `term {emb_chunks:7}` 25 → **0** → 25
+/// `terms {emb_chunks:[7]}` 25 → **0** → 25
+/// `range {emb_chunks:{gte:8}}` 25 → **0** → 25
+/// `bool{filter:[term emb_chunks 7]}` 25 → **0** → 25
+/// `bool{must:[match_all],must_not:[term 7]}` 25 → **50** → 25
+/// `terms` agg / `stats` on `emb_chunks` 25/25 on all three
///
-/// It is narrower than what it replaces — the reported case, a `keyword`
-/// tenant discriminator, is now byte-identical to the merge base — but it is
-/// the same failure shape, and closing it needs a decision this function cannot
-/// make on its own: either give the schema per-field PROVENANCE (explicit
-/// mapping vs. dynamically inferred) so an explicit `double` can outrank the
-/// name, or drop the by-name rule entirely and give up 67% of the saving. Both
-/// are larger than #328. Documented as a behaviour change in `CHANGELOG.md`
-/// and pinned by `a_numerically_declared_sibling_is_still_excluded_by_name`
-/// here and the `[long]` arm of
-/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in
-/// `tests/integration.rs`, so it cannot drift silently.
+/// The `double` arm and the DYNAMICALLY mapped numeric case (a scalar `7`/`9`
+/// under this name with no mapping at all, which dynamic mapping registers as
+/// `long`) behave identically and are pinned alongside it. It was NOT closed by
+/// per-field schema provenance and NOT by giving up the by-name rule — the two
+/// options an earlier revision of this comment called exhaustive, wrongly. It
+/// is closed by not manufacturing the ambiguous declaration in the first place
+/// (`index::is_undeclared_multi_vector_companion`), which costs nothing: the
+/// 300-doc fixture still reports `emb_chunks.fst=0` and durable 857,265–857,769 B.
+///
+/// THE LIMIT THAT REMAINS, and it fails safe: the refusal needs the
+/// `dense_vector` declared BEFORE the documents arrive. Index first and map
+/// afterwards and `emb_chunks` already carries a dynamic `double` config, which
+/// this function then yields to — measured `.emb_chunks.fst` = 9,467 B on a
+/// 50-doc × 16-dim fixture, with `exists {field: emb_chunks}` still answering
+/// 50. That costs bytes, not answers, and is pinned by
+/// `a_companion_mapped_before_its_vector_keeps_its_dictionary_and_its_answers`
+/// in `tests/integration.rs`. The nested-under-an-object spelling has the same
+/// shape of limit and the same direction.
///
/// Pinned by `a_declared_sibling_keeps_its_lexical_surface` here and by
/// `declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one` in
@@ -4238,7 +4317,7 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S
for field in crate::index::collect_dense_vector_fields(schema) {
// BY NAME, and only where no declaration contradicts it.
for synthesised in [
- format!("{field}_chunks"),
+ format!("{field}{MULTI_VECTOR_COMPANION_SUFFIX}"),
format!("{}{}", xerj_query::executor::PASSAGE_METADATA_PREFIX, field),
] {
if !declares_non_vector_shaped_field(schema, &synthesised) {
@@ -4259,33 +4338,41 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S
/// rather than using `Schema::has_field`, which only sees root names and would
/// answer `false` for every nested companion.
///
-/// THE LINE IS DRAWN AT "COULD THIS DECLARATION HAVE COME FROM THE COMPANION
-/// ITSELF", and it is drawn there because the companion is not always
-/// undeclared. `<field>_chunks` is user-supplied `_source` (RFC #148's
-/// per-document multi-vector: an array of float arrays), so DYNAMIC MAPPING
-/// registers it like any other unmapped key — `infer_field_type` walks to the
-/// first non-null element, twice, and lands on `FieldType::Double`
-/// (`index.rs`'s `Value::Array` arm). Measured: with "any declaration wins",
-/// `<seg>.emb_chunks.fst` comes back at 1,592,118 B on the 300-doc fixture and
-/// the whole 67% of the saving that lives in the companion is lost, because
-/// dynamic mapping had already declared it.
+/// THE RULE IS ITS COMPLEMENT, NOT A LIST: this yields for EVERY declared type
+/// except `dense_vector`. It is stated that way on purpose — an earlier
+/// revision enumerated `Text`/`Keyword`/`Date`/`Boolean`/`Ip`/`Chunk`/`Object`
+/// and read as exhaustive when it was not (`GeoPoint`, `Binary` and `Nested`
+/// also yielded, and `es_type_to_field_type` maps every UNRECOGNISED ES type
+/// string to `Object`, so unknown mapping types yielded too). So:
///
-/// So:
+/// * `Vector` — already in the set by declared type; `false` is a no-op, and
+/// the companion of a companion is not a thing.
+/// * anything else — a declaration is a declaration. The user mapped a field
+/// under this name, so the name rule yields and the field keeps its term
+/// dictionary and its whole lexical surface: `text`, `keyword`, `long`,
+/// `double`, `date`, `boolean`, `ip`, `geo_point`, `binary`, `object`,
+/// `nested`, `chunk`, and anything `es_type_to_field_type` did not
+/// recognise.
+/// * absent — nothing to defer to; the name rule applies. This is the
+/// companion's own case, and it is the ONLY one, because
+/// `index::evolve_schema_from_doc`/`_docs` refuse to invent a `FieldConfig`
+/// for an unmapped `<vector>_chunks` key whose value
+/// [`looks_like_multi_vector`]. Without that refusal this predicate would be
+/// answering about a declaration DYNAMIC MAPPING wrote: `infer_field_type`
+/// walks an array of float arrays to its first scalar and lands on
+/// `FieldType::Double`, indistinguishable from a user's own `double`. That
+/// is why an earlier revision had to carve `Double`/`Long` out of the yield
+/// set, and why a user's own numeric field under this name silently lost its
+/// lexical surface. The carve-out is gone; the ambiguity is prevented at the
+/// only place it was ever manufactured.
///
-/// * `Double` / `Long` — the shape `infer_field_type` produces from an array
-/// of float (or integer) arrays. Indistinguishable from the companion, so
-/// the name rule applies and this returns `false`. This is the residual
-/// case: a user's OWN numeric field named `<vector>_chunks` loses its term
-/// dictionary. It does not lose its data — doc values, `terms`/`range`
-/// aggregations and sorts are untouched — and the measured cost of the
-/// alternative is the entire companion saving.
-/// * `Vector` — already in the set by declared type; `false` is a no-op.
-/// * `Text` / `Keyword` / `Date` / `Boolean` / `Ip` / `Chunk` / `Object` — a
-/// multi-vector cannot infer to any of these, so a declaration of one is
-/// unambiguously the user's own field and the name rule yields. This is the
-/// case that matters: a tenant or ACL discriminator named `emb_chunks` keeps
-/// its whole lexical surface.
-/// * absent — nothing to defer to; the name rule applies.
+/// Lucene draws exactly this line: whether a field gets a term dictionary is
+/// read off the field's own `IndexableFieldType`
+/// (`IndexingChain.invertAndStore`,
+/// `lucene/core/src/java/org/apache/lucene/index/IndexingChain.java:1416`),
+/// never off its name, and one name meaning two different things is an outright
+/// error rather than a silent downgrade (`FieldSchema.setIndexOptions` →
+/// `assertSame` → `raiseNotSame`, `IndexingChain.java:2213`, `:2187`, `:2193`).
///
/// BOTH SPELLINGS of a nested name are resolved, because both reach the schema.
/// `put_mapping` stores a dotted path as ONE top-level `FieldConfig` named
@@ -4295,12 +4382,8 @@ pub fn lexically_typeless_fields(schema: &Schema) -> std::collections::HashSet<S
/// exercises for the vector itself. The literal lookup runs first so the flat
/// spelling cannot fall through the segmented walk and be treated as undeclared.
fn declares_non_vector_shaped_field(schema: &Schema, path: &str) -> bool {
- let is_lexical = |fc: &xerj_common::types::FieldConfig| {
- !matches!(
- fc.field_type,
- FieldType::Vector | FieldType::Double | FieldType::Long
- )
- };
+ let is_lexical =
+ |fc: &xerj_common::types::FieldConfig| !matches!(fc.field_type, FieldType::Vector);
if let Some(field) = schema.fields.iter().find(|fc| fc.name == path) {
return is_lexical(field);
}
@@ -4653,15 +4736,16 @@ mod lexically_typeless_fields_tests {
}
}
- /// THE RESIDUAL, pinned so it cannot drift into an accident. A `Double` or
- /// `Long` declaration is exactly what dynamic mapping produces from the
- /// real multi-vector (`infer_field_type` walks an array of float arrays to
- /// its first scalar), so the name rule cannot yield to it without losing
- /// the whole companion saving — measured at `<seg>.emb_chunks.fst`
- /// 1,592,118 B reappearing on the 300-doc fixture. Documented in the
- /// CHANGELOG as a behaviour change rather than fixed.
+ /// THE NUMERIC ARM, which used to be the accepted residual: a `Double` or
+ /// `Long` declaration is exactly what dynamic mapping USED to produce from
+ /// a real multi-vector, so the name rule could not yield to it without
+ /// losing the whole companion saving. It yields now, because
+ /// `index::is_undeclared_multi_vector_companion` stops that config from
+ /// being invented — so a `long`/`double` under this name is necessarily
+ /// somebody's own field. This assertion is the inverse of the one it
+ /// replaces; if the carve-out ever comes back it fails here first.
#[test]
- fn a_numerically_declared_sibling_is_still_excluded_by_name() {
+ fn a_numerically_declared_sibling_keeps_its_lexical_surface() {
for declared in [FieldType::Double, FieldType::Long] {
let mut schema = Schema::empty();
schema
@@ -4669,11 +4753,92 @@ mod lexically_typeless_fields_tests {
.unwrap();
schema.add_field(vector("emb", 8)).unwrap();
assert!(
- lexically_typeless_fields(&schema).contains("emb_chunks"),
- "a `{declared:?}` declaration is the multi-vector's own inferred \
- shape and must not rescue the name"
+ !lexically_typeless_fields(&schema).contains("emb_chunks"),
+ "a `{declared:?}` declaration is somebody's own field and the \
+ name rule must yield to it"
);
- assert!(!declares_non_vector_shaped_field(&schema, "emb_chunks"));
+ assert!(declares_non_vector_shaped_field(&schema, "emb_chunks"));
+ }
+ }
+
+ /// The yield set is the COMPLEMENT of `Vector`, not the enumerated list an
+ /// earlier doc comment gave. `GeoPoint`, `Binary` and `Nested` were never
+ /// in that list and always yielded, and `es_type_to_field_type` maps every
+ /// unrecognised ES type string to `Object`, so an unknown mapping type
+ /// yields too. Pinned across every variant so the doc and the code cannot
+ /// drift apart again.
+ #[test]
+ fn every_declared_type_except_dense_vector_keeps_its_lexical_surface() {
+ for declared in [
+ FieldType::Text,
+ FieldType::Keyword,
+ FieldType::Long,
+ FieldType::Double,
+ FieldType::Boolean,
+ FieldType::Date,
+ FieldType::Ip,
+ FieldType::GeoPoint,
+ FieldType::Binary,
+ FieldType::Object,
+ FieldType::Nested,
+ ] {
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("emb_chunks", declared))
+ .unwrap();
+ schema.add_field(vector("emb", 8)).unwrap();
+ assert!(
+ declares_non_vector_shaped_field(&schema, "emb_chunks"),
+ "`{declared:?}` is a declaration and must be yielded to"
+ );
+ let typeless = lexically_typeless_fields(&schema);
+ assert!(
+ !typeless.contains("emb_chunks"),
+ "`{declared:?}` sibling excluded anyway: {typeless:?}"
+ );
+ // …while the vector itself is excluded on every one of them.
+ assert!(typeless.contains("emb"));
+ }
+ }
+
+ /// The value-shape half of the rule, exhaustively. Only a rectangular,
+ /// non-empty array of non-empty all-numeric arrays is a multi-vector;
+ /// everything else is somebody's field and keeps its mapping.
+ #[test]
+ fn looks_like_multi_vector_accepts_only_the_rfc_148_shape() {
+ for yes in [
+ serde_json::json!([[1.0, 2.0], [3.0, 4.0]]),
+ serde_json::json!([[1, 2, 3]]),
+ serde_json::json!([[0.5]]),
+ ] {
+ assert!(looks_like_multi_vector(&yes), "should accept {yes}");
+ }
+ for no in [
+ // scalars and strings — a user's own field
+ serde_json::json!(7),
+ serde_json::json!(7.5),
+ serde_json::json!("tenant-a"),
+ serde_json::json!(true),
+ serde_json::json!(null),
+ serde_json::json!({"a": 1}),
+ // a FLAT numeric array is a single vector, not a multi-vector, and
+ // is not what this rule is about
+ serde_json::json!([1.0, 2.0, 3.0]),
+ // empty outer, empty inner
+ serde_json::json!([]),
+ serde_json::json!([[]]),
+ // RAGGED — cannot have come from one embedding model, so it is left
+ // to dynamic mapping (the fail-safe direction)
+ serde_json::json!([[1.0, 2.0], [3.0]]),
+ // non-numeric elements
+ serde_json::json!([["a", "b"]]),
+ serde_json::json!([[1.0, "b"]]),
+ serde_json::json!([[1.0], null]),
+ serde_json::json!([[1.0], 2.0]),
+ // array of objects
+ serde_json::json!([{"a": 1}]),
+ ] {
+ assert!(!looks_like_multi_vector(&no), "should reject {no}");
}
}
diff --git a/engine/crates/xerj-engine/tests/integration.rs b/engine/crates/xerj-engine/tests/integration.rs
index e7ea4ee42..849902872 100644
--- a/engine/crates/xerj-engine/tests/integration.rs
+++ b/engine/crates/xerj-engine/tests/integration.rs
@@ -250,13 +250,16 @@ async fn user_mapped_dense_vector_builds_no_fts_term_dictionary() {
//
// TWO totals, and only the second one is quotable. The whole data dir
// includes the write-ahead log, whose tail is reclaimed asynchronously, so
- // it is NOT reproducible: eight runs of this test on one machine printed a
- // whole-dir total between 858,147 B and 1,105,001 B, a 29% spread, entirely
- // because the surviving `.wal` files ranged from 256 B to 6 MB while the
- // segment bytes did not move at all. `durable-index` excludes `.wal` and is
- // reproducible to ~230 B (the run-to-run jitter in the HNSW `graph.bin`):
- // the same eight runs printed 857,843 – 858,071 B. That is the number the
- // CHANGELOG quotes and the one the ceiling below pins.
+ // it is NOT reproducible: the eight runs below printed a whole-dir total
+ // between 1,069,604 B and 1,104,599 B, and earlier revisions of this test
+ // have printed as little as 858,147 B, entirely because the surviving
+ // `.wal` files vary by megabytes while the segment bytes do not move at
+ // all. `durable-index` excludes `.wal` and is reproducible to ~500 B (the
+ // run-to-run jitter in the HNSW `graph.bin`): eight runs on THIS revision
+ // printed 857,265 / 857,529 / 857,533 / 857,545 / 857,549 / 857,657 /
+ // 857,657 / 857,769 B — a 504 B spread. That range is what the CHANGELOG
+ // quotes and the one the ceiling below pins. Re-measure all three places
+ // together (here, `memtable.rs`'s table, `CHANGELOG.md`) if it moves.
let whole_dir: u64 = files.iter().map(|(_, len)| *len).sum();
let durable: u64 = files
.iter()
@@ -921,15 +924,21 @@ async fn nested_dense_vector_is_excluded_from_its_parent_objects_term_dictionary
/// error rather than a silent downgrade (`IndexingChain.FieldSchema
/// .setIndexOptions` → `assertSame` → `raiseNotSame`, `:2213` / `:2193`).
///
-/// THREE ARMS, and the third one asserts the OPPOSITE. `keyword` and `text`
-/// yield: the sibling keeps its `.fst` and every answer above. `long` does NOT
-/// yield, because `long`/`double` is exactly what dynamic mapping infers from
-/// a real multi-vector (`infer_field_type` on an array of float arrays), so
-/// deferring to it would put `<seg>.emb_chunks.fst` back at 1,592,118 B on the
-/// 300-doc fixture and give up the larger half of #328's saving. That arm
-/// therefore requires 0 hits and `must_not` = 50 — the documented residual,
-/// asserted rather than left to chance, so it cannot silently widen. The
-/// aggregation is checked on all three arms and buckets 25/25 on all three.
+/// FOUR ARMS AND THEY ALL YIELD, including the two numeric ones. An earlier
+/// revision of this test asserted the OPPOSITE for `long`: the name rule could
+/// not yield to `long`/`double` because that is what `infer_field_type` derives
+/// from an array of float arrays, so the declaration was indistinguishable from
+/// the multi-vector's own. That residual is closed at its source —
+/// `index::is_undeclared_multi_vector_companion` stops dynamic mapping
+/// inventing the numeric `FieldConfig` in the first place, so the only `long`
+/// or `double` under this name is one a user actually wrote, and
+/// `declares_non_vector_shaped_field` now yields to every declared type except
+/// `dense_vector`. Measured here: the `[long]` arm keeps `.emb_chunks.fst` at
+/// 44 B and answers 25 on every row, while the 300-doc fixture in
+/// `user_mapped_dense_vector_builds_no_fts_term_dictionary` still reports
+/// `emb_chunks.fst=0` — the saving is not paid for with the collision.
+///
+/// The aggregation is checked on all four arms and buckets 25/25 on all four.
#[tokio::test]
async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_looks_like_one() {
const DIMS: usize = 16;
@@ -942,17 +951,18 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0
};
- for declared in [FieldType::Keyword, FieldType::Text, FieldType::Long] {
+ for declared in [
+ FieldType::Keyword,
+ FieldType::Text,
+ FieldType::Long,
+ FieldType::Double,
+ ] {
let label = match declared {
FieldType::Keyword => "keyword",
FieldType::Text => "text",
- _ => "long",
+ FieldType::Long => "long",
+ _ => "double",
};
- // `long`/`double` is the RESIDUAL arm: it is what dynamic mapping
- // infers from a real multi-vector, so the name rule cannot yield to it
- // and this field IS still excluded. Pinned here, opposite in sign to
- // the other two arms, so the limitation cannot drift into an accident.
- let yields = !matches!(declared, FieldType::Long);
let dir = TempDir::new().unwrap();
let mut schema = Schema::empty();
schema
@@ -986,9 +996,13 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
"{tenant} {}",
if d % 2 == 0 { "widgets" } else { "gadgets" }
)),
- // The residual arm: a scalar number, which is the shape
- // `infer_field_type` also produces from a real multi-vector.
- _ => json!(if d % 2 == 0 { 7 } else { 9 }),
+ // The two numeric arms. A SCALAR, which is what a user's own
+ // `long`/`double` field looks like — and, decisively, not the
+ // rectangular array-of-numeric-arrays `looks_like_multi_vector`
+ // recognises, so nothing about this value is companion-shaped
+ // either.
+ FieldType::Long => json!(if d % 2 == 0 { 7 } else { 9 }),
+ _ => json!(if d % 2 == 0 { 7.5 } else { 9.5 }),
};
idx.index_document(
Some(format!("d{d}")),
@@ -1032,15 +1046,14 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
bytes_of(".emb.fst"),
bytes_of(".body.fst"),
);
- // The DECLARED sibling keeps its term dictionary — except on the
- // residual arm, where the declaration is indistinguishable from the
- // multi-vector's own inferred type and the name rule wins.
- assert_eq!(
+ // The DECLARED sibling keeps its term dictionary. On ALL FOUR arms —
+ // this is the assertion that fails if the numeric carve-out ever comes
+ // back.
+ assert!(
files
.iter()
.any(|(name, _)| name.ends_with(".emb_chunks.fst")),
- yields,
- "[{label}] `.emb_chunks.fst` present should be {yields}; got {:?}",
+ "[{label}] a declared sibling must keep `.emb_chunks.fst`; got {:?}",
files
.iter()
.filter(|(name, _)| name.contains(".emb"))
@@ -1054,11 +1067,9 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
// ── The answers. Every row here is 25 on the merge base `ca4d75a`; the
// unconditional name-only rule answered 0 for all of them, and 50 for
- // the `must_not` shape below. On the two YIELDING arms this test
- // requires 25 — the merge base's own answer. On the residual `long`
- // arm it requires 0, which is what the name rule produces and what the
- // CHANGELOG documents as a behaviour change.
- let want = if yields { (DOCS / 2) as u64 } else { 0 };
+ // the `must_not` shape below. All four arms require 25 — the merge
+ // base's own answer.
+ let want = (DOCS / 2) as u64;
let probes: Vec<(&str, Value)> = if matches!(declared, FieldType::Keyword) {
vec![
("term", json!({"term": {"emb_chunks": "tenant-a"}})),
@@ -1089,6 +1100,16 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
json!({"bool": {"filter": [{"term": {"emb_chunks": 7}}]}}),
),
]
+ } else if matches!(declared, FieldType::Double) {
+ vec![
+ ("term", json!({"term": {"emb_chunks": 7.5}})),
+ ("terms", json!({"terms": {"emb_chunks": [7.5]}})),
+ ("range", json!({"range": {"emb_chunks": {"gte": 8}}})),
+ (
+ "bool-filter",
+ json!({"bool": {"filter": [{"term": {"emb_chunks": 7.5}}]}}),
+ ),
+ ]
} else {
vec![
("match", json!({"match": {"emb_chunks": "widgets"}})),
@@ -1136,6 +1157,11 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
"must": [{"match_all": {}}],
"must_not": [{"term": {"emb_chunks": 7}}]
}})
+ } else if matches!(declared, FieldType::Double) {
+ json!({"bool": {
+ "must": [{"match_all": {}}],
+ "must_not": [{"term": {"emb_chunks": 7.5}}]
+ }})
} else {
json!({"bool": {
"must": [{"match_all": {}}],
@@ -1146,38 +1172,30 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
.search(&make_search(must_not_probe.clone()))
.await
.unwrap();
- // On the yielding arms the exclusion still works and returns exactly the
- // other half. On the residual arm it does NOT: the clause lowers to
- // `match_none`, the Bool fold drops it as "excludes nothing", and all
- // 50 come back — the 25 the exclusion was written to remove included.
- // That is the behaviour change the CHANGELOG documents, asserted here
- // so it cannot happen by accident on a shape nobody looked at.
+ // The exclusion returns exactly the other half on all four arms. Under
+ // the name-only rule the numeric arms returned all 50 — the 25 the
+ // exclusion was written to remove included; that is the shape this
+ // assertion exists to keep out.
assert_eq!(
excluded.total.value,
- if yields {
- (DOCS / 2) as u64
- } else {
- DOCS as u64
- },
+ (DOCS / 2) as u64,
"[{label}] must_not: got {} for {must_not_probe}",
excluded.total.value
);
- if yields {
- for hit in &excluded.hits {
- let n: usize = hit.id.trim_start_matches('d').parse().unwrap();
- assert_eq!(
- n % 2,
- 1,
- "[{label}] must_not returned an excluded document: {}",
- hit.id
- );
- }
+ for hit in &excluded.hits {
+ let n: usize = hit.id.trim_start_matches('d').parse().unwrap();
+ assert_eq!(
+ n % 2,
+ 1,
+ "[{label}] must_not returned an excluded document: {}",
+ hit.id
+ );
}
- // Doc values were never in question — this row answers 25/25 on ALL
- // THREE arms, including the residual one where every lexical clause
- // says 0. That is what proves the data is present and correctly
- // populated, and that it is the planner, not the storage, that changed.
+ // Doc values were never in question — this row answers 25/25 on all
+ // four arms. It answered 25/25 under the name-only rule too, which is
+ // what proved the data was present and correctly populated while every
+ // lexical clause on it said 0: it was the planner, not the storage.
let agg = idx
.search(
&parse_request(&json!({
@@ -1228,6 +1246,281 @@ async fn declared_chunks_sibling_of_a_dense_vector_stays_queryable_unless_it_loo
}
}
+/// #328, the other half of the same rule: what dynamic mapping is allowed to do
+/// with an UNDECLARED `<vector>_chunks` key, which is what decides whether the
+/// collision test above can be true at the same time as the saving.
+///
+/// Three arms, one index each, all with `emb` declared `dense_vector` and
+/// `emb_chunks` absent from the mapping. What differs is only the VALUE, and
+/// the value is the whole rule
+/// (`memtable::looks_like_multi_vector` →
+/// `index::is_undeclared_multi_vector_companion`):
+///
+/// * `[[…],[…]]` rectangular numeric — a real RFC #148 multi-vector.
+/// Dynamic mapping must not register it, it must acquire no `FieldConfig`,
+/// it must have no `.emb_chunks.fst`, and lexical clauses on it answer 0.
+/// This is where the 26,518,005 B / 67% of #328's saving comes from.
+/// * a SCALAR string — somebody's own field that happens to sit under the
+/// name. Registered as `text` and fully lexical. This one was already safe
+/// before the fix.
+/// * a SCALAR number — somebody's own field, registered as `long`, and fully
+/// lexical. THIS is the arm that was silently broken: dynamic mapping wrote
+/// a `long` config, `declares_non_vector_shaped_field` carved `long` out of
+/// its yield set precisely because it could not tell that config from the
+/// multi-vector's, and every lexical clause on a live user field lowered to
+/// `match_none`. Nothing about the mapping changed and no error was raised.
+///
+/// The value-shape test is why all three can hold at once, and it is the same
+/// place Lucene puts the distinction: a per-document multi-vector is
+/// `LateInteractionField extends BinaryDocValuesField`
+/// (`lucene/core/src/java/org/apache/lucene/document/LateInteractionField.java:36`,
+/// `:44`, `:45`) — its own field type, never a numeric one arrived at by
+/// inference.
+#[tokio::test]
+async fn an_undeclared_chunks_key_is_unmapped_only_when_its_value_is_a_multi_vector() {
+ const DIMS: usize = 16;
+ const DOCS: usize = 50;
+
+ let component = |doc: usize, dim: usize| -> f64 {
+ let h = (doc as u64)
+ .wrapping_mul(6_364_136_223_846_793_005)
+ .wrapping_add((dim as u64).wrapping_mul(1_442_695_040_888_963_407));
+ ((h >> 11) as f64 / (1u64 << 53) as f64 * 2.0) - 1.0
+ };
+
+ // (label, value builder, expected-to-stay-lexical, probe)
+ #[allow(clippy::type_complexity)]
+ let arms: Vec<(&str, Box<dyn Fn(usize, &[f64]) -> Value>, bool, Value)> = vec![
+ (
+ "multi-vector",
+ Box::new(|_d, v: &[f64]| json!([v, v])),
+ false,
+ json!({"term": {"emb_chunks": "0"}}),
+ ),
+ (
+ "scalar-string",
+ Box::new(|d, _v: &[f64]| json!(if d % 2 == 0 { "tenant-a" } else { "tenant-b" })),
+ true,
+ json!({"term": {"emb_chunks": "tenant-a"}}),
+ ),
+ (
+ "scalar-number",
+ Box::new(|d, _v: &[f64]| json!(if d % 2 == 0 { 7 } else { 9 })),
+ true,
+ json!({"term": {"emb_chunks": 7}}),
+ ),
+ ];
+
+ for (label, value_of, stays_lexical, probe) in arms {
+ let dir = TempDir::new().unwrap();
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ let mut emb = FieldConfig::new("emb", FieldType::Vector);
+ emb.options.dimensions = Some(DIMS);
+ emb.options.similarity = Some("cosine".to_string());
+ schema.fields.push(emb);
+ // NOTE: no `emb_chunks` here. Whatever config it ends up with is one
+ // dynamic mapping decided, which is exactly what is under test.
+
+ let engine = make_engine(&dir);
+ engine.create_index("dyn", schema).unwrap();
+ let idx = engine.get_index("dyn").unwrap();
+
+ for d in 0..DOCS {
+ let v: Vec<f64> = (0..DIMS).map(|dim| component(d, dim)).collect();
+ idx.index_document(
+ Some(format!("d{d}")),
+ json!({
+ "body": format!("quarterly liquidity evidence {d}"),
+ "emb_chunks": value_of(d, &v),
+ "emb": v,
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ idx.refresh().await.unwrap();
+ idx.force_merge(1).await.unwrap();
+
+ let live = idx.schema().await;
+ let mapped = live.fields.iter().find(|f| f.name == "emb_chunks");
+
+ let mut files: Vec<(String, u64)> = Vec::new();
+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {
+ for entry in std::fs::read_dir(dir).unwrap() {
+ let entry = entry.unwrap();
+ if entry.file_type().unwrap().is_dir() {
+ walk(&entry.path(), out);
+ } else {
+ out.push((
+ entry.file_name().to_string_lossy().into_owned(),
+ entry.metadata().unwrap().len(),
+ ));
+ }
+ }
+ }
+ walk(dir.path(), &mut files);
+ let fst: u64 = files
+ .iter()
+ .filter(|(name, _)| name.ends_with(".emb_chunks.fst"))
+ .map(|(_, len)| *len)
+ .sum();
+ let hits = idx.search(&make_search(probe.clone())).await.unwrap();
+ eprintln!(
+ "#328 dynamic [{label}]: mapped={:?} emb_chunks.fst={fst} hits={}",
+ mapped.map(|f| f.field_type),
+ hits.total.value
+ );
+
+ if stays_lexical {
+ assert!(
+ mapped.is_some(),
+ "[{label}] dynamic mapping must still register an ordinary field \
+ under this name"
+ );
+ assert!(
+ fst > 0,
+ "[{label}] a dynamically mapped field of the user's own must keep \
+ its term dictionary"
+ );
+ assert_eq!(
+ hits.total.value,
+ (DOCS / 2) as u64,
+ "[{label}] a dynamically mapped field of the user's own must stay \
+ queryable, got {} for {probe}",
+ hits.total.value
+ );
+ } else {
+ assert!(
+ mapped.is_none(),
+ "[{label}] a real multi-vector must not acquire a FieldConfig; got {mapped:?}"
+ );
+ assert_eq!(fst, 0, "[{label}] a real multi-vector must have no `.fst`");
+ assert_eq!(
+ hits.total.value, 0,
+ "[{label}] a lexical clause on a real multi-vector must answer 0"
+ );
+ }
+ // The `dense_vector` itself is unaffected on every arm.
+ assert!(
+ !files.iter().any(|(name, _)| name.ends_with(".emb.fst")),
+ "[{label}] the dense_vector must still have no term dictionary"
+ );
+ }
+}
+
+/// #328, the documented LIMIT of the rule above, asserted rather than assumed.
+///
+/// `is_undeclared_multi_vector_companion` can only refuse to map a companion it
+/// can recognise, and recognising one needs the `dense_vector` to be declared
+/// ALREADY. Index the documents first and map `emb` afterwards and dynamic
+/// mapping has already written a `double` config for `emb_chunks` — which
+/// `declares_non_vector_shaped_field` then yields to, exactly as it would to a
+/// user's own declaration, because on the information in the schema the two are
+/// the same thing.
+///
+/// The consequence is bytes, not answers: `<seg>.emb_chunks.fst` survives. That
+/// is the safe direction of the trade and the reason this limit is acceptable,
+/// so the test asserts BOTH halves — the dictionary is there AND the field
+/// still answers — so that a future change cannot quietly convert it into the
+/// silent-zero direction.
+#[tokio::test]
+async fn a_companion_mapped_before_its_vector_keeps_its_dictionary_and_its_answers() {
+ const DIMS: usize = 16;
+ const DOCS: usize = 50;
+
+ let dir = TempDir::new().unwrap();
+ let mut schema = Schema::empty();
+ schema
+ .fields
+ .push(FieldConfig::new("body", FieldType::Text));
+ let engine = make_engine(&dir);
+ engine.create_index("late", schema).unwrap();
+ let idx = engine.get_index("late").unwrap();
+
+ // Documents FIRST — no `emb` in the mapping yet, so `emb_chunks` is just an
+ // array of numeric arrays and dynamic mapping registers it as `double`.
+ for d in 0..DOCS {
+ let v: Vec<f64> = (0..DIMS).map(|dim| (d + dim) as f64 * 0.25).collect();
+ idx.index_document(
+ Some(format!("d{d}")),
+ json!({
+ "body": format!("quarterly liquidity evidence {d}"),
+ "emb_chunks": [v.clone(), v.clone()],
+ }),
+ )
+ .await
+ .unwrap();
+ }
+ idx.refresh().await.unwrap();
+
+ let before = idx.schema().await;
+ let inferred = before
+ .fields
+ .iter()
+ .find(|f| f.name == "emb_chunks")
+ .map(|f| f.field_type);
+ assert_eq!(
+ inferred,
+ Some(FieldType::Double),
+ "with no `dense_vector` declared, the companion is an ordinary numeric \
+ field and must be mapped like one"
+ );
+
+ // …and only NOW is the vector declared. From this point on
+ // `lexically_typeless_fields` synthesises `emb_chunks` as a companion
+ // candidate — and finds the `double` config dynamic mapping already wrote.
+ let mut emb = FieldConfig::new("emb", FieldType::Vector);
+ emb.options.dimensions = Some(DIMS);
+ emb.options.similarity = Some("cosine".to_string());
+ idx.add_field(emb).await.unwrap();
+ idx.force_merge(1).await.unwrap();
+
+ let mut files: Vec<(String, u64)> = Vec::new();
+ fn walk(dir: &std::path::Path, out: &mut Vec<(String, u64)>) {
+ for entry in std::fs::read_dir(dir).unwrap() {
+ let entry = entry.unwrap();
+ if entry.file_type().unwrap().is_dir() {
+ walk(&entry.path(), out);
+ } else {
+ out.push((
+ entry.file_name().to_string_lossy().into_owned(),
+ entry.metadata().unwrap().len(),
+ ));
+ }
+ }
+ }
+ walk(dir.path(), &mut files);
+ let fst: u64 = files
+ .iter()
+ .filter(|(name, _)| name.ends_with(".emb_chunks.fst"))
+ .map(|(_, len)| *len)
+ .sum();
+ let hits = idx
+ .search(&make_search(json!({"exists": {"field": "emb_chunks"}})))
+ .await
+ .unwrap();
+ eprintln!(
+ "#328 mapping-after-data: emb_chunks.fst={fst} exists={}",
+ hits.total.value
+ );
+
+ // The cost of the limit: the companion's dictionary survives.
+ assert!(
+ fst > 0,
+ "mapping the vector after the data leaves the companion's dictionary in \
+ place — that is the documented cost, and it is bytes"
+ );
+ // The thing that must NOT happen: it is still a live, answering field.
+ assert_eq!(
+ hits.total.value, DOCS as u64,
+ "the companion must stay queryable when the schema says it is an \
+ ordinary numeric field"
+ );
+}
+
#[tokio::test]
async fn semantic_passage_provenance_survives_update_merge_restart_and_source_filter() {
let dir = TempDir::new().unwrap();