AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge remote-tracking branch 'origin/main' into fix/issue-204-fallback-sweep
xerj-orga5d3f0627fc8
diff --cc CHANGELOG.mdindex 6bd31b162,d110fa1d4..d03c3c776--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -178,376 -178,259 +178,627 @@@ and this project adheres to [Semantic Vfrom the moment it is allowlisted. Seed the target with snapshot/restorefirst if it needs the existing documents.+### Changed++- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot+ honour, instead of accepting it and analysing differently**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an+ existing index is opened, where a hard failure would brick it), so an+ unresolvable tokenizer silently became `standard` and an unresolvable token+ filter was dropped — the index was created, `GET /_settings` echoed the block+ back, and the field was analysed in a way nobody asked for. Index creation+ now validates the block and answers `400` with the exact constructs it cannot+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused+ (it only ever changed what `GET /_settings` echoed; the registry is built+ once, at create/open, and there is no rebuild path — Elasticsearch also+ refuses analysis updates on an open index). That refusal covers every+ spelling this handler *accepts*, not just the nested one: the flat dotted+ `{"index.analysis.analyzer.x.type": "custom"}` and half-dotted+ `{"index": {"analysis.analyzer.x.type": "custom"}}` forms that ES clients+ routinely send used to walk straight past it into the display copy, and+ `GET /_settings` then echoed back an analyzer that analysed nothing.++ **This rejects settings blocks that XERJ used to accept with a `200`, and the+ ES-YAML conformance suite does not cover them — CI being green is not+ evidence that your index still creates.** What is honoured:++ | | honoured (still `200`) |+ |---|---|+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |+ | `analysis.normalizer` | nothing — never built |++ Anything else in those positions is now a `400`. There is one important+ exception to read first: XERJ resolves a declared filter or tokenizer by+ **name**, not by `type`, so a declaration whose *name* is a built-in is still+ honoured whatever its type says. Measured, all still `200`:+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,+ `{"lowercase": {"type": "lowercase"}}`,+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which+ has a regression test pinning it. Naming a built-in analyzer on a field+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.++ What now fails is a declaration under a name that does NOT resolve — which is+ the usual ES habit of naming filters after what they do in *your* index.+ Measured blocks that returned `200` in rc.14 and return `400` from this+ release:++ - a `keyword` field's `normalizer` (any);+ - a `char_filter` (any, including `html_strip` and `mapping`), declared+ either under `analysis.char_filter` or as an analyzer's `char_filter`;+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,+ which never lowercased anything, because the analyzer referencing+ `my_lowercase` found no such filter and dropped it;+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,+ `letter`, `lowercase` or `simple_pattern`;+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,+ `fingerprint`, or a language other than `english`;+ - a `synonym` filter whose `synonyms` is a bare string rather than an array+ (it built a filter with zero rules).++ **The gate covers every spelling, at create as well as at update.** An+ `analysis` declaration written with flat dotted keys+ (`{"index.analysis.filter.my_lower.type": "lowercase"}`) or half-dotted ones+ (`{"index": {"analysis.analyzer.a.type": "custom"}}`) is the same request as+ the nested form to an Elasticsearch client, and this handler already parses+ dotted keys for other settings (`index.sort.field`). The analyzer registry+ resolves the nested spellings only, so a dotted declaration was accepted,+ echoed back by `GET /{index}/_settings`, and applied to nothing. It is now a+ `400` naming the offending key, on both `PUT /{index}` and+ `PUT /{index}/_settings`. Measured on this release, the same+ `my_lower`/`lowercase` declaration in four spellings — nested, `index.`-+ namespaced, fully dotted, half-dotted — answers `400`, `400`, `400`, `400`;+ a settings body with no analysis in it (`index.number_of_shards`,+ `index.sort.field`) still answers `200`. Nothing already on disk changes+ meaning: the dotted form is refused, not newly honoured, so no existing+ index's analysis is re-derived on reopen.++ **Upgrading:** there is no opt-out flag. Either remove the analysis block and+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only+ the constructs in the table. The `400` names every offending construct in one+ response, so one round-trip is enough to see the whole list. **Two paths are+ not covered and still accept these blocks silently: an index template's+ `template.settings.analysis` is dropped wholesale before the gate ever sees+ it, and option-level divergence inside a supported type (`synonyms_path`,+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**++- **BREAKING — `PUT /_ingest/pipeline/{id}` refuses processor configuration+ XERJ cannot honour, so a pipeline that used to be acknowledged can now be a+ `400`, and a stored one can start refusing writes**+ ([#204](https://github.com/xerj-org/xerj/issues/204)). This is the same+ change as the "`200 acknowledged` … means we will run this" entry under+ *Fixed* below, stated from the upgrader's side, because it converts `201`+ into `400` for any pipeline using an ES processor or option this build does+ not implement. Read that entry for the full list.++ **What is NOT broken, deliberately.** A definition persisted by a build with+ **no gate** was accepted by whichever build wrote it, and there is no caller+ left to answer `400` at boot. Boot replay therefore compiles *those*+ definitions in a compatibility mode that reproduces the previously shipped+ behaviour for every check this sweep added (unknown `grok` pattern name,+ unknown `convert` target, a bare string where an array is required,+ `set.copy_from`, a processor-level `if`/`on_failure`/`ignore_failure`), so+ **an upgrade alone never turns a running cluster's writes from `201` into+ `400`**. Each one is logged at ERROR naming the pipeline and the offending+ config, and recorded in `Engine::degraded_pipelines`; a re-`PUT` of the same+ definition answers `400` with the same reason, which is the repair path. A+ definition that did not compile before the upgrade either (unknown stage+ type, missing required field) still fails at boot and is recorded unrunnable,+ as before.++ **Which definitions those are is decided by provenance, not by the calendar.**+ A definition THIS build refused at `PUT` time is recorded in+ `<data_dir>/ingest-pipeline-refusals.json` and replayed at *definition*+ strictness, so the refusal survives a restart. Without that marker the replay+ could not tell "written by an older, laxer build" from "written by this build+ and answered `400`", and a plain restart overturned the answer the operator+ had already been given: measured, `PUT` a `{"remove": {"field": "secret"}}`+ pipeline, add an `if` guard to it (refused — every write `400`s), restart the+ node and nothing else, and the identical write answered `201` with `secret`+ dropped from a document the guard EXCLUDES. It covered every check the sweep+ added; only an unimplemented *stage* survived a restart, because the raw ES+ body it stores does not deserialise. This is the same pattern as an index's+ `<index>/analysis-binding.json`: the marker's ABSENCE is the load-bearing+ signal.++ The marker is provenance, not a tombstone. A marked definition that compiles+ cleanly on the new build — because that build implements what the refusing+ one could not — clears its own marker at boot and runs, with an INFO line; a+ successful re-`PUT` or a `DELETE` clears it too. If the file exists but+ cannot be read or parsed, no pipeline's provenance is known, so **every**+ persisted pipeline is replayed at definition strictness and any that fails is+ recorded unrunnable with the file named in the reason — fail closed and+ loudly, rather than silently reopening the hole. **In that state a refusal is+ also not acknowledged:** the file is written whole from the in-memory map,+ which a failed load leaves empty, so the next refused `PUT` would have+ replaced it with a one-entry map and erased every refusal it still held —+ after which it parses cleanly, every earlier refusal reads as "no provenance",+ and the fail-closed behaviour converts itself into the hole it exists to+ close after exactly one process lifetime. `PUT /_ingest/pipeline/{id}` now+ answers `500` naming the file instead, and the file is left byte-for-byte as+ found; move it aside and restart to clear the state. An older binary reading a+ data dir written by this one simply ignores the file; it is a sidecar+ precisely because `cluster_state.json`'s format-1 envelope is+ `deny_unknown_fields` and widening it would break downgrades.++ **The marker survives concurrent refusals, not just sequential ones.** The+ sidecar is rewritten whole, so two refused `PUT`s in flight at once were a+ lost update rather than a merge: each snapshotted the map, and whichever+ wrote second with the older snapshot erased the other's marker — while both+ callers were told `{"acknowledged": true}`. Measured at a live node against+ the first cut of this work: 40 concurrent refused `PUT`s, 40 acknowledged, 37+ markers on disk, and after a plain restart the three that were lost answered+ `201` and dropped `secret` from a document their `if` guard excludes. The+ snapshot and the write now happen under the same `cluster_state_write` mutex+ that `flush_cluster_state` has always taken. Measured on this release: 40+ concurrent refused `PUT`s, 40 acknowledged, 40 markers on disk, plain restart,+ 40 of 40 still `400`.++ The same rule applies one level up. Teaching the analysis resolver the+ canonical `settings.index.analysis` nesting (see the analysis entry above) is+ a fix at index *create* and a data bug at index *open*: `settings.json` keeps+ the caller's nesting verbatim, so an index created before this release with+ the canonical shape has postings that were written with `standard`, and+ honouring the declaration on reopen would tokenise every new write+ differently from everything already on disk. An index now records the+ analysis binding it was created with (`<index>/analysis-binding.json`) and+ keeps it for life. An index with no marker and analyzers declared *only*+ under `index.analysis` keeps the legacy binding if it holds documents — with+ an ERROR naming the index and saying a reindex is the repair — and adopts the+ canonical binding if it is empty, recording that so a later restart cannot+ disagree.++ ### Performance++ - **A field mapped `dense_vector` — and the undeclared `_chunks` companion the+ passage pipeline writes beside it — no longer gets a full-text term+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both+ halves are named in that headline because the companion is where most of the+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the+ mapping at all — it is matched by NAME, not by anyone declaring it+ `dense_vector`. The field was fed+ through the lexical indexing path and given an FST + postings that no query+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from+ `_source`/doc values, and `_field_caps` and highlighting never open the+ field's term dictionary. `extract_field_text`'s array arm joined the+ components with spaces, so a 128-dim vector became one enormous+ decimal-string term per document.++ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long ++ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing+ every file in the data directory except the write-ahead log:+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are+ byte-identical on both sides.++ `.wal` is excluded from that total because its tail is reclaimed+ asynchronously and it is not reproducible: on the 300-doc fixture the+ committed regression test uses, runs put the *whole directory* anywhere+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The+ test prints both and asserts on the durable one. It reports+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured by+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the+ committed test on this revision printed 857,265 / 857,529 / 857,533 /+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and+ still 1.7 MB below the before-side.++ A `dense_vector` nested under an object mapping is covered too — its+ components were landing in the *parent's* term dictionary, because the segment+ builder flattens the whole object into one text field. On the nested fixture+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`+ sub-mapping under a `passages` object).++ The exclusion covers the whole family a vector field generates, not just the+ base name: `<field>_chunks` (the per-document multi-vector) and+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides+ which fields get an HNSW graph. The companion is the bigger half —+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change+ index — so a base-name-only exclusion would leave more behind than it removes.++ **Two rules, and they do not have the same strength.** The base field is+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two+ companions are excluded BY NAME, because they arrive in `_source` with no+ mapping entry behind them, so there is no declared type to read — and a+ name-based rule YIELDS to any declaration it finds. **If you have mapped your+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,+ `object`, `nested`, or a type string XERJ does not recognise — nothing about+ it changes**: it keeps its term dictionary, every query naming it keeps+ answering exactly what it answered before, and only the `dense_vector` beside+ it loses its postings. The same holds for a field of that name XERJ mapped for+ you dynamically, whatever type it inferred, as long as the value is not itself+ a multi-vector. Measured on+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,+ `bool{must:[match body],filter:[term …]}` 25 → 25,+ `constant_score{filter:term}` 25 → 25,+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name+ XERJ mapped for you *dynamically* is exempt on the same rule — measured+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one+ (44 B `.fst`).++ **What makes "yields to any declaration" affordable** is that the companion+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied+ `_source` — an array of float arrays — so dynamic mapping used to register it+ like any other unmapped key and infer `double` from it, producing a mapping+ entry indistinguishable from one you wrote. Dynamic mapping now declines to+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is+ already mapped *and* the value is a rectangular, non-empty array of non-empty+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,+ a string, a flat array or a ragged one under that name is your field and is+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is+ mapped `long`, keeps a 44 B `.fst` and answers 25.++ Both halves were measured on their own, by disabling each in the tree and+ re-running the committed tests:++ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |+ |---|---:|---:|---|+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |++ The middle row is the behaviour rc.16 would have shipped; the bottom row is+ the 67% of the saving that an unconditional "any declaration wins" gives up.+ Neither cost is paid here.++ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now+ lowered to `match_none` at plan time rather than falling through to the+ stored-doc scan. That is the correctness half of the change, not only the+ speed half: with the postings removed and no lowering,+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**+ instead of none, because the scan renders the float array to text and every+ component contains a `0`. Measured on the corpus above, first call for each+ shape (`main` → postings-gone-without-lowering → this release):++ | query | `main` | no lowering | this release |+ |---|---|---|---|+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |++ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`+ list keeps its lexical members, and a `nested` query is deliberately not+ rewritten (its field names resolve against the element, not the root).++ **Four behaviour changes to know about:**++ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no+ lexical surface at all: every lexical leaf naming one *outright* answers 0+ hits.** "Outright" excludes one shape, measured rather than assumed: a+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because+ resolving a pattern needs the expansion universe — dynamic fields included —+ and lowering one that would have expanded onto a real lexical field is the+ silent-zero failure this release goes out of its way to avoid. So+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls+ to the stored-doc scan, which resolves the pattern against the source's+ field names and matches the rendered decimals. Unchanged in both columns, so+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`+ all answer 0. Before this release+ such a clause fell through to the stored-doc scan, which renders the float+ array back to decimal text and matches against *that*, so whether a shape+ answered 0 or answered most of the corpus depended entirely on whether the+ rendered decimals happened to contain the probe. This is a CLASS of changed+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,+ `main` → this release:++ | query | before | after |+ |---|---:|---:|+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |+ | `prefix {emb: "0"}` | 146 | 0 |+ | `prefix {emb_chunks: "0"}` | 146 | 0 |+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |+ | `term {emb: <exact component>}` numeric | 1 | 0 |+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |++ Shapes whose probe text never appeared in a rendered float (`term` with a+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,+ `query_string`) answered 0 on both sides. ES rejects these queries on a+ `dense_vector` outright and Lucene gives a field with no terms an empty+ scorer, so 0 is the answer this release adopts for all of them. The+ regression test pins the whole class rather than sampling it.+ - **A `bool` that mentions a `dense_vector` can change `_score` and+ `max_score` without changing its hit count.** Dropping the dead clause makes+ the surviving bool projectable onto the inverted index, so BM25 scores it+ where the stored-doc scan used to. Measured on the same fixture:++ | query | hits | `_score` | `max_score` |+ |---|---|---|---|+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |++ Both moves are toward consistency — on `main` these queries reported a+ `max_score` that **no returned hit carried**, and the first now scores+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces+ to. Absolute scores are not part of XERJ's compatibility surface, but+ anything comparing scores across a version boundary or asserting on+ `max_score` for a bool that touches a vector field will see this. Plain+ lexical queries are unaffected; the blast radius is bools with a+ `dense_vector` clause in them.+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that+ XERJ mapped for you.** The multi-vector companion beside a declared+ `dense_vector` used to be registered by dynamic mapping as `double` (an+ array of float arrays walks to its first scalar). It is not registered at+ all now, so it disappears from `_mapping` and `_field_caps` — the same+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads+ `_source` changes; the values are stored and returned exactly as before, and+ `knn` and `semantic` on the vector itself are untouched. A field of that+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector+ value, is unaffected and still listed.++ Aggregating and sorting on that companion still answer `200` — measured on+ the wire against a live node with `emb` mapped `dense_vector` and+ `emb_chunks` unregistered: `terms` returns real buckets, `stats` returns+ `count: 0` byte-identical to the previous release, and `sort` returns sort+ values. Nothing that worked before returns an error now.+ - **Existing indices keep their bloat until a merge rewrites them, and an+ index created by rc.16 or earlier only ever reclaims the base-vector half.**+ This is a write-side rule. Segments written by an earlier release keep their+ `.emb.*` / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate+ reads whichever shape it finds.++ `POST /<index>/_forcemerge?max_num_segments=1` reclaims `.emb.*`. **It does+ not reclaim the companion.** Measured by building an index on an rc.16+ server (`emb` mapped `dense_vector` before ingest, 500 docs × 32 dim with+ `emb_chunks` multi-vectors), then reopening the same data directory with+ this release and force-merging: `.emb.fst` 262,904 B → 0, but+ `.emb_chunks.fst` 525,200 B → 525,355 B, and+ `wildcard {emb_chunks: "0*"}` still answers 501 of 501. The reason is not a+ bug in the merge: rc.16 persisted `emb_chunks: {"type": "double"}` into the+ mapping, and the by-name rule correctly yields to a field you (or an earlier+ release) declared. Since a mapping entry cannot be dropped in place, the+ remediation for an existing index is to **reindex into a fresh index**. On+ the corpus this entry headlines that companion is 67% of the saving, so an+ in-place upgrade recovers roughly a third of the published number.++ The companion rule has an ordering limit of the same kind, and it fails in+ the safe direction: XERJ can only decline to map a companion it can+ recognise, and recognising one needs the `dense_vector` to be mapped+ *before* the documents arrive. Index first and add the mapping afterwards+ and `<vector>_chunks` already carries a dynamic `double` entry, which the+ by-name rule then yields to — so that index keeps the companion's term+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of+ reclaiming it. It costs bytes, not answers.++ *Known limitation, filed as [#382](https://github.com/xerj-org/xerj/issues/382):*+ refusing a `FieldConfig` for the companion leaves the pre-existing schema-hash+ throttle asserting the key set is unchanged, so if a later document puts a+ differently-typed value under `<vector>_chunks` it stays unregistered for up+ to 100 documents and queries on it answer zero until the throttle expires.+ Doc-order-dependent and self-healing. The throttle is pre-existing and the fix+ belongs to it.++ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,+ where XERJ answers `200`. This release removes bytes and adopts Lucene's+ zero-hit answer; it does not add the rejection.+### Fixed+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and+ every write path enforces it**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only+ warn-log a compile failure, so a pipeline that could never run was+ acknowledged and echoed back by `GET`. It now compiles first, with three+ outcomes: compiled → stored and acknowledged; a processor config we refuse →+ `400`; a processor or option this build cannot honour → accepted (as+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses+ rather than storing an untransformed document.++ Defects this surfaced, each of which answered `200`/`201` and did something+ smaller than asked: **every** ES `rename` processor failed to compile (the+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and+ renamed nothing; ES `append` was executed as `set`, replacing the field+ instead of extending it, and disagreeing with `_simulate`, which implemented+ a real append; an ES `set` processor did not override an existing value where+ Elasticsearch does (the ES→XERJ translation now writes `"override": true`+ explicitly when the processor omits it — the **native** `type: "set"` stage+ keeps its long-standing `override: false` default, because native definitions+ are recompiled from `cluster_state.json` at every boot and flipping the stage+ default would have changed what an already-running cluster's pipelines do to+ documents on a binary upgrade alone); an unknown `pii_redaction` type built an+ empty pattern set and redacted nothing; an unknown `grok` pattern name+ silently became the+ catch-all; an unknown `convert` target left the field unconverted per+ document; array-typed keys given as a bare string were dropped and the stage+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the+ definition while leaving the compiled pipeline live under the same name.+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`+ rather than being refused.++ Two more of the same, at the definition's own edges: a **pipeline-level**+ `on_failure` block (the top-level body key, not the processor-level one) was+ accepted, echoed by `GET`, and never run — the pipeline is now recorded+ unrunnable, exactly as its processor-level twin already was; and `set` with+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:+ missing value`, which told the caller their valid ES pipeline was malformed.++ **The refusal reads the value, not the key.** A processor-level option is+ refused only when what the caller *wrote* asks for behaviour this build does+ not have; writing the value XERJ already implements is the same as omitting+ it. `ignore_failure: false` is Elasticsearch's own default and ES clients emit+ it verbatim — a presence check made that pipeline register and then `400`+ every write through it, while the byte-identical processor without the key+ indexed fine. Accepted: `ignore_failure: false`, an empty `on_failure: []`,+ `ignore_missing: true` (which is exactly what `remove`/`rename`/`convert`/+ `date` already do — they pass the document through unchanged when the field+ is absent), `ignore_empty_value: false`, `allow_duplicates: true`. Refused:+ the opposite value in each case, and any `if` at all. (Modelled on Lucene's+ consume-then-complain: `AbstractAnalysisFactory.getBoolean(args, name,+ default)` at `lucene/core/src/java/org/apache/lucene/analysis/AbstractAnalysisFactory.java:213-217`+ removes the key and returns the default only when it was absent, and+ `ArabicStemFilterFactory` at+ `lucene/analysis/common/src/java/org/apache/lucene/analysis/ar/ArabicStemFilterFactory.java:44-51`+ then throws on whatever is left. Lucene is Apache-2.0, the same licence as+ XERJ; no code is reproduced.)++ **`remove` and `rename` no longer walk around the gate.** Their ES→XERJ+ translation built a fresh config object holding only the translated keys, so+ a processor-level `if` / `on_failure` / `ignore_failure` / `ignore_missing` on+ them was dropped *before* the gate ran: `{"remove": {"field": "secret", "if":+ "ctx.tenant == 'a'"}}` compiled, was acknowledged, and dropped the field on+ every document — while the identical `if` on `set` was refused. Every branch+ of the translation now edits the caller's config instead of rebuilding it.++ **`_simulate` speaks Elasticsearch's vocabulary again.** It walks the+ *stored* pipeline, and once ES `rename` started compiling the stored form+ became XERJ's internal `stages` shape — whose stage names leaked straight into+ the response, so `processor_type` read `field_rename` where ES reports+ `rename`. The stages→processors conversion now inverts both halves of the+ translation, name and config.++- **`GET /_ingest/pipeline[/{id}]` answers in Elasticsearch's `processors`+ vocabulary instead of XERJ's internal `stages` shape**+ ([#204](https://github.com/xerj-org/xerj/issues/204)). Two shapes reach the+ store: the translated `stages` body (everything that compiles) and the raw ES+ body (kept verbatim for a definition XERJ stores but cannot run). Only the+ second was ever ES-shaped, so `GET` spoke XERJ's vocabulary for every pipeline+ that *worked* — measured before this release, an ES `rename` pipeline came+ back as+ `{"p":{"description":"d","stages":[{"type":"field_rename","config":{"mappings":{"a":"b"}}}]}}`.+ This is a wire-visible change on both `GET /_ingest/pipeline/{id}` and+ `GET /_ingest/pipeline`. It also mattered beyond cosmetics: `GET`'s output is+ what an operator edits and `PUT`s back, and a `stages`-shaped body takes the+ handler's pass-through branch — which is exactly how the restart hole+ described under *Changed* was reached in ordinary use.++ `set` is rendered with an explicit `"override"`, always, because it is the one+ key whose default differs between the two vocabularies (Elasticsearch `true`,+ XERJ's native `set` stage `false`). Emitting a bare ES `set` for a native+ stage would claim it overwrites when it preserves. A pipeline created through+ `PUT /_ingest/pipeline` therefore shows `"override": true` even when the+ caller omitted it — that is the value XERJ applies, and it round-trips: PUTting+ `GET`'s own output back cannot silently flip the stage's behaviour. Relatedly,+ `_simulate`'s `set` interpreter honoured no `override` at all and always+ overwrote, so it reported a document the ingest path would never produce; it+ now applies Elasticsearch's rule (default `true`; `false` leaves an existing+ non-null value alone).++ **This is also the only read endpoint for a pipeline created through+ `PUT /v1/pipelines/{name}`** — the native surface registers no `GET` — so the+ round trip has to be lossless in xerj's own vocabulary too, and it was not.+ `PUT /_ingest/pipeline/{id}` rebuilt the stored definition as+ `{description, stages}` and dropped every other top-level key, so PUTting+ `GET`'s own output back silently discarded a natively-defined pipeline's+ `on_error` and `timeout_ms`: the error policy reverted from `pass` (keep the+ document) to the `Drop` default (discard it) under a+ `200 {"acknowledged": true}`. The translation now edits the body instead of+ rebuilding it — the same rule already applied to `remove`/`rename` one level+ down. Measured on this release, `{"description":"n3","on_error":"pass",+ "timeout_ms":250,"stages":[…]}` reads back with both keys intact, and PUTting+ that body back leaves them intact, before and after a restart. Elasticsearch's+ own pipeline metadata (`version`, `_meta`, `deprecated`) rides along the same+ way and is now echoed back rather than dropped. `PUT /v1/pipelines/{name}`+ given that ES-shaped body used to answer `500` with `internal error: missing+ field stages`; it now answers `400` naming `/_ingest/pipeline/{name}` as the+ endpoint that accepts it.++- **Four wire-visible refusals that were fixed in this sweep but not written+ down** ([#204](https://github.com/xerj-org/xerj/issues/204)). All four+ replace a silent fallback with an error, which is the point of the issue, but+ each changes a status code:+ `POST /v1/admin/backup` answers `400` on a request body that is present but+ unparseable, where it previously ran a *default* backup under a success+ response; `GET /{index}/_stats` and `GET /_all/_stats` carry an added+ non-Elasticsearch key `primaries.mappings.schema_persist_failures` inside an+ otherwise ES-shaped response; the authorization layer's response pruning+ answers `500` when it cannot parse the body it was asked to filter, instead+ of falling back to the unfiltered bytes; and the bundled console's+ `DELETE /_xerj-console/api/v1/auth/passkeys/{id}` propagates a failed delete+ instead of discarding it — it answered `204 No Content` unconditionally, so a+ revoke refused by the engine (index write block, disk flood stage, storage+ error) reported success while the passkey stayed on disk and kept+ authenticating. An already-absent credential is still a `204`; only a genuine+ failure now surfaces.++ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`+ previously read the instruction and dropped it — a redaction pipeline that+ the single-document endpoint refused was ignored in bulk, and the document+ was stored verbatim under `errors: false`. All four now run the pipeline, and+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the+ name before touching a single document, so a bad pipeline fails the request+ instead of half of it.++ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`+ copied at most 10,000 documents and swallowed every per-document write error+ before answering `{"acknowledged": true}` (now full keyset paging, and any+ failed write — or a failed source flush — fails the request); and a failed+ `_bulk` item no longer reports `"result": "deleted"` alongside its own+ `status: 400`, which claimed a deletion that never happened.++ **`ignore_missing: false` is refused on purpose, and it is not the same call+ as `ignore_failure: false`.** Both are Elasticsearch defaults that ES clients+ emit verbatim, so the difference is worth stating plainly: the rule is *accept+ the value this build actually implements*, and the two land on opposite sides+ of it. `ignore_failure: false` means "a failing processor is not ignored",+ which is exactly what XERJ does, so it is accepted. `ignore_missing: false`+ means "FAIL the document when the field is absent", which XERJ cannot do at+ all — `ProcessAction` has no error variant, and every field-reading stage+ passes the document through untouched. Accepting it would be precisely the+ silent lie this issue exists to remove, so it is refused: a `remove` /+ `rename` / `convert` / `date` processor that spells out `ignore_missing:+ false` registers and then `400`s every write through it. **Omitting the key+ is accepted** and behaves as `true`. That asymmetry — omission accepted,+ the ES default written out refused — is deliberate but is a hard `400` for+ ES-generated pipelines; strip the key, or set it to `true`, to state what+ XERJ will actually do. `ignore_empty_value` is the mirror image: omitting it+ or writing `false` is honoured exactly, `true` is refused.++ **Known gaps, stated rather than closed:** `_simulate` interprets the stored+ ES body directly and covers processors the compiled path does not, so a+ pipeline recorded as unrunnable can still show a transformation under+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`+ carries no runnability signal — it renders the stored definition in ES+ vocabulary whether or not this build can run it, so an operator only learns+ at write time; a write addressed to an **alias** does not pick up the+ backing index's `default_pipeline`; `_update_by_query` reports a `drop`+ processor as a per-document failure, because it updates in place and has no+ delete path; `Engine::degraded_pipelines` is in-memory only and has no+ HTTP surface yet, so a boot-replay degradation is visible in the startup log+ and nowhere else — unlike a definition-time refusal, which is now durable+ (see *Changed*); and a refusal marker is only read back at boot, so a data+ dir opened by two processes at once is outside what the sidecar can promise+ (the node lock already prevents that).+- **Unity assets were sampled through a 4 MiB cap, silently junking wholeobject classes.** Unity YAML is a grouped family — each `unity_class` is itsown dataset and a class's first document can sit anywhere in a scene that