Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into fix/issue-204-fallback-sweep

xerj-orga5d3f0627fc8
diff --cc CHANGELOG.md
index 6bd31b162,d110fa1d4..d03c3c776
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -178,376 -178,259 +178,627 @@@ and this project adheres to [Semantic V
from the moment it is allowlisted. Seed the target with snapshot/restore
first if it needs the existing documents.
+### Changed
+
+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot
+ honour, instead of accepting it and analysing differently**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an
+ existing index is opened, where a hard failure would brick it), so an
+ unresolvable tokenizer silently became `standard` and an unresolvable token
+ filter was dropped — the index was created, `GET /_settings` echoed the block
+ back, and the field was analysed in a way nobody asked for. Index creation
+ now validates the block and answers `400` with the exact constructs it cannot
+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused
+ (it only ever changed what `GET /_settings` echoed; the registry is built
+ once, at create/open, and there is no rebuild path — Elasticsearch also
+ refuses analysis updates on an open index). That refusal covers every
+ spelling this handler *accepts*, not just the nested one: the flat dotted
+ `{"index.analysis.analyzer.x.type": "custom"}` and half-dotted
+ `{"index": {"analysis.analyzer.x.type": "custom"}}` forms that ES clients
+ routinely send used to walk straight past it into the display copy, and
+ `GET /_settings` then echoed back an analyzer that analysed nothing.
+
+ **This rejects settings blocks that XERJ used to accept with a `200`, and the
+ ES-YAML conformance suite does not cover them — CI being green is not
+ evidence that your index still creates.** What is honoured:
+
+ | | honoured (still `200`) |
+ |---|---|
+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |
+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |
+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |
+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |
+ | `analysis.normalizer` | nothing — never built |
+
+ Anything else in those positions is now a `400`. There is one important
+ exception to read first: XERJ resolves a declared filter or tokenizer by
+ **name**, not by `type`, so a declaration whose *name* is a built-in is still
+ honoured whatever its type says. Measured, all still `200`:
+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,
+ `{"lowercase": {"type": "lowercase"}}`,
+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`
+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which
+ has a regression test pinning it. Naming a built-in analyzer on a field
+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.
+
+ What now fails is a declaration under a name that does NOT resolve — which is
+ the usual ES habit of naming filters after what they do in *your* index.
+ Measured blocks that returned `200` in rc.14 and return `400` from this
+ release:
+
+ - a `keyword` field's `normalizer` (any);
+ - a `char_filter` (any, including `html_strip` and `mapping`), declared
+ either under `analysis.char_filter` or as an analyzer's `char_filter`;
+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,
+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,
+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,
+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,
+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,
+ which never lowercased anything, because the analyzer referencing
+ `my_lowercase` found no such filter and dropped it;
+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,
+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,
+ `letter`, `lowercase` or `simple_pattern`;
+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,
+ `fingerprint`, or a language other than `english`;
+ - a `synonym` filter whose `synonyms` is a bare string rather than an array
+ (it built a filter with zero rules).
+
+ **The gate covers every spelling, at create as well as at update.** An
+ `analysis` declaration written with flat dotted keys
+ (`{"index.analysis.filter.my_lower.type": "lowercase"}`) or half-dotted ones
+ (`{"index": {"analysis.analyzer.a.type": "custom"}}`) is the same request as
+ the nested form to an Elasticsearch client, and this handler already parses
+ dotted keys for other settings (`index.sort.field`). The analyzer registry
+ resolves the nested spellings only, so a dotted declaration was accepted,
+ echoed back by `GET /{index}/_settings`, and applied to nothing. It is now a
+ `400` naming the offending key, on both `PUT /{index}` and
+ `PUT /{index}/_settings`. Measured on this release, the same
+ `my_lower`/`lowercase` declaration in four spellings — nested, `index.`-
+ namespaced, fully dotted, half-dotted — answers `400`, `400`, `400`, `400`;
+ a settings body with no analysis in it (`index.number_of_shards`,
+ `index.sort.field`) still answers `200`. Nothing already on disk changes
+ meaning: the dotted form is refused, not newly honoured, so no existing
+ index's analysis is re-derived on reopen.
+
+ **Upgrading:** there is no opt-out flag. Either remove the analysis block and
+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only
+ the constructs in the table. The `400` names every offending construct in one
+ response, so one round-trip is enough to see the whole list. **Two paths are
+ not covered and still accept these blocks silently: an index template's
+ `template.settings.analysis` is dropped wholesale before the gate ever sees
+ it, and option-level divergence inside a supported type (`synonyms_path`,
+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**
+
+- **BREAKING — `PUT /_ingest/pipeline/{id}` refuses processor configuration
+ XERJ cannot honour, so a pipeline that used to be acknowledged can now be a
+ `400`, and a stored one can start refusing writes**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)). This is the same
+ change as the "`200 acknowledged` … means we will run this" entry under
+ *Fixed* below, stated from the upgrader's side, because it converts `201`
+ into `400` for any pipeline using an ES processor or option this build does
+ not implement. Read that entry for the full list.
+
+ **What is NOT broken, deliberately.** A definition persisted by a build with
+ **no gate** was accepted by whichever build wrote it, and there is no caller
+ left to answer `400` at boot. Boot replay therefore compiles *those*
+ definitions in a compatibility mode that reproduces the previously shipped
+ behaviour for every check this sweep added (unknown `grok` pattern name,
+ unknown `convert` target, a bare string where an array is required,
+ `set.copy_from`, a processor-level `if`/`on_failure`/`ignore_failure`), so
+ **an upgrade alone never turns a running cluster's writes from `201` into
+ `400`**. Each one is logged at ERROR naming the pipeline and the offending
+ config, and recorded in `Engine::degraded_pipelines`; a re-`PUT` of the same
+ definition answers `400` with the same reason, which is the repair path. A
+ definition that did not compile before the upgrade either (unknown stage
+ type, missing required field) still fails at boot and is recorded unrunnable,
+ as before.
+
+ **Which definitions those are is decided by provenance, not by the calendar.**
+ A definition THIS build refused at `PUT` time is recorded in
+ `<data_dir>/ingest-pipeline-refusals.json` and replayed at *definition*
+ strictness, so the refusal survives a restart. Without that marker the replay
+ could not tell "written by an older, laxer build" from "written by this build
+ and answered `400`", and a plain restart overturned the answer the operator
+ had already been given: measured, `PUT` a `{"remove": {"field": "secret"}}`
+ pipeline, add an `if` guard to it (refused — every write `400`s), restart the
+ node and nothing else, and the identical write answered `201` with `secret`
+ dropped from a document the guard EXCLUDES. It covered every check the sweep
+ added; only an unimplemented *stage* survived a restart, because the raw ES
+ body it stores does not deserialise. This is the same pattern as an index's
+ `<index>/analysis-binding.json`: the marker's ABSENCE is the load-bearing
+ signal.
+
+ The marker is provenance, not a tombstone. A marked definition that compiles
+ cleanly on the new build — because that build implements what the refusing
+ one could not — clears its own marker at boot and runs, with an INFO line; a
+ successful re-`PUT` or a `DELETE` clears it too. If the file exists but
+ cannot be read or parsed, no pipeline's provenance is known, so **every**
+ persisted pipeline is replayed at definition strictness and any that fails is
+ recorded unrunnable with the file named in the reason — fail closed and
+ loudly, rather than silently reopening the hole. **In that state a refusal is
+ also not acknowledged:** the file is written whole from the in-memory map,
+ which a failed load leaves empty, so the next refused `PUT` would have
+ replaced it with a one-entry map and erased every refusal it still held —
+ after which it parses cleanly, every earlier refusal reads as "no provenance",
+ and the fail-closed behaviour converts itself into the hole it exists to
+ close after exactly one process lifetime. `PUT /_ingest/pipeline/{id}` now
+ answers `500` naming the file instead, and the file is left byte-for-byte as
+ found; move it aside and restart to clear the state. An older binary reading a
+ data dir written by this one simply ignores the file; it is a sidecar
+ precisely because `cluster_state.json`'s format-1 envelope is
+ `deny_unknown_fields` and widening it would break downgrades.
+
+ **The marker survives concurrent refusals, not just sequential ones.** The
+ sidecar is rewritten whole, so two refused `PUT`s in flight at once were a
+ lost update rather than a merge: each snapshotted the map, and whichever
+ wrote second with the older snapshot erased the other's marker — while both
+ callers were told `{"acknowledged": true}`. Measured at a live node against
+ the first cut of this work: 40 concurrent refused `PUT`s, 40 acknowledged, 37
+ markers on disk, and after a plain restart the three that were lost answered
+ `201` and dropped `secret` from a document their `if` guard excludes. The
+ snapshot and the write now happen under the same `cluster_state_write` mutex
+ that `flush_cluster_state` has always taken. Measured on this release: 40
+ concurrent refused `PUT`s, 40 acknowledged, 40 markers on disk, plain restart,
+ 40 of 40 still `400`.
+
+ The same rule applies one level up. Teaching the analysis resolver the
+ canonical `settings.index.analysis` nesting (see the analysis entry above) is
+ a fix at index *create* and a data bug at index *open*: `settings.json` keeps
+ the caller's nesting verbatim, so an index created before this release with
+ the canonical shape has postings that were written with `standard`, and
+ honouring the declaration on reopen would tokenise every new write
+ differently from everything already on disk. An index now records the
+ analysis binding it was created with (`<index>/analysis-binding.json`) and
+ keeps it for life. An index with no marker and analyzers declared *only*
+ under `index.analysis` keeps the legacy binding if it holds documents — with
+ an ERROR naming the index and saying a reindex is the repair — and adopts the
+ canonical binding if it is empty, recording that so a later restart cannot
+ disagree.
+
+ ### Performance
+
+ - **A field mapped `dense_vector` — and the undeclared `_chunks` companion the
+ passage pipeline writes beside it — no longer gets a full-text term
+ dictionary** ([#328](https://github.com/xerj-org/xerj/issues/328)). Both
+ halves are named in that headline because the companion is where most of the
+ bytes are: of the 39,796,380 B removed below, **26,518,005 B (67%) is
+ `<seg>.emb_chunks.fst`**, and in that corpus `emb_chunks` is not in the
+ mapping at all — it is matched by NAME, not by anyone declaring it
+ `dense_vector`. The field was fed
+ through the lexical indexing path and given an FST + postings that no query
+ path can read: kNN is served by `hnsw/graph.bin`, `exists` from
+ `_source`/doc values, and `_field_caps` and highlighting never open the
+ field's term dictionary. `extract_field_text`'s array arm joined the
+ components with spaces, so a 128-dim vector became one enormous
+ decimal-string term per document.
+
+ Measured on a 5,000-doc × 128-dim corpus (text + keyword + long +
+ `dense_vector` + its `_chunks` companion), after `force_merge(1)`, summing
+ every file in the data directory except the write-ahead log:
+ **54,068,549 B → 14,251,975 B (−73.6%, 3.79×)**. Per file, `<seg>.emb.fst`
+ 13,256,659 B → 0, `<seg>.emb.post` 10,834 B → 0, `<seg>.emb.norms` 48 B → 0,
+ `<seg>.emb_chunks.fst` 26,518,005 B → 0, `<seg>.emb_chunks.post` 10,834 B → 0,
+ while the three lexical `.fst` files (`body` 278 B, `cat` 53 B, `n` 209 B) are
+ byte-identical on both sides.
+
+ `.wal` is excluded from that total because its tail is reclaimed
+ asynchronously and it is not reproducible: on the 300-doc fixture the
+ committed regression test uses, runs put the *whole directory* anywhere
+ between 858,147 B and 1,105,001 B while the durable bytes moved by ~500 B. The
+ test prints both and asserts on the durable one. It reports
+ **3,249,323 B → 857,265–857,769 B** for that fixture (before-side measured by
+ reverting `index.rs` and `memtable.rs` to the merge base `ca4d75a` on the same
+ fixture), which is the same −73.6% at 1/17th the corpus. Eight runs of the
+ committed test on this revision printed 857,265 / 857,529 / 857,533 /
+ 857,545 / 857,549 / 857,657 / 857,657 / 857,769 B, a 504 B spread; the ceiling
+ it asserts is 1,500,000 B, three orders of magnitude clear of that jitter and
+ still 1.7 MB below the before-side.
+
+ A `dense_vector` nested under an object mapping is covered too — its
+ components were landing in the *parent's* term dictionary, because the segment
+ builder flattens the whole object into one text field. On the nested fixture
+ `<seg>.passages.fst` goes 398,558 B → 45 B, for both mapping shapes that reach
+ `FieldType::Vector` (a dotted top-level `passages.vec`, and a `vec`
+ sub-mapping under a `passages` object).
+
+ The exclusion covers the whole family a vector field generates, not just the
+ base name: `<field>_chunks` (the per-document multi-vector) and
+ `__xerj_passage_meta__<field>` are excluded on the same walk that decides
+ which fields get an HNSW graph. The companion is the bigger half —
+ 26,518,005 B against `emb.fst`'s 13,256,659 B, 49% of the whole pre-change
+ index — so a base-name-only exclusion would leave more behind than it removes.
+
+ **Two rules, and they do not have the same strength.** The base field is
+ excluded BY DECLARED TYPE (`dense_vector`, unconditionally). The two
+ companions are excluded BY NAME, because they arrive in `_source` with no
+ mapping entry behind them, so there is no declared type to read — and a
+ name-based rule YIELDS to any declaration it finds. **If you have mapped your
+ own field called `<vector>_chunks` or `__xerj_passage_meta__<vector>` as
+ anything at all other than `dense_vector` — `text`, `keyword`, `long`,
+ `integer`, `short`, `byte`, `unsigned_long`, `double`, `float`, `half_float`,
+ `scaled_float`, `date`, `date_nanos`, `boolean`, `ip`, `geo_point`, `binary`,
+ `object`, `nested`, or a type string XERJ does not recognise — nothing about
+ it changes**: it keeps its term dictionary, every query naming it keeps
+ answering exactly what it answered before, and only the `dense_vector` beside
+ it loses its postings. The same holds for a field of that name XERJ mapped for
+ you dynamically, whatever type it inferred, as long as the value is not itself
+ a multi-vector. Measured on
+ `{text body, keyword emb_chunks, dense_vector emb}`, 50 docs, half `tenant-a`
+ (before → after): `term {emb_chunks:"tenant-a"}` 25 → 25,
+ `terms` 25 → 25, `bool{filter:[term emb_chunks]}` 25 → 25,
+ `bool{must:[match body],filter:[term …]}` 25 → 25,
+ `constant_score{filter:term}` 25 → 25,
+ `bool{must_not:[term emb_chunks]}` 25 → 25, `<seg>.emb_chunks.fst`
+ 54 B → 54 B, `<seg>.emb.fst` 16,713 B → 0. Same with a `text` mapping:
+ `match` / `match_phrase` / `multi_match` / `simple_query_string` / `prefix`
+ all 25 → 25 and `<seg>.emb_chunks.fst` 71 B → 71 B. A `long` mapping keeps a
+ 44 B `.fst` and a `double` mapping a 48 B one, both 25 → 25 on `term`,
+ `terms`, `range`, `bool{filter}` and `bool{must_not}`. A field of that name
+ XERJ mapped for you *dynamically* is exempt on the same rule — measured
+ 25 → 25 for a `text` one (54 B `.fst`) and 25 → 25 for a `long` one
+ (44 B `.fst`).
+
+ **What makes "yields to any declaration" affordable** is that the companion
+ is stopped from ever acquiring one. `<vector>_chunks` is user-supplied
+ `_source` — an array of float arrays — so dynamic mapping used to register it
+ like any other unmapped key and infer `double` from it, producing a mapping
+ entry indistinguishable from one you wrote. Dynamic mapping now declines to
+ register a `<vector>_chunks` key when a `dense_vector` named `<vector>` is
+ already mapped *and* the value is a rectangular, non-empty array of non-empty
+ all-numeric arrays — the exact multi-vector shape, and nothing else. A scalar,
+ a string, a flat array or a ragged one under that name is your field and is
+ mapped as one. Measured on `{text body, dense_vector emb}` with `emb_chunks`
+ left out of the mapping, 50 docs: an `[[…],[…]]` value gets no mapping entry,
+ `<seg>.emb_chunks.fst` = 0 B and a lexical clause on it answers 0; a `"tenant-a"`
+ value is mapped `text`, keeps a 54 B `.fst` and answers 25; a `7` value is
+ mapped `long`, keeps a 44 B `.fst` and answers 25.
+
+ Both halves were measured on their own, by disabling each in the tree and
+ re-running the committed tests:
+
+ | | `emb_chunks.fst` (300-doc) | durable index | declared `long` sibling |
+ |---|---:|---:|---|
+ | this release | **0 B** | 857,265–857,769 B | 44 B `.fst`, answers 25 |
+ | yield restored to `long`/`double`-excluded | 0 B | 857,445 B | **0 B, answers 0, `must_not` returns 50** |
+ | dynamic-mapping refusal disabled | **1,592,118 B** | **2,451,336 B** | 44 B, answers 25 |
+
+ The middle row is the behaviour rc.16 would have shipped; the bottom row is
+ the 67% of the saving that an unconditional "any declaration wins" gives up.
+ Neither cost is paid here.
+
+ A lexical clause that NAMES a `dense_vector` (or one of its companions) is now
+ lowered to `match_none` at plan time rather than falling through to the
+ stored-doc scan. That is the correctness half of the change, not only the
+ speed half: with the postings removed and no lowering,
+ `{"multi_match":{"fields":["emb"],"query":"0"}}` returns **every document**
+ instead of none, because the scan renders the float array to text and every
+ component contains a `0`. Measured on the corpus above, first call for each
+ shape (`main` → postings-gone-without-lowering → this release):
+
+ | query | `main` | no lowering | this release |
+ |---|---|---|---|
+ | `multi_match {fields:["emb"], query:"0"}` | 0 hits, 0.409 ms | **5,000 hits**, 123.5 ms | 0 hits, 0.012 ms |
+ | `multi_match {fields:["emb_chunks"], query:"0"}` | 0 hits, 2.24 ms | **5,000 hits**, 138.5 ms | 0 hits, 0.014 ms |
+ | `term {emb: "<component>"}` | 0 hits, 89.2 ms | 0 hits, 186.8 ms | 0 hits, 0.061 ms |
+ | `range {emb: {gte:-2, lte:2}}` | 0 hits, 119.8 ms | 0 hits, 134.5 ms | 0 hits, 0.009 ms |
+ | `simple_query_string {fields:["emb"]}` | 0 hits, 0.033 ms | 0 hits, 114.5 ms | 0 hits, 0.015 ms |
+ | `constant_score {filter: term emb}` | 0 hits, 83.0 ms | 0 hits, 110.3 ms | 0 hits, 0.012 ms |
+ | `exists {field: "emb"}` | 5,000 hits, 83.0 ms | 5,000 hits, 108.0 ms | 5,000 hits, 88.5 ms |
+ | `match {body: "liquidity"}` | 5,000 hits, 5.18 ms | 5,000 hits, 5.96 ms | 5,000 hits, 4.95 ms |
+ | `term {cat: "even"}` | 2,500 hits, 2.32 ms | 2,500 hits, 2.87 ms | 2,500 hits, 2.27 ms |
+
+ `exists`, `knn` and `semantic` on the field are untouched, a mixed `fields`
+ list keeps its lexical members, and a `nested` query is deliberately not
+ rewritten (its field names resolve against the element, not the root).
+
+ **Four behaviour changes to know about:**
+
+ - **A `dense_vector` (and its undeclared `_chunks` companion) now has no
+ lexical surface at all: every lexical leaf naming one *outright* answers 0
+ hits.** "Outright" excludes one shape, measured rather than assumed: a
+ `fields` entry that is a PATTERN (`["emb.*"]`) is NOT lowered, because
+ resolving a pattern needs the expansion universe — dynamic fields included —
+ and lowering one that would have expanded onto a real lexical field is the
+ silent-zero failure this release goes out of its way to avoid. So
+ `{"multi_match":{"query":"0","fields":["emb.*"]}}` still answers 300 of 300
+ on the 300-doc fixture, exactly as on `main`: the unprojectable clause falls
+ to the stored-doc scan, which resolves the pattern against the source's
+ field names and matches the rendered decimals. Unchanged in both columns, so
+ not a regression — but not closed either. `["emb"]`, `["emb*"]` and `["*"]`
+ all answer 0. Before this release
+ such a clause fell through to the stored-doc scan, which renders the float
+ array back to decimal text and matches against *that*, so whether a shape
+ answered 0 or answered most of the corpus depended entirely on whether the
+ rendered decimals happened to contain the probe. This is a CLASS of changed
+ answers, not one row. Measured on the 300-doc × 128-dim regression fixture,
+ `main` → this release:
+
+ | query | before | after |
+ |---|---:|---:|
+ | `wildcard {emb: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb: {value:"0", fuzziness:2}}` | **300** (every document) | 0 |
+ | `wildcard {emb_chunks: "0*"}` | **300** (every document) | 0 |
+ | `fuzzy {emb_chunks: {value:"0", fuzziness:2}}` | **300** | 0 |
+ | `prefix {emb: "0"}` | 146 | 0 |
+ | `prefix {emb_chunks: "0"}` | 146 | 0 |
+ | `match_phrase_prefix {emb: "0"}` | 50 | 0 |
+ | `term {emb: <exact component>}` numeric | 1 | 0 |
+ | `terms {emb: [<exact component>]}` numeric | 1 | 0 |
+
+ Shapes whose probe text never appeared in a rendered float (`term` with a
+ string, `match`, `match_phrase`, `regexp`, `range`, `simple_query_string`,
+ `query_string`) answered 0 on both sides. ES rejects these queries on a
+ `dense_vector` outright and Lucene gives a field with no terms an empty
+ scorer, so 0 is the answer this release adopts for all of them. The
+ regression test pins the whole class rather than sampling it.
+ - **A `bool` that mentions a `dense_vector` can change `_score` and
+ `max_score` without changing its hit count.** Dropping the dead clause makes
+ the surviving bool projectable onto the inverted index, so BM25 scores it
+ where the stored-doc scan used to. Measured on the same fixture:
+
+ | query | hits | `_score` | `max_score` |
+ |---|---|---|---|
+ | `bool{should:[term emb, term cat:"even"]}` | 150 → 150 | 0.008402659 → 0.6931471 | 1.6931472 → 0.6931471 |
+ | `bool{must:[match body], must_not:[term emb]}` | 300 → 300 | 0.008402659 (unchanged) | 1.6931472 → 0.008402659 |
+
+ Both moves are toward consistency — on `main` these queries reported a
+ `max_score` that **no returned hit carried**, and the first now scores
+ exactly like `{"term":{"cat":"even"}}` on its own, which is what it reduces
+ to. Absolute scores are not part of XERJ's compatibility surface, but
+ anything comparing scores across a version boundary or asserting on
+ `max_score` for a bool that touches a vector field will see this. Plain
+ lexical queries are unaffected; the blast radius is bools with a
+ `dense_vector` clause in them.
+ - **`GET /<index>/_mapping` no longer lists a `<vector>_chunks` field that
+ XERJ mapped for you.** The multi-vector companion beside a declared
+ `dense_vector` used to be registered by dynamic mapping as `double` (an
+ array of float arrays walks to its first scalar). It is not registered at
+ all now, so it disappears from `_mapping` and `_field_caps` — the same
+ treatment `__xerj_passage_meta__<vector>` already had. Nothing that reads
+ `_source` changes; the values are stored and returned exactly as before, and
+ `knn` and `semantic` on the vector itself are untouched. A field of that
+ name that you mapped yourself, or that XERJ mapped from a non-multi-vector
+ value, is unaffected and still listed.
+
+ Aggregating and sorting on that companion still answer `200` — measured on
+ the wire against a live node with `emb` mapped `dense_vector` and
+ `emb_chunks` unregistered: `terms` returns real buckets, `stats` returns
+ `count: 0` byte-identical to the previous release, and `sort` returns sort
+ values. Nothing that worked before returns an error now.
+ - **Existing indices keep their bloat until a merge rewrites them, and an
+ index created by rc.16 or earlier only ever reclaims the base-vector half.**
+ This is a write-side rule. Segments written by an earlier release keep their
+ `.emb.*` / `.emb_chunks.*` files, and the per-segment `fts_has_field` gate
+ reads whichever shape it finds.
+
+ `POST /<index>/_forcemerge?max_num_segments=1` reclaims `.emb.*`. **It does
+ not reclaim the companion.** Measured by building an index on an rc.16
+ server (`emb` mapped `dense_vector` before ingest, 500 docs × 32 dim with
+ `emb_chunks` multi-vectors), then reopening the same data directory with
+ this release and force-merging: `.emb.fst` 262,904 B → 0, but
+ `.emb_chunks.fst` 525,200 B → 525,355 B, and
+ `wildcard {emb_chunks: "0*"}` still answers 501 of 501. The reason is not a
+ bug in the merge: rc.16 persisted `emb_chunks: {"type": "double"}` into the
+ mapping, and the by-name rule correctly yields to a field you (or an earlier
+ release) declared. Since a mapping entry cannot be dropped in place, the
+ remediation for an existing index is to **reindex into a fresh index**. On
+ the corpus this entry headlines that companion is 67% of the saving, so an
+ in-place upgrade recovers roughly a third of the published number.
+
+ The companion rule has an ordering limit of the same kind, and it fails in
+ the safe direction: XERJ can only decline to map a companion it can
+ recognise, and recognising one needs the `dense_vector` to be mapped
+ *before* the documents arrive. Index first and add the mapping afterwards
+ and `<vector>_chunks` already carries a dynamic `double` entry, which the
+ by-name rule then yields to — so that index keeps the companion's term
+ dictionary (measured 9,467 B on a 50-doc × 16-dim fixture) instead of
+ reclaiming it. It costs bytes, not answers.
+
+ *Known limitation, filed as [#382](https://github.com/xerj-org/xerj/issues/382):*
+ refusing a `FieldConfig` for the companion leaves the pre-existing schema-hash
+ throttle asserting the key set is unchanged, so if a later document puts a
+ differently-typed value under `<vector>_chunks` it stays unregistered for up
+ to 100 documents and queries on it answer zero until the throttle expires.
+ Doc-order-dependent and self-healing. The throttle is pre-existing and the fix
+ belongs to it.
+
+ *Not changed:* ES rejects `term`/`match`/`range` on a `dense_vector` outright,
+ where XERJ answers `200`. This release removes bytes and adopts Lucene's
+ zero-hit answer; it does not add the rejection.
+
### Fixed
+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and
+ every write path enforces it**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only
+ warn-log a compile failure, so a pipeline that could never run was
+ acknowledged and echoed back by `GET`. It now compiles first, with three
+ outcomes: compiled → stored and acknowledged; a processor config we refuse →
+ `400`; a processor or option this build cannot honour → accepted (as
+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses
+ rather than storing an untransformed document.
+
+ Defects this surfaced, each of which answered `200`/`201` and did something
+ smaller than asked: **every** ES `rename` processor failed to compile (the
+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and
+ renamed nothing; ES `append` was executed as `set`, replacing the field
+ instead of extending it, and disagreeing with `_simulate`, which implemented
+ a real append; an ES `set` processor did not override an existing value where
+ Elasticsearch does (the ES→XERJ translation now writes `"override": true`
+ explicitly when the processor omits it — the **native** `type: "set"` stage
+ keeps its long-standing `override: false` default, because native definitions
+ are recompiled from `cluster_state.json` at every boot and flipping the stage
+ default would have changed what an already-running cluster's pipelines do to
+ documents on a binary upgrade alone); an unknown `pii_redaction` type built an
+ empty pattern set and redacted nothing; an unknown `grok` pattern name
+ silently became the
+ catch-all; an unknown `convert` target left the field unconverted per
+ document; array-typed keys given as a bare string were dropped and the stage
+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the
+ definition while leaving the compiled pipeline live under the same name.
+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`
+ rather than being refused.
+
+ Two more of the same, at the definition's own edges: a **pipeline-level**
+ `on_failure` block (the top-level body key, not the processor-level one) was
+ accepted, echoed by `GET`, and never run — the pipeline is now recorded
+ unrunnable, exactly as its processor-level twin already was; and `set` with
+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted
+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:
+ missing value`, which told the caller their valid ES pipeline was malformed.
+
+ **The refusal reads the value, not the key.** A processor-level option is
+ refused only when what the caller *wrote* asks for behaviour this build does
+ not have; writing the value XERJ already implements is the same as omitting
+ it. `ignore_failure: false` is Elasticsearch's own default and ES clients emit
+ it verbatim — a presence check made that pipeline register and then `400`
+ every write through it, while the byte-identical processor without the key
+ indexed fine. Accepted: `ignore_failure: false`, an empty `on_failure: []`,
+ `ignore_missing: true` (which is exactly what `remove`/`rename`/`convert`/
+ `date` already do — they pass the document through unchanged when the field
+ is absent), `ignore_empty_value: false`, `allow_duplicates: true`. Refused:
+ the opposite value in each case, and any `if` at all. (Modelled on Lucene's
+ consume-then-complain: `AbstractAnalysisFactory.getBoolean(args, name,
+ default)` at `lucene/core/src/java/org/apache/lucene/analysis/AbstractAnalysisFactory.java:213-217`
+ removes the key and returns the default only when it was absent, and
+ `ArabicStemFilterFactory` at
+ `lucene/analysis/common/src/java/org/apache/lucene/analysis/ar/ArabicStemFilterFactory.java:44-51`
+ then throws on whatever is left. Lucene is Apache-2.0, the same licence as
+ XERJ; no code is reproduced.)
+
+ **`remove` and `rename` no longer walk around the gate.** Their ES→XERJ
+ translation built a fresh config object holding only the translated keys, so
+ a processor-level `if` / `on_failure` / `ignore_failure` / `ignore_missing` on
+ them was dropped *before* the gate ran: `{"remove": {"field": "secret", "if":
+ "ctx.tenant == 'a'"}}` compiled, was acknowledged, and dropped the field on
+ every document — while the identical `if` on `set` was refused. Every branch
+ of the translation now edits the caller's config instead of rebuilding it.
+
+ **`_simulate` speaks Elasticsearch's vocabulary again.** It walks the
+ *stored* pipeline, and once ES `rename` started compiling the stored form
+ became XERJ's internal `stages` shape — whose stage names leaked straight into
+ the response, so `processor_type` read `field_rename` where ES reports
+ `rename`. The stages→processors conversion now inverts both halves of the
+ translation, name and config.
+
+- **`GET /_ingest/pipeline[/{id}]` answers in Elasticsearch's `processors`
+ vocabulary instead of XERJ's internal `stages` shape**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)). Two shapes reach the
+ store: the translated `stages` body (everything that compiles) and the raw ES
+ body (kept verbatim for a definition XERJ stores but cannot run). Only the
+ second was ever ES-shaped, so `GET` spoke XERJ's vocabulary for every pipeline
+ that *worked* — measured before this release, an ES `rename` pipeline came
+ back as
+ `{"p":{"description":"d","stages":[{"type":"field_rename","config":{"mappings":{"a":"b"}}}]}}`.
+ This is a wire-visible change on both `GET /_ingest/pipeline/{id}` and
+ `GET /_ingest/pipeline`. It also mattered beyond cosmetics: `GET`'s output is
+ what an operator edits and `PUT`s back, and a `stages`-shaped body takes the
+ handler's pass-through branch — which is exactly how the restart hole
+ described under *Changed* was reached in ordinary use.
+
+ `set` is rendered with an explicit `"override"`, always, because it is the one
+ key whose default differs between the two vocabularies (Elasticsearch `true`,
+ XERJ's native `set` stage `false`). Emitting a bare ES `set` for a native
+ stage would claim it overwrites when it preserves. A pipeline created through
+ `PUT /_ingest/pipeline` therefore shows `"override": true` even when the
+ caller omitted it — that is the value XERJ applies, and it round-trips: PUTting
+ `GET`'s own output back cannot silently flip the stage's behaviour. Relatedly,
+ `_simulate`'s `set` interpreter honoured no `override` at all and always
+ overwrote, so it reported a document the ingest path would never produce; it
+ now applies Elasticsearch's rule (default `true`; `false` leaves an existing
+ non-null value alone).
+
+ **This is also the only read endpoint for a pipeline created through
+ `PUT /v1/pipelines/{name}`** — the native surface registers no `GET` — so the
+ round trip has to be lossless in xerj's own vocabulary too, and it was not.
+ `PUT /_ingest/pipeline/{id}` rebuilt the stored definition as
+ `{description, stages}` and dropped every other top-level key, so PUTting
+ `GET`'s own output back silently discarded a natively-defined pipeline's
+ `on_error` and `timeout_ms`: the error policy reverted from `pass` (keep the
+ document) to the `Drop` default (discard it) under a
+ `200 {"acknowledged": true}`. The translation now edits the body instead of
+ rebuilding it — the same rule already applied to `remove`/`rename` one level
+ down. Measured on this release, `{"description":"n3","on_error":"pass",
+ "timeout_ms":250,"stages":[…]}` reads back with both keys intact, and PUTting
+ that body back leaves them intact, before and after a restart. Elasticsearch's
+ own pipeline metadata (`version`, `_meta`, `deprecated`) rides along the same
+ way and is now echoed back rather than dropped. `PUT /v1/pipelines/{name}`
+ given that ES-shaped body used to answer `500` with `internal error: missing
+ field stages`; it now answers `400` naming `/_ingest/pipeline/{name}` as the
+ endpoint that accepts it.
+
+- **Four wire-visible refusals that were fixed in this sweep but not written
+ down** ([#204](https://github.com/xerj-org/xerj/issues/204)). All four
+ replace a silent fallback with an error, which is the point of the issue, but
+ each changes a status code:
+ `POST /v1/admin/backup` answers `400` on a request body that is present but
+ unparseable, where it previously ran a *default* backup under a success
+ response; `GET /{index}/_stats` and `GET /_all/_stats` carry an added
+ non-Elasticsearch key `primaries.mappings.schema_persist_failures` inside an
+ otherwise ES-shaped response; the authorization layer's response pruning
+ answers `500` when it cannot parse the body it was asked to filter, instead
+ of falling back to the unfiltered bytes; and the bundled console's
+ `DELETE /_xerj-console/api/v1/auth/passkeys/{id}` propagates a failed delete
+ instead of discarding it — it answered `204 No Content` unconditionally, so a
+ revoke refused by the engine (index write block, disk flood stage, storage
+ error) reported success while the passkey stayed on disk and kept
+ authenticating. An already-absent credential is still a `204`; only a genuine
+ failure now surfaces.
+
+ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**
+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,
+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`
+ previously read the instruction and dropped it — a redaction pipeline that
+ the single-document endpoint refused was ignored in bulk, and the document
+ was stored verbatim under `errors: false`. All four now run the pipeline, and
+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the
+ name before touching a single document, so a bad pipeline fails the request
+ instead of half of it.
+
+ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`
+ copied at most 10,000 documents and swallowed every per-document write error
+ before answering `{"acknowledged": true}` (now full keyset paging, and any
+ failed write — or a failed source flush — fails the request); and a failed
+ `_bulk` item no longer reports `"result": "deleted"` alongside its own
+ `status: 400`, which claimed a deletion that never happened.
+
+ **`ignore_missing: false` is refused on purpose, and it is not the same call
+ as `ignore_failure: false`.** Both are Elasticsearch defaults that ES clients
+ emit verbatim, so the difference is worth stating plainly: the rule is *accept
+ the value this build actually implements*, and the two land on opposite sides
+ of it. `ignore_failure: false` means "a failing processor is not ignored",
+ which is exactly what XERJ does, so it is accepted. `ignore_missing: false`
+ means "FAIL the document when the field is absent", which XERJ cannot do at
+ all — `ProcessAction` has no error variant, and every field-reading stage
+ passes the document through untouched. Accepting it would be precisely the
+ silent lie this issue exists to remove, so it is refused: a `remove` /
+ `rename` / `convert` / `date` processor that spells out `ignore_missing:
+ false` registers and then `400`s every write through it. **Omitting the key
+ is accepted** and behaves as `true`. That asymmetry — omission accepted,
+ the ES default written out refused — is deliberate but is a hard `400` for
+ ES-generated pipelines; strip the key, or set it to `true`, to state what
+ XERJ will actually do. `ignore_empty_value` is the mirror image: omitting it
+ or writing `false` is honoured exactly, `true` is refused.
+
+ **Known gaps, stated rather than closed:** `_simulate` interprets the stored
+ ES body directly and covers processors the compiled path does not, so a
+ pipeline recorded as unrunnable can still show a transformation under
+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`
+ carries no runnability signal — it renders the stored definition in ES
+ vocabulary whether or not this build can run it, so an operator only learns
+ at write time; a write addressed to an **alias** does not pick up the
+ backing index's `default_pipeline`; `_update_by_query` reports a `drop`
+ processor as a per-document failure, because it updates in place and has no
+ delete path; `Engine::degraded_pipelines` is in-memory only and has no
+ HTTP surface yet, so a boot-replay degradation is visible in the startup log
+ and nowhere else — unlike a definition-time refusal, which is now durable
+ (see *Changed*); and a refusal marker is only read back at boot, so a data
+ dir opened by two processes at once is outside what the sidecar can promise
+ (the node lock already prevents that).
+
- **Unity assets were sampled through a 4 MiB cap, silently junking whole
object classes.** Unity YAML is a grouped family — each `unity_class` is its
own dataset and a class's first document can sit anywhere in a scene that