Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge remote-tracking branch 'origin/main' into fix/issue-204-fallback-sweep

xerj-org4d0d645ab45c
diff --cc CHANGELOG.md
index 4e9e5110e,c46c5dcfb..4eae01ea9
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,311 -7,152 +7,457 @@@ and this project adheres to [Semantic V
## [Unreleased]
+ ### Added
+
+ - **Push a filtered subset of indices to an external ES-compatible target —
+ the single-node WAL tap** ([#320](https://github.com/xerj-org/xerj/issues/320)).
+ Nothing pushed data out of the engine before this: `_ccr/*` answered `501`,
+ reindex-from-remote was refused up front, and snapshot/restore — scheduled and
+ whole-index — was the only export path. A XERJ node used as a lightweight edge
+ collector had no way to stream a curated slice of its data up to a central
+ cluster.
+ - New `[wal_tap]` config section (10 settings, off by default) and a runtime
+ REST surface: `GET`/`PUT`/`DELETE /_xerj/wal_tap` and
+ `GET /_xerj/wal_tap/_stats`. The allowlist is glob-based and adjustable
+ without a restart. That brings the documented config surface to
+ **115 settings** (counted by `journey_zero_config`, not by hand).
+ - **Every `/_xerj/*` route is superuser-only.** `PUT /_xerj/wal_tap` is a
+ whole-node export primitive — `target_url` names any host, `indices: ["*"]`
+ names every index, and the tap attaches an operator-supplied
+ `Authorization` header — so it is data exfiltration and authenticated SSRF
+ in one call. Same reasoning `authz.rs` already applies to snapshot and
+ restore: nothing here is index-scoped, so there is no per-index decision to
+ make. The reads are covered too — `GET /_xerj/wal_tap` echoes `target_url`
+ and `_stats` enumerates the node's whole index inventory.
+ - **`target_url` may not carry credentials.** `https://user:pass@host` is an
+ ordinary URL that `reqwest` turns into a `Basic Authorization` header — i.e.
+ `target_auth` by another spelling, in the one field the API echoes back and
+ the boot log prints. It is refused at startup and by `PUT`, and redacted
+ everywhere it is rendered.
+ - The wire format is `_bulk` and nothing else, so the target can be
+ Elasticsearch, OpenSearch, or another XERJ node with nothing installed.
+ - **This is not CCR.** It is one-directional, single-node, and independent of
+ the post-GA sharding/replication track. `_ccr/*` still answers `501`.
+ - The tap adds nothing to the write path: it tails WAL files from disk on a
+ timer and takes no lock the writer wants. A poll costs the bytes appended
+ since the last one, not the size of the file — the tail reader is a sliding
+ window over a `File`, shaped after Lucene's `BufferedIndexInput.refill`
+ (`lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java:289-317`,
+ Apache-2.0). A caught-up poll reads the 16-byte WAL header and stops —
+ measured at exactly 16 bytes over a 4,176,200-byte generation, asserted
+ through `tail_shard` itself by
+ `wal::tests::a_caught_up_poll_does_not_read_the_whole_generation`. (The
+ window's 64 KiB read-ahead is deliberately not applied to that header read:
+ it was, and it made the figure 65,536 bytes per shard per poll rather
+ than 16.)
+ - Delivery is at-least-once, with `version_type: external` carrying the WAL
+ `seq_no` on every action so a redelivery converges on the same document the
+ source holds. Covered against a stub that implements the actual ES external
+ versioning rule — absent `_id` is created at any version, present `_id` is
+ overwritten only by a strictly greater one, everything else comes back
+ `409 version_conflict_engine_exception` — and asserted on both halves: four
+ documents accepted on the first delivery, four conflicts and zero extra
+ `docs_shipped` on the redelivery, watermark unchanged. Note that XERJ's own
+ `_bulk` ignores per-action `version` / `version_type` (only the single-doc
+ path honours them), so a XERJ target degrades to last-write-wins by
+ arrival; against Elasticsearch or OpenSearch the mechanism is live.
+ - `_stats` counts what the **target accepted**, per item, not what was
+ rendered into the request. `lag_seq` is derived from the highest accepted
+ `seq_no`, and each index carries a `healthy` boolean that is false both when
+ sends are failing and when the target answers `200` while applying none of
+ the actions inside it — for **any** reason, not only a version conflict.
+ A target rejecting every document with `mapper_parsing_exception` is a `200`
+ with `errors: true`, so no send fails and the cursor advances past every
+ dropped document; `last_item_rejection` in `_stats` carries the target's own
+ word for it, because "fix the mapping" and "look for a second writer" are
+ opposite responses.
+ - Cursor state is flushed once per poll cycle rather than once per index per
+ poll; a deferred flush can only cause a redelivery, which the external
+ versioning above absorbs.
+ - Deleting an index drops its cursor at the moment of deletion, so a
+ `DELETE` + `PUT` of the same name inside one poll interval — never observed
+ as an absence at a 500 ms default — cannot leave a byte offset pointing into
+ a WAL stream that no longer exists. For a delete that happened while the
+ node was down, the reader treats a cursor offset past the end of its
+ generation as the discontinuity it is (WAL files are append-only) and
+ restarts the stream with a reported gap instead of clamping to EOF.
+ - WAL retention is deliberately *not* coupled to the tap — a slow remote must
+ not be able to fill the local disk. A tap that falls far enough behind loses
+ entries and says so: `gaps` in `_stats`, plus a warning per occurrence. New
+ `wal_tap.min_retained_generations` (default `0`, unchanged behaviour) buys
+ **bounded** slack: it is a floor, not an Elasticsearch-style retention
+ lease, so it holds at most `n × storage.wal_max_size_mb` per WAL shard
+ however far behind any consumer is. The floor is enforced inside
+ `IndexStore::wal_maintain_all_verified` — the prune pass the engine actually
+ runs — using the same arithmetic and the same "inside the deletion pass, not
+ in its callers" placement as Lucene's
+ `KeepLastNCommitsDeletionPolicy.onCommit`
+ (`lucene/core/src/java/org/apache/lucene/index/KeepLastNCommitsDeletionPolicy.java:51-58`,
+ Apache-2.0). At the default `0` the loop is unchanged: no extra syscall, no
+ behaviour change.
+ - **The retention floor is live, and the API no longer promises otherwise.**
+ `min_retained_generations` is the one `[wal_tap]` setting that does not live
+ in the tap — it lives in every open index's `WalWriter`, seeded at open from
+ `Engine.config`, an `Arc<Config>` written once at boot and never mutated.
+ `PUT /_xerj/wal_tap` therefore pushes it straight onto the live writers
+ (`Engine::apply_wal_retention_floor`, after Lucene's `LiveIndexWriterConfig`
+ and `IndexFileDeleter.revisitPolicy`,
+ `lucene/core/src/java/org/apache/lucene/index/LiveIndexWriterConfig.java:39-126`
+ and `IndexFileDeleter.java:516-543`) and answers with
+ `retention_floor_applied_to_indices` — the number of writers it reached —
+ instead of a warning. `DELETE /_xerj/wal_tap` releases it the same way, and
+ `Engine::new` folds the tap's effective configuration into the boot `Config`
+ so a store opened after a restart is seeded with the persisted floor rather
+ than the file's. Verified over HTTP end to end
+ (`xerj-api/tests/wal_tap_retention_floor_is_applied.rs`): after
+ `PUT {"min_retained_generations": 3}` an index opened before the call reads
+ `3` on all 16 WAL shards, an index created after it opens with `3`, and both
+ still read `3` after a restart whose config file still says `0`.
+ - A configuration set through `PUT /_xerj/wal_tap` is persisted next to the
+ cursors and re-applied over the config file on the next boot, so "runtime
+ config, no restart" does not quietly mean "and gone after one".
+ `DELETE /_xerj/wal_tap` drops the overlay and reverts to the file — to the
+ value the tap kept verbatim at construction, not to a re-read of
+ `Engine.config`. The state file holds `target_auth`, so it is written `0600`
+ through the same writer the API-key store uses.
+ - The unapplied-batch health signal counts **polls**, not `_bulk` requests.
+ How many chunks a poll splits into is `max_batch_bytes` arithmetic: at
+ `max_batch_bytes = 1` one legitimate at-least-once redelivery of four
+ documents produced four all-conflict `_bulk` responses, tripped the
+ three-in-a-row threshold on its own, and emitted `healthy: false` with
+ `last_error: "… nothing is being replicated"` about a poll in which every
+ document was already at the target. The field is now
+ `consecutive_unapplied_polls` and rises by at most one per poll; three
+ unapplied polls in a row still reports the stall.
+ - Every multiply in the retry-backoff computation saturates.
+ `max_retry_backoff_secs.max(1) * 1000` overflowed above ~1.8e16: a panic
+ under `cargo test`'s overflow-checks that killed the spawned tap task
+ silently, and under a release profile a *wrapped* cap that can be
+ arbitrarily small — the value `18_446_744_073_709_552` wrapped to a 384 ms
+ ceiling, i.e. a retry storm aimed at an already-failing target.
+ - **Every `[wal_tap]` numeric knob is range-checked at boot as well as by
+ `PUT`,** through one shared `WalTapConfig::check_limits`. The config file
+ was the way around the API's validation: `PUT
+ {"min_retained_generations": 100}` is a `400` because the knob costs
+ `n × storage.wal_max_size_mb` per WAL shard per index, while `xerj.toml`
+ took the same `100` in silence and the node then held 100 rotated
+ generations per shard forever. Bounds are `poll_interval_ms 50..=60000`,
+ `max_retry_backoff_secs 1..=86400`, `min_retained_generations <= 64`, and
+ `max_batch_docs` / `max_batch_bytes` / `request_timeout_secs` at least 1.
+ Same precedent as `compression.block_size_docs` (#318): an out-of-range
+ value is a typo the operator should hear about at boot, not as a disk-full
+ page later.
+ - XERJ's own `.xerj*` system indices are never shipped, whatever the allowlist
+ says, and a wildcard never expands to a hidden index.
+ - There is no backfill. A new index is shipped whole; an established one ships
+ from the moment it is allowlisted. Seed the target with snapshot/restore
+ first if it needs the existing documents.
+
+### Changed
+
+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot
+ honour, instead of accepting it and analysing differently**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an
+ existing index is opened, where a hard failure would brick it), so an
+ unresolvable tokenizer silently became `standard` and an unresolvable token
+ filter was dropped — the index was created, `GET /_settings` echoed the block
+ back, and the field was analysed in a way nobody asked for. Index creation
+ now validates the block and answers `400` with the exact constructs it cannot
+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused
+ (it only ever changed what `GET /_settings` echoed; the registry is built
+ once, at create/open, and there is no rebuild path — Elasticsearch also
+ refuses analysis updates on an open index). That refusal covers every
+ spelling this handler *accepts*, not just the nested one: the flat dotted
+ `{"index.analysis.analyzer.x.type": "custom"}` and half-dotted
+ `{"index": {"analysis.analyzer.x.type": "custom"}}` forms that ES clients
+ routinely send used to walk straight past it into the display copy, and
+ `GET /_settings` then echoed back an analyzer that analysed nothing.
+
+ **This rejects settings blocks that XERJ used to accept with a `200`, and the
+ ES-YAML conformance suite does not cover them — CI being green is not
+ evidence that your index still creates.** What is honoured:
+
+ | | honoured (still `200`) |
+ |---|---|
+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |
+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |
+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |
+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |
+ | `analysis.normalizer` | nothing — never built |
+
+ Anything else in those positions is now a `400`. There is one important
+ exception to read first: XERJ resolves a declared filter or tokenizer by
+ **name**, not by `type`, so a declaration whose *name* is a built-in is still
+ honoured whatever its type says. Measured, all still `200`:
+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,
+ `{"lowercase": {"type": "lowercase"}}`,
+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`
+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which
+ has a regression test pinning it. Naming a built-in analyzer on a field
+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.
+
+ What now fails is a declaration under a name that does NOT resolve — which is
+ the usual ES habit of naming filters after what they do in *your* index.
+ Measured blocks that returned `200` in rc.14 and return `400` from this
+ release:
+
+ - a `keyword` field's `normalizer` (any);
+ - a `char_filter` (any, including `html_strip` and `mapping`), declared
+ either under `analysis.char_filter` or as an analyzer's `char_filter`;
+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,
+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,
+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,
+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,
+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,
+ which never lowercased anything, because the analyzer referencing
+ `my_lowercase` found no such filter and dropped it;
+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,
+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,
+ `letter`, `lowercase` or `simple_pattern`;
+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,
+ `fingerprint`, or a language other than `english`;
+ - a `synonym` filter whose `synonyms` is a bare string rather than an array
+ (it built a filter with zero rules).
+
+ **Upgrading:** there is no opt-out flag. Either remove the analysis block and
+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only
+ the constructs in the table. The `400` names every offending construct in one
+ response, so one round-trip is enough to see the whole list. **Two paths are
+ not covered and still accept these blocks silently: an index template's
+ `template.settings.analysis` is dropped wholesale before the gate ever sees
+ it, and option-level divergence inside a supported type (`synonyms_path`,
+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**
+
+- **BREAKING — `PUT /_ingest/pipeline/{id}` refuses processor configuration
+ XERJ cannot honour, so a pipeline that used to be acknowledged can now be a
+ `400`, and a stored one can start refusing writes**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)). This is the same
+ change as the "`200 acknowledged` … means we will run this" entry under
+ *Fixed* below, stated from the upgrader's side, because it converts `201`
+ into `400` for any pipeline using an ES processor or option this build does
+ not implement. Read that entry for the full list.
+
+ **What is NOT broken, deliberately.** A definition persisted by a build with
+ **no gate** was accepted by whichever build wrote it, and there is no caller
+ left to answer `400` at boot. Boot replay therefore compiles *those*
+ definitions in a compatibility mode that reproduces the previously shipped
+ behaviour for every check this sweep added (unknown `grok` pattern name,
+ unknown `convert` target, a bare string where an array is required,
+ `set.copy_from`, a processor-level `if`/`on_failure`/`ignore_failure`), so
+ **an upgrade alone never turns a running cluster's writes from `201` into
+ `400`**. Each one is logged at ERROR naming the pipeline and the offending
+ config, and recorded in `Engine::degraded_pipelines`; a re-`PUT` of the same
+ definition answers `400` with the same reason, which is the repair path. A
+ definition that did not compile before the upgrade either (unknown stage
+ type, missing required field) still fails at boot and is recorded unrunnable,
+ as before.
+
+ **Which definitions those are is decided by provenance, not by the calendar.**
+ A definition THIS build refused at `PUT` time is recorded in
+ `<data_dir>/ingest-pipeline-refusals.json` and replayed at *definition*
+ strictness, so the refusal survives a restart. Without that marker the replay
+ could not tell "written by an older, laxer build" from "written by this build
+ and answered `400`", and a plain restart overturned the answer the operator
+ had already been given: measured, `PUT` a `{"remove": {"field": "secret"}}`
+ pipeline, add an `if` guard to it (refused — every write `400`s), restart the
+ node and nothing else, and the identical write answered `201` with `secret`
+ dropped from a document the guard EXCLUDES. It covered every check the sweep
+ added; only an unimplemented *stage* survived a restart, because the raw ES
+ body it stores does not deserialise. This is the same pattern as an index's
+ `<index>/analysis-binding.json`: the marker's ABSENCE is the load-bearing
+ signal.
+
+ The marker is provenance, not a tombstone. A marked definition that compiles
+ cleanly on the new build — because that build implements what the refusing
+ one could not — clears its own marker at boot and runs, with an INFO line; a
+ successful re-`PUT` or a `DELETE` clears it too. If the file exists but
+ cannot be read or parsed, no pipeline's provenance is known, so **every**
+ persisted pipeline is replayed at definition strictness and any that fails is
+ recorded unrunnable with the file named in the reason — fail closed and
+ loudly, rather than silently reopening the hole. An older binary reading a
+ data dir written by this one simply ignores the file; it is a sidecar
+ precisely because `cluster_state.json`'s format-1 envelope is
+ `deny_unknown_fields` and widening it would break downgrades.
+
+ The same rule applies one level up. Teaching the analysis resolver the
+ canonical `settings.index.analysis` nesting (see the analysis entry above) is
+ a fix at index *create* and a data bug at index *open*: `settings.json` keeps
+ the caller's nesting verbatim, so an index created before this release with
+ the canonical shape has postings that were written with `standard`, and
+ honouring the declaration on reopen would tokenise every new write
+ differently from everything already on disk. An index now records the
+ analysis binding it was created with (`<index>/analysis-binding.json`) and
+ keeps it for life. An index with no marker and analyzers declared *only*
+ under `index.analysis` keeps the legacy binding if it holds documents — with
+ an ERROR naming the index and saying a reindex is the repair — and adopts the
+ canonical binding if it is empty, recording that so a later restart cannot
+ disagree.
+
+### Fixed
+
+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and
+ every write path enforces it**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only
+ warn-log a compile failure, so a pipeline that could never run was
+ acknowledged and echoed back by `GET`. It now compiles first, with three
+ outcomes: compiled → stored and acknowledged; a processor config we refuse →
+ `400`; a processor or option this build cannot honour → accepted (as
+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses
+ rather than storing an untransformed document.
+
+ Defects this surfaced, each of which answered `200`/`201` and did something
+ smaller than asked: **every** ES `rename` processor failed to compile (the
+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and
+ renamed nothing; ES `append` was executed as `set`, replacing the field
+ instead of extending it, and disagreeing with `_simulate`, which implemented
+ a real append; an ES `set` processor did not override an existing value where
+ Elasticsearch does (the ES→XERJ translation now writes `"override": true`
+ explicitly when the processor omits it — the **native** `type: "set"` stage
+ keeps its long-standing `override: false` default, because native definitions
+ are recompiled from `cluster_state.json` at every boot and flipping the stage
+ default would have changed what an already-running cluster's pipelines do to
+ documents on a binary upgrade alone); an unknown `pii_redaction` type built an
+ empty pattern set and redacted nothing; an unknown `grok` pattern name
+ silently became the
+ catch-all; an unknown `convert` target left the field unconverted per
+ document; array-typed keys given as a bare string were dropped and the stage
+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the
+ definition while leaving the compiled pipeline live under the same name.
+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`
+ rather than being refused.
+
+ Two more of the same, at the definition's own edges: a **pipeline-level**
+ `on_failure` block (the top-level body key, not the processor-level one) was
+ accepted, echoed by `GET`, and never run — the pipeline is now recorded
+ unrunnable, exactly as its processor-level twin already was; and `set` with
+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted
+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:
+ missing value`, which told the caller their valid ES pipeline was malformed.
+
+ **The refusal reads the value, not the key.** A processor-level option is
+ refused only when what the caller *wrote* asks for behaviour this build does
+ not have; writing the value XERJ already implements is the same as omitting
+ it. `ignore_failure: false` is Elasticsearch's own default and ES clients emit
+ it verbatim — a presence check made that pipeline register and then `400`
+ every write through it, while the byte-identical processor without the key
+ indexed fine. Accepted: `ignore_failure: false`, an empty `on_failure: []`,
+ `ignore_missing: true` (which is exactly what `remove`/`rename`/`convert`/
+ `date` already do — they pass the document through unchanged when the field
+ is absent), `ignore_empty_value: false`, `allow_duplicates: true`. Refused:
+ the opposite value in each case, and any `if` at all. (Modelled on Lucene's
+ consume-then-complain: `AbstractAnalysisFactory.getBoolean(args, name,
+ default)` at `lucene/core/src/java/org/apache/lucene/analysis/AbstractAnalysisFactory.java:213-217`
+ removes the key and returns the default only when it was absent, and
+ `ArabicStemFilterFactory` at
+ `lucene/analysis/common/src/java/org/apache/lucene/analysis/ar/ArabicStemFilterFactory.java:44-51`
+ then throws on whatever is left. Lucene is Apache-2.0, the same licence as
+ XERJ; no code is reproduced.)
+
+ **`remove` and `rename` no longer walk around the gate.** Their ES→XERJ
+ translation built a fresh config object holding only the translated keys, so
+ a processor-level `if` / `on_failure` / `ignore_failure` / `ignore_missing` on
+ them was dropped *before* the gate ran: `{"remove": {"field": "secret", "if":
+ "ctx.tenant == 'a'"}}` compiled, was acknowledged, and dropped the field on
+ every document — while the identical `if` on `set` was refused. Every branch
+ of the translation now edits the caller's config instead of rebuilding it.
+
+ **`_simulate` speaks Elasticsearch's vocabulary again.** It walks the
+ *stored* pipeline, and once ES `rename` started compiling the stored form
+ became XERJ's internal `stages` shape — whose stage names leaked straight into
+ the response, so `processor_type` read `field_rename` where ES reports
+ `rename`. The stages→processors conversion now inverts both halves of the
+ translation, name and config.
+
+- **`GET /_ingest/pipeline[/{id}]` answers in Elasticsearch's `processors`
+ vocabulary instead of XERJ's internal `stages` shape**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)). Two shapes reach the
+ store: the translated `stages` body (everything that compiles) and the raw ES
+ body (kept verbatim for a definition XERJ stores but cannot run). Only the
+ second was ever ES-shaped, so `GET` spoke XERJ's vocabulary for every pipeline
+ that *worked* — measured before this release, an ES `rename` pipeline came
+ back as
+ `{"p":{"description":"d","stages":[{"type":"field_rename","config":{"mappings":{"a":"b"}}}]}}`.
+ This is a wire-visible change on both `GET /_ingest/pipeline/{id}` and
+ `GET /_ingest/pipeline`. It also mattered beyond cosmetics: `GET`'s output is
+ what an operator edits and `PUT`s back, and a `stages`-shaped body takes the
+ handler's pass-through branch — which is exactly how the restart hole
+ described under *Changed* was reached in ordinary use.
+
+ `set` is rendered with an explicit `"override"`, always, because it is the one
+ key whose default differs between the two vocabularies (Elasticsearch `true`,
+ XERJ's native `set` stage `false`). Emitting a bare ES `set` for a native
+ stage would claim it overwrites when it preserves. A pipeline created through
+ `PUT /_ingest/pipeline` therefore shows `"override": true` even when the
+ caller omitted it — that is the value XERJ applies, and it round-trips: PUTting
+ `GET`'s own output back cannot silently flip the stage's behaviour. Relatedly,
+ `_simulate`'s `set` interpreter honoured no `override` at all and always
+ overwrote, so it reported a document the ingest path would never produce; it
+ now applies Elasticsearch's rule (default `true`; `false` leaves an existing
+ non-null value alone).
+
+- **Three wire-visible refusals that were fixed in this sweep but not written
+ down** ([#204](https://github.com/xerj-org/xerj/issues/204)). All three
+ replace a silent fallback with an error, which is the point of the issue, but
+ each changes a status code:
+ `POST /v1/admin/backup` answers `400` on a request body that is present but
+ unparseable, where it previously ran a *default* backup under a success
+ response; `GET /{index}/_stats` and `GET /_all/_stats` carry an added
+ non-Elasticsearch key `primaries.mappings.schema_persist_failures` inside an
+ otherwise ES-shaped response; and the authorization layer's response pruning
+ answers `500` when it cannot parse the body it was asked to filter, instead
+ of falling back to the unfiltered bytes.
+
+ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**
+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,
+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`
+ previously read the instruction and dropped it — a redaction pipeline that
+ the single-document endpoint refused was ignored in bulk, and the document
+ was stored verbatim under `errors: false`. All four now run the pipeline, and
+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the
+ name before touching a single document, so a bad pipeline fails the request
+ instead of half of it.
+
+ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`
+ copied at most 10,000 documents and swallowed every per-document write error
+ before answering `{"acknowledged": true}` (now full keyset paging, and any
+ failed write — or a failed source flush — fails the request); and a failed
+ `_bulk` item no longer reports `"result": "deleted"` alongside its own
+ `status: 400`, which claimed a deletion that never happened.
+
+ **`ignore_missing: false` is refused on purpose, and it is not the same call
+ as `ignore_failure: false`.** Both are Elasticsearch defaults that ES clients
+ emit verbatim, so the difference is worth stating plainly: the rule is *accept
+ the value this build actually implements*, and the two land on opposite sides
+ of it. `ignore_failure: false` means "a failing processor is not ignored",
+ which is exactly what XERJ does, so it is accepted. `ignore_missing: false`
+ means "FAIL the document when the field is absent", which XERJ cannot do at
+ all — `ProcessAction` has no error variant, and every field-reading stage
+ passes the document through untouched. Accepting it would be precisely the
+ silent lie this issue exists to remove, so it is refused: a `remove` /
+ `rename` / `convert` / `date` processor that spells out `ignore_missing:
+ false` registers and then `400`s every write through it. **Omitting the key
+ is accepted** and behaves as `true`. That asymmetry — omission accepted,
+ the ES default written out refused — is deliberate but is a hard `400` for
+ ES-generated pipelines; strip the key, or set it to `true`, to state what
+ XERJ will actually do. `ignore_empty_value` is the mirror image: omitting it
+ or writing `false` is honoured exactly, `true` is refused.
+
+ **Known gaps, stated rather than closed:** `_simulate` interprets the stored
+ ES body directly and covers processors the compiled path does not, so a
+ pipeline recorded as unrunnable can still show a transformation under
+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`
+ carries no runnability signal — it renders the stored definition in ES
+ vocabulary whether or not this build can run it, so an operator only learns
+ at write time; a write addressed to an **alias** does not pick up the
+ backing index's `default_pipeline`; `_update_by_query` reports a `drop`
+ processor as a per-document failure, because it updates in place and has no
+ delete path; and `Engine::degraded_pipelines` is in-memory only and has no
+ HTTP surface yet, so a boot-replay degradation is visible in the startup log
+ and nowhere else — unlike a definition-time refusal, which is now durable
+ (see *Changed*).
+
## [1.0.0-rc.16] - 2026-08-13
Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`