AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
Merge origin/main into fix/issue-204-fallback-sweep
xerj-org3e73a00c07ba
diff --cc CHANGELOG.mdindex 00d8a674a,23cc2c8fc..05c76f536--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,77 -7,138 +7,207 @@@ and this project adheres to [Semantic V## [Unreleased]+ ### Fixed++ - **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detaches+ the index, and the detach survives a restart** (#282, ported from #262). The+ null previously fell through a string-only settings reader, so the operator+ got `200 acknowledged` while the index stayed attached — to a policy whose+ delete phase then destroyed it. A detach now removes the execution cursor,+ writes a persisted tombstone (`ism_managed_indices.json` grew a+ `managed`/`detached` envelope; the old bare-map file is still read), and+ scrubs the stale `index.lifecycle.name` from the stored settings.+ - **The lifecycle delete action gained #262's safety rails.** It now refuses —+ visibly, in `explain`, never silently — to delete a dot-prefixed internal+ index (`.ds-*` backing indices exempt), a data stream's current write index,+ or an index whose age cannot be established from its execution cursor.+ - **An ILM policy naming an action the engine cannot execute is refused at PUT+ time with the action named** (400), instead of being stored with the action+ silently dropped — the accepted-and-ignored class (#204). Executable ILM+ actions are `rollover`, `delete`, `readonly`.++ ### Added++ - **`GET /_ilm/status`, `POST /_ilm/start`, `POST /_ilm/stop`** (#282): the+ operator can halt lifecycle execution without stopping the node; a stopped+ engine's tick touches nothing. The flag is in-memory — a restart resumes+ execution — while the per-index detach tombstone is the durable stop.+ - **`POST /{index}/_ilm/remove`** (#282): ES's own detach verb; goes through+ the same tombstoned detach path as the settings-null route. A literal name+ that is not an index answers 404 rather than writing a tombstone for a+ ghost.++ - **A crafted filename can no longer forge records on the agent progress+ stream** (the first rc.15 known issue below). Every externally-controlled+ string — in-flight paths, the paths and error text interpolated into human+ notes, the terminal line's reason — is stripped of control characters, bidi+ overrides and zero-width characters and bounded in length before it reaches+ any progress surface. Measured on a corpus holding crafted names: a run that+ previously emitted a forged `xerj-done ok=true exit=0 reason=completed`+ 2.2 s ahead of its real terminal line now emits one terminal line and no+ control characters at all. The same sanitisation covers the walker's+ "skipping unreadable entry" warning, which renders a path to the same stderr.+ - **`--progress json` paces the bar like `--progress plain`.** The `bar` field+ bypassed the spacing slot entirely: measured at `--progress-interval 1`, 178+ of 178 ticks carried a bar against 18 bars on the plain surface. It is now a+ string on exactly the ticks that owe a bar and `null` in between — 26 of 320+ ticks on the same corpus, against 16 on plain.+ - **The bar spacing floor is the 15 s the documentation states.** A half-tick+ tolerance made the enforced floor `interval/2` shorter — 12.5 s at the+ shipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phase+ gaps under 15 s (min 10.0 s) before, none after (min 19.3 s).++ ## [1.0.0-rc.15] - 2026-08-10++ ### Known issues in this release++ Both were found by adversarial review *after* their pull requests merged, so+ they are present in these binaries. Fixes are in flight for rc.16.++ - **A crafted filename can forge records on the agent progress stream.** The+ in-flight path is rendered onto the progress surface without control-character+ sanitisation, so a filename containing a newline can inject something that+ looks like a genuine `xerj-progress` or `xerj-done` record — including a false+ `ok=true` completion — into the stream this release tells AI agents to parse.+ Indexing a repository whose filenames you do not control is enough to trigger+ it. Until the fix lands, do not treat that stream as trustworthy when the+ corpus is untrusted.+ - **An outer repository's `.gitignore` judges files inside a nested checkout.**+ The ignore lookup walks every layer with no repository-boundary stop, unlike+ git itself, so a vendored or submoduled repository is filtered by rules that+ have no authority over it. Files you expect to be indexed may be skipped.+ `--no-ignore` bypasses it.++ Also unfixed and worth knowing: `--progress json` does not pace the bar (a JSON+ consumer sees roughly ten times as many bar lines as a plain consumer), and the+ documented "at most one bar per 15 s" floor is really 12.5 s.+++ ### Added++ - **Index lifecycle policies are executed, not just stored**+ ([#199](https://github.com/xerj-org/xerj/issues/199), contributed by+ @Vinz2168). A single internal engine modeled on OpenSearch's ISM state machine+ — named states, ordered actions, ordered transitions — exposed through two+ REST surfaces: native ISM at `_plugins/_ism/*` and the Elasticsearch-shaped+ `_ilm/*`. `spawn_lifecycle_manager()` runs the tick on+ `lifecycle.tick_interval_secs` (default 300s, matching OpenSearch ISM's own+ job interval), and the managed-index cursor survives restart. Retention now+ actually deletes and rolls over instead of being acknowledged and forgotten.+ One documented divergence: `min_age` is measured from when an index entered+ its current state rather than from rollover time, so a policy that rolls over+ *and* has a downstream phase meant to be measured from the rollover will+ advance on a different clock than Elasticsearch uses.++ - **`xerj autoindex` honours `.gitignore` and `.xerjignore`**+ ([#276](https://github.com/xerj-org/xerj/issues/276)). Nested ignore files and+ negation patterns follow git's own precedence, with build-output defaults+ (`node_modules`, `vendor`, `target`, `dist`, `build`, `.venv`, `__pycache__`)+ on top; `--no-ignore` and `--no-default-ignores` turn them off. Pointing xerj+ directly at an ignored path still indexes it — an explicit instruction beats a+ rule the user did not write for this purpose. `--dry-run` reports what was+ skipped and by which rule, because a user whose files did not appear needs to+ know why. Measured on this repository: 274,826 files walked in 1.18s becomes+ 1,385 in 0.25s.++ - **Progress is relayed to an AI agent as a drawn bar.** The stream surface+ carries a rendered bar alongside its machine-readable `key=value` fields, so+ an agent driving `xerj autoindex` on someone's machine can show a real+ progress line rather than going silent for minutes. `--progress json` keeps a+ single parseable stream; `--quiet` still writes nothing on either.++ ### Fixed++ - **Tied scores have one total order — a bounded page is a prefix of the full+ page** ([#191](https://github.com/xerj-org/xerj/issues/191)). A `size:5` page+ and a `size:1000` page disagreed about which tied document came back. The+ memtable's bounded candidates are now ranked by the same page key as the+ segment path, so the two agree. The remaining `constant_score` and post-sort+ re-sort cases are tracked separately in+ [#270](https://github.com/xerj-org/xerj/issues/270).++ - **`date_histogram` no longer aborts the process on a multi-byte `time_zone`**+ ([#272](https://github.com/xerj-org/xerj/issues/272), contributed by+ @Vinz2168). `aggs::parse_time_zone_offset` sliced a `&str` on a byte index+ that could land mid-character, and with `panic=abort` that took the whole node+ down — an unauthenticated request was enough.++ - **The legacy path-parameter form of scroll continuation works again**+ (contributed by @Vinz2168), and **`_cat/*?format=json` plus `_data_stream/**`+ wildcards** are supported (also @Vinz2168) — both are shapes real+ Elasticsearch clients send.+### Changed+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot+ honour, instead of accepting it and analysing differently**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an+ existing index is opened, where a hard failure would brick it), so an+ unresolvable tokenizer silently became `standard` and an unresolvable token+ filter was dropped — the index was created, `GET /_settings` echoed the block+ back, and the field was analysed in a way nobody asked for. Index creation+ now validates the block and answers `400` with the exact constructs it cannot+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused+ (it only ever changed what `GET /_settings` echoed; the registry is built+ once, at create/open, and there is no rebuild path — Elasticsearch also+ refuses analysis updates on an open index).++ **This rejects settings blocks that XERJ used to accept with a `200`, and the+ ES-YAML conformance suite does not cover them — CI being green is not+ evidence that your index still creates.** What is honoured:++ | | honoured (still `200`) |+ |---|---|+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |+ | `analysis.normalizer` | nothing — never built |++ Anything else in those positions is now a `400`. There is one important+ exception to read first: XERJ resolves a declared filter or tokenizer by+ **name**, not by `type`, so a declaration whose *name* is a built-in is still+ honoured whatever its type says. Measured, all still `200`:+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,+ `{"lowercase": {"type": "lowercase"}}`,+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which+ has a regression test pinning it. Naming a built-in analyzer on a field+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.++ What now fails is a declaration under a name that does NOT resolve — which is+ the usual ES habit of naming filters after what they do in *your* index.+ Measured blocks that returned `200` in rc.14 and return `400` from this+ release:++ - a `keyword` field's `normalizer` (any);+ - a `char_filter` (any, including `html_strip` and `mapping`), declared+ either under `analysis.char_filter` or as an analyzer's `char_filter`;+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,+ which never lowercased anything, because the analyzer referencing+ `my_lowercase` found no such filter and dropped it;+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,+ `letter`, `lowercase` or `simple_pattern`;+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,+ `fingerprint`, or a language other than `english`;+ - a `synonym` filter whose `synonyms` is a bare string rather than an array+ (it built a filter with zero rules).++ **Upgrading:** there is no opt-out flag. Either remove the analysis block and+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only+ the constructs in the table. The `400` names every offending construct in one+ response, so one round-trip is enough to see the whole list. **Two paths are+ not covered and still accept these blocks silently: an index template's+ `template.settings.analysis` is dropped wholesale before the gate ever sees+ it, and option-level divergence inside a supported type (`synonyms_path`,+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**+- **`autoindex` parses each PDF once per fresh run instead of twice.** Phase Aalready paid a *complete* parse for every PDF — `extract::extract` routes`Family::Pdf` straight to the isolated worker and drops the sampling limit,@@@ -149,67 -210,83 +279,145 @@@### Fixed+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and+ every write path enforces it**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only+ warn-log a compile failure, so a pipeline that could never run was+ acknowledged and echoed back by `GET`. It now compiles first, with three+ outcomes: compiled → stored and acknowledged; a processor config we refuse →+ `400`; a processor or option this build cannot honour → accepted (as+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses+ rather than storing an untransformed document.++ Defects this surfaced, each of which answered `200`/`201` and did something+ smaller than asked: **every** ES `rename` processor failed to compile (the+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and+ renamed nothing; ES `append` was executed as `set`, replacing the field+ instead of extending it, and disagreeing with `_simulate`, which implemented+ a real append; `set.override` defaulted to `false` where Elasticsearch+ defaults it to `true`; an unknown `pii_redaction` type built an empty pattern+ set and redacted nothing; an unknown `grok` pattern name silently became the+ catch-all; an unknown `convert` target left the field unconverted per+ document; array-typed keys given as a bare string were dropped and the stage+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the+ definition while leaving the compiled pipeline live under the same name.+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`+ rather than being refused.++ Two more of the same, at the definition's own edges: a **pipeline-level**+ `on_failure` block (the top-level body key, not the processor-level one) was+ accepted, echoed by `GET`, and never run — the pipeline is now recorded+ unrunnable, exactly as its processor-level twin already was; and `set` with+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:+ missing value`, which told the caller their valid ES pipeline was malformed.++ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`+ previously read the instruction and dropped it — a redaction pipeline that+ the single-document endpoint refused was ignored in bulk, and the document+ was stored verbatim under `errors: false`. All four now run the pipeline, and+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the+ name before touching a single document, so a bad pipeline fails the request+ instead of half of it.++ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`+ copied at most 10,000 documents and swallowed every per-document write error+ before answering `{"acknowledged": true}` (now full keyset paging, and any+ failed write — or a failed source flush — fails the request); and a failed+ `_bulk` item no longer reports `"result": "deleted"` alongside its own+ `status: 400`, which claimed a deletion that never happened.++ **Known gaps, stated rather than closed:** `_simulate` interprets the stored+ ES body directly and covers processors the compiled path does not, so a+ pipeline recorded as unrunnable can still show a transformation under+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`- returns the ES body with no runnability signal, so an operator only learns at- write time; a write addressed to an **alias** does not pick up the backing- index's `default_pipeline`; and `_update_by_query` reports a `drop`++ returns the stored definition (the ES body for an unrunnable pipeline, the++ compiled shape otherwise) with no runnability signal, so an operator only++ learns at write time; a write addressed to an **alias** does not pick up the++ backing index's `default_pipeline`; and `_update_by_query` reports a `drop`+ processor as a per-document failure, because it updates in place and has no+ delete path.++ - **An index whose metadata will not parse is refused instead of silently+ reopened with an empty mapping**+ ([#202](https://github.com/xerj-org/xerj/issues/202)). `Index::open` treated+ "this file is not there" and "this file does not parse" as the same thing:+ `load_schema`/`load_settings` mapped ENOENT, EACCES, EIO and malformed JSON+ alike to one anonymous error, and the caller answered all of them with a+ fresh dynamic mapping. Measured on a field deliberately mapped `keyword`:+ after `schema.json` was truncated to half its bytes the index still opened+ `Ok` with `field_count = 0`, the field came back `None`, and one further+ document re-inferred it as `long` — a mapping silently replaced by a+ different one, with nothing in any response saying so. Absent stays absent+ (indices predating create-time schema persistence legitimately have no+ `schema.json`); present-but-unparseable now fails the open with an error+ naming the file. `es_mapping.json` — the full-fidelity mapping behind+ `GET /_mapping` — was logged-and-ignored on a parse failure and now fails the+ index too, on the boot scan, on snapshot restore and on a retry.++ **Visible on upgrade:** a node whose data dir already holds a corrupt sidecar+ now boots **red** with that index unserved, where it previously came up green+ with an empty mapping. It joins the failed set+ [#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every+ surface that set feeds already reports it — measured on a node holding two+ user indices, `victim` with a truncated `schema.json`:++ ```+ GET /_cluster/health -> red, unassigned_primary_shards: 1+ GET /_cluster/health?level=indices+ -> victim red, unassigned_info.details = the+ absolute path of schema.json and the parse error+ GET /_cat/indices -> "red open victim <uuid> 1 0 0 0 0b 0b 0b"+ GET /_cluster/indices/failed -> the reason, plus the retry and delete calls+ PUT /victim, POST /victim/_doc, POST /victim/_search+ -> 503 no_shard_available_action_exception+ carrying that same reason+ GET /health/ready -> 200 "ready (degraded): 1 of 2 indices+ serving, 1 failed to open; see GET+ /_cluster/indices/failed"+ POST /healthy/_search -> 200+ ```++ The readiness counts above are the two-index test router's. A running node+ also holds its internal `.xerj_*` indices, so the same condition on the shipped+ binary reads `ready (degraded): 15 of 16 indices serving, 1 failed to open` —+ the 200 and its meaning are unchanged, only the totals move with what else the+ node holds.++ Recovery is the three doors #206 added. Two were measured end to end on the+ shipped binary: repair the file and+ `POST /_cluster/indices/failed/{index}/_retry` (503 while it is still torn,+ `200 {"reopened": true}` once it is not, health back to green), and+ `DELETE /{index}` (200, directory removed, health back to green). The third —+ restore the index from a snapshot — is the path this change touches (a+ successful restore now clears the recorded failure, which it has to or health+ would stay red after the repair worked); it is covered by an engine-level+ test rather than measured over HTTP. The node stays in kubelet rotation as+ long as any index is still serving.++ - **Concurrent sidecar writes can no longer manufacture the torn file the reader+ now refuses.** `write_file_atomic` staged every write in+ `path.with_extension("tmp")` — one shared name for all writers of that file —+ so two concurrent writers of one sidecar (`PUT /{index}/_settings` racing+ another settings update, two API-key mints for `api_keys.json`) both opened it+ `O_TRUNC`, interleaved their bytes, and each renamed it into place. The rename+ is atomic; the content being renamed was not one writer's. Measured with the+ old shared name, 4 threads × two different-length settings bodies × 200+ rounds, over four runs: **86–294 of 800 writes failed with ENOENT** (the loser+ renaming a file the winner had already moved) and, with those errors swallowed+ as the callers did, **1–16 of 200 rounds left an unparseable `settings.json`**.+ The counts are race-dependent and vary with machine load; what does not vary is+ that both are non-zero on every run with the shared name and **0 and 0** on+ every run with a unique staging name. Each write now stages+ in its own sibling file (`<file>.tmp.<pid>.<seq>`) in the target's directory+ and removes its own debris on failure — which for `api_keys.json` is key+ material. `update_settings` also writes `settings.json` through+ `write_file_atomic` rather than a plain `fs::write`; it was the last+ non-atomic sidecar writer left.+- **`index` on the mapping is now honoured instead of silently ignored**([#204](https://github.com/xerj-org/xerj/issues/204)). `"index": false` wasaccepted by `PUT /{index}`, echoed back verbatim by `GET /{index}/_mapping`,diff --cc engine/crates/xerj-api/src/es_compat.rsindex 949efe667,c451efb85..18dd66967--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@@ -22858,112 -22415,25 +23077,121 @@@ pub async fn put_ingest_pipeline}_ => proc_config.clone(),};- Some(json!({"type": xerj_type, "config": adapted_config}))- }).collect();- json!({- "description": body.get("description").and_then(Value::as_str).unwrap_or(""),- "stages": stages- })- } else {- body.clone()- };- // `Some(body)` keeps this surface's long-standing behaviour: a definition- // that will not compile is still stored and still readable through- // `GET /_ingest/pipeline/{id}` (the engine logs the compile error and- // drops the executable form, so `?pipeline={id}` behaves identically- // before and after a restart). An `Err` means it did not reach disk —- // answering `acknowledged` there would promise a pipeline that is gone at- // the next boot, which is the whole of issue #203.- match state.engine.put_pipeline(&id, xerj_config, Some(body)) {- Ok(_) => Json(json!({ "acknowledged": true })).into_response(),+ stages.push(json!({"type": xerj_type, "config": adapted_config}));+ }+ json!({+ "description": body.get("description").and_then(Value::as_str).unwrap_or(""),+ "stages": stages+ })+ } else {+ body.clone()+ };+ // Elasticsearch pipeline-level (top-level body) keys that decide what+ // happens when a processor fails. The compiled pipeline has no equivalent:+ // `PipelineConfig` carries a single `on_error` policy and no recovery+ // chain, so a top-level `on_failure` was read by nothing.+ //+ // This is the SAME defect the repair fixed one level down+ // (`UNHONOURED_PROCESSOR_KEYS` in `xerj-wasm/src/pipeline.rs`): refusing a+ // processor-level `on_failure` while silently dropping the identical key at+ // pipeline level would have left the endpoint inconsistent with itself.+ // `description`, `version`, `_meta` and `deprecated` are NOT here — they+ // are identification metadata that changes no document, the same line drawn+ // for processor-level `tag`/`description`.+ const UNHONOURED_PIPELINE_KEYS: &[(&str, &str)] = &[(+ "on_failure",+ "the pipeline-level recovery processors are not run; the compiled pipeline's \+ `on_error` policy applies instead",+ )];+ let unhonoured_top_level = UNHONOURED_PIPELINE_KEYS+ .iter()+ .find(|(key, _)| body.get(*key).is_some())+ .copied();+ // Three outcomes, and the difference between them is the whole point of+ // issue #204:+ //- // compiled → store the ES body verbatim (that is what ES- // echoes from GET, and what `_simulate` walks) and- // acknowledge.++ // compiled → store the translated config (the shape the boot++ // replay can recompile, and one `_simulate` walks++ // just as well as the ES body) and acknowledge.+ // config we refuse → 400. The caller misconfigured a processor xerj+ // does implement; storing it would mean answering+ // "acknowledged" for redaction that never runs.+ // stage we don't have → accept, exactly as ES does — but record WHY it+ // has no compiled form, so every ingest path+ // refuses loudly instead of indexing the document+ // untransformed. This is the "degrade loudly"+ // half of the issue: the capability gap is xerj's,+ // not the caller's, and it surfaces at the point+ // where a document would otherwise be silently+ // mis-ingested.- match state.engine.create_pipeline(&id, xerj_config) {- Ok(()) if unhonoured_top_level.is_some() => {++ //++ // Whichever way it goes, nothing is acknowledged before it reaches++ // `cluster_state.json` (issue #203): `put_pipeline` (with `None`, so a++ // refused definition stores nothing) and `register_unrunnable_pipeline`++ // both flush before returning, and an `Err` from either means the write++ // did not reach disk — answering `acknowledged` there would promise a++ // pipeline that is gone at the next boot.++ match state.engine.put_pipeline(&id, xerj_config, None) {++ Ok(None) if unhonoured_top_level.is_some() => {+ // Every stage compiled, but a top-level key we cannot act on came+ // with it. `register_unrunnable_pipeline` drops the compiled form- // `create_pipeline` just installed, so nothing runs half-honoured.++ // `put_pipeline` just installed, so nothing runs half-honoured.+ let (option, reason) = unhonoured_top_level.expect("guarded by the match arm");- state.engine.register_unrunnable_pipeline(++ match state.engine.register_unrunnable_pipeline(+ &id,+ body,+ format!(+ "pipeline-level [{option}] is not supported by this xerj build \+ ({reason}); no document can be ingested through pipeline [{id}]"+ ),- );- Json(json!({ "acknowledged": true })).into_response()- }- Ok(()) => {- // `create_pipeline` stores the *translated* config; put the ES- // body back so GET and _simulate see what was actually submitted.- state.engine.pipelines.insert(id.clone(), body);- Json(json!({ "acknowledged": true })).into_response()++ ) {++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),++ }+ }- Err(xerj_wasm::WasmError::UnsupportedStage { stage }) => {- state.engine.register_unrunnable_pipeline(++ Ok(None) => Json(json!({ "acknowledged": true })).into_response(),++ Ok(Some(xerj_wasm::WasmError::UnsupportedStage { stage })) => {++ match state.engine.register_unrunnable_pipeline(+ &id,+ body,+ format!(+ "processor [{stage}] is not implemented by this xerj build; \+ no document can be ingested through pipeline [{id}]"+ ),- );- Json(json!({ "acknowledged": true })).into_response()++ ) {++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),++ }+ }+ // Same shape one level down: the processor exists, an option on it does+ // not. ES accepts the definition, so we do too — and record why it+ // cannot run, so no document is written as if the option had been+ // honoured (issue #204 applied to this endpoint's own new contract).- Err(xerj_wasm::WasmError::UnsupportedOption {++ Ok(Some(xerj_wasm::WasmError::UnsupportedOption {+ stage,+ option,+ reason,- }) => {- state.engine.register_unrunnable_pipeline(++ })) => {++ match state.engine.register_unrunnable_pipeline(+ &id,+ body,+ format!(+ "processor [{stage}] option [{option}] is not supported by this xerj \+ build ({reason}); no document can be ingested through pipeline [{id}]"+ ),- );- Json(json!({ "acknowledged": true })).into_response()++ ) {++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),++ }+ }- Err(e) => {++ Ok(Some(e)) => {+ tracing::warn!(pipeline = %id, error = %e, "rejected ingest pipeline definition");+ let err = xerj_common::XerjError::invalid_mapping(format!(+ "pipeline [{id}] cannot be created: {e}"+ ));+ ApiError::new(err).into_response()+ }+ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),}}@@@ -22994,18 -22464,13 +23222,20 @@@ pub async fn delete_ingest_pipelineState(state): State<AppState>,Path(id): Path<String>,) -> impl IntoResponse {- if state.engine.pipelines.remove(&id).is_some() {- // Issue #204: only the definition used to be removed. The COMPILED- // pipeline stayed in `transform_pipelines`, so a delete that answered- // `acknowledged: true` left the stages running for any caller that- // reached them by name (`POST /v1/ingest?pipeline=`) — deletion- // acknowledged, deletion not performed.- state.engine.transform_pipelines.remove(&id);- state.engine.unrunnable_pipelines.remove(&id);- Json(json!({ "acknowledged": true })).into_response()- } else {- let e = xerj_common::XerjError::index_not_found(format!("pipeline [{id}] is missing"));- ApiError::new(e).into_response()++ // Issue #204: only the definition used to be removed here. The COMPILED++ // pipeline stayed in `transform_pipelines`, so a delete that answered++ // `acknowledged: true` left the stages running for any caller that++ // reached them by name (`POST /v1/ingest?pipeline=`) — deletion++ // acknowledged, deletion not performed. `Engine::delete_pipeline` removes++ // the definition, the compiled form and any recorded unrunnable reason++ // together, and persists the removal (issue #203).+ match state.engine.delete_pipeline(&id) {+ Ok(true) => Json(json!({ "acknowledged": true })).into_response(),+ Ok(false) => {+ let e = xerj_common::XerjError::index_not_found(format!("pipeline [{id}] is missing"));+ ApiError::new(e).into_response()+ }+ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),}}diff --cc engine/crates/xerj-engine/src/engine.rsindex ee6778ab9,08b645e04..9a348f270--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@@ -421,19 -472,17 +472,25 @@@ pub struct Enginepub templates: Arc<DashMap<String, IndexTemplate>>,/// scroll_id → ScrollContextpub scrolls: Arc<DashMap<String, ScrollContext>>,- /// pipeline_id → pipeline definition JSON+ /// pipeline_id → pipeline definition JSON. Persisted in+ /// `<data_dir>/cluster_state.json`; mutate only through+ /// [`Engine::put_pipeline`] / [`Engine::delete_pipeline`], or the change+ /// will not survive a restart (issue #203).pub pipelines: Arc<DashMap<String, Value>>,+ /// pipeline_id → why the stored definition has no compiled counterpart.+ ///+ /// Issue #204. A definition that carries a stage type this build does not+ /// implement is still stored (Elasticsearch accepts it, and+ /// `_ingest/pipeline/{id}/_simulate` still walks it), but nothing may be+ /// ingested through it: `process_through_pipeline` refuses with the reason+ /// recorded here instead of quietly indexing the untransformed document.+ pub unrunnable_pipelines: Arc<DashMap<String, String>>,/// index_name → open/closed state (true = closed)pub closed_indices: Arc<DashMap<String, bool>>,- /// data stream name → DataStream+ /// data stream name → DataStream. Persisted in+ /// `<data_dir>/cluster_state.json`; mutate only through the+ /// `*_data_stream` methods, or the change will not survive a restart+ /// (issue #203).pub data_streams: Arc<DashMap<String, DataStream>>,/// ILM policy name → raw policy JSON, kept verbatim for `GET _ilm/policy`/// round-trip fidelity. The REAL execution engine reads `ism_policies`@@@ -1635,6 -1901,583 +1910,620 @@@ impl Engineself.record_index_created_at(name);}+ // ── Cluster metadata persistence (issue #203) ────────────────────────────+ //+ // Index templates, legacy (v1) templates, component templates, ingest+ // pipelines, data streams and ILM policies used to live only in the+ // in-memory maps above: `PUT /_index_template/logs` answered `200+ // {"acknowledged": true}` and the template was gone after the next boot,+ // with no error anywhere and the next matching index silently created+ // unshaped. They are now snapshotted, as one consistent document, into+ // `<data_dir>/cluster_state.json` on every mutation and reloaded in+ // `Engine::new`.+ //+ // Durability contract: a full-file rewrite through+ // `index::write_file_atomic` — write to `cluster_state.tmp`, `fsync` it,+ // `rename` over the target, then `fsync` the parent directory so the+ // rename itself survives power loss. The rename is the commit point, so+ // a `kill -9` at any instant leaves either the previous complete+ // document or the new one, never a half-written mix. The same shape peer+ // engines use for their manifests (cf. `fjall/src/file.rs:17`+ // `fsync_directory`, which fjall calls after every atomic rewrite;+ // `sled/src/metadata_store.rs:696` discards leftover `*.tmp` files from+ // an interrupted rewrite on the next boot, which `load_cluster_state`+ // mirrors below).+ //+ // One file rather than six keeps the maps mutually consistent — a data+ // stream and the template that shaped it can never be restored from+ // different generations.++ /// Path of the persisted cluster-metadata document+ /// (`<data_dir>/cluster_state.json`).+ fn cluster_state_path(&self) -> PathBuf {+ self.data_dir.join("cluster_state.json")+ }++ /// `Err` when boot found `cluster_state.json` and could not load it — see+ /// `cluster_state_loaded` for why that has to latch.+ ///+ /// Every management mutation calls this, not only the ones that reach+ /// `flush_cluster_state` with something to write. A `DELETE` consults the+ /// in-memory map first, and while the load has failed that map is empty,+ /// so an unguarded delete answers `404 not found` about an object that is+ /// sitting in the document on disk — the same lie in a different shape.+ /// A 500 that names the file is the only honest answer while the node+ /// cannot see the operator's configuration.+ fn ensure_cluster_state_writable(&self) -> Result<()> {+ if self+ .cluster_state_loaded+ .load(std::sync::atomic::Ordering::Acquire)+ {+ return Ok(());+ }+ let path = self.cluster_state_path();+ error!(+ path = %path.display(),+ "refusing a cluster-metadata write: this node could not load \+ cluster_state.json at boot, so its management state is not the \+ operator's. Recover or move the file aside, then restart."+ );+ Err(EngineError::Common(xerj_common::XerjError::storage(+ format!(+ "refusing to write {} — it could not be loaded at boot, so \+ writing now would destroy cluster configuration that is still \+ on disk; recover or move it aside and restart xerj",+ path.display()+ ),+ )))+ }++ /// Snapshot every persisted management map into one document.+ fn cluster_state_snapshot(&self) -> PersistedClusterState {+ fn dump<V: Clone>(map: &DashMap<String, V>) -> std::collections::BTreeMap<String, V> {+ map.iter()+ .map(|e| (e.key().clone(), e.value().clone()))+ .collect()+ }+ PersistedClusterState {+ version: CLUSTER_STATE_VERSION,+ index_templates: dump(&self.templates),+ legacy_templates: dump(&self.legacy_templates),+ component_templates: dump(&self.component_templates),+ pipelines: dump(&self.pipelines),+ data_streams: dump(&self.data_streams),+ ilm_policies: dump(&self.ilm_policies),+ }+ }++ /// Durably write the current cluster metadata.+ ///+ /// Unlike `flush_aliases`, a failure here is **returned**, not swallowed:+ /// every caller is an API write that would otherwise answer+ /// `{"acknowledged": true}` for a change it cannot keep. Callers roll the+ /// in-memory change back and surface a 500, so the operator finds out at+ /// the moment of the write instead of at the next restart.+ ///+ /// Refuses outright when boot could not load the existing document — see+ /// `cluster_state_loaded`. The snapshot below is taken from the live maps,+ /// which in that state hold only what this process was told after boot, so+ /// writing it would rename an empty-ish document over configuration that is+ /// still intact on disk.+ fn flush_cluster_state(&self) -> Result<()> {+ self.ensure_cluster_state_writable()?;+ // One writer at a time — see `cluster_state_write`. Snapshotting+ // inside the lock also means the bytes that land are never older+ // than a rewrite that has already returned.+ let _writing = self.cluster_state_write.lock();+ let snapshot = self.cluster_state_snapshot();+ let bytes = serde_json::to_vec_pretty(&snapshot)?;+ crate::index::write_file_atomic(&self.cluster_state_path(), &bytes)?;+ Ok(())+ }++ /// Highest generation `N` for which a backing index `.ds-<stream>-<N>` is+ /// actually open in this data dir; `0` when there is none.+ ///+ /// Read off the open indices rather than the stream document, because the+ /// point is to catch the case where the two disagree. Suffix parsing keeps+ /// neighbouring streams apart on its own: for stream `a`, `.ds-a-b-000001`+ /// leaves `b-000001`, which is not a number.+ fn highest_backing_generation(&self, stream: &str) -> u64 {+ let prefix = format!(".ds-{stream}-");+ self.indices+ .iter()+ .filter_map(|e| {+ e.key()+ .strip_prefix(&prefix)+ .and_then(|gen| gen.parse::<u64>().ok())+ })+ .max()+ .unwrap_or(0)+ }++ /// Load `<data_dir>/cluster_state.json` back into the management maps.+ ///+ /// A missing file is normal (fresh node, or a data dir written by a build+ /// that predates this). A file that is present but does not load is loud+ /// *and* latching: it means the operator's templates, pipelines and data+ /// streams are not coming back, which is exactly the silence this whole+ /// mechanism exists to remove, so `cluster_state_loaded` goes false and+ /// every subsequent management write is refused rather than allowed to+ /// rename a snapshot of empty maps over the document on disk.+ fn load_cluster_state(&self) {+ // An interrupted rewrite can leave a partial `cluster_state.tmp`+ // behind. It is never read (the rename is the commit point), but+ // leaving it litters the data dir and invites a hand-edit that+ // "restores" a torn document — sled sweeps the same leftovers on+ // boot (`sled/src/metadata_store.rs:696`).+ let tmp = self.cluster_state_path().with_extension("tmp");+ if tmp.exists() {+ warn!(+ path = %tmp.display(),+ "discarding an incomplete cluster_state rewrite left by an unclean shutdown"+ );+ let _ = std::fs::remove_file(&tmp);+ }++ let path = self.cluster_state_path();+ let bytes = match std::fs::read(&path) {+ Ok(b) => b,+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => return,+ Err(e) => {+ // The file is *there* and its bytes are almost certainly fine+ // — `EACCES` after a uid change on a container volume, a+ // backup tool's chmod, `EIO`, `EMFILE` at boot. Do not copy it+ // aside (we cannot read it) and do not touch it: mark the load+ // failed so no later write can rename a snapshot of empty maps+ // over configuration that is still perfectly recoverable.+ self.cluster_state_loaded+ .store(false, std::sync::atomic::Ordering::Release);+ error!(+ path = %path.display(), error = %e,+ "could not READ cluster_state.json — index templates, ingest \+ pipelines, data streams and ILM policies are NOT restored, \+ and management writes will be refused until a boot loads it \+ (the file itself is left untouched)"+ );+ return;+ }+ };+ let state: PersistedClusterState = match serde_json::from_slice(&bytes) {+ Ok(s) => s,+ Err(e) => {+ // Same position as the read error above: the load failed, so+ // the live maps are not the operator's configuration and+ // nothing may be written over the document on disk. Refusing+ // rather than salvaging-then-overwriting also closes the+ // second-corruption hole in the `!salvage.exists()` guard+ // below — with an overwrite still allowed, a second damaged+ // document would be destroyed *and* not copied, because the+ // copy from the first one is already there.+ self.cluster_state_loaded+ .store(false, std::sync::atomic::Ordering::Release);+ error!(+ path = %path.display(), error = %e,+ "cluster_state.json is corrupt — index templates, ingest \+ pipelines, data streams and ILM policies are NOT restored, \+ and management writes will be refused until a boot loads it"+ );+ // Keep a copy anyway. The original is now safe from xerj, but+ // un-wedging the node means moving it aside by hand, and+ // hand-recovering a template out of a damaged document is the+ // operator's last option — leave them something to recover+ // from that survives that step. The original is deliberately+ // left in place so every boot keeps logging the error above+ // until someone deals with it, and an existing copy is never+ // clobbered: the first corruption is the informative one.+ let salvage = path.with_extension("corrupt.json");+ if !salvage.exists() {+ match std::fs::write(&salvage, &bytes) {+ Ok(()) => warn!(+ path = %salvage.display(),+ "kept a copy of the corrupt cluster_state for recovery"+ ),+ Err(e) => warn!(+ path = %salvage.display(), error = %e,+ "could not preserve a copy of the corrupt cluster_state"+ ),+ }+ }+ return;+ }+ };+ if state.version > CLUSTER_STATE_VERSION {+ warn!(+ found = state.version,+ supported = CLUSTER_STATE_VERSION,+ "cluster_state.json was written by a newer xerj; loading what \+ this build understands (downgrade may drop metadata)"+ );+ }++ for (name, tmpl) in state.index_templates {+ self.templates.insert(name, tmpl);+ }+ for (name, body) in state.legacy_templates {+ self.legacy_templates.insert(name, body);+ }+ for (name, body) in state.component_templates {+ self.component_templates.insert(name, body);+ }+ for (name, body) in state.ilm_policies {+ self.ilm_policies.insert(name, body);+ }++ // Data streams: restore the record, reconcile it against what is+ // actually on disk, then say plainly when a backing index it names is+ // not there. Reporting a stream whose data is gone as+ // `"status": "GREEN"` without a word in the log is the same quiet lie+ // this issue is about.+ let mut reconciled_a_stream = false;+ for (name, mut ds) in state.data_streams {+ // `rollover_data_stream` creates the new backing index before it+ // records the new generation, so `kill -9` in that window leaves+ // `.ds-<name>-00000N` on disk while the document still says N-1.+ // Restoring N-1 verbatim would wedge the stream permanently: the+ // next rollover computes the same name, and `create_index`+ // refuses it as `index_already_exists` — forever. Adopt the+ // highest generation that actually exists instead, which is also+ // what stops a second rollover writing into an index that already+ // holds the first one's documents.+ let on_disk = self.highest_backing_generation(&name);+ if on_disk > ds.generation {+ let adopted = format!(".ds-{name}-{on_disk:06}");+ warn!(+ data_stream = name.as_str(),+ recorded = ds.generation,+ found = on_disk,+ adopted = adopted.as_str(),+ "adopting a backing index left behind by a rollover that \+ was interrupted before its generation reached disk"+ );+ if !ds.backing_indices.contains(&adopted) {+ ds.backing_indices.push(adopted);+ }+ ds.generation = on_disk;+ reconciled_a_stream = true;+ }++ let missing: Vec<&str> = ds+ .backing_indices+ .iter()+ .filter(|b| !self.indices.contains_key(b.as_str()))+ .map(|b| b.as_str())+ .collect();+ if !missing.is_empty() {+ warn!(+ data_stream = name.as_str(),+ missing = missing.join(","),+ "restored data stream references backing indices that are \+ not present in the data dir"+ );+ }+ self.data_streams.insert(name, ds);+ }++ // Pipelines carry behaviour, not just a document to hand back from+ // `GET /_ingest/pipeline`: the executable form has to be rebuilt or+ // `?pipeline=x` would be accepted after a restart and quietly do+ // nothing. Keep the stored document either way so GET round-trips+ // exactly as it did before the restart.+ let mut pipelines_restored = 0usize;+ for (name, config) in state.pipelines {+ self.pipelines.insert(name.clone(), config.clone());+ match self.compile_pipeline(&name, config) {+ Ok(()) => pipelines_restored += 1,- Err(e) => error!(- pipeline = name.as_str(), error = %e,- "persisted ingest pipeline could not be recompiled — it is \- still visible to GET /_ingest/pipeline but will NOT \- transform anything; re-PUT the definition"- ),++ Err(e) => {++ // Issue #204: record WHY there is no executable form, so++ // every ingest path refuses loudly with the reason++ // instead of reporting a pipeline GET plainly shows as++ // "not found" — the same contract as a fresh PUT.++ self.unrunnable_pipelines.insert(++ name.clone(),++ format!(++ "the persisted definition could not be recompiled \++ at startup ({e}); no document can be ingested \++ through pipeline [{name}] — re-PUT the definition"++ ),++ );++ error!(++ pipeline = name.as_str(), error = %e,++ "persisted ingest pipeline could not be recompiled — it is \++ still visible to GET /_ingest/pipeline but will NOT \++ transform anything; re-PUT the definition"++ )++ }+ }+ }++ // Make an adopted generation durable straight away, so the recovery+ // happens once rather than on every boot from here on.+ if reconciled_a_stream {+ if let Err(e) = self.flush_cluster_state() {+ warn!(+ error = %e,+ "could not persist the reconciled data-stream generation; \+ it will be recomputed from disk on the next boot"+ );+ }+ }++ let n = self.templates.len()+ + self.legacy_templates.len()+ + self.component_templates.len()+ + self.pipelines.len()+ + self.data_streams.len()+ + self.ilm_policies.len();+ if n > 0 {+ info!(+ index_templates = self.templates.len(),+ legacy_templates = self.legacy_templates.len(),+ component_templates = self.component_templates.len(),+ pipelines = pipelines_restored,+ data_streams = self.data_streams.len(),+ ilm_policies = self.ilm_policies.len(),+ "restored persisted cluster metadata"+ );+ }+ }++ /// Name every `.ds-*` index on disk that no restored data stream claims.+ ///+ /// This does **not** reconcile anything — nothing is deleted, adopted or+ /// repaired, because an orphan may hold the only copy of somebody's data.+ /// It exists so the state is not *silent*: an orphaned backing index is+ /// unreachable through the data-stream API (`GET` and `DELETE` on the+ /// stream answer 404, while `PUT /_data_stream/<name>` answers+ /// `409 resource_already_exists_exception` naming the orphan), so without+ /// a line in the boot log the operator has no way to learn the name they+ /// have to pass to `DELETE /<backing-index>` to clear it.+ ///+ /// Expected sources: a data dir written by a build that predates+ /// `cluster_state.json` (data streams were not persisted at all then), or+ /// a `DELETE /_data_stream` interrupted by a build that recorded the+ /// removal before destroying the backing indices. Skipped entirely when+ /// the cluster state did not load — then every stream is unknown and+ /// every backing index would look orphaned.+ fn warn_orphaned_backing_indices(&self) {+ if !self+ .cluster_state_loaded+ .load(std::sync::atomic::Ordering::Acquire)+ {+ return;+ }+ // Snapshot both maps before comparing them: nothing here may hold a+ // shard guard on one map while taking one on the other.+ let backing: Vec<String> = self+ .indices+ .iter()+ .map(|e| e.key().clone())+ .filter(|n| n.starts_with(".ds-"))+ .collect();+ if backing.is_empty() {+ return;+ }+ let streams: Vec<(String, Vec<String>)> = self+ .data_streams+ .iter()+ .map(|s| (s.key().clone(), s.value().backing_indices.clone()))+ .collect();+ let claimed = |index: &str| {+ streams.iter().any(|(stream, indices)| {+ indices.iter().any(|b| b == index)+ || index+ .strip_prefix(&format!(".ds-{stream}-"))+ .is_some_and(|gen| gen.parse::<u64>().is_ok())+ })+ };+ let mut orphans: Vec<String> = backing.into_iter().filter(|n| !claimed(n)).collect();+ if orphans.is_empty() {+ return;+ }+ orphans.sort();+ warn!(+ count = orphans.len(),+ indices = orphans.join(","),+ "backing indices on disk belong to no known data stream — they are \+ unreachable through the data-stream API and are NOT deleted \+ automatically; remove each with DELETE /<index> once you have \+ confirmed the data is not wanted"+ );+ }++ /// Insert into a persisted map and durably record the result, rolling the+ /// in-memory change back if the write fails so a 500 never leaves a+ /// change that only this process can see.+ fn persisted_insert<V: Clone>(+ &self,+ map: &DashMap<String, V>,+ name: String,+ value: V,+ ) -> Result<()> {+ self.ensure_cluster_state_writable()?;+ let previous = map.insert(name.clone(), value);+ if let Err(e) = self.flush_cluster_state() {+ match previous {+ Some(old) => map.insert(name, old),+ None => map.remove(&name).map(|(_, v)| v),+ };+ return Err(e);+ }+ Ok(())+ }++ /// Remove from a persisted map and durably record the result. Returns+ /// `false` when there was nothing to remove (the caller answers 404);+ /// an `Err` means the removal did not reach disk and was undone.+ fn persisted_remove<V: Clone>(&self, map: &DashMap<String, V>, name: &str) -> Result<bool> {+ // Before the map lookup: an empty map after a failed load would+ // otherwise turn "I cannot see your configuration" into "it does not+ // exist" — see `ensure_cluster_state_writable`.+ self.ensure_cluster_state_writable()?;+ let Some((key, removed)) = map.remove(name) else {+ return Ok(false);+ };+ if let Err(e) = self.flush_cluster_state() {+ map.insert(key, removed);+ return Err(e);+ }+ Ok(true)+ }++ /// Create or replace a v2 index template (`PUT /_index_template/{name}`).+ pub fn put_index_template(&self, name: String, template: IndexTemplate) -> Result<()> {+ self.persisted_insert(&self.templates, name, template)+ }++ /// Delete a v2 index template; `false` when it did not exist.+ pub fn delete_index_template(&self, name: &str) -> Result<bool> {+ self.persisted_remove(&self.templates, name)+ }++ /// Create or replace a legacy v1 template (`PUT /_template/{name}`).+ pub fn put_legacy_template(&self, name: String, body: Value) -> Result<()> {+ self.persisted_insert(&self.legacy_templates, name, body)+ }++ /// Delete a legacy v1 template; `false` when it did not exist.+ pub fn delete_legacy_template(&self, name: &str) -> Result<bool> {+ self.persisted_remove(&self.legacy_templates, name)+ }++ /// Create or replace a component template.+ pub fn put_component_template(&self, name: String, body: Value) -> Result<()> {+ self.persisted_insert(&self.component_templates, name, body)+ }++ /// Delete a component template; `false` when it did not exist.+ pub fn delete_component_template(&self, name: &str) -> Result<bool> {+ self.persisted_remove(&self.component_templates, name)+ }++ /// Create or replace an ILM policy.+ pub fn put_ilm_policy(&self, name: String, body: Value) -> Result<()> {+ self.persisted_insert(&self.ilm_policies, name, body)+ }++ /// Delete an ILM policy; `false` when it did not exist.+ pub fn delete_ilm_policy(&self, name: &str) -> Result<bool> {+ self.persisted_remove(&self.ilm_policies, name)+ }++ /// Durably store an ingest pipeline definition and (re)build its+ /// executable form. This is the only way a user-visible pipeline should+ /// ever be created — `compile_pipeline` alone does not reach disk.+ ///+ /// `config` is the xerj-shaped pipeline config to compile, and is what+ /// gets stored (and handed back by `GET /_ingest/pipeline/{id}`) when it+ /// compiles — the shape this endpoint has always returned.+ ///+ /// `keep_if_uncompilable` says what to do when it does *not* compile, and+ /// differs by surface:+ ///+ /// * `Some(doc)` — the ES-compat surface. It has always answered+ /// `acknowledged` and left the definition readable, so `doc` (the+ /// ES-shaped body as received) is stored and the compile error is+ /// returned as `Ok(Some(err))` for the caller to log.+ /// * `None` — the xerj-native surface, which rejects a definition it+ /// cannot compile. Nothing is stored, nothing is flushed, and any+ /// pipeline already registered under this id is left untouched.+ ///+ /// Either way a compile failure drops the *executable* form under this+ /// id. Leaving the old one behind would mean `?pipeline={id}` kept+ /// running a definition `GET` no longer shows — and stopped running it at+ /// the next restart. A definition must behave the same before and after a+ /// reboot; that is the whole point of persisting it.+ ///+ /// `Err` means the definition did not reach disk. The in-memory change is+ /// rolled back first, so the caller can report the write as failed+ /// without leaving behind a pipeline only this process can see.+ pub fn put_pipeline(+ &self,+ id: &str,+ config: Value,+ keep_if_uncompilable: Option<Value>,+ ) -> Result<Option<xerj_wasm::WasmError>> {+ // Up front, so a refused write never disturbs the compiled form of a+ // pipeline that is still running.+ self.ensure_cluster_state_writable()?;+ let previous_doc = self.pipelines.get(id).map(|v| v.value().clone());+ // Taken out up front so a failed compile cannot leave a stale+ // executable behind; put back verbatim on every abort path.+ let previous_compiled = self.transform_pipelines.remove(id).map(|(_, p)| p);++ // `compile_pipeline` clears this on success; snapshot it so a failed++ // flush can put the previous definition's unrunnable reason back too++ // (issue #204 — a rolled-back write must not turn "stored but not++ // runnable, and here is why" into a bare "not found").++ let previous_reason = self.unrunnable_pipelines.get(id).map(|r| r.value().clone());++ // `compile_pipeline` mutates nothing when it fails, so an abort only+ // has to undo the two lines above.+ let compile_err = match self.compile_pipeline(id, config) {+ Ok(()) => None,+ Err(e) => match keep_if_uncompilable {+ None => {+ if let Some(p) = previous_compiled {+ self.transform_pipelines.insert(id.to_string(), p);+ }+ return Ok(Some(e));+ }+ Some(doc) => {+ self.pipelines.insert(id.to_string(), doc);++ // Issue #204: the kept definition is visible to `GET`, so++ // ingest through it must refuse WITH the reason rather++ // than claim the pipeline is missing.++ self.unrunnable_pipelines.insert(++ id.to_string(),++ format!(++ "the stored definition does not compile ({e}); no \++ document can be ingested through pipeline [{id}]"++ ),++ );+ Some(e)+ }+ },+ };++ if let Err(e) = self.flush_cluster_state() {+ match previous_doc {+ Some(d) => self.pipelines.insert(id.to_string(), d),+ None => self.pipelines.remove(id).map(|(_, v)| v),+ };+ self.transform_pipelines.remove(id);+ if let Some(p) = previous_compiled {+ self.transform_pipelines.insert(id.to_string(), p);+ }++ match previous_reason {++ Some(r) => self.unrunnable_pipelines.insert(id.to_string(), r),++ None => self.unrunnable_pipelines.remove(id).map(|(_, v)| v),++ };+ return Err(e);+ }++ if let Some(e) = &compile_err {+ warn!(pipeline = id, error = %e, "pipeline stored but failed to compile");+ } else {+ info!(name = id, "transform pipeline created");+ }+ Ok(compile_err)+ }++ /// Delete an ingest pipeline — both the stored definition and the+ /// compiled form, so `?pipeline={name}` stops working at the same moment+ /// `GET /_ingest/pipeline/{name}` starts answering 404. `false` when it+ /// did not exist.+ pub fn delete_pipeline(&self, name: &str) -> Result<bool> {+ let removed = self.persisted_remove(&self.pipelines, name)?;+ if removed {+ self.transform_pipelines.remove(name);++ // Issue #204: a deleted pipeline must not leave a stale++ // "stored but not runnable" reason behind either — the next++ // definition under this id starts from a clean slate.++ self.unrunnable_pipelines.remove(name);+ }+ Ok(removed)+ }+/// Add an alias pointing to an index.pub fn add_alias(&self, alias: &str, index: &str) {let mut entry = self.aliases.entry(alias.to_string()).or_default();@@@ -2318,41 -3294,19 +3340,45 @@@// ── Transform pipeline methods ────────────────────────────────────────────+ /// The `index.default_pipeline` setting for `index`, if it has one.+ ///+ /// One implementation for every write path. It used to live only in the+ /// `_doc` handlers, which is why `_bulk` ignored the setting entirely and+ /// stored documents untransformed under a `201` while the single-document+ /// form of the same write refused (issue #204).+ ///+ /// Handles the three shapes settings arrive in — nested+ /// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`+ /// and flat `{default_pipeline: ..}` — mirroring the `number_of_shards`+ /// lookup pattern used elsewhere.+ ///+ /// `index` is matched literally: an alias has no settings of its own, so a+ /// write addressed to an alias does not pick up its backing index's default+ /// pipeline. That is a known gap, identical on every write path.+ pub fn default_pipeline_for(&self, index: &str) -> Option<String> {+ self.index_settings.get(index).and_then(|v| {+ v.get("index")+ .and_then(|ix| ix.get("default_pipeline"))+ .or_else(|| v.get("index.default_pipeline"))+ .or_else(|| v.get("default_pipeline"))+ .and_then(Value::as_str)+ .map(str::to_string)+ })+ }+- /// Compile and register a typed transform pipeline from a JSON config.+ /// Compile and register a typed transform pipeline from a JSON config,+ /// **in memory only**.////// `config_json` must be a valid [`PipelineConfig`](xerj_wasm::pipeline::PipelineConfig)- /// object. The compiled pipeline is stored in `transform_pipelines` and- /// can be retrieved by name for use at ingest time.+ /// object. The compiled pipeline is stored in `transform_pipelines` and+ /// the raw JSON in `pipelines`, so the ES-compatible ingest API can hand+ /// it back.///- /// The raw JSON is also stored in `pipelines` so it can be returned by the- /// ES-compatible ingest pipeline API.- pub fn create_pipeline(+ /// This is the primitive underneath [`Engine::put_pipeline`] and the boot+ /// path; it deliberately does not touch disk, so it is private. Anything+ /// that serves a user request must go through `put_pipeline`, or the+ /// pipeline is gone at the next restart (issue #203).+ fn compile_pipeline(&self,name: &str,config_json: Value,@@@ -2362,34 -3316,7 +3388,68 @@@let pipeline = xerj_wasm::pipeline::Pipeline::from_config(name, &cfg)?;self.pipelines.insert(name.to_string(), config_json);self.transform_pipelines.insert(name.to_string(), pipeline);++ // Issue #204: a successful compile clears any recorded "stored but++ // not runnable" reason left by a previous definition under this id.+ self.unrunnable_pipelines.remove(name);- info!(name, "transform pipeline created");+ Ok(())+ }++ /// Store a pipeline definition that could not be compiled, together with+ /// the reason, and make sure nothing stale is left behind.+ ///+ /// Issue #204. This exists for one case: a definition xerj accepts for wire+ /// compatibility (Elasticsearch does) but cannot execute — because it names+ /// a processor this build does not implement, or sets an option on one that+ /// this build cannot honour (a processor-level `if` guard, an `on_failure`+ /// chain, a `grok` `patterns` array). Storing it keeps+ /// `GET`/`_simulate` truthful about what was submitted; recording the+ /// reason is what stops [`Self::process_through_pipeline`] from silently+ /// passing documents through untransformed. A previously-compiled pipeline+ /// under the same id is REMOVED — re-defining a pipeline must not leave the+ /// old stages quietly running under the new definition's name.- pub fn register_unrunnable_pipeline(&self, name: &str, definition: Value, reason: String) {++ ///++ /// Like [`Engine::put_pipeline`], the stored definition is flushed to++ /// `cluster_state.json` before the caller may acknowledge it (issue++ /// #203). `Err` means it did not reach disk; the in-memory change is++ /// rolled back first. The *reason* itself is in-memory only: after a++ /// restart the boot replay fails to recompile the stored definition and++ /// re-records why, so the pipeline stays unrunnable — loudly — either way.++ pub fn register_unrunnable_pipeline(++ &self,++ name: &str,++ definition: Value,++ reason: String,++ ) -> Result<()> {++ self.ensure_cluster_state_writable()?;++ let previous_doc = self.pipelines.get(name).map(|v| v.value().clone());++ let previous_compiled = self.transform_pipelines.remove(name).map(|(_, p)| p);++ let previous_reason = self++ .unrunnable_pipelines++ .get(name)++ .map(|r| r.value().clone());+ self.pipelines.insert(name.to_string(), definition);- self.transform_pipelines.remove(name);+ self.unrunnable_pipelines+ .insert(name.to_string(), reason.clone());++ if let Err(e) = self.flush_cluster_state() {++ match previous_doc {++ Some(d) => self.pipelines.insert(name.to_string(), d),++ None => self.pipelines.remove(name).map(|(_, v)| v),++ };++ match previous_reason {++ Some(r) => self.unrunnable_pipelines.insert(name.to_string(), r),++ None => self.unrunnable_pipelines.remove(name).map(|(_, v)| v),++ };++ if let Some(p) = previous_compiled {++ self.transform_pipelines.insert(name.to_string(), p);++ }++ return Err(e);++ }+ warn!(+ name,+ reason = reason.as_str(),+ "ingest pipeline stored but NOT runnable — documents may not be ingested through it"+ );+ Ok(())}/// Run `docs` through a named pipeline, returning `(action, doc)` pairs.