Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

Merge origin/main into fix/issue-204-fallback-sweep

xerj-org3e73a00c07ba
diff --cc CHANGELOG.md
index 00d8a674a,23cc2c8fc..05c76f536
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,77 -7,138 +7,207 @@@ and this project adheres to [Semantic V
## [Unreleased]
+ ### Fixed
+
+ - **`PUT /{index}/_settings {"index.lifecycle.name": null}` actually detaches
+ the index, and the detach survives a restart** (#282, ported from #262). The
+ null previously fell through a string-only settings reader, so the operator
+ got `200 acknowledged` while the index stayed attached — to a policy whose
+ delete phase then destroyed it. A detach now removes the execution cursor,
+ writes a persisted tombstone (`ism_managed_indices.json` grew a
+ `managed`/`detached` envelope; the old bare-map file is still read), and
+ scrubs the stale `index.lifecycle.name` from the stored settings.
+ - **The lifecycle delete action gained #262's safety rails.** It now refuses —
+ visibly, in `explain`, never silently — to delete a dot-prefixed internal
+ index (`.ds-*` backing indices exempt), a data stream's current write index,
+ or an index whose age cannot be established from its execution cursor.
+ - **An ILM policy naming an action the engine cannot execute is refused at PUT
+ time with the action named** (400), instead of being stored with the action
+ silently dropped — the accepted-and-ignored class (#204). Executable ILM
+ actions are `rollover`, `delete`, `readonly`.
+
+ ### Added
+
+ - **`GET /_ilm/status`, `POST /_ilm/start`, `POST /_ilm/stop`** (#282): the
+ operator can halt lifecycle execution without stopping the node; a stopped
+ engine's tick touches nothing. The flag is in-memory — a restart resumes
+ execution — while the per-index detach tombstone is the durable stop.
+ - **`POST /{index}/_ilm/remove`** (#282): ES's own detach verb; goes through
+ the same tombstoned detach path as the settings-null route. A literal name
+ that is not an index answers 404 rather than writing a tombstone for a
+ ghost.
+
+ - **A crafted filename can no longer forge records on the agent progress
+ stream** (the first rc.15 known issue below). Every externally-controlled
+ string — in-flight paths, the paths and error text interpolated into human
+ notes, the terminal line's reason — is stripped of control characters, bidi
+ overrides and zero-width characters and bounded in length before it reaches
+ any progress surface. Measured on a corpus holding crafted names: a run that
+ previously emitted a forged `xerj-done ok=true exit=0 reason=completed`
+ 2.2 s ahead of its real terminal line now emits one terminal line and no
+ control characters at all. The same sanitisation covers the walker's
+ "skipping unreadable entry" warning, which renders a path to the same stderr.
+ - **`--progress json` paces the bar like `--progress plain`.** The `bar` field
+ bypassed the spacing slot entirely: measured at `--progress-interval 1`, 178
+ of 178 ticks carried a bar against 18 bars on the plain surface. It is now a
+ string on exactly the ticks that owe a bar and `null` in between — 26 of 320
+ ticks on the same corpus, against 16 on plain.
+ - **The bar spacing floor is the 15 s the documentation states.** A half-tick
+ tolerance made the enforced floor `interval/2` shorter — 12.5 s at the
+ shipped defaults. Measured at `--progress-interval 10`: 11 of 11 same-phase
+ gaps under 15 s (min 10.0 s) before, none after (min 19.3 s).
+
+ ## [1.0.0-rc.15] - 2026-08-10
+
+ ### Known issues in this release
+
+ Both were found by adversarial review *after* their pull requests merged, so
+ they are present in these binaries. Fixes are in flight for rc.16.
+
+ - **A crafted filename can forge records on the agent progress stream.** The
+ in-flight path is rendered onto the progress surface without control-character
+ sanitisation, so a filename containing a newline can inject something that
+ looks like a genuine `xerj-progress` or `xerj-done` record — including a false
+ `ok=true` completion — into the stream this release tells AI agents to parse.
+ Indexing a repository whose filenames you do not control is enough to trigger
+ it. Until the fix lands, do not treat that stream as trustworthy when the
+ corpus is untrusted.
+ - **An outer repository's `.gitignore` judges files inside a nested checkout.**
+ The ignore lookup walks every layer with no repository-boundary stop, unlike
+ git itself, so a vendored or submoduled repository is filtered by rules that
+ have no authority over it. Files you expect to be indexed may be skipped.
+ `--no-ignore` bypasses it.
+
+ Also unfixed and worth knowing: `--progress json` does not pace the bar (a JSON
+ consumer sees roughly ten times as many bar lines as a plain consumer), and the
+ documented "at most one bar per 15 s" floor is really 12.5 s.
+
+
+ ### Added
+
+ - **Index lifecycle policies are executed, not just stored**
+ ([#199](https://github.com/xerj-org/xerj/issues/199), contributed by
+ @Vinz2168). A single internal engine modeled on OpenSearch's ISM state machine
+ — named states, ordered actions, ordered transitions — exposed through two
+ REST surfaces: native ISM at `_plugins/_ism/*` and the Elasticsearch-shaped
+ `_ilm/*`. `spawn_lifecycle_manager()` runs the tick on
+ `lifecycle.tick_interval_secs` (default 300s, matching OpenSearch ISM's own
+ job interval), and the managed-index cursor survives restart. Retention now
+ actually deletes and rolls over instead of being acknowledged and forgotten.
+ One documented divergence: `min_age` is measured from when an index entered
+ its current state rather than from rollover time, so a policy that rolls over
+ *and* has a downstream phase meant to be measured from the rollover will
+ advance on a different clock than Elasticsearch uses.
+
+ - **`xerj autoindex` honours `.gitignore` and `.xerjignore`**
+ ([#276](https://github.com/xerj-org/xerj/issues/276)). Nested ignore files and
+ negation patterns follow git's own precedence, with build-output defaults
+ (`node_modules`, `vendor`, `target`, `dist`, `build`, `.venv`, `__pycache__`)
+ on top; `--no-ignore` and `--no-default-ignores` turn them off. Pointing xerj
+ directly at an ignored path still indexes it — an explicit instruction beats a
+ rule the user did not write for this purpose. `--dry-run` reports what was
+ skipped and by which rule, because a user whose files did not appear needs to
+ know why. Measured on this repository: 274,826 files walked in 1.18s becomes
+ 1,385 in 0.25s.
+
+ - **Progress is relayed to an AI agent as a drawn bar.** The stream surface
+ carries a rendered bar alongside its machine-readable `key=value` fields, so
+ an agent driving `xerj autoindex` on someone's machine can show a real
+ progress line rather than going silent for minutes. `--progress json` keeps a
+ single parseable stream; `--quiet` still writes nothing on either.
+
+ ### Fixed
+
+ - **Tied scores have one total order — a bounded page is a prefix of the full
+ page** ([#191](https://github.com/xerj-org/xerj/issues/191)). A `size:5` page
+ and a `size:1000` page disagreed about which tied document came back. The
+ memtable's bounded candidates are now ranked by the same page key as the
+ segment path, so the two agree. The remaining `constant_score` and post-sort
+ re-sort cases are tracked separately in
+ [#270](https://github.com/xerj-org/xerj/issues/270).
+
+ - **`date_histogram` no longer aborts the process on a multi-byte `time_zone`**
+ ([#272](https://github.com/xerj-org/xerj/issues/272), contributed by
+ @Vinz2168). `aggs::parse_time_zone_offset` sliced a `&str` on a byte index
+ that could land mid-character, and with `panic=abort` that took the whole node
+ down — an unauthenticated request was enough.
+
+ - **The legacy path-parameter form of scroll continuation works again**
+ (contributed by @Vinz2168), and **`_cat/*?format=json` plus `_data_stream/**`
+ wildcards** are supported (also @Vinz2168) — both are shapes real
+ Elasticsearch clients send.
+
### Changed
+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot
+ honour, instead of accepting it and analysing differently**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an
+ existing index is opened, where a hard failure would brick it), so an
+ unresolvable tokenizer silently became `standard` and an unresolvable token
+ filter was dropped — the index was created, `GET /_settings` echoed the block
+ back, and the field was analysed in a way nobody asked for. Index creation
+ now validates the block and answers `400` with the exact constructs it cannot
+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused
+ (it only ever changed what `GET /_settings` echoed; the registry is built
+ once, at create/open, and there is no rebuild path — Elasticsearch also
+ refuses analysis updates on an open index).
+
+ **This rejects settings blocks that XERJ used to accept with a `200`, and the
+ ES-YAML conformance suite does not cover them — CI being green is not
+ evidence that your index still creates.** What is honoured:
+
+ | | honoured (still `200`) |
+ |---|---|
+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |
+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |
+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |
+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |
+ | `analysis.normalizer` | nothing — never built |
+
+ Anything else in those positions is now a `400`. There is one important
+ exception to read first: XERJ resolves a declared filter or tokenizer by
+ **name**, not by `type`, so a declaration whose *name* is a built-in is still
+ honoured whatever its type says. Measured, all still `200`:
+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,
+ `{"lowercase": {"type": "lowercase"}}`,
+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`
+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which
+ has a regression test pinning it. Naming a built-in analyzer on a field
+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.
+
+ What now fails is a declaration under a name that does NOT resolve — which is
+ the usual ES habit of naming filters after what they do in *your* index.
+ Measured blocks that returned `200` in rc.14 and return `400` from this
+ release:
+
+ - a `keyword` field's `normalizer` (any);
+ - a `char_filter` (any, including `html_strip` and `mapping`), declared
+ either under `analysis.char_filter` or as an analyzer's `char_filter`;
+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,
+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,
+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,
+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,
+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,
+ which never lowercased anything, because the analyzer referencing
+ `my_lowercase` found no such filter and dropped it;
+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,
+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,
+ `letter`, `lowercase` or `simple_pattern`;
+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,
+ `fingerprint`, or a language other than `english`;
+ - a `synonym` filter whose `synonyms` is a bare string rather than an array
+ (it built a filter with zero rules).
+
+ **Upgrading:** there is no opt-out flag. Either remove the analysis block and
+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only
+ the constructs in the table. The `400` names every offending construct in one
+ response, so one round-trip is enough to see the whole list. **Two paths are
+ not covered and still accept these blocks silently: an index template's
+ `template.settings.analysis` is dropped wholesale before the gate ever sees
+ it, and option-level divergence inside a supported type (`synonyms_path`,
+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**
+
- **`autoindex` parses each PDF once per fresh run instead of twice.** Phase A
already paid a *complete* parse for every PDF — `extract::extract` routes
`Family::Pdf` straight to the isolated worker and drops the sampling limit,
@@@ -149,67 -210,83 +279,145 @@@
### Fixed
+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and
+ every write path enforces it**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only
+ warn-log a compile failure, so a pipeline that could never run was
+ acknowledged and echoed back by `GET`. It now compiles first, with three
+ outcomes: compiled → stored and acknowledged; a processor config we refuse →
+ `400`; a processor or option this build cannot honour → accepted (as
+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses
+ rather than storing an untransformed document.
+
+ Defects this surfaced, each of which answered `200`/`201` and did something
+ smaller than asked: **every** ES `rename` processor failed to compile (the
+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and
+ renamed nothing; ES `append` was executed as `set`, replacing the field
+ instead of extending it, and disagreeing with `_simulate`, which implemented
+ a real append; `set.override` defaulted to `false` where Elasticsearch
+ defaults it to `true`; an unknown `pii_redaction` type built an empty pattern
+ set and redacted nothing; an unknown `grok` pattern name silently became the
+ catch-all; an unknown `convert` target left the field unconverted per
+ document; array-typed keys given as a bare string were dropped and the stage
+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the
+ definition while leaving the compiled pipeline live under the same name.
+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`
+ rather than being refused.
+
+ Two more of the same, at the definition's own edges: a **pipeline-level**
+ `on_failure` block (the top-level body key, not the processor-level one) was
+ accepted, echoed by `GET`, and never run — the pipeline is now recorded
+ unrunnable, exactly as its processor-level twin already was; and `set` with
+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted
+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:
+ missing value`, which told the caller their valid ES pipeline was malformed.
+
+ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**
+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,
+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`
+ previously read the instruction and dropped it — a redaction pipeline that
+ the single-document endpoint refused was ignored in bulk, and the document
+ was stored verbatim under `errors: false`. All four now run the pipeline, and
+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the
+ name before touching a single document, so a bad pipeline fails the request
+ instead of half of it.
+
+ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`
+ copied at most 10,000 documents and swallowed every per-document write error
+ before answering `{"acknowledged": true}` (now full keyset paging, and any
+ failed write — or a failed source flush — fails the request); and a failed
+ `_bulk` item no longer reports `"result": "deleted"` alongside its own
+ `status: 400`, which claimed a deletion that never happened.
+
+ **Known gaps, stated rather than closed:** `_simulate` interprets the stored
+ ES body directly and covers processors the compiled path does not, so a
+ pipeline recorded as unrunnable can still show a transformation under
+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`
- returns the ES body with no runnability signal, so an operator only learns at
- write time; a write addressed to an **alias** does not pick up the backing
- index's `default_pipeline`; and `_update_by_query` reports a `drop`
++ returns the stored definition (the ES body for an unrunnable pipeline, the
++ compiled shape otherwise) with no runnability signal, so an operator only
++ learns at write time; a write addressed to an **alias** does not pick up the
++ backing index's `default_pipeline`; and `_update_by_query` reports a `drop`
+ processor as a per-document failure, because it updates in place and has no
+ delete path.
+
+ - **An index whose metadata will not parse is refused instead of silently
+ reopened with an empty mapping**
+ ([#202](https://github.com/xerj-org/xerj/issues/202)). `Index::open` treated
+ "this file is not there" and "this file does not parse" as the same thing:
+ `load_schema`/`load_settings` mapped ENOENT, EACCES, EIO and malformed JSON
+ alike to one anonymous error, and the caller answered all of them with a
+ fresh dynamic mapping. Measured on a field deliberately mapped `keyword`:
+ after `schema.json` was truncated to half its bytes the index still opened
+ `Ok` with `field_count = 0`, the field came back `None`, and one further
+ document re-inferred it as `long` — a mapping silently replaced by a
+ different one, with nothing in any response saying so. Absent stays absent
+ (indices predating create-time schema persistence legitimately have no
+ `schema.json`); present-but-unparseable now fails the open with an error
+ naming the file. `es_mapping.json` — the full-fidelity mapping behind
+ `GET /_mapping` — was logged-and-ignored on a parse failure and now fails the
+ index too, on the boot scan, on snapshot restore and on a retry.
+
+ **Visible on upgrade:** a node whose data dir already holds a corrupt sidecar
+ now boots **red** with that index unserved, where it previously came up green
+ with an empty mapping. It joins the failed set
+ [#206](https://github.com/xerj-org/xerj/issues/206) introduced, so every
+ surface that set feeds already reports it — measured on a node holding two
+ user indices, `victim` with a truncated `schema.json`:
+
+ ```
+ GET /_cluster/health -> red, unassigned_primary_shards: 1
+ GET /_cluster/health?level=indices
+ -> victim red, unassigned_info.details = the
+ absolute path of schema.json and the parse error
+ GET /_cat/indices -> "red open victim <uuid> 1 0 0 0 0b 0b 0b"
+ GET /_cluster/indices/failed -> the reason, plus the retry and delete calls
+ PUT /victim, POST /victim/_doc, POST /victim/_search
+ -> 503 no_shard_available_action_exception
+ carrying that same reason
+ GET /health/ready -> 200 "ready (degraded): 1 of 2 indices
+ serving, 1 failed to open; see GET
+ /_cluster/indices/failed"
+ POST /healthy/_search -> 200
+ ```
+
+ The readiness counts above are the two-index test router's. A running node
+ also holds its internal `.xerj_*` indices, so the same condition on the shipped
+ binary reads `ready (degraded): 15 of 16 indices serving, 1 failed to open` —
+ the 200 and its meaning are unchanged, only the totals move with what else the
+ node holds.
+
+ Recovery is the three doors #206 added. Two were measured end to end on the
+ shipped binary: repair the file and
+ `POST /_cluster/indices/failed/{index}/_retry` (503 while it is still torn,
+ `200 {"reopened": true}` once it is not, health back to green), and
+ `DELETE /{index}` (200, directory removed, health back to green). The third —
+ restore the index from a snapshot — is the path this change touches (a
+ successful restore now clears the recorded failure, which it has to or health
+ would stay red after the repair worked); it is covered by an engine-level
+ test rather than measured over HTTP. The node stays in kubelet rotation as
+ long as any index is still serving.
+
+ - **Concurrent sidecar writes can no longer manufacture the torn file the reader
+ now refuses.** `write_file_atomic` staged every write in
+ `path.with_extension("tmp")` — one shared name for all writers of that file —
+ so two concurrent writers of one sidecar (`PUT /{index}/_settings` racing
+ another settings update, two API-key mints for `api_keys.json`) both opened it
+ `O_TRUNC`, interleaved their bytes, and each renamed it into place. The rename
+ is atomic; the content being renamed was not one writer's. Measured with the
+ old shared name, 4 threads × two different-length settings bodies × 200
+ rounds, over four runs: **86–294 of 800 writes failed with ENOENT** (the loser
+ renaming a file the winner had already moved) and, with those errors swallowed
+ as the callers did, **1–16 of 200 rounds left an unparseable `settings.json`**.
+ The counts are race-dependent and vary with machine load; what does not vary is
+ that both are non-zero on every run with the shared name and **0 and 0** on
+ every run with a unique staging name. Each write now stages
+ in its own sibling file (`<file>.tmp.<pid>.<seq>`) in the target's directory
+ and removes its own debris on failure — which for `api_keys.json` is key
+ material. `update_settings` also writes `settings.json` through
+ `write_file_atomic` rather than a plain `fs::write`; it was the last
+ non-atomic sidecar writer left.
+
- **`index` on the mapping is now honoured instead of silently ignored**
([#204](https://github.com/xerj-org/xerj/issues/204)). `"index": false` was
accepted by `PUT /{index}`, echoed back verbatim by `GET /{index}/_mapping`,
diff --cc engine/crates/xerj-api/src/es_compat.rs
index 949efe667,c451efb85..18dd66967
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@@ -22858,112 -22415,25 +23077,121 @@@ pub async fn put_ingest_pipeline
}
_ => proc_config.clone(),
};
- Some(json!({"type": xerj_type, "config": adapted_config}))
- }).collect();
- json!({
- "description": body.get("description").and_then(Value::as_str).unwrap_or(""),
- "stages": stages
- })
- } else {
- body.clone()
- };
- // `Some(body)` keeps this surface's long-standing behaviour: a definition
- // that will not compile is still stored and still readable through
- // `GET /_ingest/pipeline/{id}` (the engine logs the compile error and
- // drops the executable form, so `?pipeline={id}` behaves identically
- // before and after a restart). An `Err` means it did not reach disk —
- // answering `acknowledged` there would promise a pipeline that is gone at
- // the next boot, which is the whole of issue #203.
- match state.engine.put_pipeline(&id, xerj_config, Some(body)) {
- Ok(_) => Json(json!({ "acknowledged": true })).into_response(),
+ stages.push(json!({"type": xerj_type, "config": adapted_config}));
+ }
+ json!({
+ "description": body.get("description").and_then(Value::as_str).unwrap_or(""),
+ "stages": stages
+ })
+ } else {
+ body.clone()
+ };
+ // Elasticsearch pipeline-level (top-level body) keys that decide what
+ // happens when a processor fails. The compiled pipeline has no equivalent:
+ // `PipelineConfig` carries a single `on_error` policy and no recovery
+ // chain, so a top-level `on_failure` was read by nothing.
+ //
+ // This is the SAME defect the repair fixed one level down
+ // (`UNHONOURED_PROCESSOR_KEYS` in `xerj-wasm/src/pipeline.rs`): refusing a
+ // processor-level `on_failure` while silently dropping the identical key at
+ // pipeline level would have left the endpoint inconsistent with itself.
+ // `description`, `version`, `_meta` and `deprecated` are NOT here — they
+ // are identification metadata that changes no document, the same line drawn
+ // for processor-level `tag`/`description`.
+ const UNHONOURED_PIPELINE_KEYS: &[(&str, &str)] = &[(
+ "on_failure",
+ "the pipeline-level recovery processors are not run; the compiled pipeline's \
+ `on_error` policy applies instead",
+ )];
+ let unhonoured_top_level = UNHONOURED_PIPELINE_KEYS
+ .iter()
+ .find(|(key, _)| body.get(*key).is_some())
+ .copied();
+ // Three outcomes, and the difference between them is the whole point of
+ // issue #204:
+ //
- // compiled → store the ES body verbatim (that is what ES
- // echoes from GET, and what `_simulate` walks) and
- // acknowledge.
++ // compiled → store the translated config (the shape the boot
++ // replay can recompile, and one `_simulate` walks
++ // just as well as the ES body) and acknowledge.
+ // config we refuse → 400. The caller misconfigured a processor xerj
+ // does implement; storing it would mean answering
+ // "acknowledged" for redaction that never runs.
+ // stage we don't have → accept, exactly as ES does — but record WHY it
+ // has no compiled form, so every ingest path
+ // refuses loudly instead of indexing the document
+ // untransformed. This is the "degrade loudly"
+ // half of the issue: the capability gap is xerj's,
+ // not the caller's, and it surfaces at the point
+ // where a document would otherwise be silently
+ // mis-ingested.
- match state.engine.create_pipeline(&id, xerj_config) {
- Ok(()) if unhonoured_top_level.is_some() => {
++ //
++ // Whichever way it goes, nothing is acknowledged before it reaches
++ // `cluster_state.json` (issue #203): `put_pipeline` (with `None`, so a
++ // refused definition stores nothing) and `register_unrunnable_pipeline`
++ // both flush before returning, and an `Err` from either means the write
++ // did not reach disk — answering `acknowledged` there would promise a
++ // pipeline that is gone at the next boot.
++ match state.engine.put_pipeline(&id, xerj_config, None) {
++ Ok(None) if unhonoured_top_level.is_some() => {
+ // Every stage compiled, but a top-level key we cannot act on came
+ // with it. `register_unrunnable_pipeline` drops the compiled form
- // `create_pipeline` just installed, so nothing runs half-honoured.
++ // `put_pipeline` just installed, so nothing runs half-honoured.
+ let (option, reason) = unhonoured_top_level.expect("guarded by the match arm");
- state.engine.register_unrunnable_pipeline(
++ match state.engine.register_unrunnable_pipeline(
+ &id,
+ body,
+ format!(
+ "pipeline-level [{option}] is not supported by this xerj build \
+ ({reason}); no document can be ingested through pipeline [{id}]"
+ ),
- );
- Json(json!({ "acknowledged": true })).into_response()
- }
- Ok(()) => {
- // `create_pipeline` stores the *translated* config; put the ES
- // body back so GET and _simulate see what was actually submitted.
- state.engine.pipelines.insert(id.clone(), body);
- Json(json!({ "acknowledged": true })).into_response()
++ ) {
++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),
++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),
++ }
+ }
- Err(xerj_wasm::WasmError::UnsupportedStage { stage }) => {
- state.engine.register_unrunnable_pipeline(
++ Ok(None) => Json(json!({ "acknowledged": true })).into_response(),
++ Ok(Some(xerj_wasm::WasmError::UnsupportedStage { stage })) => {
++ match state.engine.register_unrunnable_pipeline(
+ &id,
+ body,
+ format!(
+ "processor [{stage}] is not implemented by this xerj build; \
+ no document can be ingested through pipeline [{id}]"
+ ),
- );
- Json(json!({ "acknowledged": true })).into_response()
++ ) {
++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),
++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),
++ }
+ }
+ // Same shape one level down: the processor exists, an option on it does
+ // not. ES accepts the definition, so we do too — and record why it
+ // cannot run, so no document is written as if the option had been
+ // honoured (issue #204 applied to this endpoint's own new contract).
- Err(xerj_wasm::WasmError::UnsupportedOption {
++ Ok(Some(xerj_wasm::WasmError::UnsupportedOption {
+ stage,
+ option,
+ reason,
- }) => {
- state.engine.register_unrunnable_pipeline(
++ })) => {
++ match state.engine.register_unrunnable_pipeline(
+ &id,
+ body,
+ format!(
+ "processor [{stage}] option [{option}] is not supported by this xerj \
+ build ({reason}); no document can be ingested through pipeline [{id}]"
+ ),
- );
- Json(json!({ "acknowledged": true })).into_response()
++ ) {
++ Ok(()) => Json(json!({ "acknowledged": true })).into_response(),
++ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),
++ }
+ }
- Err(e) => {
++ Ok(Some(e)) => {
+ tracing::warn!(pipeline = %id, error = %e, "rejected ingest pipeline definition");
+ let err = xerj_common::XerjError::invalid_mapping(format!(
+ "pipeline [{id}] cannot be created: {e}"
+ ));
+ ApiError::new(err).into_response()
+ }
+ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),
}
}
@@@ -22994,18 -22464,13 +23222,20 @@@ pub async fn delete_ingest_pipeline
State(state): State<AppState>,
Path(id): Path<String>,
) -> impl IntoResponse {
- if state.engine.pipelines.remove(&id).is_some() {
- // Issue #204: only the definition used to be removed. The COMPILED
- // pipeline stayed in `transform_pipelines`, so a delete that answered
- // `acknowledged: true` left the stages running for any caller that
- // reached them by name (`POST /v1/ingest?pipeline=`) — deletion
- // acknowledged, deletion not performed.
- state.engine.transform_pipelines.remove(&id);
- state.engine.unrunnable_pipelines.remove(&id);
- Json(json!({ "acknowledged": true })).into_response()
- } else {
- let e = xerj_common::XerjError::index_not_found(format!("pipeline [{id}] is missing"));
- ApiError::new(e).into_response()
++ // Issue #204: only the definition used to be removed here. The COMPILED
++ // pipeline stayed in `transform_pipelines`, so a delete that answered
++ // `acknowledged: true` left the stages running for any caller that
++ // reached them by name (`POST /v1/ingest?pipeline=`) — deletion
++ // acknowledged, deletion not performed. `Engine::delete_pipeline` removes
++ // the definition, the compiled form and any recorded unrunnable reason
++ // together, and persists the removal (issue #203).
+ match state.engine.delete_pipeline(&id) {
+ Ok(true) => Json(json!({ "acknowledged": true })).into_response(),
+ Ok(false) => {
+ let e = xerj_common::XerjError::index_not_found(format!("pipeline [{id}] is missing"));
+ ApiError::new(e).into_response()
+ }
+ Err(e) => ApiError::new(xerj_common::XerjError::from(e)).into_response(),
}
}
diff --cc engine/crates/xerj-engine/src/engine.rs
index ee6778ab9,08b645e04..9a348f270
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@@ -421,19 -472,17 +472,25 @@@ pub struct Engine
pub templates: Arc<DashMap<String, IndexTemplate>>,
/// scroll_id → ScrollContext
pub scrolls: Arc<DashMap<String, ScrollContext>>,
- /// pipeline_id → pipeline definition JSON
+ /// pipeline_id → pipeline definition JSON. Persisted in
+ /// `<data_dir>/cluster_state.json`; mutate only through
+ /// [`Engine::put_pipeline`] / [`Engine::delete_pipeline`], or the change
+ /// will not survive a restart (issue #203).
pub pipelines: Arc<DashMap<String, Value>>,
+ /// pipeline_id → why the stored definition has no compiled counterpart.
+ ///
+ /// Issue #204. A definition that carries a stage type this build does not
+ /// implement is still stored (Elasticsearch accepts it, and
+ /// `_ingest/pipeline/{id}/_simulate` still walks it), but nothing may be
+ /// ingested through it: `process_through_pipeline` refuses with the reason
+ /// recorded here instead of quietly indexing the untransformed document.
+ pub unrunnable_pipelines: Arc<DashMap<String, String>>,
/// index_name → open/closed state (true = closed)
pub closed_indices: Arc<DashMap<String, bool>>,
- /// data stream name → DataStream
+ /// data stream name → DataStream. Persisted in
+ /// `<data_dir>/cluster_state.json`; mutate only through the
+ /// `*_data_stream` methods, or the change will not survive a restart
+ /// (issue #203).
pub data_streams: Arc<DashMap<String, DataStream>>,
/// ILM policy name → raw policy JSON, kept verbatim for `GET _ilm/policy`
/// round-trip fidelity. The REAL execution engine reads `ism_policies`
@@@ -1635,6 -1901,583 +1910,620 @@@ impl Engine
self.record_index_created_at(name);
}
+ // ── Cluster metadata persistence (issue #203) ────────────────────────────
+ //
+ // Index templates, legacy (v1) templates, component templates, ingest
+ // pipelines, data streams and ILM policies used to live only in the
+ // in-memory maps above: `PUT /_index_template/logs` answered `200
+ // {"acknowledged": true}` and the template was gone after the next boot,
+ // with no error anywhere and the next matching index silently created
+ // unshaped. They are now snapshotted, as one consistent document, into
+ // `<data_dir>/cluster_state.json` on every mutation and reloaded in
+ // `Engine::new`.
+ //
+ // Durability contract: a full-file rewrite through
+ // `index::write_file_atomic` — write to `cluster_state.tmp`, `fsync` it,
+ // `rename` over the target, then `fsync` the parent directory so the
+ // rename itself survives power loss. The rename is the commit point, so
+ // a `kill -9` at any instant leaves either the previous complete
+ // document or the new one, never a half-written mix. The same shape peer
+ // engines use for their manifests (cf. `fjall/src/file.rs:17`
+ // `fsync_directory`, which fjall calls after every atomic rewrite;
+ // `sled/src/metadata_store.rs:696` discards leftover `*.tmp` files from
+ // an interrupted rewrite on the next boot, which `load_cluster_state`
+ // mirrors below).
+ //
+ // One file rather than six keeps the maps mutually consistent — a data
+ // stream and the template that shaped it can never be restored from
+ // different generations.
+
+ /// Path of the persisted cluster-metadata document
+ /// (`<data_dir>/cluster_state.json`).
+ fn cluster_state_path(&self) -> PathBuf {
+ self.data_dir.join("cluster_state.json")
+ }
+
+ /// `Err` when boot found `cluster_state.json` and could not load it — see
+ /// `cluster_state_loaded` for why that has to latch.
+ ///
+ /// Every management mutation calls this, not only the ones that reach
+ /// `flush_cluster_state` with something to write. A `DELETE` consults the
+ /// in-memory map first, and while the load has failed that map is empty,
+ /// so an unguarded delete answers `404 not found` about an object that is
+ /// sitting in the document on disk — the same lie in a different shape.
+ /// A 500 that names the file is the only honest answer while the node
+ /// cannot see the operator's configuration.
+ fn ensure_cluster_state_writable(&self) -> Result<()> {
+ if self
+ .cluster_state_loaded
+ .load(std::sync::atomic::Ordering::Acquire)
+ {
+ return Ok(());
+ }
+ let path = self.cluster_state_path();
+ error!(
+ path = %path.display(),
+ "refusing a cluster-metadata write: this node could not load \
+ cluster_state.json at boot, so its management state is not the \
+ operator's. Recover or move the file aside, then restart."
+ );
+ Err(EngineError::Common(xerj_common::XerjError::storage(
+ format!(
+ "refusing to write {} — it could not be loaded at boot, so \
+ writing now would destroy cluster configuration that is still \
+ on disk; recover or move it aside and restart xerj",
+ path.display()
+ ),
+ )))
+ }
+
+ /// Snapshot every persisted management map into one document.
+ fn cluster_state_snapshot(&self) -> PersistedClusterState {
+ fn dump<V: Clone>(map: &DashMap<String, V>) -> std::collections::BTreeMap<String, V> {
+ map.iter()
+ .map(|e| (e.key().clone(), e.value().clone()))
+ .collect()
+ }
+ PersistedClusterState {
+ version: CLUSTER_STATE_VERSION,
+ index_templates: dump(&self.templates),
+ legacy_templates: dump(&self.legacy_templates),
+ component_templates: dump(&self.component_templates),
+ pipelines: dump(&self.pipelines),
+ data_streams: dump(&self.data_streams),
+ ilm_policies: dump(&self.ilm_policies),
+ }
+ }
+
+ /// Durably write the current cluster metadata.
+ ///
+ /// Unlike `flush_aliases`, a failure here is **returned**, not swallowed:
+ /// every caller is an API write that would otherwise answer
+ /// `{"acknowledged": true}` for a change it cannot keep. Callers roll the
+ /// in-memory change back and surface a 500, so the operator finds out at
+ /// the moment of the write instead of at the next restart.
+ ///
+ /// Refuses outright when boot could not load the existing document — see
+ /// `cluster_state_loaded`. The snapshot below is taken from the live maps,
+ /// which in that state hold only what this process was told after boot, so
+ /// writing it would rename an empty-ish document over configuration that is
+ /// still intact on disk.
+ fn flush_cluster_state(&self) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
+ // One writer at a time — see `cluster_state_write`. Snapshotting
+ // inside the lock also means the bytes that land are never older
+ // than a rewrite that has already returned.
+ let _writing = self.cluster_state_write.lock();
+ let snapshot = self.cluster_state_snapshot();
+ let bytes = serde_json::to_vec_pretty(&snapshot)?;
+ crate::index::write_file_atomic(&self.cluster_state_path(), &bytes)?;
+ Ok(())
+ }
+
+ /// Highest generation `N` for which a backing index `.ds-<stream>-<N>` is
+ /// actually open in this data dir; `0` when there is none.
+ ///
+ /// Read off the open indices rather than the stream document, because the
+ /// point is to catch the case where the two disagree. Suffix parsing keeps
+ /// neighbouring streams apart on its own: for stream `a`, `.ds-a-b-000001`
+ /// leaves `b-000001`, which is not a number.
+ fn highest_backing_generation(&self, stream: &str) -> u64 {
+ let prefix = format!(".ds-{stream}-");
+ self.indices
+ .iter()
+ .filter_map(|e| {
+ e.key()
+ .strip_prefix(&prefix)
+ .and_then(|gen| gen.parse::<u64>().ok())
+ })
+ .max()
+ .unwrap_or(0)
+ }
+
+ /// Load `<data_dir>/cluster_state.json` back into the management maps.
+ ///
+ /// A missing file is normal (fresh node, or a data dir written by a build
+ /// that predates this). A file that is present but does not load is loud
+ /// *and* latching: it means the operator's templates, pipelines and data
+ /// streams are not coming back, which is exactly the silence this whole
+ /// mechanism exists to remove, so `cluster_state_loaded` goes false and
+ /// every subsequent management write is refused rather than allowed to
+ /// rename a snapshot of empty maps over the document on disk.
+ fn load_cluster_state(&self) {
+ // An interrupted rewrite can leave a partial `cluster_state.tmp`
+ // behind. It is never read (the rename is the commit point), but
+ // leaving it litters the data dir and invites a hand-edit that
+ // "restores" a torn document — sled sweeps the same leftovers on
+ // boot (`sled/src/metadata_store.rs:696`).
+ let tmp = self.cluster_state_path().with_extension("tmp");
+ if tmp.exists() {
+ warn!(
+ path = %tmp.display(),
+ "discarding an incomplete cluster_state rewrite left by an unclean shutdown"
+ );
+ let _ = std::fs::remove_file(&tmp);
+ }
+
+ let path = self.cluster_state_path();
+ let bytes = match std::fs::read(&path) {
+ Ok(b) => b,
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => return,
+ Err(e) => {
+ // The file is *there* and its bytes are almost certainly fine
+ // — `EACCES` after a uid change on a container volume, a
+ // backup tool's chmod, `EIO`, `EMFILE` at boot. Do not copy it
+ // aside (we cannot read it) and do not touch it: mark the load
+ // failed so no later write can rename a snapshot of empty maps
+ // over configuration that is still perfectly recoverable.
+ self.cluster_state_loaded
+ .store(false, std::sync::atomic::Ordering::Release);
+ error!(
+ path = %path.display(), error = %e,
+ "could not READ cluster_state.json — index templates, ingest \
+ pipelines, data streams and ILM policies are NOT restored, \
+ and management writes will be refused until a boot loads it \
+ (the file itself is left untouched)"
+ );
+ return;
+ }
+ };
+ let state: PersistedClusterState = match serde_json::from_slice(&bytes) {
+ Ok(s) => s,
+ Err(e) => {
+ // Same position as the read error above: the load failed, so
+ // the live maps are not the operator's configuration and
+ // nothing may be written over the document on disk. Refusing
+ // rather than salvaging-then-overwriting also closes the
+ // second-corruption hole in the `!salvage.exists()` guard
+ // below — with an overwrite still allowed, a second damaged
+ // document would be destroyed *and* not copied, because the
+ // copy from the first one is already there.
+ self.cluster_state_loaded
+ .store(false, std::sync::atomic::Ordering::Release);
+ error!(
+ path = %path.display(), error = %e,
+ "cluster_state.json is corrupt — index templates, ingest \
+ pipelines, data streams and ILM policies are NOT restored, \
+ and management writes will be refused until a boot loads it"
+ );
+ // Keep a copy anyway. The original is now safe from xerj, but
+ // un-wedging the node means moving it aside by hand, and
+ // hand-recovering a template out of a damaged document is the
+ // operator's last option — leave them something to recover
+ // from that survives that step. The original is deliberately
+ // left in place so every boot keeps logging the error above
+ // until someone deals with it, and an existing copy is never
+ // clobbered: the first corruption is the informative one.
+ let salvage = path.with_extension("corrupt.json");
+ if !salvage.exists() {
+ match std::fs::write(&salvage, &bytes) {
+ Ok(()) => warn!(
+ path = %salvage.display(),
+ "kept a copy of the corrupt cluster_state for recovery"
+ ),
+ Err(e) => warn!(
+ path = %salvage.display(), error = %e,
+ "could not preserve a copy of the corrupt cluster_state"
+ ),
+ }
+ }
+ return;
+ }
+ };
+ if state.version > CLUSTER_STATE_VERSION {
+ warn!(
+ found = state.version,
+ supported = CLUSTER_STATE_VERSION,
+ "cluster_state.json was written by a newer xerj; loading what \
+ this build understands (downgrade may drop metadata)"
+ );
+ }
+
+ for (name, tmpl) in state.index_templates {
+ self.templates.insert(name, tmpl);
+ }
+ for (name, body) in state.legacy_templates {
+ self.legacy_templates.insert(name, body);
+ }
+ for (name, body) in state.component_templates {
+ self.component_templates.insert(name, body);
+ }
+ for (name, body) in state.ilm_policies {
+ self.ilm_policies.insert(name, body);
+ }
+
+ // Data streams: restore the record, reconcile it against what is
+ // actually on disk, then say plainly when a backing index it names is
+ // not there. Reporting a stream whose data is gone as
+ // `"status": "GREEN"` without a word in the log is the same quiet lie
+ // this issue is about.
+ let mut reconciled_a_stream = false;
+ for (name, mut ds) in state.data_streams {
+ // `rollover_data_stream` creates the new backing index before it
+ // records the new generation, so `kill -9` in that window leaves
+ // `.ds-<name>-00000N` on disk while the document still says N-1.
+ // Restoring N-1 verbatim would wedge the stream permanently: the
+ // next rollover computes the same name, and `create_index`
+ // refuses it as `index_already_exists` — forever. Adopt the
+ // highest generation that actually exists instead, which is also
+ // what stops a second rollover writing into an index that already
+ // holds the first one's documents.
+ let on_disk = self.highest_backing_generation(&name);
+ if on_disk > ds.generation {
+ let adopted = format!(".ds-{name}-{on_disk:06}");
+ warn!(
+ data_stream = name.as_str(),
+ recorded = ds.generation,
+ found = on_disk,
+ adopted = adopted.as_str(),
+ "adopting a backing index left behind by a rollover that \
+ was interrupted before its generation reached disk"
+ );
+ if !ds.backing_indices.contains(&adopted) {
+ ds.backing_indices.push(adopted);
+ }
+ ds.generation = on_disk;
+ reconciled_a_stream = true;
+ }
+
+ let missing: Vec<&str> = ds
+ .backing_indices
+ .iter()
+ .filter(|b| !self.indices.contains_key(b.as_str()))
+ .map(|b| b.as_str())
+ .collect();
+ if !missing.is_empty() {
+ warn!(
+ data_stream = name.as_str(),
+ missing = missing.join(","),
+ "restored data stream references backing indices that are \
+ not present in the data dir"
+ );
+ }
+ self.data_streams.insert(name, ds);
+ }
+
+ // Pipelines carry behaviour, not just a document to hand back from
+ // `GET /_ingest/pipeline`: the executable form has to be rebuilt or
+ // `?pipeline=x` would be accepted after a restart and quietly do
+ // nothing. Keep the stored document either way so GET round-trips
+ // exactly as it did before the restart.
+ let mut pipelines_restored = 0usize;
+ for (name, config) in state.pipelines {
+ self.pipelines.insert(name.clone(), config.clone());
+ match self.compile_pipeline(&name, config) {
+ Ok(()) => pipelines_restored += 1,
- Err(e) => error!(
- pipeline = name.as_str(), error = %e,
- "persisted ingest pipeline could not be recompiled — it is \
- still visible to GET /_ingest/pipeline but will NOT \
- transform anything; re-PUT the definition"
- ),
++ Err(e) => {
++ // Issue #204: record WHY there is no executable form, so
++ // every ingest path refuses loudly with the reason
++ // instead of reporting a pipeline GET plainly shows as
++ // "not found" — the same contract as a fresh PUT.
++ self.unrunnable_pipelines.insert(
++ name.clone(),
++ format!(
++ "the persisted definition could not be recompiled \
++ at startup ({e}); no document can be ingested \
++ through pipeline [{name}] — re-PUT the definition"
++ ),
++ );
++ error!(
++ pipeline = name.as_str(), error = %e,
++ "persisted ingest pipeline could not be recompiled — it is \
++ still visible to GET /_ingest/pipeline but will NOT \
++ transform anything; re-PUT the definition"
++ )
++ }
+ }
+ }
+
+ // Make an adopted generation durable straight away, so the recovery
+ // happens once rather than on every boot from here on.
+ if reconciled_a_stream {
+ if let Err(e) = self.flush_cluster_state() {
+ warn!(
+ error = %e,
+ "could not persist the reconciled data-stream generation; \
+ it will be recomputed from disk on the next boot"
+ );
+ }
+ }
+
+ let n = self.templates.len()
+ + self.legacy_templates.len()
+ + self.component_templates.len()
+ + self.pipelines.len()
+ + self.data_streams.len()
+ + self.ilm_policies.len();
+ if n > 0 {
+ info!(
+ index_templates = self.templates.len(),
+ legacy_templates = self.legacy_templates.len(),
+ component_templates = self.component_templates.len(),
+ pipelines = pipelines_restored,
+ data_streams = self.data_streams.len(),
+ ilm_policies = self.ilm_policies.len(),
+ "restored persisted cluster metadata"
+ );
+ }
+ }
+
+ /// Name every `.ds-*` index on disk that no restored data stream claims.
+ ///
+ /// This does **not** reconcile anything — nothing is deleted, adopted or
+ /// repaired, because an orphan may hold the only copy of somebody's data.
+ /// It exists so the state is not *silent*: an orphaned backing index is
+ /// unreachable through the data-stream API (`GET` and `DELETE` on the
+ /// stream answer 404, while `PUT /_data_stream/<name>` answers
+ /// `409 resource_already_exists_exception` naming the orphan), so without
+ /// a line in the boot log the operator has no way to learn the name they
+ /// have to pass to `DELETE /<backing-index>` to clear it.
+ ///
+ /// Expected sources: a data dir written by a build that predates
+ /// `cluster_state.json` (data streams were not persisted at all then), or
+ /// a `DELETE /_data_stream` interrupted by a build that recorded the
+ /// removal before destroying the backing indices. Skipped entirely when
+ /// the cluster state did not load — then every stream is unknown and
+ /// every backing index would look orphaned.
+ fn warn_orphaned_backing_indices(&self) {
+ if !self
+ .cluster_state_loaded
+ .load(std::sync::atomic::Ordering::Acquire)
+ {
+ return;
+ }
+ // Snapshot both maps before comparing them: nothing here may hold a
+ // shard guard on one map while taking one on the other.
+ let backing: Vec<String> = self
+ .indices
+ .iter()
+ .map(|e| e.key().clone())
+ .filter(|n| n.starts_with(".ds-"))
+ .collect();
+ if backing.is_empty() {
+ return;
+ }
+ let streams: Vec<(String, Vec<String>)> = self
+ .data_streams
+ .iter()
+ .map(|s| (s.key().clone(), s.value().backing_indices.clone()))
+ .collect();
+ let claimed = |index: &str| {
+ streams.iter().any(|(stream, indices)| {
+ indices.iter().any(|b| b == index)
+ || index
+ .strip_prefix(&format!(".ds-{stream}-"))
+ .is_some_and(|gen| gen.parse::<u64>().is_ok())
+ })
+ };
+ let mut orphans: Vec<String> = backing.into_iter().filter(|n| !claimed(n)).collect();
+ if orphans.is_empty() {
+ return;
+ }
+ orphans.sort();
+ warn!(
+ count = orphans.len(),
+ indices = orphans.join(","),
+ "backing indices on disk belong to no known data stream — they are \
+ unreachable through the data-stream API and are NOT deleted \
+ automatically; remove each with DELETE /<index> once you have \
+ confirmed the data is not wanted"
+ );
+ }
+
+ /// Insert into a persisted map and durably record the result, rolling the
+ /// in-memory change back if the write fails so a 500 never leaves a
+ /// change that only this process can see.
+ fn persisted_insert<V: Clone>(
+ &self,
+ map: &DashMap<String, V>,
+ name: String,
+ value: V,
+ ) -> Result<()> {
+ self.ensure_cluster_state_writable()?;
+ let previous = map.insert(name.clone(), value);
+ if let Err(e) = self.flush_cluster_state() {
+ match previous {
+ Some(old) => map.insert(name, old),
+ None => map.remove(&name).map(|(_, v)| v),
+ };
+ return Err(e);
+ }
+ Ok(())
+ }
+
+ /// Remove from a persisted map and durably record the result. Returns
+ /// `false` when there was nothing to remove (the caller answers 404);
+ /// an `Err` means the removal did not reach disk and was undone.
+ fn persisted_remove<V: Clone>(&self, map: &DashMap<String, V>, name: &str) -> Result<bool> {
+ // Before the map lookup: an empty map after a failed load would
+ // otherwise turn "I cannot see your configuration" into "it does not
+ // exist" — see `ensure_cluster_state_writable`.
+ self.ensure_cluster_state_writable()?;
+ let Some((key, removed)) = map.remove(name) else {
+ return Ok(false);
+ };
+ if let Err(e) = self.flush_cluster_state() {
+ map.insert(key, removed);
+ return Err(e);
+ }
+ Ok(true)
+ }
+
+ /// Create or replace a v2 index template (`PUT /_index_template/{name}`).
+ pub fn put_index_template(&self, name: String, template: IndexTemplate) -> Result<()> {
+ self.persisted_insert(&self.templates, name, template)
+ }
+
+ /// Delete a v2 index template; `false` when it did not exist.
+ pub fn delete_index_template(&self, name: &str) -> Result<bool> {
+ self.persisted_remove(&self.templates, name)
+ }
+
+ /// Create or replace a legacy v1 template (`PUT /_template/{name}`).
+ pub fn put_legacy_template(&self, name: String, body: Value) -> Result<()> {
+ self.persisted_insert(&self.legacy_templates, name, body)
+ }
+
+ /// Delete a legacy v1 template; `false` when it did not exist.
+ pub fn delete_legacy_template(&self, name: &str) -> Result<bool> {
+ self.persisted_remove(&self.legacy_templates, name)
+ }
+
+ /// Create or replace a component template.
+ pub fn put_component_template(&self, name: String, body: Value) -> Result<()> {
+ self.persisted_insert(&self.component_templates, name, body)
+ }
+
+ /// Delete a component template; `false` when it did not exist.
+ pub fn delete_component_template(&self, name: &str) -> Result<bool> {
+ self.persisted_remove(&self.component_templates, name)
+ }
+
+ /// Create or replace an ILM policy.
+ pub fn put_ilm_policy(&self, name: String, body: Value) -> Result<()> {
+ self.persisted_insert(&self.ilm_policies, name, body)
+ }
+
+ /// Delete an ILM policy; `false` when it did not exist.
+ pub fn delete_ilm_policy(&self, name: &str) -> Result<bool> {
+ self.persisted_remove(&self.ilm_policies, name)
+ }
+
+ /// Durably store an ingest pipeline definition and (re)build its
+ /// executable form. This is the only way a user-visible pipeline should
+ /// ever be created — `compile_pipeline` alone does not reach disk.
+ ///
+ /// `config` is the xerj-shaped pipeline config to compile, and is what
+ /// gets stored (and handed back by `GET /_ingest/pipeline/{id}`) when it
+ /// compiles — the shape this endpoint has always returned.
+ ///
+ /// `keep_if_uncompilable` says what to do when it does *not* compile, and
+ /// differs by surface:
+ ///
+ /// * `Some(doc)` — the ES-compat surface. It has always answered
+ /// `acknowledged` and left the definition readable, so `doc` (the
+ /// ES-shaped body as received) is stored and the compile error is
+ /// returned as `Ok(Some(err))` for the caller to log.
+ /// * `None` — the xerj-native surface, which rejects a definition it
+ /// cannot compile. Nothing is stored, nothing is flushed, and any
+ /// pipeline already registered under this id is left untouched.
+ ///
+ /// Either way a compile failure drops the *executable* form under this
+ /// id. Leaving the old one behind would mean `?pipeline={id}` kept
+ /// running a definition `GET` no longer shows — and stopped running it at
+ /// the next restart. A definition must behave the same before and after a
+ /// reboot; that is the whole point of persisting it.
+ ///
+ /// `Err` means the definition did not reach disk. The in-memory change is
+ /// rolled back first, so the caller can report the write as failed
+ /// without leaving behind a pipeline only this process can see.
+ pub fn put_pipeline(
+ &self,
+ id: &str,
+ config: Value,
+ keep_if_uncompilable: Option<Value>,
+ ) -> Result<Option<xerj_wasm::WasmError>> {
+ // Up front, so a refused write never disturbs the compiled form of a
+ // pipeline that is still running.
+ self.ensure_cluster_state_writable()?;
+ let previous_doc = self.pipelines.get(id).map(|v| v.value().clone());
+ // Taken out up front so a failed compile cannot leave a stale
+ // executable behind; put back verbatim on every abort path.
+ let previous_compiled = self.transform_pipelines.remove(id).map(|(_, p)| p);
++ // `compile_pipeline` clears this on success; snapshot it so a failed
++ // flush can put the previous definition's unrunnable reason back too
++ // (issue #204 — a rolled-back write must not turn "stored but not
++ // runnable, and here is why" into a bare "not found").
++ let previous_reason = self.unrunnable_pipelines.get(id).map(|r| r.value().clone());
+
+ // `compile_pipeline` mutates nothing when it fails, so an abort only
+ // has to undo the two lines above.
+ let compile_err = match self.compile_pipeline(id, config) {
+ Ok(()) => None,
+ Err(e) => match keep_if_uncompilable {
+ None => {
+ if let Some(p) = previous_compiled {
+ self.transform_pipelines.insert(id.to_string(), p);
+ }
+ return Ok(Some(e));
+ }
+ Some(doc) => {
+ self.pipelines.insert(id.to_string(), doc);
++ // Issue #204: the kept definition is visible to `GET`, so
++ // ingest through it must refuse WITH the reason rather
++ // than claim the pipeline is missing.
++ self.unrunnable_pipelines.insert(
++ id.to_string(),
++ format!(
++ "the stored definition does not compile ({e}); no \
++ document can be ingested through pipeline [{id}]"
++ ),
++ );
+ Some(e)
+ }
+ },
+ };
+
+ if let Err(e) = self.flush_cluster_state() {
+ match previous_doc {
+ Some(d) => self.pipelines.insert(id.to_string(), d),
+ None => self.pipelines.remove(id).map(|(_, v)| v),
+ };
+ self.transform_pipelines.remove(id);
+ if let Some(p) = previous_compiled {
+ self.transform_pipelines.insert(id.to_string(), p);
+ }
++ match previous_reason {
++ Some(r) => self.unrunnable_pipelines.insert(id.to_string(), r),
++ None => self.unrunnable_pipelines.remove(id).map(|(_, v)| v),
++ };
+ return Err(e);
+ }
+
+ if let Some(e) = &compile_err {
+ warn!(pipeline = id, error = %e, "pipeline stored but failed to compile");
+ } else {
+ info!(name = id, "transform pipeline created");
+ }
+ Ok(compile_err)
+ }
+
+ /// Delete an ingest pipeline — both the stored definition and the
+ /// compiled form, so `?pipeline={name}` stops working at the same moment
+ /// `GET /_ingest/pipeline/{name}` starts answering 404. `false` when it
+ /// did not exist.
+ pub fn delete_pipeline(&self, name: &str) -> Result<bool> {
+ let removed = self.persisted_remove(&self.pipelines, name)?;
+ if removed {
+ self.transform_pipelines.remove(name);
++ // Issue #204: a deleted pipeline must not leave a stale
++ // "stored but not runnable" reason behind either — the next
++ // definition under this id starts from a clean slate.
++ self.unrunnable_pipelines.remove(name);
+ }
+ Ok(removed)
+ }
+
/// Add an alias pointing to an index.
pub fn add_alias(&self, alias: &str, index: &str) {
let mut entry = self.aliases.entry(alias.to_string()).or_default();
@@@ -2318,41 -3294,19 +3340,45 @@@
// ── Transform pipeline methods ────────────────────────────────────────────
+ /// The `index.default_pipeline` setting for `index`, if it has one.
+ ///
+ /// One implementation for every write path. It used to live only in the
+ /// `_doc` handlers, which is why `_bulk` ignored the setting entirely and
+ /// stored documents untransformed under a `201` while the single-document
+ /// form of the same write refused (issue #204).
+ ///
+ /// Handles the three shapes settings arrive in — nested
+ /// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`
+ /// and flat `{default_pipeline: ..}` — mirroring the `number_of_shards`
+ /// lookup pattern used elsewhere.
+ ///
+ /// `index` is matched literally: an alias has no settings of its own, so a
+ /// write addressed to an alias does not pick up its backing index's default
+ /// pipeline. That is a known gap, identical on every write path.
+ pub fn default_pipeline_for(&self, index: &str) -> Option<String> {
+ self.index_settings.get(index).and_then(|v| {
+ v.get("index")
+ .and_then(|ix| ix.get("default_pipeline"))
+ .or_else(|| v.get("index.default_pipeline"))
+ .or_else(|| v.get("default_pipeline"))
+ .and_then(Value::as_str)
+ .map(str::to_string)
+ })
+ }
+
- /// Compile and register a typed transform pipeline from a JSON config.
+ /// Compile and register a typed transform pipeline from a JSON config,
+ /// **in memory only**.
///
/// `config_json` must be a valid [`PipelineConfig`](xerj_wasm::pipeline::PipelineConfig)
- /// object. The compiled pipeline is stored in `transform_pipelines` and
- /// can be retrieved by name for use at ingest time.
+ /// object. The compiled pipeline is stored in `transform_pipelines` and
+ /// the raw JSON in `pipelines`, so the ES-compatible ingest API can hand
+ /// it back.
///
- /// The raw JSON is also stored in `pipelines` so it can be returned by the
- /// ES-compatible ingest pipeline API.
- pub fn create_pipeline(
+ /// This is the primitive underneath [`Engine::put_pipeline`] and the boot
+ /// path; it deliberately does not touch disk, so it is private. Anything
+ /// that serves a user request must go through `put_pipeline`, or the
+ /// pipeline is gone at the next restart (issue #203).
+ fn compile_pipeline(
&self,
name: &str,
config_json: Value,
@@@ -2362,34 -3316,7 +3388,68 @@@
let pipeline = xerj_wasm::pipeline::Pipeline::from_config(name, &cfg)?;
self.pipelines.insert(name.to_string(), config_json);
self.transform_pipelines.insert(name.to_string(), pipeline);
++ // Issue #204: a successful compile clears any recorded "stored but
++ // not runnable" reason left by a previous definition under this id.
+ self.unrunnable_pipelines.remove(name);
- info!(name, "transform pipeline created");
+ Ok(())
+ }
+
+ /// Store a pipeline definition that could not be compiled, together with
+ /// the reason, and make sure nothing stale is left behind.
+ ///
+ /// Issue #204. This exists for one case: a definition xerj accepts for wire
+ /// compatibility (Elasticsearch does) but cannot execute — because it names
+ /// a processor this build does not implement, or sets an option on one that
+ /// this build cannot honour (a processor-level `if` guard, an `on_failure`
+ /// chain, a `grok` `patterns` array). Storing it keeps
+ /// `GET`/`_simulate` truthful about what was submitted; recording the
+ /// reason is what stops [`Self::process_through_pipeline`] from silently
+ /// passing documents through untransformed. A previously-compiled pipeline
+ /// under the same id is REMOVED — re-defining a pipeline must not leave the
+ /// old stages quietly running under the new definition's name.
- pub fn register_unrunnable_pipeline(&self, name: &str, definition: Value, reason: String) {
++ ///
++ /// Like [`Engine::put_pipeline`], the stored definition is flushed to
++ /// `cluster_state.json` before the caller may acknowledge it (issue
++ /// #203). `Err` means it did not reach disk; the in-memory change is
++ /// rolled back first. The *reason* itself is in-memory only: after a
++ /// restart the boot replay fails to recompile the stored definition and
++ /// re-records why, so the pipeline stays unrunnable — loudly — either way.
++ pub fn register_unrunnable_pipeline(
++ &self,
++ name: &str,
++ definition: Value,
++ reason: String,
++ ) -> Result<()> {
++ self.ensure_cluster_state_writable()?;
++ let previous_doc = self.pipelines.get(name).map(|v| v.value().clone());
++ let previous_compiled = self.transform_pipelines.remove(name).map(|(_, p)| p);
++ let previous_reason = self
++ .unrunnable_pipelines
++ .get(name)
++ .map(|r| r.value().clone());
+ self.pipelines.insert(name.to_string(), definition);
- self.transform_pipelines.remove(name);
+ self.unrunnable_pipelines
+ .insert(name.to_string(), reason.clone());
++ if let Err(e) = self.flush_cluster_state() {
++ match previous_doc {
++ Some(d) => self.pipelines.insert(name.to_string(), d),
++ None => self.pipelines.remove(name).map(|(_, v)| v),
++ };
++ match previous_reason {
++ Some(r) => self.unrunnable_pipelines.insert(name.to_string(), r),
++ None => self.unrunnable_pipelines.remove(name).map(|(_, v)| v),
++ };
++ if let Some(p) = previous_compiled {
++ self.transform_pipelines.insert(name.to_string(), p);
++ }
++ return Err(e);
++ }
+ warn!(
+ name,
+ reason = reason.as_str(),
+ "ingest pipeline stored but NOT runnable — documents may not be ingested through it"
+ );
+ Ok(())
}
/// Run `docs` through a named pipeline, returning `(action, doc)` pairs.