Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(api,engine,fts,wasm): make "every ingest path refuses loudly" true, and disclose the analysis break

xerj-orgcda86c032a9e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 00ff4fc62..060973e70 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,8 +7,140 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Changed
+
+- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot
+ honour, instead of accepting it and analysing differently**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an
+ existing index is opened, where a hard failure would brick it), so an
+ unresolvable tokenizer silently became `standard` and an unresolvable token
+ filter was dropped — the index was created, `GET /_settings` echoed the block
+ back, and the field was analysed in a way nobody asked for. Index creation
+ now validates the block and answers `400` with the exact constructs it cannot
+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused
+ (it only ever changed what `GET /_settings` echoed; the registry is built
+ once, at create/open, and there is no rebuild path — Elasticsearch also
+ refuses analysis updates on an open index).
+
+ **This rejects settings blocks that XERJ used to accept with a `200`, and the
+ ES-YAML conformance suite does not cover them — CI being green is not
+ evidence that your index still creates.** What is honoured:
+
+ | | honoured (still `200`) |
+ |---|---|
+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |
+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |
+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |
+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |
+ | `analysis.normalizer` | nothing — never built |
+
+ Anything else in those positions is now a `400`. There is one important
+ exception to read first: XERJ resolves a declared filter or tokenizer by
+ **name**, not by `type`, so a declaration whose *name* is a built-in is still
+ honoured whatever its type says. Measured, all still `200`:
+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,
+ `{"lowercase": {"type": "lowercase"}}`,
+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`
+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which
+ has a regression test pinning it. Naming a built-in analyzer on a field
+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.
+
+ What now fails is a declaration under a name that does NOT resolve — which is
+ the usual ES habit of naming filters after what they do in *your* index.
+ Measured blocks that returned `200` in rc.14 and return `400` from this
+ release:
+
+ - a `keyword` field's `normalizer` (any);
+ - a `char_filter` (any, including `html_strip` and `mapping`), declared
+ either under `analysis.char_filter` or as an analyzer's `char_filter`;
+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,
+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,
+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,
+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,
+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,
+ which never lowercased anything, because the analyzer referencing
+ `my_lowercase` found no such filter and dropped it;
+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,
+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,
+ `letter`, `lowercase` or `simple_pattern`;
+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,
+ `fingerprint`, or a language other than `english`;
+ - a `synonym` filter whose `synonyms` is a bare string rather than an array
+ (it built a filter with zero rules).
+
+ **Upgrading:** there is no opt-out flag. Either remove the analysis block and
+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only
+ the constructs in the table. The `400` names every offending construct in one
+ response, so one round-trip is enough to see the whole list. **Two paths are
+ not covered and still accept these blocks silently: an index template's
+ `template.settings.analysis` is dropped wholesale before the gate ever sees
+ it, and option-level divergence inside a supported type (`synonyms_path`,
+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**
+
### Fixed
+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and
+ every write path enforces it**
+ ([#204](https://github.com/xerj-org/xerj/issues/204)).
+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only
+ warn-log a compile failure, so a pipeline that could never run was
+ acknowledged and echoed back by `GET`. It now compiles first, with three
+ outcomes: compiled → stored and acknowledged; a processor config we refuse →
+ `400`; a processor or option this build cannot honour → accepted (as
+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses
+ rather than storing an untransformed document.
+
+ Defects this surfaced, each of which answered `200`/`201` and did something
+ smaller than asked: **every** ES `rename` processor failed to compile (the
+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and
+ renamed nothing; ES `append` was executed as `set`, replacing the field
+ instead of extending it, and disagreeing with `_simulate`, which implemented
+ a real append; `set.override` defaulted to `false` where Elasticsearch
+ defaults it to `true`; an unknown `pii_redaction` type built an empty pattern
+ set and redacted nothing; an unknown `grok` pattern name silently became the
+ catch-all; an unknown `convert` target left the field unconverted per
+ document; array-typed keys given as a bare string were dropped and the stage
+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the
+ definition while leaving the compiled pipeline live under the same name.
+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`
+ rather than being refused.
+
+ Two more of the same, at the definition's own edges: a **pipeline-level**
+ `on_failure` block (the top-level body key, not the processor-level one) was
+ accepted, echoed by `GET`, and never run — the pipeline is now recorded
+ unrunnable, exactly as its processor-level twin already was; and `set` with
+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted
+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:
+ missing value`, which told the caller their valid ES pipeline was malformed.
+
+ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**
+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,
+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`
+ previously read the instruction and dropped it — a redaction pipeline that
+ the single-document endpoint refused was ignored in bulk, and the document
+ was stored verbatim under `errors: false`. All four now run the pipeline, and
+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the
+ name before touching a single document, so a bad pipeline fails the request
+ instead of half of it.
+
+ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`
+ copied at most 10,000 documents and swallowed every per-document write error
+ before answering `{"acknowledged": true}` (now full keyset paging, and any
+ failed write — or a failed source flush — fails the request); and a failed
+ `_bulk` item no longer reports `"result": "deleted"` alongside its own
+ `status: 400`, which claimed a deletion that never happened.
+
+ **Known gaps, stated rather than closed:** `_simulate` interprets the stored
+ ES body directly and covers processors the compiled path does not, so a
+ pipeline recorded as unrunnable can still show a transformation under
+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`
+ returns the ES body with no runnability signal, so an operator only learns at
+ write time; a write addressed to an **alias** does not pick up the backing
+ index's `default_pipeline`; and `_update_by_query` reports a `drop`
+ processor as a per-document failure, because it updates in place and has no
+ delete path.
+
- **`autoindex` no longer leaves immortal catalog entries for skipped files**
([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after the
resume plan was frozen is skipped and reported in the catalog — but that
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 8a55caf30..d5ca885e0 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -2201,10 +2201,9 @@ pub struct IndexDocAutoParams {
/// - `Some(name)` → that pipeline (an explicit `?pipeline=` always wins).
/// - `None` → the index's `index.default_pipeline` setting, if any.
///
-/// The default is read from `engine.index_settings`, handling the nested
-/// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`,
-/// and flat `{default_pipeline: ..}` shapes — mirroring the `number_of_shards`
-/// lookup pattern used elsewhere in this module.
+/// The default is read by [`xerj_engine::Engine::default_pipeline_for`], which
+/// `_bulk` also uses — one lookup, so the two forms of the same write cannot
+/// disagree about which pipeline applies.
fn resolve_effective_pipeline(
state: &AppState,
index: &str,
@@ -2213,14 +2212,7 @@ fn resolve_effective_pipeline(
match explicit {
Some("_none") => None,
Some(p) => Some(p.to_string()),
- None => state.engine.index_settings.get(index).and_then(|v| {
- v.get("index")
- .and_then(|ix| ix.get("default_pipeline"))
- .or_else(|| v.get("index.default_pipeline"))
- .or_else(|| v.get("default_pipeline"))
- .and_then(Value::as_str)
- .map(|s| s.to_string())
- }),
+ None => state.engine.default_pipeline_for(index),
}
}
@@ -2295,10 +2287,9 @@ pub async fn index_doc_auto(
}
transformed
}
- Ok(_) => {
- // Pipeline returned empty results — pass through original
- serde_json::from_slice(&body).unwrap_or(Value::Null)
- }
+ // No output for this document: refused, not passed through
+ // unchanged — see `pipeline_empty_result_error`.
+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),
// Issue #204, the purest form of the class on a user-DATA path:
// this used to warn and then index the ORIGINAL document. A
// pipeline whose job was to redact PII, drop a field or route the
@@ -2426,9 +2417,9 @@ pub async fn index_doc(
}
transformed
}
- // Pipeline produced no output for this document — index the
- // original body, which is what it asked for.
- Ok(_) => doc,
+ // No output for this document: refused, not passed through
+ // unchanged — see `pipeline_empty_result_error`.
+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),
// Issue #204: this arm was folded into `_ => doc`, so a pipeline
// that could not run left the document indexed UNTRANSFORMED under
// a success response. Refuse instead.
@@ -2553,9 +2544,9 @@ pub async fn create_doc(
}
transformed
}
- // Pipeline produced no output for this document — index the
- // original body, which is what it asked for.
- Ok(_) => doc,
+ // No output for this document: refused, not passed through
+ // unchanged — see `pipeline_empty_result_error`.
+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),
// Issue #204: this arm was folded into `_ => doc`, so a pipeline
// that could not run left the document indexed UNTRANSFORMED under
// a success response. Refuse instead.
@@ -13054,6 +13045,10 @@ fn bulk_opts_from_query(
_ => None,
}),
default_source_req,
+ // `?pipeline=` — the request-level default for every `index`/`create`
+ // action that names no pipeline of its own. Read but discarded before
+ // (issue #204): the bulk item was written untransformed under a `201`.
+ default_pipeline: qp.get("pipeline").cloned(),
}
}
@@ -13182,7 +13177,14 @@ async fn process_bulk_body(
index: item.index,
id: item.id,
version,
- result: item.result.unwrap_or_else(|| "deleted".to_string()),
+ // A successful action with no explicit result is a `delete`
+ // (the only one that does not set one). A FAILED action has no
+ // result at all — see `EsBulkItemResult::result`.
+ result: if item.error.is_some() {
+ None
+ } else {
+ Some(item.result.clone().unwrap_or_else(|| "deleted".to_string()))
+ },
shards: crate::responses::EsShards::single_success(),
seq_no,
primary_term: 1,
@@ -16506,6 +16508,13 @@ pub async fn index_stats(
// Aggregate across multiple indices.
let mut all_doc_count = 0u64;
let mut all_store_bytes = 0u64;
+ // Aggregated so `GET /_all/_stats` carries the same
+ // `primaries.mappings.schema_persist_failures` the per-index form does.
+ // Without it `Index::schema_persist_failures`'s doc-comment named an
+ // endpoint that did not report the counter, and a fleet-wide health
+ // check reading only `_all` saw nothing (issue #204: the claim has to
+ // match what the endpoint really answers).
+ let mut all_schema_persist_failures = 0u64;
let mut all_indices = serde_json::Map::new();
for name in &targets {
let stats = match state.engine.index_stats(name).await {
@@ -16545,6 +16554,7 @@ pub async fn index_stats(
.get_index(name)
.map(|idx| idx.schema_persist_failures())
.unwrap_or(0);
+ all_schema_persist_failures += schema_persist_failures;
let primaries = json!({
"docs": { "count": doc_count, "deleted": 0 },
"store": { "size_in_bytes": store_size_bytes },
@@ -16614,6 +16624,7 @@ pub async fn index_stats(
let all_primaries = json!({
"docs": { "count": all_doc_count, "deleted": 0 },
"store": { "size_in_bytes": all_store_bytes },
+ "mappings": { "schema_persist_failures": all_schema_persist_failures },
"dense_vector": {
"value_count": all_dv_value_count,
"off_heap": {
@@ -18309,6 +18320,14 @@ fn reindex_default_size() -> usize {
#[derive(Debug, Deserialize)]
pub struct ReindexDest {
pub index: String,
+ /// `dest.pipeline` — the ingest pipeline every copied document is run
+ /// through before it is written to the destination.
+ ///
+ /// Issue #204: this field did not exist, so serde dropped it and a
+ /// `_reindex` asking for redaction on the way in answered
+ /// `{"created": N}` having copied every document VERBATIM.
+ #[serde(default)]
+ pub pipeline: Option<String>,
}
pub async fn reindex(
@@ -18356,6 +18375,41 @@ pub async fn reindex(
let source_name = &body.source.index;
let dest_name = &body.dest.index;
+ // `dest.pipeline` is checked BEFORE a single document is copied. A name
+ // that does not resolve, or one this build cannot run, fails the whole
+ // request with nothing written — rather than copying the corpus
+ // untransformed and reporting success (issue #204). `_none` means "no
+ // pipeline", matching the write paths.
+ let dest_pipeline: Option<String> = match body.dest.pipeline.as_deref() {
+ None | Some("_none") => None,
+ Some(p) => {
+ let reason = if !state.engine.pipelines.contains_key(p) {
+ Some(format!("pipeline with id [{p}] does not exist"))
+ } else {
+ state
+ .engine
+ .unrunnable_pipelines
+ .get(p)
+ .map(|r| format!("pipeline with id [{p}] cannot be run: {}", r.value()))
+ };
+ if let Some(reason) = reason {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(json!({
+ "error": {
+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],
+ "type": "illegal_argument_exception",
+ "reason": reason,
+ },
+ "status": 400,
+ })),
+ )
+ .into_response();
+ }
+ Some(p.to_string())
+ }
+ };
+
// Get or create destination index.
let dest_idx = match state.engine.get_or_create_index(dest_name) {
Ok(i) => i,
@@ -18374,7 +18428,19 @@ pub async fn reindex(
// reliably, so un-flushed recent writes would be skipped or duplicated
// across pages. Reindex is a snapshot-style bulk op, so a single flush up
// front is an acceptable cost and makes the whole source keyset-pageable.
- let _ = source_idx.flush().await;
+ //
+ // Propagated, not swallowed, for the same reason as `clone_index_to`: the
+ // flush is a precondition of the paging being complete, so a failed flush
+ // must not be followed by a reindex that reports `"total": N` over a source
+ // it could not read consistently (issue #204).
+ if let Err(e) = source_idx.flush().await {
+ return ApiError::new(xerj_common::XerjError::internal(format!(
+ "reindex from [{source_name}] into [{dest_name}] was not attempted: flushing \
+ the source failed ({e}); the copy needs a flushed source to page over it \
+ without skipping or duplicating documents"
+ )))
+ .into_response();
+ }
let query_val = body
.source
@@ -18391,6 +18457,7 @@ pub async fn reindex(
let mut total_fetched = 0usize;
let mut created = 0usize;
let mut updated = 0usize;
+ let mut noops = 0usize;
let mut failures: Vec<Value> = Vec::new();
let mut batches = 0usize;
// Keyset cursor: `[last_id]` of the previous page (the sole sort key is
@@ -18441,6 +18508,49 @@ pub async fn reindex(
for hit in results.hits {
if !hit.source.is_null() {
+ // `dest.pipeline`, if the request named one. Already known to
+ // exist and to be runnable (checked before any copy started),
+ // so anything that goes wrong here is a per-document failure
+ // and is reported as one — never a pass-through of the
+ // untransformed source.
+ let doc_source = match dest_pipeline.as_deref() {
+ None => hit.source,
+ Some(p) => match state.engine.process_through_pipeline(p, vec![hit.source]) {
+ Ok(mut results) if !results.is_empty() => {
+ let (action, transformed) = results.remove(0);
+ if matches!(action, xerj_wasm::pipeline::ProcessAction::Drop) {
+ // ES: a pipeline that drops the document does
+ // not write it, and the run reports a noop.
+ noops += 1;
+ continue;
+ }
+ transformed
+ }
+ Ok(_) => {
+ failures.push(json!({
+ "id": hit.id,
+ "cause": {
+ "type": "reindex_error",
+ "reason": format!(
+ "pipeline [{p}] returned no document for [{}]",
+ hit.id
+ ),
+ },
+ }));
+ continue;
+ }
+ Err(e) => {
+ failures.push(json!({
+ "id": hit.id,
+ "cause": {
+ "type": "reindex_error",
+ "reason": format!("pipeline [{p}] failed: {e}"),
+ },
+ }));
+ continue;
+ }
+ },
+ };
// Check if doc already exists in dest to track created vs updated.
let exists = dest_idx
.get_document(&hit.id)
@@ -18449,7 +18559,7 @@ pub async fn reindex(
.flatten()
.is_some();
match dest_idx
- .index_document(Some(hit.id.clone()), hit.source)
+ .index_document(Some(hit.id.clone()), doc_source)
.await
{
Ok(_) => {
@@ -18490,7 +18600,7 @@ pub async fn reindex(
"batches": batches,
"failures": failures,
"throttled_millis": 0,
- "noops": 0,
+ "noops": noops,
"version_conflicts": 0,
"retries": { "bulk": 0, "search": 0 },
}))
@@ -21245,6 +21355,42 @@ pub async fn update_by_query(
// update happens in place (no duplicate-`_id` docs are appended).
let script = body.script.as_ref().map(extract_update_script);
+ // `?pipeline=` — every matched document is run through it before being
+ // written back. Issue #204: the parameter used to be read by nothing, so
+ // `_update_by_query?pipeline=redact` reported `"updated": N` having
+ // rewritten every document EXACTLY as it was. Resolved and validated
+ // before any document is touched: an unknown or unrunnable name fails the
+ // whole request rather than half of it.
+ let pipeline = match params.get("pipeline").map(String::as_str) {
+ None | Some("_none") => None,
+ Some(p) => {
+ let reason = if !state.engine.pipelines.contains_key(p) {
+ Some(format!("pipeline with id [{p}] does not exist"))
+ } else {
+ state
+ .engine
+ .unrunnable_pipelines
+ .get(p)
+ .map(|r| format!("pipeline with id [{p}] cannot be run: {}", r.value()))
+ };
+ if let Some(reason) = reason {
+ return (
+ StatusCode::BAD_REQUEST,
+ Json(json!({
+ "error": {
+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],
+ "type": "illegal_argument_exception",
+ "reason": reason,
+ },
+ "status": 400,
+ })),
+ )
+ .into_response();
+ }
+ Some((state.engine.clone(), p.to_string()))
+ }
+ };
+
// ES defaults `wait_for_completion` to true (synchronous); only an
// explicit `false` switches to the async `{"task": "node:id"}` form.
let wait_for_completion = params
@@ -21260,14 +21406,14 @@ pub async fn update_by_query(
// Same reasoning as the `_delete_by_query` spawn above: the task owns
// one already-authorized `Index` handle and can reach no other index.
tokio::spawn(async move {
- let response = run_update_by_query(&idx, &search_req, script).await;
+ let response = run_update_by_query(&idx, &search_req, script, pipeline).await;
tasks.complete(&spawned_key, response);
drop(handle);
});
return Json(json!({ "task": task_key })).into_response();
}
- let response = run_update_by_query(&idx, &search_req, script).await;
+ let response = run_update_by_query(&idx, &search_req, script, pipeline).await;
drop(handle);
by_query_response(response)
}
@@ -21280,10 +21426,46 @@ pub async fn update_by_query(
/// `wait_for_completion=false` form.
const SCRIPTED_UPDATE_BUDGET_MS: u64 = 30_000;
+/// Run `doc` through `pipeline`, or say why it could not be run.
+///
+/// Issue #204: every caller must get a document that really went through the
+/// pipeline, or an error. There is deliberately no "return the input unchanged"
+/// branch — that is precisely how an unrun redaction became a `200`.
+///
+/// A `drop` action is an error here rather than a deletion:
+/// `_update_by_query` updates documents in place and has no delete path, so
+/// honouring `drop` is not something this endpoint can do. Saying so per
+/// document beats writing the document as if the processor had not asked.
+fn apply_pipeline_or_fail(
+ engine: &xerj_engine::Engine,
+ pipeline: &str,
+ doc_id: &str,
+ doc: Value,
+) -> std::result::Result<Value, String> {
+ match engine.process_through_pipeline(pipeline, vec![doc]) {
+ Ok(mut results) if !results.is_empty() => {
+ let (action, transformed) = results.remove(0);
+ if matches!(action, xerj_wasm::pipeline::ProcessAction::Drop) {
+ return Err(format!(
+ "pipeline [{pipeline}] dropped document [{doc_id}]; _update_by_query \
+ updates in place and cannot delete it, so the document was left \
+ unchanged"
+ ));
+ }
+ Ok(transformed)
+ }
+ Ok(_) => Err(format!(
+ "pipeline [{pipeline}] returned no document for [{doc_id}]"
+ )),
+ Err(e) => Err(format!("pipeline [{pipeline}] failed: {e}")),
+ }
+}
+
async fn run_update_by_query(
idx: &xerj_engine::Index,
search_req: &xerj_query::SearchRequest,
script: Option<(String, Value)>,
+ pipeline: Option<(std::sync::Arc<xerj_engine::Engine>, String)>,
) -> Value {
let started = Instant::now();
@@ -21343,6 +21525,13 @@ async fn run_update_by_query(
let result = idx
.transform_document_serialized(&hit.id, None, |mut current| {
apply_painless_update(&mut current, src, params)?;
+ // ES runs `?pipeline=` on the result of the script,
+ // and so do we — inside the same serialized
+ // transform, so the pipeline sees exactly the
+ // document that is about to be written.
+ if let Some((engine, name)) = pipeline.as_ref() {
+ current = apply_pipeline_or_fail(engine, name, &hit.id, current)?;
+ }
Ok::<_, String>(current)
})
.await;
@@ -21359,8 +21548,24 @@ async fn run_update_by_query(
})),
}
} else {
- // No script: preserve the historical re-index-in-place behavior.
- match idx.index_document(Some(hit.id.clone()), hit.source).await {
+ // No script: preserve the historical re-index-in-place
+ // behavior, with `?pipeline=` applied when one was named.
+ let doc = match pipeline.as_ref() {
+ None => hit.source,
+ Some((engine, name)) => {
+ match apply_pipeline_or_fail(engine, name, &hit.id, hit.source) {
+ Ok(d) => d,
+ Err(reason) => {
+ failures.push(json!({
+ "id": hit.id,
+ "cause": { "reason": reason },
+ }));
+ continue;
+ }
+ }
+ }
+ };
+ match idx.index_document(Some(hit.id.clone()), doc).await {
Ok(_) => updated += 1,
Err(e) => {
failures.push(json!({
@@ -21483,6 +21688,7 @@ mod scripted_update_publication_tests {
&idx,
&xerj_query::SearchRequest::default(),
Some(("ctx._source.n += 1".into(), json!({}))),
+ None,
)
.await
}));
@@ -21761,6 +21967,21 @@ fn pipeline_execution_error(pipeline: &str, e: &xerj_wasm::WasmError) -> ApiErro
)))
}
+/// A pipeline that returned no document for the one it was given.
+///
+/// `process_through_pipeline` zips one action per input document, so this is an
+/// internal invariant break rather than a configuration a caller can write.
+/// It still must not fall through to indexing the ORIGINAL document: "the
+/// pipeline produced nothing, so store what the caller sent" is the same
+/// silent-pass-through this whole change exists to remove, and it would land on
+/// exactly the redaction case where it costs the most.
+fn pipeline_empty_result_error(pipeline: &str) -> ApiError {
+ tracing::error!(pipeline = %pipeline, "refusing write: ingest pipeline returned no document");
+ ApiError::new(xerj_common::XerjError::config(format!(
+ "document not indexed: ingest pipeline [{pipeline}] returned no document"
+ )))
+}
+
pub async fn put_ingest_pipeline(
State(state): State<AppState>,
Path(id): Path<String>,
@@ -21884,6 +22105,27 @@ pub async fn put_ingest_pipeline(
} else {
body.clone()
};
+ // Elasticsearch pipeline-level (top-level body) keys that decide what
+ // happens when a processor fails. The compiled pipeline has no equivalent:
+ // `PipelineConfig` carries a single `on_error` policy and no recovery
+ // chain, so a top-level `on_failure` was read by nothing.
+ //
+ // This is the SAME defect the repair fixed one level down
+ // (`UNHONOURED_PROCESSOR_KEYS` in `xerj-wasm/src/pipeline.rs`): refusing a
+ // processor-level `on_failure` while silently dropping the identical key at
+ // pipeline level would have left the endpoint inconsistent with itself.
+ // `description`, `version`, `_meta` and `deprecated` are NOT here — they
+ // are identification metadata that changes no document, the same line drawn
+ // for processor-level `tag`/`description`.
+ const UNHONOURED_PIPELINE_KEYS: &[(&str, &str)] = &[(
+ "on_failure",
+ "the pipeline-level recovery processors are not run; the compiled pipeline's \
+ `on_error` policy applies instead",
+ )];
+ let unhonoured_top_level = UNHONOURED_PIPELINE_KEYS
+ .iter()
+ .find(|(key, _)| body.get(*key).is_some())
+ .copied();
// Three outcomes, and the difference between them is the whole point of
// issue #204:
//
@@ -21902,6 +22144,21 @@ pub async fn put_ingest_pipeline(
// where a document would otherwise be silently
// mis-ingested.
match state.engine.create_pipeline(&id, xerj_config) {
+ Ok(()) if unhonoured_top_level.is_some() => {
+ // Every stage compiled, but a top-level key we cannot act on came
+ // with it. `register_unrunnable_pipeline` drops the compiled form
+ // `create_pipeline` just installed, so nothing runs half-honoured.
+ let (option, reason) = unhonoured_top_level.expect("guarded by the match arm");
+ state.engine.register_unrunnable_pipeline(
+ &id,
+ body,
+ format!(
+ "pipeline-level [{option}] is not supported by this xerj build \
+ ({reason}); no document can be ingested through pipeline [{id}]"
+ ),
+ );
+ Json(json!({ "acknowledged": true })).into_response()
+ }
Ok(()) => {
// `create_pipeline` stores the *translated* config; put the ES
// body back so GET and _simulate see what was actually submitted.
@@ -25151,7 +25408,19 @@ async fn clone_index_to(state: &AppState, source: &str, target: &str) -> Result<
// `max_result_window`. The source is flushed first because the in-memory
// memtable's sort path does not order by `_id` reliably, so un-flushed
// writes would be skipped or duplicated across pages.
- let _ = src_idx.flush().await;
+ //
+ // That makes the flush a CORRECTNESS precondition of the paging below, not
+ // a best-effort hint — so it is propagated rather than swallowed. A
+ // `let _ =` here would have produced exactly the defect this function was
+ // rewritten to remove: a silently partial (or duplicated) copy answering
+ // `{"acknowledged": true, "shards_acknowledged": true}`.
+ src_idx.flush().await.map_err(|e| {
+ ApiError::new(xerj_common::XerjError::internal(format!(
+ "clone of [{source}] into [{target}] was not attempted: flushing the source \
+ failed ({e}); the copy needs a flushed source to page over it without \
+ skipping or duplicating documents"
+ )))
+ })?;
const CLONE_PAGE_SIZE: usize = 1_000;
let mut search_after: Option<Value> = None;
@@ -34102,6 +34371,7 @@ mod reindex_keyset_tests {
},
dest: ReindexDest {
index: "dst".into(),
+ pipeline: None,
},
};
let resp = reindex(State(state.clone()), Json(body))
diff --git a/engine/crates/xerj-api/src/responses.rs b/engine/crates/xerj-api/src/responses.rs
index 8e7a75818..f1137b84f 100644
--- a/engine/crates/xerj-api/src/responses.rs
+++ b/engine/crates/xerj-api/src/responses.rs
@@ -437,7 +437,18 @@ pub struct EsBulkItemResult {
pub id: String,
#[serde(rename = "_version")]
pub version: u64,
- pub result: String,
+ /// `"created"` / `"updated"` / `"deleted"` / `"noop"`, and ABSENT on an
+ /// item that failed.
+ ///
+ /// Issue #204: this was a bare `String` filled from
+ /// `item.result.unwrap_or_else(|| "deleted".to_string())`, so every failed
+ /// bulk item — a rejected pipeline, a mapping error, a version conflict —
+ /// was reported as `"result": "deleted"` alongside its own `status: 400`.
+ /// A client reading `result` was told the document had been deleted when
+ /// nothing had been written or removed. Elasticsearch omits the field on a
+ /// failed item; so do we.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub result: Option<String>,
#[serde(rename = "_shards")]
pub shards: EsShards,
#[serde(rename = "_seq_no")]
diff --git a/engine/crates/xerj-api/tests/fail_closed_config_http.rs b/engine/crates/xerj-api/tests/fail_closed_config_http.rs
index 5a3914641..0ab8ea5b9 100644
--- a/engine/crates/xerj-api/tests/fail_closed_config_http.rs
+++ b/engine/crates/xerj-api/tests/fail_closed_config_http.rs
@@ -394,3 +394,337 @@ async fn clone_copies_every_document_not_the_first_ten_thousand() {
"clone must copy every document: {count}"
);
}
+
+// ── Every ingest path, not just the single-document one ──────────────────────
+//
+// The first cut of this branch made `PUT /_ingest/pipeline` mean "compiled ⇒
+// xerj honours this" and wired the refusal into the three single-document
+// handlers only. Measured against that build, `_bulk?pipeline=`, an index
+// carrying `index.default_pipeline`, `_reindex` `dest.pipeline` and
+// `_update_by_query?pipeline=` all answered 200/201 and stored the document
+// UNTRANSFORMED — including for a pipeline that was perfectly runnable.
+
+fn ndjson(path: &str, body: &str) -> Request<Body> {
+ Request::builder()
+ .method("POST")
+ .uri(path)
+ .header("content-type", "application/x-ndjson")
+ .body(Body::from(body.to_string()))
+ .expect("request")
+}
+
+/// `{"set": {"field": "env", "value": "prod"}}` — compiles and runs.
+async fn define_runnable_pipeline(app: &axum::Router, id: &str) {
+ let (status, body) = send(
+ app,
+ put(
+ &format!("/_ingest/pipeline/{id}"),
+ json!({ "processors": [{ "set": { "field": "env", "value": "prod" } }] }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+}
+
+/// An ES `grok` `patterns` array — accepted by `PUT`, recorded unrunnable.
+async fn define_unrunnable_pipeline(app: &axum::Router, id: &str) {
+ let (status, body) = send(
+ app,
+ put(
+ &format!("/_ingest/pipeline/{id}"),
+ json!({ "processors": [{ "grok": {
+ "field": "message",
+ "patterns": ["%{IP:client} %{WORD:method}"]
+ } }] }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["acknowledged"], true, "{body}");
+}
+
+#[tokio::test]
+async fn bulk_runs_the_url_level_pipeline() {
+ let (app, _dir) = app();
+ define_runnable_pipeline(&app, "setp").await;
+
+ let (status, body) = send(
+ &app,
+ ndjson(
+ "/_bulk?pipeline=setp&refresh=true",
+ "{\"index\":{\"_index\":\"b\",\"_id\":\"1\"}}\n{\"a\":1}\n",
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["errors"], false, "{body}");
+
+ let (_, doc) = send(&app, get("/b/_doc/1")).await;
+ // Pre-fix: `{"a": 1}` — the parameter was read by nothing.
+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");
+}
+
+#[tokio::test]
+async fn bulk_refuses_an_unrunnable_url_level_pipeline() {
+ let (app, _dir) = app();
+ define_unrunnable_pipeline(&app, "unrun").await;
+
+ let (status, body) = send(
+ &app,
+ ndjson(
+ "/_bulk?pipeline=unrun",
+ "{\"index\":{\"_index\":\"b\",\"_id\":\"1\"}}\n{\"message\":\"10.0.0.1 GET\"}\n",
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["errors"], true, "{body}");
+ assert_eq!(body["items"][0]["index"]["status"], 400, "{body}");
+ // A failed item carries no `result`. Pre-fix it said `"deleted"`, which is
+ // a statement about the document that was not true.
+ assert!(
+ body["items"][0]["index"].get("result").is_none(),
+ "a failed item must not claim a result: {body}"
+ );
+
+ let (status, _) = send(&app, get("/b/_doc/1")).await;
+ assert_ne!(status, StatusCode::OK, "nothing may have been written");
+}
+
+#[tokio::test]
+async fn bulk_honours_index_default_pipeline() {
+ let (app, _dir) = app();
+ define_runnable_pipeline(&app, "setp").await;
+ let (status, body) = send(
+ &app,
+ put(
+ "/dp",
+ json!({ "settings": { "index": { "default_pipeline": "setp" } } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+
+ let (_, body) = send(
+ &app,
+ ndjson(
+ "/_bulk?refresh=true",
+ "{\"index\":{\"_index\":\"dp\",\"_id\":\"1\"}}\n{\"a\":1}\n",
+ ),
+ )
+ .await;
+ assert_eq!(body["errors"], false, "{body}");
+ let (_, doc) = send(&app, get("/dp/_doc/1")).await;
+ // Pre-fix: `_bulk` never consulted the setting, so this was `{"a": 1}`
+ // while `PUT /dp/_doc/1` on the same index ran the pipeline.
+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");
+
+ // `_none` still disables it, as in Elasticsearch.
+ let (_, body) = send(
+ &app,
+ ndjson(
+ "/_bulk?pipeline=_none&refresh=true",
+ "{\"index\":{\"_index\":\"dp\",\"_id\":\"2\"}}\n{\"a\":2}\n",
+ ),
+ )
+ .await;
+ assert_eq!(body["errors"], false, "{body}");
+ let (_, doc) = send(&app, get("/dp/_doc/2")).await;
+ assert!(doc["_source"].get("env").is_none(), "{doc}");
+}
+
+#[tokio::test]
+async fn reindex_runs_and_validates_dest_pipeline() {
+ let (app, _dir) = app();
+ define_runnable_pipeline(&app, "setp").await;
+ define_unrunnable_pipeline(&app, "unrun").await;
+ let (status, _) = send(&app, put("/rsrc/_doc/1?refresh=true", json!({ "a": 1 }))).await;
+ assert!(status.is_success());
+
+ // Unrunnable: refused before anything is copied.
+ let (status, body) = send(
+ &app,
+ post(
+ "/_reindex",
+ json!({ "source": { "index": "rsrc" },
+ "dest": { "index": "rbad", "pipeline": "unrun" } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");
+ let (status, _) = send(&app, get("/rbad/_count")).await;
+ assert_ne!(status, StatusCode::OK, "no destination may have been made");
+
+ // Runnable: every copied document really went through it. Pre-fix
+ // `dest.pipeline` was not even a field on the request struct, so serde
+ // dropped it and the corpus was copied verbatim under `"created": 1`.
+ let (status, body) = send(
+ &app,
+ post(
+ "/_reindex",
+ json!({ "source": { "index": "rsrc" },
+ "dest": { "index": "rdst", "pipeline": "setp" } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["created"], 1, "{body}");
+ let (_, doc) = send(&app, get("/rdst/_doc/1")).await;
+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");
+}
+
+#[tokio::test]
+async fn update_by_query_runs_and_validates_its_pipeline() {
+ let (app, _dir) = app();
+ define_runnable_pipeline(&app, "setp").await;
+ define_unrunnable_pipeline(&app, "unrun").await;
+ let (status, _) = send(&app, put("/ubq/_doc/1?refresh=true", json!({ "a": 1 }))).await;
+ assert!(status.is_success());
+
+ let (status, body) = send(
+ &app,
+ post(
+ "/ubq/_update_by_query?pipeline=unrun",
+ json!({ "query": { "match_all": {} } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");
+ let (_, doc) = send(&app, get("/ubq/_doc/1")).await;
+ assert_eq!(doc["_version"], 1, "nothing may have been rewritten: {doc}");
+
+ let (status, body) = send(
+ &app,
+ post(
+ "/ubq/_update_by_query?pipeline=setp&refresh=true",
+ json!({ "query": { "match_all": {} } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["updated"], 1, "{body}");
+ let (_, doc) = send(&app, get("/ubq/_doc/1")).await;
+ // Pre-fix: `"updated": 1` and the document byte-identical to what it was.
+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");
+}
+
+#[tokio::test]
+async fn a_pipeline_level_on_failure_block_is_accepted_and_then_refuses_the_write() {
+ let (app, _dir) = app();
+ // The same key one level up from the processor-level `on_failure` the
+ // first cut refused. Measured against that build: `PUT` answered 200, `GET`
+ // echoed the block back, and the write succeeded with the recovery chain
+ // silently absent.
+ let (status, body) = send(
+ &app,
+ put(
+ "/_ingest/pipeline/plof",
+ json!({
+ "processors": [{ "set": { "field": "env", "value": "prod" } }],
+ "on_failure": [{ "set": { "field": "err", "value": "yes" } }],
+ }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["acknowledged"], true, "{body}");
+
+ let (status, body) = send(&app, put("/plof/_doc/1?pipeline=plof", json!({ "a": 1 }))).await;
+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");
+ assert!(
+ body["error"]["reason"]
+ .as_str()
+ .is_some_and(|r| r.contains("on_failure")),
+ "{body}"
+ );
+}
+
+#[tokio::test]
+async fn set_copy_from_is_a_capability_gap_not_a_caller_error() {
+ let (app, _dir) = app();
+ // `copy_from` is valid Elasticsearch 7.x+. Refusing the DEFINITION with
+ // `mapper_parsing_exception: missing value` told the caller their correct
+ // pipeline was malformed; the gap is xerj's, so it takes the same
+ // accept-then-refuse-the-write route as every other unsupported option.
+ let (status, body) = send(
+ &app,
+ put(
+ "/_ingest/pipeline/cf",
+ json!({ "processors": [{ "set": { "field": "b", "copy_from": "a" } }] }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(body["acknowledged"], true, "{body}");
+
+ let (status, body) = send(&app, put("/cf/_doc/1?pipeline=cf", json!({ "a": 1 }))).await;
+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");
+ assert!(
+ body["error"]["reason"]
+ .as_str()
+ .is_some_and(|r| r.contains("copy_from")),
+ "{body}"
+ );
+}
+
+#[tokio::test]
+async fn all_stats_reports_the_schema_persist_failure_counter_too() {
+ let (app, _dir) = app();
+ let (status, _) = send(&app, put("/s1/_doc/1?refresh=true", json!({ "a": 1 }))).await;
+ assert!(status.is_success());
+
+ let (status, body) = send(&app, get("/_all/_stats")).await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ // Pre-fix `_all.primaries` carried no `mappings` key at all, while
+ // `Index::schema_persist_failures`'s doc-comment named this endpoint.
+ assert_eq!(
+ body["_all"]["primaries"]["mappings"]["schema_persist_failures"], 0,
+ "{body}"
+ );
+ assert_eq!(
+ body["indices"]["s1"]["primaries"]["mappings"]["schema_persist_failures"], 0,
+ "{body}"
+ );
+}
+
+#[tokio::test]
+async fn a_synonym_filter_given_a_bare_string_is_refused_not_silently_emptied() {
+ let (app, _dir) = app();
+ // `apply_settings` reads `synonyms` with `and_then(as_array)`, so a bare
+ // string built a synonym filter with ZERO rules — registered, referenced by
+ // the analyzer, expanding nothing, reported nowhere.
+ let (status, body) = send(
+ &app,
+ put(
+ "/syn_bad",
+ json!({ "settings": { "analysis": {
+ "filter": { "s": { "type": "synonym", "synonyms": "fast,quick" } },
+ "analyzer": { "a": { "type": "custom", "tokenizer": "standard",
+ "filter": ["s"] } }
+ } } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");
+ assert!(
+ body["error"]["reason"]
+ .as_str()
+ .is_some_and(|r| r.contains("synonyms")),
+ "{body}"
+ );
+
+ // The array form is still accepted.
+ let (status, body) = send(
+ &app,
+ put(
+ "/syn_ok",
+ json!({ "settings": { "analysis": {
+ "filter": { "s": { "type": "synonym", "synonyms": ["fast,quick"] } },
+ "analyzer": { "a": { "type": "custom", "tokenizer": "standard",
+ "filter": ["s"] } }
+ } } }),
+ ),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+}
diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rs
index f4df56df8..6d82bc1e1 100644
--- a/engine/crates/xerj-engine/src/bulk.rs
+++ b/engine/crates/xerj-engine/src/bulk.rs
@@ -265,6 +265,16 @@ pub struct BulkOpts {
/// shape (see search body filter). Applied to every update action
/// that doesn't provide its own `_source` in the action meta.
pub default_source_req: Option<Value>,
+ /// URL-level `?pipeline=` — the ingest pipeline for every `index` /
+ /// `create` action that does not name one in its own action metadata.
+ /// `_none` disables the target index's `index.default_pipeline`.
+ ///
+ /// Issue #204: this used to be dropped on the floor, so
+ /// `POST /_bulk?pipeline=redact` answered `201` per item and stored the
+ /// UNREDACTED document, while the single-document form of the same write
+ /// ran the pipeline. Same for `index.default_pipeline`, which `_bulk` never
+ /// consulted at all.
+ pub default_pipeline: Option<String>,
}
pub async fn process_bulk_with_opts(
@@ -936,10 +946,49 @@ pub async fn process_bulk_with_opts(
}
}
+ // ── Effective ingest pipeline per item ──────────────────────────────────
+ //
+ // Elasticsearch resolves the pipeline for a bulk `index`/`create` action in
+ // this order, and so does xerj now:
+ //
+ // 1. the action's own `"pipeline"` metadata key
+ // 2. the request's `?pipeline=` query parameter
+ // 3. the target index's `index.default_pipeline` setting
+ //
+ // and `_none` at either of the first two levels disables the setting.
+ //
+ // Issue #204: only step 1 existed here. `POST /_bulk?pipeline=redact` and
+ // an index carrying `index.default_pipeline` both answered `201` per item
+ // and stored the document UNTRANSFORMED — no error, no log line — while
+ // `PUT /{index}/_doc/{id}?pipeline=redact` ran the pipeline and refused
+ // when it could not. One instruction, two answers, and the silent one was
+ // the high-volume path.
+ //
+ // Resolving here, before the existence check below and before the
+ // turbo-path partition (which diverts any action carrying a pipeline),
+ // means the rest of this function needs no further change: an unknown or
+ // unrunnable name fails the item exactly as an explicit one always did.
+ for action in parsed.iter_mut() {
+ if !matches!(action.action_type.as_str(), "index" | "create") {
+ // `update` and `delete` never carry a source body through a
+ // pipeline, in ES or here.
+ continue;
+ }
+ let explicit = action
+ .pipeline
+ .as_deref()
+ .or(opts.default_pipeline.as_deref());
+ action.pipeline = match explicit {
+ Some("_none") => None,
+ Some(p) => Some(p.to_string()),
+ None => engine.default_pipeline_for(&action.target_index),
+ };
+ }
+
// ── Per-item validation: pipeline existence ─────────────────────────────
//
- // A per-item `pipeline: "..."` metadata reference must resolve to a
- // registered ingest pipeline; otherwise ES fails the item with
+ // The effective pipeline resolved above must name a registered ingest
+ // pipeline; otherwise ES fails the item with
// `illegal_argument_exception: pipeline with id [...] does not exist`.
parsed.retain(|action| {
if let Some(pid) = action.pipeline.as_deref() {
@@ -1644,8 +1693,24 @@ pub async fn process_bulk_with_opts(
}
doc_body = Some(transformed);
}
- // Empty result set — index the original body unchanged.
- Ok(_) => {}
+ // No output for this document. `process_through_pipeline`
+ // zips one action per input, so this is an internal
+ // invariant break — but it must not fall through to
+ // indexing the ORIGINAL body, which is the silent
+ // pass-through issue #204 is about. Fail the item.
+ Ok(_) => {
+ items[item_idx] = Some(BulkItemResult {
+ action: action_type.to_string(),
+ index: target_index.clone(),
+ id: error_id.clone(),
+ status: 400,
+ result: None,
+ error: Some(format!("pipeline with id [{pid}] returned no document")),
+ get_source: None,
+ });
+ errors = true;
+ continue;
+ }
Err(e) => {
// A genuinely absent pipeline keeps ES's exact wording
// (the conformance suite asserts it). A pipeline that
diff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rs
index 2e4b7e8db..7b83128ad 100644
--- a/engine/crates/xerj-engine/src/engine.rs
+++ b/engine/crates/xerj-engine/src/engine.rs
@@ -1668,6 +1668,32 @@ impl Engine {
// ── Transform pipeline methods ────────────────────────────────────────────
+ /// The `index.default_pipeline` setting for `index`, if it has one.
+ ///
+ /// One implementation for every write path. It used to live only in the
+ /// `_doc` handlers, which is why `_bulk` ignored the setting entirely and
+ /// stored documents untransformed under a `201` while the single-document
+ /// form of the same write refused (issue #204).
+ ///
+ /// Handles the three shapes settings arrive in — nested
+ /// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`
+ /// and flat `{default_pipeline: ..}` — mirroring the `number_of_shards`
+ /// lookup pattern used elsewhere.
+ ///
+ /// `index` is matched literally: an alias has no settings of its own, so a
+ /// write addressed to an alias does not pick up its backing index's default
+ /// pipeline. That is a known gap, identical on every write path.
+ pub fn default_pipeline_for(&self, index: &str) -> Option<String> {
+ self.index_settings.get(index).and_then(|v| {
+ v.get("index")
+ .and_then(|ix| ix.get("default_pipeline"))
+ .or_else(|| v.get("index.default_pipeline"))
+ .or_else(|| v.get("default_pipeline"))
+ .and_then(Value::as_str)
+ .map(str::to_string)
+ })
+ }
+
/// Compile and register a typed transform pipeline from a JSON config.
///
/// `config_json` must be a valid [`PipelineConfig`](xerj_wasm::pipeline::PipelineConfig)
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 93689208c..6dd115bf2 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -17776,9 +17776,11 @@ impl Index {
/// `schema.json`. Non-zero means the on-disk mapping is behind the
/// in-memory one and a restart will lose dynamically-mapped fields.
///
- /// Read by `GET /{index}/_stats` and `GET /_all/_stats`, which report it as
- /// `primaries.mappings.schema_persist_failures` — a health check can see it
- /// without reading logs.
+ /// Read by `GET /{index}/_stats`, which reports it as
+ /// `primaries.mappings.schema_persist_failures`, and by `GET /_all/_stats`,
+ /// which reports the sum over every index at
+ /// `_all.primaries.mappings.schema_persist_failures` — so a health check
+ /// can see it without reading logs.
pub fn schema_persist_failures(&self) -> u64 {
self.schema_persist_failures.load(Ordering::Relaxed)
}
diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rs
index 490c749df..53716ff64 100644
--- a/engine/crates/xerj-fts/src/analyzer.rs
+++ b/engine/crates/xerj-fts/src/analyzer.rs
@@ -1333,7 +1333,10 @@ impl AnalyzerRegistry {
/// 6. `normalizer` — accepted by ES for `keyword` fields, never built here;
/// 7. an analyzer whose `filter` is not an array or whose `tokenizer` is
/// not a string: `apply_settings` drops the value shape-first and falls
- /// back to "no filters" / `standard`.
+ /// back to "no filters" / `standard`;
+ /// 8. a `synonym` filter whose `synonyms` is not an array of strings —
+ /// dropped the same shape-first way, leaving a filter that expands
+ /// nothing.
///
/// # What is NOT checked
///
@@ -1373,6 +1376,34 @@ impl AnalyzerRegistry {
.and_then(|v| v.as_str())
.unwrap_or("");
if SUPPORTED_FILTER_TYPES.contains(&filter_type) {
+ // A supported TYPE can still be given a wrong-shaped
+ // option, and `apply_settings` drops those shape-first:
+ // `synonyms` is read with `and_then(as_array)` and its
+ // entries with `filter_map(as_str)`, so
+ // `"synonyms": "fast,quick"` builds a synonym filter with
+ // ZERO rules — registered, referenced, expanding nothing,
+ // and reported nowhere. Same silent-loss class as the
+ // analyzer-level `filter`/`tokenizer` shapes below.
+ if filter_type == "synonym" {
+ match filter_def.get("synonyms") {
+ None => {}
+ Some(serde_json::Value::Array(arr)) => {
+ for v in arr {
+ if !v.is_string() {
+ problems.push(format!(
+ "token filter [{filter_name}]: `synonyms` entries \
+ must be rule strings, got {v} — it would be dropped"
+ ));
+ }
+ }
+ }
+ Some(other) => problems.push(format!(
+ "token filter [{filter_name}]: `synonyms` must be an array of \
+ rule strings, got {other} — it would be ignored and the filter \
+ would expand nothing"
+ )),
+ }
+ }
continue;
}
// The `type` is one `apply_settings` cannot BUILD — but that is
diff --git a/engine/crates/xerj-wasm/src/pipeline.rs b/engine/crates/xerj-wasm/src/pipeline.rs
index de8c583f0..a7b17d8b4 100644
--- a/engine/crates/xerj-wasm/src/pipeline.rs
+++ b/engine/crates/xerj-wasm/src/pipeline.rs
@@ -436,6 +436,22 @@ fn build_plugin(
"set" => {
let field = str_field(config, "field")?;
+ // Elasticsearch's `set` takes EITHER `value` or `copy_from` (7.x+),
+ // where `copy_from` names another field to read the value out of
+ // the document at ingest time. `SetPlugin` only holds a literal
+ // value, so `copy_from` cannot be honoured — but it is a valid ES
+ // processor, which makes the gap xerj's, not the caller's. Reported
+ // as an unsupported OPTION (accept the definition, refuse the
+ // write) rather than falling through to `missing 'value'`, which
+ // told the caller their correct ES pipeline was malformed.
+ if config.get("copy_from").is_some() {
+ return Err(StageBuildError::UnsupportedOption {
+ option: "copy_from".to_string(),
+ reason: "xerj's set stage assigns a literal `value`; copying the value \
+ from another field at ingest time is not implemented"
+ .to_string(),
+ });
+ }
let value = config
.get("value")
.cloned()