AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(api,engine,fts,wasm): make "every ingest path refuses loudly" true, and disclose the analysis break
xerj-orgcda86c032a9e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 00ff4fc62..060973e70 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,8 +7,140 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Changed++- **BREAKING — `PUT /{index}` now refuses an `analysis` block XERJ cannot+ honour, instead of accepting it and analysing differently**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `AnalyzerRegistry::apply_settings` is total by design (it also runs when an+ existing index is opened, where a hard failure would brick it), so an+ unresolvable tokenizer silently became `standard` and an unresolvable token+ filter was dropped — the index was created, `GET /_settings` echoed the block+ back, and the field was analysed in a way nobody asked for. Index creation+ now validates the block and answers `400` with the exact constructs it cannot+ build. `PUT /{index}/_settings` with an `analysis` block is likewise refused+ (it only ever changed what `GET /_settings` echoed; the registry is built+ once, at create/open, and there is no rebuild path — Elasticsearch also+ refuses analysis updates on an open index).++ **This rejects settings blocks that XERJ used to accept with a `200`, and the+ ES-YAML conformance suite does not cover them — CI being green is not+ evidence that your index still creates.** What is honoured:++ | | honoured (still `200`) |+ |---|---|+ | `analysis.filter.*.type` | `synonym`, `length`, `shingle`, `asciifolding` |+ | `analysis.tokenizer.*.type` | `ngram`, `edge_ngram`, `pattern` |+ | `analysis.analyzer.*.type` (non-custom) | any name the built-in registry resolves — measured: `standard`, `english`, `whitespace`, `keyword` |+ | `analysis.char_filter` | nothing — the char-filter slot is hard-coded empty |+ | `analysis.normalizer` | nothing — never built |++ Anything else in those positions is now a `400`. There is one important+ exception to read first: XERJ resolves a declared filter or tokenizer by+ **name**, not by `type`, so a declaration whose *name* is a built-in is still+ honoured whatever its type says. Measured, all still `200`:+ `{"english_stop": {"type": "stop", "stopwords": "_english_"}}`,+ `{"lowercase": {"type": "lowercase"}}`,+ `{"stemmer"|"english_stemmer": {"type": "stemmer", "language": "english"}}`+ — the canonical Elasticsearch-docs `rebuilt_english` block among them, which+ has a regression test pinning it. Naming a built-in analyzer on a field+ (`"analyzer": "english"`) needs no `analysis` block at all and is unaffected.++ What now fails is a declaration under a name that does NOT resolve — which is+ the usual ES habit of naming filters after what they do in *your* index.+ Measured blocks that returned `200` in rc.14 and return `400` from this+ release:++ - a `keyword` field's `normalizer` (any);+ - a `char_filter` (any, including `html_strip` and `mapping`), declared+ either under `analysis.char_filter` or as an analyzer's `char_filter`;+ - a custom-named token filter whose `type` is `lowercase`, `stop`, `stemmer`,+ `snowball`, `kstem`, `porter_stem`, `ngram`, `edge_ngram`,+ `word_delimiter_graph`, `trim`, `unique`, `elision`, `decimal_digit`,+ `apostrophe`, `limit`, `truncate`, `pattern_replace`, `keyword_marker`,+ `uppercase` or `reverse` — e.g. `{"my_lowercase": {"type": "lowercase"}}`,+ which never lowercased anything, because the analyzer referencing+ `my_lowercase` found no such filter and dropped it;+ - a custom-named tokenizer whose `type` is `standard`, `whitespace`,+ `keyword`, `classic`, `uax_url_email`, `path_hierarchy`, `char_group`,+ `letter`, `lowercase` or `simple_pattern`;+ - an analyzer whose non-custom `type` is `simple`, `stop`, `pattern`,+ `fingerprint`, or a language other than `english`;+ - a `synonym` filter whose `synonyms` is a bare string rather than an array+ (it built a filter with zero rules).++ **Upgrading:** there is no opt-out flag. Either remove the analysis block and+ name a built-in analyzer on the field (`"analyzer": "english"`), or keep only+ the constructs in the table. The `400` names every offending construct in one+ response, so one round-trip is enough to see the whole list. **Two paths are+ not covered and still accept these blocks silently: an index template's+ `template.settings.analysis` is dropped wholesale before the gate ever sees+ it, and option-level divergence inside a supported type (`synonyms_path`,+ `length.min`, `asciifolding.preserve_original`) is not checked at all.**+### Fixed+- **`200 acknowledged` on an ingest pipeline now means "we will run this", and+ every write path enforces it**+ ([#204](https://github.com/xerj-org/xerj/issues/204)).+ `PUT /_ingest/pipeline/{id}` used to store the raw body first and only+ warn-log a compile failure, so a pipeline that could never run was+ acknowledged and echoed back by `GET`. It now compiles first, with three+ outcomes: compiled → stored and acknowledged; a processor config we refuse →+ `400`; a processor or option this build cannot honour → accepted (as+ Elasticsearch does) but recorded as **unrunnable**, so the write refuses+ rather than storing an untransformed document.++ Defects this surfaced, each of which answered `200`/`201` and did something+ smaller than asked: **every** ES `rename` processor failed to compile (the+ handler emitted `{from, to}`, the plugin requires `{mappings: {…}}`) and+ renamed nothing; ES `append` was executed as `set`, replacing the field+ instead of extending it, and disagreeing with `_simulate`, which implemented+ a real append; `set.override` defaulted to `false` where Elasticsearch+ defaults it to `true`; an unknown `pii_redaction` type built an empty pattern+ set and redacted nothing; an unknown `grok` pattern name silently became the+ catch-all; an unknown `convert` target left the field unconverted per+ document; array-typed keys given as a bare string were dropped and the stage+ behaved as if unconfigured; and `DELETE /_ingest/pipeline/{id}` removed the+ definition while leaving the compiled pipeline live under the same name.+ ES `convert` `long`/`double` now map to XERJ's equivalent `integer`/`float`+ rather than being refused.++ Two more of the same, at the definition's own edges: a **pipeline-level**+ `on_failure` block (the top-level body key, not the processor-level one) was+ accepted, echoed by `GET`, and never run — the pipeline is now recorded+ unrunnable, exactly as its processor-level twin already was; and `set` with+ Elasticsearch's `copy_from` is reported as an unsupported *option* (accepted+ at `PUT`, refused at the write) rather than as `mapper_parsing_exception:+ missing value`, which told the caller their valid ES pipeline was malformed.++ **The refusal now covers every ingest path, not only `PUT /{index}/_doc`.**+ `POST /_bulk?pipeline=`, `index.default_pipeline` under `_bulk`,+ `_reindex` `dest.pipeline` and `POST /{index}/_update_by_query?pipeline=`+ previously read the instruction and dropped it — a redaction pipeline that+ the single-document endpoint refused was ignored in bulk, and the document+ was stored verbatim under `errors: false`. All four now run the pipeline, and+ refuse when it cannot run. `_reindex` and `_update_by_query` validate the+ name before touching a single document, so a bad pipeline fails the request+ instead of half of it.++ Two related honesty fixes on the same paths: `_clone` / `_shrink` / `_split`+ copied at most 10,000 documents and swallowed every per-document write error+ before answering `{"acknowledged": true}` (now full keyset paging, and any+ failed write — or a failed source flush — fails the request); and a failed+ `_bulk` item no longer reports `"result": "deleted"` alongside its own+ `status: 400`, which claimed a deletion that never happened.++ **Known gaps, stated rather than closed:** `_simulate` interprets the stored+ ES body directly and covers processors the compiled path does not, so a+ pipeline recorded as unrunnable can still show a transformation under+ `_simulate` that ingest refuses to perform; `GET /_ingest/pipeline/{id}`+ returns the ES body with no runnability signal, so an operator only learns at+ write time; a write addressed to an **alias** does not pick up the backing+ index's `default_pipeline`; and `_update_by_query` reports a `drop`+ processor as a per-document failure, because it updates in place and has no+ delete path.+- **`autoindex` no longer leaves immortal catalog entries for skipped files**([#238](https://github.com/xerj-org/xerj/issues/238)). A file added after theresume plan was frozen is skipped and reported in the catalog — but thatdiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 8a55caf30..d5ca885e0 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -2201,10 +2201,9 @@ pub struct IndexDocAutoParams {/// - `Some(name)` → that pipeline (an explicit `?pipeline=` always wins)./// - `None` → the index's `index.default_pipeline` setting, if any.///-/// The default is read from `engine.index_settings`, handling the nested-/// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`,-/// and flat `{default_pipeline: ..}` shapes — mirroring the `number_of_shards`-/// lookup pattern used elsewhere in this module.+/// The default is read by [`xerj_engine::Engine::default_pipeline_for`], which+/// `_bulk` also uses — one lookup, so the two forms of the same write cannot+/// disagree about which pipeline applies.fn resolve_effective_pipeline(state: &AppState,index: &str,@@ -2213,14 +2212,7 @@ fn resolve_effective_pipeline(match explicit {Some("_none") => None,Some(p) => Some(p.to_string()),- None => state.engine.index_settings.get(index).and_then(|v| {- v.get("index")- .and_then(|ix| ix.get("default_pipeline"))- .or_else(|| v.get("index.default_pipeline"))- .or_else(|| v.get("default_pipeline"))- .and_then(Value::as_str)- .map(|s| s.to_string())- }),+ None => state.engine.default_pipeline_for(index),}}@@ -2295,10 +2287,9 @@ pub async fn index_doc_auto(}transformed}- Ok(_) => {- // Pipeline returned empty results — pass through original- serde_json::from_slice(&body).unwrap_or(Value::Null)- }+ // No output for this document: refused, not passed through+ // unchanged — see `pipeline_empty_result_error`.+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),// Issue #204, the purest form of the class on a user-DATA path:// this used to warn and then index the ORIGINAL document. A// pipeline whose job was to redact PII, drop a field or route the@@ -2426,9 +2417,9 @@ pub async fn index_doc(}transformed}- // Pipeline produced no output for this document — index the- // original body, which is what it asked for.- Ok(_) => doc,+ // No output for this document: refused, not passed through+ // unchanged — see `pipeline_empty_result_error`.+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),// Issue #204: this arm was folded into `_ => doc`, so a pipeline// that could not run left the document indexed UNTRANSFORMED under// a success response. Refuse instead.@@ -2553,9 +2544,9 @@ pub async fn create_doc(}transformed}- // Pipeline produced no output for this document — index the- // original body, which is what it asked for.- Ok(_) => doc,+ // No output for this document: refused, not passed through+ // unchanged — see `pipeline_empty_result_error`.+ Ok(_) => return pipeline_empty_result_error(pipeline).into_response(),// Issue #204: this arm was folded into `_ => doc`, so a pipeline// that could not run left the document indexed UNTRANSFORMED under// a success response. Refuse instead.@@ -13054,6 +13045,10 @@ fn bulk_opts_from_query(_ => None,}),default_source_req,+ // `?pipeline=` — the request-level default for every `index`/`create`+ // action that names no pipeline of its own. Read but discarded before+ // (issue #204): the bulk item was written untransformed under a `201`.+ default_pipeline: qp.get("pipeline").cloned(),}}@@ -13182,7 +13177,14 @@ async fn process_bulk_body(index: item.index,id: item.id,version,- result: item.result.unwrap_or_else(|| "deleted".to_string()),+ // A successful action with no explicit result is a `delete`+ // (the only one that does not set one). A FAILED action has no+ // result at all — see `EsBulkItemResult::result`.+ result: if item.error.is_some() {+ None+ } else {+ Some(item.result.clone().unwrap_or_else(|| "deleted".to_string()))+ },shards: crate::responses::EsShards::single_success(),seq_no,primary_term: 1,@@ -16506,6 +16508,13 @@ pub async fn index_stats(// Aggregate across multiple indices.let mut all_doc_count = 0u64;let mut all_store_bytes = 0u64;+ // Aggregated so `GET /_all/_stats` carries the same+ // `primaries.mappings.schema_persist_failures` the per-index form does.+ // Without it `Index::schema_persist_failures`'s doc-comment named an+ // endpoint that did not report the counter, and a fleet-wide health+ // check reading only `_all` saw nothing (issue #204: the claim has to+ // match what the endpoint really answers).+ let mut all_schema_persist_failures = 0u64;let mut all_indices = serde_json::Map::new();for name in &targets {let stats = match state.engine.index_stats(name).await {@@ -16545,6 +16554,7 @@ pub async fn index_stats(.get_index(name).map(|idx| idx.schema_persist_failures()).unwrap_or(0);+ all_schema_persist_failures += schema_persist_failures;let primaries = json!({"docs": { "count": doc_count, "deleted": 0 },"store": { "size_in_bytes": store_size_bytes },@@ -16614,6 +16624,7 @@ pub async fn index_stats(let all_primaries = json!({"docs": { "count": all_doc_count, "deleted": 0 },"store": { "size_in_bytes": all_store_bytes },+ "mappings": { "schema_persist_failures": all_schema_persist_failures },"dense_vector": {"value_count": all_dv_value_count,"off_heap": {@@ -18309,6 +18320,14 @@ fn reindex_default_size() -> usize {#[derive(Debug, Deserialize)]pub struct ReindexDest {pub index: String,+ /// `dest.pipeline` — the ingest pipeline every copied document is run+ /// through before it is written to the destination.+ ///+ /// Issue #204: this field did not exist, so serde dropped it and a+ /// `_reindex` asking for redaction on the way in answered+ /// `{"created": N}` having copied every document VERBATIM.+ #[serde(default)]+ pub pipeline: Option<String>,}pub async fn reindex(@@ -18356,6 +18375,41 @@ pub async fn reindex(let source_name = &body.source.index;let dest_name = &body.dest.index;+ // `dest.pipeline` is checked BEFORE a single document is copied. A name+ // that does not resolve, or one this build cannot run, fails the whole+ // request with nothing written — rather than copying the corpus+ // untransformed and reporting success (issue #204). `_none` means "no+ // pipeline", matching the write paths.+ let dest_pipeline: Option<String> = match body.dest.pipeline.as_deref() {+ None | Some("_none") => None,+ Some(p) => {+ let reason = if !state.engine.pipelines.contains_key(p) {+ Some(format!("pipeline with id [{p}] does not exist"))+ } else {+ state+ .engine+ .unrunnable_pipelines+ .get(p)+ .map(|r| format!("pipeline with id [{p}] cannot be run: {}", r.value()))+ };+ if let Some(reason) = reason {+ return (+ StatusCode::BAD_REQUEST,+ Json(json!({+ "error": {+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],+ "type": "illegal_argument_exception",+ "reason": reason,+ },+ "status": 400,+ })),+ )+ .into_response();+ }+ Some(p.to_string())+ }+ };+// Get or create destination index.let dest_idx = match state.engine.get_or_create_index(dest_name) {Ok(i) => i,@@ -18374,7 +18428,19 @@ pub async fn reindex(// reliably, so un-flushed recent writes would be skipped or duplicated// across pages. Reindex is a snapshot-style bulk op, so a single flush up// front is an acceptable cost and makes the whole source keyset-pageable.- let _ = source_idx.flush().await;+ //+ // Propagated, not swallowed, for the same reason as `clone_index_to`: the+ // flush is a precondition of the paging being complete, so a failed flush+ // must not be followed by a reindex that reports `"total": N` over a source+ // it could not read consistently (issue #204).+ if let Err(e) = source_idx.flush().await {+ return ApiError::new(xerj_common::XerjError::internal(format!(+ "reindex from [{source_name}] into [{dest_name}] was not attempted: flushing \+ the source failed ({e}); the copy needs a flushed source to page over it \+ without skipping or duplicating documents"+ )))+ .into_response();+ }let query_val = body.source@@ -18391,6 +18457,7 @@ pub async fn reindex(let mut total_fetched = 0usize;let mut created = 0usize;let mut updated = 0usize;+ let mut noops = 0usize;let mut failures: Vec<Value> = Vec::new();let mut batches = 0usize;// Keyset cursor: `[last_id]` of the previous page (the sole sort key is@@ -18441,6 +18508,49 @@ pub async fn reindex(for hit in results.hits {if !hit.source.is_null() {+ // `dest.pipeline`, if the request named one. Already known to+ // exist and to be runnable (checked before any copy started),+ // so anything that goes wrong here is a per-document failure+ // and is reported as one — never a pass-through of the+ // untransformed source.+ let doc_source = match dest_pipeline.as_deref() {+ None => hit.source,+ Some(p) => match state.engine.process_through_pipeline(p, vec![hit.source]) {+ Ok(mut results) if !results.is_empty() => {+ let (action, transformed) = results.remove(0);+ if matches!(action, xerj_wasm::pipeline::ProcessAction::Drop) {+ // ES: a pipeline that drops the document does+ // not write it, and the run reports a noop.+ noops += 1;+ continue;+ }+ transformed+ }+ Ok(_) => {+ failures.push(json!({+ "id": hit.id,+ "cause": {+ "type": "reindex_error",+ "reason": format!(+ "pipeline [{p}] returned no document for [{}]",+ hit.id+ ),+ },+ }));+ continue;+ }+ Err(e) => {+ failures.push(json!({+ "id": hit.id,+ "cause": {+ "type": "reindex_error",+ "reason": format!("pipeline [{p}] failed: {e}"),+ },+ }));+ continue;+ }+ },+ };// Check if doc already exists in dest to track created vs updated.let exists = dest_idx.get_document(&hit.id)@@ -18449,7 +18559,7 @@ pub async fn reindex(.flatten().is_some();match dest_idx- .index_document(Some(hit.id.clone()), hit.source)+ .index_document(Some(hit.id.clone()), doc_source).await{Ok(_) => {@@ -18490,7 +18600,7 @@ pub async fn reindex("batches": batches,"failures": failures,"throttled_millis": 0,- "noops": 0,+ "noops": noops,"version_conflicts": 0,"retries": { "bulk": 0, "search": 0 },}))@@ -21245,6 +21355,42 @@ pub async fn update_by_query(// update happens in place (no duplicate-`_id` docs are appended).let script = body.script.as_ref().map(extract_update_script);+ // `?pipeline=` — every matched document is run through it before being+ // written back. Issue #204: the parameter used to be read by nothing, so+ // `_update_by_query?pipeline=redact` reported `"updated": N` having+ // rewritten every document EXACTLY as it was. Resolved and validated+ // before any document is touched: an unknown or unrunnable name fails the+ // whole request rather than half of it.+ let pipeline = match params.get("pipeline").map(String::as_str) {+ None | Some("_none") => None,+ Some(p) => {+ let reason = if !state.engine.pipelines.contains_key(p) {+ Some(format!("pipeline with id [{p}] does not exist"))+ } else {+ state+ .engine+ .unrunnable_pipelines+ .get(p)+ .map(|r| format!("pipeline with id [{p}] cannot be run: {}", r.value()))+ };+ if let Some(reason) = reason {+ return (+ StatusCode::BAD_REQUEST,+ Json(json!({+ "error": {+ "root_cause": [{ "type": "illegal_argument_exception", "reason": reason }],+ "type": "illegal_argument_exception",+ "reason": reason,+ },+ "status": 400,+ })),+ )+ .into_response();+ }+ Some((state.engine.clone(), p.to_string()))+ }+ };+// ES defaults `wait_for_completion` to true (synchronous); only an// explicit `false` switches to the async `{"task": "node:id"}` form.let wait_for_completion = params@@ -21260,14 +21406,14 @@ pub async fn update_by_query(// Same reasoning as the `_delete_by_query` spawn above: the task owns// one already-authorized `Index` handle and can reach no other index.tokio::spawn(async move {- let response = run_update_by_query(&idx, &search_req, script).await;+ let response = run_update_by_query(&idx, &search_req, script, pipeline).await;tasks.complete(&spawned_key, response);drop(handle);});return Json(json!({ "task": task_key })).into_response();}- let response = run_update_by_query(&idx, &search_req, script).await;+ let response = run_update_by_query(&idx, &search_req, script, pipeline).await;drop(handle);by_query_response(response)}@@ -21280,10 +21426,46 @@ pub async fn update_by_query(/// `wait_for_completion=false` form.const SCRIPTED_UPDATE_BUDGET_MS: u64 = 30_000;+/// Run `doc` through `pipeline`, or say why it could not be run.+///+/// Issue #204: every caller must get a document that really went through the+/// pipeline, or an error. There is deliberately no "return the input unchanged"+/// branch — that is precisely how an unrun redaction became a `200`.+///+/// A `drop` action is an error here rather than a deletion:+/// `_update_by_query` updates documents in place and has no delete path, so+/// honouring `drop` is not something this endpoint can do. Saying so per+/// document beats writing the document as if the processor had not asked.+fn apply_pipeline_or_fail(+ engine: &xerj_engine::Engine,+ pipeline: &str,+ doc_id: &str,+ doc: Value,+) -> std::result::Result<Value, String> {+ match engine.process_through_pipeline(pipeline, vec![doc]) {+ Ok(mut results) if !results.is_empty() => {+ let (action, transformed) = results.remove(0);+ if matches!(action, xerj_wasm::pipeline::ProcessAction::Drop) {+ return Err(format!(+ "pipeline [{pipeline}] dropped document [{doc_id}]; _update_by_query \+ updates in place and cannot delete it, so the document was left \+ unchanged"+ ));+ }+ Ok(transformed)+ }+ Ok(_) => Err(format!(+ "pipeline [{pipeline}] returned no document for [{doc_id}]"+ )),+ Err(e) => Err(format!("pipeline [{pipeline}] failed: {e}")),+ }+}+async fn run_update_by_query(idx: &xerj_engine::Index,search_req: &xerj_query::SearchRequest,script: Option<(String, Value)>,+ pipeline: Option<(std::sync::Arc<xerj_engine::Engine>, String)>,) -> Value {let started = Instant::now();@@ -21343,6 +21525,13 @@ async fn run_update_by_query(let result = idx.transform_document_serialized(&hit.id, None, |mut current| {apply_painless_update(&mut current, src, params)?;+ // ES runs `?pipeline=` on the result of the script,+ // and so do we — inside the same serialized+ // transform, so the pipeline sees exactly the+ // document that is about to be written.+ if let Some((engine, name)) = pipeline.as_ref() {+ current = apply_pipeline_or_fail(engine, name, &hit.id, current)?;+ }Ok::<_, String>(current)}).await;@@ -21359,8 +21548,24 @@ async fn run_update_by_query(})),}} else {- // No script: preserve the historical re-index-in-place behavior.- match idx.index_document(Some(hit.id.clone()), hit.source).await {+ // No script: preserve the historical re-index-in-place+ // behavior, with `?pipeline=` applied when one was named.+ let doc = match pipeline.as_ref() {+ None => hit.source,+ Some((engine, name)) => {+ match apply_pipeline_or_fail(engine, name, &hit.id, hit.source) {+ Ok(d) => d,+ Err(reason) => {+ failures.push(json!({+ "id": hit.id,+ "cause": { "reason": reason },+ }));+ continue;+ }+ }+ }+ };+ match idx.index_document(Some(hit.id.clone()), doc).await {Ok(_) => updated += 1,Err(e) => {failures.push(json!({@@ -21483,6 +21688,7 @@ mod scripted_update_publication_tests {&idx,&xerj_query::SearchRequest::default(),Some(("ctx._source.n += 1".into(), json!({}))),+ None,).await}));@@ -21761,6 +21967,21 @@ fn pipeline_execution_error(pipeline: &str, e: &xerj_wasm::WasmError) -> ApiErro)))}+/// A pipeline that returned no document for the one it was given.+///+/// `process_through_pipeline` zips one action per input document, so this is an+/// internal invariant break rather than a configuration a caller can write.+/// It still must not fall through to indexing the ORIGINAL document: "the+/// pipeline produced nothing, so store what the caller sent" is the same+/// silent-pass-through this whole change exists to remove, and it would land on+/// exactly the redaction case where it costs the most.+fn pipeline_empty_result_error(pipeline: &str) -> ApiError {+ tracing::error!(pipeline = %pipeline, "refusing write: ingest pipeline returned no document");+ ApiError::new(xerj_common::XerjError::config(format!(+ "document not indexed: ingest pipeline [{pipeline}] returned no document"+ )))+}+pub async fn put_ingest_pipeline(State(state): State<AppState>,Path(id): Path<String>,@@ -21884,6 +22105,27 @@ pub async fn put_ingest_pipeline(} else {body.clone()};+ // Elasticsearch pipeline-level (top-level body) keys that decide what+ // happens when a processor fails. The compiled pipeline has no equivalent:+ // `PipelineConfig` carries a single `on_error` policy and no recovery+ // chain, so a top-level `on_failure` was read by nothing.+ //+ // This is the SAME defect the repair fixed one level down+ // (`UNHONOURED_PROCESSOR_KEYS` in `xerj-wasm/src/pipeline.rs`): refusing a+ // processor-level `on_failure` while silently dropping the identical key at+ // pipeline level would have left the endpoint inconsistent with itself.+ // `description`, `version`, `_meta` and `deprecated` are NOT here — they+ // are identification metadata that changes no document, the same line drawn+ // for processor-level `tag`/`description`.+ const UNHONOURED_PIPELINE_KEYS: &[(&str, &str)] = &[(+ "on_failure",+ "the pipeline-level recovery processors are not run; the compiled pipeline's \+ `on_error` policy applies instead",+ )];+ let unhonoured_top_level = UNHONOURED_PIPELINE_KEYS+ .iter()+ .find(|(key, _)| body.get(*key).is_some())+ .copied();// Three outcomes, and the difference between them is the whole point of// issue #204://@@ -21902,6 +22144,21 @@ pub async fn put_ingest_pipeline(// where a document would otherwise be silently// mis-ingested.match state.engine.create_pipeline(&id, xerj_config) {+ Ok(()) if unhonoured_top_level.is_some() => {+ // Every stage compiled, but a top-level key we cannot act on came+ // with it. `register_unrunnable_pipeline` drops the compiled form+ // `create_pipeline` just installed, so nothing runs half-honoured.+ let (option, reason) = unhonoured_top_level.expect("guarded by the match arm");+ state.engine.register_unrunnable_pipeline(+ &id,+ body,+ format!(+ "pipeline-level [{option}] is not supported by this xerj build \+ ({reason}); no document can be ingested through pipeline [{id}]"+ ),+ );+ Json(json!({ "acknowledged": true })).into_response()+ }Ok(()) => {// `create_pipeline` stores the *translated* config; put the ES// body back so GET and _simulate see what was actually submitted.@@ -25151,7 +25408,19 @@ async fn clone_index_to(state: &AppState, source: &str, target: &str) -> Result<// `max_result_window`. The source is flushed first because the in-memory// memtable's sort path does not order by `_id` reliably, so un-flushed// writes would be skipped or duplicated across pages.- let _ = src_idx.flush().await;+ //+ // That makes the flush a CORRECTNESS precondition of the paging below, not+ // a best-effort hint — so it is propagated rather than swallowed. A+ // `let _ =` here would have produced exactly the defect this function was+ // rewritten to remove: a silently partial (or duplicated) copy answering+ // `{"acknowledged": true, "shards_acknowledged": true}`.+ src_idx.flush().await.map_err(|e| {+ ApiError::new(xerj_common::XerjError::internal(format!(+ "clone of [{source}] into [{target}] was not attempted: flushing the source \+ failed ({e}); the copy needs a flushed source to page over it without \+ skipping or duplicating documents"+ )))+ })?;const CLONE_PAGE_SIZE: usize = 1_000;let mut search_after: Option<Value> = None;@@ -34102,6 +34371,7 @@ mod reindex_keyset_tests {},dest: ReindexDest {index: "dst".into(),+ pipeline: None,},};let resp = reindex(State(state.clone()), Json(body))diff --git a/engine/crates/xerj-api/src/responses.rs b/engine/crates/xerj-api/src/responses.rsindex 8e7a75818..f1137b84f 100644--- a/engine/crates/xerj-api/src/responses.rs+++ b/engine/crates/xerj-api/src/responses.rs@@ -437,7 +437,18 @@ pub struct EsBulkItemResult {pub id: String,#[serde(rename = "_version")]pub version: u64,- pub result: String,+ /// `"created"` / `"updated"` / `"deleted"` / `"noop"`, and ABSENT on an+ /// item that failed.+ ///+ /// Issue #204: this was a bare `String` filled from+ /// `item.result.unwrap_or_else(|| "deleted".to_string())`, so every failed+ /// bulk item — a rejected pipeline, a mapping error, a version conflict —+ /// was reported as `"result": "deleted"` alongside its own `status: 400`.+ /// A client reading `result` was told the document had been deleted when+ /// nothing had been written or removed. Elasticsearch omits the field on a+ /// failed item; so do we.+ #[serde(skip_serializing_if = "Option::is_none")]+ pub result: Option<String>,#[serde(rename = "_shards")]pub shards: EsShards,#[serde(rename = "_seq_no")]diff --git a/engine/crates/xerj-api/tests/fail_closed_config_http.rs b/engine/crates/xerj-api/tests/fail_closed_config_http.rsindex 5a3914641..0ab8ea5b9 100644--- a/engine/crates/xerj-api/tests/fail_closed_config_http.rs+++ b/engine/crates/xerj-api/tests/fail_closed_config_http.rs@@ -394,3 +394,337 @@ async fn clone_copies_every_document_not_the_first_ten_thousand() {"clone must copy every document: {count}");}++// ── Every ingest path, not just the single-document one ──────────────────────+//+// The first cut of this branch made `PUT /_ingest/pipeline` mean "compiled ⇒+// xerj honours this" and wired the refusal into the three single-document+// handlers only. Measured against that build, `_bulk?pipeline=`, an index+// carrying `index.default_pipeline`, `_reindex` `dest.pipeline` and+// `_update_by_query?pipeline=` all answered 200/201 and stored the document+// UNTRANSFORMED — including for a pipeline that was perfectly runnable.++fn ndjson(path: &str, body: &str) -> Request<Body> {+ Request::builder()+ .method("POST")+ .uri(path)+ .header("content-type", "application/x-ndjson")+ .body(Body::from(body.to_string()))+ .expect("request")+}++/// `{"set": {"field": "env", "value": "prod"}}` — compiles and runs.+async fn define_runnable_pipeline(app: &axum::Router, id: &str) {+ let (status, body) = send(+ app,+ put(+ &format!("/_ingest/pipeline/{id}"),+ json!({ "processors": [{ "set": { "field": "env", "value": "prod" } }] }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+}++/// An ES `grok` `patterns` array — accepted by `PUT`, recorded unrunnable.+async fn define_unrunnable_pipeline(app: &axum::Router, id: &str) {+ let (status, body) = send(+ app,+ put(+ &format!("/_ingest/pipeline/{id}"),+ json!({ "processors": [{ "grok": {+ "field": "message",+ "patterns": ["%{IP:client} %{WORD:method}"]+ } }] }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["acknowledged"], true, "{body}");+}++#[tokio::test]+async fn bulk_runs_the_url_level_pipeline() {+ let (app, _dir) = app();+ define_runnable_pipeline(&app, "setp").await;++ let (status, body) = send(+ &app,+ ndjson(+ "/_bulk?pipeline=setp&refresh=true",+ "{\"index\":{\"_index\":\"b\",\"_id\":\"1\"}}\n{\"a\":1}\n",+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["errors"], false, "{body}");++ let (_, doc) = send(&app, get("/b/_doc/1")).await;+ // Pre-fix: `{"a": 1}` — the parameter was read by nothing.+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");+}++#[tokio::test]+async fn bulk_refuses_an_unrunnable_url_level_pipeline() {+ let (app, _dir) = app();+ define_unrunnable_pipeline(&app, "unrun").await;++ let (status, body) = send(+ &app,+ ndjson(+ "/_bulk?pipeline=unrun",+ "{\"index\":{\"_index\":\"b\",\"_id\":\"1\"}}\n{\"message\":\"10.0.0.1 GET\"}\n",+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["errors"], true, "{body}");+ assert_eq!(body["items"][0]["index"]["status"], 400, "{body}");+ // A failed item carries no `result`. Pre-fix it said `"deleted"`, which is+ // a statement about the document that was not true.+ assert!(+ body["items"][0]["index"].get("result").is_none(),+ "a failed item must not claim a result: {body}"+ );++ let (status, _) = send(&app, get("/b/_doc/1")).await;+ assert_ne!(status, StatusCode::OK, "nothing may have been written");+}++#[tokio::test]+async fn bulk_honours_index_default_pipeline() {+ let (app, _dir) = app();+ define_runnable_pipeline(&app, "setp").await;+ let (status, body) = send(+ &app,+ put(+ "/dp",+ json!({ "settings": { "index": { "default_pipeline": "setp" } } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");++ let (_, body) = send(+ &app,+ ndjson(+ "/_bulk?refresh=true",+ "{\"index\":{\"_index\":\"dp\",\"_id\":\"1\"}}\n{\"a\":1}\n",+ ),+ )+ .await;+ assert_eq!(body["errors"], false, "{body}");+ let (_, doc) = send(&app, get("/dp/_doc/1")).await;+ // Pre-fix: `_bulk` never consulted the setting, so this was `{"a": 1}`+ // while `PUT /dp/_doc/1` on the same index ran the pipeline.+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");++ // `_none` still disables it, as in Elasticsearch.+ let (_, body) = send(+ &app,+ ndjson(+ "/_bulk?pipeline=_none&refresh=true",+ "{\"index\":{\"_index\":\"dp\",\"_id\":\"2\"}}\n{\"a\":2}\n",+ ),+ )+ .await;+ assert_eq!(body["errors"], false, "{body}");+ let (_, doc) = send(&app, get("/dp/_doc/2")).await;+ assert!(doc["_source"].get("env").is_none(), "{doc}");+}++#[tokio::test]+async fn reindex_runs_and_validates_dest_pipeline() {+ let (app, _dir) = app();+ define_runnable_pipeline(&app, "setp").await;+ define_unrunnable_pipeline(&app, "unrun").await;+ let (status, _) = send(&app, put("/rsrc/_doc/1?refresh=true", json!({ "a": 1 }))).await;+ assert!(status.is_success());++ // Unrunnable: refused before anything is copied.+ let (status, body) = send(+ &app,+ post(+ "/_reindex",+ json!({ "source": { "index": "rsrc" },+ "dest": { "index": "rbad", "pipeline": "unrun" } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");+ let (status, _) = send(&app, get("/rbad/_count")).await;+ assert_ne!(status, StatusCode::OK, "no destination may have been made");++ // Runnable: every copied document really went through it. Pre-fix+ // `dest.pipeline` was not even a field on the request struct, so serde+ // dropped it and the corpus was copied verbatim under `"created": 1`.+ let (status, body) = send(+ &app,+ post(+ "/_reindex",+ json!({ "source": { "index": "rsrc" },+ "dest": { "index": "rdst", "pipeline": "setp" } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["created"], 1, "{body}");+ let (_, doc) = send(&app, get("/rdst/_doc/1")).await;+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");+}++#[tokio::test]+async fn update_by_query_runs_and_validates_its_pipeline() {+ let (app, _dir) = app();+ define_runnable_pipeline(&app, "setp").await;+ define_unrunnable_pipeline(&app, "unrun").await;+ let (status, _) = send(&app, put("/ubq/_doc/1?refresh=true", json!({ "a": 1 }))).await;+ assert!(status.is_success());++ let (status, body) = send(+ &app,+ post(+ "/ubq/_update_by_query?pipeline=unrun",+ json!({ "query": { "match_all": {} } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");+ let (_, doc) = send(&app, get("/ubq/_doc/1")).await;+ assert_eq!(doc["_version"], 1, "nothing may have been rewritten: {doc}");++ let (status, body) = send(+ &app,+ post(+ "/ubq/_update_by_query?pipeline=setp&refresh=true",+ json!({ "query": { "match_all": {} } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["updated"], 1, "{body}");+ let (_, doc) = send(&app, get("/ubq/_doc/1")).await;+ // Pre-fix: `"updated": 1` and the document byte-identical to what it was.+ assert_eq!(doc["_source"]["env"], "prod", "{doc}");+}++#[tokio::test]+async fn a_pipeline_level_on_failure_block_is_accepted_and_then_refuses_the_write() {+ let (app, _dir) = app();+ // The same key one level up from the processor-level `on_failure` the+ // first cut refused. Measured against that build: `PUT` answered 200, `GET`+ // echoed the block back, and the write succeeded with the recovery chain+ // silently absent.+ let (status, body) = send(+ &app,+ put(+ "/_ingest/pipeline/plof",+ json!({+ "processors": [{ "set": { "field": "env", "value": "prod" } }],+ "on_failure": [{ "set": { "field": "err", "value": "yes" } }],+ }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["acknowledged"], true, "{body}");++ let (status, body) = send(&app, put("/plof/_doc/1?pipeline=plof", json!({ "a": 1 }))).await;+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");+ assert!(+ body["error"]["reason"]+ .as_str()+ .is_some_and(|r| r.contains("on_failure")),+ "{body}"+ );+}++#[tokio::test]+async fn set_copy_from_is_a_capability_gap_not_a_caller_error() {+ let (app, _dir) = app();+ // `copy_from` is valid Elasticsearch 7.x+. Refusing the DEFINITION with+ // `mapper_parsing_exception: missing value` told the caller their correct+ // pipeline was malformed; the gap is xerj's, so it takes the same+ // accept-then-refuse-the-write route as every other unsupported option.+ let (status, body) = send(+ &app,+ put(+ "/_ingest/pipeline/cf",+ json!({ "processors": [{ "set": { "field": "b", "copy_from": "a" } }] }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body["acknowledged"], true, "{body}");++ let (status, body) = send(&app, put("/cf/_doc/1?pipeline=cf", json!({ "a": 1 }))).await;+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");+ assert!(+ body["error"]["reason"]+ .as_str()+ .is_some_and(|r| r.contains("copy_from")),+ "{body}"+ );+}++#[tokio::test]+async fn all_stats_reports_the_schema_persist_failure_counter_too() {+ let (app, _dir) = app();+ let (status, _) = send(&app, put("/s1/_doc/1?refresh=true", json!({ "a": 1 }))).await;+ assert!(status.is_success());++ let (status, body) = send(&app, get("/_all/_stats")).await;+ assert_eq!(status, StatusCode::OK, "{body}");+ // Pre-fix `_all.primaries` carried no `mappings` key at all, while+ // `Index::schema_persist_failures`'s doc-comment named this endpoint.+ assert_eq!(+ body["_all"]["primaries"]["mappings"]["schema_persist_failures"], 0,+ "{body}"+ );+ assert_eq!(+ body["indices"]["s1"]["primaries"]["mappings"]["schema_persist_failures"], 0,+ "{body}"+ );+}++#[tokio::test]+async fn a_synonym_filter_given_a_bare_string_is_refused_not_silently_emptied() {+ let (app, _dir) = app();+ // `apply_settings` reads `synonyms` with `and_then(as_array)`, so a bare+ // string built a synonym filter with ZERO rules — registered, referenced by+ // the analyzer, expanding nothing, reported nowhere.+ let (status, body) = send(+ &app,+ put(+ "/syn_bad",+ json!({ "settings": { "analysis": {+ "filter": { "s": { "type": "synonym", "synonyms": "fast,quick" } },+ "analyzer": { "a": { "type": "custom", "tokenizer": "standard",+ "filter": ["s"] } }+ } } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::BAD_REQUEST, "{body}");+ assert!(+ body["error"]["reason"]+ .as_str()+ .is_some_and(|r| r.contains("synonyms")),+ "{body}"+ );++ // The array form is still accepted.+ let (status, body) = send(+ &app,+ put(+ "/syn_ok",+ json!({ "settings": { "analysis": {+ "filter": { "s": { "type": "synonym", "synonyms": ["fast,quick"] } },+ "analyzer": { "a": { "type": "custom", "tokenizer": "standard",+ "filter": ["s"] } }+ } } }),+ ),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+}diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rsindex f4df56df8..6d82bc1e1 100644--- a/engine/crates/xerj-engine/src/bulk.rs+++ b/engine/crates/xerj-engine/src/bulk.rs@@ -265,6 +265,16 @@ pub struct BulkOpts {/// shape (see search body filter). Applied to every update action/// that doesn't provide its own `_source` in the action meta.pub default_source_req: Option<Value>,+ /// URL-level `?pipeline=` — the ingest pipeline for every `index` /+ /// `create` action that does not name one in its own action metadata.+ /// `_none` disables the target index's `index.default_pipeline`.+ ///+ /// Issue #204: this used to be dropped on the floor, so+ /// `POST /_bulk?pipeline=redact` answered `201` per item and stored the+ /// UNREDACTED document, while the single-document form of the same write+ /// ran the pipeline. Same for `index.default_pipeline`, which `_bulk` never+ /// consulted at all.+ pub default_pipeline: Option<String>,}pub async fn process_bulk_with_opts(@@ -936,10 +946,49 @@ pub async fn process_bulk_with_opts(}}+ // ── Effective ingest pipeline per item ──────────────────────────────────+ //+ // Elasticsearch resolves the pipeline for a bulk `index`/`create` action in+ // this order, and so does xerj now:+ //+ // 1. the action's own `"pipeline"` metadata key+ // 2. the request's `?pipeline=` query parameter+ // 3. the target index's `index.default_pipeline` setting+ //+ // and `_none` at either of the first two levels disables the setting.+ //+ // Issue #204: only step 1 existed here. `POST /_bulk?pipeline=redact` and+ // an index carrying `index.default_pipeline` both answered `201` per item+ // and stored the document UNTRANSFORMED — no error, no log line — while+ // `PUT /{index}/_doc/{id}?pipeline=redact` ran the pipeline and refused+ // when it could not. One instruction, two answers, and the silent one was+ // the high-volume path.+ //+ // Resolving here, before the existence check below and before the+ // turbo-path partition (which diverts any action carrying a pipeline),+ // means the rest of this function needs no further change: an unknown or+ // unrunnable name fails the item exactly as an explicit one always did.+ for action in parsed.iter_mut() {+ if !matches!(action.action_type.as_str(), "index" | "create") {+ // `update` and `delete` never carry a source body through a+ // pipeline, in ES or here.+ continue;+ }+ let explicit = action+ .pipeline+ .as_deref()+ .or(opts.default_pipeline.as_deref());+ action.pipeline = match explicit {+ Some("_none") => None,+ Some(p) => Some(p.to_string()),+ None => engine.default_pipeline_for(&action.target_index),+ };+ }+// ── Per-item validation: pipeline existence ─────────────────────────────//- // A per-item `pipeline: "..."` metadata reference must resolve to a- // registered ingest pipeline; otherwise ES fails the item with+ // The effective pipeline resolved above must name a registered ingest+ // pipeline; otherwise ES fails the item with// `illegal_argument_exception: pipeline with id [...] does not exist`.parsed.retain(|action| {if let Some(pid) = action.pipeline.as_deref() {@@ -1644,8 +1693,24 @@ pub async fn process_bulk_with_opts(}doc_body = Some(transformed);}- // Empty result set — index the original body unchanged.- Ok(_) => {}+ // No output for this document. `process_through_pipeline`+ // zips one action per input, so this is an internal+ // invariant break — but it must not fall through to+ // indexing the ORIGINAL body, which is the silent+ // pass-through issue #204 is about. Fail the item.+ Ok(_) => {+ items[item_idx] = Some(BulkItemResult {+ action: action_type.to_string(),+ index: target_index.clone(),+ id: error_id.clone(),+ status: 400,+ result: None,+ error: Some(format!("pipeline with id [{pid}] returned no document")),+ get_source: None,+ });+ errors = true;+ continue;+ }Err(e) => {// A genuinely absent pipeline keeps ES's exact wording// (the conformance suite asserts it). A pipeline thatdiff --git a/engine/crates/xerj-engine/src/engine.rs b/engine/crates/xerj-engine/src/engine.rsindex 2e4b7e8db..7b83128ad 100644--- a/engine/crates/xerj-engine/src/engine.rs+++ b/engine/crates/xerj-engine/src/engine.rs@@ -1668,6 +1668,32 @@ impl Engine {// ── Transform pipeline methods ────────────────────────────────────────────+ /// The `index.default_pipeline` setting for `index`, if it has one.+ ///+ /// One implementation for every write path. It used to live only in the+ /// `_doc` handlers, which is why `_bulk` ignored the setting entirely and+ /// stored documents untransformed under a `201` while the single-document+ /// form of the same write refused (issue #204).+ ///+ /// Handles the three shapes settings arrive in — nested+ /// `{index: {default_pipeline: ..}}`, dotted `{"index.default_pipeline": ..}`+ /// and flat `{default_pipeline: ..}` — mirroring the `number_of_shards`+ /// lookup pattern used elsewhere.+ ///+ /// `index` is matched literally: an alias has no settings of its own, so a+ /// write addressed to an alias does not pick up its backing index's default+ /// pipeline. That is a known gap, identical on every write path.+ pub fn default_pipeline_for(&self, index: &str) -> Option<String> {+ self.index_settings.get(index).and_then(|v| {+ v.get("index")+ .and_then(|ix| ix.get("default_pipeline"))+ .or_else(|| v.get("index.default_pipeline"))+ .or_else(|| v.get("default_pipeline"))+ .and_then(Value::as_str)+ .map(str::to_string)+ })+ }+/// Compile and register a typed transform pipeline from a JSON config.////// `config_json` must be a valid [`PipelineConfig`](xerj_wasm::pipeline::PipelineConfig)diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 93689208c..6dd115bf2 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -17776,9 +17776,11 @@ impl Index {/// `schema.json`. Non-zero means the on-disk mapping is behind the/// in-memory one and a restart will lose dynamically-mapped fields.///- /// Read by `GET /{index}/_stats` and `GET /_all/_stats`, which report it as- /// `primaries.mappings.schema_persist_failures` — a health check can see it- /// without reading logs.+ /// Read by `GET /{index}/_stats`, which reports it as+ /// `primaries.mappings.schema_persist_failures`, and by `GET /_all/_stats`,+ /// which reports the sum over every index at+ /// `_all.primaries.mappings.schema_persist_failures` — so a health check+ /// can see it without reading logs.pub fn schema_persist_failures(&self) -> u64 {self.schema_persist_failures.load(Ordering::Relaxed)}diff --git a/engine/crates/xerj-fts/src/analyzer.rs b/engine/crates/xerj-fts/src/analyzer.rsindex 490c749df..53716ff64 100644--- a/engine/crates/xerj-fts/src/analyzer.rs+++ b/engine/crates/xerj-fts/src/analyzer.rs@@ -1333,7 +1333,10 @@ impl AnalyzerRegistry {/// 6. `normalizer` — accepted by ES for `keyword` fields, never built here;/// 7. an analyzer whose `filter` is not an array or whose `tokenizer` is/// not a string: `apply_settings` drops the value shape-first and falls- /// back to "no filters" / `standard`.+ /// back to "no filters" / `standard`;+ /// 8. a `synonym` filter whose `synonyms` is not an array of strings —+ /// dropped the same shape-first way, leaving a filter that expands+ /// nothing.////// # What is NOT checked///@@ -1373,6 +1376,34 @@ impl AnalyzerRegistry {.and_then(|v| v.as_str()).unwrap_or("");if SUPPORTED_FILTER_TYPES.contains(&filter_type) {+ // A supported TYPE can still be given a wrong-shaped+ // option, and `apply_settings` drops those shape-first:+ // `synonyms` is read with `and_then(as_array)` and its+ // entries with `filter_map(as_str)`, so+ // `"synonyms": "fast,quick"` builds a synonym filter with+ // ZERO rules — registered, referenced, expanding nothing,+ // and reported nowhere. Same silent-loss class as the+ // analyzer-level `filter`/`tokenizer` shapes below.+ if filter_type == "synonym" {+ match filter_def.get("synonyms") {+ None => {}+ Some(serde_json::Value::Array(arr)) => {+ for v in arr {+ if !v.is_string() {+ problems.push(format!(+ "token filter [{filter_name}]: `synonyms` entries \+ must be rule strings, got {v} — it would be dropped"+ ));+ }+ }+ }+ Some(other) => problems.push(format!(+ "token filter [{filter_name}]: `synonyms` must be an array of \+ rule strings, got {other} — it would be ignored and the filter \+ would expand nothing"+ )),+ }+ }continue;}// The `type` is one `apply_settings` cannot BUILD — but that isdiff --git a/engine/crates/xerj-wasm/src/pipeline.rs b/engine/crates/xerj-wasm/src/pipeline.rsindex de8c583f0..a7b17d8b4 100644--- a/engine/crates/xerj-wasm/src/pipeline.rs+++ b/engine/crates/xerj-wasm/src/pipeline.rs@@ -436,6 +436,22 @@ fn build_plugin("set" => {let field = str_field(config, "field")?;+ // Elasticsearch's `set` takes EITHER `value` or `copy_from` (7.x+),+ // where `copy_from` names another field to read the value out of+ // the document at ingest time. `SetPlugin` only holds a literal+ // value, so `copy_from` cannot be honoured — but it is a valid ES+ // processor, which makes the gap xerj's, not the caller's. Reported+ // as an unsupported OPTION (accept the definition, refuse the+ // write) rather than falling through to `missing 'value'`, which+ // told the caller their correct ES pipeline was malformed.+ if config.get("copy_from").is_some() {+ return Err(StageBuildError::UnsupportedOption {+ option: "copy_from".to_string(),+ reason: "xerj's set stage assigns a literal `value`; copying the value \+ from another field at ingest time is not implemented"+ .to_string(),+ });+ }let value = config.get("value").cloned()