AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(es-compat): numeric and boolean fields enforce their declared type (#781) (#887)
xerj-org41a760a421a0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b6f6feb0d..5e5bd0eda 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,74 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A field holding both numbers and strings in one segment no longer loses the+ numeric documents from `term` / `terms`.** The doc-values builder collects a+ segment's values into a numeric map (numbers and booleans) and a keyword map+ (strings), then writes both into one column set — so a *type-mixed* field had+ its numeric column silently overwritten by the keyword one, leaving every+ number/boolean document filed as null in a column the query prefilter treats+ as exact. Those documents were then dropped from the result with no error.+ Such a field now ships **no** doc-values column at all, exactly as a+ multi-valued field already does, and every consumer falls back to the+ stored-source scan: correct, at scan cost rather than column speed. Reindex+ onto a single type to get the column back. Whether it bit you depended on+ core count — a many-core host scatters a flush across shards and each segment+ stays single-typed, so this reproduced only where documents of both kinds+ landed in one segment.++- **A field mapped `integer`, `long`, `float` or `boolean` now enforces that+ type on write** ([#781](https://github.com/xerj-org/xerj/issues/781)). The+ declared type was enforced for nothing: `1.9` into an `integer` field was+ stored as `1.9`, `9999999999` was kept exact, `"abc"` was stored as a+ string, `"yes"` went into a `boolean`, and — per the issue's follow-up — an+ entire nested object `{"bad":"x"}` was indexed into an `integer`. Every one+ of those answered `201 created`.++ The consequence was wrong hits, not just leniency. With `1.9` sitting in an+ `integer` field, `range {"i":{"gte":1.5}}` **matched** in XERJ and would not+ in ES (which indexes the truncated `1`), while `term {"i":1}` matched in ES+ and returned nothing here — the same query, over the same document, under+ the same mapping, giving different answers.++ Ingest now applies ES 8.x's own rules (`coerce` defaults to `true`): `1.9`+ → `1`, `"5"` → `5`, `"1.9"` → `1`, `"true"` → `true`; out-of-range for the+ declared width, an unparseable string, an object, an array element that is+ none of those, and anything but `true`/`false` in a `boolean` are refused+ with a 400 `document_parsing_exception` — a per-item 400 under `_bulk`. An+ explicit `"coerce": false` additionally refuses a decimal part and a string,+ and `"ignore_malformed": true` still wins: that field's bad values are+ dropped into `_ignored` rather than failing the document.++ **What this changes about stored data, and what it means on upgrade.** A+ coerced value is rewritten in `_source`: ES keeps `_source` verbatim and+ coerces only the indexed value, but XERJ indexes from the stored source, so+ matching ES's *hits* costs source fidelity here. A document written+ `{"i": 1.9, "b": "false"}` now reads back `{"i": 1, "b": false}`. An index+ that spans this upgrade therefore holds **both** spellings of the same+ logical value — documents written before hold `"false"` / `"5"`, documents+ written after hold `false` / `5` — and no reindex is performed for you.++ That mixed index still answers correctly, and closing that gap is part of+ this change. Because the canonicalisation moves a declared `boolean` from+ the string spelling to a real JSON boolean in `_source`, the **query** side+ now runs `term` / `terms` values on a `boolean` field through the same+ predicate — ES parses a query value with the same `Booleans` its field+ mapper uses, so `"true"` and `true` name one term. Without it,+ `terms {"b":["true"]}` found nothing on a doc-values-only boolean field+ while the equivalent `term` still matched. The `_source` scan comparator+ relates the two spellings in both directions (as it already did for a number+ and its string spelling), for single- and multi-valued fields alike, and a+ `terms` aggregation buckets them together — so one query sees the whole+ index rather than half of it. **Reindex** if you want a uniform `_source`;+ nothing breaks if you do not.++ Also narrowed deliberately, all documented in `xerj_common::field_coercion`:+ the index-level `index.mapping.coerce` setting is not read (only the+ field-level parameter), `scaled_float` is range-checked but not quantised by+ its `scaling_factor`, an empty string is left alone rather than dropped, and+ `_update` is not re-validated — it merges into a document already checked on+ write.+- **A top-level `knn` beside a `query` now scores as a sum over the union ofboth halves, aggregations included**([#825](https://github.com/xerj-org/xerj/issues/825)). ES 8.x defines thisdiff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 396eab9c5..e9a7288f6 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -2831,6 +2831,13 @@ pub async fn index_doc_auto(doc};+ // Declared numeric/boolean types are enforced after the pipeline has run,+ // so a pipeline-produced value is held to the same rules as a caller's.+ let doc = match enforce_field_types(&state, &index, "", doc) {+ Ok(d) => d,+ Err(resp) => return resp,+ };+let idx = match state.engine.get_or_create_index(&index) {Ok(i) => i,Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),@@ -2959,6 +2966,13 @@ pub async fn index_doc(};// ?refresh=true|wait_for — accepted silently; memtable is immediately visible.+ // Declared numeric/boolean types are enforced after the pipeline has run,+ // so a pipeline-produced value is held to the same rules as a caller's.+ let doc = match enforce_field_types(&state, &index, &id, doc) {+ Ok(d) => d,+ Err(resp) => return resp,+ };+let idx = match state.engine.get_or_create_index(&index) {Ok(i) => i,Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),@@ -3085,6 +3099,13 @@ pub async fn create_doc(doc};+ // `_create` skipped every mapping-aware validation stage; the declared+ // numeric/boolean types are now enforced here too (issue #781).+ let doc = match enforce_field_types(&state, &index, &id, doc) {+ Ok(d) => d,+ Err(resp) => return resp,+ };+let idx = match state.engine.get_or_create_index(&index) {Ok(i) => i,Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),@@ -35519,6 +35540,62 @@ pub(crate) fn rewrite_bulk_ignore_malformed(out}+/// The 400 envelope ES sends when a document cannot be parsed against the+/// mapping — the same shape the range/`copy_to` rejection above emits, with+/// the specific `caused_by` sentence ES attaches underneath.+fn document_parsing_error(reason: &str, caused_by: &str) -> Response {+ (+ StatusCode::BAD_REQUEST,+ Json(json!({+ "error": {+ "root_cause": [{ "type": "document_parsing_exception", "reason": reason }],+ "type": "document_parsing_exception",+ "reason": reason,+ "caused_by": { "type": "illegal_argument_exception", "reason": caused_by }+ },+ "status": 400+ })),+ )+ .into_response()+}++/// Enforce the declared type of every numeric/boolean field on a single-doc+/// write (issue #781).+///+/// Returns the document with ES's coercions applied (`1.9` → `1` in an+/// `integer` field, `"5"` → `5`, `"true"` → `true`), or the 400+/// `document_parsing_exception` ES answers for a value no coercion can+/// rescue — an unparseable string, an out-of-range integer, an object where a+/// number belongs, `"yes"` in a boolean.+///+/// The rules live in [`xerj_common::field_coercion`], shared with the+/// per-item `_bulk` loop in `xerj_engine::bulk`, so the two write paths cannot+/// drift apart the way the date validators once did.+pub(crate) fn enforce_field_types(+ state: &AppState,+ index: &str,+ doc_id: &str,+ doc: Value,+) -> Result<Value, Response> {+ let Some(mapping) = state.engine.index_mappings.get(index).map(|v| v.clone()) else {+ return Ok(doc);+ };+ // Borrow the properties out of the cloned blob rather than cloning them+ // again — one clone per write is what `apply_ignore_malformed` already+ // pays to keep the DashMap shard lock off the coercion walk.+ let Some(props) = xerj_common::field_coercion::mapping_properties(&mapping) else {+ return Ok(doc);+ };+ let mut obj = match doc {+ Value::Object(o) => o,+ other => return Ok(other),+ };+ match xerj_common::field_coercion::coerce_document(&mut obj, props) {+ Ok(_) => Ok(Value::Object(obj)),+ Err(bad) => Err(document_parsing_error(&bad.reason(doc_id), &bad.detail)),+ }+}+pub(crate) fn apply_ignore_malformed(state: &AppState, index: &str, doc: Value) -> Value {let Some(mapping) = state.engine.index_mappings.get(index).map(|v| v.clone()) else {return doc;diff --git a/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rs b/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rsnew file mode 100644index 000000000..9ebc3a7a7--- /dev/null+++ b/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rs@@ -0,0 +1,614 @@+//! Numeric and boolean fields enforce their declared type, from the outside —+//! issue #781.+//!+//! A field mapped `integer` used to accept and keep literally anything: `1.9`+//! stayed `1.9`, `9999999999` stayed exact, `"abc"` was stored as a string,+//! and even `{"bad": "x"}` was indexed. Every one of those was a `201`.+//!+//! The worst of it is not leniency, it is wrong hits. With `1.9` sitting in an+//! `integer` field, `range {"i": {"gte": 1.5}}` MATCHED in xerj and would not+//! in ES (which indexes the truncated `1`), while `term {"i": 1}` matched in+//! ES and returned nothing here. Same mapping, same document, same query,+//! different answers — `wrong_results_the_declared_type_used_to_produce`+//! below is that case, pinned.+//!+//! ES 8.x semantics, with `coerce` defaulting to `true`:+//!+//! * `1.9` → `1` (truncated toward zero), `"5"` → `5`, `"1.9"` → `1`+//! * `"true"`/`"false"` → the booleans+//! * out of range for the declared width → **400**+//! * unparseable string, object, array, `"yes"` in a boolean → **400**+//! * `"coerce": false` additionally refuses the decimal part and the string+//!+//! Every write path that knows the typed schema is covered here: `PUT+//! /{index}/_doc/{id}`, `POST /{index}/_doc` (auto-id), `PUT+//! /{index}/_create/{id}`, and `_bulk` — including the turbo batch path+//! auto-id bulk takes, which never parses the document body and so had to be+//! reached by a partition-time check rather than the per-item loop.+//!+//! Elasticsearch is referenced for semantics only. It is AGPL-3.0/SSPL-1.0/+//! Elastic-2.0 licensed and no code from it is reproduced here.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++async fn app() -> (axum::Router, tempfile::TempDir) {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ (xerj_api::router::build_es_compat_router(state), dir)+}++async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {+ let response = app.clone().oneshot(req).await.expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, value)+}++async fn json_req(+ app: &axum::Router,+ method: &str,+ path: &str,+ body: Value,+) -> (StatusCode, Value) {+ send(+ app,+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+}++async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {+ send(+ app,+ Request::get(path).body(Body::empty()).expect("request"),+ )+ .await+}++async fn bulk(app: &axum::Router, path: &str, ndjson: &str) -> (StatusCode, Value) {+ send(+ app,+ Request::post(path)+ .header("content-type", "application/x-ndjson")+ .body(Body::from(ndjson.to_owned()))+ .expect("request"),+ )+ .await+}++/// `i` integer, `l` long, `f` float, `b` boolean, `strict` integer with+/// `coerce: false`, `lenient` integer with `ignore_malformed: true`, plus a+/// nested object and a `keyword` this check must not touch.+async fn typed_index(name: &str) -> (axum::Router, tempfile::TempDir) {+ let (app, dir) = app().await;+ let (status, _) = json_req(+ &app,+ "PUT",+ &format!("/{name}"),+ json!({+ "mappings": {+ "properties": {+ "i": { "type": "integer" },+ "l": { "type": "long" },+ "f": { "type": "float" },+ "b": { "type": "boolean" },+ "strict": { "type": "integer", "coerce": false },+ "lenient": { "type": "integer", "ignore_malformed": true },+ "k": { "type": "keyword" },+ "inner": { "properties": { "n": { "type": "integer" } } }+ }+ }+ }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "mapping should be accepted");+ (app, dir)+}++async fn put_doc(app: &axum::Router, index: &str, id: &str, doc: Value) -> (StatusCode, Value) {+ json_req(app, "PUT", &format!("/{index}/_doc/{id}"), doc).await+}++/// Flush the memtable into a segment.+///+/// This matters for the query-side tests at the bottom of this file: while a+/// document is memtable-resident, `term`/`terms` are answered by the doc-values+/// fast path, which compares STRINGIFIED values and so is blind to the+/// boolean-versus-`"boolean"` spelling. It is the flushed segment path — a+/// doc-values prefilter refined by an exact re-test of `_source` — that+/// distinguishes them, which is why the ES-YAML case that caught this+/// (`search/390_doc_values_search.yml`) does `indices.refresh` before it+/// queries. A test that skips the refresh passes on the broken code.+async fn refresh(app: &axum::Router, index: &str) {+ let (status, body) = json_req(app, "POST", &format!("/{index}/_refresh"), json!({})).await;+ assert_eq!(status, StatusCode::OK, "refresh should succeed: {body}");+}++async fn search(app: &axum::Router, index: &str, query: Value) -> Value {+ let (status, body) = json_req(+ app,+ "POST",+ &format!("/{index}/_search"),+ json!({ "query": query }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "search should succeed: {body}");+ body+}++fn hit_count(body: &Value) -> u64 {+ body["hits"]["total"]["value"].as_u64().unwrap_or(0)+}++fn is_parsing_rejection(status: StatusCode, body: &Value, field: &str) -> bool {+ status == StatusCode::BAD_REQUEST+ && body["error"]["type"] == "document_parsing_exception"+ && body["error"]["reason"]+ .as_str()+ .map(|r| r.contains(&format!("failed to parse field [{field}]")))+ .unwrap_or(false)+}++// ─────────────────────────────────────────────────────────────────────────────+// The wrong-results case that makes this a bug and not a style preference.+// ─────────────────────────────────────────────────────────────────────────────++/// `1.9` into an `integer` field. ES truncates it to `1` at ingest, so+/// `range {gte: 1.5}` misses and `term {i: 1}` hits. Before the fix xerj+/// stored `1.9` verbatim and answered both queries the other way round.+#[tokio::test]+async fn wrong_results_the_declared_type_used_to_produce() {+ let (app, _dir) = typed_index("t").await;++ let (status, _) = put_doc(&app, "t", "1", json!({ "i": 1.9 })).await;+ assert_eq!(status, StatusCode::CREATED);++ // The stored value is the integer ES would have indexed.+ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(doc["_source"]["i"], json!(1), "1.9 should truncate to 1");++ let gte = search(&app, "t", json!({ "range": { "i": { "gte": 1.5 } } })).await;+ assert_eq!(+ hit_count(>e),+ 0,+ "an integer field holding ES's 1 must not match gte 1.5: {gte}"+ );++ let term = search(&app, "t", json!({ "term": { "i": 1 } })).await;+ assert_eq!(+ hit_count(&term),+ 1,+ "term 1 must find the truncated value: {term}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// Coercion: the values ES accepts and rewrites.+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn numeric_strings_and_boolean_spellings_are_coerced() {+ let (app, _dir) = typed_index("t").await;++ let (status, _) = put_doc(+ &app,+ "t",+ "1",+ json!({ "i": "5", "l": "-7", "f": "2.5", "b": "true", "inner": { "n": "3" } }),+ )+ .await;+ assert_eq!(status, StatusCode::CREATED);++ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(doc["_source"]["i"], json!(5));+ assert_eq!(doc["_source"]["l"], json!(-7));+ assert_eq!(doc["_source"]["f"], json!(2.5));+ assert_eq!(doc["_source"]["b"], json!(true));+ assert_eq!(doc["_source"]["inner"]["n"], json!(3), "nested too");++ // Coerced, therefore queryable as the number it now is.+ let term = search(&app, "t", json!({ "term": { "i": 5 } })).await;+ assert_eq!(hit_count(&term), 1, "coerced string is numerically indexed");+}++#[tokio::test]+async fn arrays_are_coerced_element_wise() {+ let (app, _dir) = typed_index("t").await;++ let (status, _) = put_doc(&app, "t", "1", json!({ "i": [1.9, "3", 4] })).await;+ assert_eq!(status, StatusCode::CREATED);++ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(doc["_source"]["i"], json!([1, 3, 4]));+}++#[tokio::test]+async fn values_that_already_match_the_type_are_left_alone() {+ let (app, _dir) = typed_index("t").await;++ // `2.0` is already the integer ES indexes — no `_source` churn — and a+ // `keyword` field is not this check's business at all.+ let (status, _) = put_doc(+ &app,+ "t",+ "1",+ json!({ "i": 2.0, "l": 42, "b": false, "k": 7, "unmapped": { "any": "shape" } }),+ )+ .await;+ assert_eq!(status, StatusCode::CREATED);++ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(doc["_source"]["l"], json!(42));+ assert_eq!(doc["_source"]["b"], json!(false));+ assert_eq!(doc["_source"]["k"], json!(7), "keyword is not coerced");+ assert_eq!(doc["_source"]["unmapped"], json!({ "any": "shape" }));+}++// ─────────────────────────────────────────────────────────────────────────────+// Rejection: the values ES refuses outright.+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn a_non_numeric_string_is_refused() {+ let (app, _dir) = typed_index("t").await;+ let (status, body) = put_doc(&app, "t", "1", json!({ "i": "abc" })).await;+ assert!(+ is_parsing_rejection(status, &body, "i"),+ "expected 400 document_parsing_exception, got {status}: {body}"+ );+ assert!(+ body["error"]["caused_by"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("abc"),+ "the caused_by should name the offending input: {body}"+ );+ // Nothing was written.+ assert_eq!(get(&app, "/t/_doc/1").await.0, StatusCode::NOT_FOUND);+}++#[tokio::test]+async fn an_out_of_range_value_is_refused_even_though_it_is_an_integer() {+ let (app, _dir) = typed_index("t").await;++ let (status, body) = put_doc(&app, "t", "1", json!({ "i": 9999999999i64 })).await;+ assert!(+ is_parsing_rejection(status, &body, "i"),+ "9999999999 does not fit an integer: {status} {body}"+ );++ // The same number in a `long` is perfectly fine.+ let (status, _) = put_doc(&app, "t", "2", json!({ "l": 9999999999i64 })).await;+ assert_eq!(status, StatusCode::CREATED);+}++#[tokio::test]+async fn a_boolean_takes_only_true_and_false() {+ let (app, _dir) = typed_index("t").await;++ let (status, body) = put_doc(&app, "t", "1", json!({ "b": "yes" })).await;+ assert!(+ is_parsing_rejection(status, &body, "b"),+ "\"yes\" is not a boolean: {status} {body}"+ );++ let (status, body) = put_doc(&app, "t", "2", json!({ "b": 1 })).await;+ assert!(+ is_parsing_rejection(status, &body, "b"),+ "1 is not a boolean in ES 8.x: {status} {body}"+ );+}++/// The comment on the issue: an OBJECT went into an integer field and the+/// document was indexed, so the declared type enforced nothing at all.+#[tokio::test]+async fn an_object_is_not_a_number() {+ let (app, _dir) = typed_index("t").await;+ let (status, body) = put_doc(&app, "t", "1", json!({ "i": { "bad": "x" } })).await;+ assert!(+ is_parsing_rejection(status, &body, "i"),+ "an object in an integer field must be refused: {status} {body}"+ );+ assert!(+ body["error"]["reason"]+ .as_str()+ .unwrap_or_default()+ .contains("bad=x"),+ "the preview should render the object ES-style: {body}"+ );+}++#[tokio::test]+async fn a_bad_nested_field_is_named_by_its_dotted_path() {+ let (app, _dir) = typed_index("t").await;+ let (status, body) = put_doc(&app, "t", "1", json!({ "inner": { "n": "abc" } })).await;+ assert!(+ is_parsing_rejection(status, &body, "inner.n"),+ "expected the dotted path in the reason: {status} {body}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// The two mapping parameters that change the answer.+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn coerce_false_refuses_what_the_default_would_fix() {+ let (app, _dir) = typed_index("t").await;++ for bad in [json!(1.9), json!("5")] {+ let (status, body) = put_doc(&app, "t", "1", json!({ "strict": bad })).await;+ assert!(+ is_parsing_rejection(status, &body, "strict"),+ "coerce:false should refuse {bad}: {status} {body}"+ );+ }++ // An in-range integer still goes in.+ let (status, _) = put_doc(&app, "t", "1", json!({ "strict": 3 })).await;+ assert_eq!(status, StatusCode::CREATED);+}++#[tokio::test]+async fn ignore_malformed_still_wins_over_rejection() {+ let (app, _dir) = typed_index("t").await;++ // The field asked for the bad value to be dropped into `_ignored`, not for+ // the document to be refused — that contract must survive this change.+ let (status, _) = put_doc(&app, "t", "1", json!({ "lenient": "abc", "i": 4 })).await;+ assert_eq!(status, StatusCode::CREATED, "ignore_malformed must not 400");++ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(doc["_source"]["i"], json!(4));+ assert!(+ doc["_source"].get("lenient").is_none(),+ "the malformed value should have been dropped: {doc}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// Every other write path agrees with `PUT /_doc/{id}`.+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn the_auto_id_and_create_endpoints_enforce_the_same_rules() {+ let (app, _dir) = typed_index("t").await;++ let (status, body) = json_req(&app, "POST", "/t/_doc", json!({ "i": "abc" })).await;+ assert!(+ is_parsing_rejection(status, &body, "i"),+ "POST /_doc must refuse it too: {status} {body}"+ );++ let (status, body) = json_req(&app, "PUT", "/t/_create/9", json!({ "b": "yes" })).await;+ assert!(+ is_parsing_rejection(status, &body, "b"),+ "PUT /_create/{{id}} must refuse it too: {status} {body}"+ );++ // …and coerce on the happy path.+ let (status, _) = json_req(&app, "PUT", "/t/_create/10", json!({ "i": 1.9 })).await;+ assert_eq!(status, StatusCode::CREATED);+ let (_, doc) = get(&app, "/t/_doc/10").await;+ assert_eq!(doc["_source"]["i"], json!(1));+}++/// `_bulk` reports per item and isolates failures — including on the auto-id+/// turbo batch path, which never parses the document body.+#[tokio::test]+async fn bulk_rejects_per_item_and_keeps_the_good_ones() {+ let (app, _dir) = typed_index("t").await;++ let ndjson = concat!(+ "{\"index\":{\"_id\":\"ok\"}}\n{\"i\":1.9}\n",+ "{\"index\":{\"_id\":\"bad-str\"}}\n{\"i\":\"abc\"}\n",+ "{\"index\":{\"_id\":\"bad-range\"}}\n{\"i\":9999999999}\n",+ "{\"index\":{\"_id\":\"bad-obj\"}}\n{\"i\":{\"bad\":\"x\"}}\n",+ "{\"index\":{\"_id\":\"bad-bool\"}}\n{\"b\":\"yes\"}\n",+ "{\"create\":{\"_id\":\"ok2\"}}\n{\"i\":\"7\"}\n",+ );+ let (status, body) = bulk(&app, "/t/_bulk", ndjson).await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["errors"], json!(true), "{body}");++ let items = body["items"].as_array().expect("items");+ assert_eq!(items.len(), 6);+ let code = |n: usize| -> u64 {+ items[n]+ .as_object()+ .and_then(|o| o.values().next())+ .and_then(|v| v["status"].as_u64())+ .unwrap_or(0)+ };+ assert_eq!(code(0), 201, "the coercible one lands: {body}");+ for (n, what) in [(1, "string"), (2, "range"), (3, "object"), (4, "boolean")] {+ assert_eq!(code(n), 400, "bad {what} should be a per-item 400: {body}");+ }+ assert_eq!(code(5), 201, "create coerces too: {body}");++ // Only the two good documents exist, with ES's values.+ let (_, ok) = get(&app, "/t/_doc/ok").await;+ assert_eq!(ok["_source"]["i"], json!(1));+ let (_, ok2) = get(&app, "/t/_doc/ok2").await;+ assert_eq!(ok2["_source"]["i"], json!(7));+ assert_eq!(get(&app, "/t/_doc/bad-str").await.0, StatusCode::NOT_FOUND);++ let (_, count) = get(&app, "/t/_count").await;+ assert_eq!(count["count"], json!(2), "{count}");+}++/// Auto-id bulk is the turbo batch path: the doc body is never parsed and the+/// whole group is appended in one shot, so it needed its own check. Without+/// it, every bad value in this batch was a silent 201.+#[tokio::test]+async fn the_bulk_turbo_path_is_not_a_way_around_the_check() {+ let (app, _dir) = typed_index("t").await;++ let ndjson = concat!(+ "{\"index\":{}}\n{\"i\":1.9}\n",+ "{\"index\":{}}\n{\"i\":\"abc\"}\n",+ "{\"index\":{}}\n{\"i\":\"6\"}\n",+ );+ let (status, body) = bulk(&app, "/t/_bulk", ndjson).await;+ assert_eq!(status, StatusCode::OK);+ assert_eq!(body["errors"], json!(true), "{body}");++ let (_, count) = get(&app, "/t/_count").await;+ assert_eq!(count["count"], json!(2), "only the two valid docs: {count}");++ // Both survivors were coerced, so both are numerically queryable.+ let one = search(&app, "t", json!({ "term": { "i": 1 } })).await;+ assert_eq!(hit_count(&one), 1, "{one}");+ let six = search(&app, "t", json!({ "term": { "i": 6 } })).await;+ assert_eq!(hit_count(&six), 1, "{six}");+}++/// An index with no numeric or boolean field skips the check entirely — the+/// gate that keeps the turbo path free must not change behaviour for it.+#[tokio::test]+async fn an_index_with_no_numeric_mapping_is_untouched() {+ let (app, _dir) = app().await;+ let (status, _) = json_req(+ &app,+ "PUT",+ "/plain",+ json!({ "mappings": { "properties": { "msg": { "type": "text" } } } }),+ )+ .await;+ assert_eq!(status, StatusCode::OK);++ let (status, _) = put_doc(&app, "plain", "1", json!({ "msg": "hi", "n": "abc" })).await;+ assert_eq!(status, StatusCode::CREATED);+ let (_, doc) = get(&app, "/plain/_doc/1").await;+ assert_eq!(doc["_source"]["n"], json!("abc"), "unmapped stays verbatim");+}++// ─────────────────────────────────────────────────────────────────────────────+// The query side of the same coercion. Rewriting `_source` moves the STORED+// spelling, so a query operand written the pre-coercion way has to keep+// matching — this is the case CI caught in+// `tests/es-compat-yaml/yaml/search/390_doc_values_search.yml`, and the reason+// the first cut of this change regressed conformance from 0 failed to 1.+// ─────────────────────────────────────────────────────────────────────────────++/// A document written `{"b": "true"}` is now STORED as `{"b": true}`. ES accepts+/// either spelling in a `term`/`terms` operand against a `boolean` field+/// (`terms {b: ["true"]}` is literally what the ES YAML suite asserts), so both+/// must still find it. Pre-fix, `terms` with the string operand found nothing:+/// `terms` verifies each admitted doc with exact JSON equality, and `true` is+/// not `"true"`.+#[tokio::test]+async fn a_query_in_the_pre_coercion_spelling_still_matches_the_coerced_document() {+ let (app, _dir) = typed_index("t").await;++ let (status, _) = put_doc(&app, "t", "1", json!({ "b": "true", "i": "5" })).await;+ assert_eq!(status, StatusCode::CREATED);+ let (_, doc) = get(&app, "/t/_doc/1").await;+ assert_eq!(+ doc["_source"]["b"],+ json!(true),+ "ingest coerced the spelling"+ );+ assert_eq!(doc["_source"]["i"], json!(5));+ refresh(&app, "t").await;++ for query in [+ json!({ "terms": { "b": ["true"] } }),+ json!({ "terms": { "b": [true] } }),+ json!({ "term": { "b": "true" } }),+ json!({ "term": { "b": true } }),+ json!({ "terms": { "b": ["false", "true"] } }),+ json!({ "terms": { "i": ["5"] } }),+ json!({ "terms": { "i": [5] } }),+ ] {+ let body = search(&app, "t", query.clone()).await;+ assert_eq!(+ hit_count(&body),+ 1,+ "both spellings of {query} must find the coerced document: {body}"+ );+ }++ // Tolerance, not blindness: the other boolean still must not match.+ for query in [+ json!({ "terms": { "b": ["false"] } }),+ json!({ "term": { "b": false } }),+ json!({ "terms": { "b": ["yes"] } }),+ json!({ "terms": { "i": ["6"] } }),+ ] {+ let body = search(&app, "t", query.clone()).await;+ assert_eq!(hit_count(&body), 0, "{query} must not match: {body}");+ }+}++/// An index that spans the upgrade holds BOTH representations of the same+/// logical value: documents written before this change keep `"true"` / `"5"`,+/// documents written after hold `true` / `5`. One query must not silently see+/// half the index. `_update` is the shortest way to plant the old spelling —+/// it merges into a document that was already validated on write and is+/// deliberately not re-validated, which is exactly the shape a pre-upgrade+/// document has.+#[tokio::test]+async fn an_index_holding_both_spellings_answers_every_query_with_both_docs() {+ let (app, _dir) = typed_index("t").await;++ // Post-upgrade document: coerced on the way in.+ let (status, _) = put_doc(&app, "t", "new", json!({ "b": "true", "i": "5" })).await;+ assert_eq!(status, StatusCode::CREATED);++ // Pre-upgrade document: the raw spelling, planted past the check.+ let (status, _) = put_doc(&app, "t", "legacy", json!({ "k": "a" })).await;+ assert_eq!(status, StatusCode::CREATED);+ let (status, body) = json_req(+ &app,+ "POST",+ "/t/_update/legacy",+ json!({ "doc": { "b": "true", "i": "5" } }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "update should apply: {body}");++ refresh(&app, "t").await;++ let (_, legacy) = get(&app, "/t/_doc/legacy").await;+ let (_, fresh) = get(&app, "/t/_doc/new").await;+ assert_eq!(+ legacy["_source"]["b"],+ json!("true"),+ "the legacy spelling is what makes this test meaningful"+ );+ assert_eq!(fresh["_source"]["b"], json!(true));++ for query in [+ json!({ "terms": { "b": ["true"] } }),+ json!({ "terms": { "b": [true] } }),+ json!({ "term": { "b": "true" } }),+ json!({ "term": { "b": true } }),+ json!({ "terms": { "i": ["5"] } }),+ json!({ "terms": { "i": [5] } }),+ json!({ "term": { "i": 5 } }),+ ] {+ let body = search(&app, "t", query.clone()).await;+ assert_eq!(+ hit_count(&body),+ 2,+ "{query} must see both spellings in one index: {body}"+ );+ }+}diff --git a/engine/crates/xerj-common/src/field_coercion.rs b/engine/crates/xerj-common/src/field_coercion.rsnew file mode 100644index 000000000..4094b72b3--- /dev/null+++ b/engine/crates/xerj-common/src/field_coercion.rs@@ -0,0 +1,891 @@+//! Ingest-time coercion and type enforcement for numeric and boolean fields.+//!+//! THE single predicate for "what would Elasticsearch index for this value in+//! a field of this declared type?". Every write path that knows the typed+//! schema must call it and nothing else — the single-document `_doc`/`_create`+//! handlers in `xerj-api`, the per-item `_bulk` loop in `xerj-engine`, and the+//! `ignore_malformed` walker. When two ingest paths grew independent copies of+//! a validation rule before (the date predicate, `xerj_query::dates`), the+//! *strict* one ended up the laxer of the pair; sharing one function is what+//! stops that from happening again.+//!+//! ## What ES actually does (8.x, `coerce` defaults to `true`)+//!+//! For a numeric field, `NumberFieldMapper` parses the JSON token through+//! `XContentParser::intValue(coerce)` / `doubleValue(coerce)` and then range-+//! checks against the target width:+//!+//! | input into `integer` | ES 8.x |+//! |----------------------|-------------------------------------------|+//! | `1` | `1` |+//! | `1.9` | `1` — truncated toward zero |+//! | `"5"` / `"1.9"` | `5` / `1` — string coerced then truncated |+//! | `9999999999` | **400**, out of range for an integer |+//! | `"abc"` | **400**, cannot parse |+//! | `{"bad":"x"}`, `[…]` | **400**, object/array is not a number |+//!+//! With `"coerce": false` on the field, a decimal part and a string are both+//! refused; the range check applies either way. `boolean` has no `coerce`+//! knob at all: `true`/`false` and the strings `"true"`/`"false"` are the+//! whole accepted set, and anything else — including `1`, `0` and `"yes"` —+//! is a 400.+//!+//! ## The XERJ divergence this closes (issue #781)+//!+//! XERJ stored whatever arrived, so an `integer` field holding `1.9` matched+//! `range {gte: 1.5}` and missed `term {i: 1}` — the same query over the same+//! declared-integer field returned different hits than ES. Coercing at ingest+//! is what makes the indexed value and the declared type agree.+//!+//! ## Rewriting `_source` moves the stored spelling — the query side pairs+//!+//! ES keeps `_source` byte-verbatim and coerces only the *indexed* value.+//! XERJ indexes from the stored source, so agreeing with ES on **hits** costs+//! source fidelity: ES returns `1.9` from `_source` where XERJ now returns+//! `1`, and a document written `{"b": "false"}` is stored `{"b": false}`.+//!+//! Two consequences, both handled — do not undo either half without the+//! other:+//!+//! 1. **A query operand written in the pre-coercion spelling must still+//! match.** ES accepts `terms {b: ["false"]}` against a `boolean` field+//! (`search/390_doc_values_search.yml` asserts exactly that), and once the+//! stored value is a real `false` an exact-JSON comparison stops finding+//! it. So `xerj_engine::index::rewrite_query_aliases` runs a `term`/`terms`+//! value on a declared `boolean` through [`coerce_value`] — THIS predicate,+//! on the read side — before any path sees it. The first cut of this module+//! shipped without that and regressed the conformance suite from 0 failed+//! to 1.+//! 2. **An index can hold BOTH representations.** Documents written before+//! this change keep `"true"` / `"5"`; documents written after hold `true` /+//! `5`; `_update` (deliberately not re-validated) can still merge the old+//! spelling into a new document. Canonicalising the operand alone would+//! then miss the older half, so `xerj_engine::index::json_scalar_equal`+//! also relates a boolean to its string spelling in BOTH directions —+//! for a scalar and for an element of a multi-valued field — alongside the+//! number/string pair it already had. Terms aggregations already bucket the+//! two spellings together. A reindex is still the way to make an index+//! uniform, but nothing breaks without one.+//!+//! ## Known, deliberate narrowing+//!+//! * An empty string is left alone rather than dropped. ES treats `""` in a+//! numeric field as "no value"; dropping it would rewrite `_source` for a+//! case that is not a wrong-results bug.+//! * Only the field-level `"coerce": false` is honoured, not the index-level+//! `index.mapping.coerce` setting.+//! * `scaled_float` is range-checked as a `double` and **not** quantised by+//! its `scaling_factor`. ES indexes `Math.round(value * scaling_factor)`;+//! reproducing that here would rewrite `_source` into the quantised value,+//! which loses information the caller sent for no wrong-results gain.+//! * `_update` is not re-validated: it merges into a document that was already+//! checked on write, matching how the existing date check scopes itself to+//! `index` / `create`.+//! * Enforcement applies only to **declared** mappings. `index_mappings` is+//! written by index-create-with-mappings and `PUT /_mapping` only, never by+//! dynamic inference, so a schemaless index is untouched.+//!+//! Elasticsearch is referenced for semantics only. It is AGPL-3.0/SSPL-1.0/+//! Elastic-2.0 licensed and no code from it is reproduced here.++use serde_json::{Map, Number, Value};++/// What Elasticsearch would do with one value in a typed field.+#[derive(Debug, Clone, PartialEq)]+pub enum Coercion {+ /// Index the value exactly as it arrived.+ AsIs,+ /// ES indexes something else; this is the value to store instead.+ Rewrite(Value),+ /// ES refuses the whole document. Carries the `caused_by`-style detail.+ Reject(String),+}++/// The first field of a document that its declared type refuses.+#[derive(Debug, Clone, PartialEq)]+pub struct BadField {+ /// Full dotted path of the offending field.+ pub field: String,+ /// The declared type that refused it.+ pub ftype: String,+ /// Rendered preview of the offending value, ES-style.+ pub preview: String,+ /// Why it was refused (ES's `caused_by` sentence).+ pub detail: String,+}++impl BadField {+ /// The `reason` string ES puts on a `document_parsing_exception`.+ ///+ /// Same shape as the type-clash rejection `xerj_engine::bulk` already+ /// emits, so the two read identically on the wire.+ pub fn reason(&self, doc_id: &str) -> String {+ format!(+ "failed to parse field [{}] of type [{}] in document with id '{}'. \+ Preview of field's value: '{}'",+ self.field, self.ftype, doc_id, self.preview+ )+ }+}++/// Types this module enforces. Everything else (`text`, `keyword`, `date`,+/// `ip`, `geo_point`, ranges, vectors…) is left to its own validator.+pub fn is_enforced_type(ftype: &str) -> bool {+ matches!(+ ftype,+ "byte"+ | "short"+ | "integer"+ | "long"+ | "unsigned_long"+ | "half_float"+ | "float"+ | "double"+ | "scaled_float"+ | "boolean"+ )+}++/// Inclusive integer bounds of an integral ES type, in `i128` so that both+/// `i64::MIN` and `u64::MAX` are representable exactly — an `f64` bound would+/// round `i64::MAX` up and let `9223372036854775808` through as a `long`.+fn integral_bounds(ftype: &str) -> Option<(i128, i128)> {+ Some(match ftype {+ "byte" => (i8::MIN as i128, i8::MAX as i128),+ "short" => (i16::MIN as i128, i16::MAX as i128),+ "integer" => (i32::MIN as i128, i32::MAX as i128),+ "long" => (i64::MIN as i128, i64::MAX as i128),+ "unsigned_long" => (0, u64::MAX as i128),+ _ => return None,+ })+}++/// ES-style rendering of a value inside an error message: a string shows its+/// text unquoted, an object shows `{k=v, …}`, everything else its JSON form.+pub fn preview_value(v: &Value) -> String {+ match v {+ Value::String(s) => s.clone(),+ Value::Object(o) => {+ let parts: Vec<String> = o+ .iter()+ .map(|(k, val)| match val {+ Value::String(s) => format!("{}={}", k, s),+ other => format!("{}={}", k, other),+ })+ .collect();+ format!("{{{}}}", parts.join(", "))+ }+ other => other.to_string(),+ }+}++fn out_of_range(ftype: &str, v: &Value) -> Coercion {+ Coercion::Reject(format!(+ "Value [{}] is out of range for a[n] {}",+ preview_value(v),+ ftype+ ))+}++fn not_finite(ftype: &str) -> Coercion {+ Coercion::Reject(format!("[{}] supports only finite values", ftype))+}++fn not_a_number(ftype: &str) -> Coercion {+ Coercion::Reject(format!(+ "Current token is not a numeric value, expected a[n] {}",+ ftype+ ))+}++/// The JSON number for an integral value the caller has already clamped into+/// its declared width's bounds, so it fits `i64` or `u64` exactly.+///+/// `as i64` alone would be wrong for `unsigned_long`, whose range runs past+/// `i64::MAX`: a saturating cast turns `1e19` into `i64::MAX` — a silently+/// different value, which is the class of bug this whole module exists to+/// close.+fn number_from_i128(i: i128) -> Value {+ if i > i64::MAX as i128 {+ Value::Number(Number::from(i as u64))+ } else {+ Value::Number(Number::from(i as i64))+ }+}++/// The exact integer part of the finite float `f`, and that value clamped into+/// a field's inclusive `lo..=hi` bounds.+///+/// Rust's float-to-int `as` saturates, so `f.trunc() as i128` is exact for+/// every value any of these widths can hold — no UB, no wraparound — and the+/// `clamp` then reproduces the Java narrowing cast ES applies after its own+/// (double-widened) range check. The caller compares the two: equal means the+/// float already IS the integer ES would index, different means ES's cast+/// changed it.+fn truncate_into(f: f64, lo: i128, hi: i128) -> (i128, i128) {+ let exact = f.trunc() as i128;+ (exact, exact.clamp(lo, hi))+}++/// Coerce one scalar (non-array) value for an integral field.+fn coerce_integral(ftype: &str, coerce: bool, v: &Value) -> Coercion {+ let (lo, hi) = match integral_bounds(ftype) {+ Some(b) => b,+ None => return Coercion::AsIs,+ };+ match v {+ Value::Number(n) => {+ if let Some(i) = n.as_i64() {+ if (i as i128) < lo || (i as i128) > hi {+ return out_of_range(ftype, v);+ }+ Coercion::AsIs+ } else if let Some(u) = n.as_u64() {+ if (u as i128) < lo || (u as i128) > hi {+ return out_of_range(ftype, v);+ }+ Coercion::AsIs+ } else {+ // JSON float token: 1.9, 2.0, 1e40…+ let f = match n.as_f64() {+ Some(f) if f.is_finite() => f,+ _ => return not_finite(ftype),+ };+ if f < lo as f64 || f > hi as f64 {+ return out_of_range(ftype, v);+ }+ // That range check is ES's own — and it is WIDER than the+ // declared width, because Java widens `Long.MAX_VALUE` to the+ // `double` 2^63 to make the comparison. So the float token+ // `9223372036854775808.0` survives it, and ES's `(long)` cast+ // then saturates the value to `Long.MAX_VALUE`. The+ // integer-token branch above is exact (`i128` bounds); this+ // branch has to reproduce BOTH halves of ES's behaviour, or a+ // bare `fract() == 0` shortcut stores a value the declared+ // width cannot hold — the residual leniency the `i128` bound+ // alone did not close.+ let (exact, clamped) = truncate_into(f, lo, hi);+ // Compare in the INTEGER domain. `clamped as f64 == f.trunc()`+ // would be fooled by the very rounding this guards against:+ // `i64::MAX as f64` is 2^63, so the two sides agree on exactly+ // the value that must not pass.+ if f.fract() == 0.0 && exact == clamped {+ // `2.0` already equals the integer ES would index; leave+ // `_source` alone rather than churn it to `2`.+ return Coercion::AsIs;+ }+ if f.fract() != 0.0 && !coerce {+ return Coercion::Reject(format!(+ "Cannot coerce NUMBER to {} — value [{}] has a decimal part",+ ftype, f+ ));+ }+ Coercion::Rewrite(number_from_i128(clamped))+ }+ }+ Value::String(s) => {+ if s.trim().is_empty() {+ // "no value" — see the module note on empty strings.+ return Coercion::AsIs;+ }+ if !coerce {+ return Coercion::Reject(format!("Cannot coerce a string to a[n] {}", ftype));+ }+ let f: f64 = match s.trim().parse::<f64>() {+ Ok(f) if f.is_finite() => f,+ Ok(_) => return not_finite(ftype),+ Err(_) => return Coercion::Reject(format!("For input string: \"{}\"", s)),+ };+ if f < lo as f64 || f > hi as f64 {+ return out_of_range(ftype, v);+ }+ // ES runs a string through `Double.parseDouble` and truncates, so+ // `"1.9"` lands on 1 exactly as the bare number would. Prefer the+ // exact integer parse when it succeeds, so a `long` beyond f64's+ // 2^53 exact range keeps every digit — and range-check THAT parse+ // in `i128`, not the `f64` above: `"9223372036854775808"` rounds to+ // exactly `i64::MAX as f64` and so slips through the float bound,+ // while ES's `Numbers.toLong` throws on it.+ if let Ok(i) = s.trim().parse::<i128>() {+ if i < lo || i > hi {+ return out_of_range(ftype, v);+ }+ return Coercion::Rewrite(number_from_i128(i));+ }+ Coercion::Rewrite(number_from_i128(truncate_into(f, lo, hi).1))+ }+ Value::Bool(_) | Value::Object(_) | Value::Array(_) => not_a_number(ftype),+ Value::Null => Coercion::AsIs,+ }+}++/// Coerce one scalar value for a floating-point field.+fn coerce_floating(ftype: &str, coerce: bool, v: &Value) -> Coercion {+ // ES range-checks a `float` by requiring the value to survive the cast to+ // 32-bit finitely, and a `half_float` by requiring it to survive the round+ // trip through 16-bit precision. 65504 is the largest half-float, but+ // round-to-nearest maps everything below 65520 back onto it — ES accepts+ // 65510 and rejects 65520 — so the boundary is 65520, not 65504. A plain+ // `f as f32` check (what the first cut used for both) accepts up to+ // 3.4e38 — 33 decimal orders of magnitude past where a `half_float`+ // actually overflows.+ const HALF_FLOAT_OVERFLOWS_AT: f64 = 65_520.0;+ let overflows = |f: f64| -> bool {+ match ftype {+ "float" => !(f as f32).is_finite(),+ "half_float" => f.abs() >= HALF_FLOAT_OVERFLOWS_AT,+ // `scaled_float` is deliberately NOT quantised by+ // `scaling_factor` here — see the module header.+ _ => false,+ }+ };+ let check = |f: f64, rewritten: bool| -> Coercion {+ if !f.is_finite() || overflows(f) {+ return not_finite(ftype);+ }+ if rewritten {+ match Number::from_f64(f) {+ Some(n) => Coercion::Rewrite(Value::Number(n)),+ None => not_finite(ftype),+ }+ } else {+ Coercion::AsIs+ }+ };+ match v {+ Value::Number(n) => match n.as_f64() {+ Some(f) => check(f, false),+ None => not_finite(ftype),+ },+ Value::String(s) => {+ if s.trim().is_empty() {+ return Coercion::AsIs;+ }+ if !coerce {+ return Coercion::Reject(format!("Cannot coerce a string to a[n] {}", ftype));+ }+ match s.trim().parse::<f64>() {+ Ok(f) => check(f, true),+ Err(_) => Coercion::Reject(format!("For input string: \"{}\"", s)),+ }+ }+ Value::Bool(_) | Value::Object(_) | Value::Array(_) => not_a_number(ftype),+ Value::Null => Coercion::AsIs,+ }+}++/// Coerce one scalar value for a `boolean` field. ES has no `coerce` knob+/// here: `true`/`false` and their string spellings are the entire domain.+fn coerce_boolean(v: &Value) -> Coercion {+ match v {+ Value::Bool(_) | Value::Null => Coercion::AsIs,+ Value::String(s) if s.is_empty() => Coercion::AsIs,+ Value::String(s) => match s.as_str() {+ "true" => Coercion::Rewrite(Value::Bool(true)),+ "false" => Coercion::Rewrite(Value::Bool(false)),+ other => Coercion::Reject(format!(+ "Failed to parse value [{}] as only [true] or [false] are allowed.",+ other+ )),+ },+ other => Coercion::Reject(format!(+ "Failed to parse value [{}] as only [true] or [false] are allowed.",+ preview_value(other)+ )),+ }+}++/// What ES would index for `v` in a field declared `ftype`.+///+/// `coerce` is the field's effective `"coerce"` mapping parameter (ES default+/// `true`). Types this module does not own return [`Coercion::AsIs`].+pub fn coerce_value(ftype: &str, coerce: bool, v: &Value) -> Coercion {+ match ftype {+ "byte" | "short" | "integer" | "long" | "unsigned_long" => {+ coerce_integral(ftype, coerce, v)+ }+ "half_float" | "float" | "double" | "scaled_float" => coerce_floating(ftype, coerce, v),+ "boolean" => coerce_boolean(v),+ _ => Coercion::AsIs,+ }+}++/// Apply [`coerce_value`] to a whole field value, recursing into arrays.+///+/// Returns `Ok(Some(new))` when at least one element had to be rewritten,+/// `Ok(None)` when the value stands as sent, and `Err((detail, offender))` on+/// the first element ES would refuse.+fn coerce_field(ftype: &str, coerce: bool, v: &Value) -> Result<Option<Value>, (String, Value)> {+ match v {+ Value::Array(arr) => {+ let mut out = Vec::with_capacity(arr.len());+ let mut changed = false;+ for el in arr {+ match coerce_field(ftype, coerce, el)? {+ Some(new) => {+ changed = true;+ out.push(new);+ }+ None => out.push(el.clone()),+ }+ }+ Ok(changed.then_some(Value::Array(out)))+ }+ scalar => match coerce_value(ftype, coerce, scalar) {+ Coercion::AsIs => Ok(None),+ Coercion::Rewrite(new) => Ok(Some(new)),+ Coercion::Reject(detail) => Err((detail, scalar.clone())),+ },+ }+}++/// Walk `doc` against the mapping `properties`, coercing every numeric and+/// boolean field in place.+///+/// Returns `Ok(true)` when the document was modified, `Ok(false)` when it was+/// already ES-shaped, and `Err(BadField)` at the first value ES would refuse —+/// the caller turns that into a 400 `document_parsing_exception`.+///+/// Fields carrying `"ignore_malformed": true` are skipped outright: that+/// mapping asks for the bad value to be dropped into `_ignored`, which the+/// `ignore_malformed` walker has already done by the time this runs.+pub fn coerce_document(+ doc: &mut Map<String, Value>,+ props: &Map<String, Value>,+) -> Result<bool, BadField> {+ coerce_in(doc, props, "")+}++fn coerce_in(+ doc: &mut Map<String, Value>,+ props: &Map<String, Value>,+ prefix: &str,+) -> Result<bool, BadField> {+ let mut changed = false;+ let keys: Vec<String> = doc.keys().cloned().collect();+ for field in keys {+ let Some(spec) = props.get(&field) else {+ continue;+ };+ // `"enabled": false` — ES does not parse, index or validate ANYTHING+ // inside such an object; it is stored in `_source` and otherwise+ // ignored. The object recursion below keys off `properties`, and a+ // disabled object is still allowed to declare typed sub-properties, so+ // without this guard those would be enforced where ES ignores them.+ if spec.get("enabled").and_then(Value::as_bool) == Some(false) {+ continue;+ }+ let full = if prefix.is_empty() {+ field.clone()+ } else {+ format!("{}.{}", prefix, field)+ };+ let ftype = spec.get("type").and_then(Value::as_str).unwrap_or("");++ // Sub-objects: recurse. A mapping with `properties` but no declared+ // `type` is an implicit object in ES, so key off `properties` rather+ // than off the type name.+ if let Some(child_props) = spec.get("properties").and_then(Value::as_object) {+ if ftype.is_empty() || ftype == "object" || ftype == "nested" {+ let child_props = child_props.clone();+ match doc.get_mut(&field) {+ Some(Value::Object(child)) => {+ changed |= coerce_in(child, &child_props, &full)?;+ }+ Some(Value::Array(arr)) => {+ for el in arr.iter_mut() {+ if let Value::Object(child) = el {+ changed |= coerce_in(child, &child_props, &full)?;+ }+ }+ }+ _ => {}+ }+ continue;+ }+ }++ if !is_enforced_type(ftype) {+ continue;+ }+ if spec+ .get("ignore_malformed")+ .and_then(Value::as_bool)+ .unwrap_or(false)+ {+ continue;+ }+ let coerce = spec.get("coerce").and_then(Value::as_bool).unwrap_or(true);++ let Some(value) = doc.get(&field) else {+ continue;+ };+ match coerce_field(ftype, coerce, value) {+ Ok(None) => {}+ Ok(Some(new)) => {+ doc.insert(field.clone(), new);+ changed = true;+ }+ Err((detail, bad)) => {+ return Err(BadField {+ field: full,+ ftype: ftype.to_string(),+ preview: preview_value(&bad),+ detail,+ });+ }+ }+ }+ Ok(changed)+}++/// Does this mapping declare ANY field whose type this module enforces?+///+/// The gate that keeps the `_bulk` turbo path free: an index with no numeric+/// or boolean field never needs its raw doc bytes parsed for coercion, so it+/// skips the whole check. Conservative on false positives (a user field+/// literally named `type` holding the string `"integer"` still returns true+/// and pays for one parse); never falsely negative.+pub fn mapping_has_enforced_types(mapping: &Value) -> bool {+ match mapping {+ Value::Object(m) => {+ if m.get("type")+ .and_then(Value::as_str)+ .map(is_enforced_type)+ .unwrap_or(false)+ {+ return true;+ }+ m.values().any(mapping_has_enforced_types)+ }+ Value::Array(arr) => arr.iter().any(mapping_has_enforced_types),+ _ => false,+ }+}++/// Pull the `properties` map out of a stored index mapping, which is held+/// either bare (`{"properties": …}`) or wrapped (`{"mappings": {"properties":+/// …}}`) depending on which API wrote it.+pub fn mapping_properties(mapping: &Value) -> Option<&Map<String, Value>> {+ mapping+ .get("properties")+ .or_else(|| mapping.get("mappings").and_then(|m| m.get("properties")))+ .and_then(Value::as_object)+}++#[cfg(test)]+mod tests {+ use super::*;+ use serde_json::json;++ fn coerced(ftype: &str, v: Value) -> Coercion {+ coerce_value(ftype, true, &v)+ }++ #[test]+ fn integer_truncates_a_fraction_the_way_es_does() {+ assert_eq!(coerced("integer", json!(1.9)), Coercion::Rewrite(json!(1)));+ assert_eq!(+ coerced("integer", json!(-1.9)),+ Coercion::Rewrite(json!(-1))+ );+ assert_eq!(coerced("long", json!(2.5)), Coercion::Rewrite(json!(2)));+ // Already integral: no `_source` churn.+ assert_eq!(coerced("integer", json!(2.0)), Coercion::AsIs);+ assert_eq!(coerced("integer", json!(7)), Coercion::AsIs);+ }++ #[test]+ fn numeric_strings_are_coerced_and_junk_strings_are_refused() {+ assert_eq!(coerced("integer", json!("5")), Coercion::Rewrite(json!(5)));+ assert_eq!(+ coerced("integer", json!("1.9")),+ Coercion::Rewrite(json!(1))+ );+ assert_eq!(+ coerced("double", json!("2.5")),+ Coercion::Rewrite(json!(2.5))+ );+ assert!(matches!(coerced("long", json!("abc")), Coercion::Reject(_)));+ assert!(matches!(+ coerced("double", json!("abc")),+ Coercion::Reject(_)+ ));+ }++ #[test]+ fn out_of_range_is_refused_on_every_width() {+ assert!(matches!(+ coerced("integer", json!(9999999999i64)),+ Coercion::Reject(_)+ ));+ assert!(matches!(coerced("byte", json!(200)), Coercion::Reject(_)));+ assert!(matches!(+ coerced("short", json!(40000)),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerced("unsigned_long", json!(-1)),+ Coercion::Reject(_)+ ));+ // A u64 beyond i64::MAX must not pass as a `long`; an f64 bound would+ // have rounded i64::MAX up and let this through. The same value must+ // be refused in every SPELLING it can arrive in — bare integer token,+ // string, and float token (which ES's own check admits and its cast+ // then saturates).+ assert!(matches!(+ coerced("long", json!(9223372036854775808u64)),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerced("long", json!("9223372036854775808")),+ Coercion::Reject(_)+ ));+ assert_eq!(+ coerced("long", json!("9223372036854775807")),+ Coercion::Rewrite(json!(9223372036854775807i64))+ );+ assert_eq!(+ coerced("long", json!(9223372036854775807i64)),+ Coercion::AsIs+ );+ assert_eq!(coerced("integer", json!(2147483647)), Coercion::AsIs);+ }++ #[test]+ fn an_unsigned_long_past_i64_max_truncates_without_saturating() {+ // 1e19 is inside `unsigned_long` and outside `i64`. This string form+ // reaches the truncating cast (neither `i64` nor `u64` parses it), and+ // a plain `as i64` there would have clamped it to i64::MAX — a+ // silently different value, exactly the class of bug this closes.+ let got = coerced("unsigned_long", json!("10000000000000000000.5"));+ match got {+ Coercion::Rewrite(Value::Number(n)) => {+ assert_eq!(n.as_u64(), Some(10_000_000_000_000_000_000));+ }+ other => panic!("unexpected: {other:?}"),+ }+ assert!(matches!(+ coerced("unsigned_long", json!(1.0e30f64)),+ Coercion::Reject(_)+ ));+ }++ #[test]+ fn objects_and_arrays_are_not_numbers() {+ assert!(matches!(+ coerced("integer", json!({"bad": "x"})),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerced("integer", json!(true)),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerced("boolean", json!({"a": 1})),+ Coercion::Reject(_)+ ));+ }++ #[test]+ fn coerce_false_refuses_what_coerce_true_would_fix() {+ assert!(matches!(+ coerce_value("integer", false, &json!(1.9)),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerce_value("integer", false, &json!("5")),+ Coercion::Reject(_)+ ));+ // The range check is not a coercion and applies either way.+ assert!(matches!(+ coerce_value("integer", false, &json!(9999999999i64)),+ Coercion::Reject(_)+ ));+ }++ #[test]+ fn boolean_takes_true_false_and_their_spellings_only() {+ assert_eq!(coerced("boolean", json!(true)), Coercion::AsIs);+ assert_eq!(+ coerced("boolean", json!("true")),+ Coercion::Rewrite(json!(true))+ );+ assert_eq!(+ coerced("boolean", json!("false")),+ Coercion::Rewrite(json!(false))+ );+ assert!(matches!(+ coerced("boolean", json!("yes")),+ Coercion::Reject(_)+ ));+ assert!(matches!(coerced("boolean", json!(1)), Coercion::Reject(_)));+ }++ /// The `i128` bound closes the INTEGER token. The float token has a second+ /// hole: ES compares the raw double against a bound Java widened to+ /// `double`, so `Long.MAX_VALUE` becomes 2^63 and `9223372036854775808.0`+ /// passes the check — ES's `(long)` cast then saturates it. Storing the+ /// float verbatim (what a bare `fract() == 0` shortcut does) leaves a+ /// value no `long` can hold sitting in a `long` field.+ #[test]+ fn a_float_token_past_the_width_is_saturated_the_way_es_casts_it() {+ assert_eq!(+ coerced("long", json!(9223372036854775808.0f64)),+ Coercion::Rewrite(json!(9223372036854775807i64))+ );+ assert_eq!(+ coerced("integer", json!(-2147483649.0f64)),+ Coercion::Reject("Value [-2147483649.0] is out of range for a[n] integer".to_string())+ );+ // Inside the width, an integral float is still left alone.+ assert_eq!(+ coerced("long", json!(9223372036854775807.0f64)),+ Coercion::Rewrite(json!(9223372036854775807i64))+ );+ assert_eq!(coerced("long", json!(1234.0)), Coercion::AsIs);+ }++ /// ES round-trips a `half_float` through 16-bit precision and rejects what+ /// comes back infinite. 65504 is the largest half-float and+ /// round-to-nearest maps everything under 65520 onto it, so the boundary+ /// is 65520 — an `f32` finiteness check (which accepts 3.4e38) is ~2^128+ /// too wide.+ #[test]+ fn half_float_is_bounded_at_the_half_precision_overflow() {+ assert_eq!(coerced("half_float", json!(65504.0)), Coercion::AsIs);+ assert_eq!(coerced("half_float", json!(65510.0)), Coercion::AsIs);+ assert!(matches!(+ coerced("half_float", json!(65520.0)),+ Coercion::Reject(_)+ ));+ assert!(matches!(+ coerced("half_float", json!(1.0e38)),+ Coercion::Reject(_)+ ));+ // `float` keeps its own, wider bound.+ assert_eq!(coerced("float", json!(1.0e38)), Coercion::AsIs);+ }++ /// `"enabled": false` tells ES not to parse anything inside the object.+ /// The recursion keys off `properties`, which a disabled object may still+ /// declare, so it needs its own guard.+ #[test]+ fn a_disabled_object_is_not_walked() {+ let props = json!({+ "meta": {+ "enabled": false,+ "properties": {"n": {"type": "integer"}}+ }+ });+ let mut doc = json!({"meta": {"n": "not a number at all"}})+ .as_object()+ .unwrap()+ .clone();+ assert!(!coerce_document(&mut doc, props.as_object().unwrap())+ .expect("a disabled object must not be rejected"));+ assert_eq!(doc["meta"]["n"], json!("not a number at all"));+ }++ #[test]+ fn narrow_floats_refuse_values_that_overflow_them() {+ assert!(matches!(+ coerced("float", json!(1e300)),+ Coercion::Reject(_)+ ));+ assert_eq!(coerced("double", json!(1e300)), Coercion::AsIs);+ assert_eq!(coerced("float", json!(1.5)), Coercion::AsIs);+ }++ #[test]+ fn empty_string_and_null_are_left_alone() {+ assert_eq!(coerced("integer", json!("")), Coercion::AsIs);+ assert_eq!(coerced("integer", Value::Null), Coercion::AsIs);+ assert_eq!(coerced("boolean", json!("")), Coercion::AsIs);+ assert_eq!(coerced("boolean", Value::Null), Coercion::AsIs);+ }++ #[test]+ fn a_document_is_rewritten_in_place_and_nested_fields_are_reached() {+ let props = json!({+ "i": {"type": "integer"},+ "b": {"type": "boolean"},+ "inner": {"properties": {"n": {"type": "long"}}}+ });+ let mut doc = json!({"i": 1.9, "b": "true", "inner": {"n": "42"}})+ .as_object()+ .unwrap()+ .clone();+ let changed = coerce_document(&mut doc, props.as_object().unwrap()).expect("should coerce");+ assert!(changed);+ assert_eq!(doc.get("i"), Some(&json!(1)));+ assert_eq!(doc.get("b"), Some(&json!(true)));+ assert_eq!(doc["inner"]["n"], json!(42));+ }++ #[test]+ fn a_bad_nested_field_reports_its_dotted_path() {+ let props = json!({"inner": {"properties": {"n": {"type": "integer"}}}});+ let mut doc = json!({"inner": {"n": "abc"}}).as_object().unwrap().clone();+ let bad = coerce_document(&mut doc, props.as_object().unwrap()).unwrap_err();+ assert_eq!(bad.field, "inner.n");+ assert_eq!(bad.ftype, "integer");+ assert_eq!(bad.preview, "abc");+ assert!(bad+ .reason("7")+ .contains("failed to parse field [inner.n] of type [integer] in document with id '7'"));+ }++ #[test]+ fn arrays_are_coerced_element_wise() {+ let props = json!({"i": {"type": "integer"}});+ let mut doc = json!({"i": [1.9, "3", 4]}).as_object().unwrap().clone();+ assert!(coerce_document(&mut doc, props.as_object().unwrap()).expect("ok"));+ assert_eq!(doc.get("i"), Some(&json!([1, 3, 4])));++ let mut bad_doc = json!({"i": [1, "abc"]}).as_object().unwrap().clone();+ assert!(coerce_document(&mut bad_doc, props.as_object().unwrap()).is_err());+ }++ #[test]+ fn ignore_malformed_fields_are_left_to_their_own_walker() {+ let props = json!({"i": {"type": "integer", "ignore_malformed": true}});+ let mut doc = json!({"i": "abc"}).as_object().unwrap().clone();+ assert!(!coerce_document(&mut doc, props.as_object().unwrap()).expect("no rejection"));+ assert_eq!(doc.get("i"), Some(&json!("abc")));+ }++ #[test]+ fn unmapped_and_non_numeric_fields_are_untouched() {+ let props = json!({"k": {"type": "keyword"}, "t": {"type": "text"}});+ let mut doc = json!({"k": 5, "t": "hi", "unmapped": {"any": "shape"}})+ .as_object()+ .unwrap()+ .clone();+ assert!(!coerce_document(&mut doc, props.as_object().unwrap()).expect("ok"));+ assert_eq!(doc.get("k"), Some(&json!(5)));+ }++ #[test]+ fn the_turbo_gate_fires_only_on_a_numeric_or_boolean_mapping() {+ assert!(mapping_has_enforced_types(&json!({+ "properties": {"msg": {"type": "text"}, "n": {"type": "long"}}+ })));+ assert!(mapping_has_enforced_types(&json!({+ "properties": {"o": {"properties": {"b": {"type": "boolean"}}}}+ })));+ assert!(!mapping_has_enforced_types(&json!({+ "properties": {"msg": {"type": "text"}, "k": {"type": "keyword"}}+ })));+ assert!(!mapping_has_enforced_types(&json!({})));+ }++ #[test]+ fn mapping_properties_reads_both_stored_shapes() {+ let bare = json!({"properties": {"a": {"type": "integer"}}});+ let wrapped = json!({"mappings": {"properties": {"a": {"type": "integer"}}}});+ assert!(mapping_properties(&bare).is_some());+ assert!(mapping_properties(&wrapped).is_some());+ assert!(mapping_properties(&json!({"settings": {}})).is_none());+ }+}diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex 2ca0eb71a..e60f4d636 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -17,6 +17,8 @@//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`//! - [`error`] — Unified error type ([`XerjError`])//! - [`types`] — Core domain types (documents, fields, IDs)+//! - [`field_coercion`] — ES-faithful ingest-time coercion/enforcement for+//! numeric and boolean fields (the one predicate every write path shares)//! - [`schema`] — Index schema management and mapping evolution//! - [`metrics`] — Prometheus counters, histograms, and gauges//! - [`net`] — Network trust primitives (trusted-proxy CIDR matching)@@ -25,6 +27,7 @@pub mod config;pub mod error;pub mod feedback;+pub mod field_coercion;pub mod fsio;pub mod metrics;pub mod net;diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rsindex 30d836308..55044ba0e 100644--- a/engine/crates/xerj-engine/src/bulk.rs+++ b/engine/crates/xerj-engine/src/bulk.rs@@ -1195,7 +1195,102 @@ pub async fn process_bulk_with_opts(has};- for action in parsed {+ // Same gate for numeric / boolean field types (issue #781). Unlike the+ // three above this does NOT divert to the slow path — that would cost+ // every mapped-numeric index the turbo batch — it coerces the raw doc+ // bytes in place right here and lets the action stay on turbo. The gate+ // exists so an index with no numeric or boolean field never pays for the+ // parse at all.+ //+ // The cache carries the PROPERTIES MAP, not just the boolean gate:+ // `mapping_properties(...).cloned()` deep-copies the whole properties+ // blob, and reading it per action would put an O(mapping) copy on the HTTP+ // worker for every document in the batch — the cost M5.11 took off this+ // path. Behind an `Arc` the per-action price is a refcount bump, and both+ // enforcement sites (partition loop and per-item loop) share the one+ // cache. `None` means "this index needs no check" — an unmapped index, or+ // one whose mapping declares no numeric/boolean field.+ type CoerceProps = Option<std::sync::Arc<serde_json::Map<String, Value>>>;+ let mut index_type_check_props: HashMap<String, CoerceProps> = HashMap::new();+ let typed_numeric_props =+ |target: &str, cache: &mut HashMap<String, CoerceProps>| -> CoerceProps {+ if let Some(p) = cache.get(target) {+ return p.clone();+ }+ let props: CoerceProps = engine.index_mappings.get(target).and_then(|m| {+ if !xerj_common::field_coercion::mapping_has_enforced_types(&m) {+ return None;+ }+ xerj_common::field_coercion::mapping_properties(&m)+ .cloned()+ .map(std::sync::Arc::new)+ });+ cache.insert(target.to_string(), props.clone());+ props+ };++ for mut action in parsed {+ // ── Numeric / boolean type enforcement (issue #781) ──────────────+ //+ // A field mapped `integer` used to keep whatever arrived — `1.9`,+ // `9999999999`, `"abc"`, even an object — so `range {gte: 1.5}`+ // matched a doc that ES stores as `1` and `term {i: 1}` missed it.+ // Values ES coerces are coerced here (`1.9` → `1`, `"5"` → `5`,+ // `"true"` → `true`); values ES refuses become a per-item 400+ // `document_parsing_exception`, like the date-format check below.+ //+ // Runs at partition time so it covers the turbo batch path as well+ // as the per-item loop. Actions carrying a pipeline are skipped: ES+ // runs the pipeline before parsing the document, so those are+ // checked after their pipeline in the per-item loop instead.+ // `update` merges into a doc that was already validated on write.+ if matches!(action.action_type.as_str(), "index" | "create") && action.pipeline.is_none() {+ let props = typed_numeric_props(&action.target_index, &mut index_type_check_props);+ if let Some(props) = props {+ // `index` actions defer the JSON parse to the turbo path, so+ // the body may not be built yet — parse it here, and only+ // pay the re-serialize when a value actually changed.+ let parsed_body = match action.doc_body.take() {+ Some(Value::Object(o)) => Some(o),+ Some(other) => {+ action.doc_body = Some(other);+ None+ }+ None => match serde_json::from_slice::<Value>(&action.doc_bytes) {+ Ok(Value::Object(o)) => Some(o),+ _ => None,+ },+ };+ if let Some(mut body) = parsed_body {+ match xerj_common::field_coercion::coerce_document(&mut body, &props) {+ Ok(changed) => {+ // Serialize from the map itself, not a clone of+ // it — `preserve_order` is on workspace-wide, so+ // the caller's `_source` key order survives.+ if changed {+ if let Ok(bytes) = serde_json::to_vec(&body) {+ action.doc_bytes = std::sync::Arc::from(bytes.as_slice());+ }+ }+ action.doc_body = Some(Value::Object(body));+ }+ Err(bad) => {+ items[action.item_index] = Some(BulkItemResult {+ action: action.action_type.clone(),+ index: action.target_index.clone(),+ id: action.doc_id.clone().unwrap_or_default(),+ status: 400,+ result: None,+ error: Some(bad.reason(&action.doc_id.clone().unwrap_or_default())),+ get_source: None,+ });+ errors = true;+ continue;+ }+ }+ }+ }+ }// `index` with no `if_seq_no` / `routing` goes through the// turbo batch path. Actions that carry CAS or routing metadata// must be executed individually so we can honor them — route@@ -2005,6 +2100,38 @@ pub async fn process_bulk_with_opts(}}}+ // Numeric / boolean type enforcement (issue #781). A field mapped+ // `integer` used to keep whatever arrived — `1.9`, `9999999999`,+ // `"abc"`, even an object — so the same query over the same declared+ // -integer field returned different hits than ES. Values ES coerces+ // are coerced here (`1.9` → `1`, `"5"` → `5`, `"true"` → `true`) and+ // values ES refuses become a per-item 400 `document_parsing_exception`,+ // exactly like the date-format check above. `index` / `create` only:+ // an `update` merges into a doc that was already validated on write.+ //+ // The rules live in `xerj_common::field_coercion`, shared with the+ // single-document `_doc` / `_create` handlers in `xerj-api`, so the+ // two write paths cannot drift apart.+ if matches!(action_type.as_str(), "index" | "create") {+ let mapping_props = typed_numeric_props(&target_index, &mut index_type_check_props);+ if let Some(props) = mapping_props {+ if let Some(body) = doc_body.as_mut().and_then(Value::as_object_mut) {+ if let Err(bad) = xerj_common::field_coercion::coerce_document(body, &props) {+ items[item_idx] = Some(BulkItemResult {+ action: action_type.to_string(),+ index: target_index.clone(),+ id: error_id.clone(),+ status: 400,+ result: None,+ error: Some(bad.reason(&doc_id.clone().unwrap_or_default())),+ get_source: None,+ });+ errors = true;+ continue;+ }+ }+ }+ }// doc_body already lazy-parsed above (for dynamic-templates// validation). Nothing else to do here.let result = execute_bulk_action(diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex ab16f910a..a5b554da8 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -186,6 +186,60 @@ mod collection_publication_fail_closed_tests {));}+ #[test]+ fn boolean_term_and_terms_spellings_normalise_to_the_indexed_value() {+ // #781 follow-up: ingest coercion canonicalises `"true"`/`"false"` to+ // real JSON booleans in `_source`, so a query still spelling them as+ // strings named a value no document held. On a doc-values-only boolean+ // (`390_doc_values_search.yml`) that meant `terms` returned 0 while the+ // equivalent `term` still found the doc. Both arms now run the query+ // value through the SAME `field_coercion` predicate the write path uses.+ let mut schema = Schema::empty();+ schema+ .add_field(FieldConfig::new("flag", FieldType::Boolean))+ .unwrap();+ schema+ .add_field(FieldConfig::new("name", FieldType::Keyword))+ .unwrap();++ let term = |f: &str, v: serde_json::Value| QueryNode::Term {+ field: f.into(),+ value: v,+ boost: None,+ };+ let terms = |f: &str, v: Vec<serde_json::Value>| QueryNode::Terms {+ field: f.into(),+ values: v,+ boost: None,+ };++ match rewrite_query_aliases(&term("flag", json!("false")), &schema) {+ QueryNode::Term { field, value, .. } => {+ assert_eq!(field, "flag");+ assert_eq!(value, json!(false), "string spelling names the boolean");+ }+ other => panic!("boolean term must stay a Term, got {other:?}"),+ }+ match rewrite_query_aliases(&terms("flag", vec![json!("false"), json!(true)]), &schema) {+ QueryNode::Terms { field, values, .. } => {+ assert_eq!(field, "flag");+ assert_eq!(values, vec![json!(false), json!(true)]);+ }+ other => panic!("boolean terms must stay a Terms, got {other:?}"),+ }+ // A value the ingest predicate refuses is left alone — a search must+ // not 400 on it; it simply matches nothing, exactly as before.+ match rewrite_query_aliases(&terms("flag", vec![json!("yes"), json!(1)]), &schema) {+ QueryNode::Terms { values, .. } => assert_eq!(values, vec![json!("yes"), json!(1)]),+ other => panic!("unparseable boolean values must survive, got {other:?}"),+ }+ // A NON-boolean field keeps its literal string — `"true"` is a keyword.+ match rewrite_query_aliases(&term("name", json!("true")), &schema) {+ QueryNode::Term { value, .. } => assert_eq!(value, json!("true")),+ other => panic!("keyword term must be untouched, got {other:?}"),+ }+ }+async fn assert_active_reader_waits_for_commit<F, Fut>(idx: &Arc<Index>,id: &'static str,@@ -479,6 +533,37 @@ mod collection_publication_fail_closed_tests {);}+ /// A field that is TYPE-mixed inside one segment must ship NO doc-values+ /// column. Only one column can carry a name, so the keyword pass would+ /// overwrite the numeric one and file every number/boolean doc as null —+ /// a column consumers treat as exact, which then hides those docs from the+ /// `term`/`terms` prefilter entirely.+ ///+ /// #781 made this reachable in ordinary use: ingest coercion canonicalises+ /// `"true"` to `true`, so an index written across the upgrade holds both+ /// spellings, and a flush that lands them in ONE segment (the 2-core CI+ /// shape) built the lying column — `terms {b:["true"]}` then returned the+ /// pre-upgrade document and silently dropped the coerced one.+ #[test]+ fn build_doc_value_columns_ships_nothing_for_a_type_mixed_field() {+ let legacy = json!({"b": "true", "n": "5", "ok": "kw"});+ let coerced = json!({"b": true, "n": 5, "ok": "kw"});+ let skip = std::collections::HashSet::new();+ let cols =+ super::build_doc_value_columns([Some(&legacy), Some(&coerced)].into_iter(), &skip);+ assert!(+ !cols.contains_key("b") && !cols.contains_key("n"),+ "a type-mixed field must ship no column (got {:?})",+ cols.keys().collect::<Vec<_>>()+ );+ // Not vacuous: a field that stayed one type still gets its column.+ assert!(+ cols.contains_key("ok"),+ "a single-typed field must still be shadowed (got {:?})",+ cols.keys().collect::<Vec<_>>()+ );+ }+async fn assert_flush_publication_blocks_reader_then_finishes(inject_error: bool) {let dir = tempfile::tempdir().unwrap();let mut config = Config::default();@@ -22375,6 +22460,31 @@ fn build_doc_value_columns<'a>(keyword.remove(f);}+ // Same rule for a field that arrived TYPE-mixed inside one segment — some+ // docs numeric/boolean, others a string. Each kind was collected into its+ // own map, and only one column can carry the name: the `keyword` loop below+ // runs second and would overwrite the numeric column, leaving every+ // number/boolean doc filed as NULL in a column consumers trust as exact.+ // `build_term_prefilter_cached` then returns a set that is not a superset+ // and the scan never sees the dropped docs — a confident wrong count, the+ // same failure shape the multi-valued poisoning above exists to prevent.+ //+ // This is exactly what #781's ingest coercion makes reachable: an index+ // written across that upgrade holds `"true"` in one doc and `true` in the+ // next, and a flush that lands both in ONE segment (the 2-core CI shape;+ // a many-core box scatters them and each segment stays homogeneous) built+ // the lying column. Ship no column instead; consumers take the+ // abandon-to-scan path they already take for an absent field.+ let mixed_type: Vec<String> = numeric+ .keys()+ .filter(|f| keyword.contains_key(*f))+ .cloned()+ .collect();+ for f in &mixed_type {+ numeric.remove(f);+ keyword.remove(f);+ }+// Pad every column to total_docs so position == doc_id.for col in numeric.values_mut() {pad_with_none(col, total_docs);@@ -36277,6 +36387,30 @@ fn date_string_has_subms(s: &str) -> bool {>= 4}+/// Does the schema declare `field` a `boolean`?+fn field_is_boolean(schema: &Schema, field: &str) -> bool {+ declared_field(schema, field).is_some_and(|fc| matches!(fc.field_type, FieldType::Boolean))+}++/// Normalise one `term`/`terms` value on a `boolean` field to the value the+/// WRITE path stores for it, using the same predicate the write path uses.+///+/// ES's `BooleanFieldMapper` parses a query value through `Booleans` exactly as+/// it parses an indexed one, so `"true"` and `true` name the same term. XERJ+/// indexes from the stored `_source`, and since #781 the ingest coercion+/// canonicalises `"true"`/`"false"` to real JSON booleans there — so a query+/// still spelling the value as a string names something no document holds.+/// Running the query value through `field_coercion::coerce_value` is what keeps+/// the two sides on one representation; a value the predicate refuses (`"yes"`,+/// `1`) is left untouched and simply matches nothing, because a search must not+/// 400 on a value ingest would have rejected.+fn normalize_boolean_query_value(v: &Value) -> Value {+ match xerj_common::field_coercion::coerce_value("boolean", true, v) {+ xerj_common::field_coercion::Coercion::Rewrite(canonical) => canonical,+ _ => v.clone(),+ }+}+/// Rewrite a query by resolving any field aliases to their canonical field names.////// Traverses the query tree, replacing alias field names with their target paths.@@ -36288,6 +36422,15 @@ fn rewrite_query_aliases(q: &QueryNode, schema: &Schema) -> QueryNode {boost,} => {let resolved = resolve_field_alias(schema, field);+ // A `term` on a `boolean` field names the boolean, however it is+ // spelled — see `normalize_boolean_query_value`.+ if field_is_boolean(schema, &resolved) {+ return QueryNode::Term {+ field: resolved,+ value: normalize_boolean_query_value(value),+ boost: *boost,+ };+ }// A CIDR `term` on an `ip` field means the whole subnet. The// source-scan matcher expands it (`ip_matches_cidr`), but the// segment term-dictionary lookup can't, so it silently returned 0@@ -36380,6 +36523,19 @@ fn rewrite_query_aliases(q: &QueryNode, schema: &Schema) -> QueryNode {// single-term gate), plain (non-CIDR) ips, and any value that does// not parse. Only fires on a declared `date`/`ip` field with >=1// rewritable value; otherwise the plain `terms` is returned as-is.+ //+ // boolean: a `terms` spelling its values `"true"`/`"false"` must+ // name the same term the single `term` arm does, or `terms`+ // misses what `term` finds — the #781 ingest coercion stores a+ // real JSON boolean, so the string spelling matched nothing on+ // a doc-values-only field (`390_doc_values_search.yml`).+ if field_is_boolean(schema, &resolved) {+ return QueryNode::Terms {+ field: resolved,+ values: values.iter().map(normalize_boolean_query_value).collect(),+ boost: *boost,+ };+ }let ft_is_date = declared_field(schema, &resolved).is_some_and(|fc| matches!(fc.field_type, FieldType::Date));let ft_is_ip = declared_field(schema, &resolved)@@ -42971,20 +43127,43 @@ fn json_to_str(v: &Value) -> String {fn json_values_equal(doc_val: &Option<Value>, query_val: &Value) -> bool {match doc_val {None => false,- Some(dv) => {- // Direct equality check.- if dv == query_val {- return true;- }- // Cross-type: number stored as string or vice versa.- match (dv, query_val) {- (Value::String(s), Value::Number(n)) => s.parse::<f64>().ok() == n.as_f64(),- (Value::Number(n), Value::String(s)) => n.as_f64() == s.parse::<f64>().ok(),- // Array field: any element matches.- (Value::Array(arr), _) => arr.iter().any(|elem| elem == query_val),- _ => false,- }- }+ Some(dv) => json_scalar_equal(dv, query_val),+ }+}++/// The cross-type equality of one stored value against one query operand.+///+/// Split out of [`json_values_equal`] so the array arm can recurse through it.+/// An element of a MULTI-valued field has to be compared with the same+/// tolerance a scalar gets, or the same document answers differently depending+/// on whether its field happens to hold one value or several: with a bare+/// `elem == query_val`, `{"b": "true"}` matches `term {b: true}` (via the+/// boolean arm below) while `{"b": ["true"]}` does not.+fn json_scalar_equal(dv: &Value, query_val: &Value) -> bool {+ // Direct equality check.+ if dv == query_val {+ return true;+ }+ // Cross-type: number stored as string or vice versa.+ match (dv, query_val) {+ (Value::String(s), Value::Number(n)) => s.parse::<f64>().ok() == n.as_f64(),+ (Value::Number(n), Value::String(s)) => n.as_f64() == s.parse::<f64>().ok(),+ // Cross-type: a boolean and its string spelling name the same+ // value. ES parses a query value through the same `Booleans`+ // the field mapper uses, so `"true"` and `true` are one term;+ // both directions matter here because since #781 ingest+ // canonicalises a declared `boolean` to a real JSON boolean in+ // `_source` while documents written BEFORE that still hold the+ // string. Without this the two spellings split across the+ // upgrade boundary — the doc-values fast paths already agree+ // (they stringify a boolean needle before the lookup), so the+ // `_source` scan was the one comparator that disagreed.+ (Value::Bool(b), Value::String(s)) | (Value::String(s), Value::Bool(b)) => {+ matches!((s.as_str(), *b), ("true", true) | ("false", false))+ }+ // Array field: any element matches, under the same tolerance.+ (Value::Array(arr), _) => arr.iter().any(|elem| json_scalar_equal(elem, query_val)),+ _ => false,}}