Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(es-compat): numeric and boolean fields enforce their declared type (#781) (#887)

xerj-org41a760a421a0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b6f6feb0d..5e5bd0eda 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,74 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **A field holding both numbers and strings in one segment no longer loses the
+ numeric documents from `term` / `terms`.** The doc-values builder collects a
+ segment's values into a numeric map (numbers and booleans) and a keyword map
+ (strings), then writes both into one column set — so a *type-mixed* field had
+ its numeric column silently overwritten by the keyword one, leaving every
+ number/boolean document filed as null in a column the query prefilter treats
+ as exact. Those documents were then dropped from the result with no error.
+ Such a field now ships **no** doc-values column at all, exactly as a
+ multi-valued field already does, and every consumer falls back to the
+ stored-source scan: correct, at scan cost rather than column speed. Reindex
+ onto a single type to get the column back. Whether it bit you depended on
+ core count — a many-core host scatters a flush across shards and each segment
+ stays single-typed, so this reproduced only where documents of both kinds
+ landed in one segment.
+
+- **A field mapped `integer`, `long`, `float` or `boolean` now enforces that
+ type on write** ([#781](https://github.com/xerj-org/xerj/issues/781)). The
+ declared type was enforced for nothing: `1.9` into an `integer` field was
+ stored as `1.9`, `9999999999` was kept exact, `"abc"` was stored as a
+ string, `"yes"` went into a `boolean`, and — per the issue's follow-up — an
+ entire nested object `{"bad":"x"}` was indexed into an `integer`. Every one
+ of those answered `201 created`.
+
+ The consequence was wrong hits, not just leniency. With `1.9` sitting in an
+ `integer` field, `range {"i":{"gte":1.5}}` **matched** in XERJ and would not
+ in ES (which indexes the truncated `1`), while `term {"i":1}` matched in ES
+ and returned nothing here — the same query, over the same document, under
+ the same mapping, giving different answers.
+
+ Ingest now applies ES 8.x's own rules (`coerce` defaults to `true`): `1.9`
+ → `1`, `"5"` → `5`, `"1.9"` → `1`, `"true"` → `true`; out-of-range for the
+ declared width, an unparseable string, an object, an array element that is
+ none of those, and anything but `true`/`false` in a `boolean` are refused
+ with a 400 `document_parsing_exception` — a per-item 400 under `_bulk`. An
+ explicit `"coerce": false` additionally refuses a decimal part and a string,
+ and `"ignore_malformed": true` still wins: that field's bad values are
+ dropped into `_ignored` rather than failing the document.
+
+ **What this changes about stored data, and what it means on upgrade.** A
+ coerced value is rewritten in `_source`: ES keeps `_source` verbatim and
+ coerces only the indexed value, but XERJ indexes from the stored source, so
+ matching ES's *hits* costs source fidelity here. A document written
+ `{"i": 1.9, "b": "false"}` now reads back `{"i": 1, "b": false}`. An index
+ that spans this upgrade therefore holds **both** spellings of the same
+ logical value — documents written before hold `"false"` / `"5"`, documents
+ written after hold `false` / `5` — and no reindex is performed for you.
+
+ That mixed index still answers correctly, and closing that gap is part of
+ this change. Because the canonicalisation moves a declared `boolean` from
+ the string spelling to a real JSON boolean in `_source`, the **query** side
+ now runs `term` / `terms` values on a `boolean` field through the same
+ predicate — ES parses a query value with the same `Booleans` its field
+ mapper uses, so `"true"` and `true` name one term. Without it,
+ `terms {"b":["true"]}` found nothing on a doc-values-only boolean field
+ while the equivalent `term` still matched. The `_source` scan comparator
+ relates the two spellings in both directions (as it already did for a number
+ and its string spelling), for single- and multi-valued fields alike, and a
+ `terms` aggregation buckets them together — so one query sees the whole
+ index rather than half of it. **Reindex** if you want a uniform `_source`;
+ nothing breaks if you do not.
+
+ Also narrowed deliberately, all documented in `xerj_common::field_coercion`:
+ the index-level `index.mapping.coerce` setting is not read (only the
+ field-level parameter), `scaled_float` is range-checked but not quantised by
+ its `scaling_factor`, an empty string is left alone rather than dropped, and
+ `_update` is not re-validated — it merges into a document already checked on
+ write.
+
- **A top-level `knn` beside a `query` now scores as a sum over the union of
both halves, aggregations included**
([#825](https://github.com/xerj-org/xerj/issues/825)). ES 8.x defines this
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 396eab9c5..e9a7288f6 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -2831,6 +2831,13 @@ pub async fn index_doc_auto(
doc
};
+ // Declared numeric/boolean types are enforced after the pipeline has run,
+ // so a pipeline-produced value is held to the same rules as a caller's.
+ let doc = match enforce_field_types(&state, &index, "", doc) {
+ Ok(d) => d,
+ Err(resp) => return resp,
+ };
+
let idx = match state.engine.get_or_create_index(&index) {
Ok(i) => i,
Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),
@@ -2959,6 +2966,13 @@ pub async fn index_doc(
};
// ?refresh=true|wait_for — accepted silently; memtable is immediately visible.
+ // Declared numeric/boolean types are enforced after the pipeline has run,
+ // so a pipeline-produced value is held to the same rules as a caller's.
+ let doc = match enforce_field_types(&state, &index, &id, doc) {
+ Ok(d) => d,
+ Err(resp) => return resp,
+ };
+
let idx = match state.engine.get_or_create_index(&index) {
Ok(i) => i,
Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),
@@ -3085,6 +3099,13 @@ pub async fn create_doc(
doc
};
+ // `_create` skipped every mapping-aware validation stage; the declared
+ // numeric/boolean types are now enforced here too (issue #781).
+ let doc = match enforce_field_types(&state, &index, &id, doc) {
+ Ok(d) => d,
+ Err(resp) => return resp,
+ };
+
let idx = match state.engine.get_or_create_index(&index) {
Ok(i) => i,
Err(e) => return ApiError::new(xerj_common::XerjError::from(e)).into_response(),
@@ -35519,6 +35540,62 @@ pub(crate) fn rewrite_bulk_ignore_malformed(
out
}
+/// The 400 envelope ES sends when a document cannot be parsed against the
+/// mapping — the same shape the range/`copy_to` rejection above emits, with
+/// the specific `caused_by` sentence ES attaches underneath.
+fn document_parsing_error(reason: &str, caused_by: &str) -> Response {
+ (
+ StatusCode::BAD_REQUEST,
+ Json(json!({
+ "error": {
+ "root_cause": [{ "type": "document_parsing_exception", "reason": reason }],
+ "type": "document_parsing_exception",
+ "reason": reason,
+ "caused_by": { "type": "illegal_argument_exception", "reason": caused_by }
+ },
+ "status": 400
+ })),
+ )
+ .into_response()
+}
+
+/// Enforce the declared type of every numeric/boolean field on a single-doc
+/// write (issue #781).
+///
+/// Returns the document with ES's coercions applied (`1.9` → `1` in an
+/// `integer` field, `"5"` → `5`, `"true"` → `true`), or the 400
+/// `document_parsing_exception` ES answers for a value no coercion can
+/// rescue — an unparseable string, an out-of-range integer, an object where a
+/// number belongs, `"yes"` in a boolean.
+///
+/// The rules live in [`xerj_common::field_coercion`], shared with the
+/// per-item `_bulk` loop in `xerj_engine::bulk`, so the two write paths cannot
+/// drift apart the way the date validators once did.
+pub(crate) fn enforce_field_types(
+ state: &AppState,
+ index: &str,
+ doc_id: &str,
+ doc: Value,
+) -> Result<Value, Response> {
+ let Some(mapping) = state.engine.index_mappings.get(index).map(|v| v.clone()) else {
+ return Ok(doc);
+ };
+ // Borrow the properties out of the cloned blob rather than cloning them
+ // again — one clone per write is what `apply_ignore_malformed` already
+ // pays to keep the DashMap shard lock off the coercion walk.
+ let Some(props) = xerj_common::field_coercion::mapping_properties(&mapping) else {
+ return Ok(doc);
+ };
+ let mut obj = match doc {
+ Value::Object(o) => o,
+ other => return Ok(other),
+ };
+ match xerj_common::field_coercion::coerce_document(&mut obj, props) {
+ Ok(_) => Ok(Value::Object(obj)),
+ Err(bad) => Err(document_parsing_error(&bad.reason(doc_id), &bad.detail)),
+ }
+}
+
pub(crate) fn apply_ignore_malformed(state: &AppState, index: &str, doc: Value) -> Value {
let Some(mapping) = state.engine.index_mappings.get(index).map(|v| v.clone()) else {
return doc;
diff --git a/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rs b/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rs
new file mode 100644
index 000000000..9ebc3a7a7
--- /dev/null
+++ b/engine/crates/xerj-api/tests/numeric_fields_enforce_their_type.rs
@@ -0,0 +1,614 @@
+//! Numeric and boolean fields enforce their declared type, from the outside —
+//! issue #781.
+//!
+//! A field mapped `integer` used to accept and keep literally anything: `1.9`
+//! stayed `1.9`, `9999999999` stayed exact, `"abc"` was stored as a string,
+//! and even `{"bad": "x"}` was indexed. Every one of those was a `201`.
+//!
+//! The worst of it is not leniency, it is wrong hits. With `1.9` sitting in an
+//! `integer` field, `range {"i": {"gte": 1.5}}` MATCHED in xerj and would not
+//! in ES (which indexes the truncated `1`), while `term {"i": 1}` matched in
+//! ES and returned nothing here. Same mapping, same document, same query,
+//! different answers — `wrong_results_the_declared_type_used_to_produce`
+//! below is that case, pinned.
+//!
+//! ES 8.x semantics, with `coerce` defaulting to `true`:
+//!
+//! * `1.9` → `1` (truncated toward zero), `"5"` → `5`, `"1.9"` → `1`
+//! * `"true"`/`"false"` → the booleans
+//! * out of range for the declared width → **400**
+//! * unparseable string, object, array, `"yes"` in a boolean → **400**
+//! * `"coerce": false` additionally refuses the decimal part and the string
+//!
+//! Every write path that knows the typed schema is covered here: `PUT
+//! /{index}/_doc/{id}`, `POST /{index}/_doc` (auto-id), `PUT
+//! /{index}/_create/{id}`, and `_bulk` — including the turbo batch path
+//! auto-id bulk takes, which never parses the document body and so had to be
+//! reached by a partition-time check rather than the per-item loop.
+//!
+//! Elasticsearch is referenced for semantics only. It is AGPL-3.0/SSPL-1.0/
+//! Elastic-2.0 licensed and no code from it is reproduced here.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+async fn app() -> (axum::Router, tempfile::TempDir) {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ (xerj_api::router::build_es_compat_router(state), dir)
+}
+
+async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {
+ let response = app.clone().oneshot(req).await.expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, value)
+}
+
+async fn json_req(
+ app: &axum::Router,
+ method: &str,
+ path: &str,
+ body: Value,
+) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+}
+
+async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::get(path).body(Body::empty()).expect("request"),
+ )
+ .await
+}
+
+async fn bulk(app: &axum::Router, path: &str, ndjson: &str) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::post(path)
+ .header("content-type", "application/x-ndjson")
+ .body(Body::from(ndjson.to_owned()))
+ .expect("request"),
+ )
+ .await
+}
+
+/// `i` integer, `l` long, `f` float, `b` boolean, `strict` integer with
+/// `coerce: false`, `lenient` integer with `ignore_malformed: true`, plus a
+/// nested object and a `keyword` this check must not touch.
+async fn typed_index(name: &str) -> (axum::Router, tempfile::TempDir) {
+ let (app, dir) = app().await;
+ let (status, _) = json_req(
+ &app,
+ "PUT",
+ &format!("/{name}"),
+ json!({
+ "mappings": {
+ "properties": {
+ "i": { "type": "integer" },
+ "l": { "type": "long" },
+ "f": { "type": "float" },
+ "b": { "type": "boolean" },
+ "strict": { "type": "integer", "coerce": false },
+ "lenient": { "type": "integer", "ignore_malformed": true },
+ "k": { "type": "keyword" },
+ "inner": { "properties": { "n": { "type": "integer" } } }
+ }
+ }
+ }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "mapping should be accepted");
+ (app, dir)
+}
+
+async fn put_doc(app: &axum::Router, index: &str, id: &str, doc: Value) -> (StatusCode, Value) {
+ json_req(app, "PUT", &format!("/{index}/_doc/{id}"), doc).await
+}
+
+/// Flush the memtable into a segment.
+///
+/// This matters for the query-side tests at the bottom of this file: while a
+/// document is memtable-resident, `term`/`terms` are answered by the doc-values
+/// fast path, which compares STRINGIFIED values and so is blind to the
+/// boolean-versus-`"boolean"` spelling. It is the flushed segment path — a
+/// doc-values prefilter refined by an exact re-test of `_source` — that
+/// distinguishes them, which is why the ES-YAML case that caught this
+/// (`search/390_doc_values_search.yml`) does `indices.refresh` before it
+/// queries. A test that skips the refresh passes on the broken code.
+async fn refresh(app: &axum::Router, index: &str) {
+ let (status, body) = json_req(app, "POST", &format!("/{index}/_refresh"), json!({})).await;
+ assert_eq!(status, StatusCode::OK, "refresh should succeed: {body}");
+}
+
+async fn search(app: &axum::Router, index: &str, query: Value) -> Value {
+ let (status, body) = json_req(
+ app,
+ "POST",
+ &format!("/{index}/_search"),
+ json!({ "query": query }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "search should succeed: {body}");
+ body
+}
+
+fn hit_count(body: &Value) -> u64 {
+ body["hits"]["total"]["value"].as_u64().unwrap_or(0)
+}
+
+fn is_parsing_rejection(status: StatusCode, body: &Value, field: &str) -> bool {
+ status == StatusCode::BAD_REQUEST
+ && body["error"]["type"] == "document_parsing_exception"
+ && body["error"]["reason"]
+ .as_str()
+ .map(|r| r.contains(&format!("failed to parse field [{field}]")))
+ .unwrap_or(false)
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The wrong-results case that makes this a bug and not a style preference.
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `1.9` into an `integer` field. ES truncates it to `1` at ingest, so
+/// `range {gte: 1.5}` misses and `term {i: 1}` hits. Before the fix xerj
+/// stored `1.9` verbatim and answered both queries the other way round.
+#[tokio::test]
+async fn wrong_results_the_declared_type_used_to_produce() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, _) = put_doc(&app, "t", "1", json!({ "i": 1.9 })).await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ // The stored value is the integer ES would have indexed.
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(doc["_source"]["i"], json!(1), "1.9 should truncate to 1");
+
+ let gte = search(&app, "t", json!({ "range": { "i": { "gte": 1.5 } } })).await;
+ assert_eq!(
+ hit_count(&gte),
+ 0,
+ "an integer field holding ES's 1 must not match gte 1.5: {gte}"
+ );
+
+ let term = search(&app, "t", json!({ "term": { "i": 1 } })).await;
+ assert_eq!(
+ hit_count(&term),
+ 1,
+ "term 1 must find the truncated value: {term}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Coercion: the values ES accepts and rewrites.
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn numeric_strings_and_boolean_spellings_are_coerced() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, _) = put_doc(
+ &app,
+ "t",
+ "1",
+ json!({ "i": "5", "l": "-7", "f": "2.5", "b": "true", "inner": { "n": "3" } }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(doc["_source"]["i"], json!(5));
+ assert_eq!(doc["_source"]["l"], json!(-7));
+ assert_eq!(doc["_source"]["f"], json!(2.5));
+ assert_eq!(doc["_source"]["b"], json!(true));
+ assert_eq!(doc["_source"]["inner"]["n"], json!(3), "nested too");
+
+ // Coerced, therefore queryable as the number it now is.
+ let term = search(&app, "t", json!({ "term": { "i": 5 } })).await;
+ assert_eq!(hit_count(&term), 1, "coerced string is numerically indexed");
+}
+
+#[tokio::test]
+async fn arrays_are_coerced_element_wise() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, _) = put_doc(&app, "t", "1", json!({ "i": [1.9, "3", 4] })).await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(doc["_source"]["i"], json!([1, 3, 4]));
+}
+
+#[tokio::test]
+async fn values_that_already_match_the_type_are_left_alone() {
+ let (app, _dir) = typed_index("t").await;
+
+ // `2.0` is already the integer ES indexes — no `_source` churn — and a
+ // `keyword` field is not this check's business at all.
+ let (status, _) = put_doc(
+ &app,
+ "t",
+ "1",
+ json!({ "i": 2.0, "l": 42, "b": false, "k": 7, "unmapped": { "any": "shape" } }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(doc["_source"]["l"], json!(42));
+ assert_eq!(doc["_source"]["b"], json!(false));
+ assert_eq!(doc["_source"]["k"], json!(7), "keyword is not coerced");
+ assert_eq!(doc["_source"]["unmapped"], json!({ "any": "shape" }));
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Rejection: the values ES refuses outright.
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn a_non_numeric_string_is_refused() {
+ let (app, _dir) = typed_index("t").await;
+ let (status, body) = put_doc(&app, "t", "1", json!({ "i": "abc" })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "i"),
+ "expected 400 document_parsing_exception, got {status}: {body}"
+ );
+ assert!(
+ body["error"]["caused_by"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("abc"),
+ "the caused_by should name the offending input: {body}"
+ );
+ // Nothing was written.
+ assert_eq!(get(&app, "/t/_doc/1").await.0, StatusCode::NOT_FOUND);
+}
+
+#[tokio::test]
+async fn an_out_of_range_value_is_refused_even_though_it_is_an_integer() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, body) = put_doc(&app, "t", "1", json!({ "i": 9999999999i64 })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "i"),
+ "9999999999 does not fit an integer: {status} {body}"
+ );
+
+ // The same number in a `long` is perfectly fine.
+ let (status, _) = put_doc(&app, "t", "2", json!({ "l": 9999999999i64 })).await;
+ assert_eq!(status, StatusCode::CREATED);
+}
+
+#[tokio::test]
+async fn a_boolean_takes_only_true_and_false() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, body) = put_doc(&app, "t", "1", json!({ "b": "yes" })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "b"),
+ "\"yes\" is not a boolean: {status} {body}"
+ );
+
+ let (status, body) = put_doc(&app, "t", "2", json!({ "b": 1 })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "b"),
+ "1 is not a boolean in ES 8.x: {status} {body}"
+ );
+}
+
+/// The comment on the issue: an OBJECT went into an integer field and the
+/// document was indexed, so the declared type enforced nothing at all.
+#[tokio::test]
+async fn an_object_is_not_a_number() {
+ let (app, _dir) = typed_index("t").await;
+ let (status, body) = put_doc(&app, "t", "1", json!({ "i": { "bad": "x" } })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "i"),
+ "an object in an integer field must be refused: {status} {body}"
+ );
+ assert!(
+ body["error"]["reason"]
+ .as_str()
+ .unwrap_or_default()
+ .contains("bad=x"),
+ "the preview should render the object ES-style: {body}"
+ );
+}
+
+#[tokio::test]
+async fn a_bad_nested_field_is_named_by_its_dotted_path() {
+ let (app, _dir) = typed_index("t").await;
+ let (status, body) = put_doc(&app, "t", "1", json!({ "inner": { "n": "abc" } })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "inner.n"),
+ "expected the dotted path in the reason: {status} {body}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The two mapping parameters that change the answer.
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn coerce_false_refuses_what_the_default_would_fix() {
+ let (app, _dir) = typed_index("t").await;
+
+ for bad in [json!(1.9), json!("5")] {
+ let (status, body) = put_doc(&app, "t", "1", json!({ "strict": bad })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "strict"),
+ "coerce:false should refuse {bad}: {status} {body}"
+ );
+ }
+
+ // An in-range integer still goes in.
+ let (status, _) = put_doc(&app, "t", "1", json!({ "strict": 3 })).await;
+ assert_eq!(status, StatusCode::CREATED);
+}
+
+#[tokio::test]
+async fn ignore_malformed_still_wins_over_rejection() {
+ let (app, _dir) = typed_index("t").await;
+
+ // The field asked for the bad value to be dropped into `_ignored`, not for
+ // the document to be refused — that contract must survive this change.
+ let (status, _) = put_doc(&app, "t", "1", json!({ "lenient": "abc", "i": 4 })).await;
+ assert_eq!(status, StatusCode::CREATED, "ignore_malformed must not 400");
+
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(doc["_source"]["i"], json!(4));
+ assert!(
+ doc["_source"].get("lenient").is_none(),
+ "the malformed value should have been dropped: {doc}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Every other write path agrees with `PUT /_doc/{id}`.
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn the_auto_id_and_create_endpoints_enforce_the_same_rules() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, body) = json_req(&app, "POST", "/t/_doc", json!({ "i": "abc" })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "i"),
+ "POST /_doc must refuse it too: {status} {body}"
+ );
+
+ let (status, body) = json_req(&app, "PUT", "/t/_create/9", json!({ "b": "yes" })).await;
+ assert!(
+ is_parsing_rejection(status, &body, "b"),
+ "PUT /_create/{{id}} must refuse it too: {status} {body}"
+ );
+
+ // …and coerce on the happy path.
+ let (status, _) = json_req(&app, "PUT", "/t/_create/10", json!({ "i": 1.9 })).await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (_, doc) = get(&app, "/t/_doc/10").await;
+ assert_eq!(doc["_source"]["i"], json!(1));
+}
+
+/// `_bulk` reports per item and isolates failures — including on the auto-id
+/// turbo batch path, which never parses the document body.
+#[tokio::test]
+async fn bulk_rejects_per_item_and_keeps_the_good_ones() {
+ let (app, _dir) = typed_index("t").await;
+
+ let ndjson = concat!(
+ "{\"index\":{\"_id\":\"ok\"}}\n{\"i\":1.9}\n",
+ "{\"index\":{\"_id\":\"bad-str\"}}\n{\"i\":\"abc\"}\n",
+ "{\"index\":{\"_id\":\"bad-range\"}}\n{\"i\":9999999999}\n",
+ "{\"index\":{\"_id\":\"bad-obj\"}}\n{\"i\":{\"bad\":\"x\"}}\n",
+ "{\"index\":{\"_id\":\"bad-bool\"}}\n{\"b\":\"yes\"}\n",
+ "{\"create\":{\"_id\":\"ok2\"}}\n{\"i\":\"7\"}\n",
+ );
+ let (status, body) = bulk(&app, "/t/_bulk", ndjson).await;
+ assert_eq!(status, StatusCode::OK);
+ assert_eq!(body["errors"], json!(true), "{body}");
+
+ let items = body["items"].as_array().expect("items");
+ assert_eq!(items.len(), 6);
+ let code = |n: usize| -> u64 {
+ items[n]
+ .as_object()
+ .and_then(|o| o.values().next())
+ .and_then(|v| v["status"].as_u64())
+ .unwrap_or(0)
+ };
+ assert_eq!(code(0), 201, "the coercible one lands: {body}");
+ for (n, what) in [(1, "string"), (2, "range"), (3, "object"), (4, "boolean")] {
+ assert_eq!(code(n), 400, "bad {what} should be a per-item 400: {body}");
+ }
+ assert_eq!(code(5), 201, "create coerces too: {body}");
+
+ // Only the two good documents exist, with ES's values.
+ let (_, ok) = get(&app, "/t/_doc/ok").await;
+ assert_eq!(ok["_source"]["i"], json!(1));
+ let (_, ok2) = get(&app, "/t/_doc/ok2").await;
+ assert_eq!(ok2["_source"]["i"], json!(7));
+ assert_eq!(get(&app, "/t/_doc/bad-str").await.0, StatusCode::NOT_FOUND);
+
+ let (_, count) = get(&app, "/t/_count").await;
+ assert_eq!(count["count"], json!(2), "{count}");
+}
+
+/// Auto-id bulk is the turbo batch path: the doc body is never parsed and the
+/// whole group is appended in one shot, so it needed its own check. Without
+/// it, every bad value in this batch was a silent 201.
+#[tokio::test]
+async fn the_bulk_turbo_path_is_not_a_way_around_the_check() {
+ let (app, _dir) = typed_index("t").await;
+
+ let ndjson = concat!(
+ "{\"index\":{}}\n{\"i\":1.9}\n",
+ "{\"index\":{}}\n{\"i\":\"abc\"}\n",
+ "{\"index\":{}}\n{\"i\":\"6\"}\n",
+ );
+ let (status, body) = bulk(&app, "/t/_bulk", ndjson).await;
+ assert_eq!(status, StatusCode::OK);
+ assert_eq!(body["errors"], json!(true), "{body}");
+
+ let (_, count) = get(&app, "/t/_count").await;
+ assert_eq!(count["count"], json!(2), "only the two valid docs: {count}");
+
+ // Both survivors were coerced, so both are numerically queryable.
+ let one = search(&app, "t", json!({ "term": { "i": 1 } })).await;
+ assert_eq!(hit_count(&one), 1, "{one}");
+ let six = search(&app, "t", json!({ "term": { "i": 6 } })).await;
+ assert_eq!(hit_count(&six), 1, "{six}");
+}
+
+/// An index with no numeric or boolean field skips the check entirely — the
+/// gate that keeps the turbo path free must not change behaviour for it.
+#[tokio::test]
+async fn an_index_with_no_numeric_mapping_is_untouched() {
+ let (app, _dir) = app().await;
+ let (status, _) = json_req(
+ &app,
+ "PUT",
+ "/plain",
+ json!({ "mappings": { "properties": { "msg": { "type": "text" } } } }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK);
+
+ let (status, _) = put_doc(&app, "plain", "1", json!({ "msg": "hi", "n": "abc" })).await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (_, doc) = get(&app, "/plain/_doc/1").await;
+ assert_eq!(doc["_source"]["n"], json!("abc"), "unmapped stays verbatim");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The query side of the same coercion. Rewriting `_source` moves the STORED
+// spelling, so a query operand written the pre-coercion way has to keep
+// matching — this is the case CI caught in
+// `tests/es-compat-yaml/yaml/search/390_doc_values_search.yml`, and the reason
+// the first cut of this change regressed conformance from 0 failed to 1.
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// A document written `{"b": "true"}` is now STORED as `{"b": true}`. ES accepts
+/// either spelling in a `term`/`terms` operand against a `boolean` field
+/// (`terms {b: ["true"]}` is literally what the ES YAML suite asserts), so both
+/// must still find it. Pre-fix, `terms` with the string operand found nothing:
+/// `terms` verifies each admitted doc with exact JSON equality, and `true` is
+/// not `"true"`.
+#[tokio::test]
+async fn a_query_in_the_pre_coercion_spelling_still_matches_the_coerced_document() {
+ let (app, _dir) = typed_index("t").await;
+
+ let (status, _) = put_doc(&app, "t", "1", json!({ "b": "true", "i": "5" })).await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (_, doc) = get(&app, "/t/_doc/1").await;
+ assert_eq!(
+ doc["_source"]["b"],
+ json!(true),
+ "ingest coerced the spelling"
+ );
+ assert_eq!(doc["_source"]["i"], json!(5));
+ refresh(&app, "t").await;
+
+ for query in [
+ json!({ "terms": { "b": ["true"] } }),
+ json!({ "terms": { "b": [true] } }),
+ json!({ "term": { "b": "true" } }),
+ json!({ "term": { "b": true } }),
+ json!({ "terms": { "b": ["false", "true"] } }),
+ json!({ "terms": { "i": ["5"] } }),
+ json!({ "terms": { "i": [5] } }),
+ ] {
+ let body = search(&app, "t", query.clone()).await;
+ assert_eq!(
+ hit_count(&body),
+ 1,
+ "both spellings of {query} must find the coerced document: {body}"
+ );
+ }
+
+ // Tolerance, not blindness: the other boolean still must not match.
+ for query in [
+ json!({ "terms": { "b": ["false"] } }),
+ json!({ "term": { "b": false } }),
+ json!({ "terms": { "b": ["yes"] } }),
+ json!({ "terms": { "i": ["6"] } }),
+ ] {
+ let body = search(&app, "t", query.clone()).await;
+ assert_eq!(hit_count(&body), 0, "{query} must not match: {body}");
+ }
+}
+
+/// An index that spans the upgrade holds BOTH representations of the same
+/// logical value: documents written before this change keep `"true"` / `"5"`,
+/// documents written after hold `true` / `5`. One query must not silently see
+/// half the index. `_update` is the shortest way to plant the old spelling —
+/// it merges into a document that was already validated on write and is
+/// deliberately not re-validated, which is exactly the shape a pre-upgrade
+/// document has.
+#[tokio::test]
+async fn an_index_holding_both_spellings_answers_every_query_with_both_docs() {
+ let (app, _dir) = typed_index("t").await;
+
+ // Post-upgrade document: coerced on the way in.
+ let (status, _) = put_doc(&app, "t", "new", json!({ "b": "true", "i": "5" })).await;
+ assert_eq!(status, StatusCode::CREATED);
+
+ // Pre-upgrade document: the raw spelling, planted past the check.
+ let (status, _) = put_doc(&app, "t", "legacy", json!({ "k": "a" })).await;
+ assert_eq!(status, StatusCode::CREATED);
+ let (status, body) = json_req(
+ &app,
+ "POST",
+ "/t/_update/legacy",
+ json!({ "doc": { "b": "true", "i": "5" } }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "update should apply: {body}");
+
+ refresh(&app, "t").await;
+
+ let (_, legacy) = get(&app, "/t/_doc/legacy").await;
+ let (_, fresh) = get(&app, "/t/_doc/new").await;
+ assert_eq!(
+ legacy["_source"]["b"],
+ json!("true"),
+ "the legacy spelling is what makes this test meaningful"
+ );
+ assert_eq!(fresh["_source"]["b"], json!(true));
+
+ for query in [
+ json!({ "terms": { "b": ["true"] } }),
+ json!({ "terms": { "b": [true] } }),
+ json!({ "term": { "b": "true" } }),
+ json!({ "term": { "b": true } }),
+ json!({ "terms": { "i": ["5"] } }),
+ json!({ "terms": { "i": [5] } }),
+ json!({ "term": { "i": 5 } }),
+ ] {
+ let body = search(&app, "t", query.clone()).await;
+ assert_eq!(
+ hit_count(&body),
+ 2,
+ "{query} must see both spellings in one index: {body}"
+ );
+ }
+}
diff --git a/engine/crates/xerj-common/src/field_coercion.rs b/engine/crates/xerj-common/src/field_coercion.rs
new file mode 100644
index 000000000..4094b72b3
--- /dev/null
+++ b/engine/crates/xerj-common/src/field_coercion.rs
@@ -0,0 +1,891 @@
+//! Ingest-time coercion and type enforcement for numeric and boolean fields.
+//!
+//! THE single predicate for "what would Elasticsearch index for this value in
+//! a field of this declared type?". Every write path that knows the typed
+//! schema must call it and nothing else — the single-document `_doc`/`_create`
+//! handlers in `xerj-api`, the per-item `_bulk` loop in `xerj-engine`, and the
+//! `ignore_malformed` walker. When two ingest paths grew independent copies of
+//! a validation rule before (the date predicate, `xerj_query::dates`), the
+//! *strict* one ended up the laxer of the pair; sharing one function is what
+//! stops that from happening again.
+//!
+//! ## What ES actually does (8.x, `coerce` defaults to `true`)
+//!
+//! For a numeric field, `NumberFieldMapper` parses the JSON token through
+//! `XContentParser::intValue(coerce)` / `doubleValue(coerce)` and then range-
+//! checks against the target width:
+//!
+//! | input into `integer` | ES 8.x |
+//! |----------------------|-------------------------------------------|
+//! | `1` | `1` |
+//! | `1.9` | `1` — truncated toward zero |
+//! | `"5"` / `"1.9"` | `5` / `1` — string coerced then truncated |
+//! | `9999999999` | **400**, out of range for an integer |
+//! | `"abc"` | **400**, cannot parse |
+//! | `{"bad":"x"}`, `[…]` | **400**, object/array is not a number |
+//!
+//! With `"coerce": false` on the field, a decimal part and a string are both
+//! refused; the range check applies either way. `boolean` has no `coerce`
+//! knob at all: `true`/`false` and the strings `"true"`/`"false"` are the
+//! whole accepted set, and anything else — including `1`, `0` and `"yes"` —
+//! is a 400.
+//!
+//! ## The XERJ divergence this closes (issue #781)
+//!
+//! XERJ stored whatever arrived, so an `integer` field holding `1.9` matched
+//! `range {gte: 1.5}` and missed `term {i: 1}` — the same query over the same
+//! declared-integer field returned different hits than ES. Coercing at ingest
+//! is what makes the indexed value and the declared type agree.
+//!
+//! ## Rewriting `_source` moves the stored spelling — the query side pairs
+//!
+//! ES keeps `_source` byte-verbatim and coerces only the *indexed* value.
+//! XERJ indexes from the stored source, so agreeing with ES on **hits** costs
+//! source fidelity: ES returns `1.9` from `_source` where XERJ now returns
+//! `1`, and a document written `{"b": "false"}` is stored `{"b": false}`.
+//!
+//! Two consequences, both handled — do not undo either half without the
+//! other:
+//!
+//! 1. **A query operand written in the pre-coercion spelling must still
+//! match.** ES accepts `terms {b: ["false"]}` against a `boolean` field
+//! (`search/390_doc_values_search.yml` asserts exactly that), and once the
+//! stored value is a real `false` an exact-JSON comparison stops finding
+//! it. So `xerj_engine::index::rewrite_query_aliases` runs a `term`/`terms`
+//! value on a declared `boolean` through [`coerce_value`] — THIS predicate,
+//! on the read side — before any path sees it. The first cut of this module
+//! shipped without that and regressed the conformance suite from 0 failed
+//! to 1.
+//! 2. **An index can hold BOTH representations.** Documents written before
+//! this change keep `"true"` / `"5"`; documents written after hold `true` /
+//! `5`; `_update` (deliberately not re-validated) can still merge the old
+//! spelling into a new document. Canonicalising the operand alone would
+//! then miss the older half, so `xerj_engine::index::json_scalar_equal`
+//! also relates a boolean to its string spelling in BOTH directions —
+//! for a scalar and for an element of a multi-valued field — alongside the
+//! number/string pair it already had. Terms aggregations already bucket the
+//! two spellings together. A reindex is still the way to make an index
+//! uniform, but nothing breaks without one.
+//!
+//! ## Known, deliberate narrowing
+//!
+//! * An empty string is left alone rather than dropped. ES treats `""` in a
+//! numeric field as "no value"; dropping it would rewrite `_source` for a
+//! case that is not a wrong-results bug.
+//! * Only the field-level `"coerce": false` is honoured, not the index-level
+//! `index.mapping.coerce` setting.
+//! * `scaled_float` is range-checked as a `double` and **not** quantised by
+//! its `scaling_factor`. ES indexes `Math.round(value * scaling_factor)`;
+//! reproducing that here would rewrite `_source` into the quantised value,
+//! which loses information the caller sent for no wrong-results gain.
+//! * `_update` is not re-validated: it merges into a document that was already
+//! checked on write, matching how the existing date check scopes itself to
+//! `index` / `create`.
+//! * Enforcement applies only to **declared** mappings. `index_mappings` is
+//! written by index-create-with-mappings and `PUT /_mapping` only, never by
+//! dynamic inference, so a schemaless index is untouched.
+//!
+//! Elasticsearch is referenced for semantics only. It is AGPL-3.0/SSPL-1.0/
+//! Elastic-2.0 licensed and no code from it is reproduced here.
+
+use serde_json::{Map, Number, Value};
+
+/// What Elasticsearch would do with one value in a typed field.
+#[derive(Debug, Clone, PartialEq)]
+pub enum Coercion {
+ /// Index the value exactly as it arrived.
+ AsIs,
+ /// ES indexes something else; this is the value to store instead.
+ Rewrite(Value),
+ /// ES refuses the whole document. Carries the `caused_by`-style detail.
+ Reject(String),
+}
+
+/// The first field of a document that its declared type refuses.
+#[derive(Debug, Clone, PartialEq)]
+pub struct BadField {
+ /// Full dotted path of the offending field.
+ pub field: String,
+ /// The declared type that refused it.
+ pub ftype: String,
+ /// Rendered preview of the offending value, ES-style.
+ pub preview: String,
+ /// Why it was refused (ES's `caused_by` sentence).
+ pub detail: String,
+}
+
+impl BadField {
+ /// The `reason` string ES puts on a `document_parsing_exception`.
+ ///
+ /// Same shape as the type-clash rejection `xerj_engine::bulk` already
+ /// emits, so the two read identically on the wire.
+ pub fn reason(&self, doc_id: &str) -> String {
+ format!(
+ "failed to parse field [{}] of type [{}] in document with id '{}'. \
+ Preview of field's value: '{}'",
+ self.field, self.ftype, doc_id, self.preview
+ )
+ }
+}
+
+/// Types this module enforces. Everything else (`text`, `keyword`, `date`,
+/// `ip`, `geo_point`, ranges, vectors…) is left to its own validator.
+pub fn is_enforced_type(ftype: &str) -> bool {
+ matches!(
+ ftype,
+ "byte"
+ | "short"
+ | "integer"
+ | "long"
+ | "unsigned_long"
+ | "half_float"
+ | "float"
+ | "double"
+ | "scaled_float"
+ | "boolean"
+ )
+}
+
+/// Inclusive integer bounds of an integral ES type, in `i128` so that both
+/// `i64::MIN` and `u64::MAX` are representable exactly — an `f64` bound would
+/// round `i64::MAX` up and let `9223372036854775808` through as a `long`.
+fn integral_bounds(ftype: &str) -> Option<(i128, i128)> {
+ Some(match ftype {
+ "byte" => (i8::MIN as i128, i8::MAX as i128),
+ "short" => (i16::MIN as i128, i16::MAX as i128),
+ "integer" => (i32::MIN as i128, i32::MAX as i128),
+ "long" => (i64::MIN as i128, i64::MAX as i128),
+ "unsigned_long" => (0, u64::MAX as i128),
+ _ => return None,
+ })
+}
+
+/// ES-style rendering of a value inside an error message: a string shows its
+/// text unquoted, an object shows `{k=v, …}`, everything else its JSON form.
+pub fn preview_value(v: &Value) -> String {
+ match v {
+ Value::String(s) => s.clone(),
+ Value::Object(o) => {
+ let parts: Vec<String> = o
+ .iter()
+ .map(|(k, val)| match val {
+ Value::String(s) => format!("{}={}", k, s),
+ other => format!("{}={}", k, other),
+ })
+ .collect();
+ format!("{{{}}}", parts.join(", "))
+ }
+ other => other.to_string(),
+ }
+}
+
+fn out_of_range(ftype: &str, v: &Value) -> Coercion {
+ Coercion::Reject(format!(
+ "Value [{}] is out of range for a[n] {}",
+ preview_value(v),
+ ftype
+ ))
+}
+
+fn not_finite(ftype: &str) -> Coercion {
+ Coercion::Reject(format!("[{}] supports only finite values", ftype))
+}
+
+fn not_a_number(ftype: &str) -> Coercion {
+ Coercion::Reject(format!(
+ "Current token is not a numeric value, expected a[n] {}",
+ ftype
+ ))
+}
+
+/// The JSON number for an integral value the caller has already clamped into
+/// its declared width's bounds, so it fits `i64` or `u64` exactly.
+///
+/// `as i64` alone would be wrong for `unsigned_long`, whose range runs past
+/// `i64::MAX`: a saturating cast turns `1e19` into `i64::MAX` — a silently
+/// different value, which is the class of bug this whole module exists to
+/// close.
+fn number_from_i128(i: i128) -> Value {
+ if i > i64::MAX as i128 {
+ Value::Number(Number::from(i as u64))
+ } else {
+ Value::Number(Number::from(i as i64))
+ }
+}
+
+/// The exact integer part of the finite float `f`, and that value clamped into
+/// a field's inclusive `lo..=hi` bounds.
+///
+/// Rust's float-to-int `as` saturates, so `f.trunc() as i128` is exact for
+/// every value any of these widths can hold — no UB, no wraparound — and the
+/// `clamp` then reproduces the Java narrowing cast ES applies after its own
+/// (double-widened) range check. The caller compares the two: equal means the
+/// float already IS the integer ES would index, different means ES's cast
+/// changed it.
+fn truncate_into(f: f64, lo: i128, hi: i128) -> (i128, i128) {
+ let exact = f.trunc() as i128;
+ (exact, exact.clamp(lo, hi))
+}
+
+/// Coerce one scalar (non-array) value for an integral field.
+fn coerce_integral(ftype: &str, coerce: bool, v: &Value) -> Coercion {
+ let (lo, hi) = match integral_bounds(ftype) {
+ Some(b) => b,
+ None => return Coercion::AsIs,
+ };
+ match v {
+ Value::Number(n) => {
+ if let Some(i) = n.as_i64() {
+ if (i as i128) < lo || (i as i128) > hi {
+ return out_of_range(ftype, v);
+ }
+ Coercion::AsIs
+ } else if let Some(u) = n.as_u64() {
+ if (u as i128) < lo || (u as i128) > hi {
+ return out_of_range(ftype, v);
+ }
+ Coercion::AsIs
+ } else {
+ // JSON float token: 1.9, 2.0, 1e40…
+ let f = match n.as_f64() {
+ Some(f) if f.is_finite() => f,
+ _ => return not_finite(ftype),
+ };
+ if f < lo as f64 || f > hi as f64 {
+ return out_of_range(ftype, v);
+ }
+ // That range check is ES's own — and it is WIDER than the
+ // declared width, because Java widens `Long.MAX_VALUE` to the
+ // `double` 2^63 to make the comparison. So the float token
+ // `9223372036854775808.0` survives it, and ES's `(long)` cast
+ // then saturates the value to `Long.MAX_VALUE`. The
+ // integer-token branch above is exact (`i128` bounds); this
+ // branch has to reproduce BOTH halves of ES's behaviour, or a
+ // bare `fract() == 0` shortcut stores a value the declared
+ // width cannot hold — the residual leniency the `i128` bound
+ // alone did not close.
+ let (exact, clamped) = truncate_into(f, lo, hi);
+ // Compare in the INTEGER domain. `clamped as f64 == f.trunc()`
+ // would be fooled by the very rounding this guards against:
+ // `i64::MAX as f64` is 2^63, so the two sides agree on exactly
+ // the value that must not pass.
+ if f.fract() == 0.0 && exact == clamped {
+ // `2.0` already equals the integer ES would index; leave
+ // `_source` alone rather than churn it to `2`.
+ return Coercion::AsIs;
+ }
+ if f.fract() != 0.0 && !coerce {
+ return Coercion::Reject(format!(
+ "Cannot coerce NUMBER to {} — value [{}] has a decimal part",
+ ftype, f
+ ));
+ }
+ Coercion::Rewrite(number_from_i128(clamped))
+ }
+ }
+ Value::String(s) => {
+ if s.trim().is_empty() {
+ // "no value" — see the module note on empty strings.
+ return Coercion::AsIs;
+ }
+ if !coerce {
+ return Coercion::Reject(format!("Cannot coerce a string to a[n] {}", ftype));
+ }
+ let f: f64 = match s.trim().parse::<f64>() {
+ Ok(f) if f.is_finite() => f,
+ Ok(_) => return not_finite(ftype),
+ Err(_) => return Coercion::Reject(format!("For input string: \"{}\"", s)),
+ };
+ if f < lo as f64 || f > hi as f64 {
+ return out_of_range(ftype, v);
+ }
+ // ES runs a string through `Double.parseDouble` and truncates, so
+ // `"1.9"` lands on 1 exactly as the bare number would. Prefer the
+ // exact integer parse when it succeeds, so a `long` beyond f64's
+ // 2^53 exact range keeps every digit — and range-check THAT parse
+ // in `i128`, not the `f64` above: `"9223372036854775808"` rounds to
+ // exactly `i64::MAX as f64` and so slips through the float bound,
+ // while ES's `Numbers.toLong` throws on it.
+ if let Ok(i) = s.trim().parse::<i128>() {
+ if i < lo || i > hi {
+ return out_of_range(ftype, v);
+ }
+ return Coercion::Rewrite(number_from_i128(i));
+ }
+ Coercion::Rewrite(number_from_i128(truncate_into(f, lo, hi).1))
+ }
+ Value::Bool(_) | Value::Object(_) | Value::Array(_) => not_a_number(ftype),
+ Value::Null => Coercion::AsIs,
+ }
+}
+
+/// Coerce one scalar value for a floating-point field.
+fn coerce_floating(ftype: &str, coerce: bool, v: &Value) -> Coercion {
+ // ES range-checks a `float` by requiring the value to survive the cast to
+ // 32-bit finitely, and a `half_float` by requiring it to survive the round
+ // trip through 16-bit precision. 65504 is the largest half-float, but
+ // round-to-nearest maps everything below 65520 back onto it — ES accepts
+ // 65510 and rejects 65520 — so the boundary is 65520, not 65504. A plain
+ // `f as f32` check (what the first cut used for both) accepts up to
+ // 3.4e38 — 33 decimal orders of magnitude past where a `half_float`
+ // actually overflows.
+ const HALF_FLOAT_OVERFLOWS_AT: f64 = 65_520.0;
+ let overflows = |f: f64| -> bool {
+ match ftype {
+ "float" => !(f as f32).is_finite(),
+ "half_float" => f.abs() >= HALF_FLOAT_OVERFLOWS_AT,
+ // `scaled_float` is deliberately NOT quantised by
+ // `scaling_factor` here — see the module header.
+ _ => false,
+ }
+ };
+ let check = |f: f64, rewritten: bool| -> Coercion {
+ if !f.is_finite() || overflows(f) {
+ return not_finite(ftype);
+ }
+ if rewritten {
+ match Number::from_f64(f) {
+ Some(n) => Coercion::Rewrite(Value::Number(n)),
+ None => not_finite(ftype),
+ }
+ } else {
+ Coercion::AsIs
+ }
+ };
+ match v {
+ Value::Number(n) => match n.as_f64() {
+ Some(f) => check(f, false),
+ None => not_finite(ftype),
+ },
+ Value::String(s) => {
+ if s.trim().is_empty() {
+ return Coercion::AsIs;
+ }
+ if !coerce {
+ return Coercion::Reject(format!("Cannot coerce a string to a[n] {}", ftype));
+ }
+ match s.trim().parse::<f64>() {
+ Ok(f) => check(f, true),
+ Err(_) => Coercion::Reject(format!("For input string: \"{}\"", s)),
+ }
+ }
+ Value::Bool(_) | Value::Object(_) | Value::Array(_) => not_a_number(ftype),
+ Value::Null => Coercion::AsIs,
+ }
+}
+
+/// Coerce one scalar value for a `boolean` field. ES has no `coerce` knob
+/// here: `true`/`false` and their string spellings are the entire domain.
+fn coerce_boolean(v: &Value) -> Coercion {
+ match v {
+ Value::Bool(_) | Value::Null => Coercion::AsIs,
+ Value::String(s) if s.is_empty() => Coercion::AsIs,
+ Value::String(s) => match s.as_str() {
+ "true" => Coercion::Rewrite(Value::Bool(true)),
+ "false" => Coercion::Rewrite(Value::Bool(false)),
+ other => Coercion::Reject(format!(
+ "Failed to parse value [{}] as only [true] or [false] are allowed.",
+ other
+ )),
+ },
+ other => Coercion::Reject(format!(
+ "Failed to parse value [{}] as only [true] or [false] are allowed.",
+ preview_value(other)
+ )),
+ }
+}
+
+/// What ES would index for `v` in a field declared `ftype`.
+///
+/// `coerce` is the field's effective `"coerce"` mapping parameter (ES default
+/// `true`). Types this module does not own return [`Coercion::AsIs`].
+pub fn coerce_value(ftype: &str, coerce: bool, v: &Value) -> Coercion {
+ match ftype {
+ "byte" | "short" | "integer" | "long" | "unsigned_long" => {
+ coerce_integral(ftype, coerce, v)
+ }
+ "half_float" | "float" | "double" | "scaled_float" => coerce_floating(ftype, coerce, v),
+ "boolean" => coerce_boolean(v),
+ _ => Coercion::AsIs,
+ }
+}
+
+/// Apply [`coerce_value`] to a whole field value, recursing into arrays.
+///
+/// Returns `Ok(Some(new))` when at least one element had to be rewritten,
+/// `Ok(None)` when the value stands as sent, and `Err((detail, offender))` on
+/// the first element ES would refuse.
+fn coerce_field(ftype: &str, coerce: bool, v: &Value) -> Result<Option<Value>, (String, Value)> {
+ match v {
+ Value::Array(arr) => {
+ let mut out = Vec::with_capacity(arr.len());
+ let mut changed = false;
+ for el in arr {
+ match coerce_field(ftype, coerce, el)? {
+ Some(new) => {
+ changed = true;
+ out.push(new);
+ }
+ None => out.push(el.clone()),
+ }
+ }
+ Ok(changed.then_some(Value::Array(out)))
+ }
+ scalar => match coerce_value(ftype, coerce, scalar) {
+ Coercion::AsIs => Ok(None),
+ Coercion::Rewrite(new) => Ok(Some(new)),
+ Coercion::Reject(detail) => Err((detail, scalar.clone())),
+ },
+ }
+}
+
+/// Walk `doc` against the mapping `properties`, coercing every numeric and
+/// boolean field in place.
+///
+/// Returns `Ok(true)` when the document was modified, `Ok(false)` when it was
+/// already ES-shaped, and `Err(BadField)` at the first value ES would refuse —
+/// the caller turns that into a 400 `document_parsing_exception`.
+///
+/// Fields carrying `"ignore_malformed": true` are skipped outright: that
+/// mapping asks for the bad value to be dropped into `_ignored`, which the
+/// `ignore_malformed` walker has already done by the time this runs.
+pub fn coerce_document(
+ doc: &mut Map<String, Value>,
+ props: &Map<String, Value>,
+) -> Result<bool, BadField> {
+ coerce_in(doc, props, "")
+}
+
+fn coerce_in(
+ doc: &mut Map<String, Value>,
+ props: &Map<String, Value>,
+ prefix: &str,
+) -> Result<bool, BadField> {
+ let mut changed = false;
+ let keys: Vec<String> = doc.keys().cloned().collect();
+ for field in keys {
+ let Some(spec) = props.get(&field) else {
+ continue;
+ };
+ // `"enabled": false` — ES does not parse, index or validate ANYTHING
+ // inside such an object; it is stored in `_source` and otherwise
+ // ignored. The object recursion below keys off `properties`, and a
+ // disabled object is still allowed to declare typed sub-properties, so
+ // without this guard those would be enforced where ES ignores them.
+ if spec.get("enabled").and_then(Value::as_bool) == Some(false) {
+ continue;
+ }
+ let full = if prefix.is_empty() {
+ field.clone()
+ } else {
+ format!("{}.{}", prefix, field)
+ };
+ let ftype = spec.get("type").and_then(Value::as_str).unwrap_or("");
+
+ // Sub-objects: recurse. A mapping with `properties` but no declared
+ // `type` is an implicit object in ES, so key off `properties` rather
+ // than off the type name.
+ if let Some(child_props) = spec.get("properties").and_then(Value::as_object) {
+ if ftype.is_empty() || ftype == "object" || ftype == "nested" {
+ let child_props = child_props.clone();
+ match doc.get_mut(&field) {
+ Some(Value::Object(child)) => {
+ changed |= coerce_in(child, &child_props, &full)?;
+ }
+ Some(Value::Array(arr)) => {
+ for el in arr.iter_mut() {
+ if let Value::Object(child) = el {
+ changed |= coerce_in(child, &child_props, &full)?;
+ }
+ }
+ }
+ _ => {}
+ }
+ continue;
+ }
+ }
+
+ if !is_enforced_type(ftype) {
+ continue;
+ }
+ if spec
+ .get("ignore_malformed")
+ .and_then(Value::as_bool)
+ .unwrap_or(false)
+ {
+ continue;
+ }
+ let coerce = spec.get("coerce").and_then(Value::as_bool).unwrap_or(true);
+
+ let Some(value) = doc.get(&field) else {
+ continue;
+ };
+ match coerce_field(ftype, coerce, value) {
+ Ok(None) => {}
+ Ok(Some(new)) => {
+ doc.insert(field.clone(), new);
+ changed = true;
+ }
+ Err((detail, bad)) => {
+ return Err(BadField {
+ field: full,
+ ftype: ftype.to_string(),
+ preview: preview_value(&bad),
+ detail,
+ });
+ }
+ }
+ }
+ Ok(changed)
+}
+
+/// Does this mapping declare ANY field whose type this module enforces?
+///
+/// The gate that keeps the `_bulk` turbo path free: an index with no numeric
+/// or boolean field never needs its raw doc bytes parsed for coercion, so it
+/// skips the whole check. Conservative on false positives (a user field
+/// literally named `type` holding the string `"integer"` still returns true
+/// and pays for one parse); never falsely negative.
+pub fn mapping_has_enforced_types(mapping: &Value) -> bool {
+ match mapping {
+ Value::Object(m) => {
+ if m.get("type")
+ .and_then(Value::as_str)
+ .map(is_enforced_type)
+ .unwrap_or(false)
+ {
+ return true;
+ }
+ m.values().any(mapping_has_enforced_types)
+ }
+ Value::Array(arr) => arr.iter().any(mapping_has_enforced_types),
+ _ => false,
+ }
+}
+
+/// Pull the `properties` map out of a stored index mapping, which is held
+/// either bare (`{"properties": …}`) or wrapped (`{"mappings": {"properties":
+/// …}}`) depending on which API wrote it.
+pub fn mapping_properties(mapping: &Value) -> Option<&Map<String, Value>> {
+ mapping
+ .get("properties")
+ .or_else(|| mapping.get("mappings").and_then(|m| m.get("properties")))
+ .and_then(Value::as_object)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use serde_json::json;
+
+ fn coerced(ftype: &str, v: Value) -> Coercion {
+ coerce_value(ftype, true, &v)
+ }
+
+ #[test]
+ fn integer_truncates_a_fraction_the_way_es_does() {
+ assert_eq!(coerced("integer", json!(1.9)), Coercion::Rewrite(json!(1)));
+ assert_eq!(
+ coerced("integer", json!(-1.9)),
+ Coercion::Rewrite(json!(-1))
+ );
+ assert_eq!(coerced("long", json!(2.5)), Coercion::Rewrite(json!(2)));
+ // Already integral: no `_source` churn.
+ assert_eq!(coerced("integer", json!(2.0)), Coercion::AsIs);
+ assert_eq!(coerced("integer", json!(7)), Coercion::AsIs);
+ }
+
+ #[test]
+ fn numeric_strings_are_coerced_and_junk_strings_are_refused() {
+ assert_eq!(coerced("integer", json!("5")), Coercion::Rewrite(json!(5)));
+ assert_eq!(
+ coerced("integer", json!("1.9")),
+ Coercion::Rewrite(json!(1))
+ );
+ assert_eq!(
+ coerced("double", json!("2.5")),
+ Coercion::Rewrite(json!(2.5))
+ );
+ assert!(matches!(coerced("long", json!("abc")), Coercion::Reject(_)));
+ assert!(matches!(
+ coerced("double", json!("abc")),
+ Coercion::Reject(_)
+ ));
+ }
+
+ #[test]
+ fn out_of_range_is_refused_on_every_width() {
+ assert!(matches!(
+ coerced("integer", json!(9999999999i64)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(coerced("byte", json!(200)), Coercion::Reject(_)));
+ assert!(matches!(
+ coerced("short", json!(40000)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerced("unsigned_long", json!(-1)),
+ Coercion::Reject(_)
+ ));
+ // A u64 beyond i64::MAX must not pass as a `long`; an f64 bound would
+ // have rounded i64::MAX up and let this through. The same value must
+ // be refused in every SPELLING it can arrive in — bare integer token,
+ // string, and float token (which ES's own check admits and its cast
+ // then saturates).
+ assert!(matches!(
+ coerced("long", json!(9223372036854775808u64)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerced("long", json!("9223372036854775808")),
+ Coercion::Reject(_)
+ ));
+ assert_eq!(
+ coerced("long", json!("9223372036854775807")),
+ Coercion::Rewrite(json!(9223372036854775807i64))
+ );
+ assert_eq!(
+ coerced("long", json!(9223372036854775807i64)),
+ Coercion::AsIs
+ );
+ assert_eq!(coerced("integer", json!(2147483647)), Coercion::AsIs);
+ }
+
+ #[test]
+ fn an_unsigned_long_past_i64_max_truncates_without_saturating() {
+ // 1e19 is inside `unsigned_long` and outside `i64`. This string form
+ // reaches the truncating cast (neither `i64` nor `u64` parses it), and
+ // a plain `as i64` there would have clamped it to i64::MAX — a
+ // silently different value, exactly the class of bug this closes.
+ let got = coerced("unsigned_long", json!("10000000000000000000.5"));
+ match got {
+ Coercion::Rewrite(Value::Number(n)) => {
+ assert_eq!(n.as_u64(), Some(10_000_000_000_000_000_000));
+ }
+ other => panic!("unexpected: {other:?}"),
+ }
+ assert!(matches!(
+ coerced("unsigned_long", json!(1.0e30f64)),
+ Coercion::Reject(_)
+ ));
+ }
+
+ #[test]
+ fn objects_and_arrays_are_not_numbers() {
+ assert!(matches!(
+ coerced("integer", json!({"bad": "x"})),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerced("integer", json!(true)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerced("boolean", json!({"a": 1})),
+ Coercion::Reject(_)
+ ));
+ }
+
+ #[test]
+ fn coerce_false_refuses_what_coerce_true_would_fix() {
+ assert!(matches!(
+ coerce_value("integer", false, &json!(1.9)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerce_value("integer", false, &json!("5")),
+ Coercion::Reject(_)
+ ));
+ // The range check is not a coercion and applies either way.
+ assert!(matches!(
+ coerce_value("integer", false, &json!(9999999999i64)),
+ Coercion::Reject(_)
+ ));
+ }
+
+ #[test]
+ fn boolean_takes_true_false_and_their_spellings_only() {
+ assert_eq!(coerced("boolean", json!(true)), Coercion::AsIs);
+ assert_eq!(
+ coerced("boolean", json!("true")),
+ Coercion::Rewrite(json!(true))
+ );
+ assert_eq!(
+ coerced("boolean", json!("false")),
+ Coercion::Rewrite(json!(false))
+ );
+ assert!(matches!(
+ coerced("boolean", json!("yes")),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(coerced("boolean", json!(1)), Coercion::Reject(_)));
+ }
+
+ /// The `i128` bound closes the INTEGER token. The float token has a second
+ /// hole: ES compares the raw double against a bound Java widened to
+ /// `double`, so `Long.MAX_VALUE` becomes 2^63 and `9223372036854775808.0`
+ /// passes the check — ES's `(long)` cast then saturates it. Storing the
+ /// float verbatim (what a bare `fract() == 0` shortcut does) leaves a
+ /// value no `long` can hold sitting in a `long` field.
+ #[test]
+ fn a_float_token_past_the_width_is_saturated_the_way_es_casts_it() {
+ assert_eq!(
+ coerced("long", json!(9223372036854775808.0f64)),
+ Coercion::Rewrite(json!(9223372036854775807i64))
+ );
+ assert_eq!(
+ coerced("integer", json!(-2147483649.0f64)),
+ Coercion::Reject("Value [-2147483649.0] is out of range for a[n] integer".to_string())
+ );
+ // Inside the width, an integral float is still left alone.
+ assert_eq!(
+ coerced("long", json!(9223372036854775807.0f64)),
+ Coercion::Rewrite(json!(9223372036854775807i64))
+ );
+ assert_eq!(coerced("long", json!(1234.0)), Coercion::AsIs);
+ }
+
+ /// ES round-trips a `half_float` through 16-bit precision and rejects what
+ /// comes back infinite. 65504 is the largest half-float and
+ /// round-to-nearest maps everything under 65520 onto it, so the boundary
+ /// is 65520 — an `f32` finiteness check (which accepts 3.4e38) is ~2^128
+ /// too wide.
+ #[test]
+ fn half_float_is_bounded_at_the_half_precision_overflow() {
+ assert_eq!(coerced("half_float", json!(65504.0)), Coercion::AsIs);
+ assert_eq!(coerced("half_float", json!(65510.0)), Coercion::AsIs);
+ assert!(matches!(
+ coerced("half_float", json!(65520.0)),
+ Coercion::Reject(_)
+ ));
+ assert!(matches!(
+ coerced("half_float", json!(1.0e38)),
+ Coercion::Reject(_)
+ ));
+ // `float` keeps its own, wider bound.
+ assert_eq!(coerced("float", json!(1.0e38)), Coercion::AsIs);
+ }
+
+ /// `"enabled": false` tells ES not to parse anything inside the object.
+ /// The recursion keys off `properties`, which a disabled object may still
+ /// declare, so it needs its own guard.
+ #[test]
+ fn a_disabled_object_is_not_walked() {
+ let props = json!({
+ "meta": {
+ "enabled": false,
+ "properties": {"n": {"type": "integer"}}
+ }
+ });
+ let mut doc = json!({"meta": {"n": "not a number at all"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ assert!(!coerce_document(&mut doc, props.as_object().unwrap())
+ .expect("a disabled object must not be rejected"));
+ assert_eq!(doc["meta"]["n"], json!("not a number at all"));
+ }
+
+ #[test]
+ fn narrow_floats_refuse_values_that_overflow_them() {
+ assert!(matches!(
+ coerced("float", json!(1e300)),
+ Coercion::Reject(_)
+ ));
+ assert_eq!(coerced("double", json!(1e300)), Coercion::AsIs);
+ assert_eq!(coerced("float", json!(1.5)), Coercion::AsIs);
+ }
+
+ #[test]
+ fn empty_string_and_null_are_left_alone() {
+ assert_eq!(coerced("integer", json!("")), Coercion::AsIs);
+ assert_eq!(coerced("integer", Value::Null), Coercion::AsIs);
+ assert_eq!(coerced("boolean", json!("")), Coercion::AsIs);
+ assert_eq!(coerced("boolean", Value::Null), Coercion::AsIs);
+ }
+
+ #[test]
+ fn a_document_is_rewritten_in_place_and_nested_fields_are_reached() {
+ let props = json!({
+ "i": {"type": "integer"},
+ "b": {"type": "boolean"},
+ "inner": {"properties": {"n": {"type": "long"}}}
+ });
+ let mut doc = json!({"i": 1.9, "b": "true", "inner": {"n": "42"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ let changed = coerce_document(&mut doc, props.as_object().unwrap()).expect("should coerce");
+ assert!(changed);
+ assert_eq!(doc.get("i"), Some(&json!(1)));
+ assert_eq!(doc.get("b"), Some(&json!(true)));
+ assert_eq!(doc["inner"]["n"], json!(42));
+ }
+
+ #[test]
+ fn a_bad_nested_field_reports_its_dotted_path() {
+ let props = json!({"inner": {"properties": {"n": {"type": "integer"}}}});
+ let mut doc = json!({"inner": {"n": "abc"}}).as_object().unwrap().clone();
+ let bad = coerce_document(&mut doc, props.as_object().unwrap()).unwrap_err();
+ assert_eq!(bad.field, "inner.n");
+ assert_eq!(bad.ftype, "integer");
+ assert_eq!(bad.preview, "abc");
+ assert!(bad
+ .reason("7")
+ .contains("failed to parse field [inner.n] of type [integer] in document with id '7'"));
+ }
+
+ #[test]
+ fn arrays_are_coerced_element_wise() {
+ let props = json!({"i": {"type": "integer"}});
+ let mut doc = json!({"i": [1.9, "3", 4]}).as_object().unwrap().clone();
+ assert!(coerce_document(&mut doc, props.as_object().unwrap()).expect("ok"));
+ assert_eq!(doc.get("i"), Some(&json!([1, 3, 4])));
+
+ let mut bad_doc = json!({"i": [1, "abc"]}).as_object().unwrap().clone();
+ assert!(coerce_document(&mut bad_doc, props.as_object().unwrap()).is_err());
+ }
+
+ #[test]
+ fn ignore_malformed_fields_are_left_to_their_own_walker() {
+ let props = json!({"i": {"type": "integer", "ignore_malformed": true}});
+ let mut doc = json!({"i": "abc"}).as_object().unwrap().clone();
+ assert!(!coerce_document(&mut doc, props.as_object().unwrap()).expect("no rejection"));
+ assert_eq!(doc.get("i"), Some(&json!("abc")));
+ }
+
+ #[test]
+ fn unmapped_and_non_numeric_fields_are_untouched() {
+ let props = json!({"k": {"type": "keyword"}, "t": {"type": "text"}});
+ let mut doc = json!({"k": 5, "t": "hi", "unmapped": {"any": "shape"}})
+ .as_object()
+ .unwrap()
+ .clone();
+ assert!(!coerce_document(&mut doc, props.as_object().unwrap()).expect("ok"));
+ assert_eq!(doc.get("k"), Some(&json!(5)));
+ }
+
+ #[test]
+ fn the_turbo_gate_fires_only_on_a_numeric_or_boolean_mapping() {
+ assert!(mapping_has_enforced_types(&json!({
+ "properties": {"msg": {"type": "text"}, "n": {"type": "long"}}
+ })));
+ assert!(mapping_has_enforced_types(&json!({
+ "properties": {"o": {"properties": {"b": {"type": "boolean"}}}}
+ })));
+ assert!(!mapping_has_enforced_types(&json!({
+ "properties": {"msg": {"type": "text"}, "k": {"type": "keyword"}}
+ })));
+ assert!(!mapping_has_enforced_types(&json!({})));
+ }
+
+ #[test]
+ fn mapping_properties_reads_both_stored_shapes() {
+ let bare = json!({"properties": {"a": {"type": "integer"}}});
+ let wrapped = json!({"mappings": {"properties": {"a": {"type": "integer"}}}});
+ assert!(mapping_properties(&bare).is_some());
+ assert!(mapping_properties(&wrapped).is_some());
+ assert!(mapping_properties(&json!({"settings": {}})).is_none());
+ }
+}
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index 2ca0eb71a..e60f4d636 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -17,6 +17,8 @@
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
+//! - [`field_coercion`] — ES-faithful ingest-time coercion/enforcement for
+//! numeric and boolean fields (the one predicate every write path shares)
//! - [`schema`] — Index schema management and mapping evolution
//! - [`metrics`] — Prometheus counters, histograms, and gauges
//! - [`net`] — Network trust primitives (trusted-proxy CIDR matching)
@@ -25,6 +27,7 @@
pub mod config;
pub mod error;
pub mod feedback;
+pub mod field_coercion;
pub mod fsio;
pub mod metrics;
pub mod net;
diff --git a/engine/crates/xerj-engine/src/bulk.rs b/engine/crates/xerj-engine/src/bulk.rs
index 30d836308..55044ba0e 100644
--- a/engine/crates/xerj-engine/src/bulk.rs
+++ b/engine/crates/xerj-engine/src/bulk.rs
@@ -1195,7 +1195,102 @@ pub async fn process_bulk_with_opts(
has
};
- for action in parsed {
+ // Same gate for numeric / boolean field types (issue #781). Unlike the
+ // three above this does NOT divert to the slow path — that would cost
+ // every mapped-numeric index the turbo batch — it coerces the raw doc
+ // bytes in place right here and lets the action stay on turbo. The gate
+ // exists so an index with no numeric or boolean field never pays for the
+ // parse at all.
+ //
+ // The cache carries the PROPERTIES MAP, not just the boolean gate:
+ // `mapping_properties(...).cloned()` deep-copies the whole properties
+ // blob, and reading it per action would put an O(mapping) copy on the HTTP
+ // worker for every document in the batch — the cost M5.11 took off this
+ // path. Behind an `Arc` the per-action price is a refcount bump, and both
+ // enforcement sites (partition loop and per-item loop) share the one
+ // cache. `None` means "this index needs no check" — an unmapped index, or
+ // one whose mapping declares no numeric/boolean field.
+ type CoerceProps = Option<std::sync::Arc<serde_json::Map<String, Value>>>;
+ let mut index_type_check_props: HashMap<String, CoerceProps> = HashMap::new();
+ let typed_numeric_props =
+ |target: &str, cache: &mut HashMap<String, CoerceProps>| -> CoerceProps {
+ if let Some(p) = cache.get(target) {
+ return p.clone();
+ }
+ let props: CoerceProps = engine.index_mappings.get(target).and_then(|m| {
+ if !xerj_common::field_coercion::mapping_has_enforced_types(&m) {
+ return None;
+ }
+ xerj_common::field_coercion::mapping_properties(&m)
+ .cloned()
+ .map(std::sync::Arc::new)
+ });
+ cache.insert(target.to_string(), props.clone());
+ props
+ };
+
+ for mut action in parsed {
+ // ── Numeric / boolean type enforcement (issue #781) ──────────────
+ //
+ // A field mapped `integer` used to keep whatever arrived — `1.9`,
+ // `9999999999`, `"abc"`, even an object — so `range {gte: 1.5}`
+ // matched a doc that ES stores as `1` and `term {i: 1}` missed it.
+ // Values ES coerces are coerced here (`1.9` → `1`, `"5"` → `5`,
+ // `"true"` → `true`); values ES refuses become a per-item 400
+ // `document_parsing_exception`, like the date-format check below.
+ //
+ // Runs at partition time so it covers the turbo batch path as well
+ // as the per-item loop. Actions carrying a pipeline are skipped: ES
+ // runs the pipeline before parsing the document, so those are
+ // checked after their pipeline in the per-item loop instead.
+ // `update` merges into a doc that was already validated on write.
+ if matches!(action.action_type.as_str(), "index" | "create") && action.pipeline.is_none() {
+ let props = typed_numeric_props(&action.target_index, &mut index_type_check_props);
+ if let Some(props) = props {
+ // `index` actions defer the JSON parse to the turbo path, so
+ // the body may not be built yet — parse it here, and only
+ // pay the re-serialize when a value actually changed.
+ let parsed_body = match action.doc_body.take() {
+ Some(Value::Object(o)) => Some(o),
+ Some(other) => {
+ action.doc_body = Some(other);
+ None
+ }
+ None => match serde_json::from_slice::<Value>(&action.doc_bytes) {
+ Ok(Value::Object(o)) => Some(o),
+ _ => None,
+ },
+ };
+ if let Some(mut body) = parsed_body {
+ match xerj_common::field_coercion::coerce_document(&mut body, &props) {
+ Ok(changed) => {
+ // Serialize from the map itself, not a clone of
+ // it — `preserve_order` is on workspace-wide, so
+ // the caller's `_source` key order survives.
+ if changed {
+ if let Ok(bytes) = serde_json::to_vec(&body) {
+ action.doc_bytes = std::sync::Arc::from(bytes.as_slice());
+ }
+ }
+ action.doc_body = Some(Value::Object(body));
+ }
+ Err(bad) => {
+ items[action.item_index] = Some(BulkItemResult {
+ action: action.action_type.clone(),
+ index: action.target_index.clone(),
+ id: action.doc_id.clone().unwrap_or_default(),
+ status: 400,
+ result: None,
+ error: Some(bad.reason(&action.doc_id.clone().unwrap_or_default())),
+ get_source: None,
+ });
+ errors = true;
+ continue;
+ }
+ }
+ }
+ }
+ }
// `index` with no `if_seq_no` / `routing` goes through the
// turbo batch path. Actions that carry CAS or routing metadata
// must be executed individually so we can honor them — route
@@ -2005,6 +2100,38 @@ pub async fn process_bulk_with_opts(
}
}
}
+ // Numeric / boolean type enforcement (issue #781). A field mapped
+ // `integer` used to keep whatever arrived — `1.9`, `9999999999`,
+ // `"abc"`, even an object — so the same query over the same declared
+ // -integer field returned different hits than ES. Values ES coerces
+ // are coerced here (`1.9` → `1`, `"5"` → `5`, `"true"` → `true`) and
+ // values ES refuses become a per-item 400 `document_parsing_exception`,
+ // exactly like the date-format check above. `index` / `create` only:
+ // an `update` merges into a doc that was already validated on write.
+ //
+ // The rules live in `xerj_common::field_coercion`, shared with the
+ // single-document `_doc` / `_create` handlers in `xerj-api`, so the
+ // two write paths cannot drift apart.
+ if matches!(action_type.as_str(), "index" | "create") {
+ let mapping_props = typed_numeric_props(&target_index, &mut index_type_check_props);
+ if let Some(props) = mapping_props {
+ if let Some(body) = doc_body.as_mut().and_then(Value::as_object_mut) {
+ if let Err(bad) = xerj_common::field_coercion::coerce_document(body, &props) {
+ items[item_idx] = Some(BulkItemResult {
+ action: action_type.to_string(),
+ index: target_index.clone(),
+ id: error_id.clone(),
+ status: 400,
+ result: None,
+ error: Some(bad.reason(&doc_id.clone().unwrap_or_default())),
+ get_source: None,
+ });
+ errors = true;
+ continue;
+ }
+ }
+ }
+ }
// doc_body already lazy-parsed above (for dynamic-templates
// validation). Nothing else to do here.
let result = execute_bulk_action(
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index ab16f910a..a5b554da8 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -186,6 +186,60 @@ mod collection_publication_fail_closed_tests {
));
}
+ #[test]
+ fn boolean_term_and_terms_spellings_normalise_to_the_indexed_value() {
+ // #781 follow-up: ingest coercion canonicalises `"true"`/`"false"` to
+ // real JSON booleans in `_source`, so a query still spelling them as
+ // strings named a value no document held. On a doc-values-only boolean
+ // (`390_doc_values_search.yml`) that meant `terms` returned 0 while the
+ // equivalent `term` still found the doc. Both arms now run the query
+ // value through the SAME `field_coercion` predicate the write path uses.
+ let mut schema = Schema::empty();
+ schema
+ .add_field(FieldConfig::new("flag", FieldType::Boolean))
+ .unwrap();
+ schema
+ .add_field(FieldConfig::new("name", FieldType::Keyword))
+ .unwrap();
+
+ let term = |f: &str, v: serde_json::Value| QueryNode::Term {
+ field: f.into(),
+ value: v,
+ boost: None,
+ };
+ let terms = |f: &str, v: Vec<serde_json::Value>| QueryNode::Terms {
+ field: f.into(),
+ values: v,
+ boost: None,
+ };
+
+ match rewrite_query_aliases(&term("flag", json!("false")), &schema) {
+ QueryNode::Term { field, value, .. } => {
+ assert_eq!(field, "flag");
+ assert_eq!(value, json!(false), "string spelling names the boolean");
+ }
+ other => panic!("boolean term must stay a Term, got {other:?}"),
+ }
+ match rewrite_query_aliases(&terms("flag", vec![json!("false"), json!(true)]), &schema) {
+ QueryNode::Terms { field, values, .. } => {
+ assert_eq!(field, "flag");
+ assert_eq!(values, vec![json!(false), json!(true)]);
+ }
+ other => panic!("boolean terms must stay a Terms, got {other:?}"),
+ }
+ // A value the ingest predicate refuses is left alone — a search must
+ // not 400 on it; it simply matches nothing, exactly as before.
+ match rewrite_query_aliases(&terms("flag", vec![json!("yes"), json!(1)]), &schema) {
+ QueryNode::Terms { values, .. } => assert_eq!(values, vec![json!("yes"), json!(1)]),
+ other => panic!("unparseable boolean values must survive, got {other:?}"),
+ }
+ // A NON-boolean field keeps its literal string — `"true"` is a keyword.
+ match rewrite_query_aliases(&term("name", json!("true")), &schema) {
+ QueryNode::Term { value, .. } => assert_eq!(value, json!("true")),
+ other => panic!("keyword term must be untouched, got {other:?}"),
+ }
+ }
+
async fn assert_active_reader_waits_for_commit<F, Fut>(
idx: &Arc<Index>,
id: &'static str,
@@ -479,6 +533,37 @@ mod collection_publication_fail_closed_tests {
);
}
+ /// A field that is TYPE-mixed inside one segment must ship NO doc-values
+ /// column. Only one column can carry a name, so the keyword pass would
+ /// overwrite the numeric one and file every number/boolean doc as null —
+ /// a column consumers treat as exact, which then hides those docs from the
+ /// `term`/`terms` prefilter entirely.
+ ///
+ /// #781 made this reachable in ordinary use: ingest coercion canonicalises
+ /// `"true"` to `true`, so an index written across the upgrade holds both
+ /// spellings, and a flush that lands them in ONE segment (the 2-core CI
+ /// shape) built the lying column — `terms {b:["true"]}` then returned the
+ /// pre-upgrade document and silently dropped the coerced one.
+ #[test]
+ fn build_doc_value_columns_ships_nothing_for_a_type_mixed_field() {
+ let legacy = json!({"b": "true", "n": "5", "ok": "kw"});
+ let coerced = json!({"b": true, "n": 5, "ok": "kw"});
+ let skip = std::collections::HashSet::new();
+ let cols =
+ super::build_doc_value_columns([Some(&legacy), Some(&coerced)].into_iter(), &skip);
+ assert!(
+ !cols.contains_key("b") && !cols.contains_key("n"),
+ "a type-mixed field must ship no column (got {:?})",
+ cols.keys().collect::<Vec<_>>()
+ );
+ // Not vacuous: a field that stayed one type still gets its column.
+ assert!(
+ cols.contains_key("ok"),
+ "a single-typed field must still be shadowed (got {:?})",
+ cols.keys().collect::<Vec<_>>()
+ );
+ }
+
async fn assert_flush_publication_blocks_reader_then_finishes(inject_error: bool) {
let dir = tempfile::tempdir().unwrap();
let mut config = Config::default();
@@ -22375,6 +22460,31 @@ fn build_doc_value_columns<'a>(
keyword.remove(f);
}
+ // Same rule for a field that arrived TYPE-mixed inside one segment — some
+ // docs numeric/boolean, others a string. Each kind was collected into its
+ // own map, and only one column can carry the name: the `keyword` loop below
+ // runs second and would overwrite the numeric column, leaving every
+ // number/boolean doc filed as NULL in a column consumers trust as exact.
+ // `build_term_prefilter_cached` then returns a set that is not a superset
+ // and the scan never sees the dropped docs — a confident wrong count, the
+ // same failure shape the multi-valued poisoning above exists to prevent.
+ //
+ // This is exactly what #781's ingest coercion makes reachable: an index
+ // written across that upgrade holds `"true"` in one doc and `true` in the
+ // next, and a flush that lands both in ONE segment (the 2-core CI shape;
+ // a many-core box scatters them and each segment stays homogeneous) built
+ // the lying column. Ship no column instead; consumers take the
+ // abandon-to-scan path they already take for an absent field.
+ let mixed_type: Vec<String> = numeric
+ .keys()
+ .filter(|f| keyword.contains_key(*f))
+ .cloned()
+ .collect();
+ for f in &mixed_type {
+ numeric.remove(f);
+ keyword.remove(f);
+ }
+
// Pad every column to total_docs so position == doc_id.
for col in numeric.values_mut() {
pad_with_none(col, total_docs);
@@ -36277,6 +36387,30 @@ fn date_string_has_subms(s: &str) -> bool {
>= 4
}
+/// Does the schema declare `field` a `boolean`?
+fn field_is_boolean(schema: &Schema, field: &str) -> bool {
+ declared_field(schema, field).is_some_and(|fc| matches!(fc.field_type, FieldType::Boolean))
+}
+
+/// Normalise one `term`/`terms` value on a `boolean` field to the value the
+/// WRITE path stores for it, using the same predicate the write path uses.
+///
+/// ES's `BooleanFieldMapper` parses a query value through `Booleans` exactly as
+/// it parses an indexed one, so `"true"` and `true` name the same term. XERJ
+/// indexes from the stored `_source`, and since #781 the ingest coercion
+/// canonicalises `"true"`/`"false"` to real JSON booleans there — so a query
+/// still spelling the value as a string names something no document holds.
+/// Running the query value through `field_coercion::coerce_value` is what keeps
+/// the two sides on one representation; a value the predicate refuses (`"yes"`,
+/// `1`) is left untouched and simply matches nothing, because a search must not
+/// 400 on a value ingest would have rejected.
+fn normalize_boolean_query_value(v: &Value) -> Value {
+ match xerj_common::field_coercion::coerce_value("boolean", true, v) {
+ xerj_common::field_coercion::Coercion::Rewrite(canonical) => canonical,
+ _ => v.clone(),
+ }
+}
+
/// Rewrite a query by resolving any field aliases to their canonical field names.
///
/// Traverses the query tree, replacing alias field names with their target paths.
@@ -36288,6 +36422,15 @@ fn rewrite_query_aliases(q: &QueryNode, schema: &Schema) -> QueryNode {
boost,
} => {
let resolved = resolve_field_alias(schema, field);
+ // A `term` on a `boolean` field names the boolean, however it is
+ // spelled — see `normalize_boolean_query_value`.
+ if field_is_boolean(schema, &resolved) {
+ return QueryNode::Term {
+ field: resolved,
+ value: normalize_boolean_query_value(value),
+ boost: *boost,
+ };
+ }
// A CIDR `term` on an `ip` field means the whole subnet. The
// source-scan matcher expands it (`ip_matches_cidr`), but the
// segment term-dictionary lookup can't, so it silently returned 0
@@ -36380,6 +36523,19 @@ fn rewrite_query_aliases(q: &QueryNode, schema: &Schema) -> QueryNode {
// single-term gate), plain (non-CIDR) ips, and any value that does
// not parse. Only fires on a declared `date`/`ip` field with >=1
// rewritable value; otherwise the plain `terms` is returned as-is.
+ //
+ // boolean: a `terms` spelling its values `"true"`/`"false"` must
+ // name the same term the single `term` arm does, or `terms`
+ // misses what `term` finds — the #781 ingest coercion stores a
+ // real JSON boolean, so the string spelling matched nothing on
+ // a doc-values-only field (`390_doc_values_search.yml`).
+ if field_is_boolean(schema, &resolved) {
+ return QueryNode::Terms {
+ field: resolved,
+ values: values.iter().map(normalize_boolean_query_value).collect(),
+ boost: *boost,
+ };
+ }
let ft_is_date = declared_field(schema, &resolved)
.is_some_and(|fc| matches!(fc.field_type, FieldType::Date));
let ft_is_ip = declared_field(schema, &resolved)
@@ -42971,20 +43127,43 @@ fn json_to_str(v: &Value) -> String {
fn json_values_equal(doc_val: &Option<Value>, query_val: &Value) -> bool {
match doc_val {
None => false,
- Some(dv) => {
- // Direct equality check.
- if dv == query_val {
- return true;
- }
- // Cross-type: number stored as string or vice versa.
- match (dv, query_val) {
- (Value::String(s), Value::Number(n)) => s.parse::<f64>().ok() == n.as_f64(),
- (Value::Number(n), Value::String(s)) => n.as_f64() == s.parse::<f64>().ok(),
- // Array field: any element matches.
- (Value::Array(arr), _) => arr.iter().any(|elem| elem == query_val),
- _ => false,
- }
- }
+ Some(dv) => json_scalar_equal(dv, query_val),
+ }
+}
+
+/// The cross-type equality of one stored value against one query operand.
+///
+/// Split out of [`json_values_equal`] so the array arm can recurse through it.
+/// An element of a MULTI-valued field has to be compared with the same
+/// tolerance a scalar gets, or the same document answers differently depending
+/// on whether its field happens to hold one value or several: with a bare
+/// `elem == query_val`, `{"b": "true"}` matches `term {b: true}` (via the
+/// boolean arm below) while `{"b": ["true"]}` does not.
+fn json_scalar_equal(dv: &Value, query_val: &Value) -> bool {
+ // Direct equality check.
+ if dv == query_val {
+ return true;
+ }
+ // Cross-type: number stored as string or vice versa.
+ match (dv, query_val) {
+ (Value::String(s), Value::Number(n)) => s.parse::<f64>().ok() == n.as_f64(),
+ (Value::Number(n), Value::String(s)) => n.as_f64() == s.parse::<f64>().ok(),
+ // Cross-type: a boolean and its string spelling name the same
+ // value. ES parses a query value through the same `Booleans`
+ // the field mapper uses, so `"true"` and `true` are one term;
+ // both directions matter here because since #781 ingest
+ // canonicalises a declared `boolean` to a real JSON boolean in
+ // `_source` while documents written BEFORE that still hold the
+ // string. Without this the two spellings split across the
+ // upgrade boundary — the doc-values fast paths already agree
+ // (they stringify a boolean needle before the lookup), so the
+ // `_source` scan was the one comparator that disagreed.
+ (Value::Bool(b), Value::String(s)) | (Value::String(s), Value::Bool(b)) => {
+ matches!((s.as_str(), *b), ("true", true) | ("false", false))
+ }
+ // Array field: any element matches, under the same tolerance.
+ (Value::Array(arr), _) => arr.iter().any(|elem| json_scalar_equal(elem, query_val)),
+ _ => false,
}
}