Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(vector): honour or refuse a dense_vector quantization, never ignore it

xerj-org9b2f9ae0a4e1
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 963bbc7fb..f7a85057d 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -1013,7 +1013,16 @@ pub async fn create_index(
}
let schema = if let Some(props) = mappings_val.get("properties") {
- es_properties_to_schema(props)
+ // #275: a field asked for something this build cannot deliver (an
+ // unimplemented or unknown `quantization`). Refuse the create rather
+ // than acknowledge it and echo the value back from `GET /_mapping`.
+ match es_properties_to_schema(props) {
+ Ok(schema) => schema,
+ Err(reason) => {
+ let e = xerj_common::XerjError::invalid_mapping(reason);
+ return ApiError::new(e).into_response();
+ }
+ }
} else {
Schema::empty()
};
@@ -1171,6 +1180,16 @@ fn validate_properties(props: &serde_json::Map<String, Value>) -> Result<(), Str
"Failed to parse mapping: The mapper type [{t}] declared on field [{name}] does not exist. It might have been created within a future version or requires a plugin to be installed. Check the documentation."
));
}
+ // #275: the same `quantization` guard `es_properties_to_fields`
+ // enforces, applied on the validation walk as well. Not
+ // belt-and-braces: this walk recurses into `fields` (multi-fields)
+ // where the schema builder does not, so a `dense_vector` declared
+ // as a multi-field would otherwise still be accepted, echoed back
+ // by `GET /_mapping`, and dropped on the floor. One shared
+ // resolver means one accepted set and one error message.
+ if t == "dense_vector" {
+ resolve_vector_quantization(name, spec)?;
+ }
}
// Recurse into sub-properties and multi-fields.
if let Some(sub) = obj.get("properties").and_then(Value::as_object) {
@@ -2283,7 +2302,18 @@ pub async fn put_mapping(
if let Some(properties) = body.get("properties") {
let current = idx.schema().await;
let mut candidate = current.clone();
- for field in es_properties_to_fields(properties) {
+ // #275: refuse here too, and refuse in the plan phase — this loop
+ // runs per target index, so an unimplemented `quantization` in a
+ // multi-index `PUT _mapping` must abort before any of it is
+ // published rather than after half of it took effect.
+ let parsed = match es_properties_to_fields(properties) {
+ Ok(parsed) => parsed,
+ Err(reason) => {
+ let e = xerj_common::XerjError::invalid_mapping(reason);
+ return ApiError::new(e).into_response();
+ }
+ };
+ for field in parsed {
if let Some(existing_field) = current.field(&field.name) {
if existing_field.field_type != field.field_type {
let e = xerj_common::XerjError::invalid_mapping(format!(
@@ -14903,19 +14933,104 @@ mod script_query_end_to_end_tests {
// Schema conversion helpers
// ─────────────────────────────────────────────────────────────────────────────
-fn es_properties_to_schema(properties: &Value) -> Schema {
+/// Decide what a `dense_vector` field's `quantization` means — the single
+/// place that knows which schemes this build actually implements.
+///
+/// **`quantization` is strict; `index_options.type` is not, and the asymmetry
+/// is deliberate.** `index_options.type` is Elasticsearch's key: a real ES
+/// mapping carrying `hnsw`, `flat`, `bbq_hnsw` or `int4_hnsw` must still
+/// create an index here, because refusing it would break the wire
+/// compatibility this surface exists for. Those families are served by the
+/// exact f32 scan, which still answers the contract ES states for them (a
+/// nearest-neighbour result set) — a substitution that costs memory, not
+/// answers. `quantization` is XERJ's own key with no ES meaning, so nothing is
+/// owed to compatibility and everything is owed to the operator: the value is
+/// a request for a specific memory/recall trade, and the only honest answers
+/// are "done" and "no".
+///
+/// Issue #275: this match used to end in `_ => {}`. `"scalar4"`, `"binary"`,
+/// `"int4"` and typos such as `"sq8"` were accepted with a `200`, echoed back
+/// verbatim by `GET /_mapping`, and then ignored — the field stored
+/// full-precision f32 while the mapping the operator read back to confirm it
+/// said otherwise. Only `Some("scalar8")` is ever read by the serving path
+/// (`xerj-engine/src/index.rs:10440` and `:10847`), so every other value was
+/// inert. `Config::validate` has refused an unimplemented `[vector]
+/// default_quantization` since #207; this is the same guard at the other
+/// surface that sets the same field.
+fn resolve_vector_quantization(
+ field_name: &str,
+ field_def: &Value,
+) -> Result<Option<String>, String> {
+ // ES expresses 8-bit scalar quantization through `index_options.type`:
+ // the `int8_*` families map onto our serving-path SQ8 code store. Every
+ // other family keeps the exact f32 scan, so `quantization` stays unset for
+ // them and their behaviour is byte-identical to an unquantized field.
+ let mut quant: Option<String> = None;
+ if let Some(io_type) = field_def
+ .get("index_options")
+ .and_then(|io| io.get("type"))
+ .and_then(Value::as_str)
+ {
+ if matches!(io_type, "int8_hnsw" | "int8_flat") {
+ quant = Some("scalar8".to_string());
+ }
+ }
+
+ let Some(q) = field_def.get("quantization") else {
+ return Ok(quant);
+ };
+ let Some(q) = q.as_str() else {
+ return Err(format!(
+ "Failed to parse mapping: [quantization] on field [{field_name}] must be a string; \
+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"
+ ));
+ };
+ match q {
+ // Honoured: a `scalar8` field keeps a per-field u8 code store and the
+ // kNN path scores candidates by decoding those codes.
+ "scalar8" | "int8" => Ok(Some("scalar8".to_string())),
+ // Explicit opt-out, and it overrides an `int8_*` index_options.type.
+ "none" => Ok(None),
+ // Named but unimplemented. `binary` is refused for exactly the reason
+ // `Config::validate` refuses `default_quantization = "binary"`: there
+ // is no BinaryQuantizer, so honouring it would store full precision
+ // while the operator sized a cluster on a 32× reduction. `scalar4` /
+ // `int4` do have a `Scalar4Quantizer` in xerj-vector, but nothing in
+ // the serving path reaches it — an unreachable quantizer is not an
+ // implemented one, and accepting the name is the same lie in a
+ // smaller font.
+ "binary" | "scalar4" | "int4" => Err(format!(
+ "Failed to parse mapping: quantization [{q}] on field [{field_name}] is not \
+ implemented in this build and would silently store full-precision vectors; \
+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"
+ )),
+ other => Err(format!(
+ "Failed to parse mapping: unknown quantization [{other}] on field [{field_name}]; \
+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"
+ )),
+ }
+}
+
+fn es_properties_to_schema(properties: &Value) -> Result<Schema, String> {
let mut schema = Schema::empty();
- for field in es_properties_to_fields(properties) {
+ for field in es_properties_to_fields(properties)? {
let _ = schema.add_field(field);
}
- schema
+ Ok(schema)
}
-fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {
+/// Convert an ES `properties` object into engine [`FieldConfig`]s.
+///
+/// Returns `Err` with an ES-shaped `mapper_parsing_exception` reason when a
+/// field asks for something this build cannot deliver. The check lives *here*,
+/// at the site every engine schema is built from, and not only beside the
+/// handlers: #207's sibling defect survived a fix placed next to one handler
+/// because a second entry point kept building schemas the old way.
+fn es_properties_to_fields(properties: &Value) -> Result<Vec<FieldConfig>, String> {
let mut fields = Vec::new();
let obj = match properties.as_object() {
Some(o) => o,
- None => return fields,
+ None => return Ok(fields),
};
for (field_name, field_def) in obj {
let es_type = field_def
@@ -14936,7 +15051,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {
let native_type = es_type_to_native(es_type);
let mut fc = FieldConfig::new(field_name.clone(), native_type);
if let Some(sub_props) = field_def.get("properties") {
- fc.fields = es_properties_to_fields(sub_props);
+ fc.fields = es_properties_to_fields(sub_props)?;
}
// `doc_values` — carry the DECLARED mapping intent onto the field.
@@ -15064,32 +15179,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {
fc.options.similarity = Some("cosine".to_string());
}
- // Opt-in scalar8 (SQ8) quantization. ES expresses this via
- // `index_options.type`: the `int8_*` families are 8-bit scalar
- // quantized, which we map to our serving-path SQ8 code store. A
- // direct `quantization` string on the field def is also honoured
- // (and overrides). Only `scalar8` changes the serving path — every
- // other family (hnsw/flat/bbq*/int4*) keeps the exact f32 scan, so
- // we leave `quantization` unset (None) for them and default
- // behaviour stays byte-identical.
- let mut quant: Option<String> = None;
- if let Some(io_type) = field_def
- .get("index_options")
- .and_then(|io| io.get("type"))
- .and_then(Value::as_str)
- {
- if matches!(io_type, "int8_hnsw" | "int8_flat") {
- quant = Some("scalar8".to_string());
- }
- }
- if let Some(q) = field_def.get("quantization").and_then(Value::as_str) {
- match q {
- "scalar8" | "int8" => quant = Some("scalar8".to_string()),
- "none" => quant = None,
- _ => {}
- }
- }
- fc.options.quantization = quant;
+ fc.options.quantization = resolve_vector_quantization(field_name, field_def)?;
}
fields.push(fc);
@@ -15119,7 +15209,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {
})
.collect();
fields.extend(semantic_companions);
- fields
+ Ok(fields)
}
fn field_configs_contain_logical_path(fields: &[FieldConfig], target: &str) -> bool {
@@ -15175,7 +15265,8 @@ mod semantic_companion_schema_tests {
"similarity": "dot_product"
},
"ports": { "type": "object" }
- }));
+ }))
+ .expect("valid mapping");
let body = fields.iter().find(|field| field.name == "body").unwrap();
assert_eq!(
@@ -15209,7 +15300,8 @@ mod semantic_companion_schema_tests {
"dims": 7,
"similarity": "l2_norm"
}
- }));
+ }))
+ .expect("valid mapping");
assert_eq!(
fields
@@ -15313,7 +15405,8 @@ mod semantic_companion_schema_tests {
"dimensions": 16
},
"measurements": { "type": "object" }
- }));
+ }))
+ .expect("valid mapping");
let vectors: Vec<_> = fields
.iter()
diff --git a/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rs b/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rs
new file mode 100644
index 000000000..7350b6337
--- /dev/null
+++ b/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rs
@@ -0,0 +1,514 @@
+//! A `dense_vector` mapping's `quantization` value, from the outside — issue
+//! #275, another instance of the accepted-and-ignored class tracked in #204.
+//!
+//! Before the fix, `es_properties_to_fields` ended its `quantization` match on
+//! `_ => {}`. `"scalar4"`, `"binary"`, `"int4"` and typos such as `"sq8"` were
+//! taken with a `200 acknowledged`, echoed back verbatim by
+//! `GET /{index}/_mapping`, and then ignored — the field stored full-precision
+//! f32 while the mapping the operator read back to confirm it said otherwise.
+//! Only `Some("scalar8")` is ever read by the serving path
+//! (`xerj-engine/src/index.rs:10440`, `:10847`), so every other value was
+//! inert — a kNN over such a field scored bit-identically to a field carrying
+//! no `quantization` key at all.
+//!
+//! `Config::validate` has refused an unimplemented `[vector]
+//! default_quantization` since #207. These tests pin the same guard on the
+//! per-field mapping key, in all three of its branches:
+//!
+//! * **honoured** — `scalar8` / `int8` / `index_options.type: int8_hnsw`
+//! accepted AND observably quantized; `none` accepted and observably not;
+//! * **refused-unimplemented** — `binary`, `scalar4`, `int4`: named schemes
+//! with no reachable quantizer behind them;
+//! * **refused-unknown** — anything else, including case variants and
+//! non-strings.
+//!
+//! The tests live at the crate boundary rather than beside the function on
+//! purpose: #207's sibling defect survived a fix placed next to one handler,
+//! so both entry points that build a schema (`PUT /{index}` and
+//! `PUT /{index}/_mapping`) are exercised over real HTTP here.
+//!
+//! Elasticsearch is referenced for wire semantics only. It is AGPL-3.0/
+//! SSPL-1.0/Elastic-2.0 licensed and no code from it is reproduced here.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+async fn app() -> (axum::Router, tempfile::TempDir) {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ (xerj_api::router::build_es_compat_router(state), dir)
+}
+
+async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {
+ let response = app.clone().oneshot(req).await.expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, value)
+}
+
+async fn json_req(
+ app: &axum::Router,
+ method: &str,
+ path: &str,
+ body: Value,
+) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+}
+
+async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::get(path).body(Body::empty()).expect("request"),
+ )
+ .await
+}
+
+/// `PUT /{index}` with a single `dense_vector` field carrying `field_def`.
+async fn create_vector_index(
+ app: &axum::Router,
+ index: &str,
+ field_def: Value,
+) -> (StatusCode, Value) {
+ json_req(
+ app,
+ "PUT",
+ &format!("/{index}"),
+ json!({ "mappings": { "properties": { "v": field_def } } }),
+ )
+ .await
+}
+
+/// Index a small corpus and return the kNN scores, in hit order.
+///
+/// The scores are the observable: a `scalar8` field is scored by decoding its
+/// u8 codes, so its scores move off the exact-f32 values. A field that only
+/// *claims* quantization scores bit-identically to one that claims nothing.
+async fn knn_scores(app: &axum::Router, index: &str) -> Vec<f64> {
+ for i in 1..=5u32 {
+ let (status, body) = json_req(
+ app,
+ "POST",
+ &format!("/{index}/_doc/{i}"),
+ json!({ "v": [0.1 * i as f64, 0.2, 0.3, 0.4] }),
+ )
+ .await;
+ assert!(status.is_success(), "index doc {i}: {status} {body}");
+ }
+ let (status, _) = json_req(app, "POST", &format!("/{index}/_refresh"), json!({})).await;
+ assert!(status.is_success(), "refresh: {status}");
+
+ let (status, body) = json_req(
+ app,
+ "POST",
+ &format!("/{index}/_search"),
+ json!({
+ "knn": {
+ "field": "v",
+ "query_vector": [0.15, 0.25, 0.3, 0.4],
+ "k": 5,
+ "num_candidates": 50
+ },
+ "_source": false
+ }),
+ )
+ .await;
+ assert!(status.is_success(), "knn on {index}: {status} {body}");
+ let hits = body["hits"]["hits"]
+ .as_array()
+ .unwrap_or_else(|| panic!("no hits on {index}: {body}"));
+ assert_eq!(hits.len(), 5, "expected the whole corpus back: {body}");
+ hits.iter()
+ .map(|h| h["_score"].as_f64().expect("score"))
+ .collect()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 1. accepted-and-honoured
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `scalar8` is the one value with a quantizer the serving path actually
+/// reads, so it must be accepted — and accepted has to mean *honoured*, which
+/// is what the score comparison establishes. `int8` is its ES-flavoured alias
+/// and `index_options.type: int8_hnsw` is how an ES mapping asks for the same
+/// thing; all three must land on the same behaviour.
+#[tokio::test]
+async fn scalar8_is_accepted_and_observably_quantizes() {
+ let (app, _dir) = app().await;
+
+ for (index, field_def) in [
+ (
+ "exact",
+ json!({ "type": "dense_vector", "dims": 4, "quantization": "none" }),
+ ),
+ ("unset", json!({ "type": "dense_vector", "dims": 4 })),
+ (
+ "sq8",
+ json!({ "type": "dense_vector", "dims": 4, "quantization": "scalar8" }),
+ ),
+ (
+ "sq8-alias",
+ json!({ "type": "dense_vector", "dims": 4, "quantization": "int8" }),
+ ),
+ (
+ "sq8-es",
+ json!({
+ "type": "dense_vector",
+ "dims": 4,
+ "index_options": { "type": "int8_hnsw" }
+ }),
+ ),
+ ] {
+ let (status, body) = create_vector_index(&app, index, field_def).await;
+ assert_eq!(status, StatusCode::OK, "create {index}: {body}");
+ }
+
+ let exact = knn_scores(&app, "exact").await;
+ let unset = knn_scores(&app, "unset").await;
+ let sq8 = knn_scores(&app, "sq8").await;
+ let alias = knn_scores(&app, "sq8-alias").await;
+ let es = knn_scores(&app, "sq8-es").await;
+
+ // `none` is not a synonym for "some quantization": it must be bit-equal to
+ // the field that declared nothing at all.
+ assert_eq!(
+ exact, unset,
+ "quantization: none must score identically to an unquantized field"
+ );
+
+ // The honoured branch: SQ8 decode moves the scores off exact f32. If this
+ // ever comes back equal, `scalar8` has silently stopped being honoured and
+ // #275 is back in a new form.
+ assert_ne!(
+ sq8, exact,
+ "quantization: scalar8 must change the scores — equal scores mean the \
+ SQ8 code store was never consulted"
+ );
+ assert_eq!(alias, sq8, "int8 is an alias for scalar8");
+ assert_eq!(
+ es, sq8,
+ "index_options.type: int8_hnsw must reach the same quantizer"
+ );
+
+ // The values are close, not arbitrary — a wrong quantizer would not land
+ // within a percent of the exact scores. (Recall/RAM effects are not
+ // claimed here; nothing in this test measures them.)
+ for (q, e) in sq8.iter().zip(exact.iter()) {
+ assert!(
+ (q - e).abs() < 0.01,
+ "SQ8 score {q} is not a plausible approximation of exact {e}"
+ );
+ }
+
+ // And the mapping still round-trips the value it honoured.
+ let (status, body) = get(&app, "/sq8/_mapping").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert_eq!(
+ body["sq8"]["mappings"]["properties"]["v"]["quantization"],
+ json!("scalar8"),
+ "{body}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 2. refused-unimplemented
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `binary` has no `BinaryQuantizer` at all; `scalar4`/`int4` have a
+/// `Scalar4Quantizer` in xerj-vector that nothing in the serving path reaches.
+/// Either way the operator would get f32 while sizing a cluster on a 4×–32×
+/// reduction, which is precisely what `Config::validate` refuses at startup for
+/// `default_quantization = "binary"`. Refuse it here too, and say why.
+#[tokio::test]
+async fn unimplemented_quantization_is_refused_with_a_reason() {
+ let (app, _dir) = app().await;
+
+ for (i, q) in ["binary", "scalar4", "int4"].iter().enumerate() {
+ let index = format!("bad{i}");
+ let (status, body) = create_vector_index(
+ &app,
+ &index,
+ json!({ "type": "dense_vector", "dims": 4, "quantization": q }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "quantization {q} must be refused, got {status}: {body}"
+ );
+ let reason = body.to_string();
+ assert!(
+ reason.contains("not implemented"),
+ "the error must say the scheme is unimplemented, got: {reason}"
+ );
+ assert!(
+ reason.contains(q) && reason.contains("scalar8"),
+ "the error must name the rejected value and the accepted set, got: {reason}"
+ );
+
+ // Refused means not created — no index left behind to be discovered
+ // later with a mapping that lies. (`GET /{index}/_mapping` is
+ // wildcard-lenient and answers `200 {}` for a missing index, so the
+ // existence question is asked of `GET /{index}`, which 404s.)
+ let (status, body) = get(&app, &format!("/{index}")).await;
+ assert_eq!(
+ status,
+ StatusCode::NOT_FOUND,
+ "a refused mapping must not create the index: {body}"
+ );
+ let (_, body) = get(&app, &format!("/{index}/_mapping")).await;
+ assert!(
+ body.get(&index).is_none(),
+ "a refused value must never be echoed back: {body}"
+ );
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 3. refused-unknown
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// A typo is the common case and the worst one: `"sq8"` looks like it worked.
+/// Reject anything outside the implemented set, including case variants (the
+/// key is not case-insensitive anywhere else) and non-string values.
+#[tokio::test]
+async fn unknown_quantization_is_refused_and_never_echoed() {
+ let (app, _dir) = app().await;
+
+ for (i, q) in ["sq8", "SCALAR8", "Scalar8", "int_8", "", "pq", "true"]
+ .iter()
+ .enumerate()
+ {
+ let index = format!("typo{i}");
+ let (status, body) = create_vector_index(
+ &app,
+ &index,
+ json!({ "type": "dense_vector", "dims": 4, "quantization": q }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "quantization {q:?} must be refused, got {status}: {body}"
+ );
+ assert!(
+ body.to_string().contains("scalar8"),
+ "the error must name what this build does implement, got: {body}"
+ );
+ let (status, _) = get(&app, &format!("/{index}")).await;
+ assert_eq!(
+ status,
+ StatusCode::NOT_FOUND,
+ "{q:?} must not create {index}"
+ );
+ let (_, body) = get(&app, &format!("/{index}/_mapping")).await;
+ assert!(
+ body.get(&index).is_none(),
+ "a refused value must never be echoed back: {body}"
+ );
+ }
+
+ // A non-string is a different mistake with the same consequence.
+ let (status, body) = create_vector_index(
+ &app,
+ "nonstring",
+ json!({ "type": "dense_vector", "dims": 4, "quantization": 8 }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "a non-string quantization must be refused, got {status}: {body}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 4. the guard is structural, not per-handler
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `PUT /{index}/_mapping` is the second way a schema gets built, and #207's
+/// sibling defect survived a fix that only covered the first. Adding a bad
+/// vector field to an existing index must be refused there too — and refused
+/// in the plan phase, so nothing about the request is published.
+#[tokio::test]
+async fn put_mapping_refuses_the_same_values() {
+ let (app, _dir) = app().await;
+ let (status, body) = json_req(&app, "PUT", "/later", json!({})).await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ "/later/_mapping",
+ json!({ "properties": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "binary"
+ }}}),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "PUT _mapping must refuse it too: {body}"
+ );
+
+ // Nothing was published: the field is absent, not present-and-lying.
+ let (status, body) = get(&app, "/later/_mapping").await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+ assert!(
+ body["later"]["mappings"]["properties"]["v"].is_null(),
+ "a refused PUT _mapping must not leave the field behind: {body}"
+ );
+
+ // The good value still lands on the same endpoint.
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ "/later/_mapping",
+ json!({ "properties": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"
+ }}}),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "{body}");
+}
+
+/// A `dense_vector` nested under an object, and one declared as a multi-field,
+/// go through the same recursive parse. The guard must reach them: a mapping
+/// that hides the bad value one level down is the obvious way a same-file fix
+/// gets bypassed.
+#[tokio::test]
+async fn nested_and_multi_field_vectors_are_checked_too() {
+ let (app, _dir) = app().await;
+
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ "/nested",
+ json!({ "mappings": { "properties": { "doc": {
+ "type": "object",
+ "properties": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "scalar4"
+ }}
+ }}}}),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "a nested dense_vector must be checked: {body}"
+ );
+ assert!(body.to_string().contains("scalar4"), "{body}");
+
+ // A multi-field `dense_vector` never reaches the schema builder at all —
+ // `es_properties_to_fields` recurses into `properties` but not `fields` —
+ // so before #275 it was the one shape that could be accepted, echoed back
+ // by `GET /_mapping`, and dropped entirely. The validation walk does
+ // recurse into `fields`, which is why the guard is wired there too.
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ "/multifield",
+ json!({ "mappings": { "properties": { "t": {
+ "type": "keyword",
+ "fields": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "binary"
+ }}
+ }}}}),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::BAD_REQUEST,
+ "a multi-field dense_vector must be checked: {body}"
+ );
+ assert!(body.to_string().contains("binary"), "{body}");
+
+ // And the good value still creates in both shapes.
+ for (index, mapping) in [
+ (
+ "nested-ok",
+ json!({ "mappings": { "properties": { "doc": {
+ "type": "object",
+ "properties": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"
+ }}
+ }}}}),
+ ),
+ (
+ "multifield-ok",
+ json!({ "mappings": { "properties": { "t": {
+ "type": "keyword",
+ "fields": { "v": {
+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"
+ }}
+ }}}}),
+ ),
+ ] {
+ let (status, body) = json_req(&app, "PUT", &format!("/{index}"), mapping).await;
+ assert_eq!(status, StatusCode::OK, "create {index}: {body}");
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 5. ES wire compatibility is NOT tightened
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The strictness belongs on XERJ's own `quantization` key only. ES's
+/// `index_options.type` must stay permissive: `hnsw`, `flat`, `bbq_*` and
+/// `int4_*` are real Elasticsearch mappings, and a migrating user's index
+/// definition has to keep creating. Those families are served by the exact f32
+/// scan, which satisfies the contract ES states for them (a nearest-neighbour
+/// result set) at recall 1.00 — a substitution that costs memory, not answers,
+/// unlike a quantization that was asked for and skipped.
+#[tokio::test]
+async fn es_index_options_families_still_create() {
+ let (app, _dir) = app().await;
+
+ for (i, io) in [
+ "hnsw",
+ "flat",
+ "int8_hnsw",
+ "int8_flat",
+ "int4_hnsw",
+ "int4_flat",
+ "bbq_hnsw",
+ "bbq_flat",
+ ]
+ .iter()
+ .enumerate()
+ {
+ let (status, body) = create_vector_index(
+ &app,
+ &format!("es{i}"),
+ json!({
+ "type": "dense_vector",
+ "dims": 4,
+ "index_options": { "type": io, "m": 16, "ef_construction": 100 }
+ }),
+ )
+ .await;
+ assert_eq!(
+ status,
+ StatusCode::OK,
+ "ES index_options.type {io} must still create an index: {body}"
+ );
+ }
+}
diff --git a/landing/docs/vectors.html b/landing/docs/vectors.html
index 7e298f549..89097d09c 100644
--- a/landing/docs/vectors.html
+++ b/landing/docs/vectors.html
@@ -157,7 +157,8 @@
<li><strong>binary</strong> — 1-bit per dimension. Not implemented: the server <strong>refuses to start</strong> on <code class="inline">default_quantization = "binary"</code> rather than silently storing full precision while claiming a 32× saving.</li>
</ul>
<p>A 4-bit quantizer exists in the vector crate but no config or mapping value reaches it, so <code class="inline">scalar4</code> is not a mode you can select in this build.</p>
- <p><strong>Known gap — a mapping's <code class="inline">quantization</code> value is not validated</strong> (<a href="https://github.com/xerj-org/xerj/issues/275">#275</a>). Startup refuses an unimplemented <code class="inline">[vector] default_quantization</code>, but the per-field mapping key does not: <code class="inline">"quantization": "scalar4"</code> — or <code class="inline">"binary"</code>, or a typo such as <code class="inline">"sq8"</code> — is accepted with a <code class="inline">200</code>, echoed back verbatim by <code class="inline">GET /_mapping</code>, and then ignored, so the field is stored at full-precision f32 while the mapping you read back says otherwise. Measured on rc.14. Until it is a <code class="inline">400</code>, <code class="inline">"scalar8"</code> (or <code class="inline">"int8"</code>, or <code class="inline">index_options.type: int8_hnsw</code>/<code class="inline">int8_flat</code>) is the only value that changes anything, and reading the mapping back is not a way to confirm it took.</p>
+ <p><strong>A mapping's <code class="inline">quantization</code> value is honoured or refused — never accepted and ignored</strong> (<a href="https://github.com/xerj-org/xerj/issues/275">#275</a>, fixed). <code class="inline">"scalar8"</code> (alias <code class="inline">"int8"</code>) and <code class="inline">"none"</code> are the values this build implements. Anything else — <code class="inline">"binary"</code>, <code class="inline">"scalar4"</code>, <code class="inline">"int4"</code>, or a typo such as <code class="inline">"sq8"</code> — is now a <code class="inline">400 mapper_parsing_exception</code> naming the value and the accepted set, on both <code class="inline">PUT /&lt;index&gt;</code> and <code class="inline">PUT /&lt;index&gt;/_mapping</code>, and the index is not created. Before the fix those values answered <code class="inline">200</code>, were echoed back verbatim by <code class="inline">GET /_mapping</code>, and were then ignored, so the field stored full-precision f32 while the mapping you read back to confirm it said otherwise. This is the same guard <code class="inline">Config::validate</code> applies to <code class="inline">[vector] default_quantization</code> at startup, now at the per-field surface that sets the same field.</p>
+ <p>The strictness is confined to XERJ's own <code class="inline">quantization</code> key. Elasticsearch's <code class="inline">index_options.type</code> stays permissive — <code class="inline">hnsw</code>, <code class="inline">flat</code>, <code class="inline">int4_*</code> and <code class="inline">bbq_*</code> mappings still create, served by the exact f32 scan — because an ES mapping must not be rejected on a wire-compatibility surface.</p>
<h2 id="knn-query">KNN query</h2>
<pre class="code">{