AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(vector): honour or refuse a dense_vector quantization, never ignore it
xerj-org9b2f9ae0a4e1
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 963bbc7fb..f7a85057d 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -1013,7 +1013,16 @@ pub async fn create_index(}let schema = if let Some(props) = mappings_val.get("properties") {- es_properties_to_schema(props)+ // #275: a field asked for something this build cannot deliver (an+ // unimplemented or unknown `quantization`). Refuse the create rather+ // than acknowledge it and echo the value back from `GET /_mapping`.+ match es_properties_to_schema(props) {+ Ok(schema) => schema,+ Err(reason) => {+ let e = xerj_common::XerjError::invalid_mapping(reason);+ return ApiError::new(e).into_response();+ }+ }} else {Schema::empty()};@@ -1171,6 +1180,16 @@ fn validate_properties(props: &serde_json::Map<String, Value>) -> Result<(), Str"Failed to parse mapping: The mapper type [{t}] declared on field [{name}] does not exist. It might have been created within a future version or requires a plugin to be installed. Check the documentation."));}+ // #275: the same `quantization` guard `es_properties_to_fields`+ // enforces, applied on the validation walk as well. Not+ // belt-and-braces: this walk recurses into `fields` (multi-fields)+ // where the schema builder does not, so a `dense_vector` declared+ // as a multi-field would otherwise still be accepted, echoed back+ // by `GET /_mapping`, and dropped on the floor. One shared+ // resolver means one accepted set and one error message.+ if t == "dense_vector" {+ resolve_vector_quantization(name, spec)?;+ }}// Recurse into sub-properties and multi-fields.if let Some(sub) = obj.get("properties").and_then(Value::as_object) {@@ -2283,7 +2302,18 @@ pub async fn put_mapping(if let Some(properties) = body.get("properties") {let current = idx.schema().await;let mut candidate = current.clone();- for field in es_properties_to_fields(properties) {+ // #275: refuse here too, and refuse in the plan phase — this loop+ // runs per target index, so an unimplemented `quantization` in a+ // multi-index `PUT _mapping` must abort before any of it is+ // published rather than after half of it took effect.+ let parsed = match es_properties_to_fields(properties) {+ Ok(parsed) => parsed,+ Err(reason) => {+ let e = xerj_common::XerjError::invalid_mapping(reason);+ return ApiError::new(e).into_response();+ }+ };+ for field in parsed {if let Some(existing_field) = current.field(&field.name) {if existing_field.field_type != field.field_type {let e = xerj_common::XerjError::invalid_mapping(format!(@@ -14903,19 +14933,104 @@ mod script_query_end_to_end_tests {// Schema conversion helpers// ─────────────────────────────────────────────────────────────────────────────-fn es_properties_to_schema(properties: &Value) -> Schema {+/// Decide what a `dense_vector` field's `quantization` means — the single+/// place that knows which schemes this build actually implements.+///+/// **`quantization` is strict; `index_options.type` is not, and the asymmetry+/// is deliberate.** `index_options.type` is Elasticsearch's key: a real ES+/// mapping carrying `hnsw`, `flat`, `bbq_hnsw` or `int4_hnsw` must still+/// create an index here, because refusing it would break the wire+/// compatibility this surface exists for. Those families are served by the+/// exact f32 scan, which still answers the contract ES states for them (a+/// nearest-neighbour result set) — a substitution that costs memory, not+/// answers. `quantization` is XERJ's own key with no ES meaning, so nothing is+/// owed to compatibility and everything is owed to the operator: the value is+/// a request for a specific memory/recall trade, and the only honest answers+/// are "done" and "no".+///+/// Issue #275: this match used to end in `_ => {}`. `"scalar4"`, `"binary"`,+/// `"int4"` and typos such as `"sq8"` were accepted with a `200`, echoed back+/// verbatim by `GET /_mapping`, and then ignored — the field stored+/// full-precision f32 while the mapping the operator read back to confirm it+/// said otherwise. Only `Some("scalar8")` is ever read by the serving path+/// (`xerj-engine/src/index.rs:10440` and `:10847`), so every other value was+/// inert. `Config::validate` has refused an unimplemented `[vector]+/// default_quantization` since #207; this is the same guard at the other+/// surface that sets the same field.+fn resolve_vector_quantization(+ field_name: &str,+ field_def: &Value,+) -> Result<Option<String>, String> {+ // ES expresses 8-bit scalar quantization through `index_options.type`:+ // the `int8_*` families map onto our serving-path SQ8 code store. Every+ // other family keeps the exact f32 scan, so `quantization` stays unset for+ // them and their behaviour is byte-identical to an unquantized field.+ let mut quant: Option<String> = None;+ if let Some(io_type) = field_def+ .get("index_options")+ .and_then(|io| io.get("type"))+ .and_then(Value::as_str)+ {+ if matches!(io_type, "int8_hnsw" | "int8_flat") {+ quant = Some("scalar8".to_string());+ }+ }++ let Some(q) = field_def.get("quantization") else {+ return Ok(quant);+ };+ let Some(q) = q.as_str() else {+ return Err(format!(+ "Failed to parse mapping: [quantization] on field [{field_name}] must be a string; \+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"+ ));+ };+ match q {+ // Honoured: a `scalar8` field keeps a per-field u8 code store and the+ // kNN path scores candidates by decoding those codes.+ "scalar8" | "int8" => Ok(Some("scalar8".to_string())),+ // Explicit opt-out, and it overrides an `int8_*` index_options.type.+ "none" => Ok(None),+ // Named but unimplemented. `binary` is refused for exactly the reason+ // `Config::validate` refuses `default_quantization = "binary"`: there+ // is no BinaryQuantizer, so honouring it would store full precision+ // while the operator sized a cluster on a 32× reduction. `scalar4` /+ // `int4` do have a `Scalar4Quantizer` in xerj-vector, but nothing in+ // the serving path reaches it — an unreachable quantizer is not an+ // implemented one, and accepting the name is the same lie in a+ // smaller font.+ "binary" | "scalar4" | "int4" => Err(format!(+ "Failed to parse mapping: quantization [{q}] on field [{field_name}] is not \+ implemented in this build and would silently store full-precision vectors; \+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"+ )),+ other => Err(format!(+ "Failed to parse mapping: unknown quantization [{other}] on field [{field_name}]; \+ this build implements [\"none\"] and [\"scalar8\"] (alias [\"int8\"])"+ )),+ }+}++fn es_properties_to_schema(properties: &Value) -> Result<Schema, String> {let mut schema = Schema::empty();- for field in es_properties_to_fields(properties) {+ for field in es_properties_to_fields(properties)? {let _ = schema.add_field(field);}- schema+ Ok(schema)}-fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {+/// Convert an ES `properties` object into engine [`FieldConfig`]s.+///+/// Returns `Err` with an ES-shaped `mapper_parsing_exception` reason when a+/// field asks for something this build cannot deliver. The check lives *here*,+/// at the site every engine schema is built from, and not only beside the+/// handlers: #207's sibling defect survived a fix placed next to one handler+/// because a second entry point kept building schemas the old way.+fn es_properties_to_fields(properties: &Value) -> Result<Vec<FieldConfig>, String> {let mut fields = Vec::new();let obj = match properties.as_object() {Some(o) => o,- None => return fields,+ None => return Ok(fields),};for (field_name, field_def) in obj {let es_type = field_def@@ -14936,7 +15051,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {let native_type = es_type_to_native(es_type);let mut fc = FieldConfig::new(field_name.clone(), native_type);if let Some(sub_props) = field_def.get("properties") {- fc.fields = es_properties_to_fields(sub_props);+ fc.fields = es_properties_to_fields(sub_props)?;}// `doc_values` — carry the DECLARED mapping intent onto the field.@@ -15064,32 +15179,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {fc.options.similarity = Some("cosine".to_string());}- // Opt-in scalar8 (SQ8) quantization. ES expresses this via- // `index_options.type`: the `int8_*` families are 8-bit scalar- // quantized, which we map to our serving-path SQ8 code store. A- // direct `quantization` string on the field def is also honoured- // (and overrides). Only `scalar8` changes the serving path — every- // other family (hnsw/flat/bbq*/int4*) keeps the exact f32 scan, so- // we leave `quantization` unset (None) for them and default- // behaviour stays byte-identical.- let mut quant: Option<String> = None;- if let Some(io_type) = field_def- .get("index_options")- .and_then(|io| io.get("type"))- .and_then(Value::as_str)- {- if matches!(io_type, "int8_hnsw" | "int8_flat") {- quant = Some("scalar8".to_string());- }- }- if let Some(q) = field_def.get("quantization").and_then(Value::as_str) {- match q {- "scalar8" | "int8" => quant = Some("scalar8".to_string()),- "none" => quant = None,- _ => {}- }- }- fc.options.quantization = quant;+ fc.options.quantization = resolve_vector_quantization(field_name, field_def)?;}fields.push(fc);@@ -15119,7 +15209,7 @@ fn es_properties_to_fields(properties: &Value) -> Vec<FieldConfig> {}).collect();fields.extend(semantic_companions);- fields+ Ok(fields)}fn field_configs_contain_logical_path(fields: &[FieldConfig], target: &str) -> bool {@@ -15175,7 +15265,8 @@ mod semantic_companion_schema_tests {"similarity": "dot_product"},"ports": { "type": "object" }- }));+ }))+ .expect("valid mapping");let body = fields.iter().find(|field| field.name == "body").unwrap();assert_eq!(@@ -15209,7 +15300,8 @@ mod semantic_companion_schema_tests {"dims": 7,"similarity": "l2_norm"}- }));+ }))+ .expect("valid mapping");assert_eq!(fields@@ -15313,7 +15405,8 @@ mod semantic_companion_schema_tests {"dimensions": 16},"measurements": { "type": "object" }- }));+ }))+ .expect("valid mapping");let vectors: Vec<_> = fields.iter()diff --git a/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rs b/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rsnew file mode 100644index 000000000..7350b6337--- /dev/null+++ b/engine/crates/xerj-api/tests/quantization_is_honoured_or_refused.rs@@ -0,0 +1,514 @@+//! A `dense_vector` mapping's `quantization` value, from the outside — issue+//! #275, another instance of the accepted-and-ignored class tracked in #204.+//!+//! Before the fix, `es_properties_to_fields` ended its `quantization` match on+//! `_ => {}`. `"scalar4"`, `"binary"`, `"int4"` and typos such as `"sq8"` were+//! taken with a `200 acknowledged`, echoed back verbatim by+//! `GET /{index}/_mapping`, and then ignored — the field stored full-precision+//! f32 while the mapping the operator read back to confirm it said otherwise.+//! Only `Some("scalar8")` is ever read by the serving path+//! (`xerj-engine/src/index.rs:10440`, `:10847`), so every other value was+//! inert — a kNN over such a field scored bit-identically to a field carrying+//! no `quantization` key at all.+//!+//! `Config::validate` has refused an unimplemented `[vector]+//! default_quantization` since #207. These tests pin the same guard on the+//! per-field mapping key, in all three of its branches:+//!+//! * **honoured** — `scalar8` / `int8` / `index_options.type: int8_hnsw`+//! accepted AND observably quantized; `none` accepted and observably not;+//! * **refused-unimplemented** — `binary`, `scalar4`, `int4`: named schemes+//! with no reachable quantizer behind them;+//! * **refused-unknown** — anything else, including case variants and+//! non-strings.+//!+//! The tests live at the crate boundary rather than beside the function on+//! purpose: #207's sibling defect survived a fix placed next to one handler,+//! so both entry points that build a schema (`PUT /{index}` and+//! `PUT /{index}/_mapping`) are exercised over real HTTP here.+//!+//! Elasticsearch is referenced for wire semantics only. It is AGPL-3.0/+//! SSPL-1.0/Elastic-2.0 licensed and no code from it is reproduced here.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++async fn app() -> (axum::Router, tempfile::TempDir) {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ (xerj_api::router::build_es_compat_router(state), dir)+}++async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {+ let response = app.clone().oneshot(req).await.expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, value)+}++async fn json_req(+ app: &axum::Router,+ method: &str,+ path: &str,+ body: Value,+) -> (StatusCode, Value) {+ send(+ app,+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+}++async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {+ send(+ app,+ Request::get(path).body(Body::empty()).expect("request"),+ )+ .await+}++/// `PUT /{index}` with a single `dense_vector` field carrying `field_def`.+async fn create_vector_index(+ app: &axum::Router,+ index: &str,+ field_def: Value,+) -> (StatusCode, Value) {+ json_req(+ app,+ "PUT",+ &format!("/{index}"),+ json!({ "mappings": { "properties": { "v": field_def } } }),+ )+ .await+}++/// Index a small corpus and return the kNN scores, in hit order.+///+/// The scores are the observable: a `scalar8` field is scored by decoding its+/// u8 codes, so its scores move off the exact-f32 values. A field that only+/// *claims* quantization scores bit-identically to one that claims nothing.+async fn knn_scores(app: &axum::Router, index: &str) -> Vec<f64> {+ for i in 1..=5u32 {+ let (status, body) = json_req(+ app,+ "POST",+ &format!("/{index}/_doc/{i}"),+ json!({ "v": [0.1 * i as f64, 0.2, 0.3, 0.4] }),+ )+ .await;+ assert!(status.is_success(), "index doc {i}: {status} {body}");+ }+ let (status, _) = json_req(app, "POST", &format!("/{index}/_refresh"), json!({})).await;+ assert!(status.is_success(), "refresh: {status}");++ let (status, body) = json_req(+ app,+ "POST",+ &format!("/{index}/_search"),+ json!({+ "knn": {+ "field": "v",+ "query_vector": [0.15, 0.25, 0.3, 0.4],+ "k": 5,+ "num_candidates": 50+ },+ "_source": false+ }),+ )+ .await;+ assert!(status.is_success(), "knn on {index}: {status} {body}");+ let hits = body["hits"]["hits"]+ .as_array()+ .unwrap_or_else(|| panic!("no hits on {index}: {body}"));+ assert_eq!(hits.len(), 5, "expected the whole corpus back: {body}");+ hits.iter()+ .map(|h| h["_score"].as_f64().expect("score"))+ .collect()+}++// ─────────────────────────────────────────────────────────────────────────────+// 1. accepted-and-honoured+// ─────────────────────────────────────────────────────────────────────────────++/// `scalar8` is the one value with a quantizer the serving path actually+/// reads, so it must be accepted — and accepted has to mean *honoured*, which+/// is what the score comparison establishes. `int8` is its ES-flavoured alias+/// and `index_options.type: int8_hnsw` is how an ES mapping asks for the same+/// thing; all three must land on the same behaviour.+#[tokio::test]+async fn scalar8_is_accepted_and_observably_quantizes() {+ let (app, _dir) = app().await;++ for (index, field_def) in [+ (+ "exact",+ json!({ "type": "dense_vector", "dims": 4, "quantization": "none" }),+ ),+ ("unset", json!({ "type": "dense_vector", "dims": 4 })),+ (+ "sq8",+ json!({ "type": "dense_vector", "dims": 4, "quantization": "scalar8" }),+ ),+ (+ "sq8-alias",+ json!({ "type": "dense_vector", "dims": 4, "quantization": "int8" }),+ ),+ (+ "sq8-es",+ json!({+ "type": "dense_vector",+ "dims": 4,+ "index_options": { "type": "int8_hnsw" }+ }),+ ),+ ] {+ let (status, body) = create_vector_index(&app, index, field_def).await;+ assert_eq!(status, StatusCode::OK, "create {index}: {body}");+ }++ let exact = knn_scores(&app, "exact").await;+ let unset = knn_scores(&app, "unset").await;+ let sq8 = knn_scores(&app, "sq8").await;+ let alias = knn_scores(&app, "sq8-alias").await;+ let es = knn_scores(&app, "sq8-es").await;++ // `none` is not a synonym for "some quantization": it must be bit-equal to+ // the field that declared nothing at all.+ assert_eq!(+ exact, unset,+ "quantization: none must score identically to an unquantized field"+ );++ // The honoured branch: SQ8 decode moves the scores off exact f32. If this+ // ever comes back equal, `scalar8` has silently stopped being honoured and+ // #275 is back in a new form.+ assert_ne!(+ sq8, exact,+ "quantization: scalar8 must change the scores — equal scores mean the \+ SQ8 code store was never consulted"+ );+ assert_eq!(alias, sq8, "int8 is an alias for scalar8");+ assert_eq!(+ es, sq8,+ "index_options.type: int8_hnsw must reach the same quantizer"+ );++ // The values are close, not arbitrary — a wrong quantizer would not land+ // within a percent of the exact scores. (Recall/RAM effects are not+ // claimed here; nothing in this test measures them.)+ for (q, e) in sq8.iter().zip(exact.iter()) {+ assert!(+ (q - e).abs() < 0.01,+ "SQ8 score {q} is not a plausible approximation of exact {e}"+ );+ }++ // And the mapping still round-trips the value it honoured.+ let (status, body) = get(&app, "/sq8/_mapping").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(+ body["sq8"]["mappings"]["properties"]["v"]["quantization"],+ json!("scalar8"),+ "{body}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// 2. refused-unimplemented+// ─────────────────────────────────────────────────────────────────────────────++/// `binary` has no `BinaryQuantizer` at all; `scalar4`/`int4` have a+/// `Scalar4Quantizer` in xerj-vector that nothing in the serving path reaches.+/// Either way the operator would get f32 while sizing a cluster on a 4×–32×+/// reduction, which is precisely what `Config::validate` refuses at startup for+/// `default_quantization = "binary"`. Refuse it here too, and say why.+#[tokio::test]+async fn unimplemented_quantization_is_refused_with_a_reason() {+ let (app, _dir) = app().await;++ for (i, q) in ["binary", "scalar4", "int4"].iter().enumerate() {+ let index = format!("bad{i}");+ let (status, body) = create_vector_index(+ &app,+ &index,+ json!({ "type": "dense_vector", "dims": 4, "quantization": q }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "quantization {q} must be refused, got {status}: {body}"+ );+ let reason = body.to_string();+ assert!(+ reason.contains("not implemented"),+ "the error must say the scheme is unimplemented, got: {reason}"+ );+ assert!(+ reason.contains(q) && reason.contains("scalar8"),+ "the error must name the rejected value and the accepted set, got: {reason}"+ );++ // Refused means not created — no index left behind to be discovered+ // later with a mapping that lies. (`GET /{index}/_mapping` is+ // wildcard-lenient and answers `200 {}` for a missing index, so the+ // existence question is asked of `GET /{index}`, which 404s.)+ let (status, body) = get(&app, &format!("/{index}")).await;+ assert_eq!(+ status,+ StatusCode::NOT_FOUND,+ "a refused mapping must not create the index: {body}"+ );+ let (_, body) = get(&app, &format!("/{index}/_mapping")).await;+ assert!(+ body.get(&index).is_none(),+ "a refused value must never be echoed back: {body}"+ );+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// 3. refused-unknown+// ─────────────────────────────────────────────────────────────────────────────++/// A typo is the common case and the worst one: `"sq8"` looks like it worked.+/// Reject anything outside the implemented set, including case variants (the+/// key is not case-insensitive anywhere else) and non-string values.+#[tokio::test]+async fn unknown_quantization_is_refused_and_never_echoed() {+ let (app, _dir) = app().await;++ for (i, q) in ["sq8", "SCALAR8", "Scalar8", "int_8", "", "pq", "true"]+ .iter()+ .enumerate()+ {+ let index = format!("typo{i}");+ let (status, body) = create_vector_index(+ &app,+ &index,+ json!({ "type": "dense_vector", "dims": 4, "quantization": q }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "quantization {q:?} must be refused, got {status}: {body}"+ );+ assert!(+ body.to_string().contains("scalar8"),+ "the error must name what this build does implement, got: {body}"+ );+ let (status, _) = get(&app, &format!("/{index}")).await;+ assert_eq!(+ status,+ StatusCode::NOT_FOUND,+ "{q:?} must not create {index}"+ );+ let (_, body) = get(&app, &format!("/{index}/_mapping")).await;+ assert!(+ body.get(&index).is_none(),+ "a refused value must never be echoed back: {body}"+ );+ }++ // A non-string is a different mistake with the same consequence.+ let (status, body) = create_vector_index(+ &app,+ "nonstring",+ json!({ "type": "dense_vector", "dims": 4, "quantization": 8 }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "a non-string quantization must be refused, got {status}: {body}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// 4. the guard is structural, not per-handler+// ─────────────────────────────────────────────────────────────────────────────++/// `PUT /{index}/_mapping` is the second way a schema gets built, and #207's+/// sibling defect survived a fix that only covered the first. Adding a bad+/// vector field to an existing index must be refused there too — and refused+/// in the plan phase, so nothing about the request is published.+#[tokio::test]+async fn put_mapping_refuses_the_same_values() {+ let (app, _dir) = app().await;+ let (status, body) = json_req(&app, "PUT", "/later", json!({})).await;+ assert_eq!(status, StatusCode::OK, "{body}");++ let (status, body) = json_req(+ &app,+ "PUT",+ "/later/_mapping",+ json!({ "properties": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "binary"+ }}}),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "PUT _mapping must refuse it too: {body}"+ );++ // Nothing was published: the field is absent, not present-and-lying.+ let (status, body) = get(&app, "/later/_mapping").await;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert!(+ body["later"]["mappings"]["properties"]["v"].is_null(),+ "a refused PUT _mapping must not leave the field behind: {body}"+ );++ // The good value still lands on the same endpoint.+ let (status, body) = json_req(+ &app,+ "PUT",+ "/later/_mapping",+ json!({ "properties": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"+ }}}),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "{body}");+}++/// A `dense_vector` nested under an object, and one declared as a multi-field,+/// go through the same recursive parse. The guard must reach them: a mapping+/// that hides the bad value one level down is the obvious way a same-file fix+/// gets bypassed.+#[tokio::test]+async fn nested_and_multi_field_vectors_are_checked_too() {+ let (app, _dir) = app().await;++ let (status, body) = json_req(+ &app,+ "PUT",+ "/nested",+ json!({ "mappings": { "properties": { "doc": {+ "type": "object",+ "properties": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "scalar4"+ }}+ }}}}),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "a nested dense_vector must be checked: {body}"+ );+ assert!(body.to_string().contains("scalar4"), "{body}");++ // A multi-field `dense_vector` never reaches the schema builder at all —+ // `es_properties_to_fields` recurses into `properties` but not `fields` —+ // so before #275 it was the one shape that could be accepted, echoed back+ // by `GET /_mapping`, and dropped entirely. The validation walk does+ // recurse into `fields`, which is why the guard is wired there too.+ let (status, body) = json_req(+ &app,+ "PUT",+ "/multifield",+ json!({ "mappings": { "properties": { "t": {+ "type": "keyword",+ "fields": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "binary"+ }}+ }}}}),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::BAD_REQUEST,+ "a multi-field dense_vector must be checked: {body}"+ );+ assert!(body.to_string().contains("binary"), "{body}");++ // And the good value still creates in both shapes.+ for (index, mapping) in [+ (+ "nested-ok",+ json!({ "mappings": { "properties": { "doc": {+ "type": "object",+ "properties": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"+ }}+ }}}}),+ ),+ (+ "multifield-ok",+ json!({ "mappings": { "properties": { "t": {+ "type": "keyword",+ "fields": { "v": {+ "type": "dense_vector", "dims": 4, "quantization": "scalar8"+ }}+ }}}}),+ ),+ ] {+ let (status, body) = json_req(&app, "PUT", &format!("/{index}"), mapping).await;+ assert_eq!(status, StatusCode::OK, "create {index}: {body}");+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// 5. ES wire compatibility is NOT tightened+// ─────────────────────────────────────────────────────────────────────────────++/// The strictness belongs on XERJ's own `quantization` key only. ES's+/// `index_options.type` must stay permissive: `hnsw`, `flat`, `bbq_*` and+/// `int4_*` are real Elasticsearch mappings, and a migrating user's index+/// definition has to keep creating. Those families are served by the exact f32+/// scan, which satisfies the contract ES states for them (a nearest-neighbour+/// result set) at recall 1.00 — a substitution that costs memory, not answers,+/// unlike a quantization that was asked for and skipped.+#[tokio::test]+async fn es_index_options_families_still_create() {+ let (app, _dir) = app().await;++ for (i, io) in [+ "hnsw",+ "flat",+ "int8_hnsw",+ "int8_flat",+ "int4_hnsw",+ "int4_flat",+ "bbq_hnsw",+ "bbq_flat",+ ]+ .iter()+ .enumerate()+ {+ let (status, body) = create_vector_index(+ &app,+ &format!("es{i}"),+ json!({+ "type": "dense_vector",+ "dims": 4,+ "index_options": { "type": io, "m": 16, "ef_construction": 100 }+ }),+ )+ .await;+ assert_eq!(+ status,+ StatusCode::OK,+ "ES index_options.type {io} must still create an index: {body}"+ );+ }+}diff --git a/landing/docs/vectors.html b/landing/docs/vectors.htmlindex 7e298f549..89097d09c 100644--- a/landing/docs/vectors.html+++ b/landing/docs/vectors.html@@ -157,7 +157,8 @@<li><strong>binary</strong> — 1-bit per dimension. Not implemented: the server <strong>refuses to start</strong> on <code class="inline">default_quantization = "binary"</code> rather than silently storing full precision while claiming a 32× saving.</li></ul><p>A 4-bit quantizer exists in the vector crate but no config or mapping value reaches it, so <code class="inline">scalar4</code> is not a mode you can select in this build.</p>- <p><strong>Known gap — a mapping's <code class="inline">quantization</code> value is not validated</strong> (<a href="https://github.com/xerj-org/xerj/issues/275">#275</a>). Startup refuses an unimplemented <code class="inline">[vector] default_quantization</code>, but the per-field mapping key does not: <code class="inline">"quantization": "scalar4"</code> — or <code class="inline">"binary"</code>, or a typo such as <code class="inline">"sq8"</code> — is accepted with a <code class="inline">200</code>, echoed back verbatim by <code class="inline">GET /_mapping</code>, and then ignored, so the field is stored at full-precision f32 while the mapping you read back says otherwise. Measured on rc.14. Until it is a <code class="inline">400</code>, <code class="inline">"scalar8"</code> (or <code class="inline">"int8"</code>, or <code class="inline">index_options.type: int8_hnsw</code>/<code class="inline">int8_flat</code>) is the only value that changes anything, and reading the mapping back is not a way to confirm it took.</p>+ <p><strong>A mapping's <code class="inline">quantization</code> value is honoured or refused — never accepted and ignored</strong> (<a href="https://github.com/xerj-org/xerj/issues/275">#275</a>, fixed). <code class="inline">"scalar8"</code> (alias <code class="inline">"int8"</code>) and <code class="inline">"none"</code> are the values this build implements. Anything else — <code class="inline">"binary"</code>, <code class="inline">"scalar4"</code>, <code class="inline">"int4"</code>, or a typo such as <code class="inline">"sq8"</code> — is now a <code class="inline">400 mapper_parsing_exception</code> naming the value and the accepted set, on both <code class="inline">PUT /<index></code> and <code class="inline">PUT /<index>/_mapping</code>, and the index is not created. Before the fix those values answered <code class="inline">200</code>, were echoed back verbatim by <code class="inline">GET /_mapping</code>, and were then ignored, so the field stored full-precision f32 while the mapping you read back to confirm it said otherwise. This is the same guard <code class="inline">Config::validate</code> applies to <code class="inline">[vector] default_quantization</code> at startup, now at the per-field surface that sets the same field.</p>+ <p>The strictness is confined to XERJ's own <code class="inline">quantization</code> key. Elasticsearch's <code class="inline">index_options.type</code> stays permissive — <code class="inline">hnsw</code>, <code class="inline">flat</code>, <code class="inline">int4_*</code> and <code class="inline">bbq_*</code> mappings still create, served by the exact f32 scan — because an ES mapping must not be rejected on a wire-compatibility surface.</p><h2 id="knn-query">KNN query</h2><pre class="code">{