Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(vectors): say that a mapping's quantization value is not validated

xerj-org7bb6eb77f575
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 80b35923e..760fd5877 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -100,6 +100,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`Config::default()`. An example that deliberately differs (the blocks whose
point is switching TLS or cluster mode on) has to carry `# not a default` on
the line, so the reader is told as well as the test.
+ - **Known gap, found while correcting the `scalar4` prose and disclosed
+ rather than quietly fixed:** the guard reads `config.html` against
+ `Config::default()`, so it covers the *config* half only. Nothing checks a
+ *mapping*, and a mapping is not validated either —
+ `es_compat.rs` matches `"scalar8" | "int8" | "none"` on a `dense_vector`
+ field's `quantization` and falls through `_ => {}`, so
+ `"quantization": "scalar4"` (or `"binary"`, or a typo like `"sq8"`) is
+ accepted with a 200, echoed back verbatim by `GET /_mapping`, and ignored
+ — the field is stored at full-precision f32 while the mapping says it is
+ quantized. Measured through the ES-compat router at this commit. It is
+ pre-existing and outside this diff; it is another instance of the
+ accepted-and-ignored class in
+ [#204](https://github.com/xerj-org/xerj/issues/204), is filed with the
+ repro and a proposed 400 as
+ [#275](https://github.com/xerj-org/xerj/issues/275), and
+ `landing/docs/vectors.html` now says so on the page instead of leaving
+ "not a mode you can select" to imply the value is rejected.
- **Four other claims from [#207](https://github.com/xerj-org/xerj/issues/207)
now match the source.**
diff --git a/landing/docs/vectors.html b/landing/docs/vectors.html
index bb0c4fb00..702ba5de2 100644
--- a/landing/docs/vectors.html
+++ b/landing/docs/vectors.html
@@ -157,6 +157,7 @@
<li><strong>binary</strong> — 1-bit per dimension. Not implemented: the server <strong>refuses to start</strong> on <code class="inline">default_quantization = "binary"</code> rather than silently storing full precision while claiming a 32× saving.</li>
</ul>
<p>A 4-bit quantizer exists in the vector crate but no config or mapping value reaches it, so <code class="inline">scalar4</code> is not a mode you can select in this build.</p>
+ <p><strong>Known gap — a mapping's <code class="inline">quantization</code> value is not validated</strong> (<a href="https://github.com/xerj-org/xerj/issues/275">#275</a>). Startup refuses an unimplemented <code class="inline">[vector] default_quantization</code>, but the per-field mapping key does not: <code class="inline">"quantization": "scalar4"</code> — or <code class="inline">"binary"</code>, or a typo such as <code class="inline">"sq8"</code> — is accepted with a <code class="inline">200</code>, echoed back verbatim by <code class="inline">GET /_mapping</code>, and then ignored, so the field is stored at full-precision f32 while the mapping you read back says otherwise. Measured on rc.14. Until it is a <code class="inline">400</code>, <code class="inline">"scalar8"</code> (or <code class="inline">"int8"</code>, or <code class="inline">index_options.type: int8_hnsw</code>/<code class="inline">int8_flat</code>) is the only value that changes anything, and reading the mapping back is not a way to confirm it took.</p>
<h2 id="knn-query">KNN query</h2>
<pre class="code">{