AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
knn: scalar8 codes written at ingest, slot-addressed (#392)
xerj-org46b58eef99fe
diff --git a/ROADMAP.md b/ROADMAP.mdindex e9861550f..e31f0e69d 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -106,7 +106,7 @@ The 1.0 bar: **every public claim verified against the release binary, and every**Zero-token** is the name for work XERJ does locally, inside the engine, so that a person or an agent spends no model tokens *finding*, *judging*, *watching for* or *handing over* an answer. Search already works that way — no model runs to answer a query. This section is the rest of that idea, as product work (long form: [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)), with each item's status checked against `main` on 2026-09-18 (`4d8dadbf`, 32 commits after `v1.0.0-rc.74`) by reading the code named beside it. "In flight" means a pushed branch exists and nothing of it is on `main`; "planned" means no code exists. Tracking issue: [#941](https://github.com/xerj-org/xerj/issues/941).-**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` now):+**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` as of v1.0.0-rc.75):- **Judged search — a rerank stage.** Retrieve with BM25 or `hybrid`, then have a second stage re-judge the top *N* before the page is returned. *On `main` since rc.75:* the opt-in `rerank` block on `_search`, inert until an operator configures a key; a LOCAL cross-encoder provider was measured and did NOT beat the shipped hybrid (SciFact 0.7021 hybrid vs 0.7186 local-base, interval spanning zero; NFCorpus 0.3445 vs 0.3597 local-small), so it stays opt-in and unmerged. *Before rc.75:* the ES `rescore` block (query rescorer) and the `hybrid` query type with `rrf|linear`; there is no rerank stage. *In flight:* branch `feat/rerank-stage` — a `rerank` block on `_search` that hands the top hits to an external relevance judge, opt-in per request and inert until an operator configures a key. It is the only *search-time* feature that would send document text off the machine — proxy embeddings (`[embedding] default_endpoint`) send it at write time and the WAL tap replays writes to an external `_bulk` endpoint, both operator-configured and off by default ([every outbound connection](./docs/RERANK.md#every-way-data-leaves-a-xerj-node)) — which is exactly why a *local* judge is on this list. *The bar for a local model is a measured one.* On the BEIR test splits with the opt-in neural embedder (`--embed-mode neural`, all-MiniLM-L6-v2, CPU) the shipped `hybrid` RRF query is already the best arm we have — nDCG@10 **0.699–0.704 on SciFact and 0.345 on NFCorpus** over three runs, against 0.657 / 0.302 for BM25 alone, 0.676 / 0.329 for vectors alone, and 0.686 / 0.332 for "BM25 top-30 re-ordered by the same bi-encoder", which is *worse* than hybrid on both. So re-ordering with the embedder we already ship is not worth building, and **a local cross-encoder ships only if it beats 0.699 / 0.345 on the same harness, by more than the run-to-run spread** ([`benchmarks/neural-path-triage/`](./benchmarks/neural-path-triage/)). Those figures are with the neural embedder; the default embedder is lexical feature hashing and was not part of this run. The hybrid figure is the only arm that does not reproduce — 0.6993, 0.7023 and 0.7044 on SciFact across three runs of unchanged indices — because tied RRF scores are ordered by a per-process hash seed ([#940](https://github.com/xerj-org/xerj/issues/940)); a gain inside that spread is not a gain.- **Share links and a guest reading room.** Hand one person a link to one corpus or one document, read-only, without creating them an account. *On `main` since rc.75:* `xerj share` with scoped, expiring links, a guest reading room, and a `--tunnel` mode that names Cloudflare as a reader of the traffic. *Before rc.75:* nothing — there was no share or guest code in the tree, and reading a corpus needs an account session or an API key. *In flight:* branches `feat/share-links` (scoped, expiring links; a share can never name a system index) and `feat/console-corpus-reader` (the corpus browser and the reader the link opens). Both branches carry commits their authors marked unverified; they are not claims yet.diff --git a/demo/playbooks/ES_COMPATIBILITY.md b/demo/playbooks/ES_COMPATIBILITY.mdindex 8e28f360e..8d2e08744 100644--- a/demo/playbooks/ES_COMPATIBILITY.md+++ b/demo/playbooks/ES_COMPATIBILITY.md@@ -384,6 +384,6 @@ The sections above predate the **rc-3** release and the **RC4** hardening pass.- **`percolate`** — a real reverse-search doc-scan (each indexed doc stores a query; the supplied `document`/`documents` are matched against it), replacing the former hard-coded `MatchNone`.- **`script_fields`** — real per-hit Painless evaluation with `index.max_script_fields` enforcement (was a silent `null`, fixed `65ce844`). Relatedly, **`scripted_metric`** now runs a real single-shard `init`/`map`/`combine`/`reduce` interpreter, and **`significant_terms`** computes real JLH significance scoring — both previously empty/`null`.- **`has_child` / `has_parent`** — now **fail loud (400)** at parse time instead of returning unfiltered flat hits.-- **HNSW-served kNN** — unfiltered, cosine, full-precision kNN on indexes ≥1,024 docs is now served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the bench query; filtered/nested/SQ8/small-index shapes keep the exact brute-force scan). **`scalar8` (`int8_hnsw`)** has a real serving path — scores are computed from 1-byte-per-dimension codes, not from the f32 vector (recall@10 ≈ 0.998, measured) — but it buys **precision, not memory**: the scan reads the full-precision vector from `_source` and quantizes it per query, so nothing gets smaller in RAM. The earlier "~4× smaller vector working set" claim described caches that [#371](https://github.com/xerj-org/xerj/issues/371) removed — both the per-document code map and the per-field codebook were write-once, and between them kept scoring updated documents from stale data. The ingest-time, ordinal-addressed code array that would make a memory claim true is tracked in [#392](https://github.com/xerj-org/xerj/issues/392), which also covers the two consequences of fitting the codebook per query: `scalar8` still disqualifies a field from HNSW-served kNN, and a `scalar8` `_score` varies with the candidate set (bounded by SQ8's own 1/255 quantization step) where Elasticsearch's, fixed per segment at index time, does not. `binary` (1-bit) is still rejected at startup.+- **HNSW-served kNN** — unfiltered, cosine, full-precision kNN on indexes ≥1,024 docs is now served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the bench query; filtered/nested/SQ8/small-index shapes keep the exact brute-force scan). **`scalar8` (`int8_hnsw`)** has a real serving path — scores are computed from 1-byte-per-dimension codes written at INGEST time into a flat, slot-addressed u8 array ([#392](https://github.com/xerj-org/xerj/issues/392); recall@10 ≈ 0.998, measured), so the kNN scoring working set is the codes (1 byte/dim) instead of the f32 `_source` vectors (4 bytes/dim) — the "~4× smaller vector working set" claim is true of that path now, observable as `primaries.sq8.fields.<f>.codes_bytes` in `_stats`. Total resident memory does not shrink 4× (`_source` keeps the f32 originals), and Elasticsearch still differs in mechanism (per-segment quantization at merge time, codes on disk). The codebook is fitted from the ingested vectors and widened with a re-encode when a value lands outside it — never clamped, which is the defect class [#371](https://github.com/xerj-org/xerj/issues/371) fixed by removing the write-once caches. A `scalar8` `_score` is now a function of index state alone and no longer varies with the candidate set; the one #392 consequence that remains open: `scalar8` still disqualifies a field from HNSW-served kNN. `binary` (1-bit) is still rejected at startup.Still **not** compatible (unchanged, single-node by design): true multi-node sharding/replication, CCR, and full ML. The float-to-integer truncation and scored-`hits.total` defects flagged in §1/§8 are tracked separately — this addendum does not claim them closed. See ROADMAP.md and STUB_AUDIT.md for the full tracked list.diff --git a/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md b/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.mdindex 7c7968eeb..7719adeff 100644--- a/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md+++ b/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md@@ -92,7 +92,7 @@ These are the strategic decisions for rc5, stated without spin.- `role_descriptors` accepted then ignored — every API key is effectively superuser.- ~~No `cargo-audit`/`cargo-deny` gate (528 unscanned dep crates) and no `cargo-fuzz` harnesses.~~**CLOSED 2026-08-09 (#207):** CI job `security-audit` runs `cargo audit` on every push and PR, and job `fuzz` builds and runs the libFuzzer harnesses in `engine/fuzz/` (ES DSL, `query_string`, date math, index-name date math, SQL, Painless, and the aggregation script tokenisers). `cargo-deny` is still absent.-- Binary/RaBitQ (BBQ) quantization unimplemented; `bbq_*`/`int4_*` mappings silently keep full f32 (no memory saving) — HNSW RAM wall NOT mitigated by SQ8 either — `scalar8` changes scoring precision, not resident memory (the scan quantizes from `_source` per query); see [#392](https://github.com/xerj-org/xerj/issues/392). No DiskANN.+- Binary/RaBitQ (BBQ) quantization unimplemented; `bbq_*`/`int4_*` mappings silently keep full f32 (no memory saving) — HNSW RAM wall NOT mitigated by SQ8 either: `scalar8` kNN scores from ingest-time 1-byte/dim codes ([#392](https://github.com/xerj-org/xerj/issues/392)), which shrinks the scoring working set ~4×, but `_source` keeps the f32 originals and the HNSW graph still holds f32. No DiskANN.**Marketing numbers to reconcile with measured reality (honesty-posture debt):**`<500MB RSS / 1M docs` (measured 596MB / 200k) · `~30MB Docker image` (~75–100MB) · `2–5× compression` (~1.6×) · `2 files/segment` (~4) · `TLS enabled out of the box` (off by default) · `OpenAPI generated from code` (static hand-maintained, 404 live) · `no deep-pagination cliff` (identical 10k cap) · `cost-based planner` (rule/structure-based) · error links use `xerj.io` vs canonical `xerj.org`.diff --git a/docs/examples/vector-quantization/quant_demo.py b/docs/examples/vector-quantization/quant_demo.pyindex b2dd31e3d..a348d08b3 100644--- a/docs/examples/vector-quantization/quant_demo.py+++ b/docs/examples/vector-quantization/quant_demo.py@@ -7,11 +7,13 @@ int8_hnsw`) makes the kNN *serving* path score against 1-byte-per-dimensioncodes instead of 4-byte floats, with almost no recall loss. `_source` stillreturns the original vectors.-NOTE: this changes PRECISION, not memory. XERJ reads the full-precision-vector from `_source` and quantizes it per query, so `scalar8` does not-shrink the resident vector working set today — see issue #392. The footprint-number printed below is the size of the int8 ENCODING, not a saving XERJ-currently realises.+Since issue #392 the codes are written at INGEST time into a flat,+slot-addressed u8 array and the kNN scan scores against them, so the+serving working set IS the 1-byte-per-dim codes rather than the f32+`_source` vectors. The run below measures the resident array straight off+the server (`GET /{index}/_stats` → `primaries.sq8.fields.v.codes_bytes`).+`_source` keeps the f32 originals, so this is the scoring working set that+shrinks 4x — not total process memory.This recipe embeds the 40 real KB articles (demo/data/ai_kb.ndjson) into128-dim vectors with a small deterministic feature-hasher (same idea as@@ -21,9 +23,9 @@ shows that:1. kNN returns the same top results from both,2. recall@10 of the quantized index vs the exact index stays >= 0.90,- 3. what the int8 encoding costs vs float32 — MEASURED by actually- encoding every vector both ways and comparing the real byte totals- (not a hardcoded ratio). See the NOTE above on what this is not.+ 3. the resident SQ8 code array the server actually holds — measured+ from `_stats` (1 byte/dim/live doc), alongside the encoding-size+ comparison measured by encoding every vector both ways client-side.Usage:xerj --insecure --data-dir ./data & # start XERJ@@ -178,11 +180,17 @@ def main():total += len(exact)recall = hits / total if total else 0.0- # ── 5. Measure the real byte size of each ENCODING. ──────────────────+ # ── 5. Measure the RESIDENT codes the server holds (#392). ──────────+ # The ingest-time, slot-addressed SQ8 array is server state — read it+ # back from _stats rather than stipulating its size.+ stats = call("GET", f"/{SQ8_INDEX}/_stats")+ field_stats = stats["indices"][SQ8_INDEX]["primaries"]["sq8"]["fields"]["v"]+ codes_bytes = field_stats["codes_bytes"]+ live = field_stats["live"]++ # ── 6. Measure the real byte size of each ENCODING (client side). ────# Encode every corpus vector both ways and compare the actual byte totals- # — a genuine measurement, not a stipulated 4x. It is a measurement of the- # encodings, though, NOT of XERJ's resident footprint: the serving path- # reads the f32 vector from `_source` and quantizes per query (issue #392).+ # — a genuine measurement, not a stipulated 4x.f32_total = sum(len(encode_f32(d["v"])) for d in docs)i8_total = sum(len(encode_i8(d["v"])) for d in docs)ratio = f32_total / i8_total if i8_total else 0.0@@ -190,6 +198,11 @@ def main():i8_per = i8_total // len(docs)print(f"recall@10 (scalar8 vs float32 ground truth): {recall:.3f}")+ print(+ f"resident SQ8 codes (from _stats, {live} live docs): "+ f"{codes_bytes} B ({codes_bytes // max(live, 1)} B/vec) — "+ f"serving={field_stats['serving']}"+ )print(f"encoding size over {len(docs)} vecs: "f"float32 = {f32_total} B ({f32_per} B/vec) → "@@ -197,8 +210,14 @@ def main():)if recall < 0.90:raise SystemExit(f"FAIL: recall {recall:.3f} < 0.90")- print("\nOK — recall preserved through 1-byte-per-dim codes. `_source` still holds")- print("the originals. scalar8 changes precision, not resident memory (issue #392).")+ if not field_stats["serving"] or not field_stats["ready"]:+ raise SystemExit("FAIL: SQ8 code store not serving after ingest (#392)")+ if codes_bytes != len(docs) * DIM:+ raise SystemExit(+ f"FAIL: codes_bytes {codes_bytes} != {len(docs)} docs x {DIM} dim"+ )+ print("\nOK — recall preserved through 1-byte-per-dim codes, served from the")+ print("ingest-time slot-addressed array (#392). `_source` still holds the originals.")if __name__ == "__main__":diff --git a/docs/recipes/README.md b/docs/recipes/README.mdindex c9f926134..38a5392fe 100644--- a/docs/recipes/README.md+++ b/docs/recipes/README.md@@ -16,7 +16,7 @@ These are practical "how do I actually do X" recipes, not an API reference (that| [Passage retrieval on long docs](./passage-retrieval.md) | `semantic_text` auto-embeds every overlapping passage; a long doc competes on any one of its sections via best-passage (max-sim) scoring — 98% top-3 vs 32% pooled | [`passage_demo.py`](../examples/passage-retrieval/passage_demo.py) || [Log analytics](./log-analytics.md) | From raw logs to dashboards — error rates, p95 latency, top services via aggregations | [`log_analytics.py`](../examples/log-analytics/log_analytics.py) || [Vector search (kNN)](./vector-search-knn.md) | Nearest-neighbor similarity search over `dense_vector` (HNSW), with filters | [`knn_demo.py`](../examples/vector-search-knn/knn_demo.py) |-| [Vector quantization](./vector-quantization.md) | Opt a `dense_vector` field into scalar8 (`int8_hnsw`) — kNN scored from 1-byte-per-dim codes at recall@10 ≈ 0.998 (precision, **not** memory: see [#392](https://github.com/xerj-org/xerj/issues/392)) | [`quant_demo.py`](../examples/vector-quantization/quant_demo.py) |+| [Vector quantization](./vector-quantization.md) | Opt a `dense_vector` field into scalar8 (`int8_hnsw`) — kNN scored from ingest-time 1-byte-per-dim codes at recall@10 ≈ 0.998; the scoring working set is 4× smaller than the f32 vectors (see [#392](https://github.com/xerj-org/xerj/issues/392)) | [`quant_demo.py`](../examples/vector-quantization/quant_demo.py) || [Hybrid search](./hybrid-search.md) | Keyword + vector in one query — results neither BM25 nor kNN finds alone | [`hybrid_search.py`](../examples/hybrid-search/hybrid_search.py) || [Anomaly detection](./anomaly-detection.md) | Statistical `_ml` detectors that flag spikes in metrics/logs | [`anomaly_detection.py`](../examples/anomaly-detection/anomaly_detection.py) || [Continuous anomaly datafeeds](./continuous-anomaly-datafeeds.md) | A live `_ml` datafeed that re-scores an index on a timer and stores new anomaly records you poll | [`datafeed_demo.py`](../examples/continuous-anomaly-datafeeds/datafeed_demo.py) |diff --git a/docs/recipes/vector-quantization.md b/docs/recipes/vector-quantization.mdindex 409ee4005..1c1553b20 100644--- a/docs/recipes/vector-quantization.md+++ b/docs/recipes/vector-quantization.md@@ -1,13 +1,15 @@# Vector quantization: int8 scoring at recall@10 ≈ 0.998-> **Read this first.** `scalar8` on XERJ today changes **precision, not-> memory**. Scores are computed from 1-byte-per-dimension codes, so the field-> has the recall profile of int8 — but the serving path still reads the-> full-precision vector out of `_source` and quantizes it per query, so-> nothing gets smaller in RAM. The ingest-time code array that would make it-> a memory win is tracked in-> [#392](https://github.com/xerj-org/xerj/issues/392). If you came here for a-> smaller working set, that issue is the one to watch, not this recipe.+> **Read this first.** `scalar8` scores come from 1-byte-per-dimension+> codes written at INGEST time into a flat, slot-addressed u8 array+> ([#392](https://github.com/xerj-org/xerj/issues/392)) — the kNN serving+> working set is those codes (1 byte/dim), not the f32 `_source` vectors+> (4 bytes/dim), and the score a document gets is a function of the index+> state alone, not of the filter or candidate set of the query that found it.+> `_source` keeps the original f32 vectors, so this shrinks the scoring+> working set ~4×, not total process memory. When the code store cannot+> serve (right after open, mid-rebuild, or after a wrong-dimension write+> breaks coverage), queries fall back to the exact `_source` scan.## The problem@@ -34,10 +36,13 @@ On a real 128-dim corpus the recall cost is negligible: **recall@10 = 0.998**against the exact float32 index. That number is computed by the run below,not stipulated.-What that costs in bytes *as an encoding* is 128 rather than 512 per vector,-and the run measures that too — but see the note at the top: XERJ does not-hold those codes resident today, so treat it as the size of the encoding, not-as a saving you get.+The codes are 128 bytes rather than 512 per vector, and since #392 they are+held resident by the engine: the run reads the live array's byte size+straight off the server (`GET /{index}/_stats` →+`primaries.sq8.fields.<field>.codes_bytes`) and checks it is exactly+`docs × dim`. The kNN scan scores against that array without touching the f32+vectors — the ~4× smaller vector working set is a property of the serving+path now, measured, not a ratio the encoding merely implies.## The solution@@ -95,22 +100,23 @@ query: 'how do I stop an agent's context window from overflowing?'── scalar8 (quantized)0.67938 Long-context windows do not replace memory- 0.60021 p95 latency budgets for interactive RAG agents- 0.59731 SOC 2 controls that apply to vector workloads+ 0.59983 p95 latency budgets for interactive RAG agents+ 0.59766 SOC 2 controls that apply to vector workloadsrecall@10 (scalar8 vs float32 ground truth): 0.998+resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=Trueencoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)-OK — recall preserved through 1-byte-per-dim codes. `_source` still holds-the originals. scalar8 changes precision, not resident memory (issue #392).+OK — recall preserved through 1-byte-per-dim codes, served from the+ingest-time slot-addressed array (#392). `_source` still holds the originals.```-The encoding-size line is a real measurement: the run encodes every corpus-vector as float32 bytes (`struct`) and as int8 codes and compares the actual-byte totals — 20480 B vs 5120 B, exactly 4.00×. It is the cost of the two-*encodings*, measured in the client. It is **not** a measurement of XERJ's-resident footprint, and XERJ does not currently realise it as one-([#392](https://github.com/xerj-org/xerj/issues/392)).+Both size lines are real measurements. The client-side one encodes every+corpus vector as float32 bytes (`struct`) and as int8 codes and compares the+byte totals — 20480 B vs 5120 B, exactly 4.00×. The `resident SQ8 codes` line+is read off the server (`_stats` → `primaries.sq8.fields.v.codes_bytes`) and+matches it byte-for-byte: 40 docs × 128 dim = 5120 B, `serving=True` — the+array the kNN scan actually scores against ([#392](https://github.com/xerj-org/xerj/issues/392)).## Reproduce it yourself@@ -127,6 +133,7 @@ overrides the KB path (default: auto-discovered `demo/data/ai_kb.ndjson`).The embedder and corpus are deterministic, so a customer should see exactly:- `recall@10 (scalar8 vs float32 ground truth): 0.998`+- `resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=True`- `encoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)`These numbers are stable run-to-run (verified across repeated runs — no@@ -142,41 +149,41 @@ variance); the printed kNN scores are likewise identical each run.is rejected at startup rather than silently storing full precision.- **Cosine is normalised** before quantizing for the tightest code range;`dot_product` and `l2_norm` similarities are supported too.-- **No memory saving yet.** The serving path reads the full-precision vector- from `_source` and quantizes it per query, so `scalar8` costs the same RAM- as float32 and a little more CPU. Tracked in+- **The serving working set shrinks ~4×; total memory does not.** The kNN+ scan scores against the ingest-time code array (1 byte/dim) instead of the+ f32 `_source` vectors (4 bytes/dim), but `_source` keeps the originals for+ retrieval, so the process's total footprint does not fall 4×. The array+ itself is observable: `GET /{index}/_stats` → `primaries.sq8` (per field:+ `dim`, `live`, `expected`, `covered`, `ready`, `serving`, `refits`,+ `codes_bytes`).+- **Codes are written at ingest and rewritten on update.** One slot per+ document, addressed `slot * dim`, tombstoned on delete — the same slot+ discipline the HNSW slab uses. A vector that lands outside the fitted+ per-dimension range WIDENS the codebook and re-encodes every live code+ (decode → encode, each value moves by at most the new quantization step);+ nothing is ever clamped, which is the defect class+ [#371](https://github.com/xerj-org/xerj/issues/371) fixed by deleting the+ old write-once caches.+- **`_score` no longer depends on the filter.** The codebook is fitted from+ the ingested vectors and lives with the data, so the same document carries+ the same `_score` whether or not a filter removed other documents — the+ property Elasticsearch has and per-query fitting lacked. Pinned over HTTP in+ `engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs`+ (drift is exactly 0, not merely small).+- **Fallback windows keep the exact scan.** Until the open-time walk converges+ (a restart re-derives the store in the background — WAL replay never+ re-runs vector indexing), while a publication race is possible, or after a+ wrong-dimension write breaks the store's coverage gate, queries are served+ by the exact `_source` scan. In that fallback the SQ8 branch still fits its+ codebook per query over the candidates being scored, so scores there move+ by up to SQ8's own quantization step (1/255 of the fitted per-dimension+ range) when the candidate set changes — measured on the pre-#392 path:+ max |Δ_score| 1.976e-05 with order swaps at 19/30 positions when an+ unrelated-only filter was added. That fallback is correct, just not+ filter-independent; `primaries.sq8.fields.<f>.serving` tells you which+ path you are on.+- **`scalar8` still disables ANN.** A quantized field remains excluded from+ HNSW-served kNN (the graph scores f32) and takes the codes scan above for+ unfiltered queries or the exact brute-force scan otherwise. Teaching the+ graph to traverse against the codes is the remaining open half of[#392](https://github.com/xerj-org/xerj/issues/392).-- **`scalar8` disables ANN.** A quantized field is excluded from HNSW-served- kNN and always takes the exact brute-force scan, so it is slower, not- faster, than leaving the field full-precision. Also [#392](https://github.com/xerj-org/xerj/issues/392).-- **A `scalar8` `_score` — and the order it produces — depends on the- candidate set.** The codebook is fitted per query over the candidates being- scored, which is what keeps an updated document from being scored on a stale- codebook ([#371](https://github.com/xerj-org/xerj/issues/371)). Three things- follow, all of them measured, none of them true of a full-precision field:-- - **A `filter` reorders the documents it does not remove.** On a 60-document- 4-dim cosine corpus (30 tightly clustered `grp:a` documents, 30 spread- `grp:b` ones), adding `filter: {"term": {"grp": "a"}}` — which removes only- `grp:b` — returned the same 30 documents with a maximum `_score` difference- of **1.976e-05**, but in a **different order at 19 of the 30 positions**.- The score delta is tiny; the reordering is what a caller actually sees.- - **The trigger is the candidate set, not the `filter` keyword.** Indexing- one more unrelated document moves existing documents too. Same corpus, one- document added far from the query: maximum `_score` difference **7.100e-06**- over the 30 pre-existing documents, and a reordered top 10.- - **Unfiltered corpora over 1000 documents also score differently from- v1.0.0-rc.17**, which fitted the codebook from the first ≤1000 candidates- and then cached it for the life of the process. On 1500 documents, **all 40- of the top-40 `_score` values changed** (maximum difference **4.880e-05**)- and two adjacent ranks swapped. Below 1000 documents, unfiltered and never- updated, scores are byte-identical to rc.17.-- Every individual difference is bounded by SQ8's own quantization step (1/255- of the fitted per-dimension range), i.e. inside the approximation error- `scalar8` already carries — but documents whose true scores are close will- swap places, so treat `scalar8` ordering as stable only for a fixed corpus- and a fixed filter. Elasticsearch fits its codebook per segment at index time- and does not have this property;- [#392](https://github.com/xerj-org/xerj/issues/392) is what would bring XERJ- in line.diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rsindex 712da8dc3..680cdc96e 100644--- a/engine/crates/xerj-api/src/es_compat.rs+++ b/engine/crates/xerj-api/src/es_compat.rs@@ -20297,6 +20297,16 @@ pub async fn index_stats(Ok(idx) => idx.hnsw_stats().await,Err(_) => json!({ "present": false }),};+ // XERJ extension (issue #392): the ingest-time SQ8 code stores. `fields`+ // is empty for indices with no `scalar8` mapping; per field it reports+ // the dimension, live slot count, coverage denominator, the+ // ready/serving gates, codebook refits, and the resident `codes_bytes`+ // — the flat u8 array the slot addressing is for, observable instead+ // of notional.+ let sq8_stats = match state.engine.get_index(&index) {+ Ok(idx) => json!({ "fields": idx.sq8_stats() }),+ Err(_) => json!({ "fields": {} }),+ };// Merge status + real segment count so benchmarks can verify the// index is merge-quiescent (merges.current == 0) before measuring.let (seg_count, merges_current) = state@@ -20326,6 +20336,7 @@ pub async fn index_stats("store": { "size_in_bytes": store_size_bytes },"dense_vector": dense_vector_stats,"hnsw": hnsw_stats,+ "sq8": sq8_stats,"mappings": { "schema_persist_failures": schema_persist_failures },"indexing": {// Real cumulative indexing counters (RC4 W4 item 1): plumbed fromdiff --git a/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs b/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rsnew file mode 100644index 000000000..e89110c92--- /dev/null+++ b/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs@@ -0,0 +1,378 @@+//! Issue #392 — `scalar8` codes must be written at INGEST time, addressed by+//! a dense slot, and scored against a codebook that is a property of the+//! index rather than of the query's candidate set.+//!+//! Two observable consequences, each pinned here from the outside over real+//! HTTP:+//!+//! * `sq8_codes_exist_at_ingest_before_any_query` — after indexing (and+//! before a single kNN), `GET /{index}/_stats` reports a live, ready SQ8+//! code store for the field. Before #392 nothing quantized anything at+//! ingest: the serving path read each candidate's f32 vector out of+//! `_source` and quantized it per query, so no such store existed and the+//! stats section is absent (null).+//!+//! * `scores_do_not_depend_on_the_filter` — the same document must carry the+//! same `_score` under two queries whose only difference is a filter that+//! removes OTHER documents. Before #392 the SQ8 codebook was fitted per+//! query over the candidates being scored, so shrinking the candidate set+//! moved every surviving document's score (measured on the unfixed code:+//! max |Δ_score| 1.976e-05 with order swaps at 19/30 positions — inside+//! SQ8's advertised approximation error, but a real behavioural difference+//! from Elasticsearch, where the codebook is fixed at index time).+//!+//! * `codes_survive_a_restart` — the code store is re-derived from the live+//! documents when the index reopens (WAL replay never re-runs vector+//! indexing), and once the rebuild lands the same query returns the same+//! neighbours. A regression pin for the rebuild, not a fail-before: the+//! per-query-fit path that preceded it also survives a restart.+//!+//! * `a_wrong_dimension_document_does_not_break_the_field` — a document+//! whose vector has the wrong dimensionality breaks the store's coverage+//! invariant; queries must then fall back to the exact path and still+//! return correct results (the malformed document is skipped, exactly as+//! the brute-force scan skips it).+//!+//! Elasticsearch is referenced for wire semantics only. It is AGPL-3.0/+//! SSPL-1.0/Elastic-2.0 licensed and no code from it is reproduced here.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++const DIM: usize = 8;+const INDEX: &str = "sq8";++/// 25 corpus documents whose every dimension lies in `[0, 1)` — the documents+/// both queries score. Deterministic prime-stride walk, all values distinct+/// and not representable on both SQ8 grids (that is what makes the unfixed+/// code's score drift observable rather than coincidentally zero).+const KEEP: usize = 25;++/// 5 documents carrying `tag: "wide"` whose dimension 0 sits far outside the+/// keep documents' range. They widen the per-query-fitted codebook only when+/// present, which is exactly the candidate-set dependency under test.+const WIDE_IDS: [usize; 5] = [100, 101, 102, 103, 104];++fn keep_vector(i: usize) -> Vec<f32> {+ (0..DIM)+ .map(|d| ((i * 37 + d * 101) % 199) as f32 / 199.0)+ .collect()+}++fn wide_vector(i: usize) -> Vec<f32> {+ let mut v = vec![0.0f32; DIM];+ v[0] = 5.0 + i as f32;+ for d in 1..DIM {+ v[d] = ((i * 53 + d * 97) % 199) as f32 / 199.0;+ }+ v+}++fn doc(i: usize) -> Value {+ if WIDE_IDS.contains(&i) {+ json!({ "tag": "wide", "v": wide_vector(i) })+ } else {+ json!({ "tag": "keep", "v": keep_vector(i) })+ }+}++async fn app() -> (axum::Router, tempfile::TempDir) {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config, engine, metrics);+ (xerj_api::router::build_es_compat_router(state), dir)+}++async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {+ let response = app.clone().oneshot(req).await.expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);+ (status, value)+}++async fn json_req(+ app: &axum::Router,+ method: &str,+ path: &str,+ body: Value,+) -> (StatusCode, Value) {+ send(+ app,+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+}++async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {+ send(+ app,+ Request::get(path).body(Body::empty()).expect("request"),+ )+ .await+}++async fn create_and_index(app: &axum::Router) {+ let (status, body) = json_req(+ app,+ "PUT",+ &format!("/{INDEX}"),+ json!({+ "mappings": { "properties": {+ "v": { "type": "dense_vector", "dims": DIM, "quantization": "scalar8" }+ } }+ }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "create: {body}");+ for i in 0..KEEP {+ let (status, body) = json_req(app, "PUT", &format!("/{INDEX}/_doc/{i}"), doc(i)).await;+ assert!(status.is_success(), "index {i}: {status} {body}");+ }+ for &i in &WIDE_IDS {+ let (status, body) = json_req(app, "PUT", &format!("/{INDEX}/_doc/{i}"), doc(i)).await;+ assert!(status.is_success(), "index {i}: {status} {body}");+ }+ let (status, body) = json_req(app, "POST", &format!("/{INDEX}/_refresh"), json!({})).await;+ assert!(status.is_success(), "refresh: {status} {body}");+}++/// The query vector: keep document 0's own vector. All keep documents score+/// distinctly against it (deterministic corpus), so score identity between+/// the two queries is not satisfiable by accident.+fn query_vector() -> Vec<f32> {+ keep_vector(0)+}++/// Run the kNN, returning `(id, _score)` per hit.+async fn knn(app: &axum::Router, filter: Option<Value>) -> Vec<(String, f64)> {+ let mut knn_body = json!({+ "field": "v",+ "query_vector": query_vector(),+ "k": KEEP + WIDE_IDS.len(),+ "num_candidates": 100+ });+ if let Some(f) = filter {+ knn_body["filter"] = f;+ }+ let (status, body) = json_req(+ app,+ "POST",+ &format!("/{INDEX}/_search"),+ json!({ "knn": knn_body, "size": KEEP + WIDE_IDS.len(), "_source": false }),+ )+ .await;+ assert_eq!(status, StatusCode::OK, "knn: {status} {body}");+ body["hits"]["hits"]+ .as_array()+ .unwrap_or_else(|| panic!("no hits: {body}"))+ .iter()+ .map(|h| {+ (+ h["_id"].as_str().expect("_id").to_string(),+ h["_score"].as_f64().expect("_score"),+ )+ })+ .collect()+}++// ─────────────────────────────────────────────────────────────────────────────+// 1. Codes exist at ingest — before any query is served+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn sq8_codes_exist_at_ingest_before_any_query() {+ let (app, _dir) = app().await;+ create_and_index(&app).await;++ // No kNN has run. If SQ8 codes are written at ingest, the per-index stats+ // section reports them; on the per-query-fit path this whole section is+ // absent because nothing exists to report.+ let (status, body) = get(&app, &format!("/{INDEX}/_stats")).await;+ assert_eq!(status, StatusCode::OK, "stats: {status} {body}");+ let field = &body["indices"][INDEX]["primaries"]["sq8"]["fields"]["v"];+ assert!(+ field.is_object(),+ "no ingest-time SQ8 code store in stats (issue #392): {body}"+ );+ assert_eq!(field["dim"], json!(DIM), "{body}");+ assert_eq!(+ field["live"],+ json!(KEEP + WIDE_IDS.len()),+ "every indexed document must hold a live code slot: {body}"+ );+ assert_eq!(+ field["expected"],+ json!(KEEP + WIDE_IDS.len()),+ "coverage numerator and denominator must agree: {body}"+ );+ assert_eq!(field["ready"], json!(true), "{body}");+ // 1 byte per dimension per live document — the resident array itself.+ assert_eq!(+ field["codes_bytes"],+ json!((KEEP + WIDE_IDS.len()) * DIM),+ "{body}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// 2. A document's _score does not depend on the filter+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn scores_do_not_depend_on_the_filter() {+ let (app, _dir) = app().await;+ create_and_index(&app).await;++ let unfiltered = knn(&app, None).await;+ let filtered = knn(&app, Some(json!({ "term": { "tag": "keep" } }))).await;++ assert_eq!(unfiltered.len(), KEEP + WIDE_IDS.len());+ assert_eq!(filtered.len(), KEEP, "filter must remove the wide docs");++ let score_of = |hits: &[(String, f64)], id: &str| {+ hits.iter()+ .find(|(hit_id, _)| hit_id == id)+ .map(|(_, s)| *s)+ .unwrap_or_else(|| panic!("{id} missing"))+ };++ let mut drift: f64 = 0.0;+ for i in 0..KEEP {+ let id = i.to_string();+ let a = score_of(&unfiltered, &id);+ let b = score_of(&filtered, &id);+ drift = drift.max((a - b).abs());+ assert_eq!(+ a, b,+ "doc {id} scores {a} unfiltered but {b} filtered — the SQ8 \+ codebook is being fitted per query over the candidate set, so \+ _score depends on the filter (issue #392)"+ );+ }+ // Belt: on the unfixed path the two fits differ by the wide docs' whole+ // range on dimension 0 (≥ 5.0 over 255 levels ⇒ Δ ≫ 1e-3), so a partial+ // regression still trips the assert above long before this line.+ assert!(drift == 0.0, "drift must be exactly zero, got {drift}");+}++// ─────────────────────────────────────────────────────────────────────────────+// 3. The code store is re-derived when the index reopens+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn codes_survive_a_restart() {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;++ let before = {+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config.clone(), engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state.clone());+ create_and_index(&app).await;+ let hits = knn(&app, None).await;+ state.engine.flush_all_force().await;+ hits+ };++ let (app, _keep) = {+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = xerj_api::state::AppState::new(config.clone(), engine, metrics);+ let app = xerj_api::router::build_es_compat_router(state.clone());+ // The open-time rebuild is background work; wait for it so the+ // assertion observes the codes path, not a mid-rebuild fallback.+ if let Ok(idx) = state.engine.get_index(INDEX) {+ idx.await_sq8_rebuilds().await;+ }+ (app, ())+ };++ let after = knn(&app, None).await;++ // Same neighbours in the same order: quantizing the same live vectors+ // under a codebook spanning the same per-dimension ranges reproduces the+ // ranking. Scores may shift by at most the SQ8 quantization step (the+ // re-fit walk re-encodes from the true f32 sources, while the live path+ // may have re-encoded through a decode), so ids/order are asserted+ // strictly and scores within a generous step-sized tolerance.+ assert_eq!(+ before.len(),+ after.len(),+ "restart must not lose documents: {before:?} vs {after:?}"+ );+ let order_before: Vec<&str> = before.iter().map(|(id, _)| id.as_str()).collect();+ let order_after: Vec<&str> = after.iter().map(|(id, _)| id.as_str()).collect();+ assert_eq!(+ order_before, order_after,+ "ranking changed across restart: {before:?} vs {after:?}"+ );+ for ((_, a), (_, b)) in before.iter().zip(after.iter()) {+ assert!(+ (a - b).abs() < 1e-2,+ "score for the same document moved from {a} to {b} across restart"+ );+ }++ // And the re-derived store is again serving-shape.+ let (status, body) = get(&app, &format!("/{INDEX}/_stats")).await;+ assert_eq!(status, StatusCode::OK);+ let field = &body["indices"][INDEX]["primaries"]["sq8"]["fields"]["v"];+ assert_eq!(field["ready"], json!(true), "{body}");+ assert_eq!(field["live"], json!(KEEP + WIDE_IDS.len()), "{body}");+}++// ─────────────────────────────────────────────────────────────────────────────+// 4. Coverage broken → exact fallback, correct results+// ─────────────────────────────────────────────────────────────────────────────++#[tokio::test]+async fn a_wrong_dimension_document_does_not_break_the_field() {+ let (app, _dir) = app().await;+ create_and_index(&app).await;++ // A document whose vector has the wrong dimensionality. The brute-force+ // scan has always skipped it (`doc_vec.len() != dim`); the code store+ // must not serve around it either — coverage breaks and queries fall+ // back to the exact path rather than returning wrong results.+ let (status, body) = json_req(+ &app,+ "PUT",+ &format!("/{INDEX}/_doc/broken"),+ json!({ "tag": "keep", "v": vec![0.5f32; DIM + 3] }),+ )+ .await;+ assert!(status.is_success(), "index broken: {status} {body}");+ let (status, body) = json_req(&app, "POST", &format!("/{INDEX}/_refresh"), json!({})).await;+ assert!(status.is_success(), "{status} {body}");++ let hits = knn(&app, Some(json!({ "term": { "tag": "keep" } }))).await;+ assert_eq!(+ hits.len(),+ KEEP,+ "the malformed doc must be skipped: {hits:?}"+ );+ assert!(+ !hits.iter().any(|(id, _)| id == "broken"),+ "malformed document must not appear in hits: {hits:?}"+ );+ // The control: doc 0 is the query vector and must still top the list.+ assert_eq!(hits[0].0, "0", "query-vector doc must rank first: {hits:?}");+ assert!(hits[0].1 > 0.99, "expected ~1.0, got {}", hits[0].1);+}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex fdb3ed173..c50f85aa7 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -303,16 +303,15 @@ impl Config {}// Vector: `scalar8` (SQ8) quantization is wired into the kNN serving- // path (`Index::run_knn_brute_force`): a `scalar8` dense_vector field- // scores candidates from 1-byte-per-dimension codes rather than their- // f32 values, so it has the recall profile of SQ8. It does not reduce- // resident memory today — the scan quantizes each candidate's current- // vector per query rather than caching codes, which is what keeps an- // updated document from being scored on a stale code (#371).- // `none` and `scalar8` are therefore- // accepted. `binary` (1-bit) has no implemented quantizer, so honouring- // it would silently store full-precision vectors while claiming a 32×- // saving — it stays rejected until a BinaryQuantizer lands.+ // path (`Index::run_knn_brute_force` / `run_knn_sq8_codes_scan`): a+ // `scalar8` dense_vector field scores candidates from+ // 1-byte-per-dimension codes written at ingest into a slot-addressed+ // store (#392), so it has the recall profile of SQ8 and the kNN+ // scoring working set is the codes rather than the f32 `_source`+ // vectors. `none` and `scalar8` are therefore accepted. `binary`+ // (1-bit) has no implemented quantizer, so honouring it would silently+ // store full-precision vectors while claiming a 32× saving — it stays+ // rejected until a BinaryQuantizer lands.if self.vector.default_quantization == VectorQuantization::Binary {return Err(XerjError::config("vector.default_quantization: binary (1-bit) quantization is not implemented in \@@ -1191,17 +1190,18 @@ pub struct VectorConfig {/// **not implemented in this build** and is rejected at startup.////// - `"none"` — full float32 vectors (highest accuracy, most memory).- /// - `"scalar8"` — 8-bit scalar quantization — WIRED into the kNN serving- /// path, but it buys **precision, not memory**. A `scalar8` dense_vector- /// field scores candidates from 1-byte-per-dimension codes, so it has the- /// recall profile of int8 (measured recall@10 ≈ 0.998). It does NOT- /// reduce resident memory: the scan reads the full-precision vector from- /// `_source` and quantizes it per query, which is what keeps an updated- /// document from being scored on stale codes or a stale codebook (#371).- /// The ingest-time code array that would make a memory claim true is- /// tracked in #392. Typically opted into per field via- /// `index_options.type: int8_hnsw` on the mapping; this global default- /// applies the same scheme index-wide.+ /// - `"scalar8"` — 8-bit scalar quantization, WIRED into the kNN serving+ /// path. A `scalar8` dense_vector field is scored from+ /// 1-byte-per-dimension SQ8 codes written at INGEST time into a flat,+ /// slot-addressed u8 array (#392), so it has the recall profile of int8+ /// and the kNN scoring working set is the codes (1 byte/dim) instead of+ /// the f32 `_source` vectors (4 bytes/dim). `_source` keeps the f32+ /// originals, so this shrinks the scoring working set, not total resident+ /// memory. When the code store cannot serve (open-time walk not yet+ /// converged, a publication race, coverage broken by a wrong-dimension+ /// write), queries fall back to the exact `_source` scan. Typically+ /// opted into per field via `index_options.type: int8_hnsw` on the+ /// mapping; this global default applies the same scheme index-wide./// - `"binary"` — 1-bit binary quantization (~32× memory reduction) — NOT/// YET IMPLEMENTED (no `BinaryQuantizer` exists).///diff --git a/engine/crates/xerj-common/src/types.rs b/engine/crates/xerj-common/src/types.rsindex 8f731aaf0..41a210155 100644--- a/engine/crates/xerj-common/src/types.rs+++ b/engine/crates/xerj-common/src/types.rs@@ -317,14 +317,21 @@ pub struct FieldOptions {pub similarity: Option<String>,/// Vector quantization scheme for this dense_vector field.///- /// `Some("scalar8")` makes the brute-force kNN scan score this field from- /// 1-byte-per-dimension codes, quantized per query from the document's- /// current vector — precision, not memory: nothing smaller is held- /// resident, and no SQ8 state outlives the query (#371; #392 tracks the- /// ingest-time code array that would make it a memory win). `None`/absent- /// keeps the exact full-precision f32 brute-force path. Set from the- /// mapping's `index_options.type` (`int8_hnsw`/`int8_flat` → `scalar8`) in- /// es_compat.+ /// `Some("scalar8")` makes the kNN paths score this field from+ /// 1-byte-per-dimension SQ8 codes written at INGEST time into a flat,+ /// slot-addressed u8 array (`Sq8CodeStore`, #392) — one slot per document,+ /// rewritten on update, codebook fitted from the ingested vectors (widened+ /// and re-encoded when a value lands outside it, never clamped). The kNN+ /// scoring working set is those codes (1 byte/dim) instead of the f32+ /// `_source` vectors (4 bytes/dim), and a document's `_score` is a function+ /// of the index state alone, not of the query's candidate set. `_source`+ /// keeps the original f32 vectors, so total resident memory does not shrink+ /// 4× — the vector working set the scoring path touches does. When the+ /// store cannot serve (open-time walk not yet converged, a publication+ /// race, coverage broken by a wrong-dimension write), queries fall back to+ /// the exact `_source` scan. `None`/absent keeps the exact full-precision+ /// f32 brute-force path. Set from the mapping's `index_options.type`+ /// (`int8_hnsw`/`int8_flat` → `scalar8`) in es_compat.#[serde(default, skip_serializing_if = "Option::is_none")]pub quantization: Option<String>,/// Null value to substitute when the field is missing from a document.diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rsindex 0910053a2..758f1d960 100644--- a/engine/crates/xerj-engine/src/index.rs+++ b/engine/crates/xerj-engine/src/index.rs@@ -34,7 +34,7 @@ use xerj_storage::segment::SectionType;use xerj_storage::wal::{SyncMode, WalEntry};use xerj_vector::distance::DistanceMetric;use xerj_vector::hnsw::{HnswIndex, HnswParams};-use xerj_vector::Sq8Params;+use xerj_vector::{Sq8CodeStore, Sq8Params};use crate::aggs::run_aggs_with_all;use crate::segment_cache_budget::{@@ -7855,6 +7855,10 @@ pub struct Index {/// `abort_background_tasks` so an unfinished rebuild can't hold the/// runtime (or the index Arc) alive across shutdown.hnsw_rebuild_task: Arc<parking_lot::Mutex<Option<tokio::task::JoinHandle<()>>>>,+ /// Per-field ingest-time SQ8 code stores (#392): field name → handle.+ /// Empty unless the mapping declares `quantization: "scalar8"` — every+ /// other index pays one empty-map read per write.+ sq8_stores: Arc<parking_lot::RwLock<HashMap<String, Arc<Sq8StoreHandle>>>>,// ── Per-index concurrency control ─────────────────────────────────────────/// Semaphore that limits the number of queries executing concurrently/// against this index. The default is 64 permits, matching the global@@ -8497,6 +8501,9 @@ impl Index {let segment_hydration_budget = index_segment_hydration_budget(config);let passage_chunk_fields_init = passage_chunk_fields_from_schema(&managed.schema);let date_field_scales_snapshot = date_field_scales(&managed.schema);+ // #392: kept back from the struct literal so the SQ8 handle setup at+ // the end of create can read the mapping without re-locking.+ let managed_schema_for_sq8 = managed.schema.clone();let index = Arc::new(Self {name,schema: Arc::new(RwLock::new(managed)),@@ -8588,6 +8595,7 @@ impl Index {hnsw_stale: Arc::new(std::sync::atomic::AtomicBool::new(false)),hnsw_rebuilding: Arc::new(std::sync::atomic::AtomicBool::new(false)),hnsw_rebuild_task: Arc::new(parking_lot::Mutex::new(None)),+ sq8_stores: Arc::new(parking_lot::RwLock::new(HashMap::new())),metric_query_count: Arc::new(AtomicU64::new(0)),metric_query_total_ms: Arc::new(AtomicU64::new(0)),metric_index_count: Arc::new(AtomicU64::new(0)),@@ -8656,6 +8664,12 @@ impl Index {idx.on_segments_changed();}})));+ // #392: SQ8 code stores for scalar8 fields. A fresh index carries+ // no documents, so the walk converges (and publishes `ready`)+ // immediately; ingest maintains the store from the first write.+ for field in index.ensure_sq8_stores_from_schema(&managed_schema_for_sq8) {+ index.spawn_sq8_walk(&field);+ }Ok(index)}@@ -8814,6 +8828,10 @@ impl Index {Self::resolve_flush_thresholds(config.storage.flush_size_mb);let flush_idle_secs = config.storage.flush_idle_secs;+ // #392: kept back from the struct literal (which moves `schema`) so+ // the SQ8 handle setup at the end of open can read the mapping.+ let schema_for_sq8 = schema.schema.clone();+// Try to reload a previously-persisted HNSW snapshot. If both// graph.bin and ids.json exist, validate, and the pinned field is// dense_vector-mapped (RC4 W2 item 16 — unmapped/legacy graphs are@@ -8984,6 +9002,7 @@ impl Index {hnsw_stale: Arc::new(std::sync::atomic::AtomicBool::new(hnsw_stale_init)),hnsw_rebuilding: Arc::new(std::sync::atomic::AtomicBool::new(false)),hnsw_rebuild_task: Arc::new(parking_lot::Mutex::new(None)),+ sq8_stores: Arc::new(parking_lot::RwLock::new(HashMap::new())),metric_query_count: Arc::new(AtomicU64::new(0)),metric_query_total_ms: Arc::new(AtomicU64::new(0)),metric_index_count: Arc::new(AtomicU64::new(0)),@@ -9059,6 +9078,12 @@ impl Index {// full graph-maintenance cost while the ANN path never served).// Heal it in the background from the authoritative doc set.index.spawn_hnsw_rebuild_if_stale();+ // #392: create SQ8 code-store handles for any scalar8 field and+ // make them serving-ready from the authoritative doc set (WAL+ // replay never re-runs vector indexing).+ for field in index.ensure_sq8_stores_from_schema(&schema_for_sq8) {+ index.spawn_sq8_walk(&field);+ }Ok(index)}@@ -12620,6 +12645,16 @@ impl Index {if let Some(handle) = self.hnsw_rebuild_task.lock().take() {handle.abort();}+ // #392: same treatment for in-flight SQ8 code-store walks. An+ // interrupted walk leaves `ready` false (exact scan serves) and the+ // next open retries.+ let walk_handles: Vec<Arc<Sq8StoreHandle>> =+ self.sq8_stores.read().values().cloned().collect();+ for handle in walk_handles {+ if let Some(handle) = handle.walk_task.lock().take() {+ handle.abort();+ }+ }}/// Update a document with upsert support.@@ -12871,6 +12906,9 @@ impl Index {/// node before inserting the fresh vector — otherwise the graph would/// keep serving the pre-update vector forever.async fn index_vectors(&self, doc_id: &str, source: &Value) {+ // #392: SQ8 code stores are maintained for EVERY scalar8 field on+ // every write, independent of the single field the HNSW graph pins.+ self.sq8_ingest(doc_id, source).await;let obj = match source.as_object() {Some(o) => o,None => return,@@ -13010,6 +13048,319 @@ impl Index {true}+ // ── SQ8 ingest-time code stores (#392) ──────────────────────────────+ //+ // One `Sq8CodeStore` per `quantization: "scalar8"` dense_vector field:+ // one u8 per dimension per document, written by the ingest hook the+ // moment a write publishes, addressed by dense slot (`slot * dim..`) —+ // the same slot discipline the HNSW slab uses for its f32 vectors. The+ // codebook is fitted from the vectors as they are ingested (widening ++ // re-encode on out-of-range values, never clamping), so a document's+ // quantized score is a function of index state alone — not of the+ // candidate set a query scans. That is the Lucene property #392 asks+ // for; qdrant's `EncodedVectorsU8` does the same at build time.+ //+ // Fail-safe direction everywhere: a store that is missing, not yet+ // converged, or whose coverage gate is broken (a wrong-dimension write)+ // leaves the kNN paths on the exact `_source` scan — nothing is ever+ // served from incomplete codes.++ /// The SQ8 code store handle for one field, if the mapping declared+ /// `scalar8` and the handle exists.+ fn sq8_store_for(&self, field: &str) -> Option<Arc<Sq8StoreHandle>> {+ self.sq8_stores.read().get(field).cloned()+ }++ /// Create code-store handles for every `scalar8` field the schema+ /// declares and none exists for yet. Returns the NEW fields — the caller+ /// spawns the authoritative doc walk for those (an existing index may+ /// carry documents that predate the handle; a fresh index converges on+ /// an empty walk).+ async fn ensure_sq8_stores(&self) -> Vec<String> {+ let schema = self.schema.read().await;+ self.ensure_sq8_stores_from_schema(&schema.schema)+ }++ /// Sync variant for call sites that already hold the schema.+ fn ensure_sq8_stores_from_schema(&self, schema: &Schema) -> Vec<String> {+ let scalar8_fields = collect_scalar8_fields(schema);+ if scalar8_fields.is_empty() {+ return Vec::new();+ }+ let mut created = Vec::new();+ let mut guard = self.sq8_stores.write();+ for field in scalar8_fields {+ if guard.contains_key(&field) {+ continue;+ }+ let similarity = lookup_vector_similarity(schema, &field);+ let dim = declared_field(schema, &field)+ .and_then(|fc| fc.options.dimensions)+ .unwrap_or(0);+ let handle = Arc::new(Sq8StoreHandle {+ store: parking_lot::RwLock::new(Sq8CodeStore::new(dim)),+ normalize: !matches!(+ similarity.as_str(),+ "l2_norm" | "dot_product" | "max_inner_product"+ ),+ ready: std::sync::atomic::AtomicBool::new(false),+ walking: std::sync::atomic::AtomicBool::new(false),+ walk_task: parking_lot::Mutex::new(None),+ });+ guard.insert(field.clone(), Arc::clone(&handle));+ created.push(field);+ }+ drop(guard);+ created+ }++ /// Spawn the authoritative doc walk that makes a (new) code store+ /// serving-ready. Single-flight per handle. The walk re-derives codes+ /// for every live document — WAL replay never re-runs vector indexing,+ /// so this is also the restart path — and then publishes `ready` when+ /// coverage is intact.+ fn spawn_sq8_walk(self: &Arc<Self>, field: &str) {+ let Some(handle) = self.sq8_store_for(field) else {+ return;+ };+ if handle+ .walking+ .swap(true, std::sync::atomic::Ordering::AcqRel)+ {+ return;+ }+ let weak = Arc::downgrade(self);+ let field = field.to_string();+ let index_name = self.name.as_str().to_string();+ let walk_handle = Arc::clone(&handle);+ let join = tokio::spawn(async move {+ let started = std::time::Instant::now();+ let mut converged = false;+ if let Some(idx) = weak.upgrade() {+ converged = idx.sq8_rebuild_one_field(&walk_handle, &field).await;+ }+ walk_handle+ .walking+ .store(false, std::sync::atomic::Ordering::Release);+ if converged {+ info!(+ index = %index_name,+ field = %field,+ elapsed_ms = started.elapsed().as_millis() as u64,+ "SQ8 code store rebuilt — codes serving path enabled"+ );+ } else {+ warn!(+ index = %index_name,+ field = %field,+ "SQ8 code store rebuild did not converge — staying on exact scan"+ );+ }+ });+ let mut slot = handle.walk_task.lock();+ if let Some(prev) = slot.replace(join) {+ prev.abort();+ }+ }++ /// Walk every live document once and make `handle`'s store describe+ /// exactly the live corpus: upsert each doc that carries `field`, then+ /// reconcile the remaining slots against the authoritative doc state.+ /// Returns `false` (store left not-ready, exact scan keeps serving) if+ /// any segment was unreadable or a suspect doc could not be fetched.+ async fn sq8_rebuild_one_field(+ self: &Arc<Self>,+ handle: &Arc<Sq8StoreHandle>,+ field: &str,+ ) -> bool {+ // Live-doc walk, same discipline as `rebuild_hnsw_from_docs`:+ // memtable first, then every flushed segment, dedup by doc id,+ // skipping tombstoned and seq-superseded copies.+ let mut seen: HashSet<String> = HashSet::new();+ let mem_docs = self.memtable.all_docs_with_sources();+ for (doc_id, src) in &mem_docs {+ seen.insert(doc_id.clone());+ if let Some(mut vector) = extract_numeric_vector(src, field) {+ if handle.normalize {+ l2_normalize_vec(&mut vector);+ }+ handle.store.write().upsert(doc_id, &vector);+ }+ }+ drop(mem_docs);++ let snap = self.store.snapshot();+ for meta in snap.segments.iter() {+ let Some(docs_arc) = self.stored_values_for_async(&meta.id).await else {+ warn!(+ index = self.name.as_str(),+ segment = meta.id.as_str(),+ field,+ "SQ8 rebuild: segment not readable — not ready"+ );+ return false;+ };+ for doc in docs_arc.iter() {+ let Some(id) = doc.get("_id").and_then(Value::as_str) else {+ continue;+ };+ if let Some(ver) = self.store.version_map.get(id) {+ if ver.deleted {+ continue;+ }+ if let Some(doc_seq) = doc.get("_seq_no").and_then(Value::as_u64) {+ if doc_seq < ver.seq_no {+ continue;+ }+ }+ }+ if !seen.insert(id.to_string()) {+ continue;+ }+ let src_owned;+ let src: &Value = match doc.get("_source") {+ Some(s) => s,+ None => {+ let mut d = doc.clone();+ if let Some(obj) = d.as_object_mut() {+ obj.remove("_id");+ }+ src_owned = d;+ &src_owned+ }+ };+ if let Some(mut vector) = extract_numeric_vector(src, field) {+ if handle.normalize {+ l2_normalize_vec(&mut vector);+ }+ handle.store.write().upsert(id, &vector);+ }+ }+ }++ // Reconcile slots the walk did not confirm. A slot is dropped when+ // its doc is gone (deleted, or no longer in the authoritative view)+ // or no longer carries the field; a slot whose doc raced the walk+ // (updated after its segment was passed) is refreshed from the live+ // source rather than trusted or dropped blind.+ let suspects: Vec<String> = {+ let store = handle.store.read();+ store+ .iter_live()+ .map(|(id, _)| id.to_string())+ .filter(|id| !seen.contains(id))+ .collect()+ };+ for id in suspects {+ let gone = self+ .store+ .version_map+ .get(&id)+ .map(|e| e.deleted)+ .unwrap_or(true);+ if gone {+ handle.store.write().remove(&id);+ continue;+ }+ match self.get_document_uncounted(&id).await {+ Ok(Some(src)) => match extract_numeric_vector(&src, field) {+ Some(mut vector) => {+ if handle.normalize {+ l2_normalize_vec(&mut vector);+ }+ handle.store.write().upsert(&id, &vector);+ }+ None => {+ // Live doc no longer carries the field.+ handle.store.write().remove(&id);+ }+ },+ Ok(None) => {+ // Not in the authoritative view any more.+ handle.store.write().remove(&id);+ }+ Err(e) => {+ warn!(+ index = self.name.as_str(),+ field,+ doc_id = %id,+ error = %e,+ "SQ8 rebuild: suspect doc not fetchable — not ready"+ );+ return false;+ }+ }+ }++ let coverage = handle.store.read().coverage_ok();+ handle.ready.store(coverage, Ordering::Release);+ coverage+ }++ /// Await all in-flight SQ8 code-store walks (deterministic test hook,+ /// mirroring [`Self::await_hnsw_rebuild`]).+ pub async fn await_sq8_rebuilds(&self) {+ let handles: Vec<Arc<Sq8StoreHandle>> = self.sq8_stores.read().values().cloned().collect();+ for handle in handles {+ let join = handle.walk_task.lock().take();+ if let Some(join) = join {+ let _ = join.await;+ }+ }+ }++ /// Maintain every scalar8 field's code store for one document write.+ /// Runs inside the write's publication interval (the same bracket the+ /// HNSW inserts use), so a codes-served query either sees the slot or+ /// falls back — never a half-published vector. A doc that stops+ /// carrying the field loses its slot in the same step.+ async fn sq8_ingest(&self, doc_id: &str, source: &Value) {+ let handles: Vec<(String, Arc<Sq8StoreHandle>)> = {+ let guard = self.sq8_stores.read();+ guard+ .iter()+ .map(|(f, h)| (f.clone(), Arc::clone(h)))+ .collect()+ };+ if handles.is_empty() {+ return;+ }+ for (field, handle) in handles {+ let Some(mut vector) = extract_numeric_vector(source, &field) else {+ handle.store.write().remove(doc_id);+ continue;+ };+ if handle.normalize {+ l2_normalize_vec(&mut vector);+ }+ handle.store.write().upsert(doc_id, &vector);+ }+ }++ /// Per-field SQ8 code-store health (#392), surfaced by `GET+ /// /{index}/_stats` next to the HNSW section.+ pub fn sq8_stats(&self) -> Value {+ let guard = self.sq8_stores.read();+ let mut fields = serde_json::Map::new();+ for (field, handle) in guard.iter() {+ let store = handle.store.read();+ fields.insert(+ field.clone(),+ serde_json::json!({+ "dim": store.dim(),+ "live": store.live_len(),+ "expected": store.expected(),+ "covered": store.coverage_ok(),+ "ready": handle.ready.load(Ordering::Acquire),+ "serving": handle.serving_fresh(),+ "refits": store.refits(),+ "codes_bytes": store.codes_bytes(),+ }),+ );+ }+ Value::Object(fields)+ }+/// Perform KNN search over the HNSW index.////// Returns the top-`k` (doc_id, distance) pairs.@@ -13757,6 +14108,154 @@ impl Index {.await}+ /// #392 serving fast path: score an UNFILTERED scalar8 kNN entirely+ /// from the ingest-time code store — no `_source` candidates are+ /// collected, no f32 vectors are read, nothing is quantized per query.+ ///+ /// The whole scoring loop runs under ONE read guard of the store, with+ /// no await inside it, so it observes the codes and the codebook of a+ /// single consistent generation (a concurrent ingest waits on the write+ /// guard; a codebook re-fit cannot interleave). Only the top `k`+ /// survivors are hydrated from stored docs afterwards — any hydration+ /// anomaly (doc deleted mid-scan, unreadable segment, a chunked-passage+ /// doc) abandons the path and the caller falls back to the exact scan,+ /// mirroring `run_knn_hnsw`'s never-a-partial-page discipline.+ ///+ /// Returns `None` whenever the path cannot serve: store missing, walk+ /// not converged, coverage broken (a wrong-dimension write), dim+ /// mismatch, deadline, or a publication racing source visibility. The+ /// fail-safe direction is always the exact `_source` scan.+ #[allow(clippy::too_many_arguments)] // mirrors run_knn_brute_force_with_deadline+ async fn run_knn_sq8_codes_scan(+ &self,+ request: &SearchRequest,+ deadline: std::time::Instant,+ field: &str,+ query_vec: &[f32],+ k: usize,+ similarity: &str,+ boost: Option<f32>,+ min_similarity: Option<f32>,+ started: std::time::Instant,+ ) -> Option<SearchResult> {+ if std::time::Instant::now() >= deadline {+ return None;+ }+ // Chunked-passage docs score by best passage in the exact path; the+ // codes hold the pooled vector, so hand those back to it.+ if self.passage_chunks_require_exact(field) {+ return None;+ }+ let handle = self.sq8_store_for(field)?;+ if !handle.serving_fresh() {+ return None;+ }+ // Same publication discipline as `run_knn_hnsw`: never serve codes+ // from a window that overlapped a vector publication.+ let publication_generation = self.hnsw_publication_generation.load(Ordering::Acquire);+ if self.hnsw_publications_in_flight.load(Ordering::Acquire) != 0 {+ return None;+ }++ // ── Score every live slot from the codes ─────────────────────+ let mut ranked: Vec<(f32, String)> = Vec::new();+ {+ let store = handle.store.read();+ // Re-check the gates under the guard: the freshness check above+ // raced a concurrent walk/ingest if this now fails.+ if !store.coverage_ok() || store.dim() != query_vec.len() {+ return None;+ }+ let codebook = store.codebook();+ let mut decoded = vec![0.0f32; store.dim()];+ ranked.reserve(store.live_len() as usize);+ for (id, codes) in store.iter_live() {+ // Cheap deadline poll — no await can run under this guard.+ // On expiry hand the request to the exact path (which owns+ // the timed_out semantics), never serve a partial scan.+ if ranked.len() & 8191 == 8191 && std::time::Instant::now() >= deadline {+ return None;+ }+ codebook.decode_into(codes, &mut decoded);+ let score = compute_vector_similarity(similarity, query_vec, &decoded);+ ranked.push((score, id.to_string()));+ }+ }+ debug!(+ index = self.name.as_str(),+ field,+ scored = ranked.len(),+ "SQ8 kNN served from ingest-time codes (#392)"+ );++ // ── Same cutoff → boost → rank pipeline as the exact path ─────+ if let Some(raw) = min_similarity {+ let cut = raw_similarity_to_score(similarity, raw);+ ranked.retain(|(score, _)| *score >= cut);+ }+ if let Some(b) = boost {+ if (b - 1.0).abs() > f32::EPSILON {+ for (score, _) in ranked.iter_mut() {+ *score *= b;+ }+ }+ }+ // Deterministic order for equal scores (the slot walk's HashMap+ // order is not), then cap the pool at k — only these are hydrated.+ ranked.sort_by(|a, b| {+ b.0.partial_cmp(&a.0)+ .unwrap_or(std::cmp::Ordering::Equal)+ .then_with(|| a.1.cmp(&b.1))+ });+ ranked.truncate(k.max(1));++ // ── Hydrate the top-k from stored docs ────────────────────────+ let chunk_field = format!("{field}_chunks");+ let mut scored: Vec<(String, f32, Value, Option<u32>)> = Vec::with_capacity(ranked.len());+ for (score, doc_id) in ranked {+ let remaining = deadline.saturating_duration_since(std::time::Instant::now());+ if remaining.is_zero() {+ return None;+ }+ let src =+ match tokio::time::timeout(remaining, self.get_document_uncounted(&doc_id)).await {+ Ok(Ok(Some(source))) => source,+ _ => return None,+ };+ // A chunked doc must be scored by its best passage, not the+ // pooled codes — same bail as `run_knn_hnsw`.+ if get_field_value(&src, &chunk_field).is_some() {+ return None;+ }+ scored.push((doc_id, score, src, None));+ }+ if std::time::Instant::now() >= deadline {+ return None;+ }+ if self.passage_chunks_require_exact(field) {+ return None;+ }+ if self.hnsw_publications_in_flight.load(Ordering::Acquire) != 0+ || self.hnsw_publication_generation.load(Ordering::Acquire) != publication_generation+ {+ return None;+ }++ let generated_companion_fields = {+ let schema = self.schema.read().await;+ generated_embedding_companion_fields(&schema.schema)+ };+ Some(knn_result_from_scored(+ self,+ request,+ field,+ scored,+ k,+ started,+ &generated_companion_fields,+ ))+ }+#[allow(clippy::too_many_arguments)]async fn run_knn_brute_force_with_deadline(&self,@@ -13787,6 +14286,42 @@ impl Index {// this is the whole filtered-kNN path. Owned clone; the lock is released.let dmq_schema = self.schema().await;+ // ── Determine whether this field opts into SQ8 (scalar8) ──────+ // Default fields keep the exact f32 brute-force scan below,+ // byte-identical to before. A `scalar8` field may take the codes+ // fast path instead (#392). Determined up front so the fast path+ // can skip the candidate collection altogether.+ let use_sq8 = {+ let schema = self.schema.read().await;+ lookup_vector_quantization(&schema.schema, field).as_deref() == Some("scalar8")+ };++ // ── #392 fast path: unfiltered scalar8 kNN from ingest-time codes ──+ // Scores every live slot straight out of the code store — no+ // `_source` candidates collected, no per-query quantization — and+ // hydrates only the top k. Any gate miss (store not converged,+ // coverage broken, publication racing) falls through to the exact+ // scan below; filtered queries score from the same codes inside+ // the use_sq8 branch further down.+ if use_sq8 && filter.is_none() {+ if let Some(result) = self+ .run_knn_sq8_codes_scan(+ request,+ deadline,+ field,+ query_vec,+ k,+ similarity,+ boost,+ min_similarity,+ started,+ )+ .await+ {+ return Ok(result);+ }+ }+// ── Collect all candidate (doc_id, source) pairs ──────────────let mut candidates: Vec<(String, Value)> = Vec::new();// Memtable first (newest writes).@@ -13889,17 +14424,8 @@ impl Index {}let collect_elapsed = started.elapsed();- // ── Determine whether this field opts into SQ8 (scalar8) ──────- // Default fields keep the exact f32 brute-force scan below,- // byte-identical to before. A `scalar8` field takes the branch above- // instead: it reads the same live f32 vector out of `_source`, but- // rounds it through 1-byte-per-dimension SQ8 codes before scoring, so- // the score carries `scalar8`'s precision loss. Nothing is cached —- // neither the codes nor the codebook outlive the query (#371).- let use_sq8 = {- let schema = self.schema.read().await;- lookup_vector_quantization(&schema.schema, field).as_deref() == Some("scalar8")- };+ // `use_sq8` was determined before candidate collection (the codes+ // fast path above may already have returned).// ── Score each candidate against the query vector ─────────────let mut scored: Vec<(String, f32, Value, Option<u32>)> =@@ -13948,78 +14474,130 @@ impl Index {cand.push((id, src, doc_vec));}- // Fit this field's SQ8 codebook over the candidate vectors this- // query is about to score, and drop it when the query ends. No- // codec state survives a query (#371).+ // ── #392: score from the INGEST-TIME codes when they cover the+ // candidate set ──────────────────────────────────────────────//- // The codebook used to be fitted from the first ≤1000 vectors the- // field was ever scanned with and then kept for the life of the- // process, which is the same write-once defect as the per-document- // code map one level up: a vector written afterwards outside the- // fitted per-dimension range is CLAMPED into it, and when that- // range is narrow the clamped decode is indistinguishable from the- // vector the document used to hold. A corpus whose dimension 0- // never left +1.0 fitted `[1,1]` there, so overwriting a document- // with its exact negation decoded straight back to +1.0 and it- // stayed top at cosine 1.000000 — verbatim the reported symptom,- // with the per-document map already gone.- //- // Fitting over exactly the set being encoded also makes clamping- // structurally impossible here rather than merely unlikely: every- // value passed to `encode_into` is inside `[min,max]` by- // construction. An empty candidate set fits degenerate params, but- // the scoring loop below is then empty too, so nothing observes- // them — the old code had to guard against publishing that codec- // because it outlived the query that produced it.- //- // ACCEPTED CONSEQUENCE: the codebook now depends on the candidate- // set, and that changes the RETURNED ORDER, not merely the score.- // Each individual score moves by at most SQ8's own quantization- // step (1/255 of the fitted range per dimension), which is inside- // the approximation error `scalar8` already advertises — but- // near-tied documents swap, so a caller sees a different ranking.- // Measured at the HTTP boundary on a 60-document 4-dim cosine- // field: adding a `filter` that removes only unrelated documents- // returned the same 30 survivors with max |Δ_score| 1.976e-05 but- // a different order at 19 of 30 positions. The trigger is the- // candidate set, not the `filter` keyword — indexing one more- // unrelated document moved the same corpus by max 7.100e-06 and- // reordered its top 10 — and on an unfiltered corpus over 1000- // documents every score also differs from rc.17, which fitted from- // the first <=1000 candidates and cached that (1500 documents: all- // of the top 40 changed, max 4.880e-05, one adjacent-rank swap).- // Documented in docs/recipes/vector-quantization.md and #392.- //- // This is NOT the dependency Lucene has. Lucene fits per segment at- // INDEX time, so a Lucene score is a function of the index state- // alone; here it is a function of the query's candidate set too.- // #392 is what would close that gap.- let params = Sq8Params::fit_borrowed(cand.iter().map(|(_, _, v)| v.as_slice()), dim);- debug!(- field,- dim,- candidates = cand.len(),- normalize,- "SQ8 codec fitted for this query"- );+ // When the field's code store is serving-fresh and every+ // post-filter candidate holds a live slot, each candidate is+ // scored by decoding the SAME bytes under the SAME codebook the+ // unfiltered path (`run_knn_sq8_codes_scan`) scores with. That+ // is the point of #392: a document's `_score` becomes a+ // function of the index state alone and stops depending on+ // which other documents a filter removed. Any miss — store not+ // converged, coverage broken, a candidate without codes, a dim+ // mismatch — falls back to the per-query fit below, which is+ // still correct (it is the pre-#392 behaviour) but+ // filter-dependent.+ let mut stored_codes: Option<(Sq8Params, Vec<Vec<u8>>)> = None;+ if let Some(handle) = self.sq8_store_for(field) {+ if handle.serving_fresh() {+ let store = handle.store.read();+ if store.dim() == dim && store.coverage_ok() {+ let codebook = store.codebook();+ let mut all = Vec::with_capacity(cand.len());+ let mut complete = true;+ for (id, _, _) in cand.iter() {+ match store.codes_for(id) {+ Some(c) => all.push(c.to_vec()),+ None => {+ complete = false;+ break;+ }+ }+ }+ if complete {+ stored_codes = Some((codebook, all));+ }+ }+ }+ }- // Score by quantizing each candidate's CURRENT vector and decoding- // it straight back — 1 byte/dim, so `scalar8` keeps its recall- // profile, and the score describes the vector the document holds- // right now rather than one a previous query cached. Both buffers- // are reused across the scan, so this allocates nothing per- // document. `v.len() == dim` was established when `cand` was built.- let mut codes = vec![0u8; dim];- let mut decoded = vec![0.0f32; dim];- for (position, (id, src, v)) in cand.into_iter().enumerate() {- if position & 127 == 0 && self.exact_scan_checkpoint(position, deadline).await {- timed_out = true;- break;+ match stored_codes {+ Some((codebook, codes)) => {+ debug!(+ field,+ dim,+ candidates = codes.len(),+ "SQ8 filtered kNN scored from ingest-time codes (#392)"+ );+ // `dim` bytes per candidate were copied out under a+ // short read guard, so this loop holds no lock across+ // its deadline checkpoints.+ let mut decoded = vec![0.0f32; dim];+ for (position, ((id, src, _), codes)) in cand.into_iter().zip(codes).enumerate()+ {+ if position & 127 == 0+ && self.exact_scan_checkpoint(position, deadline).await+ {+ timed_out = true;+ break;+ }+ codebook.decode_into(&codes, &mut decoded);+ let score = compute_vector_similarity(similarity, query_vec, &decoded);+ scored.push((id, score, src, None));+ }+ }+ None => {+ // ── Legacy per-query fit (pre-#392 fallback) ────────+ //+ // Fit this field's SQ8 codebook over the candidate vectors+ // this query is about to score, and drop it when the query+ // ends. No codec state survives a query (#371).+ //+ // The codebook used to be fitted from the first ≤1000+ // vectors the field was ever scanned with and then kept+ // for the life of the process, which is the same+ // write-once defect as the per-document code map one+ // level up: a vector written afterwards outside the+ // fitted per-dimension range is CLAMPED into it.+ // Fitting over exactly the set being encoded makes+ // clamping structurally impossible here: every value+ // passed to `encode_into` is inside `[min,max]` by+ // construction.+ //+ // REMAINING CONSEQUENCE (#392): the fitted codebook+ // depends on the candidate set, so each score moves by up+ // to SQ8's quantization step when unrelated documents are+ // added or filtered away, and near-tied documents can+ // swap. Measured at the HTTP boundary on a 60-document+ // 4-dim cosine field: adding a `filter` that removed only+ // unrelated documents returned the same 30 survivors with+ // max |Δ_score| 1.976e-05 but a different order at 19 of+ // 30 positions. Lucene fits per segment at INDEX time, so+ // its scores are a function of index state alone. This+ // branch is now reached only when the ingest-time store+ // cannot serve; #392's codes path is what closes the gap.+ let params =+ Sq8Params::fit_borrowed(cand.iter().map(|(_, _, v)| v.as_slice()), dim);+ debug!(+ field,+ dim,+ candidates = cand.len(),+ normalize,+ "SQ8 codec fitted for this query (ingest-time codes unavailable)"+ );++ // Score by quantizing each candidate's CURRENT vector and+ // decoding it straight back — 1 byte/dim, so `scalar8`+ // keeps its recall profile, and the score describes the+ // vector the document holds right now. Both buffers are+ // reused across the scan. `v.len() == dim` was+ // established when `cand` was built.+ let mut codes = vec![0u8; dim];+ let mut decoded = vec![0.0f32; dim];+ for (position, (id, src, v)) in cand.into_iter().enumerate() {+ if position & 127 == 0+ && self.exact_scan_checkpoint(position, deadline).await+ {+ timed_out = true;+ break;+ }+ params.encode_into(&v, &mut codes);+ params.decode_into(&codes, &mut decoded);+ let score = compute_vector_similarity(similarity, query_vec, &decoded);+ scored.push((id, score, src, None));+ }}- params.encode_into(&v, &mut codes);- params.decode_into(&codes, &mut decoded);- let score = compute_vector_similarity(similarity, query_vec, &decoded);- scored.push((id, score, src, None));}} else {// Per-chunk (passage) companion: multi-chunk `semantic_text` docs@@ -16472,6 +17050,15 @@ impl Index {}).ok();}+ // #392: drop the deleted doc's SQ8 code slots so the per-field+ // coverage gates stay exact (mirrors the vector_doc_count decrement+ // above).+ {+ let guard = self.sq8_stores.read();+ for handle in guard.values() {+ handle.store.write().remove(id);+ }+ }#[cfg(test)]self.publication_test_point(id, PublicationTestPoint::AfterHnsw);@@ -22650,7 +23237,7 @@ impl Index {}/// Add a field to the schema.- pub async fn add_field(&self, field: FieldConfig) -> Result<()> {+ pub async fn add_field(self: &Arc<Self>, field: FieldConfig) -> Result<()> {self.add_fields(vec![field]).await}@@ -22660,7 +23247,7 @@ impl Index {/// companion together. Validate and persist the complete candidate schema/// before publishing it so a later-field failure cannot leave half of that/// contract visible.- pub async fn add_fields(&self, fields: Vec<FieldConfig>) -> Result<()> {+ pub async fn add_fields(self: &Arc<Self>, fields: Vec<FieldConfig>) -> Result<()> {if fields.is_empty() {return Ok(());}@@ -22724,6 +23311,13 @@ impl Index {*self.embedder.write().await = embedder;}*schema = candidate;+ drop(schema);+ // #392: a mapping update can introduce a `scalar8` field — create+ // its code store now and walk the live docs (the index may already+ // carry documents predating the handle).+ for field in self.ensure_sq8_stores().await {+ self.spawn_sq8_walk(&field);+ }Ok(())}@@ -35631,6 +36225,64 @@ pub(crate) fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {out}+/// One `scalar8` field's ingest-time SQ8 code store plus its readiness gate+/// (#392). The engine holds one handle per quantized field in+/// [`Index::sq8_stores`].+struct Sq8StoreHandle {+ /// The slot-addressed code array + codebook. Written by the ingest hook+ /// (`sq8_ingest`), the delete hook, and the authoritative doc walk;+ /// read by the kNN serving paths under a read guard.+ store: parking_lot::RwLock<Sq8CodeStore>,+ /// L2-normalize vectors before encoding — cosine-family similarity,+ /// exactly the rule the per-query serving path applies+ /// (`!matches!(similarity, "l2_norm" | "dot_product" | "max_inner_product")`).+ /// Captured from the mapping when the handle is created; changing a+ /// live field's similarity requires a reindex (as in ES).+ normalize: bool,+ /// Set once an authoritative walk over the live document set has+ /// converged (empty indexes converge trivially). The serving paths may+ /// only score from the codes when this holds AND the store's coverage+ /// gate passes; otherwise they fall back to the exact scan.+ ready: std::sync::atomic::AtomicBool,+ /// Single-flight guard for the authoritative walk.+ walking: std::sync::atomic::AtomicBool,+ /// JoinHandle of the in-flight walk, aborted by `abort_background_tasks`.+ walk_task: parking_lot::Mutex<Option<tokio::task::JoinHandle<()>>>,+}++impl Sq8StoreHandle {+ /// The serving gate: the walk has converged and every doc id that ever+ /// carried (or attempted) this field holds a live slot.+ fn serving_fresh(&self) -> bool {+ self.ready.load(Ordering::Acquire) && self.store.read().coverage_ok()+ }+}++/// All dense_vector fields whose mapping opts into SQ8 (`quantization:+/// "scalar8"`, set from `index_options.type: int8_hnsw|int8_flat` by the+/// compat layer). Same recursive walk as [`collect_dense_vector_fields`].+fn collect_scalar8_fields(schema: &Schema) -> Vec<String> {+ fn walk(fields: &[FieldConfig], prefix: &str, out: &mut Vec<String>) {+ for fc in fields {+ let path = if prefix.is_empty() {+ fc.name.clone()+ } else {+ format!("{prefix}.{}", fc.name)+ };+ if matches!(fc.field_type, FieldType::Vector)+ && fc.options.quantization.as_deref() == Some("scalar8")+ {+ out.push(path);+ } else if !fc.fields.is_empty() {+ walk(&fc.fields, &path, out);+ }+ }+ }+ let mut out = Vec::new();+ walk(&schema.fields, "", &mut out);+ out+}+/// A top-level KNN query unwrapped for the short-circuit executor.#[derive(Debug, Clone)]struct PeeledKnn {diff --git a/engine/crates/xerj-vector/src/lib.rs b/engine/crates/xerj-vector/src/lib.rsindex 994be0c2e..302ce6c0a 100644--- a/engine/crates/xerj-vector/src/lib.rs+++ b/engine/crates/xerj-vector/src/lib.rs@@ -12,6 +12,7 @@ pub mod distance;pub mod hnsw;pub mod quantizer;pub mod search;+pub mod sq8_codes;pub use distance::{compute_distance, cosine, dot_product, l2_squared, DistanceMetric};pub use hnsw::{HnswIndex, HnswParams};@@ -19,5 +20,6 @@ pub use quantizer::{NoneQuantizer, QuantizedData, QuantizedVectors, Quantizer, Scalar8Quantizer, Sq8Params,};pub use search::{SearchResult, VectorSearcher};+pub use sq8_codes::Sq8CodeStore;pub use xerj_common::Result;diff --git a/engine/crates/xerj-vector/src/quantizer.rs b/engine/crates/xerj-vector/src/quantizer.rsindex 72a450179..95f4b2534 100644--- a/engine/crates/xerj-vector/src/quantizer.rs+++ b/engine/crates/xerj-vector/src/quantizer.rs@@ -183,11 +183,11 @@ impl Quantizer for NoneQuantizer {/// Each dimension gets its own `min` and `scale` computed from the training/// set, so the quantization is adaptive to the actual data distribution.///-/// Encoding size: 1 byte per dimension vs. 4 bytes for f32 — 4x. That is a-/// property of this codec, not of the engine: the kNN serving path does not-/// hold SQ8 codes resident, it quantizes each candidate's f32 vector per query-/// (see [`Sq8Params`], and issue #392 for the ingest-time code array that would-/// turn this ratio into a resident saving).+/// Encoding size: 1 byte per dimension vs. 4 bytes for f32 — 4x. On the kNN+/// serving path that ratio is realized by [`crate::sq8_codes::Sq8CodeStore`]+/// (issue #392): codes are written at ingest into a flat, slot-addressed u8+/// array with a codebook fitted from the ingested vectors, and queries score+/// against those codes instead of re-quantizing each candidate's f32 vector.#[derive(Debug, Default, Clone)]pub struct Scalar8Quantizer;@@ -320,25 +320,38 @@ impl Quantizer for Scalar8Quantizer {////// This is the serving-path counterpart to [`Scalar8Quantizer`]: instead of/// packing a whole batch into one blob, it exposes a fitted codec cheap enough-/// to build per query. The brute-force kNN scan fits one of these over exactly-/// the candidate vectors it is about to score ([`Sq8Params::fit_borrowed`]),-/// quantizes each candidate's current vector through it, dequantizes straight-/// back, and drops it — so scores come from **1 byte/dim** codes while **no-/// SQ8 state outlives the query** (issue #371).+/// to build per query.///-/// That is deliberate. Both a per-document `doc_id -> Vec<u8>` code map and a-/// per-field cached codebook were tried, and both were write-once in practice:-/// the map never recomputed a document's codes after an update, and the-/// codebook, fitted from the first ~1000 vectors a field was ever scanned-/// with, silently *clamped* every later vector into that range — so a document-/// overwritten with its own negation decoded straight back to its old value-/// and kept ranking first at cosine 1.000000. Fitting over the set being-/// encoded also makes clamping impossible here rather than merely unlikely.+/// Since issue #392 the PRIMARY serving path no longer builds one of these at+/// all: [`crate::sq8_codes::Sq8CodeStore`] holds the codes and the codebook at+/// ingest time (one flat u8 array, slot-addressed, widened and re-encoded when+/// a vector lands outside the fitted range), and kNN queries score by decoding+/// those codes. A document's score is therefore a function of the index state+/// alone, not of the candidate set the query happens to scan — the property+/// Lucene has and the per-query fit lacked.///-/// The cost is that `scalar8` does not reduce resident memory: the scan reads-/// the full-precision vector from `_source` either way. Making codes live with-/// the data, addressed by ordinal and written at ingest as Lucene does, is-/// tracked in issue #392.+/// `fit_borrowed` remains as the FALLBACK codec for the windows where the+/// store cannot serve — the open-time walk has not converged yet, a+/// publication is racing source visibility, or coverage is broken by a+/// wrong-dimension write. There the brute-force scan fits one of these over+/// exactly the candidate vectors it is about to score, quantizes each+/// candidate's current vector through it, dequantizes straight back, and drops+/// it. That keeps the #371 guarantee — **no SQ8 state outlives the query**, so+/// an updated document can never be scored on a stale code — at the cost that+/// scores in that fallback window depend on the candidate set (each by at most+/// SQ8's own 1/255 quantization step).+///+/// History (#371): both a per-document `doc_id -> Vec<u8>` code map and a+/// per-field cached codebook were tried before this type existed, and both+/// were write-once in practice: the map never recomputed a document's codes+/// after an update, and the codebook, fitted from the first ~1000 vectors a+/// field was ever scanned with, silently *clamped* every later vector into+/// that range — so a document overwritten with its own negation decoded+/// straight back to its old value and kept ranking first at cosine 1.000000.+/// The ingest-time store does not repeat that defect class structurally: an+/// out-of-range value extends the range and every live code is re-encoded+/// through a decode/encode round trip (each stored value moves by at most the+/// new quantization step), so clamping cannot occur.////// `mins[d]`/`scales[d]` are the per-dimension minimum and range (`max-min`)/// observed in the fitting sample. `encode` maps `v[d]` linearly to a u8 in@@ -787,9 +800,10 @@ mod tests {}/// (b) MEMORY: SQ8 codes are ~4× smaller than the f32 vectors they encode.- /// This is a property of the codec, measured here on the codec. The engine- /// does not currently realize it as a resident saving on the kNN path — see- /// the note on [`Sq8Params`].+ /// This is a property of the codec, measured here on the codec; since #392+ /// the engine realizes it on the kNN path too — the serving scan scores+ /// against the ingest-time code array ([`crate::sq8_codes::Sq8CodeStore`]),+ /// not the f32 `_source` vectors.#[test]fn sq8_store_is_quarter_of_f32() {let n = 2000usize;diff --git a/engine/crates/xerj-vector/src/sq8_codes.rs b/engine/crates/xerj-vector/src/sq8_codes.rsnew file mode 100644index 000000000..e2e407201--- /dev/null+++ b/engine/crates/xerj-vector/src/sq8_codes.rs@@ -0,0 +1,404 @@+//! Ingest-time, slot-addressed SQ8 code store — issue #392.+//!+//! [`Sq8CodeStore`] is the durable counterpart of the per-query codec+//! ([`Sq8Params`](crate::quantizer::Sq8Params)): one u8 code per dimension+//! per document, written when the document is indexed and rewritten when it+//! is updated, in a single flat `Vec<u8>` addressed by a dense slot —+//! `codes[slot * dim .. slot * dim + dim]` — the same slot discipline the+//! HNSW slab uses for its f32 vectors. The codebook is fitted from the+//! vectors as they are ingested and lives with the data, so a document's+//! quantized score is a function of the index state alone and not of the+//! candidate set a particular query happens to scan (#392's second half).+//!+//! Reference points (retrieved, not copied):+//! * qdrant `lib/quantization/src/encoded_vectors_u8.rs` (Apache-2.0) —+//! quantized codes are written once at build time into contiguous+//! storage, fetched by `get_vector_data(offset)` at+//! `offset * quantized_vector_size`; the codebook (alpha/offset) is+//! fitted at build time and persisted next to the codes; the query is+//! encoded once and scored against stored codes without touching f32.+//! * Lucene `OffHeapScalarQuantizedVectorValues.vectorValue(targetOrd)`+//! seeks `targetOrd * byteSize` into a per-segment slice written at index+//! time; requantization happens at merge time, never at query time+//! (cited from issue #392; Elasticsearch/Lucene is AGPL/SSPL — approach+//! only).+//!+//! The one structural difference from both: XERJ maintains the store+//! incrementally per document instead of rebuilding it per segment merge, so+//! the codebook WIDENS whenever an ingested vector falls outside the fitted+//! range, and widening re-encodes every stored code through a decode/encode+//! round trip (each stored value moves by at most the new quantization+//! step). That is the same order of accuracy work a merge-time requantization+//! does, paid at ingest, and it is what makes clamping structurally+//! impossible: the codebook always spans every live vector (#371's defect+//! class cannot recur — a vector written outside the range extends the range+//! rather than being clamped into it).++use std::collections::HashMap;++use crate::quantizer::Sq8Params;++/// Slot-addressed SQ8 code store for ONE dense_vector field.+///+/// All mutating operations are `&mut self`; the engine keeps each store+/// behind a lock and serves scans under a read guard, so a scan observes the+/// codes and the codebook of one consistent generation.+#[derive(Debug)]+pub struct Sq8CodeStore {+ dim: usize,+ /// The live codebook. `maxs` are kept alongside so an out-of-range+ /// ingest can extend the range exactly rather than through the+ /// dequantized grid.+ mins: Vec<f32>,+ maxs: Vec<f32>,+ /// slot → doc id. Slots of tombstoned documents keep their last id+ /// (never read again; `slot_of` is the live set).+ ids: Vec<String>,+ /// doc id → slot (the live set).+ slot_of: HashMap<String, u32>,+ /// Flat code array, `codes[slot * dim .. slot * dim + dim]`.+ codes: Vec<u8>,+ /// Soft delete marks, one per slot (the slab's discipline: append-only,+ /// tombstone on delete, never physically free).+ tomb: Vec<bool>,+ /// Coverage-gate denominator: number of DISTINCT doc ids ever offered to+ /// [`Self::upsert`] and not since removed. Mirrors the engine's+ /// `vector_doc_count` (counted on attempt): a rejected vector (wrong+ /// dimension) increments this without creating a slot, so+ /// `live_len() != expected` and consumers fall back to exact paths.+ expected: u64,+ /// Number of codebook widenings (observability).+ refits: u64,+}++impl Sq8CodeStore {+ /// An empty store for `dim`-dimensional vectors.+ pub fn new(dim: usize) -> Self {+ Sq8CodeStore {+ dim,+ mins: vec![0.0; dim],+ maxs: vec![0.0; dim],+ ids: Vec::new(),+ slot_of: HashMap::new(),+ codes: Vec::new(),+ tomb: Vec::new(),+ expected: 0,+ refits: 0,+ }+ }++ /// Vector dimensionality this store encodes.+ pub fn dim(&self) -> usize {+ self.dim+ }++ /// The current codebook (mins/scales), for decoding at query time.+ pub fn codebook(&self) -> Sq8Params {+ let scales = self+ .mins+ .iter()+ .zip(self.maxs.iter())+ .map(|(&mn, &mx)| mx - mn)+ .collect();+ Sq8Params {+ mins: self.mins.clone(),+ scales,+ }+ }++ /// Live documents holding a code slot (the coverage-gate numerator).+ pub fn live_len(&self) -> u64 {+ self.slot_of.len() as u64+ }++ /// Distinct doc ids ever accepted for this field minus removals (the+ /// coverage-gate denominator). See the field docs for the attempt+ /// semantics.+ pub fn expected(&self) -> u64 {+ self.expected+ }++ /// The serving gate: every doc id that ever carried (or attempted to+ /// carry) this field holds a live slot. Consumers that score from the+ /// codes must fall back to an exact path when this is false — a doc with+ /// a rejected vector would otherwise be silently missing from results.+ pub fn coverage_ok(&self) -> bool {+ self.live_len() == self.expected+ }++ /// Codebook widenings since construction (observability).+ pub fn refits(&self) -> u64 {+ self.refits+ }++ /// Bytes occupied by the flat code array (including tombstoned slots —+ /// the array is append-only).+ pub fn codes_bytes(&self) -> usize {+ self.codes.len()+ }++ /// The live codes of one document, `dim` bytes at `slot * dim`.+ pub fn codes_for(&self, doc_id: &str) -> Option<&[u8]> {+ let &slot = self.slot_of.get(doc_id)?;+ Some(&self.codes[slot as usize * self.dim..][..self.dim])+ }++ /// Iterate every LIVE `(doc id, codes)` pair. Slot-addressed order.+ pub fn iter_live(&self) -> impl Iterator<Item = (&str, &[u8])> {+ let dim = self.dim;+ self.slot_of+ .iter()+ .map(move |(id, &slot)| (id.as_str(), &self.codes[slot as usize * dim..][..dim]))+ }++ /// Write (or rewrite, on update) one document's codes.+ ///+ /// Returns `false` when `vector` does not have exactly [`Self::dim`]+ /// dimensions — the attempt is still counted in `expected` (no slot is+ /// created), which breaks [`Self::coverage_ok`] and keeps consumers on+ /// exact paths: the same wrong-dimension document the brute-force scan+ /// skips must not be half-represented here.+ ///+ /// A vector outside the fitted per-dimension range widens the codebook+ /// and re-encodes every live code (see the module docs). Re-encoding+ /// goes decode-old → encode-new, bounded by the new quantization step.+ pub fn upsert(&mut self, doc_id: &str, vector: &[f32]) -> bool {+ if vector.len() != self.dim {+ // Count the attempt for a NEW id only: an id that never had a+ // slot must not leak a phantom expectation, and an id whose+ // field silently appeared with the wrong arity keeps its slot+ // (remove() is the API for that).+ if !self.slot_of.contains_key(doc_id) {+ self.expected += 1;+ }+ return false;+ }+ // Snapshot the codebook BEFORE widening: the re-encode decodes each+ // live code under the range it was encoded with.+ let old_codebook = self.codebook();+ if self.extend_codebook_for(vector) {+ self.refits += 1;+ self.reencode_all(&old_codebook);+ }+ let codebook = self.codebook();+ match self.slot_of.get(doc_id).copied() {+ Some(slot) => {+ let start = slot as usize * self.dim;+ codebook.encode_into(vector, &mut self.codes[start..][..self.dim]);+ self.tomb[slot as usize] = false;+ true+ }+ None => {+ let slot = self.ids.len() as u32;+ let mut code = vec![0u8; self.dim];+ codebook.encode_into(vector, &mut code);+ self.ids.push(doc_id.to_string());+ self.codes.extend_from_slice(&code);+ self.tomb.push(false);+ self.slot_of.insert(doc_id.to_string(), slot);+ self.expected += 1;+ true+ }+ }+ }++ /// Drop a document's live slot (delete, or the document no longer+ /// carries the field). Returns whether a slot was live. The slot is+ /// tombstoned, not freed — same append-only discipline as the HNSW slab.+ pub fn remove(&mut self, doc_id: &str) -> bool {+ match self.slot_of.remove(doc_id) {+ Some(slot) => {+ self.tomb[slot as usize] = true;+ self.expected = self.expected.saturating_sub(1);+ true+ }+ None => false,+ }+ }++ /// Widen `mins`/`maxs` to include `vector` where it falls outside.+ /// Returns whether anything moved. An empty store adopts `vector`'s+ /// per-dimension values as its initial (degenerate, zero-scale) range —+ /// `min == max` encodes every value to code 0 and decodes it back to+ /// `min` exactly, so the first document carries no quantization error.+ fn extend_codebook_for(&mut self, vector: &[f32]) -> bool {+ let mut extended = false;+ if self.ids.is_empty() {+ self.mins.copy_from_slice(vector);+ self.maxs.copy_from_slice(vector);+ return false;+ }+ for (d, &x) in vector.iter().enumerate() {+ if x < self.mins[d] {+ self.mins[d] = x;+ extended = true;+ }+ if x > self.maxs[d] {+ self.maxs[d] = x;+ extended = true;+ }+ }+ extended+ }++ /// Re-encode every live slot under the current (just widened) codebook,+ /// decoding under `old`. Each stored value moves by at most the new+ /// quantization step (see the module docs).+ fn reencode_all(&mut self, old: &Sq8Params) {+ if self.slot_of.is_empty() {+ return;+ }+ let new = self.codebook();+ let dim = self.dim;+ let mut decoded = vec![0.0f32; dim];+ let mut code = vec![0u8; dim];+ // Live slots only: tombstoned codes are unreachable through+ // `slot_of` and re-encoding them would be wasted work.+ let slots: Vec<u32> = self.slot_of.values().copied().collect();+ for slot in slots {+ let start = slot as usize * dim;+ old.decode_into(&self.codes[start..][..dim], &mut decoded);+ new.encode_into(&decoded, &mut code);+ self.codes[start..][..dim].copy_from_slice(&code);+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn store_with(vectors: &[Vec<f32>]) -> Sq8CodeStore {+ let mut s = Sq8CodeStore::new(vectors[0].len());+ for (i, v) in vectors.iter().enumerate() {+ assert!(s.upsert(&i.to_string(), v), "upsert {i} must succeed");+ }+ s+ }++ #[test]+ fn first_vector_encodes_exactly() {+ // Degenerate range [v, v] per dim → code 0 → decode back to v.+ let v = vec![0.25, -0.5, 1.75];+ let s = store_with(&[v.clone()]);+ let codebook = s.codebook();+ let codes = s.codes_for("0").unwrap();+ assert_eq!(codes, vec![0, 0, 0]);+ let decoded = codebook.decode(codes);+ assert_eq!(decoded, v);+ }++ #[test]+ fn roundtrip_within_one_step() {+ let vectors: Vec<Vec<f32>> = (0..20)+ .map(|i| {+ (0..4)+ .map(|d| ((i * 37 + d * 101) % 199) as f32 / 99.0 - 1.0)+ .collect()+ })+ .collect();+ let s = store_with(&vectors);+ let codebook = s.codebook();+ for (i, v) in vectors.iter().enumerate() {+ let decoded = codebook.decode(s.codes_for(&i.to_string()).unwrap());+ let step: f32 = codebook.scales.iter().cloned().sum();+ let err: f32 = v.iter().zip(decoded).map(|(a, b)| (a - b).abs()).sum();+ assert!(+ err < step,+ "doc {i}: |err| {err} must be < one total step {step}"+ );+ }+ }++ #[test]+ fn out_of_range_vector_extends_rather_than_clamps() {+ // #371's defect class: a corpus whose dim 0 never leaves +1.0, then a+ // document rewritten to -1.0. A clamping (stale) codebook decodes+ // -1.0 back to +1.0; a widening one decodes it near -1.0.+ let mut s = store_with(&[vec![1.0, 0.0], vec![1.0, 0.5]]);+ let before_refits = s.refits();+ assert!(s.upsert("0", &[-1.0, 0.0]));+ assert_eq!(s.refits(), before_refits + 1, "must have widened");+ let decoded = s.codebook().decode(s.codes_for("0").unwrap());+ assert!(+ (decoded[0] - (-1.0)).abs() < 2.0 / 255.0,+ "dim 0 decoded to {} — the codebook clamped (#371 defect class)",+ decoded[0]+ );+ // The untouched dimension-1 values of the OTHER document survive the+ // re-encode within a step.+ let d1 = s.codebook().decode(s.codes_for("1").unwrap());+ assert!((d1[0] - 1.0).abs() < 2.0 / 255.0);+ assert!((d1[1] - 0.5).abs() < 2.0 / 255.0);+ }++ #[test]+ fn update_rewrites_the_slot() {+ let mut s = store_with(&[vec![0.0, 1.0], vec![1.0, 0.0]]);+ let codebook_before = s.codebook();+ let old = codebook_before.decode(s.codes_for("0").unwrap());+ assert_eq!(old, vec![0.0, 1.0]);+ assert!(s.upsert("0", &[1.0, 1.0]));+ // Both dims' ranges now [0,1]; doc 0 re-encoded in place.+ assert_eq!(s.live_len(), 2, "update must not grow the live set");+ let decoded = s.codebook().decode(s.codes_for("0").unwrap());+ assert!((decoded[0] - 1.0).abs() < 1.0 / 255.0);+ assert!((decoded[1] - 1.0).abs() < 1.0 / 255.0);+ }++ #[test]+ fn remove_breaks_coverage_and_delete_reinsert_restores_it() {+ let mut s = store_with(&[vec![0.0], vec![1.0]]);+ assert!(s.coverage_ok());+ assert!(s.remove("0"));+ assert_eq!(s.live_len(), 1);+ assert!(s.coverage_ok(), "remove decrements both sides");+ assert!(!s.remove("0"), "second remove is a no-op");+ assert!(s.upsert("0", &[0.5]));+ assert!(s.coverage_ok());+ assert_eq!(s.live_len(), 2);+ }++ #[test]+ fn wrong_dim_attempt_breaks_coverage() {+ let mut s = store_with(&[vec![0.0, 0.0]]);+ assert!(!s.upsert("bad", &[0.0, 0.0, 0.0]), "mismatch rejected");+ assert_eq!(s.live_len(), 1);+ assert_eq!(s.expected(), 2);+ assert!(+ !s.coverage_ok(),+ "a rejected vector must keep consumers on exact paths"+ );+ assert!(s.codes_for("bad").is_none());+ }++ #[test]+ fn codes_bytes_is_one_byte_per_dim_per_slot() {+ let s = store_with(&[vec![0.0; 8], vec![1.0; 8], vec![0.5; 8]]);+ assert_eq!(s.codes_bytes(), 3 * 8);+ }++ #[test]+ fn iter_live_skips_removed_and_addresses_by_slot() {+ let mut s = store_with(&[vec![0.0, 0.1], vec![0.2, 0.3]]);+ assert!(s.remove("0"));+ let live: Vec<&str> = s.iter_live().map(|(id, _)| id).collect();+ assert_eq!(live, vec!["1"]);+ let (_, codes) = s.iter_live().next().unwrap();+ assert_eq!(codes.len(), 2);+ }++ #[test]+ fn codebook_is_order_independent() {+ // The final range spans the same values regardless of ingest order,+ // so a restart rebuild produces the same codebook (codes may differ+ // in the last bit through decode/encode re-encode, the ranges do not).+ let a: Vec<Vec<f32>> = vec![vec![0.0, 1.0], vec![-2.0, 0.5], vec![3.0, -1.0]];+ let mut b = a.clone();+ b.reverse();+ let sa = store_with(&a);+ let sb = store_with(&b);+ assert_eq!(sa.codebook().mins, sb.codebook().mins);+ assert_eq!(sa.codebook().scales, sb.codebook().scales);+ }+}diff --git a/recipes/vector_quantization.py b/recipes/vector_quantization.pyindex 5ee7a4f6a..250b2c2ab 100644--- a/recipes/vector_quantization.py+++ b/recipes/vector_quantization.py@@ -4,8 +4,16 @@XERJ stores dense vectors at full float32 precision by default. Opting a`dense_vector` field into scalar8 quantization (`index_options.type:int8_hnsw`) makes the kNN *serving* path score against 1-byte-per-dimension-codes instead of 4-byte floats — the recall profile of int8, with-almost no recall loss. `_source` still returns the original vectors.+codes instead of 4-byte floats, with almost no recall loss. `_source` still+returns the original vectors.++Since issue #392 the codes are written at INGEST time into a flat,+slot-addressed u8 array and the kNN scan scores against them, so the+serving working set IS the 1-byte-per-dim codes rather than the f32+`_source` vectors. The run below measures the resident array straight off+the server (`GET /{index}/_stats` → `primaries.sq8.fields.v.codes_bytes`).+`_source` keeps the f32 originals, so this is the scoring working set that+shrinks 4x — not total process memory.This recipe embeds the 40 real KB articles (demo/data/ai_kb.ndjson) into128-dim vectors with a small deterministic feature-hasher (same idea as@@ -15,19 +23,18 @@ shows that:1. kNN returns the same top results from both,2. recall@10 of the quantized index vs the exact index stays >= 0.90,- 3. what the int8 encoding costs vs float32 (1 vs 4 bytes/dim).--NOTE: this changes PRECISION, not memory. XERJ reads the full-precision-vector from `_source` and quantizes it per query, so `scalar8` does not-shrink the resident vector working set today — see issue #392. The-footprint line below is the size of the ENCODING, not a saving XERJ-currently realises.+ 3. the resident SQ8 code array the server actually holds — measured+ from `_stats` (1 byte/dim/live doc), alongside the encoding-size+ comparison measured by encoding every vector both ways client-side.Usage:xerj --insecure --data-dir ./data & # start XERJpython3 recipes/vector_quantization.py- XERJ_URL (default http://localhost:9200)+Environment:+ XERJ_URL server URL (default http://localhost:9200)+ XERJ_KB path to ai_kb.ndjson (default: auto-discovered by walking up+ from this script to the repo's demo/data/ai_kb.ndjson)"""import hashlib@@ -35,16 +42,34 @@ import jsonimport mathimport osimport pathlib+import structimport urllib.errorimport urllib.requestXERJ = os.environ.get("XERJ_URL", "http://localhost:9200")-KB = pathlib.Path(__file__).resolve().parent.parent / "demo" / "data" / "ai_kb.ndjson"DIM = 128NONE_INDEX = "vq-none"SQ8_INDEX = "vq-scalar8"+def _find_kb():+ """Locate demo/data/ai_kb.ndjson robustly, regardless of where this copy+ of the script lives (docs/examples/... or recipes/...). Honours XERJ_KB."""+ env = os.environ.get("XERJ_KB")+ if env:+ return pathlib.Path(env)+ here = pathlib.Path(__file__).resolve()+ for base in (here.parent, *here.parents):+ cand = base / "demo" / "data" / "ai_kb.ndjson"+ if cand.exists():+ return cand+ # Fall back to the original relative guess so the error message is useful.+ return here.parent.parent / "demo" / "data" / "ai_kb.ndjson"+++KB = _find_kb()++def call(method, path, body=None):"""One tiny HTTP helper instead of a client library."""data = None@@ -108,6 +133,17 @@ def knn(name, qv, k=10):return [(h["_id"], h["_score"]) for h in r["hits"]["hits"]]+def encode_f32(vec):+ """Exact wire size of a full-precision vector: 4 bytes per dimension."""+ return struct.pack(f"<{len(vec)}f", *vec)+++def encode_i8(vec):+ """Scalar8 code: symmetric int8 quantization → 1 byte per dimension.+ (Cosine vectors are already L2-normalised, so |x| <= 1.)"""+ return bytes((max(-127, min(127, round(x * 127))) & 0xFF) for x in vec)++def main():# ── 1. Embed the real KB into 128-dim vectors. ───────────────────────docs = []@@ -143,12 +179,45 @@ def main():hits += len(exact & approx)total += len(exact)recall = hits / total if total else 0.0++ # ── 5. Measure the RESIDENT codes the server holds (#392). ──────────+ # The ingest-time, slot-addressed SQ8 array is server state — read it+ # back from _stats rather than stipulating its size.+ stats = call("GET", f"/{SQ8_INDEX}/_stats")+ field_stats = stats["indices"][SQ8_INDEX]["primaries"]["sq8"]["fields"]["v"]+ codes_bytes = field_stats["codes_bytes"]+ live = field_stats["live"]++ # ── 6. Measure the real byte size of each ENCODING (client side). ────+ # Encode every corpus vector both ways and compare the actual byte totals+ # — a genuine measurement, not a stipulated 4x.+ f32_total = sum(len(encode_f32(d["v"])) for d in docs)+ i8_total = sum(len(encode_i8(d["v"])) for d in docs)+ ratio = f32_total / i8_total if i8_total else 0.0+ f32_per = f32_total // len(docs)+ i8_per = i8_total // len(docs)+print(f"recall@10 (scalar8 vs float32 ground truth): {recall:.3f}")- print(f"encoding size: float32 = {DIM * 4} B/vec → scalar8 = {DIM} B/vec (4x smaller)")+ print(+ f"resident SQ8 codes (from _stats, {live} live docs): "+ f"{codes_bytes} B ({codes_bytes // max(live, 1)} B/vec) — "+ f"serving={field_stats['serving']}"+ )+ print(+ f"encoding size over {len(docs)} vecs: "+ f"float32 = {f32_total} B ({f32_per} B/vec) → "+ f"scalar8 = {i8_total} B ({i8_per} B/vec) ({ratio:.2f}x smaller)"+ )if recall < 0.90:raise SystemExit(f"FAIL: recall {recall:.3f} < 0.90")- print("\nOK — recall preserved through 1-byte-per-dim codes. `_source` still holds")- print("the originals. scalar8 changes precision, not resident memory (issue #392).")+ if not field_stats["serving"] or not field_stats["ready"]:+ raise SystemExit("FAIL: SQ8 code store not serving after ingest (#392)")+ if codes_bytes != len(docs) * DIM:+ raise SystemExit(+ f"FAIL: codes_bytes {codes_bytes} != {len(docs)} docs x {DIM} dim"+ )+ print("\nOK — recall preserved through 1-byte-per-dim codes, served from the")+ print("ingest-time slot-addressed array (#392). `_source` still holds the originals.")if __name__ == "__main__":