Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

knn: scalar8 codes written at ingest, slot-addressed (#392)

xerj-org46b58eef99fe
diff --git a/ROADMAP.md b/ROADMAP.md
index e9861550f..e31f0e69d 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -106,7 +106,7 @@ The 1.0 bar: **every public claim verified against the release binary, and every
**Zero-token** is the name for work XERJ does locally, inside the engine, so that a person or an agent spends no model tokens *finding*, *judging*, *watching for* or *handing over* an answer. Search already works that way — no model runs to answer a query. This section is the rest of that idea, as product work (long form: [docs/ZERO_TOKEN_DIRECTION.md](./docs/ZERO_TOKEN_DIRECTION.md)), with each item's status checked against `main` on 2026-09-18 (`4d8dadbf`, 32 commits after `v1.0.0-rc.74`) by reading the code named beside it. "In flight" means a pushed branch exists and nothing of it is on `main`; "planned" means no code exists. Tracking issue: [#941](https://github.com/xerj-org/xerj/issues/941).
-**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` now):
+**Stage 1 — all of it merged and released in rc.75** (each landed as its own PR; the status lines below say what is on `main` as of v1.0.0-rc.75):
- **Judged search — a rerank stage.** Retrieve with BM25 or `hybrid`, then have a second stage re-judge the top *N* before the page is returned. *On `main` since rc.75:* the opt-in `rerank` block on `_search`, inert until an operator configures a key; a LOCAL cross-encoder provider was measured and did NOT beat the shipped hybrid (SciFact 0.7021 hybrid vs 0.7186 local-base, interval spanning zero; NFCorpus 0.3445 vs 0.3597 local-small), so it stays opt-in and unmerged. *Before rc.75:* the ES `rescore` block (query rescorer) and the `hybrid` query type with `rrf|linear`; there is no rerank stage. *In flight:* branch `feat/rerank-stage` — a `rerank` block on `_search` that hands the top hits to an external relevance judge, opt-in per request and inert until an operator configures a key. It is the only *search-time* feature that would send document text off the machine — proxy embeddings (`[embedding] default_endpoint`) send it at write time and the WAL tap replays writes to an external `_bulk` endpoint, both operator-configured and off by default ([every outbound connection](./docs/RERANK.md#every-way-data-leaves-a-xerj-node)) — which is exactly why a *local* judge is on this list. *The bar for a local model is a measured one.* On the BEIR test splits with the opt-in neural embedder (`--embed-mode neural`, all-MiniLM-L6-v2, CPU) the shipped `hybrid` RRF query is already the best arm we have — nDCG@10 **0.699–0.704 on SciFact and 0.345 on NFCorpus** over three runs, against 0.657 / 0.302 for BM25 alone, 0.676 / 0.329 for vectors alone, and 0.686 / 0.332 for "BM25 top-30 re-ordered by the same bi-encoder", which is *worse* than hybrid on both. So re-ordering with the embedder we already ship is not worth building, and **a local cross-encoder ships only if it beats 0.699 / 0.345 on the same harness, by more than the run-to-run spread** ([`benchmarks/neural-path-triage/`](./benchmarks/neural-path-triage/)). Those figures are with the neural embedder; the default embedder is lexical feature hashing and was not part of this run. The hybrid figure is the only arm that does not reproduce — 0.6993, 0.7023 and 0.7044 on SciFact across three runs of unchanged indices — because tied RRF scores are ordered by a per-process hash seed ([#940](https://github.com/xerj-org/xerj/issues/940)); a gain inside that spread is not a gain.
- **Share links and a guest reading room.** Hand one person a link to one corpus or one document, read-only, without creating them an account. *On `main` since rc.75:* `xerj share` with scoped, expiring links, a guest reading room, and a `--tunnel` mode that names Cloudflare as a reader of the traffic. *Before rc.75:* nothing — there was no share or guest code in the tree, and reading a corpus needs an account session or an API key. *In flight:* branches `feat/share-links` (scoped, expiring links; a share can never name a system index) and `feat/console-corpus-reader` (the corpus browser and the reader the link opens). Both branches carry commits their authors marked unverified; they are not claims yet.
diff --git a/demo/playbooks/ES_COMPATIBILITY.md b/demo/playbooks/ES_COMPATIBILITY.md
index 8e28f360e..8d2e08744 100644
--- a/demo/playbooks/ES_COMPATIBILITY.md
+++ b/demo/playbooks/ES_COMPATIBILITY.md
@@ -384,6 +384,6 @@ The sections above predate the **rc-3** release and the **RC4** hardening pass.
- **`percolate`** — a real reverse-search doc-scan (each indexed doc stores a query; the supplied `document`/`documents` are matched against it), replacing the former hard-coded `MatchNone`.
- **`script_fields`** — real per-hit Painless evaluation with `index.max_script_fields` enforcement (was a silent `null`, fixed `65ce844`). Relatedly, **`scripted_metric`** now runs a real single-shard `init`/`map`/`combine`/`reduce` interpreter, and **`significant_terms`** computes real JLH significance scoring — both previously empty/`null`.
- **`has_child` / `has_parent`** — now **fail loud (400)** at parse time instead of returning unfiltered flat hits.
-- **HNSW-served kNN** — unfiltered, cosine, full-precision kNN on indexes ≥1,024 docs is now served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the bench query; filtered/nested/SQ8/small-index shapes keep the exact brute-force scan). **`scalar8` (`int8_hnsw`)** has a real serving path — scores are computed from 1-byte-per-dimension codes, not from the f32 vector (recall@10 ≈ 0.998, measured) — but it buys **precision, not memory**: the scan reads the full-precision vector from `_source` and quantizes it per query, so nothing gets smaller in RAM. The earlier "~4× smaller vector working set" claim described caches that [#371](https://github.com/xerj-org/xerj/issues/371) removed — both the per-document code map and the per-field codebook were write-once, and between them kept scoring updated documents from stale data. The ingest-time, ordinal-addressed code array that would make a memory claim true is tracked in [#392](https://github.com/xerj-org/xerj/issues/392), which also covers the two consequences of fitting the codebook per query: `scalar8` still disqualifies a field from HNSW-served kNN, and a `scalar8` `_score` varies with the candidate set (bounded by SQ8's own 1/255 quantization step) where Elasticsearch's, fixed per segment at index time, does not. `binary` (1-bit) is still rejected at startup.
+- **HNSW-served kNN** — unfiltered, cosine, full-precision kNN on indexes ≥1,024 docs is now served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the bench query; filtered/nested/SQ8/small-index shapes keep the exact brute-force scan). **`scalar8` (`int8_hnsw`)** has a real serving path — scores are computed from 1-byte-per-dimension codes written at INGEST time into a flat, slot-addressed u8 array ([#392](https://github.com/xerj-org/xerj/issues/392); recall@10 ≈ 0.998, measured), so the kNN scoring working set is the codes (1 byte/dim) instead of the f32 `_source` vectors (4 bytes/dim) — the "~4× smaller vector working set" claim is true of that path now, observable as `primaries.sq8.fields.<f>.codes_bytes` in `_stats`. Total resident memory does not shrink 4× (`_source` keeps the f32 originals), and Elasticsearch still differs in mechanism (per-segment quantization at merge time, codes on disk). The codebook is fitted from the ingested vectors and widened with a re-encode when a value lands outside it — never clamped, which is the defect class [#371](https://github.com/xerj-org/xerj/issues/371) fixed by removing the write-once caches. A `scalar8` `_score` is now a function of index state alone and no longer varies with the candidate set; the one #392 consequence that remains open: `scalar8` still disqualifies a field from HNSW-served kNN. `binary` (1-bit) is still rejected at startup.
Still **not** compatible (unchanged, single-node by design): true multi-node sharding/replication, CCR, and full ML. The float-to-integer truncation and scored-`hits.total` defects flagged in §1/§8 are tracked separately — this addendum does not claim them closed. See ROADMAP.md and STUB_AUDIT.md for the full tracked list.
diff --git a/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md b/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md
index 7c7968eeb..7719adeff 100644
--- a/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md
+++ b/demo/playbooks/USER_FEEDBACK_SCORECARD_RC4.md
@@ -92,7 +92,7 @@ These are the strategic decisions for rc5, stated without spin.
- `role_descriptors` accepted then ignored — every API key is effectively superuser.
- ~~No `cargo-audit`/`cargo-deny` gate (528 unscanned dep crates) and no `cargo-fuzz` harnesses.~~
**CLOSED 2026-08-09 (#207):** CI job `security-audit` runs `cargo audit` on every push and PR, and job `fuzz` builds and runs the libFuzzer harnesses in `engine/fuzz/` (ES DSL, `query_string`, date math, index-name date math, SQL, Painless, and the aggregation script tokenisers). `cargo-deny` is still absent.
-- Binary/RaBitQ (BBQ) quantization unimplemented; `bbq_*`/`int4_*` mappings silently keep full f32 (no memory saving) — HNSW RAM wall NOT mitigated by SQ8 either — `scalar8` changes scoring precision, not resident memory (the scan quantizes from `_source` per query); see [#392](https://github.com/xerj-org/xerj/issues/392). No DiskANN.
+- Binary/RaBitQ (BBQ) quantization unimplemented; `bbq_*`/`int4_*` mappings silently keep full f32 (no memory saving) — HNSW RAM wall NOT mitigated by SQ8 either: `scalar8` kNN scores from ingest-time 1-byte/dim codes ([#392](https://github.com/xerj-org/xerj/issues/392)), which shrinks the scoring working set ~4×, but `_source` keeps the f32 originals and the HNSW graph still holds f32. No DiskANN.
**Marketing numbers to reconcile with measured reality (honesty-posture debt):**
`<500MB RSS / 1M docs` (measured 596MB / 200k) · `~30MB Docker image` (~75–100MB) · `2–5× compression` (~1.6×) · `2 files/segment` (~4) · `TLS enabled out of the box` (off by default) · `OpenAPI generated from code` (static hand-maintained, 404 live) · `no deep-pagination cliff` (identical 10k cap) · `cost-based planner` (rule/structure-based) · error links use `xerj.io` vs canonical `xerj.org`.
diff --git a/docs/examples/vector-quantization/quant_demo.py b/docs/examples/vector-quantization/quant_demo.py
index b2dd31e3d..a348d08b3 100644
--- a/docs/examples/vector-quantization/quant_demo.py
+++ b/docs/examples/vector-quantization/quant_demo.py
@@ -7,11 +7,13 @@ int8_hnsw`) makes the kNN *serving* path score against 1-byte-per-dimension
codes instead of 4-byte floats, with almost no recall loss. `_source` still
returns the original vectors.
-NOTE: this changes PRECISION, not memory. XERJ reads the full-precision
-vector from `_source` and quantizes it per query, so `scalar8` does not
-shrink the resident vector working set today — see issue #392. The footprint
-number printed below is the size of the int8 ENCODING, not a saving XERJ
-currently realises.
+Since issue #392 the codes are written at INGEST time into a flat,
+slot-addressed u8 array and the kNN scan scores against them, so the
+serving working set IS the 1-byte-per-dim codes rather than the f32
+`_source` vectors. The run below measures the resident array straight off
+the server (`GET /{index}/_stats` → `primaries.sq8.fields.v.codes_bytes`).
+`_source` keeps the f32 originals, so this is the scoring working set that
+shrinks 4x — not total process memory.
This recipe embeds the 40 real KB articles (demo/data/ai_kb.ndjson) into
128-dim vectors with a small deterministic feature-hasher (same idea as
@@ -21,9 +23,9 @@ shows that:
1. kNN returns the same top results from both,
2. recall@10 of the quantized index vs the exact index stays >= 0.90,
- 3. what the int8 encoding costs vs float32 — MEASURED by actually
- encoding every vector both ways and comparing the real byte totals
- (not a hardcoded ratio). See the NOTE above on what this is not.
+ 3. the resident SQ8 code array the server actually holds — measured
+ from `_stats` (1 byte/dim/live doc), alongside the encoding-size
+ comparison measured by encoding every vector both ways client-side.
Usage:
xerj --insecure --data-dir ./data & # start XERJ
@@ -178,11 +180,17 @@ def main():
total += len(exact)
recall = hits / total if total else 0.0
- # ── 5. Measure the real byte size of each ENCODING. ──────────────────
+ # ── 5. Measure the RESIDENT codes the server holds (#392). ──────────
+ # The ingest-time, slot-addressed SQ8 array is server state — read it
+ # back from _stats rather than stipulating its size.
+ stats = call("GET", f"/{SQ8_INDEX}/_stats")
+ field_stats = stats["indices"][SQ8_INDEX]["primaries"]["sq8"]["fields"]["v"]
+ codes_bytes = field_stats["codes_bytes"]
+ live = field_stats["live"]
+
+ # ── 6. Measure the real byte size of each ENCODING (client side). ────
# Encode every corpus vector both ways and compare the actual byte totals
- # — a genuine measurement, not a stipulated 4x. It is a measurement of the
- # encodings, though, NOT of XERJ's resident footprint: the serving path
- # reads the f32 vector from `_source` and quantizes per query (issue #392).
+ # — a genuine measurement, not a stipulated 4x.
f32_total = sum(len(encode_f32(d["v"])) for d in docs)
i8_total = sum(len(encode_i8(d["v"])) for d in docs)
ratio = f32_total / i8_total if i8_total else 0.0
@@ -190,6 +198,11 @@ def main():
i8_per = i8_total // len(docs)
print(f"recall@10 (scalar8 vs float32 ground truth): {recall:.3f}")
+ print(
+ f"resident SQ8 codes (from _stats, {live} live docs): "
+ f"{codes_bytes} B ({codes_bytes // max(live, 1)} B/vec) — "
+ f"serving={field_stats['serving']}"
+ )
print(
f"encoding size over {len(docs)} vecs: "
f"float32 = {f32_total} B ({f32_per} B/vec) → "
@@ -197,8 +210,14 @@ def main():
)
if recall < 0.90:
raise SystemExit(f"FAIL: recall {recall:.3f} < 0.90")
- print("\nOK — recall preserved through 1-byte-per-dim codes. `_source` still holds")
- print("the originals. scalar8 changes precision, not resident memory (issue #392).")
+ if not field_stats["serving"] or not field_stats["ready"]:
+ raise SystemExit("FAIL: SQ8 code store not serving after ingest (#392)")
+ if codes_bytes != len(docs) * DIM:
+ raise SystemExit(
+ f"FAIL: codes_bytes {codes_bytes} != {len(docs)} docs x {DIM} dim"
+ )
+ print("\nOK — recall preserved through 1-byte-per-dim codes, served from the")
+ print("ingest-time slot-addressed array (#392). `_source` still holds the originals.")
if __name__ == "__main__":
diff --git a/docs/recipes/README.md b/docs/recipes/README.md
index c9f926134..38a5392fe 100644
--- a/docs/recipes/README.md
+++ b/docs/recipes/README.md
@@ -16,7 +16,7 @@ These are practical "how do I actually do X" recipes, not an API reference (that
| [Passage retrieval on long docs](./passage-retrieval.md) | `semantic_text` auto-embeds every overlapping passage; a long doc competes on any one of its sections via best-passage (max-sim) scoring — 98% top-3 vs 32% pooled | [`passage_demo.py`](../examples/passage-retrieval/passage_demo.py) |
| [Log analytics](./log-analytics.md) | From raw logs to dashboards — error rates, p95 latency, top services via aggregations | [`log_analytics.py`](../examples/log-analytics/log_analytics.py) |
| [Vector search (kNN)](./vector-search-knn.md) | Nearest-neighbor similarity search over `dense_vector` (HNSW), with filters | [`knn_demo.py`](../examples/vector-search-knn/knn_demo.py) |
-| [Vector quantization](./vector-quantization.md) | Opt a `dense_vector` field into scalar8 (`int8_hnsw`) — kNN scored from 1-byte-per-dim codes at recall@10 ≈ 0.998 (precision, **not** memory: see [#392](https://github.com/xerj-org/xerj/issues/392)) | [`quant_demo.py`](../examples/vector-quantization/quant_demo.py) |
+| [Vector quantization](./vector-quantization.md) | Opt a `dense_vector` field into scalar8 (`int8_hnsw`) — kNN scored from ingest-time 1-byte-per-dim codes at recall@10 ≈ 0.998; the scoring working set is 4× smaller than the f32 vectors (see [#392](https://github.com/xerj-org/xerj/issues/392)) | [`quant_demo.py`](../examples/vector-quantization/quant_demo.py) |
| [Hybrid search](./hybrid-search.md) | Keyword + vector in one query — results neither BM25 nor kNN finds alone | [`hybrid_search.py`](../examples/hybrid-search/hybrid_search.py) |
| [Anomaly detection](./anomaly-detection.md) | Statistical `_ml` detectors that flag spikes in metrics/logs | [`anomaly_detection.py`](../examples/anomaly-detection/anomaly_detection.py) |
| [Continuous anomaly datafeeds](./continuous-anomaly-datafeeds.md) | A live `_ml` datafeed that re-scores an index on a timer and stores new anomaly records you poll | [`datafeed_demo.py`](../examples/continuous-anomaly-datafeeds/datafeed_demo.py) |
diff --git a/docs/recipes/vector-quantization.md b/docs/recipes/vector-quantization.md
index 409ee4005..1c1553b20 100644
--- a/docs/recipes/vector-quantization.md
+++ b/docs/recipes/vector-quantization.md
@@ -1,13 +1,15 @@
# Vector quantization: int8 scoring at recall@10 ≈ 0.998
-> **Read this first.** `scalar8` on XERJ today changes **precision, not
-> memory**. Scores are computed from 1-byte-per-dimension codes, so the field
-> has the recall profile of int8 — but the serving path still reads the
-> full-precision vector out of `_source` and quantizes it per query, so
-> nothing gets smaller in RAM. The ingest-time code array that would make it
-> a memory win is tracked in
-> [#392](https://github.com/xerj-org/xerj/issues/392). If you came here for a
-> smaller working set, that issue is the one to watch, not this recipe.
+> **Read this first.** `scalar8` scores come from 1-byte-per-dimension
+> codes written at INGEST time into a flat, slot-addressed u8 array
+> ([#392](https://github.com/xerj-org/xerj/issues/392)) — the kNN serving
+> working set is those codes (1 byte/dim), not the f32 `_source` vectors
+> (4 bytes/dim), and the score a document gets is a function of the index
+> state alone, not of the filter or candidate set of the query that found it.
+> `_source` keeps the original f32 vectors, so this shrinks the scoring
+> working set ~4×, not total process memory. When the code store cannot
+> serve (right after open, mid-rebuild, or after a wrong-dimension write
+> breaks coverage), queries fall back to the exact `_source` scan.
## The problem
@@ -34,10 +36,13 @@ On a real 128-dim corpus the recall cost is negligible: **recall@10 = 0.998**
against the exact float32 index. That number is computed by the run below,
not stipulated.
-What that costs in bytes *as an encoding* is 128 rather than 512 per vector,
-and the run measures that too — but see the note at the top: XERJ does not
-hold those codes resident today, so treat it as the size of the encoding, not
-as a saving you get.
+The codes are 128 bytes rather than 512 per vector, and since #392 they are
+held resident by the engine: the run reads the live array's byte size
+straight off the server (`GET /{index}/_stats` →
+`primaries.sq8.fields.<field>.codes_bytes`) and checks it is exactly
+`docs × dim`. The kNN scan scores against that array without touching the f32
+vectors — the ~4× smaller vector working set is a property of the serving
+path now, measured, not a ratio the encoding merely implies.
## The solution
@@ -95,22 +100,23 @@ query: 'how do I stop an agent's context window from overflowing?'
── scalar8 (quantized)
0.67938 Long-context windows do not replace memory
- 0.60021 p95 latency budgets for interactive RAG agents
- 0.59731 SOC 2 controls that apply to vector workloads
+ 0.59983 p95 latency budgets for interactive RAG agents
+ 0.59766 SOC 2 controls that apply to vector workloads
recall@10 (scalar8 vs float32 ground truth): 0.998
+resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=True
encoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)
-OK — recall preserved through 1-byte-per-dim codes. `_source` still holds
-the originals. scalar8 changes precision, not resident memory (issue #392).
+OK — recall preserved through 1-byte-per-dim codes, served from the
+ingest-time slot-addressed array (#392). `_source` still holds the originals.
```
-The encoding-size line is a real measurement: the run encodes every corpus
-vector as float32 bytes (`struct`) and as int8 codes and compares the actual
-byte totals — 20480 B vs 5120 B, exactly 4.00×. It is the cost of the two
-*encodings*, measured in the client. It is **not** a measurement of XERJ's
-resident footprint, and XERJ does not currently realise it as one
-([#392](https://github.com/xerj-org/xerj/issues/392)).
+Both size lines are real measurements. The client-side one encodes every
+corpus vector as float32 bytes (`struct`) and as int8 codes and compares the
+byte totals — 20480 B vs 5120 B, exactly 4.00×. The `resident SQ8 codes` line
+is read off the server (`_stats` → `primaries.sq8.fields.v.codes_bytes`) and
+matches it byte-for-byte: 40 docs × 128 dim = 5120 B, `serving=True` — the
+array the kNN scan actually scores against ([#392](https://github.com/xerj-org/xerj/issues/392)).
## Reproduce it yourself
@@ -127,6 +133,7 @@ overrides the KB path (default: auto-discovered `demo/data/ai_kb.ndjson`).
The embedder and corpus are deterministic, so a customer should see exactly:
- `recall@10 (scalar8 vs float32 ground truth): 0.998`
+- `resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=True`
- `encoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)`
These numbers are stable run-to-run (verified across repeated runs — no
@@ -142,41 +149,41 @@ variance); the printed kNN scores are likewise identical each run.
is rejected at startup rather than silently storing full precision.
- **Cosine is normalised** before quantizing for the tightest code range;
`dot_product` and `l2_norm` similarities are supported too.
-- **No memory saving yet.** The serving path reads the full-precision vector
- from `_source` and quantizes it per query, so `scalar8` costs the same RAM
- as float32 and a little more CPU. Tracked in
+- **The serving working set shrinks ~4×; total memory does not.** The kNN
+ scan scores against the ingest-time code array (1 byte/dim) instead of the
+ f32 `_source` vectors (4 bytes/dim), but `_source` keeps the originals for
+ retrieval, so the process's total footprint does not fall 4×. The array
+ itself is observable: `GET /{index}/_stats` → `primaries.sq8` (per field:
+ `dim`, `live`, `expected`, `covered`, `ready`, `serving`, `refits`,
+ `codes_bytes`).
+- **Codes are written at ingest and rewritten on update.** One slot per
+ document, addressed `slot * dim`, tombstoned on delete — the same slot
+ discipline the HNSW slab uses. A vector that lands outside the fitted
+ per-dimension range WIDENS the codebook and re-encodes every live code
+ (decode → encode, each value moves by at most the new quantization step);
+ nothing is ever clamped, which is the defect class
+ [#371](https://github.com/xerj-org/xerj/issues/371) fixed by deleting the
+ old write-once caches.
+- **`_score` no longer depends on the filter.** The codebook is fitted from
+ the ingested vectors and lives with the data, so the same document carries
+ the same `_score` whether or not a filter removed other documents — the
+ property Elasticsearch has and per-query fitting lacked. Pinned over HTTP in
+ `engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs`
+ (drift is exactly 0, not merely small).
+- **Fallback windows keep the exact scan.** Until the open-time walk converges
+ (a restart re-derives the store in the background — WAL replay never
+ re-runs vector indexing), while a publication race is possible, or after a
+ wrong-dimension write breaks the store's coverage gate, queries are served
+ by the exact `_source` scan. In that fallback the SQ8 branch still fits its
+ codebook per query over the candidates being scored, so scores there move
+ by up to SQ8's own quantization step (1/255 of the fitted per-dimension
+ range) when the candidate set changes — measured on the pre-#392 path:
+ max |Δ_score| 1.976e-05 with order swaps at 19/30 positions when an
+ unrelated-only filter was added. That fallback is correct, just not
+ filter-independent; `primaries.sq8.fields.<f>.serving` tells you which
+ path you are on.
+- **`scalar8` still disables ANN.** A quantized field remains excluded from
+ HNSW-served kNN (the graph scores f32) and takes the codes scan above for
+ unfiltered queries or the exact brute-force scan otherwise. Teaching the
+ graph to traverse against the codes is the remaining open half of
[#392](https://github.com/xerj-org/xerj/issues/392).
-- **`scalar8` disables ANN.** A quantized field is excluded from HNSW-served
- kNN and always takes the exact brute-force scan, so it is slower, not
- faster, than leaving the field full-precision. Also [#392](https://github.com/xerj-org/xerj/issues/392).
-- **A `scalar8` `_score` — and the order it produces — depends on the
- candidate set.** The codebook is fitted per query over the candidates being
- scored, which is what keeps an updated document from being scored on a stale
- codebook ([#371](https://github.com/xerj-org/xerj/issues/371)). Three things
- follow, all of them measured, none of them true of a full-precision field:
-
- - **A `filter` reorders the documents it does not remove.** On a 60-document
- 4-dim cosine corpus (30 tightly clustered `grp:a` documents, 30 spread
- `grp:b` ones), adding `filter: {"term": {"grp": "a"}}` — which removes only
- `grp:b` — returned the same 30 documents with a maximum `_score` difference
- of **1.976e-05**, but in a **different order at 19 of the 30 positions**.
- The score delta is tiny; the reordering is what a caller actually sees.
- - **The trigger is the candidate set, not the `filter` keyword.** Indexing
- one more unrelated document moves existing documents too. Same corpus, one
- document added far from the query: maximum `_score` difference **7.100e-06**
- over the 30 pre-existing documents, and a reordered top 10.
- - **Unfiltered corpora over 1000 documents also score differently from
- v1.0.0-rc.17**, which fitted the codebook from the first ≤1000 candidates
- and then cached it for the life of the process. On 1500 documents, **all 40
- of the top-40 `_score` values changed** (maximum difference **4.880e-05**)
- and two adjacent ranks swapped. Below 1000 documents, unfiltered and never
- updated, scores are byte-identical to rc.17.
-
- Every individual difference is bounded by SQ8's own quantization step (1/255
- of the fitted per-dimension range), i.e. inside the approximation error
- `scalar8` already carries — but documents whose true scores are close will
- swap places, so treat `scalar8` ordering as stable only for a fixed corpus
- and a fixed filter. Elasticsearch fits its codebook per segment at index time
- and does not have this property;
- [#392](https://github.com/xerj-org/xerj/issues/392) is what would bring XERJ
- in line.
diff --git a/engine/crates/xerj-api/src/es_compat.rs b/engine/crates/xerj-api/src/es_compat.rs
index 712da8dc3..680cdc96e 100644
--- a/engine/crates/xerj-api/src/es_compat.rs
+++ b/engine/crates/xerj-api/src/es_compat.rs
@@ -20297,6 +20297,16 @@ pub async fn index_stats(
Ok(idx) => idx.hnsw_stats().await,
Err(_) => json!({ "present": false }),
};
+ // XERJ extension (issue #392): the ingest-time SQ8 code stores. `fields`
+ // is empty for indices with no `scalar8` mapping; per field it reports
+ // the dimension, live slot count, coverage denominator, the
+ // ready/serving gates, codebook refits, and the resident `codes_bytes`
+ // — the flat u8 array the slot addressing is for, observable instead
+ // of notional.
+ let sq8_stats = match state.engine.get_index(&index) {
+ Ok(idx) => json!({ "fields": idx.sq8_stats() }),
+ Err(_) => json!({ "fields": {} }),
+ };
// Merge status + real segment count so benchmarks can verify the
// index is merge-quiescent (merges.current == 0) before measuring.
let (seg_count, merges_current) = state
@@ -20326,6 +20336,7 @@ pub async fn index_stats(
"store": { "size_in_bytes": store_size_bytes },
"dense_vector": dense_vector_stats,
"hnsw": hnsw_stats,
+ "sq8": sq8_stats,
"mappings": { "schema_persist_failures": schema_persist_failures },
"indexing": {
// Real cumulative indexing counters (RC4 W4 item 1): plumbed from
diff --git a/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs b/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs
new file mode 100644
index 000000000..e89110c92
--- /dev/null
+++ b/engine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs
@@ -0,0 +1,378 @@
+//! Issue #392 — `scalar8` codes must be written at INGEST time, addressed by
+//! a dense slot, and scored against a codebook that is a property of the
+//! index rather than of the query's candidate set.
+//!
+//! Two observable consequences, each pinned here from the outside over real
+//! HTTP:
+//!
+//! * `sq8_codes_exist_at_ingest_before_any_query` — after indexing (and
+//! before a single kNN), `GET /{index}/_stats` reports a live, ready SQ8
+//! code store for the field. Before #392 nothing quantized anything at
+//! ingest: the serving path read each candidate's f32 vector out of
+//! `_source` and quantized it per query, so no such store existed and the
+//! stats section is absent (null).
+//!
+//! * `scores_do_not_depend_on_the_filter` — the same document must carry the
+//! same `_score` under two queries whose only difference is a filter that
+//! removes OTHER documents. Before #392 the SQ8 codebook was fitted per
+//! query over the candidates being scored, so shrinking the candidate set
+//! moved every surviving document's score (measured on the unfixed code:
+//! max |Δ_score| 1.976e-05 with order swaps at 19/30 positions — inside
+//! SQ8's advertised approximation error, but a real behavioural difference
+//! from Elasticsearch, where the codebook is fixed at index time).
+//!
+//! * `codes_survive_a_restart` — the code store is re-derived from the live
+//! documents when the index reopens (WAL replay never re-runs vector
+//! indexing), and once the rebuild lands the same query returns the same
+//! neighbours. A regression pin for the rebuild, not a fail-before: the
+//! per-query-fit path that preceded it also survives a restart.
+//!
+//! * `a_wrong_dimension_document_does_not_break_the_field` — a document
+//! whose vector has the wrong dimensionality breaks the store's coverage
+//! invariant; queries must then fall back to the exact path and still
+//! return correct results (the malformed document is skipped, exactly as
+//! the brute-force scan skips it).
+//!
+//! Elasticsearch is referenced for wire semantics only. It is AGPL-3.0/
+//! SSPL-1.0/Elastic-2.0 licensed and no code from it is reproduced here.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+const DIM: usize = 8;
+const INDEX: &str = "sq8";
+
+/// 25 corpus documents whose every dimension lies in `[0, 1)` — the documents
+/// both queries score. Deterministic prime-stride walk, all values distinct
+/// and not representable on both SQ8 grids (that is what makes the unfixed
+/// code's score drift observable rather than coincidentally zero).
+const KEEP: usize = 25;
+
+/// 5 documents carrying `tag: "wide"` whose dimension 0 sits far outside the
+/// keep documents' range. They widen the per-query-fitted codebook only when
+/// present, which is exactly the candidate-set dependency under test.
+const WIDE_IDS: [usize; 5] = [100, 101, 102, 103, 104];
+
+fn keep_vector(i: usize) -> Vec<f32> {
+ (0..DIM)
+ .map(|d| ((i * 37 + d * 101) % 199) as f32 / 199.0)
+ .collect()
+}
+
+fn wide_vector(i: usize) -> Vec<f32> {
+ let mut v = vec![0.0f32; DIM];
+ v[0] = 5.0 + i as f32;
+ for d in 1..DIM {
+ v[d] = ((i * 53 + d * 97) % 199) as f32 / 199.0;
+ }
+ v
+}
+
+fn doc(i: usize) -> Value {
+ if WIDE_IDS.contains(&i) {
+ json!({ "tag": "wide", "v": wide_vector(i) })
+ } else {
+ json!({ "tag": "keep", "v": keep_vector(i) })
+ }
+}
+
+async fn app() -> (axum::Router, tempfile::TempDir) {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ (xerj_api::router::build_es_compat_router(state), dir)
+}
+
+async fn send(app: &axum::Router, req: Request<Body>) -> (StatusCode, Value) {
+ let response = app.clone().oneshot(req).await.expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ let value: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null);
+ (status, value)
+}
+
+async fn json_req(
+ app: &axum::Router,
+ method: &str,
+ path: &str,
+ body: Value,
+) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+}
+
+async fn get(app: &axum::Router, path: &str) -> (StatusCode, Value) {
+ send(
+ app,
+ Request::get(path).body(Body::empty()).expect("request"),
+ )
+ .await
+}
+
+async fn create_and_index(app: &axum::Router) {
+ let (status, body) = json_req(
+ app,
+ "PUT",
+ &format!("/{INDEX}"),
+ json!({
+ "mappings": { "properties": {
+ "v": { "type": "dense_vector", "dims": DIM, "quantization": "scalar8" }
+ } }
+ }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "create: {body}");
+ for i in 0..KEEP {
+ let (status, body) = json_req(app, "PUT", &format!("/{INDEX}/_doc/{i}"), doc(i)).await;
+ assert!(status.is_success(), "index {i}: {status} {body}");
+ }
+ for &i in &WIDE_IDS {
+ let (status, body) = json_req(app, "PUT", &format!("/{INDEX}/_doc/{i}"), doc(i)).await;
+ assert!(status.is_success(), "index {i}: {status} {body}");
+ }
+ let (status, body) = json_req(app, "POST", &format!("/{INDEX}/_refresh"), json!({})).await;
+ assert!(status.is_success(), "refresh: {status} {body}");
+}
+
+/// The query vector: keep document 0's own vector. All keep documents score
+/// distinctly against it (deterministic corpus), so score identity between
+/// the two queries is not satisfiable by accident.
+fn query_vector() -> Vec<f32> {
+ keep_vector(0)
+}
+
+/// Run the kNN, returning `(id, _score)` per hit.
+async fn knn(app: &axum::Router, filter: Option<Value>) -> Vec<(String, f64)> {
+ let mut knn_body = json!({
+ "field": "v",
+ "query_vector": query_vector(),
+ "k": KEEP + WIDE_IDS.len(),
+ "num_candidates": 100
+ });
+ if let Some(f) = filter {
+ knn_body["filter"] = f;
+ }
+ let (status, body) = json_req(
+ app,
+ "POST",
+ &format!("/{INDEX}/_search"),
+ json!({ "knn": knn_body, "size": KEEP + WIDE_IDS.len(), "_source": false }),
+ )
+ .await;
+ assert_eq!(status, StatusCode::OK, "knn: {status} {body}");
+ body["hits"]["hits"]
+ .as_array()
+ .unwrap_or_else(|| panic!("no hits: {body}"))
+ .iter()
+ .map(|h| {
+ (
+ h["_id"].as_str().expect("_id").to_string(),
+ h["_score"].as_f64().expect("_score"),
+ )
+ })
+ .collect()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 1. Codes exist at ingest — before any query is served
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn sq8_codes_exist_at_ingest_before_any_query() {
+ let (app, _dir) = app().await;
+ create_and_index(&app).await;
+
+ // No kNN has run. If SQ8 codes are written at ingest, the per-index stats
+ // section reports them; on the per-query-fit path this whole section is
+ // absent because nothing exists to report.
+ let (status, body) = get(&app, &format!("/{INDEX}/_stats")).await;
+ assert_eq!(status, StatusCode::OK, "stats: {status} {body}");
+ let field = &body["indices"][INDEX]["primaries"]["sq8"]["fields"]["v"];
+ assert!(
+ field.is_object(),
+ "no ingest-time SQ8 code store in stats (issue #392): {body}"
+ );
+ assert_eq!(field["dim"], json!(DIM), "{body}");
+ assert_eq!(
+ field["live"],
+ json!(KEEP + WIDE_IDS.len()),
+ "every indexed document must hold a live code slot: {body}"
+ );
+ assert_eq!(
+ field["expected"],
+ json!(KEEP + WIDE_IDS.len()),
+ "coverage numerator and denominator must agree: {body}"
+ );
+ assert_eq!(field["ready"], json!(true), "{body}");
+ // 1 byte per dimension per live document — the resident array itself.
+ assert_eq!(
+ field["codes_bytes"],
+ json!((KEEP + WIDE_IDS.len()) * DIM),
+ "{body}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 2. A document's _score does not depend on the filter
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn scores_do_not_depend_on_the_filter() {
+ let (app, _dir) = app().await;
+ create_and_index(&app).await;
+
+ let unfiltered = knn(&app, None).await;
+ let filtered = knn(&app, Some(json!({ "term": { "tag": "keep" } }))).await;
+
+ assert_eq!(unfiltered.len(), KEEP + WIDE_IDS.len());
+ assert_eq!(filtered.len(), KEEP, "filter must remove the wide docs");
+
+ let score_of = |hits: &[(String, f64)], id: &str| {
+ hits.iter()
+ .find(|(hit_id, _)| hit_id == id)
+ .map(|(_, s)| *s)
+ .unwrap_or_else(|| panic!("{id} missing"))
+ };
+
+ let mut drift: f64 = 0.0;
+ for i in 0..KEEP {
+ let id = i.to_string();
+ let a = score_of(&unfiltered, &id);
+ let b = score_of(&filtered, &id);
+ drift = drift.max((a - b).abs());
+ assert_eq!(
+ a, b,
+ "doc {id} scores {a} unfiltered but {b} filtered — the SQ8 \
+ codebook is being fitted per query over the candidate set, so \
+ _score depends on the filter (issue #392)"
+ );
+ }
+ // Belt: on the unfixed path the two fits differ by the wide docs' whole
+ // range on dimension 0 (≥ 5.0 over 255 levels ⇒ Δ ≫ 1e-3), so a partial
+ // regression still trips the assert above long before this line.
+ assert!(drift == 0.0, "drift must be exactly zero, got {drift}");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 3. The code store is re-derived when the index reopens
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn codes_survive_a_restart() {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+
+ let before = {
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config.clone(), engine, metrics);
+ let app = xerj_api::router::build_es_compat_router(state.clone());
+ create_and_index(&app).await;
+ let hits = knn(&app, None).await;
+ state.engine.flush_all_force().await;
+ hits
+ };
+
+ let (app, _keep) = {
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config.clone(), engine, metrics);
+ let app = xerj_api::router::build_es_compat_router(state.clone());
+ // The open-time rebuild is background work; wait for it so the
+ // assertion observes the codes path, not a mid-rebuild fallback.
+ if let Ok(idx) = state.engine.get_index(INDEX) {
+ idx.await_sq8_rebuilds().await;
+ }
+ (app, ())
+ };
+
+ let after = knn(&app, None).await;
+
+ // Same neighbours in the same order: quantizing the same live vectors
+ // under a codebook spanning the same per-dimension ranges reproduces the
+ // ranking. Scores may shift by at most the SQ8 quantization step (the
+ // re-fit walk re-encodes from the true f32 sources, while the live path
+ // may have re-encoded through a decode), so ids/order are asserted
+ // strictly and scores within a generous step-sized tolerance.
+ assert_eq!(
+ before.len(),
+ after.len(),
+ "restart must not lose documents: {before:?} vs {after:?}"
+ );
+ let order_before: Vec<&str> = before.iter().map(|(id, _)| id.as_str()).collect();
+ let order_after: Vec<&str> = after.iter().map(|(id, _)| id.as_str()).collect();
+ assert_eq!(
+ order_before, order_after,
+ "ranking changed across restart: {before:?} vs {after:?}"
+ );
+ for ((_, a), (_, b)) in before.iter().zip(after.iter()) {
+ assert!(
+ (a - b).abs() < 1e-2,
+ "score for the same document moved from {a} to {b} across restart"
+ );
+ }
+
+ // And the re-derived store is again serving-shape.
+ let (status, body) = get(&app, &format!("/{INDEX}/_stats")).await;
+ assert_eq!(status, StatusCode::OK);
+ let field = &body["indices"][INDEX]["primaries"]["sq8"]["fields"]["v"];
+ assert_eq!(field["ready"], json!(true), "{body}");
+ assert_eq!(field["live"], json!(KEEP + WIDE_IDS.len()), "{body}");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// 4. Coverage broken → exact fallback, correct results
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[tokio::test]
+async fn a_wrong_dimension_document_does_not_break_the_field() {
+ let (app, _dir) = app().await;
+ create_and_index(&app).await;
+
+ // A document whose vector has the wrong dimensionality. The brute-force
+ // scan has always skipped it (`doc_vec.len() != dim`); the code store
+ // must not serve around it either — coverage breaks and queries fall
+ // back to the exact path rather than returning wrong results.
+ let (status, body) = json_req(
+ &app,
+ "PUT",
+ &format!("/{INDEX}/_doc/broken"),
+ json!({ "tag": "keep", "v": vec![0.5f32; DIM + 3] }),
+ )
+ .await;
+ assert!(status.is_success(), "index broken: {status} {body}");
+ let (status, body) = json_req(&app, "POST", &format!("/{INDEX}/_refresh"), json!({})).await;
+ assert!(status.is_success(), "{status} {body}");
+
+ let hits = knn(&app, Some(json!({ "term": { "tag": "keep" } }))).await;
+ assert_eq!(
+ hits.len(),
+ KEEP,
+ "the malformed doc must be skipped: {hits:?}"
+ );
+ assert!(
+ !hits.iter().any(|(id, _)| id == "broken"),
+ "malformed document must not appear in hits: {hits:?}"
+ );
+ // The control: doc 0 is the query vector and must still top the list.
+ assert_eq!(hits[0].0, "0", "query-vector doc must rank first: {hits:?}");
+ assert!(hits[0].1 > 0.99, "expected ~1.0, got {}", hits[0].1);
+}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index fdb3ed173..c50f85aa7 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -303,16 +303,15 @@ impl Config {
}
// Vector: `scalar8` (SQ8) quantization is wired into the kNN serving
- // path (`Index::run_knn_brute_force`): a `scalar8` dense_vector field
- // scores candidates from 1-byte-per-dimension codes rather than their
- // f32 values, so it has the recall profile of SQ8. It does not reduce
- // resident memory today — the scan quantizes each candidate's current
- // vector per query rather than caching codes, which is what keeps an
- // updated document from being scored on a stale code (#371).
- // `none` and `scalar8` are therefore
- // accepted. `binary` (1-bit) has no implemented quantizer, so honouring
- // it would silently store full-precision vectors while claiming a 32×
- // saving — it stays rejected until a BinaryQuantizer lands.
+ // path (`Index::run_knn_brute_force` / `run_knn_sq8_codes_scan`): a
+ // `scalar8` dense_vector field scores candidates from
+ // 1-byte-per-dimension codes written at ingest into a slot-addressed
+ // store (#392), so it has the recall profile of SQ8 and the kNN
+ // scoring working set is the codes rather than the f32 `_source`
+ // vectors. `none` and `scalar8` are therefore accepted. `binary`
+ // (1-bit) has no implemented quantizer, so honouring it would silently
+ // store full-precision vectors while claiming a 32× saving — it stays
+ // rejected until a BinaryQuantizer lands.
if self.vector.default_quantization == VectorQuantization::Binary {
return Err(XerjError::config(
"vector.default_quantization: binary (1-bit) quantization is not implemented in \
@@ -1191,17 +1190,18 @@ pub struct VectorConfig {
/// **not implemented in this build** and is rejected at startup.
///
/// - `"none"` — full float32 vectors (highest accuracy, most memory).
- /// - `"scalar8"` — 8-bit scalar quantization — WIRED into the kNN serving
- /// path, but it buys **precision, not memory**. A `scalar8` dense_vector
- /// field scores candidates from 1-byte-per-dimension codes, so it has the
- /// recall profile of int8 (measured recall@10 ≈ 0.998). It does NOT
- /// reduce resident memory: the scan reads the full-precision vector from
- /// `_source` and quantizes it per query, which is what keeps an updated
- /// document from being scored on stale codes or a stale codebook (#371).
- /// The ingest-time code array that would make a memory claim true is
- /// tracked in #392. Typically opted into per field via
- /// `index_options.type: int8_hnsw` on the mapping; this global default
- /// applies the same scheme index-wide.
+ /// - `"scalar8"` — 8-bit scalar quantization, WIRED into the kNN serving
+ /// path. A `scalar8` dense_vector field is scored from
+ /// 1-byte-per-dimension SQ8 codes written at INGEST time into a flat,
+ /// slot-addressed u8 array (#392), so it has the recall profile of int8
+ /// and the kNN scoring working set is the codes (1 byte/dim) instead of
+ /// the f32 `_source` vectors (4 bytes/dim). `_source` keeps the f32
+ /// originals, so this shrinks the scoring working set, not total resident
+ /// memory. When the code store cannot serve (open-time walk not yet
+ /// converged, a publication race, coverage broken by a wrong-dimension
+ /// write), queries fall back to the exact `_source` scan. Typically
+ /// opted into per field via `index_options.type: int8_hnsw` on the
+ /// mapping; this global default applies the same scheme index-wide.
/// - `"binary"` — 1-bit binary quantization (~32× memory reduction) — NOT
/// YET IMPLEMENTED (no `BinaryQuantizer` exists).
///
diff --git a/engine/crates/xerj-common/src/types.rs b/engine/crates/xerj-common/src/types.rs
index 8f731aaf0..41a210155 100644
--- a/engine/crates/xerj-common/src/types.rs
+++ b/engine/crates/xerj-common/src/types.rs
@@ -317,14 +317,21 @@ pub struct FieldOptions {
pub similarity: Option<String>,
/// Vector quantization scheme for this dense_vector field.
///
- /// `Some("scalar8")` makes the brute-force kNN scan score this field from
- /// 1-byte-per-dimension codes, quantized per query from the document's
- /// current vector — precision, not memory: nothing smaller is held
- /// resident, and no SQ8 state outlives the query (#371; #392 tracks the
- /// ingest-time code array that would make it a memory win). `None`/absent
- /// keeps the exact full-precision f32 brute-force path. Set from the
- /// mapping's `index_options.type` (`int8_hnsw`/`int8_flat` → `scalar8`) in
- /// es_compat.
+ /// `Some("scalar8")` makes the kNN paths score this field from
+ /// 1-byte-per-dimension SQ8 codes written at INGEST time into a flat,
+ /// slot-addressed u8 array (`Sq8CodeStore`, #392) — one slot per document,
+ /// rewritten on update, codebook fitted from the ingested vectors (widened
+ /// and re-encoded when a value lands outside it, never clamped). The kNN
+ /// scoring working set is those codes (1 byte/dim) instead of the f32
+ /// `_source` vectors (4 bytes/dim), and a document's `_score` is a function
+ /// of the index state alone, not of the query's candidate set. `_source`
+ /// keeps the original f32 vectors, so total resident memory does not shrink
+ /// 4× — the vector working set the scoring path touches does. When the
+ /// store cannot serve (open-time walk not yet converged, a publication
+ /// race, coverage broken by a wrong-dimension write), queries fall back to
+ /// the exact `_source` scan. `None`/absent keeps the exact full-precision
+ /// f32 brute-force path. Set from the mapping's `index_options.type`
+ /// (`int8_hnsw`/`int8_flat` → `scalar8`) in es_compat.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub quantization: Option<String>,
/// Null value to substitute when the field is missing from a document.
diff --git a/engine/crates/xerj-engine/src/index.rs b/engine/crates/xerj-engine/src/index.rs
index 0910053a2..758f1d960 100644
--- a/engine/crates/xerj-engine/src/index.rs
+++ b/engine/crates/xerj-engine/src/index.rs
@@ -34,7 +34,7 @@ use xerj_storage::segment::SectionType;
use xerj_storage::wal::{SyncMode, WalEntry};
use xerj_vector::distance::DistanceMetric;
use xerj_vector::hnsw::{HnswIndex, HnswParams};
-use xerj_vector::Sq8Params;
+use xerj_vector::{Sq8CodeStore, Sq8Params};
use crate::aggs::run_aggs_with_all;
use crate::segment_cache_budget::{
@@ -7855,6 +7855,10 @@ pub struct Index {
/// `abort_background_tasks` so an unfinished rebuild can't hold the
/// runtime (or the index Arc) alive across shutdown.
hnsw_rebuild_task: Arc<parking_lot::Mutex<Option<tokio::task::JoinHandle<()>>>>,
+ /// Per-field ingest-time SQ8 code stores (#392): field name → handle.
+ /// Empty unless the mapping declares `quantization: "scalar8"` — every
+ /// other index pays one empty-map read per write.
+ sq8_stores: Arc<parking_lot::RwLock<HashMap<String, Arc<Sq8StoreHandle>>>>,
// ── Per-index concurrency control ─────────────────────────────────────────
/// Semaphore that limits the number of queries executing concurrently
/// against this index. The default is 64 permits, matching the global
@@ -8497,6 +8501,9 @@ impl Index {
let segment_hydration_budget = index_segment_hydration_budget(config);
let passage_chunk_fields_init = passage_chunk_fields_from_schema(&managed.schema);
let date_field_scales_snapshot = date_field_scales(&managed.schema);
+ // #392: kept back from the struct literal so the SQ8 handle setup at
+ // the end of create can read the mapping without re-locking.
+ let managed_schema_for_sq8 = managed.schema.clone();
let index = Arc::new(Self {
name,
schema: Arc::new(RwLock::new(managed)),
@@ -8588,6 +8595,7 @@ impl Index {
hnsw_stale: Arc::new(std::sync::atomic::AtomicBool::new(false)),
hnsw_rebuilding: Arc::new(std::sync::atomic::AtomicBool::new(false)),
hnsw_rebuild_task: Arc::new(parking_lot::Mutex::new(None)),
+ sq8_stores: Arc::new(parking_lot::RwLock::new(HashMap::new())),
metric_query_count: Arc::new(AtomicU64::new(0)),
metric_query_total_ms: Arc::new(AtomicU64::new(0)),
metric_index_count: Arc::new(AtomicU64::new(0)),
@@ -8656,6 +8664,12 @@ impl Index {
idx.on_segments_changed();
}
})));
+ // #392: SQ8 code stores for scalar8 fields. A fresh index carries
+ // no documents, so the walk converges (and publishes `ready`)
+ // immediately; ingest maintains the store from the first write.
+ for field in index.ensure_sq8_stores_from_schema(&managed_schema_for_sq8) {
+ index.spawn_sq8_walk(&field);
+ }
Ok(index)
}
@@ -8814,6 +8828,10 @@ impl Index {
Self::resolve_flush_thresholds(config.storage.flush_size_mb);
let flush_idle_secs = config.storage.flush_idle_secs;
+ // #392: kept back from the struct literal (which moves `schema`) so
+ // the SQ8 handle setup at the end of open can read the mapping.
+ let schema_for_sq8 = schema.schema.clone();
+
// Try to reload a previously-persisted HNSW snapshot. If both
// graph.bin and ids.json exist, validate, and the pinned field is
// dense_vector-mapped (RC4 W2 item 16 — unmapped/legacy graphs are
@@ -8984,6 +9002,7 @@ impl Index {
hnsw_stale: Arc::new(std::sync::atomic::AtomicBool::new(hnsw_stale_init)),
hnsw_rebuilding: Arc::new(std::sync::atomic::AtomicBool::new(false)),
hnsw_rebuild_task: Arc::new(parking_lot::Mutex::new(None)),
+ sq8_stores: Arc::new(parking_lot::RwLock::new(HashMap::new())),
metric_query_count: Arc::new(AtomicU64::new(0)),
metric_query_total_ms: Arc::new(AtomicU64::new(0)),
metric_index_count: Arc::new(AtomicU64::new(0)),
@@ -9059,6 +9078,12 @@ impl Index {
// full graph-maintenance cost while the ANN path never served).
// Heal it in the background from the authoritative doc set.
index.spawn_hnsw_rebuild_if_stale();
+ // #392: create SQ8 code-store handles for any scalar8 field and
+ // make them serving-ready from the authoritative doc set (WAL
+ // replay never re-runs vector indexing).
+ for field in index.ensure_sq8_stores_from_schema(&schema_for_sq8) {
+ index.spawn_sq8_walk(&field);
+ }
Ok(index)
}
@@ -12620,6 +12645,16 @@ impl Index {
if let Some(handle) = self.hnsw_rebuild_task.lock().take() {
handle.abort();
}
+ // #392: same treatment for in-flight SQ8 code-store walks. An
+ // interrupted walk leaves `ready` false (exact scan serves) and the
+ // next open retries.
+ let walk_handles: Vec<Arc<Sq8StoreHandle>> =
+ self.sq8_stores.read().values().cloned().collect();
+ for handle in walk_handles {
+ if let Some(handle) = handle.walk_task.lock().take() {
+ handle.abort();
+ }
+ }
}
/// Update a document with upsert support.
@@ -12871,6 +12906,9 @@ impl Index {
/// node before inserting the fresh vector — otherwise the graph would
/// keep serving the pre-update vector forever.
async fn index_vectors(&self, doc_id: &str, source: &Value) {
+ // #392: SQ8 code stores are maintained for EVERY scalar8 field on
+ // every write, independent of the single field the HNSW graph pins.
+ self.sq8_ingest(doc_id, source).await;
let obj = match source.as_object() {
Some(o) => o,
None => return,
@@ -13010,6 +13048,319 @@ impl Index {
true
}
+ // ── SQ8 ingest-time code stores (#392) ──────────────────────────────
+ //
+ // One `Sq8CodeStore` per `quantization: "scalar8"` dense_vector field:
+ // one u8 per dimension per document, written by the ingest hook the
+ // moment a write publishes, addressed by dense slot (`slot * dim..`) —
+ // the same slot discipline the HNSW slab uses for its f32 vectors. The
+ // codebook is fitted from the vectors as they are ingested (widening +
+ // re-encode on out-of-range values, never clamping), so a document's
+ // quantized score is a function of index state alone — not of the
+ // candidate set a query scans. That is the Lucene property #392 asks
+ // for; qdrant's `EncodedVectorsU8` does the same at build time.
+ //
+ // Fail-safe direction everywhere: a store that is missing, not yet
+ // converged, or whose coverage gate is broken (a wrong-dimension write)
+ // leaves the kNN paths on the exact `_source` scan — nothing is ever
+ // served from incomplete codes.
+
+ /// The SQ8 code store handle for one field, if the mapping declared
+ /// `scalar8` and the handle exists.
+ fn sq8_store_for(&self, field: &str) -> Option<Arc<Sq8StoreHandle>> {
+ self.sq8_stores.read().get(field).cloned()
+ }
+
+ /// Create code-store handles for every `scalar8` field the schema
+ /// declares and none exists for yet. Returns the NEW fields — the caller
+ /// spawns the authoritative doc walk for those (an existing index may
+ /// carry documents that predate the handle; a fresh index converges on
+ /// an empty walk).
+ async fn ensure_sq8_stores(&self) -> Vec<String> {
+ let schema = self.schema.read().await;
+ self.ensure_sq8_stores_from_schema(&schema.schema)
+ }
+
+ /// Sync variant for call sites that already hold the schema.
+ fn ensure_sq8_stores_from_schema(&self, schema: &Schema) -> Vec<String> {
+ let scalar8_fields = collect_scalar8_fields(schema);
+ if scalar8_fields.is_empty() {
+ return Vec::new();
+ }
+ let mut created = Vec::new();
+ let mut guard = self.sq8_stores.write();
+ for field in scalar8_fields {
+ if guard.contains_key(&field) {
+ continue;
+ }
+ let similarity = lookup_vector_similarity(schema, &field);
+ let dim = declared_field(schema, &field)
+ .and_then(|fc| fc.options.dimensions)
+ .unwrap_or(0);
+ let handle = Arc::new(Sq8StoreHandle {
+ store: parking_lot::RwLock::new(Sq8CodeStore::new(dim)),
+ normalize: !matches!(
+ similarity.as_str(),
+ "l2_norm" | "dot_product" | "max_inner_product"
+ ),
+ ready: std::sync::atomic::AtomicBool::new(false),
+ walking: std::sync::atomic::AtomicBool::new(false),
+ walk_task: parking_lot::Mutex::new(None),
+ });
+ guard.insert(field.clone(), Arc::clone(&handle));
+ created.push(field);
+ }
+ drop(guard);
+ created
+ }
+
+ /// Spawn the authoritative doc walk that makes a (new) code store
+ /// serving-ready. Single-flight per handle. The walk re-derives codes
+ /// for every live document — WAL replay never re-runs vector indexing,
+ /// so this is also the restart path — and then publishes `ready` when
+ /// coverage is intact.
+ fn spawn_sq8_walk(self: &Arc<Self>, field: &str) {
+ let Some(handle) = self.sq8_store_for(field) else {
+ return;
+ };
+ if handle
+ .walking
+ .swap(true, std::sync::atomic::Ordering::AcqRel)
+ {
+ return;
+ }
+ let weak = Arc::downgrade(self);
+ let field = field.to_string();
+ let index_name = self.name.as_str().to_string();
+ let walk_handle = Arc::clone(&handle);
+ let join = tokio::spawn(async move {
+ let started = std::time::Instant::now();
+ let mut converged = false;
+ if let Some(idx) = weak.upgrade() {
+ converged = idx.sq8_rebuild_one_field(&walk_handle, &field).await;
+ }
+ walk_handle
+ .walking
+ .store(false, std::sync::atomic::Ordering::Release);
+ if converged {
+ info!(
+ index = %index_name,
+ field = %field,
+ elapsed_ms = started.elapsed().as_millis() as u64,
+ "SQ8 code store rebuilt — codes serving path enabled"
+ );
+ } else {
+ warn!(
+ index = %index_name,
+ field = %field,
+ "SQ8 code store rebuild did not converge — staying on exact scan"
+ );
+ }
+ });
+ let mut slot = handle.walk_task.lock();
+ if let Some(prev) = slot.replace(join) {
+ prev.abort();
+ }
+ }
+
+ /// Walk every live document once and make `handle`'s store describe
+ /// exactly the live corpus: upsert each doc that carries `field`, then
+ /// reconcile the remaining slots against the authoritative doc state.
+ /// Returns `false` (store left not-ready, exact scan keeps serving) if
+ /// any segment was unreadable or a suspect doc could not be fetched.
+ async fn sq8_rebuild_one_field(
+ self: &Arc<Self>,
+ handle: &Arc<Sq8StoreHandle>,
+ field: &str,
+ ) -> bool {
+ // Live-doc walk, same discipline as `rebuild_hnsw_from_docs`:
+ // memtable first, then every flushed segment, dedup by doc id,
+ // skipping tombstoned and seq-superseded copies.
+ let mut seen: HashSet<String> = HashSet::new();
+ let mem_docs = self.memtable.all_docs_with_sources();
+ for (doc_id, src) in &mem_docs {
+ seen.insert(doc_id.clone());
+ if let Some(mut vector) = extract_numeric_vector(src, field) {
+ if handle.normalize {
+ l2_normalize_vec(&mut vector);
+ }
+ handle.store.write().upsert(doc_id, &vector);
+ }
+ }
+ drop(mem_docs);
+
+ let snap = self.store.snapshot();
+ for meta in snap.segments.iter() {
+ let Some(docs_arc) = self.stored_values_for_async(&meta.id).await else {
+ warn!(
+ index = self.name.as_str(),
+ segment = meta.id.as_str(),
+ field,
+ "SQ8 rebuild: segment not readable — not ready"
+ );
+ return false;
+ };
+ for doc in docs_arc.iter() {
+ let Some(id) = doc.get("_id").and_then(Value::as_str) else {
+ continue;
+ };
+ if let Some(ver) = self.store.version_map.get(id) {
+ if ver.deleted {
+ continue;
+ }
+ if let Some(doc_seq) = doc.get("_seq_no").and_then(Value::as_u64) {
+ if doc_seq < ver.seq_no {
+ continue;
+ }
+ }
+ }
+ if !seen.insert(id.to_string()) {
+ continue;
+ }
+ let src_owned;
+ let src: &Value = match doc.get("_source") {
+ Some(s) => s,
+ None => {
+ let mut d = doc.clone();
+ if let Some(obj) = d.as_object_mut() {
+ obj.remove("_id");
+ }
+ src_owned = d;
+ &src_owned
+ }
+ };
+ if let Some(mut vector) = extract_numeric_vector(src, field) {
+ if handle.normalize {
+ l2_normalize_vec(&mut vector);
+ }
+ handle.store.write().upsert(id, &vector);
+ }
+ }
+ }
+
+ // Reconcile slots the walk did not confirm. A slot is dropped when
+ // its doc is gone (deleted, or no longer in the authoritative view)
+ // or no longer carries the field; a slot whose doc raced the walk
+ // (updated after its segment was passed) is refreshed from the live
+ // source rather than trusted or dropped blind.
+ let suspects: Vec<String> = {
+ let store = handle.store.read();
+ store
+ .iter_live()
+ .map(|(id, _)| id.to_string())
+ .filter(|id| !seen.contains(id))
+ .collect()
+ };
+ for id in suspects {
+ let gone = self
+ .store
+ .version_map
+ .get(&id)
+ .map(|e| e.deleted)
+ .unwrap_or(true);
+ if gone {
+ handle.store.write().remove(&id);
+ continue;
+ }
+ match self.get_document_uncounted(&id).await {
+ Ok(Some(src)) => match extract_numeric_vector(&src, field) {
+ Some(mut vector) => {
+ if handle.normalize {
+ l2_normalize_vec(&mut vector);
+ }
+ handle.store.write().upsert(&id, &vector);
+ }
+ None => {
+ // Live doc no longer carries the field.
+ handle.store.write().remove(&id);
+ }
+ },
+ Ok(None) => {
+ // Not in the authoritative view any more.
+ handle.store.write().remove(&id);
+ }
+ Err(e) => {
+ warn!(
+ index = self.name.as_str(),
+ field,
+ doc_id = %id,
+ error = %e,
+ "SQ8 rebuild: suspect doc not fetchable — not ready"
+ );
+ return false;
+ }
+ }
+ }
+
+ let coverage = handle.store.read().coverage_ok();
+ handle.ready.store(coverage, Ordering::Release);
+ coverage
+ }
+
+ /// Await all in-flight SQ8 code-store walks (deterministic test hook,
+ /// mirroring [`Self::await_hnsw_rebuild`]).
+ pub async fn await_sq8_rebuilds(&self) {
+ let handles: Vec<Arc<Sq8StoreHandle>> = self.sq8_stores.read().values().cloned().collect();
+ for handle in handles {
+ let join = handle.walk_task.lock().take();
+ if let Some(join) = join {
+ let _ = join.await;
+ }
+ }
+ }
+
+ /// Maintain every scalar8 field's code store for one document write.
+ /// Runs inside the write's publication interval (the same bracket the
+ /// HNSW inserts use), so a codes-served query either sees the slot or
+ /// falls back — never a half-published vector. A doc that stops
+ /// carrying the field loses its slot in the same step.
+ async fn sq8_ingest(&self, doc_id: &str, source: &Value) {
+ let handles: Vec<(String, Arc<Sq8StoreHandle>)> = {
+ let guard = self.sq8_stores.read();
+ guard
+ .iter()
+ .map(|(f, h)| (f.clone(), Arc::clone(h)))
+ .collect()
+ };
+ if handles.is_empty() {
+ return;
+ }
+ for (field, handle) in handles {
+ let Some(mut vector) = extract_numeric_vector(source, &field) else {
+ handle.store.write().remove(doc_id);
+ continue;
+ };
+ if handle.normalize {
+ l2_normalize_vec(&mut vector);
+ }
+ handle.store.write().upsert(doc_id, &vector);
+ }
+ }
+
+ /// Per-field SQ8 code-store health (#392), surfaced by `GET
+ /// /{index}/_stats` next to the HNSW section.
+ pub fn sq8_stats(&self) -> Value {
+ let guard = self.sq8_stores.read();
+ let mut fields = serde_json::Map::new();
+ for (field, handle) in guard.iter() {
+ let store = handle.store.read();
+ fields.insert(
+ field.clone(),
+ serde_json::json!({
+ "dim": store.dim(),
+ "live": store.live_len(),
+ "expected": store.expected(),
+ "covered": store.coverage_ok(),
+ "ready": handle.ready.load(Ordering::Acquire),
+ "serving": handle.serving_fresh(),
+ "refits": store.refits(),
+ "codes_bytes": store.codes_bytes(),
+ }),
+ );
+ }
+ Value::Object(fields)
+ }
+
/// Perform KNN search over the HNSW index.
///
/// Returns the top-`k` (doc_id, distance) pairs.
@@ -13757,6 +14108,154 @@ impl Index {
.await
}
+ /// #392 serving fast path: score an UNFILTERED scalar8 kNN entirely
+ /// from the ingest-time code store — no `_source` candidates are
+ /// collected, no f32 vectors are read, nothing is quantized per query.
+ ///
+ /// The whole scoring loop runs under ONE read guard of the store, with
+ /// no await inside it, so it observes the codes and the codebook of a
+ /// single consistent generation (a concurrent ingest waits on the write
+ /// guard; a codebook re-fit cannot interleave). Only the top `k`
+ /// survivors are hydrated from stored docs afterwards — any hydration
+ /// anomaly (doc deleted mid-scan, unreadable segment, a chunked-passage
+ /// doc) abandons the path and the caller falls back to the exact scan,
+ /// mirroring `run_knn_hnsw`'s never-a-partial-page discipline.
+ ///
+ /// Returns `None` whenever the path cannot serve: store missing, walk
+ /// not converged, coverage broken (a wrong-dimension write), dim
+ /// mismatch, deadline, or a publication racing source visibility. The
+ /// fail-safe direction is always the exact `_source` scan.
+ #[allow(clippy::too_many_arguments)] // mirrors run_knn_brute_force_with_deadline
+ async fn run_knn_sq8_codes_scan(
+ &self,
+ request: &SearchRequest,
+ deadline: std::time::Instant,
+ field: &str,
+ query_vec: &[f32],
+ k: usize,
+ similarity: &str,
+ boost: Option<f32>,
+ min_similarity: Option<f32>,
+ started: std::time::Instant,
+ ) -> Option<SearchResult> {
+ if std::time::Instant::now() >= deadline {
+ return None;
+ }
+ // Chunked-passage docs score by best passage in the exact path; the
+ // codes hold the pooled vector, so hand those back to it.
+ if self.passage_chunks_require_exact(field) {
+ return None;
+ }
+ let handle = self.sq8_store_for(field)?;
+ if !handle.serving_fresh() {
+ return None;
+ }
+ // Same publication discipline as `run_knn_hnsw`: never serve codes
+ // from a window that overlapped a vector publication.
+ let publication_generation = self.hnsw_publication_generation.load(Ordering::Acquire);
+ if self.hnsw_publications_in_flight.load(Ordering::Acquire) != 0 {
+ return None;
+ }
+
+ // ── Score every live slot from the codes ─────────────────────
+ let mut ranked: Vec<(f32, String)> = Vec::new();
+ {
+ let store = handle.store.read();
+ // Re-check the gates under the guard: the freshness check above
+ // raced a concurrent walk/ingest if this now fails.
+ if !store.coverage_ok() || store.dim() != query_vec.len() {
+ return None;
+ }
+ let codebook = store.codebook();
+ let mut decoded = vec![0.0f32; store.dim()];
+ ranked.reserve(store.live_len() as usize);
+ for (id, codes) in store.iter_live() {
+ // Cheap deadline poll — no await can run under this guard.
+ // On expiry hand the request to the exact path (which owns
+ // the timed_out semantics), never serve a partial scan.
+ if ranked.len() & 8191 == 8191 && std::time::Instant::now() >= deadline {
+ return None;
+ }
+ codebook.decode_into(codes, &mut decoded);
+ let score = compute_vector_similarity(similarity, query_vec, &decoded);
+ ranked.push((score, id.to_string()));
+ }
+ }
+ debug!(
+ index = self.name.as_str(),
+ field,
+ scored = ranked.len(),
+ "SQ8 kNN served from ingest-time codes (#392)"
+ );
+
+ // ── Same cutoff → boost → rank pipeline as the exact path ─────
+ if let Some(raw) = min_similarity {
+ let cut = raw_similarity_to_score(similarity, raw);
+ ranked.retain(|(score, _)| *score >= cut);
+ }
+ if let Some(b) = boost {
+ if (b - 1.0).abs() > f32::EPSILON {
+ for (score, _) in ranked.iter_mut() {
+ *score *= b;
+ }
+ }
+ }
+ // Deterministic order for equal scores (the slot walk's HashMap
+ // order is not), then cap the pool at k — only these are hydrated.
+ ranked.sort_by(|a, b| {
+ b.0.partial_cmp(&a.0)
+ .unwrap_or(std::cmp::Ordering::Equal)
+ .then_with(|| a.1.cmp(&b.1))
+ });
+ ranked.truncate(k.max(1));
+
+ // ── Hydrate the top-k from stored docs ────────────────────────
+ let chunk_field = format!("{field}_chunks");
+ let mut scored: Vec<(String, f32, Value, Option<u32>)> = Vec::with_capacity(ranked.len());
+ for (score, doc_id) in ranked {
+ let remaining = deadline.saturating_duration_since(std::time::Instant::now());
+ if remaining.is_zero() {
+ return None;
+ }
+ let src =
+ match tokio::time::timeout(remaining, self.get_document_uncounted(&doc_id)).await {
+ Ok(Ok(Some(source))) => source,
+ _ => return None,
+ };
+ // A chunked doc must be scored by its best passage, not the
+ // pooled codes — same bail as `run_knn_hnsw`.
+ if get_field_value(&src, &chunk_field).is_some() {
+ return None;
+ }
+ scored.push((doc_id, score, src, None));
+ }
+ if std::time::Instant::now() >= deadline {
+ return None;
+ }
+ if self.passage_chunks_require_exact(field) {
+ return None;
+ }
+ if self.hnsw_publications_in_flight.load(Ordering::Acquire) != 0
+ || self.hnsw_publication_generation.load(Ordering::Acquire) != publication_generation
+ {
+ return None;
+ }
+
+ let generated_companion_fields = {
+ let schema = self.schema.read().await;
+ generated_embedding_companion_fields(&schema.schema)
+ };
+ Some(knn_result_from_scored(
+ self,
+ request,
+ field,
+ scored,
+ k,
+ started,
+ &generated_companion_fields,
+ ))
+ }
+
#[allow(clippy::too_many_arguments)]
async fn run_knn_brute_force_with_deadline(
&self,
@@ -13787,6 +14286,42 @@ impl Index {
// this is the whole filtered-kNN path. Owned clone; the lock is released.
let dmq_schema = self.schema().await;
+ // ── Determine whether this field opts into SQ8 (scalar8) ──────
+ // Default fields keep the exact f32 brute-force scan below,
+ // byte-identical to before. A `scalar8` field may take the codes
+ // fast path instead (#392). Determined up front so the fast path
+ // can skip the candidate collection altogether.
+ let use_sq8 = {
+ let schema = self.schema.read().await;
+ lookup_vector_quantization(&schema.schema, field).as_deref() == Some("scalar8")
+ };
+
+ // ── #392 fast path: unfiltered scalar8 kNN from ingest-time codes ──
+ // Scores every live slot straight out of the code store — no
+ // `_source` candidates collected, no per-query quantization — and
+ // hydrates only the top k. Any gate miss (store not converged,
+ // coverage broken, publication racing) falls through to the exact
+ // scan below; filtered queries score from the same codes inside
+ // the use_sq8 branch further down.
+ if use_sq8 && filter.is_none() {
+ if let Some(result) = self
+ .run_knn_sq8_codes_scan(
+ request,
+ deadline,
+ field,
+ query_vec,
+ k,
+ similarity,
+ boost,
+ min_similarity,
+ started,
+ )
+ .await
+ {
+ return Ok(result);
+ }
+ }
+
// ── Collect all candidate (doc_id, source) pairs ──────────────
let mut candidates: Vec<(String, Value)> = Vec::new();
// Memtable first (newest writes).
@@ -13889,17 +14424,8 @@ impl Index {
}
let collect_elapsed = started.elapsed();
- // ── Determine whether this field opts into SQ8 (scalar8) ──────
- // Default fields keep the exact f32 brute-force scan below,
- // byte-identical to before. A `scalar8` field takes the branch above
- // instead: it reads the same live f32 vector out of `_source`, but
- // rounds it through 1-byte-per-dimension SQ8 codes before scoring, so
- // the score carries `scalar8`'s precision loss. Nothing is cached —
- // neither the codes nor the codebook outlive the query (#371).
- let use_sq8 = {
- let schema = self.schema.read().await;
- lookup_vector_quantization(&schema.schema, field).as_deref() == Some("scalar8")
- };
+ // `use_sq8` was determined before candidate collection (the codes
+ // fast path above may already have returned).
// ── Score each candidate against the query vector ─────────────
let mut scored: Vec<(String, f32, Value, Option<u32>)> =
@@ -13948,78 +14474,130 @@ impl Index {
cand.push((id, src, doc_vec));
}
- // Fit this field's SQ8 codebook over the candidate vectors this
- // query is about to score, and drop it when the query ends. No
- // codec state survives a query (#371).
+ // ── #392: score from the INGEST-TIME codes when they cover the
+ // candidate set ──────────────────────────────────────────────
//
- // The codebook used to be fitted from the first ≤1000 vectors the
- // field was ever scanned with and then kept for the life of the
- // process, which is the same write-once defect as the per-document
- // code map one level up: a vector written afterwards outside the
- // fitted per-dimension range is CLAMPED into it, and when that
- // range is narrow the clamped decode is indistinguishable from the
- // vector the document used to hold. A corpus whose dimension 0
- // never left +1.0 fitted `[1,1]` there, so overwriting a document
- // with its exact negation decoded straight back to +1.0 and it
- // stayed top at cosine 1.000000 — verbatim the reported symptom,
- // with the per-document map already gone.
- //
- // Fitting over exactly the set being encoded also makes clamping
- // structurally impossible here rather than merely unlikely: every
- // value passed to `encode_into` is inside `[min,max]` by
- // construction. An empty candidate set fits degenerate params, but
- // the scoring loop below is then empty too, so nothing observes
- // them — the old code had to guard against publishing that codec
- // because it outlived the query that produced it.
- //
- // ACCEPTED CONSEQUENCE: the codebook now depends on the candidate
- // set, and that changes the RETURNED ORDER, not merely the score.
- // Each individual score moves by at most SQ8's own quantization
- // step (1/255 of the fitted range per dimension), which is inside
- // the approximation error `scalar8` already advertises — but
- // near-tied documents swap, so a caller sees a different ranking.
- // Measured at the HTTP boundary on a 60-document 4-dim cosine
- // field: adding a `filter` that removes only unrelated documents
- // returned the same 30 survivors with max |Δ_score| 1.976e-05 but
- // a different order at 19 of 30 positions. The trigger is the
- // candidate set, not the `filter` keyword — indexing one more
- // unrelated document moved the same corpus by max 7.100e-06 and
- // reordered its top 10 — and on an unfiltered corpus over 1000
- // documents every score also differs from rc.17, which fitted from
- // the first <=1000 candidates and cached that (1500 documents: all
- // of the top 40 changed, max 4.880e-05, one adjacent-rank swap).
- // Documented in docs/recipes/vector-quantization.md and #392.
- //
- // This is NOT the dependency Lucene has. Lucene fits per segment at
- // INDEX time, so a Lucene score is a function of the index state
- // alone; here it is a function of the query's candidate set too.
- // #392 is what would close that gap.
- let params = Sq8Params::fit_borrowed(cand.iter().map(|(_, _, v)| v.as_slice()), dim);
- debug!(
- field,
- dim,
- candidates = cand.len(),
- normalize,
- "SQ8 codec fitted for this query"
- );
+ // When the field's code store is serving-fresh and every
+ // post-filter candidate holds a live slot, each candidate is
+ // scored by decoding the SAME bytes under the SAME codebook the
+ // unfiltered path (`run_knn_sq8_codes_scan`) scores with. That
+ // is the point of #392: a document's `_score` becomes a
+ // function of the index state alone and stops depending on
+ // which other documents a filter removed. Any miss — store not
+ // converged, coverage broken, a candidate without codes, a dim
+ // mismatch — falls back to the per-query fit below, which is
+ // still correct (it is the pre-#392 behaviour) but
+ // filter-dependent.
+ let mut stored_codes: Option<(Sq8Params, Vec<Vec<u8>>)> = None;
+ if let Some(handle) = self.sq8_store_for(field) {
+ if handle.serving_fresh() {
+ let store = handle.store.read();
+ if store.dim() == dim && store.coverage_ok() {
+ let codebook = store.codebook();
+ let mut all = Vec::with_capacity(cand.len());
+ let mut complete = true;
+ for (id, _, _) in cand.iter() {
+ match store.codes_for(id) {
+ Some(c) => all.push(c.to_vec()),
+ None => {
+ complete = false;
+ break;
+ }
+ }
+ }
+ if complete {
+ stored_codes = Some((codebook, all));
+ }
+ }
+ }
+ }
- // Score by quantizing each candidate's CURRENT vector and decoding
- // it straight back — 1 byte/dim, so `scalar8` keeps its recall
- // profile, and the score describes the vector the document holds
- // right now rather than one a previous query cached. Both buffers
- // are reused across the scan, so this allocates nothing per
- // document. `v.len() == dim` was established when `cand` was built.
- let mut codes = vec![0u8; dim];
- let mut decoded = vec![0.0f32; dim];
- for (position, (id, src, v)) in cand.into_iter().enumerate() {
- if position & 127 == 0 && self.exact_scan_checkpoint(position, deadline).await {
- timed_out = true;
- break;
+ match stored_codes {
+ Some((codebook, codes)) => {
+ debug!(
+ field,
+ dim,
+ candidates = codes.len(),
+ "SQ8 filtered kNN scored from ingest-time codes (#392)"
+ );
+ // `dim` bytes per candidate were copied out under a
+ // short read guard, so this loop holds no lock across
+ // its deadline checkpoints.
+ let mut decoded = vec![0.0f32; dim];
+ for (position, ((id, src, _), codes)) in cand.into_iter().zip(codes).enumerate()
+ {
+ if position & 127 == 0
+ && self.exact_scan_checkpoint(position, deadline).await
+ {
+ timed_out = true;
+ break;
+ }
+ codebook.decode_into(&codes, &mut decoded);
+ let score = compute_vector_similarity(similarity, query_vec, &decoded);
+ scored.push((id, score, src, None));
+ }
+ }
+ None => {
+ // ── Legacy per-query fit (pre-#392 fallback) ────────
+ //
+ // Fit this field's SQ8 codebook over the candidate vectors
+ // this query is about to score, and drop it when the query
+ // ends. No codec state survives a query (#371).
+ //
+ // The codebook used to be fitted from the first ≤1000
+ // vectors the field was ever scanned with and then kept
+ // for the life of the process, which is the same
+ // write-once defect as the per-document code map one
+ // level up: a vector written afterwards outside the
+ // fitted per-dimension range is CLAMPED into it.
+ // Fitting over exactly the set being encoded makes
+ // clamping structurally impossible here: every value
+ // passed to `encode_into` is inside `[min,max]` by
+ // construction.
+ //
+ // REMAINING CONSEQUENCE (#392): the fitted codebook
+ // depends on the candidate set, so each score moves by up
+ // to SQ8's quantization step when unrelated documents are
+ // added or filtered away, and near-tied documents can
+ // swap. Measured at the HTTP boundary on a 60-document
+ // 4-dim cosine field: adding a `filter` that removed only
+ // unrelated documents returned the same 30 survivors with
+ // max |Δ_score| 1.976e-05 but a different order at 19 of
+ // 30 positions. Lucene fits per segment at INDEX time, so
+ // its scores are a function of index state alone. This
+ // branch is now reached only when the ingest-time store
+ // cannot serve; #392's codes path is what closes the gap.
+ let params =
+ Sq8Params::fit_borrowed(cand.iter().map(|(_, _, v)| v.as_slice()), dim);
+ debug!(
+ field,
+ dim,
+ candidates = cand.len(),
+ normalize,
+ "SQ8 codec fitted for this query (ingest-time codes unavailable)"
+ );
+
+ // Score by quantizing each candidate's CURRENT vector and
+ // decoding it straight back — 1 byte/dim, so `scalar8`
+ // keeps its recall profile, and the score describes the
+ // vector the document holds right now. Both buffers are
+ // reused across the scan. `v.len() == dim` was
+ // established when `cand` was built.
+ let mut codes = vec![0u8; dim];
+ let mut decoded = vec![0.0f32; dim];
+ for (position, (id, src, v)) in cand.into_iter().enumerate() {
+ if position & 127 == 0
+ && self.exact_scan_checkpoint(position, deadline).await
+ {
+ timed_out = true;
+ break;
+ }
+ params.encode_into(&v, &mut codes);
+ params.decode_into(&codes, &mut decoded);
+ let score = compute_vector_similarity(similarity, query_vec, &decoded);
+ scored.push((id, score, src, None));
+ }
}
- params.encode_into(&v, &mut codes);
- params.decode_into(&codes, &mut decoded);
- let score = compute_vector_similarity(similarity, query_vec, &decoded);
- scored.push((id, score, src, None));
}
} else {
// Per-chunk (passage) companion: multi-chunk `semantic_text` docs
@@ -16472,6 +17050,15 @@ impl Index {
})
.ok();
}
+ // #392: drop the deleted doc's SQ8 code slots so the per-field
+ // coverage gates stay exact (mirrors the vector_doc_count decrement
+ // above).
+ {
+ let guard = self.sq8_stores.read();
+ for handle in guard.values() {
+ handle.store.write().remove(id);
+ }
+ }
#[cfg(test)]
self.publication_test_point(id, PublicationTestPoint::AfterHnsw);
@@ -22650,7 +23237,7 @@ impl Index {
}
/// Add a field to the schema.
- pub async fn add_field(&self, field: FieldConfig) -> Result<()> {
+ pub async fn add_field(self: &Arc<Self>, field: FieldConfig) -> Result<()> {
self.add_fields(vec![field]).await
}
@@ -22660,7 +23247,7 @@ impl Index {
/// companion together. Validate and persist the complete candidate schema
/// before publishing it so a later-field failure cannot leave half of that
/// contract visible.
- pub async fn add_fields(&self, fields: Vec<FieldConfig>) -> Result<()> {
+ pub async fn add_fields(self: &Arc<Self>, fields: Vec<FieldConfig>) -> Result<()> {
if fields.is_empty() {
return Ok(());
}
@@ -22724,6 +23311,13 @@ impl Index {
*self.embedder.write().await = embedder;
}
*schema = candidate;
+ drop(schema);
+ // #392: a mapping update can introduce a `scalar8` field — create
+ // its code store now and walk the live docs (the index may already
+ // carry documents predating the handle).
+ for field in self.ensure_sq8_stores().await {
+ self.spawn_sq8_walk(&field);
+ }
Ok(())
}
@@ -35631,6 +36225,64 @@ pub(crate) fn collect_dense_vector_fields(schema: &Schema) -> Vec<String> {
out
}
+/// One `scalar8` field's ingest-time SQ8 code store plus its readiness gate
+/// (#392). The engine holds one handle per quantized field in
+/// [`Index::sq8_stores`].
+struct Sq8StoreHandle {
+ /// The slot-addressed code array + codebook. Written by the ingest hook
+ /// (`sq8_ingest`), the delete hook, and the authoritative doc walk;
+ /// read by the kNN serving paths under a read guard.
+ store: parking_lot::RwLock<Sq8CodeStore>,
+ /// L2-normalize vectors before encoding — cosine-family similarity,
+ /// exactly the rule the per-query serving path applies
+ /// (`!matches!(similarity, "l2_norm" | "dot_product" | "max_inner_product")`).
+ /// Captured from the mapping when the handle is created; changing a
+ /// live field's similarity requires a reindex (as in ES).
+ normalize: bool,
+ /// Set once an authoritative walk over the live document set has
+ /// converged (empty indexes converge trivially). The serving paths may
+ /// only score from the codes when this holds AND the store's coverage
+ /// gate passes; otherwise they fall back to the exact scan.
+ ready: std::sync::atomic::AtomicBool,
+ /// Single-flight guard for the authoritative walk.
+ walking: std::sync::atomic::AtomicBool,
+ /// JoinHandle of the in-flight walk, aborted by `abort_background_tasks`.
+ walk_task: parking_lot::Mutex<Option<tokio::task::JoinHandle<()>>>,
+}
+
+impl Sq8StoreHandle {
+ /// The serving gate: the walk has converged and every doc id that ever
+ /// carried (or attempted) this field holds a live slot.
+ fn serving_fresh(&self) -> bool {
+ self.ready.load(Ordering::Acquire) && self.store.read().coverage_ok()
+ }
+}
+
+/// All dense_vector fields whose mapping opts into SQ8 (`quantization:
+/// "scalar8"`, set from `index_options.type: int8_hnsw|int8_flat` by the
+/// compat layer). Same recursive walk as [`collect_dense_vector_fields`].
+fn collect_scalar8_fields(schema: &Schema) -> Vec<String> {
+ fn walk(fields: &[FieldConfig], prefix: &str, out: &mut Vec<String>) {
+ for fc in fields {
+ let path = if prefix.is_empty() {
+ fc.name.clone()
+ } else {
+ format!("{prefix}.{}", fc.name)
+ };
+ if matches!(fc.field_type, FieldType::Vector)
+ && fc.options.quantization.as_deref() == Some("scalar8")
+ {
+ out.push(path);
+ } else if !fc.fields.is_empty() {
+ walk(&fc.fields, &path, out);
+ }
+ }
+ }
+ let mut out = Vec::new();
+ walk(&schema.fields, "", &mut out);
+ out
+}
+
/// A top-level KNN query unwrapped for the short-circuit executor.
#[derive(Debug, Clone)]
struct PeeledKnn {
diff --git a/engine/crates/xerj-vector/src/lib.rs b/engine/crates/xerj-vector/src/lib.rs
index 994be0c2e..302ce6c0a 100644
--- a/engine/crates/xerj-vector/src/lib.rs
+++ b/engine/crates/xerj-vector/src/lib.rs
@@ -12,6 +12,7 @@ pub mod distance;
pub mod hnsw;
pub mod quantizer;
pub mod search;
+pub mod sq8_codes;
pub use distance::{compute_distance, cosine, dot_product, l2_squared, DistanceMetric};
pub use hnsw::{HnswIndex, HnswParams};
@@ -19,5 +20,6 @@ pub use quantizer::{
NoneQuantizer, QuantizedData, QuantizedVectors, Quantizer, Scalar8Quantizer, Sq8Params,
};
pub use search::{SearchResult, VectorSearcher};
+pub use sq8_codes::Sq8CodeStore;
pub use xerj_common::Result;
diff --git a/engine/crates/xerj-vector/src/quantizer.rs b/engine/crates/xerj-vector/src/quantizer.rs
index 72a450179..95f4b2534 100644
--- a/engine/crates/xerj-vector/src/quantizer.rs
+++ b/engine/crates/xerj-vector/src/quantizer.rs
@@ -183,11 +183,11 @@ impl Quantizer for NoneQuantizer {
/// Each dimension gets its own `min` and `scale` computed from the training
/// set, so the quantization is adaptive to the actual data distribution.
///
-/// Encoding size: 1 byte per dimension vs. 4 bytes for f32 — 4x. That is a
-/// property of this codec, not of the engine: the kNN serving path does not
-/// hold SQ8 codes resident, it quantizes each candidate's f32 vector per query
-/// (see [`Sq8Params`], and issue #392 for the ingest-time code array that would
-/// turn this ratio into a resident saving).
+/// Encoding size: 1 byte per dimension vs. 4 bytes for f32 — 4x. On the kNN
+/// serving path that ratio is realized by [`crate::sq8_codes::Sq8CodeStore`]
+/// (issue #392): codes are written at ingest into a flat, slot-addressed u8
+/// array with a codebook fitted from the ingested vectors, and queries score
+/// against those codes instead of re-quantizing each candidate's f32 vector.
#[derive(Debug, Default, Clone)]
pub struct Scalar8Quantizer;
@@ -320,25 +320,38 @@ impl Quantizer for Scalar8Quantizer {
///
/// This is the serving-path counterpart to [`Scalar8Quantizer`]: instead of
/// packing a whole batch into one blob, it exposes a fitted codec cheap enough
-/// to build per query. The brute-force kNN scan fits one of these over exactly
-/// the candidate vectors it is about to score ([`Sq8Params::fit_borrowed`]),
-/// quantizes each candidate's current vector through it, dequantizes straight
-/// back, and drops it — so scores come from **1 byte/dim** codes while **no
-/// SQ8 state outlives the query** (issue #371).
+/// to build per query.
///
-/// That is deliberate. Both a per-document `doc_id -> Vec<u8>` code map and a
-/// per-field cached codebook were tried, and both were write-once in practice:
-/// the map never recomputed a document's codes after an update, and the
-/// codebook, fitted from the first ~1000 vectors a field was ever scanned
-/// with, silently *clamped* every later vector into that range — so a document
-/// overwritten with its own negation decoded straight back to its old value
-/// and kept ranking first at cosine 1.000000. Fitting over the set being
-/// encoded also makes clamping impossible here rather than merely unlikely.
+/// Since issue #392 the PRIMARY serving path no longer builds one of these at
+/// all: [`crate::sq8_codes::Sq8CodeStore`] holds the codes and the codebook at
+/// ingest time (one flat u8 array, slot-addressed, widened and re-encoded when
+/// a vector lands outside the fitted range), and kNN queries score by decoding
+/// those codes. A document's score is therefore a function of the index state
+/// alone, not of the candidate set the query happens to scan — the property
+/// Lucene has and the per-query fit lacked.
///
-/// The cost is that `scalar8` does not reduce resident memory: the scan reads
-/// the full-precision vector from `_source` either way. Making codes live with
-/// the data, addressed by ordinal and written at ingest as Lucene does, is
-/// tracked in issue #392.
+/// `fit_borrowed` remains as the FALLBACK codec for the windows where the
+/// store cannot serve — the open-time walk has not converged yet, a
+/// publication is racing source visibility, or coverage is broken by a
+/// wrong-dimension write. There the brute-force scan fits one of these over
+/// exactly the candidate vectors it is about to score, quantizes each
+/// candidate's current vector through it, dequantizes straight back, and drops
+/// it. That keeps the #371 guarantee — **no SQ8 state outlives the query**, so
+/// an updated document can never be scored on a stale code — at the cost that
+/// scores in that fallback window depend on the candidate set (each by at most
+/// SQ8's own 1/255 quantization step).
+///
+/// History (#371): both a per-document `doc_id -> Vec<u8>` code map and a
+/// per-field cached codebook were tried before this type existed, and both
+/// were write-once in practice: the map never recomputed a document's codes
+/// after an update, and the codebook, fitted from the first ~1000 vectors a
+/// field was ever scanned with, silently *clamped* every later vector into
+/// that range — so a document overwritten with its own negation decoded
+/// straight back to its old value and kept ranking first at cosine 1.000000.
+/// The ingest-time store does not repeat that defect class structurally: an
+/// out-of-range value extends the range and every live code is re-encoded
+/// through a decode/encode round trip (each stored value moves by at most the
+/// new quantization step), so clamping cannot occur.
///
/// `mins[d]`/`scales[d]` are the per-dimension minimum and range (`max-min`)
/// observed in the fitting sample. `encode` maps `v[d]` linearly to a u8 in
@@ -787,9 +800,10 @@ mod tests {
}
/// (b) MEMORY: SQ8 codes are ~4× smaller than the f32 vectors they encode.
- /// This is a property of the codec, measured here on the codec. The engine
- /// does not currently realize it as a resident saving on the kNN path — see
- /// the note on [`Sq8Params`].
+ /// This is a property of the codec, measured here on the codec; since #392
+ /// the engine realizes it on the kNN path too — the serving scan scores
+ /// against the ingest-time code array ([`crate::sq8_codes::Sq8CodeStore`]),
+ /// not the f32 `_source` vectors.
#[test]
fn sq8_store_is_quarter_of_f32() {
let n = 2000usize;
diff --git a/engine/crates/xerj-vector/src/sq8_codes.rs b/engine/crates/xerj-vector/src/sq8_codes.rs
new file mode 100644
index 000000000..e2e407201
--- /dev/null
+++ b/engine/crates/xerj-vector/src/sq8_codes.rs
@@ -0,0 +1,404 @@
+//! Ingest-time, slot-addressed SQ8 code store — issue #392.
+//!
+//! [`Sq8CodeStore`] is the durable counterpart of the per-query codec
+//! ([`Sq8Params`](crate::quantizer::Sq8Params)): one u8 code per dimension
+//! per document, written when the document is indexed and rewritten when it
+//! is updated, in a single flat `Vec<u8>` addressed by a dense slot —
+//! `codes[slot * dim .. slot * dim + dim]` — the same slot discipline the
+//! HNSW slab uses for its f32 vectors. The codebook is fitted from the
+//! vectors as they are ingested and lives with the data, so a document's
+//! quantized score is a function of the index state alone and not of the
+//! candidate set a particular query happens to scan (#392's second half).
+//!
+//! Reference points (retrieved, not copied):
+//! * qdrant `lib/quantization/src/encoded_vectors_u8.rs` (Apache-2.0) —
+//! quantized codes are written once at build time into contiguous
+//! storage, fetched by `get_vector_data(offset)` at
+//! `offset * quantized_vector_size`; the codebook (alpha/offset) is
+//! fitted at build time and persisted next to the codes; the query is
+//! encoded once and scored against stored codes without touching f32.
+//! * Lucene `OffHeapScalarQuantizedVectorValues.vectorValue(targetOrd)`
+//! seeks `targetOrd * byteSize` into a per-segment slice written at index
+//! time; requantization happens at merge time, never at query time
+//! (cited from issue #392; Elasticsearch/Lucene is AGPL/SSPL — approach
+//! only).
+//!
+//! The one structural difference from both: XERJ maintains the store
+//! incrementally per document instead of rebuilding it per segment merge, so
+//! the codebook WIDENS whenever an ingested vector falls outside the fitted
+//! range, and widening re-encodes every stored code through a decode/encode
+//! round trip (each stored value moves by at most the new quantization
+//! step). That is the same order of accuracy work a merge-time requantization
+//! does, paid at ingest, and it is what makes clamping structurally
+//! impossible: the codebook always spans every live vector (#371's defect
+//! class cannot recur — a vector written outside the range extends the range
+//! rather than being clamped into it).
+
+use std::collections::HashMap;
+
+use crate::quantizer::Sq8Params;
+
+/// Slot-addressed SQ8 code store for ONE dense_vector field.
+///
+/// All mutating operations are `&mut self`; the engine keeps each store
+/// behind a lock and serves scans under a read guard, so a scan observes the
+/// codes and the codebook of one consistent generation.
+#[derive(Debug)]
+pub struct Sq8CodeStore {
+ dim: usize,
+ /// The live codebook. `maxs` are kept alongside so an out-of-range
+ /// ingest can extend the range exactly rather than through the
+ /// dequantized grid.
+ mins: Vec<f32>,
+ maxs: Vec<f32>,
+ /// slot → doc id. Slots of tombstoned documents keep their last id
+ /// (never read again; `slot_of` is the live set).
+ ids: Vec<String>,
+ /// doc id → slot (the live set).
+ slot_of: HashMap<String, u32>,
+ /// Flat code array, `codes[slot * dim .. slot * dim + dim]`.
+ codes: Vec<u8>,
+ /// Soft delete marks, one per slot (the slab's discipline: append-only,
+ /// tombstone on delete, never physically free).
+ tomb: Vec<bool>,
+ /// Coverage-gate denominator: number of DISTINCT doc ids ever offered to
+ /// [`Self::upsert`] and not since removed. Mirrors the engine's
+ /// `vector_doc_count` (counted on attempt): a rejected vector (wrong
+ /// dimension) increments this without creating a slot, so
+ /// `live_len() != expected` and consumers fall back to exact paths.
+ expected: u64,
+ /// Number of codebook widenings (observability).
+ refits: u64,
+}
+
+impl Sq8CodeStore {
+ /// An empty store for `dim`-dimensional vectors.
+ pub fn new(dim: usize) -> Self {
+ Sq8CodeStore {
+ dim,
+ mins: vec![0.0; dim],
+ maxs: vec![0.0; dim],
+ ids: Vec::new(),
+ slot_of: HashMap::new(),
+ codes: Vec::new(),
+ tomb: Vec::new(),
+ expected: 0,
+ refits: 0,
+ }
+ }
+
+ /// Vector dimensionality this store encodes.
+ pub fn dim(&self) -> usize {
+ self.dim
+ }
+
+ /// The current codebook (mins/scales), for decoding at query time.
+ pub fn codebook(&self) -> Sq8Params {
+ let scales = self
+ .mins
+ .iter()
+ .zip(self.maxs.iter())
+ .map(|(&mn, &mx)| mx - mn)
+ .collect();
+ Sq8Params {
+ mins: self.mins.clone(),
+ scales,
+ }
+ }
+
+ /// Live documents holding a code slot (the coverage-gate numerator).
+ pub fn live_len(&self) -> u64 {
+ self.slot_of.len() as u64
+ }
+
+ /// Distinct doc ids ever accepted for this field minus removals (the
+ /// coverage-gate denominator). See the field docs for the attempt
+ /// semantics.
+ pub fn expected(&self) -> u64 {
+ self.expected
+ }
+
+ /// The serving gate: every doc id that ever carried (or attempted to
+ /// carry) this field holds a live slot. Consumers that score from the
+ /// codes must fall back to an exact path when this is false — a doc with
+ /// a rejected vector would otherwise be silently missing from results.
+ pub fn coverage_ok(&self) -> bool {
+ self.live_len() == self.expected
+ }
+
+ /// Codebook widenings since construction (observability).
+ pub fn refits(&self) -> u64 {
+ self.refits
+ }
+
+ /// Bytes occupied by the flat code array (including tombstoned slots —
+ /// the array is append-only).
+ pub fn codes_bytes(&self) -> usize {
+ self.codes.len()
+ }
+
+ /// The live codes of one document, `dim` bytes at `slot * dim`.
+ pub fn codes_for(&self, doc_id: &str) -> Option<&[u8]> {
+ let &slot = self.slot_of.get(doc_id)?;
+ Some(&self.codes[slot as usize * self.dim..][..self.dim])
+ }
+
+ /// Iterate every LIVE `(doc id, codes)` pair. Slot-addressed order.
+ pub fn iter_live(&self) -> impl Iterator<Item = (&str, &[u8])> {
+ let dim = self.dim;
+ self.slot_of
+ .iter()
+ .map(move |(id, &slot)| (id.as_str(), &self.codes[slot as usize * dim..][..dim]))
+ }
+
+ /// Write (or rewrite, on update) one document's codes.
+ ///
+ /// Returns `false` when `vector` does not have exactly [`Self::dim`]
+ /// dimensions — the attempt is still counted in `expected` (no slot is
+ /// created), which breaks [`Self::coverage_ok`] and keeps consumers on
+ /// exact paths: the same wrong-dimension document the brute-force scan
+ /// skips must not be half-represented here.
+ ///
+ /// A vector outside the fitted per-dimension range widens the codebook
+ /// and re-encodes every live code (see the module docs). Re-encoding
+ /// goes decode-old → encode-new, bounded by the new quantization step.
+ pub fn upsert(&mut self, doc_id: &str, vector: &[f32]) -> bool {
+ if vector.len() != self.dim {
+ // Count the attempt for a NEW id only: an id that never had a
+ // slot must not leak a phantom expectation, and an id whose
+ // field silently appeared with the wrong arity keeps its slot
+ // (remove() is the API for that).
+ if !self.slot_of.contains_key(doc_id) {
+ self.expected += 1;
+ }
+ return false;
+ }
+ // Snapshot the codebook BEFORE widening: the re-encode decodes each
+ // live code under the range it was encoded with.
+ let old_codebook = self.codebook();
+ if self.extend_codebook_for(vector) {
+ self.refits += 1;
+ self.reencode_all(&old_codebook);
+ }
+ let codebook = self.codebook();
+ match self.slot_of.get(doc_id).copied() {
+ Some(slot) => {
+ let start = slot as usize * self.dim;
+ codebook.encode_into(vector, &mut self.codes[start..][..self.dim]);
+ self.tomb[slot as usize] = false;
+ true
+ }
+ None => {
+ let slot = self.ids.len() as u32;
+ let mut code = vec![0u8; self.dim];
+ codebook.encode_into(vector, &mut code);
+ self.ids.push(doc_id.to_string());
+ self.codes.extend_from_slice(&code);
+ self.tomb.push(false);
+ self.slot_of.insert(doc_id.to_string(), slot);
+ self.expected += 1;
+ true
+ }
+ }
+ }
+
+ /// Drop a document's live slot (delete, or the document no longer
+ /// carries the field). Returns whether a slot was live. The slot is
+ /// tombstoned, not freed — same append-only discipline as the HNSW slab.
+ pub fn remove(&mut self, doc_id: &str) -> bool {
+ match self.slot_of.remove(doc_id) {
+ Some(slot) => {
+ self.tomb[slot as usize] = true;
+ self.expected = self.expected.saturating_sub(1);
+ true
+ }
+ None => false,
+ }
+ }
+
+ /// Widen `mins`/`maxs` to include `vector` where it falls outside.
+ /// Returns whether anything moved. An empty store adopts `vector`'s
+ /// per-dimension values as its initial (degenerate, zero-scale) range —
+ /// `min == max` encodes every value to code 0 and decodes it back to
+ /// `min` exactly, so the first document carries no quantization error.
+ fn extend_codebook_for(&mut self, vector: &[f32]) -> bool {
+ let mut extended = false;
+ if self.ids.is_empty() {
+ self.mins.copy_from_slice(vector);
+ self.maxs.copy_from_slice(vector);
+ return false;
+ }
+ for (d, &x) in vector.iter().enumerate() {
+ if x < self.mins[d] {
+ self.mins[d] = x;
+ extended = true;
+ }
+ if x > self.maxs[d] {
+ self.maxs[d] = x;
+ extended = true;
+ }
+ }
+ extended
+ }
+
+ /// Re-encode every live slot under the current (just widened) codebook,
+ /// decoding under `old`. Each stored value moves by at most the new
+ /// quantization step (see the module docs).
+ fn reencode_all(&mut self, old: &Sq8Params) {
+ if self.slot_of.is_empty() {
+ return;
+ }
+ let new = self.codebook();
+ let dim = self.dim;
+ let mut decoded = vec![0.0f32; dim];
+ let mut code = vec![0u8; dim];
+ // Live slots only: tombstoned codes are unreachable through
+ // `slot_of` and re-encoding them would be wasted work.
+ let slots: Vec<u32> = self.slot_of.values().copied().collect();
+ for slot in slots {
+ let start = slot as usize * dim;
+ old.decode_into(&self.codes[start..][..dim], &mut decoded);
+ new.encode_into(&decoded, &mut code);
+ self.codes[start..][..dim].copy_from_slice(&code);
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn store_with(vectors: &[Vec<f32>]) -> Sq8CodeStore {
+ let mut s = Sq8CodeStore::new(vectors[0].len());
+ for (i, v) in vectors.iter().enumerate() {
+ assert!(s.upsert(&i.to_string(), v), "upsert {i} must succeed");
+ }
+ s
+ }
+
+ #[test]
+ fn first_vector_encodes_exactly() {
+ // Degenerate range [v, v] per dim → code 0 → decode back to v.
+ let v = vec![0.25, -0.5, 1.75];
+ let s = store_with(&[v.clone()]);
+ let codebook = s.codebook();
+ let codes = s.codes_for("0").unwrap();
+ assert_eq!(codes, vec![0, 0, 0]);
+ let decoded = codebook.decode(codes);
+ assert_eq!(decoded, v);
+ }
+
+ #[test]
+ fn roundtrip_within_one_step() {
+ let vectors: Vec<Vec<f32>> = (0..20)
+ .map(|i| {
+ (0..4)
+ .map(|d| ((i * 37 + d * 101) % 199) as f32 / 99.0 - 1.0)
+ .collect()
+ })
+ .collect();
+ let s = store_with(&vectors);
+ let codebook = s.codebook();
+ for (i, v) in vectors.iter().enumerate() {
+ let decoded = codebook.decode(s.codes_for(&i.to_string()).unwrap());
+ let step: f32 = codebook.scales.iter().cloned().sum();
+ let err: f32 = v.iter().zip(decoded).map(|(a, b)| (a - b).abs()).sum();
+ assert!(
+ err < step,
+ "doc {i}: |err| {err} must be < one total step {step}"
+ );
+ }
+ }
+
+ #[test]
+ fn out_of_range_vector_extends_rather_than_clamps() {
+ // #371's defect class: a corpus whose dim 0 never leaves +1.0, then a
+ // document rewritten to -1.0. A clamping (stale) codebook decodes
+ // -1.0 back to +1.0; a widening one decodes it near -1.0.
+ let mut s = store_with(&[vec![1.0, 0.0], vec![1.0, 0.5]]);
+ let before_refits = s.refits();
+ assert!(s.upsert("0", &[-1.0, 0.0]));
+ assert_eq!(s.refits(), before_refits + 1, "must have widened");
+ let decoded = s.codebook().decode(s.codes_for("0").unwrap());
+ assert!(
+ (decoded[0] - (-1.0)).abs() < 2.0 / 255.0,
+ "dim 0 decoded to {} — the codebook clamped (#371 defect class)",
+ decoded[0]
+ );
+ // The untouched dimension-1 values of the OTHER document survive the
+ // re-encode within a step.
+ let d1 = s.codebook().decode(s.codes_for("1").unwrap());
+ assert!((d1[0] - 1.0).abs() < 2.0 / 255.0);
+ assert!((d1[1] - 0.5).abs() < 2.0 / 255.0);
+ }
+
+ #[test]
+ fn update_rewrites_the_slot() {
+ let mut s = store_with(&[vec![0.0, 1.0], vec![1.0, 0.0]]);
+ let codebook_before = s.codebook();
+ let old = codebook_before.decode(s.codes_for("0").unwrap());
+ assert_eq!(old, vec![0.0, 1.0]);
+ assert!(s.upsert("0", &[1.0, 1.0]));
+ // Both dims' ranges now [0,1]; doc 0 re-encoded in place.
+ assert_eq!(s.live_len(), 2, "update must not grow the live set");
+ let decoded = s.codebook().decode(s.codes_for("0").unwrap());
+ assert!((decoded[0] - 1.0).abs() < 1.0 / 255.0);
+ assert!((decoded[1] - 1.0).abs() < 1.0 / 255.0);
+ }
+
+ #[test]
+ fn remove_breaks_coverage_and_delete_reinsert_restores_it() {
+ let mut s = store_with(&[vec![0.0], vec![1.0]]);
+ assert!(s.coverage_ok());
+ assert!(s.remove("0"));
+ assert_eq!(s.live_len(), 1);
+ assert!(s.coverage_ok(), "remove decrements both sides");
+ assert!(!s.remove("0"), "second remove is a no-op");
+ assert!(s.upsert("0", &[0.5]));
+ assert!(s.coverage_ok());
+ assert_eq!(s.live_len(), 2);
+ }
+
+ #[test]
+ fn wrong_dim_attempt_breaks_coverage() {
+ let mut s = store_with(&[vec![0.0, 0.0]]);
+ assert!(!s.upsert("bad", &[0.0, 0.0, 0.0]), "mismatch rejected");
+ assert_eq!(s.live_len(), 1);
+ assert_eq!(s.expected(), 2);
+ assert!(
+ !s.coverage_ok(),
+ "a rejected vector must keep consumers on exact paths"
+ );
+ assert!(s.codes_for("bad").is_none());
+ }
+
+ #[test]
+ fn codes_bytes_is_one_byte_per_dim_per_slot() {
+ let s = store_with(&[vec![0.0; 8], vec![1.0; 8], vec![0.5; 8]]);
+ assert_eq!(s.codes_bytes(), 3 * 8);
+ }
+
+ #[test]
+ fn iter_live_skips_removed_and_addresses_by_slot() {
+ let mut s = store_with(&[vec![0.0, 0.1], vec![0.2, 0.3]]);
+ assert!(s.remove("0"));
+ let live: Vec<&str> = s.iter_live().map(|(id, _)| id).collect();
+ assert_eq!(live, vec!["1"]);
+ let (_, codes) = s.iter_live().next().unwrap();
+ assert_eq!(codes.len(), 2);
+ }
+
+ #[test]
+ fn codebook_is_order_independent() {
+ // The final range spans the same values regardless of ingest order,
+ // so a restart rebuild produces the same codebook (codes may differ
+ // in the last bit through decode/encode re-encode, the ranges do not).
+ let a: Vec<Vec<f32>> = vec![vec![0.0, 1.0], vec![-2.0, 0.5], vec![3.0, -1.0]];
+ let mut b = a.clone();
+ b.reverse();
+ let sa = store_with(&a);
+ let sb = store_with(&b);
+ assert_eq!(sa.codebook().mins, sb.codebook().mins);
+ assert_eq!(sa.codebook().scales, sb.codebook().scales);
+ }
+}
diff --git a/recipes/vector_quantization.py b/recipes/vector_quantization.py
index 5ee7a4f6a..250b2c2ab 100644
--- a/recipes/vector_quantization.py
+++ b/recipes/vector_quantization.py
@@ -4,8 +4,16 @@
XERJ stores dense vectors at full float32 precision by default. Opting a
`dense_vector` field into scalar8 quantization (`index_options.type:
int8_hnsw`) makes the kNN *serving* path score against 1-byte-per-dimension
-codes instead of 4-byte floats — the recall profile of int8, with
-almost no recall loss. `_source` still returns the original vectors.
+codes instead of 4-byte floats, with almost no recall loss. `_source` still
+returns the original vectors.
+
+Since issue #392 the codes are written at INGEST time into a flat,
+slot-addressed u8 array and the kNN scan scores against them, so the
+serving working set IS the 1-byte-per-dim codes rather than the f32
+`_source` vectors. The run below measures the resident array straight off
+the server (`GET /{index}/_stats` → `primaries.sq8.fields.v.codes_bytes`).
+`_source` keeps the f32 originals, so this is the scoring working set that
+shrinks 4x — not total process memory.
This recipe embeds the 40 real KB articles (demo/data/ai_kb.ndjson) into
128-dim vectors with a small deterministic feature-hasher (same idea as
@@ -15,19 +23,18 @@ shows that:
1. kNN returns the same top results from both,
2. recall@10 of the quantized index vs the exact index stays >= 0.90,
- 3. what the int8 encoding costs vs float32 (1 vs 4 bytes/dim).
-
-NOTE: this changes PRECISION, not memory. XERJ reads the full-precision
-vector from `_source` and quantizes it per query, so `scalar8` does not
-shrink the resident vector working set today — see issue #392. The
-footprint line below is the size of the ENCODING, not a saving XERJ
-currently realises.
+ 3. the resident SQ8 code array the server actually holds — measured
+ from `_stats` (1 byte/dim/live doc), alongside the encoding-size
+ comparison measured by encoding every vector both ways client-side.
Usage:
xerj --insecure --data-dir ./data & # start XERJ
python3 recipes/vector_quantization.py
- XERJ_URL (default http://localhost:9200)
+Environment:
+ XERJ_URL server URL (default http://localhost:9200)
+ XERJ_KB path to ai_kb.ndjson (default: auto-discovered by walking up
+ from this script to the repo's demo/data/ai_kb.ndjson)
"""
import hashlib
@@ -35,16 +42,34 @@ import json
import math
import os
import pathlib
+import struct
import urllib.error
import urllib.request
XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
-KB = pathlib.Path(__file__).resolve().parent.parent / "demo" / "data" / "ai_kb.ndjson"
DIM = 128
NONE_INDEX = "vq-none"
SQ8_INDEX = "vq-scalar8"
+def _find_kb():
+ """Locate demo/data/ai_kb.ndjson robustly, regardless of where this copy
+ of the script lives (docs/examples/... or recipes/...). Honours XERJ_KB."""
+ env = os.environ.get("XERJ_KB")
+ if env:
+ return pathlib.Path(env)
+ here = pathlib.Path(__file__).resolve()
+ for base in (here.parent, *here.parents):
+ cand = base / "demo" / "data" / "ai_kb.ndjson"
+ if cand.exists():
+ return cand
+ # Fall back to the original relative guess so the error message is useful.
+ return here.parent.parent / "demo" / "data" / "ai_kb.ndjson"
+
+
+KB = _find_kb()
+
+
def call(method, path, body=None):
"""One tiny HTTP helper instead of a client library."""
data = None
@@ -108,6 +133,17 @@ def knn(name, qv, k=10):
return [(h["_id"], h["_score"]) for h in r["hits"]["hits"]]
+def encode_f32(vec):
+ """Exact wire size of a full-precision vector: 4 bytes per dimension."""
+ return struct.pack(f"<{len(vec)}f", *vec)
+
+
+def encode_i8(vec):
+ """Scalar8 code: symmetric int8 quantization → 1 byte per dimension.
+ (Cosine vectors are already L2-normalised, so |x| <= 1.)"""
+ return bytes((max(-127, min(127, round(x * 127))) & 0xFF) for x in vec)
+
+
def main():
# ── 1. Embed the real KB into 128-dim vectors. ───────────────────────
docs = []
@@ -143,12 +179,45 @@ def main():
hits += len(exact & approx)
total += len(exact)
recall = hits / total if total else 0.0
+
+ # ── 5. Measure the RESIDENT codes the server holds (#392). ──────────
+ # The ingest-time, slot-addressed SQ8 array is server state — read it
+ # back from _stats rather than stipulating its size.
+ stats = call("GET", f"/{SQ8_INDEX}/_stats")
+ field_stats = stats["indices"][SQ8_INDEX]["primaries"]["sq8"]["fields"]["v"]
+ codes_bytes = field_stats["codes_bytes"]
+ live = field_stats["live"]
+
+ # ── 6. Measure the real byte size of each ENCODING (client side). ────
+ # Encode every corpus vector both ways and compare the actual byte totals
+ # — a genuine measurement, not a stipulated 4x.
+ f32_total = sum(len(encode_f32(d["v"])) for d in docs)
+ i8_total = sum(len(encode_i8(d["v"])) for d in docs)
+ ratio = f32_total / i8_total if i8_total else 0.0
+ f32_per = f32_total // len(docs)
+ i8_per = i8_total // len(docs)
+
print(f"recall@10 (scalar8 vs float32 ground truth): {recall:.3f}")
- print(f"encoding size: float32 = {DIM * 4} B/vec → scalar8 = {DIM} B/vec (4x smaller)")
+ print(
+ f"resident SQ8 codes (from _stats, {live} live docs): "
+ f"{codes_bytes} B ({codes_bytes // max(live, 1)} B/vec) — "
+ f"serving={field_stats['serving']}"
+ )
+ print(
+ f"encoding size over {len(docs)} vecs: "
+ f"float32 = {f32_total} B ({f32_per} B/vec) → "
+ f"scalar8 = {i8_total} B ({i8_per} B/vec) ({ratio:.2f}x smaller)"
+ )
if recall < 0.90:
raise SystemExit(f"FAIL: recall {recall:.3f} < 0.90")
- print("\nOK — recall preserved through 1-byte-per-dim codes. `_source` still holds")
- print("the originals. scalar8 changes precision, not resident memory (issue #392).")
+ if not field_stats["serving"] or not field_stats["ready"]:
+ raise SystemExit("FAIL: SQ8 code store not serving after ingest (#392)")
+ if codes_bytes != len(docs) * DIM:
+ raise SystemExit(
+ f"FAIL: codes_bytes {codes_bytes} != {len(docs)} docs x {DIM} dim"
+ )
+ print("\nOK — recall preserved through 1-byte-per-dim codes, served from the")
+ print("ingest-time slot-addressed array (#392). `_source` still holds the originals.")
if __name__ == "__main__":