AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Vector quantization: int8 scoring at recall@10 ≈ 0.998
Read this first.
scalar8scores come from 1-byte-per-dimension codes written at INGEST time into a flat, slot-addressed u8 array (#392) — the kNN serving working set is those codes (1 byte/dim), not the f32_sourcevectors (4 bytes/dim), and the score a document gets is a function of the index state alone, not of the filter or candidate set of the query that found it._sourcekeeps the original f32 vectors, so this shrinks the scoring working set ~4×, not total process memory. When the code store cannot serve (right after open, mid-rebuild, or after a wrong-dimension write breaks coverage), queries fall back to the exact_sourcescan.
The problem
Dense vectors are heavy. A 768-dim float32 embedding is ~3 KB; a million
of them is ~3 GB of vector data that has to be resident to serve low-latency
kNN. Scale to tens of millions and the vector working set — not the text,
not the postings — becomes the thing that decides how much RAM you rent.
The standard fix is scalar quantization: store each dimension in one byte instead of four. The catch everyone worries about is recall — does compressing the vectors quietly wreck ranking quality? That is the question this recipe answers, and the answer is no.
Why XERJ
XERJ lets you opt a dense_vector field into scalar8 (int8)
quantization per field. When you do, the kNN serving path scores against
1-byte-per-dimension codes instead of 4-byte floats, while _source still
returns the original vectors for retrieval. It's off by default (full
float32), so you choose the precision model per field, spelled exactly like
Elasticsearch's int8_hnsw.
On a real 128-dim corpus the recall cost is negligible: recall@10 = 0.998 against the exact float32 index. That number is computed by the run below, not stipulated.
The codes are 128 bytes rather than 512 per vector, and since #392 they are
held resident by the engine: the run reads the live array's byte size
straight off the server (GET /{index}/_stats →
primaries.sq8.fields.<field>.codes_bytes) and checks it is exactly
docs × dim. The kNN scan scores against that array without touching the f32
vectors — the ~4× smaller vector working set is a property of the serving
path now, measured, not a ratio the encoding merely implies.
The solution
Opt a field in at mapping time with index_options.type: int8_hnsw:
curl -sX PUT "$XERJ_URL/docs" -H 'content-type: application/json' -d '{
"mappings": {
"properties": {
"title": { "type": "text" },
"v": {
"type": "dense_vector",
"dims": 128,
"similarity": "cosine",
"index_options": { "type": "int8_hnsw" }
}
}
}
}'
Index and query exactly as you would a full-precision field — nothing else changes:
curl -sX POST "$XERJ_URL/docs/_search" -H 'content-type: application/json' -d '{
"knn": { "field": "v", "query_vector": [0.12, 0.08, -0.31, "..."], "k": 10 }
}'
The scores come back slightly different from an exact float32 index (that's the quantization at work — a query that exactly matches a stored vector scores ~0.99999 instead of 1.0), but the ranking is the same.
Try it
docs/examples/vector-quantization/quant_demo.py (the mirrored
recipes/vector_quantization.py runs the same demo) embeds the 40 real KB
articles into 128-dim vectors, indexes the same vectors into a float32 index
and a scalar8 index, and prints the side-by-side top hits, the measured
recall@10, and the measured byte footprint of each encoding:
$ python3 docs/examples/vector-quantization/quant_demo.py
embedded 40 real KB articles into 128-dim vectors
indexed into `vq-none` (float32) and `vq-scalar8` (int8_hnsw / scalar8)
query: 'how do I stop an agent's context window from overflowing?'
── float32 (exact)
0.67958 Long-context windows do not replace memory
0.60029 p95 latency budgets for interactive RAG agents
0.59712 SOC 2 controls that apply to vector workloads
── scalar8 (quantized)
0.67938 Long-context windows do not replace memory
0.59983 p95 latency budgets for interactive RAG agents
0.59766 SOC 2 controls that apply to vector workloads
recall@10 (scalar8 vs float32 ground truth): 0.998
resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=True
encoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)
OK — recall preserved through 1-byte-per-dim codes, served from the
ingest-time slot-addressed array (#392). `_source` still holds the originals.
Both size lines are real measurements. The client-side one encodes every
corpus vector as float32 bytes (struct) and as int8 codes and compares the
byte totals — 20480 B vs 5120 B, exactly 4.00×. The resident SQ8 codes line
is read off the server (_stats → primaries.sq8.fields.v.codes_bytes) and
matches it byte-for-byte: 40 docs × 128 dim = 5120 B, serving=True — the
array the kNN scan actually scores against (#392).
Reproduce it yourself
# 1. Start XERJ (dev mode, default ES-compat port 9200)
xerj --insecure --data-dir ./data &
# 2. Run the demo (stdlib-only Python 3, no packages, no API keys)
python3 docs/examples/vector-quantization/quant_demo.py
XERJ_URL overrides the server (default http://localhost:9200); XERJ_KB
overrides the KB path (default: auto-discovered demo/data/ai_kb.ndjson).
The embedder and corpus are deterministic, so a customer should see exactly:
recall@10 (scalar8 vs float32 ground truth): 0.998resident SQ8 codes (from _stats, 40 live docs): 5120 B (128 B/vec) — serving=Trueencoding size over 40 vecs: float32 = 20480 B (512 B/vec) → scalar8 = 5120 B (128 B/vec) (4.00x smaller)
These numbers are stable run-to-run (verified across repeated runs — no variance); the printed kNN scores are likewise identical each run.
Notes and limits
- Opt-in per field. Fields without
int8_hnswkeep exact float32 scoring, byte-for-byte unchanged. _sourceis never quantized. Retrieval returns the vectors you indexed; only the scoring path uses the compact codes.scalar8is wired;binaryis not yet. Binary (1-bit) quantization is rejected at startup rather than silently storing full precision.- Cosine is normalised before quantizing for the tightest code range;
dot_productandl2_normsimilarities are supported too. - The serving working set shrinks ~4×; total memory does not. The kNN
scan scores against the ingest-time code array (1 byte/dim) instead of the
f32
_sourcevectors (4 bytes/dim), but_sourcekeeps the originals for retrieval, so the process's total footprint does not fall 4×. The array itself is observable:GET /{index}/_stats→primaries.sq8(per field:dim,live,expected,covered,ready,serving,refits,codes_bytes). - Codes are written at ingest and rewritten on update. One slot per
document, addressed
slot * dim, tombstoned on delete — the same slot discipline the HNSW slab uses. A vector that lands outside the fitted per-dimension range WIDENS the codebook and re-encodes every live code (decode → encode, each value moves by at most the new quantization step); nothing is ever clamped, which is the defect class #371 fixed by deleting the old write-once caches. _scoreno longer depends on the filter. The codebook is fitted from the ingested vectors and lives with the data, so the same document carries the same_scorewhether or not a filter removed other documents — the property Elasticsearch has and per-query fitting lacked. Pinned over HTTP inengine/crates/xerj-api/tests/sq8_codes_are_ingest_time_and_filter_independent.rs(drift is exactly 0, not merely small).- Fallback windows keep the exact scan. Until the open-time walk converges
(a restart re-derives the store in the background — WAL replay never
re-runs vector indexing), while a publication race is possible, or after a
wrong-dimension write breaks the store's coverage gate, queries are served
by the exact
_sourcescan. In that fallback the SQ8 branch still fits its codebook per query over the candidates being scored, so scores there move by up to SQ8's own quantization step (1/255 of the fitted per-dimension range) when the candidate set changes — measured on the pre-#392 path: max |Δ_score| 1.976e-05 with order swaps at 19/30 positions when an unrelated-only filter was added. That fallback is correct, just not filter-independent;primaries.sq8.fields.<f>.servingtells you which path you are on. scalar8still disables ANN. A quantized field remains excluded from HNSW-served kNN (the graph scores f32) and takes the codes scan above for unfiltered queries or the exact brute-force scan otherwise. Teaching the graph to traverse against the codes is the remaining open half of #392.