AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
test(perf): correct the text doc-values evidence — the target is semantic_text
xerj-org61a52c4b5608
diff --git a/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.mdindex 55eeee431..24babbaf7 100644--- a/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md+++ b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md@@ -99,9 +99,25 @@ curl -s -X POST localhost:9400/rc12bench/_search -H 'Content-Type: application/j-d '{"size":1,"sort":[{"body":"asc"}]}'```-**Both succeed**, served from the `.dv` column. Elasticsearch rejects both by-default (`Fielddata is disabled on text fields by default`) because neither is-meaningful on analyzed text.+**Both succeed**, served from the `.dv` column. Elasticsearch rejects both **by+default** (`Fielddata is disabled on text fields by default`).++> **Correction (important).** An earlier draft of this file said the whole-value+> bucketing below is "semantically wrong" and that ES refuses it outright. That is+> true only of *default* ES and of classic Lucene. Modern Elasticsearch supports+> text doc-values behind the cluster feature `mapper.text.doc_values`, by attaching+> a separate `SORTED_SET` doc-values field — and **XERJ's own conformance suite+> already encodes that behaviour**:+> `tests/es-compat-yaml/yaml/aggregations/terms_text_docvalues.yml` maps+> `{type: text, index: false, doc_values: true}` and asserts whole-value buckets+> (`"foo bar": 2`, `"baz qux": 1`).+>+> So whole-value bucketing is the *correct* answer when a user explicitly asks for+> `doc_values: true` on a text field. The defect is not the shape of the answer —+> it is that XERJ builds the column **unconditionally**, for every text field,+> whether or not anyone asked. That reframes the lever from "never build+> doc-values for text" to **"default off, and honour the flag"**, which is both the+> real ES behaviour and the only version that keeps the 1360/0/3 gate green.| operation | engine `took` | result ||---|---:|---|@@ -128,25 +144,43 @@ why `sum_other_doc_count` is 65,533 and the answer carries no information.## What this establishes-The text doc-values column is not a trade. It is a **triple defect**:+The text doc-values column is built **unconditionally**, and that costs on two axes:1. **Disk.** 37.1% of the index on the 500k mixed corpus, 39.7% on a pure-textindex. The largest single artifact in both.-2. **Semantics.** The two operations it enables return answers that are wrong in- kind, not merely imprecise — bucketing by document instead of by term. ES- refuses them for exactly this reason.-3. **Latency.** Those same two operations are the slowest in the engine by a wide- margin (7.8 s and 1.65 s against `took: 0` everywhere else).--So the read-path interaction that would have made this a trade does not exist in-the form feared. Nothing *useful* reads the text `.dv`. Dropping it removes the-largest artifact, removes two wrong answers, and removes the two slowest-operations.--**Consequence to decide explicitly:** after the change, `terms`-agg and `sort` on a-`text` field must **error** the way ES does, rather than silently returning fewer-results. That is a deliberate, documented behaviour change and it needs an ES-YAML-conformance check — the suite must stay at 1360 passed / 0 failed / 3 skipped.+2. **Latency.** The two operations it enables are the slowest in the engine by a+ wide margin (7.8 s and 1.65 s against `took: 0` everywhere else) — and today+ every index pays the disk cost for them whether or not anyone ever runs one.++Nothing reads this column unless a user explicitly aggregates or sorts on a text+field, which is rare and which ES requires you to opt into.++### The design this implies — and the trap in the RFC++Discussion #148 says "ES/Lucene never builds doc_values for `text`; honouring+`doc_values: false` drops the 22.9 MB `.dv` entirely." Implemented literally that+**fails the conformance gate**, because of the `terms_text_docvalues.yml` cases+above. The correct shape is:++- **default off** for `text` (matching ES's default), and+- **honour an explicit `doc_values: true`**, which still builds the column and+ still returns whole-value buckets.++### The detail that decides whether this lever is worth anything++Measured across live corpora by parsing the `DV01` envelope and joining to each+index's `_mapping`:++| corpus | `semantic_text` | `text` | keyword | numeric |+|---|---:|---:|---:|---:|+| `xc-lucene*` (81 indices, 23.8 MiB `.dv`) | 86.1% | 4.7% | 8.2% | 1.0% |+| `xc-dataplane-cilium-vendor*` (83 indices, 61.0 MiB `.dv`) | 91.9% | 5.2% | — | — |++**`semantic_text`, not `text`, is where the bytes are.** Both map to+`FieldType::Text` (`es_compat.rs:13678`), so one schema-level check captures+90.8–97.1%. A policy keyed on the literal string `"text"` would capture under 5%+and be nearly worthless. The RFC does not mention this distinction at all, and it+is the single most important implementation detail in the lever.**Still to determine:** whether the doc-values *prefilters* (term/terms/range/bool)or the `size:0` count shortcut ever bind to a `text` field. Those paths are real anddiff --git a/demo/playbooks/rc12/measure_rc12.sh b/demo/playbooks/rc12/measure_rc12.shindex cbe6fd01f..5da52f653 100755--- a/demo/playbooks/rc12/measure_rc12.sh+++ b/demo/playbooks/rc12/measure_rc12.sh@@ -34,8 +34,15 @@ echo " data: $DATA"# ── fresh instance ───────────────────────────────────────────────────────────# There is no --port flag; the listen ports come from a TOML config.-pkill -f "[x]erj -c $DATA/xerj.toml" 2>/dev/null || true-sleep 1+# Free the PORT, not just this label's data dir: a previous run under a+# DIFFERENT label still owns the socket, and killing only our own config path+# leaves it up. The new server then fails to bind and exits, while every+# subsequent request silently hits the OLD instance — which surfaces as a+# baffling 409 on index creation rather than as a bind error.+for pid in $(lsof -ti tcp:"$PORT" 2>/dev/null); do+ kill "$pid" 2>/dev/null || true+done+sleep 2rm -rf "$DATA"; mkdir -p "$DATA"cat > "$DATA/xerj.toml" <<EOF[server]@@ -44,7 +51,15 @@ es_compat_port = $PORTrest_port = $((PORT + 1))grpc_port = $((PORT + 2))EOF-nohup "$BIN" -c "$DATA/xerj.toml" --data-dir "$DATA" --insecure \+# XERJ_DISABLE_QUERY_CACHE=1 is MANDATORY, not optional. Without it the+# whole-result cache clones the answer for a repeated identical query body, and+# since this harness fires 40 identical repeats per shape, every number becomes+# the cache's latency instead of the engine's. A first run of this harness+# without it reported `took: 0` for all 17 shapes and nearly led to the+# conclusion that no query work was measurable at all. See+# demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md.+nohup env XERJ_DISABLE_QUERY_CACHE=1 \+ "$BIN" -c "$DATA/xerj.toml" --data-dir "$DATA" --insecure \> "$DATA/server.log" 2>&1 &disown@@ -232,6 +247,22 @@ QUERIES = {"interval":250}}}},"agg_nested_terms_stats":{"size":0,"aggs":{"m":{"terms":{"field":"model"},"aggs":{"s":{"stats":{"field":"cost_usd"}}}}}},+ # --- the two read families READ_SCORECARD_2026-07-09.md leaves UNFIXED ---+ # Both still brute-scan and score every document: boosting ~163ms,+ # function_score ~157ms on 100k docs = 64.5% of that scorecard's total read+ # time. These are the largest remaining query-latency levers in the engine.+ "boosting": {"query":{"boosting":{+ "positive":{"match":{"body":"index0000"}},+ "negative":{"match":{"body":"merge0000"}},+ "negative_boost":0.2}}},+ "function_score": {"query":{"function_score":{+ "query":{"match":{"body":"index0000"}},+ "functions":[{"filter":{"term":{"status":"ok"}},"weight":2}]}}},+ # phrase families improved 5-6x in that sweep but remain super-linear (~40ms)+ "match_phrase_prefix": {"query":{"match_phrase_prefix":{"body":"index0000 segment"}}},+ "fuzzy_text": {"query":{"fuzzy":{"body":{"value":"index0001","fuzziness":1}}}},+ "wildcard_text": {"query":{"wildcard":{"body":"index00*"}}},+ "prefix_text": {"query":{"prefix":{"body":"index00"}}},# --- pathological: served from the TEXT doc-values column ---# ES rejects both (fielddata disabled on text by default). XERJ answers them# from `.dv`, keyed by the WHOLE document body, and they are by three orders ofdiff --git a/demo/playbooks/rc12/results-baseline-nocache.json b/demo/playbooks/rc12/results-baseline-nocache.jsonnew file mode 100644index 000000000..a19c97eb3--- /dev/null+++ b/demo/playbooks/rc12/results-baseline-nocache.json@@ -0,0 +1,24 @@+{+ "label": "baseline-nocache",+ "docs": 500000,+ "raw_source_bytes": 356593195,+ "index_total_bytes": 154413167,+ "index_over_raw": 0.433,+ "ingest_seconds": 31.86,+ "forcemerge_seconds": 139.13,+ "by_artifact": {+ "dv": 57357593,+ "seg": 51449817,+ "post": 40951589,+ "ids": 3057388,+ "meta": 1464327,+ "fst": 64485,+ "json": 35337,+ "norms": 28305,+ "wal": 4119,+ "toml": 104,+ "xerj_master_key": 64,+ "sidx": 32,+ "lock": 7+ }+}\ No newline at end of file