Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

test(perf): correct the text doc-values evidence — the target is semantic_text

xerj-org61a52c4b5608
diff --git a/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md
index 55eeee431..24babbaf7 100644
--- a/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md
+++ b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md
@@ -99,9 +99,25 @@ curl -s -X POST localhost:9400/rc12bench/_search -H 'Content-Type: application/j
-d '{"size":1,"sort":[{"body":"asc"}]}'
```
-**Both succeed**, served from the `.dv` column. Elasticsearch rejects both by
-default (`Fielddata is disabled on text fields by default`) because neither is
-meaningful on analyzed text.
+**Both succeed**, served from the `.dv` column. Elasticsearch rejects both **by
+default** (`Fielddata is disabled on text fields by default`).
+
+> **Correction (important).** An earlier draft of this file said the whole-value
+> bucketing below is "semantically wrong" and that ES refuses it outright. That is
+> true only of *default* ES and of classic Lucene. Modern Elasticsearch supports
+> text doc-values behind the cluster feature `mapper.text.doc_values`, by attaching
+> a separate `SORTED_SET` doc-values field — and **XERJ's own conformance suite
+> already encodes that behaviour**:
+> `tests/es-compat-yaml/yaml/aggregations/terms_text_docvalues.yml` maps
+> `{type: text, index: false, doc_values: true}` and asserts whole-value buckets
+> (`"foo bar": 2`, `"baz qux": 1`).
+>
+> So whole-value bucketing is the *correct* answer when a user explicitly asks for
+> `doc_values: true` on a text field. The defect is not the shape of the answer —
+> it is that XERJ builds the column **unconditionally**, for every text field,
+> whether or not anyone asked. That reframes the lever from "never build
+> doc-values for text" to **"default off, and honour the flag"**, which is both the
+> real ES behaviour and the only version that keeps the 1360/0/3 gate green.
| operation | engine `took` | result |
|---|---:|---|
@@ -128,25 +144,43 @@ why `sum_other_doc_count` is 65,533 and the answer carries no information.
## What this establishes
-The text doc-values column is not a trade. It is a **triple defect**:
+The text doc-values column is built **unconditionally**, and that costs on two axes:
1. **Disk.** 37.1% of the index on the 500k mixed corpus, 39.7% on a pure-text
index. The largest single artifact in both.
-2. **Semantics.** The two operations it enables return answers that are wrong in
- kind, not merely imprecise — bucketing by document instead of by term. ES
- refuses them for exactly this reason.
-3. **Latency.** Those same two operations are the slowest in the engine by a wide
- margin (7.8 s and 1.65 s against `took: 0` everywhere else).
-
-So the read-path interaction that would have made this a trade does not exist in
-the form feared. Nothing *useful* reads the text `.dv`. Dropping it removes the
-largest artifact, removes two wrong answers, and removes the two slowest
-operations.
-
-**Consequence to decide explicitly:** after the change, `terms`-agg and `sort` on a
-`text` field must **error** the way ES does, rather than silently returning fewer
-results. That is a deliberate, documented behaviour change and it needs an ES-YAML
-conformance check — the suite must stay at 1360 passed / 0 failed / 3 skipped.
+2. **Latency.** The two operations it enables are the slowest in the engine by a
+ wide margin (7.8 s and 1.65 s against `took: 0` everywhere else) — and today
+ every index pays the disk cost for them whether or not anyone ever runs one.
+
+Nothing reads this column unless a user explicitly aggregates or sorts on a text
+field, which is rare and which ES requires you to opt into.
+
+### The design this implies — and the trap in the RFC
+
+Discussion #148 says "ES/Lucene never builds doc_values for `text`; honouring
+`doc_values: false` drops the 22.9 MB `.dv` entirely." Implemented literally that
+**fails the conformance gate**, because of the `terms_text_docvalues.yml` cases
+above. The correct shape is:
+
+- **default off** for `text` (matching ES's default), and
+- **honour an explicit `doc_values: true`**, which still builds the column and
+ still returns whole-value buckets.
+
+### The detail that decides whether this lever is worth anything
+
+Measured across live corpora by parsing the `DV01` envelope and joining to each
+index's `_mapping`:
+
+| corpus | `semantic_text` | `text` | keyword | numeric |
+|---|---:|---:|---:|---:|
+| `xc-lucene*` (81 indices, 23.8 MiB `.dv`) | 86.1% | 4.7% | 8.2% | 1.0% |
+| `xc-dataplane-cilium-vendor*` (83 indices, 61.0 MiB `.dv`) | 91.9% | 5.2% | — | — |
+
+**`semantic_text`, not `text`, is where the bytes are.** Both map to
+`FieldType::Text` (`es_compat.rs:13678`), so one schema-level check captures
+90.8–97.1%. A policy keyed on the literal string `"text"` would capture under 5%
+and be nearly worthless. The RFC does not mention this distinction at all, and it
+is the single most important implementation detail in the lever.
**Still to determine:** whether the doc-values *prefilters* (term/terms/range/bool)
or the `size:0` count shortcut ever bind to a `text` field. Those paths are real and
diff --git a/demo/playbooks/rc12/measure_rc12.sh b/demo/playbooks/rc12/measure_rc12.sh
index cbe6fd01f..5da52f653 100755
--- a/demo/playbooks/rc12/measure_rc12.sh
+++ b/demo/playbooks/rc12/measure_rc12.sh
@@ -34,8 +34,15 @@ echo " data: $DATA"
# ── fresh instance ───────────────────────────────────────────────────────────
# There is no --port flag; the listen ports come from a TOML config.
-pkill -f "[x]erj -c $DATA/xerj.toml" 2>/dev/null || true
-sleep 1
+# Free the PORT, not just this label's data dir: a previous run under a
+# DIFFERENT label still owns the socket, and killing only our own config path
+# leaves it up. The new server then fails to bind and exits, while every
+# subsequent request silently hits the OLD instance — which surfaces as a
+# baffling 409 on index creation rather than as a bind error.
+for pid in $(lsof -ti tcp:"$PORT" 2>/dev/null); do
+ kill "$pid" 2>/dev/null || true
+done
+sleep 2
rm -rf "$DATA"; mkdir -p "$DATA"
cat > "$DATA/xerj.toml" <<EOF
[server]
@@ -44,7 +51,15 @@ es_compat_port = $PORT
rest_port = $((PORT + 1))
grpc_port = $((PORT + 2))
EOF
-nohup "$BIN" -c "$DATA/xerj.toml" --data-dir "$DATA" --insecure \
+# XERJ_DISABLE_QUERY_CACHE=1 is MANDATORY, not optional. Without it the
+# whole-result cache clones the answer for a repeated identical query body, and
+# since this harness fires 40 identical repeats per shape, every number becomes
+# the cache's latency instead of the engine's. A first run of this harness
+# without it reported `took: 0` for all 17 shapes and nearly led to the
+# conclusion that no query work was measurable at all. See
+# demo/playbooks/CRITICAL_FINDING_read_perf_cache_mirage.md.
+nohup env XERJ_DISABLE_QUERY_CACHE=1 \
+ "$BIN" -c "$DATA/xerj.toml" --data-dir "$DATA" --insecure \
> "$DATA/server.log" 2>&1 &
disown
@@ -232,6 +247,22 @@ QUERIES = {
"interval":250}}}},
"agg_nested_terms_stats":{"size":0,"aggs":{"m":{"terms":{"field":"model"},
"aggs":{"s":{"stats":{"field":"cost_usd"}}}}}},
+ # --- the two read families READ_SCORECARD_2026-07-09.md leaves UNFIXED ---
+ # Both still brute-scan and score every document: boosting ~163ms,
+ # function_score ~157ms on 100k docs = 64.5% of that scorecard's total read
+ # time. These are the largest remaining query-latency levers in the engine.
+ "boosting": {"query":{"boosting":{
+ "positive":{"match":{"body":"index0000"}},
+ "negative":{"match":{"body":"merge0000"}},
+ "negative_boost":0.2}}},
+ "function_score": {"query":{"function_score":{
+ "query":{"match":{"body":"index0000"}},
+ "functions":[{"filter":{"term":{"status":"ok"}},"weight":2}]}}},
+ # phrase families improved 5-6x in that sweep but remain super-linear (~40ms)
+ "match_phrase_prefix": {"query":{"match_phrase_prefix":{"body":"index0000 segment"}}},
+ "fuzzy_text": {"query":{"fuzzy":{"body":{"value":"index0001","fuzziness":1}}}},
+ "wildcard_text": {"query":{"wildcard":{"body":"index00*"}}},
+ "prefix_text": {"query":{"prefix":{"body":"index00"}}},
# --- pathological: served from the TEXT doc-values column ---
# ES rejects both (fielddata disabled on text by default). XERJ answers them
# from `.dv`, keyed by the WHOLE document body, and they are by three orders of
diff --git a/demo/playbooks/rc12/results-baseline-nocache.json b/demo/playbooks/rc12/results-baseline-nocache.json
new file mode 100644
index 000000000..a19c97eb3
--- /dev/null
+++ b/demo/playbooks/rc12/results-baseline-nocache.json
@@ -0,0 +1,24 @@
+{
+ "label": "baseline-nocache",
+ "docs": 500000,
+ "raw_source_bytes": 356593195,
+ "index_total_bytes": 154413167,
+ "index_over_raw": 0.433,
+ "ingest_seconds": 31.86,
+ "forcemerge_seconds": 139.13,
+ "by_artifact": {
+ "dv": 57357593,
+ "seg": 51449817,
+ "post": 40951589,
+ "ids": 3057388,
+ "meta": 1464327,
+ "fst": 64485,
+ "json": 35337,
+ "norms": 28305,
+ "wal": 4119,
+ "toml": 104,
+ "xerj_master_key": 64,
+ "sidx": 32,
+ "lock": 7
+ }
+}
\ No newline at end of file