Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

test(perf): measure the FTS reader cache — 2.4-2.9x on full-text, 1.03x on the suite

xerj-orga26c7a96222f
diff --git a/demo/playbooks/rc12/RESULT-fts-reader-cache.md b/demo/playbooks/rc12/RESULT-fts-reader-cache.md
new file mode 100644
index 000000000..924e881e7
--- /dev/null
+++ b/demo/playbooks/rc12/RESULT-fts-reader-cache.md
@@ -0,0 +1,96 @@
+# Measured: per-segment FTS reader cache
+
+A/B on the same seeded 500k-doc corpus, force-merged to one segment,
+`XERJ_DISABLE_QUERY_CACHE=1`, 40 runs per shape after 5 warm-ups.
+
+- before: `results-baseline-nocache.json` (`main` @ 8a6fa38)
+- after: `results-ftscache.json` (branch `perf/fts-reader-cache`)
+
+The binary measured predates the `_nodes/stats` reporting commit, which changes
+only JSON output and cannot affect query latency.
+
+---
+
+## Result
+
+| query | base p50 ms | cached p50 ms | speedup |
+|---|---:|---:|---:|
+| `match_text_common` | 166.29 | **57.85** | **2.87x** |
+| `large_page` | 168.96 | **64.05** | **2.64x** |
+| `prefix_text` | 170.91 | **69.22** | **2.47x** |
+| `wildcard_text` | 180.52 | **74.79** | **2.41x** |
+| `fuzzy_text` | 306.16 | **195.25** | **1.57x** |
+| `match_phrase` | 495.58 | 339.90 | 1.46x |
+| `match_text_multi` | 493.90 | 342.99 | 1.44x |
+| `bool_must_filter` | 2846.62 | 2764.02 | 1.03x |
+| `agg_terms_on_text` | 3414.46 | 3370.50 | 1.01x |
+| `boosting` | 2708.92 | 2704.39 | 1.00x |
+| `function_score` | 4608.30 | 4588.56 | 1.00x |
+| `sort_on_text` | 1535.11 | 1576.16 | 0.97x |
+| `deep_page` | 56.48 | 59.06 | 0.96x |
+| `match_phrase_prefix` | 4369.24 | 4684.25 | **0.93x** |
+| **sum of 25 shapes** | 21,560.6 | 20,930.8 | **1.03x** |
+
+Disk is unchanged (154,413,167 → 154,413,175 B, 8 bytes on 147 MiB). Ingest
+31.86 → 31.24 s and force-merge 139.13 → 127.45 s are unaffected by design — the
+cache is read-path only, and both deltas are within run-to-run variance.
+
+---
+
+## What it does and does not buy
+
+**It buys 2.4–2.9x on the mid-tier full-text shapes**, which is exactly the set
+whose cost was dominated by reopening the segment side-cars. That confirms the
+mechanism: `FtsIndexReader::open` inflating the whole `.post` (ZPS1) and `.meta`
+(ZFM4) per query was the dominant term for those queries and is now paid once.
+
+**It buys nothing on the four multi-second shapes** (`function_score`,
+`match_phrase_prefix`, `agg_terms_on_text`, `boosting`, `bool_must_filter`),
+because their cost is brute scanning and scoring every document, not opening a
+reader. Those need separate work.
+
+### The aggregate is 1.03x, and that is the honest number for "the suite"
+
+Because the four brute-scan shapes dominate the sum of p50s, the total barely
+moves. This is a concrete demonstration that **"sum of independent p50s" is a bad
+headline metric** — the same criticism an adversarial verifier levelled at the
+2.70x figure in `READ_SCORECARD_2026-07-09.md`. Quote the per-shape numbers, or
+quote a real workload mix; never quote the sum.
+
+**Defensible claim for rc.12:** *"2.4–2.9x on common full-text queries
+(`match`, `prefix`, `wildcard`, large result pages); 1.4–1.6x on phrase and
+fuzzy; no change on brute-scan families."*
+
+---
+
+## The one regression, stated plainly
+
+`match_phrase_prefix` moved 4369.24 → 4684.25 ms (0.93x). This is **above noise
+by the p99 test**: the cached p50 (4684) exceeds the base p99 (4651).
+
+I investigated the obvious mechanism — that the new cache competes for the shared
+`SegmentHydrationBudget` and starves the stored-scan caches — and **refuted it**.
+On the post-change instance:
+
+```
+limit_in_bytes 25,607,935,590
+current_in_bytes 3,830,604,688 (15% of budget)
+admission_refusals 0
+```
+
+Nothing was refused and the budget has ample headroom, so budget competition is
+not the cause.
+
+**What I have not established** is the actual cause. The two runs are separate
+server processes over separate data directories, so page-cache state and merge
+scheduling differ (force-merge took 127 s vs 139 s, implying a different merge
+schedule). For a 4.4-second brute-scan query, 7% run-to-run variance between
+independent instances is plausible but unproven. The small moves elsewhere
+(`sort_on_text` 0.97x, `deep_page` 0.96x, `agg_stats_numeric` 0.20→0.27 ms) fit
+the same variance pattern.
+
+**This must be settled by repeated runs before the change merges.** Either it
+reproduces — in which case it is a real regression in the multi-term expansion
+path and needs a cause — or it does not, in which case the harness needs more
+repetitions per shape for multi-second queries to be trustworthy at all. Do not
+merge on the assumption that it is noise.
diff --git a/demo/playbooks/rc12/results-ftscache.json b/demo/playbooks/rc12/results-ftscache.json
index dd497e922..9b853ddf0 100644
--- a/demo/playbooks/rc12/results-ftscache.json
+++ b/demo/playbooks/rc12/results-ftscache.json
@@ -20,5 +20,157 @@
"xerj_master_key": 64,
"sidx": 32,
"lock": 8
+ },
+ "queries": {
+ "term_keyword_low_card": {
+ "p50_ms": 0.626,
+ "p99_ms": 0.745,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "term_keyword_high_card": {
+ "p50_ms": 1.467,
+ "p99_ms": 1.643,
+ "took_ms": 1,
+ "hits": 1
+ },
+ "range_date_narrow": {
+ "p50_ms": 0.323,
+ "p99_ms": 0.419,
+ "took_ms": 0,
+ "hits": 50
+ },
+ "range_long_wide": {
+ "p50_ms": 0.359,
+ "p99_ms": 0.49,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "match_text_common": {
+ "p50_ms": 57.85,
+ "p99_ms": 59.141,
+ "took_ms": 57,
+ "hits": 10000
+ },
+ "match_text_multi": {
+ "p50_ms": 342.994,
+ "p99_ms": 366.262,
+ "took_ms": 340,
+ "hits": 10000
+ },
+ "match_phrase": {
+ "p50_ms": 339.905,
+ "p99_ms": 360.655,
+ "took_ms": 337,
+ "hits": 10000
+ },
+ "bool_must_filter": {
+ "p50_ms": 2764.016,
+ "p99_ms": 2790.642,
+ "took_ms": 2730,
+ "hits": 10000
+ },
+ "deep_page": {
+ "p50_ms": 59.059,
+ "p99_ms": 60.854,
+ "took_ms": 49,
+ "hits": 10000
+ },
+ "large_page": {
+ "p50_ms": 64.047,
+ "p99_ms": 65.32,
+ "took_ms": 61,
+ "hits": 10000
+ },
+ "sort_by_numeric": {
+ "p50_ms": 2.45,
+ "p99_ms": 2.634,
+ "took_ms": 1,
+ "hits": 10000
+ },
+ "agg_terms_low_card": {
+ "p50_ms": 0.441,
+ "p99_ms": 0.594,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "agg_terms_high_card": {
+ "p50_ms": 2.892,
+ "p99_ms": 3.573,
+ "took_ms": 2,
+ "hits": 10000
+ },
+ "agg_stats_numeric": {
+ "p50_ms": 0.269,
+ "p99_ms": 0.403,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "agg_date_histogram": {
+ "p50_ms": 17.162,
+ "p99_ms": 17.328,
+ "took_ms": 16,
+ "hits": 10000
+ },
+ "agg_histogram_numeric": {
+ "p50_ms": 10.517,
+ "p99_ms": 10.959,
+ "took_ms": 10,
+ "hits": 10000
+ },
+ "agg_nested_terms_stats": {
+ "p50_ms": 3.319,
+ "p99_ms": 3.785,
+ "took_ms": 3,
+ "hits": 10000
+ },
+ "boosting": {
+ "p50_ms": 2704.395,
+ "p99_ms": 2730.551,
+ "took_ms": 2728,
+ "hits": 10000
+ },
+ "function_score": {
+ "p50_ms": 4588.564,
+ "p99_ms": 4642.287,
+ "took_ms": 4628,
+ "hits": 10000
+ },
+ "match_phrase_prefix": {
+ "p50_ms": 4684.251,
+ "p99_ms": 4902.213,
+ "took_ms": 4674,
+ "hits": 10000
+ },
+ "fuzzy_text": {
+ "p50_ms": 195.252,
+ "p99_ms": 200.411,
+ "took_ms": 194,
+ "hits": 10000
+ },
+ "wildcard_text": {
+ "p50_ms": 74.787,
+ "p99_ms": 76.372,
+ "took_ms": 73,
+ "hits": 10000
+ },
+ "prefix_text": {
+ "p50_ms": 69.219,
+ "p99_ms": 72.784,
+ "took_ms": 66,
+ "hits": 10000
+ },
+ "agg_terms_on_text": {
+ "p50_ms": 3370.501,
+ "p99_ms": 3531.283,
+ "took_ms": 3417,
+ "hits": 10000
+ },
+ "sort_on_text": {
+ "p50_ms": 1576.156,
+ "p99_ms": 1599.107,
+ "took_ms": 1577,
+ "hits": 10000
+ }
}
}
\ No newline at end of file