Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

test(perf): repeat run — the match_phrase_prefix regression did not reproduce

xerj-orga5b1c8a9565d
diff --git a/demo/playbooks/rc12/RESULT-fts-reader-cache.md b/demo/playbooks/rc12/RESULT-fts-reader-cache.md
index 924e881e7..3b3beb2c9 100644
--- a/demo/playbooks/rc12/RESULT-fts-reader-cache.md
+++ b/demo/playbooks/rc12/RESULT-fts-reader-cache.md
@@ -63,7 +63,46 @@ fuzzy; no change on brute-scan families."*
---
-## The one regression, stated plainly
+## Repeat run — the regression did not reproduce
+
+A second independent run (`results-ftscache-rep2.json`, fresh instance, fresh
+data dir, same seeded corpus) settles the `match_phrase_prefix` question:
+
+| query | base | run 1 | run 2 | run 2 vs base |
+|---|---:|---:|---:|---:|
+| `match_phrase_prefix` | 4369.24 | **4684.25** | **4460.21** | 0.98x |
+| `match_text_common` | 166.29 | 57.85 | 57.21 | **2.91x** |
+| `prefix_text` | 170.91 | 69.22 | 66.94 | **2.55x** |
+| `wildcard_text` | 180.52 | 74.79 | 73.52 | **2.46x** |
+| `large_page` | 168.96 | 64.05 | 64.36 | **2.63x** |
+| `fuzzy_text` | 306.16 | 195.25 | 197.24 | **1.55x** |
+| `match_phrase` | 495.58 | 339.90 | 327.65 | **1.51x** |
+| `match_text_multi` | 493.90 | 342.99 | 334.22 | **1.48x** |
+| `function_score` | 4608.30 | 4588.56 | 4564.46 | 1.01x |
+| `boosting` | 2708.92 | 2704.39 | 2713.27 | 1.00x |
+
+`match_phrase_prefix` moved 4684 → 4460 between the two cache runs, a 5% swing
+with no code change. Its earlier 0.93x was **run-to-run variance on a
+multi-second brute-scan query, not a regression** — as suspected but, correctly,
+not assumed.
+
+**The wins are stable to within ~2% across both runs**, which is the more
+important result: the 2.4–2.9x on the mid-tier full-text shapes reproduces
+exactly, and the brute-scan families are flat in both.
+
+### What this says about the harness
+
+Multi-second shapes carry ±5% run-to-run noise at 40 repetitions. Any future
+claim about `function_score`, `boosting`, `match_phrase_prefix`,
+`agg_terms_on_text`, `bool_must_filter` or `sort_on_text` needs either more
+repetitions or repeated runs — a single run cannot resolve anything smaller than
+about 10% on those. The sub-millisecond shapes are noisy in relative terms for
+the opposite reason (`agg_stats_numeric` 0.20 → 0.27 ms reads as "0.75x" but is
+70 microseconds); they should be read as a pass/fail guard, not as ratios.
+
+---
+
+## The one regression, stated plainly (superseded by the repeat run above)
`match_phrase_prefix` moved 4369.24 → 4684.25 ms (0.93x). This is **above noise
by the p99 test**: the cached p50 (4684) exceeds the base p99 (4651).
diff --git a/demo/playbooks/rc12/results-ftscache-rep2.json b/demo/playbooks/rc12/results-ftscache-rep2.json
new file mode 100644
index 000000000..531d1a1b3
--- /dev/null
+++ b/demo/playbooks/rc12/results-ftscache-rep2.json
@@ -0,0 +1,176 @@
+{
+ "label": "ftscache-rep2",
+ "docs": 500000,
+ "raw_source_bytes": 356593195,
+ "index_total_bytes": 154413171,
+ "index_over_raw": 0.433,
+ "ingest_seconds": 30.15,
+ "forcemerge_seconds": 105.75,
+ "by_artifact": {
+ "dv": 57357593,
+ "seg": 51449822,
+ "post": 40951589,
+ "ids": 3057388,
+ "meta": 1464327,
+ "fst": 64485,
+ "json": 35338,
+ "norms": 28305,
+ "wal": 4116,
+ "toml": 104,
+ "xerj_master_key": 64,
+ "sidx": 32,
+ "lock": 8
+ },
+ "queries": {
+ "term_keyword_low_card": {
+ "p50_ms": 0.623,
+ "p99_ms": 0.768,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "term_keyword_high_card": {
+ "p50_ms": 1.429,
+ "p99_ms": 1.617,
+ "took_ms": 1,
+ "hits": 1
+ },
+ "range_date_narrow": {
+ "p50_ms": 0.407,
+ "p99_ms": 0.526,
+ "took_ms": 0,
+ "hits": 50
+ },
+ "range_long_wide": {
+ "p50_ms": 0.38,
+ "p99_ms": 0.505,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "match_text_common": {
+ "p50_ms": 57.206,
+ "p99_ms": 57.949,
+ "took_ms": 56,
+ "hits": 10000
+ },
+ "match_text_multi": {
+ "p50_ms": 334.223,
+ "p99_ms": 347.116,
+ "took_ms": 333,
+ "hits": 10000
+ },
+ "match_phrase": {
+ "p50_ms": 327.649,
+ "p99_ms": 337.466,
+ "took_ms": 315,
+ "hits": 10000
+ },
+ "bool_must_filter": {
+ "p50_ms": 2750.581,
+ "p99_ms": 2775.305,
+ "took_ms": 2736,
+ "hits": 10000
+ },
+ "deep_page": {
+ "p50_ms": 61.19,
+ "p99_ms": 64.421,
+ "took_ms": 62,
+ "hits": 10000
+ },
+ "large_page": {
+ "p50_ms": 64.361,
+ "p99_ms": 65.596,
+ "took_ms": 59,
+ "hits": 10000
+ },
+ "sort_by_numeric": {
+ "p50_ms": 2.502,
+ "p99_ms": 2.687,
+ "took_ms": 1,
+ "hits": 10000
+ },
+ "agg_terms_low_card": {
+ "p50_ms": 0.463,
+ "p99_ms": 0.706,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "agg_terms_high_card": {
+ "p50_ms": 2.906,
+ "p99_ms": 3.712,
+ "took_ms": 2,
+ "hits": 10000
+ },
+ "agg_stats_numeric": {
+ "p50_ms": 0.272,
+ "p99_ms": 0.411,
+ "took_ms": 0,
+ "hits": 10000
+ },
+ "agg_date_histogram": {
+ "p50_ms": 17.208,
+ "p99_ms": 17.379,
+ "took_ms": 16,
+ "hits": 10000
+ },
+ "agg_histogram_numeric": {
+ "p50_ms": 10.518,
+ "p99_ms": 11.056,
+ "took_ms": 10,
+ "hits": 10000
+ },
+ "agg_nested_terms_stats": {
+ "p50_ms": 3.384,
+ "p99_ms": 3.885,
+ "took_ms": 3,
+ "hits": 10000
+ },
+ "boosting": {
+ "p50_ms": 2713.273,
+ "p99_ms": 2748.389,
+ "took_ms": 2723,
+ "hits": 10000
+ },
+ "function_score": {
+ "p50_ms": 4564.462,
+ "p99_ms": 4601.268,
+ "took_ms": 4577,
+ "hits": 10000
+ },
+ "match_phrase_prefix": {
+ "p50_ms": 4460.21,
+ "p99_ms": 4633.372,
+ "took_ms": 4311,
+ "hits": 10000
+ },
+ "fuzzy_text": {
+ "p50_ms": 197.238,
+ "p99_ms": 200.498,
+ "took_ms": 196,
+ "hits": 10000
+ },
+ "wildcard_text": {
+ "p50_ms": 73.52,
+ "p99_ms": 76.535,
+ "took_ms": 73,
+ "hits": 10000
+ },
+ "prefix_text": {
+ "p50_ms": 66.945,
+ "p99_ms": 70.127,
+ "took_ms": 65,
+ "hits": 10000
+ },
+ "agg_terms_on_text": {
+ "p50_ms": 3443.724,
+ "p99_ms": 3935.99,
+ "took_ms": 3422,
+ "hits": 10000
+ },
+ "sort_on_text": {
+ "p50_ms": 1586.369,
+ "p99_ms": 1657.992,
+ "took_ms": 1651,
+ "hits": 10000
+ }
+ }
+}
\ No newline at end of file