AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
test(perf): add the rc.12 measurement harness and capture the baseline
xerj-orga28302c60d61
diff --git a/demo/playbooks/rc12/BASELINE.md b/demo/playbooks/rc12/BASELINE.mdnew file mode 100644index 000000000..fb03f3941--- /dev/null+++ b/demo/playbooks/rc12/BASELINE.md@@ -0,0 +1,119 @@+# rc.12 baseline — measured 2026-08-07 on `main` @ 8a6fa38 (v1.0.0-rc.11)++Produced by `demo/playbooks/rc12/measure_rc12.sh baseline-main 500000`.+Raw numbers: `results-baseline-main.json`.++**Corpus.** 500,000 documents, one index, force-merged to a single segment so we+measure steady state. Mapping is explicit (6 keyword, 1 analyzed `text`, 4+numeric, 1 date, 1 boolean). The `body` field is 40 tokens drawn **Zipfian from a+60,000-term vocabulary** — this matters: an earlier version of this harness used a+24-word vocabulary and produced a 0.0595x ratio and uniformly sub-0.5ms queries,+which flattered both axes into meaninglessness. Realistic term diversity is a+precondition for this measurement being worth anything.++---++## Axis 1 — disk++| | value |+|---|---:|+| raw source bytes | 356,593,195 (340 MiB) |+| index on disk | 154,413,168 (147 MiB) |+| **index / raw** | **0.433x** |+| ingest | 33.0 s (≈15.2k docs/s) |+| force-merge to 1 segment | **116.52 s** |++Where the bytes are:++| artifact | bytes | share | what it is |+|---|---:|---:|---|+| **`.dv`** | **57,357,593** | **37.1%** | doc-values — full-value column for every string, *including the analyzed body* |+| `.seg` | 51,449,817 | 33.3% | stored `_source`, zstd |+| `.post` | 40,951,589 | 26.5% | postings incl. positions |+| `.ids` | 3,057,388 | 2.0% | external-id → ordinal index, stored raw |+| `.meta` | 1,464,327 | 0.9% | per-term metadata |+| `.fst` | 64,485 | 0.0% | term dictionary |+| `.norms` | 28,305 | 0.0% | per-doc field length |++**`.dv` is the largest artifact at 37.1%**, independently corroborating discussion+#148's 34% measurement on a completely different corpus (WooCommerce source). See+`EVIDENCE-doc-values-on-text.md` for the isolated experiment proving doc-values are+built for `text` fields at all, and that `"doc_values": false` is silently ignored.++---++## Axis 2 — query latency, and why "2x" cannot honestly be claimed here++| query | p50 ms | p99 ms | engine `took` | hits |+|---|---:|---:|---:|---:|+| term_keyword_low_card | 0.30 | 0.49 | **0** | 10,000 |+| term_keyword_high_card | 0.21 | 0.56 | **0** | 1 |+| range_date_narrow | 0.33 | 0.58 | **0** | 50 |+| range_long_wide | 0.30 | 0.46 | **0** | 10,000 |+| match_text_common | 0.29 | 0.36 | **0** | 10,000 |+| match_text_multi | 0.31 | 0.41 | **0** | 10,000 |+| match_phrase | 0.28 | 0.54 | **0** | 10,000 |+| bool_must_filter | 0.37 | 0.74 | **0** | 10,000 |+| deep_page (from 9000) | 1.04 | 1.16 | **0** | 10,000 |+| large_page (size 500) | **4.72** | 5.94 | **0** | 10,000 |+| sort_by_numeric | 1.17 | 1.38 | **0** | 10,000 |+| agg_terms_low_card | 0.21 | 0.30 | **0** | 10,000 |+| agg_terms_high_card | 0.24 | 0.35 | **0** | 10,000 |+| agg_stats_numeric | 0.19 | 0.31 | **0** | 10,000 |+| agg_date_histogram | 0.33 | 0.46 | **0** | 10,000 |+| agg_histogram_numeric | 0.23 | 0.38 | **0** | 10,000 |+| agg_nested_terms_stats | 0.21 | 0.61 | **0** | 10,000 |++### The engine reports `took: 0` for every single query.++That is the headline, and it constrains what rc.12 may claim. At 500k documents on+a warm single segment, XERJ's own timer cannot resolve any of these queries, and+the 0.2–0.6 ms wall-clock is HTTP, JSON serialisation, and loopback — not engine+work. **You cannot demonstrate a 2x speedup on a quantity that already measures+zero.** Any "2x faster" claim built on these shapes would be measuring the harness,+not the engine, which is exactly the saturation artifact that invalidated an+earlier round of XERJ benchmarking.++Only three shapes do measurable work, and they are dominated by *materialisation*+rather than matching:++- `large_page` (4.72 ms) — hydrating 500 documents from `_source`+- `sort_by_numeric` (1.17 ms) — sorting 500k docs by a numeric column+- `deep_page` (1.04 ms) — skipping 9,000 documents++Note also that `hits` saturates at exactly **10,000** for every broad query; that is+the scored-total cap, not a corpus property.++### Where a real, measurable 2x actually lives++On this evidence the honest performance targets for rc.12 are **not** point-query+latency. They are:++1. **Force-merge: 116.52 s for 500k docs.** Real, large, and directly measurable.+ This is also the number that any merge-time compression lever must be scored+ against — the 2026-07-09 playbook's merge-path zstd-19 proposal raises merge CPU+ ~8x, and it must not be scheduled without measuring it here first.+2. **Ingest: 33.0 s (15.2k docs/s).**+3. **Hydration/paging cost**, visible in `large_page`.++To make point-query latency measurable at all would require a substantially larger+corpus or a closed-loop concurrent load generator. Until such a harness exists, the+correct statement for rc.12 is that query latency is **at the measurement floor and+must not regress** — a guard, not an improvement target.++---++## How to reproduce and compare++```sh+# baseline (already captured)+demo/playbooks/rc12/measure_rc12.sh baseline-main 500000++# after a change, on the same corpus and doc count+demo/playbooks/rc12/measure_rc12.sh <change-label> 500000++# then diff results-baseline-main.json against results-<change-label>.json+```++The corpus generator is seeded (`random.Random(20261207)`), so both runs index a+byte-identical corpus and the A/B is valid.diff --git a/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.mdnew file mode 100644index 000000000..17fa6672d--- /dev/null+++ b/demo/playbooks/rc12/EVIDENCE-doc-values-on-text.md@@ -0,0 +1,97 @@+# Evidence: doc-values are built for `text` fields, and `"doc_values": false` is ignored++Measured 2026-08-07 against `xerj v1.0.0-rc.11` (commit `8a6fa38`), live instance.+Two independent experiments. Both are reproducible with the commands shown.++This file records *empirical* evidence for two claims in discussion #148. It does+not project a saving — see the rc.12 plan for that arithmetic.++---++## Experiment 1 — `"doc_values": false` is accepted, stored, and silently ignored++This is a **correctness bug**, not only a footprint issue: XERJ accepts a mapping+option, echoes it back on `GET _mapping`, and then does not honour it.++```sh+curl -s -X PUT localhost:9200/dvtest -H 'Content-Type: application/json' -d '{+ "mappings":{"properties":{+ "kw_dv_off":{"type":"keyword","doc_values":false},+ "kw_dv_on":{"type":"keyword"},+ "body":{"type":"text"}}}}'+```++`GET /dvtest/_mapping` returns the field with `"doc_values": false` intact — so the+option survives round-trip. Then, with two documents indexed:++```sh+# Aggregate on the doc_values:false field+curl -s -X POST localhost:9200/dvtest/_search -H 'Content-Type: application/json' \+ -d '{"size":0,"aggs":{"a":{"terms":{"field":"kw_dv_off"}}}}'+```++**Result: it succeeds**, returning `buckets: [{alpha,1},{beta,1}]`. Sorting on the+same field also succeeds and returns a populated `sort` key.++Elasticsearch fails both with an explicit error, because doc-values are the only+structure that can serve an aggregation or a sort on a keyword field. XERJ+succeeding proves the doc-values were built regardless of the mapping.++**Why this matters beyond bytes.** A user who sets `"doc_values": false` is making+an explicit storage decision. Today they get neither the saving nor an error, and+nothing in the response tells them the option did nothing. Whatever we decide about+the footprint lever, the honest options are to *honour* the flag or to *reject* it —+silently ignoring an accepted option is the one behaviour that cannot be defended.++---++## Experiment 2 — a `text`-only index still builds a full `.dv` column++The sharper test: an index whose mapping contains **exactly one field, of type+`text`**. There is no keyword, no numeric, no date. Lucene builds **zero**+doc-values for such an index, because you cannot sort or aggregate on an analyzed+body and the bytes would be unusable.++```sh+curl -s -X PUT localhost:9200/dvtext -H 'Content-Type: application/json' \+ -d '{"mappings":{"properties":{"body":{"type":"text"}}}}'+# 2,000 docs, each a 60-token body drawn from a 5,000-term vocabulary+# ... bulk index ... then:+curl -s -X POST "localhost:9200/dvtext/_forcemerge?max_num_segments=1"+curl -s -X POST "localhost:9200/dvtext/_flush"+```++Segment artifacts after force-merge to a single segment:++| artifact | bytes | share |+|---|---:|---:|+| `.seg` stored `_source` | 39,656 | 43.3% |+| **`.dv` doc-values** | **36,333** | **39.7%** |+| `.post` postings | 9,142 | 10.0% |+| `.fst` term dictionary | 187 | 0.2% |+| **total** | **85,318** | |++**The doc-values column is four times the size of the postings** on a pure-text+index, and it is storing the full value of every analyzed body — the one thing+that can never be sorted or aggregated on.++This is consistent with the RFC's measurement on the WooCommerce corpus (`.dv` =+22.92 MiB of 68.14 MiB = 34%). The isolated experiment shows the effect is not an+artifact of that corpus's keyword fields: it is inherent to how XERJ treats `text`.++---++## What this does and does not establish++**Established:** doc-values are built for analyzed `text` fields; the+`"doc_values": false` mapping option is accepted and ignored; on a pure-text index+the resulting column is ~40% of the segment.++**Not established here:** the net saving on a realistic mixed corpus, and — the+crux — what currently *reads* `.dv` for a text field. XERJ's read path grew+doc-values prefilters for `term`/`terms`/`range`/`bool`, and the `size:0` count+shortcut reads from doc-values too. If any of those fire on `text` fields, naively+dropping the column moves queries onto a slower path instead of saving anything,+and the disk win would be paid for in latency. That interaction must be settled+before this lever is scheduled — it is the difference between a real win and a+trade.diff --git a/demo/playbooks/rc12/measure_rc12.sh b/demo/playbooks/rc12/measure_rc12.shnew file mode 100755index 000000000..bf6f401f8--- /dev/null+++ b/demo/playbooks/rc12/measure_rc12.sh@@ -0,0 +1,277 @@+#!/usr/bin/env bash+# rc.12 gate harness — measures BOTH axes the release must defend:+# (1) on-disk bytes per artifact type for the same logical corpus+# (2) query latency across the shapes rc.12 claims to improve+#+# Methodology follows demo/playbooks/DISK_SIZE_2026-07-09.md so the numbers are+# comparable to that baseline: deterministic corpus, single index, force-merged+# to one segment so we measure STEADY STATE rather than a transient many-segment+# ingest state (that conflation is what produced the bogus "XERJ ~2x ES disk"+# claim historically).+#+# Usage:+# measure_rc12.sh <label> [docs]+# Writes results to demo/playbooks/rc12/results-<label>.json and prints a table.+#+# Run it once on the base commit and once after a change; compare the JSONs.+set -euo pipefail++LABEL="${1:?usage: measure_rc12.sh <label> [docs]}"+DOCS="${2:-100000}"+PORT="${XERJ_BENCH_PORT:-9400}"+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"+BIN="${XERJ_BIN:-$ROOT/engine/target/release/xerj}"+DATA="${XERJ_BENCH_DATA:-/tmp/claude-1001/xerj-rc12-bench/$LABEL}"+OUT="$ROOT/demo/playbooks/rc12/results-$LABEL.json"+IDX="rc12bench"++[ -x "$BIN" ] || { echo "no xerj binary at $BIN — build it first" >&2; exit 2; }+mkdir -p "$(dirname "$OUT")"++echo "==> rc.12 harness: label=$LABEL docs=$DOCS port=$PORT"+echo " binary: $BIN"+echo " data: $DATA"++# ── fresh instance ───────────────────────────────────────────────────────────+# There is no --port flag; the listen ports come from a TOML config.+pkill -f "[x]erj -c $DATA/xerj.toml" 2>/dev/null || true+sleep 1+rm -rf "$DATA"; mkdir -p "$DATA"+cat > "$DATA/xerj.toml" <<EOF+[server]+bind_address = "127.0.0.1"+es_compat_port = $PORT+rest_port = $((PORT + 1))+grpc_port = $((PORT + 2))+EOF+nohup "$BIN" -c "$DATA/xerj.toml" --data-dir "$DATA" --insecure \+ > "$DATA/server.log" 2>&1 &+disown++URL="http://localhost:$PORT"+for _ in $(seq 1 120); do+ curl -fsS -m 3 "$URL/_cluster/health" 2>/dev/null | grep -q '"status":"\(green\|yellow\)"' && break+ sleep 1+done+curl -fsS -m 5 "$URL/_cluster/health" >/dev/null || { echo "server never came up" >&2; exit 1; }+echo "==> server up"++# ── corpus: deterministic, mixed-type (keyword + text + numeric + date + bool) ─+# Mirrors the 2026-07-09 baseline shape so artifact shares are comparable.+CORPUS="$DATA/corpus.ndjson"+python3 - "$DOCS" "$IDX" "$CORPUS" <<'PY'+import json, random, sys+n, idx, out = int(sys.argv[1]), sys.argv[2], sys.argv[3]++# The `body` text must have REALISTIC term diversity or the measurement is+# worthless: a small vocabulary makes postings compress absurdly well and makes+# every text query trivially fast, which flatters both axes. Real text is+# Zipfian over a large vocabulary, so we synthesise that explicitly.+# Seeded, so the corpus is byte-identical between runs and A/B is valid.+rng = random.Random(20261207)+VOCAB_N = 60000+STEMS = ("index segment posting token merge flush shard query filter score rank "+ "cache buffer stream codec vector cluster commit replica offset column "+ "record field parse encode decode compress bitmap ordinal dictionary").split()+VOCAB = [f"{STEMS[i % len(STEMS)]}{i//len(STEMS):04d}" for i in range(VOCAB_N)]+# Zipf weights: a few very common terms, a long tail of rare ones.+# Draw ONE large pool up front rather than sampling per token: per-call weighted+# sampling over a 60k vocabulary rebuilds the cumulative table every time and is+# far too slow at corpus scale. A rolling window over the pool preserves the+# distribution while making generation linear.+WEIGHTS = [1.0 / (i + 1) ** 1.07 for i in range(VOCAB_N)]+POOL_N = 2_000_003 # prime, so the rolling window does not resonate+POOL = rng.choices(VOCAB, weights=WEIGHTS, k=POOL_N)++MODELS = ["fable-5", "opus-5", "sonnet-5", "haiku-4-5"]+STATUS = ["ok", "error", "timeout", "throttled"]+REGION = ["us-east-1", "eu-west-2", "ap-south-1"]+with open(out, "w") as f:+ for i in range(n):+ # ~40 tokens drawn Zipfian — drives realistic positional postings and+ # gives phrase/match queries a real term-frequency distribution.+ off = (i * 41) % (POOL_N - 40)+ body = " ".join(POOL[off:off + 40])+ doc = {+ "doc_id": f"doc-{i:08d}", # high-cardinality keyword+ "model": MODELS[i % len(MODELS)], # low-cardinality keyword+ "status": STATUS[i % len(STATUS)],+ "region": REGION[i % len(REGION)],+ "session": f"sess-{i % 5000:05d}",+ "tenant": f"tenant-{i % 97:03d}",+ "body": body, # analyzed text+ "latency_ms": (i * 37) % 5000, # integer-structured+ "context_tokens": 1000 + (i % 100000), # monotonic-ish+ "cost_usd": round((i % 997) / 1000.0, 4),# float+ "retries": i % 5, # tiny range+ "score": round(((i * 13) % 1000) / 100.0, 2),+ "ts": 1700000000000 + i * 1000, # epoch-ms, monotonic+ "cached": (i % 3 == 0), # boolean+ }+ f.write(json.dumps({"index": {"_index": idx, "_id": doc["doc_id"]}}) + "\n")+ f.write(json.dumps(doc) + "\n")+print(f"corpus: {n} docs -> {out}", file=sys.stderr)+PY++RAW_BYTES=$(python3 - "$CORPUS" <<'PY'+import json,sys+# raw = the source documents only, excluding bulk action lines+tot=0+with open(sys.argv[1]) as f:+ for i,l in enumerate(f):+ if i % 2 == 1: tot += len(l.encode())+print(tot)+PY+)+echo "==> corpus raw source bytes: $RAW_BYTES"++# ── mapping: explicit, so the measurement is not at the mercy of inference ────+curl -fsS -X PUT "$URL/$IDX" -H 'Content-Type: application/json' -d '{+ "mappings": {"properties": {+ "doc_id":{"type":"keyword"}, "model":{"type":"keyword"},+ "status":{"type":"keyword"}, "region":{"type":"keyword"},+ "session":{"type":"keyword"}, "tenant":{"type":"keyword"},+ "body":{"type":"text"},+ "latency_ms":{"type":"long"}, "context_tokens":{"type":"long"},+ "cost_usd":{"type":"double"}, "retries":{"type":"long"},+ "score":{"type":"double"}, "ts":{"type":"date"}, "cached":{"type":"boolean"}+ }}}' > /dev/null+echo "==> mapping created"++# ── ingest (timed) ───────────────────────────────────────────────────────────+INGEST_START=$(date +%s.%N)+split -l 20000 "$CORPUS" "$DATA/chunk-"+for c in "$DATA"/chunk-*; do+ curl -fsS -X POST "$URL/_bulk" -H 'Content-Type: application/x-ndjson' \+ --data-binary "@$c" > /dev/null+done+curl -fsS -X POST "$URL/$IDX/_refresh" > /dev/null+INGEST_END=$(date +%s.%N)+INGEST_S=$(python3 -c "print(round($INGEST_END-$INGEST_START,2))")+echo "==> ingest: ${INGEST_S}s"++# ── force-merge to steady state ──────────────────────────────────────────────+MERGE_START=$(date +%s.%N)+curl -fsS -X POST "$URL/$IDX/_forcemerge?max_num_segments=1" > /dev/null || true+curl -fsS -X POST "$URL/$IDX/_flush" > /dev/null || true+sleep 10+MERGE_END=$(date +%s.%N)+MERGE_S=$(python3 -c "print(round($MERGE_END-$MERGE_START,2))")+echo "==> force-merge: ${MERGE_S}s"++COUNT=$(curl -fsS "$URL/$IDX/_count" | sed 's/.*"count":\([0-9]*\).*/\1/')+echo "==> indexed docs: $COUNT"++# ── disk by artifact type ────────────────────────────────────────────────────+python3 - "$DATA" "$IDX" "$RAW_BYTES" "$COUNT" "$INGEST_S" "$MERGE_S" "$LABEL" "$OUT" <<'PY'+import json, os, sys, collections+data, idx, raw, count, ing, mrg, label, out = sys.argv[1:9]+by_ext = collections.Counter()+total = 0+for dirpath, _, files in os.walk(data):+ if "corpus.ndjson" in files or dirpath.endswith(data): pass+ for fn in files:+ if fn in ("corpus.ndjson","server.log") or fn.startswith("chunk-"): continue+ p = os.path.join(dirpath, fn)+ try: sz = os.path.getsize(p)+ except OSError: continue+ ext = fn.rsplit(".", 1)[-1] if "." in fn else "(noext)"+ by_ext[ext] += sz; total += sz+res = {+ "label": label, "docs": int(count), "raw_source_bytes": int(raw),+ "index_total_bytes": total,+ "index_over_raw": round(total / int(raw), 4) if int(raw) else None,+ "ingest_seconds": float(ing), "forcemerge_seconds": float(mrg),+ "by_artifact": {k: v for k, v in by_ext.most_common()},+}+json.dump(res, open(out, "w"), indent=2)+print(f"\n{'artifact':<12}{'bytes':>14}{'share':>9}")+for k, v in by_ext.most_common(12):+ print(f"{k:<12}{v:>14,}{v/total*100:>8.1f}%")+print(f"{'TOTAL':<12}{total:>14,}")+print(f"\nraw source : {int(raw):,} B")+print(f"index/raw : {res['index_over_raw']}x")+PY++# ── query latency suite ──────────────────────────────────────────────────────+echo ""+echo "==> query latency (p50/p99 ms over 40 runs each)"+python3 - "$URL" "$IDX" "$OUT" <<'PY'+import json, sys, time, urllib.request+url, idx, out = sys.argv[1], sys.argv[2], sys.argv[3]+# Cheap point-lookup shapes were all sub-0.5ms at 100k docs — at that level HTTP+# and JSON overhead dominate and no engine change is measurable. So the suite+# deliberately includes EXPENSIVE shapes (large result sets, deep pages, high+# cardinality, phrase over diverse text) where engine work actually shows up.+QUERIES = {+ # --- cheap shapes: regression guards, not improvement targets ---+ "term_keyword_low_card": {"query":{"term":{"model":"opus-5"}}},+ "term_keyword_high_card":{"query":{"term":{"doc_id":"doc-00050000"}}},+ "range_date_narrow": {"query":{"range":{"ts":{"gte":1700000000000,"lt":1700000050000}}}},+ # --- expensive shapes: where a 2x would have to come from ---+ "range_long_wide": {"query":{"range":{"latency_ms":{"gte":0,"lt":4900}}}},+ "match_text_common": {"query":{"match":{"body":"index0000"}}},+ "match_text_multi": {"query":{"match":{"body":"index0000 segment0000 token0000"}}},+ "match_phrase": {"query":{"match_phrase":{"body":"index0000 segment0000"}}},+ "bool_must_filter": {"query":{"bool":{"must":[{"match":{"body":"index0000"}}],+ "filter":[{"term":{"status":"ok"}},+ {"range":{"latency_ms":{"lt":2500}}}]}}},+ "deep_page": {"from":9000,"size":100,+ "query":{"range":{"latency_ms":{"gte":0,"lt":4900}}}},+ "large_page": {"size":500,"query":{"match":{"body":"index0000"}}},+ "sort_by_numeric": {"size":100,"sort":[{"latency_ms":"desc"}],+ "query":{"match_all":{}}},+ # --- aggregations ---+ "agg_terms_low_card": {"size":0,"aggs":{"m":{"terms":{"field":"model"}}}},+ "agg_terms_high_card": {"size":0,"aggs":{"s":{"terms":{"field":"session","size":100}}}},+ "agg_stats_numeric": {"size":0,"aggs":{"s":{"stats":{"field":"latency_ms"}}}},+ "agg_date_histogram": {"size":0,"aggs":{"h":{"date_histogram":{"field":"ts",+ "fixed_interval":"1h"}}}},+ # numeric histogram is still the brute-force O(N) path (task #35) — a named target+ "agg_histogram_numeric": {"size":0,"aggs":{"h":{"histogram":{"field":"latency_ms",+ "interval":250}}}},+ "agg_nested_terms_stats":{"size":0,"aggs":{"m":{"terms":{"field":"model"},+ "aggs":{"s":{"stats":{"field":"cost_usd"}}}}}},+}+def run(body):+ d = json.dumps(body).encode()+ r = urllib.request.Request(f"{url}/{idx}/_search", data=d,+ headers={"Content-Type":"application/json"})+ t0 = time.perf_counter()+ with urllib.request.urlopen(r) as resp: resp.read()+ return (time.perf_counter()-t0)*1000+# Report the engine's own `took` alongside wall-clock: when wall-clock is+# sub-millisecond, transport dominates and `took` is the honest engine signal.+def took(body):+ d = json.dumps(body).encode()+ r = urllib.request.Request(f"{url}/{idx}/_search", data=d,+ headers={"Content-Type":"application/json"})+ with urllib.request.urlopen(r) as resp:+ return json.loads(resp.read()).get("took", -1)+results = {}+print(f"{'query':<26}{'p50':>9}{'p99':>9}{'took':>8}{'hits':>12}")+for name, body in QUERIES.items():+ try:+ for _ in range(5): run(body) # warm+ ts = sorted(run(body) for _ in range(40))+ p50, p99 = ts[len(ts)//2], ts[int(len(ts)*0.99)-1]+ tk = took(body)+ d = json.dumps(body).encode()+ r = urllib.request.Request(f"{url}/{idx}/_search", data=d,+ headers={"Content-Type":"application/json"})+ with urllib.request.urlopen(r) as resp:+ tot = json.loads(resp.read()).get("hits",{}).get("total",{}).get("value",-1)+ results[name] = {"p50_ms": round(p50,3), "p99_ms": round(p99,3),+ "took_ms": tk, "hits": tot}+ print(f"{name:<26}{p50:>9.2f}{p99:>9.2f}{tk:>8}{tot:>12,}")+ except Exception as e:+ results[name] = {"error": str(e)[:200]}+ print(f"{name:<26}{'ERROR':>9} {str(e)[:60]}")+res = json.load(open(out)); res["queries"] = results+json.dump(res, open(out,"w"), indent=2)+print(f"\nwrote {out}")+PY++echo ""+echo "==> done. server still running on $PORT (data: $DATA)"+echo " stop it with: pkill -f 'xerj --data-dir $DATA'"diff --git a/demo/playbooks/rc12/results-baseline-main.json b/demo/playbooks/rc12/results-baseline-main.jsonnew file mode 100644index 000000000..fb17f5619--- /dev/null+++ b/demo/playbooks/rc12/results-baseline-main.json@@ -0,0 +1,128 @@+{+ "label": "baseline-main",+ "docs": 500000,+ "raw_source_bytes": 356593195,+ "index_total_bytes": 154413168,+ "index_over_raw": 0.433,+ "ingest_seconds": 33.0,+ "forcemerge_seconds": 116.52,+ "by_artifact": {+ "dv": 57357593,+ "seg": 51449817,+ "post": 40951589,+ "ids": 3057388,+ "meta": 1464327,+ "fst": 64485,+ "json": 35338,+ "norms": 28305,+ "wal": 4119,+ "toml": 104,+ "xerj_master_key": 64,+ "sidx": 32,+ "lock": 7+ },+ "queries": {+ "term_keyword_low_card": {+ "p50_ms": 0.297,+ "p99_ms": 0.488,+ "took_ms": 0,+ "hits": 10000+ },+ "term_keyword_high_card": {+ "p50_ms": 0.208,+ "p99_ms": 0.559,+ "took_ms": 0,+ "hits": 1+ },+ "range_date_narrow": {+ "p50_ms": 0.327,+ "p99_ms": 0.584,+ "took_ms": 0,+ "hits": 50+ },+ "range_long_wide": {+ "p50_ms": 0.302,+ "p99_ms": 0.455,+ "took_ms": 0,+ "hits": 10000+ },+ "match_text_common": {+ "p50_ms": 0.287,+ "p99_ms": 0.361,+ "took_ms": 0,+ "hits": 10000+ },+ "match_text_multi": {+ "p50_ms": 0.313,+ "p99_ms": 0.406,+ "took_ms": 0,+ "hits": 10000+ },+ "match_phrase": {+ "p50_ms": 0.282,+ "p99_ms": 0.544,+ "took_ms": 0,+ "hits": 10000+ },+ "bool_must_filter": {+ "p50_ms": 0.368,+ "p99_ms": 0.743,+ "took_ms": 0,+ "hits": 10000+ },+ "deep_page": {+ "p50_ms": 1.043,+ "p99_ms": 1.161,+ "took_ms": 0,+ "hits": 10000+ },+ "large_page": {+ "p50_ms": 4.718,+ "p99_ms": 5.937,+ "took_ms": 0,+ "hits": 10000+ },+ "sort_by_numeric": {+ "p50_ms": 1.171,+ "p99_ms": 1.38,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_terms_low_card": {+ "p50_ms": 0.208,+ "p99_ms": 0.296,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_terms_high_card": {+ "p50_ms": 0.239,+ "p99_ms": 0.348,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_stats_numeric": {+ "p50_ms": 0.194,+ "p99_ms": 0.308,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_date_histogram": {+ "p50_ms": 0.329,+ "p99_ms": 0.456,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_histogram_numeric": {+ "p50_ms": 0.229,+ "p99_ms": 0.379,+ "took_ms": 0,+ "hits": 10000+ },+ "agg_nested_terms_stats": {+ "p50_ms": 0.213,+ "p99_ms": 0.61,+ "took_ms": 0,+ "hits": 10000+ }+ }+}\ No newline at end of file