Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: use-case recipes, each verified end-to-end against a live XERJ

xerj-org2d173ed2c115
diff --git a/.gitignore b/.gitignore
index 42372c8a7..083b99d6b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -96,3 +96,4 @@ landing/resources/xerj-sales-kit.zip
# Re-include the public architecture doc (Markdown is ignored by default above).
!docs/ARCHITECTURE.md
!ROADMAP.md
+!docs/**/*.md
diff --git a/docs/examples/agentic-memory/agent_memory.py b/docs/examples/agentic-memory/agent_memory.py
new file mode 100644
index 000000000..1d404b78b
--- /dev/null
+++ b/docs/examples/agentic-memory/agent_memory.py
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+"""Long-term memory for an AI agent, backed by XERJ's /_memory API.
+
+Simulates a coding-assistant agent that, across a session:
+ 1. STORES observations/preferences it learns about the project
+ (each as text + a small embedding vector + metadata),
+ 2. later RECALLS the most relevant memories to answer a new question
+ -- by vector similarity AND by plain text,
+ 3. does a metadata-filtered recall (only "preference" memories),
+ 4. FORGETS a memory that is no longer true,
+ 5. proves multi-namespace isolation: a second agent cannot see the
+ first agent's memories.
+
+No LLM and no pip installs -- stdlib only. The "embedder" here is a tiny
+deterministic hashed-character-trigram vectorizer so the demo is fully
+self-contained and cosine similarity is meaningful (it is lexical, not
+neural -- see the note by embed()). In production you bring your own vectors
+(OpenAI/Cohere/your model) -- XERJ stores and kNN-searches them the same way,
+which is exactly what lets XERJ replace a bolt-on vector DB for agent memory.
+"""
+
+import json
+import math
+import re
+import urllib.request
+from urllib.error import HTTPError
+
+BASE = "http://localhost:9481"
+DIM = 96 # embedding dimensionality for the toy hashing embedder
+
+
+# --------------------------------------------------------------------------- #
+# Tiny self-contained "embedder": deterministic hashed character trigrams.
+# Character n-grams give the demo enough lexical-semantic overlap that related
+# wording (service/services, prefer/prefers) lands nearby -- WITHOUT any model
+# download. It is NOT a neural embedder: real synonymy ("car" ~ "automobile")
+# needs real embeddings. In production you pass vectors from your own model
+# (OpenAI/Cohere/local); XERJ stores and kNN-searches them exactly the same,
+# which is what lets it stand in for a bolt-on vector DB.
+# --------------------------------------------------------------------------- #
+def embed(text: str) -> list[float]:
+ s = " " + re.sub(r"[^a-z0-9]+", " ", text.lower()) + " "
+ vec = [0.0] * DIM
+ for i in range(len(s) - 2):
+ h = 0
+ for ch in s[i:i + 3]: # hash each character trigram into a bucket
+ h = (h * 131 + ord(ch)) & 0xFFFFFFFF
+ vec[h % DIM] += 1.0
+ norm = math.sqrt(sum(v * v for v in vec)) or 1.0
+ return [round(v / norm, 6) for v in vec]
+
+
+# --------------------------------------------------------------------------- #
+# Thin HTTP helpers over XERJ's REST surface.
+# --------------------------------------------------------------------------- #
+def _req(method: str, path: str, body: dict | None = None) -> dict:
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(
+ BASE + path, data=data, method=method,
+ headers={"Content-Type": "application/json"},
+ )
+ try:
+ with urllib.request.urlopen(req) as r:
+ return json.loads(r.read() or b"{}")
+ except HTTPError as e:
+ return json.loads(e.read() or b"{}")
+
+
+def remember(ns: str, text: str, metadata: dict, mem_id: str) -> dict:
+ """Store one memory: text is BM25-indexed, vector enables kNN recall."""
+ return _req("POST", f"/_memory/{ns}", {
+ "text": text, "vector": embed(text), "metadata": metadata, "id": mem_id,
+ })
+
+
+def recall_vec(ns: str, query: str, k: int = 3, filt: dict | None = None) -> list:
+ """Semantic recall: embed the query, kNN over stored vectors."""
+ body = {"vector": embed(query), "k": k}
+ if filt:
+ body["filter"] = filt
+ return _req("POST", f"/_memory/{ns}/_recall", body)["hits"]
+
+
+def recall_text(ns: str, query: str, k: int = 3) -> list:
+ """Lexical recall: BM25 over the stored memory text."""
+ return _req("POST", f"/_memory/{ns}/_recall", {"query": query, "k": k})["hits"]
+
+
+def forget(ns: str, mem_id: str) -> dict:
+ return _req("DELETE", f"/_memory/{ns}/{mem_id}")
+
+
+def drop(ns: str) -> dict:
+ return _req("DELETE", f"/_memory/{ns}")
+
+
+def show(label: str, hits: list) -> None:
+ print(f"\n{label}")
+ for h in hits:
+ kind = (h.get("metadata") or {}).get("kind", "-")
+ print(f" [{h['score']:.3f}] ({kind:10}) {h['text']}")
+
+
+# --------------------------------------------------------------------------- #
+# The scenario.
+# --------------------------------------------------------------------------- #
+def main() -> None:
+ ADA = "agent-ada" # our coding assistant
+ BILLING = "agent-billing" # an unrelated agent, to prove isolation
+
+ # Clean slate for a repeatable demo.
+ drop(ADA)
+ drop(BILLING)
+
+ # --- Turn 1..N: the agent learns things and writes them to memory. ------ #
+ print("== Storing what the agent learned this session ==")
+ facts = [
+ ("The team prefers Rust over Go for new backend services.",
+ {"kind": "preference", "topic": "language"}),
+ ("Production runs on AWS in us-east-1; staging is us-west-2.",
+ {"kind": "fact", "topic": "infra"}),
+ ("The team prefers tabs over spaces and a 100-column line limit.",
+ {"kind": "preference", "topic": "style"}),
+ ("The payments service owns the Postgres 'ledger' database.",
+ {"kind": "fact", "topic": "ownership"}),
+ ("Deploys are gated on the full integration suite passing in CI.",
+ {"kind": "fact", "topic": "process"}),
+ ("The team prefers OpenTelemetry for tracing, not vendor SDKs.",
+ {"kind": "preference", "topic": "observability"}),
+ ]
+ for i, (text, md) in enumerate(facts, 1):
+ r = remember(ADA, text, md, f"ada-{i}")
+ print(f" stored {r['id']:8} created={r['created']}")
+
+ # A different agent stores its own (finance) memory in its own namespace.
+ remember(BILLING, "Invoices over $10k require CFO approval.",
+ {"kind": "policy"}, "bill-1")
+
+ # --- A later turn: answer a question by recalling relevant memory. ------ #
+ q1 = "What programming language should I use for a new backend service?"
+ show(f'== Semantic recall for: "{q1}"', recall_vec(ADA, q1, k=3))
+
+ q2 = "Which region is production deployed in?"
+ show(f'== Lexical (BM25) recall for: "{q2}"', recall_text(ADA, q2, k=3))
+
+ # --- Metadata-filtered recall: only surface team *preferences*. --------- #
+ q3 = "How should I set up the new service?"
+ show(f'== Preference-only recall for: "{q3}"',
+ recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}}))
+
+ # --- Isolation: agent-billing cannot see agent-ada's memories. ---------- #
+ print("\n== Namespace isolation ==")
+ leak = recall_vec(BILLING, q1, k=3)
+ billing_texts = [h["text"] for h in leak]
+ print(f" agent-billing recall for a coding question -> {billing_texts}")
+ assert all("Rust" not in t and "AWS" not in t for t in billing_texts), \
+ "ISOLATION VIOLATION: agent-billing saw agent-ada's memories"
+ print(" OK: agent-billing only ever sees its own namespace.")
+
+ # --- Forgetting: the team switched tracing tools; retract that memory. -- #
+ print("\n== Forgetting a memory that is no longer true ==")
+ before = recall_vec(ADA, "what do we use for tracing", k=3)
+ show(" before forget:", before)
+ print(" forget ->", forget(ADA, "ada-6"))
+ after = recall_vec(ADA, "what do we use for tracing", k=3)
+ show(" after forget:", after)
+ assert all(h["id"] != "ada-6" for h in after), "ada-6 was not forgotten"
+ print(" OK: the retracted memory is gone.")
+
+ # --- Correctness checks so a green run really means it worked. ---------- #
+ top_lang = recall_vec(ADA, q1, k=1)[0]
+ assert "Rust" in top_lang["text"], f"expected Rust preference on top, got {top_lang}"
+ top_region = recall_text(ADA, q2, k=1)[0]
+ assert "us-east-1" in top_region["text"], f"expected region fact on top, got {top_region}"
+ prefs = recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}})
+ assert prefs and all((h["metadata"] or {}).get("kind") == "preference" for h in prefs), \
+ "preference filter leaked non-preferences"
+ print("\nAll assertions passed.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/examples/anomaly-detection/anomaly_detection.py b/docs/examples/anomaly-detection/anomaly_detection.py
new file mode 100644
index 000000000..3783dd0fa
--- /dev/null
+++ b/docs/examples/anomaly-detection/anomaly_detection.py
@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""
+Anomaly detection on a metric series with XERJ.
+
+Scenario: we collect per-host CPU readings into `cpu_metrics`. For fifteen
+minutes CPU sits around ~20%. At minute 12 a runaway process pins the box and
+the per-minute mean jumps to ~96%, then recovers. We want XERJ to flag that one
+minute as anomalous - automatically, without us hand-picking a threshold on the
+raw value.
+
+XERJ's on-demand anomaly detector does exactly this: it buckets the source
+index by time, computes the metric per bucket (here mean(cpu)), builds a moving
+baseline from prior *normal* buckets, and scores each bucket by how many
+standard deviations it sits from that baseline.
+
+Run: python3 anomaly_detection.py # assumes XERJ on :9486
+ XERJ=http://localhost:9200 python3 anomaly_detection.py
+
+Uses only the Python 3 standard library. No pip installs.
+"""
+import json
+import os
+import urllib.request
+
+XERJ = os.environ.get("XERJ", "http://localhost:9486").rstrip("/")
+INDEX = "cpu_metrics"
+JOB = "cpu-spike"
+
+
+def req(method, path, body=None, ndjson=False):
+ data = None
+ headers = {}
+ if ndjson:
+ data = body.encode()
+ headers["Content-Type"] = "application/x-ndjson"
+ elif body is not None:
+ data = json.dumps(body).encode()
+ headers["Content-Type"] = "application/json"
+ r = urllib.request.Request(XERJ + path, data=data, method=method, headers=headers)
+ with urllib.request.urlopen(r) as resp:
+ return json.loads(resp.read().decode())
+
+
+# 1. Map the index: @timestamp as a date, cpu as a numeric we can average.
+try:
+ req("DELETE", "/" + INDEX)
+except Exception:
+ pass
+req("PUT", "/" + INDEX, {
+ "mappings": {
+ "properties": {
+ "@timestamp": {"type": "date"},
+ "host": {"type": "keyword"},
+ "cpu": {"type": "double"},
+ }
+ }
+})
+
+# 2. Build 16 one-minute buckets of 4 host readings each.
+# Minutes 0-11 and 13-15 are normal (~20%); minute 12 is the injected spike.
+# Readings per minute are chosen so the per-minute mean is exact & stable.
+NORMAL = [19, 20, 20, 21] # mean 20.0
+JITTER = { # a little bucket-to-bucket variance
+ 0: 0, 1: 1, 2: -1, 3: 0, 4: 2, 5: -2, 6: 1, 7: 0,
+ 8: -1, 9: 1, 10: 0, 11: 2, 13: 0, 14: 1, 15: -1,
+}
+SPIKE = [94, 96, 96, 98] # mean 96.0 <-- the anomaly
+
+lines = []
+for minute in range(16):
+ ts = "2026-07-06T00:%02d:00Z" % minute
+ if minute == 12:
+ readings = SPIKE
+ else:
+ readings = [v + JITTER[minute] for v in NORMAL]
+ for h, cpu in enumerate(readings):
+ lines.append(json.dumps({"index": {"_index": INDEX}}))
+ lines.append(json.dumps({"@timestamp": ts, "host": "web-%d" % h, "cpu": cpu}))
+bulk = req("POST", "/_bulk", "\n".join(lines) + "\n", ndjson=True)
+assert not bulk.get("errors"), bulk
+req("POST", "/" + INDEX + "/_refresh")
+
+# 3. Create the detector: mean(cpu) over 1-minute buckets.
+job = req("PUT", "/_ml/anomaly_detectors/" + JOB, {
+ "source_index": INDEX,
+ "time_field": "@timestamp",
+ "function": "mean",
+ "field": "cpu",
+ "bucket_span": "1m",
+ "description": "CPU mean per minute",
+})
+print("== detector created ==")
+print(json.dumps({
+ "job_id": job["job_id"],
+ "function": job["analysis_config"]["detectors"][0]["function"],
+ "bucket_span": job["analysis_config"]["bucket_span"],
+ "anomaly_threshold": job["anomaly_threshold"],
+}, indent=2))
+
+# 4. Score it. Baseline warm-up is the first 4 buckets (no score yet); every
+# later bucket is scored against the mean/std of prior *normal* buckets.
+res = req("POST", "/_ml/anomaly_detectors/" + JOB + "/_score")
+
+print("\n== per-bucket scores ==")
+print("%-20s %8s %9s %8s %7s %5s" %
+ ("minute (UTC)", "actual", "expected", "z_score", "score", "flag"))
+for r in res["records"]:
+ exp = r["expected"]
+ z = r["z_score"]
+ print("%-20s %8.1f %9s %8s %7.1f %5s" % (
+ r["timestamp_iso"][11:19],
+ r["actual"],
+ "-" if exp is None else "%.1f" % exp,
+ "-" if z is None else "%.1f" % z,
+ r["anomaly_score"],
+ "***" if r["is_anomaly"] else "",
+ ))
+
+print("\n== anomalies (%d of %d buckets) ==" % (res["anomaly_count"], res["bucket_count"]))
+for a in res["anomalies"]:
+ print(" %s actual=%.1f expected=%.1f z=%.1f score=%.1f" % (
+ a["timestamp_iso"], a["actual"], a["expected"], a["z_score"], a["anomaly_score"]))
+
+# 5. Verify the result is correct.
+spike = next(r for r in res["records"] if r["timestamp_iso"].startswith("2026-07-06T00:12"))
+normals = [r for r in res["records"]
+ if r["expected"] is not None and not r["timestamp_iso"].startswith("2026-07-06T00:12")]
+
+assert res["anomaly_count"] == 1, "expected exactly one anomaly, got %d" % res["anomaly_count"]
+assert spike["is_anomaly"], "spike bucket was not flagged"
+assert spike["anomaly_score"] >= 99.0, "spike score too low: %s" % spike["anomaly_score"]
+assert abs(spike["actual"] - 96.0) < 1e-6, "spike mean wrong: %s" % spike["actual"]
+# is_anomaly is the gate: only the spike crosses the threshold; jitter buckets
+# never do, even if their normalized score creeps up near the threshold line.
+assert all(not r["is_anomaly"] for r in normals), "a normal bucket was flagged"
+max_normal = max(r["anomaly_score"] for r in normals)
+assert spike["anomaly_score"] >= 2 * max_normal, \
+ "spike (%.1f) not clearly separated from top normal (%.1f)" % (
+ spike["anomaly_score"], max_normal)
+print("\nOK: spike at 00:12 flagged (score %.1f); %d normal buckets, none flagged"
+ " (top normal score %.1f)." % (spike["anomaly_score"], len(normals), max_normal))
diff --git a/docs/examples/hybrid-search/hybrid_search.py b/docs/examples/hybrid-search/hybrid_search.py
new file mode 100644
index 000000000..edac4d6e8
--- /dev/null
+++ b/docs/examples/hybrid-search/hybrid_search.py
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+"""
+Hybrid search on XERJ — keyword (BM25) + vector (kNN) fused in ONE query.
+
+Scenario: a help-desk search box. A user types "reset password".
+We show that:
+ * pure BM25 misses a highly relevant article that shares no keywords
+ with the query ("Regain entry to a locked-out account"),
+ * pure kNN (top-k) misses a keyword-exact article whose topic vector
+ is far from the query ("Password complexity and rotation policy"),
+ * a single `hybrid` query surfaces BOTH — no second system, no
+ client-side result stitching.
+
+Vectors here are hand-authored 4-dim "topic" vectors so the demo is
+deterministic. In production you'd fill dense_vector fields with a real
+embedding model (or use XERJ's semantic_text auto-embedding).
+Topic axes: [auth_recovery, security_policy, cooking, generic]
+"""
+import json
+import sys
+import urllib.request
+
+BASE = "http://localhost:9485"
+INDEX = "helpdesk"
+
+
+def req(method, path, body=None):
+ data = json.dumps(body).encode() if body is not None else None
+ r = urllib.request.Request(
+ BASE + path, data=data, method=method,
+ headers={"Content-Type": "application/json"},
+ )
+ with urllib.request.urlopen(r) as resp:
+ return json.loads(resp.read().decode())
+
+
+def ids(res):
+ return [h["_id"] for h in res["hits"]["hits"]]
+
+
+def table(res):
+ for h in res["hits"]["hits"]:
+ print(f" {h['_id']:4} score={h['_score']:.4f} {h['_source']['title']}")
+
+
+# ---- 1. fresh index: a text field + a dense_vector field -----------------
+try:
+ req("DELETE", f"/{INDEX}")
+except urllib.error.HTTPError:
+ pass
+
+req("PUT", f"/{INDEX}", {
+ "mappings": {
+ "properties": {
+ "title": {"type": "text"},
+ "vec": {"type": "dense_vector", "dims": 4, "similarity": "cosine"},
+ }
+ }
+})
+
+# ---- 2. corpus: title (for BM25) + topic vector (for kNN) ----------------
+DOCS = [
+ ("d1", "Reset your password", [1.00, 0.15, 0.0, 0.10]),
+ ("d2", "Regain entry to a locked-out account", [0.95, 0.10, 0.0, 0.15]),
+ ("d3", "Password complexity and rotation policy", [0.15, 1.00, 0.0, 0.10]),
+ ("d4", "How to bake sourdough bread", [0.00, 0.00, 1.0, 0.10]),
+ ("d5", "Change your account password", [0.85, 0.25, 0.0, 0.20]),
+]
+
+bulk = ""
+for did, title, vec in DOCS:
+ bulk += json.dumps({"index": {"_index": INDEX, "_id": did}}) + "\n"
+ bulk += json.dumps({"title": title, "vec": vec}) + "\n"
+
+r = urllib.request.Request(
+ BASE + "/_bulk", data=bulk.encode(), method="POST",
+ headers={"Content-Type": "application/x-ndjson"},
+)
+with urllib.request.urlopen(r) as resp:
+ assert not json.loads(resp.read())["errors"], "bulk had errors"
+req("POST", f"/{INDEX}/_refresh")
+
+# The user's intent, in both modalities:
+QUERY_TEXT = "reset password"
+QUERY_VEC = [1.0, 0.20, 0.0, 0.10] # "auth recovery" intent
+
+# ---- 3a. Pure keyword (BM25) ---------------------------------------------
+kw = req("POST", f"/{INDEX}/_search", {
+ "size": 3,
+ "query": {"match": {"title": QUERY_TEXT}},
+})
+print("Pure BM25 (match 'reset password'):")
+table(kw)
+kw_ids = ids(kw)
+
+# ---- 3b. Pure vector (kNN, top-3) ----------------------------------------
+# knn scores every doc that has the field by cosine; `size` is the cutoff the
+# user actually sees, so size:3 == "show me the 3 nearest".
+vec = req("POST", f"/{INDEX}/_search", {
+ "size": 3,
+ "query": {"knn": {"field": "vec", "query_vector": QUERY_VEC, "k": 3}},
+})
+print("\nPure kNN (top-3 nearest topic vectors):")
+table(vec)
+vec_ids = ids(vec)
+
+# ---- 3c. Hybrid — both, fused with Reciprocal Rank Fusion ----------------
+hyb = req("POST", f"/{INDEX}/_search", {
+ "size": 5,
+ "query": {
+ "hybrid": {
+ "queries": [
+ {"query": {"match": {"title": QUERY_TEXT}}, "weight": 1.0},
+ {"query": {"knn": {"field": "vec", "query_vector": QUERY_VEC, "k": 3}}, "weight": 1.0},
+ ],
+ "fusion": "rrf"
+ }
+ },
+})
+print("\nHybrid (BM25 + kNN, RRF fusion):")
+table(hyb)
+hyb_ids = ids(hyb)
+
+# ---- 4. Prove the point --------------------------------------------------
+print("\n--- assertions ---")
+# d2 is relevant (account recovery) but shares NO keywords -> BM25 can't see it
+assert "d2" not in kw_ids, f"expected BM25 to miss d2, got {kw_ids}"
+print("PASS BM25 alone MISSES d2 (locked-out account: no shared keywords)")
+
+# d3 has the keyword 'password' but its topic vector is far -> kNN top-3 drops it
+assert "d3" not in vec_ids, f"expected kNN top-3 to miss d3, got {vec_ids}"
+print("PASS kNN alone MISSES d3 (password policy: topic vector too far)")
+
+# hybrid recovers BOTH in one request
+assert "d2" in hyb_ids and "d3" in hyb_ids, f"hybrid should surface d2 AND d3, got {hyb_ids}"
+print("PASS Hybrid surfaces BOTH d2 and d3 in a single query")
+
+# and the exact double-match still ranks first
+assert hyb_ids[0] == "d1", f"expected d1 first in hybrid, got {hyb_ids}"
+print("PASS d1 (keyword + vector match) still ranks #1 under fusion")
+
+print("\nBM25 ids :", kw_ids)
+print("kNN ids :", vec_ids)
+print("Hybrid ids:", hyb_ids)
+print("\nOK")
diff --git a/docs/examples/log-analytics/log_analytics.py b/docs/examples/log-analytics/log_analytics.py
new file mode 100644
index 000000000..4c88baacd
--- /dev/null
+++ b/docs/examples/log-analytics/log_analytics.py
@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+"""
+Log analytics with XERJ — from raw logs to dashboards.
+
+Bulk-ingests a few hundred synthetic structured log events, then answers
+real operational questions with Elasticsearch-style aggregations:
+ 1. Error rate over time -> date_histogram + filters sub-agg
+ 2. p95 / p99 latency -> percentiles
+ 3. Top services by volume -> terms + stats sub-agg
+
+Stdlib only (urllib + json). No pip installs. Point it at a live XERJ.
+"""
+import json
+import random
+import sys
+import urllib.request
+from datetime import datetime, timedelta, timezone
+
+BASE = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:9483"
+INDEX = "logs-app"
+
+
+def req(method, path, body=None, ndjson=False):
+ data = None
+ headers = {}
+ if ndjson:
+ data = body.encode()
+ headers["Content-Type"] = "application/x-ndjson"
+ elif body is not None:
+ data = json.dumps(body).encode()
+ headers["Content-Type"] = "application/json"
+ r = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
+ with urllib.request.urlopen(r) as resp:
+ return json.loads(resp.read().decode())
+
+
+# ---------------------------------------------------------------- setup
+# Clean slate, then create an explicit mapping so timestamp is a date,
+# latency_ms is numeric, and service/level/status are keywords for aggs.
+try:
+ req("DELETE", "/" + INDEX)
+except urllib.error.HTTPError:
+ pass
+
+req("PUT", "/" + INDEX, {
+ "mappings": {
+ "properties": {
+ "@timestamp": {"type": "date"},
+ "level": {"type": "keyword"},
+ "service": {"type": "keyword"},
+ "latency_ms": {"type": "integer"},
+ "status": {"type": "integer"},
+ "message": {"type": "text"},
+ }
+ }
+})
+
+# ---------------------------------------------------------------- generate
+# 600 events across 6 hours and 4 services. Each service has its own
+# baseline latency and error propensity so the aggregations tell a story.
+random.seed(7)
+services = {
+ "checkout": {"lat": 120, "err": 0.08},
+ "search": {"lat": 45, "err": 0.02},
+ "catalog": {"lat": 30, "err": 0.01},
+ "payments": {"lat": 200, "err": 0.05},
+}
+start = datetime(2026, 7, 6, 8, 0, 0, tzinfo=timezone.utc)
+N = 600
+lines = []
+for i in range(N):
+ svc = random.choices(list(services), weights=[3, 4, 4, 2])[0]
+ cfg = services[svc]
+ ts = start + timedelta(seconds=random.randint(0, 6 * 3600))
+ is_err = random.random() < cfg["err"]
+ if is_err:
+ level, status = "ERROR", random.choice([500, 502, 503])
+ latency = int(cfg["lat"] * random.uniform(2.0, 5.0)) # errors are slow
+ else:
+ level = random.choices(["INFO", "WARN"], weights=[9, 1])[0]
+ status = 200
+ latency = int(cfg["lat"] * random.uniform(0.5, 1.8))
+ doc = {
+ "@timestamp": ts.isoformat(),
+ "level": level, "service": svc,
+ "latency_ms": latency, "status": status,
+ "message": f"{svc} handled request status={status}",
+ }
+ lines.append(json.dumps({"index": {"_index": INDEX}}))
+ lines.append(json.dumps(doc))
+
+resp = req("POST", "/_bulk", "\n".join(lines) + "\n", ndjson=True)
+assert not resp.get("errors"), "bulk had errors: " + json.dumps(resp)[:400]
+req("POST", "/" + INDEX + "/_refresh")
+count = req("GET", "/" + INDEX + "/_count")["count"]
+print(f"Ingested {count} log events into '{INDEX}'\n")
+assert count == N, f"expected {N}, got {count}"
+
+# ---------------------------------------------------------------- Q1
+# Error rate over time: bucket by hour, and inside each bucket count how
+# many were ERROR with a `filter` sub-agg. errors/doc_count = error rate.
+print("=" * 60)
+print("Q1. Error rate over time (hourly)")
+print("=" * 60)
+r = req("POST", "/" + INDEX + "/_search", {
+ "size": 0,
+ "aggs": {
+ "per_hour": {
+ "date_histogram": {"field": "@timestamp", "fixed_interval": "1h"},
+ "aggs": {
+ "errors": {"filter": {"term": {"level": "ERROR"}}}
+ }
+ }
+ }
+})
+print(f"{'hour (UTC)':<22}{'total':>8}{'errors':>8}{'err_rate':>10}")
+for b in r["aggregations"]["per_hour"]["buckets"]:
+ total = b["doc_count"]
+ errs = b["errors"]["doc_count"]
+ rate = (errs / total * 100) if total else 0.0
+ hour = b["key_as_string"][11:16]
+ print(f"{hour:<22}{total:>8}{errs:>8}{rate:>9.1f}%")
+
+# ---------------------------------------------------------------- Q2
+# Latency percentiles across the whole fleet, then p95 broken out per
+# service so you can see which service is dragging the tail.
+print()
+print("=" * 60)
+print("Q2. Latency percentiles (ms)")
+print("=" * 60)
+r = req("POST", "/" + INDEX + "/_search", {
+ "size": 0,
+ "aggs": {
+ "latency": {"percentiles": {"field": "latency_ms", "percents": [50, 95, 99]}},
+ "by_service": {
+ "terms": {"field": "service"},
+ "aggs": {"p95": {"percentiles": {"field": "latency_ms", "percents": [95]}}}
+ }
+ }
+})
+pct = r["aggregations"]["latency"]["values"]
+print(f"fleet-wide p50={pct['50.0']:.0f} p95={pct['95.0']:.0f} p99={pct['99.0']:.0f}")
+print(f"{'service':<12}{'p95 latency':>14}")
+for b in r["aggregations"]["by_service"]["buckets"]:
+ p95 = b["p95"]["values"]["95.0"]
+ print(f"{b['key']:<12}{p95:>13.0f}")
+
+# ---------------------------------------------------------------- Q3
+# Top services by volume, with a stats sub-agg (count/avg/min/max) so one
+# call gives you the leaderboard AND the latency profile of each service.
+print()
+print("=" * 60)
+print("Q3. Top services by volume (+ latency stats)")
+print("=" * 60)
+r = req("POST", "/" + INDEX + "/_search", {
+ "size": 0,
+ "aggs": {
+ "top_services": {
+ "terms": {"field": "service", "size": 10, "order": {"_count": "desc"}},
+ "aggs": {"lat": {"stats": {"field": "latency_ms"}}}
+ }
+ }
+})
+print(f"{'service':<12}{'volume':>8}{'avg_ms':>9}{'min':>6}{'max':>7}")
+for b in r["aggregations"]["top_services"]["buckets"]:
+ s = b["lat"]
+ print(f"{b['key']:<12}{b['doc_count']:>8}{s['avg']:>9.0f}{int(s['min']):>6}{int(s['max']):>7}")
+
+print("\nAll aggregations returned. Log analytics on one engine, ES wire API.")
diff --git a/docs/examples/migrate-from-elasticsearch/migrate_demo.sh b/docs/examples/migrate-from-elasticsearch/migrate_demo.sh
new file mode 100755
index 000000000..b4faca7ab
--- /dev/null
+++ b/docs/examples/migrate-from-elasticsearch/migrate_demo.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# migrate_demo.sh — prove standard Elasticsearch REST calls run UNCHANGED against XERJ.
+#
+# The only thing that changed vs. a real Elasticsearch cluster is $ES:
+# point it at XERJ's wire port and every request below is byte-for-byte
+# the same JSON you'd send Elasticsearch.
+#
+# ES=http://localhost:9487 ./migrate_demo.sh
+#
+# Requires: curl, and (for pretty output) python3 — both stdlib-only.
+set -euo pipefail
+
+ES="${ES:-http://localhost:9200}" # <-- the ONE thing you change to migrate
+IDX=products
+
+pp() { python3 -m json.tool; } # pretty-print JSON from stdin
+say() { printf '\n=== %s ===\n' "$1"; }
+
+# 0) Clean slate (ignore 404 on first run)
+curl -s -X DELETE "$ES/$IDX" >/dev/null || true
+
+# 1) Create an index with an explicit mapping — standard ES PUT.
+say "1. PUT /$IDX (create index + mapping)"
+curl -s -X PUT "$ES/$IDX" \
+ -H 'Content-Type: application/json' -d '{
+ "mappings": {
+ "properties": {
+ "name": { "type": "text" },
+ "brand": { "type": "keyword" },
+ "price": { "type": "float" },
+ "in_stock": { "type": "boolean" }
+ }
+ }
+}' | pp
+
+# 2) Bulk-load documents — standard ES _bulk NDJSON (action line + source line).
+say "2. POST /_bulk (index 6 docs)"
+curl -s -X POST "$ES/_bulk" \
+ -H 'Content-Type: application/x-ndjson' --data-binary '
+{"index":{"_index":"products","_id":"1"}}
+{"name":"Aluminum water bottle","brand":"Klean","price":24.99,"in_stock":true}
+{"index":{"_index":"products","_id":"2"}}
+{"name":"Insulated steel water bottle","brand":"Hydro","price":39.95,"in_stock":true}
+{"index":{"_index":"products","_id":"3"}}
+{"name":"Glass water carafe","brand":"Bodum","price":19.50,"in_stock":false}
+{"index":{"_index":"products","_id":"4"}}
+{"name":"Steel travel mug","brand":"Hydro","price":29.00,"in_stock":true}
+{"index":{"_index":"products","_id":"5"}}
+{"name":"Ceramic coffee mug","brand":"Bodum","price":12.00,"in_stock":true}
+{"index":{"_index":"products","_id":"6"}}
+{"name":"Plastic sports bottle","brand":"Klean","price":9.99,"in_stock":false}
+' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("errors:",d["errors"],"items:",len(d["items"]))'
+
+# Make freshly-indexed docs searchable (ES semantics: refresh).
+curl -s -X POST "$ES/$IDX/_refresh" >/dev/null
+
+# 3) match query — full-text search on the analyzed "name" field.
+say "3. _search match: name ~ \"water bottle\""
+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{
+ "query": { "match": { "name": "water bottle" } },
+ "_source": ["name","brand","price"]
+}' | python3 -c '
+import sys,json
+d=json.load(sys.stdin)
+print("total:", d["hits"]["total"]["value"])
+for h in d["hits"]["hits"]:
+ print(f" {h[_SCORE]:.3f} {h[_SRC][NAME]}".replace("_SCORE",chr(39)+"_score"+chr(39)).replace("_SRC",chr(39)+"_source"+chr(39)).replace("NAME",chr(39)+"name"+chr(39)))
+' 2>/dev/null || curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{"query":{"match":{"name":"water bottle"}},"_source":["name"]}' | pp
+
+# 4) bool query: must match + term filter + range filter — the classic ES combo.
+say "4. _search bool: match \"bottle\" AND brand=Hydro-or-Klean AND price 10..40"
+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{
+ "query": {
+ "bool": {
+ "must": [ { "match": { "name": "bottle" } } ],
+ "filter": [
+ { "terms": { "brand": ["Hydro","Klean"] } },
+ { "range": { "price": { "gte": 10, "lte": 40 } } }
+ ]
+ }
+ },
+ "_source": ["name","brand","price"]
+}' | pp
+
+# 5) terms aggregation — count products per brand (no query hits, just the agg).
+say "5. _search terms agg: products per brand"
+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{
+ "size": 0,
+ "aggs": {
+ "by_brand": { "terms": { "field": "brand" } }
+ }
+}' | pp
+
+# 6) _cat/indices — operational sanity check, human-readable table.
+say "6. GET /_cat/indices?v"
+curl -s "$ES/_cat/indices?v"
+
+echo
+echo "All standard Elasticsearch calls succeeded against XERJ at $ES"
diff --git a/docs/examples/semantic-search-rag/rag_demo.py b/docs/examples/semantic-search-rag/rag_demo.py
new file mode 100644
index 000000000..de58f7322
--- /dev/null
+++ b/docs/examples/semantic-search-rag/rag_demo.py
@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""
+Semantic search / RAG retrieval with XERJ — no separate vector DB.
+
+Indexes a tiny knowledge base into a `semantic_text` field (auto-embedded on
+ingest, zero external config), then retrieves passages by *meaning* using the
+`semantic` query. The retrieved passages are the chunks you'd hand to an LLM as
+grounding context for RAG.
+
+Stdlib only (urllib + json). Point BASE at a running XERJ es_compat port.
+
+ python3 rag_demo.py # defaults to http://localhost:9482
+ BASE=http://localhost:9200 python3 rag_demo.py
+"""
+import json
+import os
+import sys
+import time
+import urllib.request
+import urllib.error
+
+BASE = os.environ.get("BASE", "http://localhost:9482")
+INDEX = "kb"
+
+
+def req(method, path, body=None):
+ data = json.dumps(body).encode() if body is not None else None
+ r = urllib.request.Request(
+ BASE + path, data=data, method=method,
+ headers={"Content-Type": "application/json"},
+ )
+ try:
+ with urllib.request.urlopen(r) as resp:
+ return resp.status, json.loads(resp.read().decode())
+ except urllib.error.HTTPError as e:
+ return e.code, json.loads(e.read().decode())
+
+
+# The knowledge base: short docs an LLM might need to answer support questions.
+# Note the wording — the queries below deliberately paraphrase these.
+DOCS = [
+ {"id": "kb-1", "title": "Rotating an API key",
+ "body": "To change your API credentials, open Settings, choose Security, "
+ "and click Regenerate token. The old token stops working immediately."},
+ {"id": "kb-2", "title": "Refund policy",
+ "body": "Customers can request their money back within 30 days of purchase. "
+ "Refunds are issued to the original payment method within five business days."},
+ {"id": "kb-3", "title": "Resetting a forgotten password",
+ "body": "If you cannot sign in, use the Forgot password link on the login page "
+ "to receive a reset email and choose new credentials."},
+ {"id": "kb-4", "title": "Supported file formats for upload",
+ "body": "You may upload documents as PDF, DOCX, or plain text. Spreadsheets "
+ "in CSV and XLSX are also accepted. Images are not indexed."},
+ {"id": "kb-5", "title": "Rate limits",
+ "body": "The API permits 600 requests per minute per key. Exceeding the "
+ "throttle returns HTTP 429; back off and retry after the reset window."},
+]
+
+
+def main():
+ # 1) Map `body` as semantic_text -> auto-embedded on ingest (no external service).
+ req("DELETE", "/" + INDEX)
+ status, resp = req("PUT", "/" + INDEX, {
+ "mappings": {
+ "properties": {
+ "title": {"type": "text"},
+ "body": {"type": "semantic_text"},
+ }
+ }
+ })
+ print(f"create index -> {status} {json.dumps(resp)[:120]}")
+ assert status == 200, resp
+
+ # 2) Bulk-ingest. XERJ embeds `body` into a companion vector at index time.
+ lines = []
+ for d in DOCS:
+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": d["id"]}}))
+ lines.append(json.dumps({"title": d["title"], "body": d["body"]}))
+ ndjson = "\n".join(lines) + "\n"
+ r = urllib.request.Request(
+ BASE + "/_bulk", data=ndjson.encode(), method="POST",
+ headers={"Content-Type": "application/x-ndjson"},
+ )
+ with urllib.request.urlopen(r) as resp:
+ bulk = json.loads(resp.read().decode())
+ print(f"bulk -> errors={bulk.get('errors')} items={len(bulk.get('items', []))}")
+ assert bulk.get("errors") is False, bulk
+ req("POST", f"/{INDEX}/_refresh")
+
+ # 3) Semantic retrieval. The query PARAPHRASES the KB — different words, same meaning.
+ def semantic(query, k=3):
+ status, resp = req("POST", f"/{INDEX}/_search", {
+ "size": k,
+ "query": {"semantic": {"field": "body", "query": query, "k": k}},
+ })
+ assert status == 200, resp
+ return resp["hits"]["hits"]
+
+ def keyword(query, k=3):
+ status, resp = req("POST", f"/{INDEX}/_search", {
+ "size": k,
+ "query": {"match": {"body": query}},
+ })
+ assert status == 200, resp
+ return resp["hits"]["hits"]
+
+ print("\n=== SEMANTIC RETRIEVAL (retrieve by meaning) ===")
+ checks = [
+ # (paraphrased user question, id we expect back at rank 1)
+ ("how do I regenerate my API token", "kb-1"),
+ ("can I get my money back after buying", "kb-2"),
+ ("I forgot my login and can't sign in", "kb-3"),
+ ("which file formats are supported for uploads", "kb-4"),
+ ("how many calls per minute am I allowed", "kb-5"),
+ ]
+ passed = 0
+ for q, expect in checks:
+ hits = semantic(q, k=3)
+ top = hits[0]
+ ok = top["_id"] == expect
+ passed += ok
+ print(f"\nQ: {q!r}")
+ print(f" top -> {top['_id']} '{top['_source']['title']}' "
+ f"score={top['_score']:.4f} {'OK' if ok else 'MISS(expected '+expect+')'}")
+ for h in hits:
+ print(f" {h['_id']:6} {h['_score']:.4f} {h['_source']['title']}")
+
+ # 4) Show the RAG context bundle you'd feed an LLM for the first question.
+ q = "how do I regenerate my API token"
+ hits = semantic(q, k=2)
+ context = "\n\n".join(
+ f"[{h['_id']}] {h['_source']['title']}\n{h['_source']['body']}" for h in hits
+ )
+ print("\n=== RAG CONTEXT BUNDLE (top-2 passages -> LLM prompt) ===")
+ print(f"User question: {q}\n")
+ print("Retrieved context:\n" + context)
+
+ # 5) Contrast: a paraphrase with NO shared keywords is where lexical differs.
+ print("\n=== KEYWORD vs SEMANTIC (same paraphrase) ===")
+ q = "get my money back after buying"
+ kw = keyword(q, k=1)
+ sm = semantic(q, k=1)
+ print(f"Q: {q!r}")
+ print(f" match top -> {(kw[0]['_id']+' '+kw[0]['_source']['title']) if kw else 'NO HITS'}")
+ print(f" semantic top -> {sm[0]['_id']} {sm[0]['_source']['title']}")
+
+ # 6) Honest limitation: the BUILT-IN embedder is lexical. A true paraphrase
+ # with no token overlap against the embedded *body* text can miss. Here
+ # the KB titled "Rotating an API key" is stored, but its body never says
+ # "rotate", so a "rotate" paraphrase does not retrieve it. Wire an
+ # external /v1/embeddings endpoint for production-grade neural semantics.
+ print("\n=== HONEST LIMITATION (lexical embedder, no body-token overlap) ===")
+ q = "how do I rotate my API key"
+ sm = semantic(q, k=1)
+ print(f"Q: {q!r}")
+ print(f" semantic top -> {sm[0]['_id']} {sm[0]['_source']['title']} "
+ f"(expected kb-1; the body never uses the word 'rotate')")
+
+ print(f"\nRESULT: {passed}/{len(checks)} semantic retrievals returned the right doc at rank 1")
+ if passed != len(checks):
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/examples/vector-search-knn/knn_demo.py b/docs/examples/vector-search-knn/knn_demo.py
new file mode 100644
index 000000000..03cd67d0c
--- /dev/null
+++ b/docs/examples/vector-search-knn/knn_demo.py
@@ -0,0 +1,156 @@
+#!/usr/bin/env python3
+"""
+Vector similarity search (kNN) against a live XERJ node.
+
+Use case: you have items (products, docs, images) each represented as an
+embedding vector, and you want "find the N most similar to this one".
+
+This script uses tiny, hand-made 4-dimensional vectors so the result is
+100% reproducible and you can eyeball why each neighbor was returned.
+The 4 dims are a toy "feature space":
+
+ [ citrus , berry , engine , wheels ]
+
+- Fruits score high on the fruit axes, ~0 on the vehicle axes.
+- Vehicles score high on engine/wheels, ~0 on the fruit axes.
+
+So a query near the "citrus" corner should return oranges/lemons first,
+and never a truck. That's the recall check.
+
+No external deps: Python 3 stdlib only (urllib + json).
+"""
+import json
+import urllib.request
+import urllib.error
+
+BASE = "http://localhost:9484"
+INDEX = "catalog"
+
+
+def req(method, path, body=None):
+ data = None
+ headers = {}
+ if body is not None:
+ if isinstance(body, str): # ndjson (bulk)
+ data = body.encode()
+ headers["Content-Type"] = "application/x-ndjson"
+ else:
+ data = json.dumps(body).encode()
+ headers["Content-Type"] = "application/json"
+ r = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
+ try:
+ with urllib.request.urlopen(r) as resp:
+ return resp.status, json.loads(resp.read().decode())
+ except urllib.error.HTTPError as e:
+ return e.code, json.loads(e.read().decode())
+
+
+def hits(resp):
+ """(id, score, name, category, in_stock) for each hit, in ranked order."""
+ out = []
+ for h in resp["hits"]["hits"]:
+ s = h["_source"]
+ out.append((h["_id"], round(h["_score"], 4), s["name"], s["category"], s["in_stock"]))
+ return out
+
+
+# ── 1. Fresh index with a dense_vector field ───────────────────────
+req("DELETE", "/" + INDEX)
+status, _ = req("PUT", "/" + INDEX, {
+ "mappings": {
+ "properties": {
+ "name": {"type": "text"},
+ "category": {"type": "keyword"},
+ "in_stock": {"type": "boolean"},
+ # dims MUST match the vectors you index. similarity defaults to
+ # cosine (magnitude-invariant) — the right choice for most
+ # normalized embeddings. Other options: l2, dot_product.
+ "embedding": {"type": "dense_vector", "dims": 4, "similarity": "cosine"},
+ }
+ }
+})
+print("create index:", status)
+
+# ── 2. Index a tiny catalog. Vectors are [citrus, berry, engine, wheels]
+catalog = [
+ ("1", "Navel Orange", "fruit", True, [0.90, 0.10, 0.00, 0.00]),
+ ("2", "Meyer Lemon", "fruit", True, [0.95, 0.05, 0.00, 0.00]),
+ ("3", "Blood Orange", "fruit", False, [0.85, 0.20, 0.00, 0.00]),
+ ("4", "Strawberry", "fruit", True, [0.10, 0.92, 0.00, 0.00]),
+ ("5", "Blueberry", "fruit", True, [0.05, 0.95, 0.00, 0.00]),
+ ("6", "Pickup Truck", "vehicle", True, [0.00, 0.00, 0.90, 0.85]),
+ ("7", "Sports Car", "vehicle", False, [0.00, 0.00, 0.95, 0.80]),
+ ("8", "Electric Sedan", "vehicle", True, [0.00, 0.05, 0.40, 0.88]),
+]
+lines = []
+for _id, name, cat, stock, vec in catalog:
+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": _id}}))
+ lines.append(json.dumps({"name": name, "category": cat, "in_stock": stock, "embedding": vec}))
+status, resp = req("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")
+print("bulk:", status, "errors=", resp.get("errors"))
+
+# ── 3. Plain kNN: "find items most like a citrus fruit" ──────────────
+query_vec = [0.92, 0.08, 0.00, 0.00] # sits right in the citrus corner
+status, resp = req("POST", "/%s/_search" % INDEX, {
+ "knn": {"field": "embedding", "query_vector": query_vec, "k": 3, "num_candidates": 10}
+})
+print("\n=== kNN k=3, query ~citrus ===")
+knn_hits = hits(resp)
+for h in knn_hits:
+ print(" ", h)
+
+top3_ids = [h[0] for h in knn_hits]
+top3_cats = [h[3] for h in knn_hits]
+# RECALL CHECK: the 3 nearest MUST be the 3 citrus fruits (ids 1,2,3),
+# and NO vehicle should appear.
+assert set(top3_ids) == {"1", "2", "3"}, top3_ids
+assert all(c == "fruit" for c in top3_cats), top3_cats
+# Cosine similarity of near-identical direction should be ~1.0.
+assert knn_hits[0][1] > 0.99, knn_hits[0]
+print(" OK: top-3 are exactly the citrus fruits, no vehicles leaked in")
+
+# ── 4. kNN + filter: same query, but only items in stock ─────────────
+# To combine kNN with a filter, wrap the knn in a `bool` and put the
+# filter in a sibling `filter` clause. bool.filter runs as a PRE-filter:
+# XERJ restricts the candidate set, THEN ranks by vector similarity.
+#
+# "Blood Orange" (id 3) is the 3rd-closest citrus but is OUT of stock, so
+# with the filter it drops out and a farther in-stock item (a berry)
+# takes its place — proving the filter runs, not just the ranker.
+#
+# NOTE: the ES-8 shorthand `knn: {..., "filter": {...}}` is parsed but the
+# filter is currently ignored on this path — use the bool wrapper below.
+status, resp = req("POST", "/%s/_search" % INDEX, {
+ "size": 3,
+ "query": {
+ "bool": {
+ "must": [
+ {"knn": {"field": "embedding", "query_vector": query_vec,
+ "k": 3, "num_candidates": 10}}
+ ],
+ "filter": [{"term": {"in_stock": True}}],
+ }
+ },
+})
+print("\n=== kNN + bool.filter in_stock:true ===")
+filtered = hits(resp)
+for h in filtered:
+ print(" ", h)
+filtered_ids = [h[0] for h in filtered]
+assert all(h[4] is True for h in filtered), filtered # every hit in stock
+assert "3" not in filtered_ids, filtered_ids # out-of-stock citrus excluded
+assert filtered_ids[:2] == ["1", "2"], filtered_ids # two in-stock citrus rank first
+print(" OK: out-of-stock Blood Orange excluded; only in-stock neighbors returned")
+
+# ── 5. Query the other corner to show the space really separates ───────
+status, resp = req("POST", "/%s/_search" % INDEX, {
+ "knn": {"field": "embedding", "query_vector": [0.0, 0.0, 0.9, 0.85], "k": 2, "num_candidates": 10}
+})
+print("\n=== kNN k=2, query ~vehicle ===")
+veh = hits(resp)
+for h in veh:
+ print(" ", h)
+assert all(h[3] == "vehicle" for h in veh), veh
+print(" OK: vehicle query returns only vehicles")
+
+print("\nALL ASSERTIONS PASSED")
diff --git a/docs/recipes/README.md b/docs/recipes/README.md
new file mode 100644
index 000000000..4104dae68
--- /dev/null
+++ b/docs/recipes/README.md
@@ -0,0 +1,17 @@
+# XERJ Recipes
+
+Use-case-driven guides for building real things with XERJ. Every recipe was **verified end-to-end against a live XERJ** before being written — the runnable example for each lives under [`docs/examples/`](../examples).
+
+These are practical "how do I actually do X" recipes, not an API reference (that follows separately).
+
+| Recipe | What you build | Example |
+|---|---|---|
+| [Give an AI agent long-term memory](./agentic-memory.md) | A memory-backed agent using the `/_memory` API (store, semantic + keyword recall, metadata filters, forgetting, per-agent isolation) | [`agent_memory.py`](../examples/agentic-memory/agent_memory.py) |
+| [Semantic search & RAG](./semantic-search-rag.md) | Retrieval by meaning with `semantic_text` (auto-embed on ingest, no separate vector DB) | [`rag_demo.py`](../examples/semantic-search-rag/rag_demo.py) |
+| [Log analytics](./log-analytics.md) | From raw logs to dashboards — error rates, p95 latency, top services via aggregations | [`log_analytics.py`](../examples/log-analytics/log_analytics.py) |
+| [Vector search (kNN)](./vector-search-knn.md) | Nearest-neighbor similarity search over `dense_vector` (HNSW), with filters | [`knn_demo.py`](../examples/vector-search-knn/knn_demo.py) |
+| [Hybrid search](./hybrid-search.md) | Keyword + vector in one query — results neither BM25 nor kNN finds alone | [`hybrid_search.py`](../examples/hybrid-search/hybrid_search.py) |
+| [Anomaly detection](./anomaly-detection.md) | Statistical `_ml` detectors that flag spikes in metrics/logs | [`anomaly_detection.py`](../examples/anomaly-detection/anomaly_detection.py) |
+| [Migrate from Elasticsearch](./migrate-from-elasticsearch.md) | Point your existing ES client at XERJ — same wire, change the URL | [`migrate_demo.sh`](../examples/migrate-from-elasticsearch/migrate_demo.sh) |
+
+Every example is stdlib-only Python 3 or plain `curl` — no dependencies to install. Start XERJ (`./target/release/xerj --data-dir ./data --insecure`), then run any example.
diff --git a/docs/recipes/agentic-memory.md b/docs/recipes/agentic-memory.md
new file mode 100644
index 000000000..37b1162f9
--- /dev/null
+++ b/docs/recipes/agentic-memory.md
@@ -0,0 +1,334 @@
+# Give an AI agent long-term memory with XERJ
+
+**Use case:** your agent learns things during a session — user preferences, project
+facts, decisions — and needs to *remember* them on a later turn to answer well.
+The usual answer is to bolt a separate vector database next to your search stack.
+XERJ folds that job into the engine you already run: a namespaced, offline
+**agent-memory API** (`/_memory/{ns}`) that stores text + an embedding + metadata
+and recalls the most relevant memories by **vector similarity** or **plain text**,
+with metadata filtering, forgetting, and hard per-agent isolation.
+
+This recipe builds a tiny simulated coding-assistant agent (no LLM required) that:
+
+1. **stores** what it learns as memories (`POST /_memory/{ns}`),
+2. **recalls** the right memory to answer a new question — semantically (kNN) and
+ lexically (BM25),
+3. does a **metadata-filtered** recall ("only surface team *preferences*"),
+4. **forgets** a memory that is no longer true, and
+5. proves **multi-namespace isolation**: a second agent can never see the first
+ agent's memories.
+
+Everything below was run end-to-end against a live XERJ. No pip installs — the
+example is Python 3 stdlib only.
+
+---
+
+## Why this replaces a bolt-on vector DB
+
+A memory namespace is just a reserved XERJ index (`.xerj-memory-{ns}`) under the
+hood, so recall reuses the same `dense_vector`/HNSW kNN, BM25, and metadata-filter
+paths that serve the rest of the engine. That means:
+
+- **One system to run and back up.** No second datastore, no sync job keeping a
+ vector index consistent with your source of truth.
+- **kNN *and* BM25 in the same store.** Recall by embedding similarity, by keyword,
+ or filter both by metadata — no glue code.
+- **Namespaces are physical isolation.** Each agent (or tenant, or user) gets its
+ own backing index; a recall in namespace `A` literally cannot read namespace `B`.
+- **Offline and zero-config.** You supply the vectors (from your own model); XERJ
+ never phones out to embed.
+
+---
+
+## The memory API, in four calls
+
+| Verb + path | What it does | Body / result |
+|---|---|---|
+| `POST /_memory/{ns}` | store a memory | `{text, vector?, metadata?, id?}` → `{id, namespace, created}` |
+| `POST /_memory/{ns}/_recall` | recall top-k | `{vector? \| query?, k?, filter?}` → `{hits:[{id,text,metadata,score}]}` |
+| `GET /_memory/{ns}` | list recent (bounded) | → `{count, entries:[…]}` |
+| `DELETE /_memory/{ns}/{id}` | forget one | → `{id, forgotten}` |
+| `DELETE /_memory/{ns}` | drop the namespace | → `{namespace, dropped}` |
+
+`_recall` picks the path from the body: a `vector` runs kNN; otherwise `query`
+runs BM25 over the text; an empty body returns recent memories (`match_all`).
+`filter` is a normal ES query clause applied as a `bool` filter, so it narrows
+without affecting the score. Recall of an unknown namespace returns `{"hits": []}` —
+that is how isolation stays clean.
+
+### Storing a memory (raw wire)
+
+```bash
+curl -XPOST localhost:9481/_memory/agent-ada -H 'content-type: application/json' -d '{
+ "text": "The team prefers Rust over Go for new backend services.",
+ "vector": [0.12, 0.0, 0.34],
+ "metadata": {"kind": "preference", "topic": "language"},
+ "id": "demo-1"
+}'
+```
+```json
+{"created":true,"id":"demo-1","namespace":"agent-ada"}
+```
+
+### Recalling by vector, filtered to preferences (raw wire)
+
+```bash
+curl -XPOST localhost:9481/_memory/agent-ada/_recall -H 'content-type: application/json' -d '{
+ "vector": [0.12, 0.0, 0.34],
+ "k": 2,
+ "filter": {"term": {"metadata.kind": "preference"}}
+}'
+```
+```json
+{"hits":[{"id":"demo-1",
+ "metadata":{"kind":"preference","topic":"language"},
+ "score":1.0,
+ "text":"The team prefers Rust over Go for new backend services."}],
+ "namespace":"agent-ada"}
+```
+
+Note `metadata.kind` filters straight out of the box — nested metadata subfields
+are queryable with a normal `term` clause, no explicit mapping needed.
+
+---
+
+## The full agent
+
+The script below is the whole example. The one thing to understand about it: it
+carries a **tiny self-contained "embedder"** (deterministic hashed character
+trigrams) so the demo needs no model download and cosine similarity is meaningful.
+
+> **Honesty check.** That toy embedder is *lexical*, not neural: it rewards shared
+> character n-grams (so `service`/`services`, `prefer`/`prefers` land near each
+> other), but it does not know that "car" ≈ "automobile". For real semantic recall
+> you pass vectors from your own model (OpenAI/Cohere/a local model) — the store and
+> recall calls are byte-for-byte the same. XERJ's own built-in `semantic_text`
+> embedder is likewise lexical unless you configure an external `/v1/embeddings`
+> endpoint; the memory API deliberately takes *your* vectors so quality is your
+> choice.
+
+```python
+#!/usr/bin/env python3
+"""Long-term memory for an AI agent, backed by XERJ's /_memory API."""
+
+import json
+import math
+import re
+import urllib.request
+from urllib.error import HTTPError
+
+BASE = "http://localhost:9481"
+DIM = 96 # embedding dimensionality for the toy hashing embedder
+
+
+# Tiny self-contained "embedder": deterministic hashed character trigrams.
+# NOT neural -- real synonymy needs real embeddings. In production you pass
+# vectors from your own model; XERJ stores and kNN-searches them the same way.
+def embed(text: str) -> list[float]:
+ s = " " + re.sub(r"[^a-z0-9]+", " ", text.lower()) + " "
+ vec = [0.0] * DIM
+ for i in range(len(s) - 2):
+ h = 0
+ for ch in s[i:i + 3]: # hash each character trigram into a bucket
+ h = (h * 131 + ord(ch)) & 0xFFFFFFFF
+ vec[h % DIM] += 1.0
+ norm = math.sqrt(sum(v * v for v in vec)) or 1.0
+ return [round(v / norm, 6) for v in vec]
+
+
+def _req(method, path, body=None):
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(BASE + path, data=data, method=method,
+ headers={"Content-Type": "application/json"})
+ try:
+ with urllib.request.urlopen(req) as r:
+ return json.loads(r.read() or b"{}")
+ except HTTPError as e:
+ return json.loads(e.read() or b"{}")
+
+
+def remember(ns, text, metadata, mem_id):
+ return _req("POST", f"/_memory/{ns}",
+ {"text": text, "vector": embed(text), "metadata": metadata, "id": mem_id})
+
+def recall_vec(ns, query, k=3, filt=None): # semantic (kNN)
+ body = {"vector": embed(query), "k": k}
+ if filt:
+ body["filter"] = filt
+ return _req("POST", f"/_memory/{ns}/_recall", body)["hits"]
+
+def recall_text(ns, query, k=3): # lexical (BM25)
+ return _req("POST", f"/_memory/{ns}/_recall", {"query": query, "k": k})["hits"]
+
+def forget(ns, mem_id):
+ return _req("DELETE", f"/_memory/{ns}/{mem_id}")
+
+def drop(ns):
+ return _req("DELETE", f"/_memory/{ns}")
+
+def show(label, hits):
+ print(f"\n{label}")
+ for h in hits:
+ kind = (h.get("metadata") or {}).get("kind", "-")
+ print(f" [{h['score']:.3f}] ({kind:10}) {h['text']}")
+
+
+def main():
+ ADA, BILLING = "agent-ada", "agent-billing"
+ drop(ADA); drop(BILLING) # clean slate for a repeatable demo
+
+ print("== Storing what the agent learned this session ==")
+ facts = [
+ ("The team prefers Rust over Go for new backend services.",
+ {"kind": "preference", "topic": "language"}),
+ ("Production runs on AWS in us-east-1; staging is us-west-2.",
+ {"kind": "fact", "topic": "infra"}),
+ ("The team prefers tabs over spaces and a 100-column line limit.",
+ {"kind": "preference", "topic": "style"}),
+ ("The payments service owns the Postgres 'ledger' database.",
+ {"kind": "fact", "topic": "ownership"}),
+ ("Deploys are gated on the full integration suite passing in CI.",
+ {"kind": "fact", "topic": "process"}),
+ ("The team prefers OpenTelemetry for tracing, not vendor SDKs.",
+ {"kind": "preference", "topic": "observability"}),
+ ]
+ for i, (text, md) in enumerate(facts, 1):
+ r = remember(ADA, text, md, f"ada-{i}")
+ print(f" stored {r['id']:8} created={r['created']}")
+
+ remember(BILLING, "Invoices over $10k require CFO approval.", {"kind": "policy"}, "bill-1")
+
+ q1 = "What programming language should I use for a new backend service?"
+ show(f'== Semantic recall for: "{q1}"', recall_vec(ADA, q1, k=3))
+
+ q2 = "Which region is production deployed in?"
+ show(f'== Lexical (BM25) recall for: "{q2}"', recall_text(ADA, q2, k=3))
+
+ q3 = "How should I set up the new service?"
+ show(f'== Preference-only recall for: "{q3}"',
+ recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}}))
+
+ print("\n== Namespace isolation ==")
+ leak = recall_vec(BILLING, q1, k=3)
+ billing_texts = [h["text"] for h in leak]
+ print(f" agent-billing recall for a coding question -> {billing_texts}")
+ assert all("Rust" not in t and "AWS" not in t for t in billing_texts)
+ print(" OK: agent-billing only ever sees its own namespace.")
+
+ print("\n== Forgetting a memory that is no longer true ==")
+ show(" before forget:", recall_vec(ADA, "what do we use for tracing", k=3))
+ print(" forget ->", forget(ADA, "ada-6"))
+ after = recall_vec(ADA, "what do we use for tracing", k=3)
+ show(" after forget:", after)
+ assert all(h["id"] != "ada-6" for h in after)
+ print(" OK: the retracted memory is gone.")
+
+ # Correctness checks so a green run really means it worked.
+ assert "Rust" in recall_vec(ADA, q1, k=1)[0]["text"]
+ assert "us-east-1" in recall_text(ADA, q2, k=1)[0]["text"]
+ prefs = recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}})
+ assert prefs and all((h["metadata"] or {}).get("kind") == "preference" for h in prefs)
+ print("\nAll assertions passed.")
+
+
+if __name__ == "__main__":
+ main()
+```
+
+---
+
+## Run it
+
+Start XERJ (single node, no TLS/auth) and run the script:
+
+```bash
+# 1. boot XERJ
+printf '[server]\nes_compat_port = 9481\n' > /tmp/xerj.toml
+./engine/target/release/xerj --insecure --data-dir /tmp/xerj-mem --config /tmp/xerj.toml &
+
+# 2. run the agent
+python3 docs/examples/agentic-memory/agent_memory.py
+```
+
+### Real output
+
+```text
+== Storing what the agent learned this session ==
+ stored ada-1 created=True
+ stored ada-2 created=True
+ stored ada-3 created=True
+ stored ada-4 created=True
+ stored ada-5 created=True
+ stored ada-6 created=True
+
+== Semantic recall for: "What programming language should I use for a new backend service?"
+ [0.779] (preference) The team prefers Rust over Go for new backend services.
+ [0.741] (fact ) The payments service owns the Postgres 'ledger' database.
+ [0.731] (preference) The team prefers tabs over spaces and a 100-column line limit.
+
+== Lexical (BM25) recall for: "Which region is production deployed in?"
+ [3.746] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.
+ [1.010] (fact ) Deploys are gated on the full integration suite passing in CI.
+
+== Preference-only recall for: "How should I set up the new service?"
+ [0.765] (preference) The team prefers Rust over Go for new backend services.
+ [0.692] (preference) The team prefers tabs over spaces and a 100-column line limit.
+ [0.666] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.
+
+== Namespace isolation ==
+ agent-billing recall for a coding question -> ['Invoices over $10k require CFO approval.']
+ OK: agent-billing only ever sees its own namespace.
+
+== Forgetting a memory that is no longer true ==
+
+ before forget:
+ [0.755] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.
+ [0.692] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.
+ [0.683] (preference) The team prefers Rust over Go for new backend services.
+ forget -> {'forgotten': True, 'id': 'ada-6', 'namespace': 'agent-ada'}
+
+ after forget:
+ [0.692] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.
+ [0.683] (preference) The team prefers Rust over Go for new backend services.
+ [0.661] (preference) The team prefers tabs over spaces and a 100-column line limit.
+ OK: the retracted memory is gone.
+
+All assertions passed.
+```
+
+Read the run top to bottom:
+
+- **Semantic recall** put the Rust preference on top for a question that never
+ says "Rust" or "prefer" — trigram overlap on *backend / service / new* pulled it
+ up over unrelated facts.
+- **BM25 recall** answered the region question exactly, no vectors involved.
+- **Preference-only recall** returned three memories, *all* `kind:"preference"` —
+ the `metadata.kind` filter did its job.
+- **Isolation** held: `agent-billing`, asked a coding question, only saw its own
+ single finance memory.
+- **Forgetting** removed `ada-6` (the tracing preference); the follow-up recall no
+ longer surfaces it.
+
+---
+
+## Notes, limits, and going to production
+
+- **Bring your own embeddings.** The memory API stores whatever vector you pass and
+ kNN-searches it with cosine similarity. Swap the toy `embed()` for a call to your
+ model's embeddings endpoint and everything else is unchanged. Keep the vector
+ dimension consistent within a namespace — the backing index's `dense_vector`
+ field is sized from the first stored vector.
+- **Vector *or* text.** A memory needs at least `text` or a `vector`. Text is always
+ BM25-indexed; a vector additionally enables kNN recall. Recall with `vector` uses
+ kNN; without it, `query` uses BM25.
+- **Recency.** The store blends similarity with a recency signal internally, so
+ fresh memories surface even when marginally less similar — useful for agents whose
+ latest observation should win ties.
+- **Isolation is structural, not a filter.** Namespaces are separate physical
+ indices, so there is no query you can write in one namespace that reads another.
+ Use one namespace per agent/tenant/user.
+- **`GET /_memory/{ns}`** lists up to 100 most-recent entries for debugging/audit;
+ it is bounded and recent-first, not a full export.
+- **Wire-compatible.** Because a namespace is a real index, you can inspect or
+ operate on it with the standard ES-compatible surface if you ever need to — the
+ memory API is a thin, honest adapter over the same machinery.
\ No newline at end of file
diff --git a/docs/recipes/anomaly-detection.md b/docs/recipes/anomaly-detection.md
new file mode 100644
index 000000000..34464a9a8
--- /dev/null
+++ b/docs/recipes/anomaly-detection.md
@@ -0,0 +1,226 @@
+# Detect a spike in a metric series
+
+**Goal:** you have a timestamped metric — CPU, latency, request rate, error
+count — and you want to know *when it went wrong* without hand-tuning a fixed
+red-line. A CPU box that idles at 20% is fine at 20% and alarming at 96%, but
+the "alarming" number depends on the box. You want the engine to learn the
+normal band and flag the minute it broke out.
+
+XERJ ships an on-demand anomaly detector (`_ml/anomaly_detectors`, Elasticsearch
+wire-shape). You point it at an index, tell it the time field, the metric
+(`mean(cpu)`), and a bucket span. When you score it, XERJ buckets the data by
+time, computes the metric per bucket, builds a **moving baseline from prior
+normal buckets**, and scores each bucket by how many standard deviations it sits
+from that baseline. A spike lands many sigmas out and gets flagged; ordinary
+jitter does not.
+
+> **Honest scope.** This is **on-demand batch scoring**: you POST `_score` and
+> get the full timeline back in one response. There is **no continuous datafeed
+> or background job** yet — nothing runs on a timer, and there is no alerting.
+> To monitor live, call `_score` on a schedule (cron, your alertmanager, a
+> `/loop`) over a recent time window and act on `anomaly_count`. The model is a
+> transparent moving-window z-score, not a seasonal/multivariate ML model — it's
+> excellent for "this metric jumped relative to its recent self," which covers
+> the large majority of ops spikes.
+
+## The scenario
+
+`cpu_metrics` collects four host readings per minute. For fifteen minutes CPU
+sits around 20%. At **minute 12** a runaway process pins a box and the per-minute
+mean jumps to **96%**, then recovers. We want XERJ to flag exactly that minute.
+
+## 1. Map the index
+
+The time field must be a `date`; the metric must be numeric so the detector can
+average it.
+
+```bash
+curl -XPUT localhost:9486/cpu_metrics -H 'Content-Type: application/json' -d '{
+ "mappings": { "properties": {
+ "@timestamp": { "type": "date" },
+ "host": { "type": "keyword" },
+ "cpu": { "type": "double" }
+ }}
+}'
+```
+
+## 2. Bulk-ingest the series (spike injected at minute 12)
+
+Standard `_bulk` NDJSON — 16 minutes × 4 hosts. Normal minutes read ~20%; minute
+12 reads ~96%. (Full generator in the script below.)
+
+```
+{"index":{"_index":"cpu_metrics"}}
+{"@timestamp":"2026-07-06T00:00:00Z","host":"web-0","cpu":19}
+...
+{"index":{"_index":"cpu_metrics"}}
+{"@timestamp":"2026-07-06T00:12:00Z","host":"web-1","cpu":96} <- the spike
+...
+```
+
+## 3. Create the detector
+
+`function: mean` over the `cpu` field, one-minute buckets. `anomaly_threshold`
+is the z-score gate (defaults to `3.0` — a bucket is flagged when it sits more
+than 3σ from the baseline).
+
+```bash
+curl -XPUT localhost:9486/_ml/anomaly_detectors/cpu-spike \
+ -H 'Content-Type: application/json' -d '{
+ "source_index": "cpu_metrics",
+ "time_field": "@timestamp",
+ "function": "mean",
+ "field": "cpu",
+ "bucket_span": "1m",
+ "description": "CPU mean per minute"
+ }'
+```
+
+Response (the config, ES-shaped):
+
+```json
+{
+ "job_id": "cpu-spike",
+ "job_type": "anomaly_detector",
+ "analysis_config": {
+ "bucket_span": "1m",
+ "detectors": [{ "function": "mean", "field_name": "cpu" }]
+ },
+ "data_description": { "time_field": "@timestamp" },
+ "source_index": ["cpu_metrics"],
+ "anomaly_threshold": 3.0
+}
+```
+
+Supported `function` values: `count` (no field needed — buckets by doc count,
+great for request/error rate), `mean`/`avg`, `min`, `max`, `sum`.
+
+## 4. Score it
+
+```bash
+curl -XPOST localhost:9486/_ml/anomaly_detectors/cpu-spike/_score
+```
+
+XERJ returns the whole timeline plus a ranked `anomalies` list. Rendered per
+bucket:
+
+```
+minute (UTC) actual expected z_score score flag
+00:00:00 20.0 - - 0.0 <- warm-up
+00:01:00 21.0 - - 0.0 <- warm-up
+00:02:00 19.0 - - 0.0 <- warm-up
+00:03:00 20.0 - - 0.0 <- warm-up
+00:04:00 22.0 20.0 2.8 48.0
+00:05:00 18.0 20.4 -2.4 41.9
+00:06:00 21.0 20.0 0.8 16.4
+00:07:00 20.0 20.1 -0.1 2.6
+00:08:00 19.0 20.1 -1.0 20.0
+00:09:00 21.0 20.0 0.9 18.1
+00:10:00 20.0 20.1 -0.1 2.0
+00:11:00 22.0 20.1 1.8 33.5
+00:12:00 96.0 20.2 65.1 100.0 *** <- SPIKE
+00:13:00 20.0 20.2 -0.2 4.8
+00:14:00 21.0 20.2 0.7 14.7
+00:15:00 19.0 20.3 -1.2 23.7
+
+== anomalies (1 of 16 buckets) ==
+ 2026-07-06T00:12:00.000Z actual=96.0 expected=20.2 z=65.1 score=100.0
+```
+
+The spike record in full:
+
+```json
+{
+ "actual": 96.0,
+ "expected": 20.25,
+ "std_dev": 1.1636866703140785,
+ "z_score": 65.09484205018445,
+ "anomaly_score": 100.0,
+ "is_anomaly": true,
+ "timestamp_iso": "2026-07-06T00:12:00.000Z"
+}
+```
+
+## How the scoring works (and how to read the numbers)
+
+- **Warm-up.** The first 4 buckets seed the baseline and are not scored
+ (`"expected": null`, `"note": "insufficient_baseline"`). Give the detector a
+ span of history before the interesting part.
+- **`expected` / `std_dev`.** The mean and standard deviation of prior **normal**
+ buckets. Crucially, a flagged bucket is **not folded back into the baseline** —
+ one spike can't poison the expectation for the minutes after it. Notice
+ `expected` stays ~20 straight through the spike.
+- **`z_score`.** `(actual − expected) / std_dev`. The spike is 65σ out; normal
+ jitter stays within a couple of sigma.
+- **`is_anomaly`** is the decision, gated purely on `|z| > anomaly_threshold`.
+ Only the spike crosses it here.
+- **`anomaly_score` (0–100)** is a normalized, human-friendly severity: it maps
+ |z| = threshold to ~50 and saturates toward 100 for large deviations. It is
+ **not** the flag. That's why the `00:04` jitter bucket shows a mid score of 48
+ (its z of 2.8 is close to the threshold of 3) yet is correctly *not* flagged.
+ **Alert on `is_anomaly` / `anomaly_count`; use `anomaly_score` for ranking and
+ dashboards.** In this run the spike (100.0) is more than 2× the top normal
+ bucket (48.0) — clean separation.
+
+## Narrow or re-tune a run without recreating the job
+
+`_score` accepts two per-run overrides in the POST body:
+
+- **`query`** — restrict which docs feed the buckets (e.g. one `host`, one
+ service). Any XERJ query works: `{"query": {"term": {"host": "web-1"}}}`.
+- **`anomaly_threshold`** — override the flag sensitivity for this run only.
+
+```bash
+# Same data, stricter gate (10σ) — the 65σ spike still stands alone.
+curl -XPOST localhost:9486/_ml/anomaly_detectors/cpu-spike/_score \
+ -H 'Content-Type: application/json' -d '{"anomaly_threshold": 10}'
+# -> "anomaly_threshold": 10.0, "anomaly_count": 1
+```
+
+## Count instead of mean (rate spikes)
+
+For "too many errors this minute" you don't need a metric field — use
+`function: count`, drop `field`, and point the detector at your log index. Each
+bucket's `actual` becomes the doc count, scored the same way. Same recipe, same
+response shape.
+
+## List detectors
+
+```bash
+curl localhost:9486/_cat/ml/anomaly_detectors
+# cpu-spike opened cpu_metrics mean 1m
+```
+
+`GET /_ml/anomaly_detectors/{id}` and `DELETE /_ml/anomaly_detectors/{id}` round
+out lifecycle management.
+
+## Run it
+
+```bash
+# XERJ listening on :9486 (adjust XERJ=... for another port)
+python3 docs/examples/anomaly-detection/anomaly_detection.py
+```
+
+The script maps the index, ingests the series with the injected spike, creates
+the detector, scores it, prints the table above, and **asserts** the result:
+exactly one anomaly, at 00:12, score 100, mean 96 — and no normal bucket
+flagged. It exits non-zero if any of that regresses, so it doubles as a
+smoke test.
+
+## Endpoint cheat-sheet
+
+| Method | Path | Purpose |
+|---|---|---|
+| `PUT` | `/_ml/anomaly_detectors/{id}` | create/replace a detector |
+| `GET` | `/_ml/anomaly_detectors/{id}` | fetch one detector's config |
+| `GET` | `/_ml/anomaly_detectors` | list all detectors |
+| `DELETE` | `/_ml/anomaly_detectors/{id}` | remove a detector |
+| `POST` | `/_ml/anomaly_detectors/{id}/_score` | score now; body may set `query`, `anomaly_threshold` |
+| `GET` | `/_cat/ml/anomaly_detectors` | one line per detector |
+
+Config fields: `source_index` (required), `time_field` (default `@timestamp`),
+`function` (`count`/`mean`/`min`/`max`/`sum`, default `count`), `field`
+(required for everything but `count`), `bucket_span` (default `1h`),
+`anomaly_threshold` (z-gate, default `3.0`), `description`. ES-style
+`analysis_config.detectors[]` and `data_description.time_field` are also
+accepted for drop-in configs.
diff --git a/docs/recipes/hybrid-search.md b/docs/recipes/hybrid-search.md
new file mode 100644
index 000000000..785060375
--- /dev/null
+++ b/docs/recipes/hybrid-search.md
@@ -0,0 +1,203 @@
+# Hybrid search: keyword + vector in one query
+
+## The problem
+
+You run a help-desk search box. A user types **"reset password"**. Two
+things are true at once:
+
+- The best answer might share **no keywords** with the query. An article
+ titled *"Regain entry to a locked-out account"* is exactly what they
+ need, but BM25 never sees it — there's no lexical overlap.
+- A keyword-exact article might be **off-topic**. *"Password complexity
+ and rotation policy"* contains "password," so BM25 loves it, but a
+ semantic ranker knows it's about policy, not recovery.
+
+Keyword search (BM25) is precise on exact terms and useless on synonyms.
+Vector search (kNN) captures meaning but drops exact-term matches whose
+embeddings drift. The usual "fix" is to run **two systems** — a lexical
+engine and a vector DB — then stitch and re-rank the results in your
+application. That's two deployments, two writes on every ingest, and a
+fusion step you have to build and maintain yourself.
+
+## Why XERJ
+
+XERJ indexes the text field and the `dense_vector` field in the **same
+index**, and fuses BM25 and kNN in a **single `hybrid` query**. One
+request goes out, one ranked list comes back — already fused with
+Reciprocal Rank Fusion (RRF). No second datastore, no client-side
+re-ranking.
+
+## The solution
+
+### 1. One index, two modalities
+
+A `text` field for BM25 and a `dense_vector` field for kNN:
+
+```json
+PUT /helpdesk
+{
+ "mappings": {
+ "properties": {
+ "title": { "type": "text" },
+ "vec": { "type": "dense_vector", "dims": 4, "similarity": "cosine" }
+ }
+ }
+}
+```
+
+> The vectors in this recipe are hand-authored 4-dim *topic* vectors so
+> the demo is deterministic and you can see exactly why each result
+> ranks where it does. In production you fill `vec` with a real embedding
+> model — or skip the vectors entirely and use a `semantic_text` field,
+> which auto-embeds on ingest. (XERJ's built-in embedder is lexical, good
+> for demos and keyword-semantic overlap; point `inference_id` at an
+> external `/v1/embeddings` endpoint for neural-quality vectors.)
+
+### 2. The corpus
+
+Each doc has a title (for BM25) and a topic vector (for kNN). Topic axes
+are `[auth_recovery, security_policy, cooking, generic]`:
+
+| id | title | vector | keyword "password"? | near query vector? |
+|----|-------|--------|:--:|:--:|
+| d1 | Reset your password | `[1.00, 0.15, 0, 0.10]` | yes | yes |
+| d2 | Regain entry to a locked-out account | `[0.95, 0.10, 0, 0.15]` | **no** | yes |
+| d3 | Password complexity and rotation policy | `[0.15, 1.00, 0, 0.10]` | yes | **no** |
+| d4 | How to bake sourdough bread | `[0.00, 0.00, 1.0, 0.10]` | no | no |
+| d5 | Change your account password | `[0.85, 0.25, 0, 0.20]` | yes | yes |
+
+The user's intent in both modalities: text `"reset password"`, vector
+`[1.0, 0.20, 0.0, 0.10]` (an "auth recovery" direction).
+
+### 3. Watch each modality fail on its own
+
+**Pure BM25** — `{"match": {"title": "reset password"}}`:
+
+```
+d1 score=0.5754 Reset your password
+d3 score=0.2877 Password complexity and rotation policy
+d5 score=0.2877 Change your account password
+```
+
+It **misses d2** entirely — "Regain entry to a locked-out account" has no
+shared keywords, so it can never be a BM25 hit.
+
+**Pure kNN** — `{"knn": {"field": "vec", "query_vector": [...], "k": 3}}`,
+showing the 3 nearest:
+
+```
+d1 score=0.9994 Reset your password
+d2 score=0.9971 Regain entry to a locked-out account
+d5 score=0.9942 Change your account password
+```
+
+It surfaces d2 (great!) but **misses d3** — the password-policy article's
+topic vector points at "security_policy," far from the query, so it falls
+below the top-3 cutoff.
+
+### 4. Hybrid recovers both — one query
+
+```json
+POST /helpdesk/_search
+{
+ "size": 5,
+ "query": {
+ "hybrid": {
+ "queries": [
+ { "query": { "match": { "title": "reset password" } }, "weight": 1.0 },
+ { "query": { "knn": { "field": "vec",
+ "query_vector": [1.0, 0.2, 0.0, 0.1],
+ "k": 3 } }, "weight": 1.0 }
+ ],
+ "fusion": "rrf"
+ }
+ }
+}
+```
+
+Real response (trimmed to `title`):
+
+```json
+{
+ "hits": {
+ "total": { "value": 5, "relation": "eq" },
+ "max_score": 0.03278,
+ "hits": [
+ { "_id": "d1", "_score": 0.03278, "_source": { "title": "Reset your password" } },
+ { "_id": "d5", "_score": 0.03200, "_source": { "title": "Change your account password" } },
+ { "_id": "d3", "_score": 0.03150, "_source": { "title": "Password complexity and rotation policy" } },
+ { "_id": "d2", "_score": 0.01613, "_source": { "title": "Regain entry to a locked-out account" } },
+ { "_id": "d4", "_score": 0.01538, "_source": { "title": "How to bake sourdough bread" } }
+ ]
+ }
+}
+```
+
+**d2** (only kNN found it) and **d3** (only BM25 found it) are now *both*
+on the page, and **d1** — the doc that matched on *both* keyword and
+vector — is still ranked #1. That's the whole point: fusion rewards
+agreement between the two signals and still admits the strong
+single-signal hits that either method alone would have dropped.
+
+## How the fusion works
+
+`"fusion": "rrf"` uses Reciprocal Rank Fusion: each sub-query contributes
+`weight / (k + rank)` for a doc (default `k = 60`), summed across
+sub-queries. It ranks by *position*, not raw score, so BM25's `~0.5`
+scores and kNN's `~0.99` cosines never have to be normalized onto a
+common scale — a persistent headache when you fuse two systems yourself.
+
+Other `fusion` options:
+
+- `"rrf"` or `{"type": "rrf", "k": 40}` — rank-based, the safe default.
+- `"linear"` — weighted sum of the raw sub-query scores (use `weight` to
+ tilt toward keyword or vector; requires comparable score scales).
+
+Per-query `weight` lets you bias the blend, e.g. `1.5` on the `match`
+sub-query to favor exact terms.
+
+## Run it
+
+No dependencies beyond Python 3 stdlib. Point it at a running XERJ:
+
+```bash
+# start XERJ (insecure, local data dir) on port 9485
+xerj --insecure --data-dir /tmp/xerj-hybrid --config <(printf '[server]\nes_compat_port = 9485\n')
+
+# in another shell
+python3 docs/examples/hybrid-search/hybrid_search.py
+```
+
+Expected tail:
+
+```
+--- assertions ---
+PASS BM25 alone MISSES d2 (locked-out account: no shared keywords)
+PASS kNN alone MISSES d3 (password policy: topic vector too far)
+PASS Hybrid surfaces BOTH d2 and d3 in a single query
+PASS d1 (keyword + vector match) still ranks #1 under fusion
+
+BM25 ids : ['d1', 'd3', 'd5']
+kNN ids : ['d1', 'd2', 'd5']
+Hybrid ids: ['d1', 'd3', 'd5', 'd2', 'd4']
+
+OK
+```
+
+## Notes & limits
+
+- **`knn` in a query scores every doc that has the field**, ranked by
+ cosine; `size` (or `from`+`size`) is the cutoff the user sees. So `k:3`
+ plus `size:3` means "show the 3 nearest." For a hard candidate cap
+ independent of `size`, keep `k` small — it's carried on the sub-query.
+- **Score scales differ by fusion.** RRF returns small fused scores
+ (`~0.01–0.03` here) — compare docs *within* a response, not against
+ BM25/cosine absolutes.
+- **Tie ordering is not guaranteed** for docs with near-identical fused
+ scores (d3 and d5 differ by `0.00005` here), so exact positions among
+ ties can vary run to run. Set membership — the point of this recipe —
+ is stable.
+- **Vector quality is your embeddings' job.** This recipe supplies
+ vectors directly for a deterministic demo. For real semantic recall,
+ generate `dense_vector` values with a production embedding model, or
+ use a `semantic_text` field with an external `inference_id`.
diff --git a/docs/recipes/log-analytics.md b/docs/recipes/log-analytics.md
new file mode 100644
index 000000000..b1b989195
--- /dev/null
+++ b/docs/recipes/log-analytics.md
@@ -0,0 +1,207 @@
+# Log analytics with XERJ: from raw logs to dashboards
+
+**Goal:** You have a stream of structured application logs — timestamp, level,
+service, latency, status — and you need to answer the questions an on-call
+engineer actually asks: *Is the error rate climbing? Which service owns the
+latency tail? Who's taking the most traffic?* This recipe bulk-ingests a few
+hundred synthetic log events and answers all three with plain Elasticsearch
+aggregations against a single XERJ instance — no separate metrics store, no
+second query language.
+
+Everything below was run end-to-end against a live XERJ. The numbers and JSON
+are the real captured output.
+
+## Why XERJ for this
+
+XERJ speaks the Elasticsearch wire protocol, so the aggregation surface you
+already know — `date_histogram`, `percentiles`, `terms`, `stats`, and `filter`
+sub-aggs — works as-is. You ingest with `_bulk` and query with `_search`. One
+Rust engine handles the logs, the aggregations, and (if you want them later)
+full-text and vector search on the same documents. No Logstash/metrics-store
+split to operate.
+
+## Set up the index
+
+Give the log fields explicit types up front. `@timestamp` must be a `date` for
+`date_histogram`, `latency_ms`/`status` numeric for math, and `service`/`level`
+`keyword` so they bucket exactly instead of being analyzed.
+
+```bash
+curl -XPUT localhost:9483/logs-app -H 'Content-Type: application/json' -d '{
+ "mappings": { "properties": {
+ "@timestamp": {"type": "date"},
+ "level": {"type": "keyword"},
+ "service": {"type": "keyword"},
+ "latency_ms": {"type": "integer"},
+ "status": {"type": "integer"},
+ "message": {"type": "text"}
+ }}
+}'
+```
+
+## Ingest with `_bulk`
+
+Logs arrive in volume, so use the bulk API: alternating action + document
+NDJSON lines. The example script generates 600 events across 6 hours and 4
+services (`checkout`, `search`, `catalog`, `payments`), each with its own
+baseline latency and error propensity so the aggregations tell a real story.
+One `_bulk` call ships them all:
+
+```
+POST /_bulk (Content-Type: application/x-ndjson)
+{"index":{"_index":"logs-app"}}
+{"@timestamp":"2026-07-06T08:14:03+00:00","level":"INFO","service":"search","latency_ms":61,"status":200,"message":"search handled request status=200"}
+{"index":{"_index":"logs-app"}}
+{"@timestamp":"2026-07-06T08:22:41+00:00","level":"ERROR","service":"payments","latency_ms":740,"status":503,"message":"payments handled request status=503"}
+...
+```
+
+Check `errors` in the response is `false`, then `_refresh` to make the docs
+searchable. `GET /logs-app/_count` confirms 600 landed.
+
+## Q1 — Error rate over time
+
+Bucket events by hour with `date_histogram`, and inside each bucket count the
+ERROR events with a `filter` sub-agg. `errors.doc_count / doc_count` is the
+error rate for that hour — exactly the shape a dashboard panel needs.
+
+```json
+POST /logs-app/_search
+{
+ "size": 0,
+ "aggs": {
+ "per_hour": {
+ "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" },
+ "aggs": {
+ "errors": { "filter": { "term": { "level": "ERROR" } } }
+ }
+ }
+ }
+}
+```
+
+`size: 0` means "no hits, just the aggregations." Real response (first two
+buckets, `took` in ms):
+
+```json
+{
+ "took": 4,
+ "buckets": [
+ { "key_as_string": "2026-07-06T08:00:00.000Z", "doc_count": 102,
+ "errors": { "doc_count": 7 } },
+ { "key_as_string": "2026-07-06T09:00:00.000Z", "doc_count": 101,
+ "errors": { "doc_count": 3 } }
+ ]
+}
+```
+
+Rolled up across all six buckets:
+
+```
+hour (UTC) total errors err_rate
+08:00 102 7 6.9%
+09:00 101 3 3.0%
+10:00 106 5 4.7%
+11:00 91 2 2.2%
+12:00 93 2 2.2%
+13:00 107 5 4.7%
+```
+
+## Q2 — Latency percentiles (p50 / p95 / p99)
+
+Averages hide tail latency; percentiles don't. Ask for fleet-wide percentiles,
+and break out p95 per service with a `terms` + `percentiles` sub-agg to see who
+owns the tail.
+
+```json
+POST /logs-app/_search
+{
+ "size": 0,
+ "aggs": {
+ "latency": { "percentiles": { "field": "latency_ms", "percents": [50, 95, 99] } },
+ "by_service": {
+ "terms": { "field": "service" },
+ "aggs": { "p95": { "percentiles": { "field": "latency_ms", "percents": [95] } } }
+ }
+ }
+}
+```
+
+The fleet-wide block comes back as ES-style keyed percentiles:
+
+```json
+{ "latency": { "values": { "50.0": 56.0, "95.0": 321.15, "99.0": 565.22 } } }
+```
+
+Per-service p95 makes the culprits obvious — `checkout` and `payments` carry
+the tail while `catalog` and `search` stay flat:
+
+```
+fleet-wide p50=56 p95=321 p99=565
+service p95 latency
+catalog 51
+search 79
+checkout 447
+payments 529
+```
+
+## Q3 — Top services by volume (with latency stats)
+
+One `terms` agg ranks services by event count; a `stats` sub-agg attaches
+count/avg/min/max latency to each in the same round trip — leaderboard and
+latency profile from a single query.
+
+```json
+POST /logs-app/_search
+{
+ "size": 0,
+ "aggs": {
+ "top_services": {
+ "terms": { "field": "service", "size": 10, "order": { "_count": "desc" } },
+ "aggs": { "lat": { "stats": { "field": "latency_ms" } } }
+ }
+ }
+}
+```
+
+```
+service volume avg_ms min max
+catalog 190 35 15 118
+search 187 51 22 114
+checkout 150 167 60 590
+payments 73 269 103 958
+```
+
+`catalog` and `search` take the most traffic but are cheap; `payments` is
+low-volume but has the worst worst-case (958 ms). That's the whole triage
+picture from three `_search` calls.
+
+## Run it
+
+```bash
+# 1. Boot XERJ (insecure/local) on a port of your choice
+printf '[server]\nes_compat_port = 9483\n' > /tmp/xerj.toml
+./engine/target/release/xerj --insecure --data-dir /tmp/xerj-logs --config /tmp/xerj.toml &
+
+# 2. Run the recipe (stdlib only — no pip installs)
+python3 docs/examples/log-analytics/log_analytics.py http://localhost:9483
+```
+
+The script creates the mapping, bulk-ingests 600 events, runs all three
+queries, and prints the tables above. It asserts the ingest count and that
+`_bulk` reported no errors, so a green run means the aggregations are real.
+
+## Notes and honest limits
+
+- These aggregations are computed on demand per query — XERJ is not running a
+ continuous rollup or a background job. That's exactly what you want for
+ ad-hoc dashboards and alerting queries; if you need pre-materialized rollups
+ at very high cardinality, aggregate on a schedule and index the results.
+- `fixed_interval: "1h"` gives evenly spaced buckets; use `calendar_interval`
+ (`"day"`, `"month"`) when you want calendar-aware boundaries and DST handling.
+- Percentiles use an approximate (t-digest-style) algorithm, matching ES
+ semantics — the values are estimates, not exact order statistics, which is
+ why p95 reads as `321.15` rather than a whole millisecond.
+- Same documents, same engine: because `message` is a `text` field you can add
+ `match`/`match_phrase` full-text filters to any of these queries (e.g. only
+ aggregate logs matching `"timeout"`) without moving data anywhere.
diff --git a/docs/recipes/migrate-from-elasticsearch.md b/docs/recipes/migrate-from-elasticsearch.md
new file mode 100644
index 000000000..1b14f0bbf
--- /dev/null
+++ b/docs/recipes/migrate-from-elasticsearch.md
@@ -0,0 +1,209 @@
+# Migrate from Elasticsearch: change the URL, not your app
+
+**Use case:** You have a service that talks to Elasticsearch — it creates indices with mappings, bulk-loads documents, runs `match`/`term`/`range`/`bool` searches, computes `terms` aggregations, and checks `_cat/indices` in ops dashboards. You want to move to XERJ without rewriting any of that.
+
+**The whole migration is one line:** point your client's base URL at XERJ. XERJ speaks the Elasticsearch REST wire protocol on the same `:9200`, so the exact JSON your app already sends works unchanged. No SDK swap, no query rewrites, no re-mapping.
+
+```diff
+- ES_URL = "http://elasticsearch:9200"
++ ES_URL = "http://xerj:9200"
+```
+
+This recipe proves it by running the standard ES request shapes — verbatim — against a live XERJ and showing the real responses.
+
+## Why XERJ for this
+
+- **Same wire, same client.** The official `elasticsearch-py`, `@elastic/elasticsearch`, or a raw `curl` all work because XERJ implements the ES REST API surface. XERJ even reports itself as version `8.13.0` at `GET /`, so version-sniffing clients connect cleanly.
+- **Faster on the same workload.** On an identical corpus and machine, XERJ beats a real Elasticsearch 8.13 on both ingest and reads — e.g. bulk ingest ~1.5–1.7× and `match` queries ~2.5× lower p50 latency (see [Benchmarks](#benchmarks)).
+- **Honest scope.** XERJ is *wire-compatible*, not a byte-for-byte fork. As of the last full run it passes **1,325 / 1,326** executed cases of the 1,329-case ES REST conformance suite (3 skipped, 1 known fail). Response *shapes* match; some internal details (exact BM25 float scores, segment-merge timing) are XERJ's own. See [Limitations](#limitations-be-honest).
+
+## The working solution
+
+Everything below is a plain `curl` in the exact shape you'd send Elasticsearch. The only variable is `$ES`.
+
+### 1. Create an index with a mapping — `PUT /products`
+
+```bash
+curl -s -X PUT "$ES/products" -H 'Content-Type: application/json' -d '{
+ "mappings": {
+ "properties": {
+ "name": { "type": "text" },
+ "brand": { "type": "keyword" },
+ "price": { "type": "float" },
+ "in_stock": { "type": "boolean" }
+ }
+ }
+}'
+```
+
+Response — the standard ES acknowledgement:
+
+```json
+{ "acknowledged": true, "shards_acknowledged": true, "index": "products" }
+```
+
+### 2. Bulk-load documents — `POST /_bulk`
+
+Standard NDJSON: an action line, then a source line, repeated. `Content-Type: application/x-ndjson`, trailing newline.
+
+```bash
+curl -s -X POST "$ES/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary '
+{"index":{"_index":"products","_id":"1"}}
+{"name":"Aluminum water bottle","brand":"Klean","price":24.99,"in_stock":true}
+{"index":{"_index":"products","_id":"2"}}
+{"name":"Insulated steel water bottle","brand":"Hydro","price":39.95,"in_stock":true}
+... 4 more ...
+'
+```
+
+Response is the ES bulk envelope (`{"took":..,"errors":false,"items":[...]}`) — here, `errors: false, items: 6`.
+
+> As in Elasticsearch, freshly indexed docs become searchable after a refresh. Call `POST /products/_refresh` (or wait for the interval) before the search steps.
+
+### 3. Full-text `match` — `POST /products/_search`
+
+```bash
+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{
+ "query": { "match": { "name": "water bottle" } },
+ "_source": ["name","brand","price"]
+}'
+```
+
+Real result — BM25-ranked, standard `hits.total.value` / `_score` shape:
+
+```
+total: 4
+ 0.575 Aluminum water bottle
+ 0.575 Insulated steel water bottle
+ 0.288 Glass water carafe # matched "water"
+ 0.288 Plastic sports bottle # matched "bottle"
+```
+
+Note the classic OR semantics of `match`: "Glass water carafe" and "Plastic sports bottle" each match one term and rank below the docs matching both. The full hit shape is exactly ES's:
+
+```json
+{
+ "hits": {
+ "total": { "value": 4, "relation": "eq" },
+ "max_score": 1.4168,
+ "hits": [
+ { "_index": "products", "_id": "1", "_score": 1.4168,
+ "_source": { "name": "Aluminum water bottle", "brand": "Klean", "price": 24.99 } }
+ ]
+ }
+}
+```
+
+### 4. The classic `bool` combo — `must` + `terms` filter + `range` filter
+
+The bread-and-butter Elasticsearch query: full-text relevance in `must`, non-scoring constraints in `filter`.
+
+```bash
+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{
+ "query": {
+ "bool": {
+ "must": [ { "match": { "name": "bottle" } } ],
+ "filter": [
+ { "terms": { "brand": ["Hydro","Klean"] } },
+ { "range": { "price": { "gte": 10, "lte": 40 } } }
+ ]
+ }
+ },
+ "_source": ["name","brand","price"]
+}'
+```
+
+Real result — only the docs that match the text **and** pass both filters:
+
+```
+total: 2
+ Klean 24.99 Aluminum water bottle
+ Hydro 39.95 Insulated steel water bottle
+```
+
+(The Klean "Plastic sports bottle" is excluded — its price `9.99` is below the `10` floor.)
+
+### 5. `terms` aggregation — `POST /products/_search` with `size: 0`
+
+```bash
+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{
+ "size": 0,
+ "aggs": { "by_brand": { "terms": { "field": "brand" } } }
+}'
+```
+
+Real result — the standard `buckets` shape with `doc_count_error_upper_bound` and `sum_other_doc_count`:
+
+```json
+{
+ "hits": { "total": { "value": 6, "relation": "eq" }, "hits": [] },
+ "aggregations": {
+ "by_brand": {
+ "doc_count_error_upper_bound": 0,
+ "sum_other_doc_count": 0,
+ "buckets": [
+ { "key": "Bodum", "doc_count": 2 },
+ { "key": "Hydro", "doc_count": 2 },
+ { "key": "Klean", "doc_count": 2 }
+ ]
+ }
+ }
+}
+```
+
+### 6. Ops check — `GET /_cat/indices?v`
+
+The human-readable `_cat` table your dashboards and shell scripts parse:
+
+```
+health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
+green open products f5c15d1... 1 0 6 0 10901b 10901b
+```
+
+`_cat/indices` also lists XERJ's internal `.xerj_*` system indices (sessions, users, dashboards, etc.), analogous to Elasticsearch's own `.security`/`.kibana` dot-indices. Filter them the same way you would in ES: `GET /_cat/indices/products?v` or a `products*` pattern.
+
+## Run it
+
+The full script is [`docs/examples/migrate-from-elasticsearch/migrate_demo.sh`](../examples/migrate-from-elasticsearch/migrate_demo.sh). Boot XERJ and run every step above end-to-end:
+
+```bash
+# 1. Boot XERJ (insecure = no TLS/auth, for local demo)
+xerj --insecure --data-dir /tmp/xerj-demo &
+
+# 2. Run the standard-ES demo against it — the ONLY change is the URL
+ES=http://localhost:9200 ./migrate_demo.sh
+```
+
+It prints each request's result and ends with `All standard Elasticsearch calls succeeded against XERJ`. To A/B against a real cluster, run the identical script with `ES=http://your-elasticsearch:9200` — same script, same output shape.
+
+## What to change in your real app
+
+Almost nothing:
+
+1. **Base URL** → your XERJ host. That's the migration.
+2. **Auth/TLS** → XERJ supports them; the demo uses `--insecure` for brevity. Point your client's existing credentials/CA config at XERJ as you would any ES endpoint.
+3. **Client library** → keep it. `elasticsearch-py`, the JS client, Logstash/Beats outputs, or raw HTTP all speak this wire.
+
+## Limitations (be honest)
+
+- **Wire-compatible, not byte-identical.** XERJ passes **1,325 / 1,326** executed cases of the 1,329-case ES REST conformance suite (3 skipped, 1 known fail), last full run. Response *structure* matches ES; validate the specific endpoints and query types your app depends on against your own data before cutting over. Per-suite conformance and caveats are tracked in `demo/playbooks/ES_COMPATIBILITY.md`.
+- **Relevance scores are XERJ's own BM25.** Ranking order matches ES intent, but exact `_score` float values differ, and can even shift slightly for the *same* query as background segment merges change collection statistics — the same class of internal difference you'd see comparing two ES minor versions. If you assert on exact scores in tests, relax those assertions.
+- **A few query/agg types differ or aren't implemented.** XERJ covers the common set (`match`, `term`, `terms`, `range`, `bool`, `match_phrase`, `multi_match`, `terms`/`stats`/`date_histogram`/... aggs). Some ES features are partial or unsupported (e.g. vector-suite conformance is low). Check the compatibility report for your specific needs.
+
+## Benchmarks
+
+Same corpus, same machine, single node, security off — XERJ vs. a real Elasticsearch 8.13:
+
+| Operation | XERJ | ES 8.13 | XERJ advantage |
+|---|---|---|---|
+| ingest 1M docs, 1 client (docs/s) | 119,031 | 70,320 | **1.69×** |
+| ingest 100k docs, 8 clients (docs/s) | 382,301 | 253,961 | **1.51×** |
+| `match(model)` read (p50 ms) | 0.57 | 1.45 | **2.53×** |
+| `match_all` read (p50 ms) | 0.88 | 1.72 | **1.94×** |
+| `query_string` read (p50 ms) | 1.35 | 9.35 | **6.92×** |
+
+Full head-to-head matrix and methodology (identical keep-alive HTTP client applied to both engines): `demo/playbooks/BENCHMARK_VS_ES.md`.
+
+---
+
+*Verified end-to-end against a live XERJ (merged `main` binary, ES-wire port). Every request/response above is real captured output.*
diff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.md
new file mode 100644
index 000000000..d3a0c3e41
--- /dev/null
+++ b/docs/recipes/semantic-search-rag.md
@@ -0,0 +1,188 @@
+# Semantic search / RAG retrieval with XERJ — no separate vector DB
+
+**Goal:** you have a knowledge base (support docs, product FAQs, policy pages) and
+you want an LLM to answer questions *grounded* in it. The retrieval half of RAG —
+"find the passages that mean the same thing as the user's question" — normally
+means standing up a separate vector database, an embedding service, and glue code
+to keep them in sync with your search index.
+
+With XERJ you don't. You map one field as `semantic_text`, ingest your docs with a
+normal `_bulk`, and XERJ **embeds each document on ingest** with a built-in,
+zero-config embedder. Then a `semantic` query retrieves by meaning from the same
+index you already full-text search. One engine, one wire protocol (Elasticsearch
+8.x), no extra moving parts.
+
+This recipe indexes a tiny KB, retrieves passages with paraphrased questions
+(different words, same meaning), assembles the context bundle you'd hand an LLM,
+and is honest about where the built-in lexical embedder stops and a real neural
+endpoint takes over.
+
+---
+
+## 1. Map one field as `semantic_text`
+
+```bash
+curl -X PUT localhost:9482/kb -H 'Content-Type: application/json' -d '{
+ "mappings": {
+ "properties": {
+ "title": { "type": "text" },
+ "body": { "type": "semantic_text" }
+ }
+ }
+}'
+```
+
+`body` is now auto-embedded. On ingest XERJ vectorizes the field's text into a
+companion HNSW-indexed vector field (`body_vector`, 384 dims by default, cosine
+similarity) — you never call an embedding API yourself. No `dense_vector`
+declaration, no dimension bookkeeping, no second datastore.
+
+## 2. Ingest with a plain `_bulk`
+
+Nothing special — the embedding happens under the hood:
+
+```bash
+curl -X POST localhost:9482/_bulk -H 'Content-Type: application/x-ndjson' --data-binary '
+{"index":{"_index":"kb","_id":"kb-1"}}
+{"title":"Rotating an API key","body":"To change your API credentials, open Settings, choose Security, and click Regenerate token. The old token stops working immediately."}
+{"index":{"_index":"kb","_id":"kb-2"}}
+{"title":"Refund policy","body":"Customers can request their money back within 30 days of purchase. Refunds are issued to the original payment method within five business days."}
+'
+```
+
+## 3. Retrieve by meaning with the `semantic` query
+
+The user's question rarely uses your doc's exact words. A `match` query on
+`"get my money back after buying"` returns **nothing** — none of those tokens are
+in the "Refund policy" doc analyzed the same way. The `semantic` query retrieves
+it anyway:
+
+```bash
+curl localhost:9482/kb/_search -H 'Content-Type: application/json' -d '{
+ "size": 2,
+ "query": { "semantic": { "field": "body", "query": "how do I regenerate my API token", "k": 2 } }
+}'
+```
+
+Real response (trimmed):
+
+```json
+{
+ "hits": {
+ "hits": [
+ {
+ "_id": "kb-1",
+ "_score": 0.6625697,
+ "_source": {
+ "title": "Rotating an API key",
+ "body": "To change your API credentials, open Settings, choose Security, and click Regenerate token. The old token stops working immediately.",
+ "body_vector": [ 0.0, 0.0, -0.2164437, ... ]
+ }
+ }
+ ]
+ }
+}
+```
+
+`field` is the `semantic_text` field, `query` is the natural-language question,
+`k` is how many nearest passages to pull from the vector index. Add a `filter`
+(any XERJ query) to constrain candidates, or a `boost` to weight it inside a
+`bool`.
+
+> **`_source` note.** The companion `body_vector` rides along in `_source` on
+> semantic hits (source filtering is applied on the lexical path but the vector
+> field is re-attached on the retrieval path). It's just noise for RAG — read the
+> fields you want (`title`, `body`) and ignore it when you build the LLM prompt.
+
+## 4. The RAG context bundle
+
+Retrieval done, RAG is just "concatenate the top passages into the prompt." Pull
+`k` hits and format them with their ids so the model can cite them:
+
+```
+User question: how do I regenerate my API token
+
+Retrieved context:
+[kb-1] Rotating an API key
+To change your API credentials, open Settings, choose Security, and click
+Regenerate token. The old token stops working immediately.
+
+[kb-5] Rate limits
+The API permits 600 requests per minute per key. Exceeding the throttle returns
+HTTP 429; back off and retry after the reset window.
+```
+
+Feed that block to your LLM as grounding context alongside the question. That's
+the whole retrieval side of RAG — no vector DB in the diagram.
+
+## 5. Results — paraphrases land on the right doc
+
+Running five paraphrased questions (none share the doc's full wording) against
+the 5-doc KB, each returns the correct passage at rank 1:
+
+```
+Q: 'how do I regenerate my API token' -> kb-1 Rotating an API key OK
+Q: 'can I get my money back after buying' -> kb-2 Refund policy OK
+Q: "I forgot my login and can't sign in" -> kb-3 Resetting a forgotten password OK
+Q: 'which file formats are supported for uploads' -> kb-4 Supported file formats OK
+Q: 'how many calls per minute am I allowed' -> kb-5 Rate limits OK
+
+RESULT: 5/5 semantic retrievals returned the right doc at rank 1
+```
+
+For contrast, the same paraphrase through a keyword `match`:
+
+```
+Q: 'get my money back after buying'
+ match top -> NO HITS
+ semantic top -> kb-2 Refund policy
+```
+
+## Honest limitation: the built-in embedder is *lexical*, not neural
+
+The zero-config embedder is a **lexical** model — it maps overlapping and related
+vocabulary into vector space. It shines when the question and the answer share
+*some* terms (a synonym here, a rephrase there), which covers a lot of real
+support/FAQ traffic. It is **not** a neural sentence embedder and won't capture
+deep semantics with no lexical bridge.
+
+Concretely, from this same KB:
+
+```
+Q: 'how do I rotate my API key'
+ semantic top -> kb-5 Rate limits (expected kb-1)
+```
+
+The doc is *titled* "Rotating an API key," but only its **body** is embedded, and
+the body never uses the word "rotate" — so a pure "rotate" paraphrase misses. Two
+honest takeaways:
+
+- **Embed the text that carries the meaning.** If titles matter, index them into
+ the `semantic_text` field too (or add a second `semantic_text` field and combine
+ with `hybrid`), so the vocabulary the user might use is actually in the vector.
+- **For production-grade semantics, wire an external embeddings endpoint.** XERJ
+ will call any OpenAI-compatible `/v1/embeddings` service. Set
+ `embedding.default_endpoint` in config, or point a field at it in the mapping
+ via `inference_endpoint` / `inference_id`. The `semantic` query and the ingest
+ path are identical — you just get neural-quality vectors instead of the built-in
+ lexical ones. (Anomaly-style continuous re-embedding isn't a thing here;
+ embedding happens at ingest time.)
+
+Everything above is `semantic_text` + the built-in embedder — no external service
+was configured for this recipe.
+
+---
+
+## Run it
+
+Boot XERJ (Elasticsearch-compatible port shown as 9482 here), then:
+
+```bash
+BASE=http://localhost:9482 python3 docs/examples/semantic-search-rag/rag_demo.py
+```
+
+The script (stdlib only — `urllib` + `json`, no pip) creates the index, bulk-loads
+the KB, runs the five semantic retrievals with assertions, prints the RAG context
+bundle, shows the keyword-vs-semantic contrast, demonstrates the lexical-embedder
+miss, and exits non-zero if any of the five retrievals regress. Point `BASE` at any
+XERJ (or real Elasticsearch with a `semantic_text`/inference setup) to compare.
diff --git a/docs/recipes/vector-search-knn.md b/docs/recipes/vector-search-knn.md
new file mode 100644
index 000000000..bcb7c6523
--- /dev/null
+++ b/docs/recipes/vector-search-knn.md
@@ -0,0 +1,251 @@
+# Vector similarity search (kNN) with XERJ
+
+**Goal:** you have a set of items — products, articles, images, support
+tickets — and each one is represented by an *embedding*: a fixed-length
+list of floats. You want the classic operation behind recommendations,
+"more like this", dedup, and semantic retrieval: **given one vector, find
+the N most similar items.**
+
+XERJ answers this natively. It stores a `dense_vector` field, builds an
+HNSW graph over the vectors, and serves the Elasticsearch-8 top-level
+`knn` query — so the exact same request bodies (and ES client libraries)
+work unchanged. No sidecar vector database, no separate service.
+
+This recipe uses **tiny, hand-made 4-dimensional vectors** so every result
+is reproducible and you can see *why* each neighbor came back. In
+production your vectors come from a real embedding model (768, 1024,
+1536 dims…) — the API is identical.
+
+---
+
+## The toy feature space
+
+Real embeddings are opaque. To make kNN legible, we hand-build vectors in
+a 4-axis "feature space":
+
+```
+[ citrus , berry , engine , wheels ]
+```
+
+Fruits load on the fruit axes and are ~0 on the vehicle axes; vehicles do
+the reverse. So a query sitting in the *citrus* corner should return
+oranges and lemons first and **never** a truck. That's our recall check.
+
+| id | name | category | in_stock | embedding |
+|----|----------------|----------|----------|----------------------------|
+| 1 | Navel Orange | fruit | true | `[0.90, 0.10, 0.00, 0.00]` |
+| 2 | Meyer Lemon | fruit | true | `[0.95, 0.05, 0.00, 0.00]` |
+| 3 | Blood Orange | fruit | **false**| `[0.85, 0.20, 0.00, 0.00]` |
+| 4 | Strawberry | fruit | true | `[0.10, 0.92, 0.00, 0.00]` |
+| 5 | Blueberry | fruit | true | `[0.05, 0.95, 0.00, 0.00]` |
+| 6 | Pickup Truck | vehicle | true | `[0.00, 0.00, 0.90, 0.85]` |
+| 7 | Sports Car | vehicle | false | `[0.00, 0.00, 0.95, 0.80]` |
+| 8 | Electric Sedan | vehicle | true | `[0.00, 0.05, 0.40, 0.88]` |
+
+---
+
+## 1. Map a `dense_vector` field
+
+```bash
+curl -X PUT http://localhost:9200/catalog -H 'Content-Type: application/json' -d '{
+ "mappings": {
+ "properties": {
+ "name": { "type": "text" },
+ "category": { "type": "keyword" },
+ "in_stock": { "type": "boolean" },
+ "embedding": { "type": "dense_vector", "dims": 4, "similarity": "cosine" }
+ }
+ }
+}'
+```
+
+Two things matter:
+
+- **`dims`** must equal the length of every vector you index. Indexing a
+ wrong-length vector is silently skipped from the ANN scan.
+- **`similarity`** defaults to `cosine` (magnitude-invariant — the right
+ default for most normalized embeddings). Other options: `l2`,
+ `dot_product`. Scores come back as ES-normalized cosine in `[0, 1]`:
+ identical direction ≈ `1.0`, orthogonal ≈ `0.5`.
+
+Load the rows with the Bulk API (`{"index":{...}}` + source line pairs),
+adding `?refresh=true` so they're searchable immediately.
+
+---
+
+## 2. Plain kNN — "find items most like a citrus fruit"
+
+Put the query vector right in the citrus corner and ask for the 3 nearest:
+
+```bash
+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
+ "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 }
+}'
+```
+
+Real response (trimmed):
+
+```json
+{
+ "hits": { "hits": [
+ { "_id": "1", "_score": 0.9998569, "_source": { "name": "Navel Orange", "category": "fruit" } },
+ { "_id": "2", "_score": 0.9997083, "_source": { "name": "Meyer Lemon", "category": "fruit" } },
+ { "_id": "3", "_score": 0.9948, "_source": { "name": "Blood Orange", "category": "fruit" } }
+ ] }
+}
+```
+
+The three nearest are **exactly** the three citrus fruits, in order of
+closeness, and no vehicle leaked in. `k` is how many neighbors you want
+back; `num_candidates` is how many the graph search explores before
+picking the top `k` (bigger = higher recall, more work).
+
+Query the *other* corner and the space cleanly separates:
+
+```bash
+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
+ "knn": { "field": "embedding", "query_vector": [0.0, 0.0, 0.9, 0.85], "k": 2, "num_candidates": 10 }
+}'
+# → Pickup Truck (1.0), Sports Car (0.999) — only vehicles
+```
+
+---
+
+## 3. kNN + a filter — "…but only what's in stock"
+
+This is the request people actually get wrong. To constrain a kNN search
+(price range, in-stock, tenant, language…), **wrap the `knn` in a `bool`
+and put the constraint in a sibling `filter` clause.** `bool.filter` runs
+as a **pre-filter**: XERJ narrows the candidate set first, *then* ranks
+the survivors by similarity.
+
+```bash
+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
+ "size": 3,
+ "query": {
+ "bool": {
+ "must": [ { "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 } } ],
+ "filter": [ { "term": { "in_stock": true } } ]
+ }
+ }
+}'
+```
+
+Real response (trimmed):
+
+```
+1 0.9999 Navel Orange in_stock=True
+2 0.9997 Meyer Lemon in_stock=True
+4 0.5969 Strawberry in_stock=True
+```
+
+Blood Orange (id 3) is the 3rd-closest citrus but is **out of stock**, so
+it drops out and a farther *in-stock* item (a berry) takes the slot. The
+filter genuinely restricted the pool — it didn't just re-score.
+
+> **Heads-up on the ES-8 shorthand.** Elasticsearch also lets you inline
+> the filter *inside* the knn clause: `"knn": { …, "filter": {…} }`. XERJ
+> parses that form but **currently ignores the inlined filter** on the
+> top-level-`knn` path (it returns unfiltered neighbors). Until that's
+> wired up, use the `bool` + `filter` wrapper shown above — it's the
+> reliable way to get pre-filtered kNN today, and it's also how XERJ's own
+> agent-memory recall builds filtered vector queries internally.
+
+---
+
+## Run it
+
+Boot XERJ (`target/release/xerj --insecure --data-dir ./data`, default
+port 9200) and run the self-checking script — stdlib only, no pip:
+
+```bash
+python3 docs/examples/vector-search-knn/knn_demo.py
+```
+
+It creates the index, bulk-loads the catalog, runs all three searches,
+and asserts the recall and filter behavior. Expected tail:
+
+```
+=== kNN k=3, query ~citrus ===
+ ('1', 0.9999, 'Navel Orange', 'fruit', True)
+ ('2', 0.9997, 'Meyer Lemon', 'fruit', True)
+ ('3', 0.9948, 'Blood Orange', 'fruit', False)
+ OK: top-3 are exactly the citrus fruits, no vehicles leaked in
+
+=== kNN + bool.filter in_stock:true ===
+ ('1', 0.9999, 'Navel Orange', 'fruit', True)
+ ('2', 0.9997, 'Meyer Lemon', 'fruit', True)
+ ('4', 0.5969, 'Strawberry', 'fruit', True)
+ OK: out-of-stock Blood Orange excluded; only in-stock neighbors returned
+
+=== kNN k=2, query ~vehicle ===
+ ('6', 1.0, 'Pickup Truck', 'vehicle', True)
+ ('7', 0.9992, 'Sports Car', 'vehicle', False)
+ OK: vehicle query returns only vehicles
+
+ALL ASSERTIONS PASSED
+```
+
+(The script targets port 9484; change `BASE` if you booted on 9200.)
+
+---
+
+## How the search works: HNSW
+
+XERJ indexes vectors in an **HNSW** graph (Hierarchical Navigable Small
+World) — the same family of ANN index Elasticsearch/Lucene use. Instead
+of comparing your query against every stored vector (O(N), fine for our
+8-row demo, painful at millions), HNSW builds a layered proximity graph
+and greedily hops toward the nearest region, giving roughly O(log N)
+search that stays fast as the corpus grows. XERJ builds with
+`ef_construction = 200` for high recall; at query time `num_candidates`
+controls the search-time breadth (the ANN analog of `ef_search`) — raise
+it to trade latency for recall, lower it for speed.
+
+Because it's *approximate*, ANN can in principle miss a true neighbor.
+For a corpus this small the result is exact; the knobs above only start
+to matter at scale.
+
+---
+
+## Raw kNN vs. `semantic_text`: which do I use?
+
+Both end up doing vector search. The difference is **who makes the
+vectors.**
+
+- **Raw `dense_vector` + `knn` (this recipe)** — *you* supply the
+ embeddings, on both the ingest side and the query side. Use it when you
+ already run an embedding model (OpenAI, a local sentence-transformer,
+ a CLIP image encoder, your own recommender) and just need fast storage
+ and nearest-neighbor retrieval. Maximum control; works for any modality,
+ not just text.
+
+- **`semantic_text` field + `semantic` query** — XERJ embeds text
+ **for** you at ingest time and at query time, so you send plain strings
+ and never touch a vector. Great when your data is text and you want
+ semantic search with zero embedding plumbing.
+ *Honest caveat:* the built-in embedder is **lexical, not neural** —
+ excellent for demos and keyword-semantic overlap, but for
+ production-grade semantic quality point XERJ at an external
+ OpenAI-compatible `/v1/embeddings` endpoint
+ (`embedding.default_endpoint`), after which both ingest and query use
+ that higher-quality model.
+
+Rule of thumb: **already have embeddings → raw `knn`. Have text and want
+it embedded for you → `semantic_text`.** Need to blend keyword relevance
+(BM25) with vector similarity in one request? Reach for the `hybrid`
+query.
+
+---
+
+## Gotchas
+
+- **`dims` must match your vectors exactly** — a length mismatch means the
+ doc is skipped from the scan, not an error.
+- **Filter a kNN with `bool` + `filter`**, not the inlined `knn.filter`
+ (currently ignored on the top-level path — see §3).
+- **`k` vs `num_candidates`**: `k` = results returned; `num_candidates` =
+ how many the graph explores first. `num_candidates >= k`; raise it if
+ recall matters.
+- Scores are **ES-normalized cosine in `[0,1]`** (≈1.0 identical, ≈0.5
+ orthogonal), not raw cosine in `[-1,1]`.