AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: use-case recipes, each verified end-to-end against a live XERJ
xerj-org2d173ed2c115
diff --git a/.gitignore b/.gitignoreindex 42372c8a7..083b99d6b 100644--- a/.gitignore+++ b/.gitignore@@ -96,3 +96,4 @@ landing/resources/xerj-sales-kit.zip# Re-include the public architecture doc (Markdown is ignored by default above).!docs/ARCHITECTURE.md!ROADMAP.md+!docs/**/*.mddiff --git a/docs/examples/agentic-memory/agent_memory.py b/docs/examples/agentic-memory/agent_memory.pynew file mode 100644index 000000000..1d404b78b--- /dev/null+++ b/docs/examples/agentic-memory/agent_memory.py@@ -0,0 +1,182 @@+#!/usr/bin/env python3+"""Long-term memory for an AI agent, backed by XERJ's /_memory API.++Simulates a coding-assistant agent that, across a session:+ 1. STORES observations/preferences it learns about the project+ (each as text + a small embedding vector + metadata),+ 2. later RECALLS the most relevant memories to answer a new question+ -- by vector similarity AND by plain text,+ 3. does a metadata-filtered recall (only "preference" memories),+ 4. FORGETS a memory that is no longer true,+ 5. proves multi-namespace isolation: a second agent cannot see the+ first agent's memories.++No LLM and no pip installs -- stdlib only. The "embedder" here is a tiny+deterministic hashed-character-trigram vectorizer so the demo is fully+self-contained and cosine similarity is meaningful (it is lexical, not+neural -- see the note by embed()). In production you bring your own vectors+(OpenAI/Cohere/your model) -- XERJ stores and kNN-searches them the same way,+which is exactly what lets XERJ replace a bolt-on vector DB for agent memory.+"""++import json+import math+import re+import urllib.request+from urllib.error import HTTPError++BASE = "http://localhost:9481"+DIM = 96 # embedding dimensionality for the toy hashing embedder+++# --------------------------------------------------------------------------- #+# Tiny self-contained "embedder": deterministic hashed character trigrams.+# Character n-grams give the demo enough lexical-semantic overlap that related+# wording (service/services, prefer/prefers) lands nearby -- WITHOUT any model+# download. It is NOT a neural embedder: real synonymy ("car" ~ "automobile")+# needs real embeddings. In production you pass vectors from your own model+# (OpenAI/Cohere/local); XERJ stores and kNN-searches them exactly the same,+# which is what lets it stand in for a bolt-on vector DB.+# --------------------------------------------------------------------------- #+def embed(text: str) -> list[float]:+ s = " " + re.sub(r"[^a-z0-9]+", " ", text.lower()) + " "+ vec = [0.0] * DIM+ for i in range(len(s) - 2):+ h = 0+ for ch in s[i:i + 3]: # hash each character trigram into a bucket+ h = (h * 131 + ord(ch)) & 0xFFFFFFFF+ vec[h % DIM] += 1.0+ norm = math.sqrt(sum(v * v for v in vec)) or 1.0+ return [round(v / norm, 6) for v in vec]+++# --------------------------------------------------------------------------- #+# Thin HTTP helpers over XERJ's REST surface.+# --------------------------------------------------------------------------- #+def _req(method: str, path: str, body: dict | None = None) -> dict:+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(+ BASE + path, data=data, method=method,+ headers={"Content-Type": "application/json"},+ )+ try:+ with urllib.request.urlopen(req) as r:+ return json.loads(r.read() or b"{}")+ except HTTPError as e:+ return json.loads(e.read() or b"{}")+++def remember(ns: str, text: str, metadata: dict, mem_id: str) -> dict:+ """Store one memory: text is BM25-indexed, vector enables kNN recall."""+ return _req("POST", f"/_memory/{ns}", {+ "text": text, "vector": embed(text), "metadata": metadata, "id": mem_id,+ })+++def recall_vec(ns: str, query: str, k: int = 3, filt: dict | None = None) -> list:+ """Semantic recall: embed the query, kNN over stored vectors."""+ body = {"vector": embed(query), "k": k}+ if filt:+ body["filter"] = filt+ return _req("POST", f"/_memory/{ns}/_recall", body)["hits"]+++def recall_text(ns: str, query: str, k: int = 3) -> list:+ """Lexical recall: BM25 over the stored memory text."""+ return _req("POST", f"/_memory/{ns}/_recall", {"query": query, "k": k})["hits"]+++def forget(ns: str, mem_id: str) -> dict:+ return _req("DELETE", f"/_memory/{ns}/{mem_id}")+++def drop(ns: str) -> dict:+ return _req("DELETE", f"/_memory/{ns}")+++def show(label: str, hits: list) -> None:+ print(f"\n{label}")+ for h in hits:+ kind = (h.get("metadata") or {}).get("kind", "-")+ print(f" [{h['score']:.3f}] ({kind:10}) {h['text']}")+++# --------------------------------------------------------------------------- #+# The scenario.+# --------------------------------------------------------------------------- #+def main() -> None:+ ADA = "agent-ada" # our coding assistant+ BILLING = "agent-billing" # an unrelated agent, to prove isolation++ # Clean slate for a repeatable demo.+ drop(ADA)+ drop(BILLING)++ # --- Turn 1..N: the agent learns things and writes them to memory. ------ #+ print("== Storing what the agent learned this session ==")+ facts = [+ ("The team prefers Rust over Go for new backend services.",+ {"kind": "preference", "topic": "language"}),+ ("Production runs on AWS in us-east-1; staging is us-west-2.",+ {"kind": "fact", "topic": "infra"}),+ ("The team prefers tabs over spaces and a 100-column line limit.",+ {"kind": "preference", "topic": "style"}),+ ("The payments service owns the Postgres 'ledger' database.",+ {"kind": "fact", "topic": "ownership"}),+ ("Deploys are gated on the full integration suite passing in CI.",+ {"kind": "fact", "topic": "process"}),+ ("The team prefers OpenTelemetry for tracing, not vendor SDKs.",+ {"kind": "preference", "topic": "observability"}),+ ]+ for i, (text, md) in enumerate(facts, 1):+ r = remember(ADA, text, md, f"ada-{i}")+ print(f" stored {r['id']:8} created={r['created']}")++ # A different agent stores its own (finance) memory in its own namespace.+ remember(BILLING, "Invoices over $10k require CFO approval.",+ {"kind": "policy"}, "bill-1")++ # --- A later turn: answer a question by recalling relevant memory. ------ #+ q1 = "What programming language should I use for a new backend service?"+ show(f'== Semantic recall for: "{q1}"', recall_vec(ADA, q1, k=3))++ q2 = "Which region is production deployed in?"+ show(f'== Lexical (BM25) recall for: "{q2}"', recall_text(ADA, q2, k=3))++ # --- Metadata-filtered recall: only surface team *preferences*. --------- #+ q3 = "How should I set up the new service?"+ show(f'== Preference-only recall for: "{q3}"',+ recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}}))++ # --- Isolation: agent-billing cannot see agent-ada's memories. ---------- #+ print("\n== Namespace isolation ==")+ leak = recall_vec(BILLING, q1, k=3)+ billing_texts = [h["text"] for h in leak]+ print(f" agent-billing recall for a coding question -> {billing_texts}")+ assert all("Rust" not in t and "AWS" not in t for t in billing_texts), \+ "ISOLATION VIOLATION: agent-billing saw agent-ada's memories"+ print(" OK: agent-billing only ever sees its own namespace.")++ # --- Forgetting: the team switched tracing tools; retract that memory. -- #+ print("\n== Forgetting a memory that is no longer true ==")+ before = recall_vec(ADA, "what do we use for tracing", k=3)+ show(" before forget:", before)+ print(" forget ->", forget(ADA, "ada-6"))+ after = recall_vec(ADA, "what do we use for tracing", k=3)+ show(" after forget:", after)+ assert all(h["id"] != "ada-6" for h in after), "ada-6 was not forgotten"+ print(" OK: the retracted memory is gone.")++ # --- Correctness checks so a green run really means it worked. ---------- #+ top_lang = recall_vec(ADA, q1, k=1)[0]+ assert "Rust" in top_lang["text"], f"expected Rust preference on top, got {top_lang}"+ top_region = recall_text(ADA, q2, k=1)[0]+ assert "us-east-1" in top_region["text"], f"expected region fact on top, got {top_region}"+ prefs = recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}})+ assert prefs and all((h["metadata"] or {}).get("kind") == "preference" for h in prefs), \+ "preference filter leaked non-preferences"+ print("\nAll assertions passed.")+++if __name__ == "__main__":+ main()diff --git a/docs/examples/anomaly-detection/anomaly_detection.py b/docs/examples/anomaly-detection/anomaly_detection.pynew file mode 100644index 000000000..3783dd0fa--- /dev/null+++ b/docs/examples/anomaly-detection/anomaly_detection.py@@ -0,0 +1,141 @@+#!/usr/bin/env python3+"""+Anomaly detection on a metric series with XERJ.++Scenario: we collect per-host CPU readings into `cpu_metrics`. For fifteen+minutes CPU sits around ~20%. At minute 12 a runaway process pins the box and+the per-minute mean jumps to ~96%, then recovers. We want XERJ to flag that one+minute as anomalous - automatically, without us hand-picking a threshold on the+raw value.++XERJ's on-demand anomaly detector does exactly this: it buckets the source+index by time, computes the metric per bucket (here mean(cpu)), builds a moving+baseline from prior *normal* buckets, and scores each bucket by how many+standard deviations it sits from that baseline.++Run: python3 anomaly_detection.py # assumes XERJ on :9486+ XERJ=http://localhost:9200 python3 anomaly_detection.py++Uses only the Python 3 standard library. No pip installs.+"""+import json+import os+import urllib.request++XERJ = os.environ.get("XERJ", "http://localhost:9486").rstrip("/")+INDEX = "cpu_metrics"+JOB = "cpu-spike"+++def req(method, path, body=None, ndjson=False):+ data = None+ headers = {}+ if ndjson:+ data = body.encode()+ headers["Content-Type"] = "application/x-ndjson"+ elif body is not None:+ data = json.dumps(body).encode()+ headers["Content-Type"] = "application/json"+ r = urllib.request.Request(XERJ + path, data=data, method=method, headers=headers)+ with urllib.request.urlopen(r) as resp:+ return json.loads(resp.read().decode())+++# 1. Map the index: @timestamp as a date, cpu as a numeric we can average.+try:+ req("DELETE", "/" + INDEX)+except Exception:+ pass+req("PUT", "/" + INDEX, {+ "mappings": {+ "properties": {+ "@timestamp": {"type": "date"},+ "host": {"type": "keyword"},+ "cpu": {"type": "double"},+ }+ }+})++# 2. Build 16 one-minute buckets of 4 host readings each.+# Minutes 0-11 and 13-15 are normal (~20%); minute 12 is the injected spike.+# Readings per minute are chosen so the per-minute mean is exact & stable.+NORMAL = [19, 20, 20, 21] # mean 20.0+JITTER = { # a little bucket-to-bucket variance+ 0: 0, 1: 1, 2: -1, 3: 0, 4: 2, 5: -2, 6: 1, 7: 0,+ 8: -1, 9: 1, 10: 0, 11: 2, 13: 0, 14: 1, 15: -1,+}+SPIKE = [94, 96, 96, 98] # mean 96.0 <-- the anomaly++lines = []+for minute in range(16):+ ts = "2026-07-06T00:%02d:00Z" % minute+ if minute == 12:+ readings = SPIKE+ else:+ readings = [v + JITTER[minute] for v in NORMAL]+ for h, cpu in enumerate(readings):+ lines.append(json.dumps({"index": {"_index": INDEX}}))+ lines.append(json.dumps({"@timestamp": ts, "host": "web-%d" % h, "cpu": cpu}))+bulk = req("POST", "/_bulk", "\n".join(lines) + "\n", ndjson=True)+assert not bulk.get("errors"), bulk+req("POST", "/" + INDEX + "/_refresh")++# 3. Create the detector: mean(cpu) over 1-minute buckets.+job = req("PUT", "/_ml/anomaly_detectors/" + JOB, {+ "source_index": INDEX,+ "time_field": "@timestamp",+ "function": "mean",+ "field": "cpu",+ "bucket_span": "1m",+ "description": "CPU mean per minute",+})+print("== detector created ==")+print(json.dumps({+ "job_id": job["job_id"],+ "function": job["analysis_config"]["detectors"][0]["function"],+ "bucket_span": job["analysis_config"]["bucket_span"],+ "anomaly_threshold": job["anomaly_threshold"],+}, indent=2))++# 4. Score it. Baseline warm-up is the first 4 buckets (no score yet); every+# later bucket is scored against the mean/std of prior *normal* buckets.+res = req("POST", "/_ml/anomaly_detectors/" + JOB + "/_score")++print("\n== per-bucket scores ==")+print("%-20s %8s %9s %8s %7s %5s" %+ ("minute (UTC)", "actual", "expected", "z_score", "score", "flag"))+for r in res["records"]:+ exp = r["expected"]+ z = r["z_score"]+ print("%-20s %8.1f %9s %8s %7.1f %5s" % (+ r["timestamp_iso"][11:19],+ r["actual"],+ "-" if exp is None else "%.1f" % exp,+ "-" if z is None else "%.1f" % z,+ r["anomaly_score"],+ "***" if r["is_anomaly"] else "",+ ))++print("\n== anomalies (%d of %d buckets) ==" % (res["anomaly_count"], res["bucket_count"]))+for a in res["anomalies"]:+ print(" %s actual=%.1f expected=%.1f z=%.1f score=%.1f" % (+ a["timestamp_iso"], a["actual"], a["expected"], a["z_score"], a["anomaly_score"]))++# 5. Verify the result is correct.+spike = next(r for r in res["records"] if r["timestamp_iso"].startswith("2026-07-06T00:12"))+normals = [r for r in res["records"]+ if r["expected"] is not None and not r["timestamp_iso"].startswith("2026-07-06T00:12")]++assert res["anomaly_count"] == 1, "expected exactly one anomaly, got %d" % res["anomaly_count"]+assert spike["is_anomaly"], "spike bucket was not flagged"+assert spike["anomaly_score"] >= 99.0, "spike score too low: %s" % spike["anomaly_score"]+assert abs(spike["actual"] - 96.0) < 1e-6, "spike mean wrong: %s" % spike["actual"]+# is_anomaly is the gate: only the spike crosses the threshold; jitter buckets+# never do, even if their normalized score creeps up near the threshold line.+assert all(not r["is_anomaly"] for r in normals), "a normal bucket was flagged"+max_normal = max(r["anomaly_score"] for r in normals)+assert spike["anomaly_score"] >= 2 * max_normal, \+ "spike (%.1f) not clearly separated from top normal (%.1f)" % (+ spike["anomaly_score"], max_normal)+print("\nOK: spike at 00:12 flagged (score %.1f); %d normal buckets, none flagged"+ " (top normal score %.1f)." % (spike["anomaly_score"], len(normals), max_normal))diff --git a/docs/examples/hybrid-search/hybrid_search.py b/docs/examples/hybrid-search/hybrid_search.pynew file mode 100644index 000000000..edac4d6e8--- /dev/null+++ b/docs/examples/hybrid-search/hybrid_search.py@@ -0,0 +1,145 @@+#!/usr/bin/env python3+"""+Hybrid search on XERJ — keyword (BM25) + vector (kNN) fused in ONE query.++Scenario: a help-desk search box. A user types "reset password".+We show that:+ * pure BM25 misses a highly relevant article that shares no keywords+ with the query ("Regain entry to a locked-out account"),+ * pure kNN (top-k) misses a keyword-exact article whose topic vector+ is far from the query ("Password complexity and rotation policy"),+ * a single `hybrid` query surfaces BOTH — no second system, no+ client-side result stitching.++Vectors here are hand-authored 4-dim "topic" vectors so the demo is+deterministic. In production you'd fill dense_vector fields with a real+embedding model (or use XERJ's semantic_text auto-embedding).+Topic axes: [auth_recovery, security_policy, cooking, generic]+"""+import json+import sys+import urllib.request++BASE = "http://localhost:9485"+INDEX = "helpdesk"+++def req(method, path, body=None):+ data = json.dumps(body).encode() if body is not None else None+ r = urllib.request.Request(+ BASE + path, data=data, method=method,+ headers={"Content-Type": "application/json"},+ )+ with urllib.request.urlopen(r) as resp:+ return json.loads(resp.read().decode())+++def ids(res):+ return [h["_id"] for h in res["hits"]["hits"]]+++def table(res):+ for h in res["hits"]["hits"]:+ print(f" {h['_id']:4} score={h['_score']:.4f} {h['_source']['title']}")+++# ---- 1. fresh index: a text field + a dense_vector field -----------------+try:+ req("DELETE", f"/{INDEX}")+except urllib.error.HTTPError:+ pass++req("PUT", f"/{INDEX}", {+ "mappings": {+ "properties": {+ "title": {"type": "text"},+ "vec": {"type": "dense_vector", "dims": 4, "similarity": "cosine"},+ }+ }+})++# ---- 2. corpus: title (for BM25) + topic vector (for kNN) ----------------+DOCS = [+ ("d1", "Reset your password", [1.00, 0.15, 0.0, 0.10]),+ ("d2", "Regain entry to a locked-out account", [0.95, 0.10, 0.0, 0.15]),+ ("d3", "Password complexity and rotation policy", [0.15, 1.00, 0.0, 0.10]),+ ("d4", "How to bake sourdough bread", [0.00, 0.00, 1.0, 0.10]),+ ("d5", "Change your account password", [0.85, 0.25, 0.0, 0.20]),+]++bulk = ""+for did, title, vec in DOCS:+ bulk += json.dumps({"index": {"_index": INDEX, "_id": did}}) + "\n"+ bulk += json.dumps({"title": title, "vec": vec}) + "\n"++r = urllib.request.Request(+ BASE + "/_bulk", data=bulk.encode(), method="POST",+ headers={"Content-Type": "application/x-ndjson"},+)+with urllib.request.urlopen(r) as resp:+ assert not json.loads(resp.read())["errors"], "bulk had errors"+req("POST", f"/{INDEX}/_refresh")++# The user's intent, in both modalities:+QUERY_TEXT = "reset password"+QUERY_VEC = [1.0, 0.20, 0.0, 0.10] # "auth recovery" intent++# ---- 3a. Pure keyword (BM25) ---------------------------------------------+kw = req("POST", f"/{INDEX}/_search", {+ "size": 3,+ "query": {"match": {"title": QUERY_TEXT}},+})+print("Pure BM25 (match 'reset password'):")+table(kw)+kw_ids = ids(kw)++# ---- 3b. Pure vector (kNN, top-3) ----------------------------------------+# knn scores every doc that has the field by cosine; `size` is the cutoff the+# user actually sees, so size:3 == "show me the 3 nearest".+vec = req("POST", f"/{INDEX}/_search", {+ "size": 3,+ "query": {"knn": {"field": "vec", "query_vector": QUERY_VEC, "k": 3}},+})+print("\nPure kNN (top-3 nearest topic vectors):")+table(vec)+vec_ids = ids(vec)++# ---- 3c. Hybrid — both, fused with Reciprocal Rank Fusion ----------------+hyb = req("POST", f"/{INDEX}/_search", {+ "size": 5,+ "query": {+ "hybrid": {+ "queries": [+ {"query": {"match": {"title": QUERY_TEXT}}, "weight": 1.0},+ {"query": {"knn": {"field": "vec", "query_vector": QUERY_VEC, "k": 3}}, "weight": 1.0},+ ],+ "fusion": "rrf"+ }+ },+})+print("\nHybrid (BM25 + kNN, RRF fusion):")+table(hyb)+hyb_ids = ids(hyb)++# ---- 4. Prove the point --------------------------------------------------+print("\n--- assertions ---")+# d2 is relevant (account recovery) but shares NO keywords -> BM25 can't see it+assert "d2" not in kw_ids, f"expected BM25 to miss d2, got {kw_ids}"+print("PASS BM25 alone MISSES d2 (locked-out account: no shared keywords)")++# d3 has the keyword 'password' but its topic vector is far -> kNN top-3 drops it+assert "d3" not in vec_ids, f"expected kNN top-3 to miss d3, got {vec_ids}"+print("PASS kNN alone MISSES d3 (password policy: topic vector too far)")++# hybrid recovers BOTH in one request+assert "d2" in hyb_ids and "d3" in hyb_ids, f"hybrid should surface d2 AND d3, got {hyb_ids}"+print("PASS Hybrid surfaces BOTH d2 and d3 in a single query")++# and the exact double-match still ranks first+assert hyb_ids[0] == "d1", f"expected d1 first in hybrid, got {hyb_ids}"+print("PASS d1 (keyword + vector match) still ranks #1 under fusion")++print("\nBM25 ids :", kw_ids)+print("kNN ids :", vec_ids)+print("Hybrid ids:", hyb_ids)+print("\nOK")diff --git a/docs/examples/log-analytics/log_analytics.py b/docs/examples/log-analytics/log_analytics.pynew file mode 100644index 000000000..4c88baacd--- /dev/null+++ b/docs/examples/log-analytics/log_analytics.py@@ -0,0 +1,169 @@+#!/usr/bin/env python3+"""+Log analytics with XERJ — from raw logs to dashboards.++Bulk-ingests a few hundred synthetic structured log events, then answers+real operational questions with Elasticsearch-style aggregations:+ 1. Error rate over time -> date_histogram + filters sub-agg+ 2. p95 / p99 latency -> percentiles+ 3. Top services by volume -> terms + stats sub-agg++Stdlib only (urllib + json). No pip installs. Point it at a live XERJ.+"""+import json+import random+import sys+import urllib.request+from datetime import datetime, timedelta, timezone++BASE = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:9483"+INDEX = "logs-app"+++def req(method, path, body=None, ndjson=False):+ data = None+ headers = {}+ if ndjson:+ data = body.encode()+ headers["Content-Type"] = "application/x-ndjson"+ elif body is not None:+ data = json.dumps(body).encode()+ headers["Content-Type"] = "application/json"+ r = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)+ with urllib.request.urlopen(r) as resp:+ return json.loads(resp.read().decode())+++# ---------------------------------------------------------------- setup+# Clean slate, then create an explicit mapping so timestamp is a date,+# latency_ms is numeric, and service/level/status are keywords for aggs.+try:+ req("DELETE", "/" + INDEX)+except urllib.error.HTTPError:+ pass++req("PUT", "/" + INDEX, {+ "mappings": {+ "properties": {+ "@timestamp": {"type": "date"},+ "level": {"type": "keyword"},+ "service": {"type": "keyword"},+ "latency_ms": {"type": "integer"},+ "status": {"type": "integer"},+ "message": {"type": "text"},+ }+ }+})++# ---------------------------------------------------------------- generate+# 600 events across 6 hours and 4 services. Each service has its own+# baseline latency and error propensity so the aggregations tell a story.+random.seed(7)+services = {+ "checkout": {"lat": 120, "err": 0.08},+ "search": {"lat": 45, "err": 0.02},+ "catalog": {"lat": 30, "err": 0.01},+ "payments": {"lat": 200, "err": 0.05},+}+start = datetime(2026, 7, 6, 8, 0, 0, tzinfo=timezone.utc)+N = 600+lines = []+for i in range(N):+ svc = random.choices(list(services), weights=[3, 4, 4, 2])[0]+ cfg = services[svc]+ ts = start + timedelta(seconds=random.randint(0, 6 * 3600))+ is_err = random.random() < cfg["err"]+ if is_err:+ level, status = "ERROR", random.choice([500, 502, 503])+ latency = int(cfg["lat"] * random.uniform(2.0, 5.0)) # errors are slow+ else:+ level = random.choices(["INFO", "WARN"], weights=[9, 1])[0]+ status = 200+ latency = int(cfg["lat"] * random.uniform(0.5, 1.8))+ doc = {+ "@timestamp": ts.isoformat(),+ "level": level, "service": svc,+ "latency_ms": latency, "status": status,+ "message": f"{svc} handled request status={status}",+ }+ lines.append(json.dumps({"index": {"_index": INDEX}}))+ lines.append(json.dumps(doc))++resp = req("POST", "/_bulk", "\n".join(lines) + "\n", ndjson=True)+assert not resp.get("errors"), "bulk had errors: " + json.dumps(resp)[:400]+req("POST", "/" + INDEX + "/_refresh")+count = req("GET", "/" + INDEX + "/_count")["count"]+print(f"Ingested {count} log events into '{INDEX}'\n")+assert count == N, f"expected {N}, got {count}"++# ---------------------------------------------------------------- Q1+# Error rate over time: bucket by hour, and inside each bucket count how+# many were ERROR with a `filter` sub-agg. errors/doc_count = error rate.+print("=" * 60)+print("Q1. Error rate over time (hourly)")+print("=" * 60)+r = req("POST", "/" + INDEX + "/_search", {+ "size": 0,+ "aggs": {+ "per_hour": {+ "date_histogram": {"field": "@timestamp", "fixed_interval": "1h"},+ "aggs": {+ "errors": {"filter": {"term": {"level": "ERROR"}}}+ }+ }+ }+})+print(f"{'hour (UTC)':<22}{'total':>8}{'errors':>8}{'err_rate':>10}")+for b in r["aggregations"]["per_hour"]["buckets"]:+ total = b["doc_count"]+ errs = b["errors"]["doc_count"]+ rate = (errs / total * 100) if total else 0.0+ hour = b["key_as_string"][11:16]+ print(f"{hour:<22}{total:>8}{errs:>8}{rate:>9.1f}%")++# ---------------------------------------------------------------- Q2+# Latency percentiles across the whole fleet, then p95 broken out per+# service so you can see which service is dragging the tail.+print()+print("=" * 60)+print("Q2. Latency percentiles (ms)")+print("=" * 60)+r = req("POST", "/" + INDEX + "/_search", {+ "size": 0,+ "aggs": {+ "latency": {"percentiles": {"field": "latency_ms", "percents": [50, 95, 99]}},+ "by_service": {+ "terms": {"field": "service"},+ "aggs": {"p95": {"percentiles": {"field": "latency_ms", "percents": [95]}}}+ }+ }+})+pct = r["aggregations"]["latency"]["values"]+print(f"fleet-wide p50={pct['50.0']:.0f} p95={pct['95.0']:.0f} p99={pct['99.0']:.0f}")+print(f"{'service':<12}{'p95 latency':>14}")+for b in r["aggregations"]["by_service"]["buckets"]:+ p95 = b["p95"]["values"]["95.0"]+ print(f"{b['key']:<12}{p95:>13.0f}")++# ---------------------------------------------------------------- Q3+# Top services by volume, with a stats sub-agg (count/avg/min/max) so one+# call gives you the leaderboard AND the latency profile of each service.+print()+print("=" * 60)+print("Q3. Top services by volume (+ latency stats)")+print("=" * 60)+r = req("POST", "/" + INDEX + "/_search", {+ "size": 0,+ "aggs": {+ "top_services": {+ "terms": {"field": "service", "size": 10, "order": {"_count": "desc"}},+ "aggs": {"lat": {"stats": {"field": "latency_ms"}}}+ }+ }+})+print(f"{'service':<12}{'volume':>8}{'avg_ms':>9}{'min':>6}{'max':>7}")+for b in r["aggregations"]["top_services"]["buckets"]:+ s = b["lat"]+ print(f"{b['key']:<12}{b['doc_count']:>8}{s['avg']:>9.0f}{int(s['min']):>6}{int(s['max']):>7}")++print("\nAll aggregations returned. Log analytics on one engine, ES wire API.")diff --git a/docs/examples/migrate-from-elasticsearch/migrate_demo.sh b/docs/examples/migrate-from-elasticsearch/migrate_demo.shnew file mode 100755index 000000000..b4faca7ab--- /dev/null+++ b/docs/examples/migrate-from-elasticsearch/migrate_demo.sh@@ -0,0 +1,99 @@+#!/usr/bin/env bash+# migrate_demo.sh — prove standard Elasticsearch REST calls run UNCHANGED against XERJ.+#+# The only thing that changed vs. a real Elasticsearch cluster is $ES:+# point it at XERJ's wire port and every request below is byte-for-byte+# the same JSON you'd send Elasticsearch.+#+# ES=http://localhost:9487 ./migrate_demo.sh+#+# Requires: curl, and (for pretty output) python3 — both stdlib-only.+set -euo pipefail++ES="${ES:-http://localhost:9200}" # <-- the ONE thing you change to migrate+IDX=products++pp() { python3 -m json.tool; } # pretty-print JSON from stdin+say() { printf '\n=== %s ===\n' "$1"; }++# 0) Clean slate (ignore 404 on first run)+curl -s -X DELETE "$ES/$IDX" >/dev/null || true++# 1) Create an index with an explicit mapping — standard ES PUT.+say "1. PUT /$IDX (create index + mapping)"+curl -s -X PUT "$ES/$IDX" \+ -H 'Content-Type: application/json' -d '{+ "mappings": {+ "properties": {+ "name": { "type": "text" },+ "brand": { "type": "keyword" },+ "price": { "type": "float" },+ "in_stock": { "type": "boolean" }+ }+ }+}' | pp++# 2) Bulk-load documents — standard ES _bulk NDJSON (action line + source line).+say "2. POST /_bulk (index 6 docs)"+curl -s -X POST "$ES/_bulk" \+ -H 'Content-Type: application/x-ndjson' --data-binary '+{"index":{"_index":"products","_id":"1"}}+{"name":"Aluminum water bottle","brand":"Klean","price":24.99,"in_stock":true}+{"index":{"_index":"products","_id":"2"}}+{"name":"Insulated steel water bottle","brand":"Hydro","price":39.95,"in_stock":true}+{"index":{"_index":"products","_id":"3"}}+{"name":"Glass water carafe","brand":"Bodum","price":19.50,"in_stock":false}+{"index":{"_index":"products","_id":"4"}}+{"name":"Steel travel mug","brand":"Hydro","price":29.00,"in_stock":true}+{"index":{"_index":"products","_id":"5"}}+{"name":"Ceramic coffee mug","brand":"Bodum","price":12.00,"in_stock":true}+{"index":{"_index":"products","_id":"6"}}+{"name":"Plastic sports bottle","brand":"Klean","price":9.99,"in_stock":false}+' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("errors:",d["errors"],"items:",len(d["items"]))'++# Make freshly-indexed docs searchable (ES semantics: refresh).+curl -s -X POST "$ES/$IDX/_refresh" >/dev/null++# 3) match query — full-text search on the analyzed "name" field.+say "3. _search match: name ~ \"water bottle\""+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{+ "query": { "match": { "name": "water bottle" } },+ "_source": ["name","brand","price"]+}' | python3 -c '+import sys,json+d=json.load(sys.stdin)+print("total:", d["hits"]["total"]["value"])+for h in d["hits"]["hits"]:+ print(f" {h[_SCORE]:.3f} {h[_SRC][NAME]}".replace("_SCORE",chr(39)+"_score"+chr(39)).replace("_SRC",chr(39)+"_source"+chr(39)).replace("NAME",chr(39)+"name"+chr(39)))+' 2>/dev/null || curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{"query":{"match":{"name":"water bottle"}},"_source":["name"]}' | pp++# 4) bool query: must match + term filter + range filter — the classic ES combo.+say "4. _search bool: match \"bottle\" AND brand=Hydro-or-Klean AND price 10..40"+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{+ "query": {+ "bool": {+ "must": [ { "match": { "name": "bottle" } } ],+ "filter": [+ { "terms": { "brand": ["Hydro","Klean"] } },+ { "range": { "price": { "gte": 10, "lte": 40 } } }+ ]+ }+ },+ "_source": ["name","brand","price"]+}' | pp++# 5) terms aggregation — count products per brand (no query hits, just the agg).+say "5. _search terms agg: products per brand"+curl -s "$ES/$IDX/_search" -H 'Content-Type: application/json' -d '{+ "size": 0,+ "aggs": {+ "by_brand": { "terms": { "field": "brand" } }+ }+}' | pp++# 6) _cat/indices — operational sanity check, human-readable table.+say "6. GET /_cat/indices?v"+curl -s "$ES/_cat/indices?v"++echo+echo "All standard Elasticsearch calls succeeded against XERJ at $ES"diff --git a/docs/examples/semantic-search-rag/rag_demo.py b/docs/examples/semantic-search-rag/rag_demo.pynew file mode 100644index 000000000..de58f7322--- /dev/null+++ b/docs/examples/semantic-search-rag/rag_demo.py@@ -0,0 +1,165 @@+#!/usr/bin/env python3+"""+Semantic search / RAG retrieval with XERJ — no separate vector DB.++Indexes a tiny knowledge base into a `semantic_text` field (auto-embedded on+ingest, zero external config), then retrieves passages by *meaning* using the+`semantic` query. The retrieved passages are the chunks you'd hand to an LLM as+grounding context for RAG.++Stdlib only (urllib + json). Point BASE at a running XERJ es_compat port.++ python3 rag_demo.py # defaults to http://localhost:9482+ BASE=http://localhost:9200 python3 rag_demo.py+"""+import json+import os+import sys+import time+import urllib.request+import urllib.error++BASE = os.environ.get("BASE", "http://localhost:9482")+INDEX = "kb"+++def req(method, path, body=None):+ data = json.dumps(body).encode() if body is not None else None+ r = urllib.request.Request(+ BASE + path, data=data, method=method,+ headers={"Content-Type": "application/json"},+ )+ try:+ with urllib.request.urlopen(r) as resp:+ return resp.status, json.loads(resp.read().decode())+ except urllib.error.HTTPError as e:+ return e.code, json.loads(e.read().decode())+++# The knowledge base: short docs an LLM might need to answer support questions.+# Note the wording — the queries below deliberately paraphrase these.+DOCS = [+ {"id": "kb-1", "title": "Rotating an API key",+ "body": "To change your API credentials, open Settings, choose Security, "+ "and click Regenerate token. The old token stops working immediately."},+ {"id": "kb-2", "title": "Refund policy",+ "body": "Customers can request their money back within 30 days of purchase. "+ "Refunds are issued to the original payment method within five business days."},+ {"id": "kb-3", "title": "Resetting a forgotten password",+ "body": "If you cannot sign in, use the Forgot password link on the login page "+ "to receive a reset email and choose new credentials."},+ {"id": "kb-4", "title": "Supported file formats for upload",+ "body": "You may upload documents as PDF, DOCX, or plain text. Spreadsheets "+ "in CSV and XLSX are also accepted. Images are not indexed."},+ {"id": "kb-5", "title": "Rate limits",+ "body": "The API permits 600 requests per minute per key. Exceeding the "+ "throttle returns HTTP 429; back off and retry after the reset window."},+]+++def main():+ # 1) Map `body` as semantic_text -> auto-embedded on ingest (no external service).+ req("DELETE", "/" + INDEX)+ status, resp = req("PUT", "/" + INDEX, {+ "mappings": {+ "properties": {+ "title": {"type": "text"},+ "body": {"type": "semantic_text"},+ }+ }+ })+ print(f"create index -> {status} {json.dumps(resp)[:120]}")+ assert status == 200, resp++ # 2) Bulk-ingest. XERJ embeds `body` into a companion vector at index time.+ lines = []+ for d in DOCS:+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": d["id"]}}))+ lines.append(json.dumps({"title": d["title"], "body": d["body"]}))+ ndjson = "\n".join(lines) + "\n"+ r = urllib.request.Request(+ BASE + "/_bulk", data=ndjson.encode(), method="POST",+ headers={"Content-Type": "application/x-ndjson"},+ )+ with urllib.request.urlopen(r) as resp:+ bulk = json.loads(resp.read().decode())+ print(f"bulk -> errors={bulk.get('errors')} items={len(bulk.get('items', []))}")+ assert bulk.get("errors") is False, bulk+ req("POST", f"/{INDEX}/_refresh")++ # 3) Semantic retrieval. The query PARAPHRASES the KB — different words, same meaning.+ def semantic(query, k=3):+ status, resp = req("POST", f"/{INDEX}/_search", {+ "size": k,+ "query": {"semantic": {"field": "body", "query": query, "k": k}},+ })+ assert status == 200, resp+ return resp["hits"]["hits"]++ def keyword(query, k=3):+ status, resp = req("POST", f"/{INDEX}/_search", {+ "size": k,+ "query": {"match": {"body": query}},+ })+ assert status == 200, resp+ return resp["hits"]["hits"]++ print("\n=== SEMANTIC RETRIEVAL (retrieve by meaning) ===")+ checks = [+ # (paraphrased user question, id we expect back at rank 1)+ ("how do I regenerate my API token", "kb-1"),+ ("can I get my money back after buying", "kb-2"),+ ("I forgot my login and can't sign in", "kb-3"),+ ("which file formats are supported for uploads", "kb-4"),+ ("how many calls per minute am I allowed", "kb-5"),+ ]+ passed = 0+ for q, expect in checks:+ hits = semantic(q, k=3)+ top = hits[0]+ ok = top["_id"] == expect+ passed += ok+ print(f"\nQ: {q!r}")+ print(f" top -> {top['_id']} '{top['_source']['title']}' "+ f"score={top['_score']:.4f} {'OK' if ok else 'MISS(expected '+expect+')'}")+ for h in hits:+ print(f" {h['_id']:6} {h['_score']:.4f} {h['_source']['title']}")++ # 4) Show the RAG context bundle you'd feed an LLM for the first question.+ q = "how do I regenerate my API token"+ hits = semantic(q, k=2)+ context = "\n\n".join(+ f"[{h['_id']}] {h['_source']['title']}\n{h['_source']['body']}" for h in hits+ )+ print("\n=== RAG CONTEXT BUNDLE (top-2 passages -> LLM prompt) ===")+ print(f"User question: {q}\n")+ print("Retrieved context:\n" + context)++ # 5) Contrast: a paraphrase with NO shared keywords is where lexical differs.+ print("\n=== KEYWORD vs SEMANTIC (same paraphrase) ===")+ q = "get my money back after buying"+ kw = keyword(q, k=1)+ sm = semantic(q, k=1)+ print(f"Q: {q!r}")+ print(f" match top -> {(kw[0]['_id']+' '+kw[0]['_source']['title']) if kw else 'NO HITS'}")+ print(f" semantic top -> {sm[0]['_id']} {sm[0]['_source']['title']}")++ # 6) Honest limitation: the BUILT-IN embedder is lexical. A true paraphrase+ # with no token overlap against the embedded *body* text can miss. Here+ # the KB titled "Rotating an API key" is stored, but its body never says+ # "rotate", so a "rotate" paraphrase does not retrieve it. Wire an+ # external /v1/embeddings endpoint for production-grade neural semantics.+ print("\n=== HONEST LIMITATION (lexical embedder, no body-token overlap) ===")+ q = "how do I rotate my API key"+ sm = semantic(q, k=1)+ print(f"Q: {q!r}")+ print(f" semantic top -> {sm[0]['_id']} {sm[0]['_source']['title']} "+ f"(expected kb-1; the body never uses the word 'rotate')")++ print(f"\nRESULT: {passed}/{len(checks)} semantic retrievals returned the right doc at rank 1")+ if passed != len(checks):+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/docs/examples/vector-search-knn/knn_demo.py b/docs/examples/vector-search-knn/knn_demo.pynew file mode 100644index 000000000..03cd67d0c--- /dev/null+++ b/docs/examples/vector-search-knn/knn_demo.py@@ -0,0 +1,156 @@+#!/usr/bin/env python3+"""+Vector similarity search (kNN) against a live XERJ node.++Use case: you have items (products, docs, images) each represented as an+embedding vector, and you want "find the N most similar to this one".++This script uses tiny, hand-made 4-dimensional vectors so the result is+100% reproducible and you can eyeball why each neighbor was returned.+The 4 dims are a toy "feature space":++ [ citrus , berry , engine , wheels ]++- Fruits score high on the fruit axes, ~0 on the vehicle axes.+- Vehicles score high on engine/wheels, ~0 on the fruit axes.++So a query near the "citrus" corner should return oranges/lemons first,+and never a truck. That's the recall check.++No external deps: Python 3 stdlib only (urllib + json).+"""+import json+import urllib.request+import urllib.error++BASE = "http://localhost:9484"+INDEX = "catalog"+++def req(method, path, body=None):+ data = None+ headers = {}+ if body is not None:+ if isinstance(body, str): # ndjson (bulk)+ data = body.encode()+ headers["Content-Type"] = "application/x-ndjson"+ else:+ data = json.dumps(body).encode()+ headers["Content-Type"] = "application/json"+ r = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)+ try:+ with urllib.request.urlopen(r) as resp:+ return resp.status, json.loads(resp.read().decode())+ except urllib.error.HTTPError as e:+ return e.code, json.loads(e.read().decode())+++def hits(resp):+ """(id, score, name, category, in_stock) for each hit, in ranked order."""+ out = []+ for h in resp["hits"]["hits"]:+ s = h["_source"]+ out.append((h["_id"], round(h["_score"], 4), s["name"], s["category"], s["in_stock"]))+ return out+++# ── 1. Fresh index with a dense_vector field ───────────────────────+req("DELETE", "/" + INDEX)+status, _ = req("PUT", "/" + INDEX, {+ "mappings": {+ "properties": {+ "name": {"type": "text"},+ "category": {"type": "keyword"},+ "in_stock": {"type": "boolean"},+ # dims MUST match the vectors you index. similarity defaults to+ # cosine (magnitude-invariant) — the right choice for most+ # normalized embeddings. Other options: l2, dot_product.+ "embedding": {"type": "dense_vector", "dims": 4, "similarity": "cosine"},+ }+ }+})+print("create index:", status)++# ── 2. Index a tiny catalog. Vectors are [citrus, berry, engine, wheels]+catalog = [+ ("1", "Navel Orange", "fruit", True, [0.90, 0.10, 0.00, 0.00]),+ ("2", "Meyer Lemon", "fruit", True, [0.95, 0.05, 0.00, 0.00]),+ ("3", "Blood Orange", "fruit", False, [0.85, 0.20, 0.00, 0.00]),+ ("4", "Strawberry", "fruit", True, [0.10, 0.92, 0.00, 0.00]),+ ("5", "Blueberry", "fruit", True, [0.05, 0.95, 0.00, 0.00]),+ ("6", "Pickup Truck", "vehicle", True, [0.00, 0.00, 0.90, 0.85]),+ ("7", "Sports Car", "vehicle", False, [0.00, 0.00, 0.95, 0.80]),+ ("8", "Electric Sedan", "vehicle", True, [0.00, 0.05, 0.40, 0.88]),+]+lines = []+for _id, name, cat, stock, vec in catalog:+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": _id}}))+ lines.append(json.dumps({"name": name, "category": cat, "in_stock": stock, "embedding": vec}))+status, resp = req("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")+print("bulk:", status, "errors=", resp.get("errors"))++# ── 3. Plain kNN: "find items most like a citrus fruit" ──────────────+query_vec = [0.92, 0.08, 0.00, 0.00] # sits right in the citrus corner+status, resp = req("POST", "/%s/_search" % INDEX, {+ "knn": {"field": "embedding", "query_vector": query_vec, "k": 3, "num_candidates": 10}+})+print("\n=== kNN k=3, query ~citrus ===")+knn_hits = hits(resp)+for h in knn_hits:+ print(" ", h)++top3_ids = [h[0] for h in knn_hits]+top3_cats = [h[3] for h in knn_hits]+# RECALL CHECK: the 3 nearest MUST be the 3 citrus fruits (ids 1,2,3),+# and NO vehicle should appear.+assert set(top3_ids) == {"1", "2", "3"}, top3_ids+assert all(c == "fruit" for c in top3_cats), top3_cats+# Cosine similarity of near-identical direction should be ~1.0.+assert knn_hits[0][1] > 0.99, knn_hits[0]+print(" OK: top-3 are exactly the citrus fruits, no vehicles leaked in")++# ── 4. kNN + filter: same query, but only items in stock ─────────────+# To combine kNN with a filter, wrap the knn in a `bool` and put the+# filter in a sibling `filter` clause. bool.filter runs as a PRE-filter:+# XERJ restricts the candidate set, THEN ranks by vector similarity.+#+# "Blood Orange" (id 3) is the 3rd-closest citrus but is OUT of stock, so+# with the filter it drops out and a farther in-stock item (a berry)+# takes its place — proving the filter runs, not just the ranker.+#+# NOTE: the ES-8 shorthand `knn: {..., "filter": {...}}` is parsed but the+# filter is currently ignored on this path — use the bool wrapper below.+status, resp = req("POST", "/%s/_search" % INDEX, {+ "size": 3,+ "query": {+ "bool": {+ "must": [+ {"knn": {"field": "embedding", "query_vector": query_vec,+ "k": 3, "num_candidates": 10}}+ ],+ "filter": [{"term": {"in_stock": True}}],+ }+ },+})+print("\n=== kNN + bool.filter in_stock:true ===")+filtered = hits(resp)+for h in filtered:+ print(" ", h)+filtered_ids = [h[0] for h in filtered]+assert all(h[4] is True for h in filtered), filtered # every hit in stock+assert "3" not in filtered_ids, filtered_ids # out-of-stock citrus excluded+assert filtered_ids[:2] == ["1", "2"], filtered_ids # two in-stock citrus rank first+print(" OK: out-of-stock Blood Orange excluded; only in-stock neighbors returned")++# ── 5. Query the other corner to show the space really separates ───────+status, resp = req("POST", "/%s/_search" % INDEX, {+ "knn": {"field": "embedding", "query_vector": [0.0, 0.0, 0.9, 0.85], "k": 2, "num_candidates": 10}+})+print("\n=== kNN k=2, query ~vehicle ===")+veh = hits(resp)+for h in veh:+ print(" ", h)+assert all(h[3] == "vehicle" for h in veh), veh+print(" OK: vehicle query returns only vehicles")++print("\nALL ASSERTIONS PASSED")diff --git a/docs/recipes/README.md b/docs/recipes/README.mdnew file mode 100644index 000000000..4104dae68--- /dev/null+++ b/docs/recipes/README.md@@ -0,0 +1,17 @@+# XERJ Recipes++Use-case-driven guides for building real things with XERJ. Every recipe was **verified end-to-end against a live XERJ** before being written — the runnable example for each lives under [`docs/examples/`](../examples).++These are practical "how do I actually do X" recipes, not an API reference (that follows separately).++| Recipe | What you build | Example |+|---|---|---|+| [Give an AI agent long-term memory](./agentic-memory.md) | A memory-backed agent using the `/_memory` API (store, semantic + keyword recall, metadata filters, forgetting, per-agent isolation) | [`agent_memory.py`](../examples/agentic-memory/agent_memory.py) |+| [Semantic search & RAG](./semantic-search-rag.md) | Retrieval by meaning with `semantic_text` (auto-embed on ingest, no separate vector DB) | [`rag_demo.py`](../examples/semantic-search-rag/rag_demo.py) |+| [Log analytics](./log-analytics.md) | From raw logs to dashboards — error rates, p95 latency, top services via aggregations | [`log_analytics.py`](../examples/log-analytics/log_analytics.py) |+| [Vector search (kNN)](./vector-search-knn.md) | Nearest-neighbor similarity search over `dense_vector` (HNSW), with filters | [`knn_demo.py`](../examples/vector-search-knn/knn_demo.py) |+| [Hybrid search](./hybrid-search.md) | Keyword + vector in one query — results neither BM25 nor kNN finds alone | [`hybrid_search.py`](../examples/hybrid-search/hybrid_search.py) |+| [Anomaly detection](./anomaly-detection.md) | Statistical `_ml` detectors that flag spikes in metrics/logs | [`anomaly_detection.py`](../examples/anomaly-detection/anomaly_detection.py) |+| [Migrate from Elasticsearch](./migrate-from-elasticsearch.md) | Point your existing ES client at XERJ — same wire, change the URL | [`migrate_demo.sh`](../examples/migrate-from-elasticsearch/migrate_demo.sh) |++Every example is stdlib-only Python 3 or plain `curl` — no dependencies to install. Start XERJ (`./target/release/xerj --data-dir ./data --insecure`), then run any example.diff --git a/docs/recipes/agentic-memory.md b/docs/recipes/agentic-memory.mdnew file mode 100644index 000000000..37b1162f9--- /dev/null+++ b/docs/recipes/agentic-memory.md@@ -0,0 +1,334 @@+# Give an AI agent long-term memory with XERJ++**Use case:** your agent learns things during a session — user preferences, project+facts, decisions — and needs to *remember* them on a later turn to answer well.+The usual answer is to bolt a separate vector database next to your search stack.+XERJ folds that job into the engine you already run: a namespaced, offline+**agent-memory API** (`/_memory/{ns}`) that stores text + an embedding + metadata+and recalls the most relevant memories by **vector similarity** or **plain text**,+with metadata filtering, forgetting, and hard per-agent isolation.++This recipe builds a tiny simulated coding-assistant agent (no LLM required) that:++1. **stores** what it learns as memories (`POST /_memory/{ns}`),+2. **recalls** the right memory to answer a new question — semantically (kNN) and+ lexically (BM25),+3. does a **metadata-filtered** recall ("only surface team *preferences*"),+4. **forgets** a memory that is no longer true, and+5. proves **multi-namespace isolation**: a second agent can never see the first+ agent's memories.++Everything below was run end-to-end against a live XERJ. No pip installs — the+example is Python 3 stdlib only.++---++## Why this replaces a bolt-on vector DB++A memory namespace is just a reserved XERJ index (`.xerj-memory-{ns}`) under the+hood, so recall reuses the same `dense_vector`/HNSW kNN, BM25, and metadata-filter+paths that serve the rest of the engine. That means:++- **One system to run and back up.** No second datastore, no sync job keeping a+ vector index consistent with your source of truth.+- **kNN *and* BM25 in the same store.** Recall by embedding similarity, by keyword,+ or filter both by metadata — no glue code.+- **Namespaces are physical isolation.** Each agent (or tenant, or user) gets its+ own backing index; a recall in namespace `A` literally cannot read namespace `B`.+- **Offline and zero-config.** You supply the vectors (from your own model); XERJ+ never phones out to embed.++---++## The memory API, in four calls++| Verb + path | What it does | Body / result |+|---|---|---|+| `POST /_memory/{ns}` | store a memory | `{text, vector?, metadata?, id?}` → `{id, namespace, created}` |+| `POST /_memory/{ns}/_recall` | recall top-k | `{vector? \| query?, k?, filter?}` → `{hits:[{id,text,metadata,score}]}` |+| `GET /_memory/{ns}` | list recent (bounded) | → `{count, entries:[…]}` |+| `DELETE /_memory/{ns}/{id}` | forget one | → `{id, forgotten}` |+| `DELETE /_memory/{ns}` | drop the namespace | → `{namespace, dropped}` |++`_recall` picks the path from the body: a `vector` runs kNN; otherwise `query`+runs BM25 over the text; an empty body returns recent memories (`match_all`).+`filter` is a normal ES query clause applied as a `bool` filter, so it narrows+without affecting the score. Recall of an unknown namespace returns `{"hits": []}` —+that is how isolation stays clean.++### Storing a memory (raw wire)++```bash+curl -XPOST localhost:9481/_memory/agent-ada -H 'content-type: application/json' -d '{+ "text": "The team prefers Rust over Go for new backend services.",+ "vector": [0.12, 0.0, 0.34],+ "metadata": {"kind": "preference", "topic": "language"},+ "id": "demo-1"+}'+```+```json+{"created":true,"id":"demo-1","namespace":"agent-ada"}+```++### Recalling by vector, filtered to preferences (raw wire)++```bash+curl -XPOST localhost:9481/_memory/agent-ada/_recall -H 'content-type: application/json' -d '{+ "vector": [0.12, 0.0, 0.34],+ "k": 2,+ "filter": {"term": {"metadata.kind": "preference"}}+}'+```+```json+{"hits":[{"id":"demo-1",+ "metadata":{"kind":"preference","topic":"language"},+ "score":1.0,+ "text":"The team prefers Rust over Go for new backend services."}],+ "namespace":"agent-ada"}+```++Note `metadata.kind` filters straight out of the box — nested metadata subfields+are queryable with a normal `term` clause, no explicit mapping needed.++---++## The full agent++The script below is the whole example. The one thing to understand about it: it+carries a **tiny self-contained "embedder"** (deterministic hashed character+trigrams) so the demo needs no model download and cosine similarity is meaningful.++> **Honesty check.** That toy embedder is *lexical*, not neural: it rewards shared+> character n-grams (so `service`/`services`, `prefer`/`prefers` land near each+> other), but it does not know that "car" ≈ "automobile". For real semantic recall+> you pass vectors from your own model (OpenAI/Cohere/a local model) — the store and+> recall calls are byte-for-byte the same. XERJ's own built-in `semantic_text`+> embedder is likewise lexical unless you configure an external `/v1/embeddings`+> endpoint; the memory API deliberately takes *your* vectors so quality is your+> choice.++```python+#!/usr/bin/env python3+"""Long-term memory for an AI agent, backed by XERJ's /_memory API."""++import json+import math+import re+import urllib.request+from urllib.error import HTTPError++BASE = "http://localhost:9481"+DIM = 96 # embedding dimensionality for the toy hashing embedder+++# Tiny self-contained "embedder": deterministic hashed character trigrams.+# NOT neural -- real synonymy needs real embeddings. In production you pass+# vectors from your own model; XERJ stores and kNN-searches them the same way.+def embed(text: str) -> list[float]:+ s = " " + re.sub(r"[^a-z0-9]+", " ", text.lower()) + " "+ vec = [0.0] * DIM+ for i in range(len(s) - 2):+ h = 0+ for ch in s[i:i + 3]: # hash each character trigram into a bucket+ h = (h * 131 + ord(ch)) & 0xFFFFFFFF+ vec[h % DIM] += 1.0+ norm = math.sqrt(sum(v * v for v in vec)) or 1.0+ return [round(v / norm, 6) for v in vec]+++def _req(method, path, body=None):+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(BASE + path, data=data, method=method,+ headers={"Content-Type": "application/json"})+ try:+ with urllib.request.urlopen(req) as r:+ return json.loads(r.read() or b"{}")+ except HTTPError as e:+ return json.loads(e.read() or b"{}")+++def remember(ns, text, metadata, mem_id):+ return _req("POST", f"/_memory/{ns}",+ {"text": text, "vector": embed(text), "metadata": metadata, "id": mem_id})++def recall_vec(ns, query, k=3, filt=None): # semantic (kNN)+ body = {"vector": embed(query), "k": k}+ if filt:+ body["filter"] = filt+ return _req("POST", f"/_memory/{ns}/_recall", body)["hits"]++def recall_text(ns, query, k=3): # lexical (BM25)+ return _req("POST", f"/_memory/{ns}/_recall", {"query": query, "k": k})["hits"]++def forget(ns, mem_id):+ return _req("DELETE", f"/_memory/{ns}/{mem_id}")++def drop(ns):+ return _req("DELETE", f"/_memory/{ns}")++def show(label, hits):+ print(f"\n{label}")+ for h in hits:+ kind = (h.get("metadata") or {}).get("kind", "-")+ print(f" [{h['score']:.3f}] ({kind:10}) {h['text']}")+++def main():+ ADA, BILLING = "agent-ada", "agent-billing"+ drop(ADA); drop(BILLING) # clean slate for a repeatable demo++ print("== Storing what the agent learned this session ==")+ facts = [+ ("The team prefers Rust over Go for new backend services.",+ {"kind": "preference", "topic": "language"}),+ ("Production runs on AWS in us-east-1; staging is us-west-2.",+ {"kind": "fact", "topic": "infra"}),+ ("The team prefers tabs over spaces and a 100-column line limit.",+ {"kind": "preference", "topic": "style"}),+ ("The payments service owns the Postgres 'ledger' database.",+ {"kind": "fact", "topic": "ownership"}),+ ("Deploys are gated on the full integration suite passing in CI.",+ {"kind": "fact", "topic": "process"}),+ ("The team prefers OpenTelemetry for tracing, not vendor SDKs.",+ {"kind": "preference", "topic": "observability"}),+ ]+ for i, (text, md) in enumerate(facts, 1):+ r = remember(ADA, text, md, f"ada-{i}")+ print(f" stored {r['id']:8} created={r['created']}")++ remember(BILLING, "Invoices over $10k require CFO approval.", {"kind": "policy"}, "bill-1")++ q1 = "What programming language should I use for a new backend service?"+ show(f'== Semantic recall for: "{q1}"', recall_vec(ADA, q1, k=3))++ q2 = "Which region is production deployed in?"+ show(f'== Lexical (BM25) recall for: "{q2}"', recall_text(ADA, q2, k=3))++ q3 = "How should I set up the new service?"+ show(f'== Preference-only recall for: "{q3}"',+ recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}}))++ print("\n== Namespace isolation ==")+ leak = recall_vec(BILLING, q1, k=3)+ billing_texts = [h["text"] for h in leak]+ print(f" agent-billing recall for a coding question -> {billing_texts}")+ assert all("Rust" not in t and "AWS" not in t for t in billing_texts)+ print(" OK: agent-billing only ever sees its own namespace.")++ print("\n== Forgetting a memory that is no longer true ==")+ show(" before forget:", recall_vec(ADA, "what do we use for tracing", k=3))+ print(" forget ->", forget(ADA, "ada-6"))+ after = recall_vec(ADA, "what do we use for tracing", k=3)+ show(" after forget:", after)+ assert all(h["id"] != "ada-6" for h in after)+ print(" OK: the retracted memory is gone.")++ # Correctness checks so a green run really means it worked.+ assert "Rust" in recall_vec(ADA, q1, k=1)[0]["text"]+ assert "us-east-1" in recall_text(ADA, q2, k=1)[0]["text"]+ prefs = recall_vec(ADA, q3, k=5, filt={"term": {"metadata.kind": "preference"}})+ assert prefs and all((h["metadata"] or {}).get("kind") == "preference" for h in prefs)+ print("\nAll assertions passed.")+++if __name__ == "__main__":+ main()+```++---++## Run it++Start XERJ (single node, no TLS/auth) and run the script:++```bash+# 1. boot XERJ+printf '[server]\nes_compat_port = 9481\n' > /tmp/xerj.toml+./engine/target/release/xerj --insecure --data-dir /tmp/xerj-mem --config /tmp/xerj.toml &++# 2. run the agent+python3 docs/examples/agentic-memory/agent_memory.py+```++### Real output++```text+== Storing what the agent learned this session ==+ stored ada-1 created=True+ stored ada-2 created=True+ stored ada-3 created=True+ stored ada-4 created=True+ stored ada-5 created=True+ stored ada-6 created=True++== Semantic recall for: "What programming language should I use for a new backend service?"+ [0.779] (preference) The team prefers Rust over Go for new backend services.+ [0.741] (fact ) The payments service owns the Postgres 'ledger' database.+ [0.731] (preference) The team prefers tabs over spaces and a 100-column line limit.++== Lexical (BM25) recall for: "Which region is production deployed in?"+ [3.746] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.+ [1.010] (fact ) Deploys are gated on the full integration suite passing in CI.++== Preference-only recall for: "How should I set up the new service?"+ [0.765] (preference) The team prefers Rust over Go for new backend services.+ [0.692] (preference) The team prefers tabs over spaces and a 100-column line limit.+ [0.666] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.++== Namespace isolation ==+ agent-billing recall for a coding question -> ['Invoices over $10k require CFO approval.']+ OK: agent-billing only ever sees its own namespace.++== Forgetting a memory that is no longer true ==++ before forget:+ [0.755] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.+ [0.692] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.+ [0.683] (preference) The team prefers Rust over Go for new backend services.+ forget -> {'forgotten': True, 'id': 'ada-6', 'namespace': 'agent-ada'}++ after forget:+ [0.692] (fact ) Production runs on AWS in us-east-1; staging is us-west-2.+ [0.683] (preference) The team prefers Rust over Go for new backend services.+ [0.661] (preference) The team prefers tabs over spaces and a 100-column line limit.+ OK: the retracted memory is gone.++All assertions passed.+```++Read the run top to bottom:++- **Semantic recall** put the Rust preference on top for a question that never+ says "Rust" or "prefer" — trigram overlap on *backend / service / new* pulled it+ up over unrelated facts.+- **BM25 recall** answered the region question exactly, no vectors involved.+- **Preference-only recall** returned three memories, *all* `kind:"preference"` —+ the `metadata.kind` filter did its job.+- **Isolation** held: `agent-billing`, asked a coding question, only saw its own+ single finance memory.+- **Forgetting** removed `ada-6` (the tracing preference); the follow-up recall no+ longer surfaces it.++---++## Notes, limits, and going to production++- **Bring your own embeddings.** The memory API stores whatever vector you pass and+ kNN-searches it with cosine similarity. Swap the toy `embed()` for a call to your+ model's embeddings endpoint and everything else is unchanged. Keep the vector+ dimension consistent within a namespace — the backing index's `dense_vector`+ field is sized from the first stored vector.+- **Vector *or* text.** A memory needs at least `text` or a `vector`. Text is always+ BM25-indexed; a vector additionally enables kNN recall. Recall with `vector` uses+ kNN; without it, `query` uses BM25.+- **Recency.** The store blends similarity with a recency signal internally, so+ fresh memories surface even when marginally less similar — useful for agents whose+ latest observation should win ties.+- **Isolation is structural, not a filter.** Namespaces are separate physical+ indices, so there is no query you can write in one namespace that reads another.+ Use one namespace per agent/tenant/user.+- **`GET /_memory/{ns}`** lists up to 100 most-recent entries for debugging/audit;+ it is bounded and recent-first, not a full export.+- **Wire-compatible.** Because a namespace is a real index, you can inspect or+ operate on it with the standard ES-compatible surface if you ever need to — the+ memory API is a thin, honest adapter over the same machinery.\ No newline at end of filediff --git a/docs/recipes/anomaly-detection.md b/docs/recipes/anomaly-detection.mdnew file mode 100644index 000000000..34464a9a8--- /dev/null+++ b/docs/recipes/anomaly-detection.md@@ -0,0 +1,226 @@+# Detect a spike in a metric series++**Goal:** you have a timestamped metric — CPU, latency, request rate, error+count — and you want to know *when it went wrong* without hand-tuning a fixed+red-line. A CPU box that idles at 20% is fine at 20% and alarming at 96%, but+the "alarming" number depends on the box. You want the engine to learn the+normal band and flag the minute it broke out.++XERJ ships an on-demand anomaly detector (`_ml/anomaly_detectors`, Elasticsearch+wire-shape). You point it at an index, tell it the time field, the metric+(`mean(cpu)`), and a bucket span. When you score it, XERJ buckets the data by+time, computes the metric per bucket, builds a **moving baseline from prior+normal buckets**, and scores each bucket by how many standard deviations it sits+from that baseline. A spike lands many sigmas out and gets flagged; ordinary+jitter does not.++> **Honest scope.** This is **on-demand batch scoring**: you POST `_score` and+> get the full timeline back in one response. There is **no continuous datafeed+> or background job** yet — nothing runs on a timer, and there is no alerting.+> To monitor live, call `_score` on a schedule (cron, your alertmanager, a+> `/loop`) over a recent time window and act on `anomaly_count`. The model is a+> transparent moving-window z-score, not a seasonal/multivariate ML model — it's+> excellent for "this metric jumped relative to its recent self," which covers+> the large majority of ops spikes.++## The scenario++`cpu_metrics` collects four host readings per minute. For fifteen minutes CPU+sits around 20%. At **minute 12** a runaway process pins a box and the per-minute+mean jumps to **96%**, then recovers. We want XERJ to flag exactly that minute.++## 1. Map the index++The time field must be a `date`; the metric must be numeric so the detector can+average it.++```bash+curl -XPUT localhost:9486/cpu_metrics -H 'Content-Type: application/json' -d '{+ "mappings": { "properties": {+ "@timestamp": { "type": "date" },+ "host": { "type": "keyword" },+ "cpu": { "type": "double" }+ }}+}'+```++## 2. Bulk-ingest the series (spike injected at minute 12)++Standard `_bulk` NDJSON — 16 minutes × 4 hosts. Normal minutes read ~20%; minute+12 reads ~96%. (Full generator in the script below.)++```+{"index":{"_index":"cpu_metrics"}}+{"@timestamp":"2026-07-06T00:00:00Z","host":"web-0","cpu":19}+...+{"index":{"_index":"cpu_metrics"}}+{"@timestamp":"2026-07-06T00:12:00Z","host":"web-1","cpu":96} <- the spike+...+```++## 3. Create the detector++`function: mean` over the `cpu` field, one-minute buckets. `anomaly_threshold`+is the z-score gate (defaults to `3.0` — a bucket is flagged when it sits more+than 3σ from the baseline).++```bash+curl -XPUT localhost:9486/_ml/anomaly_detectors/cpu-spike \+ -H 'Content-Type: application/json' -d '{+ "source_index": "cpu_metrics",+ "time_field": "@timestamp",+ "function": "mean",+ "field": "cpu",+ "bucket_span": "1m",+ "description": "CPU mean per minute"+ }'+```++Response (the config, ES-shaped):++```json+{+ "job_id": "cpu-spike",+ "job_type": "anomaly_detector",+ "analysis_config": {+ "bucket_span": "1m",+ "detectors": [{ "function": "mean", "field_name": "cpu" }]+ },+ "data_description": { "time_field": "@timestamp" },+ "source_index": ["cpu_metrics"],+ "anomaly_threshold": 3.0+}+```++Supported `function` values: `count` (no field needed — buckets by doc count,+great for request/error rate), `mean`/`avg`, `min`, `max`, `sum`.++## 4. Score it++```bash+curl -XPOST localhost:9486/_ml/anomaly_detectors/cpu-spike/_score+```++XERJ returns the whole timeline plus a ranked `anomalies` list. Rendered per+bucket:++```+minute (UTC) actual expected z_score score flag+00:00:00 20.0 - - 0.0 <- warm-up+00:01:00 21.0 - - 0.0 <- warm-up+00:02:00 19.0 - - 0.0 <- warm-up+00:03:00 20.0 - - 0.0 <- warm-up+00:04:00 22.0 20.0 2.8 48.0+00:05:00 18.0 20.4 -2.4 41.9+00:06:00 21.0 20.0 0.8 16.4+00:07:00 20.0 20.1 -0.1 2.6+00:08:00 19.0 20.1 -1.0 20.0+00:09:00 21.0 20.0 0.9 18.1+00:10:00 20.0 20.1 -0.1 2.0+00:11:00 22.0 20.1 1.8 33.5+00:12:00 96.0 20.2 65.1 100.0 *** <- SPIKE+00:13:00 20.0 20.2 -0.2 4.8+00:14:00 21.0 20.2 0.7 14.7+00:15:00 19.0 20.3 -1.2 23.7++== anomalies (1 of 16 buckets) ==+ 2026-07-06T00:12:00.000Z actual=96.0 expected=20.2 z=65.1 score=100.0+```++The spike record in full:++```json+{+ "actual": 96.0,+ "expected": 20.25,+ "std_dev": 1.1636866703140785,+ "z_score": 65.09484205018445,+ "anomaly_score": 100.0,+ "is_anomaly": true,+ "timestamp_iso": "2026-07-06T00:12:00.000Z"+}+```++## How the scoring works (and how to read the numbers)++- **Warm-up.** The first 4 buckets seed the baseline and are not scored+ (`"expected": null`, `"note": "insufficient_baseline"`). Give the detector a+ span of history before the interesting part.+- **`expected` / `std_dev`.** The mean and standard deviation of prior **normal**+ buckets. Crucially, a flagged bucket is **not folded back into the baseline** —+ one spike can't poison the expectation for the minutes after it. Notice+ `expected` stays ~20 straight through the spike.+- **`z_score`.** `(actual − expected) / std_dev`. The spike is 65σ out; normal+ jitter stays within a couple of sigma.+- **`is_anomaly`** is the decision, gated purely on `|z| > anomaly_threshold`.+ Only the spike crosses it here.+- **`anomaly_score` (0–100)** is a normalized, human-friendly severity: it maps+ |z| = threshold to ~50 and saturates toward 100 for large deviations. It is+ **not** the flag. That's why the `00:04` jitter bucket shows a mid score of 48+ (its z of 2.8 is close to the threshold of 3) yet is correctly *not* flagged.+ **Alert on `is_anomaly` / `anomaly_count`; use `anomaly_score` for ranking and+ dashboards.** In this run the spike (100.0) is more than 2× the top normal+ bucket (48.0) — clean separation.++## Narrow or re-tune a run without recreating the job++`_score` accepts two per-run overrides in the POST body:++- **`query`** — restrict which docs feed the buckets (e.g. one `host`, one+ service). Any XERJ query works: `{"query": {"term": {"host": "web-1"}}}`.+- **`anomaly_threshold`** — override the flag sensitivity for this run only.++```bash+# Same data, stricter gate (10σ) — the 65σ spike still stands alone.+curl -XPOST localhost:9486/_ml/anomaly_detectors/cpu-spike/_score \+ -H 'Content-Type: application/json' -d '{"anomaly_threshold": 10}'+# -> "anomaly_threshold": 10.0, "anomaly_count": 1+```++## Count instead of mean (rate spikes)++For "too many errors this minute" you don't need a metric field — use+`function: count`, drop `field`, and point the detector at your log index. Each+bucket's `actual` becomes the doc count, scored the same way. Same recipe, same+response shape.++## List detectors++```bash+curl localhost:9486/_cat/ml/anomaly_detectors+# cpu-spike opened cpu_metrics mean 1m+```++`GET /_ml/anomaly_detectors/{id}` and `DELETE /_ml/anomaly_detectors/{id}` round+out lifecycle management.++## Run it++```bash+# XERJ listening on :9486 (adjust XERJ=... for another port)+python3 docs/examples/anomaly-detection/anomaly_detection.py+```++The script maps the index, ingests the series with the injected spike, creates+the detector, scores it, prints the table above, and **asserts** the result:+exactly one anomaly, at 00:12, score 100, mean 96 — and no normal bucket+flagged. It exits non-zero if any of that regresses, so it doubles as a+smoke test.++## Endpoint cheat-sheet++| Method | Path | Purpose |+|---|---|---|+| `PUT` | `/_ml/anomaly_detectors/{id}` | create/replace a detector |+| `GET` | `/_ml/anomaly_detectors/{id}` | fetch one detector's config |+| `GET` | `/_ml/anomaly_detectors` | list all detectors |+| `DELETE` | `/_ml/anomaly_detectors/{id}` | remove a detector |+| `POST` | `/_ml/anomaly_detectors/{id}/_score` | score now; body may set `query`, `anomaly_threshold` |+| `GET` | `/_cat/ml/anomaly_detectors` | one line per detector |++Config fields: `source_index` (required), `time_field` (default `@timestamp`),+`function` (`count`/`mean`/`min`/`max`/`sum`, default `count`), `field`+(required for everything but `count`), `bucket_span` (default `1h`),+`anomaly_threshold` (z-gate, default `3.0`), `description`. ES-style+`analysis_config.detectors[]` and `data_description.time_field` are also+accepted for drop-in configs.diff --git a/docs/recipes/hybrid-search.md b/docs/recipes/hybrid-search.mdnew file mode 100644index 000000000..785060375--- /dev/null+++ b/docs/recipes/hybrid-search.md@@ -0,0 +1,203 @@+# Hybrid search: keyword + vector in one query++## The problem++You run a help-desk search box. A user types **"reset password"**. Two+things are true at once:++- The best answer might share **no keywords** with the query. An article+ titled *"Regain entry to a locked-out account"* is exactly what they+ need, but BM25 never sees it — there's no lexical overlap.+- A keyword-exact article might be **off-topic**. *"Password complexity+ and rotation policy"* contains "password," so BM25 loves it, but a+ semantic ranker knows it's about policy, not recovery.++Keyword search (BM25) is precise on exact terms and useless on synonyms.+Vector search (kNN) captures meaning but drops exact-term matches whose+embeddings drift. The usual "fix" is to run **two systems** — a lexical+engine and a vector DB — then stitch and re-rank the results in your+application. That's two deployments, two writes on every ingest, and a+fusion step you have to build and maintain yourself.++## Why XERJ++XERJ indexes the text field and the `dense_vector` field in the **same+index**, and fuses BM25 and kNN in a **single `hybrid` query**. One+request goes out, one ranked list comes back — already fused with+Reciprocal Rank Fusion (RRF). No second datastore, no client-side+re-ranking.++## The solution++### 1. One index, two modalities++A `text` field for BM25 and a `dense_vector` field for kNN:++```json+PUT /helpdesk+{+ "mappings": {+ "properties": {+ "title": { "type": "text" },+ "vec": { "type": "dense_vector", "dims": 4, "similarity": "cosine" }+ }+ }+}+```++> The vectors in this recipe are hand-authored 4-dim *topic* vectors so+> the demo is deterministic and you can see exactly why each result+> ranks where it does. In production you fill `vec` with a real embedding+> model — or skip the vectors entirely and use a `semantic_text` field,+> which auto-embeds on ingest. (XERJ's built-in embedder is lexical, good+> for demos and keyword-semantic overlap; point `inference_id` at an+> external `/v1/embeddings` endpoint for neural-quality vectors.)++### 2. The corpus++Each doc has a title (for BM25) and a topic vector (for kNN). Topic axes+are `[auth_recovery, security_policy, cooking, generic]`:++| id | title | vector | keyword "password"? | near query vector? |+|----|-------|--------|:--:|:--:|+| d1 | Reset your password | `[1.00, 0.15, 0, 0.10]` | yes | yes |+| d2 | Regain entry to a locked-out account | `[0.95, 0.10, 0, 0.15]` | **no** | yes |+| d3 | Password complexity and rotation policy | `[0.15, 1.00, 0, 0.10]` | yes | **no** |+| d4 | How to bake sourdough bread | `[0.00, 0.00, 1.0, 0.10]` | no | no |+| d5 | Change your account password | `[0.85, 0.25, 0, 0.20]` | yes | yes |++The user's intent in both modalities: text `"reset password"`, vector+`[1.0, 0.20, 0.0, 0.10]` (an "auth recovery" direction).++### 3. Watch each modality fail on its own++**Pure BM25** — `{"match": {"title": "reset password"}}`:++```+d1 score=0.5754 Reset your password+d3 score=0.2877 Password complexity and rotation policy+d5 score=0.2877 Change your account password+```++It **misses d2** entirely — "Regain entry to a locked-out account" has no+shared keywords, so it can never be a BM25 hit.++**Pure kNN** — `{"knn": {"field": "vec", "query_vector": [...], "k": 3}}`,+showing the 3 nearest:++```+d1 score=0.9994 Reset your password+d2 score=0.9971 Regain entry to a locked-out account+d5 score=0.9942 Change your account password+```++It surfaces d2 (great!) but **misses d3** — the password-policy article's+topic vector points at "security_policy," far from the query, so it falls+below the top-3 cutoff.++### 4. Hybrid recovers both — one query++```json+POST /helpdesk/_search+{+ "size": 5,+ "query": {+ "hybrid": {+ "queries": [+ { "query": { "match": { "title": "reset password" } }, "weight": 1.0 },+ { "query": { "knn": { "field": "vec",+ "query_vector": [1.0, 0.2, 0.0, 0.1],+ "k": 3 } }, "weight": 1.0 }+ ],+ "fusion": "rrf"+ }+ }+}+```++Real response (trimmed to `title`):++```json+{+ "hits": {+ "total": { "value": 5, "relation": "eq" },+ "max_score": 0.03278,+ "hits": [+ { "_id": "d1", "_score": 0.03278, "_source": { "title": "Reset your password" } },+ { "_id": "d5", "_score": 0.03200, "_source": { "title": "Change your account password" } },+ { "_id": "d3", "_score": 0.03150, "_source": { "title": "Password complexity and rotation policy" } },+ { "_id": "d2", "_score": 0.01613, "_source": { "title": "Regain entry to a locked-out account" } },+ { "_id": "d4", "_score": 0.01538, "_source": { "title": "How to bake sourdough bread" } }+ ]+ }+}+```++**d2** (only kNN found it) and **d3** (only BM25 found it) are now *both*+on the page, and **d1** — the doc that matched on *both* keyword and+vector — is still ranked #1. That's the whole point: fusion rewards+agreement between the two signals and still admits the strong+single-signal hits that either method alone would have dropped.++## How the fusion works++`"fusion": "rrf"` uses Reciprocal Rank Fusion: each sub-query contributes+`weight / (k + rank)` for a doc (default `k = 60`), summed across+sub-queries. It ranks by *position*, not raw score, so BM25's `~0.5`+scores and kNN's `~0.99` cosines never have to be normalized onto a+common scale — a persistent headache when you fuse two systems yourself.++Other `fusion` options:++- `"rrf"` or `{"type": "rrf", "k": 40}` — rank-based, the safe default.+- `"linear"` — weighted sum of the raw sub-query scores (use `weight` to+ tilt toward keyword or vector; requires comparable score scales).++Per-query `weight` lets you bias the blend, e.g. `1.5` on the `match`+sub-query to favor exact terms.++## Run it++No dependencies beyond Python 3 stdlib. Point it at a running XERJ:++```bash+# start XERJ (insecure, local data dir) on port 9485+xerj --insecure --data-dir /tmp/xerj-hybrid --config <(printf '[server]\nes_compat_port = 9485\n')++# in another shell+python3 docs/examples/hybrid-search/hybrid_search.py+```++Expected tail:++```+--- assertions ---+PASS BM25 alone MISSES d2 (locked-out account: no shared keywords)+PASS kNN alone MISSES d3 (password policy: topic vector too far)+PASS Hybrid surfaces BOTH d2 and d3 in a single query+PASS d1 (keyword + vector match) still ranks #1 under fusion++BM25 ids : ['d1', 'd3', 'd5']+kNN ids : ['d1', 'd2', 'd5']+Hybrid ids: ['d1', 'd3', 'd5', 'd2', 'd4']++OK+```++## Notes & limits++- **`knn` in a query scores every doc that has the field**, ranked by+ cosine; `size` (or `from`+`size`) is the cutoff the user sees. So `k:3`+ plus `size:3` means "show the 3 nearest." For a hard candidate cap+ independent of `size`, keep `k` small — it's carried on the sub-query.+- **Score scales differ by fusion.** RRF returns small fused scores+ (`~0.01–0.03` here) — compare docs *within* a response, not against+ BM25/cosine absolutes.+- **Tie ordering is not guaranteed** for docs with near-identical fused+ scores (d3 and d5 differ by `0.00005` here), so exact positions among+ ties can vary run to run. Set membership — the point of this recipe —+ is stable.+- **Vector quality is your embeddings' job.** This recipe supplies+ vectors directly for a deterministic demo. For real semantic recall,+ generate `dense_vector` values with a production embedding model, or+ use a `semantic_text` field with an external `inference_id`.diff --git a/docs/recipes/log-analytics.md b/docs/recipes/log-analytics.mdnew file mode 100644index 000000000..b1b989195--- /dev/null+++ b/docs/recipes/log-analytics.md@@ -0,0 +1,207 @@+# Log analytics with XERJ: from raw logs to dashboards++**Goal:** You have a stream of structured application logs — timestamp, level,+service, latency, status — and you need to answer the questions an on-call+engineer actually asks: *Is the error rate climbing? Which service owns the+latency tail? Who's taking the most traffic?* This recipe bulk-ingests a few+hundred synthetic log events and answers all three with plain Elasticsearch+aggregations against a single XERJ instance — no separate metrics store, no+second query language.++Everything below was run end-to-end against a live XERJ. The numbers and JSON+are the real captured output.++## Why XERJ for this++XERJ speaks the Elasticsearch wire protocol, so the aggregation surface you+already know — `date_histogram`, `percentiles`, `terms`, `stats`, and `filter`+sub-aggs — works as-is. You ingest with `_bulk` and query with `_search`. One+Rust engine handles the logs, the aggregations, and (if you want them later)+full-text and vector search on the same documents. No Logstash/metrics-store+split to operate.++## Set up the index++Give the log fields explicit types up front. `@timestamp` must be a `date` for+`date_histogram`, `latency_ms`/`status` numeric for math, and `service`/`level`+`keyword` so they bucket exactly instead of being analyzed.++```bash+curl -XPUT localhost:9483/logs-app -H 'Content-Type: application/json' -d '{+ "mappings": { "properties": {+ "@timestamp": {"type": "date"},+ "level": {"type": "keyword"},+ "service": {"type": "keyword"},+ "latency_ms": {"type": "integer"},+ "status": {"type": "integer"},+ "message": {"type": "text"}+ }}+}'+```++## Ingest with `_bulk`++Logs arrive in volume, so use the bulk API: alternating action + document+NDJSON lines. The example script generates 600 events across 6 hours and 4+services (`checkout`, `search`, `catalog`, `payments`), each with its own+baseline latency and error propensity so the aggregations tell a real story.+One `_bulk` call ships them all:++```+POST /_bulk (Content-Type: application/x-ndjson)+{"index":{"_index":"logs-app"}}+{"@timestamp":"2026-07-06T08:14:03+00:00","level":"INFO","service":"search","latency_ms":61,"status":200,"message":"search handled request status=200"}+{"index":{"_index":"logs-app"}}+{"@timestamp":"2026-07-06T08:22:41+00:00","level":"ERROR","service":"payments","latency_ms":740,"status":503,"message":"payments handled request status=503"}+...+```++Check `errors` in the response is `false`, then `_refresh` to make the docs+searchable. `GET /logs-app/_count` confirms 600 landed.++## Q1 — Error rate over time++Bucket events by hour with `date_histogram`, and inside each bucket count the+ERROR events with a `filter` sub-agg. `errors.doc_count / doc_count` is the+error rate for that hour — exactly the shape a dashboard panel needs.++```json+POST /logs-app/_search+{+ "size": 0,+ "aggs": {+ "per_hour": {+ "date_histogram": { "field": "@timestamp", "fixed_interval": "1h" },+ "aggs": {+ "errors": { "filter": { "term": { "level": "ERROR" } } }+ }+ }+ }+}+```++`size: 0` means "no hits, just the aggregations." Real response (first two+buckets, `took` in ms):++```json+{+ "took": 4,+ "buckets": [+ { "key_as_string": "2026-07-06T08:00:00.000Z", "doc_count": 102,+ "errors": { "doc_count": 7 } },+ { "key_as_string": "2026-07-06T09:00:00.000Z", "doc_count": 101,+ "errors": { "doc_count": 3 } }+ ]+}+```++Rolled up across all six buckets:++```+hour (UTC) total errors err_rate+08:00 102 7 6.9%+09:00 101 3 3.0%+10:00 106 5 4.7%+11:00 91 2 2.2%+12:00 93 2 2.2%+13:00 107 5 4.7%+```++## Q2 — Latency percentiles (p50 / p95 / p99)++Averages hide tail latency; percentiles don't. Ask for fleet-wide percentiles,+and break out p95 per service with a `terms` + `percentiles` sub-agg to see who+owns the tail.++```json+POST /logs-app/_search+{+ "size": 0,+ "aggs": {+ "latency": { "percentiles": { "field": "latency_ms", "percents": [50, 95, 99] } },+ "by_service": {+ "terms": { "field": "service" },+ "aggs": { "p95": { "percentiles": { "field": "latency_ms", "percents": [95] } } }+ }+ }+}+```++The fleet-wide block comes back as ES-style keyed percentiles:++```json+{ "latency": { "values": { "50.0": 56.0, "95.0": 321.15, "99.0": 565.22 } } }+```++Per-service p95 makes the culprits obvious — `checkout` and `payments` carry+the tail while `catalog` and `search` stay flat:++```+fleet-wide p50=56 p95=321 p99=565+service p95 latency+catalog 51+search 79+checkout 447+payments 529+```++## Q3 — Top services by volume (with latency stats)++One `terms` agg ranks services by event count; a `stats` sub-agg attaches+count/avg/min/max latency to each in the same round trip — leaderboard and+latency profile from a single query.++```json+POST /logs-app/_search+{+ "size": 0,+ "aggs": {+ "top_services": {+ "terms": { "field": "service", "size": 10, "order": { "_count": "desc" } },+ "aggs": { "lat": { "stats": { "field": "latency_ms" } } }+ }+ }+}+```++```+service volume avg_ms min max+catalog 190 35 15 118+search 187 51 22 114+checkout 150 167 60 590+payments 73 269 103 958+```++`catalog` and `search` take the most traffic but are cheap; `payments` is+low-volume but has the worst worst-case (958 ms). That's the whole triage+picture from three `_search` calls.++## Run it++```bash+# 1. Boot XERJ (insecure/local) on a port of your choice+printf '[server]\nes_compat_port = 9483\n' > /tmp/xerj.toml+./engine/target/release/xerj --insecure --data-dir /tmp/xerj-logs --config /tmp/xerj.toml &++# 2. Run the recipe (stdlib only — no pip installs)+python3 docs/examples/log-analytics/log_analytics.py http://localhost:9483+```++The script creates the mapping, bulk-ingests 600 events, runs all three+queries, and prints the tables above. It asserts the ingest count and that+`_bulk` reported no errors, so a green run means the aggregations are real.++## Notes and honest limits++- These aggregations are computed on demand per query — XERJ is not running a+ continuous rollup or a background job. That's exactly what you want for+ ad-hoc dashboards and alerting queries; if you need pre-materialized rollups+ at very high cardinality, aggregate on a schedule and index the results.+- `fixed_interval: "1h"` gives evenly spaced buckets; use `calendar_interval`+ (`"day"`, `"month"`) when you want calendar-aware boundaries and DST handling.+- Percentiles use an approximate (t-digest-style) algorithm, matching ES+ semantics — the values are estimates, not exact order statistics, which is+ why p95 reads as `321.15` rather than a whole millisecond.+- Same documents, same engine: because `message` is a `text` field you can add+ `match`/`match_phrase` full-text filters to any of these queries (e.g. only+ aggregate logs matching `"timeout"`) without moving data anywhere.diff --git a/docs/recipes/migrate-from-elasticsearch.md b/docs/recipes/migrate-from-elasticsearch.mdnew file mode 100644index 000000000..1b14f0bbf--- /dev/null+++ b/docs/recipes/migrate-from-elasticsearch.md@@ -0,0 +1,209 @@+# Migrate from Elasticsearch: change the URL, not your app++**Use case:** You have a service that talks to Elasticsearch — it creates indices with mappings, bulk-loads documents, runs `match`/`term`/`range`/`bool` searches, computes `terms` aggregations, and checks `_cat/indices` in ops dashboards. You want to move to XERJ without rewriting any of that.++**The whole migration is one line:** point your client's base URL at XERJ. XERJ speaks the Elasticsearch REST wire protocol on the same `:9200`, so the exact JSON your app already sends works unchanged. No SDK swap, no query rewrites, no re-mapping.++```diff+- ES_URL = "http://elasticsearch:9200"++ ES_URL = "http://xerj:9200"+```++This recipe proves it by running the standard ES request shapes — verbatim — against a live XERJ and showing the real responses.++## Why XERJ for this++- **Same wire, same client.** The official `elasticsearch-py`, `@elastic/elasticsearch`, or a raw `curl` all work because XERJ implements the ES REST API surface. XERJ even reports itself as version `8.13.0` at `GET /`, so version-sniffing clients connect cleanly.+- **Faster on the same workload.** On an identical corpus and machine, XERJ beats a real Elasticsearch 8.13 on both ingest and reads — e.g. bulk ingest ~1.5–1.7× and `match` queries ~2.5× lower p50 latency (see [Benchmarks](#benchmarks)).+- **Honest scope.** XERJ is *wire-compatible*, not a byte-for-byte fork. As of the last full run it passes **1,325 / 1,326** executed cases of the 1,329-case ES REST conformance suite (3 skipped, 1 known fail). Response *shapes* match; some internal details (exact BM25 float scores, segment-merge timing) are XERJ's own. See [Limitations](#limitations-be-honest).++## The working solution++Everything below is a plain `curl` in the exact shape you'd send Elasticsearch. The only variable is `$ES`.++### 1. Create an index with a mapping — `PUT /products`++```bash+curl -s -X PUT "$ES/products" -H 'Content-Type: application/json' -d '{+ "mappings": {+ "properties": {+ "name": { "type": "text" },+ "brand": { "type": "keyword" },+ "price": { "type": "float" },+ "in_stock": { "type": "boolean" }+ }+ }+}'+```++Response — the standard ES acknowledgement:++```json+{ "acknowledged": true, "shards_acknowledged": true, "index": "products" }+```++### 2. Bulk-load documents — `POST /_bulk`++Standard NDJSON: an action line, then a source line, repeated. `Content-Type: application/x-ndjson`, trailing newline.++```bash+curl -s -X POST "$ES/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary '+{"index":{"_index":"products","_id":"1"}}+{"name":"Aluminum water bottle","brand":"Klean","price":24.99,"in_stock":true}+{"index":{"_index":"products","_id":"2"}}+{"name":"Insulated steel water bottle","brand":"Hydro","price":39.95,"in_stock":true}+... 4 more ...+'+```++Response is the ES bulk envelope (`{"took":..,"errors":false,"items":[...]}`) — here, `errors: false, items: 6`.++> As in Elasticsearch, freshly indexed docs become searchable after a refresh. Call `POST /products/_refresh` (or wait for the interval) before the search steps.++### 3. Full-text `match` — `POST /products/_search`++```bash+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{+ "query": { "match": { "name": "water bottle" } },+ "_source": ["name","brand","price"]+}'+```++Real result — BM25-ranked, standard `hits.total.value` / `_score` shape:++```+total: 4+ 0.575 Aluminum water bottle+ 0.575 Insulated steel water bottle+ 0.288 Glass water carafe # matched "water"+ 0.288 Plastic sports bottle # matched "bottle"+```++Note the classic OR semantics of `match`: "Glass water carafe" and "Plastic sports bottle" each match one term and rank below the docs matching both. The full hit shape is exactly ES's:++```json+{+ "hits": {+ "total": { "value": 4, "relation": "eq" },+ "max_score": 1.4168,+ "hits": [+ { "_index": "products", "_id": "1", "_score": 1.4168,+ "_source": { "name": "Aluminum water bottle", "brand": "Klean", "price": 24.99 } }+ ]+ }+}+```++### 4. The classic `bool` combo — `must` + `terms` filter + `range` filter++The bread-and-butter Elasticsearch query: full-text relevance in `must`, non-scoring constraints in `filter`.++```bash+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{+ "query": {+ "bool": {+ "must": [ { "match": { "name": "bottle" } } ],+ "filter": [+ { "terms": { "brand": ["Hydro","Klean"] } },+ { "range": { "price": { "gte": 10, "lte": 40 } } }+ ]+ }+ },+ "_source": ["name","brand","price"]+}'+```++Real result — only the docs that match the text **and** pass both filters:++```+total: 2+ Klean 24.99 Aluminum water bottle+ Hydro 39.95 Insulated steel water bottle+```++(The Klean "Plastic sports bottle" is excluded — its price `9.99` is below the `10` floor.)++### 5. `terms` aggregation — `POST /products/_search` with `size: 0`++```bash+curl -s "$ES/products/_search" -H 'Content-Type: application/json' -d '{+ "size": 0,+ "aggs": { "by_brand": { "terms": { "field": "brand" } } }+}'+```++Real result — the standard `buckets` shape with `doc_count_error_upper_bound` and `sum_other_doc_count`:++```json+{+ "hits": { "total": { "value": 6, "relation": "eq" }, "hits": [] },+ "aggregations": {+ "by_brand": {+ "doc_count_error_upper_bound": 0,+ "sum_other_doc_count": 0,+ "buckets": [+ { "key": "Bodum", "doc_count": 2 },+ { "key": "Hydro", "doc_count": 2 },+ { "key": "Klean", "doc_count": 2 }+ ]+ }+ }+}+```++### 6. Ops check — `GET /_cat/indices?v`++The human-readable `_cat` table your dashboards and shell scripts parse:++```+health status index uuid pri rep docs.count docs.deleted store.size pri.store.size+green open products f5c15d1... 1 0 6 0 10901b 10901b+```++`_cat/indices` also lists XERJ's internal `.xerj_*` system indices (sessions, users, dashboards, etc.), analogous to Elasticsearch's own `.security`/`.kibana` dot-indices. Filter them the same way you would in ES: `GET /_cat/indices/products?v` or a `products*` pattern.++## Run it++The full script is [`docs/examples/migrate-from-elasticsearch/migrate_demo.sh`](../examples/migrate-from-elasticsearch/migrate_demo.sh). Boot XERJ and run every step above end-to-end:++```bash+# 1. Boot XERJ (insecure = no TLS/auth, for local demo)+xerj --insecure --data-dir /tmp/xerj-demo &++# 2. Run the standard-ES demo against it — the ONLY change is the URL+ES=http://localhost:9200 ./migrate_demo.sh+```++It prints each request's result and ends with `All standard Elasticsearch calls succeeded against XERJ`. To A/B against a real cluster, run the identical script with `ES=http://your-elasticsearch:9200` — same script, same output shape.++## What to change in your real app++Almost nothing:++1. **Base URL** → your XERJ host. That's the migration.+2. **Auth/TLS** → XERJ supports them; the demo uses `--insecure` for brevity. Point your client's existing credentials/CA config at XERJ as you would any ES endpoint.+3. **Client library** → keep it. `elasticsearch-py`, the JS client, Logstash/Beats outputs, or raw HTTP all speak this wire.++## Limitations (be honest)++- **Wire-compatible, not byte-identical.** XERJ passes **1,325 / 1,326** executed cases of the 1,329-case ES REST conformance suite (3 skipped, 1 known fail), last full run. Response *structure* matches ES; validate the specific endpoints and query types your app depends on against your own data before cutting over. Per-suite conformance and caveats are tracked in `demo/playbooks/ES_COMPATIBILITY.md`.+- **Relevance scores are XERJ's own BM25.** Ranking order matches ES intent, but exact `_score` float values differ, and can even shift slightly for the *same* query as background segment merges change collection statistics — the same class of internal difference you'd see comparing two ES minor versions. If you assert on exact scores in tests, relax those assertions.+- **A few query/agg types differ or aren't implemented.** XERJ covers the common set (`match`, `term`, `terms`, `range`, `bool`, `match_phrase`, `multi_match`, `terms`/`stats`/`date_histogram`/... aggs). Some ES features are partial or unsupported (e.g. vector-suite conformance is low). Check the compatibility report for your specific needs.++## Benchmarks++Same corpus, same machine, single node, security off — XERJ vs. a real Elasticsearch 8.13:++| Operation | XERJ | ES 8.13 | XERJ advantage |+|---|---|---|---|+| ingest 1M docs, 1 client (docs/s) | 119,031 | 70,320 | **1.69×** |+| ingest 100k docs, 8 clients (docs/s) | 382,301 | 253,961 | **1.51×** |+| `match(model)` read (p50 ms) | 0.57 | 1.45 | **2.53×** |+| `match_all` read (p50 ms) | 0.88 | 1.72 | **1.94×** |+| `query_string` read (p50 ms) | 1.35 | 9.35 | **6.92×** |++Full head-to-head matrix and methodology (identical keep-alive HTTP client applied to both engines): `demo/playbooks/BENCHMARK_VS_ES.md`.++---++*Verified end-to-end against a live XERJ (merged `main` binary, ES-wire port). Every request/response above is real captured output.*diff --git a/docs/recipes/semantic-search-rag.md b/docs/recipes/semantic-search-rag.mdnew file mode 100644index 000000000..d3a0c3e41--- /dev/null+++ b/docs/recipes/semantic-search-rag.md@@ -0,0 +1,188 @@+# Semantic search / RAG retrieval with XERJ — no separate vector DB++**Goal:** you have a knowledge base (support docs, product FAQs, policy pages) and+you want an LLM to answer questions *grounded* in it. The retrieval half of RAG —+"find the passages that mean the same thing as the user's question" — normally+means standing up a separate vector database, an embedding service, and glue code+to keep them in sync with your search index.++With XERJ you don't. You map one field as `semantic_text`, ingest your docs with a+normal `_bulk`, and XERJ **embeds each document on ingest** with a built-in,+zero-config embedder. Then a `semantic` query retrieves by meaning from the same+index you already full-text search. One engine, one wire protocol (Elasticsearch+8.x), no extra moving parts.++This recipe indexes a tiny KB, retrieves passages with paraphrased questions+(different words, same meaning), assembles the context bundle you'd hand an LLM,+and is honest about where the built-in lexical embedder stops and a real neural+endpoint takes over.++---++## 1. Map one field as `semantic_text`++```bash+curl -X PUT localhost:9482/kb -H 'Content-Type: application/json' -d '{+ "mappings": {+ "properties": {+ "title": { "type": "text" },+ "body": { "type": "semantic_text" }+ }+ }+}'+```++`body` is now auto-embedded. On ingest XERJ vectorizes the field's text into a+companion HNSW-indexed vector field (`body_vector`, 384 dims by default, cosine+similarity) — you never call an embedding API yourself. No `dense_vector`+declaration, no dimension bookkeeping, no second datastore.++## 2. Ingest with a plain `_bulk`++Nothing special — the embedding happens under the hood:++```bash+curl -X POST localhost:9482/_bulk -H 'Content-Type: application/x-ndjson' --data-binary '+{"index":{"_index":"kb","_id":"kb-1"}}+{"title":"Rotating an API key","body":"To change your API credentials, open Settings, choose Security, and click Regenerate token. The old token stops working immediately."}+{"index":{"_index":"kb","_id":"kb-2"}}+{"title":"Refund policy","body":"Customers can request their money back within 30 days of purchase. Refunds are issued to the original payment method within five business days."}+'+```++## 3. Retrieve by meaning with the `semantic` query++The user's question rarely uses your doc's exact words. A `match` query on+`"get my money back after buying"` returns **nothing** — none of those tokens are+in the "Refund policy" doc analyzed the same way. The `semantic` query retrieves+it anyway:++```bash+curl localhost:9482/kb/_search -H 'Content-Type: application/json' -d '{+ "size": 2,+ "query": { "semantic": { "field": "body", "query": "how do I regenerate my API token", "k": 2 } }+}'+```++Real response (trimmed):++```json+{+ "hits": {+ "hits": [+ {+ "_id": "kb-1",+ "_score": 0.6625697,+ "_source": {+ "title": "Rotating an API key",+ "body": "To change your API credentials, open Settings, choose Security, and click Regenerate token. The old token stops working immediately.",+ "body_vector": [ 0.0, 0.0, -0.2164437, ... ]+ }+ }+ ]+ }+}+```++`field` is the `semantic_text` field, `query` is the natural-language question,+`k` is how many nearest passages to pull from the vector index. Add a `filter`+(any XERJ query) to constrain candidates, or a `boost` to weight it inside a+`bool`.++> **`_source` note.** The companion `body_vector` rides along in `_source` on+> semantic hits (source filtering is applied on the lexical path but the vector+> field is re-attached on the retrieval path). It's just noise for RAG — read the+> fields you want (`title`, `body`) and ignore it when you build the LLM prompt.++## 4. The RAG context bundle++Retrieval done, RAG is just "concatenate the top passages into the prompt." Pull+`k` hits and format them with their ids so the model can cite them:++```+User question: how do I regenerate my API token++Retrieved context:+[kb-1] Rotating an API key+To change your API credentials, open Settings, choose Security, and click+Regenerate token. The old token stops working immediately.++[kb-5] Rate limits+The API permits 600 requests per minute per key. Exceeding the throttle returns+HTTP 429; back off and retry after the reset window.+```++Feed that block to your LLM as grounding context alongside the question. That's+the whole retrieval side of RAG — no vector DB in the diagram.++## 5. Results — paraphrases land on the right doc++Running five paraphrased questions (none share the doc's full wording) against+the 5-doc KB, each returns the correct passage at rank 1:++```+Q: 'how do I regenerate my API token' -> kb-1 Rotating an API key OK+Q: 'can I get my money back after buying' -> kb-2 Refund policy OK+Q: "I forgot my login and can't sign in" -> kb-3 Resetting a forgotten password OK+Q: 'which file formats are supported for uploads' -> kb-4 Supported file formats OK+Q: 'how many calls per minute am I allowed' -> kb-5 Rate limits OK++RESULT: 5/5 semantic retrievals returned the right doc at rank 1+```++For contrast, the same paraphrase through a keyword `match`:++```+Q: 'get my money back after buying'+ match top -> NO HITS+ semantic top -> kb-2 Refund policy+```++## Honest limitation: the built-in embedder is *lexical*, not neural++The zero-config embedder is a **lexical** model — it maps overlapping and related+vocabulary into vector space. It shines when the question and the answer share+*some* terms (a synonym here, a rephrase there), which covers a lot of real+support/FAQ traffic. It is **not** a neural sentence embedder and won't capture+deep semantics with no lexical bridge.++Concretely, from this same KB:++```+Q: 'how do I rotate my API key'+ semantic top -> kb-5 Rate limits (expected kb-1)+```++The doc is *titled* "Rotating an API key," but only its **body** is embedded, and+the body never uses the word "rotate" — so a pure "rotate" paraphrase misses. Two+honest takeaways:++- **Embed the text that carries the meaning.** If titles matter, index them into+ the `semantic_text` field too (or add a second `semantic_text` field and combine+ with `hybrid`), so the vocabulary the user might use is actually in the vector.+- **For production-grade semantics, wire an external embeddings endpoint.** XERJ+ will call any OpenAI-compatible `/v1/embeddings` service. Set+ `embedding.default_endpoint` in config, or point a field at it in the mapping+ via `inference_endpoint` / `inference_id`. The `semantic` query and the ingest+ path are identical — you just get neural-quality vectors instead of the built-in+ lexical ones. (Anomaly-style continuous re-embedding isn't a thing here;+ embedding happens at ingest time.)++Everything above is `semantic_text` + the built-in embedder — no external service+was configured for this recipe.++---++## Run it++Boot XERJ (Elasticsearch-compatible port shown as 9482 here), then:++```bash+BASE=http://localhost:9482 python3 docs/examples/semantic-search-rag/rag_demo.py+```++The script (stdlib only — `urllib` + `json`, no pip) creates the index, bulk-loads+the KB, runs the five semantic retrievals with assertions, prints the RAG context+bundle, shows the keyword-vs-semantic contrast, demonstrates the lexical-embedder+miss, and exits non-zero if any of the five retrievals regress. Point `BASE` at any+XERJ (or real Elasticsearch with a `semantic_text`/inference setup) to compare.diff --git a/docs/recipes/vector-search-knn.md b/docs/recipes/vector-search-knn.mdnew file mode 100644index 000000000..bcb7c6523--- /dev/null+++ b/docs/recipes/vector-search-knn.md@@ -0,0 +1,251 @@+# Vector similarity search (kNN) with XERJ++**Goal:** you have a set of items — products, articles, images, support+tickets — and each one is represented by an *embedding*: a fixed-length+list of floats. You want the classic operation behind recommendations,+"more like this", dedup, and semantic retrieval: **given one vector, find+the N most similar items.**++XERJ answers this natively. It stores a `dense_vector` field, builds an+HNSW graph over the vectors, and serves the Elasticsearch-8 top-level+`knn` query — so the exact same request bodies (and ES client libraries)+work unchanged. No sidecar vector database, no separate service.++This recipe uses **tiny, hand-made 4-dimensional vectors** so every result+is reproducible and you can see *why* each neighbor came back. In+production your vectors come from a real embedding model (768, 1024,+1536 dims…) — the API is identical.++---++## The toy feature space++Real embeddings are opaque. To make kNN legible, we hand-build vectors in+a 4-axis "feature space":++```+[ citrus , berry , engine , wheels ]+```++Fruits load on the fruit axes and are ~0 on the vehicle axes; vehicles do+the reverse. So a query sitting in the *citrus* corner should return+oranges and lemons first and **never** a truck. That's our recall check.++| id | name | category | in_stock | embedding |+|----|----------------|----------|----------|----------------------------|+| 1 | Navel Orange | fruit | true | `[0.90, 0.10, 0.00, 0.00]` |+| 2 | Meyer Lemon | fruit | true | `[0.95, 0.05, 0.00, 0.00]` |+| 3 | Blood Orange | fruit | **false**| `[0.85, 0.20, 0.00, 0.00]` |+| 4 | Strawberry | fruit | true | `[0.10, 0.92, 0.00, 0.00]` |+| 5 | Blueberry | fruit | true | `[0.05, 0.95, 0.00, 0.00]` |+| 6 | Pickup Truck | vehicle | true | `[0.00, 0.00, 0.90, 0.85]` |+| 7 | Sports Car | vehicle | false | `[0.00, 0.00, 0.95, 0.80]` |+| 8 | Electric Sedan | vehicle | true | `[0.00, 0.05, 0.40, 0.88]` |++---++## 1. Map a `dense_vector` field++```bash+curl -X PUT http://localhost:9200/catalog -H 'Content-Type: application/json' -d '{+ "mappings": {+ "properties": {+ "name": { "type": "text" },+ "category": { "type": "keyword" },+ "in_stock": { "type": "boolean" },+ "embedding": { "type": "dense_vector", "dims": 4, "similarity": "cosine" }+ }+ }+}'+```++Two things matter:++- **`dims`** must equal the length of every vector you index. Indexing a+ wrong-length vector is silently skipped from the ANN scan.+- **`similarity`** defaults to `cosine` (magnitude-invariant — the right+ default for most normalized embeddings). Other options: `l2`,+ `dot_product`. Scores come back as ES-normalized cosine in `[0, 1]`:+ identical direction ≈ `1.0`, orthogonal ≈ `0.5`.++Load the rows with the Bulk API (`{"index":{...}}` + source line pairs),+adding `?refresh=true` so they're searchable immediately.++---++## 2. Plain kNN — "find items most like a citrus fruit"++Put the query vector right in the citrus corner and ask for the 3 nearest:++```bash+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{+ "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 }+}'+```++Real response (trimmed):++```json+{+ "hits": { "hits": [+ { "_id": "1", "_score": 0.9998569, "_source": { "name": "Navel Orange", "category": "fruit" } },+ { "_id": "2", "_score": 0.9997083, "_source": { "name": "Meyer Lemon", "category": "fruit" } },+ { "_id": "3", "_score": 0.9948, "_source": { "name": "Blood Orange", "category": "fruit" } }+ ] }+}+```++The three nearest are **exactly** the three citrus fruits, in order of+closeness, and no vehicle leaked in. `k` is how many neighbors you want+back; `num_candidates` is how many the graph search explores before+picking the top `k` (bigger = higher recall, more work).++Query the *other* corner and the space cleanly separates:++```bash+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{+ "knn": { "field": "embedding", "query_vector": [0.0, 0.0, 0.9, 0.85], "k": 2, "num_candidates": 10 }+}'+# → Pickup Truck (1.0), Sports Car (0.999) — only vehicles+```++---++## 3. kNN + a filter — "…but only what's in stock"++This is the request people actually get wrong. To constrain a kNN search+(price range, in-stock, tenant, language…), **wrap the `knn` in a `bool`+and put the constraint in a sibling `filter` clause.** `bool.filter` runs+as a **pre-filter**: XERJ narrows the candidate set first, *then* ranks+the survivors by similarity.++```bash+curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{+ "size": 3,+ "query": {+ "bool": {+ "must": [ { "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 } } ],+ "filter": [ { "term": { "in_stock": true } } ]+ }+ }+}'+```++Real response (trimmed):++```+1 0.9999 Navel Orange in_stock=True+2 0.9997 Meyer Lemon in_stock=True+4 0.5969 Strawberry in_stock=True+```++Blood Orange (id 3) is the 3rd-closest citrus but is **out of stock**, so+it drops out and a farther *in-stock* item (a berry) takes the slot. The+filter genuinely restricted the pool — it didn't just re-score.++> **Heads-up on the ES-8 shorthand.** Elasticsearch also lets you inline+> the filter *inside* the knn clause: `"knn": { …, "filter": {…} }`. XERJ+> parses that form but **currently ignores the inlined filter** on the+> top-level-`knn` path (it returns unfiltered neighbors). Until that's+> wired up, use the `bool` + `filter` wrapper shown above — it's the+> reliable way to get pre-filtered kNN today, and it's also how XERJ's own+> agent-memory recall builds filtered vector queries internally.++---++## Run it++Boot XERJ (`target/release/xerj --insecure --data-dir ./data`, default+port 9200) and run the self-checking script — stdlib only, no pip:++```bash+python3 docs/examples/vector-search-knn/knn_demo.py+```++It creates the index, bulk-loads the catalog, runs all three searches,+and asserts the recall and filter behavior. Expected tail:++```+=== kNN k=3, query ~citrus ===+ ('1', 0.9999, 'Navel Orange', 'fruit', True)+ ('2', 0.9997, 'Meyer Lemon', 'fruit', True)+ ('3', 0.9948, 'Blood Orange', 'fruit', False)+ OK: top-3 are exactly the citrus fruits, no vehicles leaked in++=== kNN + bool.filter in_stock:true ===+ ('1', 0.9999, 'Navel Orange', 'fruit', True)+ ('2', 0.9997, 'Meyer Lemon', 'fruit', True)+ ('4', 0.5969, 'Strawberry', 'fruit', True)+ OK: out-of-stock Blood Orange excluded; only in-stock neighbors returned++=== kNN k=2, query ~vehicle ===+ ('6', 1.0, 'Pickup Truck', 'vehicle', True)+ ('7', 0.9992, 'Sports Car', 'vehicle', False)+ OK: vehicle query returns only vehicles++ALL ASSERTIONS PASSED+```++(The script targets port 9484; change `BASE` if you booted on 9200.)++---++## How the search works: HNSW++XERJ indexes vectors in an **HNSW** graph (Hierarchical Navigable Small+World) — the same family of ANN index Elasticsearch/Lucene use. Instead+of comparing your query against every stored vector (O(N), fine for our+8-row demo, painful at millions), HNSW builds a layered proximity graph+and greedily hops toward the nearest region, giving roughly O(log N)+search that stays fast as the corpus grows. XERJ builds with+`ef_construction = 200` for high recall; at query time `num_candidates`+controls the search-time breadth (the ANN analog of `ef_search`) — raise+it to trade latency for recall, lower it for speed.++Because it's *approximate*, ANN can in principle miss a true neighbor.+For a corpus this small the result is exact; the knobs above only start+to matter at scale.++---++## Raw kNN vs. `semantic_text`: which do I use?++Both end up doing vector search. The difference is **who makes the+vectors.**++- **Raw `dense_vector` + `knn` (this recipe)** — *you* supply the+ embeddings, on both the ingest side and the query side. Use it when you+ already run an embedding model (OpenAI, a local sentence-transformer,+ a CLIP image encoder, your own recommender) and just need fast storage+ and nearest-neighbor retrieval. Maximum control; works for any modality,+ not just text.++- **`semantic_text` field + `semantic` query** — XERJ embeds text+ **for** you at ingest time and at query time, so you send plain strings+ and never touch a vector. Great when your data is text and you want+ semantic search with zero embedding plumbing.+ *Honest caveat:* the built-in embedder is **lexical, not neural** —+ excellent for demos and keyword-semantic overlap, but for+ production-grade semantic quality point XERJ at an external+ OpenAI-compatible `/v1/embeddings` endpoint+ (`embedding.default_endpoint`), after which both ingest and query use+ that higher-quality model.++Rule of thumb: **already have embeddings → raw `knn`. Have text and want+it embedded for you → `semantic_text`.** Need to blend keyword relevance+(BM25) with vector similarity in one request? Reach for the `hybrid`+query.++---++## Gotchas++- **`dims` must match your vectors exactly** — a length mismatch means the+ doc is skipped from the scan, not an error.+- **Filter a kNN with `bool` + `filter`**, not the inlined `knn.filter`+ (currently ignored on the top-level path — see §3).+- **`k` vs `num_candidates`**: `k` = results returned; `num_candidates` =+ how many the graph explores first. `num_candidates >= k`; raise it if+ recall matters.+- Scores are **ES-normalized cosine in `[0,1]`** (≈1.0 identical, ≈0.5+ orthogonal), not raw cosine in `[-1,1]`.