Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

recipes: real AI apps exercising XERJ on real workloads (zero-dependency)

xerj-orgde2d3dcfca1b
diff --git a/recipes/README.md b/recipes/README.md
new file mode 100644
index 000000000..1b1a3a305
--- /dev/null
+++ b/recipes/README.md
@@ -0,0 +1,68 @@
+# XERJ recipes — real AI apps, real workloads
+
+Four small, dependency-free programs that use XERJ the way an AI
+application or agent actually would. Every recipe is a single Python file
+using only the standard library (`urllib`), runs against a stock XERJ
+binary, and uses **real data that ships in this repo** — no mock servers,
+no API keys, no `pip install`.
+
+They all work **fully offline**: XERJ has a built-in deterministic
+embedder, so `semantic_text` fields, `semantic` queries, and agent memory
+all function with zero external services. Where a recipe can *optionally*
+use a real neural embedding model or an LLM, it says so and reads the
+endpoint from an environment variable.
+
+## Run them
+
+```bash
+# 1. get a binary (or `cargo build --release -p xerj-server`)
+curl -fsSL https://xerj.org/get | sh
+
+# 2. start XERJ
+xerj --insecure --data-dir ./data &
+
+# 3. run any recipe
+python3 recipes/semantic_search.py
+python3 recipes/rag_app.py
+python3 recipes/memory_agent.py # run twice — memory persists
+python3 recipes/log_anomaly.py
+```
+
+Every recipe honors `XERJ_URL` (default `http://localhost:9200`).
+
+## The recipes
+
+| File | What it shows | Real workload |
+|---|---|---|
+| [`semantic_search.py`](semantic_search.py) | BM25 vs semantic vs hybrid (RRF) retrieval on one `semantic_text` field | `demo/data/ai_kb.ndjson` — 40 hand-written AI-engineering articles |
+| [`rag_app.py`](rag_app.py) | A full RAG pipeline: HTML→text→chunk→index→hybrid-retrieve→answer, with a retrieval eval and citations | XERJ's own docs (`landing/docs/*.html`) — ~96 chunks over 23 pages |
+| [`memory_agent.py`](memory_agent.py) | A triage agent with **persistent** long-term memory via the `/_memory` API: recall similar past incidents, decide, remember | Real OpenSSH attack capture (`engine/demo-data/ssh_one.ndjson`, logpai/loghub) |
+| [`log_anomaly.py`](log_anomaly.py) | Statistical anomaly detection (`_ml`) finding brute-force spikes over a moving baseline, then attributing the worst hour | Same real SSH capture (655k events) |
+
+## What each one exercises in the engine
+
+- **semantic_text auto-embedding** — documents embed at ingest with no
+ config; `semantic` queries embed the question the same way.
+- **hybrid queries** — `bool`/`match` fused with `semantic`/`knn` via
+ Reciprocal Rank Fusion in one query tree.
+- **Agent-Memory API** — `POST /_memory/{ns}`, `_recall`, backed by a
+ real per-namespace index, so memory survives restarts.
+- **`_ml` anomaly detection** — `PUT /_ml/anomaly_detectors/{id}` +
+ `_score`: date-histogram buckets scored against a moving mean/stddev.
+- **ordinary ES aggregations** — `terms`/`range` used alongside the AI
+ features (attribution in the anomaly recipe).
+
+## Honest notes
+
+- The built-in embedder is **lexical** (feature-hashing over word +
+ character n-grams, L2-normalized), not neural. It captures vocabulary
+ overlap well and is perfect for offline demos and CI, but it won't
+ match true synonyms. For production semantics, point XERJ at an
+ OpenAI-compatible embeddings endpoint (`[embedding]` in `xerj.toml`) —
+ the recipes need no change.
+- `memory_agent.py` recalls with **text (BM25) relevance** scores
+ (unbounded, higher = better). For pure vector recall, pass a `vector`
+ to `_recall`.
+- The SSH capture is real production honeypot-style traffic, so it is
+ dominated by attacks — which is exactly why the anomaly detector lights
+ up. That is the real signal, not an artifact.
diff --git a/recipes/log_anomaly.py b/recipes/log_anomaly.py
new file mode 100644
index 000000000..531a1ba3e
--- /dev/null
+++ b/recipes/log_anomaly.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Find real brute-force attacks with statistical anomaly detection.
+
+Ingests the public OpenSSH log capture from logpai/loghub that ships in
+this repo (655k real events from a production server) and uses XERJ's
+`_ml` anomaly-detection API to find the hours where failed-authentication
+volume spikes far above its moving baseline — i.e. the actual brute-force
+attacks in the capture. No training, no external services: the detector
+is a deterministic moving mean/stddev baseline scored per time bucket.
+
+Run it:
+ xerj --insecure --data-dir ./data & # start XERJ
+ python3 recipes/log_anomaly.py
+
+Optional env:
+ XERJ_URL (default http://localhost:9200)
+"""
+
+import datetime
+import json
+import os
+import pathlib
+import urllib.error
+import urllib.request
+
+XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
+LOGS = pathlib.Path(__file__).resolve().parent.parent / "engine" / "demo-data" / "ssh_one.ndjson"
+INDEX = "ssh-auth"
+DETECTOR = "ssh-bruteforce"
+BATCH = 5000
+
+
+def call(method: str, path: str, body=None):
+ data = None
+ headers = {}
+ if body is not None:
+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()
+ headers["Content-Type"] = (
+ "application/x-ndjson" if isinstance(body, str) else "application/json"
+ )
+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)
+ with urllib.request.urlopen(req) as resp:
+ return json.loads(resp.read())
+
+
+def ingest():
+ try:
+ call("DELETE", f"/{INDEX}")
+ except urllib.error.HTTPError:
+ pass
+ call("PUT", f"/{INDEX}", {
+ "mappings": {"properties": {
+ "@timestamp": {"type": "date"},
+ "event": {"type": "keyword"},
+ "src_ip": {"type": "keyword"},
+ "user": {"type": "keyword"},
+ }}
+ })
+ batch, n = [], 0
+ with LOGS.open() as f:
+ for line in f:
+ ev = json.loads(line)
+ # keep only auth outcomes; drop the free-text message for speed
+ if ev.get("event") not in {"failed_password", "invalid_user", "auth_failure"}:
+ continue
+ batch.append(json.dumps({"index": {"_index": INDEX}}))
+ batch.append(json.dumps({k: ev.get(k) for k in ("@timestamp", "event", "src_ip", "user")}))
+ if len(batch) >= 2 * BATCH:
+ call("POST", "/_bulk", "\n".join(batch) + "\n")
+ n += BATCH
+ batch = []
+ if batch:
+ call("POST", "/_bulk?refresh=true", "\n".join(batch) + "\n")
+ n += len(batch) // 2
+ else:
+ call("POST", f"/{INDEX}/_refresh")
+ print(f"indexed {call('GET', f'/{INDEX}/_count')['count']} real failed-auth events\n")
+
+
+def detect():
+ # A detector is just: which index, which clock field, what metric,
+ # what bucket size, and how many stddevs count as anomalous.
+ call("PUT", f"/_ml/anomaly_detectors/{DETECTOR}", {
+ "source_index": INDEX,
+ "time_field": "@timestamp",
+ "function": "count",
+ "bucket_span": "1h",
+ "anomaly_threshold": 3.0,
+ "description": "failed ssh auth volume vs moving baseline",
+ })
+ result = call("POST", f"/_ml/anomaly_detectors/{DETECTOR}/_score")
+ records = [r for r in result.get("records", []) if r.get("is_anomaly")]
+ records.sort(key=lambda r: -r["anomaly_score"])
+
+ print(f"{len(records)} anomalous hours found (threshold: 3.0σ over moving baseline)\n")
+ print(" score when failed auths expected")
+ for r in records[:10]:
+ ts = datetime.datetime.fromtimestamp(r["timestamp"] / 1000, datetime.timezone.utc)
+ print(f" {r['anomaly_score']:5.1f} {ts:%Y-%m-%d %H:%M} UTC {r['actual']:>10.0f} {r['expected']:>8.1f}")
+
+ if records:
+ # Who caused the worst hour? Ordinary ES aggregation on the bucket.
+ worst = records[0]
+ start = worst["timestamp"]
+ top = call("POST", f"/{INDEX}/_search", {
+ "size": 0,
+ "query": {"range": {"@timestamp": {"gte": start, "lt": start + 3_600_000}}},
+ "aggs": {"attackers": {"terms": {"field": "src_ip", "size": 3}}},
+ })["aggregations"]["attackers"]["buckets"]
+ print("\nworst hour, top attacking IPs:")
+ for b in top:
+ print(f" {b['key']:<18} {b['doc_count']} attempts")
+
+
+if __name__ == "__main__":
+ ingest()
+ detect()
diff --git a/recipes/memory_agent.py b/recipes/memory_agent.py
new file mode 100644
index 000000000..016e0514c
--- /dev/null
+++ b/recipes/memory_agent.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+"""A security-triage agent with persistent semantic memory — zero dependencies.
+
+A small SOC (security operations) agent that works through *real* attack
+traffic — the public OpenSSH log capture from logpai/loghub that ships in
+this repo — and remembers what it decided, using XERJ's Agent-Memory API
+(`/_memory/{namespace}`) as its long-term memory:
+
+ 1. It groups the raw sshd events into per-attacker incidents.
+ 2. Before deciding, it RECALLS similar past incidents. Text recall is
+ relevance-ranked (BM25); pass a `vector` to `_recall` instead for
+ fully semantic matching over stored embeddings.
+ 3. It decides (block / rate-limit / watch), explains whether memory
+ changed the decision, and STORES the outcome as a new memory.
+
+Memory is backed by a real index, so it survives restarts: run this
+script twice and the second run recalls the first run's decisions.
+
+Run it:
+ xerj --insecure --data-dir ./data & # start XERJ
+ python3 recipes/memory_agent.py # first run: cold memory
+ python3 recipes/memory_agent.py # second run: warm memory
+
+Optional env:
+ XERJ_URL (default http://localhost:9200)
+"""
+
+import collections
+import json
+import os
+import pathlib
+import urllib.request
+
+XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
+LOGS = pathlib.Path(__file__).resolve().parent.parent / "engine" / "demo-data" / "ssh_one.ndjson"
+NAMESPACE = "soc-agent"
+EVENTS_TO_READ = 80_000 # first slice of the real capture
+INCIDENTS_TO_TRIAGE = 8
+
+
+def call(method: str, path: str, body=None):
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(
+ f"{XERJ}{path}", data=data,
+ headers={"Content-Type": "application/json"} if body else {},
+ method=method,
+ )
+ with urllib.request.urlopen(req) as resp:
+ return json.loads(resp.read())
+
+
+# ── 1. Build incidents from the real sshd capture ───────────────────────────
+def load_incidents():
+ attackers = collections.defaultdict(lambda: {"events": 0, "users": set(), "kinds": set()})
+ with LOGS.open() as f:
+ for i, line in enumerate(f):
+ if i >= EVENTS_TO_READ:
+ break
+ ev = json.loads(line)
+ ip = ev.get("src_ip")
+ if not ip or ev.get("event") not in {
+ "invalid_user", "failed_password", "possible_break_in", "auth_failure",
+ }:
+ continue
+ a = attackers[ip]
+ a["events"] += 1
+ a["kinds"].add(ev["event"])
+ if ev.get("user"):
+ a["users"].add(ev["user"])
+
+ ranked = sorted(attackers.items(), key=lambda kv: -kv[1]["events"])
+ incidents = []
+ for ip, a in ranked[:INCIDENTS_TO_TRIAGE]:
+ users = sorted(a["users"])
+ incidents.append({
+ "src_ip": ip,
+ "attempts": a["events"],
+ "kinds": sorted(a["kinds"]),
+ "summary": (
+ f"host {ip} produced {a['events']} failed ssh authentications "
+ f"({', '.join(sorted(a['kinds']))}) trying accounts like "
+ f"{', '.join(users[:5]) or 'root'}"
+ ),
+ })
+ return incidents
+
+
+# ── 2. The agent: recall → decide → remember ────────────────────────────────
+def decide(incident, recalled):
+ # Memory-informed path: a strongly relevant past incident that was
+ # blocked means this is a known pattern — block immediately.
+ # (Recall scores are BM25 relevance — unbounded, higher is better.)
+ for m in recalled:
+ if m["score"] > 1.0 and m["metadata"].get("action") == "block":
+ return "block", f"recalled similar blocked incident ({m['metadata']['src_ip']})"
+ # Cold-start policy on the raw numbers.
+ if incident["attempts"] >= 200:
+ return "block", "fresh analysis: sustained brute force"
+ if incident["attempts"] >= 30:
+ return "rate-limit", "fresh analysis: repeated failures"
+ return "watch", "fresh analysis: low volume"
+
+
+def main():
+ before = call("GET", f"/_memory/{NAMESPACE}").get("count", 0)
+ print(f"agent memory `{NAMESPACE}`: {before} memories at start\n")
+
+ for inc in load_incidents():
+ recalled = call("POST", f"/_memory/{NAMESPACE}/_recall", {
+ "query": inc["summary"],
+ "k": 3,
+ }).get("hits", [])
+
+ action, why = decide(inc, recalled)
+ print(f"incident {inc['src_ip']:<16} {inc['attempts']:>5} attempts → {action:<10} ({why})")
+ for m in recalled[:1]:
+ print(f" remembered: [{m['score']:.2f}] {m['text'][:90]}…")
+
+ call("POST", f"/_memory/{NAMESPACE}", {
+ "text": f"{inc['summary']} — decision: {action}",
+ "metadata": {
+ "src_ip": inc["src_ip"],
+ "action": action,
+ "attempts": inc["attempts"],
+ },
+ })
+
+ after = call("GET", f"/_memory/{NAMESPACE}").get("count", 0)
+ print(f"\nagent memory `{NAMESPACE}`: {after} memories at end "
+ f"(+{after - before} this run)")
+ if before:
+ print("second run used warm memory — decisions above cite recalled incidents")
+ else:
+ print("run me again: the next run recalls these memories semantically")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/recipes/rag_app.py b/recipes/rag_app.py
new file mode 100644
index 000000000..7eb7c719d
--- /dev/null
+++ b/recipes/rag_app.py
@@ -0,0 +1,204 @@
+#!/usr/bin/env python3
+"""A complete RAG pipeline over real documentation — zero dependencies.
+
+Retrieval-augmented generation against XERJ's own manual: strips the HTML
+docs that ship in this repo (landing/docs/*.html), chunks them with
+overlap, indexes every chunk into a `semantic_text` field (XERJ embeds at
+ingest — no API keys needed), and answers questions with hybrid retrieval
+(BM25 + semantic, RRF-fused) plus citations.
+
+Works fully offline. If you export an OpenAI-compatible chat endpoint the
+final answer is generated by an LLM grounded in the retrieved chunks;
+otherwise the recipe prints the retrieved passages (extractive mode).
+
+Run it:
+ xerj --insecure --data-dir ./data & # start XERJ
+ python3 recipes/rag_app.py # index + eval + one answer
+ python3 recipes/rag_app.py "your question here"
+
+Optional env:
+ XERJ_URL (default http://localhost:9200)
+ LLM_ENDPOINT OpenAI-compatible /v1/chat/completions URL
+ LLM_API_KEY bearer token for LLM_ENDPOINT
+ LLM_MODEL (default gpt-4o-mini)
+"""
+
+import html.parser
+import json
+import os
+import pathlib
+import re
+import sys
+import urllib.error
+import urllib.request
+
+XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
+DOCS = pathlib.Path(__file__).resolve().parent.parent / "landing" / "docs"
+INDEX = "xerj-manual"
+CHUNK_CHARS, OVERLAP = 900, 150
+
+
+def call(method: str, path: str, body=None):
+ data = None
+ headers = {}
+ if body is not None:
+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()
+ headers["Content-Type"] = (
+ "application/x-ndjson" if isinstance(body, str) else "application/json"
+ )
+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)
+ with urllib.request.urlopen(req) as resp:
+ return json.loads(resp.read())
+
+
+# ── 1. Turn the shipped HTML docs into plain text ───────────────────────────
+class TextExtractor(html.parser.HTMLParser):
+ """Keep visible text from <main>, drop scripts/styles/nav."""
+
+ SKIP = {"script", "style", "nav", "footer", "header"}
+
+ def __init__(self):
+ super().__init__()
+ self.parts, self._skip_depth, self.title = [], 0, ""
+ self._in_title = False
+
+ def handle_starttag(self, tag, attrs):
+ if tag in self.SKIP:
+ self._skip_depth += 1
+ if tag == "title":
+ self._in_title = True
+
+ def handle_endtag(self, tag):
+ if tag in self.SKIP and self._skip_depth:
+ self._skip_depth -= 1
+ if tag == "title":
+ self._in_title = False
+
+ def handle_data(self, data):
+ if self._in_title:
+ self.title += data
+ elif not self._skip_depth:
+ self.parts.append(data)
+
+
+def page_text(path: pathlib.Path):
+ p = TextExtractor()
+ p.feed(path.read_text(errors="ignore"))
+ text = re.sub(r"\s+", " ", " ".join(p.parts)).strip()
+ title = p.title.split("·")[-1].strip() or path.stem
+ return title, text
+
+
+def chunk(text: str):
+ """Overlapping chunks, preferring sentence boundaries — the same idea
+ XERJ's internal ingest chunker uses."""
+ out, start = [], 0
+ while start < len(text):
+ end = min(start + CHUNK_CHARS, len(text))
+ if end < len(text):
+ cut = text.rfind(". ", start, end)
+ if cut > start + CHUNK_CHARS // 2:
+ end = cut + 1
+ out.append(text[start:end].strip())
+ if end >= len(text):
+ break
+ start = end - OVERLAP
+ return out
+
+
+def build_index():
+ try:
+ call("DELETE", f"/{INDEX}")
+ except urllib.error.HTTPError:
+ pass
+ call("PUT", f"/{INDEX}", {
+ "mappings": {
+ "properties": {
+ "page": {"type": "keyword"},
+ "title": {"type": "keyword"},
+ "text": {"type": "semantic_text", "dimensions": 384},
+ }
+ }
+ })
+ lines, n_pages = [], 0
+ for f in sorted(DOCS.glob("*.html")):
+ title, text = page_text(f)
+ if len(text) < 200:
+ continue
+ n_pages += 1
+ for i, c in enumerate(chunk(text)):
+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": f"{f.stem}-{i}"}}))
+ lines.append(json.dumps({"page": f.stem, "title": title, "text": c}))
+ call("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")
+ print(f"indexed {len(lines) // 2} chunks from {n_pages} real docs pages\n")
+
+
+# ── 2. Retrieval: hybrid = BM25 + semantic, fused with RRF ──────────────────
+def retrieve(question: str, k: int = 4):
+ hits = call("POST", f"/{INDEX}/_search", {
+ "size": k,
+ "query": {"hybrid": {"queries": [
+ {"query": {"match": {"text": question}}, "weight": 1.0},
+ {"query": {"semantic": {"field": "text", "query": question, "k": 20}}, "weight": 1.0},
+ ]}},
+ })["hits"]["hits"]
+ return [(h["_source"]["page"], h["_source"]["title"], h["_source"]["text"]) for h in hits]
+
+
+# ── 3. Generation: LLM if configured, extractive otherwise ─────────────────
+def answer(question: str):
+ ctx = retrieve(question)
+ print(f"Q: {question}\n")
+ endpoint = os.environ.get("LLM_ENDPOINT")
+ if endpoint:
+ prompt = "Answer strictly from the context.\n\n" + "\n---\n".join(t for _, _, t in ctx)
+ req = urllib.request.Request(
+ endpoint,
+ data=json.dumps({
+ "model": os.environ.get("LLM_MODEL", "gpt-4o-mini"),
+ "messages": [
+ {"role": "system", "content": prompt},
+ {"role": "user", "content": question},
+ ],
+ }).encode(),
+ headers={
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {os.environ.get('LLM_API_KEY', '')}",
+ },
+ )
+ with urllib.request.urlopen(req) as resp:
+ print(json.loads(resp.read())["choices"][0]["message"]["content"], "\n")
+ else:
+ print("(no LLM_ENDPOINT set — extractive mode)\n")
+ for page, title, text in ctx[:2]:
+ print(f" {text[:300]}…\n")
+ print("sources:", ", ".join(dict.fromkeys(f"docs/{p}.html" for p, _, _ in ctx)), "\n")
+
+
+# ── 4. A tiny honest retrieval eval on real questions ──────────────────────
+EVAL = [
+ ("how do I run xerj as a systemd service?", "install"),
+ ("which quantization modes exist for vectors?", "vectors"),
+ ("how do I migrate an existing Elasticsearch index?", "migration-from-es"),
+ ("what does the bool query's filter clause do?", "queries"),
+ ("how do I take a backup to S3?", "backup-restore"),
+]
+
+
+def run_eval():
+ ok = 0
+ for q, expect in EVAL:
+ pages = [p for p, _, _ in retrieve(q, k=3)]
+ hit = expect in pages
+ ok += hit
+ print(f" {'✓' if hit else '✗'} {q} → {pages}")
+ print(f"\nretrieval hit@3: {ok}/{len(EVAL)}\n")
+
+
+if __name__ == "__main__":
+ build_index()
+ if len(sys.argv) > 1:
+ answer(" ".join(sys.argv[1:]))
+ else:
+ run_eval()
+ answer("how do I keep memory usage down on a vector-heavy index?")
diff --git a/recipes/run_all.sh b/recipes/run_all.sh
new file mode 100755
index 000000000..2915a12bf
--- /dev/null
+++ b/recipes/run_all.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+# Smoke-run every recipe against a throwaway XERJ instance.
+# Usage: recipes/run_all.sh [path-to-xerj-binary]
+# Exits non-zero if any recipe fails. Used by CI and as a one-shot demo.
+set -eu
+
+ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
+XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"
+[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {
+ echo "error: no xerj binary (build with: cargo build --release -p xerj-server)" >&2
+ exit 1
+}
+
+DATA=$(mktemp -d)
+# Isolated ports (config-only; the binary has no port flags) so a smoke
+# run never collides with a dev instance on 8080/9200/9300.
+ESPORT=9219
+cat >"$DATA/xerj.toml" <<EOF
+[server]
+data_dir = "$DATA"
+rest_port = 8319
+grpc_port = 9419
+es_compat_port = $ESPORT
+EOF
+export XERJ_URL="http://127.0.0.1:${ESPORT}"
+
+"$XERJ_BIN" --insecure --config "$DATA/xerj.toml" >"$DATA/server.log" 2>&1 &
+SRV=$!
+trap 'kill "$SRV" 2>/dev/null || true; rm -rf "$DATA"' EXIT INT TERM
+
+i=0
+until curl -fsS "$XERJ_URL/" >/dev/null 2>&1; do
+ i=$((i + 1)); [ "$i" -gt 60 ] && { echo "server never came up"; cat "$DATA/server.log"; exit 1; }
+ sleep 0.5
+done
+
+rc=0
+for r in semantic_search rag_app memory_agent log_anomaly; do
+ printf '\n═══ %s ═══\n' "$r"
+ if python3 "$ROOT/recipes/$r.py"; then :; else echo "FAILED: $r"; rc=1; fi
+done
+exit "$rc"
diff --git a/recipes/semantic_search.py b/recipes/semantic_search.py
new file mode 100644
index 000000000..c815f5761
--- /dev/null
+++ b/recipes/semantic_search.py
@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""Semantic + hybrid search over a real knowledge base — zero dependencies.
+
+Indexes the 40-article AI engineering KB that ships in this repo
+(demo/data/ai_kb.ndjson) into a `semantic_text` field. XERJ embeds every
+document at ingest with its built-in embedder (no external services, no
+API keys), then answers the same question three ways so you can see what
+each retrieval mode is good at:
+
+ 1. match — classic BM25 keyword search
+ 2. semantic — embed the query, k-NN over the document vectors
+ 3. hybrid — BM25 + semantic fused with Reciprocal Rank Fusion
+
+Run it:
+ xerj --insecure --data-dir ./data & # start XERJ
+ python3 recipes/semantic_search.py
+
+Optional env:
+ XERJ_URL (default http://localhost:9200)
+
+To use a real neural embedding model instead of the built-in lexical
+embedder, point XERJ at any OpenAI-compatible endpoint in xerj.toml:
+ [embedding]
+ default_endpoint = "https://api.openai.com/v1/embeddings"
+ default_model = "text-embedding-3-small"
+"""
+
+import json
+import os
+import pathlib
+import urllib.error
+import urllib.request
+
+XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
+KB = pathlib.Path(__file__).resolve().parent.parent / "demo" / "data" / "ai_kb.ndjson"
+INDEX = "ai-kb"
+
+
+def call(method: str, path: str, body=None):
+ """One tiny HTTP helper instead of a client library."""
+ data = None
+ headers = {}
+ if body is not None:
+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()
+ headers["Content-Type"] = (
+ "application/x-ndjson" if isinstance(body, str) else "application/json"
+ )
+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)
+ with urllib.request.urlopen(req) as resp:
+ return json.loads(resp.read())
+
+
+def main():
+ # ── 1. Fresh index: `content` is semantic_text, so XERJ embeds it at
+ # ingest time and `semantic` queries embed the question the same
+ # way. `category`/`source` stay keyword for exact filtering.
+ try:
+ call("DELETE", f"/{INDEX}")
+ except urllib.error.HTTPError:
+ pass # didn't exist yet
+ call("PUT", f"/{INDEX}", {
+ "mappings": {
+ "properties": {
+ "title": {"type": "text"},
+ "content": {"type": "semantic_text", "dimensions": 384},
+ "category": {"type": "keyword"},
+ "source": {"type": "keyword"},
+ }
+ }
+ })
+
+ # ── 2. Bulk-ingest the real KB (40 hand-written articles).
+ lines = []
+ for raw in KB.read_text().splitlines():
+ doc = json.loads(raw)
+ doc.pop("embedding", None) # let semantic_text embed the content itself
+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": str(doc["id"])}}))
+ lines.append(json.dumps(doc))
+ call("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")
+ total = call("GET", f"/{INDEX}/_count")["count"]
+ print(f"indexed {total} KB articles into `{INDEX}`\n")
+
+ # ── 3. Ask the same question three ways.
+ question = "how do I keep an LLM agent's context from overflowing?"
+
+ modes = {
+ "match (BM25)": {"match": {"content": question}},
+ "semantic": {"semantic": {"field": "content", "query": question, "k": 5}},
+ "hybrid (RRF)": {"hybrid": {"queries": [
+ {"query": {"match": {"content": question}}, "weight": 1.0},
+ {"query": {"semantic": {"field": "content", "query": question, "k": 10}}, "weight": 1.0},
+ ]}},
+ }
+
+ print(f"Q: {question}\n")
+ for name, query in modes.items():
+ hits = call("POST", f"/{INDEX}/_search", {"query": query, "size": 3})["hits"]["hits"]
+ print(f"── {name}")
+ for h in hits:
+ src = h["_source"]
+ print(f" {h['_score']:6.3f} [{src['category']:>9}] {src['title']}")
+ print()
+
+
+if __name__ == "__main__":
+ main()