AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
recipes: real AI apps exercising XERJ on real workloads (zero-dependency)
xerj-orgde2d3dcfca1b
diff --git a/recipes/README.md b/recipes/README.mdnew file mode 100644index 000000000..1b1a3a305--- /dev/null+++ b/recipes/README.md@@ -0,0 +1,68 @@+# XERJ recipes — real AI apps, real workloads++Four small, dependency-free programs that use XERJ the way an AI+application or agent actually would. Every recipe is a single Python file+using only the standard library (`urllib`), runs against a stock XERJ+binary, and uses **real data that ships in this repo** — no mock servers,+no API keys, no `pip install`.++They all work **fully offline**: XERJ has a built-in deterministic+embedder, so `semantic_text` fields, `semantic` queries, and agent memory+all function with zero external services. Where a recipe can *optionally*+use a real neural embedding model or an LLM, it says so and reads the+endpoint from an environment variable.++## Run them++```bash+# 1. get a binary (or `cargo build --release -p xerj-server`)+curl -fsSL https://xerj.org/get | sh++# 2. start XERJ+xerj --insecure --data-dir ./data &++# 3. run any recipe+python3 recipes/semantic_search.py+python3 recipes/rag_app.py+python3 recipes/memory_agent.py # run twice — memory persists+python3 recipes/log_anomaly.py+```++Every recipe honors `XERJ_URL` (default `http://localhost:9200`).++## The recipes++| File | What it shows | Real workload |+|---|---|---|+| [`semantic_search.py`](semantic_search.py) | BM25 vs semantic vs hybrid (RRF) retrieval on one `semantic_text` field | `demo/data/ai_kb.ndjson` — 40 hand-written AI-engineering articles |+| [`rag_app.py`](rag_app.py) | A full RAG pipeline: HTML→text→chunk→index→hybrid-retrieve→answer, with a retrieval eval and citations | XERJ's own docs (`landing/docs/*.html`) — ~96 chunks over 23 pages |+| [`memory_agent.py`](memory_agent.py) | A triage agent with **persistent** long-term memory via the `/_memory` API: recall similar past incidents, decide, remember | Real OpenSSH attack capture (`engine/demo-data/ssh_one.ndjson`, logpai/loghub) |+| [`log_anomaly.py`](log_anomaly.py) | Statistical anomaly detection (`_ml`) finding brute-force spikes over a moving baseline, then attributing the worst hour | Same real SSH capture (655k events) |++## What each one exercises in the engine++- **semantic_text auto-embedding** — documents embed at ingest with no+ config; `semantic` queries embed the question the same way.+- **hybrid queries** — `bool`/`match` fused with `semantic`/`knn` via+ Reciprocal Rank Fusion in one query tree.+- **Agent-Memory API** — `POST /_memory/{ns}`, `_recall`, backed by a+ real per-namespace index, so memory survives restarts.+- **`_ml` anomaly detection** — `PUT /_ml/anomaly_detectors/{id}` ++ `_score`: date-histogram buckets scored against a moving mean/stddev.+- **ordinary ES aggregations** — `terms`/`range` used alongside the AI+ features (attribution in the anomaly recipe).++## Honest notes++- The built-in embedder is **lexical** (feature-hashing over word ++ character n-grams, L2-normalized), not neural. It captures vocabulary+ overlap well and is perfect for offline demos and CI, but it won't+ match true synonyms. For production semantics, point XERJ at an+ OpenAI-compatible embeddings endpoint (`[embedding]` in `xerj.toml`) —+ the recipes need no change.+- `memory_agent.py` recalls with **text (BM25) relevance** scores+ (unbounded, higher = better). For pure vector recall, pass a `vector`+ to `_recall`.+- The SSH capture is real production honeypot-style traffic, so it is+ dominated by attacks — which is exactly why the anomaly detector lights+ up. That is the real signal, not an artifact.diff --git a/recipes/log_anomaly.py b/recipes/log_anomaly.pynew file mode 100644index 000000000..531a1ba3e--- /dev/null+++ b/recipes/log_anomaly.py@@ -0,0 +1,117 @@+#!/usr/bin/env python3+"""Find real brute-force attacks with statistical anomaly detection.++Ingests the public OpenSSH log capture from logpai/loghub that ships in+this repo (655k real events from a production server) and uses XERJ's+`_ml` anomaly-detection API to find the hours where failed-authentication+volume spikes far above its moving baseline — i.e. the actual brute-force+attacks in the capture. No training, no external services: the detector+is a deterministic moving mean/stddev baseline scored per time bucket.++Run it:+ xerj --insecure --data-dir ./data & # start XERJ+ python3 recipes/log_anomaly.py++Optional env:+ XERJ_URL (default http://localhost:9200)+"""++import datetime+import json+import os+import pathlib+import urllib.error+import urllib.request++XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")+LOGS = pathlib.Path(__file__).resolve().parent.parent / "engine" / "demo-data" / "ssh_one.ndjson"+INDEX = "ssh-auth"+DETECTOR = "ssh-bruteforce"+BATCH = 5000+++def call(method: str, path: str, body=None):+ data = None+ headers = {}+ if body is not None:+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()+ headers["Content-Type"] = (+ "application/x-ndjson" if isinstance(body, str) else "application/json"+ )+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)+ with urllib.request.urlopen(req) as resp:+ return json.loads(resp.read())+++def ingest():+ try:+ call("DELETE", f"/{INDEX}")+ except urllib.error.HTTPError:+ pass+ call("PUT", f"/{INDEX}", {+ "mappings": {"properties": {+ "@timestamp": {"type": "date"},+ "event": {"type": "keyword"},+ "src_ip": {"type": "keyword"},+ "user": {"type": "keyword"},+ }}+ })+ batch, n = [], 0+ with LOGS.open() as f:+ for line in f:+ ev = json.loads(line)+ # keep only auth outcomes; drop the free-text message for speed+ if ev.get("event") not in {"failed_password", "invalid_user", "auth_failure"}:+ continue+ batch.append(json.dumps({"index": {"_index": INDEX}}))+ batch.append(json.dumps({k: ev.get(k) for k in ("@timestamp", "event", "src_ip", "user")}))+ if len(batch) >= 2 * BATCH:+ call("POST", "/_bulk", "\n".join(batch) + "\n")+ n += BATCH+ batch = []+ if batch:+ call("POST", "/_bulk?refresh=true", "\n".join(batch) + "\n")+ n += len(batch) // 2+ else:+ call("POST", f"/{INDEX}/_refresh")+ print(f"indexed {call('GET', f'/{INDEX}/_count')['count']} real failed-auth events\n")+++def detect():+ # A detector is just: which index, which clock field, what metric,+ # what bucket size, and how many stddevs count as anomalous.+ call("PUT", f"/_ml/anomaly_detectors/{DETECTOR}", {+ "source_index": INDEX,+ "time_field": "@timestamp",+ "function": "count",+ "bucket_span": "1h",+ "anomaly_threshold": 3.0,+ "description": "failed ssh auth volume vs moving baseline",+ })+ result = call("POST", f"/_ml/anomaly_detectors/{DETECTOR}/_score")+ records = [r for r in result.get("records", []) if r.get("is_anomaly")]+ records.sort(key=lambda r: -r["anomaly_score"])++ print(f"{len(records)} anomalous hours found (threshold: 3.0σ over moving baseline)\n")+ print(" score when failed auths expected")+ for r in records[:10]:+ ts = datetime.datetime.fromtimestamp(r["timestamp"] / 1000, datetime.timezone.utc)+ print(f" {r['anomaly_score']:5.1f} {ts:%Y-%m-%d %H:%M} UTC {r['actual']:>10.0f} {r['expected']:>8.1f}")++ if records:+ # Who caused the worst hour? Ordinary ES aggregation on the bucket.+ worst = records[0]+ start = worst["timestamp"]+ top = call("POST", f"/{INDEX}/_search", {+ "size": 0,+ "query": {"range": {"@timestamp": {"gte": start, "lt": start + 3_600_000}}},+ "aggs": {"attackers": {"terms": {"field": "src_ip", "size": 3}}},+ })["aggregations"]["attackers"]["buckets"]+ print("\nworst hour, top attacking IPs:")+ for b in top:+ print(f" {b['key']:<18} {b['doc_count']} attempts")+++if __name__ == "__main__":+ ingest()+ detect()diff --git a/recipes/memory_agent.py b/recipes/memory_agent.pynew file mode 100644index 000000000..016e0514c--- /dev/null+++ b/recipes/memory_agent.py@@ -0,0 +1,138 @@+#!/usr/bin/env python3+"""A security-triage agent with persistent semantic memory — zero dependencies.++A small SOC (security operations) agent that works through *real* attack+traffic — the public OpenSSH log capture from logpai/loghub that ships in+this repo — and remembers what it decided, using XERJ's Agent-Memory API+(`/_memory/{namespace}`) as its long-term memory:++ 1. It groups the raw sshd events into per-attacker incidents.+ 2. Before deciding, it RECALLS similar past incidents. Text recall is+ relevance-ranked (BM25); pass a `vector` to `_recall` instead for+ fully semantic matching over stored embeddings.+ 3. It decides (block / rate-limit / watch), explains whether memory+ changed the decision, and STORES the outcome as a new memory.++Memory is backed by a real index, so it survives restarts: run this+script twice and the second run recalls the first run's decisions.++Run it:+ xerj --insecure --data-dir ./data & # start XERJ+ python3 recipes/memory_agent.py # first run: cold memory+ python3 recipes/memory_agent.py # second run: warm memory++Optional env:+ XERJ_URL (default http://localhost:9200)+"""++import collections+import json+import os+import pathlib+import urllib.request++XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")+LOGS = pathlib.Path(__file__).resolve().parent.parent / "engine" / "demo-data" / "ssh_one.ndjson"+NAMESPACE = "soc-agent"+EVENTS_TO_READ = 80_000 # first slice of the real capture+INCIDENTS_TO_TRIAGE = 8+++def call(method: str, path: str, body=None):+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(+ f"{XERJ}{path}", data=data,+ headers={"Content-Type": "application/json"} if body else {},+ method=method,+ )+ with urllib.request.urlopen(req) as resp:+ return json.loads(resp.read())+++# ── 1. Build incidents from the real sshd capture ───────────────────────────+def load_incidents():+ attackers = collections.defaultdict(lambda: {"events": 0, "users": set(), "kinds": set()})+ with LOGS.open() as f:+ for i, line in enumerate(f):+ if i >= EVENTS_TO_READ:+ break+ ev = json.loads(line)+ ip = ev.get("src_ip")+ if not ip or ev.get("event") not in {+ "invalid_user", "failed_password", "possible_break_in", "auth_failure",+ }:+ continue+ a = attackers[ip]+ a["events"] += 1+ a["kinds"].add(ev["event"])+ if ev.get("user"):+ a["users"].add(ev["user"])++ ranked = sorted(attackers.items(), key=lambda kv: -kv[1]["events"])+ incidents = []+ for ip, a in ranked[:INCIDENTS_TO_TRIAGE]:+ users = sorted(a["users"])+ incidents.append({+ "src_ip": ip,+ "attempts": a["events"],+ "kinds": sorted(a["kinds"]),+ "summary": (+ f"host {ip} produced {a['events']} failed ssh authentications "+ f"({', '.join(sorted(a['kinds']))}) trying accounts like "+ f"{', '.join(users[:5]) or 'root'}"+ ),+ })+ return incidents+++# ── 2. The agent: recall → decide → remember ────────────────────────────────+def decide(incident, recalled):+ # Memory-informed path: a strongly relevant past incident that was+ # blocked means this is a known pattern — block immediately.+ # (Recall scores are BM25 relevance — unbounded, higher is better.)+ for m in recalled:+ if m["score"] > 1.0 and m["metadata"].get("action") == "block":+ return "block", f"recalled similar blocked incident ({m['metadata']['src_ip']})"+ # Cold-start policy on the raw numbers.+ if incident["attempts"] >= 200:+ return "block", "fresh analysis: sustained brute force"+ if incident["attempts"] >= 30:+ return "rate-limit", "fresh analysis: repeated failures"+ return "watch", "fresh analysis: low volume"+++def main():+ before = call("GET", f"/_memory/{NAMESPACE}").get("count", 0)+ print(f"agent memory `{NAMESPACE}`: {before} memories at start\n")++ for inc in load_incidents():+ recalled = call("POST", f"/_memory/{NAMESPACE}/_recall", {+ "query": inc["summary"],+ "k": 3,+ }).get("hits", [])++ action, why = decide(inc, recalled)+ print(f"incident {inc['src_ip']:<16} {inc['attempts']:>5} attempts → {action:<10} ({why})")+ for m in recalled[:1]:+ print(f" remembered: [{m['score']:.2f}] {m['text'][:90]}…")++ call("POST", f"/_memory/{NAMESPACE}", {+ "text": f"{inc['summary']} — decision: {action}",+ "metadata": {+ "src_ip": inc["src_ip"],+ "action": action,+ "attempts": inc["attempts"],+ },+ })++ after = call("GET", f"/_memory/{NAMESPACE}").get("count", 0)+ print(f"\nagent memory `{NAMESPACE}`: {after} memories at end "+ f"(+{after - before} this run)")+ if before:+ print("second run used warm memory — decisions above cite recalled incidents")+ else:+ print("run me again: the next run recalls these memories semantically")+++if __name__ == "__main__":+ main()diff --git a/recipes/rag_app.py b/recipes/rag_app.pynew file mode 100644index 000000000..7eb7c719d--- /dev/null+++ b/recipes/rag_app.py@@ -0,0 +1,204 @@+#!/usr/bin/env python3+"""A complete RAG pipeline over real documentation — zero dependencies.++Retrieval-augmented generation against XERJ's own manual: strips the HTML+docs that ship in this repo (landing/docs/*.html), chunks them with+overlap, indexes every chunk into a `semantic_text` field (XERJ embeds at+ingest — no API keys needed), and answers questions with hybrid retrieval+(BM25 + semantic, RRF-fused) plus citations.++Works fully offline. If you export an OpenAI-compatible chat endpoint the+final answer is generated by an LLM grounded in the retrieved chunks;+otherwise the recipe prints the retrieved passages (extractive mode).++Run it:+ xerj --insecure --data-dir ./data & # start XERJ+ python3 recipes/rag_app.py # index + eval + one answer+ python3 recipes/rag_app.py "your question here"++Optional env:+ XERJ_URL (default http://localhost:9200)+ LLM_ENDPOINT OpenAI-compatible /v1/chat/completions URL+ LLM_API_KEY bearer token for LLM_ENDPOINT+ LLM_MODEL (default gpt-4o-mini)+"""++import html.parser+import json+import os+import pathlib+import re+import sys+import urllib.error+import urllib.request++XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")+DOCS = pathlib.Path(__file__).resolve().parent.parent / "landing" / "docs"+INDEX = "xerj-manual"+CHUNK_CHARS, OVERLAP = 900, 150+++def call(method: str, path: str, body=None):+ data = None+ headers = {}+ if body is not None:+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()+ headers["Content-Type"] = (+ "application/x-ndjson" if isinstance(body, str) else "application/json"+ )+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)+ with urllib.request.urlopen(req) as resp:+ return json.loads(resp.read())+++# ── 1. Turn the shipped HTML docs into plain text ───────────────────────────+class TextExtractor(html.parser.HTMLParser):+ """Keep visible text from <main>, drop scripts/styles/nav."""++ SKIP = {"script", "style", "nav", "footer", "header"}++ def __init__(self):+ super().__init__()+ self.parts, self._skip_depth, self.title = [], 0, ""+ self._in_title = False++ def handle_starttag(self, tag, attrs):+ if tag in self.SKIP:+ self._skip_depth += 1+ if tag == "title":+ self._in_title = True++ def handle_endtag(self, tag):+ if tag in self.SKIP and self._skip_depth:+ self._skip_depth -= 1+ if tag == "title":+ self._in_title = False++ def handle_data(self, data):+ if self._in_title:+ self.title += data+ elif not self._skip_depth:+ self.parts.append(data)+++def page_text(path: pathlib.Path):+ p = TextExtractor()+ p.feed(path.read_text(errors="ignore"))+ text = re.sub(r"\s+", " ", " ".join(p.parts)).strip()+ title = p.title.split("·")[-1].strip() or path.stem+ return title, text+++def chunk(text: str):+ """Overlapping chunks, preferring sentence boundaries — the same idea+ XERJ's internal ingest chunker uses."""+ out, start = [], 0+ while start < len(text):+ end = min(start + CHUNK_CHARS, len(text))+ if end < len(text):+ cut = text.rfind(". ", start, end)+ if cut > start + CHUNK_CHARS // 2:+ end = cut + 1+ out.append(text[start:end].strip())+ if end >= len(text):+ break+ start = end - OVERLAP+ return out+++def build_index():+ try:+ call("DELETE", f"/{INDEX}")+ except urllib.error.HTTPError:+ pass+ call("PUT", f"/{INDEX}", {+ "mappings": {+ "properties": {+ "page": {"type": "keyword"},+ "title": {"type": "keyword"},+ "text": {"type": "semantic_text", "dimensions": 384},+ }+ }+ })+ lines, n_pages = [], 0+ for f in sorted(DOCS.glob("*.html")):+ title, text = page_text(f)+ if len(text) < 200:+ continue+ n_pages += 1+ for i, c in enumerate(chunk(text)):+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": f"{f.stem}-{i}"}}))+ lines.append(json.dumps({"page": f.stem, "title": title, "text": c}))+ call("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")+ print(f"indexed {len(lines) // 2} chunks from {n_pages} real docs pages\n")+++# ── 2. Retrieval: hybrid = BM25 + semantic, fused with RRF ──────────────────+def retrieve(question: str, k: int = 4):+ hits = call("POST", f"/{INDEX}/_search", {+ "size": k,+ "query": {"hybrid": {"queries": [+ {"query": {"match": {"text": question}}, "weight": 1.0},+ {"query": {"semantic": {"field": "text", "query": question, "k": 20}}, "weight": 1.0},+ ]}},+ })["hits"]["hits"]+ return [(h["_source"]["page"], h["_source"]["title"], h["_source"]["text"]) for h in hits]+++# ── 3. Generation: LLM if configured, extractive otherwise ─────────────────+def answer(question: str):+ ctx = retrieve(question)+ print(f"Q: {question}\n")+ endpoint = os.environ.get("LLM_ENDPOINT")+ if endpoint:+ prompt = "Answer strictly from the context.\n\n" + "\n---\n".join(t for _, _, t in ctx)+ req = urllib.request.Request(+ endpoint,+ data=json.dumps({+ "model": os.environ.get("LLM_MODEL", "gpt-4o-mini"),+ "messages": [+ {"role": "system", "content": prompt},+ {"role": "user", "content": question},+ ],+ }).encode(),+ headers={+ "Content-Type": "application/json",+ "Authorization": f"Bearer {os.environ.get('LLM_API_KEY', '')}",+ },+ )+ with urllib.request.urlopen(req) as resp:+ print(json.loads(resp.read())["choices"][0]["message"]["content"], "\n")+ else:+ print("(no LLM_ENDPOINT set — extractive mode)\n")+ for page, title, text in ctx[:2]:+ print(f" {text[:300]}…\n")+ print("sources:", ", ".join(dict.fromkeys(f"docs/{p}.html" for p, _, _ in ctx)), "\n")+++# ── 4. A tiny honest retrieval eval on real questions ──────────────────────+EVAL = [+ ("how do I run xerj as a systemd service?", "install"),+ ("which quantization modes exist for vectors?", "vectors"),+ ("how do I migrate an existing Elasticsearch index?", "migration-from-es"),+ ("what does the bool query's filter clause do?", "queries"),+ ("how do I take a backup to S3?", "backup-restore"),+]+++def run_eval():+ ok = 0+ for q, expect in EVAL:+ pages = [p for p, _, _ in retrieve(q, k=3)]+ hit = expect in pages+ ok += hit+ print(f" {'✓' if hit else '✗'} {q} → {pages}")+ print(f"\nretrieval hit@3: {ok}/{len(EVAL)}\n")+++if __name__ == "__main__":+ build_index()+ if len(sys.argv) > 1:+ answer(" ".join(sys.argv[1:]))+ else:+ run_eval()+ answer("how do I keep memory usage down on a vector-heavy index?")diff --git a/recipes/run_all.sh b/recipes/run_all.shnew file mode 100755index 000000000..2915a12bf--- /dev/null+++ b/recipes/run_all.sh@@ -0,0 +1,42 @@+#!/bin/sh+# Smoke-run every recipe against a throwaway XERJ instance.+# Usage: recipes/run_all.sh [path-to-xerj-binary]+# Exits non-zero if any recipe fails. Used by CI and as a one-shot demo.+set -eu++ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)+XERJ_BIN="${1:-$ROOT/engine/target/release/xerj}"+[ -x "$XERJ_BIN" ] || XERJ_BIN=$(command -v xerj) || {+ echo "error: no xerj binary (build with: cargo build --release -p xerj-server)" >&2+ exit 1+}++DATA=$(mktemp -d)+# Isolated ports (config-only; the binary has no port flags) so a smoke+# run never collides with a dev instance on 8080/9200/9300.+ESPORT=9219+cat >"$DATA/xerj.toml" <<EOF+[server]+data_dir = "$DATA"+rest_port = 8319+grpc_port = 9419+es_compat_port = $ESPORT+EOF+export XERJ_URL="http://127.0.0.1:${ESPORT}"++"$XERJ_BIN" --insecure --config "$DATA/xerj.toml" >"$DATA/server.log" 2>&1 &+SRV=$!+trap 'kill "$SRV" 2>/dev/null || true; rm -rf "$DATA"' EXIT INT TERM++i=0+until curl -fsS "$XERJ_URL/" >/dev/null 2>&1; do+ i=$((i + 1)); [ "$i" -gt 60 ] && { echo "server never came up"; cat "$DATA/server.log"; exit 1; }+ sleep 0.5+done++rc=0+for r in semantic_search rag_app memory_agent log_anomaly; do+ printf '\n═══ %s ═══\n' "$r"+ if python3 "$ROOT/recipes/$r.py"; then :; else echo "FAILED: $r"; rc=1; fi+done+exit "$rc"diff --git a/recipes/semantic_search.py b/recipes/semantic_search.pynew file mode 100644index 000000000..c815f5761--- /dev/null+++ b/recipes/semantic_search.py@@ -0,0 +1,106 @@+#!/usr/bin/env python3+"""Semantic + hybrid search over a real knowledge base — zero dependencies.++Indexes the 40-article AI engineering KB that ships in this repo+(demo/data/ai_kb.ndjson) into a `semantic_text` field. XERJ embeds every+document at ingest with its built-in embedder (no external services, no+API keys), then answers the same question three ways so you can see what+each retrieval mode is good at:++ 1. match — classic BM25 keyword search+ 2. semantic — embed the query, k-NN over the document vectors+ 3. hybrid — BM25 + semantic fused with Reciprocal Rank Fusion++Run it:+ xerj --insecure --data-dir ./data & # start XERJ+ python3 recipes/semantic_search.py++Optional env:+ XERJ_URL (default http://localhost:9200)++To use a real neural embedding model instead of the built-in lexical+embedder, point XERJ at any OpenAI-compatible endpoint in xerj.toml:+ [embedding]+ default_endpoint = "https://api.openai.com/v1/embeddings"+ default_model = "text-embedding-3-small"+"""++import json+import os+import pathlib+import urllib.error+import urllib.request++XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")+KB = pathlib.Path(__file__).resolve().parent.parent / "demo" / "data" / "ai_kb.ndjson"+INDEX = "ai-kb"+++def call(method: str, path: str, body=None):+ """One tiny HTTP helper instead of a client library."""+ data = None+ headers = {}+ if body is not None:+ data = body.encode() if isinstance(body, str) else json.dumps(body).encode()+ headers["Content-Type"] = (+ "application/x-ndjson" if isinstance(body, str) else "application/json"+ )+ req = urllib.request.Request(f"{XERJ}{path}", data=data, headers=headers, method=method)+ with urllib.request.urlopen(req) as resp:+ return json.loads(resp.read())+++def main():+ # ── 1. Fresh index: `content` is semantic_text, so XERJ embeds it at+ # ingest time and `semantic` queries embed the question the same+ # way. `category`/`source` stay keyword for exact filtering.+ try:+ call("DELETE", f"/{INDEX}")+ except urllib.error.HTTPError:+ pass # didn't exist yet+ call("PUT", f"/{INDEX}", {+ "mappings": {+ "properties": {+ "title": {"type": "text"},+ "content": {"type": "semantic_text", "dimensions": 384},+ "category": {"type": "keyword"},+ "source": {"type": "keyword"},+ }+ }+ })++ # ── 2. Bulk-ingest the real KB (40 hand-written articles).+ lines = []+ for raw in KB.read_text().splitlines():+ doc = json.loads(raw)+ doc.pop("embedding", None) # let semantic_text embed the content itself+ lines.append(json.dumps({"index": {"_index": INDEX, "_id": str(doc["id"])}}))+ lines.append(json.dumps(doc))+ call("POST", "/_bulk?refresh=true", "\n".join(lines) + "\n")+ total = call("GET", f"/{INDEX}/_count")["count"]+ print(f"indexed {total} KB articles into `{INDEX}`\n")++ # ── 3. Ask the same question three ways.+ question = "how do I keep an LLM agent's context from overflowing?"++ modes = {+ "match (BM25)": {"match": {"content": question}},+ "semantic": {"semantic": {"field": "content", "query": question, "k": 5}},+ "hybrid (RRF)": {"hybrid": {"queries": [+ {"query": {"match": {"content": question}}, "weight": 1.0},+ {"query": {"semantic": {"field": "content", "query": question, "k": 10}}, "weight": 1.0},+ ]}},+ }++ print(f"Q: {question}\n")+ for name, query in modes.items():+ hits = call("POST", f"/{INDEX}/_search", {"query": query, "size": 3})["hits"]["hits"]+ print(f"── {name}")+ for h in hits:+ src = h["_source"]+ print(f" {h['_score']:6.3f} [{src['category']:>9}] {src['title']}")+ print()+++if __name__ == "__main__":+ main()