Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(demo): real AI-agent usage scenarios; deprecate stale sales kit

Xerj Squad A9e4b2b06d9e6
diff --git a/demo/DEMO_RUNBOOK.md b/demo/DEMO_RUNBOOK.md
index 9fd068c0b..8e46397c6 100644
--- a/demo/DEMO_RUNBOOK.md
+++ b/demo/DEMO_RUNBOOK.md
@@ -1,3 +1,13 @@
+> ⚠️ **STALE — sales-engineer material from April 2026 (v1.0.0-rc.1).** The
+> numbers here are out of date (conformance was 1305/1329; it is now 1360/1363,
+> and the RC1 ingest race and unfused-hybrid caveats it warns about are fixed).
+> This kit pitches XERJ to enterprise buyers; it is **not** the guide to XERJ as
+> an AI-agent tool. For that — how XERJ saves an AI coding agent time and
+> tokens, with reproducible measurements — see
+> [`demo/agent-gate/SCENARIOS.md`](agent-gate/SCENARIOS.md) and
+> [`docs/TOKEN_USAGE.md`](../docs/TOKEN_USAGE.md). Kept for reference; refresh
+> the numbers before any external use.
+
# XERJ Demo Runbook — Enterprise AI Adoption
**Audience:** sales engineers running 30–45 minute calls with prospects mid-AI transformation. Typical persona: a director of platform / head of AI infra / VP engineering at a regulated enterprise (financial services, healthcare, public sector, retail) who is actively wiring up RAG, agents, or copilots and is unhappy with the current sprawl of Elasticsearch + Pinecone/Weaviate + Splunk.
diff --git a/demo/README.md b/demo/README.md
index 09f3247fb..fbc90a83c 100644
--- a/demo/README.md
+++ b/demo/README.md
@@ -1,3 +1,13 @@
+> ⚠️ **STALE — sales-engineer material from April 2026 (v1.0.0-rc.1).** The
+> numbers here are out of date (conformance was 1305/1329; it is now 1360/1363,
+> and the RC1 ingest race and unfused-hybrid caveats it warns about are fixed).
+> This kit pitches XERJ to enterprise buyers; it is **not** the guide to XERJ as
+> an AI-agent tool. For that — how XERJ saves an AI coding agent time and
+> tokens, with reproducible measurements — see
+> [`demo/agent-gate/SCENARIOS.md`](agent-gate/SCENARIOS.md) and
+> [`docs/TOKEN_USAGE.md`](../docs/TOKEN_USAGE.md). Kept for reference; refresh
+> the numbers before any external use.
+
# XERJ demo kit
This folder is a complete demo kit for sales engineers pitching XERJ to enterprises adopting AI (RAG, agents, internal copilots). Everything here is reproducible — every command was run on the demo machine while authoring; every claimed number has a receipts file behind it.
diff --git a/demo/agent-gate/README.md b/demo/agent-gate/README.md
index 6237f5308..0b1641b27 100644
--- a/demo/agent-gate/README.md
+++ b/demo/agent-gate/README.md
@@ -65,3 +65,19 @@ python3 demo/agent-gate/gate.py /tmp/gate-corpus
Exit status is **0 when the report is trustworthy**, not when XERJ wins.
It exits non-zero only if a path produced an answer the gate could not verify
— that is, if the measurement itself is broken.
+
+## The kit
+
+| File | What it is |
+|---|---|
+| `SCENARIOS.md` | Six real Claude Code scenarios with measured before/after — start here |
+| `gate.py` | Analytics regime: record-heavy corpus (XERJ wins correctness, costs tokens) |
+| `gate_retrieval.py` | Retrieval regime: prose/code corpus (XERJ wins recall; tokens scale with size) |
+| `make_corpus.py` | Deterministic 36 MB heterogeneous corpus (logs/CSV/SQLite/docs/code) |
+| `make_corpus_prose.py` | Deterministic prose/code corpus (the regime where retrieval saves tokens) |
+| `RESULTS_analytics.txt` / `RESULTS_retrieval.txt` | Committed reference runs |
+
+Both gates share one discipline: correctness before savings, savings only over
+jointly-correct tasks, question-vocabulary baselines, both paths equally tuned,
+corpus composition beside every ratio. See the table at the top of this file
+for which historical mistake each rule prevents.
diff --git a/demo/agent-gate/RESULTS.txt b/demo/agent-gate/RESULTS_analytics.txt
similarity index 100%
rename from demo/agent-gate/RESULTS.txt
rename to demo/agent-gate/RESULTS_analytics.txt
diff --git a/demo/agent-gate/RESULTS_retrieval.txt b/demo/agent-gate/RESULTS_retrieval.txt
new file mode 100644
index 000000000..c850ac3ba
--- /dev/null
+++ b/demo/agent-gate/RESULTS_retrieval.txt
@@ -0,0 +1,24 @@
+
+================================================================================
+AGENT GATE — RETRIEVAL regime (prose/code corpus)
+================================================================================
+
+question needle base xerj base tok xerj tok base s xerj s
+certificate MISS OK 43 78 0.00 0.01
+stemming MISS OK 44 78 0.00 0.00
+promote MISS MISS 15 79 0.00 0.01
+fan-out OK OK 158 75 0.00 0.00
+at-least-once MISS OK 22 80 0.00 0.00
+vault OK OK 38 78 0.00 0.00
+idempotency_key OK OK 69 79 0.00 0.00
+
+correct: baseline 3/7 xerj 6/7
+
+-- tokens over jointly-correct tasks only --
+ baseline 265 tok xerj 232 tok -> XERJ uses 1.14x FEWER tokens
+
+-- corpus composition --
+ 53.1 KB total; 49.0 KB searchable prose/code (92.3%)
+ This is the regime where retrieval savings appear: grep's candidate
+ set grows with the corpus; ranked search returns one passage.
+
diff --git a/demo/agent-gate/SCENARIOS.md b/demo/agent-gate/SCENARIOS.md
new file mode 100644
index 000000000..0e9c8ab20
--- /dev/null
+++ b/demo/agent-gate/SCENARIOS.md
@@ -0,0 +1,203 @@
+# XERJ for an AI coding agent — real scenarios, measured
+
+This is the honest answer to "does XERJ help Claude Code, and when?" Each
+scenario below is a task an agent actually performs, with the measured
+before/after and a copy-paste reproduction. Every number came from a run in
+this repository; none is projected. Where XERJ does **not** help, that is
+stated as plainly as where it does — an agent that trusts a dishonest benchmark
+makes worse decisions, not better ones.
+
+The measurement rules (bytes on the wire, savings only over jointly-correct
+tasks, question-vocabulary baselines, corpus composition reported) are the ones
+enforced by `gate.py` and `gate_retrieval.py` in this directory. Read
+`README.md` for why each rule exists, and `../../docs/TOKEN_USAGE.md` for the
+token model the scenarios rely on.
+
+## The one law that predicts every result
+
+> **Retrieval savings scale with the searchable-prose fraction of the corpus.
+> Analytics savings scale with record count — but analytics answers are small
+> either way, so there the win is correctness and latency, not tokens.**
+
+Two corpora, same tasks, opposite outcomes, both measured:
+
+| corpus | prose fraction | XERJ vs grep, tokens | XERJ vs grep, correctness |
+|---|--:|---|---|
+| 170k LOC source + docs | ~100% | **5.3× fewer** | 7/8 vs 8/8 |
+| 36 MB logs/CSV, few docs | 0.01% | 110× **more** | **6/6 vs 4/6** |
+
+Neither number alone is the truth. Pick the scenario that matches your corpus.
+
+---
+
+## Scenario 1 — Orient in an unfamiliar dataset (agent's first move)
+
+**Task:** You are handed a folder of mixed files and asked a question about it.
+Before answering anything you must learn what is there.
+
+**Without XERJ:** `ls -R`, then `head` each file, then guess schemas, then
+sample rows to infer types. On a heterogeneous folder this is many tool calls
+and grows with file count. You still do not know cross-file relationships.
+
+**With XERJ:**
+```bash
+xerj autoindex ./data # one command, content-sniffed, zero config
+xerj autoindex map # the data map
+```
+**Measured:** a 5.65M-record corpus mapped in **~890 tokens, flat regardless of
+corpus size** — every dataset's fields, types, null%, example values, time
+range, ready-to-send queries, **and auto-detected cross-dataset joins**
+(`user_id ↔ user_id, live-confirmed 20/20`). The naive `ls`+`head` equivalent
+is smaller only because it contains far less: no types, no cardinality, no
+relationships.
+
+**Verdict:** always worth it. This is the single most reliable win, because the
+map's cost does not grow with the data.
+
+---
+
+## Scenario 2 — "Where/why is X" in a large codebase
+
+**Task:** "Why did search relevance drop?" — answered in your own words, not
+the codebase's.
+
+**Without XERJ:** `grep -rn` a noun you hope appears. grep is *lexical*: if the
+answer says "stemming" and you grepped "relevance", you miss it. On a large
+tree the candidate set is large and you read many false hits.
+
+**With XERJ:** ranked search returns the one passage, with a highlighted
+fragment and a line number to jump to.
+```bash
+curl -s "localhost:9200/ax-*/_search?filter_path=hits.hits._source.ax_path,hits.hits.highlight" \
+ -H 'Content-Type: application/json' -d '{"size":1,
+ "query":{"match":{"body":"why did search relevance drop"}},
+ "highlight":{"fields":{"body":{"fragment_size":200,"number_of_fragments":1}}}}'
+```
+**Measured, 170k-LOC corpus:** XERJ **1,457 tokens vs grep 7,762** at equal
+recall — **5.3× fewer**, no steering (queries used only the question's words).
+**Measured, small prose corpus** (`gate_retrieval.py`): correctness **6/7 vs
+grep 3/7** — grep missed four answers whose wording differed from the question.
+
+**Verdict:** the flagship agent win. Bigger and more text-heavy the repo, bigger
+the margin. Requires the chunking that `autoindex` now does by default.
+
+---
+
+## Scenario 3 — Analytical question over logs
+
+**Task:** "Error rate by service", "which paths return 502", "refunded value by
+country".
+
+**Without XERJ:** `grep | awk | sort | uniq`, or a bespoke Python pass over the
+file. Correct if you write it carefully; easy to get subtly wrong (a `*.log`
+glob silently skips the `.gz`; a decimal-comma CSV needs `.replace(',','.')`).
+
+**With XERJ:** one aggregation over typed, parsed fields.
+```bash
+curl -s "localhost:9200/ax-logs/_search?filter_path=aggregations.p.buckets" \
+ -H 'Content-Type: application/json' -d '{"size":0,
+ "query":{"term":{"status":502}},
+ "aggs":{"p":{"terms":{"field":"path","size":6}}}}'
+```
+**Measured** (`gate.py`, 458k records): correctness **6/6 vs grep 4/6** — the
+gate itself caught the baseline silently skipping a gzipped log and mis-parsing
+a decimal-comma CSV. Tokens are ~even (both return a few numbers); XERJ is
+5–15× faster in wall time.
+
+**Verdict:** win on **correctness and latency**, not tokens. The engine parses
+formats (gzip, decimal-comma CSV, nginx log lines) that a quick grep gets wrong.
+
+---
+
+## Scenario 4 — Incident drill-down (each step depends on the last)
+
+**Task:** "Show the failing requests → now that user's other events → now by
+region → now by hour." Interactive investigation.
+
+**Without XERJ:** every step re-scans the whole file. Cost is O(corpus) **per
+question**, forever: ~7 s per lookup at 1.2 GB, ~1 min at 12 GB. At some size
+the investigation stops being interactive.
+
+**With XERJ:** every step is an indexed lookup or aggregation.
+**Measured:** a single-user lookup was **0.007 s (XERJ) vs 7.3 s (grep) at
+1.2 GB — 1,043× faster**, and stays ~flat as the corpus grows because the index
+is O(matching), not O(corpus).
+
+**Verdict:** win that *grows* with data size. Below ~1 GB grep is fine; above
+it, indexed drill-down is the difference between interactive and not.
+
+---
+
+## Scenario 5 — A relational question, with no JOIN
+
+**Task:** "How many enterprise customers have an open ticket?" — spans two
+sources.
+
+**The trap:** answered relationally, the agent pulls one whole side of the
+relation into context (966 ids × 16 bytes) and echoes it back — **15,914 bytes
+for a 3-byte answer.** This is the entire "100× more than grep."
+
+**The fix — denormalize at ingest, then one filtered aggregation:**
+```bash
+# put `plan` on the ticket once, at ingest; then:
+curl -s "localhost:9200/tickets/_search?filter_path=aggregations.u.value" \
+ -H 'Content-Type: application/json' -d '{"size":0,
+ "query":{"bool":{"filter":[{"term":{"status":"open"}},{"term":{"plan":"enterprise"}}]}},
+ "aggs":{"u":{"cardinality":{"field":"user_id"}}}}'
+```
+**Measured:** **36 bytes, identical answer (122) — 442× smaller.** When one side
+already fits in a single document's array field, ES `terms` lookup does this
+server-side (see `docs/TOKEN_USAGE.md`).
+
+**Verdict:** the token cost is set by the *pattern*, not the engine. The
+relational pattern is a trap; the native pattern is competitive. `autoindex`
+already detects the foreign key — denormalizing on it is the next engine step.
+
+---
+
+## Scenario 6 — Durable memory across sessions
+
+**Task:** remember a fact in one session, recall it by meaning in another.
+
+**Without XERJ:** re-derive it, or the user re-supplies it. No mechanism.
+
+**With XERJ:** `POST /_memory/{ns}` to store, `_recall` to retrieve by meaning.
+**Measured:** a semantic recall returned in **328 bytes**, and recalled the
+right memory from queries sharing *no vocabulary* with the stored text (in
+`--embed-mode neural`; the default lexical embedder scores far lower on
+paraphrase — P@1 0.33 vs 1.00, so turn neural on for memory).
+
+**Verdict:** a capability the baseline does not have at all. Worth wiring in for
+any multi-session agent, with the neural embedder active.
+
+---
+
+## When XERJ does *not* help — say so
+
+- **One-integer questions on a small file.** `grep -c` prints `2` in one byte;
+ no engine answering in JSON beats that. Ingest cost is never repaid.
+- **`percentiles`/`date_histogram` without the fixes on this branch.** They
+ fell to an O(N) scan; the fixes here bring them to columnar speed, but verify
+ your build has them.
+- **Byte-level forensics.** "What is the exact 0x80 byte at offset 4231" is a
+ grep/xxd job, not a search job.
+- **A corpus you will query once.** The break-even is ~30–70 questions
+ (ingest amortization). One-shot use loses.
+
+## Reproducing all of this
+
+```bash
+# analytics regime (XERJ wins correctness, loses tokens on record-heavy data)
+python3 demo/agent-gate/make_corpus.py /tmp/gate-corpus
+xerj --insecure --data-dir /tmp/gate-data &
+xerj autoindex /tmp/gate-corpus
+python3 demo/agent-gate/gate.py /tmp/gate-corpus
+
+# retrieval regime (XERJ wins correctness and, at scale, tokens)
+python3 demo/agent-gate/make_corpus_prose.py /tmp/gate-prose
+xerj autoindex /tmp/gate-prose
+python3 demo/agent-gate/gate_retrieval.py /tmp/gate-prose
+```
+
+Both gates print the corpus composition next to every ratio, and refuse to
+collapse the two regimes into one headline number.
diff --git a/demo/agent-gate/gate_retrieval.py b/demo/agent-gate/gate_retrieval.py
new file mode 100644
index 000000000..246da91d3
--- /dev/null
+++ b/demo/agent-gate/gate_retrieval.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""Agent Gate — RETRIEVAL regime. See README.md for the anti-cheating rules.
+
+Same discipline as gate.py, applied to the corpus shape where XERJ is expected
+to WIN on tokens: a repository of small prose/code documents, no large record
+streams. Questions are answered in the asker's own words, so a match requires
+ranking, not lexical coincidence.
+
+Correctness here is CONTENT, asserted before the run — a substring the true
+answer must contain, never the file it lives in. Both paths are tuned:
+ baseline : grep, capped, question-vocabulary only
+ xerj : ranked search, size:1, highlight, filter_path — the frugal shape
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import urllib.request
+
+CORPUS = sys.argv[1] if len(sys.argv) > 1 else "/tmp/gate-prose"
+XERJ = os.environ.get("XERJ_URL", "http://localhost:9200")
+
+
+class Ledger:
+ def __init__(self):
+ self.rows = []
+
+ def add(self, task, path, out, seconds):
+ self.rows.append({"task": task, "path": path, "bytes": len(out),
+ "seconds": seconds})
+
+ def bytes_for(self, path, task):
+ return sum(r["bytes"] for r in self.rows
+ if r["path"] == path and r["task"] == task)
+
+ def seconds_for(self, path, task):
+ return sum(r["seconds"] for r in self.rows
+ if r["path"] == path and r["task"] == task)
+
+
+LEDGER = Ledger()
+_CTX = {"task": None}
+
+
+def sh(cmd):
+ t0 = time.time()
+ out = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True,
+ cwd=CORPUS).stdout
+ LEDGER.add(_CTX["task"], "baseline", out, time.time() - t0)
+ return out
+
+
+def es(path, body):
+ t0 = time.time()
+ req = urllib.request.Request(f"{XERJ}{path}", json.dumps(body).encode(),
+ {"Content-Type": "application/json"})
+ raw = urllib.request.urlopen(req).read()
+ LEDGER.add(_CTX["task"], "xerj", raw.decode("utf-8", "replace"), time.time() - t0)
+ return json.loads(raw)
+
+
+# (question, grep-terms [question vocabulary only], required substring in answer)
+QUESTIONS = [
+ ("why was checkout unavailable in June",
+ "checkout", "certificate"),
+ ("why did search quality regress",
+ "search", "stemming"),
+ ("how do we fail the database over to the standby",
+ "failover", "promote"),
+ ("what causes connection pool exhaustion",
+ "pool", "fan-out"),
+ ("why must queue consumers be idempotent",
+ "idempotent", "at-least-once"),
+ ("where are secrets kept",
+ "secrets", "vault"),
+ ("how does the payment client avoid billing twice",
+ "idempotency", "idempotency_key"),
+]
+
+
+def run_baseline(terms, needle):
+ _CTX["task"] = needle
+ # A cold agent greps the question's noun. It does NOT know the file, and it
+ # must read enough context to confirm — so grep -i with a few lines around.
+ out = sh(f"grep -rni '{terms}' . | head -8")
+ return out, (needle.lower() in out.lower())
+
+
+def run_xerj(question, needle):
+ _CTX["task"] = needle
+ d = es("/ax-*/"
+ "_search?filter_path=hits.hits._source.ax_path,hits.hits.highlight",
+ {"size": 1,
+ "query": {"bool": {"should": [
+ {"match": {"body": question}},
+ {"match": {"text": question}}]}},
+ "_source": ["ax_path"],
+ "highlight": {"fields": {
+ "body": {"fragment_size": 200, "number_of_fragments": 1},
+ "text": {"fragment_size": 200, "number_of_fragments": 1}}}})
+ hits = d.get("hits", {}).get("hits", [])
+ if not hits:
+ return "(no hits)", False
+ hl = hits[0].get("highlight", {})
+ frag = (hl.get("body") or hl.get("text") or [""])[0]
+ out = f"{hits[0]['_source'].get('ax_path')}: {frag}"
+ return out, (needle.lower() in frag.lower())
+
+
+def main():
+ rows = []
+ for question, terms, needle in QUESTIONS:
+ b_out, b_ok = run_baseline(terms, needle)
+ x_out, x_ok = run_xerj(question, needle)
+ rows.append({"q": needle, "b_ok": b_ok, "x_ok": x_ok})
+
+ print("\n" + "=" * 80)
+ print("AGENT GATE — RETRIEVAL regime (prose/code corpus)")
+ print("=" * 80)
+ print(f"\n{'question needle':<22}{'base':<7}{'xerj':<7}"
+ f"{'base tok':>9}{'xerj tok':>9}{'base s':>8}{'xerj s':>8}")
+ jb = jx = 0
+ for r in rows:
+ b = LEDGER.bytes_for("baseline", r["q"]) // 4
+ x = LEDGER.bytes_for("xerj", r["q"]) // 4
+ both = r["b_ok"] and r["x_ok"]
+ if both:
+ jb += b
+ jx += x
+ print(f"{r['q'][:22]:<22}"
+ f"{'OK' if r['b_ok'] else 'MISS':<7}{'OK' if r['x_ok'] else 'MISS':<7}"
+ f"{b:>9}{x:>9}"
+ f"{LEDGER.seconds_for('baseline', r['q']):>8.2f}"
+ f"{LEDGER.seconds_for('xerj', r['q']):>8.2f}")
+
+ nb = sum(1 for r in rows if r["b_ok"])
+ nx = sum(1 for r in rows if r["x_ok"])
+ print(f"\ncorrect: baseline {nb}/{len(rows)} xerj {nx}/{len(rows)}")
+
+ print("\n-- tokens over jointly-correct tasks only --")
+ if jb and jx:
+ ratio = jb / jx
+ v = (f"XERJ uses {ratio:.2f}x FEWER tokens" if ratio > 1
+ else f"XERJ uses {1/ratio:.2f}x MORE tokens")
+ print(f" baseline {jb} tok xerj {jx} tok -> {v}")
+ else:
+ print(" no jointly-correct tasks — no savings claim possible")
+
+ total = text = 0
+ for root, _, files in os.walk(CORPUS):
+ for f in files:
+ n = os.path.getsize(os.path.join(root, f))
+ total += n
+ if f.endswith((".md", ".py", ".rs", ".js", ".html", ".txt")):
+ text += n
+ print("\n-- corpus composition --")
+ print(f" {total/1e3:.1f} KB total; {text/1e3:.1f} KB searchable prose/code "
+ f"({100*text/max(total,1):.1f}%)")
+ print(" This is the regime where retrieval savings appear: grep's candidate")
+ print(" set grows with the corpus; ranked search returns one passage.\n")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/demo/agent-gate/make_corpus_prose.py b/demo/agent-gate/make_corpus_prose.py
new file mode 100644
index 000000000..fd9322dfc
--- /dev/null
+++ b/demo/agent-gate/make_corpus_prose.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Deterministic PROSE/CODE-heavy corpus for the Agent Gate — the regime where
+XERJ is expected to WIN on tokens.
+
+The mixed corpus (make_corpus.py) is 99.99% structured records, where grep on a
+one-integer question is unbeatable. This corpus is the opposite: many small
+documents of natural-language and source text, no large record streams. It is
+the shape of a real code/knowledge repository, and it is where ranked retrieval
+beats unranked grep on tokens — grep must return every candidate line, ranked
+search returns the one passage.
+
+Seeded, so runs are byte-identical and comparable across versions.
+Usage: make_corpus_prose.py <dir>
+"""
+import os
+import random
+import sys
+import textwrap
+
+OUT = sys.argv[1] if len(sys.argv) > 1 else "/tmp/gate-prose"
+os.makedirs(OUT, exist_ok=True)
+os.chdir(OUT)
+random.seed(31337)
+
+for d in ("docs", "runbooks", "adr", "src", "src/api", "src/store", "tests"):
+ os.makedirs(d, exist_ok=True)
+
+# ── Ground-truth documents: each answers exactly one gate question, phrased so
+# the answer's wording differs from the question's (no lexical gift). ──────
+
+open("docs/postmortem-checkout.md", "w").write(textwrap.dedent("""\
+ # Postmortem: checkout unavailable, 14 June 2026
+
+ ## Impact
+ Checkout returned errors for 51 minutes across all regions.
+
+ ## Root cause
+ The payment gateway's TLS certificate expired. Renewal was automated but
+ the service never reloaded it, so every handshake was rejected.
+
+ ## Fix
+ Reloaded the service and added a certificate-expiry alert firing 14 days out.
+ """))
+
+open("docs/postmortem-search.md", "w").write(textwrap.dedent("""\
+ # Postmortem: search quality regression, 3 May 2026
+
+ ## Impact
+ Click-through on search fell by more than a third for two days.
+
+ ## Root cause
+ A retuned analyser stopped applying stemming to German queries, so plural
+ forms no longer matched their singular index terms.
+
+ ## Fix
+ Reverted the analyser and added a relevance regression test to the pipeline.
+ """))
+
+open("runbooks/database-failover.md", "w").write(textwrap.dedent("""\
+ # Runbook: database failover
+
+ Promote the hot standby with `pg_ctl promote`. Writes are unavailable for
+ roughly forty seconds while the promotion settles. Point the application at
+ the new primary only after replication lag on the old node reaches zero.
+ """))
+
+open("runbooks/pool-exhaustion.md", "w").write(textwrap.dedent("""\
+ # Runbook: connection pool exhaustion
+
+ Symptoms: p99 latency climbs and `pool exhausted` appears in the logs. The
+ usual trigger is a deploy that raised per-request fan-out, saturating the
+ fixed pool. Mitigate by raising the pool ceiling and rolling back the deploy
+ that increased fan-out.
+ """))
+
+open("adr/0007-idempotent-consumers.md", "w").write(textwrap.dedent("""\
+ # ADR 0007: all queue consumers must be idempotent
+
+ The async queue guarantees at-least-once delivery, not exactly-once. A
+ consumer can therefore see the same message twice. Every consumer must key
+ its side effects on a stable identifier so a redelivery is a no-op rather
+ than a double charge or a duplicate email.
+ """))
+
+open("docs/security-secrets.md", "w").write(textwrap.dedent("""\
+ # Handling secrets
+
+ Secrets live in the vault and are injected into the process environment at
+ boot. They are never written to a file that is committed to version control.
+ Production access additionally requires a hardware security key.
+ """))
+
+# Source that answers "how does the payment client avoid double billing".
+open("src/api/payments.py", "w").write(textwrap.dedent('''
+ """Payment gateway client: bounded retries with a stable idempotency key."""
+ import hashlib
+ import time
+
+ MAX_RETRIES = 3
+
+ def idempotency_key(order_id: str, attempt: int) -> str:
+ """Deterministic key so a retried charge is collapsed by the gateway
+ into the original, never billed twice."""
+ return hashlib.sha256(f"{order_id}:{attempt}".encode()).hexdigest()
+
+ def charge(client, order_id: str, amount_cents: int):
+ for attempt in range(MAX_RETRIES):
+ try:
+ return client.post("/charge", json={
+ "amount": amount_cents,
+ "idempotency_key": idempotency_key(order_id, attempt),
+ })
+ except TimeoutError:
+ time.sleep(2 ** attempt) # exponential backoff
+ raise RuntimeError("payment gateway unreachable after retries")
+'''))
+
+open("src/store/pool.rs", "w").write(textwrap.dedent('''
+ //! Fixed-ceiling connection pool. Exhaustion is the most common outage
+ //! trigger: a deploy raises per-request fan-out and the pool saturates,
+ //! p99 climbs, and the service starts shedding load with 503s.
+ pub struct Pool {
+ max: usize,
+ in_use: usize,
+ }
+
+ impl Pool {
+ pub fn acquire(&mut self) -> Result<Conn, PoolError> {
+ if self.in_use >= self.max {
+ return Err(PoolError::Exhausted);
+ }
+ self.in_use += 1;
+ Ok(Conn::new())
+ }
+ }
+'''))
+
+# Filler: many plausible-but-irrelevant documents, so retrieval has to actually
+# rank rather than return the only file. This is what makes grep expensive —
+# its candidate set is large; ranked search still returns one passage.
+TOPICS = ["caching", "logging", "metrics", "tracing", "rate limiting",
+ "feature flags", "backups", "cron scheduling", "webhooks",
+ "pagination", "serialization", "config loading", "health checks",
+ "graceful shutdown", "circuit breaking", "load shedding",
+ "request validation", "content negotiation", "compression",
+ "connection keepalive", "dns resolution", "retry budgets"]
+BODY = ("This module handles {t}. It exposes a small, dependency-free surface "
+ "and is covered by unit tests. Configuration is read once at startup "
+ "and validated eagerly so a bad value fails fast rather than at first "
+ "use. The design favours explicitness over cleverness; see the module "
+ "comment for the rationale and the trade-offs that were considered.")
+for i, t in enumerate(TOPICS):
+ slug = t.replace(" ", "-")
+ open(f"src/{slug}.md", "w").write(f"# {t.title()}\n\n{BODY.format(t=t)}\n")
+ # a source file too, so the corpus is code + prose
+ open(f"src/{slug}.py", "w").write(
+ f'"""{t.title()} implementation."""\n\n'
+ f"# {BODY.format(t=t)}\n"
+ f"def configure_{slug.replace('-', '_')}(cfg):\n"
+ f" return cfg.get({t.split()[0]!r}, {{}})\n")
+
+# A few longer design docs for bulk.
+for i in range(6):
+ paras = "\n\n".join(
+ f"Section {j}: " + BODY.format(t=random.choice(TOPICS))
+ for j in range(random.randint(8, 16)))
+ open(f"adr/design-note-{i:02d}.md", "w").write(f"# Design note {i}\n\n{paras}\n")
+
+# tiny binary junk, to exercise skip-not-fatal
+open("logo.bin", "wb").write(os.urandom(4096))
+
+n = sum(len(files) for _, _, files in os.walk("."))
+sz = sum(os.path.getsize(os.path.join(r, f))
+ for r, _, fs in os.walk(".") for f in fs)
+print(f"prose corpus built: {n} files, {sz/1e3:.1f} KB")