Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

usecase(doc-index): Claude+XERJ vs grep-only agent on a recursive doc folder

xerj-org8871b49cb20b
diff --git a/demo/usecases/doc-index/.gitignore b/demo/usecases/doc-index/.gitignore
new file mode 100644
index 000000000..c36f570e7
--- /dev/null
+++ b/demo/usecases/doc-index/.gitignore
@@ -0,0 +1,3 @@
+# generated by gen-corpus.mjs (idempotent); regenerate with: node gen-corpus.mjs
+corpus/
+.index_build_ms
diff --git a/demo/usecases/doc-index/README.md b/demo/usecases/doc-index/README.md
new file mode 100644
index 000000000..25deb6e74
--- /dev/null
+++ b/demo/usecases/doc-index/README.md
@@ -0,0 +1,243 @@
+# Doc-index — search a recursive folder of mixed-format documents with XERJ
+
+Point an AI agent at a real, messy shared drive — an HR handbook in **PDF**, an
+on-call runbook in **Word**, a product FAQ in **HTML**, architecture notes in
+**Markdown**, meeting notes in **plain text** — and let it answer questions with
+*cited passages*, regardless of format.
+
+This directory is a self-contained, reproducible use-case deliverable. One
+`xerj-index` pass walks the folder, extracts text from **every** format, chunks +
+auto-embeds each chunk, and bulk-indexes it into a single `docfolder` index. A
+comparison harness then scores XERJ against the honest shell-only baseline (`rg`
+over the raw files) and writes a measured scorecard.
+
+**Everything here runs live and reports measured numbers.** The published recipe
+page — <https://xerj.org/docs/recipes/document-folder-index.html> — cites only the
+numbers this harness emits into `results.json` / `SCORECARD.md`.
+
+---
+
+## What gets compared
+
+- **Baseline — "Claude Code with scripts only":** answer questions by running
+ `rg`/`grep` over the raw files. Its ceiling is real and structural: (1) PDF/DOCX
+ are compressed binary, so `rg` sees NUL bytes and matches nothing; (2) literal
+ substring only — no meaning/semantic retrieval; (3) to *quote* an answer it must
+ read whole files into context; (4) no ranking; (5) re-scans the whole tree every
+ query.
+- **XERJ-backed — "Claude + XERJ":** a one-time `xerj-index` pass extracts text
+ from every format, chunks (~800 chars, ~100 overlap), auto-embeds with XERJ's
+ built-in embedder, and bulk-indexes. The agent then queries XERJ (lexical +
+ semantic + hybrid) and gets ranked, cited passages back.
+
+The comparison is deliberately fair: same corpus, same 22-question ground-truth
+set, and the baseline is given ripgrep (a good plain-shell tool). Per the
+fair-baseline rule, the grep baseline searches the union of each query's curated
+keywords AND the salient tokens of the question itself — it is never capped to a
+keyword list that dodges the answer's own line. See `SPEC.md` for the full
+contract (index mappings, chunking, query schema, scoring, and the gate).
+
+---
+
+## Measured result (from the live run)
+
+| Metric | XERJ | Baseline (grep) |
+|--------|------|-----------------|
+| Overall coverage | **21/22 (95.5%)** | 14/22 (63.6%) |
+| **binary_only** (PDF/DOCX-only answers) — **HEADLINE** | **6/7** | 0/7 |
+| large_literal (answer buried in a ≥60 KB doc) | 4/4 | 3/4 |
+| robustness (differently phrased) | 5/5 | 5/5 |
+| literal (plaintext substring) | 6/6 | 6/6 |
+| Query latency p50 / mean / max (ms) | 61.23 / 64.9 / 116.09 | — |
+| Index build time | 29 files → 420 chunks in **1339 ms** | — |
+
+**Gate: PASS** — the whole corpus indexed end-to-end with 0 crashes; XERJ's overall
+coverage ≥ baseline (21 ≥ 14); XERJ is strictly higher on the decisive
+`binary_only` set (6 > 0); and the `large_literal` context ratio is **17.24× > 1×**.
+`robustness` is reported honestly as a **tie** and is not gated.
+
+Honest notes carried in the scorecard:
+
+- **The decisive win is format coverage (`binary_only`), not "meaning".** grep reads
+ raw bytes, so a PDF/DOCX answer is invisible to it (0/7); XERJ extracted and
+ indexed that text and answers 6/7. This is a capability grep structurally lacks.
+- **Context efficiency is a real win — but scale-dependent, not flat.** The baseline is
+ charged **only the single answer-containing file it must open** (`statSync(answer_path).size`,
+ and **0 bytes on any query it cannot answer**) — never the false-positive files its broad
+ terms also matched. Measured that way the ratio VARIES by document size. Over the 3
+ `large_literal` queries the fair baseline actually answers, it must open **192,782 bytes**
+ of whole answer files to quote/verify the line, while XERJ returns **11,180 bytes** of
+ ranked passages — **17.24×** less context (85.68× counting only the single best passage).
+ But on the 6 small-file `literal` queries the SAME metric INVERTS: XERJ returns **20,965
+ bytes** of passages against the baseline's **4,741 bytes** of tiny answer files — just
+ **0.23×** (≈ no win — on a tiny plaintext file you'd simply read the whole thing). Side
+ by side — large_literal **17.24×** vs `literal` **0.23×** — IS the scale-dependence,
+ measured not asserted. Across all 14 answerable queries the ratio is **3.98×**; the naive
+ whole-corpus ratio (**2.53×**) FLATTERS the blind baseline (it opens 0 bytes on every query
+ it cannot answer) and is **NOT** a claim. _(Diagnostic only, never used in any ratio or
+ claim: had the baseline instead been charged every file its broad terms matched — false
+ positives included — the total would be 4,218,746 bytes.)_
+- **`robustness` ≈ TIE (honest).** On differently-phrased answers a fair grep of the
+ question's own salient terms matches the same lines XERJ's lexical embedder finds
+ — 5/5 vs 5/5. XERJ's built-in embedder is **lexical feature-hashing** (384-dim
+ cosine), **not neural**, so its "semantic" matching is word/sub-word overlap, not
+ comprehension. We do **not** claim a semantic-understanding win here.
+- kNN is **exact brute-force** at query time (fine at this corpus size).
+- **One miss (q05):** "What is the acknowledgement SLA for a Sev-1 incident?"
+ (answer "within 5 minutes" in `oncall-runbook.docx`). The chunk *was* extracted
+ and indexed — a direct `match_phrase` on the live index finds it — but for this
+ phrasing the hybrid ranking pushes it out of the top-5 (the answer chunk lacks the
+ word "SLA"; larger incident-heavy docs outrank it). Reported truthfully as
+ `binary_only` 6/7, still strictly > the baseline's 0/7. Not massaged.
+
+---
+
+## Prerequisites
+
+- **Node** 18+ (tested on v24) — no npm dependencies; all scripts are stdlib-only.
+- **`pdftotext`** (poppler-utils) — PDF text extraction.
+- **`soffice`** / LibreOffice — DOCX extraction (indexer falls back to `unzip`).
+- **`rg`** (ripgrep) — the baseline retriever.
+- A **XERJ** binary on your PATH (or adjust the `xerj` command below to its path).
+
+The `corpus/` folder and `queries.json` are already checked in, so you can skip
+`gen-corpus.mjs` unless you want to regenerate them (that step needs `pdftotext`
+and `soffice`).
+
+---
+
+## Reproduce it end-to-end
+
+All commands run from this directory (`demo/usecases/doc-index/`).
+
+```bash
+# 1. Start a throwaway XERJ (ES-compat wire; default port :9200).
+# --insecure disables TLS + auth for local dev. Use a scratch data dir.
+xerj --insecure --data-dir ./docfolder-data
+
+# 2. In another shell, point the scripts at your node's ES-compat port.
+# The scripts default to :9209; a fresh XERJ listens on :9200.
+cd demo/usecases/doc-index
+export XERJ_PORT=9200
+
+# 3. (Optional) regenerate the corpus + ground-truth queries.json.
+# Needs pdftotext + soffice; skip it to use the checked-in corpus.
+node gen-corpus.mjs
+
+# 4. Extract, chunk, auto-embed and bulk-index the whole folder.
+# --recreate deletes + rebuilds the index so re-runs are clean.
+node xerj-index.mjs --recreate
+
+# 5. (Optional) record the index build time so the scorecard can cite it.
+# xerj-index prints "elapsed ms : <N>" — write that number here:
+echo 1339 > .index_build_ms # replace 1339 with YOUR run's elapsed ms
+
+# 6. Score XERJ vs the grep baseline. Writes results.json + SCORECARD.md.
+node compare.mjs
+
+# 7. Read the measured verdict.
+cat SCORECARD.md
+```
+
+When you're done, stop the node (Ctrl-C) and remove the scratch data dir
+(`rm -rf ./docfolder-data`).
+
+### Expected output
+
+`xerj-index.mjs --recreate` prints a summary like:
+
+```
+===== xerj-index summary =====
+files walked : 29
+chunks produced : 420
+chunks indexed : 420
+chunks errored : 0
+files skipped : 0
+elapsed ms : 1339
+```
+
+Confirm the index is populated:
+
+```bash
+curl -s "localhost:${XERJ_PORT}/docfolder/_count"
+# {"count":420, ...}
+```
+
+`compare.mjs` writes `results.json` (machine-readable rows + aggregates + gate) and
+`SCORECARD.md` (per-query table + honest verdict), then prints where it wrote them.
+It exits `0` whether or not the gate passes — a gate failure is reported honestly,
+never massaged. It exits `1` only if it *couldn't run* (node unreachable,
+`docfolder` missing, `queries.json`/`grep-baseline.mjs` missing).
+
+---
+
+## Poke at it directly
+
+The three retrieval shapes the harness uses (substitute your own question):
+
+```bash
+# Lexical — BM25 over the extracted text.
+curl -s "localhost:${XERJ_PORT}/docfolder/_search" -H 'Content-Type: application/json' -d '{
+ "size": 5, "query": { "match": { "body_text": "default API rate limit" } }
+}'
+
+# Semantic — meaning-ish retrieval over the auto-embedded body.
+curl -s "localhost:${XERJ_PORT}/docfolder/_search" -H 'Content-Type: application/json' -d '{
+ "size": 5, "query": { "semantic": { "field": "body",
+ "query": "policy on remote work versus coming into the office", "k": 5 } }
+}'
+
+# Hybrid — fuse lexical + semantic with RRF (what compare.mjs scores).
+curl -s "localhost:${XERJ_PORT}/docfolder/_search" -H 'Content-Type: application/json' -d '{
+ "size": 5, "query": { "hybrid": { "fusion": "rrf", "queries": [
+ { "query": { "match": { "body_text": "weeks of paid parental leave" } } },
+ { "query": { "semantic": { "field": "body", "query": "weeks of paid parental leave", "k": 5 } } }
+ ] } }
+}'
+```
+
+And the honest baseline, for contrast — note it finds **nothing** for the
+parental-leave question because the answer lives only in a binary PDF:
+
+```bash
+node grep-baseline.mjs q01 # binary_only → miss (answer locked in a binary PDF)
+node grep-baseline.mjs l01 # literal → hit (finds the answer line in a .md)
+```
+
+---
+
+## Files in this directory
+
+| File | Role |
+|------|------|
+| `SPEC.md` | The contract: index mappings, chunking, query schema, scoring, gate. |
+| `gen-corpus.mjs` | Generates `corpus/` + `queries.json` (needs pdftotext + soffice). |
+| `corpus/` | The generated fictional-company KB (PDF/DOCX/HTML/MD/TXT). |
+| `queries.json` | 22 ground-truth queries (7 binary_only, 4 large_literal, 5 robustness, 6 literal). |
+| `xerj-index.mjs` | The indexer: walk → extract → chunk → bulk-index into XERJ. |
+| `grep-baseline.mjs` | The shell-only baseline retriever (ripgrep over raw files). |
+| `compare.mjs` | Runs both over `queries.json`, scores, writes `results.json` + `SCORECARD.md`. |
+| `results.json` | Measured comparison output (machine-readable). |
+| `SCORECARD.md` | Measured, human-readable scorecard + verdict. |
+
+### Script options worth knowing
+
+- `xerj-index.mjs`: `--dir <folder>` (default `./corpus`), `--recreate`,
+ `--batch <n>` (docs per `_bulk`, default 200), `--dry-run` (extract + chunk +
+ print stats, don't touch XERJ). Env: `XERJ_PORT` (default 9209).
+- `compare.mjs`: env `XERJ_PORT` / `XERJ_URL`, `INDEX_BUILD_MS` (or
+ `--index-build-ms=<N>`, or the `.index_build_ms` file) to cite build time.
+- `grep-baseline.mjs`: env `DOC_INDEX_CORPUS`, `DOC_INDEX_QUERIES`, `RG_BINARY`.
+
+---
+
+## Notes on fairness and honesty
+
+A script agent *could* shell out to `pdftotext`/`soffice` to defeat the binary
+blindness — this baseline deliberately does not, because the point is the naive
+shell-only agent. But even a `pdftotext`-augmented baseline would still lack
+ranking, per-chunk retrieval, and the semantic layer, and would keep paying the
+whole-file context cost. XERJ's win is not "it can read PDFs" alone; it is
+format-agnostic extraction **plus** chunked, ranked, context-aware retrieval with a
+real (if shallow, lexical) semantic layer. Every claim above is reproducible from
+the run this README describes.
diff --git a/demo/usecases/doc-index/SCORECARD.md b/demo/usecases/doc-index/SCORECARD.md
new file mode 100644
index 000000000..70066520d
--- /dev/null
+++ b/demo/usecases/doc-index/SCORECARD.md
@@ -0,0 +1,86 @@
+# Doc-index scorecard — Claude + XERJ vs. shell-only (grep) baseline
+
+_Measured live on 2026-07-09T05:12:07.563Z against `http://localhost:9209` (index `docfolder`, 420 chunks). Retrieval mode: **hybrid**. Every number below is from this live run._
+
+> **Honest framing (REVISION 2).** The decisive, capability-based win is **binary_only** — a fair grep structurally cannot read PDF/DOCX bytes. **Context efficiency** is a real win, but only shows up on **large_literal** (big documents). **robustness** ≈ **TIE** under a fair baseline, because XERJ's built-in embedder is a LEXICAL feature-hashing model (384-dim cosine), NOT neural — a diligent grep of the question's own terms matches the same lines. **literal** is the honest control. No overstated semantic claim is made.
+
+## Per-query results
+
+| ID | Type | Fmt | XERJ | Base | XERJ ms | XERJ psg B | Best psg B | Base answer-file B | Base matched B (diag) | Question |
+|----|------|-----|------|------|--------:|-----------:|-----------:|-------------------:|----------------------:|----------|
+| q01 | binary_only | pdf | hit | — | 116.09 | 3433 | 737 | 0 | 197133 | How many weeks of paid parental leave do employees receive? |
+| q02 | binary_only | pdf | hit | — | 71.74 | 3771 | 737 | 0 | 197808 | How many days of paid time off do full-time employees acc... |
+| q03 | binary_only | docx | hit | — | 63.15 | 3621 | 693 | 0 | 197796 | How many days of bereavement leave are provided per event? |
+| q04 | binary_only | docx | hit | — | 74.96 | 3661 | 733 | 0 | 198770 | After how long does an unacknowledged page escalate to th... |
+| q05 | binary_only | docx | — | — | 61.92 | 3703 | 809 | 0 | 196317 | What is the acknowledgement SLA for a Sev-1 incident? |
+| q06 | binary_only | pdf | hit | — | 70.71 | 3780 | 710 | 0 | 197649 | How long must customer transaction records be retained? |
+| q07 | binary_only | pdf | hit | — | 62.48 | 3728 | 693 | 0 | 196826 | What is the domestic meal per diem rate? |
+| ll01 | large_literal | html | hit | hit | 59.57 | 3702 | 786 | 62204 | 195057 | How much is the employee referral bonus for a successful ... |
+| ll02 | large_literal | md | hit | hit | 60.41 | 3765 | 765 | 64837 | 195245 | What is the default value of the ingest max_batch_size pa... |
+| ll03 | large_literal | txt | hit | hit | 60.08 | 3713 | 699 | 65741 | 195532 | How quickly must a public status-page update be posted af... |
+| ll04 | large_literal | pdf | hit | — | 58.56 | 3425 | 673 | 0 | 196546 | Within how many days must critical vulnerabilities be rem... |
+| s01 | robustness | md | hit | hit | 62.32 | 3834 | 751 | 770 | 204563 | What is Northwind's policy on remote work versus coming i... |
+| s02 | robustness | html | hit | hit | 62.99 | 3699 | 661 | 842 | 196355 | How does a user recover access to a locked account? |
+| s03 | robustness | md | hit | hit | 60.06 | 3781 | 738 | 832 | 200772 | What developer hardware is issued to a newly hired engine... |
+| s04 | robustness | md | hit | hit | 60.04 | 3476 | 760 | 1105 | 195368 | Which data store backs the inventory and SKU records? |
+| s05 | robustness | html | hit | hit | 60.88 | 3815 | 781 | 991 | 198030 | What can a customer do if they want to send a purchased u... |
+| l01 | literal | md | hit | hit | 59.43 | 3788 | 743 | 783 | 196826 | What is the default API rate limit? |
+| l02 | literal | md | hit | hit | 61.11 | 3867 | 760 | 1105 | 200764 | What port does the ingest service listen on? |
+| l03 | literal | md | hit | hit | 60.09 | 2991 | 605 | 643 | 195610 | When is the weekly engineering sync held? |
+| l04 | literal | html | hit | hit | 58.16 | 3623 | 781 | 991 | 68849 | What uptime does the platform SLA guarantee? |
+| l05 | literal | html | hit | hit | 61.34 | 3280 | 548 | 722 | 198316 | Who must approve purchases over the standard threshold? |
+| l06 | literal | txt | hit | hit | 61.68 | 3416 | 493 | 497 | 198614 | What is the Q3 target for warehouse pick accuracy? |
+
+_"Base answer-file B" is the HONEST context charge: the size of the single answer-containing file the baseline must open, and **0 when the baseline cannot answer** (it reads nothing toward an answer it cannot find). "Base matched B (diag)" is a DIAGNOSTIC only — the size of every file the baseline's broad terms matched, false positives included — and is used in no ratio or claim._
+
+## Aggregate coverage by match type
+
+| Metric | XERJ | Baseline | Note |
+|--------|------|----------|------|
+| Overall coverage | 21/22 (95.5%) | 14/22 (63.6%) | XERJ ≥ baseline |
+| **binary_only** (answer only in PDF/DOCX) | 6/7 | 0/7 | **HEADLINE — capability grep lacks** |
+| large_literal (buried in a ≥60 KB doc) | 4/4 | 3/4 | both hit; see context win below |
+| robustness (differently-phrased) | 5/5 | 5/5 | ≈ TIE under fair baseline (lexical embedder) |
+| literal (plaintext substring) | 6/6 | 6/6 | honest control — both read plain text |
+
+### Coverage by answer format
+
+| Format | Queries | XERJ | Baseline |
+|--------|--------:|------|----------|
+| docx | 3 | 2/3 (66.7%) | 0/3 (0%) |
+| html | 5 | 5/5 (100%) | 5/5 (100%) |
+| md | 7 | 7/7 (100%) | 7/7 (100%) |
+| pdf | 5 | 5/5 (100%) | 0/5 (0%) |
+| txt | 2 | 2/2 (100%) | 2/2 (100%) |
+
+### Context efficiency (measured honestly)
+
+Context ratio = baseline bytes to open ÷ XERJ bytes returned. The baseline is charged ONLY the single answer-containing file it must open (`statSync(answer_path)`), **never** the false-positive files its broad terms also matched. `grep -n` yields a matching line, but to quote/verify an answer reliably an agent opens the whole answer file — so that single file is the charge (stated plainly as the assumption).
+
+| View | Baseline bytes | XERJ bytes | Ratio | What it means |
+|------|---------------:|-----------:|:-----:|---------------|
+| **large_literal** (returned passages) | 192,782 | 11,180 | **17.24×** | THE REAL WIN — big files vs. ranked passages (over 3 query/queries the baseline answers) |
+| large_literal (single best passage) | 192,782 | 2,250 | 85.68× | one passage the agent would actually quote |
+| **literal** (returned passages) | 4,741 | 20,965 | **0.23×** | SCALE-DEPENDENCE: on tiny plaintext files the win INVERTS — 5 returned passages ≈/exceed the whole small file (over 6 query/queries) |
+| literal (single best passage) | 4,741 | 3,930 | 1.21× | even a single passage ≈ a tiny file |
+| answerable (all queries baseline answers) | 202,063 | 50,750 | 3.98× | fair whole-corpus view over 14 answerable queries |
+| naive overall (all queries) | 202,063 | 79,872 | 2.53× | **flatters the blind baseline — it literally opens 0 bytes on every query it cannot answer** — NOT a claim |
+
+**Scale-dependence, side by side:** large_literal **17.24×** (answers buried in ≥60 KB docs) vs. literal **0.23×** (tiny plaintext files). The context win is real on large documents and INVERTS on small ones — demonstrated on measured data, not asserted.
+
+_Diagnostic (NOT a claim): had the baseline instead been charged every file its broad terms matched — false positives included — the total would be 4,218,746 bytes. That inflated charge is exactly what this scorecard does NOT use._
+
+_Latency: XERJ query p50 / mean / max = 61.23 / 64.9 / 116.09 ms. Index build time: 1339 ms._
+
+### Gate
+
+- (#2) Overall coverage ≥ baseline: **PASS**
+- (#2) binary_only strictly higher (the capability win): **PASS**
+- (#3) large_literal context ratio > 1×: **PASS** (17.24×)
+- (context, informational) literal ratio ≈1× or below — the small-file inversion that proves scale-dependence: 0.23× — **NOT gated**.
+- robustness is reported honestly and may TIE — **NOT gated**.
+- **GATE: PASS**
+
+## Verdict
+
+On this 22-query set, XERJ answered 21/22 (95.5%) versus the fair shell-only baseline's 14/22 (63.6%). **Headline (the one decisive, capability-based win): binary_only.** Answers that live only inside PDF/DOCX are invisible to ripgrep — it sees compressed/binary bytes and matches nothing — so the baseline gets 0/7, while XERJ, which extracted and indexed that text, gets 6/7 (+6). This is a capability grep structurally lacks, not a tuning artifact. **Context efficiency — real, but scale-dependent (shown, not asserted).** Over the 3 large_literal query/queries (answers buried in ≥60 KB docs that a fair grep DOES find), the baseline must open 192,782 bytes of whole answer files to quote/verify the line, while XERJ returns 11,180 bytes of ranked passages — **17.24×** less context (85.68× counting only the single best passage the agent would actually quote). On the 6 tiny-file **literal** query/queries the SAME metric INVERTS: XERJ returns 20,965 bytes of passages against the baseline's 4,741 bytes of tiny answer files — just **0.23×** (1.21× counting only the single best passage), because five returned passages meet or exceed a small whole file. Side by side — large_literal **17.24×** vs. literal **0.23×** — IS the scale-dependence, measured rather than claimed. Throughout, the baseline is charged ONLY the single answer-containing file it must open (`statSync(answer_path)`), never the false-positive files its broad terms also matched; `grep -n` yields a matching line, but to quote/verify an answer reliably an agent opens the whole file. The naive whole-corpus ratio is 2.53×, but it flatters the blind baseline — which now literally opens 0 bytes on every query it cannot answer. **robustness ≈ TIE (honest).** On differently-phrased answers the fair baseline — which greps the union of the curated keywords AND the salient tokens of the question itself — gets 5/5, and XERJ gets 5/5 (+0). XERJ's built-in embedder is a LEXICAL feature-hashing model (384-dim cosine), NOT neural, so its "semantic" matching is word/sub-word overlap, not deep understanding — a diligent grep of the question's own terms matches the same lines. We report this as a single-query convenience/robustness tie, NOT a semantic-understanding win. **literal — honest control.** On plaintext substring cases both approaches read plain text (XERJ 6/6, baseline 6/6); this confirms XERJ is not inflating the easy cases. **Caveats.** kNN is exact brute-force at query time (fine at this corpus size). The baseline could be upgraded to shell out to `pdftotext`/`soffice` to close the binary gap — but it would still lack ranking, per-chunk retrieval, and the (shallow, lexical) semantic layer, and would keep paying the whole-file context cost on large documents. **Gate: PASS.**
diff --git a/demo/usecases/doc-index/SPEC.md b/demo/usecases/doc-index/SPEC.md
new file mode 100644
index 000000000..b3ac04321
--- /dev/null
+++ b/demo/usecases/doc-index/SPEC.md
@@ -0,0 +1,177 @@
+# Use case: "Claude + XERJ — search a recursive folder of mixed-format documents"
+
+**One-line:** Point an AI agent at a wide, recursive folder of real documents
+(PDF, DOCX, HTML, Markdown, TXT). With XERJ it gets a format-agnostic,
+meaning-aware, ranked, cited search index. With only shell scripts (grep/rg) it
+is blind to binary formats, blind to meaning, and blows up its context window.
+
+This directory is the production deliverable. Everything here must run **live**
+against a real XERJ node and report **measured** numbers — no assertions, no
+fabricated results, honest caveats.
+
+## The two approaches being compared
+
+- **Baseline — "Claude Code with scripts only":** the agent answers questions by
+ running `rg`/`grep` over the raw files and reading matches into its context.
+ Honest capability ceiling: (1) PDF/DOCX are binary → `rg` sees gibberish and
+ matches nothing; (2) only literal substring match — no semantic/meaning
+ retrieval; (3) to "read" a hit it must load whole files into context (tokens);
+ (4) no ranking; (5) re-scans the whole tree every query.
+- **XERJ-backed — "Claude + XERJ":** a one-time `xerj-index` pass walks the
+ folder, extracts text from **every** format, chunks + embeds via XERJ, and bulk
+ indexes. The agent then queries XERJ (lexical + semantic + hybrid) and gets
+ ranked, cited passages back — format-agnostic, meaning-aware, no context blowup,
+ incrementally re-indexable.
+
+The comparison must be **fair**: same corpus, same query set, baseline given the
+best plain-shell tools available (`rg`, `pdftotext` is NOT given to the baseline —
+the whole point is that a naive script agent greps raw bytes; but we DO note in
+the writeup that a script agent *could* shell out to `pdftotext`, and explain why
+that still loses on format coverage, ranking, and context cost).
+
+### REVISION 2 — honesty fix (an adversarial audit found the v1 comparison unfair)
+
+The v1 run rigged the `semantic_only` set: the baseline was handed a curated
+keyword list that dodged the answer line, while the answer actually shared a line
+with the **question's own words**. A fair shell agent greps the question's terms
+and finds those answers. So the corrected, HONEST value proposition is:
+
+1. **Format coverage — the decisive, unassailable win.** grep reads raw bytes, so
+ PDF/DOCX/other binary formats are invisible (0 hits). XERJ extracts + indexes
+ them. This is a capability grep structurally lacks. THIS is the headline.
+2. **Context efficiency at document scale — a real win, but only on large docs.**
+ To answer from a big document a shell agent must load the whole file into
+ context; XERJ returns one ranked passage. On small files this INVERTS (passage
+ overhead > tiny file), so the corpus MUST include realistically large documents
+ to demonstrate it honestly.
+3. **Ranked, single-query retrieval — convenience/robustness, NOT a capability
+ gap.** One query ranks passages across the whole corpus; the shell agent must
+ guess terms/synonyms. XERJ's built-in LEXICAL embedder adds modest sub-word
+ robustness — but it is NOT neural, so on plaintext a diligent grep of the
+ question's own terms can TIE it. Do NOT claim a decisive semantic-understanding
+ win. Report the fair (≈tie) semantic result truthfully.
+
+**Fair-baseline rule (mandatory):** `grep-baseline.mjs` must derive its search
+terms from BOTH the curated `keywords` AND the salient content tokens of the
+`question` itself, and search all of them. It must never be capped to a keyword
+list that structurally excludes the answer's own line.
+
+## Honesty bar (non-negotiable — this repo has a standing honesty standard)
+
+- XERJ's built-in embedder is **lexical feature-hashing (384-dim cosine), not
+ neural**. Semantic wins here are word/sub-word overlap, not deep meaning. Say so.
+- kNN is **exact brute-force at query time**. Fine at this corpus size; disclose.
+- Do NOT claim XERJ "understands" documents. Claim: format-agnostic extraction +
+ ranked retrieval + context efficiency + a real (if shallow) semantic layer.
+- Every number in the recipe/website MUST come from `results.json` produced by a
+ live run. If the gate fails, report the real result, do not massage it.
+
+## Directory layout (all under demo/usecases/doc-index/)
+
+- `gen-corpus.mjs` — generates the corpus + `queries.json` (runs pdftotext/soffice)
+- `corpus/` — generated: `<topic>/<name>.{pdf,docx,html,md,txt}`
+- `queries.json` — ground-truth query set (see schema below)
+- `xerj-index.mjs` — the indexer: walk → extract → chunk → bulk-index into XERJ
+- `grep-baseline.mjs`— the baseline "scripts" retriever (rg over raw files)
+- `compare.mjs` — runs both over queries.json, scores, writes results.json + SCORECARD.md
+- `results.json` — measured comparison output
+- `SCORECARD.md` — human-readable measured scorecard
+- `README.md` — how to run it end-to-end (community-facing)
+
+## XERJ index contract (indexer + compare MUST agree)
+
+- Node listen: ES-compat REST at `http://localhost:${XERJ_PORT:-9209}` (`--insecure`).
+- Index name: `docfolder`.
+- Create with mappings:
+ - `path` : keyword (absolute or repo-relative file path)
+ - `dir` : keyword (parent folder, for folder-scoped search)
+ - `format` : keyword (pdf|docx|html|md|txt)
+ - `title` : text (derived from filename / first heading)
+ - `chunk_id` : integer
+ - `body` : `semantic_text` (auto-embed; built-in lexical embedder) — for semantic/hybrid
+ - `body_text`: text (SAME chunk text; for BM25 lexical / match / match_phrase)
+- Chunking: ~800 chars, ~100 char overlap, split on paragraph/sentence boundaries
+ where possible. Store both `body` and `body_text` = the chunk text.
+- Ingest via `POST /docfolder/_bulk` (NDJSON), batched (e.g. 500 docs/batch).
+- Extraction per format (in the INDEXER — it is allowed the good tools):
+ - `.pdf` → `pdftotext -layout <f> -`
+ - `.docx` → `soffice --headless --convert-to txt:Text --outdir <tmp> <f>` then read .txt (fallback: unzip word/document.xml + strip tags)
+ - `.html` → strip tags/script/style to text (stdlib parser or a small tokenizer)
+ - `.md`/`.txt` → read as-is (strip md syntax lightly for .md)
+- Be robust: skip unreadable files with a logged warning; never crash the run.
+
+## Query set contract (`queries.json`)
+
+Array of objects; **~18 queries** spanning the trap cases:
+```
+{
+ "id": "q01",
+ "question": "How many weeks of parental leave do employees get?",
+ "keywords": ["parental leave", "weeks"], // salient terms the baseline would grep
+ "expect_substring": "16 weeks", // ground-truth answer text (case-insensitive)
+ "answer_path": "corpus/hr/handbook.pdf", // file the answer lives in
+ "answer_format": "pdf",
+ "match_type": "binary_only" // binary_only | semantic_only | literal
+}
+```
+Distribution (REVISION 2 — reflects the honest value prop above):
+- **binary_only** (answer lives ONLY in a `.pdf` or `.docx`): ~7 queries. XERJ
+ finds them; the grep baseline structurally cannot. **This is the headline win.**
+- **large_literal** (answer is a literal string buried deep inside a REALISTICALLY
+ LARGE document, ≥60 KB, any format): ~4 queries. A FAIR baseline finds the line
+ but must load the whole large file into context to use/verify it; XERJ returns
+ one ranked passage. This is where the **context-efficiency** win is demonstrated
+ honestly — so both approaches should HIT, but XERJ's returned bytes ≪ the file
+ the baseline must open.
+- **robustness** (formerly "semantic_only"; answer phrased differently than the
+ query, in html/md/txt): ~4 queries. Report the FAIR result: a diligent grep of
+ the question's own terms will likely TIE XERJ here — that is the honest finding
+ (lexical embedder ≈ fuzzy word-overlap). Keep them to show XERJ's single-query
+ convenience, NOT to claim a semantic-understanding win.
+- **literal** (answer is a literal string in a small plaintext html/md/txt file):
+ ~5 queries. BOTH find these (honest control — XERJ isn't cheating on easy cases).
+
+`match_type` values: `binary_only | large_literal | robustness | literal`.
+
+Corpus domain: a realistic fictional company internal KB — "Northwind Robotics".
+HR handbook (PDF), engineering/on-call runbook (DOCX), product FAQ (HTML),
+architecture notes (MD), meeting/decision notes (TXT). Content must be coherent and
+plausible, NOT lorem ipsum. NO real personal names / usernames.
+
+## Scoring (in `compare.mjs`)
+
+For each query, per approach, obtain the top passages/hits and check:
+- **hit** = any returned passage/line contains `expect_substring` (case-insensitive).
+- XERJ: run the best supported retrieval (probe the live node — prefer `hybrid`
+ {match on body_text + semantic on body, rrf}; fall back to semantic-only merged
+ with match if hybrid errors). size=5.
+- Baseline (FAIR): `rg -i -n` for the union of the query `keywords` AND salient
+ content tokens extracted from the `question` itself, across `corpus/`; a hit
+ requires a matched line containing `expect_substring`. Binary files → no match
+ (that is the real capability gap, not a rigged one).
+- **Context cost (measured HONESTLY):** compute the token ratio ONLY over the
+ queries the BASELINE actually answers (it is meaningless on queries it can't
+ answer at all). For each such query: baseline cost = bytes of the file(s) it must
+ open to read/verify the answer (≈ chars/4 tokens); XERJ cost = bytes of the
+ ranked passage(s) it returns for that query. Report `context_ratio =
+ baseline_bytes / xerj_bytes` over answerable queries, AND break it out for the
+ `large_literal` set (where it should be ≫ 1×). Also report the naive overall
+ ratio but LABEL it as flattering the blind baseline.
+- Emit per-query rows + aggregates: coverage% (each approach), per-format coverage,
+ per-match_type coverage (binary_only / large_literal / robustness / literal),
+ context_ratio (answerable + large_literal), index build time, query latency.
+ SCORECARD verdict must state the honest framing: headline = binary_only; context
+ win shown on large_literal; robustness ≈ tie under fair baseline (lexical embedder).
+
+## GATE (must all hold on the live run, or report the honest failure)
+
+1. `xerj-index` indexes the whole corpus end-to-end against a live node, 0 crashes.
+2. XERJ overall coverage **≥** baseline, AND **strictly higher on `binary_only`**
+ (the decisive, capability-based differentiator). `robustness` is reported
+ honestly and may TIE — do NOT rig it to win.
+3. `context_ratio` on the `large_literal` set is **> 1×** (XERJ returns far fewer
+ tokens than the whole large files the baseline must load).
+4. Every number in SCORECARD.md / results.json is from the live run (reproducible),
+ and the SCORECARD verdict carries the honest framing (no overstated semantic win).
+5. Recipe + website + README carry the honesty caveats and cite only measured
+ numbers, and `README.md` lets a community user reproduce the whole thing.
diff --git a/demo/usecases/doc-index/compare.mjs b/demo/usecases/doc-index/compare.mjs
new file mode 100644
index 000000000..118c92901
--- /dev/null
+++ b/demo/usecases/doc-index/compare.mjs
@@ -0,0 +1,772 @@
+// compare.mjs — the COMPARISON HARNESS for the doc-index use case.
+//
+// Scores TWO retrieval approaches over the same ground-truth query set
+// (queries.json) and emits a measured, honest scorecard:
+//
+// * XERJ — Claude + XERJ. Queries a LIVE XERJ node (ES-compat REST) with a
+// HYBRID query (BM25 match on `body_text` + semantic on `body`,
+// fused with RRF). If the live node rejects the hybrid shape, it
+// falls back to running the semantic and match queries separately
+// and fusing them client-side with RRF. size=5.
+// * Baseline — "Claude Code with only shell scripts". Delegates to
+// grep-baseline.mjs's scoreQuery() (ripgrep over the RAW corpus).
+// Binary PDF/DOCX are invisible to rg; literal-only, no ranking.
+//
+// A query "hits" for an approach iff any returned passage/line contains the query's
+// ground-truth `expect_substring` (case-insensitive).
+//
+// Outputs (both MEASURED from this live run — nothing fabricated):
+// * results.json — machine-readable rows + aggregates + gate result.
+// * SCORECARD.md — human-readable per-query table, aggregate table, honest verdict.
+//
+// Honesty caveats carried into the verdict (per SPEC.md, non-negotiable):
+// * XERJ's built-in embedder is LEXICAL feature-hashing (384-dim cosine), NOT neural.
+// "Semantic" wins here are word/sub-word overlap, not deep meaning.
+// * kNN is EXACT brute-force at query time (fine at this corpus size).
+//
+// Exit codes:
+// 0 — scored successfully (WHETHER OR NOT the gate passed; a gate failure is reported
+// honestly in results.json/SCORECARD.md, never massaged).
+// 1 — could not run: XERJ node unreachable, `docfolder` index missing, queries.json
+// or grep-baseline.mjs missing/broken. The runner must notice these.
+//
+// `node --check compare.mjs` is clean. Pure Node (v18+); no external deps.
+
+import { readFileSync, writeFileSync, existsSync, statSync } from 'node:fs';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+import { dirname, join, isAbsolute } from 'node:path';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+
+// ── Configuration (all overridable via env so a runner can stage things) ──────────────
+const PORT = process.env.XERJ_PORT || '9209';
+const BASE = process.env.XERJ_URL || `http://localhost:${PORT}`;
+const INDEX = process.env.DOC_INDEX_INDEX || 'docfolder';
+const SIZE = 5; // top-K passages per query (SPEC-fixed)
+const RRF_K = 60; // reciprocal-rank-fusion constant (matches XERJ's server-side RRF default)
+const REQ_TIMEOUT_MS = Number(process.env.DOC_INDEX_TIMEOUT_MS || 20000);
+
+const QUERIES_PATH = resolvePath(process.env.DOC_INDEX_QUERIES, join(HERE, 'queries.json'));
+const BASELINE_PATH = join(HERE, 'grep-baseline.mjs');
+const RESULTS_PATH = join(HERE, 'results.json');
+const SCORECARD_PATH = join(HERE, 'SCORECARD.md');
+const INDEX_BUILD_FILE = join(HERE, '.index_build_ms');
+// The real corpus dir the baseline greps (staged elsewhere via DOC_INDEX_CORPUS, or the
+// co-located ./corpus). Used to resolve an answer_path to the SAME file on disk the baseline
+// reads, so the honest context charge stats the right bytes.
+const CORPUS_DIR = resolvePath(process.env.DOC_INDEX_CORPUS, join(HERE, 'corpus'));
+
+function resolvePath(envVal, fallback) {
+ if (!envVal) return fallback;
+ return isAbsolute(envVal) ? envVal : join(HERE, envVal);
+}
+
+// Resolve a queries.json answer_path ("corpus/hr/handbook.pdf") to the absolute path of the
+// real file on disk, honoring a staged corpus dir (DOC_INDEX_CORPUS) so the honest baseline
+// charge stats the SAME file the baseline greps.
+function answerAbsPath(answerPath) {
+ if (!answerPath || typeof answerPath !== 'string') return null;
+ if (isAbsolute(answerPath)) return answerPath;
+ const norm = answerPath.replace(/^\.[\\/]/, '').replace(/\\/g, '/');
+ if (norm.startsWith('corpus/')) return join(CORPUS_DIR, norm.slice('corpus/'.length));
+ return join(HERE, answerPath);
+}
+
+// HONEST baseline context charge (SPEC REVISION 2 audit fix): the size of the SINGLE
+// answer-containing file the baseline must open to read the answer. Returns the file's byte
+// size (0 if unreadable/missing). The caller charges this ONLY when the baseline actually
+// hits — else 0, because it cannot answer and so reads nothing toward an answer.
+function answerFileBytes(answerPath) {
+ const abs = answerAbsPath(answerPath);
+ if (!abs) return 0;
+ try {
+ return statSync(abs).size;
+ } catch {
+ return 0; // file vanished/unreadable — charge nothing rather than crash
+ }
+}
+
+function fatal(msg, code = 1) {
+ process.stderr.write(`[compare] FATAL: ${msg}\n`);
+ process.exit(code);
+}
+
+// ── XERJ query bodies (shapes verified against xerj-query/src/parser.rs) ──────────────
+//
+// Hybrid: fan out a BM25 match on the lexical `body_text` field and a `semantic` kNN on
+// the auto-embedded `body` field, then fuse with RRF server-side.
+function hybridBody(text) {
+ return {
+ size: SIZE,
+ query: {
+ hybrid: {
+ queries: [
+ { query: { match: { body_text: text } } },
+ { query: { semantic: { field: 'body', query: text, k: SIZE } } },
+ ],
+ fusion: 'rrf',
+ },
+ },
+ };
+}
+function semanticBody(text) {
+ return { size: SIZE, query: { semantic: { field: 'body', query: text, k: SIZE } } };
+}
+function matchBody(text) {
+ return { size: SIZE, query: { match: { body_text: text } } };
+}
+
+// ── Low-level: POST a search body, return {ok, status, json, latencyMs, networkError} ──
+async function esSearch(body) {
+ const start = performance.now();
+ let res;
+ try {
+ res = await fetch(`${BASE}/${INDEX}/_search`, {
+ method: 'POST',
+ headers: { 'content-type': 'application/json' },
+ body: JSON.stringify(body),
+ signal: AbortSignal.timeout(REQ_TIMEOUT_MS),
+ });
+ } catch (err) {
+ return { ok: false, networkError: err, latencyMs: performance.now() - start };
+ }
+ const latencyMs = performance.now() - start;
+ const raw = await res.text();
+ let json = null;
+ try {
+ json = JSON.parse(raw);
+ } catch {
+ /* non-JSON body (e.g. a plain error); leave json null */
+ }
+ return { ok: res.ok, status: res.status, json, raw, latencyMs };
+}
+
+function hitsFrom(json) {
+ const h = json && json.hits && json.hits.hits;
+ return Array.isArray(h) ? h : [];
+}
+
+// A passage's text — prefer the lexical chunk (`body_text`), then the semantic `body`,
+// then the whole _source. Used both for byte-cost accounting and the substring check.
+function passageText(hit) {
+ const s = (hit && hit._source) || {};
+ if (typeof s.body_text === 'string') return s.body_text;
+ if (typeof s.body === 'string') return s.body;
+ return JSON.stringify(s);
+}
+
+// Byte length of the SINGLE most-relevant returned passage: the first returned hit whose
+// text actually contains the expected answer substring (the passage the agent would quote),
+// else the top-ranked hit. Used to make the large_literal context win vivid (one passage
+// vs. a whole ≥60 KB file). Returns 0 when there are no hits.
+function bestPassageBytes(hits, expectLc) {
+ if (!hits.length) return 0;
+ let best = hits[0];
+ if (expectLc) {
+ const m = hits.find((h) => passageText(h).toLowerCase().includes(expectLc));
+ if (m) best = m;
+ }
+ return Buffer.byteLength(passageText(best), 'utf8');
+}
+
+// Everything a returned hit exposes, concatenated, for the case-insensitive
+// expect_substring test (body_text + body + highlight + full _source).
+function searchableBlob(hit) {
+ const s = (hit && hit._source) || {};
+ const parts = [];
+ if (typeof s.body_text === 'string') parts.push(s.body_text);
+ if (typeof s.body === 'string') parts.push(s.body);
+ if (hit && hit.highlight) parts.push(JSON.stringify(hit.highlight));
+ parts.push(JSON.stringify(s));
+ return parts.join('\n');
+}
+
+function citePaths(hits) {
+ const seen = [];
+ for (const h of hits) {
+ const s = (h && h._source) || {};
+ const p = s.path || s.dir || null;
+ if (p && !seen.includes(p)) seen.push(p);
+ }
+ return seen;
+}
+
+// Client-side RRF over two ranked hit lists (used only in the fallback path).
+function rrfMerge(listA, listB, k = RRF_K, size = SIZE) {
+ const acc = new Map(); // id -> { score, hit }
+ for (const list of [listA, listB]) {
+ list.forEach((hit, i) => {
+ const id = (hit && hit._id) || JSON.stringify(hit && hit._source);
+ const rank = i + 1;
+ const cur = acc.get(id) || { score: 0, hit };
+ cur.score += 1 / (k + rank);
+ if (!cur.hit) cur.hit = hit;
+ acc.set(id, cur);
+ });
+ }
+ return [...acc.values()].sort((a, b) => b.score - a.score).slice(0, size).map((x) => x.hit);
+}
+
+// Retrieval mode is decided once, on first use, and reused for consistent latency.
+// null -> undecided; 'hybrid' -> server-side RRF works; 'fallback' -> client-side RRF.
+let RETRIEVAL_MODE = null;
+let SEMANTIC_ERRORED = false; // note if the semantic sub-query itself failed (fallback degrades to lexical)
+
+/**
+ * Retrieve top-SIZE passages for `text` from the live XERJ node.
+ * @returns {{hits: object[], latencyMs: number, mode: string, error: string|null}}
+ * @throws the underlying network error if the node is unreachable (caller treats as fatal).
+ */
+async function xerjRetrieve(text) {
+ // Try the hybrid shape first (unless we've already learned it's unsupported).
+ if (RETRIEVAL_MODE === null || RETRIEVAL_MODE === 'hybrid') {
+ const r = await esSearch(hybridBody(text));
+ if (r.networkError) throw r.networkError;
+ if (r.ok && r.json && !r.json.error) {
+ RETRIEVAL_MODE = 'hybrid';
+ return { hits: hitsFrom(r.json).slice(0, SIZE), latencyMs: r.latencyMs, mode: 'hybrid', error: null };
+ }
+ // Hybrid rejected by this node — fall back for this and all subsequent queries.
+ if (RETRIEVAL_MODE === null) {
+ const why = (r.json && r.json.error && (r.json.error.reason || JSON.stringify(r.json.error))) || `HTTP ${r.status}`;
+ process.stderr.write(`[compare] hybrid query not supported by node (${why}); falling back to client-side RRF of semantic+match.\n`);
+ }
+ RETRIEVAL_MODE = 'fallback';
+ }
+
+ // Fallback: run semantic + match separately, fuse client-side with RRF.
+ const [rs, rm] = await Promise.all([esSearch(semanticBody(text)), esSearch(matchBody(text))]);
+ if (rs.networkError) throw rs.networkError;
+ if (rm.networkError) throw rm.networkError;
+
+ let error = null;
+ const semHits = rs.ok && rs.json && !rs.json.error ? hitsFrom(rs.json) : [];
+ const matchHits = rm.ok && rm.json && !rm.json.error ? hitsFrom(rm.json) : [];
+ if (!(rs.ok && rs.json && !rs.json.error)) {
+ SEMANTIC_ERRORED = true;
+ error = 'semantic sub-query failed; scoring lexical-only';
+ }
+ if (!(rm.ok && rm.json && !rm.json.error) && semHits.length === 0) {
+ error = 'both semantic and match sub-queries failed';
+ }
+ const merged = rrfMerge(semHits, matchHits);
+ return { hits: merged.slice(0, SIZE), latencyMs: rs.latencyMs + rm.latencyMs, mode: 'fallback', error };
+}
+
+// ── Preflight: is the node up and does the index exist? (fatal, non-zero, if not) ─────
+async function preflight() {
+ try {
+ await fetch(`${BASE}/`, { signal: AbortSignal.timeout(5000) });
+ } catch (err) {
+ fatal(`cannot reach XERJ node at ${BASE} (${err.message}). Is the node running with --insecure on port ${PORT}?`);
+ }
+ let countRes;
+ try {
+ countRes = await fetch(`${BASE}/${INDEX}/_count`, { signal: AbortSignal.timeout(5000) });
+ } catch (err) {
+ fatal(`cannot reach XERJ node at ${BASE} (${err.message}).`);
+ }
+ if (countRes.status === 404) {
+ fatal(`index '${INDEX}' not found at ${BASE}. Run xerj-index.mjs to build it first.`);
+ }
+ let docCount = null;
+ try {
+ const cj = JSON.parse(await countRes.text());
+ docCount = typeof cj.count === 'number' ? cj.count : null;
+ } catch {
+ /* ignore */
+ }
+ return { docCount };
+}
+
+// ── index_build_ms: env INDEX_BUILD_MS / --index-build-ms=N / file .index_build_ms ────
+function readIndexBuildMs() {
+ if (process.env.INDEX_BUILD_MS && !Number.isNaN(Number(process.env.INDEX_BUILD_MS))) {
+ return Number(process.env.INDEX_BUILD_MS);
+ }
+ const arg = process.argv.find((a) => a.startsWith('--index-build-ms='));
+ if (arg) {
+ const n = Number(arg.split('=')[1]);
+ if (!Number.isNaN(n)) return n;
+ }
+ if (existsSync(INDEX_BUILD_FILE)) {
+ const n = Number(readFileSync(INDEX_BUILD_FILE, 'utf8').trim());
+ if (!Number.isNaN(n)) return n;
+ }
+ return null;
+}
+
+// ── Small numeric helpers ─────────────────────────────────────────────────────────────
+function pct(n, d) {
+ return d === 0 ? 0 : (n / d) * 100;
+}
+function round(n, dp = 1) {
+ if (n === null || n === undefined || Number.isNaN(n)) return null;
+ const f = 10 ** dp;
+ return Math.round(n * f) / f;
+}
+function median(nums) {
+ if (!nums.length) return null;
+ const s = [...nums].sort((a, b) => a - b);
+ const mid = Math.floor(s.length / 2);
+ return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
+}
+function coverage(rows, key) {
+ const total = rows.length;
+ const hits = rows.filter((r) => r[key]).length;
+ return { total, hits, pct: round(pct(hits, total)) };
+}
+
+// ── Main ──────────────────────────────────────────────────────────────────────────────
+async function main() {
+ // Load inputs (any missing → fatal non-zero).
+ if (!existsSync(QUERIES_PATH)) fatal(`queries.json not found at ${QUERIES_PATH}. Generate it with gen-corpus.mjs.`);
+ let queries;
+ try {
+ queries = JSON.parse(readFileSync(QUERIES_PATH, 'utf8'));
+ } catch (err) {
+ fatal(`could not parse queries.json (${err.message}).`);
+ }
+ if (!Array.isArray(queries) || queries.length === 0) fatal('queries.json is empty or not an array.');
+
+ if (!existsSync(BASELINE_PATH)) fatal(`grep-baseline.mjs not found at ${BASELINE_PATH}.`);
+ let scoreQuery;
+ try {
+ const mod = await import(pathToFileURL(BASELINE_PATH).href);
+ scoreQuery = mod.scoreQuery || (mod.default && mod.default.scoreQuery) || mod.default;
+ } catch (err) {
+ fatal(`could not import grep-baseline.mjs (${err.message}).`);
+ }
+ if (typeof scoreQuery !== 'function') fatal('grep-baseline.mjs must export a scoreQuery() function.');
+
+ // Node + index must be live.
+ const { docCount } = await preflight();
+
+ // Score every query.
+ const rows = [];
+ for (const q of queries) {
+ const expect = String(q.expect_substring || '');
+ const expectLc = expect.toLowerCase();
+ const question = String(q.question || '');
+
+ // XERJ retrieval (network error here is fatal — the runner must notice).
+ let xerj;
+ try {
+ xerj = await xerjRetrieve(question);
+ } catch (err) {
+ fatal(`lost connection to XERJ node mid-run (${err.message}).`);
+ }
+ const xerjBlobs = xerj.hits.map(searchableBlob);
+ const xerjHit = expectLc !== '' && xerjBlobs.some((b) => b.toLowerCase().includes(expectLc));
+ const xerjReturnedBytes = xerj.hits.reduce((sum, h) => sum + Buffer.byteLength(passageText(h), 'utf8'), 0);
+ const xerjBestPassageBytes = bestPassageBytes(xerj.hits, expectLc);
+
+ // Baseline retrieval (grep-baseline.scoreQuery is synchronous).
+ let base = { hits: [], bytesToRead: 0 };
+ try {
+ const r = scoreQuery(q);
+ if (r && typeof r === 'object') base = r;
+ } catch (err) {
+ process.stderr.write(`[compare] baseline scoreQuery failed for ${q.id}: ${err.message}\n`);
+ }
+ const baseLines = Array.isArray(base.hits) ? base.hits.map((h) => String((h && h.text) ?? h ?? '')) : [];
+ const baselineHit = expectLc !== '' && baseLines.some((t) => t.toLowerCase().includes(expectLc));
+
+ // HONEST baseline context charge (SPEC REVISION 2): the size of the SINGLE answer-
+ // containing file the baseline must open to read the answer, and ONLY when the baseline
+ // actually hits — else 0 (it cannot answer, so it reads nothing toward an answer). We do
+ // NOT charge the false-positive files its broad terms also matched (that inflated every
+ // ratio to a flat ~52× and broke the scale-dependence claim — the defect this fixes).
+ const answerBytes = answerFileBytes(q.answer_path);
+ const baselineContextBytes = baselineHit ? answerBytes : 0;
+ // DIAGNOSTIC ONLY — grep-baseline's bytesToRead: the sum of EVERY file the broad terms
+ // matched (false-positive files included). Kept for transparency; NEVER a charge/claim.
+ const baselineAllMatchedBytes = Number(
+ base.bytesToRead ?? base.bytes_to_read ?? base.baseline_bytes_to_read ?? base.bytes ?? 0,
+ ) || 0;
+
+ rows.push({
+ id: q.id,
+ question,
+ match_type: q.match_type || 'unknown',
+ answer_format: q.answer_format || 'unknown',
+ answer_path: q.answer_path || null,
+ expect_substring: expect,
+ xerj_hit: xerjHit,
+ baseline_hit: baselineHit,
+ xerj_latency_ms: round(xerj.latencyMs, 2),
+ xerj_returned_bytes: xerjReturnedBytes,
+ xerj_best_passage_bytes: xerjBestPassageBytes,
+ answer_file_bytes: answerBytes, // size of the answer-containing file (regardless of hit)
+ baseline_context_bytes: baselineContextBytes, // HONEST charge: answer file size if hit, else 0
+ baseline_all_matched_bytes: baselineAllMatchedBytes, // DIAGNOSTIC: every matched file (FP incl.)
+ xerj_returned: xerj.hits.length,
+ xerj_cite_paths: citePaths(xerj.hits),
+ xerj_error: xerj.error,
+ });
+ }
+
+ // ── Aggregate ──────────────────────────────────────────────────────────────────────
+ const N = rows.length;
+ const xerjCov = coverage(rows, 'xerj_hit');
+ const baseCov = coverage(rows, 'baseline_hit');
+
+ // REVISION 2 canonical match types (SPEC §Query set). Any other type found in
+ // queries.json is still aggregated, appended after these.
+ const MATCH_TYPES = ['binary_only', 'large_literal', 'robustness', 'literal'];
+ const perMatchType = {};
+ for (const mt of MATCH_TYPES.concat(
+ [...new Set(rows.map((r) => r.match_type))].filter((m) => !MATCH_TYPES.includes(m)),
+ )) {
+ const sub = rows.filter((r) => r.match_type === mt);
+ if (!sub.length) continue;
+ perMatchType[mt] = {
+ total: sub.length,
+ xerj_hits: sub.filter((r) => r.xerj_hit).length,
+ baseline_hits: sub.filter((r) => r.baseline_hit).length,
+ xerj_only: sub.filter((r) => r.xerj_hit && !r.baseline_hit).length, // "wins"
+ xerj_coverage_pct: coverage(sub, 'xerj_hit').pct,
+ baseline_coverage_pct: coverage(sub, 'baseline_hit').pct,
+ };
+ }
+
+ const perFormat = {};
+ for (const fmt of [...new Set(rows.map((r) => r.answer_format))].sort()) {
+ const sub = rows.filter((r) => r.answer_format === fmt);
+ perFormat[fmt] = {
+ total: sub.length,
+ xerj_hits: sub.filter((r) => r.xerj_hit).length,
+ baseline_hits: sub.filter((r) => r.baseline_hit).length,
+ xerj_coverage_pct: coverage(sub, 'xerj_hit').pct,
+ baseline_coverage_pct: coverage(sub, 'baseline_hit').pct,
+ };
+ }
+
+ const g = (mt, key) => (perMatchType[mt] ? perMatchType[mt][key] : 0);
+ const mtHits = (mt) => ({
+ total: g(mt, 'total'),
+ xerj: g(mt, 'xerj_hits'),
+ baseline: g(mt, 'baseline_hits'),
+ xerj_only_wins: g(mt, 'xerj_only'),
+ });
+
+ // ── Context cost (bytes → tokens ≈ /4; the /4 cancels in the ratio) ──────────────────
+ // HONEST baseline charge (SPEC REVISION 2 audit fix): per query, the baseline's context
+ // cost = the size of the SINGLE answer-containing file it must open, charged ONLY when the
+ // baseline actually hits (else 0 — it reads nothing toward an answer it cannot find). The
+ // false-positive files the broad terms also matched are NOT charged; they are kept ONLY as
+ // a labelled diagnostic (`baseline_all_matched_bytes`). Charging them inflated every ratio
+ // to a flat ~52× and made the scale-dependence claim false — the defect this fixes.
+ //
+ // Views, per SPEC §Scoring / REVISION 2:
+ // 1. NAIVE overall — every query. FLATTERS the blind baseline: on queries it cannot
+ // answer it opens 0 bytes (now literally true), dragging its total down. Not a claim.
+ // 2. ANSWERABLE — only queries the baseline actually answers.
+ // 3. LARGE_LITERAL — the real context win: answers buried in ≥60 KB docs a fair grep DOES
+ // find but must open the whole file to quote; XERJ returns one ranked passage. ≫ 1×.
+ // 4. LITERAL — small plaintext files: XERJ's 5 returned passages ≈/exceed a tiny file, so
+ // the ratio is ~1× or below. Shown beside large_literal to make the scale-dependence
+ // VISIBLE (headline the conservative returned-passages number, best-passage too).
+ const contextBucket = (pred) => {
+ const sub = rows.filter(pred);
+ const baseline_bytes = sub.reduce((s, r) => s + r.baseline_context_bytes, 0);
+ const xerj_returned_bytes = sub.reduce((s, r) => s + r.xerj_returned_bytes, 0);
+ const xerj_best_passage_bytes = sub.reduce((s, r) => s + r.xerj_best_passage_bytes, 0);
+ return {
+ queries: sub.length,
+ baseline_bytes,
+ xerj_returned_bytes,
+ xerj_best_passage_bytes,
+ baseline_tokens_approx: Math.round(baseline_bytes / 4),
+ xerj_tokens_approx: Math.round(xerj_returned_bytes / 4),
+ context_ratio: xerj_returned_bytes > 0 ? round(baseline_bytes / xerj_returned_bytes, 2) : null,
+ context_ratio_best_passage:
+ xerj_best_passage_bytes > 0 ? round(baseline_bytes / xerj_best_passage_bytes, 2) : null,
+ };
+ };
+
+ const naiveBucket = contextBucket(() => true);
+ const answerableBucket = contextBucket((r) => r.baseline_hit);
+ // Only queries the baseline actually answers count toward each context win (by design a
+ // fair grep finds the literal/large_literal lines; guarded in case one is missed).
+ const largeLiteralBucket = contextBucket((r) => r.match_type === 'large_literal' && r.baseline_hit);
+ const literalBucket = contextBucket((r) => r.match_type === 'literal' && r.baseline_hit);
+
+ const naiveRatio = naiveBucket.context_ratio;
+ const contextRatioAnswerable = answerableBucket.context_ratio;
+ const contextRatioLargeLiteral = largeLiteralBucket.context_ratio;
+ const contextRatioLargeLiteralBest = largeLiteralBucket.context_ratio_best_passage;
+ const contextRatioLiteral = literalBucket.context_ratio;
+ const contextRatioLiteralBest = literalBucket.context_ratio_best_passage;
+
+ // DIAGNOSTIC ONLY — the OLD (inflated) charge: sum over ALL queries of every file the
+ // baseline's broad terms matched (false positives included). Never used in a ratio/claim.
+ const baselineAllMatchedTotal = rows.reduce((s, r) => s + r.baseline_all_matched_bytes, 0);
+
+ const latencies = rows.map((r) => r.xerj_latency_ms).filter((x) => typeof x === 'number');
+ const latency = {
+ xerj_p50_ms: round(median(latencies), 2),
+ xerj_mean_ms: round(latencies.reduce((s, x) => s + x, 0) / (latencies.length || 1), 2),
+ xerj_max_ms: latencies.length ? round(Math.max(...latencies), 2) : null,
+ };
+
+ const indexBuildMs = readIndexBuildMs();
+
+ // ── GATE (SPEC §GATE, REVISION 2) ───────────────────────────────────────────────────
+ // #2 overall coverage ≥ baseline AND strictly higher on binary_only (the decisive,
+ // capability-based differentiator: grep structurally cannot read PDF/DOCX bytes).
+ // #3 context_ratio on the large_literal set is > 1× (XERJ returns far fewer tokens
+ // than the whole large files the baseline must load).
+ // `robustness` is reported HONESTLY and may TIE under a fair baseline — it is NOT gated.
+ const checkOverall = xerjCov.hits >= baseCov.hits;
+ const checkBinary = g('binary_only', 'xerj_hits') > g('binary_only', 'baseline_hits');
+ const checkLargeLiteralContext =
+ contextRatioLargeLiteral !== null && contextRatioLargeLiteral > 1;
+ const gatePass = checkOverall && checkBinary && checkLargeLiteralContext;
+
+ const results = {
+ generated_at: new Date().toISOString(),
+ node: BASE,
+ index: INDEX,
+ index_doc_count: docCount,
+ size: SIZE,
+ retrieval_mode: RETRIEVAL_MODE || 'none',
+ semantic_subquery_errored: SEMANTIC_ERRORED,
+ xerj_query_example: RETRIEVAL_MODE === 'fallback' ? { semantic: semanticBody('<question>'), match: matchBody('<question>') } : hybridBody('<question>'),
+ index_build_ms: indexBuildMs,
+ honesty: {
+ embedder: 'XERJ built-in embedder is lexical feature-hashing (384-dim cosine), NOT neural; "semantic"/robustness wins are word/sub-word overlap, not deep meaning. On plaintext a fair grep of the question\'s own terms ties it.',
+ knn: 'kNN is exact brute-force at query time (fine at this corpus size).',
+ baseline: 'Baseline is a FAIR shell-only agent: ripgrep over raw bytes for the union of the curated keywords AND the salient tokens of the question itself. It could shell out to pdftotext, but would still lose on format coverage at scale, ranking, per-chunk retrieval, and whole-file context cost.',
+ context: 'Context efficiency is measured ONLY over queries the baseline answers, charging the baseline ONLY the single answer-containing file it must open (statSync(answer_path).size) — never the false-positive files its broad terms also matched (kept as a labelled diagnostic). It is a real win only on large documents (large_literal ≫ 1×) and INVERTS on tiny plaintext files (literal ≈ 1× or below), which is why BOTH buckets are shown side by side. The naive overall ratio flatters the blind baseline — it now literally opens 0 bytes on every query it cannot answer — and is reported for transparency, not as a claim.',
+ },
+ totals: {
+ queries: N,
+ xerj_hits: xerjCov.hits,
+ baseline_hits: baseCov.hits,
+ xerj_coverage_pct: xerjCov.pct,
+ baseline_coverage_pct: baseCov.pct,
+ },
+ per_match_type: perMatchType,
+ per_format: perFormat,
+ match_type_hits: {
+ binary_only: mtHits('binary_only'),
+ large_literal: mtHits('large_literal'),
+ robustness: mtHits('robustness'),
+ literal: mtHits('literal'),
+ },
+ context_cost: {
+ method:
+ 'Baseline is charged ONLY the size of the single answer-containing file it must open (statSync(answer_path).size) when it hits, else 0. False-positive term-matched files are NOT charged; their total is kept as a labelled diagnostic (diagnostic_all_matched_bytes). grep -n yields a matching line, but to quote/verify an answer reliably an agent opens the whole answer file — that single file is the charge.',
+ // (1) NAIVE overall — labelled; flatters the blind baseline.
+ naive_overall: {
+ note: 'flatters the blind baseline — it now literally opens 0 bytes on every query it cannot answer — NOT a claim',
+ baseline_total_bytes: naiveBucket.baseline_bytes,
+ xerj_total_bytes: naiveBucket.xerj_returned_bytes,
+ baseline_tokens_approx: naiveBucket.baseline_tokens_approx,
+ xerj_tokens_approx: naiveBucket.xerj_tokens_approx,
+ ratio_baseline_over_xerj: naiveRatio,
+ },
+ // (2) ANSWERABLE — only queries the baseline actually answers.
+ answerable: {
+ queries: answerableBucket.queries,
+ baseline_bytes: answerableBucket.baseline_bytes,
+ xerj_returned_bytes: answerableBucket.xerj_returned_bytes,
+ baseline_tokens_approx: answerableBucket.baseline_tokens_approx,
+ xerj_tokens_approx: answerableBucket.xerj_tokens_approx,
+ context_ratio_answerable: contextRatioAnswerable,
+ },
+ // (3) LARGE_LITERAL — the real, honest context win (expected ≫ 1×).
+ large_literal: {
+ queries: largeLiteralBucket.queries,
+ baseline_bytes: largeLiteralBucket.baseline_bytes,
+ xerj_returned_bytes: largeLiteralBucket.xerj_returned_bytes,
+ xerj_best_passage_bytes: largeLiteralBucket.xerj_best_passage_bytes,
+ context_ratio_large_literal: contextRatioLargeLiteral,
+ context_ratio_large_literal_best_passage: contextRatioLargeLiteralBest,
+ },
+ // (4) LITERAL — small plaintext files; the win INVERTS to ≈1× or below. Shown beside
+ // large_literal to demonstrate scale-dependence. Headline = returned-passages ratio
+ // (conservative — does not flatter XERJ); the single-best-passage variant is shown too.
+ literal: {
+ queries: literalBucket.queries,
+ baseline_bytes: literalBucket.baseline_bytes,
+ xerj_returned_bytes: literalBucket.xerj_returned_bytes,
+ xerj_best_passage_bytes: literalBucket.xerj_best_passage_bytes,
+ context_ratio_literal: contextRatioLiteral,
+ context_ratio_literal_best_passage: contextRatioLiteralBest,
+ },
+ // DIAGNOSTIC — the OLD inflated all-matched-bytes total. Labelled; NOT a claim.
+ diagnostic_all_matched_bytes: {
+ note: "DIAGNOSTIC ONLY — sum over ALL queries of the size of EVERY file the baseline's broad terms matched (false-positive files included). This is the pre-fix inflated charge; NOT used in any ratio or claim.",
+ baseline_all_matched_bytes_total: baselineAllMatchedTotal,
+ },
+ },
+ latency,
+ gate: {
+ pass: gatePass,
+ checks: {
+ overall_coverage_ge_baseline: checkOverall,
+ binary_only_strictly_higher: checkBinary,
+ large_literal_context_ratio_gt_1: checkLargeLiteralContext,
+ },
+ // Informational (NOT gated): robustness is expected to ≈ tie under a fair baseline.
+ robustness_tie_note: `robustness reported honestly (XERJ ${g('robustness', 'xerj_hits')}/${g('robustness', 'total')} vs baseline ${g('robustness', 'baseline_hits')}/${g('robustness', 'total')}); a lexical embedder ties a fair grep — not gated.`,
+ },
+ queries: rows,
+ };
+
+ writeFileSync(RESULTS_PATH, JSON.stringify(results, null, 2));
+ writeFileSync(SCORECARD_PATH, renderScorecard(results));
+
+ // Console summary.
+ process.stdout.write(
+ `[compare] mode=${results.retrieval_mode} | XERJ ${xerjCov.hits}/${N} (${xerjCov.pct}%) vs baseline ${baseCov.hits}/${N} (${baseCov.pct}%) | ` +
+ `binary_only ${g('binary_only', 'xerj_hits')}-${g('binary_only', 'baseline_hits')} (headline) | robustness ${g('robustness', 'xerj_hits')}-${g('robustness', 'baseline_hits')} (≈tie) | ` +
+ `ctx large_literal ${contextRatioLargeLiteral ?? 'n/a'}× (naive overall ${naiveRatio ?? 'n/a'}×) | GATE ${gatePass ? 'PASS' : 'FAIL'}\n`,
+ );
+ process.stdout.write(`[compare] wrote ${RESULTS_PATH} and ${SCORECARD_PATH}\n`);
+ // Gate failures are reported honestly, NOT turned into a non-zero exit (SPEC).
+ process.exit(0);
+}
+
+// ── SCORECARD.md renderer (all values come straight from `results`) ───────────────────
+function renderScorecard(r) {
+ const yn = (b) => (b ? 'hit' : '—');
+ const t = r.totals;
+ const mth = r.match_type_hits;
+ const cc = r.context_cost;
+ const no = cc.naive_overall;
+ const ans = cc.answerable;
+ const ll = cc.large_literal;
+ const lit = cc.literal;
+ const diag = cc.diagnostic_all_matched_bytes;
+ const num = (x) => (typeof x === 'number' ? x.toLocaleString() : String(x ?? 'n/a'));
+ const x = (v) => (v === null || v === undefined ? 'n/a' : `${v}×`);
+
+ const lines = [];
+ lines.push('# Doc-index scorecard — Claude + XERJ vs. shell-only (grep) baseline');
+ lines.push('');
+ lines.push(`_Measured live on ${r.generated_at} against \`${r.node}\` (index \`${r.index}\`, ${r.index_doc_count ?? '?'} chunks). Retrieval mode: **${r.retrieval_mode}**. Every number below is from this live run._`);
+ lines.push('');
+ lines.push('> **Honest framing (REVISION 2).** The decisive, capability-based win is **binary_only** — a fair grep structurally cannot read PDF/DOCX bytes. **Context efficiency** is a real win, but only shows up on **large_literal** (big documents). **robustness** ≈ **TIE** under a fair baseline, because XERJ\'s built-in embedder is a LEXICAL feature-hashing model (384-dim cosine), NOT neural — a diligent grep of the question\'s own terms matches the same lines. **literal** is the honest control. No overstated semantic claim is made.');
+ lines.push('');
+
+ // Per-query table.
+ lines.push('## Per-query results');
+ lines.push('');
+ lines.push('| ID | Type | Fmt | XERJ | Base | XERJ ms | XERJ psg B | Best psg B | Base answer-file B | Base matched B (diag) | Question |');
+ lines.push('|----|------|-----|------|------|--------:|-----------:|-----------:|-------------------:|----------------------:|----------|');
+ for (const q of r.queries) {
+ const ql = q.question.length > 60 ? q.question.slice(0, 57) + '...' : q.question;
+ lines.push(
+ `| ${q.id} | ${q.match_type} | ${q.answer_format} | ${yn(q.xerj_hit)} | ${yn(q.baseline_hit)} | ${q.xerj_latency_ms ?? '?'} | ${q.xerj_returned_bytes} | ${q.xerj_best_passage_bytes ?? '?'} | ${q.baseline_context_bytes ?? 0} | ${q.baseline_all_matched_bytes ?? 0} | ${ql.replace(/\|/g, '\\|')} |`,
+ );
+ }
+ lines.push('');
+ lines.push('_"Base answer-file B" is the HONEST context charge: the size of the single answer-containing file the baseline must open, and **0 when the baseline cannot answer** (it reads nothing toward an answer it cannot find). "Base matched B (diag)" is a DIAGNOSTIC only — the size of every file the baseline\'s broad terms matched, false positives included — and is used in no ratio or claim._');
+ lines.push('');
+
+ // Aggregate table.
+ lines.push('## Aggregate coverage by match type');
+ lines.push('');
+ lines.push('| Metric | XERJ | Baseline | Note |');
+ lines.push('|--------|------|----------|------|');
+ lines.push(`| Overall coverage | ${t.xerj_hits}/${t.queries} (${t.xerj_coverage_pct}%) | ${t.baseline_hits}/${t.queries} (${t.baseline_coverage_pct}%) | XERJ ≥ baseline |`);
+ lines.push(`| **binary_only** (answer only in PDF/DOCX) | ${mth.binary_only.xerj}/${mth.binary_only.total} | ${mth.binary_only.baseline}/${mth.binary_only.total} | **HEADLINE — capability grep lacks** |`);
+ lines.push(`| large_literal (buried in a ≥60 KB doc) | ${mth.large_literal.xerj}/${mth.large_literal.total} | ${mth.large_literal.baseline}/${mth.large_literal.total} | both hit; see context win below |`);
+ lines.push(`| robustness (differently-phrased) | ${mth.robustness.xerj}/${mth.robustness.total} | ${mth.robustness.baseline}/${mth.robustness.total} | ≈ TIE under fair baseline (lexical embedder) |`);
+ lines.push(`| literal (plaintext substring) | ${mth.literal.xerj}/${mth.literal.total} | ${mth.literal.baseline}/${mth.literal.total} | honest control — both read plain text |`);
+ lines.push('');
+
+ // Per-format coverage.
+ lines.push('### Coverage by answer format');
+ lines.push('');
+ lines.push('| Format | Queries | XERJ | Baseline |');
+ lines.push('|--------|--------:|------|----------|');
+ for (const [fmt, v] of Object.entries(r.per_format)) {
+ lines.push(`| ${fmt} | ${v.total} | ${v.xerj_hits}/${v.total} (${v.xerj_coverage_pct}%) | ${v.baseline_hits}/${v.total} (${v.baseline_coverage_pct}%) |`);
+ }
+ lines.push('');
+
+ // Context cost — the HONEST views (scale-dependence shown, not asserted).
+ lines.push('### Context efficiency (measured honestly)');
+ lines.push('');
+ lines.push('Context ratio = baseline bytes to open ÷ XERJ bytes returned. The baseline is charged ONLY the single answer-containing file it must open (`statSync(answer_path)`), **never** the false-positive files its broad terms also matched. `grep -n` yields a matching line, but to quote/verify an answer reliably an agent opens the whole answer file — so that single file is the charge (stated plainly as the assumption).');
+ lines.push('');
+ lines.push('| View | Baseline bytes | XERJ bytes | Ratio | What it means |');
+ lines.push('|------|---------------:|-----------:|:-----:|---------------|');
+ lines.push(`| **large_literal** (returned passages) | ${num(ll.baseline_bytes)} | ${num(ll.xerj_returned_bytes)} | **${x(ll.context_ratio_large_literal)}** | THE REAL WIN — big files vs. ranked passages (over ${ll.queries} query/queries the baseline answers) |`);
+ lines.push(`| large_literal (single best passage) | ${num(ll.baseline_bytes)} | ${num(ll.xerj_best_passage_bytes)} | ${x(ll.context_ratio_large_literal_best_passage)} | one passage the agent would actually quote |`);
+ lines.push(`| **literal** (returned passages) | ${num(lit.baseline_bytes)} | ${num(lit.xerj_returned_bytes)} | **${x(lit.context_ratio_literal)}** | SCALE-DEPENDENCE: on tiny plaintext files the win INVERTS — 5 returned passages ≈/exceed the whole small file (over ${lit.queries} query/queries) |`);
+ lines.push(`| literal (single best passage) | ${num(lit.baseline_bytes)} | ${num(lit.xerj_best_passage_bytes)} | ${x(lit.context_ratio_literal_best_passage)} | even a single passage ≈ a tiny file |`);
+ lines.push(`| answerable (all queries baseline answers) | ${num(ans.baseline_bytes)} | ${num(ans.xerj_returned_bytes)} | ${x(ans.context_ratio_answerable)} | fair whole-corpus view over ${ans.queries} answerable queries |`);
+ lines.push(`| naive overall (all queries) | ${num(no.baseline_total_bytes)} | ${num(no.xerj_total_bytes)} | ${x(no.ratio_baseline_over_xerj)} | **flatters the blind baseline — it literally opens 0 bytes on every query it cannot answer** — NOT a claim |`);
+ lines.push('');
+ lines.push(`**Scale-dependence, side by side:** large_literal **${x(ll.context_ratio_large_literal)}** (answers buried in ≥60 KB docs) vs. literal **${x(lit.context_ratio_literal)}** (tiny plaintext files). The context win is real on large documents and INVERTS on small ones — demonstrated on measured data, not asserted.`);
+ lines.push('');
+ lines.push(`_Diagnostic (NOT a claim): had the baseline instead been charged every file its broad terms matched — false positives included — the total would be ${num(diag.baseline_all_matched_bytes_total)} bytes. That inflated charge is exactly what this scorecard does NOT use._`);
+ lines.push('');
+ lines.push(`_Latency: XERJ query p50 / mean / max = ${r.latency.xerj_p50_ms} / ${r.latency.xerj_mean_ms} / ${r.latency.xerj_max_ms} ms. Index build time: ${r.index_build_ms ?? 'not recorded'} ms._`);
+ lines.push('');
+
+ // Gate.
+ lines.push('### Gate');
+ lines.push('');
+ lines.push(`- (#2) Overall coverage ≥ baseline: **${r.gate.checks.overall_coverage_ge_baseline ? 'PASS' : 'FAIL'}**`);
+ lines.push(`- (#2) binary_only strictly higher (the capability win): **${r.gate.checks.binary_only_strictly_higher ? 'PASS' : 'FAIL'}**`);
+ lines.push(`- (#3) large_literal context ratio > 1×: **${r.gate.checks.large_literal_context_ratio_gt_1 ? 'PASS' : 'FAIL'}** (${x(ll.context_ratio_large_literal)})`);
+ lines.push(`- (context, informational) literal ratio ≈1× or below — the small-file inversion that proves scale-dependence: ${x(lit.context_ratio_literal)} — **NOT gated**.`);
+ lines.push(`- robustness is reported honestly and may TIE — **NOT gated**.`);
+ lines.push(`- **GATE: ${r.gate.pass ? 'PASS' : 'FAIL'}**`);
+ lines.push('');
+
+ // Honest verdict paragraph — generated from the measured numbers.
+ lines.push('## Verdict');
+ lines.push('');
+ const winBinary = mth.binary_only.xerj - mth.binary_only.baseline;
+ const winRobust = mth.robustness.xerj - mth.robustness.baseline;
+ const verdict = [];
+ verdict.push(
+ `On this ${t.queries}-query set, XERJ answered ${t.xerj_hits}/${t.queries} (${t.xerj_coverage_pct}%) versus the fair shell-only baseline's ${t.baseline_hits}/${t.queries} (${t.baseline_coverage_pct}%).`,
+ );
+ verdict.push(
+ `**Headline (the one decisive, capability-based win): binary_only.** Answers that live only inside PDF/DOCX are invisible to ripgrep — it sees compressed/binary bytes and matches nothing — so the baseline gets ${mth.binary_only.baseline}/${mth.binary_only.total}, while XERJ, which extracted and indexed that text, gets ${mth.binary_only.xerj}/${mth.binary_only.total} (+${winBinary}). This is a capability grep structurally lacks, not a tuning artifact.`,
+ );
+ if (ll.queries > 0 && ll.context_ratio_large_literal !== null) {
+ const litClause = lit.context_ratio_literal !== null
+ ? `On the ${lit.queries} tiny-file **literal** query/queries the SAME metric INVERTS: XERJ returns ${num(lit.xerj_returned_bytes)} bytes of passages against the baseline's ${num(lit.baseline_bytes)} bytes of tiny answer files — just **${x(lit.context_ratio_literal)}** (${x(lit.context_ratio_literal_best_passage)} counting only the single best passage), because five returned passages meet or exceed a small whole file.`
+ : `(No answerable literal queries were available to show the small-file inversion in this run.)`;
+ verdict.push(
+ `**Context efficiency — real, but scale-dependent (shown, not asserted).** Over the ${ll.queries} large_literal query/queries (answers buried in ≥60 KB docs that a fair grep DOES find), the baseline must open ${num(ll.baseline_bytes)} bytes of whole answer files to quote/verify the line, while XERJ returns ${num(ll.xerj_returned_bytes)} bytes of ranked passages — **${x(ll.context_ratio_large_literal)}** less context (${x(ll.context_ratio_large_literal_best_passage)} counting only the single best passage the agent would actually quote). ${litClause} Side by side — large_literal **${x(ll.context_ratio_large_literal)}** vs. literal **${x(lit.context_ratio_literal)}** — IS the scale-dependence, measured rather than claimed. Throughout, the baseline is charged ONLY the single answer-containing file it must open (\`statSync(answer_path)\`), never the false-positive files its broad terms also matched; \`grep -n\` yields a matching line, but to quote/verify an answer reliably an agent opens the whole file. The naive whole-corpus ratio is ${x(no.ratio_baseline_over_xerj)}, but it flatters the blind baseline — which now literally opens 0 bytes on every query it cannot answer.`,
+ );
+ } else {
+ const why = mth.large_literal.total === 0
+ ? 'No large_literal queries were present in this run'
+ : `large_literal queries were present (${mth.large_literal.total}) but the baseline answered none of them, so the whole-file-vs-passage ratio could not be measured on the answerable basis`;
+ verdict.push(
+ `**Context efficiency — real, but only on large documents.** ${why}, so the scale-dependent context win could not be demonstrated here; the naive whole-corpus ratio (${x(no.ratio_baseline_over_xerj)}) is reported for transparency only and flatters the blind baseline, which now literally opens 0 bytes on every query it cannot answer. Throughout, the baseline is charged ONLY the single answer-containing file it must open, never the false-positive files its broad terms also matched.`,
+ );
+ }
+ verdict.push(
+ `**robustness ≈ TIE (honest).** On differently-phrased answers the fair baseline — which greps the union of the curated keywords AND the salient tokens of the question itself — gets ${mth.robustness.baseline}/${mth.robustness.total}, and XERJ gets ${mth.robustness.xerj}/${mth.robustness.total} (${winRobust >= 0 ? '+' : ''}${winRobust}). XERJ's built-in embedder is a LEXICAL feature-hashing model (384-dim cosine), NOT neural, so its "semantic" matching is word/sub-word overlap, not deep understanding — a diligent grep of the question's own terms matches the same lines. We report this as a single-query convenience/robustness tie, NOT a semantic-understanding win.`,
+ );
+ verdict.push(
+ `**literal — honest control.** On plaintext substring cases both approaches read plain text (XERJ ${mth.literal.xerj}/${mth.literal.total}, baseline ${mth.literal.baseline}/${mth.literal.total}); this confirms XERJ is not inflating the easy cases.`,
+ );
+ verdict.push(
+ `**Caveats.** kNN is exact brute-force at query time (fine at this corpus size). The baseline could be upgraded to shell out to \`pdftotext\`/\`soffice\` to close the binary gap — but it would still lack ranking, per-chunk retrieval, and the (shallow, lexical) semantic layer, and would keep paying the whole-file context cost on large documents.`,
+ );
+ verdict.push(`**Gate: ${r.gate.pass ? 'PASS' : 'FAIL'}.**`);
+ if (r.semantic_subquery_errored) {
+ verdict.push(`_Note: the semantic sub-query errored on this run; XERJ was scored lexical-only._`);
+ }
+ lines.push(verdict.join(' '));
+ lines.push('');
+
+ return lines.join('\n');
+}
+
+main().catch((err) => {
+ fatal(`unexpected error: ${err && err.stack ? err.stack : err}`);
+});
diff --git a/demo/usecases/doc-index/gen-corpus.mjs b/demo/usecases/doc-index/gen-corpus.mjs
new file mode 100644
index 000000000..587ca661b
--- /dev/null
+++ b/demo/usecases/doc-index/gen-corpus.mjs
@@ -0,0 +1,2116 @@
+#!/usr/bin/env node
+// gen-corpus.mjs — Northwind Robotics internal-KB corpus + ground-truth query set.
+//
+// Produces a wide, recursive folder of MIXED-format documents under ./corpus/
+// (.pdf .docx .html .md .txt) plus ./queries.json, for the doc-index use case
+// (see SPEC.md). The .pdf/.docx are REAL binaries: we author a flat-ODF (.fodt)
+// source and convert it with LibreOffice (soffice), so the same extraction the
+// indexer uses (pdftotext / soffice->txt) round-trips the answer text.
+//
+// The script is deterministic + idempotent: it wipes and regenerates corpus/
+// every run, and it SELF-VERIFIES the ground-truth invariants that make the
+// XERJ-vs-grep comparison honest (see verify() below). If any invariant is
+// violated it throws — the corpus is never left in a subtly-wrong state.
+//
+// Requirements: node >= 18, soffice/libreoffice, pdftotext (poppler).
+//
+// Usage: node gen-corpus.mjs
+//
+// Honesty note: this is synthetic but coherent prose for a fictional company
+// ("Northwind Robotics"). No real personal names; roles only.
+
+import { execFileSync } from "node:child_process";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const CORPUS = path.join(HERE, "corpus");
+const QUERIES_JSON = path.join(HERE, "queries.json");
+// Scratch build area (soffice profile + intermediate .fodt / conversion output).
+const BUILD = fs.mkdtempSync(path.join(os.tmpdir(), "nwr-corpus-"));
+const PROFILE = path.join(BUILD, "profile");
+const SOFFICE_ENV = `-env:UserInstallation=file://${PROFILE}`;
+
+// ---------------------------------------------------------------------------
+// Text-format corpus files (.html .md .txt). Written verbatim.
+// `body` is the exact file content. Keep answer lines that BOTH approaches must
+// find on a SINGLE physical line (baseline greps raw lines).
+// ---------------------------------------------------------------------------
+const TEXT_FILES = [
+ // ---- HR -----------------------------------------------------------------
+ {
+ path: "hr/code-of-conduct.md",
+ body: `# Northwind Robotics — Code of Conduct
+
+We build autonomous fulfillment robots, and we hold ourselves to the same
+standard of reliability we design into our machines. Every employee is expected
+to act with honesty, respect, and good judgement.
+
+## Respect in the workplace
+
+Harassment, discrimination, and retaliation of any kind are prohibited. Treat
+teammates, customers, and partners with courtesy regardless of role or seniority.
+
+## Conflicts of interest
+
+Disclose any outside activity or financial interest that could conflict with your
+responsibilities at Northwind. When in doubt, raise it with People Operations.
+
+## Reporting concerns
+
+Concerns can be raised confidentially through the ethics line. Northwind does not
+tolerate retaliation against anyone who reports a concern in good faith.
+`,
+ },
+ {
+ path: "hr/benefits/health-plan.md",
+ body: `# Health and Wellbeing Benefits
+
+Northwind Robotics offers a comprehensive benefits package to all full-time
+team members, effective on the first day of employment.
+
+## Medical, dental, and vision
+
+The company pays 90% of the medical premium for employees and 70% for
+dependents. Dental and vision coverage are included at no additional cost.
+
+## Retirement
+
+Northwind matches 401(k) contributions dollar-for-dollar up to 5% of salary.
+Matching contributions vest immediately.
+
+## Wellbeing stipend
+
+Every team member receives an annual wellbeing stipend of $600, usable for gym
+memberships, ergonomic equipment, or mental-health services.
+`,
+ },
+ // ---- Engineering --------------------------------------------------------
+ {
+ path: "engineering/onboarding.md",
+ body: `# Engineering Onboarding Guide
+
+Welcome to the Northwind engineering organization. This guide walks incoming
+engineers through their first two weeks.
+
+## Day one — equipment
+
+Every incoming software hire is issued a 16-inch developer workstation on their first day.
+It ships pre-imaged with the standard toolchain, the VPN client, and access to
+the internal package registry. If anything is missing, open a ticket with IT.
+
+## First week
+
+New hires pair with an onboarding buddy, complete the security-awareness module,
+and ship a small documentation fix as their first pull request. By the end of the
+first week you should have local builds running and be able to deploy to staging.
+
+## Getting help
+
+The engineering handbook lives in the internal wiki. For anything urgent, ask in
+the team channel — someone is always around.
+`,
+ },
+ {
+ path: "engineering/architecture/system-overview.md",
+ body: `# Platform Architecture Overview
+
+The Northwind platform coordinates fleets of warehouse robots and the software
+services that plan, dispatch, and track their work. This note summarizes the
+major services and how state flows between them.
+
+## Services
+
+- **Ingest service** — receives fulfillment events from the warehouse floor.
+ The ingest service listens on port 8412 for inbound NDJSON batches.
+- **Planner** — turns orders into pick routes and assigns them to robots.
+- **Inventory service** — the authoritative record of stock levels and locations.
+
+## Persistence
+
+We favor boring, well-understood storage. The inventory service stores its SKU records in a PostgreSQL 15 cluster.
+A read replica serves reporting queries so that analytics never compete with the
+live picking path. Event history is kept in an append-only object-store bucket.
+
+## State flow
+
+Floor events land in the ingest service, are normalized, and are fanned out to the
+planner and the inventory service. The planner never writes stock levels directly;
+it asks the inventory service, which is the single source of truth.
+`,
+ },
+ {
+ path: "engineering/architecture/data-pipeline.md",
+ body: `# Data Pipeline Notes
+
+The analytics pipeline turns raw floor events into the dashboards operations
+teams rely on. It is intentionally simple and replayable.
+
+## Ingestion limits
+
+To protect the live path, inbound traffic is throttled per client.
+The default API rate limit is 1000 requests per minute per client key.
+Clients that exceed the limit receive a 429 and should back off exponentially.
+
+## Stages
+
+1. **Collect** — floor events are appended to the event log.
+2. **Normalize** — events are validated and enriched with warehouse metadata.
+3. **Aggregate** — five-minute rollups feed the operations dashboards.
+
+## Replay
+
+Because every stage reads from the append-only log, the pipeline can be replayed
+from any offset to rebuild derived tables after a schema change.
+`,
+ },
+ {
+ path: "engineering/postmortems/incident-4471.txt",
+ body: `INCIDENT 4471 — Delayed pick routes in Fulfillment Center 3
+Status: Resolved
+
+Summary
+-------
+For roughly 40 minutes, the planner issued pick routes several seconds late,
+slowing throughput in one fulfillment center. No orders were lost.
+
+Root cause
+----------
+A slow reporting query saturated the reporting replica and, through a shared
+connection pool, starved the planner of connections. The planner fell back to
+serial planning, which is correct but slow.
+
+Resolution
+----------
+The offending query was moved to a dedicated read pool. Connection pools for the
+live picking path are now isolated from reporting workloads.
+
+Follow-ups
+----------
+- Add an alert for reporting-replica connection saturation.
+- Document pool isolation in the architecture overview.
+`,
+ },
+ // ---- Product ------------------------------------------------------------
+ {
+ path: "product/faq.html",
+ body: `<!doctype html>
+<html lang="en">
+<head><meta charset="utf-8"><title>Northwind Robotics — Product FAQ</title></head>
+<body>
+<h1>Northwind Robotics — Product FAQ</h1>
+<p>Answers to the questions we hear most often from operations teams evaluating
+the Northwind fulfillment platform.</p>
+
+<dl>
+<dt>What uptime does the platform guarantee?</dt>
+<dd>Our platform SLA guarantees 99.9% uptime measured monthly, excluding scheduled maintenance windows.</dd>
+
+<dt>How quickly can we deploy?</dt>
+<dd>A standard rollout takes two to four weeks, including floor mapping and a
+supervised pilot before full production traffic.</dd>
+
+<dt>What if a customer is unhappy with a unit?</dt>
+<dd>Customers may send back any unit within 30 days of delivery for a full reimbursement, no questions asked.</dd>
+
+<dt>Can the robots operate alongside people?</dt>
+<dd>Yes. The fleet is designed for mixed human-and-robot floors, with safety
+zones and speed limits enforced in software.</dd>
+</dl>
+</body>
+</html>
+`,
+ },
+ {
+ path: "product/release-notes.md",
+ body: `# Release Notes — Fulfillment Platform
+
+## 2044.2 (current)
+
+- Planner now batches short pick routes, improving throughput on dense shelves.
+- Added a live heatmap of floor congestion to the operations dashboard.
+- Reduced cold-start time for newly powered-on robots.
+
+## 2044.1
+
+- Introduced per-client throttling on the ingest path.
+- Reporting queries moved to a dedicated read replica.
+- Fixed a rare mis-sort when two robots contended for the same aisle.
+
+## 2043.4
+
+- Initial support for multi-zone warehouses.
+- Faster firmware updates over the maintenance network.
+`,
+ },
+ {
+ path: "product/roadmap.txt",
+ body: `NORTHWIND ROBOTICS — PRODUCT ROADMAP (INTERNAL)
+
+Now
+---
+- Ship the congestion heatmap to all production customers.
+- Harden the planner against connection-pool starvation (see incident 4471).
+
+Next (Q3)
+---------
+- Q3 target: reach 99.5% pick accuracy across all warehouse fulfillment centers.
+- Pilot autonomous charging hand-off between robots.
+- Cut planner tail latency by half on dense shelves.
+
+Later
+-----
+- Multi-site fleet balancing.
+- Predictive maintenance from vibration telemetry.
+`,
+ },
+ // ---- Security -----------------------------------------------------------
+ {
+ path: "security/access-control.html",
+ body: `<!doctype html>
+<html lang="en">
+<head><meta charset="utf-8"><title>Access Control Guide</title></head>
+<body>
+<h1>Access Control Guide</h1>
+<p>How access to Northwind systems is granted, reviewed, and recovered.</p>
+
+<h2>Single sign-on</h2>
+<p>All internal tools sit behind single sign-on. Access is granted by role, and
+group membership is reviewed every quarter.</p>
+
+<h2>Locked out?</h2>
+<p>To recover access to a locked account, use the credential self-service portal and follow the emailed recovery link.</p>
+<p>If the self-service flow does not resolve it, contact the IT service desk and
+they will verify your identity before restoring access.</p>
+
+<h2>Privileged access</h2>
+<p>Administrative access to production is time-boxed and requires a second
+approver. Every elevation is logged and expires automatically.</p>
+</body>
+</html>
+`,
+ },
+ // ---- Finance ------------------------------------------------------------
+ {
+ path: "finance/procurement.html",
+ body: `<!doctype html>
+<html lang="en">
+<head><meta charset="utf-8"><title>Procurement Guide</title></head>
+<body>
+<h1>Procurement Guide</h1>
+<p>How to buy things at Northwind without slowing your team down.</p>
+
+<h2>Approval thresholds</h2>
+<p>Purchases under $500 can be made by any manager. Purchases over $5,000 must be approved by the Director of Operations before a purchase order is issued.</p>
+
+<h2>Preferred vendors</h2>
+<p>Use a preferred vendor where one exists; the terms are already negotiated.
+New vendors must complete a security and financial review first.</p>
+
+<h2>Receiving</h2>
+<p>Match every delivery against its purchase order and flag discrepancies to
+finance within five business days.</p>
+</body>
+</html>
+`,
+ },
+ // ---- Operations ---------------------------------------------------------
+ {
+ path: "operations/facilities.md",
+ body: `# Facilities and Ways of Working
+
+Northwind's people are split across two hubs and a growing set of customer
+fulfillment sites. This note covers how and where we work.
+
+## Where we work
+
+Northwind runs a distributed-first culture, and staff may work from home up to three days a week.
+Teams come into the office on the remaining days for planning, workshops, and
+hands-on time with hardware. Field engineers are on site at customer warehouses
+as their assignments require.
+
+## Office access
+
+Badges are issued on your first day. Visitors must be signed in at the front desk
+and escorted while on the floor near active robots.
+
+## Facilities requests
+
+Anything from a broken chair to a lab-space booking goes through the facilities
+queue, which is triaged every morning.
+`,
+ },
+ {
+ path: "operations/vendor-directory.txt",
+ body: `NORTHWIND ROBOTICS — APPROVED VENDOR DIRECTORY (INTERNAL)
+
+Category Vendor Notes
+------------------ --------------------- ----------------------------------
+Cloud hosting Meridian Cloud Primary region us-central
+Motors and drives Kesten Drive Systems Long-lead items, order early
+Batteries Voltline Cells Preferred for all new robots
+Logistics Harbor Freightways Pallet and crate shipping
+Office supplies Deskworks Standard catalog only
+
+Notes
+-----
+- Always confirm current pricing in the finance portal before raising an order.
+- Report quality issues to procurement so the scorecard stays accurate.
+`,
+ },
+ // ---- Meetings -----------------------------------------------------------
+ {
+ path: "meetings/2044-q1-planning.txt",
+ body: `Q1 2044 PLANNING — NOTES
+Attendees: Engineering, Product, Operations leads
+
+Themes
+------
+- Reliability first: no throughput feature ships without a rollback plan.
+- Close out the connection-pool isolation work from incident 4471.
+- Prepare fulfillment centers for the Q3 pick-accuracy push.
+
+Decisions
+---------
+- Freeze non-critical schema changes during peak season.
+- Stand up a dedicated reporting replica in every region.
+- Operations to publish a weekly congestion report.
+
+Open questions
+--------------
+- Do we need a second on-call rotation for the analytics pipeline?
+- What is the staffing plan for the new fulfillment site?
+`,
+ },
+ {
+ path: "meetings/decision-log.md",
+ body: `# Engineering Decision Log
+
+A running log of decisions with enough context to understand them later.
+
+## Cadence
+
+The weekly engineering sync is held every Wednesday at 10:00 in the main conference room.
+Decisions made outside the sync are recorded here within one business day.
+
+## Recent decisions
+
+- **D-118** Isolate live-path connection pools from reporting. Rationale:
+ incident 4471. Owner: platform team.
+- **D-121** Adopt an append-only event log as the pipeline source of truth so
+ derived tables can be rebuilt by replay.
+- **D-124** Standardize on a single-sign-on group per service to make quarterly
+ access reviews tractable.
+`,
+ },
+ // ---- Legal --------------------------------------------------------------
+ {
+ path: "legal/ip-policy.html",
+ body: `<!doctype html>
+<html lang="en">
+<head><meta charset="utf-8"><title>Intellectual Property Policy</title></head>
+<body>
+<h1>Intellectual Property Policy</h1>
+<p>How Northwind treats the ideas and work created by its people.</p>
+
+<h2>Ownership</h2>
+<p>Work created in the course of employment, including designs, firmware, and
+documentation, is owned by Northwind Robotics.</p>
+
+<h2>Open source</h2>
+<p>Contributing to open source is encouraged. Get sign-off from engineering
+leadership before publishing anything derived from internal code.</p>
+
+<h2>Prior inventions</h2>
+<p>List any inventions you made before joining on your onboarding form so they are
+clearly excluded from company ownership.</p>
+</body>
+</html>
+`,
+ },
+ // ---- Support ------------------------------------------------------------
+ {
+ path: "support/troubleshooting.md",
+ body: `# Field Troubleshooting Guide
+
+Quick checks for the most common issues field engineers see on the warehouse
+floor. Escalate to platform on-call if these do not resolve the problem.
+
+## Robot will not start a route
+
+1. Confirm the robot has network on the maintenance VLAN.
+2. Check that the planner shows the robot as available.
+3. Power-cycle the robot if firmware is mid-update.
+
+## Slow picking in one aisle
+
+Usually aisle congestion. Check the congestion heatmap; if a single robot is
+stuck, clear the obstruction and it will re-plan automatically.
+
+## Dashboards look stale
+
+The analytics pipeline is replayable; a stale dashboard is almost always the
+rollup stage catching up, not lost data.
+`,
+ },
+ {
+ path: "support/known-issues.txt",
+ body: `NORTHWIND ROBOTICS — KNOWN ISSUES (INTERNAL)
+
+- KI-207: Congestion heatmap can lag by up to a minute on very large floors.
+ Workaround: refresh the dashboard. Fix planned for 2044.3.
+
+- KI-212: Firmware update over a weak maintenance signal may retry several times.
+ Workaround: move the robot closer to an access point before updating.
+
+- KI-219: Reporting queries during peak can still queue briefly even with the
+ dedicated replica. Being tracked under the pool-isolation work.
+`,
+ },
+];
+
+// ---------------------------------------------------------------------------
+// Binary-format corpus files (.pdf .docx). Authored as flat-ODF blocks and
+// converted with soffice. Answer sentences are kept SHORT (one printed line)
+// so pdftotext -layout / soffice->txt round-trip them without line-wrap breaks.
+// `blocks`: array of { h1 | h2 | p : string }.
+// ---------------------------------------------------------------------------
+const BINARIES = [
+ {
+ path: "hr/handbook.pdf",
+ format: "pdf",
+ blocks: [
+ { h1: "Northwind Robotics — Employee Handbook" },
+ { p: "This handbook summarizes the policies that apply to all Northwind Robotics employees. It complements, and does not replace, your individual offer letter." },
+ { h2: "Paid time off" },
+ { p: "Full-time staff accrue 22 days of paid time off each year." },
+ { p: "Paid time off begins accruing on your first day and may be carried over up to ten days into the following calendar year. Requests are approved by your manager through the people portal." },
+ { h2: "Parental leave" },
+ { p: "Paid parental leave is 16 weeks for eligible full-time staff." },
+ { p: "Parental leave is available to all full-time employees after one year of continuous service and may be taken following the birth or adoption of a child." },
+ { h2: "Working hours" },
+ { p: "Standard working hours are 40 per week. Northwind supports flexible scheduling arrangements subject to manager approval and team coverage." },
+ { h2: "Standards of conduct" },
+ { p: "Employees are expected to act with integrity, protect confidential information, and treat colleagues and customers with respect at all times." },
+ ],
+ },
+ {
+ path: "hr/benefits/leave-policy.docx",
+ format: "docx",
+ blocks: [
+ { h1: "Leave and Absence Policy" },
+ { p: "This policy describes the categories of leave available to Northwind Robotics employees beyond standard paid time off." },
+ { h2: "Bereavement leave" },
+ { p: "Bereavement leave grants up to 5 working days per event." },
+ { p: "Bereavement leave applies to the loss of an immediate family member. Additional unpaid time can be arranged with People Operations on a case-by-case basis." },
+ { h2: "Sick leave" },
+ { p: "The company provides ten days of paid sick leave each year. Unused sick leave does not carry over but is refreshed at the start of each calendar year." },
+ { h2: "Jury duty" },
+ { p: "Employees called for jury service are granted paid time for the duration of their civic obligation, with no impact on their paid-time-off balance." },
+ ],
+ },
+ {
+ path: "engineering/oncall-runbook.docx",
+ format: "docx",
+ blocks: [
+ { h1: "Platform On-Call Runbook" },
+ { p: "This runbook is the first thing an on-call engineer should open when paged. It defines severity levels, response expectations, and common remediations." },
+ { h2: "Severity levels" },
+ { p: "Sev-1 is a full customer-facing outage. Sev-2 is degraded service. Sev-3 is a minor issue with no customer impact." },
+ { p: "On-call must acknowledge a Sev-1 within 5 minutes of the page." },
+ { h2: "Escalation" },
+ { p: "Unacknowledged pages escalate to secondary on-call after 15 minutes." },
+ { p: "If the secondary also does not acknowledge, the alert escalates to the engineering manager. Keep the incident channel updated at every step." },
+ { h2: "Common remediations" },
+ { p: "For a saturated reporting replica, shed reporting load and confirm the live picking path has a dedicated connection pool. For ingest backpressure, scale the ingest workers and verify clients are honoring throttling." },
+ ],
+ },
+ {
+ path: "security/data-classification.pdf",
+ format: "pdf",
+ blocks: [
+ { h1: "Data Classification and Retention Standard" },
+ { p: "This standard defines how Northwind Robotics classifies information and how long each category must be kept." },
+ { h2: "Classification tiers" },
+ { p: "Data is classified as Public, Internal, Confidential, or Restricted. The tier determines handling, access, and encryption requirements." },
+ { h2: "Retention" },
+ { p: "Customer transaction records are retained for 7 years." },
+ { p: "Retention satisfies financial-audit and regulatory obligations, after which records are securely destroyed. Retention periods for other categories are listed in the appendix." },
+ { h2: "Encryption" },
+ { p: "All Restricted data must be encrypted at rest and in transit. Encryption keys are rotated on a regular schedule and stored in the managed key service." },
+ ],
+ },
+ {
+ path: "finance/expense-policy.pdf",
+ format: "pdf",
+ blocks: [
+ { h1: "Travel and Expense Policy" },
+ { p: "This policy explains what Northwind Robotics reimburses when you travel for work and how to submit an expense report." },
+ { h2: "Meal per diem" },
+ { p: "Domestic travel meals are reimbursed at $75 per day." },
+ { p: "International per-diem rates vary by destination and are published in the finance portal. Alcohol is not reimbursable under the per-diem allowance." },
+ { h2: "Airfare and lodging" },
+ { p: "Economy class is the default for flights under six hours. Book lodging within the published nightly cap for the destination city." },
+ { h2: "Submitting expenses" },
+ { p: "Submit expense reports within 30 days of travel, with itemized receipts for anything above the minimum threshold. Reimbursement is paid with the next payroll cycle." },
+ ],
+ },
+ // ---- Binary distractors (no query targets them, but XERJ still indexes them) ----
+ {
+ path: "product/datasheet.pdf",
+ format: "pdf",
+ blocks: [
+ { h1: "Northwind R-Series Robot — Datasheet" },
+ { p: "The R-Series is Northwind's flagship warehouse fulfillment robot, designed for mixed human-and-robot floors." },
+ { h2: "Specifications" },
+ { p: "Payload capacity is up to 40 kilograms. Top safe speed on the warehouse floor is limited in software for shared aisles." },
+ { p: "A full charge supports a typical eight-hour shift, with opportunistic top-up charging between routes." },
+ { h2: "Safety" },
+ { p: "The R-Series enforces speed and separation limits near people, with redundant sensors and an independent emergency stop." },
+ ],
+ },
+ {
+ path: "legal/contractor-terms.docx",
+ format: "docx",
+ blocks: [
+ { h1: "Independent Contractor Terms" },
+ { p: "These standard terms govern engagements between Northwind Robotics and independent contractors." },
+ { h2: "Confidentiality" },
+ { p: "Contractors must keep all non-public Northwind information confidential during and after the engagement." },
+ { h2: "Ownership of work" },
+ { p: "All deliverables created under the engagement are assigned to Northwind Robotics upon creation." },
+ { h2: "Term and termination" },
+ { p: "Either party may end the engagement with two weeks written notice. Confidentiality obligations survive termination." },
+ ],
+ },
+];
+
+// ===========================================================================
+// LARGE DOCUMENTS (>=60 KB each) — the context-efficiency demonstrators.
+//
+// The files above are deliberately small, which makes the "load the whole file
+// vs. one ranked passage" context argument INVERT (a passage costs more than a
+// 1 KB file). To demonstrate the context-efficiency win HONESTLY we need
+// realistically large documents. Each of the four below buries ONE unique,
+// plain-greppable answer line deep inside a >=60 KB body, so a FAIR grep
+// baseline CAN find the line but must open the whole large file to use it,
+// whereas XERJ returns a single ranked ~800-char passage.
+//
+// - hr/employee-handbook.html (HTML) — referral-bonus figure
+// - engineering/api-reference.md (MD) — a config default deep in a
+// big API/config reference
+// - engineering/runbooks/major-incident-runbook.txt (TXT) — a comms deadline
+// - security/information-security-policy.pdf (PDF) — a remediation SLA. Being
+// a PDF it is ALSO invisible to grep, so this one demonstrates format
+// coverage AND scale at once.
+//
+// All four share ONE content model — an array of blocks ({h1|h2|h3|p|li|code}) —
+// rendered to HTML / MD / TXT, or to a PDF via the existing FODT->soffice path.
+// Prose is coherent Northwind-Robotics content built from compact real specs and
+// realistic policy/reference scaffolding — verbose like the real thing, not
+// lorem ipsum, no personal names.
+// ===========================================================================
+const MIN_LARGE_BYTES = 60 * 1024; // 61,440 — the ">=60 KB" floor (SPEC + self-verify).
+
+// ---- block -> format renderers -------------------------------------------
+function htmlEscape(s) {
+ return String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
+}
+function renderHtml(title, blocks) {
+ const out = ["<!doctype html>", '<html lang="en">',
+ `<head><meta charset="utf-8"><title>${htmlEscape(title)}</title></head>`, "<body>"];
+ let inList = false;
+ const closeList = () => { if (inList) { out.push("</ul>"); inList = false; } };
+ for (const b of blocks) {
+ if (b.li == null) closeList();
+ if (b.h1 != null) out.push(`<h1>${htmlEscape(b.h1)}</h1>`);
+ else if (b.h2 != null) out.push(`<h2>${htmlEscape(b.h2)}</h2>`);
+ else if (b.h3 != null) out.push(`<h3>${htmlEscape(b.h3)}</h3>`);
+ else if (b.p != null) out.push(`<p>${htmlEscape(b.p)}</p>`);
+ else if (b.code != null) out.push(`<pre><code>${htmlEscape(b.code)}</code></pre>`);
+ else if (b.li != null) { if (!inList) { out.push("<ul>"); inList = true; } out.push(`<li>${htmlEscape(b.li)}</li>`); }
+ }
+ closeList();
+ out.push("</body>", "</html>");
+ return out.join("\n") + "\n";
+}
+function renderMd(blocks) {
+ const out = [];
+ for (const b of blocks) {
+ if (b.h1 != null) out.push("# " + b.h1, "");
+ else if (b.h2 != null) out.push("## " + b.h2, "");
+ else if (b.h3 != null) out.push("### " + b.h3, "");
+ else if (b.p != null) out.push(b.p, "");
+ else if (b.code != null) out.push("```json", b.code, "```", "");
+ else if (b.li != null) out.push("- " + b.li);
+ }
+ // keep list items tight, but a blank line after a list group
+ return out.join("\n").replace(/\n{3,}/g, "\n\n").trim() + "\n";
+}
+function renderTxt(blocks) {
+ const out = [];
+ for (const b of blocks) {
+ if (b.h1 != null) out.push(b.h1.toUpperCase(), "=".repeat(Math.min(b.h1.length, 72)), "");
+ else if (b.h2 != null) out.push("", b.h2, "-".repeat(Math.min(b.h2.length, 72)), "");
+ else if (b.h3 != null) out.push("", b.h3, "");
+ else if (b.p != null) out.push(b.p, ""); // one paragraph == one physical line (keeps answers greppable)
+ else if (b.code != null) out.push(b.code.split("\n").map((l) => " " + l).join("\n"), "");
+ else if (b.li != null) out.push(" - " + b.li);
+ }
+ return out.join("\n").replace(/\n{3,}/g, "\n\n").trim() + "\n";
+}
+
+// ---- 1) Employee handbook (HTML) -----------------------------------------
+// Compact specs: [h2 title, purpose sentence, [key points]]. A scaffold expands
+// each into a realistic multi-paragraph handbook section. One deep section
+// carries the unique referral-bonus figure.
+const HANDBOOK_POLICIES = [
+ ["Purpose of this handbook", "This handbook summarizes the policies, benefits, and expectations that apply to everyone who works at Northwind Robotics.", ["It complements, and does not replace, your individual offer letter or any policy published on the internal wiki.", "Where a specific policy document conflicts with this summary, the specific policy governs.", "People Operations maintains this handbook and updates it as policies change."]],
+ ["Our values", "Northwind builds autonomous fulfillment robots, and we hold ourselves to the reliability we design into our machines.", ["We value honesty, sound judgement, and respect for the people we work with and for.", "We prefer boring, well-understood solutions and clear communication over cleverness.", "We take ownership of our commitments and raise problems early."]],
+ ["Equal opportunity", "Northwind is an equal-opportunity employer and does not tolerate discrimination or harassment.", ["Employment decisions are based on merit, qualifications, and business need.", "Harassment, discrimination, and retaliation of any kind are prohibited and will be investigated.", "Reasonable accommodations are available through People Operations on request."]],
+ ["Employment basics", "Most roles at Northwind are full-time and at-will, subject to the terms of your offer letter.", ["Your first ninety days are an onboarding and mutual-fit period with regular check-ins.", "Changes to your role, level, or compensation are confirmed in writing.", "Employment records are kept confidential and handled under our data-protection standards."]],
+ ["Working hours and flexibility", "Northwind supports flexible scheduling so long as team coverage and commitments are met.", ["Core collaboration hours are agreed within each team so meetings land when people are available.", "Flexible arrangements are approved by your manager and revisited if business needs change.", "Non-exempt staff must record hours accurately, including any overtime."]],
+ ["Remote and hybrid work", "Northwind runs a distributed-first culture, and most teams work in a hybrid pattern.", ["Where a role allows it, staff may work remotely for part of the week by agreement with their manager.", "Teams come together on shared days for planning, workshops, and hands-on time with hardware.", "Field engineers work on site at customer warehouses as their assignments require."]],
+ ["Attendance and time recording", "We rely on each other to be available and predictable during agreed hours.", ["Notify your manager and team as early as possible if you will be unexpectedly unavailable.", "Recurring absence is handled supportively through People Operations, not punitively.", "Accurate time and leave records keep payroll and coverage correct."]],
+ ["Compensation and pay", "Compensation reflects the role, level, market data, and individual performance.", ["Salaries are paid on a regular cycle published in the payroll portal.", "Pay is reviewed annually and may be adjusted following a promotion or a significant change in scope.", "Questions about your pay are confidential and go to People Operations."]],
+ ["Performance and growth", "We want everyone to know how they are doing and what growth looks like.", ["Managers hold regular one-to-ones and a lightweight periodic performance conversation.", "Growth is measured against a published set of role expectations for each level.", "Feedback is expected to be specific, timely, and kind."]],
+ ["Promotions", "Promotions recognize sustained performance at the next level, not tenure alone.", ["A promotion case is written up by your manager and reviewed by a calibration panel.", "Calibration exists to keep the bar consistent across teams.", "If a case is not yet ready, you will get concrete, actionable feedback."]],
+ ["Learning and development", "Northwind invests in the growth of its people through time and budget for learning.", ["Each employee has an annual professional-development budget for courses, books, and conferences.", "Time for approved learning is treated as work time, not personal time.", "Sharing what you learn back with your team is strongly encouraged."]],
+ ["Referral program", "Great people know great people, and referrals are one of our best sources of hires.", ["Anyone may refer a candidate through the referral portal, subject to the usual fair-hiring rules.", "Referrers are kept informed of their candidate's progress at a high level.", "The program excludes hiring-manager and People-Operations referrals for roles they own."]],
+ ["Code of conduct", "Every employee is expected to act with integrity and to treat others with respect.", ["Protect confidential and customer information at all times.", "Disclose conflicts of interest to People Operations when in doubt.", "Concerns can be raised confidentially through the ethics line without fear of retaliation."]],
+ ["Health, safety, and the floor", "Safety around robots and hardware is a shared, non-negotiable responsibility.", ["Follow posted safety procedures in labs and on customer warehouse floors at all times.", "Visitors must be signed in and escorted while near active robots.", "Report near-misses and hazards immediately so we can fix them before harm occurs."]],
+ ["Wellbeing", "We want Northwind to be a sustainable place to do your best work.", ["An annual wellbeing stipend can be used for fitness, ergonomics, or mental-health services.", "Confidential support is available through the employee assistance program.", "Managers are expected to model healthy working patterns, including real time off."]],
+ ["Benefits overview", "Full-time team members are eligible for a comprehensive benefits package from day one.", ["Details of medical, dental, vision, and retirement benefits are in the benefits guide.", "Open enrollment happens once a year, with mid-year changes allowed for qualifying life events.", "The People Operations team can walk you through your options."]],
+ ["Time off and leave", "Northwind offers paid time off plus several categories of protected leave.", ["Paid time off, holidays, and sick leave are described in the leave and absence policy.", "Family and medical leave are available to eligible employees under that policy.", "Plan longer absences with your manager so the team can arrange coverage."]],
+ ["Equipment and IT", "Northwind issues the equipment you need and expects you to look after it.", ["Standard-issue hardware ships pre-imaged with the approved toolchain and security agent.", "Report lost or stolen devices to IT immediately so access can be revoked.", "Company equipment is for company work; incidental personal use is fine within the acceptable-use policy."]],
+ ["Information security for everyone", "Security is part of every role, not just the security team's job.", ["Use single sign-on and multi-factor authentication on every internal system.", "Never share credentials, and store secrets only in the approved secret manager.", "Report anything suspicious, including phishing, to the security team right away."]],
+ ["Travel and expenses", "When you travel for work, Northwind reimburses reasonable, documented costs.", ["Book within the published caps and use preferred vendors where they exist.", "Submit expenses promptly with itemized receipts through the finance portal.", "The travel and expense policy has the full rules, including per-diem details."]],
+ ["Communications and social media", "Speak about Northwind with the same honesty we expect internally.", ["Do not share confidential or customer information in public forums.", "Make clear when you are speaking personally rather than for the company.", "Route press and analyst inquiries to the communications team."]],
+ ["Grievances and disputes", "Everyone deserves a fair, confidential way to raise a workplace concern.", ["Raise concerns with your manager, People Operations, or the ethics line, whichever feels right.", "Concerns are handled promptly, confidentially, and without retaliation.", "Serious matters are escalated to an impartial reviewer."]],
+ ["Leaving Northwind", "When someone moves on, we want the transition to be clean and respectful.", ["Give the notice stated in your offer letter and help hand over your work.", "Return company equipment and confirm your access has been revoked.", "Confidentiality obligations continue after your employment ends."]],
+ ["Diversity and inclusion", "We want Northwind to be a place where people of every background can do their best work.", ["Hiring and promotion decisions are made on merit and are reviewed for fairness.", "Employee community groups are supported and open to all.", "Inclusive language and behaviour are expected of everyone, in every setting."]],
+ ["Anti-harassment", "Harassment has no place at Northwind and is taken seriously whenever it is reported.", ["Harassment of any kind, in person or online, is prohibited.", "Reports are investigated promptly and confidentially.", "Retaliation against anyone who reports in good faith is itself a serious violation."]],
+ ["Data protection", "We handle personal data lawfully, carefully, and only for legitimate purposes.", ["Access personal data only where you need it for your work.", "Store and share personal data using approved, secured tools.", "Report any suspected data loss to the security team immediately."]],
+ ["Acceptable use of systems", "Company systems and equipment are provided for company work and must be used responsibly.", ["Keep incidental personal use reasonable and lawful.", "Do not install unapproved software on company equipment.", "Do not use company systems to store or share unlawful or inappropriate content."]],
+ ["Expenses and reimbursement", "Northwind reimburses reasonable, documented business expenses promptly.", ["Spend company money as if it were your own.", "Keep itemized receipts and submit them through the finance portal.", "When a rule is unclear, ask finance before you commit the spend."]],
+ ["Working with contractors", "Contractors are valued members of the team and are engaged under clear terms.", ["Contractor access is scoped to what the engagement needs.", "Confidentiality obligations apply to contractors as they do to employees.", "Deliverables created under an engagement belong to Northwind."]],
+ ["Intellectual property", "Ideas and work created at Northwind are protected and, generally, owned by the company.", ["Work created in the course of employment is owned by Northwind.", "Get sign-off before publishing anything derived from internal work.", "List prior inventions on your onboarding form so they are clearly excluded."]],
+ ["Health and safety responsibilities", "Everyone shares responsibility for a safe workplace, on hardware floors especially.", ["Follow posted safety procedures and never bypass a safety control.", "Report hazards and near-misses so they can be fixed.", "Complete required safety training before working near active robots."]],
+ ["Environmental responsibility", "We try to run Northwind in a way that is considerate of its environmental impact.", ["Reduce waste and recycle where facilities allow.", "Prefer efficient options for travel and shipping.", "Suggest improvements through the facilities or sustainability channels."]],
+ ["Onboarding and first days", "Your first days are designed to help you settle in and become productive without pressure.", ["You are paired with an onboarding buddy and given a clear first task.", "Equipment and access are arranged before you start where possible.", "Regular check-ins during onboarding make sure you have what you need."]],
+ ["Feedback culture", "We give and receive feedback often, so nobody is surprised at review time.", ["Feedback is specific, timely, and kind.", "It flows in every direction, not just from managers.", "Acting on feedback matters more than merely collecting it."]],
+ ["Meeting hygiene", "Meetings are a tool, not a default; we try to use them well and respect people's time.", ["Every meeting has a purpose and, where useful, an agenda.", "Decisions and actions are recorded so absentees stay informed.", "Prefer asynchronous updates when a meeting is not needed."]],
+ ["Acknowledgement", "By working at Northwind you agree to follow the policies summarized here.", ["Ask People Operations if anything in this handbook is unclear.", "Policies are reviewed regularly and you will be told when material changes are made.", "The current version of every policy always lives on the internal wiki."]],
+];
+const HANDBOOK_GLOSSARY = [
+ ["Accrual", "the gradual build-up of a leave balance over the year as it is earned."],
+ ["At-will employment", "an arrangement where either party may end the employment relationship, subject to your offer terms and applicable law."],
+ ["Calibration", "a cross-team review that keeps performance and promotion decisions consistent."],
+ ["Confidential information", "any non-public information about Northwind, its people, its customers, or its technology."],
+ ["Conflict of interest", "a situation where a personal interest could improperly influence your work decisions."],
+ ["Core hours", "the agreed window when a team expects members to be available for collaboration."],
+ ["Dependent", "a family member eligible for coverage under Northwind's benefit plans."],
+ ["Employee assistance program", "a confidential service offering counselling and practical support."],
+ ["Ethics line", "a confidential channel for raising concerns about conduct or compliance."],
+ ["Exempt employee", "a salaried role not eligible for overtime under applicable law."],
+ ["Full-time", "a role scheduled for the standard full working week and eligible for the full benefits package."],
+ ["Hybrid work", "a pattern that mixes on-site and remote days by agreement with your manager."],
+ ["Immediate family", "the close relatives defined in the leave and absence policy for bereavement and family leave."],
+ ["Level", "a role's position on the career framework, with published expectations."],
+ ["Non-exempt employee", "an hourly role eligible for overtime, required to record hours accurately."],
+ ["Onboarding period", "the initial mutual-fit period after joining, with extra check-ins and support."],
+ ["Open enrollment", "the annual window to choose or change your benefit elections."],
+ ["People Operations", "the team that owns hiring, benefits, policy, and employee support."],
+ ["Per diem", "a fixed daily allowance for meals and incidentals while travelling for work."],
+ ["Preferred vendor", "a supplier with pre-negotiated terms that should be used where one exists."],
+ ["Probationary check-in", "a structured conversation during the onboarding period about fit and progress."],
+ ["Qualifying life event", "a change such as marriage or a new child that allows mid-year benefit changes."],
+ ["Reasonable accommodation", "an adjustment that enables an employee to perform their role."],
+ ["Retaliation", "any adverse action taken against someone for raising a concern in good faith; strictly prohibited."],
+ ["Single sign-on", "one authenticated identity used to access internal tools, protected by multi-factor authentication."],
+ ["Stipend", "a fixed allowance provided for a specific purpose, such as wellbeing."],
+ ["Vesting", "the point at which a benefit, such as an employer retirement match, becomes fully yours."],
+ ["Wellbeing stipend", "an annual allowance for fitness, ergonomics, or mental-health services."],
+ ["Acceptable use", "the rules governing appropriate use of company systems and equipment."],
+ ["Time in lieu", "time off granted to balance approved additional hours worked."],
+ ["Benefits portal", "the online system where you review and enrol in your benefits and find in-network providers."],
+ ["People portal", "the system used to request time off, view leave balances, and update personal details."],
+ ["Finance portal", "the system used to submit expenses, claim stipends, and manage travel bookings."],
+ ["Career framework", "the published set of role expectations for each level that guides growth and promotion."],
+ ["One-to-one", "a regular private conversation between an employee and their manager."],
+ ["Hub", "one of Northwind's main offices, as distinct from a customer fulfillment site."],
+ ["Field engineer", "an engineer who works on site at customer warehouses as assignments require."],
+ ["Offer letter", "the document setting out your individual terms of employment, which governs where it is more specific than this handbook."],
+ ["Notice period", "the amount of advance notice you or the company must give to end employment, stated in your offer letter."],
+ ["Open source contribution", "work published to a public project, which requires leadership sign-off when derived from internal code."],
+ ["Prior invention", "an invention made before joining Northwind, which you list on onboarding so it is excluded from company ownership."],
+ ["Escort policy", "the requirement that visitors are signed in and accompanied while near active robots."],
+ ["Ethics line", "a confidential channel for raising conduct or compliance concerns without fear of retaliation."],
+ ["Income protection", "insurance that replaces part of your income during extended illness or disability."],
+];
+const HANDBOOK_FAQ = [
+ ["When do benefits start?", "Eligible full-time team members are covered from their first day of employment."],
+ ["How do I request time off?", "Submit the request to your manager through the people portal; approval keeps the team's coverage clear."],
+ ["Can I carry time off into next year?", "A limited carry-over is allowed; the exact amount is set out in the leave and absence policy."],
+ ["How often is pay reviewed?", "Pay is reviewed once a year and may also change after a promotion or a significant change in scope."],
+ ["What is the professional-development budget for?", "Approved courses, books, certifications, and conferences that help you grow in your role."],
+ ["Who do I talk to about a benefits question?", "People Operations can walk you through medical, dental, vision, and retirement options."],
+ ["What should I do if I lose my laptop?", "Report it to IT immediately so device access can be revoked and a replacement issued."],
+ ["How do I raise a concern about conduct?", "Speak to your manager or People Operations, or use the confidential ethics line."],
+ ["Is retaliation really prohibited?", "Yes. Retaliation against anyone who raises a concern in good faith is strictly prohibited."],
+ ["Can I work fully remotely?", "It depends on the role; where it is possible, arrange it with your manager and keep team coverage in mind."],
+ ["Do I record my hours?", "Non-exempt employees must record hours accurately, including any overtime; exempt roles do not."],
+ ["What happens during my first ninety days?", "A structured onboarding and mutual-fit period with regular check-ins from your manager."],
+ ["How do promotions work?", "Your manager writes a case that a calibration panel reviews to keep the bar consistent across teams."],
+ ["Can I contribute to open source?", "Yes, with sign-off from engineering leadership before publishing anything derived from internal code."],
+ ["What is the wellbeing stipend?", "An annual allowance you can spend on fitness, ergonomic equipment, or mental-health services."],
+ ["How do I book work travel?", "Book within the published caps, prefer negotiated vendors, and keep itemized receipts for expenses."],
+ ["When are expenses reimbursed?", "Submit promptly through the finance portal; approved expenses are paid with the next payroll cycle."],
+ ["Who approves flexible schedules?", "Your manager, provided team coverage and commitments are maintained."],
+ ["What if I have a qualifying life event?", "You can change your benefit elections mid-year; contact People Operations to make the change."],
+ ["How do I refer someone?", "Submit them through the referral portal; you will be kept informed of their progress at a high level."],
+ ["Are references to policies binding?", "The full policy documents on the wiki govern; this handbook is a plain-language summary."],
+ ["What should I do about a phishing email?", "Report it to the security team right away and do not click links or enter credentials."],
+ ["Can I use my work laptop for personal things?", "Incidental personal use is fine within the acceptable-use policy; keep it reasonable and lawful."],
+ ["Who owns work I create at Northwind?", "Work created in the course of employment is owned by Northwind, as set out in the IP policy."],
+ ["What notice do I give if I resign?", "Give the notice stated in your offer letter and help hand over your responsibilities cleanly."],
+ ["Where is the latest version of a policy?", "Always on the internal wiki; if the handbook and a policy differ, the policy wins."],
+ ["Do visitors need to be escorted?", "Yes. Visitors are signed in at the front desk and escorted while near active robots."],
+ ["How is my personal data handled?", "Under Northwind's data-protection standards, confidentially and only for legitimate purposes."],
+ ["Can I get an ergonomic chair?", "Facilities requests, including ergonomic equipment, go through the facilities queue or the wellbeing stipend."],
+ ["What if I disagree with a performance rating?", "Discuss it with your manager first; unresolved concerns can go to People Operations for an impartial review."],
+ ["How much notice do I need to book time off?", "There is no fixed minimum, but book as early as you reasonably can so your team can arrange coverage around your absence."],
+ ["Can I take time off during my first ninety days?", "Yes, subject to the same approval as any other time; just plan it with your manager as part of settling in."],
+ ["What happens to unused sick leave at year end?", "Sick leave does not carry over; it is refreshed at the start of each calendar year rather than accumulated."],
+ ["Do I get paid for public holidays I work?", "If your role requires cover on a public holiday, time in lieu is arranged so you are not out of pocket."],
+ ["How do I change my benefit elections?", "During open enrollment, or mid-year if you have a qualifying life event; make the change through the benefits portal or People Operations."],
+ ["Who counts as a dependent for benefits?", "Eligibility is defined in the plan documents; People Operations can confirm whether a specific family member qualifies."],
+ ["Can I expense a home-office purchase?", "Use the home-office allowance for setup items and the wellbeing stipend for ergonomics; check the eligible-purchase list first."],
+ ["What is the difference between a hub and a fulfillment site?", "A hub is a Northwind office; a fulfillment site is a customer warehouse where field engineers work on assignment."],
+ ["How do I report a safety hazard?", "Report it immediately through the facilities queue or to a manager; near-misses should be reported too, not just injuries."],
+ ["Can I bring a visitor into the office?", "Yes, but sign them in at the front desk and escort them at all times while near active robots."],
+ ["What should I do before I travel for work?", "Book within the caps, use preferred vendors, and record your itinerary in the travel portal so travel insurance applies."],
+ ["How do I claim the wellbeing stipend?", "Buy an eligible item or service and submit the claim through the finance portal; the eligible list is deliberately broad."],
+ ["Is my use of the assistance program private?", "Yes; the employee assistance program is confidential and does not go through your manager."],
+ ["Can I change my retirement contribution rate?", "Yes, at any time through the payroll portal; the company match applies up to the published percentage."],
+ ["What if my equipment breaks?", "Open a ticket with IT; if it cannot be fixed quickly a replacement is arranged so you are not blocked."],
+ ["How do I request an internal move?", "Talk to your manager and watch the internal postings; open roles are advertised internally first."],
+ ["What is expected during my notice period?", "Give the notice in your offer letter, hand over your work cleanly, and return company equipment before you leave."],
+ ["Can I keep my accounts after leaving?", "No; access is revoked on your last day, and confidentiality obligations continue afterwards."],
+ ["How are pay reviews decided?", "They consider your role, level, market data, and performance, and are confirmed to you in writing."],
+ ["What if I am asked to do something that feels wrong?", "Raise it with your manager, People Operations, or the ethics line; you will not face retaliation for a good-faith concern."],
+ ["Do contractors follow the same policies?", "Contractors follow the security, confidentiality, and conduct policies; their engagement terms cover the specifics."],
+ ["Can I speak at a conference about my work?", "Often yes, with sign-off from leadership so that nothing confidential is disclosed; the communications team can help."],
+ ["How do I find the current holiday list?", "It is published annually on the internal wiki for each hub's location."],
+ ["What if I need to work adjusted hours for a while?", "Talk to your manager; flexible and adjusted arrangements are supported as long as team coverage is maintained."],
+ ["How do I nominate someone for recognition?", "Use the recognition program on the wiki; nominations are quick and genuinely appreciated."],
+ ["Can I volunteer during work hours?", "Yes; a set amount of paid volunteering time is available each year, arranged with your manager."],
+ ["What ergonomic support can I get at home?", "Request a remote ergonomic assessment through facilities and use the home-office allowance for recommended items."],
+ ["Who owns a side project I build in my own time?", "If it is unrelated to Northwind's business and uses no company resources it is generally yours; check the IP policy and disclose if unsure."],
+ ["How do I escalate a facilities problem?", "Log it in the facilities queue, which is triaged every morning; flag anything urgent to facilities directly."],
+ ["What if I cannot resolve a dispute with a colleague?", "Try a direct conversation first; if that does not work, your manager or People Operations can help mediate."],
+ ["Can I work from another country temporarily?", "Sometimes, but it has tax and legal implications, so clear it with People Operations well in advance rather than assuming."],
+ ["How do I get added to a community group?", "Community groups are employee-led and open; find them on the wiki and just reach out to join."],
+ ["What counts as confidential information?", "Any non-public information about Northwind, its people, its customers, or its technology; when unsure, treat it as confidential."],
+ ["Can I accept a gift from a vendor?", "Modest, occasional gifts may be fine, but disclose anything that could look like it influences a business decision."],
+ ["How do I set up multi-factor authentication?", "IT provisions it during onboarding; enrol a strong second factor and keep a backup method registered."],
+ ["What should I do if I receive a phishing email?", "Report it with one click and do not interact with it; the security team would rather see ten false alarms than miss one real attack."],
+ ["Do I need approval to buy software for work?", "Yes; unapproved software must not be installed on company equipment, and IT can help you find an approved alternative."],
+ ["How often are performance conversations held?", "Lightweight conversations happen regularly through one-to-ones, with a periodic more formal check-in against role expectations."],
+ ["Can I mentor or be mentored?", "Yes to both; mentoring is encouraged and People Operations can help match you with someone."],
+ ["What if my role changes significantly?", "A meaningful change in scope is confirmed in writing and may be reflected in your level and pay at the next review."],
+ ["How do I book a meeting room or lab space?", "Through the facilities queue or the room-booking system; lab space near hardware may need extra safety sign-off."],
+ ["Are lunch-and-learns recorded?", "Where the presenter is comfortable, yes, so distributed and future colleagues can catch up; check before assuming."],
+ ["What is the process for a sabbatical?", "Apply ahead of time; approval depends on tenure, team coverage, and a clean handover of your responsibilities."],
+ ["Can I donate my volunteering time to a team effort?", "Yes; teams sometimes pool volunteering time for a shared cause, arranged with your managers."],
+ ["How do I update my emergency contact?", "Keep it current in the people portal so we can reach the right person if something happens at work."],
+ ["What if I am injured at work?", "Get first aid, report it immediately, and let People Operations know; workplace injuries are taken seriously and followed up."],
+ ["Can I use company equipment for a side project?", "Incidental use is fine within the acceptable-use policy, but company equipment and time should not power an unrelated venture."],
+ ["How are decisions communicated to remote staff?", "Decisions are recorded in writing so that people who were not in the room, including remote and future colleagues, stay informed."],
+ ["What happens if I lose my badge?", "Report it to facilities immediately so it can be deactivated and a replacement issued."],
+ ["Where can I ask a question that is not covered here?", "Ask your manager or People Operations, or search the internal wiki, which holds the authoritative version of every policy."],
+];
+const HANDBOOK_BENEFITS = [
+ ["Medical cover", "Northwind pays the large majority of the medical premium for employees and a substantial share for dependents, and coverage begins on your first day with no waiting period. The plan covers preventive care, specialist visits, hospital treatment, and prescriptions. You can review the plan documents and find in-network providers through the benefits portal, and People Operations can help you choose the option that fits your circumstances."],
+ ["Dental cover", "Routine and major dental care are covered under the standard plan at no additional premium to the employee, and preventive visits are fully covered to encourage regular care. The plan includes checkups, cleanings, fillings, and a share of the cost of major work. Keeping up with routine visits is the cheapest way to avoid larger problems later."],
+ ["Vision cover", "Eye examinations and an allowance toward lenses or frames are included each year, and staff in screen-heavy roles are reminded to use the benefit. The allowance can also be applied to prescription safety eyewear where a role requires it. Book through any in-network provider and claim any balance through the portal."],
+ ["Retirement matching", "The company matches retirement-plan contributions up to a published percentage of salary, and the match vests immediately, so it is yours from day one. You choose your own contribution rate and can change it at any time through the payroll portal. Even a small regular contribution compounds meaningfully over a career, and the match is effectively part of your total compensation."],
+ ["Life and disability", "Basic life cover and income protection are provided at no cost, giving you and your family a financial cushion in the worst cases. You can buy additional cover for yourself or dependents during open enrollment. Keep your beneficiary details current in the portal so cover pays out as you intend."],
+ ["Employee assistance", "A confidential assistance program offers counselling and practical support for you and your household, available around the clock. It covers everything from stress and relationships to legal and financial questions. Using it is private, does not go through your manager, and is there precisely for the hard moments."],
+ ["Wellbeing stipend", "An annual stipend can be spent on fitness, ergonomic equipment, or mental-health services, and is claimed through the finance portal. The list of eligible purchases is deliberately broad because wellbeing looks different for everyone. Spend it on what genuinely helps you stay healthy and do good work."],
+ ["Learning budget", "Each employee has an annual budget for courses, books, certifications, and conferences that support growth in their role. Approved learning happens on work time, not your own, and sharing what you learn with your team is strongly encouraged. Talk to your manager about how to spend it in a way that advances both your goals and the team's."],
+ ["Home-office allowance", "A one-time allowance helps remote and hybrid staff set up a safe, productive workspace at home. It can go toward a desk, chair, monitor, or other ergonomic essentials. If you are not sure what you need, request an ergonomic assessment first."],
+ ["Commuter support", "Where offered locally, pre-tax commuter benefits or a transit allowance help offset the cost of getting to a hub. The exact scheme depends on your location and is described in the local benefits guide. Enrol through the portal before the deadline for the tax advantage to apply."],
+ ["Parental support", "Beyond statutory leave, new parents get a phased return-to-work option and access to family-support resources. A gradual return helps you and your team adjust without a cliff edge. People Operations can walk you through leave, pay, and the practicalities well before your leave begins."],
+ ["Fertility and family building", "The plan includes support toward fertility and family-building services, handled confidentially through the benefits provider. It recognizes that families are formed in many ways. Details of what is covered and how to access it are in the benefits portal, and the process is private."],
+ ["Mental-health days", "A small number of dedicated mental-health days each year are separate from sick leave and need no explanation. They exist so that looking after your mental health does not have to compete with time off for illness. Use them proactively rather than waiting until you are running on empty."],
+ ["Volunteering time", "Employees may take paid time each year to volunteer with a cause they care about, arranged with their manager. Teams sometimes volunteer together as a way to give back and to spend time together off the clock. Log volunteering time like any other planned absence so coverage is clear."],
+ ["Referral bonus", "Employees who refer a candidate who is hired receive a referral bonus, subject to the fair-hiring rules in the referral policy. Referrals are one of our best sources of great colleagues. The full mechanics, including timing and eligibility, are described in the referral-program section above."],
+ ["Recognition awards", "Peer and manager recognition programs celebrate outstanding work with small awards throughout the year. Recognition is most meaningful when it is specific and timely, so do not wait for a formal cycle to thank a colleague. Nominations are quick to submit and genuinely appreciated."],
+ ["Sabbatical option", "Longer-tenured employees may apply for a sabbatical, subject to team coverage and manager approval. A sabbatical is a chance to rest deeply, learn something new, or pursue a personal project. Plan it well ahead so your team can prepare and so you can truly switch off."],
+ ["Equipment refresh", "Standard-issue hardware is refreshed on a regular cycle so your tools stay reliable and safe. If your equipment is slowing you down before its refresh date, raise it with IT rather than struggling on. Returned equipment is securely wiped and responsibly recycled or redeployed."],
+ ["Discounted services", "From time to time Northwind negotiates discounts on services that are useful to staff, and these are listed on the wiki. Availability varies by location and changes over time. Check the current list before assuming a discount applies."],
+ ["Travel insurance", "Business travel is covered by company travel insurance, so keep your itinerary in the travel portal so cover applies. The policy covers medical emergencies, cancellations, and lost belongings while you are travelling for work. Familiarize yourself with the emergency contact details before you depart."],
+];
+const HANDBOOK_LEAVE = [
+ ["Paid time off", "Full-time staff accrue paid time off through the year and request it through the people portal, and a limited balance can be carried into the next year. Book it early where you can so your team can plan around your absence. Time off is there to be used; managers are expected to model taking real breaks."],
+ ["Public holidays", "Northwind observes the public holidays of each hub's location, and the current list is published annually on the wiki. Where a role requires cover on a public holiday, time in lieu is arranged. Distributed teams are mindful that holidays differ across locations when scheduling."],
+ ["Sick leave", "Paid sick leave is available each year for your own illness or medical appointments and is refreshed at the start of the calendar year. Let your manager know as early as you can so the team can cover, and focus on getting better. For longer illness, medical leave and income protection take over."],
+ ["Parental leave", "Eligible employees receive paid parental leave following the birth or adoption of a child, with the specifics set out in the leave and absence policy. A phased return-to-work option helps you ease back in. Start the conversation with People Operations well before your leave so everything is arranged in good time."],
+ ["Family and caregiver leave", "Time to care for an immediate family member is available, and longer arrangements can be made with People Operations. Caring responsibilities are a normal part of life and are treated with understanding. Talk to your manager about the flexibility you need."],
+ ["Bereavement leave", "Paid time is provided following the loss of an immediate family member, with additional unpaid time available on request. Grief does not follow a schedule, so the policy is applied compassionately. People Operations can help with the practicalities when you are ready."],
+ ["Jury and civic duty", "Employees called for jury service or other civic duty are granted paid time for the duration of the obligation, with no impact on their paid-time-off balance. Simply share your summons with your manager and People Operations. Serving is a civic responsibility we support."],
+ ["Medical and recovery leave", "Extended medical leave is available for serious health conditions and is coordinated with income-protection cover. The focus is on your recovery, not on paperwork. People Operations manages the process confidentially and keeps it as simple as possible."],
+ ["Unpaid leave", "Where the paid categories do not fit, unpaid leave can be arranged with your manager and People Operations. It is useful for planned personal projects, extended travel, or circumstances that do not fall under another category. Agree the duration and return date in advance so your role can be covered."],
+ ["Compassionate leave", "Short-notice compassionate leave is available for personal emergencies, so talk to your manager as early as you can. The aim is to remove work worries when something urgent happens in your life. Formalities can wait until the immediate situation is under control."],
+ ["Study leave", "Time for approved study or examinations related to your role may be granted alongside the learning budget. It recognizes that some growth needs focused, uninterrupted time. Agree the plan with your manager so it dovetails with your other commitments."],
+ ["Sabbatical leave", "A longer, planned absence for rest or a personal project is available to longer-tenured staff by application. A sabbatical works best when it is planned well ahead and your responsibilities are handed over cleanly. It is a chance to come back refreshed with a new perspective."],
+];
+const HANDBOOK_PERKS = [
+ ["Flexible hours", "Within agreed core hours, shape your day around when you do your best work and when your team needs you. Some people are sharpest early, others late; flexibility lets both thrive. The only firm expectation is that you are reachable and reliable during the hours your team has agreed."],
+ ["Hybrid working", "Split your week between a hub and home where your role allows, by agreement with your manager. In-person days are for the things that are simply better together, like workshops and hands-on hardware time. Remote days are protected for focused, heads-down work."],
+ ["Quiet and focus spaces", "Hubs provide quiet areas for deep work alongside collaboration spaces for workshops and pairing. Use the space that fits the task rather than defaulting to your desk. Respect the quiet zones as genuinely quiet."],
+ ["On-site refreshments", "Hubs stock tea, coffee, and healthy snacks, and kitchens are shared spaces, so please keep them tidy. Good coffee and a moment away from the screen are small things that add up. Clean up after yourself so the space stays pleasant for everyone."],
+ ["Team socials", "Each team has a modest budget for regular get-togethers, in person or online for distributed teams. Socials work best when they are inclusive and not always centred on one activity. Rotate who plans them so everyone gets a say."],
+ ["Hardware choice", "Within the standard, safe catalog you can choose the workstation configuration that suits your work. The catalog exists so equipment stays supportable and secure, not to limit you unnecessarily. If your work genuinely needs something outside it, make the case to IT."],
+ ["Internal mobility", "Open roles are posted internally first, so talk to your manager if you are curious about a move. Growing by moving across the company is encouraged, not seen as disloyalty. A good manager will help you find your next step even when it is on another team."],
+ ["Mentoring", "Formal and informal mentoring connects you with people who can help you grow. A mentor need not be senior to you in the org chart, only ahead of you in something you want to learn. People Operations can help you find a match."],
+ ["Lunch-and-learns", "Regular informal sessions let people share what they are working on or learning. They are low-pressure and open to everyone, whatever your level. Volunteering to present is a great, gentle way to build confidence and share knowledge."],
+ ["Ergonomic assessments", "Request an ergonomic assessment for your workstation, at a hub or at home, through facilities. A few small adjustments prevent a lot of discomfort over a career. Do not wait until something aches to ask."],
+ ["Pet-friendly days", "Some hubs run pet-friendly days, so check local guidelines and be considerate of colleagues. Not everyone is comfortable around animals, and that is respected. Where they run, they are a cheerful addition to the week."],
+ ["Community groups", "Employee community groups bring together people with shared interests and backgrounds. They are employee-led, open to allies, and supported by the company. They are a good way to meet people beyond your immediate team."],
+];
+function buildHandbookBlocks() {
+ const b = [];
+ const H1 = "Northwind Robotics — Employee Handbook";
+ b.push({ h1: H1 });
+ b.push({ p: "Welcome to Northwind Robotics. This handbook is your plain-language guide to how we work together, what you can expect from us, and what we ask of you. Please read it during onboarding and keep it handy — you will not need to memorize it, but you should know where to look." });
+ b.push({ p: "It is organized as a series of short policy sections, followed by a glossary and a set of frequently asked questions. Every section points to the fuller policy on the internal wiki where more detail is needed." });
+ const WHY_FRAMES = [
+ (t) => `Why it matters. Getting "${t}" right protects both you and your colleagues and keeps Northwind a place people are glad to work.`,
+ (t) => `Why it matters. Clear expectations around "${t}" prevent misunderstandings and let people focus on their work with confidence.`,
+ (t) => `Why it matters. Handling "${t}" consistently is part of treating each other, and our customers, fairly.`,
+ (t) => `Why it matters. When "${t}" is well understood, small issues are resolved early instead of becoming bigger problems.`,
+ ];
+ const WHERE_FRAMES = [
+ "If anything here is unclear for your situation, check with your manager or People Operations before acting; the authoritative version lives on the internal wiki.",
+ "When in doubt, ask first — People Operations would far rather answer a question than untangle a misunderstanding. The full policy is on the wiki.",
+ "Your manager and People Operations are there to help you apply this sensibly. The complete, current policy is always on the internal wiki.",
+ "Common sense and this summary usually agree; when they seem not to, raise it rather than guessing. The full policy is maintained on the wiki.",
+ ];
+ HANDBOOK_POLICIES.forEach(([title, purpose, points], i) => {
+ b.push({ h2: title });
+ b.push({ p: purpose });
+ b.push({ p: "What this means in practice. This section sets the expectation; the points below spell out how it works day to day." });
+ for (const pt of points) b.push({ li: pt.charAt(0).toUpperCase() + pt.slice(1) + "." });
+ b.push({ p: WHY_FRAMES[i % WHY_FRAMES.length](title.toLowerCase()) });
+ b.push({ p: WHERE_FRAMES[i % WHERE_FRAMES.length] });
+ if (title === "Referral program") {
+ // Buried unique answer line (single <p> => single physical line).
+ b.push({ p: "Northwind pays an employee referral bonus of $3,000 per successful hire, paid after the referred candidate completes ninety days of employment." });
+ b.push({ p: "The bonus applies to most non-executive roles, is split evenly if two employees refer the same candidate, and is subject to the fair-hiring rules described above. There is no limit on how many people you may refer in a year." });
+ }
+ });
+ b.push({ h2: "Benefits in detail" });
+ b.push({ p: "The full details of every benefit, including eligibility and how to enrol, are in the benefits guide on the wiki. This is a plain-language overview of what is on offer." });
+ for (const [name, desc] of HANDBOOK_BENEFITS) { b.push({ h3: name }); b.push({ p: desc }); }
+ b.push({ h2: "Leave types at a glance" });
+ b.push({ p: "Northwind offers paid time off plus several categories of protected and supported leave. The leave and absence policy is the authoritative source for eligibility and durations; plan longer absences with your manager so the team can arrange coverage." });
+ for (const [name, desc] of HANDBOOK_LEAVE) { b.push({ h3: name }); b.push({ p: desc }); }
+ b.push({ h2: "Ways of working and perks" });
+ b.push({ p: "Beyond formal benefits, a number of everyday things make Northwind a good place to do your best work." });
+ for (const [name, desc] of HANDBOOK_PERKS) { b.push({ h3: name }); b.push({ p: desc }); }
+ b.push({ h2: "Glossary" });
+ b.push({ p: "A quick reference to terms used throughout this handbook and the underlying policies." });
+ for (const [t, d] of HANDBOOK_GLOSSARY) b.push({ li: `${t}: ${d}` });
+ b.push({ h2: "Frequently asked questions" });
+ b.push({ p: "Answers to the questions People Operations hears most often. If yours is not here, just ask." });
+ for (const [q, a] of HANDBOOK_FAQ) { b.push({ h3: q }); b.push({ p: a }); }
+ return { path: "hr/employee-handbook.html", format: "html", title: H1, blocks: b };
+}
+
+// ---- 2) API & configuration reference (Markdown) -------------------------
+// A big reference is naturally long and enumerated: many endpoints with request
+// parameters, response fields, and JSON examples, plus a configuration section.
+// The unique answer (a config default) is buried deep in the config reference.
+const API_RESOURCES = [
+ { s: "robot", p: "robots", base: "/v1/robots", about: "a physical fulfillment robot enrolled in a fleet", filter: "fleet_id, status, and model",
+ fields: [["id", "string", "stable robot identifier"], ["model", "string", "hardware model, for example R-200"], ["fleet_id", "string", "fleet the robot belongs to"], ["status", "string", "one of idle, picking, charging, maintenance, or offline"], ["battery_pct", "integer", "current charge from 0 to 100"], ["firmware_version", "string", "running firmware build"], ["last_seen", "timestamp", "time of the last heartbeat"]],
+ action: ["recall", "post", "Recall the robot to its charging dock and mark it unavailable for new routes."] },
+ { s: "fleet", p: "fleets", base: "/v1/fleets", about: "a logical group of robots managed together in one warehouse", filter: "warehouse_id and status",
+ fields: [["id", "string", "fleet identifier"], ["name", "string", "human-readable fleet name"], ["warehouse_id", "string", "warehouse the fleet operates in"], ["robot_count", "integer", "number of enrolled robots"], ["status", "string", "active or paused"]],
+ action: ["pause", "post", "Pause dispatch to every robot in the fleet without unenrolling them."] },
+ { s: "warehouse", p: "warehouses", base: "/v1/warehouses", about: "a physical fulfillment site", filter: "region",
+ fields: [["id", "string", "warehouse identifier"], ["name", "string", "site name"], ["region", "string", "geographic region code"], ["zone_count", "integer", "number of mapped zones"], ["active", "boolean", "whether the site is live"]] },
+ { s: "zone", p: "zones", base: "/v1/zones", about: "a mapped area within a warehouse", filter: "warehouse_id",
+ fields: [["id", "string", "zone identifier"], ["warehouse_id", "string", "parent warehouse"], ["name", "string", "zone label"], ["kind", "string", "one of storage, staging, or charging"], ["congestion", "number", "recent congestion score from 0 to 1"]] },
+ { s: "location", p: "locations", base: "/v1/locations", about: "a specific shelf or bin coordinate", filter: "zone_id and occupied",
+ fields: [["id", "string", "location identifier"], ["zone_id", "string", "zone containing the location"], ["aisle", "string", "aisle label"], ["occupied", "boolean", "whether stock is present"], ["sku_id", "string", "SKU currently stored, if any"]] },
+ { s: "sku", p: "skus", base: "/v1/skus", about: "a stock-keeping unit in the catalog", filter: "category",
+ fields: [["id", "string", "SKU identifier"], ["title", "string", "product title"], ["category", "string", "catalog category"], ["weight_g", "integer", "unit weight in grams"], ["hazmat", "boolean", "whether special handling is required"]] },
+ { s: "inventory record", p: "inventory", base: "/v1/inventory", about: "the authoritative stock level for a SKU at a location", filter: "sku_id and location_id",
+ fields: [["id", "string", "record identifier"], ["sku_id", "string", "the SKU"], ["location_id", "string", "the storage location"], ["on_hand", "integer", "units physically present"], ["reserved", "integer", "units reserved for orders"]] },
+ { s: "order", p: "orders", base: "/v1/orders", about: "a customer fulfillment order", filter: "status and warehouse_id",
+ fields: [["id", "string", "order identifier"], ["warehouse_id", "string", "fulfilling warehouse"], ["status", "string", "one of received, planned, picking, packed, or shipped"], ["line_count", "integer", "number of order lines"], ["created_at", "timestamp", "when the order was received"]],
+ action: ["cancel", "post", "Cancel an order that has not yet entered the picking stage."] },
+ { s: "route", p: "routes", base: "/v1/routes", about: "a planned pick route assigned to a robot", filter: "robot_id and status",
+ fields: [["id", "string", "route identifier"], ["robot_id", "string", "assigned robot"], ["order_id", "string", "order being fulfilled"], ["stop_count", "integer", "number of pick stops"], ["status", "string", "one of planned, active, or done"]] },
+ { s: "pick", p: "picks", base: "/v1/picks", about: "a single pick action within a route", filter: "route_id",
+ fields: [["id", "string", "pick identifier"], ["route_id", "string", "parent route"], ["location_id", "string", "location picked from"], ["sku_id", "string", "SKU picked"], ["quantity", "integer", "units picked"]] },
+ { s: "task", p: "tasks", base: "/v1/tasks", about: "an asynchronous background job", filter: "kind and status",
+ fields: [["id", "string", "task identifier"], ["kind", "string", "job type, for example reindex or export"], ["status", "string", "one of queued, running, done, or failed"], ["progress", "integer", "percent complete from 0 to 100"], ["created_at", "timestamp", "when the task was created"]] },
+ { s: "alert", p: "alerts", base: "/v1/alerts", about: "an operational alert raised by the platform", filter: "severity and acknowledged",
+ fields: [["id", "string", "alert identifier"], ["severity", "string", "one of info, warning, or critical"], ["source", "string", "service that raised the alert"], ["acknowledged", "boolean", "whether an operator has acknowledged it"], ["raised_at", "timestamp", "when the alert fired"]],
+ action: ["acknowledge", "post", "Acknowledge the alert so it stops paging and is marked as owned."] },
+ { s: "incident", p: "incidents", base: "/v1/incidents", about: "a tracked operational incident", filter: "severity and status",
+ fields: [["id", "string", "incident identifier"], ["severity", "string", "one of sev-1, sev-2, or sev-3"], ["status", "string", "open, mitigated, or resolved"], ["summary", "string", "short description"], ["opened_at", "timestamp", "when the incident was declared"]] },
+ { s: "firmware build", p: "firmware", base: "/v1/firmware", about: "a robot firmware image available for rollout", filter: "channel",
+ fields: [["id", "string", "build identifier"], ["version", "string", "semantic version string"], ["channel", "string", "one of stable, beta, or canary"], ["size_bytes", "integer", "image size in bytes"], ["signed", "boolean", "whether the image signature is valid"]] },
+ { s: "webhook", p: "webhooks", base: "/v1/webhooks", about: "a subscription that delivers events to your endpoint", filter: "event and active",
+ fields: [["id", "string", "webhook identifier"], ["url", "string", "your HTTPS delivery endpoint"], ["event", "string", "event type to deliver"], ["active", "boolean", "whether delivery is enabled"], ["created_at", "timestamp", "when the subscription was created"]] },
+ { s: "API key", p: "api-keys", base: "/v1/api-keys", about: "a credential used to authenticate API calls", filter: "active",
+ fields: [["id", "string", "key identifier"], ["label", "string", "human-readable label"], ["scopes", "string", "space-separated permission scopes"], ["active", "boolean", "whether the key may be used"], ["created_at", "timestamp", "when the key was issued"]] },
+ { s: "reservation", p: "reservations", base: "/v1/reservations", about: "a temporary hold placed on stock while an order is fulfilled", filter: "sku_id and status",
+ fields: [["id", "string", "reservation identifier"], ["sku_id", "string", "the reserved SKU"], ["order_id", "string", "the order the hold is for"], ["quantity", "integer", "units held"], ["status", "string", "one of held, consumed, or expired"], ["expires_at", "timestamp", "when the hold lapses if unused"]] },
+ { s: "shipment", p: "shipments", base: "/v1/shipments", about: "a packed order handed to a carrier", filter: "carrier and status",
+ fields: [["id", "string", "shipment identifier"], ["order_id", "string", "the fulfilled order"], ["carrier", "string", "the carrier handling delivery"], ["tracking", "string", "carrier tracking reference"], ["status", "string", "one of packed, dispatched, or delivered"]] },
+ { s: "audit event", p: "audit-events", base: "/v1/audit-events", about: "an immutable record of a change made through the API", filter: "actor and action",
+ fields: [["id", "string", "event identifier"], ["actor", "string", "the API key or principal that acted"], ["action", "string", "the operation performed"], ["target", "string", "the resource affected"], ["at", "timestamp", "when the action occurred"]] },
+ { s: "metric series", p: "metrics", base: "/v1/metrics", about: "a named time series of operational measurements", filter: "name and warehouse_id",
+ fields: [["id", "string", "series identifier"], ["name", "string", "metric name, for example picks_per_minute"], ["warehouse_id", "string", "warehouse the series belongs to"], ["unit", "string", "unit of measurement"], ["last_value", "number", "most recent sampled value"]] },
+];
+const API_CONFIG = [
+ ["ingest", [
+ ["ingest.listen_port", "integer", "8412", "TCP port the ingest service binds for inbound NDJSON batches from the warehouse floor."],
+ ["ingest.max_batch_size", "integer", "4096", "__BURY_BATCH__"],
+ ["ingest.worker_threads", "integer", "8", "Number of worker threads that normalize and fan out inbound events."],
+ ["ingest.queue_high_watermark", "integer", "50000", "Queue depth at which the service applies backpressure and returns 503 to new batches."],
+ ["ingest.accept_timeout_ms", "integer", "2000", "How long a client connection may block waiting for the service to accept a batch."],
+ ]],
+ ["planner", [
+ ["planner.max_stops_per_route", "integer", "24", "Upper bound on the number of pick stops the planner will place on one route."],
+ ["planner.replan_interval_ms", "integer", "500", "How often the planner re-evaluates open routes against current floor state."],
+ ["planner.connection_pool_size", "integer", "32", "Size of the dedicated database connection pool reserved for the live picking path."],
+ ["planner.tail_latency_budget_ms", "integer", "150", "Target ceiling for planner tail latency before it sheds optional work."],
+ ]],
+ ["inventory", [
+ ["inventory.db_url", "string", "postgres://inventory", "Connection string for the authoritative inventory PostgreSQL cluster."],
+ ["inventory.read_replica_url", "string", "postgres://inventory-ro", "Connection string for the read replica that serves reporting queries."],
+ ["inventory.reservation_ttl_s", "integer", "900", "How long a stock reservation is held before it expires if the order does not progress."],
+ ["inventory.low_stock_threshold", "integer", "5", "On-hand level at or below which a low-stock alert is raised for a SKU."],
+ ]],
+ ["api", [
+ ["api.listen_port", "integer", "9209", "Port the public REST API binds."],
+ ["api.rate_limit_per_min", "integer", "1000", "Default per-client request ceiling per minute before requests are throttled with 429."],
+ ["api.max_page_size", "integer", "200", "Largest page size a client may request on a list endpoint."],
+ ["api.request_timeout_ms", "integer", "30000", "Server-side timeout applied to a single API request."],
+ ["api.cors_allowed_origins", "string", "*", "Comma-separated list of origins permitted to call the API from a browser."],
+ ]],
+ ["storage", [
+ ["storage.event_bucket", "string", "nwr-events", "Object-store bucket holding the append-only event log."],
+ ["storage.retention_days", "integer", "365", "How long derived export artifacts are kept before lifecycle deletion."],
+ ["storage.multipart_threshold_mb", "integer", "64", "Object size above which uploads switch to multipart."],
+ ]],
+ ["telemetry", [
+ ["telemetry.sample_rate", "number", "0.1", "Fraction of traces sampled and exported to the observability backend."],
+ ["telemetry.metrics_interval_s", "integer", "15", "How often service metrics are flushed to the metrics pipeline."],
+ ["telemetry.log_level", "string", "info", "Default log level; one of debug, info, warn, or error."],
+ ]],
+ ["auth", [
+ ["auth.token_ttl_s", "integer", "3600", "Lifetime of an issued access token in seconds."],
+ ["auth.mfa_required", "boolean", "true", "Whether multi-factor authentication is required for interactive sign-in."],
+ ["auth.session_idle_timeout_s", "integer", "1800", "Idle time after which an interactive session is invalidated."],
+ ]],
+];
+function apiSampleValue(name, type) {
+ if (type === "integer") return name.includes("pct") ? 87 : name.includes("port") ? 8412 : name.includes("count") ? 12 : 3;
+ if (type === "boolean") return true;
+ if (type === "number") return 0.42;
+ if (type === "timestamp") return "2044-05-01T12:00:00Z";
+ const strings = { id: "r-1a2b3c", model: "R-200", status: "idle", name: "north-fleet", version: "2044.2.0", severity: "warning", kind: "storage", channel: "stable", url: "https://example.com/hooks/nwr", label: "reporting-key", scopes: "read:robots read:orders" };
+ return strings[name] || `${name.replace(/[^a-z]/gi, "")}-sample`;
+}
+function apiExample(r, includeAll) {
+ const o = {};
+ for (const [name, type] of r.fields) {
+ if (!includeAll && (name === "id" || type === "timestamp")) continue;
+ o[name] = apiSampleValue(name, type);
+ }
+ return JSON.stringify(o, null, 2);
+}
+function buildApiRefBlocks() {
+ const b = [];
+ const H1 = "Northwind Robotics Platform API — Reference";
+ b.push({ h1: H1 });
+ b.push({ p: "This is the complete reference for the Northwind Robotics Platform API. It documents every resource, its endpoints, the request parameters and response fields, and the configuration keys that govern each service. It is intended for integrators building on top of the fulfillment platform." });
+ b.push({ p: "The API is a conventional JSON REST API. All requests and responses use UTF-8 JSON, all timestamps are RFC 3339 in UTC, and all identifiers are opaque strings that you should treat as case-sensitive and not parse." });
+ b.push({ h2: "Base URL and versioning" });
+ b.push({ p: "The API is served at https://api.northwind.example over TLS only. The version is embedded in the path, for example /v1/robots. Breaking changes are introduced under a new version prefix; additive changes may appear within a version without notice, so clients must ignore unknown fields." });
+ b.push({ h2: "Authentication" });
+ b.push({ p: "Every request must carry an API key in the Authorization header as a bearer token. Keys are scoped: a key may only call endpoints covered by its scopes, and a request outside a key's scopes returns 403. Rotate keys regularly and never embed them in client-side code." });
+ b.push({ code: 'GET /v1/robots\nAuthorization: Bearer nwr_live_9f8e7d6c5b4a\nAccept: application/json' });
+ b.push({ h2: "Request and response conventions" });
+ b.push({ p: "List endpoints return a paginated envelope with a data array and a page object. Single-resource endpoints return the resource object directly. Write endpoints accept a JSON body and return the created or updated resource. Successful writes return 200 or 201; deletes return 204 with no body." });
+ b.push({ h2: "Pagination" });
+ b.push({ p: "List endpoints accept page and page_size query parameters. The default page size is 50 and the maximum is 200. The response page object reports the current page, the page size, and the total number of matching records so a client can iterate deterministically." });
+ b.push({ h2: "Filtering and sorting" });
+ b.push({ p: "Most list endpoints accept filter parameters named after indexed fields, and a sort parameter of the form field or -field for descending order. Filters combine with logical AND. Unknown filter parameters are rejected with 400 so typos fail loudly rather than silently returning everything." });
+ b.push({ h2: "Rate limiting" });
+ b.push({ p: "Requests are limited per API key per minute. When a client exceeds its ceiling the API returns 429 with a Retry-After header, and the client should back off exponentially. The default ceiling and other limits are listed in the configuration reference below." });
+ b.push({ h2: "Idempotency" });
+ b.push({ p: "Unsafe create requests may include an Idempotency-Key header. If the same key is replayed within twenty-four hours the API returns the original result instead of creating a duplicate, which makes retries after a network error safe." });
+ b.push({ h2: "Errors" });
+ b.push({ p: "Errors use standard HTTP status codes and a JSON body with a code and a human-readable message. The common codes are listed below." });
+ for (const [code, meaning] of [["400 Bad Request", "the request was malformed or a parameter was invalid"], ["401 Unauthorized", "the API key was missing or invalid"], ["403 Forbidden", "the API key lacks the required scope"], ["404 Not Found", "no resource exists with the given identifier"], ["409 Conflict", "the write conflicts with the current state, for example a duplicate"], ["413 Payload Too Large", "the request body exceeded a configured size limit"], ["422 Unprocessable Entity", "the body was valid JSON but failed a business rule"], ["429 Too Many Requests", "the client exceeded its rate limit"], ["500 Internal Server Error", "an unexpected error; safe to retry with backoff"], ["503 Service Unavailable", "the service is applying backpressure; retry later"]]) {
+ b.push({ li: `${code} — ${meaning}.` });
+ }
+ b.push({ h2: "Endpoints" });
+ for (const r of API_RESOURCES) {
+ b.push({ h3: `The ${r.p} resource` });
+ b.push({ p: `A ${r.s} represents ${r.about}.` });
+ const writable = r.fields.filter(([n, t]) => n !== "id" && t !== "timestamp");
+ const fieldBullets = (fields) => { for (const [n, t, d] of fields) b.push({ li: `\`${n}\` (${t}) — ${d}.` }); };
+ // list
+ b.push({ h3: `GET ${r.base}` });
+ b.push({ p: `List ${r.p} visible to the caller. Results are paginated and may be filtered by ${r.filter} and sorted by any indexed field. Use this endpoint to enumerate ${r.p} for dashboards and reconciliation jobs.` });
+ b.push({ p: "Query parameters:" });
+ b.push({ li: "`page` (integer, optional) — 1-based page number; defaults to 1." });
+ b.push({ li: "`page_size` (integer, optional) — items per page, up to 200; defaults to 50." });
+ b.push({ li: "`sort` (string, optional) — an indexed field, prefixed with `-` for descending order." });
+ b.push({ p: `Each ${r.s} in the \`data\` array has the following fields:` });
+ fieldBullets(r.fields);
+ b.push({ p: "Example response:" });
+ b.push({ code: `{\n "data": [\n${apiExample(r, true).split("\n").map((l) => " " + l).join("\n")}\n ],\n "page": { "page": 1, "page_size": 50, "total": 128 }\n}` });
+ // create
+ b.push({ h3: `POST ${r.base}` });
+ b.push({ p: `Create a new ${r.s}. The request body carries the writable fields; server-managed fields such as \`id\` and timestamps are ignored if supplied and set by the server. Returns 201 with the created ${r.s}.` });
+ b.push({ p: "Request body fields:" });
+ fieldBullets(writable);
+ b.push({ p: "Example request body:" });
+ b.push({ code: apiExample(r, false) });
+ // get
+ b.push({ h3: `GET ${r.base}/{id}` });
+ b.push({ p: `Fetch a single ${r.s} by its identifier. Returns the full ${r.s} representation, or 404 if no ${r.s} exists with that id. The response includes every field listed above.` });
+ b.push({ li: "`id` (string, path) — the identifier of the ${r.s} to fetch." });
+ // update
+ b.push({ h3: `PATCH ${r.base}/{id}` });
+ b.push({ p: `Partially update a ${r.s}. Only the fields present in the body are changed; omitted fields are left as they are. Returns the updated ${r.s}, or 409 if the update conflicts with the current state. Any of these writable fields may be supplied:` });
+ fieldBullets(writable);
+ // delete
+ b.push({ h3: `DELETE ${r.base}/{id}` });
+ b.push({ p: `Delete a ${r.s}. Returns 204 on success and is idempotent: deleting an already-deleted ${r.s} also returns 204. Deletion may be rejected with 409 if other records still reference it.` });
+ // optional action
+ if (r.action) {
+ const [name, method, desc] = r.action;
+ b.push({ h3: `${method.toUpperCase()} ${r.base}/{id}/${name}` });
+ b.push({ p: `${desc} This action is asynchronous where it affects hardware and returns the updated ${r.s} with its new status. A 409 is returned if the ${r.s} is in a state where the action does not apply.` });
+ }
+ }
+ b.push({ h2: "Configuration reference" });
+ b.push({ p: "Each service reads its configuration from a namespaced key space. Keys may be set through the config file, an environment variable, or the admin API; the admin API takes precedence, then the environment, then the file. The tables below list every key, its type, its default, and what it controls. Defaults are chosen to be safe for a mid-sized single-warehouse deployment; tune them for your scale." });
+ for (const [svc, keys] of API_CONFIG) {
+ b.push({ h3: `${svc} configuration` });
+ b.push({ p: `Keys in the ${svc} namespace control the ${svc} service. Restart the service after changing a file-based key; admin-API changes take effect immediately.` });
+ for (const [key, type, def, desc] of keys) {
+ if (desc === "__BURY_BATCH__") {
+ // Buried unique answer line (single list item => single physical line).
+ b.push({ li: `\`${key}\` (${type}, default \`${def}\`) — Maximum records accepted in a single bulk request; the ingest max_batch_size parameter defaults to 4096 records per bulk request, and larger batches are rejected with HTTP 413.` });
+ } else {
+ b.push({ li: `\`${key}\` (${type}, default \`${def}\`) — ${desc}` });
+ }
+ }
+ }
+ b.push({ h2: "Webhooks" });
+ b.push({ p: "Subscribe to events by creating a webhook with an HTTPS endpoint and an event type. Northwind delivers each event as a signed JSON POST and retries with exponential backoff for up to twenty-four hours if your endpoint does not return 2xx. Verify the signature header on every delivery before trusting the payload." });
+ b.push({ h2: "SDKs and changelog" });
+ b.push({ p: "Official SDKs wrap this API for common languages and handle authentication, pagination, and retries for you. The changelog on the developer portal records every additive change within a version and every new version prefix, with migration notes." });
+ return { path: "engineering/api-reference.md", format: "md", title: H1, blocks: b };
+}
+
+// ---- 3) Major-incident runbook (TXT) -------------------------------------
+// A long operational runbook: severity model, roles, comms, a bank of service
+// playbooks, an alert reference, and checklists. The unique answer (a comms
+// deadline) is buried in the communications section.
+const RUNBOOK_PLAYBOOKS = [
+ ["Ingest backpressure / rising queue depth", "New floor events are accepted slowly or rejected, and dashboards fall behind real time.",
+ ["Queue depth on the ingest service is climbing toward the high watermark.", "Clients are seeing 503 responses on batch submission.", "The event lag metric is rising steadily."],
+ ["Check ingest.worker_threads utilization and CPU headroom.", "Confirm downstream (planner, inventory) are consuming, not stalled.", "Look for a single misbehaving client flooding batches."],
+ ["Scale ingest workers horizontally to drain the queue.", "Throttle or block the offending client key at the edge.", "If a downstream is stalled, treat that as the primary incident and fail this one over to it."]],
+ ["Planner connection-pool starvation", "Pick routes are issued late and throughput in one or more centers drops.",
+ ["Planner tail latency exceeds its budget.", "Database connection-wait time on the live path is elevated.", "The planner logs show fallbacks to serial planning."],
+ ["Check whether reporting queries are saturating a shared pool.", "Confirm the live path is using its dedicated pool, not the reporting pool.", "Inspect for a long-running or runaway query on the replica."],
+ ["Shed reporting load and move heavy queries to the dedicated read pool.", "Confirm pool isolation between the live path and reporting.", "If a runaway query is found, terminate it and capture it for follow-up."]],
+ ["Inventory divergence", "Reported stock levels disagree with physical counts or reservations do not release.",
+ ["Reservations exceed on-hand for one or more SKUs.", "The reconciliation job reports a growing delta.", "Operators report picks failing at supposedly-stocked locations."],
+ ["Check the reservation TTL and whether expiries are being processed.", "Confirm the event log replayed cleanly after any recent schema change.", "Look for a stuck normalize or aggregate stage."],
+ ["Restart the stalled pipeline stage and let it catch up from the log.", "Manually expire stale reservations for affected SKUs after confirming with operations.", "Trigger a targeted reconciliation for the affected locations."]],
+ ["API elevated error rate", "External integrators see 5xx responses or high latency from the public API.",
+ ["The API 5xx rate is above its alert threshold.", "Latency p99 on the API is elevated.", "One dependency (auth, inventory) is slow or erroring."],
+ ["Identify whether errors are concentrated on one endpoint or dependency.", "Check the rate-limiter for false positives after a config change.", "Confirm auth token issuance is healthy."],
+ ["If one dependency is at fault, mitigate that dependency and shed load from the API.", "Roll back a recent API deploy if the error rate tracks it.", "Communicate a degraded-service status to affected integrators."]],
+ ["Robot fleet offline in one zone", "A cluster of robots stops responding and picking halts in part of a warehouse.",
+ ["Multiple robots in one zone show as offline within a short window.", "The maintenance VLAN in that zone looks unhealthy.", "No corresponding software deploy occurred."],
+ ["Check the zone's network access points and switch health.", "Confirm it is not a firmware rollout gone wrong.", "Verify power to the charging infrastructure in that zone."],
+ ["Engage facilities and network on the physical layer as the likely cause.", "Pause the affected fleet to stop dispatching into a dead zone.", "Fail routes over to an adjacent healthy zone if layout allows."]],
+ ["Bad firmware rollout", "Robots that took a new firmware build misbehave or drop offline after an update.",
+ ["Failures correlate with a specific firmware version.", "Robots on the previous build are healthy.", "The rollout channel recently advanced."],
+ ["Confirm the version boundary between healthy and unhealthy robots.", "Check that the build signature was valid before rollout.", "Review the firmware changelog for the suspect build."],
+ ["Halt the rollout immediately on all channels.", "Roll affected robots back to the last known-good build.", "Quarantine the bad build so it cannot be re-selected."]],
+ ["Reporting replica saturation", "Analytics and dashboards slow down and may impact the live path if pools are shared.",
+ ["Replica CPU or IO is pegged.", "Reporting query latency is high.", "Live-path latency rises if isolation is imperfect."],
+ ["Identify the heaviest reporting queries.", "Confirm the live path is isolated from reporting connections.", "Check for a missing index after a schema change."],
+ ["Move or kill the heaviest queries and add the missing index.", "Confirm and, if needed, harden pool isolation.", "Rate-limit ad-hoc reporting during peak."]],
+ ["Object-store / event-log unavailable", "The append-only event log cannot be read or written, stalling the pipeline.",
+ ["Writes to the event bucket fail or time out.", "The pipeline cannot advance past collect.", "Object-store health checks are red."],
+ ["Confirm whether it is a provider outage or a credential/permission problem.", "Check bucket-level throttling or quota.", "Verify network path to the object store."],
+ ["If provider-side, follow the vendor's status and buffer events locally if possible.", "Rotate or fix credentials if it is an auth failure.", "Once restored, let the pipeline replay from the last committed offset."]],
+ ["Authentication / SSO outage", "Users and services cannot obtain tokens, blocking sign-in and internal calls.",
+ ["Token issuance fails or is very slow.", "Interactive sign-in fails across tools.", "Service-to-service calls return 401."],
+ ["Check the identity provider's health and recent changes.", "Confirm clock skew is within tolerance for token validation.", "Look for an expired signing certificate."],
+ ["If a certificate expired, rotate it and redeploy.", "Engage the identity provider if it is provider-side.", "Communicate the outage and expected recovery to all staff."]],
+ ["Deploy-induced regression", "Service health degrades immediately after a deploy to production.",
+ ["Error rate or latency steps up at the deploy timestamp.", "The previous version was healthy.", "The change is in the suspect service's path."],
+ ["Correlate the metric change precisely with the deploy time.", "Check the deploy's diff for the likely culprit.", "Confirm no coincident infrastructure change."],
+ ["Roll back to the previous version as the first action.", "Confirm health recovers after rollback before investigating further.", "Open a follow-up to fix forward safely."]],
+ ["Certificate expiry", "TLS connections fail because a certificate has expired or is about to.",
+ ["Clients report certificate-validation errors.", "A certificate's not-after date has passed or is imminent.", "The affected endpoint served fine until the expiry moment."],
+ ["Identify the exact certificate and where it is terminated.", "Confirm the renewal automation did or did not run.", "Check for multiple endpoints sharing the certificate."],
+ ["Issue and deploy a renewed certificate immediately.", "Fix the renewal automation so it does not recur.", "Audit for other certificates nearing expiry."]],
+ ["Disk / storage exhaustion", "A service degrades or crashes because a volume has filled up.",
+ ["A volume is at or near 100 percent used.", "Writes fail or the service restarts in a loop.", "Log or spool growth is the likely cause."],
+ ["Identify what is consuming the space.", "Check whether log rotation or cleanup stopped.", "Confirm it is not a runaway export or dump."],
+ ["Free space by rotating or shipping logs and removing safe temporary files.", "Grow the volume if the growth is legitimate.", "Restore the cleanup job and add a headroom alert."]],
+ ["Database primary failover", "The inventory primary becomes unavailable and writes stall until a replica is promoted.",
+ ["Writes to the inventory service fail or time out.", "The primary is unreachable or in a crash loop.", "Replication lag on standbys is being watched closely."],
+ ["Confirm the primary is genuinely down, not just slow.", "Check which standby is most caught up.", "Verify the failover automation is armed, not paused."],
+ ["Let the automation promote the healthiest standby, or promote manually if it is paused.", "Repoint the inventory service at the new primary.", "Confirm write path health, then rebuild a fresh standby."]],
+ ["Message backlog on the event bus", "Downstream consumers fall behind and derived data goes stale.",
+ ["Consumer lag is growing across one or more topics.", "Dashboards and rollups are behind real time.", "No consumer errors, just slowness."],
+ ["Identify the slowest consumer group.", "Check whether a single partition is hot.", "Confirm consumers are not blocked on a downstream call."],
+ ["Scale the lagging consumer group out.", "Rebalance partitions if one is hot.", "If a downstream is the bottleneck, treat that as the primary incident."]],
+ ["Cache stampede", "A cache expiry or flush causes a surge of expensive origin queries.",
+ ["Origin query load spikes sharply.", "Latency rises across cache-backed endpoints.", "Cache hit rate drops suddenly."],
+ ["Confirm whether a mass expiry or a flush just occurred.", "Check for a thundering-herd pattern on one key.", "Verify the origin is not itself degraded."],
+ ["Enable request coalescing so only one origin fetch runs per key.", "Warm the hottest keys before re-enabling traffic.", "Stagger future expiries to avoid synchronized misses."]],
+ ["Time synchronization drift", "Clock skew across hosts breaks token validation, ordering, or scheduling.",
+ ["Token validation fails intermittently.", "Event ordering looks wrong.", "Hosts report differing times."],
+ ["Check the time-sync service on the affected hosts.", "Measure the actual skew against a trusted source.", "Confirm it is not a single bad host."],
+ ["Restart or fix time synchronization on drifting hosts.", "Isolate any host that cannot be corrected.", "Re-validate token flows once clocks agree."]],
+ ["Third-party dependency outage", "An external provider Northwind relies on is degraded or down.",
+ ["Calls to the provider fail or time out.", "The provider's status page reports an issue.", "Only features using that provider are affected."],
+ ["Confirm it is provider-side, not our integration.", "Check whether a cached or degraded mode is available.", "Estimate blast radius across features."],
+ ["Switch affected features to degraded mode where one exists.", "Communicate the limited impact to customers.", "Follow the provider's status and retry once restored."]],
+ ["Runaway background job", "A background job consumes excessive resources and starves foreground work.",
+ ["CPU, memory, or IO is dominated by one job.", "Foreground latency rises while the job runs.", "The job started around the degradation."],
+ ["Identify the job and what it is doing.", "Check whether it is looping or processing an unexpected volume.", "Confirm it is safe to pause."],
+ ["Pause or throttle the runaway job.", "Cap its resource budget so it cannot starve foreground work.", "Fix the input or logic that caused the runaway before re-enabling it."]],
+ ["Load-balancer misroute", "Traffic is routed to unhealthy or wrong backends after a change.",
+ ["A subset of requests fail while others succeed.", "Health checks disagree with reality.", "A routing or config change preceded the issue."],
+ ["Compare the live routing config against the intended one.", "Check backend health-check definitions.", "Confirm which backends are actually receiving traffic."],
+ ["Revert the routing change if it caused the misroute.", "Drain and replace genuinely unhealthy backends.", "Fix the health-check definition so it reflects real health."]],
+ ["Data-export pipeline failure", "A scheduled customer or analytics export does not produce its output.",
+ ["An export job is marked failed or is stuck.", "The expected output artifact is missing.", "Downstream consumers of the export are blocked."],
+ ["Read the export job's error and last successful run.", "Check source-data availability and permissions.", "Confirm the destination is writable."],
+ ["Re-run the export once the cause is fixed.", "Backfill any missed windows in order.", "Add a freshness alert so a silent failure is caught sooner."]],
+ ["Security alert during an incident", "A security signal fires while an operational incident is in progress.",
+ ["An unexpected access pattern or alert appears.", "The signal may or may not relate to the incident.", "It arrives amid the operational response."],
+ ["Do not dismiss it as noise; capture it for the security team.", "Assess whether it changes the incident's severity.", "Preserve evidence rather than wiping affected systems."],
+ ["Engage the security on-call in parallel with the operational response.", "Contain rather than destroy if compromise is suspected.", "Let security drive any breach-notification assessment."]],
+ ["Elevated latency with no obvious cause", "Requests are slow across a service but nothing is clearly broken.",
+ ["Latency is up while error rate is normal.", "No single dependency looks unhealthy.", "Throughput is roughly unchanged."],
+ ["Check saturation of CPU, memory, threads, and connection pools.", "Look for a slow dependency hidden in the tail.", "Confirm no noisy-neighbour effect on shared infrastructure."],
+ ["Relieve the saturated resource, scaling out if needed.", "Add capacity to a starved pool.", "If a neighbour is the cause, isolate the workload."]],
+ ["Partial region degradation", "One region is unhealthy while others are fine.",
+ ["Errors or latency are concentrated in a single region.", "Other regions serve normally.", "A regional dependency or change is suspected."],
+ ["Confirm the impact is genuinely region-scoped.", "Check for a region-local change or provider issue.", "Verify cross-region failover is available and healthy."],
+ ["Fail the affected region's traffic over to a healthy region if safe.", "Address the region-local cause once traffic is protected.", "Fail back carefully after confirming recovery."]],
+ ["Corrupted or unexpected input data", "Malformed input causes downstream processing to fail or misbehave.",
+ ["A stage rejects or mishandles records.", "The problem started when a new data source or format appeared.", "Reprocessing the same input reproduces it."],
+ ["Identify the offending records and their source.", "Confirm whether validation should have caught them.", "Check whether the input contract changed upstream."],
+ ["Quarantine the bad input so the pipeline can proceed.", "Tighten validation to reject rather than corrupt.", "Coordinate a fix with the upstream data producer."]],
+];
+const RUNBOOK_ALERTS = [
+ ["INGEST_QUEUE_HIGH", "Ingest queue depth is above the high watermark", "Check downstream consumers first, then scale ingest workers."],
+ ["INGEST_CLIENT_FLOOD", "A single client is submitting batches far above its norm", "Throttle or block the client key at the edge."],
+ ["PLANNER_TAIL_LATENCY", "Planner tail latency exceeds its budget", "Check for connection-pool contention with reporting."],
+ ["PLANNER_SERIAL_FALLBACK", "Planner has fallen back to serial planning", "Investigate database connection availability on the live path."],
+ ["INVENTORY_DELTA", "Reconciliation delta is growing", "Check for a stalled pipeline stage or stuck reservations."],
+ ["RESERVATION_LEAK", "Reservations are not expiring as expected", "Confirm the reservation TTL processor is running."],
+ ["API_5XX_RATE", "API 5xx error rate above threshold", "Isolate the failing endpoint or dependency; consider rollback."],
+ ["API_LATENCY_P99", "API p99 latency elevated", "Check dependencies and recent deploys."],
+ ["RATE_LIMIT_SPIKE", "A surge of 429 responses", "Confirm it is genuine load and not a limiter misconfiguration."],
+ ["ROBOT_ZONE_OFFLINE", "Many robots offline in one zone", "Suspect network or power in that zone; engage facilities."],
+ ["FIRMWARE_ROLLOUT_ERRORS", "Errors correlate with a firmware version", "Halt the rollout and roll affected robots back."],
+ ["REPLICA_SATURATION", "Reporting replica CPU or IO saturated", "Shed or kill heavy queries; verify pool isolation."],
+ ["EVENTLOG_UNAVAILABLE", "The event log cannot be read or written", "Determine provider outage versus credentials; buffer if possible."],
+ ["SSO_TOKEN_FAILURE", "Token issuance failing", "Check the identity provider and signing certificate."],
+ ["CERT_EXPIRY_SOON", "A certificate expires soon", "Renew and deploy before the not-after date."],
+ ["DISK_USAGE_HIGH", "A volume is nearly full", "Free space, then grow the volume if growth is legitimate."],
+ ["DEPLOY_REGRESSION", "Health degraded right after a deploy", "Roll back first, investigate second."],
+ ["HEARTBEAT_GAP", "A service stopped sending heartbeats", "Check the process and its host before assuming a wider outage."],
+ ["BACKUP_FAILED", "A scheduled backup did not complete", "Re-run the backup and investigate the cause before the next window."],
+ ["QUEUE_STUCK", "A background task queue is not draining", "Check the worker pool and for a poison message."],
+];
+const RUNBOOK_DEPS = [
+ ["ingest service", "receives floor events; depends on the event log and downstream consumers."],
+ ["planner", "assigns routes; depends on the inventory service and its dedicated database pool."],
+ ["inventory service", "authoritative stock; depends on the primary PostgreSQL cluster."],
+ ["reporting replica", "serves analytics; must stay isolated from the live picking path."],
+ ["event log", "append-only history in the object store; source of truth for replay."],
+ ["public API", "external surface; depends on auth, inventory, and the rate limiter."],
+ ["auth / SSO", "issues tokens; a hard dependency for interactive and service calls."],
+ ["object store", "durable storage for the event log and exports."],
+ ["maintenance network", "carries firmware and robot control traffic per zone."],
+ ["observability backend", "receives metrics, logs, and traces; degraded observability slows every incident."],
+];
+const RUNBOOK_COMMS = [
+ ["Initial external update (Sev-1)", "We are investigating an issue affecting order fulfillment for some customers. Our team is engaged and we will post another update shortly. No action is needed from you at this time."],
+ ["Follow-up external update", "We have identified the cause of the fulfillment delays and are applying a fix. Some orders may be processed more slowly than usual until the fix is fully in effect. Thank you for your patience."],
+ ["Mitigated external update", "A fix has been applied and fulfillment is recovering. We are monitoring closely and will confirm full resolution once we are confident the issue is resolved. No customer action is required."],
+ ["Resolved external update", "This incident is resolved. Fulfillment has returned to normal and no data was lost. We will publish a summary once our review is complete. Thank you for bearing with us."],
+ ["Initial internal update", "Sev-1 declared for degraded fulfillment. Incident commander and communications lead are assigned. Current hypothesis and owner are in the channel. Next update in fifteen minutes or sooner if the picture changes."],
+ ["Internal handover note", "Handing over incident command. Current state, actions taken, hypotheses ruled in and out, and the immediate next step are pinned in the channel. Communications lead is unchanged. The action log is up to date."],
+];
+const RUNBOOK_DIAGNOSTICS = [
+ ["Confirm the blast radius", "before diving in, establish which customers, warehouses, or services are affected, and which are not."],
+ ["Check recent changes", "list deploys, config changes, and infrastructure events in the last hour; a coincident change is the most common cause."],
+ ["Read the golden signals", "look at latency, traffic, errors, and saturation for the suspect service before drilling into logs."],
+ ["Follow the dependency chain", "trace from the failing symptom toward its upstream dependencies to find the true cause."],
+ ["Inspect queue depths", "rising queues point to a downstream that has stalled or slowed."],
+ ["Check connection-pool utilization", "pool exhaustion on the live path is a recurring cause of latency spikes."],
+ ["Compare a healthy peer", "diff a healthy instance or zone against the unhealthy one to isolate what differs."],
+ ["Verify certificate validity", "an expired certificate produces sudden, total failures on an otherwise-healthy endpoint."],
+ ["Check clock skew", "skewed clocks silently break token validation and event ordering."],
+ ["Look for a poison message", "a single malformed message can stall a whole consumer group."],
+ ["Confirm pool isolation", "verify the live picking path is not sharing a connection pool with reporting."],
+ ["Check disk and inode headroom", "a full volume or exhausted inodes crashes services in confusing ways."],
+ ["Review the error budget", "decide how aggressively to intervene based on how much budget remains."],
+ ["Capture evidence early", "snapshot logs, metrics, and state before you change anything, for the review and for security."],
+ ["Test the rollback path", "confirm you can roll back the suspect change before you need to."],
+ ["Validate the fix on one instance", "apply a change to a single instance and confirm recovery before rolling it out widely."],
+ ["Watch for secondary failures", "a fix under load can trigger a new bottleneck; keep watching after you act."],
+ ["Confirm data integrity", "for any incident touching stored data, verify integrity before declaring resolution."],
+ ["Check replication lag", "before a failover, know which replica is most caught up."],
+ ["Re-run health checks", "confirm health checks reflect reality and are not themselves misconfigured."],
+ ["Narrow the time window", "pin the exact minute impact began; it usually lines up with a change or event."],
+ ["Bisect the change set", "if several changes landed together, isolate which one by reverting or disabling them one at a time."],
+ ["Check upstream provider status", "rule in or out a third-party outage before spending time on your own stack."],
+ ["Inspect thread and connection pools", "saturated pools produce latency that looks like a downstream problem but is not."],
+ ["Look at the slowest requests", "the tail often reveals the cause that averages hide."],
+ ["Confirm feature-flag state", "a flag flipped at the wrong scale can degrade a service without any deploy."],
+ ["Check for retry storms", "aggressive client retries can turn a small blip into a self-sustaining overload."],
+ ["Validate config actually loaded", "confirm the running process is using the config you think it is."],
+];
+function buildRunbookBlocks() {
+ const b = [];
+ const H1 = "Northwind Robotics — Major-Incident Runbook";
+ b.push({ h1: H1 });
+ b.push({ p: "This runbook is the first thing an on-call engineer opens when paged for a potential major incident. It defines the severity model, the incident roles, how we communicate, a bank of service-specific playbooks, an alert reference, and the checklists we use to open and close an incident. Read the top sections now, before you are paged, so that under pressure you only need the playbook and the checklists." });
+ b.push({ h2: "How to use this runbook" });
+ b.push({ p: "When you are paged, first classify the severity, then declare an incident if it qualifies, then open the incident channel and assign roles. Only after that do you dive into the relevant service playbook. The single most common mistake under pressure is to start fixing before declaring and communicating; resist it." });
+ b.push({ h2: "Severity levels" });
+ b.push({ p: "Sev-1 is a full customer-facing outage or data-integrity risk. Sev-2 is significant degraded service with a workaround. Sev-3 is a minor issue with limited or no customer impact. When in doubt, declare the higher severity; it is cheap to downgrade and expensive to under-respond." });
+ b.push({ li: "Sev-1 — customer-facing outage or data at risk; all-hands response, executive awareness." });
+ b.push({ li: "Sev-2 — degraded service or a major feature down with a workaround; focused response." });
+ b.push({ li: "Sev-3 — minor or internal-only impact; handled in normal working hours." });
+ b.push({ h2: "Incident roles" });
+ b.push({ p: "Every declared incident has, at minimum, an incident commander and a communications lead. The incident commander owns the response and makes the calls; they do not fix things themselves. The communications lead owns updates to stakeholders and the status page. On larger incidents we also add an operations lead and a scribe." });
+ b.push({ li: "Incident commander — coordinates the response and owns decisions; delegates the hands-on work." });
+ b.push({ li: "Communications lead — owns stakeholder updates and the public status page." });
+ b.push({ li: "Operations lead — directs the hands-on remediation work." });
+ b.push({ li: "Scribe — keeps a timestamped log of actions and decisions for the review." });
+ b.push({ h2: "Communications" });
+ b.push({ p: "Clear, frequent communication is the difference between a well-run incident and a chaotic one. The communications lead posts updates on a fixed rhythm so stakeholders never have to ask for status. Internal updates go to the incident channel; external updates go to the public status page in plain, non-technical language." });
+ // Buried unique answer line (single paragraph => single physical line; TXT is not wrapped).
+ b.push({ p: "Post a public status-page update within twenty minutes of declaring a Sev-1, and refresh it at least every thirty minutes until the incident is resolved." });
+ b.push({ p: "Do not speculate about causes in external updates, never share customer data, and always say what customers should do, even if that is simply to wait. The incident commander approves every external update before it is posted." });
+ b.push({ h2: "Escalation timeline" });
+ b.push({ p: "Pages escalate automatically if they are not acknowledged. If the primary on-call does not acknowledge within ten minutes, the page goes to the secondary. If the secondary does not acknowledge within a further twenty minutes, it escalates to the engineering manager. Keep the incident channel updated at every step so escalation is never a surprise." });
+ b.push({ h2: "Service playbooks" });
+ b.push({ p: "Each playbook below covers one common failure mode: how it looks, what to check, and how to remediate. Playbooks are starting points, not scripts; use judgement, and if two playbooks seem to apply, treat the upstream cause as the primary incident." });
+ for (const [title, impact, symptoms, checks, remediation] of RUNBOOK_PLAYBOOKS) {
+ b.push({ h3: title });
+ b.push({ p: "Impact. " + impact + " Classify the severity from the customer impact, not from how alarming the internal signals look." });
+ b.push({ p: "Detection. This failure mode usually announces itself through one or more of the following signals; the more of them you see together, the more confident you can be:" });
+ for (const s of symptoms) b.push({ li: s });
+ b.push({ p: "Triage. Work through these checks in order before you change anything. The aim is to find the upstream cause rather than to treat a downstream symptom, because treating symptoms tends to prolong incidents:" });
+ for (const c of checks) b.push({ li: c });
+ b.push({ p: "Remediation. Once you have identified the cause, apply the smallest change that restores service, and prefer a reversible action over a clever one:" });
+ for (const rr of remediation) b.push({ li: rr });
+ b.push({ p: "Verification. After remediating, do not stand down immediately. Confirm the primary signal has recovered, confirm no secondary bottleneck has appeared, and confirm any data touched by the incident is consistent. Only then consider the incident mitigated." });
+ b.push({ p: "Rollback and escalation. If remediation does not restore service within a reasonable window, escalate per the timeline above and prepare to roll back the most recent change on this path. Keep the incident channel updated so the incident commander can decide whether to widen the response or pull in another team." });
+ b.push({ p: "Follow-up. Record a timeline, note which mitigations were temporary, and open follow-ups so this failure is less likely and less severe next time. A playbook that had to be improvised is itself a follow-up: update this document." });
+ }
+ b.push({ h2: "Communication templates" });
+ b.push({ p: "Use these as starting points, not scripts. Fill in the specifics, keep external updates free of jargon and speculation, and always have the incident commander approve an external update before it is posted." });
+ for (const [label, text] of RUNBOOK_COMMS) { b.push({ h3: label }); b.push({ p: text }); }
+ b.push({ h2: "Diagnostic reference" });
+ b.push({ p: "A checklist of high-value diagnostic moves that apply across many incidents. When you are not sure where to look, work down this list." });
+ for (const [name, how] of RUNBOOK_DIAGNOSTICS) b.push({ li: `${name} — ${how}` });
+ b.push({ h2: "Alert reference" });
+ b.push({ p: "The alerts below page or notify on-call. For each, the immediate first response is given; the relevant playbook has the full procedure." });
+ for (const [name, meaning, first] of RUNBOOK_ALERTS) b.push({ li: `${name}: ${meaning}. First response: ${first}` });
+ b.push({ h2: "Service dependency reference" });
+ b.push({ p: "Knowing what depends on what lets you find the upstream cause quickly instead of chasing symptoms." });
+ for (const [svc, dep] of RUNBOOK_DEPS) b.push({ li: `${svc} — ${dep}` });
+ b.push({ h2: "On-call preparation" });
+ b.push({ p: "The best incident response happens before the page. When you go on call, confirm you can receive pages, that your access to production tooling and the incident channel works, and that you know where this runbook lives. A five-minute check at the start of a shift saves precious minutes later." });
+ b.push({ p: "Keep a personal on-call kit: a charged phone, a way to get online quickly, and bookmarks to the dashboards, the status page, and the escalation contacts. Know who your secondary is and how to reach the incident commander of the week. If you are new to on-call, shadow an experienced engineer through at least one real or simulated incident first." });
+ b.push({ p: "Rest matters. If you are too tired or unwell to respond effectively, hand over early rather than muddling through. There is no heroism in a slow, error-prone response; there is a lot of value in a clear-headed one." });
+ b.push({ h2: "Severity decision examples" });
+ b.push({ p: "Severity is judged by customer impact. These worked examples show how to classify quickly under pressure; when a case sits on a boundary, choose the higher severity." });
+ for (const ex of [
+ "A whole fulfillment center cannot pick orders: Sev-1, because customers are directly and broadly affected.",
+ "The public API returns errors for all integrators: Sev-1, because external customers are unable to operate.",
+ "Dashboards are stale but picking continues normally: Sev-3, because there is no direct customer impact.",
+ "One robot is offline in an otherwise-healthy zone: Sev-3, handled in normal hours unless it spreads.",
+ "Order throughput is degraded with a working slow path: Sev-2, because service is impaired but not down.",
+ "A reporting replica is saturated but the live path is isolated and healthy: Sev-3, watched closely.",
+ "Sign-in fails company-wide because SSO is down: Sev-1, because it blocks both staff and services.",
+ "A single background export failed but can be re-run: Sev-3, tracked but not an emergency.",
+ ]) b.push({ li: ex });
+ b.push({ h2: "Incident glossary" });
+ b.push({ p: "Shared vocabulary keeps a busy incident channel clear." });
+ for (const [t, d] of [
+ ["Incident commander", "the person coordinating the response and owning decisions; does not fix things personally."],
+ ["Communications lead", "the person who owns stakeholder updates and the public status page."],
+ ["Blast radius", "the set of customers, services, or data affected by an incident."],
+ ["Mitigation", "an action that reduces or removes customer impact, even if the root cause remains."],
+ ["Remediation", "the action that addresses the actual cause of the incident."],
+ ["Rollback", "reverting a recent change to restore a known-good state."],
+ ["Escalation", "bringing in additional people or authority when the current response is not enough."],
+ ["Golden signals", "latency, traffic, errors, and saturation — the first metrics to check."],
+ ["Error budget", "the tolerance for unreliability that guides how aggressively to intervene."],
+ ["Backpressure", "a system slowing or rejecting input to protect itself from overload."],
+ ["Poison message", "a single malformed item that repeatedly stalls a consumer."],
+ ["Failover", "switching to a standby component when the primary is unavailable."],
+ ["Replication lag", "how far behind a replica is from its primary."],
+ ["Post-incident review", "the blameless review that turns an incident into durable improvements."],
+ ["Standing down", "formally ending the active response once service is confirmed healthy."],
+ ["Time in lieu", "compensating time off for hours worked responding out of hours."],
+ ]) b.push({ li: `${t} — ${d}` });
+ b.push({ h2: "Incident-open checklist" });
+ b.push({ p: "Run through this list in the first few minutes of a declared incident." });
+ for (const item of ["Classify the severity and declare the incident.", "Open the incident channel and pin the summary.", "Assign the incident commander and communications lead.", "Post the first status update within the communications deadline.", "Start a timestamped action log.", "Identify the most likely upstream cause before making changes."]) b.push({ li: item });
+ b.push({ h2: "Incident-close checklist" });
+ b.push({ p: "Before you stand down, confirm each of these." });
+ for (const item of ["Service is confirmed healthy for at least one full check interval.", "A final status update has been posted and the status page is green.", "The action log and timeline are complete.", "Temporary mitigations are recorded with follow-ups to make them permanent or remove them.", "A post-incident review is scheduled with an owner."]) b.push({ li: item });
+ b.push({ h2: "Post-incident review" });
+ b.push({ p: "Every Sev-1 and Sev-2 gets a blameless review within a week. The review establishes the timeline, the contributing factors, and a small set of high-leverage follow-ups with owners and dates. The goal is to make the same failure less likely and less severe, not to assign blame." });
+ return { path: "engineering/runbooks/major-incident-runbook.txt", format: "txt", title: H1, blocks: b };
+}
+
+// ---- 4) Information security policy (PDF via FODT) -----------------------
+// A long policy is a catalogue of mandatory controls; each control expands via a
+// realistic scaffold. The unique answer (a remediation SLA) is buried in the
+// vulnerability-management control. Being a PDF, it is ALSO invisible to grep.
+const SEC_CONTROLS = [
+ ["AC-1", "Access control policy", "Access to Northwind systems and data is granted on the principle of least privilege and only for a legitimate business need.", ["access is granted by role rather than to individuals wherever possible", "every grant has an identifiable owner who is accountable for it", "access that is no longer needed is removed promptly"]],
+ ["AC-2", "Account management", "User and service accounts are created, changed, and removed through a controlled, auditable process.", ["accounts are provisioned only after an approved request", "shared accounts are prohibited except where technically unavoidable and explicitly approved", "leavers' accounts are disabled on their last day"]],
+ ["AC-3", "Privileged access", "Administrative access to production is tightly restricted, time-boxed, and fully logged.", ["privileged access requires a second approver", "elevated sessions are time-boxed and expire automatically", "every privileged action is logged and reviewed"]],
+ ["AC-4", "Multi-factor authentication", "Interactive access to internal systems requires multi-factor authentication.", ["all staff enrol at least one strong second factor", "single-factor exceptions require documented risk acceptance", "phishing-resistant factors are preferred for privileged roles"]],
+ ["AC-5", "Session management", "Interactive sessions are protected against hijacking and abandonment.", ["idle sessions time out automatically", "sessions are invalidated on sign-out and on credential change", "concurrent-session limits apply to privileged roles"]],
+ ["IA-1", "Password standard", "Where passwords are used they meet a minimum strength standard and are never reused across systems.", ["minimum length and complexity are enforced by the identity provider", "known-breached passwords are rejected", "passwords are stored only as salted, hashed values"]],
+ ["IA-2", "Secrets management", "Application secrets are stored in the approved secret manager, never in code or configuration files.", ["secrets are injected at runtime, not baked into images", "secrets are rotated on a schedule and after any suspected exposure", "access to secrets is scoped and logged"]],
+ ["IA-3", "Key management", "Cryptographic keys are generated, stored, rotated, and destroyed under a documented lifecycle.", ["keys are stored in the managed key service", "keys are rotated on a regular schedule", "retired keys are destroyed once no longer needed for decryption"]],
+ ["IA-4", "Certificate management", "TLS certificates are issued, tracked, and renewed before expiry through automation.", ["certificates are inventoried centrally", "renewal is automated with alerting well before expiry", "private keys never leave their intended host or service"]],
+ ["CR-1", "Encryption in transit", "All data in transit over untrusted networks is encrypted with current, strong protocols.", ["TLS is required for all external and internal service traffic", "obsolete protocol versions and ciphers are disabled", "certificate validation is enforced, not bypassed"]],
+ ["CR-2", "Encryption at rest", "Confidential and restricted data is encrypted at rest.", ["storage-level encryption is enabled on all data stores", "restricted datasets carry application-level encryption in addition", "encryption keys are managed under the key-management control"]],
+ ["NW-1", "Network segmentation", "Networks are segmented so that a compromise in one zone does not grant free movement to others.", ["production, corporate, and maintenance networks are separated", "traffic between segments is default-deny and explicitly allowed", "the robot maintenance network is isolated from the corporate network"]],
+ ["NW-2", "Firewall and edge control", "Inbound and outbound network access is controlled at well-defined boundaries.", ["only required ports and destinations are permitted", "rules are reviewed periodically and stale rules removed", "changes to edge rules follow the change-management control"]],
+ ["NW-3", "Remote access", "Remote access to internal systems is authenticated, encrypted, and limited to what is needed.", ["remote access uses the approved VPN or zero-trust gateway", "device posture is checked before access is granted", "split-tunnel exceptions require approval"]],
+ ["SD-1", "Secure development lifecycle", "Security is built into how software is designed, built, and released.", ["threat modelling is applied to significant new features", "security requirements are captured alongside functional ones", "releases pass automated security checks before production"]],
+ ["SD-2", "Code review", "Code changes are peer-reviewed before they reach production.", ["every change requires at least one independent approval", "security-sensitive changes get an additional security review", "review comments are resolved, not dismissed silently"]],
+ ["SD-3", "Dependency management", "Third-party dependencies are inventoried and kept free of known vulnerabilities.", ["a software bill of materials is maintained per service", "dependencies are scanned continuously for known issues", "unmaintained or high-risk dependencies are replaced"]],
+ ["VM-1", "Vulnerability management", "Vulnerabilities are discovered, prioritized by severity, and remediated within defined timeframes.", ["automated scanning runs continuously across systems and images", "findings are triaged and assigned an owner", "remediation deadlines are tracked to closure"]],
+ ["VM-2", "Penetration testing", "Independent testing validates the effectiveness of controls.", ["external penetration tests are performed on a regular cadence", "significant findings feed the vulnerability-management process", "retests confirm that fixes are effective"]],
+ ["CM-1", "Patch management", "Systems and software are kept current with security patches.", ["security patches are applied within severity-based windows", "unattended systems are covered by automated patching", "exceptions are documented with compensating controls"]],
+ ["CM-2", "Change management", "Changes to production are controlled, reviewed, and reversible.", ["changes are peer-reviewed and recorded", "high-risk changes have a rollback plan and a change window", "emergency changes are reviewed retrospectively"]],
+ ["CM-3", "Configuration and hardening", "Systems are configured to a documented secure baseline.", ["hardening baselines are defined for each platform", "configuration is managed as code and drift is detected", "unused services and default credentials are removed"]],
+ ["EP-1", "Endpoint protection", "Employee and server endpoints run approved protection and are centrally managed.", ["endpoints run the approved security agent", "disk encryption is enforced on all laptops", "endpoints out of compliance are flagged and remediated"]],
+ ["EP-2", "Mobile device management", "Mobile devices accessing company data are enrolled and controlled.", ["company data is accessed only from enrolled devices", "a lost device can be remotely wiped of company data", "a screen lock and encryption are enforced"]],
+ ["EP-3", "Email and phishing defence", "Email is filtered and staff are prepared to recognize phishing.", ["inbound mail is scanned for malware and spoofing", "external mail is clearly marked", "staff can report suspected phishing with one click"]],
+ ["DP-1", "Data classification", "Information is classified so that it is protected in proportion to its sensitivity.", ["data is classified as public, internal, confidential, or restricted", "the classification determines handling and access", "owners are responsible for classifying their data"]],
+ ["DP-2", "Data retention", "Data is kept only as long as it is needed or legally required, then securely destroyed.", ["retention periods are defined per data category", "records past their retention are securely destroyed", "legal holds override normal retention when in force"]],
+ ["DP-3", "Data loss prevention", "Controls reduce the risk of confidential data leaving the organization improperly.", ["egress of restricted data is monitored and controlled", "sharing of confidential data externally requires justification", "removable-media use is restricted"]],
+ ["DP-4", "Privacy and data protection", "Personal data is processed lawfully, fairly, and only for legitimate purposes.", ["personal data is collected only where there is a lawful basis", "access to personal data is limited to those who need it", "data-subject requests are handled within legal timeframes"]],
+ ["BC-1", "Backup and recovery", "Critical data is backed up and recoveries are tested.", ["backups run on a defined schedule with monitored success", "backups are stored durably and, for critical data, off-site", "restores are tested regularly, not assumed to work"]],
+ ["BC-2", "Business continuity", "The business can continue operating through significant disruptions.", ["continuity plans exist for critical services", "recovery objectives are defined and reviewed", "plans are exercised and updated after each exercise"]],
+ ["BC-3", "Disaster recovery", "Critical systems can be recovered within their defined objectives after a disaster.", ["recovery-time and recovery-point objectives are documented", "recovery procedures are kept current", "disaster-recovery exercises validate the procedures"]],
+ ["LM-1", "Logging and monitoring", "Security-relevant events are logged, retained, and monitored.", ["systems emit security-relevant logs to a central store", "logs are protected from tampering", "logs are monitored for signs of compromise"]],
+ ["LM-2", "Security event management", "Alerts are triaged and investigated by defined owners.", ["security alerts are routed to an accountable team", "alerts are triaged against a documented severity model", "investigations and outcomes are recorded"]],
+ ["IR-1", "Incident response", "Security incidents are detected, contained, eradicated, and reviewed.", ["an incident-response plan defines roles and steps", "incidents are contained before eradication and recovery", "every incident gets a blameless review with follow-ups"]],
+ ["IR-2", "Breach notification", "Where required, affected parties and regulators are notified promptly.", ["notification obligations are understood in advance", "a breach is assessed against those obligations quickly", "notifications are made within the required timeframes"]],
+ ["TP-1", "Vendor risk management", "Third parties that handle Northwind data are assessed and monitored.", ["vendors are risk-assessed before onboarding", "contracts include appropriate security and privacy terms", "vendor access is reviewed periodically"]],
+ ["TP-2", "Third-party access", "External parties are granted only the access they need, for only as long as they need it.", ["third-party access is time-boxed and least-privilege", "external accounts are clearly identifiable", "access is revoked at the end of the engagement"]],
+ ["PS-1", "Physical security", "Facilities and hardware are protected against unauthorized physical access.", ["access to offices and labs is badge-controlled", "visitors are signed in and escorted near active robots", "server and network equipment is kept in secured areas"]],
+ ["PS-2", "Media handling and disposal", "Storage media are handled and disposed of so that data cannot be recovered improperly.", ["media carrying confidential data are tracked", "media are securely wiped or destroyed before disposal", "disposal is documented for restricted data"]],
+ ["HR-1", "Personnel security", "Security responsibilities are built into the employment lifecycle.", ["background checks are performed where lawful and appropriate", "confidentiality obligations are agreed on joining", "access is adjusted promptly on role change or exit"]],
+ ["HR-2", "Security awareness training", "Staff are trained to understand and meet their security responsibilities.", ["all staff complete security training on joining and annually", "role-specific training is provided where needed", "training completion is tracked"]],
+ ["HR-3", "Acceptable use", "Company systems and equipment are used appropriately and lawfully.", ["the acceptable-use policy governs use of company systems", "incidental personal use is kept reasonable and lawful", "misuse is handled under the disciplinary process"]],
+ ["CP-1", "Compliance and audit", "The security programme is measured against its obligations and improved.", ["controls are audited on a defined cadence", "findings are tracked to closure", "the policy is updated as obligations and risks change"]],
+];
+const SEC_DEFS = [
+ ["Confidentiality", "ensuring information is accessible only to those authorized to have access."],
+ ["Integrity", "safeguarding the accuracy and completeness of information and processing methods."],
+ ["Availability", "ensuring authorized users have access to information and systems when required."],
+ ["Least privilege", "granting only the minimum access necessary to perform a task."],
+ ["Restricted data", "the most sensitive classification, requiring the strongest handling and encryption."],
+ ["Confidential data", "sensitive information whose disclosure could harm Northwind or its customers."],
+ ["Internal data", "information intended for use within Northwind but not publicly disclosed."],
+ ["Public data", "information approved for public release."],
+ ["Control owner", "the person accountable for a control operating effectively."],
+ ["Compensating control", "an alternative safeguard used when a primary control cannot be met."],
+ ["Exception", "a documented, time-boxed, and approved deviation from a control."],
+ ["Recovery-time objective", "the target time to restore a service after a disruption."],
+ ["Recovery-point objective", "the maximum acceptable amount of data loss measured in time."],
+ ["Threat model", "a structured analysis of how a system could be attacked and how it is defended."],
+ ["Software bill of materials", "an inventory of the components and dependencies in a piece of software."],
+];
+function buildSecurityBlocks() {
+ const b = [];
+ const H1 = "Northwind Robotics — Information Security Policy";
+ b.push({ h1: H1 });
+ b.push({ p: "This Information Security Policy defines how Northwind Robotics protects the confidentiality, integrity, and availability of the information and systems that run its autonomous fulfillment platform. It applies to every employee, contractor, and third party that accesses Northwind systems or data, regardless of where the system is hosted or where the person is working." });
+ b.push({ p: "The policy is organized as a catalogue of mandatory controls. Each control states its purpose, the systems and people it applies to, its specific requirements, how it is implemented, and how compliance is verified. Controls are cumulative: satisfying one does not excuse non-compliance with another." });
+ b.push({ p: "Northwind treats security as an engineering discipline. Controls are automated and enforced in code wherever possible, exceptions are documented and time-boxed, and every control is reviewed on a fixed cadence and after any material change or related incident." });
+ b.push({ h2: "Scope and applicability" });
+ b.push({ p: "This policy covers all production, staging, and corporate systems, all customer and employee data, and all networks operated by or on behalf of Northwind Robotics. Where a customer contract or applicable law imposes a stricter requirement than this policy, the stricter requirement governs." });
+ b.push({ h2: "Roles and responsibilities" });
+ b.push({ p: "The Chief Information Security Officer owns this policy and approves exceptions. Engineering managers are accountable for the controls that apply to the systems they operate. Every individual is responsible for completing security training, protecting the credentials and data entrusted to them, and reporting suspected incidents promptly." });
+ b.push({ p: "A security review board meets monthly to review exceptions, incidents, and proposed changes to this policy, and its decisions are recorded in the governance register." });
+ b.push({ h2: "Security controls" });
+ for (const [id, title, purpose, reqs] of SEC_CONTROLS) {
+ b.push({ h3: `${id} — ${title}` });
+ b.push({ p: purpose });
+ b.push({ p: `This control applies to all Northwind Robotics production and corporate systems in scope, and compliance is mandatory unless a documented, time-boxed exception has been approved by the Chief Information Security Officer.` });
+ b.push({ p: "Requirements. " + reqs.map((r, i) => (i === 0 ? "First, " : i === 1 ? "second, " : i === 2 ? "third, " : "and additionally, ") + r).join("; ") + "." });
+ b.push({ p: "Implementation is enforced through configuration and tooling managed as code, so that the control is applied consistently and any change is peer-reviewed and logged before it takes effect in production." });
+ b.push({ p: "Compliance is assessed on a defined cadence through automated checks and a documented review, and the resulting evidence is retained in the governance register so that internal and external auditors can confirm the control is operating effectively." });
+ if (id === "VM-1") {
+ // Buried unique answer line — short so pdftotext keeps it on one line.
+ b.push({ p: "Critical vulnerabilities must be remediated within seven days of confirmed disclosure." });
+ b.push({ p: "High-severity findings are remediated within thirty days, and medium and low findings are scheduled into the normal maintenance cycle. Every remediation deadline starts when a finding is confirmed, not when it is first reported, and slipping a deadline requires documented risk acceptance." });
+ }
+ }
+ b.push({ h2: "Appendix A — Definitions" });
+ b.push({ p: "The terms below are used throughout this policy." });
+ for (const [t, d] of SEC_DEFS) b.push({ li: `${t} — ${d}` });
+ b.push({ h2: "Appendix B — Review and exceptions" });
+ b.push({ p: "This policy is reviewed at least annually and whenever a material change to the environment, the business, or the threat landscape warrants it. Exceptions to any control must be requested in writing, carry a business justification and a compensating control, be time-boxed, and be approved by the Chief Information Security Officer. Expired exceptions are treated as non-compliance." });
+ return { path: "security/information-security-policy.pdf", format: "pdf", title: H1, blocks: b };
+}
+
+// The four large documents, materialised by build().
+const LARGE_DOCS = [buildHandbookBlocks(), buildApiRefBlocks(), buildRunbookBlocks(), buildSecurityBlocks()];
+
+// ---------------------------------------------------------------------------
+// Ground-truth query set. Distribution per SPEC (REVISION 2 — honesty fix):
+// binary_only (answer ONLY in a pdf/docx) — 7
+// large_literal (answer buried in a >=60 KB doc; both can hit, — 4
+// but the baseline must open the whole large file)
+// robustness (answer worded differently; a FAIR grep of the — 5
+// question's own tokens can TIE — not a semantic win)
+// literal (plain string in a small html/md/txt; both find) — 6
+// keywords = curated terms the baseline greps (UNION'd with question tokens).
+// expect_substring = the ground-truth answer text (case-insensitive check).
+// ---------------------------------------------------------------------------
+const QUERIES = [
+ // ---------- binary_only (answer lives ONLY in a pdf/docx) ----------
+ {
+ id: "q01",
+ question: "How many weeks of paid parental leave do employees receive?",
+ keywords: ["parental leave", "paid leave"],
+ expect_substring: "16 weeks",
+ answer_path: "corpus/hr/handbook.pdf",
+ answer_format: "pdf",
+ match_type: "binary_only",
+ },
+ {
+ id: "q02",
+ question: "How many days of paid time off do full-time employees accrue per year?",
+ keywords: ["paid time off", "PTO"],
+ expect_substring: "22 days",
+ answer_path: "corpus/hr/handbook.pdf",
+ answer_format: "pdf",
+ match_type: "binary_only",
+ },
+ {
+ id: "q03",
+ question: "How many days of bereavement leave are provided per event?",
+ keywords: ["bereavement leave", "bereavement"],
+ expect_substring: "5 working days",
+ answer_path: "corpus/hr/benefits/leave-policy.docx",
+ answer_format: "docx",
+ match_type: "binary_only",
+ },
+ {
+ id: "q04",
+ question: "After how long does an unacknowledged page escalate to the secondary on-call engineer?",
+ keywords: ["escalation", "secondary on-call"],
+ expect_substring: "15 minutes",
+ answer_path: "corpus/engineering/oncall-runbook.docx",
+ answer_format: "docx",
+ match_type: "binary_only",
+ },
+ {
+ id: "q05",
+ question: "What is the acknowledgement SLA for a Sev-1 incident?",
+ keywords: ["Sev-1", "acknowledge"],
+ expect_substring: "within 5 minutes",
+ answer_path: "corpus/engineering/oncall-runbook.docx",
+ answer_format: "docx",
+ match_type: "binary_only",
+ },
+ {
+ id: "q06",
+ question: "How long must customer transaction records be retained?",
+ keywords: ["retention", "retained"],
+ expect_substring: "7 years",
+ answer_path: "corpus/security/data-classification.pdf",
+ answer_format: "pdf",
+ match_type: "binary_only",
+ },
+ {
+ id: "q07",
+ question: "What is the domestic meal per diem rate?",
+ keywords: ["per diem", "meal allowance"],
+ expect_substring: "$75 per day",
+ answer_path: "corpus/finance/expense-policy.pdf",
+ answer_format: "pdf",
+ match_type: "binary_only",
+ },
+
+ // ---------- large_literal (answer buried deep in a >=60 KB document) ----------
+ // A FAIR baseline CAN grep these answer lines, but to read/verify the answer it
+ // must open the whole large file (context blowup); XERJ returns one ranked
+ // passage. ll04 additionally lives in a PDF, so grep is doubly blind there.
+ {
+ id: "ll01",
+ question: "How much is the employee referral bonus for a successful hire?",
+ keywords: ["referral bonus", "referral"],
+ expect_substring: "$3,000 per successful hire",
+ answer_path: "corpus/hr/employee-handbook.html",
+ answer_format: "html",
+ match_type: "large_literal",
+ },
+ {
+ id: "ll02",
+ question: "What is the default value of the ingest max_batch_size parameter?",
+ keywords: ["max_batch_size", "batch size"],
+ expect_substring: "defaults to 4096 records",
+ answer_path: "corpus/engineering/api-reference.md",
+ answer_format: "md",
+ match_type: "large_literal",
+ },
+ {
+ id: "ll03",
+ question: "How quickly must a public status-page update be posted after declaring a Sev-1?",
+ keywords: ["status page", "status-page update"],
+ expect_substring: "within twenty minutes",
+ answer_path: "corpus/engineering/runbooks/major-incident-runbook.txt",
+ answer_format: "txt",
+ match_type: "large_literal",
+ },
+ {
+ id: "ll04",
+ question: "Within how many days must critical vulnerabilities be remediated?",
+ keywords: ["vulnerability remediation", "critical vulnerabilities"],
+ expect_substring: "within seven days",
+ answer_path: "corpus/security/information-security-policy.pdf",
+ answer_format: "pdf",
+ match_type: "large_literal",
+ },
+
+ // ---------- robustness (answer phrased differently; FAIR grep of the question's
+ // own tokens can TIE — kept to show single-query convenience, NOT a semantic win) ----------
+ {
+ id: "s01",
+ question: "What is Northwind's policy on remote work versus coming into the office?",
+ keywords: ["remote work", "telecommute"],
+ expect_substring: "three days a week",
+ answer_path: "corpus/operations/facilities.md",
+ answer_format: "md",
+ match_type: "robustness",
+ },
+ {
+ id: "s02",
+ question: "How does a user recover access to a locked account?",
+ keywords: ["reset password", "forgot password"],
+ expect_substring: "self-service portal",
+ answer_path: "corpus/security/access-control.html",
+ answer_format: "html",
+ match_type: "robustness",
+ },
+ {
+ id: "s03",
+ question: "What developer hardware is issued to a newly hired engineer on their first day?",
+ keywords: ["laptop", "company computer"],
+ expect_substring: "16-inch developer workstation",
+ answer_path: "corpus/engineering/onboarding.md",
+ answer_format: "md",
+ match_type: "robustness",
+ },
+ {
+ id: "s04",
+ question: "Which data store backs the inventory and SKU records?",
+ keywords: ["database", "product catalog"],
+ expect_substring: "PostgreSQL 15 cluster",
+ answer_path: "corpus/engineering/architecture/system-overview.md",
+ answer_format: "md",
+ match_type: "robustness",
+ },
+ {
+ id: "s05",
+ question: "What can a customer do if they want to send a purchased unit back?",
+ keywords: ["refund", "return policy"],
+ expect_substring: "full reimbursement",
+ answer_path: "corpus/product/faq.html",
+ answer_format: "html",
+ match_type: "robustness",
+ },
+
+ // ---------- literal (plain string in html/md/txt; BOTH approaches find) ----------
+ {
+ id: "l01",
+ question: "What is the default API rate limit?",
+ keywords: ["rate limit", "API rate limit"],
+ expect_substring: "1000 requests per minute",
+ answer_path: "corpus/engineering/architecture/data-pipeline.md",
+ answer_format: "md",
+ match_type: "literal",
+ },
+ {
+ id: "l02",
+ question: "What port does the ingest service listen on?",
+ keywords: ["port", "ingest service"],
+ expect_substring: "port 8412",
+ answer_path: "corpus/engineering/architecture/system-overview.md",
+ answer_format: "md",
+ match_type: "literal",
+ },
+ {
+ id: "l03",
+ question: "When is the weekly engineering sync held?",
+ keywords: ["weekly engineering sync", "engineering sync"],
+ expect_substring: "Wednesday at 10:00",
+ answer_path: "corpus/meetings/decision-log.md",
+ answer_format: "md",
+ match_type: "literal",
+ },
+ {
+ id: "l04",
+ question: "What uptime does the platform SLA guarantee?",
+ keywords: ["uptime", "SLA"],
+ expect_substring: "99.9% uptime",
+ answer_path: "corpus/product/faq.html",
+ answer_format: "html",
+ match_type: "literal",
+ },
+ {
+ id: "l05",
+ question: "Who must approve purchases over the standard threshold?",
+ keywords: ["approve", "purchase"],
+ expect_substring: "Director of Operations",
+ answer_path: "corpus/finance/procurement.html",
+ answer_format: "html",
+ match_type: "literal",
+ },
+ {
+ id: "l06",
+ question: "What is the Q3 target for warehouse pick accuracy?",
+ keywords: ["pick accuracy", "warehouse"],
+ expect_substring: "99.5% pick accuracy",
+ answer_path: "corpus/product/roadmap.txt",
+ answer_format: "txt",
+ match_type: "literal",
+ },
+];
+
+// ---------------------------------------------------------------------------
+// FODT authoring + soffice conversion
+// ---------------------------------------------------------------------------
+function xmlEscape(s) {
+ return s
+ .replace(/&/g, "&amp;")
+ .replace(/</g, "&lt;")
+ .replace(/>/g, "&gt;");
+}
+
+function blocksToFodt(blocks) {
+ const parts = [];
+ for (const b of blocks) {
+ if (b.h1 != null) parts.push(`<text:h text:outline-level="1">${xmlEscape(b.h1)}</text:h>`);
+ else if (b.h2 != null) parts.push(`<text:h text:outline-level="2">${xmlEscape(b.h2)}</text:h>`);
+ else if (b.h3 != null) parts.push(`<text:h text:outline-level="3">${xmlEscape(b.h3)}</text:h>`);
+ else if (b.p != null) parts.push(`<text:p>${xmlEscape(b.p)}</text:p>`);
+ else if (b.li != null) parts.push(`<text:p>• ${xmlEscape(b.li)}</text:p>`);
+ else if (b.code != null) parts.push(`<text:p>${xmlEscape(b.code)}</text:p>`);
+ }
+ return `<?xml version="1.0" encoding="UTF-8"?>
+<office:document xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" office:version="1.3" office:mimetype="application/vnd.oasis.opendocument.text">
+<office:body><office:text>
+${parts.join("\n")}
+</office:text></office:body>
+</office:document>
+`;
+}
+
+function soffice(args) {
+ return execFileSync("soffice", ["--headless", SOFFICE_ENV, ...args], {
+ stdio: ["ignore", "pipe", "pipe"],
+ timeout: 120000,
+ }).toString();
+}
+
+// Convert a list of source files to `fmt`, writing into outDir. Returns after
+// verifying every expected output exists; retries any missing one-by-one.
+function convertBatch(sources, convertTo, outDir, expectedExt) {
+ fs.mkdirSync(outDir, { recursive: true });
+ const expected = sources.map(
+ (s) => path.join(outDir, path.basename(s).replace(/\.[^.]+$/, `.${expectedExt}`))
+ );
+ try {
+ soffice(["--convert-to", convertTo, "--outdir", outDir, ...sources]);
+ } catch (e) {
+ // fall through to per-file retry below
+ }
+ for (let i = 0; i < sources.length; i++) {
+ if (!fs.existsSync(expected[i])) {
+ soffice(["--convert-to", convertTo, "--outdir", outDir, sources[i]]);
+ }
+ if (!fs.existsSync(expected[i])) {
+ throw new Error(`soffice failed to produce ${expected[i]} from ${sources[i]}`);
+ }
+ }
+ return expected;
+}
+
+// ---------------------------------------------------------------------------
+// Extraction helpers — MUST mirror what xerj-index.mjs uses, so verification
+// checks the exact text the indexer will see.
+// ---------------------------------------------------------------------------
+function extractPdf(file) {
+ return execFileSync("pdftotext", ["-layout", file, "-"], { timeout: 60000 }).toString();
+}
+function extractDocx(file) {
+ const out = path.join(BUILD, "docx-extract");
+ fs.mkdirSync(out, { recursive: true });
+ soffice(["--convert-to", "txt:Text", "--outdir", out, file]);
+ const txt = path.join(out, path.basename(file).replace(/\.docx$/, ".txt"));
+ return fs.readFileSync(txt, "utf8").replace(/^/, "");
+}
+function stripHtml(html) {
+ return html
+ .replace(/<script[\s\S]*?<\/script>/gi, " ")
+ .replace(/<style[\s\S]*?<\/style>/gi, " ")
+ .replace(/<[^>]+>/g, " ")
+ .replace(/&amp;/g, "&")
+ .replace(/&lt;/g, "<")
+ .replace(/&gt;/g, ">")
+ .replace(/&#39;/g, "'")
+ .replace(/&quot;/g, '"')
+ .replace(/\s+/g, " ")
+ .trim();
+}
+function extractedText(absPath, fmt) {
+ if (fmt === "pdf") return extractPdf(absPath);
+ if (fmt === "docx") return extractDocx(absPath);
+ if (fmt === "html") return stripHtml(fs.readFileSync(absPath, "utf8"));
+ return fs.readFileSync(absPath, "utf8"); // md / txt
+}
+
+const ci = (s) => s.toLowerCase();
+
+// ---------------------------------------------------------------------------
+// Build
+// ---------------------------------------------------------------------------
+function build() {
+ // Idempotent: wipe + recreate corpus/.
+ fs.rmSync(CORPUS, { recursive: true, force: true });
+ fs.mkdirSync(CORPUS, { recursive: true });
+
+ // 1) Text files (small) + large TEXT documents (html/md/txt) rendered from blocks.
+ for (const f of TEXT_FILES) {
+ const abs = path.join(CORPUS, f.path);
+ fs.mkdirSync(path.dirname(abs), { recursive: true });
+ fs.writeFileSync(abs, f.body);
+ }
+ for (const d of LARGE_DOCS) {
+ if (d.format === "pdf") continue; // the large PDF is built via the binary path below
+ const abs = path.join(CORPUS, d.path);
+ fs.mkdirSync(path.dirname(abs), { recursive: true });
+ const content =
+ d.format === "html" ? renderHtml(d.title, d.blocks)
+ : d.format === "md" ? renderMd(d.blocks)
+ : renderTxt(d.blocks);
+ fs.writeFileSync(abs, content);
+ }
+
+ // 2) Binary files via FODT -> soffice. The small BINARIES plus any large PDF docs.
+ const binaries = [
+ ...BINARIES,
+ ...LARGE_DOCS.filter((d) => d.format === "pdf").map((d) => ({ path: d.path, format: "pdf", blocks: d.blocks })),
+ ];
+ const srcDir = path.join(BUILD, "fodt");
+ fs.mkdirSync(srcDir, { recursive: true });
+ // unique intermediate name per binary to avoid basename collisions
+ const jobs = binaries.map((b, i) => {
+ const stem = `bin${String(i).padStart(2, "0")}`;
+ const fodt = path.join(srcDir, `${stem}.fodt`);
+ fs.writeFileSync(fodt, blocksToFodt(b.blocks));
+ return { ...b, fodt, stem };
+ });
+
+ for (const fmt of ["pdf", "docx"]) {
+ const group = jobs.filter((j) => j.format === fmt);
+ if (group.length === 0) continue;
+ const outDir = path.join(BUILD, `out-${fmt}`);
+ const produced = convertBatch(group.map((j) => j.fodt), fmt, outDir, fmt);
+ group.forEach((j, idx) => {
+ const dst = path.join(CORPUS, j.path);
+ fs.mkdirSync(path.dirname(dst), { recursive: true });
+ fs.copyFileSync(produced[idx], dst);
+ });
+ }
+
+ // 3) queries.json
+ fs.writeFileSync(QUERIES_JSON, JSON.stringify(QUERIES, null, 2) + "\n");
+}
+
+// ---------------------------------------------------------------------------
+// Verify the ground-truth invariants that keep the comparison honest.
+// Throws on any violation.
+// ---------------------------------------------------------------------------
+function listTextFiles() {
+ // All non-binary files the grep baseline can actually read.
+ const out = [];
+ const walk = (dir) => {
+ for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
+ const p = path.join(dir, e.name);
+ if (e.isDirectory()) walk(p);
+ else if (/\.(html|md|txt)$/i.test(e.name)) out.push(p);
+ }
+ };
+ walk(CORPUS);
+ return out;
+}
+
+// --- FAIR-baseline term model (mirrors grep-baseline.mjs) ------------------
+// The honest baseline greps the UNION of the curated keywords AND the salient
+// content tokens of the QUESTION itself. We reproduce that here so the
+// self-verify can assert that a fair grep really could (robustness/large_literal)
+// or really could not (binary_only) reach an answer line — the exact fairness the
+// audit demanded. Keep this in sync with grep-baseline.mjs.
+const QUESTION_STOPWORDS = new Set([
+ "a", "about", "above", "after", "again", "against", "all", "am", "an", "and", "any",
+ "are", "as", "at", "be", "because", "been", "before", "being", "below", "between",
+ "both", "but", "by", "can", "cannot", "could", "day", "did", "do", "does", "doing",
+ "done", "down", "during", "each", "few", "for", "from", "further", "get", "gets",
+ "getting", "got", "had", "has", "have", "having", "he", "her", "here", "hers", "him",
+ "his", "how", "i", "if", "in", "into", "is", "it", "its", "let", "long", "many", "may",
+ "me", "might", "more", "most", "much", "must", "my", "no", "nor", "not", "of", "off",
+ "on", "once", "one", "only", "or", "other", "our", "ours", "out", "over", "own", "per",
+ "same", "shall", "she", "should", "so", "some", "such", "than", "that", "the", "their",
+ "theirs", "them", "then", "there", "these", "they", "this", "those", "through", "to",
+ "too", "under", "until", "up", "upon", "us", "very", "versus", "vs", "was", "we",
+ "were", "what", "when", "where", "which", "while", "who", "whom", "why", "will", "with",
+ "would", "you", "your", "yours",
+]);
+function tokenizeQuestion(question) {
+ return String(question)
+ .toLowerCase()
+ .replace(/['’]/g, " ")
+ .match(/[a-z0-9](?:[a-z0-9%$.\-]*[a-z0-9%])?/g) || [];
+}
+function isSalientToken(tok) {
+ if (!tok || QUESTION_STOPWORDS.has(tok)) return false;
+ if (/[0-9%$]/.test(tok)) return true;
+ return /^[a-z][a-z\-]*$/.test(tok) && tok.length >= 3;
+}
+function questionTerms(question) {
+ if (typeof question !== "string" || question.trim() === "") return [];
+ const terms = [];
+ for (const m of question.matchAll(/["“”']([^"“”']+)["“”']/g)) {
+ const phrase = m[1].trim();
+ if (phrase) terms.push(phrase);
+ }
+ const toks = tokenizeQuestion(question);
+ for (let i = 0; i < toks.length; i++) {
+ if (isSalientToken(toks[i])) terms.push(toks[i]);
+ if (i + 1 < toks.length && isSalientToken(toks[i]) && isSalientToken(toks[i + 1])) {
+ terms.push(`${toks[i]} ${toks[i + 1]}`);
+ }
+ }
+ return terms;
+}
+function fairTerms(q) {
+ const union = [...(q.keywords || []), ...questionTerms(q.question)];
+ const seen = new Set();
+ const terms = [];
+ for (const t of union) {
+ const key = ci(t);
+ if (!seen.has(key)) { seen.add(key); terms.push(t); }
+ }
+ return terms;
+}
+// Would a FAIR line-oriented grep of `terms` surface a line containing `sub`?
+// (rg matches a line by any term; a "hit" needs that line to also hold the answer.)
+function baselineLineHit(rawText, terms, sub) {
+ const subLc = ci(sub);
+ for (const raw of rawText.split(/\r?\n/)) {
+ const ln = ci(raw);
+ if (!ln.includes(subLc)) continue;
+ if (terms.some((t) => ln.includes(ci(t)))) return true;
+ }
+ return false;
+}
+
+function verify() {
+ const problems = [];
+ const textFiles = listTextFiles();
+ const textBlob = ci(textFiles.map((f) => fs.readFileSync(f, "utf8")).join("\n"));
+
+ for (const q of QUERIES) {
+ const abs = path.join(HERE, q.answer_path);
+ if (!fs.existsSync(abs)) {
+ problems.push(`[${q.id}] answer_path missing: ${q.answer_path}`);
+ continue;
+ }
+ const text = extractedText(abs, q.answer_format);
+ const sub = ci(q.expect_substring);
+
+ // (A) The answer substring must appear in the extracted text of its file,
+ // using the SAME extraction the indexer uses.
+ if (!ci(text).includes(sub)) {
+ problems.push(`[${q.id}] expect_substring "${q.expect_substring}" NOT in extracted ${q.answer_path}`);
+ }
+
+ if (q.match_type === "binary_only") {
+ // (B) The answer must live ONLY in the binary: the substring must not
+ // appear in ANY text file (else the grep baseline could find it).
+ if (textBlob.includes(sub)) {
+ problems.push(`[${q.id}] binary_only substring "${q.expect_substring}" leaked into a text file`);
+ }
+ if (!(q.answer_format === "pdf" || q.answer_format === "docx")) {
+ problems.push(`[${q.id}] binary_only answer_format must be pdf/docx, got ${q.answer_format}`);
+ }
+ } else if (q.match_type === "large_literal") {
+ // (C) Context-efficiency demonstrator. The source file must be genuinely
+ // LARGE (>=60 KB) so that "load the whole file" >> "one ranked passage",
+ // and the answer must be greppable in the extracted plaintext (check A
+ // already asserts that). For non-binary formats, a FAIR baseline must be
+ // able to reach the line (both approaches HIT — that is the point); for a
+ // pdf/docx it is additionally invisible to grep, which is fine.
+ const bytes = fs.statSync(abs).size;
+ if (bytes < MIN_LARGE_BYTES) {
+ problems.push(`[${q.id}] large_literal source ${q.answer_path} is ${bytes} B (< ${MIN_LARGE_BYTES} B floor)`);
+ }
+ if (!(q.answer_format === "pdf" || q.answer_format === "docx")) {
+ const raw = fs.readFileSync(abs, "utf8");
+ if (!baselineLineHit(raw, fairTerms(q), q.expect_substring)) {
+ problems.push(`[${q.id}] large_literal: a FAIR baseline could not reach "${q.expect_substring}" in ${q.answer_path}`);
+ }
+ }
+ } else if (q.match_type === "robustness") {
+ // (D) Formerly "semantic_only". The HONEST finding: a diligent grep of the
+ // QUESTION's own tokens can TIE here (lexical overlap, not deep meaning).
+ // So we assert the OPPOSITE of the old rigged check: the answer line MUST
+ // be reachable by the question's own salient tokens (never rigged to miss).
+ const raw = fs.readFileSync(abs, "utf8");
+ if (!baselineLineHit(raw, questionTerms(q.question), q.expect_substring)) {
+ problems.push(`[${q.id}] robustness: answer "${q.expect_substring}" is NOT reachable by the question's own tokens in ${q.answer_path} (would be a rigged "semantic win")`);
+ }
+ if (q.answer_format === "pdf" || q.answer_format === "docx") {
+ problems.push(`[${q.id}] robustness should live in html/md/txt, got ${q.answer_format}`);
+ }
+ } else if (q.match_type === "literal") {
+ // (D) The baseline greps keywords and a hit needs a matched LINE that also
+ // contains the substring. Require >=1 keyword co-occurring with the
+ // substring on a single raw line of the answer file.
+ const rawLines = fs.readFileSync(abs, "utf8").split(/\r?\n/).map(ci);
+ const ok = q.keywords.some((kw) =>
+ rawLines.some((ln) => ln.includes(ci(kw)) && ln.includes(sub))
+ );
+ if (!ok) {
+ problems.push(`[${q.id}] literal: no single raw line holds both a keyword and "${q.expect_substring}" (baseline would miss)`);
+ }
+ if (q.answer_format === "pdf" || q.answer_format === "docx") {
+ problems.push(`[${q.id}] literal should live in html/md/txt, got ${q.answer_format}`);
+ }
+ } else {
+ problems.push(`[${q.id}] unknown match_type ${q.match_type}`);
+ }
+ }
+
+ if (problems.length) {
+ throw new Error("Ground-truth verification FAILED:\n - " + problems.join("\n - "));
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Report
+// ---------------------------------------------------------------------------
+function countFormats() {
+ const counts = {};
+ const walk = (dir) => {
+ for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
+ const p = path.join(dir, e.name);
+ if (e.isDirectory()) walk(p);
+ else {
+ const ext = path.extname(e.name).slice(1).toLowerCase();
+ counts[ext] = (counts[ext] || 0) + 1;
+ }
+ }
+ };
+ walk(CORPUS);
+ return counts;
+}
+
+function main() {
+ console.log(`[gen-corpus] build dir: ${BUILD}`);
+ build();
+ for (const d of LARGE_DOCS) {
+ const bytes = fs.statSync(path.join(CORPUS, d.path)).size;
+ console.log(`[gen-corpus] large ${d.format.toUpperCase().padEnd(4)} ${d.path} = ${bytes} B (${(bytes / 1024).toFixed(1)} KB)`);
+ }
+ console.log("[gen-corpus] corpus written, verifying ground truth (live extraction)...");
+ verify();
+
+ const formats = countFormats();
+ const total = Object.values(formats).reduce((a, b) => a + b, 0);
+ const byType = QUERIES.reduce((m, q) => ((m[q.match_type] = (m[q.match_type] || 0) + 1), m), {});
+
+ // Sizes of the large context-efficiency documents (bytes on disk).
+ const largeDocs = LARGE_DOCS.map((d) => ({
+ path: d.path,
+ format: d.format,
+ bytes: fs.statSync(path.join(CORPUS, d.path)).size,
+ }));
+
+ const schema = {
+ corpus_dir: CORPUS,
+ queries_path: QUERIES_JSON,
+ total_docs: total,
+ formats,
+ total_queries: QUERIES.length,
+ binary_only: byType.binary_only || 0,
+ large_literal: byType.large_literal || 0,
+ robustness: byType.robustness || 0,
+ literal: byType.literal || 0,
+ large_docs: largeDocs,
+ min_large_bytes: MIN_LARGE_BYTES,
+ verified_extractable: true,
+ };
+ console.log("[gen-corpus] VERIFICATION PASSED.");
+ console.log("CORPUS_SCHEMA=" + JSON.stringify(schema, null, 2));
+
+ // best-effort cleanup of the scratch build dir
+ try {
+ fs.rmSync(BUILD, { recursive: true, force: true });
+ } catch {}
+}
+
+main();
diff --git a/demo/usecases/doc-index/grep-baseline.mjs b/demo/usecases/doc-index/grep-baseline.mjs
new file mode 100644
index 000000000..03211cf77
--- /dev/null
+++ b/demo/usecases/doc-index/grep-baseline.mjs
@@ -0,0 +1,387 @@
+// grep-baseline.mjs — the BASELINE retriever for the doc-index use case.
+//
+// This is the honest "Claude Code with only shell scripts" approach: to answer a
+// question, the agent runs ripgrep over the RAW corpus files and reads the matching
+// lines (and, to actually quote an answer, the whole matched file) into its context.
+//
+// HONEST CAPABILITY CEILING (per SPEC.md — do not paper over these):
+// 1. BINARY BLINDNESS. PDF and DOCX are binary containers (PDF streams are usually
+// FlateDecode-compressed; DOCX is a ZIP of deflated XML). ripgrep detects the NUL
+// bytes, treats the file as binary, and prints nothing. So any answer that lives
+// ONLY in a .pdf/.docx is structurally unreachable here. We do NOT special-case or
+// exclude those formats — we just run rg over the whole tree and let its real
+// binary detection skip them. The asymmetry is REAL, not simulated.
+// 2. LITERAL ONLY (no meaning layer). rg matches literal substrings (we use
+// -F/fixed-strings, exactly how a naive agent greps salient terms). To be FAIR we
+// grep the UNION of the curated keywords AND the salient content tokens of the
+// QUESTION itself (see termsFor / questionTerms below) — a diligent shell agent
+// would obviously grep the words in its own question, not just a canned keyword
+// list. So when an answer is phrased differently but still shares surface words
+// with the question, this baseline finds it (and ties XERJ's shallow lexical
+// embedder). The REAL, residual gap: it is pure surface overlap. If the answer
+// shares NO literal token with the question or keywords, grep misses it — there is
+// no synonym/semantic understanding, and (see #4) no ranking of the matches.
+// 3. CONTEXT BLOWUP. A matched line is not an answer; to quote/verify the answer the
+// agent must open the whole file into its context window. `bytesToRead` measures
+// that cost (total size of every file it must open). Big files = big token bills.
+// 4. NO RANKING. rg returns matches in file/line order, not by relevance.
+// 5. NO INDEX. Every query re-scans the entire tree from scratch.
+//
+// FAIRNESS NOTE (belongs in the writeup): a script agent *could* shell out to
+// `pdftotext`/`soffice` to defeat problem #1. We deliberately do NOT do that here,
+// because the point of the baseline is the naive shell-only agent. And even a
+// pdftotext-augmented baseline still loses on #2 (no semantics), #3 (still must load
+// whole extracted files to answer — no chunk-level retrieval), and #4 (no ranking).
+// XERJ's win is not "it can read PDFs" alone; it is format-agnostic extraction PLUS
+// chunked, ranked, context-cheap retrieval with a (shallow, lexical) semantic layer.
+//
+// Public API:
+// scoreQuery(query) -> { hits: [{path, line, text}], filesTouched: [path...], bytesToRead }
+// CLI:
+// node grep-baseline.mjs "<query text | query id | keyword>"
+//
+// Env overrides:
+// DOC_INDEX_CORPUS - corpus dir to grep (default: ./corpus)
+// DOC_INDEX_QUERIES - queries.json path (default: ./queries.json)
+// RG_BINARY - explicit ripgrep binary (default: `rg` on PATH)
+//
+// `node --check grep-baseline.mjs` is clean.
+
+import { execFileSync } from 'node:child_process';
+import { statSync, existsSync, readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { dirname, join, relative, isAbsolute } from 'node:path';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+
+// Corpus + query-set locations. Overridable so the compare harness can point us at a
+// staged corpus, but default to the co-located deliverable files.
+const CORPUS_DIR = resolveDir(process.env.DOC_INDEX_CORPUS, join(HERE, 'corpus'));
+const QUERIES_PATH = process.env.DOC_INDEX_QUERIES || join(HERE, 'queries.json');
+
+function resolveDir(envVal, fallback) {
+ if (!envVal) return fallback;
+ return isAbsolute(envVal) ? envVal : join(HERE, envVal);
+}
+
+/**
+ * Turn an absolute path into a repo-/deliverable-relative one for readable output,
+ * matching the "corpus/hr/handbook.pdf" style used in queries.json. Falls back to the
+ * absolute path if it lives outside the deliverable dir.
+ */
+function prettyPath(absPath) {
+ const rel = relative(HERE, absPath);
+ return rel.startsWith('..') ? absPath : rel;
+}
+
+// English function words + interrogatives/auxiliaries that carry no retrieval signal.
+// Dropped from QUESTION tokens so we grep the salient content, not "how"/"the"/"is".
+// This is DELIBERATELY a conservative function-word list: it drops only articles,
+// pronouns, prepositions, conjunctions, auxiliary/quantifier question-words, etc. It
+// does NOT drop content nouns/verbs/number-words (e.g. "day", "one", "long"), because
+// dropping a content word that happens to sit on the answer's line would re-introduce
+// exactly the structural exclusion the audit condemned. When in doubt, KEEP the token.
+// NOTE: this only filters tokens we mine from the question — the curated `keywords`
+// are always kept verbatim (they may legitimately be short, e.g. "PTO", "SLA").
+const QUESTION_STOPWORDS = new Set([
+ 'a', 'about', 'above', 'after', 'again', 'against', 'all', 'am', 'an', 'and', 'any',
+ 'are', 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between',
+ 'both', 'but', 'by', 'can', 'cannot', 'could', 'did', 'do', 'does', 'doing', 'done',
+ 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'get', 'gets', 'getting',
+ 'got', 'had', 'has', 'have', 'having', 'he', 'her', 'here', 'hers', 'him', 'his', 'how',
+ 'i', 'if', 'in', 'into', 'is', 'it', 'its', 'let', 'many', 'may', 'me', 'might', 'more',
+ 'most', 'much', 'must', 'my', 'no', 'nor', 'not', 'of', 'off', 'on', 'once', 'only',
+ 'or', 'other', 'our', 'ours', 'out', 'over', 'own', 'per', 'same', 'shall', 'she',
+ 'should', 'so', 'some', 'such', 'than', 'that', 'the', 'their', 'theirs', 'them',
+ 'then', 'there', 'these', 'they', 'this', 'those', 'through', 'to', 'too', 'under',
+ 'until', 'up', 'upon', 'us', 'very', 'versus', 'vs', 'was', 'we', 'were', 'what',
+ 'when', 'where', 'which', 'while', 'who', 'whom', 'why', 'will', 'with', 'would', 'you',
+ 'your', 'yours',
+]);
+
+// Tokenize the question text. Lowercases, drops apostrophes (so "Northwind's" → the
+// token "northwind"), and keeps word-internal hyphens/dots/percent/dollar so realistic
+// literals survive whole: "16-inch", "sev-1", "99.9%", "$75". Returns tokens in their
+// original order (used both for unigrams and for adjacent bigrams).
+function tokenizeQuestion(question) {
+ return String(question)
+ .toLowerCase()
+ .replace(/['’]/g, ' ') // possessives/contractions → boundaries, not glued tokens
+ .match(/[a-z0-9](?:[a-z0-9%$.\-]*[a-z0-9%])?/g) || [];
+}
+
+// A question token is a "salient content token" (approximating a noun/number/keyword) if
+// it is not a stopword AND is either numeric, contains a digit/%/$ (a literal like "$75"
+// or "sev-1"), or is an alphabetic word of length ≥ 3. This keeps meaning-bearing words
+// and drops filler without needing a POS tagger.
+function isSalientToken(tok) {
+ if (!tok || QUESTION_STOPWORDS.has(tok)) return false;
+ if (/[0-9%$]/.test(tok)) return true; // numbers and numeric literals are salient
+ return /^[a-z][a-z\-]*$/.test(tok) && tok.length >= 3;
+}
+
+/**
+ * Derive salient search terms from the QUESTION itself: content unigrams (stopwords
+ * dropped), quoted phrases kept whole, and adjacent content-word bigrams (e.g. "remote
+ * work", "locked account") which a shell agent would naturally grep for. This is the
+ * heart of the fairness fix — the baseline is NEVER restricted to a curated keyword list
+ * that structurally dodges the answer's own line; it also greps the words in the
+ * question, so a differently-phrased answer that still shares the question's surface
+ * words IS found.
+ */
+function questionTerms(question) {
+ if (typeof question !== 'string' || question.trim() === '') return [];
+ const terms = [];
+ // 1. Quoted phrases (single or double) are high-value literals — keep them whole.
+ for (const m of question.matchAll(/["“”']([^"“”']+)["“”']/g)) {
+ const phrase = m[1].trim();
+ if (phrase) terms.push(phrase);
+ }
+ // 2. Content unigrams + adjacent content bigrams.
+ const toks = tokenizeQuestion(question);
+ for (let i = 0; i < toks.length; i++) {
+ if (isSalientToken(toks[i])) terms.push(toks[i]);
+ if (i + 1 < toks.length && isSalientToken(toks[i]) && isSalientToken(toks[i + 1])) {
+ terms.push(`${toks[i]} ${toks[i + 1]}`); // bigram, matched literally with rg -F
+ }
+ }
+ return terms;
+}
+
+/**
+ * Search terms for a query = the UNION of the curated `keywords` AND the salient content
+ * tokens mined from the `question` itself (deduped, case-insensitive). This is exactly
+ * what a diligent shell-only agent would grep: its keyword hunches PLUS the words in its
+ * own question. It is deliberately NOT capped to `keywords` — the v1 baseline was capped
+ * to a curated list that dodged the answer line, which an audit correctly flagged as
+ * rigged. A bare string caller (CLI ad-hoc) greps the whole string as one literal term.
+ */
+function termsFor(query) {
+ if (typeof query === 'string') return [query];
+
+ const union = [];
+ if (Array.isArray(query.keywords)) {
+ for (const k of query.keywords) {
+ if (typeof k === 'string' && k.trim() !== '') union.push(k.trim());
+ }
+ }
+ union.push(...questionTerms(query.question));
+
+ // Dedupe case-insensitively, preserving first-seen order for stable/readable output.
+ const seen = new Set();
+ const terms = [];
+ for (const t of union) {
+ const key = t.toLowerCase();
+ if (seen.has(key)) continue;
+ seen.add(key);
+ terms.push(t);
+ }
+
+ // Last-resort fallback: if a query somehow has neither keywords nor a usable question,
+ // grep the raw question string so we never silently search for nothing.
+ if (terms.length === 0 && typeof query.question === 'string' && query.question.trim() !== '') {
+ return [query.question.trim()];
+ }
+ return terms;
+}
+
+/**
+ * Resolve how to invoke ripgrep, memoized. A community user just has `rg` on PATH —
+ * that is the normal path. We also support `RG_BINARY=/path/to/rg` as an override, and
+ * a fallback for the Claude Code sandbox, where `rg` is a shell function (not a binary
+ * on PATH) that dispatches to ripgrep embedded in the `claude` executable via argv0=rg;
+ * node can reproduce that with the spawn `argv0` option.
+ * @returns {{cmd:string, opts:object}|null}
+ */
+let _rg; // memo: undefined = unprobed, null = unavailable, object = resolved
+function resolveRg() {
+ if (_rg !== undefined) return _rg;
+ const probe = (cmd, opts) => {
+ try {
+ execFileSync(cmd, ['--version'], { stdio: 'ignore', ...opts });
+ return true;
+ } catch {
+ return false;
+ }
+ };
+ // 1. explicit override
+ if (process.env.RG_BINARY && probe(process.env.RG_BINARY, {})) {
+ return (_rg = { cmd: process.env.RG_BINARY, opts: {} });
+ }
+ // 2. a real `rg` binary on PATH (the normal community case)
+ if (probe('rg', {})) return (_rg = { cmd: 'rg', opts: {} });
+ // 3. Claude Code sandbox: ripgrep is embedded in the `claude` binary (argv0=rg)
+ const claudeBin = process.env.CLAUDE_CODE_EXECPATH || '/home/claude/.local/bin/claude';
+ if (probe(claudeBin, { argv0: 'rg' })) {
+ return (_rg = { cmd: claudeBin, opts: { argv0: 'rg' } });
+ }
+ return (_rg = null);
+}
+
+/**
+ * Run ripgrep for the given literal terms over the corpus and return raw matches.
+ * Flags: -i (case-insensitive), -n (line numbers), --no-heading (path per line),
+ * -F (fixed/literal strings — a naive substring grep), one `-e TERM` per search term (OR).
+ * We do NOT pass -a/--text: default binary detection is what makes PDF/DOCX yield
+ * nothing, which is the honest behavior of `rg` on those formats.
+ */
+function runRipgrep(terms) {
+ if (!terms.length) return '';
+ if (!existsSync(CORPUS_DIR)) {
+ process.stderr.write(
+ `[grep-baseline] corpus dir not found: ${CORPUS_DIR} (generate it with gen-corpus.mjs)\n`,
+ );
+ return '';
+ }
+ const rg = resolveRg();
+ if (!rg) {
+ process.stderr.write(
+ '[grep-baseline] ripgrep not found. Install rg, or set RG_BINARY=/path/to/rg.\n',
+ );
+ return '';
+ }
+ const args = ['-i', '-n', '--no-heading', '-F'];
+ for (const t of terms) args.push('-e', t);
+ args.push(CORPUS_DIR);
+ try {
+ return execFileSync(rg.cmd, args, {
+ encoding: 'utf8',
+ maxBuffer: 64 * 1024 * 1024,
+ ...rg.opts,
+ });
+ } catch (err) {
+ // ripgrep exit codes: 0 = matches, 1 = no matches (NORMAL, not an error),
+ // 2 = actual error. Only surface real errors.
+ if (err && err.status === 1) return '';
+ if (err && typeof err.status === 'number') {
+ process.stderr.write(
+ `[grep-baseline] rg exited ${err.status}: ${String(err.stderr || err.message).trim()}\n`,
+ );
+ return '';
+ }
+ // spawn failure.
+ process.stderr.write(`[grep-baseline] could not run rg: ${err && err.message}\n`);
+ return '';
+ }
+}
+
+/**
+ * Parse a single `path:line:text` ripgrep line (produced by -n --no-heading).
+ * Absolute POSIX paths contain no ':', so the first two colons are the separators.
+ */
+function parseRgLine(raw) {
+ const i1 = raw.indexOf(':');
+ if (i1 < 0) return null;
+ const rest = raw.slice(i1 + 1);
+ const i2 = rest.indexOf(':');
+ if (i2 < 0) return null;
+ const lineNo = Number.parseInt(rest.slice(0, i2), 10);
+ if (!Number.isFinite(lineNo)) return null;
+ return {
+ absPath: raw.slice(0, i1),
+ line: lineNo,
+ text: rest.slice(i2 + 1),
+ };
+}
+
+/**
+ * BASELINE retrieval for one query.
+ * @param {object|string} query - a queries.json entry (or a bare keyword string).
+ * @returns {{hits: Array<{path:string,line:number,text:string}>, filesTouched: string[], bytesToRead: number}}
+ * hits - every matched {path, line, text} (binary files contribute none).
+ * filesTouched - unique files the agent would have to open to read answers.
+ * bytesToRead - total bytes of those files (context cost; ~chars/4 => tokens).
+ */
+export function scoreQuery(query) {
+ const terms = termsFor(query);
+ const out = runRipgrep(terms);
+
+ const hits = [];
+ const touchedAbs = new Set();
+ for (const raw of out.split('\n')) {
+ if (raw === '') continue;
+ const parsed = parseRgLine(raw);
+ if (!parsed) continue;
+ touchedAbs.add(parsed.absPath);
+ hits.push({ path: prettyPath(parsed.absPath), line: parsed.line, text: parsed.text });
+ }
+
+ let bytesToRead = 0;
+ const filesTouched = [];
+ for (const abs of touchedAbs) {
+ filesTouched.push(prettyPath(abs));
+ try {
+ bytesToRead += statSync(abs).size;
+ } catch {
+ /* file vanished mid-run; skip its bytes rather than crash */
+ }
+ }
+
+ return { hits, filesTouched, bytesToRead };
+}
+
+/**
+ * Load queries.json (if present) so the CLI can resolve a human query/id to its
+ * ground-truth entry. Returns [] if the file is missing/unreadable.
+ */
+function loadQueries() {
+ try {
+ if (!existsSync(QUERIES_PATH)) return [];
+ return JSON.parse(readFileSync(QUERIES_PATH, 'utf8'));
+ } catch (err) {
+ process.stderr.write(`[grep-baseline] could not read queries.json: ${err.message}\n`);
+ return [];
+ }
+}
+
+/** Resolve a CLI argument to a query object: match by id, exact question, or substring. */
+function resolveCliQuery(arg) {
+ const queries = loadQueries();
+ const lc = arg.toLowerCase();
+ const byId = queries.find((q) => q.id && q.id.toLowerCase() === lc);
+ if (byId) return byId;
+ const byQ = queries.find((q) => q.question && q.question.toLowerCase() === lc);
+ if (byQ) return byQ;
+ const byPartial = queries.find((q) => q.question && q.question.toLowerCase().includes(lc));
+ if (byPartial) return byPartial;
+ // Not a known query — treat the raw string as a single literal keyword.
+ return { id: '(adhoc)', question: arg, keywords: [arg] };
+}
+
+function runCli(argv) {
+ const arg = argv.slice(2).join(' ').trim();
+ if (!arg) {
+ process.stderr.write('usage: node grep-baseline.mjs "<query text | query id | keyword>"\n');
+ process.exit(2);
+ }
+ const query = resolveCliQuery(arg);
+ const terms = termsFor(query);
+ const { hits, filesTouched, bytesToRead } = scoreQuery(query);
+
+ const tokensApprox = Math.ceil(bytesToRead / 4);
+ process.stdout.write(`query : ${query.question || arg}\n`);
+ if (query.id) process.stdout.write(`query id : ${query.id}\n`);
+ process.stdout.write(`grep terms : ${terms.map((t) => JSON.stringify(t)).join(', ')}\n`);
+ process.stdout.write(`hits : ${hits.length} line(s) across ${filesTouched.length} file(s)\n`);
+ process.stdout.write(
+ `bytesToRead: ${bytesToRead} bytes to open (~${tokensApprox} tokens to read answers)\n`,
+ );
+ process.stdout.write('---\n');
+ const shown = hits.slice(0, 20);
+ for (const h of shown) {
+ process.stdout.write(`${h.path}:${h.line}: ${h.text}\n`);
+ }
+ if (hits.length > shown.length) {
+ process.stdout.write(`... (${hits.length - shown.length} more)\n`);
+ }
+ if (hits.length === 0) {
+ process.stdout.write(
+ '(no matches — binary PDF/DOCX are invisible to rg, and literal grep finds no differently-phrased answers)\n',
+ );
+ }
+}
+
+// Only run the CLI when invoked directly (not when imported by compare.mjs).
+if (import.meta.url === `file://${process.argv[1]}`) {
+ runCli(process.argv);
+}
diff --git a/demo/usecases/doc-index/results.json b/demo/usecases/doc-index/results.json
new file mode 100644
index 000000000..ecfd33fd7
--- /dev/null
+++ b/demo/usecases/doc-index/results.json
@@ -0,0 +1,712 @@
+{
+ "generated_at": "2026-07-09T05:12:07.563Z",
+ "node": "http://localhost:9209",
+ "index": "docfolder",
+ "index_doc_count": 420,
+ "size": 5,
+ "retrieval_mode": "hybrid",
+ "semantic_subquery_errored": false,
+ "xerj_query_example": {
+ "size": 5,
+ "query": {
+ "hybrid": {
+ "queries": [
+ {
+ "query": {
+ "match": {
+ "body_text": "<question>"
+ }
+ }
+ },
+ {
+ "query": {
+ "semantic": {
+ "field": "body",
+ "query": "<question>",
+ "k": 5
+ }
+ }
+ }
+ ],
+ "fusion": "rrf"
+ }
+ }
+ },
+ "index_build_ms": 1339,
+ "honesty": {
+ "embedder": "XERJ built-in embedder is lexical feature-hashing (384-dim cosine), NOT neural; \"semantic\"/robustness wins are word/sub-word overlap, not deep meaning. On plaintext a fair grep of the question's own terms ties it.",
+ "knn": "kNN is exact brute-force at query time (fine at this corpus size).",
+ "baseline": "Baseline is a FAIR shell-only agent: ripgrep over raw bytes for the union of the curated keywords AND the salient tokens of the question itself. It could shell out to pdftotext, but would still lose on format coverage at scale, ranking, per-chunk retrieval, and whole-file context cost.",
+ "context": "Context efficiency is measured ONLY over queries the baseline answers, charging the baseline ONLY the single answer-containing file it must open (statSync(answer_path).size) — never the false-positive files its broad terms also matched (kept as a labelled diagnostic). It is a real win only on large documents (large_literal ≫ 1×) and INVERTS on tiny plaintext files (literal ≈ 1× or below), which is why BOTH buckets are shown side by side. The naive overall ratio flatters the blind baseline — it now literally opens 0 bytes on every query it cannot answer — and is reported for transparency, not as a claim."
+ },
+ "totals": {
+ "queries": 22,
+ "xerj_hits": 21,
+ "baseline_hits": 14,
+ "xerj_coverage_pct": 95.5,
+ "baseline_coverage_pct": 63.6
+ },
+ "per_match_type": {
+ "binary_only": {
+ "total": 7,
+ "xerj_hits": 6,
+ "baseline_hits": 0,
+ "xerj_only": 6,
+ "xerj_coverage_pct": 85.7,
+ "baseline_coverage_pct": 0
+ },
+ "large_literal": {
+ "total": 4,
+ "xerj_hits": 4,
+ "baseline_hits": 3,
+ "xerj_only": 1,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 75
+ },
+ "robustness": {
+ "total": 5,
+ "xerj_hits": 5,
+ "baseline_hits": 5,
+ "xerj_only": 0,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 100
+ },
+ "literal": {
+ "total": 6,
+ "xerj_hits": 6,
+ "baseline_hits": 6,
+ "xerj_only": 0,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 100
+ }
+ },
+ "per_format": {
+ "docx": {
+ "total": 3,
+ "xerj_hits": 2,
+ "baseline_hits": 0,
+ "xerj_coverage_pct": 66.7,
+ "baseline_coverage_pct": 0
+ },
+ "html": {
+ "total": 5,
+ "xerj_hits": 5,
+ "baseline_hits": 5,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 100
+ },
+ "md": {
+ "total": 7,
+ "xerj_hits": 7,
+ "baseline_hits": 7,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 100
+ },
+ "pdf": {
+ "total": 5,
+ "xerj_hits": 5,
+ "baseline_hits": 0,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 0
+ },
+ "txt": {
+ "total": 2,
+ "xerj_hits": 2,
+ "baseline_hits": 2,
+ "xerj_coverage_pct": 100,
+ "baseline_coverage_pct": 100
+ }
+ },
+ "match_type_hits": {
+ "binary_only": {
+ "total": 7,
+ "xerj": 6,
+ "baseline": 0,
+ "xerj_only_wins": 6
+ },
+ "large_literal": {
+ "total": 4,
+ "xerj": 4,
+ "baseline": 3,
+ "xerj_only_wins": 1
+ },
+ "robustness": {
+ "total": 5,
+ "xerj": 5,
+ "baseline": 5,
+ "xerj_only_wins": 0
+ },
+ "literal": {
+ "total": 6,
+ "xerj": 6,
+ "baseline": 6,
+ "xerj_only_wins": 0
+ }
+ },
+ "context_cost": {
+ "method": "Baseline is charged ONLY the size of the single answer-containing file it must open (statSync(answer_path).size) when it hits, else 0. False-positive term-matched files are NOT charged; their total is kept as a labelled diagnostic (diagnostic_all_matched_bytes). grep -n yields a matching line, but to quote/verify an answer reliably an agent opens the whole answer file — that single file is the charge.",
+ "naive_overall": {
+ "note": "flatters the blind baseline — it now literally opens 0 bytes on every query it cannot answer — NOT a claim",
+ "baseline_total_bytes": 202063,
+ "xerj_total_bytes": 79872,
+ "baseline_tokens_approx": 50516,
+ "xerj_tokens_approx": 19968,
+ "ratio_baseline_over_xerj": 2.53
+ },
+ "answerable": {
+ "queries": 14,
+ "baseline_bytes": 202063,
+ "xerj_returned_bytes": 50750,
+ "baseline_tokens_approx": 50516,
+ "xerj_tokens_approx": 12688,
+ "context_ratio_answerable": 3.98
+ },
+ "large_literal": {
+ "queries": 3,
+ "baseline_bytes": 192782,
+ "xerj_returned_bytes": 11180,
+ "xerj_best_passage_bytes": 2250,
+ "context_ratio_large_literal": 17.24,
+ "context_ratio_large_literal_best_passage": 85.68
+ },
+ "literal": {
+ "queries": 6,
+ "baseline_bytes": 4741,
+ "xerj_returned_bytes": 20965,
+ "xerj_best_passage_bytes": 3930,
+ "context_ratio_literal": 0.23,
+ "context_ratio_literal_best_passage": 1.21
+ },
+ "diagnostic_all_matched_bytes": {
+ "note": "DIAGNOSTIC ONLY — sum over ALL queries of the size of EVERY file the baseline's broad terms matched (false-positive files included). This is the pre-fix inflated charge; NOT used in any ratio or claim.",
+ "baseline_all_matched_bytes_total": 4218746
+ }
+ },
+ "latency": {
+ "xerj_p50_ms": 61.23,
+ "xerj_mean_ms": 64.9,
+ "xerj_max_ms": 116.09
+ },
+ "gate": {
+ "pass": true,
+ "checks": {
+ "overall_coverage_ge_baseline": true,
+ "binary_only_strictly_higher": true,
+ "large_literal_context_ratio_gt_1": true
+ },
+ "robustness_tie_note": "robustness reported honestly (XERJ 5/5 vs baseline 5/5); a lexical embedder ties a fair grep — not gated."
+ },
+ "queries": [
+ {
+ "id": "q01",
+ "question": "How many weeks of paid parental leave do employees receive?",
+ "match_type": "binary_only",
+ "answer_format": "pdf",
+ "answer_path": "corpus/hr/handbook.pdf",
+ "expect_substring": "16 weeks",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 116.09,
+ "xerj_returned_bytes": 3433,
+ "xerj_best_passage_bytes": 737,
+ "answer_file_bytes": 32959,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 197133,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/hr/handbook.pdf",
+ "corpus/hr/benefits/leave-policy.docx",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q02",
+ "question": "How many days of paid time off do full-time employees accrue per year?",
+ "match_type": "binary_only",
+ "answer_format": "pdf",
+ "answer_path": "corpus/hr/handbook.pdf",
+ "expect_substring": "22 days",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 71.74,
+ "xerj_returned_bytes": 3771,
+ "xerj_best_passage_bytes": 737,
+ "answer_file_bytes": 32959,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 197808,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/hr/handbook.pdf",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q03",
+ "question": "How many days of bereavement leave are provided per event?",
+ "match_type": "binary_only",
+ "answer_format": "docx",
+ "answer_path": "corpus/hr/benefits/leave-policy.docx",
+ "expect_substring": "5 working days",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 63.15,
+ "xerj_returned_bytes": 3621,
+ "xerj_best_passage_bytes": 693,
+ "answer_file_bytes": 6096,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 197796,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/hr/benefits/leave-policy.docx",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q04",
+ "question": "After how long does an unacknowledged page escalate to the secondary on-call engineer?",
+ "match_type": "binary_only",
+ "answer_format": "docx",
+ "answer_path": "corpus/engineering/oncall-runbook.docx",
+ "expect_substring": "15 minutes",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 74.96,
+ "xerj_returned_bytes": 3661,
+ "xerj_best_passage_bytes": 733,
+ "answer_file_bytes": 6163,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 198770,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/engineering/oncall-runbook.docx",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q05",
+ "question": "What is the acknowledgement SLA for a Sev-1 incident?",
+ "match_type": "binary_only",
+ "answer_format": "docx",
+ "answer_path": "corpus/engineering/oncall-runbook.docx",
+ "expect_substring": "within 5 minutes",
+ "xerj_hit": false,
+ "baseline_hit": false,
+ "xerj_latency_ms": 61.92,
+ "xerj_returned_bytes": 3703,
+ "xerj_best_passage_bytes": 809,
+ "answer_file_bytes": 6163,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 196317,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/meetings/2044-q1-planning.txt",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q06",
+ "question": "How long must customer transaction records be retained?",
+ "match_type": "binary_only",
+ "answer_format": "pdf",
+ "answer_path": "corpus/security/data-classification.pdf",
+ "expect_substring": "7 years",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 70.71,
+ "xerj_returned_bytes": 3780,
+ "xerj_best_passage_bytes": 710,
+ "answer_file_bytes": 29326,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 197649,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/security/data-classification.pdf",
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "q07",
+ "question": "What is the domestic meal per diem rate?",
+ "match_type": "binary_only",
+ "answer_format": "pdf",
+ "answer_path": "corpus/finance/expense-policy.pdf",
+ "expect_substring": "$75 per day",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 62.48,
+ "xerj_returned_bytes": 3728,
+ "xerj_best_passage_bytes": 693,
+ "answer_file_bytes": 31751,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 196826,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/finance/expense-policy.pdf",
+ "corpus/engineering/architecture/data-pipeline.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "ll01",
+ "question": "How much is the employee referral bonus for a successful hire?",
+ "match_type": "large_literal",
+ "answer_format": "html",
+ "answer_path": "corpus/hr/employee-handbook.html",
+ "expect_substring": "$3,000 per successful hire",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 59.57,
+ "xerj_returned_bytes": 3702,
+ "xerj_best_passage_bytes": 786,
+ "answer_file_bytes": 62204,
+ "baseline_context_bytes": 62204,
+ "baseline_all_matched_bytes": 195057,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/hr/employee-handbook.html",
+ "corpus/meetings/2044-q1-planning.txt",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "ll02",
+ "question": "What is the default value of the ingest max_batch_size parameter?",
+ "match_type": "large_literal",
+ "answer_format": "md",
+ "answer_path": "corpus/engineering/api-reference.md",
+ "expect_substring": "defaults to 4096 records",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.41,
+ "xerj_returned_bytes": 3765,
+ "xerj_best_passage_bytes": 765,
+ "answer_file_bytes": 64837,
+ "baseline_context_bytes": 64837,
+ "baseline_all_matched_bytes": 195245,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "ll03",
+ "question": "How quickly must a public status-page update be posted after declaring a Sev-1?",
+ "match_type": "large_literal",
+ "answer_format": "txt",
+ "answer_path": "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "expect_substring": "within twenty minutes",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.08,
+ "xerj_returned_bytes": 3713,
+ "xerj_best_passage_bytes": 699,
+ "answer_file_bytes": 65741,
+ "baseline_context_bytes": 65741,
+ "baseline_all_matched_bytes": 195532,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/engineering/oncall-runbook.docx",
+ "corpus/engineering/api-reference.md"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "ll04",
+ "question": "Within how many days must critical vulnerabilities be remediated?",
+ "match_type": "large_literal",
+ "answer_format": "pdf",
+ "answer_path": "corpus/security/information-security-policy.pdf",
+ "expect_substring": "within seven days",
+ "xerj_hit": true,
+ "baseline_hit": false,
+ "xerj_latency_ms": 58.56,
+ "xerj_returned_bytes": 3425,
+ "xerj_best_passage_bytes": 673,
+ "answer_file_bytes": 129864,
+ "baseline_context_bytes": 0,
+ "baseline_all_matched_bytes": 196546,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/security/information-security-policy.pdf",
+ "corpus/hr/employee-handbook.html",
+ "corpus/finance/expense-policy.pdf"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "s01",
+ "question": "What is Northwind's policy on remote work versus coming into the office?",
+ "match_type": "robustness",
+ "answer_format": "md",
+ "answer_path": "corpus/operations/facilities.md",
+ "expect_substring": "three days a week",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 62.32,
+ "xerj_returned_bytes": 3834,
+ "xerj_best_passage_bytes": 751,
+ "answer_file_bytes": 770,
+ "baseline_context_bytes": 770,
+ "baseline_all_matched_bytes": 204563,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/hr/employee-handbook.html",
+ "corpus/operations/facilities.md"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "s02",
+ "question": "How does a user recover access to a locked account?",
+ "match_type": "robustness",
+ "answer_format": "html",
+ "answer_path": "corpus/security/access-control.html",
+ "expect_substring": "self-service portal",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 62.99,
+ "xerj_returned_bytes": 3699,
+ "xerj_best_passage_bytes": 661,
+ "answer_file_bytes": 842,
+ "baseline_context_bytes": 842,
+ "baseline_all_matched_bytes": 196355,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/security/access-control.html",
+ "corpus/hr/employee-handbook.html",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/security/information-security-policy.pdf"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "s03",
+ "question": "What developer hardware is issued to a newly hired engineer on their first day?",
+ "match_type": "robustness",
+ "answer_format": "md",
+ "answer_path": "corpus/engineering/onboarding.md",
+ "expect_substring": "16-inch developer workstation",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.06,
+ "xerj_returned_bytes": 3781,
+ "xerj_best_passage_bytes": 738,
+ "answer_file_bytes": 832,
+ "baseline_context_bytes": 832,
+ "baseline_all_matched_bytes": 200772,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/onboarding.md",
+ "corpus/operations/facilities.md",
+ "corpus/hr/employee-handbook.html"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "s04",
+ "question": "Which data store backs the inventory and SKU records?",
+ "match_type": "robustness",
+ "answer_format": "md",
+ "answer_path": "corpus/engineering/architecture/system-overview.md",
+ "expect_substring": "PostgreSQL 15 cluster",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.04,
+ "xerj_returned_bytes": 3476,
+ "xerj_best_passage_bytes": 760,
+ "answer_file_bytes": 1105,
+ "baseline_context_bytes": 1105,
+ "baseline_all_matched_bytes": 195368,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/architecture/system-overview.md"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "s05",
+ "question": "What can a customer do if they want to send a purchased unit back?",
+ "match_type": "robustness",
+ "answer_format": "html",
+ "answer_path": "corpus/product/faq.html",
+ "expect_substring": "full reimbursement",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.88,
+ "xerj_returned_bytes": 3815,
+ "xerj_best_passage_bytes": 781,
+ "answer_file_bytes": 991,
+ "baseline_context_bytes": 991,
+ "baseline_all_matched_bytes": 198030,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/product/faq.html",
+ "corpus/hr/employee-handbook.html",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l01",
+ "question": "What is the default API rate limit?",
+ "match_type": "literal",
+ "answer_format": "md",
+ "answer_path": "corpus/engineering/architecture/data-pipeline.md",
+ "expect_substring": "1000 requests per minute",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 59.43,
+ "xerj_returned_bytes": 3788,
+ "xerj_best_passage_bytes": 743,
+ "answer_file_bytes": 783,
+ "baseline_context_bytes": 783,
+ "baseline_all_matched_bytes": 196826,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/architecture/data-pipeline.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l02",
+ "question": "What port does the ingest service listen on?",
+ "match_type": "literal",
+ "answer_format": "md",
+ "answer_path": "corpus/engineering/architecture/system-overview.md",
+ "expect_substring": "port 8412",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 61.11,
+ "xerj_returned_bytes": 3867,
+ "xerj_best_passage_bytes": 760,
+ "answer_file_bytes": 1105,
+ "baseline_context_bytes": 1105,
+ "baseline_all_matched_bytes": 200764,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/architecture/system-overview.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l03",
+ "question": "When is the weekly engineering sync held?",
+ "match_type": "literal",
+ "answer_format": "md",
+ "answer_path": "corpus/meetings/decision-log.md",
+ "expect_substring": "Wednesday at 10:00",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 60.09,
+ "xerj_returned_bytes": 2991,
+ "xerj_best_passage_bytes": 605,
+ "answer_file_bytes": 643,
+ "baseline_context_bytes": 643,
+ "baseline_all_matched_bytes": 195610,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/meetings/decision-log.md",
+ "corpus/engineering/onboarding.md",
+ "corpus/meetings/2044-q1-planning.txt",
+ "corpus/engineering/runbooks/major-incident-runbook.txt"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l04",
+ "question": "What uptime does the platform SLA guarantee?",
+ "match_type": "literal",
+ "answer_format": "html",
+ "answer_path": "corpus/product/faq.html",
+ "expect_substring": "99.9% uptime",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 58.16,
+ "xerj_returned_bytes": 3623,
+ "xerj_best_passage_bytes": 781,
+ "answer_file_bytes": 991,
+ "baseline_context_bytes": 991,
+ "baseline_all_matched_bytes": 68849,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/product/faq.html",
+ "corpus/support/troubleshooting.md",
+ "corpus/engineering/api-reference.md",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/security/information-security-policy.pdf"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l05",
+ "question": "Who must approve purchases over the standard threshold?",
+ "match_type": "literal",
+ "answer_format": "html",
+ "answer_path": "corpus/finance/procurement.html",
+ "expect_substring": "Director of Operations",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 61.34,
+ "xerj_returned_bytes": 3280,
+ "xerj_best_passage_bytes": 548,
+ "answer_file_bytes": 722,
+ "baseline_context_bytes": 722,
+ "baseline_all_matched_bytes": 198316,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/finance/procurement.html",
+ "corpus/engineering/runbooks/major-incident-runbook.txt",
+ "corpus/security/data-classification.pdf",
+ "corpus/hr/employee-handbook.html",
+ "corpus/legal/contractor-terms.docx"
+ ],
+ "xerj_error": null
+ },
+ {
+ "id": "l06",
+ "question": "What is the Q3 target for warehouse pick accuracy?",
+ "match_type": "literal",
+ "answer_format": "txt",
+ "answer_path": "corpus/product/roadmap.txt",
+ "expect_substring": "99.5% pick accuracy",
+ "xerj_hit": true,
+ "baseline_hit": true,
+ "xerj_latency_ms": 61.68,
+ "xerj_returned_bytes": 3416,
+ "xerj_best_passage_bytes": 493,
+ "answer_file_bytes": 497,
+ "baseline_context_bytes": 497,
+ "baseline_all_matched_bytes": 198614,
+ "xerj_returned": 5,
+ "xerj_cite_paths": [
+ "corpus/meetings/2044-q1-planning.txt",
+ "corpus/engineering/architecture/system-overview.md",
+ "corpus/engineering/api-reference.md",
+ "corpus/product/roadmap.txt"
+ ],
+ "xerj_error": null
+ }
+ ]
+}
\ No newline at end of file
diff --git a/demo/usecases/doc-index/xerj-index.mjs b/demo/usecases/doc-index/xerj-index.mjs
new file mode 100644
index 000000000..84fb81e09
--- /dev/null
+++ b/demo/usecases/doc-index/xerj-index.mjs
@@ -0,0 +1,567 @@
+#!/usr/bin/env node
+// -----------------------------------------------------------------------------
+// xerj-index.mjs — the INDEXER for the "Claude + XERJ, search a recursive folder
+// of mixed-format documents" use case.
+//
+// Walks a folder recursively, extracts text from every supported format
+// (.pdf/.docx/.html/.md/.txt), chunks it (~800 chars / ~100 overlap on
+// sentence/paragraph boundaries), creates the `docfolder` index with the SPEC
+// mappings, and bulk-indexes each chunk into a live XERJ node. Both the semantic
+// `body` field and the lexical `body_text` field are set to the SAME chunk text.
+//
+// Contract: see demo/usecases/doc-index/SPEC.md ("XERJ index contract").
+//
+// No external npm dependencies. Uses:
+// - node:child_process (execFileSync) to shell out to pdftotext / soffice
+// - node:http to talk to XERJ (ES-compatible REST)
+//
+// Target node: http://localhost:${XERJ_PORT:-9209} (XERJ started with --insecure)
+//
+// Usage:
+// node xerj-index.mjs [--dir <folder>] [--recreate] [--batch <n>] [--dry-run]
+//
+// --dir <folder> Folder to walk. Default: ./corpus (next to this script).
+// --recreate If the `docfolder` index exists, DELETE + recreate it so
+// runs are reproducible (no duplicate chunks on re-index).
+// --batch <n> Docs per _bulk request. Default: 200.
+// --dry-run Extract + chunk + print stats only. Do NOT touch XERJ.
+// (Handy for testing extraction without a running node.)
+// --help Show this help.
+//
+// Extraction tools required on PATH for full corpus coverage:
+// pdftotext (poppler-utils) — for .pdf
+// soffice (LibreOffice) — for .docx (falls back to `unzip -p`)
+// .html/.md/.txt are handled in pure Node with no external tools.
+// -----------------------------------------------------------------------------
+
+import { execFileSync } from 'node:child_process';
+import http from 'node:http';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+// ---------------------------------------------------------------------------
+// Config / constants (indexer and compare.mjs MUST agree on these).
+// ---------------------------------------------------------------------------
+const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
+const INDEX = 'docfolder';
+const PORT = parseInt(process.env.XERJ_PORT || '9209', 10);
+const HOST = 'localhost';
+const SUPPORTED = new Set(['.pdf', '.docx', '.html', '.md', '.txt']);
+
+const CHUNK_TARGET = 800; // ~800 chars per chunk
+const CHUNK_OVERLAP = 100; // ~100 chars carried into the next chunk
+
+// ---------------------------------------------------------------------------
+// CLI args
+// ---------------------------------------------------------------------------
+function parseArgs(argv) {
+ const a = {
+ dir: path.join(SCRIPT_DIR, 'corpus'),
+ recreate: false,
+ batch: 200,
+ dryRun: false,
+ help: false,
+ };
+ for (let i = 0; i < argv.length; i++) {
+ const t = argv[i];
+ if (t === '--dir') a.dir = path.resolve(argv[++i]);
+ else if (t === '--recreate') a.recreate = true;
+ else if (t === '--batch') a.batch = Math.max(1, parseInt(argv[++i], 10) || 200);
+ else if (t === '--dry-run' || t === '--dryrun') a.dryRun = true;
+ else if (t === '--help' || t === '-h') a.help = true;
+ else console.warn(`[warn] ignoring unknown arg: ${t}`);
+ }
+ return a;
+}
+
+const HELP = `xerj-index.mjs — walk a folder, extract+chunk docs, bulk-index into XERJ.
+
+Usage:
+ node xerj-index.mjs [--dir <folder>] [--recreate] [--batch <n>] [--dry-run]
+
+ --dir <folder> Folder to walk (recursive). Default: ./corpus
+ --recreate Delete + recreate the '${INDEX}' index for a clean run.
+ --batch <n> Docs per _bulk request (default 200).
+ --dry-run Extract + chunk + print stats only; do not touch XERJ.
+ --help Show this help.
+
+Env:
+ XERJ_PORT XERJ ES-compat port (default 9209).`;
+
+// ---------------------------------------------------------------------------
+// Filesystem walk — recursive, skips symlinks/hidden entries, target ext only.
+// ---------------------------------------------------------------------------
+function walk(dir) {
+ const out = [];
+ let entries;
+ try {
+ entries = fs.readdirSync(dir, { withFileTypes: true });
+ } catch (e) {
+ console.warn(`[warn] cannot read dir ${dir}: ${e.message}`);
+ return out;
+ }
+ for (const ent of entries) {
+ if (ent.name.startsWith('.')) continue; // skip hidden
+ if (ent.isSymbolicLink()) continue; // avoid loops
+ const full = path.join(dir, ent.name);
+ if (ent.isDirectory()) {
+ out.push(...walk(full));
+ } else if (ent.isFile()) {
+ const ext = path.extname(ent.name).toLowerCase();
+ if (SUPPORTED.has(ext)) out.push(full);
+ }
+ }
+ return out;
+}
+
+// ---------------------------------------------------------------------------
+// Extraction — one function per format. Each returns plain text or throws.
+// The caller catches per-file so a single bad file never crashes the run.
+// ---------------------------------------------------------------------------
+const EXEC_OPTS = { encoding: 'utf8', maxBuffer: 128 * 1024 * 1024 };
+
+function extractPdf(file) {
+ // `pdftotext -layout <f> -` → text on stdout (preserves column layout).
+ return execFileSync('pdftotext', ['-layout', file, '-'], EXEC_OPTS);
+}
+
+// One shared LibreOffice user-profile dir per run avoids the per-invocation
+// profile-lock dance; conversions run sequentially so a single profile is safe.
+let SOFFICE_PROFILE = null;
+function sofficeProfileDir() {
+ if (!SOFFICE_PROFILE) {
+ SOFFICE_PROFILE = fs.mkdtempSync(path.join(os.tmpdir(), 'xerj-lo-profile-'));
+ }
+ return SOFFICE_PROFILE;
+}
+
+function extractDocx(file) {
+ // Primary: soffice --headless --convert-to txt into a fresh temp outdir,
+ // then read the produced <base>.txt.
+ const outdir = fs.mkdtempSync(path.join(os.tmpdir(), 'xerj-docx-'));
+ try {
+ execFileSync('soffice', [
+ '--headless',
+ `-env:UserInstallation=file://${sofficeProfileDir()}`,
+ '--convert-to', 'txt:Text',
+ '--outdir', outdir,
+ file,
+ ], { ...EXEC_OPTS, stdio: ['ignore', 'ignore', 'pipe'] });
+ const base = path.basename(file, path.extname(file));
+ const txtPath = path.join(outdir, `${base}.txt`);
+ if (fs.existsSync(txtPath)) {
+ return fs.readFileSync(txtPath, 'utf8');
+ }
+ throw new Error('soffice produced no .txt output');
+ } catch (primaryErr) {
+ // Fallback: unzip word/document.xml out of the .docx and strip its tags.
+ try {
+ const xml = execFileSync('unzip', ['-p', file, 'word/document.xml'], EXEC_OPTS);
+ // <w:p> paragraphs → newlines; <w:tab> → space; then strip all tags.
+ const text = xml
+ .replace(/<\/w:p>/g, '\n')
+ .replace(/<w:tab[^>]*\/>/g, ' ')
+ .replace(/<[^>]+>/g, '')
+ .replace(/&amp;/g, '&').replace(/&lt;/g, '<').replace(/&gt;/g, '>')
+ .replace(/&quot;/g, '"').replace(/&#39;/g, "'");
+ if (text.trim()) return text;
+ throw new Error('empty document.xml after strip');
+ } catch (fallbackErr) {
+ throw new Error(`docx extract failed (soffice: ${primaryErr.message}; unzip fallback: ${fallbackErr.message})`);
+ }
+ } finally {
+ try { fs.rmSync(outdir, { recursive: true, force: true }); } catch { /* ignore */ }
+ }
+}
+
+const HTML_ENTITIES = {
+ '&amp;': '&', '&lt;': '<', '&gt;': '>', '&quot;': '"', '&#39;': "'",
+ '&apos;': "'", '&nbsp;': ' ', '&mdash;': '—', '&ndash;': '–', '&hellip;': '…',
+};
+function decodeEntities(s) {
+ return s
+ .replace(/&#(\d+);/g, (_, n) => String.fromCodePoint(parseInt(n, 10)))
+ .replace(/&#x([0-9a-fA-F]+);/g, (_, n) => String.fromCodePoint(parseInt(n, 16)))
+ .replace(/&[a-zA-Z]+;|&#39;/g, (m) => HTML_ENTITIES[m] ?? m);
+}
+function extractHtml(file) {
+ const raw = fs.readFileSync(file, 'utf8');
+ const text = raw
+ .replace(/<!--[\s\S]*?-->/g, ' ') // comments
+ .replace(/<script[\s\S]*?<\/script>/gi, ' ') // drop scripts
+ .replace(/<style[\s\S]*?<\/style>/gi, ' ') // drop styles
+ .replace(/<(?:br|p|div|li|tr|h[1-6])[^>]*>/gi, '\n') // block tags → newline
+ .replace(/<[^>]+>/g, ' ') // strip remaining tags
+ .replace(/[ \t]+/g, ' ');
+ return decodeEntities(text);
+}
+
+function extractMd(file) {
+ // Light markdown de-syntaxing: keep the words, drop the punctuation noise.
+ const raw = fs.readFileSync(file, 'utf8');
+ return raw
+ .replace(/```[\s\S]*?```/g, (m) => m.replace(/```/g, ' ')) // keep fenced text, drop fences
+ .replace(/^#{1,6}\s+/gm, '') // heading markers
+ .replace(/^\s{0,3}[-*+]\s+/gm, '') // bullet markers
+ .replace(/^\s{0,3}\d+\.\s+/gm, '') // ordered-list markers
+ .replace(/^\s{0,3}>\s?/gm, '') // blockquote markers
+ .replace(/!\[([^\]]*)\]\([^)]*\)/g, '$1') // images → alt text
+ .replace(/\[([^\]]+)\]\([^)]*\)/g, '$1') // links → link text
+ .replace(/[*_]{1,3}([^*_]+)[*_]{1,3}/g, '$1')// bold/italic
+ .replace(/`([^`]+)`/g, '$1') // inline code
+ .replace(/^\s*[-*_]{3,}\s*$/gm, ''); // horizontal rules
+}
+
+function extractTxt(file) {
+ return fs.readFileSync(file, 'utf8');
+}
+
+function extractByFormat(file, format) {
+ switch (format) {
+ case 'pdf': return extractPdf(file);
+ case 'docx': return extractDocx(file);
+ case 'html': return extractHtml(file);
+ case 'md': return extractMd(file);
+ case 'txt': return extractTxt(file);
+ default: throw new Error(`unsupported format: ${format}`);
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Title derivation — first markdown/HTML heading, else prettified filename.
+// ---------------------------------------------------------------------------
+function deriveTitle(file, format, rawText) {
+ if (format === 'md') {
+ const m = rawText.match(/^\s{0,3}#{1,6}\s+(.+?)\s*$/m);
+ if (m) return m[1].trim();
+ }
+ const base = path.basename(file, path.extname(file));
+ return base.replace(/[-_]+/g, ' ').replace(/\s+/g, ' ').trim();
+}
+
+// ---------------------------------------------------------------------------
+// Chunking — ~800 chars, ~100 char overlap, on sentence/paragraph boundaries.
+// ---------------------------------------------------------------------------
+function splitSentences(paragraph) {
+ // Break after ., !, ? when followed by whitespace. Coarse but robust.
+ const parts = paragraph.split(/(?<=[.!?])\s+(?=[^\s])/);
+ return parts.map((s) => s.trim()).filter(Boolean);
+}
+
+function chunkText(text, target = CHUNK_TARGET, overlap = CHUNK_OVERLAP) {
+ // Normalise: split into paragraphs on blank lines, collapse inner whitespace.
+ const paragraphs = text
+ .split(/\n\s*\n+/)
+ .map((p) => p.replace(/\s+/g, ' ').trim())
+ .filter(Boolean);
+
+ // Flatten into sentence-level units (paragraph boundaries already respected
+ // because we never merge across paragraphs without going through a sentence).
+ const units = [];
+ for (const p of paragraphs) units.push(...splitSentences(p));
+
+ const chunks = [];
+ let cur = [];
+ let curLen = 0;
+
+ const flush = () => {
+ if (cur.length) {
+ const c = cur.join(' ').trim();
+ if (c) chunks.push(c);
+ }
+ };
+
+ for (const unit of units) {
+ if (unit.length > target) {
+ // A single oversized sentence: flush, then hard-split it with overlap.
+ flush(); cur = []; curLen = 0;
+ const step = Math.max(1, target - overlap);
+ for (let i = 0; i < unit.length; i += step) {
+ chunks.push(unit.slice(i, i + target).trim());
+ }
+ continue;
+ }
+ if (cur.length && curLen + unit.length + 1 > target) {
+ // Emit current chunk, then seed the next one with trailing sentences that
+ // sum to ~overlap chars (sentence-boundary overlap, not a raw char cut).
+ flush();
+ const carry = [];
+ let cl = 0;
+ for (let i = cur.length - 1; i >= 0; i--) {
+ if (cl >= overlap && carry.length) break;
+ carry.unshift(cur[i]);
+ cl += cur[i].length + 1;
+ }
+ cur = carry;
+ curLen = cur.reduce((a, s) => a + s.length + 1, 0);
+ }
+ cur.push(unit);
+ curLen += unit.length + 1;
+ }
+ flush();
+ return chunks.filter((c) => c.trim().length > 0);
+}
+
+// ---------------------------------------------------------------------------
+// XERJ HTTP helpers (node:http; ES-compatible REST on localhost).
+// ---------------------------------------------------------------------------
+function xerjRequest(method, reqPath, body, contentType = 'application/json') {
+ return new Promise((resolve, reject) => {
+ const payload = body == null
+ ? null
+ : (typeof body === 'string' ? body : JSON.stringify(body));
+ const req = http.request(
+ {
+ host: HOST,
+ port: PORT,
+ method,
+ path: reqPath,
+ headers: payload == null ? {} : {
+ 'Content-Type': contentType,
+ 'Content-Length': Buffer.byteLength(payload),
+ },
+ },
+ (res) => {
+ let data = '';
+ res.setEncoding('utf8');
+ res.on('data', (d) => { data += d; });
+ res.on('end', () => resolve({ status: res.statusCode, body: data }));
+ },
+ );
+ req.on('error', reject);
+ if (payload != null) req.write(payload);
+ req.end();
+ });
+}
+
+async function xerjJson(method, reqPath, body) {
+ const { status, body: text } = await xerjRequest(method, reqPath, body);
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { /* leave null */ }
+ return { status, json, text };
+}
+
+const MAPPINGS = {
+ mappings: {
+ properties: {
+ path: { type: 'keyword' },
+ dir: { type: 'keyword' },
+ format: { type: 'keyword' },
+ title: { type: 'text' },
+ chunk_id: { type: 'integer' },
+ body: { type: 'semantic_text' }, // auto-embed (built-in lexical embedder)
+ body_text: { type: 'text' }, // BM25 lexical / match / match_phrase
+ },
+ },
+};
+
+async function ensureIndex(recreate) {
+ const head = await xerjRequest('HEAD', `/${INDEX}`);
+ const exists = head.status === 200;
+ if (exists && recreate) {
+ console.log(`[index] '${INDEX}' exists → deleting (--recreate)`);
+ const del = await xerjJson('DELETE', `/${INDEX}`);
+ if (del.status >= 300) throw new Error(`delete ${INDEX} failed: ${del.status} ${del.text}`);
+ } else if (exists) {
+ console.log(`[index] '${INDEX}' exists → appending (pass --recreate for a clean run)`);
+ return; // keep existing mappings
+ }
+ const create = await xerjJson('PUT', `/${INDEX}`, MAPPINGS);
+ if (create.status >= 300) {
+ throw new Error(`create ${INDEX} failed: ${create.status} ${create.text}`);
+ }
+ console.log(`[index] created '${INDEX}' with SPEC mappings`);
+}
+
+// Send one batch of docs via POST /docfolder/_bulk (NDJSON). Returns
+// { indexed, errored } counts and logs per-item errors.
+async function bulkIndex(docs) {
+ const lines = [];
+ for (const doc of docs) {
+ lines.push(JSON.stringify({ index: { _index: INDEX } }));
+ lines.push(JSON.stringify(doc));
+ }
+ const ndjson = lines.join('\n') + '\n';
+ const { status, body } = await xerjRequest(
+ 'POST', `/${INDEX}/_bulk`, ndjson, 'application/x-ndjson',
+ );
+ if (status >= 300) {
+ console.warn(`[bulk] HTTP ${status} for ${docs.length}-doc batch — skipped`);
+ return { indexed: 0, errored: docs.length };
+ }
+ let resp = null;
+ try { resp = JSON.parse(body); } catch { /* ignore */ }
+ if (!resp || !Array.isArray(resp.items)) {
+ console.warn('[bulk] unparseable response — counting batch as errored');
+ return { indexed: 0, errored: docs.length };
+ }
+ let indexed = 0; let errored = 0;
+ for (const item of resp.items) {
+ const op = item.index || item.create || {};
+ if (op.error) {
+ errored++;
+ if (errored <= 3) console.warn(`[bulk] item error: ${JSON.stringify(op.error)}`);
+ } else {
+ indexed++;
+ }
+ }
+ return { indexed, errored };
+}
+
+// ---------------------------------------------------------------------------
+// Main
+// ---------------------------------------------------------------------------
+async function main() {
+ const args = parseArgs(process.argv.slice(2));
+ if (args.help) { console.log(HELP); return; }
+
+ const t0 = Date.now();
+ console.log(`[start] dir=${args.dir} index=${INDEX} node=http://${HOST}:${PORT} dryRun=${args.dryRun}`);
+
+ const files = walk(args.dir);
+ if (files.length === 0) {
+ console.warn(`[warn] no supported files found under ${args.dir}`);
+ }
+
+ // Reproducible order.
+ files.sort();
+
+ if (!args.dryRun) {
+ await ensureIndex(args.recreate);
+ }
+
+ const perFormatFiles = {}; // format → files seen
+ const perFormatChunks = {}; // format → chunks produced
+ const skipped = [];
+ let totalChunks = 0;
+ let totalIndexed = 0;
+ let totalErrored = 0;
+
+ let batch = [];
+ const flushBatch = async () => {
+ if (args.dryRun || batch.length === 0) { batch = []; return; }
+ const { indexed, errored } = await bulkIndex(batch);
+ totalIndexed += indexed;
+ totalErrored += errored;
+ batch = [];
+ };
+
+ for (const file of files) {
+ const format = path.extname(file).slice(1).toLowerCase();
+ perFormatFiles[format] = (perFormatFiles[format] || 0) + 1;
+
+ // Repo-relative path when the file lives under the deliverable dir, so the
+ // stored `path` matches queries.json's answer_path (e.g. corpus/hr/x.pdf).
+ let relPath = path.relative(SCRIPT_DIR, file);
+ if (relPath.startsWith('..')) relPath = file; // outside → keep absolute
+ const relDir = path.dirname(relPath);
+
+ let rawText;
+ try {
+ rawText = extractByFormat(file, format);
+ } catch (e) {
+ console.warn(`[skip] ${relPath}: ${e.message}`);
+ skipped.push({ path: relPath, reason: e.message });
+ continue;
+ }
+ if (!rawText || !rawText.trim()) {
+ console.warn(`[skip] ${relPath}: no text extracted`);
+ skipped.push({ path: relPath, reason: 'empty extraction' });
+ continue;
+ }
+
+ const title = deriveTitle(file, format, rawText);
+ const chunks = chunkText(rawText);
+ perFormatChunks[format] = (perFormatChunks[format] || 0) + chunks.length;
+ totalChunks += chunks.length;
+
+ console.log(`[ok] ${relPath} (${format}) → ${chunks.length} chunks`);
+
+ for (let ci = 0; ci < chunks.length; ci++) {
+ const text = chunks[ci];
+ const doc = {
+ path: relPath,
+ dir: relDir,
+ format,
+ title,
+ chunk_id: ci,
+ body: text, // semantic_text: auto-embedded on ingest
+ body_text: text, // same text for BM25 / match / match_phrase
+ };
+ batch.push(doc);
+ if (batch.length >= args.batch) await flushBatch();
+ }
+ }
+ await flushBatch();
+
+ // Refresh once so freshly-indexed chunks are searchable for the compare phase.
+ if (!args.dryRun && totalIndexed > 0) {
+ await xerjRequest('POST', `/${INDEX}/_refresh`);
+ }
+
+ const elapsedMs = Date.now() - t0;
+
+ // -------------------------------------------------------------------------
+ // Summary
+ // -------------------------------------------------------------------------
+ console.log('\n===== xerj-index summary =====');
+ console.log(`files walked : ${files.length}`);
+ console.log('per-format files :', JSON.stringify(perFormatFiles));
+ console.log('per-format chunks :', JSON.stringify(perFormatChunks));
+ console.log(`chunks produced : ${totalChunks}`);
+ if (!args.dryRun) {
+ console.log(`chunks indexed : ${totalIndexed}`);
+ console.log(`chunks errored : ${totalErrored}`);
+ } else {
+ console.log('chunks indexed : (dry-run — XERJ not contacted)');
+ }
+ console.log(`files skipped : ${skipped.length}`);
+ if (skipped.length) {
+ for (const s of skipped) console.log(` - ${s.path}: ${s.reason}`);
+ }
+ console.log(`elapsed ms : ${elapsedMs}`);
+
+ // Machine-readable one-liner for downstream tooling / compare.mjs.
+ const summary = {
+ index: INDEX,
+ dir: args.dir,
+ dryRun: args.dryRun,
+ filesWalked: files.length,
+ perFormatFiles,
+ perFormatChunks,
+ chunksProduced: totalChunks,
+ chunksIndexed: args.dryRun ? null : totalIndexed,
+ chunksErrored: args.dryRun ? null : totalErrored,
+ filesSkipped: skipped.length,
+ elapsedMs,
+ };
+ console.log('SUMMARY_JSON ' + JSON.stringify(summary));
+
+ // Clean up the shared soffice profile dir.
+ if (SOFFICE_PROFILE) {
+ try { fs.rmSync(SOFFICE_PROFILE, { recursive: true, force: true }); } catch { /* ignore */ }
+ }
+
+ // Non-zero exit if a live run indexed nothing (something is wrong).
+ if (!args.dryRun && files.length > 0 && totalIndexed === 0) {
+ process.exitCode = 1;
+ }
+}
+
+// Run only when invoked directly (`node xerj-index.mjs ...`); when imported by
+// tooling/tests the extraction + chunking helpers below are reused instead.
+const invokedDirectly = process.argv[1]
+ && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url);
+if (invokedDirectly) {
+ main().catch((e) => {
+ console.error(`[fatal] ${e.stack || e.message}`);
+ process.exitCode = 1;
+ });
+}
+
+export {
+ walk, extractByFormat, extractPdf, extractDocx, extractHtml, extractMd, extractTxt,
+ chunkText, splitSentences, deriveTitle, MAPPINGS, INDEX,
+};
diff --git a/landing/docs/recipes/document-folder-index.html b/landing/docs/recipes/document-folder-index.html
new file mode 100644
index 000000000..82326242e
--- /dev/null
+++ b/landing/docs/recipes/document-folder-index.html
@@ -0,0 +1,445 @@
+<!DOCTYPE html>
+<html lang="en" data-theme="night">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>XERJ · Docs · Recipe · Index a folder of PDFs, Word docs &amp; web pages</title>
+<meta name="description" content="Point an AI agent at a recursive folder of mixed-format documents (PDF, DOCX, HTML, Markdown, TXT). One xerj-index pass extracts, chunks and auto-embeds every file, then lexical, semantic and hybrid search return ranked, cited passages — where plain grep is structurally blind to PDF/DOCX bytes and must load whole large files into context.">
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
+<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>">
+<link rel="stylesheet" href="/style.css?v=90f643e">
+</head>
+<body>
+
+
+<nav class="nav" aria-label="Primary">
+ <a href="/index.html" class="brand">XERJ.AI</a>
+ <a href="/index.html">HOME</a>
+ <a href="/product.html">PRODUCT</a>
+ <span class="nav-drop">
+ <a href="/use-cases.html">USE CASES</a>
+ </span>
+ <span class="spacer"></span>
+ <a href="/for-agents.html">FOR AGENTS</a>
+ <a href="/docs/index.html">DOCS</a>
+ <a href="/docs/recipes/index.html" class="active">RECIPES</a>
+ <a href="/playground.html">DASHBOARDS</a>
+ <a href="https://github.com/xerj-org/xerj" rel="external">GITHUB</a>
+ <span class="spacer"></span>
+ <span class="theme">
+ <button type="button" data-theme="day">DAY</button>
+ <span class="dash">·</span>
+ <button type="button" data-theme="night" class="active">NIGHT</button>
+ </span>
+</nav>
+
+<div class="docs-search" id="docsSearch">
+ <input type="text" id="docsSearchInput" placeholder="Search docs · keys, flags, endpoints, playbooks…" aria-label="Search docs" autocomplete="off" spellcheck="false">
+ <span class="hint">⌘ K</span>
+ <div class="results" id="docsSearchResults" aria-live="polite"></div>
+</div>
+
+<div class="docs-shell">
+<nav class="docs-nav" aria-label="Docs"> <div class="group">
+ <span class="label">START</span>
+ <a href="/docs/index.html" class=""><span class="eyebrow">01</span>Overview</a>
+ <a href="/docs/quickstart.html" class=""><span class="eyebrow">02</span>Quickstart</a>
+ <a href="/docs/install.html" class=""><span class="eyebrow">03</span>Install</a>
+ <a href="/docs/migration-from-es.html" class=""><span class="eyebrow">04</span>Migrate from ES</a>
+ </div>
+ <div class="group">
+ <span class="label">REFERENCE</span>
+ <a href="/docs/cli.html" class=""><span class="eyebrow">01</span>CLI</a>
+ <a href="/docs/config.html" class=""><span class="eyebrow">02</span>Config TOML</a>
+ <a href="/docs/env.html" class=""><span class="eyebrow">03</span>Environment</a>
+ <a href="/docs/api-native.html" class=""><span class="eyebrow">04</span>Native REST API</a>
+ <a href="/docs/api-es-compat.html" class=""><span class="eyebrow">05</span>ES-compat API</a>
+ </div>
+ <div class="group">
+ <span class="label">DATA MODEL</span>
+ <a href="/docs/queries.html" class=""><span class="eyebrow">01</span>Query types</a>
+ <a href="/docs/aggregations.html" class=""><span class="eyebrow">02</span>Aggregations</a>
+ <a href="/docs/analyzers.html" class=""><span class="eyebrow">03</span>Analyzers</a>
+ <a href="/docs/vectors.html" class=""><span class="eyebrow">04</span>Vectors & HNSW</a>
+ </div>
+ <div class="group">
+ <span class="label">ENGINE</span>
+ <a href="/docs/ingest.html" class=""><span class="eyebrow">01</span>Ingest pipelines</a>
+ <a href="/docs/storage.html" class=""><span class="eyebrow">02</span>Storage & WAL</a>
+ <a href="/docs/compression.html" class=""><span class="eyebrow">03</span>Compression</a>
+ <a href="/docs/clustering.html" class=""><span class="eyebrow">04</span>Clustering</a>
+ <a href="/docs/metrics.html" class=""><span class="eyebrow">05</span>Metrics</a>
+ <a href="/docs/security.html" class=""><span class="eyebrow">06</span>Auth & TLS</a>
+ </div>
+ <div class="group">
+ <span class="label">OPERATIONS</span>
+ <a href="/docs/operations.html" class=""><span class="eyebrow">01</span>Running in production</a>
+ <a href="/docs/backup-restore.html" class=""><span class="eyebrow">02</span>Backup & restore</a>
+ <a href="/docs/upgrades.html" class=""><span class="eyebrow">03</span>Upgrades</a>
+ <a href="/docs/troubleshooting.html" class=""><span class="eyebrow">04</span>Troubleshooting</a>
+ </div>
+ <div class="group">
+ <span class="label">PLAYBOOKS</span>
+ <a href="/docs/playbooks/siem.html" class=""><span class="eyebrow">01</span>SIEM</a>
+ <a href="/docs/playbooks/log-analytics.html" class=""><span class="eyebrow">02</span>Log analytics</a>
+ <a href="/docs/playbooks/vector-search.html" class=""><span class="eyebrow">03</span>Vector · RAG</a>
+ <a href="/docs/playbooks/full-text.html" class=""><span class="eyebrow">04</span>Full-text search</a>
+ <a href="/docs/playbooks/observability.html" class=""><span class="eyebrow">05</span>Observability</a>
+ </div>
+ <div class="group">
+ <span class="label">RECIPES</span>
+ <a href="/docs/recipes/index.html" class=""><span class="eyebrow">00</span>All recipes</a>
+ <a href="/docs/recipes/semantic-search-rag.html" class=""><span class="eyebrow">01</span>Semantic search &amp; RAG</a>
+ <a href="/docs/recipes/agentic-memory.html" class=""><span class="eyebrow">02</span>Agent memory</a>
+ <a href="/docs/recipes/passage-retrieval.html" class=""><span class="eyebrow">03</span>Passage retrieval</a>
+ <a href="/docs/recipes/vector-search-knn.html" class=""><span class="eyebrow">04</span>Vector search &middot; kNN</a>
+ <a href="/docs/recipes/vector-quantization.html" class=""><span class="eyebrow">05</span>Vector quantization</a>
+ <a href="/docs/recipes/hybrid-search.html" class=""><span class="eyebrow">06</span>Hybrid search</a>
+ <a href="/docs/recipes/log-analytics.html" class=""><span class="eyebrow">07</span>Log analytics</a>
+ <a href="/docs/recipes/anomaly-detection.html" class=""><span class="eyebrow">08</span>Anomaly detection</a>
+ <a href="/docs/recipes/continuous-anomaly-datafeeds.html" class=""><span class="eyebrow">09</span>Anomaly datafeeds</a>
+ <a href="/docs/recipes/migrate-from-elasticsearch.html" class=""><span class="eyebrow">10</span>Migrate from ES</a>
+ <a href="/docs/recipes/document-folder-index.html" class="active"><span class="eyebrow">11</span>Doc folder index</a>
+ </div>
+</nav>
+
+<main class="docs-content">
+ <!-- Print-only branded header (hidden on screen, visible in PDF) -->
+ <div class="print-header" style="display:none;">XERJ.AI</div>
+
+ <div class="breadcrumb"><a href="/docs/index.html">DOCS</a><span class="sep">·</span><a href="/docs/recipes/index.html">RECIPES</a><span class="sep">·</span>INDEX A FOLDER OF DOCS
+ <button class="pdf-btn" onclick="window.print()" title="Save this page as PDF">
+ <svg viewBox="0 0 24 24"><path d="M12 3v12M12 15l-4-4M12 15l4-4"/><path d="M4 17v2a2 2 0 002 2h12a2 2 0 002-2v-2"/></svg>
+ PDF
+ </button>
+ </div>
+ <div class="kicker"><span>RECIPE · 11</span></div>
+ <h1>Index a folder of PDFs, Word docs &amp; web pages</h1>
+
+
+ <p><strong>Goal:</strong> point an AI agent at a real, messy shared drive — an HR handbook in <strong>PDF</strong>, an on-call runbook in <strong>Word</strong>, a product FAQ in <strong>HTML</strong>, architecture notes in <strong>Markdown</strong>, meeting notes in <strong>plain text</strong> — and let it answer questions with <em>cited passages</em>, regardless of format.</p>
+
+ <p>The obvious shell-only approach is to <code class="inline">grep</code>/<code class="inline">rg</code> the files. It hits a hard ceiling almost immediately: it <strong>cannot read a PDF or a <code class="inline">.docx</code></strong> (they are compressed binary containers, so the answer is simply invisible), and to actually quote an answer buried in a large document it must slurp the <strong>whole file</strong> into the model's context window. (A fair grep of the question's own terms <em>does</em> find differently-phrased plaintext answers — so we do not claim a meaning-understanding edge; see the honest scorecard below.)</p>
+
+ <p>With XERJ, one <code class="inline">xerj-index</code> pass walks the tree, extracts text from <em>every</em> format, chunks it, auto-embeds each chunk with the built-in embedder, and bulk-indexes into a single <code class="inline">docfolder</code> index. The agent then searches it three ways — <strong>lexical</strong>, <strong>semantic</strong>, and <strong>hybrid</strong> — and gets back ranked, cited passages.</p>
+
+ <blockquote>
+ <p><strong>Everything on this page was measured live.</strong> The numbers come from a real run of the indexer + comparison harness in <a href="https://github.com/xerj-org/xerj/tree/main/demo/usecases/doc-index"><code class="inline">demo/usecases/doc-index</code></a> against a live XERJ node — a fictional-company corpus ("Northwind Robotics") of <strong>29 files</strong> across PDF/DOCX/HTML/MD/TXT (including four realistically large ≥60 KB documents), indexed into <strong>420 chunks</strong>, scored over a <strong>22-question</strong> ground-truth set. Nothing here is hand-tuned or aspirational; the raw <code class="inline">results.json</code> and <code class="inline">SCORECARD.md</code> ship next to the scripts so you can reproduce it. The grep baseline is run fairly — it searches the union of each query's keywords <em>and</em> the salient tokens of the question itself.</p>
+ </blockquote>
+
+ <hr>
+
+ <h2>The problem, made concrete: grep can't read a PDF</h2>
+ <p>The HR handbook is a real PDF. Ask it "how many weeks of paid parental leave do employees receive?" and grep the salient terms — <code class="inline">rg</code> returns <strong>nothing</strong>, because the file is a FlateDecode-compressed binary stream, not text:</p>
+<pre class="code">$ file corpus/hr/handbook.pdf
+corpus/hr/handbook.pdf: PDF document, version 1.7, 5 page(s)
+
+$ head -c 96 corpus/hr/handbook.pdf | cat -v
+%PDF-1.7
+%M-CM-$M-CM-&lt;M-CM-6M-CM-^_
+2 0 obj
+&lt;&lt;/Length 3 0 R/Filter/FlateDecode&gt;&gt;
+stream
+x^]XMM-KM-dF... # compressed bytes — no readable words
+
+$ rg -i -F -e 'parental leave' -e 'paid leave' corpus/
+ # (exit 1, no matches — rg detects the NUL bytes and skips the file)</pre>
+
+ <p>The answer <em>is</em> in there — it is just locked inside the binary. Run a proper extractor and you can see it:</p>
+<pre class="code">$ pdftotext -layout corpus/hr/handbook.pdf - | grep -i 'parental leave'
+Parental leave
+Paid parental leave is 16 weeks for eligible full-time staff.</pre>
+
+ <p>That extraction step is exactly what <code class="inline">xerj-index</code> does for every file — and then it chunks, embeds, and indexes the result so you can search it. A naive grep agent is structurally blind to it.</p>
+
+ <hr>
+
+ <h2>1. Create the <code class="inline">docfolder</code> index</h2>
+ <p>Two fields carry the document text. <code class="inline">body</code> is mapped as <code class="inline">semantic_text</code> — XERJ auto-embeds it on ingest for meaning-aware retrieval — and <code class="inline">body_text</code> is a plain <code class="inline">text</code> field holding the <em>same</em> chunk for BM25 lexical / <code class="inline">match</code>. The rest are cheap keyword/metadata fields so you can filter by folder or format and cite the source path.</p>
+<pre class="code">curl -X PUT localhost:9209/docfolder -H 'Content-Type: application/json' -d '{
+ "mappings": {
+ "properties": {
+ "path": { "type": "keyword" },
+ "dir": { "type": "keyword" },
+ "format": { "type": "keyword" },
+ "title": { "type": "text" },
+ "chunk_id": { "type": "integer" },
+ "body": { "type": "semantic_text" },
+ "body_text": { "type": "text" }
+ }
+ }
+}'</pre>
+
+ <blockquote>
+ <p><strong>Port note.</strong> This walkthrough talks to a node on <code class="inline">:9209</code> — the port the demo harness uses so it never collides with anything else on the box. XERJ's default ES-compat port is <code class="inline">:9200</code>; use whichever you started the node on (the scripts honour <code class="inline">XERJ_PORT</code>).</p>
+ </blockquote>
+
+ <h2>2. Index the folder in one pass</h2>
+ <p>The <code class="inline">xerj-index.mjs</code> indexer (pure Node, no npm dependencies) walks the folder recursively, extracts text per format, chunks it to ~800 characters with ~100 characters of sentence-boundary overlap, and bulk-indexes each chunk. Extraction uses the right tool for each format:</p>
+ <ul>
+ <li><code class="inline">.pdf</code> → <code class="inline">pdftotext -layout</code> (poppler-utils)</li>
+ <li><code class="inline">.docx</code> → <code class="inline">soffice --headless --convert-to txt</code> (LibreOffice), falling back to unzipping <code class="inline">word/document.xml</code> and stripping tags</li>
+ <li><code class="inline">.html</code> → strip <code class="inline">&lt;script&gt;</code>/<code class="inline">&lt;style&gt;</code>/tags to text (pure Node)</li>
+ <li><code class="inline">.md</code> / <code class="inline">.txt</code> → read as text (light Markdown de-syntaxing)</li>
+ </ul>
+<pre class="code">$ XERJ_PORT=9209 node xerj-index.mjs --recreate
+[index] created 'docfolder' with SPEC mappings
+[ok] corpus/hr/handbook.pdf (pdf) → 4 chunks
+[ok] corpus/engineering/oncall-runbook.docx (docx) → 2 chunks
+[ok] corpus/product/faq.html (html) → 1 chunks
+...
+===== xerj-index summary =====
+files walked : 29
+chunks produced : 420
+chunks indexed : 420
+chunks errored : 0
+files skipped : 0
+elapsed ms : 1339</pre>
+
+ <p>Measured result: <strong>29 files → 420 chunks, 0 errored, 0 skipped, in 1339 ms</strong>. Confirm the index is populated:</p>
+<pre class="code">$ curl -s localhost:9209/docfolder/_count
+{"count":420,"_shards":{"total":1,"successful":1,"failed":0}}</pre>
+
+ <p>A single file lands as several rows (one per chunk), each with its own <code class="inline">path</code>, <code class="inline">format</code>, <code class="inline">chunk_id</code>, and an auto-embedded <code class="inline">body</code>. That is what makes retrieval <em>passage-level</em>: you get the paragraph that answers the question, not the whole 5-page PDF.</p>
+
+ <h2>3. Search it three ways</h2>
+
+ <h3>Lexical — <code class="inline">match</code> on <code class="inline">body_text</code></h3>
+ <p>Plain BM25 over the extracted text. Good when the query words are literally in the doc:</p>
+<pre class="code">curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
+ "size": 5,
+ "query": { "match": { "body_text": "default API rate limit" } }
+}'</pre>
+<pre class="code">{
+ "hits": { "hits": [
+ { "_index": "docfolder", "_source": {
+ "path": "corpus/engineering/architecture/data-pipeline.md",
+ "format": "md", "title": "Data Pipeline Notes",
+ "body_text": "... The default API rate limit is 1000 requests per minute per client key. ..."
+ } }
+ ] }
+}
+// _score + body_vector trimmed for brevity — top hit shown</pre>
+
+ <h3>Semantic — <code class="inline">semantic</code> on <code class="inline">body</code></h3>
+ <p>The user rarely uses the doc's exact words. Ask "what is Northwind's policy on remote work versus coming into the office?" — the facilities note never says "remote work," it says staff "may work from home up to three days a week." The <code class="inline">semantic</code> query retrieves it by embedding (word/sub-word) overlap in one shot, without your having to guess synonyms. Be honest about the scope, though: a diligent grep of the question's <em>own</em> salient terms ("work", "office") lands the same line — on differently-phrased plaintext the two approaches <strong>tie</strong> (see the robustness row below). The convenience is single-query retrieval, not a capability grep lacks:</p>
+<pre class="code">curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
+ "size": 5,
+ "query": { "semantic": { "field": "body",
+ "query": "policy on remote work versus coming into the office", "k": 5 } }
+}'</pre>
+<pre class="code">{
+ "hits": { "hits": [
+ { "_index": "docfolder", "_source": {
+ "path": "corpus/operations/facilities.md",
+ "format": "md", "title": "Facilities and Ways of Working",
+ "body_text": "... Northwind runs a distributed-first culture, and staff may work
+ from home up to three days a week. Teams come into the office on the
+ remaining days for planning, workshops, and hands-on time ..."
+ } }
+ ] }
+}</pre>
+
+ <h3>Hybrid — fuse both with RRF</h3>
+ <p>The default the harness scores with: fan out a <code class="inline">match</code> on <code class="inline">body_text</code> and a <code class="inline">semantic</code> query on <code class="inline">body</code>, then fuse the two rankings with Reciprocal Rank Fusion. This is what answers the parental-leave question that grep could not touch:</p>
+<pre class="code">curl -s localhost:9209/docfolder/_search -H 'Content-Type: application/json' -d '{
+ "size": 5,
+ "query": {
+ "hybrid": {
+ "queries": [
+ { "query": { "match": { "body_text": "weeks of paid parental leave" } } },
+ { "query": { "semantic": { "field": "body",
+ "query": "weeks of paid parental leave", "k": 5 } } }
+ ],
+ "fusion": "rrf"
+ }
+ }
+}'</pre>
+ <p>Measured response — the answer chunk from the binary PDF ranks #1, and every hit carries its source path so the agent can cite it (query latency for this one was <strong>116.09 ms</strong>):</p>
+<pre class="code">{
+ "hits": { "hits": [
+ { "_index": "docfolder", "_source": {
+ "path": "corpus/hr/handbook.pdf", "format": "pdf", "title": "handbook",
+ "body_text": "... Parental leave. Paid parental leave is 16 weeks for eligible
+ full-time staff. Parental leave is available to all full-time
+ employees after one year of continuous service ..."
+ } },
+ { "_source": { "path": "corpus/hr/benefits/leave-policy.docx", "format": "docx" } },
+ { "_source": { "path": "corpus/hr/employee-handbook.html", "format": "html" } }
+ ] }
+}
+// 5 chunks returned across 3 files (top 3 shown); RRF _score + body_vector trimmed</pre>
+
+ <hr>
+
+ <h2>The measured scorecard: XERJ vs. a fair shell-only grep</h2>
+ <p>The harness runs both approaches over the same 22 ground-truth questions. A query "hits" if any returned passage contains the known answer substring. The grep baseline is <strong>fair</strong>: it searches the union of each query's curated keywords <em>and</em> the salient tokens of the question itself, so it is never rigged to dodge the answer's own line. The questions are split into four honest buckets:</p>
+ <ul>
+ <li><strong>binary_only</strong> (7) — the answer lives <em>only</em> inside a PDF or DOCX. XERJ extracted it; grep is structurally blind. <strong>This is the headline — a capability grep lacks.</strong></li>
+ <li><strong>large_literal</strong> (4) — the answer is a literal string buried inside a realistically <em>large</em> (≥60 KB) document. A fair grep <em>finds</em> the line, but must load the whole file to quote/verify it; XERJ returns one ranked passage. This is where the context-efficiency win shows up honestly.</li>
+ <li><strong>robustness</strong> (5) — the answer is phrased differently than the question, in a plaintext file grep can read. A fair grep of the question's own terms matches the same line XERJ's lexical embedder finds, so this is expected to <strong>tie</strong> — kept to show single-query convenience, not a meaning win.</li>
+ <li><strong>literal</strong> (6) — the answer is a literal substring in a small plaintext file. Both approaches should find these; the honest control that checks XERJ isn't just inflating the easy cases.</li>
+ </ul>
+
+<pre class="code">Metric XERJ Baseline (grep)
+------------------------------------- ------------ ---------------
+Overall coverage 21/22 (95.5%) 14/22 (63.6%)
+binary_only (PDF/DOCX-only) HEADLINE 6/7 0/7
+large_literal (buried in ≥60 KB doc) 4/4 3/4
+robustness (differently phrased) 5/5 5/5 ← tie
+literal (plaintext substring) 6/6 6/6
+Query latency p50 / mean / max (ms) 61.23 / 64.9 / 116.09
+Index build time (ms) 1339</pre>
+
+ <p>Coverage by the format the answer lives in — the binary formats are the whole story:</p>
+<pre class="code">Format Queries XERJ Baseline
+------ ------- ----------- ----------
+docx 3 2/3 (66.7%) 0/3 (0%)
+pdf 5 5/5 (100%) 0/5 (0%)
+html 5 5/5 (100%) 5/5 (100%)
+md 7 7/7 (100%) 7/7 (100%)
+txt 2 2/2 (100%) 2/2 (100%)</pre>
+
+ <p>The decisive gap is exactly where you'd predict, and only there: on answers locked in PDF/DOCX, grep gets <strong>0/7</strong> and XERJ gets <strong>6/7</strong> (it misses one — see the honest miss below). On the plaintext buckets a <em>fair</em> grep keeps up — <strong>robustness ties 5/5</strong> and the <strong>literal control ties 6/6</strong> — so XERJ is not winning the cases grep can actually read. What XERJ adds on those is single-query, ranked retrieval and, on large documents, far less context.</p>
+
+ <h3>Context efficiency — real, but scale-dependent (not flat)</h3>
+ <p>To quote or verify an answer, the grep baseline must open the answer-containing file the hit lives in; XERJ returns just the ranked passage(s). The baseline is charged <strong>only that single answer file</strong> (<code class="inline">statSync(answer_path)</code>), and <strong>0 bytes on any query it cannot answer</strong> — never the false-positive files its broad terms also matched. Measured that way, the win is genuine but <strong>scale-dependent</strong>: large on big documents, and it INVERTS on tiny ones:</p>
+<pre class="code">View Baseline B XERJ B Ratio
+-------------------------------------- ---------- ------- --------
+large_literal (returned passages) 192,782 11,180 17.24× ← THE REAL WIN
+large_literal (single best passage) 192,782 2,250 85.68×
+literal (returned passages) 4,741 20,965 0.23× ← INVERTS on tiny files
+literal (single best passage) 4,741 3,930 1.21×
+answerable (all 14 baseline answers) 202,063 50,750 3.98×
+naive overall (all 22 queries) 202,063 79,872 2.53× ← NOT a claim*</pre>
+ <p>* The naive whole-corpus ratio <strong>flatters the blind baseline</strong> — it opens 0 bytes on every query it cannot answer — so we do not lean on it. The headline rests on the <code class="inline">large_literal</code> ratio (<strong>17.24×</strong>); on the small-file <code class="inline">literal</code> set the same metric drops to <strong>0.23×</strong>, because a handful of returned passages can meet or exceed a tiny whole file. Side by side — large_literal <strong>17.24×</strong> vs. literal <strong>0.23×</strong> — <em>is</em> the scale-dependence, measured rather than asserted; that is why the corpus deliberately includes four ≥60 KB documents, to show the win where it is real. <em>(Diagnostic only, used in no ratio or claim: had the baseline instead been charged every file its broad terms matched — false positives included — the total would be 4,218,746 bytes.)</em></p>
+
+ <hr>
+
+ <h2>Honest caveats</h2>
+ <p>This repo has a standing honesty bar. Read these before you rely on any of the above.</p>
+
+ <ul>
+ <li><strong>The built-in embedder is <em>lexical</em>, not neural.</strong> XERJ's zero-config embedder is a deterministic feature-hashing model (384 dimensions, L2-normalised → cosine). It captures word / sub-word overlap, not deep meaning. That is exactly why the <code class="inline">robustness</code> bucket is an honest <strong>5/5 tie</strong>: a fair grep of the question's own terms matches the same lines XERJ's lexical embedder finds. We make <strong>no</strong> "beats grep on meaning" claim. For production-grade semantics, point the <code class="inline">semantic_text</code> field at an external OpenAI-compatible <code class="inline">/v1/embeddings</code> endpoint; the ingest and query paths are identical.</li>
+ <li><strong>kNN is exact brute-force at query time.</strong> The semantic sub-query scans every chunk vector. That's fine (and gives exact results) at this corpus size — the measured max query latency was <strong>116.09 ms</strong> over 420 chunks — but latency grows with the number of vectors scanned.</li>
+ <li><strong>Extraction quality is only as good as the extractor.</strong> <code class="inline">pdftotext</code>/<code class="inline">soffice</code> handle digital PDFs and Word files well, but a <em>scanned</em> PDF (an image of text) yields nothing without an OCR step, which this recipe does not include. Garbled extraction = garbled index. The indexer logs and skips files it cannot read rather than crashing.</li>
+ <li><strong>Context efficiency is real, but scale-dependent — not free on tiny files.</strong> The baseline is charged only the single answer-containing file it must open (and 0 bytes when it cannot answer at all). On the <code class="inline">large_literal</code> set XERJ returns <strong>17.24×</strong> less context than the whole answer files a fair grep must open (85.68× on a single best passage). But on the small-file <code class="inline">literal</code> set the same metric INVERTS to <strong>0.23×</strong> — a few returned passages can exceed a tiny whole file — and the naive whole-corpus ratio (2.53×) flatters the blind baseline, so we headline the measured large-document number, not the naive one. The context win appears as documents grow, and honestly disappears on tiny ones.</li>
+ <li><strong>The one miss (q05).</strong> "What is the acknowledgement SLA for a Sev-1 incident?" — the answer, "within 5 minutes," lives in <code class="inline">oncall-runbook.docx</code>. It <em>was</em> extracted and indexed (a direct <code class="inline">match_phrase</code> on the live index returns exactly that chunk), but for this phrasing hybrid ranking pushes it out of the top-5: the answer chunk lacks the word "SLA," and larger incident-heavy docs outrank it. So <code class="inline">binary_only</code> is honestly XERJ 6/7 — still strictly &gt; the baseline's 0/7. We did not change the retrieval strategy to force this to pass.</li>
+ <li><strong>The baseline could be upgraded.</strong> A script agent <em>could</em> shell out to <code class="inline">pdftotext</code>/<code class="inline">soffice</code> and close the binary gap. But it would still have no ranking, no per-chunk retrieval, no semantic layer, and would keep paying the whole-file context cost. XERJ's win is not "it can read PDFs" alone — it's format-agnostic extraction <em>plus</em> chunked, ranked, context-aware retrieval with a real (if shallow) semantic layer.</li>
+ </ul>
+
+ <hr>
+
+ <h2>Reproduce it yourself</h2>
+ <p>The whole thing — corpus generator, indexer, grep baseline, and comparison harness — lives in <a href="https://github.com/xerj-org/xerj/tree/main/demo/usecases/doc-index"><code class="inline">demo/usecases/doc-index</code></a>. Prereqs: Node 18+ (tested on v24), <code class="inline">pdftotext</code> (poppler-utils), <code class="inline">soffice</code>/LibreOffice, and <code class="inline">rg</code> (ripgrep). Then:</p>
+<pre class="code"># 1. Start a throwaway XERJ (ES-compat wire; default port :9200)
+xerj --insecure --data-dir ./docfolder-data
+
+# 2. Point the scripts at your node's port (the demo assumes :9209)
+export XERJ_PORT=9200
+
+# 3. (Optional) regenerate the fictional corpus + ground-truth queries.json
+node gen-corpus.mjs
+
+# 4. Extract, chunk, embed and bulk-index the whole folder
+node xerj-index.mjs --recreate
+
+# 5. Score XERJ vs the grep baseline → writes results.json + SCORECARD.md
+node compare.mjs</pre>
+
+ <p>Open <code class="inline">SCORECARD.md</code> for the per-query table and the honest verdict, or <code class="inline">results.json</code> for the machine-readable numbers. The full contract these scripts obey — index mappings, chunking, query schema, scoring, and the gate — is documented in <code class="inline">SPEC.md</code> and the deliverable's <code class="inline">README.md</code>. Every figure on this page is emitted by that run; nothing is hardcoded.</p>
+
+ <!-- Print-only branded footer (hidden on screen, visible in PDF) -->
+ <div class="print-footer" style="display:none;">
+ <span>XERJ.AI · V0.1 · 2026-04-16</span>
+ <a href="https://xerj.org/docs/">https://xerj.org/docs/</a>
+ <span>Recipe · Index a folder of docs</span>
+ </div>
+</main>
+</div>
+
+<footer class="footer">
+ <span>XERJ.AI · DOCS · V1.0.0-RC.1</span>
+ <span>READ THE SOURCE</span>
+ <span><a href="/index.html">HOME</a> · <a href="/product.html">PRODUCT</a> · <a href="/brand.html">BRAND</a> · <a href="/playground.html">DASHBOARDS</a> &middot; <a href="/llms.txt">LLMS.TXT</a></span>
+</footer>
+
+<script type="application/json" id="docs-index">[{"title":"Overview","slug":"/docs/index.html","kicker":"START","snip":""},{"title":"Quickstart","slug":"/docs/quickstart.html","kicker":"START","snip":""},{"title":"Install","slug":"/docs/install.html","kicker":"START","snip":""},{"title":"Migrate from ES","slug":"/docs/migration-from-es.html","kicker":"START","snip":""},{"title":"CLI","slug":"/docs/cli.html","kicker":"REFERENCE","snip":""},{"title":"Config TOML","slug":"/docs/config.html","kicker":"REFERENCE","snip":""},{"title":"Environment","slug":"/docs/env.html","kicker":"REFERENCE","snip":""},{"title":"Native REST API","slug":"/docs/api-native.html","kicker":"REFERENCE","snip":""},{"title":"ES-compat API","slug":"/docs/api-es-compat.html","kicker":"REFERENCE","snip":""},{"title":"Query types","slug":"/docs/queries.html","kicker":"DATA MODEL","snip":""},{"title":"Aggregations","slug":"/docs/aggregations.html","kicker":"DATA MODEL","snip":""},{"title":"Analyzers","slug":"/docs/analyzers.html","kicker":"DATA MODEL","snip":""},{"title":"Vectors & HNSW","slug":"/docs/vectors.html","kicker":"DATA MODEL","snip":""},{"title":"Ingest pipelines","slug":"/docs/ingest.html","kicker":"ENGINE","snip":""},{"title":"Storage & WAL","slug":"/docs/storage.html","kicker":"ENGINE","snip":""},{"title":"Compression","slug":"/docs/compression.html","kicker":"ENGINE","snip":""},{"title":"Clustering","slug":"/docs/clustering.html","kicker":"ENGINE","snip":""},{"title":"Metrics","slug":"/docs/metrics.html","kicker":"ENGINE","snip":""},{"title":"Auth & TLS","slug":"/docs/security.html","kicker":"ENGINE","snip":""},{"title":"Running in production","slug":"/docs/operations.html","kicker":"OPERATIONS","snip":""},{"title":"Backup & restore","slug":"/docs/backup-restore.html","kicker":"OPERATIONS","snip":""},{"title":"Upgrades","slug":"/docs/upgrades.html","kicker":"OPERATIONS","snip":""},{"title":"Troubleshooting","slug":"/docs/troubleshooting.html","kicker":"OPERATIONS","snip":""},{"title":"SIEM","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOKS","snip":""},{"title":"Log analytics","slug":"/docs/playbooks/log-analytics.html","kicker":"PLAYBOOKS","snip":""},{"title":"Vector · RAG","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOKS","snip":""},{"title":"Full-text search","slug":"/docs/playbooks/full-text.html","kicker":"PLAYBOOKS","snip":""},{"title":"Observability","slug":"/docs/playbooks/observability.html","kicker":"PLAYBOOKS","snip":""},{"title":"CLI · --config","slug":"/docs/cli.html#config-flag","kicker":"CLI","snip":"-c, --config <PATH> Path to the TOML config file."},{"title":"CLI · --data-dir","slug":"/docs/cli.html#data-dir-flag","kicker":"CLI","snip":"-d, --data-dir <PATH> Override [server] data_dir at the command line."},{"title":"CLI · --insecure","slug":"/docs/cli.html#insecure-flag","kicker":"CLI","snip":"-k, --insecure Disable TLS and API-key auth. Dev only."},{"title":"[server] rest_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 8080 · Native REST API listener port."},{"title":"[server] es_compat_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 9200 · Elasticsearch-wire-format port. 0 to disable."},{"title":"[storage] wal_sync","slug":"/docs/config.html#storage","kicker":"CONFIG","snip":"enum · default batched · sync · batched · async. Durability knob."},{"title":"[vector] hnsw_ef_search","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"u32 · default 100 · Accepted for ES-compat; query kNN is exact brute-force."},{"title":"[vector] default_quantization","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"enum · none · scalar8 · scalar4 · binary."},{"title":"GET /v1/health","slug":"/docs/api-native.html#health","kicker":"NATIVE API","snip":"Liveness + readiness. {status, uptime_s, version}."},{"title":"GET /v1/metrics","slug":"/docs/api-native.html#metrics","kicker":"NATIVE API","snip":"Prometheus text format. Scrape-friendly."},{"title":"POST /v1/indices/:name/turbo-ingest","slug":"/docs/api-native.html#turbo-ingest","kicker":"NATIVE API","snip":"NDJSON bulk path with parallel tokenization."},{"title":"POST /v1/indices/:name/search","slug":"/docs/api-native.html#search","kicker":"NATIVE API","snip":"Unified query endpoint — full-text, term, range, KNN, hybrid."},{"title":"POST /:index/_bulk","slug":"/docs/api-es-compat.html#bulk","kicker":"ES API","snip":"NDJSON bulk. Actions: index, create, update, delete."},{"title":"POST /:index/_search","slug":"/docs/api-es-compat.html#search","kicker":"ES API","snip":"ES query DSL. Drop-in migration path."},{"title":"match_all","slug":"/docs/queries.html#match-all","kicker":"QUERY","snip":"Everything in the index."},{"title":"bool","slug":"/docs/queries.html#bool","kicker":"QUERY","snip":"must / should / must_not / filter clauses."},{"title":"knn","slug":"/docs/queries.html#knn","kicker":"QUERY","snip":"Exact (brute-force) vector kNN; HNSW built at ingest."},{"title":"semantic_search","slug":"/docs/queries.html#semantic-search","kicker":"QUERY","snip":"Embed the query at query time."},{"title":"hybrid","slug":"/docs/queries.html#hybrid","kicker":"QUERY","snip":"RRF / linear fusion of sub-queries."},{"title":"terms aggregation","slug":"/docs/aggregations.html#terms","kicker":"AGG","snip":"Exact, pre-computed terms histograms over keyword fields."},{"title":"date_histogram","slug":"/docs/aggregations.html#date-histogram","kicker":"AGG","snip":"Time-bucketed counts."},{"title":"HNSW","slug":"/docs/vectors.html#hnsw","kicker":"VECTORS","snip":"Hierarchical graph index, built at ingest. m, ef_construction."},{"title":"SQ8 quantization","slug":"/docs/vectors.html#quantization","kicker":"VECTORS","snip":"8-bit scalar quantization. 4× memory reduction."},{"title":"turbo-ingest","slug":"/docs/ingest.html#turbo","kicker":"INGEST","snip":"NDJSON bulk path. Parallel tokenization across cores."},{"title":"delta-of-delta","slug":"/docs/compression.html#delta-of-delta","kicker":"COMPRESSION","snip":"Timestamp encoding. Apache/Nginx/ISO auto-detect."},{"title":"dictionary encoding","slug":"/docs/compression.html#dictionary","kicker":"COMPRESSION","snip":"Up to 256 unique strings per column."},{"title":"api_key","slug":"/docs/security.html#api-key","kicker":"SECURITY","snip":"Static admin key. Auto-generated on first run."},{"title":"SIEM playbook","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOK","snip":"Terms agg over source IPs for SIEM, with detection rules."},{"title":"RAG playbook","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOK","snip":"Embeddings, HNSW, hybrid search, BM25 + kNN fusion."},{"title":"[cluster] enabled","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"bool · default false · Enable multi-node Raft cluster mode."},{"title":"[cluster] peers","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"array · \"node_id=host:port\" · At least one peer per node."},{"title":"[cluster] port","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"u16 · default 9300 · Intra-cluster Raft + search port."},{"title":"[embedding] default_endpoint","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · OpenAI-compatible embeddings URL for auto-embedding on ingest."},{"title":"[embedding] default_model","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · Model name (e.g. text-embedding-3-small, nomic-embed-text)."},{"title":"Raft","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Embedded Raft — leader election, log replication, metadata only."},{"title":"Node draining","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"POST /v1/cluster/nodes/:id/drain before maintenance."},{"title":"Region split/merge","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Range partitions auto-split at byte threshold, merge back when small."},{"title":"Systemd unit","slug":"/docs/operations.html","kicker":"OPS","snip":"Production unit file with hardening directives."},{"title":"Readiness probe","slug":"/docs/operations.html","kicker":"OPS","snip":"GET /v1/health/ready — only 200 once WAL replay is complete."},{"title":"Config reload","slug":"/docs/operations.html","kicker":"OPS","snip":"SIGHUP reloads most keys without restart."},{"title":"Hot backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"rsync the data dir after POST /v1/admin/flush."},{"title":"S3 backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"POST /v1/admin/backup → any S3-compatible endpoint."},{"title":"Rolling upgrade","slug":"/docs/upgrades.html","kicker":"OPS","snip":"Drain → stop → upgrade → start → activate, per node."},{"title":"Merge pressure","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Segment count climbing → raise merge.max_concurrent + io_rate."},{"title":"Cluster flapping","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Leader changes every few seconds → raise cluster.tick_ms."},{"title":"WAL replay failed","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"xerj verify --repair-wal truncates at last valid entry."},{"title": "Recipes", "slug": "/docs/recipes/index.html", "kicker": "RECIPES", "snip": "Task-oriented AI & search recipes, each verified end-to-end against a live XERJ."},{"title": "Semantic search & RAG", "slug": "/docs/recipes/semantic-search-rag.html", "kicker": "RECIPE", "snip": "Retrieval by meaning with semantic_text — auto-embed on ingest, no separate vector DB."},{"title": "Agent memory", "slug": "/docs/recipes/agentic-memory.html", "kicker": "RECIPE", "snip": "Long-term memory for agents via /_memory — store, recall, forget, per-agent isolation."},{"title": "Passage retrieval", "slug": "/docs/recipes/passage-retrieval.html", "kicker": "RECIPE", "snip": "Auto-chunk long docs; best-passage (max-sim) scoring — 98% top-3."},{"title": "Vector search · kNN", "slug": "/docs/recipes/vector-search-knn.html", "kicker": "RECIPE", "snip": "Nearest-neighbor over dense_vector (HNSW) with filters."},{"title": "Vector quantization", "slug": "/docs/recipes/vector-quantization.html", "kicker": "RECIPE", "snip": "int8_hnsw scalar8 — 4× smaller vectors, recall@10 ≈ 0.99."},{"title": "Hybrid search", "slug": "/docs/recipes/hybrid-search.html", "kicker": "RECIPE", "snip": "Keyword + vector in one query — RRF fusion."},{"title": "Log analytics", "slug": "/docs/recipes/log-analytics.html", "kicker": "RECIPE", "snip": "Raw logs to dashboards — error rates, p95 latency, top services."},{"title": "Anomaly detection", "slug": "/docs/recipes/anomaly-detection.html", "kicker": "RECIPE", "snip": "Statistical _ml detectors that flag spikes in metrics/logs."},{"title": "Anomaly datafeeds", "slug": "/docs/recipes/continuous-anomaly-datafeeds.html", "kicker": "RECIPE", "snip": "Live _ml datafeed re-scores an index on a timer, stores anomaly records."},{"title": "Migrate from ES", "slug": "/docs/recipes/migrate-from-elasticsearch.html", "kicker": "RECIPE", "snip": "Point your existing ES client at XERJ — same wire, change the URL."},{"title": "Index a folder of mixed docs", "slug": "/docs/recipes/document-folder-index.html", "kicker": "RECIPE", "snip": "Walk a folder of PDF/DOCX/HTML/MD/TXT, extract + chunk + embed, then lexical/semantic/hybrid search — grep can't read a PDF."}]</script>
+
+<script>
+ // IIFE — js-auth navigates between decrypted pages in the same
+ // JS realm, so const bindings must be lexically scoped.
+ (function () {
+ const root = document.documentElement;
+ for (const btn of document.querySelectorAll('[data-theme]')) {
+ btn.addEventListener('click', () => {
+ const t = btn.getAttribute('data-theme');
+ root.setAttribute('data-theme', t);
+ for (const b of document.querySelectorAll('[data-theme]')) {
+ b.classList.toggle('active', b.getAttribute('data-theme') === t);
+ }
+ });
+ }
+
+ // Client-side docs search — filters the inline JSON index.
+ const searchEl = document.getElementById('docsSearch');
+ const inputEl = document.getElementById('docsSearchInput');
+ const resultsEl = document.getElementById('docsSearchResults');
+ const indexJson = document.getElementById('docs-index');
+ if (searchEl && inputEl && resultsEl && indexJson) {
+ let index = [];
+ try { index = JSON.parse(indexJson.textContent || '[]'); } catch {}
+ const render = (q) => {
+ const query = q.trim().toLowerCase();
+ if (!query) { searchEl.classList.remove('active'); resultsEl.innerHTML = ''; return; }
+ const scored = [];
+ for (const e of index) {
+ const hay = (e.title + ' ' + (e.kicker || '') + ' ' + (e.snip || '')).toLowerCase();
+ if (hay.includes(query)) {
+ let score = 0;
+ if (e.title.toLowerCase().includes(query)) score += 10;
+ if ((e.kicker || '').toLowerCase() === query.toUpperCase()) score += 5;
+ if ((e.snip || '').toLowerCase().includes(query)) score += 1;
+ scored.push({ e, score });
+ }
+ }
+ scored.sort((a, b) => b.score - a.score);
+ const top = scored.slice(0, 20);
+ if (top.length === 0) {
+ resultsEl.innerHTML = '<div class="no-results">NO MATCHES — TRY A SHORTER QUERY</div>';
+ } else {
+ resultsEl.innerHTML = top.map(({e}) => {
+ const kicker = e.kicker || '';
+ const snip = e.snip ? '<div class="snip">' + escHtml(e.snip) + '</div>' : '';
+ return '<a href="' + e.slug + '">' +
+ '<span class="kicker">' + escHtml(kicker) + '</span>' +
+ '<span class="title">' + escHtml(e.title) + '</span>' +
+ snip + '</a>';
+ }).join('');
+ }
+ searchEl.classList.add('active');
+ };
+ const escHtml = (s) => String(s).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;');
+ inputEl.addEventListener('input', (e) => render(e.target.value));
+ inputEl.addEventListener('focus', () => { if (inputEl.value) render(inputEl.value); });
+ document.addEventListener('keydown', (e) => {
+ if ((e.key === 'k' || e.key === 'K') && (e.metaKey || e.ctrlKey)) {
+ e.preventDefault();
+ inputEl.focus();
+ inputEl.select();
+ }
+ if (e.key === 'Escape' && document.activeElement === inputEl) {
+ inputEl.value = '';
+ searchEl.classList.remove('active');
+ resultsEl.innerHTML = '';
+ inputEl.blur();
+ }
+ });
+ }
+ })();
+</script>
+
+</body>
+</html>
diff --git a/landing/docs/recipes/index.html b/landing/docs/recipes/index.html
index 383642459..915ade323 100644
--- a/landing/docs/recipes/index.html
+++ b/landing/docs/recipes/index.html
@@ -101,6 +101,7 @@
<a href="/docs/recipes/anomaly-detection.html" class=""><span class="eyebrow">08</span>Anomaly detection</a>
<a href="/docs/recipes/continuous-anomaly-datafeeds.html" class=""><span class="eyebrow">09</span>Anomaly datafeeds</a>
<a href="/docs/recipes/migrate-from-elasticsearch.html" class=""><span class="eyebrow">10</span>Migrate from ES</a>
+ <a href="/docs/recipes/document-folder-index.html" class=""><span class="eyebrow">11</span>Doc folder index</a>
</div>
</nav>
@@ -171,6 +172,11 @@
<span class="title">Migrate from Elasticsearch</span>
<p class="lead">Point your existing ES client at XERJ — same wire, change the URL.</p>
</a>
+ <a class="recipe" href="/docs/recipes/document-folder-index.html">
+ <span class="num">11</span>
+ <span class="title">Index a folder of PDFs, Word docs &amp; web pages</span>
+ <p class="lead">One <code class="inline">xerj-index</code> pass walks a recursive folder of PDF/DOCX/HTML/MD/TXT, extracts + chunks + auto-embeds every file, then lexical, semantic &amp; hybrid search return ranked, cited passages — measured 21/22 vs a fair grep baseline's 14/22, with the decisive win on binary_only (6/7 vs 0/7) because grep can't read a PDF/DOCX.</p>
+ </a>
</div>
<!-- Print-only branded footer (hidden on screen, visible in PDF) -->
@@ -188,7 +194,7 @@
<span><a href="/index.html">HOME</a> · <a href="/product.html">PRODUCT</a> · <a href="/brand.html">BRAND</a> · <a href="/playground.html">DASHBOARDS</a> &middot; <a href="/llms.txt">LLMS.TXT</a></span>
</footer>
-<script type="application/json" id="docs-index">[{"title":"Overview","slug":"/docs/index.html","kicker":"START","snip":""},{"title":"Quickstart","slug":"/docs/quickstart.html","kicker":"START","snip":""},{"title":"Install","slug":"/docs/install.html","kicker":"START","snip":""},{"title":"Migrate from ES","slug":"/docs/migration-from-es.html","kicker":"START","snip":""},{"title":"CLI","slug":"/docs/cli.html","kicker":"REFERENCE","snip":""},{"title":"Config TOML","slug":"/docs/config.html","kicker":"REFERENCE","snip":""},{"title":"Environment","slug":"/docs/env.html","kicker":"REFERENCE","snip":""},{"title":"Native REST API","slug":"/docs/api-native.html","kicker":"REFERENCE","snip":""},{"title":"ES-compat API","slug":"/docs/api-es-compat.html","kicker":"REFERENCE","snip":""},{"title":"Query types","slug":"/docs/queries.html","kicker":"DATA MODEL","snip":""},{"title":"Aggregations","slug":"/docs/aggregations.html","kicker":"DATA MODEL","snip":""},{"title":"Analyzers","slug":"/docs/analyzers.html","kicker":"DATA MODEL","snip":""},{"title":"Vectors & HNSW","slug":"/docs/vectors.html","kicker":"DATA MODEL","snip":""},{"title":"Ingest pipelines","slug":"/docs/ingest.html","kicker":"ENGINE","snip":""},{"title":"Storage & WAL","slug":"/docs/storage.html","kicker":"ENGINE","snip":""},{"title":"Compression","slug":"/docs/compression.html","kicker":"ENGINE","snip":""},{"title":"Clustering","slug":"/docs/clustering.html","kicker":"ENGINE","snip":""},{"title":"Metrics","slug":"/docs/metrics.html","kicker":"ENGINE","snip":""},{"title":"Auth & TLS","slug":"/docs/security.html","kicker":"ENGINE","snip":""},{"title":"Running in production","slug":"/docs/operations.html","kicker":"OPERATIONS","snip":""},{"title":"Backup & restore","slug":"/docs/backup-restore.html","kicker":"OPERATIONS","snip":""},{"title":"Upgrades","slug":"/docs/upgrades.html","kicker":"OPERATIONS","snip":""},{"title":"Troubleshooting","slug":"/docs/troubleshooting.html","kicker":"OPERATIONS","snip":""},{"title":"SIEM","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOKS","snip":""},{"title":"Log analytics","slug":"/docs/playbooks/log-analytics.html","kicker":"PLAYBOOKS","snip":""},{"title":"Vector · RAG","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOKS","snip":""},{"title":"Full-text search","slug":"/docs/playbooks/full-text.html","kicker":"PLAYBOOKS","snip":""},{"title":"Observability","slug":"/docs/playbooks/observability.html","kicker":"PLAYBOOKS","snip":""},{"title":"CLI · --config","slug":"/docs/cli.html#config-flag","kicker":"CLI","snip":"-c, --config <PATH> Path to the TOML config file."},{"title":"CLI · --data-dir","slug":"/docs/cli.html#data-dir-flag","kicker":"CLI","snip":"-d, --data-dir <PATH> Override [server] data_dir at the command line."},{"title":"CLI · --insecure","slug":"/docs/cli.html#insecure-flag","kicker":"CLI","snip":"-k, --insecure Disable TLS and API-key auth. Dev only."},{"title":"[server] rest_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 8080 · Native REST API listener port."},{"title":"[server] es_compat_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 9200 · Elasticsearch-wire-format port. 0 to disable."},{"title":"[storage] wal_sync","slug":"/docs/config.html#storage","kicker":"CONFIG","snip":"enum · default batched · sync · batched · async. Durability knob."},{"title":"[vector] hnsw_ef_search","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"u32 · default 100 · Accepted for ES-compat; query kNN is exact brute-force."},{"title":"[vector] default_quantization","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"enum · none · scalar8 · scalar4 · binary."},{"title":"GET /v1/health","slug":"/docs/api-native.html#health","kicker":"NATIVE API","snip":"Liveness + readiness. {status, uptime_s, version}."},{"title":"GET /v1/metrics","slug":"/docs/api-native.html#metrics","kicker":"NATIVE API","snip":"Prometheus text format. Scrape-friendly."},{"title":"POST /v1/indices/:name/turbo-ingest","slug":"/docs/api-native.html#turbo-ingest","kicker":"NATIVE API","snip":"NDJSON bulk path with parallel tokenization."},{"title":"POST /v1/indices/:name/search","slug":"/docs/api-native.html#search","kicker":"NATIVE API","snip":"Unified query endpoint — full-text, term, range, KNN, hybrid."},{"title":"POST /:index/_bulk","slug":"/docs/api-es-compat.html#bulk","kicker":"ES API","snip":"NDJSON bulk. Actions: index, create, update, delete."},{"title":"POST /:index/_search","slug":"/docs/api-es-compat.html#search","kicker":"ES API","snip":"ES query DSL. Drop-in migration path."},{"title":"match_all","slug":"/docs/queries.html#match-all","kicker":"QUERY","snip":"Everything in the index."},{"title":"bool","slug":"/docs/queries.html#bool","kicker":"QUERY","snip":"must / should / must_not / filter clauses."},{"title":"knn","slug":"/docs/queries.html#knn","kicker":"QUERY","snip":"Exact (brute-force) vector kNN; HNSW built at ingest."},{"title":"semantic_search","slug":"/docs/queries.html#semantic-search","kicker":"QUERY","snip":"Embed the query at query time."},{"title":"hybrid","slug":"/docs/queries.html#hybrid","kicker":"QUERY","snip":"RRF / linear fusion of sub-queries."},{"title":"terms aggregation","slug":"/docs/aggregations.html#terms","kicker":"AGG","snip":"Exact, pre-computed terms histograms over keyword fields."},{"title":"date_histogram","slug":"/docs/aggregations.html#date-histogram","kicker":"AGG","snip":"Time-bucketed counts."},{"title":"HNSW","slug":"/docs/vectors.html#hnsw","kicker":"VECTORS","snip":"Hierarchical graph index, built at ingest. m, ef_construction."},{"title":"SQ8 quantization","slug":"/docs/vectors.html#quantization","kicker":"VECTORS","snip":"8-bit scalar quantization. 4× memory reduction."},{"title":"turbo-ingest","slug":"/docs/ingest.html#turbo","kicker":"INGEST","snip":"NDJSON bulk path. Parallel tokenization across cores."},{"title":"delta-of-delta","slug":"/docs/compression.html#delta-of-delta","kicker":"COMPRESSION","snip":"Timestamp encoding. Apache/Nginx/ISO auto-detect."},{"title":"dictionary encoding","slug":"/docs/compression.html#dictionary","kicker":"COMPRESSION","snip":"Up to 256 unique strings per column."},{"title":"api_key","slug":"/docs/security.html#api-key","kicker":"SECURITY","snip":"Static admin key. Auto-generated on first run."},{"title":"SIEM playbook","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOK","snip":"Terms agg over source IPs for SIEM, with detection rules."},{"title":"RAG playbook","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOK","snip":"Embeddings, HNSW, hybrid search, BM25 + kNN fusion."},{"title":"[cluster] enabled","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"bool · default false · Enable multi-node Raft cluster mode."},{"title":"[cluster] peers","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"array · \"node_id=host:port\" · At least one peer per node."},{"title":"[cluster] port","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"u16 · default 9300 · Intra-cluster Raft + search port."},{"title":"[embedding] default_endpoint","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · OpenAI-compatible embeddings URL for auto-embedding on ingest."},{"title":"[embedding] default_model","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · Model name (e.g. text-embedding-3-small, nomic-embed-text)."},{"title":"Raft","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Embedded Raft — leader election, log replication, metadata only."},{"title":"Node draining","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"POST /v1/cluster/nodes/:id/drain before maintenance."},{"title":"Region split/merge","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Range partitions auto-split at byte threshold, merge back when small."},{"title":"Systemd unit","slug":"/docs/operations.html","kicker":"OPS","snip":"Production unit file with hardening directives."},{"title":"Readiness probe","slug":"/docs/operations.html","kicker":"OPS","snip":"GET /v1/health/ready — only 200 once WAL replay is complete."},{"title":"Config reload","slug":"/docs/operations.html","kicker":"OPS","snip":"SIGHUP reloads most keys without restart."},{"title":"Hot backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"rsync the data dir after POST /v1/admin/flush."},{"title":"S3 backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"POST /v1/admin/backup → any S3-compatible endpoint."},{"title":"Rolling upgrade","slug":"/docs/upgrades.html","kicker":"OPS","snip":"Drain → stop → upgrade → start → activate, per node."},{"title":"Merge pressure","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Segment count climbing → raise merge.max_concurrent + io_rate."},{"title":"Cluster flapping","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Leader changes every few seconds → raise cluster.tick_ms."},{"title":"WAL replay failed","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"xerj verify --repair-wal truncates at last valid entry."},{"title": "Recipes", "slug": "/docs/recipes/index.html", "kicker": "RECIPES", "snip": "Task-oriented AI & search recipes, each verified end-to-end against a live XERJ."},{"title": "Semantic search & RAG", "slug": "/docs/recipes/semantic-search-rag.html", "kicker": "RECIPE", "snip": "Retrieval by meaning with semantic_text — auto-embed on ingest, no separate vector DB."},{"title": "Agent memory", "slug": "/docs/recipes/agentic-memory.html", "kicker": "RECIPE", "snip": "Long-term memory for agents via /_memory — store, recall, forget, per-agent isolation."},{"title": "Passage retrieval", "slug": "/docs/recipes/passage-retrieval.html", "kicker": "RECIPE", "snip": "Auto-chunk long docs; best-passage (max-sim) scoring — 98% top-3."},{"title": "Vector search · kNN", "slug": "/docs/recipes/vector-search-knn.html", "kicker": "RECIPE", "snip": "Nearest-neighbor over dense_vector (HNSW) with filters."},{"title": "Vector quantization", "slug": "/docs/recipes/vector-quantization.html", "kicker": "RECIPE", "snip": "int8_hnsw scalar8 — 4× smaller vectors, recall@10 ≈ 0.99."},{"title": "Hybrid search", "slug": "/docs/recipes/hybrid-search.html", "kicker": "RECIPE", "snip": "Keyword + vector in one query — RRF fusion."},{"title": "Log analytics", "slug": "/docs/recipes/log-analytics.html", "kicker": "RECIPE", "snip": "Raw logs to dashboards — error rates, p95 latency, top services."},{"title": "Anomaly detection", "slug": "/docs/recipes/anomaly-detection.html", "kicker": "RECIPE", "snip": "Statistical _ml detectors that flag spikes in metrics/logs."},{"title": "Anomaly datafeeds", "slug": "/docs/recipes/continuous-anomaly-datafeeds.html", "kicker": "RECIPE", "snip": "Live _ml datafeed re-scores an index on a timer, stores anomaly records."},{"title": "Migrate from ES", "slug": "/docs/recipes/migrate-from-elasticsearch.html", "kicker": "RECIPE", "snip": "Point your existing ES client at XERJ — same wire, change the URL."}]</script>
+<script type="application/json" id="docs-index">[{"title":"Overview","slug":"/docs/index.html","kicker":"START","snip":""},{"title":"Quickstart","slug":"/docs/quickstart.html","kicker":"START","snip":""},{"title":"Install","slug":"/docs/install.html","kicker":"START","snip":""},{"title":"Migrate from ES","slug":"/docs/migration-from-es.html","kicker":"START","snip":""},{"title":"CLI","slug":"/docs/cli.html","kicker":"REFERENCE","snip":""},{"title":"Config TOML","slug":"/docs/config.html","kicker":"REFERENCE","snip":""},{"title":"Environment","slug":"/docs/env.html","kicker":"REFERENCE","snip":""},{"title":"Native REST API","slug":"/docs/api-native.html","kicker":"REFERENCE","snip":""},{"title":"ES-compat API","slug":"/docs/api-es-compat.html","kicker":"REFERENCE","snip":""},{"title":"Query types","slug":"/docs/queries.html","kicker":"DATA MODEL","snip":""},{"title":"Aggregations","slug":"/docs/aggregations.html","kicker":"DATA MODEL","snip":""},{"title":"Analyzers","slug":"/docs/analyzers.html","kicker":"DATA MODEL","snip":""},{"title":"Vectors & HNSW","slug":"/docs/vectors.html","kicker":"DATA MODEL","snip":""},{"title":"Ingest pipelines","slug":"/docs/ingest.html","kicker":"ENGINE","snip":""},{"title":"Storage & WAL","slug":"/docs/storage.html","kicker":"ENGINE","snip":""},{"title":"Compression","slug":"/docs/compression.html","kicker":"ENGINE","snip":""},{"title":"Clustering","slug":"/docs/clustering.html","kicker":"ENGINE","snip":""},{"title":"Metrics","slug":"/docs/metrics.html","kicker":"ENGINE","snip":""},{"title":"Auth & TLS","slug":"/docs/security.html","kicker":"ENGINE","snip":""},{"title":"Running in production","slug":"/docs/operations.html","kicker":"OPERATIONS","snip":""},{"title":"Backup & restore","slug":"/docs/backup-restore.html","kicker":"OPERATIONS","snip":""},{"title":"Upgrades","slug":"/docs/upgrades.html","kicker":"OPERATIONS","snip":""},{"title":"Troubleshooting","slug":"/docs/troubleshooting.html","kicker":"OPERATIONS","snip":""},{"title":"SIEM","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOKS","snip":""},{"title":"Log analytics","slug":"/docs/playbooks/log-analytics.html","kicker":"PLAYBOOKS","snip":""},{"title":"Vector · RAG","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOKS","snip":""},{"title":"Full-text search","slug":"/docs/playbooks/full-text.html","kicker":"PLAYBOOKS","snip":""},{"title":"Observability","slug":"/docs/playbooks/observability.html","kicker":"PLAYBOOKS","snip":""},{"title":"CLI · --config","slug":"/docs/cli.html#config-flag","kicker":"CLI","snip":"-c, --config <PATH> Path to the TOML config file."},{"title":"CLI · --data-dir","slug":"/docs/cli.html#data-dir-flag","kicker":"CLI","snip":"-d, --data-dir <PATH> Override [server] data_dir at the command line."},{"title":"CLI · --insecure","slug":"/docs/cli.html#insecure-flag","kicker":"CLI","snip":"-k, --insecure Disable TLS and API-key auth. Dev only."},{"title":"[server] rest_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 8080 · Native REST API listener port."},{"title":"[server] es_compat_port","slug":"/docs/config.html#server","kicker":"CONFIG","snip":"u16 · default 9200 · Elasticsearch-wire-format port. 0 to disable."},{"title":"[storage] wal_sync","slug":"/docs/config.html#storage","kicker":"CONFIG","snip":"enum · default batched · sync · batched · async. Durability knob."},{"title":"[vector] hnsw_ef_search","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"u32 · default 100 · Accepted for ES-compat; query kNN is exact brute-force."},{"title":"[vector] default_quantization","slug":"/docs/config.html#vector","kicker":"CONFIG","snip":"enum · none · scalar8 · scalar4 · binary."},{"title":"GET /v1/health","slug":"/docs/api-native.html#health","kicker":"NATIVE API","snip":"Liveness + readiness. {status, uptime_s, version}."},{"title":"GET /v1/metrics","slug":"/docs/api-native.html#metrics","kicker":"NATIVE API","snip":"Prometheus text format. Scrape-friendly."},{"title":"POST /v1/indices/:name/turbo-ingest","slug":"/docs/api-native.html#turbo-ingest","kicker":"NATIVE API","snip":"NDJSON bulk path with parallel tokenization."},{"title":"POST /v1/indices/:name/search","slug":"/docs/api-native.html#search","kicker":"NATIVE API","snip":"Unified query endpoint — full-text, term, range, KNN, hybrid."},{"title":"POST /:index/_bulk","slug":"/docs/api-es-compat.html#bulk","kicker":"ES API","snip":"NDJSON bulk. Actions: index, create, update, delete."},{"title":"POST /:index/_search","slug":"/docs/api-es-compat.html#search","kicker":"ES API","snip":"ES query DSL. Drop-in migration path."},{"title":"match_all","slug":"/docs/queries.html#match-all","kicker":"QUERY","snip":"Everything in the index."},{"title":"bool","slug":"/docs/queries.html#bool","kicker":"QUERY","snip":"must / should / must_not / filter clauses."},{"title":"knn","slug":"/docs/queries.html#knn","kicker":"QUERY","snip":"Exact (brute-force) vector kNN; HNSW built at ingest."},{"title":"semantic_search","slug":"/docs/queries.html#semantic-search","kicker":"QUERY","snip":"Embed the query at query time."},{"title":"hybrid","slug":"/docs/queries.html#hybrid","kicker":"QUERY","snip":"RRF / linear fusion of sub-queries."},{"title":"terms aggregation","slug":"/docs/aggregations.html#terms","kicker":"AGG","snip":"Exact, pre-computed terms histograms over keyword fields."},{"title":"date_histogram","slug":"/docs/aggregations.html#date-histogram","kicker":"AGG","snip":"Time-bucketed counts."},{"title":"HNSW","slug":"/docs/vectors.html#hnsw","kicker":"VECTORS","snip":"Hierarchical graph index, built at ingest. m, ef_construction."},{"title":"SQ8 quantization","slug":"/docs/vectors.html#quantization","kicker":"VECTORS","snip":"8-bit scalar quantization. 4× memory reduction."},{"title":"turbo-ingest","slug":"/docs/ingest.html#turbo","kicker":"INGEST","snip":"NDJSON bulk path. Parallel tokenization across cores."},{"title":"delta-of-delta","slug":"/docs/compression.html#delta-of-delta","kicker":"COMPRESSION","snip":"Timestamp encoding. Apache/Nginx/ISO auto-detect."},{"title":"dictionary encoding","slug":"/docs/compression.html#dictionary","kicker":"COMPRESSION","snip":"Up to 256 unique strings per column."},{"title":"api_key","slug":"/docs/security.html#api-key","kicker":"SECURITY","snip":"Static admin key. Auto-generated on first run."},{"title":"SIEM playbook","slug":"/docs/playbooks/siem.html","kicker":"PLAYBOOK","snip":"Terms agg over source IPs for SIEM, with detection rules."},{"title":"RAG playbook","slug":"/docs/playbooks/vector-search.html","kicker":"PLAYBOOK","snip":"Embeddings, HNSW, hybrid search, BM25 + kNN fusion."},{"title":"[cluster] enabled","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"bool · default false · Enable multi-node Raft cluster mode."},{"title":"[cluster] peers","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"array · \"node_id=host:port\" · At least one peer per node."},{"title":"[cluster] port","slug":"/docs/config.html#cluster","kicker":"CONFIG","snip":"u16 · default 9300 · Intra-cluster Raft + search port."},{"title":"[embedding] default_endpoint","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · OpenAI-compatible embeddings URL for auto-embedding on ingest."},{"title":"[embedding] default_model","slug":"/docs/config.html#embedding","kicker":"CONFIG","snip":"string · Model name (e.g. text-embedding-3-small, nomic-embed-text)."},{"title":"Raft","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Embedded Raft — leader election, log replication, metadata only."},{"title":"Node draining","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"POST /v1/cluster/nodes/:id/drain before maintenance."},{"title":"Region split/merge","slug":"/docs/clustering.html","kicker":"CLUSTER","snip":"Range partitions auto-split at byte threshold, merge back when small."},{"title":"Systemd unit","slug":"/docs/operations.html","kicker":"OPS","snip":"Production unit file with hardening directives."},{"title":"Readiness probe","slug":"/docs/operations.html","kicker":"OPS","snip":"GET /v1/health/ready — only 200 once WAL replay is complete."},{"title":"Config reload","slug":"/docs/operations.html","kicker":"OPS","snip":"SIGHUP reloads most keys without restart."},{"title":"Hot backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"rsync the data dir after POST /v1/admin/flush."},{"title":"S3 backup","slug":"/docs/backup-restore.html","kicker":"OPS","snip":"POST /v1/admin/backup → any S3-compatible endpoint."},{"title":"Rolling upgrade","slug":"/docs/upgrades.html","kicker":"OPS","snip":"Drain → stop → upgrade → start → activate, per node."},{"title":"Merge pressure","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Segment count climbing → raise merge.max_concurrent + io_rate."},{"title":"Cluster flapping","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"Leader changes every few seconds → raise cluster.tick_ms."},{"title":"WAL replay failed","slug":"/docs/troubleshooting.html","kicker":"OPS","snip":"xerj verify --repair-wal truncates at last valid entry."},{"title": "Recipes", "slug": "/docs/recipes/index.html", "kicker": "RECIPES", "snip": "Task-oriented AI & search recipes, each verified end-to-end against a live XERJ."},{"title": "Semantic search & RAG", "slug": "/docs/recipes/semantic-search-rag.html", "kicker": "RECIPE", "snip": "Retrieval by meaning with semantic_text — auto-embed on ingest, no separate vector DB."},{"title": "Agent memory", "slug": "/docs/recipes/agentic-memory.html", "kicker": "RECIPE", "snip": "Long-term memory for agents via /_memory — store, recall, forget, per-agent isolation."},{"title": "Passage retrieval", "slug": "/docs/recipes/passage-retrieval.html", "kicker": "RECIPE", "snip": "Auto-chunk long docs; best-passage (max-sim) scoring — 98% top-3."},{"title": "Vector search · kNN", "slug": "/docs/recipes/vector-search-knn.html", "kicker": "RECIPE", "snip": "Nearest-neighbor over dense_vector (HNSW) with filters."},{"title": "Vector quantization", "slug": "/docs/recipes/vector-quantization.html", "kicker": "RECIPE", "snip": "int8_hnsw scalar8 — 4× smaller vectors, recall@10 ≈ 0.99."},{"title": "Hybrid search", "slug": "/docs/recipes/hybrid-search.html", "kicker": "RECIPE", "snip": "Keyword + vector in one query — RRF fusion."},{"title": "Log analytics", "slug": "/docs/recipes/log-analytics.html", "kicker": "RECIPE", "snip": "Raw logs to dashboards — error rates, p95 latency, top services."},{"title": "Anomaly detection", "slug": "/docs/recipes/anomaly-detection.html", "kicker": "RECIPE", "snip": "Statistical _ml detectors that flag spikes in metrics/logs."},{"title": "Anomaly datafeeds", "slug": "/docs/recipes/continuous-anomaly-datafeeds.html", "kicker": "RECIPE", "snip": "Live _ml datafeed re-scores an index on a timer, stores anomaly records."},{"title": "Migrate from ES", "slug": "/docs/recipes/migrate-from-elasticsearch.html", "kicker": "RECIPE", "snip": "Point your existing ES client at XERJ — same wire, change the URL."},{"title": "Index a folder of mixed docs", "slug": "/docs/recipes/document-folder-index.html", "kicker": "RECIPE", "snip": "Walk a folder of PDF/DOCX/HTML/MD/TXT, extract + chunk + embed, then lexical/semantic/hybrid search — grep can't read a PDF."}]</script>
<script>
// IIFE — js-auth navigates between decrypted pages in the same
diff --git a/landing/for-agents.html b/landing/for-agents.html
index c62ab2336..99d1f7af7 100644
--- a/landing/for-agents.html
+++ b/landing/for-agents.html
@@ -149,6 +149,60 @@ $ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \
</p>
</section>
+<!-- WORKED EXAMPLE · DOCUMENT FOLDER ------------------------->
+<section class="scene-section" style="border-top:0;">
+ <div class="kicker"><span>WORKED EXAMPLE · GIVE AN AGENT A SEARCHABLE DOCUMENT FOLDER</span></div>
+ <p class="lead" style="max-width:760px;">
+ Point the agent at a messy folder of real documents — PDF, DOCX, HTML,
+ Markdown, plain text. One indexing pass walks the tree, extracts the text
+ from <em>every</em> format, chunks it, and bulk-indexes into XERJ with an
+ auto-embedded <code class="inline">semantic_text</code> body. After that the
+ agent just asks questions over <code class="inline">:9200</code> and gets
+ back ranked, cited passages — no whole-file re-reads, no format blind spots.
+ </p>
+<pre class="code"># one-time: walk ./corpus, extract every format, chunk, bulk-index into `docfolder`
+$ node xerj-index.mjs ./corpus # 29 files → 420 chunks · 1339 ms
+
+# the agent asks a question — hybrid = BM25 on body_text, fused (rrf)
+# with semantic recall on the auto-embedded body
+$ curl -sXPOST localhost:9200/docfolder/_search \
+ -H 'content-type: application/json' \
+ -d '{"size":5,"query":{"hybrid":{"queries":[
+ {"query":{"match":{"body_text":"weeks of paid parental leave?"}}},
+ {"query":{"semantic":{"field":"body","query":"weeks of paid parental leave?","k":5}}}],
+ "fusion":"rrf"}}}'</pre>
+ <p class="lead" style="max-width:760px;">
+ On a 22-question set over that folder, this beat a <em>fair</em> shell-only
+ agent that greps the raw files (searching the question's own terms, not a
+ rigged keyword list): <strong>21/22 (95.5%)</strong> answered versus
+ <strong>14/22 (63.6%)</strong>. The decisive, capability-based gap is on
+ answers that live only inside a PDF or DOCX — grep sees compressed bytes and
+ matches nothing: <strong>6/7 vs 0/7</strong>. On the plaintext buckets a fair
+ grep keeps up: differently-phrased answers <strong>tie 5/5 vs 5/5</strong>,
+ and the literal control <strong>ties 6/6 vs 6/6</strong> — so XERJ is not
+ winning the cases grep can actually read. What it adds there is single-query,
+ ranked retrieval and, on large documents, far less context: over the
+ large-document set it returns <strong>17.24×</strong> fewer bytes than the
+ whole answer files a fair grep must open — where the baseline is charged only
+ the single answer-containing file it must open to read the line, nothing else.
+ Query latency was p50 <strong>61.23&nbsp;ms</strong> (max 116.09&nbsp;ms).
+ </p>
+ <p class="lead" style="max-width:760px;">
+ Honest scope, the same standard as everywhere on this page: the built-in
+ <code class="inline">semantic_text</code> embedder is a
+ <strong>lexical feature-hashing model (384-dim cosine), not neural</strong>,
+ so its matching is word/sub-word overlap, not deep understanding — which is
+ exactly why the differently-phrased set is a tie, not a XERJ win;
+ <code class="inline">knn</code> is exact brute-force at this corpus size. And
+ the context win is <strong>scale-dependent</strong>: on the large-document set
+ it is <strong>17.24×</strong>, but on tiny plaintext files it INVERTS to
+ <strong>0.23×</strong> — returning a few ranked passages can cost <em>more</em>
+ than reading the small file whole — so we don't claim it there. The full
+ corpus, query set, scorecard, and run steps are in the
+ <a href="/docs/recipes/document-folder-index.html">document-folder-index recipe</a>.
+ </p>
+</section>
+
<!-- HONESTY -------------------------------------------------->
<section class="scene-section" style="border-top:0;">
<div class="kicker"><span>WHAT IS HONEST TO CLAIM</span></div>
@@ -191,6 +245,11 @@ $ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \
<span class="title">Semantic search &amp; RAG</span>
<p class="lead">Retrieval by meaning with <code class="inline">semantic_text</code> — no separate vector DB.</p>
</a>
+ <a class="recipe" href="/docs/recipes/document-folder-index.html">
+ <span class="num">RECIPE</span>
+ <span class="title">Index a document folder</span>
+ <p class="lead">Give an agent format-agnostic search over a folder of PDF, DOCX, HTML &amp; Markdown — walk, extract, chunk, hybrid-query.</p>
+ </a>
<a class="recipe" href="/docs/api-es-compat.html">
<span class="num">REFERENCE</span>
<span class="title">ES-compatible API</span>
diff --git a/landing/llms.txt b/landing/llms.txt
index 823248704..03aef5419 100644
--- a/landing/llms.txt
+++ b/landing/llms.txt
@@ -72,6 +72,7 @@ The six canonical operations an agent uses, each grounded in a real endpoint:
- [Anomaly detection](https://xerj.org/docs/recipes/anomaly-detection.html): score records against an anomaly detector.
- [Continuous anomaly datafeeds](https://xerj.org/docs/recipes/continuous-anomaly-datafeeds.html): stream data through `_ml/datafeeds`.
- [Migrate from Elasticsearch](https://xerj.org/docs/recipes/migrate-from-elasticsearch.html): move an ES workload onto XERJ.
+- [Index a folder of mixed-format docs](https://xerj.org/docs/recipes/document-folder-index.html): walk a recursive folder of PDF/DOCX/HTML/MD/TXT → extract + chunk + auto-embed → lexical/semantic/hybrid search returns ranked, cited passages (measured 21/22 answered vs a fair grep baseline's 14/22; the decisive win is binary_only 6/7 vs 0/7 — grep is structurally blind to binary PDF/DOCX; the differently-phrased set is an honest tie because the built-in embedder is lexical, not neural).
## Optional