Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

benchmarks: grep-vs-retrieval harness for Lucene (no results yet)

xerj-orgbdaa73b66cc2
diff --git a/benchmarks/grep-vs-retrieval-lucene/README.md b/benchmarks/grep-vs-retrieval-lucene/README.md
new file mode 100644
index 000000000..749b27cd1
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/README.md
@@ -0,0 +1,338 @@
+# grep-vs-retrieval benchmark (Apache Lucene)
+
+> **Status: harness only. No results are published here yet.**
+> The runner, the question set and the analyzer are complete and tested against a
+> live XERJ instance. The measurement matrix has **not** finished, so this
+> directory deliberately contains **no** `results.json`, `runs/` or `logs/`.
+> Numbers will be added in a follow-up once every run is graded.
+
+This fills a gap the reference-coding study named explicitly: *"The measurement
+harness - the three-arm runner, the purpose-built reference libraries, the
+hidden-test task specs - is **not** in this repository; only its outputs are."*
+(`docs/case-studies/reference-coding/CASE_STUDY.md`). This is the equivalent
+harness, shipped.
+
+## What it measures
+
+For real questions about Lucene internals, how many tokens a coding agent spends
+answering them three ways:
+
+| arm | how it answers |
+|---|---|
+| `native` | Claude Code with Grep/Read over the real Lucene tree |
+| `xerj` | passages retrieved from a XERJ symbol index, injected |
+| `hybrid` | both available |
+
+Token and cost figures come from the `claude` CLI's own accounting
+(`claude -p --output-format json`), not from estimates or byte proxies.
+
+## Honesty properties built in
+
+- **Answers are blind-judged.** `analyze.py` refuses to score a run it has not
+ graded and reports `ungraded` as UNKNOWN rather than assuming success. A
+ cheaper wrong answer is a loss, not a win.
+- **The native arm is not handicapped.** It runs with `cwd` set to the real
+ 6,012-file Lucene checkout and keeps its normal tools.
+- **No answer leakage.** The retrieval arm receives passages, never the answer.
+- **Prior-hostile questions are marked as such**, so questions the model may
+ already know from training are separated from ones it cannot recall.
+
+This repository has a standing precedent for publishing where XERJ *loses*
+(`docs/research/wordpress-verification-and-xerj-vs-grep.md` concluded the
+structured call graph was worse than grep for reachability, with silent false
+negatives). The same rule applies to whatever this harness measures.
+
+---
+
+## Original harness notes
+
+Measures the claim the post wants to make. It does not assume it.
+
+**Target claim (user's words, to be corrected by data):** *"your claude code / codex wastes
+40% tokens just on grep/sed files; XERJ reduces it many times."*
+
+**Status: harness built and validated end-to-end. Pilot = 1 question x 2 arms, real runs.
+Full matrix not yet run.** Everything below marked PILOT is measured; everything else is
+design. The pilot already contradicts part of the claim — see "What the pilot already shows".
+
+---
+
+## 0. What `claude -p --output-format json` actually returns
+
+Verified by running it, not assumed. Real keys from a live throwaway run:
+
+```
+is_error duration_api_ms duration_ms num_turns stop_reason session_id uuid subtype
+total_cost_usd usage modelUsage permission_denials terminal_reason api_error_status
+fast_mode_state fast_mode_disabled_reason result ttft_ms ttft_stream_ms time_to_request_ms type
+```
+
+`usage` contains:
+```json
+{"input_tokens":6,"cache_creation_input_tokens":5956,"cache_read_input_tokens":68601,
+ "output_tokens":608,"output_tokens_details":{"thinking_tokens":89},
+ "server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},
+ "service_tier":"standard",
+ "cache_creation":{"ephemeral_1h_input_tokens":5956,"ephemeral_5m_input_tokens":0},
+ "iterations":[ ... ]}
+```
+
+`modelUsage` is per-model: `inputTokens, outputTokens, cacheReadInputTokens,
+cacheCreationInputTokens, costUSD, contextWindow, maxOutputTokens, canonicalModel, provider`.
+
+### What it does NOT expose — this is the crux
+
+1. **No per-tool token attribution whatsoever.** There is no field anywhere that says how
+ many tokens a Grep or Read cost. The "40% on grep" number **cannot be read off this
+ JSON**. It must be reconstructed from the transcript, and that reconstruction is an
+ estimate. This is the single most important fact for honesty: the headline must rest on
+ something exact, not on the reconstructed share.
+2. **`input_tokens` is a trap.** It was `2` on a real run whose actual billed input was
+ ~24,000 tokens. Nearly everything lands in `cache_creation_input_tokens` and
+ `cache_read_input_tokens`. Quoting `input_tokens` would understate cost ~1000x.
+ Billed input = `input_tokens + cache_creation_input_tokens + cache_read_input_tokens`.
+3. **`usage.iterations` is not a full per-call log.** On a 3-API-call run it contained
+ **1** entry (the last). Per-call data must come from `--output-format stream-json
+ --verbose`, deduping `assistant` events by `message.id`.
+4. **Streamed `output_tokens` are partial.** Summing them gave **9** against a true **608**.
+ Output must be taken from the final `result` event. (`ledger.py` asserts this: input and
+ cache counters reconcile exactly, output is taken from `result`.)
+5. **A second model is billed.** Every run also invokes `claude-haiku-4-5` (~$0.001).
+ It is unrelated to retrieval strategy and is reported separately, never charged to an arm.
+
+---
+
+## 1. The questions (10, ground truth verified by reading the source)
+
+In `questions.jsonl`. Each carries the exact file, line, and the fact that makes an answer
+right or wrong. All were verified against the corpus before being written down.
+
+| id | class | asks | ground truth (verified) |
+|---|---|---|---|
+| q01_block_size | prior_hostile | `ForUtil.BLOCK_SIZE` in the *current* default postings format | **256**, `lucene104/ForUtil.java:34` — five backward-codec copies say 128 |
+| q02_hnsw_level | mechanism | how HNSW picks a new node's level | `ml = M==1?1:1/Math.log(1.0*M)` :183; `(int)(-log(u)*ml)` :536-541 |
+| q03_mmap_chunk | prior_hostile | `DEFAULT_MAX_CHUNK_SIZE` in MMapDirectory | not a constant: `JRE_IS_64BIT ? (1L<<34) : (1L<<28)` :185 |
+| q04_merge_score | mechanism | how TieredMergePolicy scores a merge | `mergeScore=skew`, `*= pow(totAfterMergeBytes, 0.05)` :713-717; `deletesPctAllowed=20.0` :82 |
+| q05_hnsw_limits | lookup | hard caps in Lucene99HnswVectorsFormat | `MAXIMUM_MAX_CONN=512` :99, `MAXIMUM_BEAM_WIDTH=3200` :109, `HNSW_GRAPH_THRESHOLD=100` :134 |
+| q06_bm25_k3 | prior_hostile | query-side saturation param in BM25Similarity | **`k3` exists**, default `-1f` (disabled), `((k3+1)*qtf)/(k3+qtf)` :51,122,126 |
+| q07_iwc_flush | lookup | default flush triggers | `DEFAULT_RAM_BUFFER_SIZE_MB=16.0` :83; `DEFAULT_MAX_BUFFERED_DOCS=DISABLE_AUTO_FLUSH` :78 |
+| q08_osq_params | lookup | OptimizedScalarQuantizer defaults | `DEFAULT_LAMBDA=0.1f` :57, `DEFAULT_ITERS=5` :59, `MINIMUM_MSE_GRID` :45 |
+| q09_exhaustive_count | **index_hostile** | count files containing `implements Accountable` | **43** (`grep -rl ... \| wc -l`) |
+| q10_tragic_event | multi_hop | which method records a tragic exception, per call path | `tragicEvent(...)` :1869 vs `onTragicEvent(...)` :713 — two *different* methods |
+
+**Why these and not trivia.** Every one is a question an engineer actually asks before
+changing indexing or vector-search behaviour: what block size will my postings use, how do I
+size an HNSW graph, why did my merge not fire, what flushes my writer, why is my quantizer
+lossy. None can be answered by reading a README.
+
+**Three question classes exist on purpose:**
+
+- `prior_hostile` (q01, q03, q06) — the checkout **contradicts** what a model remembers about
+ Lucene. q01 is 256 where the world says 128. q06's `k3` does not exist in upstream Lucene
+ at all. An agent that answers from memory scores **zero** here. These exist because of a
+ problem found while building this (section 4.1).
+- `index_hostile` (q09) — deliberately built to make XERJ lose. Exhaustive counting across a
+ tree is exactly what top-k retrieval cannot do and what grep does perfectly. It is in the
+ set so the result can be reported against us, following the precedent in
+ `docs/research/wordpress-verification-and-xerj-vs-grep.md`, where XERJ's call graph was
+ found **worse** than grep for reachability with silent false negatives.
+- `lookup` / `mechanism` / `multi_hop` — the ordinary middle.
+
+---
+
+## 2. The arms
+
+| arm | tools | cwd | what it represents |
+|---|---|---|---|
+| `native` | `Grep, Glob, Read` (Bash **denied**) | the real checkout | what a Claude Code user has today |
+| `xerj` | `Bash(xq:*)` only; Grep/Glob/Read **denied** | an **empty** dir | can the index alone answer it? |
+| `hybrid` | `Grep, Glob, Read, Bash(xq:*)` | the real checkout | the realistic product experience |
+
+`xerj` runs in an empty directory so it physically cannot fall back to reading the tree.
+`hybrid` is the fairest head-to-head (both sides agentic, each picks its own strategy) and is
+the number that should anchor a public post; `xerj` is the strict floor.
+
+**Both arms are agentic and write their own queries.** Neither gets a human-tuned query. `xq`
+(the only retrieval affordance) is deliberately generic — one fixed ES template, no
+per-question logic. If `xq` contained question-specific tuning the benchmark would be rigged.
+
+### Exact commands
+
+```bash
+cd outreach/bench
+./calibrate.sh # re-derive chars/token for this corpus
+REPEATS=3 ./run.sh all all # 3 arms x 10 questions x 3 repeats = 90 runs
+python3 analyze.py # emits judge_queue.jsonl
+# blind-grade judge_queue.jsonl -> judge_verdicts.jsonl, then:
+python3 analyze.py
+```
+
+Under the hood, per arm (verbatim from `run.sh`):
+
+```bash
+# native
+cd /home/claude/.xerj-code/corpora/lucene-meta/lucene && claude -p "$RULES\n\nQUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Grep,Glob,Read" --disallowedTools "Bash,WebSearch,WebFetch,Task"
+
+# xerj
+cd outreach/bench/.sandbox && claude -p "$RULES ... xq \"<query>\" [k] ...\n\nQUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Bash(xq:*)" --disallowedTools "Grep,Glob,Read,WebSearch,WebFetch,Task"
+
+# hybrid
+cd /home/claude/.xerj-code/corpora/lucene-meta/lucene && claude -p "$RULES ... xq ...\n\nQUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Grep,Glob,Read,Bash(xq:*)" --disallowedTools "WebSearch,WebFetch,Task"
+```
+
+Every run keeps its full raw `stream-json` transcript in `runs/`. Nothing is summarised at
+capture time, so any number in the writeup can be re-derived from the evidence.
+
+---
+
+## 3. What is measured, and how it is attributed
+
+**Primary (exact, from the CLI's own billing counters):**
+- `total_tokens_billed` = billed input + output.
+- **`tokens per CORRECT answer`** = total tokens spent by an arm / number it got right.
+ This is the headline. Tokens-per-*run* would reward an arm for giving up or hallucinating.
+- `cost_usd_cache_neutral` — see 4.2.
+
+**Secondary (estimated — label it as such in public):**
+- tool-payload share of tokens, via `len(tool_result) / 2.785`.
+ 2.785 chars/token was **measured** on this corpus by `calibrate.sh`: two prompts differing
+ by 20,000 chars of real Lucene source differed by 7,181 billed tokens. The usual chars/4
+ rule understates Java source by ~30%.
+- An alternative estimator — the `cache_creation` delta between consecutive calls — was
+ tried and **rejected**: with extended thinking on it is dominated by re-sent thinking-block
+ signatures, reading 1,613 tokens for 819 chars of grep output (~5x too high). Kept only as
+ a loose upper bound.
+
+Three different shares are reported because they answer different questions: share of unique
+context, share of *billed* tokens (a tool result pulled in at turn 2 is re-billed on every
+later call), and share of dollars (cache reads cost 10% of fresh input). **The user's "40%"
+is a billed/dollar-share style claim; say which one you mean.**
+
+`ledger.py` refuses to emit numbers it cannot reconcile against the CLI's own totals.
+
+---
+
+## 4. What would make this dishonest, and what stops it
+
+### 4.1 Giving the model the answer in the prompt
+Neither arm's prompt names a file, class, or constant beyond what the question asks. `xq` is
+generic. **Found while building this:** on an early probe the model answered a Lucene question
+with *zero tool calls*, straight from memory ("a codec header is `9 + codecName.length`
+bytes"). A benchmark of memorised Lucene trivia measures nothing about retrieval and would
+have shown the native arm using ~0 grep tokens. Mitigations: an explicit instruction to answer
+only from what is retrieved in-session, mandatory file:line citation, and three
+`prior_hostile` questions where memory gives a **wrong** answer, so parametric leakage is
+detectable rather than silent.
+
+### 4.2 Cache warmth — the biggest threat, and it is real
+On the pilot, `xerj` used **1.5x fewer tokens** but cost **2.4x more dollars**
+($0.2345 vs $0.0999). Cause: `native` ran against a warm prefix cache (57,945 cheap
+cache-*reads*) while `xerj` ran cold (21,693 expensive cache-*writes*). The arms have
+different prompt prefixes (different tool sets, different cwd) so they can *never* share a
+cache, and warmth depends on run order and a 5-min/1-hr TTL.
+
+Consequences, all applied: **raw dollars are not a fair cross-arm metric.** Report tokens, and
+report `cost_usd_cache_neutral` (every input token priced as fresh input), which removes
+warmth luck. On the pilot the cache-neutral ratio (1.52x) matches the token ratio (1.53x),
+while raw dollars invert. Also: interleave arms, repeat 3x, use medians, discard warm-ups.
+
+The price model was **derived, not assumed** — solving the CLI's reported `costUSD` against
+its own counters recovered exactly $5.00/Mtok input and $25.00/Mtok output. That also caught
+an error: these runs use the **1-hour** cache (2.0x write multiplier), not the 5-minute cache
+(1.25x). Assuming 1.25x produced a *negative* output price.
+
+### 4.3 Cherry-picking questions that suit an index
+`q09` exists solely to make XERJ lose, and results are reported per question class so a win
+driven entirely by one class is visible. Ground truth was written **before** any arm ran.
+Questions are never dropped after seeing results; `results.json` keeps every run.
+
+### 4.4 Letting the native arm behave unlike a real user
+The native arm gets exactly `Grep, Glob, Read` in the real checkout — no more, no less. It is
+not handicapped (not forced to read whole files) and not helped (no Bash, no pre-seeded paths).
+Note `xq` output is capped at 1,600 chars/hit so the index arm cannot dump the corpus either.
+
+### 4.5 Grading the arm you like more generously
+Grading is blind: `analyze.py` writes `judge_queue.jsonl` with arm labels stripped and rows
+shuffled under a fixed seed. Mechanical string checks can only mark a run FAIL or
+NEEDS-JUDGE — never correct — because containing "256" is not the same as asserting it.
+
+### 4.6 Ignoring that retrieval can actively mislead
+Real, observed: for q01 `xq` returned an irrelevant `CharBlockArray` first, and the **wrong**
+`ForUtil` (backward-codecs lucene103, `BLOCK_SIZE = 128`) second, with the correct lucene104
+one third. The index contains backward-codecs and test near-duplicates of many classes. If
+that misleads the model in the full run, it must be reported.
+
+### 4.7 The indexing cost the post claims is free
+"no tokens used for indexing" is true only of *LLM* tokens. Index build is CPU/disk/wall time
+over 6,012 files, plus 75,578 docs kept live. If the post says "free", it should say **"no LLM
+tokens"**, not "free".
+
+---
+
+## 5. What the pilot already shows (PILOT — n=1 question, 1 repeat/arm)
+
+Both arms answered q01 **correctly (256)**.
+
+| | native | xerj |
+|---|---|---|
+| total tokens billed | 64,487 | 42,068 |
+| tool calls | 2 (Glob+Grep) | 1 (`xq`) |
+| est. tool-output tokens | **145** | 1,072 |
+| tool share of billed tokens | **0.5%** | 5.1% |
+| cache-neutral cost | $0.3283 | $0.2153 |
+| raw cost | $0.0999 | $0.2345 |
+
+**This does not support "40% wasted on grep."** On a well-targeted question the native arm
+spent ~145 tokens on grep output — **0.5%** of its billed tokens. The dominant cost in both
+arms is the **fixed ~24k-token system prompt and tool definitions, re-billed on every API
+call** (3 calls native vs 2 xerj). XERJ's token win here (1.53x) comes mostly from finishing
+in **fewer turns**, not from grep output being large.
+
+Interpretation to carry into the writeup: any large win is likely to come from *turn
+reduction*, and any large "grep share" will only appear on exploration-heavy questions where
+the agent hunts across many files. Whether that is 40%, 18%, or 3% is exactly what the full
+matrix is for.
+
+---
+
+## 6. What would DISPROVE the user's claim
+
+The claim has two independent halves. Either can fail on its own.
+
+**Half A — "40% of tokens go to grep/sed/read."** Disproved if the median estimated
+tool-output share of billed tokens in the `native` arm is **materially below 40%** (say <25%)
+across the 10 questions. *The pilot measured 0.5% on q01, so this half is already in serious
+trouble and the number in the post must change.* It is confirmed only if the median lands
+near 40%.
+
+**Half B — "XERJ reduces it many times."** Disproved if any of:
+- median **tokens per correct answer** for `hybrid`/`xerj` is not meaningfully below `native`
+ (a ratio under ~1.3x does not survive being called "many times");
+- XERJ wins tokens but loses **accuracy** — cheaper wrong answers are not a win;
+- the win exists in raw dollars only via cache luck and vanishes under `cost_usd_cache_neutral`;
+- the win is carried by one question class and reverses on others (q09 is expected to go
+ against XERJ; if it loses on `mechanism` and `multi_hop` too, there is no story).
+
+**Pre-committed reporting rule.** Whatever the measured numbers are, they go in the post
+unchanged, including q09 if XERJ loses it, and including a corrected figure for the 40%.
+If Half A collapses but Half B holds, the honest post is *"grep isn't where your tokens go —
+turns are; retrieval cut tokens-per-correct-answer N x"*, which is a **better** and more
+defensible claim than the original.
+
+---
+
+## Files
+
+- `questions.jsonl` — 10 questions + verified ground truth + grader notes
+- `run.sh` — runs the arms, keeps raw transcripts
+- `xq` — the only retrieval affordance (intentionally generic)
+- `ledger.py` — transcript to auditable token ledger; reconciles or refuses
+- `analyze.py` — blind grading + medians + per-class breakdown
+- `calibrate.sh` — measures chars/token for this corpus
+- `runs/`, `logs/`, `results.json`
diff --git a/benchmarks/grep-vs-retrieval-lucene/analyze.py b/benchmarks/grep-vs-retrieval-lucene/analyze.py
new file mode 100644
index 000000000..1d1ff6558
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/analyze.py
@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+"""
+analyze.py -- grade the runs and produce the head-to-head table.
+
+GRADING IS TWO-STAGE AND BLIND.
+Stage 1 (mechanical): must_match / must_not_match string checks from questions.jsonl.
+ Cheap, deterministic, auditable. It can only mark a run FAIL-certain or NEEDS-JUDGE;
+ it never marks a run correct on its own, because substring presence is not
+ comprehension (an answer can contain "256" while asserting the wrong thing).
+Stage 2 (blind judge): every NEEDS-JUDGE answer is written to judge_queue.jsonl with
+ the arm label STRIPPED and the runs shuffled, so whoever/whatever grades cannot see
+ which arm produced which answer. Verdicts are read back from judge_verdicts.jsonl.
+
+The headline metric is TOKENS PER CORRECT ANSWER, not tokens per run. An arm that is
+cheap because it gives up or hallucinates must not win. Runs graded wrong contribute
+their cost to the arm but no credit, which is the only way "N x fewer tokens" means
+anything.
+
+Medians, not means: a single runaway agentic run would otherwise decide the headline.
+"""
+import json
+import os
+import random
+import statistics
+import sys
+
+BENCH = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, BENCH)
+from ledger import build # noqa: E402
+
+ARMS = ("native", "xerj", "hybrid")
+
+
+def questions():
+ return {json.loads(l)["id"]: json.loads(l) for l in open(f"{BENCH}/questions.jsonl")}
+
+
+def mechanical(ans, q):
+ """Return 'FAIL' (certainly wrong) or 'JUDGE' (needs blind grading)."""
+ a = ans.lower()
+ for bad in q.get("must_not_match", []):
+ # a forbidden token is only fatal if the required one is absent
+ if bad.lower() in a and not any(m.lower() in a for m in q["must_match"]):
+ return "FAIL"
+ if not all(m.lower() in a for m in q["must_match"]):
+ return "FAIL"
+ if "not found" in a:
+ return "FAIL"
+ return "JUDGE"
+
+
+def collect():
+ qs = questions()
+ rows = []
+ for fn in sorted(os.listdir(f"{BENCH}/runs")):
+ if not fn.endswith(".jsonl"):
+ continue
+ parts = fn[:-6].split("__")
+ if len(parts) != 3:
+ continue
+ arm, qid, rep = parts
+ if qid not in qs:
+ continue
+ try:
+ led = build(f"{BENCH}/runs/{fn}")
+ except SystemExit as e:
+ print(f"WARN {fn}: {e}", file=sys.stderr)
+ continue
+ if not led["reconciled"]:
+ print(f"WARN {fn}: ledger did not reconcile {led['mismatches']}", file=sys.stderr)
+ rows.append({
+ "arm": arm, "qid": qid, "rep": rep, "class": qs[qid]["class"],
+ "tokens": led["total_tokens_billed"],
+ "cost_neutral": led["cost_usd_cache_neutral"],
+ "cost_raw": led["cost_usd_total"],
+ "tool_calls": led["tool_calls"],
+ "api_calls": led["api_calls"],
+ "tool_tokens": led["tool_tokens_est"],
+ "share_billed": led["share_of_billed_pct"],
+ "answer": led["answer"],
+ "mech": mechanical(led["answer"], qs[qid]),
+ })
+ return rows, qs
+
+
+def emit_judge_queue(rows, qs):
+ """Write blind grading tasks: arm label removed, order shuffled."""
+ tasks = []
+ for i, r in enumerate(rows):
+ if r["mech"] != "JUDGE":
+ continue
+ tasks.append({
+ "key": f"{r['arm']}__{r['qid']}__{r['rep']}", # opaque to the judge if hidden
+ "qid": r["qid"],
+ "question": qs[r["qid"]]["ask"],
+ "ground_truth": qs[r["qid"]]["truth_fact"],
+ "grader_note": qs[r["qid"]]["grader_note"],
+ "answer": r["answer"],
+ })
+ random.Random(20260818).shuffle(tasks)
+ with open(f"{BENCH}/judge_queue.jsonl", "w") as fh:
+ for t in tasks:
+ # the judge is shown question/truth/answer only; 'key' is for merge-back
+ fh.write(json.dumps(t) + "\n")
+ return len(tasks)
+
+
+def verdicts():
+ p = f"{BENCH}/judge_verdicts.jsonl"
+ if not os.path.exists(p):
+ return {}
+ out = {}
+ for l in open(p):
+ d = json.loads(l)
+ out[d["key"]] = d["correct"]
+ return out
+
+
+def main():
+ rows, qs = collect()
+ if not rows:
+ raise SystemExit("no runs found -- run ./run.sh first")
+ n = emit_judge_queue(rows, qs)
+ v = verdicts()
+
+ for r in rows:
+ key = f"{r['arm']}__{r['qid']}__{r['rep']}"
+ r["correct"] = False if r["mech"] == "FAIL" else v.get(key)
+
+ ungraded = [r for r in rows if r["correct"] is None]
+ print(f"runs={len(rows)} queued_for_blind_judging={n} ungraded={len(ungraded)}")
+ if ungraded:
+ print("-> grade judge_queue.jsonl into judge_verdicts.jsonl "
+ '({"key":...,"correct":true|false}) then re-run. '
+ "Numbers below treat ungraded as UNKNOWN and exclude them.\n")
+
+ graded = [r for r in rows if r["correct"] is not None]
+
+ print(f"{'arm':8s} {'n':>3s} {'acc':>6s} {'med_tok':>9s} {'med_$neut':>10s} "
+ f"{'tok/correct':>12s} {'med_tools':>9s} {'tool_share%':>11s}")
+ per_arm = {}
+ for arm in ARMS:
+ g = [r for r in graded if r["arm"] == arm]
+ if not g:
+ continue
+ ok = [r for r in g if r["correct"]]
+ acc = len(ok) / len(g)
+ med_tok = statistics.median(r["tokens"] for r in g)
+ med_cost = statistics.median(r["cost_neutral"] for r in g)
+ # tokens per correct answer: total spend by the arm / number it got right
+ tpc = (sum(r["tokens"] for r in g) / len(ok)) if ok else float("inf")
+ med_tools = statistics.median(r["tool_calls"] for r in g)
+ shares = [r["share_billed"] for r in g if r["share_billed"] is not None]
+ med_share = statistics.median(shares) if shares else 0
+ per_arm[arm] = {"acc": acc, "med_tok": med_tok, "tpc": tpc,
+ "med_cost": med_cost, "n": len(g)}
+ print(f"{arm:8s} {len(g):3d} {acc:6.0%} {med_tok:9.0f} {med_cost:10.4f} "
+ f"{tpc:12.0f} {med_tools:9.1f} {med_share:11.1f}")
+
+ if "native" in per_arm:
+ base = per_arm["native"]
+ print("\nHEADLINE (vs native, tokens per CORRECT answer):")
+ for arm in ("xerj", "hybrid"):
+ if arm in per_arm and per_arm[arm]["tpc"]:
+ print(f" {arm:7s} {base['tpc']/per_arm[arm]['tpc']:.2f}x "
+ f"(accuracy {per_arm[arm]['acc']:.0%} vs {base['acc']:.0%})")
+
+ print("\nPER QUESTION CLASS (median tokens; accuracy):")
+ classes = sorted({r["class"] for r in graded})
+ print(f"{'class':16s} " + " ".join(f"{a:>18s}" for a in ARMS))
+ for c in classes:
+ cells = []
+ for arm in ARMS:
+ g = [r for r in graded if r["class"] == c and r["arm"] == arm]
+ if not g:
+ cells.append(f"{'-':>18s}"); continue
+ m = statistics.median(r["tokens"] for r in g)
+ a = sum(1 for r in g if r["correct"]) / len(g)
+ cells.append(f"{m:11.0f}/{a:5.0%}")
+ print(f"{c:16s} " + " ".join(cells))
+
+ print("\nPER QUESTION (median tokens by arm, and who was correct):")
+ for qid in [q for q in questions()]:
+ line = f" {qid:22s}"
+ for arm in ARMS:
+ g = [r for r in graded if r["qid"] == qid and r["arm"] == arm]
+ if not g:
+ line += f" {arm}:- "
+ continue
+ m = statistics.median(r["tokens"] for r in g)
+ a = sum(1 for r in g if r["correct"]) / len(g)
+ line += f" {arm}:{m:6.0f}/{a:3.0%}"
+ print(line)
+
+ with open(f"{BENCH}/results.json", "w") as fh:
+ json.dump({"rows": rows, "per_arm": per_arm}, fh, indent=1)
+ print("\nwrote results.json")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/grep-vs-retrieval-lucene/attribute.py b/benchmarks/grep-vs-retrieval-lucene/attribute.py
new file mode 100755
index 000000000..8a68ef5f6
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/attribute.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Per-tool token attribution from the Claude Code transcript.
+
+WHY NOT THE RESULT JSON: `claude -p --output-format json` returns ONLY a
+session-level aggregate. Verified on this machine: a 2-turn run reported
+num_turns=2 but usage.iterations had length 1, and iterations[0].cache_creation
+(7907) != usage.cache_creation_input_tokens (13341). The iterations array is
+INCOMPLETE and must not be used for attribution. There is NO per-tool field.
+
+WHAT WE USE INSTEAD: the transcript JSONL, where every assistant row carries its
+own usage and every user row carries the tool_result content. Between API call i
+and call i+1 the only things added to the context are (a) call i's own output and
+(b) the tool_results for call i's tool_uses. So:
+
+ prompt_i = input_tokens + cache_creation_input_tokens + cache_read_input_tokens
+ ingested_i = prompt_{i+1} - prompt_i - output_tokens_i
+
+`ingested_i` is the provider's own billing meter measuring the tool output, so it
+needs no tokenizer and no estimate. If call i issued several tools, ingested_i is
+split across them by tool_result character share (the only approximation here; it
+is reported, and cells where a single call issued >1 tool are flagged).
+"""
+import json, sys, collections
+
+SEARCH_TOOLS = {"Grep", "Glob", "Read", "NotebookRead"}
+SEARCH_BASH = ("grep", "rg", "sed", "awk", "cat", "head", "tail", "find", "ls", "wc")
+
+def bash_is_search(inp):
+ cmd = (inp or {}).get("command", "")
+ first = cmd.strip().split()[0] if cmd.strip() else ""
+ return any(first.endswith(b) for b in SEARCH_BASH) or any(
+ (" %s " % b) in cmd or ("|%s " % b) in cmd or ("| %s " % b) in cmd for b in SEARCH_BASH)
+
+def load(path):
+ return [json.loads(l) for l in open(path) if l.strip()]
+
+def analyze(path):
+ rows = load(path)
+ calls = [] # one per assistant API call
+ pending = None
+ for r in rows:
+ m = r.get("message") or {}
+ if r.get("type") == "assistant" and m.get("usage"):
+ u = m["usage"]
+ prompt = (u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0)
+ + u.get("cache_read_input_tokens", 0))
+ tools = []
+ for c in (m.get("content") or []):
+ if isinstance(c, dict) and c.get("type") == "tool_use":
+ tools.append({"name": c.get("name"), "input": c.get("input"),
+ "id": c.get("id")})
+ # consecutive assistant rows can share one usage object (thinking + tool_use
+ # arrive as separate rows from the SAME API call); merge on identical usage
+ if calls and calls[-1]["usage_sig"] == json.dumps(u, sort_keys=True):
+ calls[-1]["tools"].extend(tools)
+ continue
+ calls.append({"prompt": prompt, "out": u.get("output_tokens", 0),
+ "tools": tools, "usage_sig": json.dumps(u, sort_keys=True),
+ "results": {}})
+ elif r.get("type") == "user":
+ cont = m.get("content") or []
+ if not isinstance(cont, list):
+ cont = []
+ for c in cont:
+ if isinstance(c, dict) and c.get("type") == "tool_result" and calls:
+ calls[-1]["results"][c.get("tool_use_id")] = len(json.dumps(c.get("content")))
+
+ per_tool = collections.Counter()
+ multi_tool_calls = 0
+ for i in range(len(calls) - 1):
+ ingested = calls[i + 1]["prompt"] - calls[i]["prompt"] - calls[i]["out"]
+ if ingested <= 0:
+ continue
+ tools = calls[i]["tools"]
+ if not tools:
+ per_tool["_nontool"] += ingested
+ continue
+ if len(tools) > 1:
+ multi_tool_calls += 1
+ chars = {t["id"]: max(1, calls[i]["results"].get(t["id"], 1)) for t in tools}
+ tot = sum(chars.values())
+ for t in tools:
+ share = ingested * chars[t["id"]] / tot
+ name = t["name"]
+ if name == "Bash":
+ name = "Bash(search)" if bash_is_search(t["input"]) else "Bash(other)"
+ per_tool[name] += share
+
+ search_tokens = sum(v for k, v in per_tool.items()
+ if k in SEARCH_TOOLS or k == "Bash(search)")
+ final_prompt = calls[-1]["prompt"] if calls else 0
+ billed_prompt = sum(c["prompt"] for c in calls)
+ return {
+ "n_api_calls": len(calls),
+ "per_tool_tokens": {k: round(v) for k, v in per_tool.items()},
+ "search_tool_tokens": round(search_tokens),
+ "final_context_tokens": final_prompt,
+ "billed_prompt_tokens": billed_prompt,
+ "output_tokens": sum(c["out"] for c in calls),
+ # (A) headline: fraction of the agent's final context that is search output
+ "share_of_context": round(search_tokens / final_prompt, 4) if final_prompt else None,
+ # (B) cost view: search tokens re-charged on every later call
+ "share_of_billed": round(
+ sum(min(search_tokens, c["prompt"]) for c in calls) / billed_prompt, 4)
+ if billed_prompt else None,
+ "multi_tool_calls_approximated": multi_tool_calls,
+ }
+
+if __name__ == "__main__":
+ print(json.dumps(analyze(sys.argv[1]), indent=1))
diff --git a/benchmarks/grep-vs-retrieval-lucene/build_scales.py b/benchmarks/grep-vs-retrieval-lucene/build_scales.py
new file mode 100644
index 000000000..d04e801f9
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/build_scales.py
@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+"""Build nested corpus scales S1<S2<S3<S4 from the Lucene tree.
+
+Exactly one variable changes across scales: the NUMBER OF FILES visible.
+The 9 answer-bearing files are pinned into every scale, so every question is
+answerable at every scale. Distractors are sampled with a fixed seed from the
+real tree (not synthesized), so decoy density at S4 is the true density.
+"""
+import os, random, shutil, json, sys
+
+ROOT = "/home/claude/.xerj-code/corpora/lucene-meta/lucene"
+OUT = "/home/claude/tasks/speakers-FTS-vector-search/outreach/bench/scales"
+SEED = 20260814
+
+ANSWER_FILES = [
+ "lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java",
+ "lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java",
+ "lucene/core/src/java/org/apache/lucene/codecs/lucene90/IndexedDISI.java",
+ "lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java",
+ "lucene/core/src/java/org/apache/lucene/index/LeafMetaData.java",
+ "lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java",
+ "lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java",
+ "lucene/core/src/java/org/apache/lucene/index/IndexWriterConfig.java",
+ "lucene/core/src/java/org/apache/lucene/util/quantization/ScalarQuantizer.java",
+]
+SIZES = {"S1": 64, "S2": 256, "S3": 1024, "S4": None} # None = whole tree
+
+def main():
+ allj = []
+ for dp, dn, fn in os.walk(ROOT):
+ for f in fn:
+ if f.endswith(".java"):
+ allj.append(os.path.relpath(os.path.join(dp, f), ROOT))
+ allj.sort()
+ for a in ANSWER_FILES:
+ assert os.path.exists(os.path.join(ROOT, a)), "missing answer file: " + a
+ pool = [p for p in allj if p not in set(ANSWER_FILES)]
+ random.Random(SEED).shuffle(pool)
+
+ manifest = {}
+ prev = list(ANSWER_FILES)
+ for name in ["S1", "S2", "S3", "S4"]:
+ n = SIZES[name]
+ if n is None:
+ files = allj
+ else:
+ need = n - len(ANSWER_FILES)
+ files = sorted(set(ANSWER_FILES) | set(pool[:need]))
+ assert set(prev).issubset(set(files)), name + " is not a superset of the previous scale"
+ prev = files
+ dest = os.path.join(OUT, name)
+ if os.path.exists(dest):
+ shutil.rmtree(dest)
+ for rel in files:
+ d = os.path.join(dest, rel)
+ os.makedirs(os.path.dirname(d), exist_ok=True)
+ shutil.copy2(os.path.join(ROOT, rel), d)
+ manifest[name] = {"n_files": len(files), "dir": dest, "files": files}
+ print("%s: %d files -> %s" % (name, len(files), dest))
+ with open(os.path.join(OUT, "manifest.json"), "w") as fh:
+ json.dump({"seed": SEED, "root": ROOT, "answer_files": ANSWER_FILES,
+ "scales": {k: {"n_files": v["n_files"], "files": v["files"]}
+ for k, v in manifest.items()}}, fh, indent=1)
+ print("manifest written")
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/grep-vs-retrieval-lucene/calibrate.sh b/benchmarks/grep-vs-retrieval-lucene/calibrate.sh
new file mode 100755
index 000000000..647d94160
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/calibrate.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Measures chars-per-token for THIS corpus, by sending two identical prompts that
+# differ only by a known number of characters of real Lucene source and diffing the
+# cache_creation_input_tokens the CLI bills. No API key and no tokenizer needed.
+#
+# Result recorded 2026-08-18: 20000 chars -> 7181 tokens => 2.785 chars/token.
+# (The common chars/4 rule understates Java source token counts by ~30%.)
+set -euo pipefail
+L=/home/claude/.xerj-code/corpora/lucene-meta/lucene/lucene
+SRC=$L/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java
+TMP=$(mktemp -d)
+for n in 2000 22000; do
+ { printf 'Reply with only the word OK. Ignore this reference material:\n<ref>\n'
+ head -c $n "$SRC"; printf '\n</ref>\n'; } > "$TMP/p$n.txt"
+ claude -p "$(cat "$TMP/p$n.txt")" --output-format json \
+ | python3 -c "import sys,json;u=json.load(sys.stdin)['usage'];print($n,u['cache_creation_input_tokens'])"
+done
+echo "chars_per_token = (22000-2000) / (cc_22000 - cc_2000)"
diff --git a/benchmarks/grep-vs-retrieval-lucene/calibration.json b/benchmarks/grep-vs-retrieval-lucene/calibration.json
new file mode 100644
index 000000000..82f702643
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/calibration.json
@@ -0,0 +1,13 @@
+{
+ "method": "Two-point measurement against the CLI's own accounting. A payload of known char length is injected via --append-system-prompt; cache_creation_input_tokens is read from --output-format json and the no-payload baseline subtracted. Uses the real production tokenizer; no third-party tokenizer is involved.",
+ "baseline_cc_tokens": 6246,
+ "measured": [
+ {"category": "java_source", "chars": 20000, "cc": 14924, "delta_tokens": 8678, "chars_per_token": 2.305},
+ {"category": "java_source", "chars": 40004, "cc": 22812, "delta_tokens": 16566, "chars_per_token": 2.415},
+ {"category": "grep_output", "chars": 20000, "cc": 17118, "delta_tokens": 10872, "chars_per_token": 1.840},
+ {"category": "xerj_json", "chars": 13207, "cc": 11986, "delta_tokens": 5740, "chars_per_token": 2.301}
+ ],
+ "ratios_chars_per_token": {"java_source": 2.35, "grep_output": 1.84, "xerj_json": 2.30, "default": 2.20},
+ "linearity_check": "20k java predicts 17356 tokens at 40k; measured 16566 (-4.6%). Deviation is composition, not nonlinearity: the second 20k is IndexWriter.java, which is javadoc-heavy English (less dense than code).",
+ "finding": "grep output is 1.84 chars/token vs 2.35 for Java source - it is ~25% denser per character because file paths, colons and line numbers fragment into many tokens. Attributing grep output at a source-code ratio would UNDERCOUNT the cost of the search arm by about a quarter."
+}
diff --git a/benchmarks/grep-vs-retrieval-lucene/grade.py b/benchmarks/grep-vs-retrieval-lucene/grade.py
new file mode 100755
index 000000000..fdfcd2f9c
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/grade.py
@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+"""Deterministic oracle. No LLM judge: substring/regex only, case-insensitive."""
+import json, sys, os, re
+B = os.path.dirname(os.path.abspath(__file__))
+QS = {q["id"]: q for q in json.load(open(f"{B}/questions.json"))["questions"]}
+
+def grade(qid, answer):
+ q = QS[qid]; a = answer.lower()
+ missing = [e for e in q["expect"] if e.lower() not in a]
+ tripped = [f for f in q.get("forbid", []) if f.lower() in a]
+ return {"qid": qid, "correct": not missing and not tripped,
+ "missing_expected": missing, "tripped_forbidden": tripped}
+
+if __name__ == "__main__":
+ out = []
+ for p in sys.argv[1:]:
+ qid = os.path.basename(p).split("_A_")[0].split("_B_")[0].split("_C_")[0]
+ d = json.load(open(p))
+ out.append({**grade(qid, d.get("result", "")), "file": os.path.basename(p)})
+ print(json.dumps(out, indent=2))
diff --git a/benchmarks/grep-vs-retrieval-lucene/ledger.py b/benchmarks/grep-vs-retrieval-lucene/ledger.py
new file mode 100644
index 000000000..3eb9fa152
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/ledger.py
@@ -0,0 +1,254 @@
+#!/usr/bin/env python3
+"""
+ledger.py -- turn one `claude -p --output-format stream-json --verbose` transcript
+into an auditable token ledger.
+
+WHY THIS FILE EXISTS
+--------------------
+`claude -p --output-format json` gives you TOTALS only. It has NO per-tool token
+field. There is no "grep cost you N tokens" anywhere in the payload. So the claim
+"agents burn X% of tokens on grep/read" cannot be read off the JSON -- it has to be
+RECONSTRUCTED. This does that reconstruction, and refuses to emit numbers it cannot
+reconcile against the CLI's own billing totals.
+
+METHOD
+------
+Per-API-call usage is recovered from the stream's `assistant` events, deduped by
+message.id (each id emits multiple events during streaming; we keep the max of each
+counter, which is the final value for that message).
+
+Billed input for call i = input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
+
+Tool-result attribution is CHARACTER-BASED against a constant measured on this exact
+corpus by calibrate.sh: two prompts differing by 20,000 chars of real Lucene source
+differed by 7,181 billed tokens => 2.785 chars/token. (The usual chars/4 rule of thumb
+understates Java source by ~30%; using it would have understated grep cost.)
+
+ tool_tokens(i) = len(tool_result_i) / CHARS_PER_TOKEN
+
+An alternative estimator -- the cache_creation delta between consecutive calls -- was
+tried and REJECTED as the primary: with extended thinking on, that delta is dominated
+by re-sent thinking-block signature blobs, not by tool output (it read 1613 tokens for
+819 chars of grep output, ~5x too high). It is retained only as a loose UPPER BOUND.
+
+HONESTY BOUNDARY -- READ THIS BEFORE QUOTING ANY PERCENTAGE
+-----------------------------------------------------------
+The arm-vs-arm TOTALS (tokens, dollars) are EXACT: they come straight from the CLI's
+billing counters and reconcile to the token. The per-tool SHARE is an ESTIMATE built
+on the calibration above. So the headline claim must be the exact one
+("N x fewer tokens for the same answer"), and any "% spent on grep" must be quoted as
+an estimate with its method stated. Do not lead a public post with the estimate.
+
+TWO DIFFERENT SHARES ARE REPORTED, BECAUSE THEY ANSWER DIFFERENT QUESTIONS
+-------------------------------------------------------------------------
+unique_share : tool tokens / total unique context tokens.
+ "Of everything the agent read, how much was grep/read output?"
+billed_share : sum over calls of (tool tokens resident in that call's context)
+ / total billed input tokens.
+ "Of what you actually PAID for, how much was grep/read output?"
+ Higher than unique_share, because a tool result pulled in at turn 2 is
+ re-billed (at cache-read rates) on every later call.
+dollar_share : same as billed_share but weighted by the real price of each token
+ class, since cache reads are ~10% the price of fresh input.
+
+The user's "40%" is a billed_share/dollar_share style claim. Report which one you mean.
+"""
+
+import json
+import sys
+import os
+
+# Measured on this corpus by calibrate.sh (2026-08-18): 20000 chars -> 7181 tokens.
+CHARS_PER_TOKEN = 2.785
+
+# Price model DERIVED (not assumed) by solving the CLI's own reported costUSD against
+# its own token counters across two real runs; it recovered exactly $5.00/Mtok input and
+# $25.00/Mtok output, confirming the multipliers below. The runs use the 1-HOUR cache
+# (usage.cache_creation.ephemeral_1h_input_tokens), whose write multiplier is 2.0x, NOT
+# the 1.25x of the 5-minute cache. Assuming 1.25x yields a negative output price, which
+# is how the error was caught.
+USD_PER_INPUT_TOK = 5.00 / 1e6
+USD_PER_OUTPUT_TOK = 25.00 / 1e6
+CACHE_WRITE_MULT = 2.0
+CACHE_READ_MULT = 0.10
+
+
+def load(path):
+ """Parse a stream-json transcript into (per_call_usage, tool_results, result_event)."""
+ calls = {} # message.id -> usage dict (max of each counter seen)
+ order = [] # message ids in first-seen order
+ tools = [] # {'after_call': idx, 'name': str, 'chars': int}
+ pending_tool_names = {} # tool_use_id -> tool name
+ result = None
+
+ with open(path) as fh:
+ for line in fh:
+ line = line.strip()
+ if not line:
+ continue
+ try:
+ ev = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+
+ t = ev.get("type")
+
+ if t == "assistant":
+ msg = ev.get("message", {})
+ mid = msg.get("id")
+ if not mid:
+ continue
+ if mid not in calls:
+ calls[mid] = {"input_tokens": 0, "output_tokens": 0,
+ "cache_creation_input_tokens": 0,
+ "cache_read_input_tokens": 0,
+ "model": msg.get("model")}
+ order.append(mid)
+ u = msg.get("usage") or {}
+ for k in ("input_tokens", "output_tokens",
+ "cache_creation_input_tokens", "cache_read_input_tokens"):
+ if k in u and u[k] is not None:
+ calls[mid][k] = max(calls[mid][k], u[k])
+ for c in msg.get("content", []):
+ if c.get("type") == "tool_use":
+ pending_tool_names[c.get("id")] = c.get("name")
+
+ elif t == "user":
+ content = ev.get("message", {}).get("content")
+ if isinstance(content, list):
+ for c in content:
+ if c.get("type") == "tool_result":
+ payload = c.get("content")
+ text = payload if isinstance(payload, str) else json.dumps(payload)
+ tools.append({
+ "after_call": len(order) - 1,
+ "name": pending_tool_names.get(c.get("tool_use_id"), "?"),
+ "chars": len(text),
+ })
+
+ elif t == "result":
+ result = ev
+
+ seq = [dict(calls[m], id=m) for m in order]
+ return seq, tools, result
+
+
+def build(path, main_model_hint="opus"):
+ seq, tools, result = load(path)
+ if result is None:
+ raise SystemExit(f"{path}: no result event -- run did not complete")
+
+ # --- reconcile our reconstructed per-call ledger against the CLI's own totals ---
+ # NOTE: output_tokens in streamed assistant events are PARTIAL (streaming deltas)
+ # and do not sum to the true total, so output is taken from the authoritative
+ # result event. Input/cache counters DO reconcile exactly and are asserted.
+ tot = result.get("usage", {})
+ recon = {
+ "input_tokens": sum(c["input_tokens"] for c in seq),
+ "cache_creation_input_tokens": sum(c["cache_creation_input_tokens"] for c in seq),
+ "cache_read_input_tokens": sum(c["cache_read_input_tokens"] for c in seq),
+ }
+ mismatches = {k: (recon[k], tot.get(k)) for k in recon if recon[k] != tot.get(k)}
+ recon["output_tokens"] = tot.get("output_tokens", 0) # authoritative
+
+ # --- isolate the main model; the CLI also bills a haiku sidecar we must not blame
+ # on either arm's retrieval strategy ---
+ mu = result.get("modelUsage", {})
+ main = {k: v for k, v in mu.items() if main_model_hint in k}
+ side = {k: v for k, v in mu.items() if main_model_hint not in k}
+ main_cost = sum(v.get("costUSD", 0) for v in main.values())
+ side_cost = sum(v.get("costUSD", 0) for v in side.values())
+
+ # --- tool token attribution: calibrated char-based (primary) ---
+ tool_tokens_by_call = {}
+ for tr in tools:
+ i = tr["after_call"]
+ tool_tokens_by_call[i] = tool_tokens_by_call.get(i, 0) + tr["chars"] / CHARS_PER_TOKEN
+ tool_tokens_by_call = {i: int(round(v)) for i, v in tool_tokens_by_call.items()}
+
+ tool_tokens_total = sum(tool_tokens_by_call.values())
+ tool_chars_total = sum(t["chars"] for t in tools)
+
+ # loose upper bound from cache_creation deltas (contaminated by thinking blobs)
+ ub = 0
+ for i in tool_tokens_by_call:
+ if i + 1 < len(seq):
+ ub += max(0, seq[i + 1]["cache_creation_input_tokens"])
+
+ # --- shares ---
+ unique_ctx = recon["input_tokens"] + recon["cache_creation_input_tokens"]
+ billed_in = unique_ctx + recon["cache_read_input_tokens"]
+
+ # billed exposure: tokens introduced at call i are re-billed on every later call
+ billed_exposure = 0
+ for i, n in tool_tokens_by_call.items():
+ later_calls = len(seq) - (i + 1)
+ billed_exposure += n * max(0, later_calls)
+
+ # dollar weighting: cache read ~0.1x, cache write ~1.25x of base input
+ PRICE = {"fresh": 1.0, "write": CACHE_WRITE_MULT, "read": CACHE_READ_MULT}
+ dollar_units_total = (recon["input_tokens"] * PRICE["fresh"]
+ + recon["cache_creation_input_tokens"] * PRICE["write"]
+ + recon["cache_read_input_tokens"] * PRICE["read"])
+ dollar_units_tool = 0.0
+ for i, n in tool_tokens_by_call.items():
+ dollar_units_tool += n * PRICE["write"]
+ dollar_units_tool += n * max(0, len(seq) - (i + 1)) * PRICE["read"]
+
+ def pct(a, b):
+ return round(100.0 * a / b, 1) if b else None
+
+ return {
+ "file": os.path.basename(path),
+ "ok": result.get("is_error") is False and result.get("subtype") == "success",
+ "answer": result.get("result", ""),
+ "num_turns": result.get("num_turns"),
+ "api_calls": len(seq),
+ "tool_calls": len(tools),
+ "tool_breakdown": _by_name(tools),
+ "duration_ms": result.get("duration_ms"),
+ "totals": recon,
+ "cli_totals": {k: tot.get(k) for k in recon},
+ "reconciled": not mismatches,
+ "mismatches": mismatches,
+ "billed_input_tokens": billed_in,
+ "unique_context_tokens": unique_ctx,
+ "output_tokens": recon["output_tokens"],
+ "total_tokens_billed": billed_in + recon["output_tokens"],
+ "cost_usd_total": result.get("total_cost_usd"),
+ # Cache-neutral cost: every input token priced as fresh input. This removes
+ # cache-warmth luck, which otherwise dominates arm-vs-arm dollar comparisons
+ # (an arm that happens to run against a cold cache pays 2.0x on writes while a
+ # warm arm pays 0.1x on reads -- observed as a 2.4x cost inversion on q01 even
+ # though the cheaper-looking arm used MORE tokens). Use this for fair $ claims.
+ "cost_usd_cache_neutral": round(billed_in * USD_PER_INPUT_TOK
+ + recon["output_tokens"] * USD_PER_OUTPUT_TOK, 6),
+ "cache_write_tokens": recon["cache_creation_input_tokens"],
+ "cache_read_tokens": recon["cache_read_input_tokens"],
+ "cost_usd_main_model": round(main_cost, 6),
+ "cost_usd_sidecar": round(side_cost, 6),
+ "sidecar_models": list(side.keys()),
+ "tool_tokens_est": tool_tokens_total,
+ "tool_tokens_upper_bound": ub,
+ "tool_chars": tool_chars_total,
+ "chars_per_token_used": CHARS_PER_TOKEN,
+ "share_of_unique_ctx_pct": pct(tool_tokens_total, unique_ctx),
+ "share_of_billed_pct": pct(tool_tokens_total + billed_exposure, billed_in),
+ "share_of_dollars_pct": pct(dollar_units_tool, dollar_units_total),
+ }
+
+
+def _by_name(tools):
+ out = {}
+ for t in tools:
+ d = out.setdefault(t["name"], {"calls": 0, "chars": 0})
+ d["calls"] += 1
+ d["chars"] += t["chars"]
+ return out
+
+
+if __name__ == "__main__":
+ if len(sys.argv) < 2:
+ raise SystemExit("usage: ledger.py <transcript.jsonl> [...]")
+ for p in sys.argv[1:]:
+ print(json.dumps(build(p)))
diff --git a/benchmarks/grep-vs-retrieval-lucene/questions.json b/benchmarks/grep-vs-retrieval-lucene/questions.json
new file mode 100644
index 000000000..67d267376
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/questions.json
@@ -0,0 +1,112 @@
+{
+ "corpus": "/home/claude/.xerj-code/corpora/lucene-meta/lucene",
+ "corpus_java_files": 6012,
+ "index": "refsym",
+ "frozen": "2026-08-18",
+ "note": "Frozen BEFORE any arm was run. Oracles are deterministic string/regex checks, not LLM judgments. 'expect' = must appear; 'forbid' = marks a wrong/hallucinated answer.",
+ "questions": [
+ {
+ "id": "q1_hnsw_maxconn",
+ "type": "point-constant",
+ "favors": "retrieval",
+ "q": "When this codebase builds an HNSW graph, what is the default maximum number of connections per node? Give the constant name, its value, and the file that defines it.",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java",
+ "truth_line": 51,
+ "expect": ["DEFAULT_MAX_CONN", "16", "HnswGraphBuilder"],
+ "forbid": ["32", "64"]
+ },
+ {
+ "id": "q2_bm25_idf",
+ "type": "single-method-body",
+ "favors": "retrieval",
+ "q": "What exact formula does the default BM25 scoring implementation use to compute IDF from docFreq and docCount? Write the expression.",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java",
+ "truth_line": 105,
+ "expect": ["docCount - docFreq + 0.5", "docFreq + 0.5", "log"],
+ "forbid": ["docCount / docFreq"]
+ },
+ {
+ "id": "q3_max_term_len",
+ "type": "two-hop-constant",
+ "favors": "neutral",
+ "q": "What is the maximum length, in bytes, of a single indexed term, and how is that limit derived? Give the numeric value and the two constants involved.",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/index/IndexWriter.java",
+ "truth_line": 258,
+ "expect": ["MAX_TERM_LENGTH", "BYTE_BLOCK_SIZE", "32766"],
+ "forbid": ["32768 is the max term", "65535"],
+ "why": "Requires resolving MAX_TERM_LENGTH = BYTE_BLOCK_SIZE - 2 across two files (IndexWriter -> ByteBlockPool). Tests reference-chasing, where symbol-level top-k can stop one hop short."
+ },
+ {
+ "id": "q4_ram_buffer",
+ "type": "point-constant",
+ "favors": "retrieval",
+ "q": "What is the default RAM buffer size (in MB) that the index writer uses before flushing, and which constant holds it?",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/index/IndexWriterConfig.java",
+ "truth_line": 83,
+ "expect": ["DEFAULT_RAM_BUFFER_SIZE_MB", "16"],
+ "forbid": ["32", "64", "128"]
+ },
+ {
+ "id": "q5_fsdir_choice",
+ "type": "semantic-no-keyword",
+ "favors": "retrieval",
+ "q": "When you open a filesystem directory without naming a concrete implementation, what decides which implementation you actually get, and what are the two possible outcomes?",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/store/FSDirectory.java",
+ "truth_line": 155,
+ "expect": ["JRE_IS_64BIT", "MMapDirectory", "NIOFSDirectory"],
+ "forbid": ["SimpleFSDirectory", "operating system"],
+ "why": "The question deliberately contains none of the answer's identifiers. Tests semantic matching, not keyword grep."
+ },
+ {
+ "id": "q6_maxclause_locality",
+ "type": "misleading-locality",
+ "favors": "grep",
+ "q": "What is the default limit on the number of clauses a boolean query may contain, and in which class is that limit actually defined?",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java",
+ "truth_line": 80,
+ "expect": ["1024", "IndexSearcher"],
+ "forbid": ["defined in BooleanQuery", "BooleanQuery.java:83"],
+ "why": "Every engineer expects this in BooleanQuery; it lives in IndexSearcher. BooleanQuery only READS it. A retriever that returns the highest-scoring BooleanQuery hit answers with the wrong file."
+ },
+ {
+ "id": "q7_fsdir_subclasses",
+ "type": "exhaustive-enumeration",
+ "favors": "grep",
+ "q": "List every concrete non-test class in this repository that directly extends the abstract filesystem Directory base class. Give all of them.",
+ "truth_file": "multiple",
+ "expect": ["MMapDirectory", "NIOFSDirectory", "RAFDirectory"],
+ "forbid": ["CommandLineUtil"],
+ "why": "Exhaustiveness. A top-k retriever silently truncates and returns 2 of 3, with no signal that it was incomplete. RAFDirectory lives in lucene/misc, far from the other two."
+ },
+ {
+ "id": "q8_deletes_pct",
+ "type": "point-constant",
+ "favors": "retrieval",
+ "q": "In the default merge policy, what percentage of deleted documents is tolerated in the index before merges are triggered to reclaim them, and what is the field called?",
+ "truth_file": "lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java",
+ "truth_line": 82,
+ "expect": ["deletesPctAllowed", "20"],
+ "forbid": ["10.0 is the deletes", "50"]
+ },
+ {
+ "id": "q9_min_java",
+ "type": "outside-index-build-config",
+ "favors": "grep",
+ "q": "What is the minimum Java language version this project requires to build, and which file declares it?",
+ "truth_file": "gradle/libs.versions.toml",
+ "expect": ["25", "libs.versions.toml"],
+ "forbid": ["17", "21", "11"],
+ "why": "GUARANTEED RETRIEVAL LOSS, included deliberately. The index holds only parsed source symbols; .toml/.gradle files are not in it at all. Any honest post must report that the index cannot answer build-config questions."
+ },
+ {
+ "id": "q10_forcemerge_callers",
+ "type": "cross-file-reachability",
+ "favors": "grep",
+ "q": "Outside of tests and benchmarks, which classes in this repository call forceMerge on an index writer? Name them.",
+ "truth_file": "multiple",
+ "expect": ["IndexMergeTool", "SpellChecker", "DocumentBatch", "IndexUtils"],
+ "forbid": [],
+ "why": "Reachability. docs/research/wordpress-verification-and-xerj-vs-grep.md already found XERJ's structured call graph WORSE than grep here, with silent false negatives. Included so that precedent is re-tested, not dodged."
+ }
+ ]
+}
diff --git a/benchmarks/grep-vs-retrieval-lucene/questions.jsonl b/benchmarks/grep-vs-retrieval-lucene/questions.jsonl
new file mode 100644
index 000000000..0e0cf4b10
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/questions.jsonl
@@ -0,0 +1,10 @@
+{"id":"q01_block_size","class":"prior_hostile","ask":"In this Lucene checkout, what is the value of ForUtil.BLOCK_SIZE used by the CURRENT default postings format (the lucene104 package, not the backward-codecs copies)? Give the number and the file path.","truth_file":"lucene/core/src/java/org/apache/lucene/codecs/lucene104/ForUtil.java","truth_line":34,"truth_fact":"BLOCK_SIZE = 256","must_match":["256"],"must_not_match":["128"],"grader_note":"Correct iff it states 256 for lucene104. Answering 128 is the parametric-memory answer and is WRONG here; five backward-codecs copies (lucene90/99/101/103/912) say 128, so a careless grep also fails. Must not hedge by giving both without committing to 256 for lucene104."}
+{"id":"q02_hnsw_level","class":"mechanism","ask":"In this checkout, how does HnswGraphBuilder decide which level a newly inserted node goes to? Give the exact formula for the ml field and the exact expression returned by getRandomGraphLevel, with file and line.","truth_file":"lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java","truth_line":183,"truth_fact":"ml = M == 1 ? 1 : 1 / Math.log(1.0 * M); getRandomGraphLevel returns ((int) (-log(randDouble) * ml))","must_match":["Math.log","-log","ml"],"must_not_match":[],"grader_note":"Correct iff BOTH: (a) ml = 1/ln(M) with the M==1 special case, (b) level = (int)(-ln(u) * ml). Stating only 'random exponential level' without the two expressions is WRONG (unverifiable hand-wave)."}
+{"id":"q03_mmap_chunk","class":"prior_hostile","ask":"What is DEFAULT_MAX_CHUNK_SIZE in MMapDirectory in this checkout, exactly as written? Give the literal expression, both resulting sizes, and the file:line.","truth_file":"lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java","truth_line":185,"truth_fact":"public static final long DEFAULT_MAX_CHUNK_SIZE = Constants.JRE_IS_64BIT ? (1L << 34) : (1L << 28);","must_match":["1L << 34","JRE_IS_64BIT"],"must_not_match":[],"grader_note":"Correct iff it reports the CONDITIONAL expression (not a single constant) and identifies 1L<<34 = 16 GiB for 64-bit, 1L<<28 = 256 MiB for 32-bit. Answering a single fixed number is WRONG."}
+{"id":"q04_merge_score","class":"mechanism","ask":"In TieredMergePolicy in this checkout, how is a candidate merge scored? Give the exact starting value of mergeScore, the exact exponent applied to total merge size, how skew is computed in the hitTooLarge branch, and the default deletesPctAllowed.","truth_file":"lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java","truth_line":713,"truth_fact":"mergeScore = skew; mergeScore *= Math.pow(totAfterMergeBytes, 0.05); hitTooLarge -> skew = 1.0/mergeFactor where mergeFactor=(int)segsPerTier; deletesPctAllowed default 20.0","must_match":["0.05","skew"],"must_not_match":[],"grader_note":"Correct iff it gives exponent 0.05 AND deletesPctAllowed default 20.0 AND that lower mergeScore is better. Missing the 0.05 exponent = WRONG."}
+{"id":"q05_hnsw_limits","class":"lookup","ask":"In Lucene99HnswVectorsFormat in this checkout, what are the hard maximum values accepted for maxConn and beamWidth, and what is the tiny-segment threshold constant? Give names, values, file:line.","truth_file":"lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java","truth_line":99,"truth_fact":"MAXIMUM_MAX_CONN = 512; MAXIMUM_BEAM_WIDTH = 3200; HNSW_GRAPH_THRESHOLD = 100","must_match":["512","3200"],"must_not_match":[],"grader_note":"Correct iff all three: 512, 3200, 100. Two of three = WRONG (scored strictly)."}
+{"id":"q06_bm25_k3","class":"prior_hostile","ask":"Does BM25Similarity in this checkout support a query-side saturation parameter? If so, name it, give its default value, the exact formula in its javadoc, and how it is disabled. File:line required.","truth_file":"lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java","truth_line":51,"truth_fact":"k3 exists; 4-arg ctor BM25Similarity(float k1,float b,boolean discountOverlaps,float k3); no-arg ctor passes -1f; formula ((k3 + 1) * qtf) / (k3 + qtf); negative disables","must_match":["k3","-1"],"must_not_match":[],"grader_note":"Correct iff it identifies k3, default -1f (disabled), and the formula ((k3+1)*qtf)/(k3+qtf). NOTE: upstream Apache Lucene BM25Similarity has NO k3, so a model answering from memory will say it does not exist -- that answer is WRONG for this tree. This is the strongest parametric-memory discriminator in the set."}
+{"id":"q07_iwc_flush","class":"lookup","ask":"What are the default flush triggers on IndexWriterConfig in this checkout: the default RAM buffer size and the default max buffered docs? Give the constant names, their values, and file:line.","truth_file":"lucene/core/src/java/org/apache/lucene/index/IndexWriterConfig.java","truth_line":83,"truth_fact":"DEFAULT_RAM_BUFFER_SIZE_MB = 16.0; DEFAULT_MAX_BUFFERED_DOCS = DISABLE_AUTO_FLUSH (i.e. disabled, -1)","must_match":["16.0","DISABLE_AUTO_FLUSH"],"must_not_match":[],"grader_note":"Correct iff 16.0 MB AND that max buffered docs defaults to DISABLE_AUTO_FLUSH (disabled), not a positive number."}
+{"id":"q08_osq_params","class":"lookup","ask":"In OptimizedScalarQuantizer in this checkout, what are the default lambda and the default iteration count used by the convenience constructor, and what is the name of the static table of precomputed constants it uses? File:line required.","truth_file":"lucene/core/src/java/org/apache/lucene/util/quantization/OptimizedScalarQuantizer.java","truth_line":57,"truth_fact":"DEFAULT_LAMBDA = 0.1f; DEFAULT_ITERS = 5; MINIMUM_MSE_GRID","must_match":["0.1","5","MINIMUM_MSE_GRID"],"must_not_match":[],"grader_note":"Correct iff all three named/valued correctly."}
+{"id":"q09_exhaustive_count","class":"index_hostile","ask":"How many .java files under lucene/core/src/java contain the exact text 'implements Accountable'? Give the integer count.","truth_file":"(whole tree scan)","truth_line":0,"truth_fact":"43","must_match":["43"],"must_not_match":[],"grader_note":"DELIBERATE ANTI-XERJ CONTROL. Ground truth is mechanically defined as: grep -rl 'implements Accountable' lucene/core/src/java | wc -l == 43. Exhaustive counting is exactly what top-k retrieval cannot do and what grep does perfectly. If XERJ loses here, that MUST be reported. Exact integer only."}
+{"id":"q10_tragic_event","class":"multi_hop","ask":"In IndexWriter in this checkout, when a tragic exception occurs during deleteDocuments(Term...), which method is invoked to record it, and what is the name of the different method invoked from the getReader path? Give both method names and file:line for each call site.","truth_file":"lucene/core/src/java/org/apache/lucene/index/IndexWriter.java","truth_line":1869,"truth_fact":"deleteDocuments(Term..) calls tragicEvent(tragedy, \"deleteDocuments(Term..)\") at line 1869; getReader path calls onTragicEvent(t, \"getReader\") at line 713","must_match":["tragicEvent","onTragicEvent"],"must_not_match":[],"grader_note":"Correct iff it distinguishes the two distinct methods tragicEvent vs onTragicEvent and attaches each to the right call site. Conflating them into one name = WRONG."}
diff --git a/benchmarks/grep-vs-retrieval-lucene/retrieve.py b/benchmarks/grep-vs-retrieval-lucene/retrieve.py
new file mode 100644
index 000000000..45c17586a
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/retrieve.py
@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""XERJ retrieval for the xerj arm.
+
+HONESTY CONTRACT — this script sees ONLY the question text.
+It never sees answer_file, answer_line, answer_text, or the grader regex.
+It is a fixed, question-agnostic query template. No per-question tuning.
+"""
+import json, sys, urllib.request
+
+ES = "http://localhost:9200/refsym/_search"
+TOPK = 8
+
+def search(question, allowed_paths=None):
+ must = [{"multi_match": {
+ "query": question,
+ "fields": ["code", "name_text^2", "sig^1.5", "doc"],
+ "type": "best_fields"}}]
+ # repo filter is MANDATORY: the index also contains `usearch`, which is NOT
+ # on disk for the native arm. Without it the xerj arm sees a corpus its peer cannot.
+ flt = [{"term": {"repo": "lucene"}}]
+ if allowed_paths:
+ flt.append({"terms": {"path": allowed_paths}})
+ body = {"size": TOPK, "query": {"bool": {"filter": flt, "must": must}},
+ "_source": ["path", "name", "kind", "line", "end_line", "sig", "code"]}
+ r = urllib.request.Request(ES, data=json.dumps(body).encode(),
+ headers={"Content-Type": "application/json"})
+ return json.load(urllib.request.urlopen(r))["hits"]["hits"]
+
+def render(hits):
+ out = []
+ for h in hits:
+ s = h["_source"]
+ out.append("// %s:%s-%s (%s %s)\n%s" %
+ (s["path"], s["line"], s.get("end_line"), s["kind"], s["name"], s["code"]))
+ return "\n\n".join(out)
+
+if __name__ == "__main__":
+ q = sys.argv[1]
+ paths = json.load(open(sys.argv[2])) if len(sys.argv) > 2 else None
+ hits = search(q, paths)
+ print(render(hits))
diff --git a/benchmarks/grep-vs-retrieval-lucene/run.py b/benchmarks/grep-vs-retrieval-lucene/run.py
new file mode 100644
index 000000000..678c84def
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/run.py
@@ -0,0 +1,160 @@
+#!/usr/bin/env python3
+"""Context-ceiling benchmark: native grep vs XERJ retrieval vs bare, across corpus scales.
+
+ python3 run.py --arms native,xerj,bare --scales S1,S2,S3,S4 --reps 3
+
+Cells are (arm x question x scale x rep). Each cell is one `claude -p` process
+with its own --session-id so its transcript can be located and attributed.
+"""
+import argparse, json, os, re, subprocess, sys, time, uuid, shutil, hashlib
+
+BENCH = os.path.dirname(os.path.abspath(__file__))
+SCALES = os.path.join(BENCH, "scales")
+RUNS = os.path.join(BENCH, "runs")
+CLAUDE = "/home/claude/.local/bin/claude"
+EMPTY = os.path.join(BENCH, "empty_cwd")
+
+ANSWER_SCHEMA = json.dumps({
+ "type": "object",
+ "properties": {
+ "answer": {"type": "string"},
+ "evidence_path": {"type": "string"},
+ "evidence_line": {"type": "integer"},
+ "confidence": {"type": "string", "enum": ["high", "medium", "low", "unknown"]},
+ },
+ "required": ["answer", "evidence_path", "confidence"],
+})
+
+# Identical for every arm. The only difference between arms is what is on disk /
+# what is prepended. No arm is told which files matter.
+SYSTEM = (
+ "You are answering a precise question about the Apache Lucene source code. "
+ "Answer ONLY from the code you can actually see. "
+ "If you cannot find the answer in the material available to you, set "
+ "confidence to \"unknown\" and say so in `answer` rather than recalling it "
+ "from general knowledge. A confident wrong answer is worse than \"unknown\"."
+)
+
+def transcript_dir(cwd):
+ return os.path.join("/home/claude/.claude/projects",
+ "-" + cwd.strip("/").replace("/", "-").replace("_", "-"))
+
+def find_transcript(sid):
+ base = "/home/claude/.claude/projects"
+ for d in os.listdir(base):
+ p = os.path.join(base, d, sid + ".jsonl")
+ if os.path.exists(p):
+ return p
+ return None
+
+def build_prompt(arm, q, scale):
+ if arm == "xerj":
+ paths = json.load(open(os.path.join(SCALES, "manifest.json")))["scales"][scale]["files"]
+ pf = os.path.join(RUNS, "_paths_%s.json" % scale)
+ if not os.path.exists(pf):
+ json.dump(paths, open(pf, "w"))
+ snip = subprocess.run([sys.executable, os.path.join(BENCH, "retrieve.py"),
+ q["question"], pf], capture_output=True, text=True).stdout
+ return ("Reference material retrieved from a XERJ index of the Lucene source.\n"
+ "This is all the source you have; there is no repository on disk.\n\n"
+ "```java\n" + snip + "\n```\n\nQuestion: " + q["question"])
+ if arm == "bare":
+ return ("You have no repository and no tools. Question: " + q["question"])
+ return ("The Apache Lucene source tree is in the current directory. "
+ "Question: " + q["question"])
+
+def run_cell(arm, q, scale, rep):
+ sid = str(uuid.UUID(hashlib.md5(("%s|%s|%s|%d" % (arm, q["id"], scale, rep))
+ .encode()).hexdigest()))
+ if arm == "native":
+ cwd = os.path.join(SCALES, scale)
+ tools = ["--tools", "Bash,Read,Grep,Glob"]
+ elif arm == "xerj":
+ os.makedirs(EMPTY, exist_ok=True)
+ cwd = EMPTY
+ tools = ["--tools", "Bash,Read,Grep,Glob"] # same agent as native, empty disk
+ else:
+ os.makedirs(EMPTY, exist_ok=True)
+ cwd = EMPTY
+ tools = ["--disallowedTools", "Bash,Read,Grep,Glob,WebSearch,WebFetch,Task"]
+
+ cmd = [CLAUDE, "-p", build_prompt(arm, q, scale),
+ "--output-format", "json",
+ "--safe-mode", # no CLAUDE.md, skills, hooks, MCP
+ "--permission-mode", "bypassPermissions",
+ "--system-prompt", SYSTEM,
+ "--json-schema", ANSWER_SCHEMA,
+ "--session-id", sid,
+ "--max-budget-usd", "1.50"] + tools
+
+ t0 = time.time()
+ p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=1800)
+ wall = time.time() - t0
+ try:
+ res = json.loads(p.stdout)
+ except Exception:
+ res = {"is_error": True, "raw": p.stdout[-4000:], "stderr": p.stderr[-2000:]}
+ tr = find_transcript(sid)
+ attr = None
+ if tr:
+ sys.path.insert(0, BENCH)
+ import attribute
+ attr = attribute.analyze(tr)
+ return {"arm": arm, "qid": q["id"], "qclass": q["class"], "scale": scale, "rep": rep,
+ "session_id": sid, "transcript": tr, "wall_s": round(wall, 1),
+ "result": res.get("result"), "is_error": res.get("is_error"),
+ "num_turns": res.get("num_turns"),
+ "total_cost_usd": res.get("total_cost_usd"),
+ "usage": res.get("usage"), "modelUsage": res.get("modelUsage"),
+ "attribution": attr}
+
+def grade(cell, q):
+ """Regex grading over the model's own `answer` field. No LLM judge."""
+ raw = cell.get("result") or ""
+ try:
+ obj = json.loads(raw)
+ txt = " ".join(str(obj.get(k, "")) for k in ("answer", "evidence_path"))
+ conf = obj.get("confidence")
+ except Exception:
+ txt, conf = raw, None
+ correct = bool(re.search(q["correct"], txt, re.I | re.S))
+ guessed = bool(q.get("guess") and re.search(q["guess"], txt, re.I | re.S))
+ abstained = (conf == "unknown")
+ return {"correct": correct and not guessed, "confabulated": guessed and not correct,
+ "abstained": abstained, "confidence": conf,
+ # the metric that matters for a security audience:
+ "confidently_wrong": (not correct) and conf in ("high", "medium")}
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--arms", default="native,xerj,bare")
+ ap.add_argument("--scales", default="S1,S2,S3,S4")
+ ap.add_argument("--questions", default="")
+ ap.add_argument("--reps", type=int, default=3)
+ ap.add_argument("--out", default=os.path.join(RUNS, "results.jsonl"))
+ a = ap.parse_args()
+ os.makedirs(RUNS, exist_ok=True)
+ qs = json.load(open(os.path.join(BENCH, "questions.json")))["questions"]
+ if a.questions:
+ keep = set(a.questions.split(","))
+ qs = [q for q in qs if q["id"] in keep]
+ fh = open(a.out, "a")
+ for rep in range(a.reps):
+ for scale in a.scales.split(","):
+ for arm in a.arms.split(","):
+ # bare has no corpus: run it once, at S1 only, and reuse
+ if arm == "bare" and scale != a.scales.split(",")[0]:
+ continue
+ for q in qs:
+ cell = run_cell(arm, q, scale, rep)
+ cell["grade"] = grade(cell, q)
+ fh.write(json.dumps(cell) + "\n"); fh.flush()
+ g = cell["grade"]
+ at = cell.get("attribution") or {}
+ print("%-6s %-3s %-3s r%d correct=%-5s conf=%-7s search_share=%-6s cost=%.4f"
+ % (arm, q["id"], scale, rep, g["correct"], g["confidence"],
+ at.get("share_of_context"), cell.get("total_cost_usd") or 0))
+ fh.close()
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/grep-vs-retrieval-lucene/run.sh b/benchmarks/grep-vs-retrieval-lucene/run.sh
new file mode 100755
index 000000000..faf1f6de1
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/run.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# run.sh -- execute the head-to-head. Usage:
+# ./run.sh <arm> <question_id|all> [repeat_index]
+# ./run.sh all all # full matrix, 3 arms x 10 questions x REPEATS
+#
+# Arms:
+# native grep/glob/read on the real checkout. No index. What a Claude Code user has today.
+# xerj retrieval ONLY (xq). No file tools at all. Strict test of "can the index alone answer".
+# hybrid both xq and file tools; the model chooses. The realistic product experience.
+#
+# Every run writes a raw stream-json transcript to runs/. Nothing is summarised here --
+# summarising happens in analyze.py, from the transcripts, so the raw evidence survives.
+set -uo pipefail
+
+BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+LUCENE=/home/claude/.xerj-code/corpora/lucene-meta/lucene
+CLAUDE=/home/claude/.local/bin/claude
+REPEATS="${REPEATS:-3}"
+export PATH="$BENCH:$PATH"
+
+# A neutral empty cwd for the xerj arm: it must NOT be able to fall back to the tree.
+SANDBOX="$BENCH/.sandbox"; mkdir -p "$SANDBOX"
+
+RULES='Answer the question about the Apache Lucene source. Rules: (1) Answer ONLY from what you actually retrieve or read in this session - do NOT answer from prior knowledge of Lucene, because this checkout may differ from the version you remember. (2) Cite file path and line number for every fact. (3) Be concise: the answer, then the citation. (4) If you cannot find it, say NOT FOUND.'
+
+run_one () {
+ local arm="$1" qid="$2" rep="$3"
+ local q; q=$(python3 -c "
+import json,sys
+for l in open('$BENCH/questions.jsonl'):
+ d=json.loads(l)
+ if d['id']=='$qid': print(d['ask']); break
+")
+ [ -z "$q" ] && { echo "unknown question $qid" >&2; return 1; }
+ local out="$BENCH/runs/${arm}__${qid}__r${rep}.jsonl"
+ [ -s "$out" ] && { echo "skip (exists) $out"; return 0; }
+
+ case "$arm" in
+ native)
+ ( cd "$LUCENE" && $CLAUDE -p "$RULES
+
+QUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Grep,Glob,Read" \
+ --disallowedTools "Bash,WebSearch,WebFetch,Task" ) > "$out" 2>"$BENCH/logs/${arm}__${qid}__r${rep}.err"
+ ;;
+ xerj)
+ ( cd "$SANDBOX" && $CLAUDE -p "$RULES
+
+You have NO access to the source tree. You have exactly one retrieval tool, a shell command:
+ xq \"<free text query>\" [k]
+It searches a prebuilt symbol-level index of this Lucene checkout and returns matching
+symbols as: path:line, kind, signature, and the symbol body. Call it via Bash. Query it
+as many times as you need, refining your query. Note the index contains backward-codecs
+and test copies of many classes, so check the path of each hit before trusting it.
+
+QUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Bash(xq:*)" \
+ --disallowedTools "Grep,Glob,Read,WebSearch,WebFetch,Task" ) > "$out" 2>"$BENCH/logs/${arm}__${qid}__r${rep}.err"
+ ;;
+ hybrid)
+ ( cd "$LUCENE" && $CLAUDE -p "$RULES
+
+In addition to the normal file tools you have a prebuilt symbol index of this checkout,
+queried by the shell command:
+ xq \"<free text query>\" [k]
+which returns path:line, kind, signature and body for matching symbols. Use whichever
+approach you judge cheapest for this question.
+
+QUESTION: $q" \
+ --output-format stream-json --verbose \
+ --allowedTools "Grep,Glob,Read,Bash(xq:*)" \
+ --disallowedTools "WebSearch,WebFetch,Task" ) > "$out" 2>"$BENCH/logs/${arm}__${qid}__r${rep}.err"
+ ;;
+ *) echo "unknown arm $arm" >&2; return 1;;
+ esac
+ echo "wrote $out"
+}
+
+ARM="${1:-all}"; QID="${2:-all}"
+ARMS=$([ "$ARM" = all ] && echo "native xerj hybrid" || echo "$ARM")
+QIDS=$([ "$QID" = all ] && python3 -c "
+import json
+print(' '.join(json.loads(l)['id'] for l in open('$BENCH/questions.jsonl')))" || echo "$QID")
+
+for r in $(seq 1 "$REPEATS"); do
+ for a in $ARMS; do
+ for q in $QIDS; do
+ run_one "$a" "$q" "$r"
+ done
+ done
+done
diff --git a/benchmarks/grep-vs-retrieval-lucene/run_arm.sh b/benchmarks/grep-vs-retrieval-lucene/run_arm.sh
new file mode 100755
index 000000000..c391ce213
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/run_arm.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Usage: run_arm.sh <arm: A|B|C|BASE> <question_id> <repeat_n>
+set -euo pipefail
+B="$(cd "$(dirname "$0")" && pwd)"
+CORPUS=/home/claude/.xerj-code/corpora/lucene-meta/lucene
+ARM="$1"; QID="${2:-}"; REP="${3:-1}"
+SID=$(python3 -c "import uuid;print(uuid.uuid4())")
+
+Q=$(python3 -c "
+import json,sys
+d=json.load(open('$B/questions.json'))
+print(next(q['q'] for q in d['questions'] if q['id']=='$QID'))
+" 2>/dev/null || echo "Reply with exactly: ok")
+
+XERJ_PROMPT='You are answering questions about a Java codebase. You have NO filesystem access to it.
+The ONLY way to see code is the XERJ index, an Elasticsearch-compatible endpoint at
+http://localhost:9200/refsym/_search . Query it with curl and standard ES DSL.
+Fields: repo, path, file, language, kind (method|class|interface|enum|...), name (keyword),
+name_text (text), line, end_line, loc, sig, doc, code. The `code` field holds the full
+symbol body, so a hit already contains the implementation - there is nothing to open afterwards.
+Always request _source so you get path and line numbers to cite.
+Answer from retrieval results only. If retrieval does not contain the answer, say so explicitly
+rather than guessing.'
+
+COMMON=(--output-format json --session-id "$SID" --permission-mode bypassPermissions
+ --max-budget-usd 3 --model opus --effort medium)
+
+case "$ARM" in
+ A) # native: the tools a real Claude Code user has, no XERJ, no hints
+ cd "$CORPUS"
+ claude -p "$Q" "${COMMON[@]}" \
+ --allowedTools "Bash Read Grep Glob" \
+ --disallowedTools "WebSearch WebFetch Edit Write" \
+ > "$B/runs/${QID}_A_${REP}.json" ;;
+ B) # retrieval-only: XERJ is the sole source of code. No grep, no file reads.
+ cd /home/claude
+ claude -p "$Q" "${COMMON[@]}" \
+ --append-system-prompt "$XERJ_PROMPT" \
+ --allowedTools "Bash" \
+ --disallowedTools "Read Grep Glob Edit Write WebSearch WebFetch" \
+ --add-dir /home/claude \
+ > "$B/runs/${QID}_B_${REP}.json" ;;
+ C) # realistic: XERJ available AND the normal toolset as fallback
+ cd "$CORPUS"
+ claude -p "$Q" "${COMMON[@]}" \
+ --append-system-prompt "$XERJ_PROMPT Standard file tools are also available as a fallback." \
+ --allowedTools "Bash Read Grep Glob" \
+ --disallowedTools "WebSearch WebFetch Edit Write" \
+ > "$B/runs/${QID}_C_${REP}.json" ;;
+ BASE_A) cd "$CORPUS"
+ claude -p "Reply with exactly: ok" "${COMMON[@]}" \
+ --allowedTools "Bash Read Grep Glob" --disallowedTools "WebSearch WebFetch Edit Write" \
+ > "$B/runs/baseline_A.json" ;;
+ BASE_B) cd /home/claude
+ claude -p "Reply with exactly: ok" "${COMMON[@]}" \
+ --append-system-prompt "$XERJ_PROMPT" --allowedTools "Bash" \
+ --disallowedTools "Read Grep Glob Edit Write WebSearch WebFetch" \
+ > "$B/runs/baseline_B.json" ;;
+esac
+echo "$SID"
diff --git a/benchmarks/grep-vs-retrieval-lucene/xq b/benchmarks/grep-vs-retrieval-lucene/xq
new file mode 100755
index 000000000..c0edce892
--- /dev/null
+++ b/benchmarks/grep-vs-retrieval-lucene/xq
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""
+xq -- the ONLY retrieval affordance the XERJ arms get.
+
+Usage: xq "<free text query>" [k]
+
+Deliberately dumb and question-agnostic: it takes whatever query string the model
+supplies and runs one fixed ES query template against the refsym index. There is no
+per-question tuning, no hand-written query, no reranking we authored for a specific
+answer. If this file contained question-specific logic the benchmark would be rigged,
+so it must stay generic -- that is the point.
+
+Returns symbol-level hits: path:line, kind, signature, and the actual body, capped so
+a single call cannot dump the corpus into context.
+"""
+import json
+import sys
+import urllib.request
+
+ENDPOINT = "http://localhost:9200/refsym/_search"
+MAX_CODE_CHARS = 1600 # per hit; keeps one call from being a whole-file dump
+DEFAULT_K = 5
+
+
+def main():
+ if len(sys.argv) < 2:
+ print("usage: xq \"<query>\" [k]", file=sys.stderr)
+ return 2
+ q = sys.argv[1]
+ k = int(sys.argv[2]) if len(sys.argv) > 2 else DEFAULT_K
+ k = max(1, min(k, 10))
+
+ body = {
+ "size": k,
+ "_source": ["path", "name", "kind", "line", "end_line", "sig", "doc", "code"],
+ "query": {
+ "bool": {
+ "should": [
+ {"match": {"code": {"query": q}}},
+ {"match": {"name_text": {"query": q, "boost": 3}}},
+ {"match": {"sig": {"query": q, "boost": 2}}},
+ {"match": {"doc": {"query": q}}},
+ ],
+ "minimum_should_match": 1,
+ }
+ },
+ }
+
+ req = urllib.request.Request(
+ ENDPOINT, data=json.dumps(body).encode(),
+ headers={"Content-Type": "application/json"}, method="POST")
+ try:
+ with urllib.request.urlopen(req, timeout=30) as r:
+ res = json.load(r)
+ except Exception as e:
+ print(f"xq: query failed: {e}", file=sys.stderr)
+ return 1
+
+ hits = res.get("hits", {}).get("hits", [])
+ if not hits:
+ print("no hits")
+ return 0
+
+ for h in hits:
+ s = h["_source"]
+ code = (s.get("code") or "")
+ truncated = len(code) > MAX_CODE_CHARS
+ if truncated:
+ code = code[:MAX_CODE_CHARS]
+ print(f"--- {s.get('path')}:{s.get('line')} [{s.get('kind')}] {s.get('name')} score={h['_score']:.1f}")
+ if s.get("sig"):
+ print(f"sig: {s['sig']}")
+ print(code)
+ if truncated:
+ print("... [truncated]")
+ print()
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())