Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(bench): 3-model reference-coding benchmark + SWE-bench turnkey harness

xerj-orgf635ee51bd4b
diff --git a/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md b/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md
new file mode 100644
index 000000000..288031a11
--- /dev/null
+++ b/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md
@@ -0,0 +1,172 @@
+# Reference-coding impact — 3-model controlled benchmark (2026-08-18)
+
+**Question:** what is the *measured* impact of XERJ reference-coding on real coding
+tasks, across Opus 4.6 / 4.8 / 5.0 — on tasks a model **cannot pass without it**,
+and on token/cost — reported so it is **defensible**, not marketing.
+
+**One-line answer.** On unfamiliar-API tasks with **one shot**, memory alone cannot
+produce working code (Opus 4.6 & 4.8: **0/12**); XERJ takes it to **12/12**. As a
+model's recall improves the *correctness* gap shrinks (Opus 5.0 one-shots **7/12**),
+but the **efficiency** win is universal and consistent: XERJ spends **~1.4× fewer
+output tokens than a grep-driven agent** and **4–10× fewer than memory-with-retries**,
+at lower cost, on every model. The value is **gated by memorization** — on code the
+model already knows, retrieval is neutral-to-harmful (measured, §5).
+
+---
+
+## 1. Method — why this is trustworthy
+
+Three arms of the **same** Claude Code, same tasks, differing only by the reference:
+
+| arm | what it has | isolates |
+|---|---|---|
+| `bare` | memory only, no tools (retries on its own compile errors) | parametric recall |
+| `native` | the reference repo on disk — **must grep to find the API** | full agentic Claude Code |
+| `xerj` | **same agent, XERJ pre-retrieved the passage** | Claude Code + reference coding |
+
+`native` vs `xerj` is the headline (identical agent; only difference is whether XERJ
+pre-retrieved or the agent had to search). `bare` is the memory floor.
+
+- **Objective, hidden verdict.** Generated code must **compile and pass a `cargo test`
+ suite the model never sees** — not a similarity score, not an LLM judge.
+- **Contamination-free tasks.** 6 tasks over purpose-built crates (`sift`, `spool`,
+ `trellis`, `quill`, `weft`) — real code written for the study, so in **no** model's
+ training set. Memory has to *guess* the API. Soundness was proven first:
+ `validate_sift.py` → golden solution passes, no-seal variant fails the test, guessed
+ API won't compile → **ALL SOUND**.
+- **Real tokens & dollars** from `claude -p --output-format json`; the model that ran
+ each call is verified from the JSON `modelUsage` field (not assumed).
+- **Integrity.** During the run, two `bare`-arm codegen subprocesses tried to obtain
+ the reference (`sift`, `weft`) via cross-session messages. Both were **refused** —
+ feeding a `bare` arm the reference would contaminate it. Every `bare` `weft-scan`
+ trial in the final data was checked: Opus 4.6/4.8 → genuine compile-errors; Opus 5.0's
+ one `weft` pass predates the message and matches its overall 7/12 one-shot strength.
+ **No contamination in the reported numbers.**
+
+Matrix: 6 tasks × 3 arms × {5-round, pass@1} × Opus 4.6/4.8/5.0 × 2 trials.
+
+---
+
+## 2. Correctness — "cannot pass without it" (pass@1, one shot)
+
+| model | `bare` (memory) | `xerj` (reference-coded) |
+|---|---:|---:|
+| **Opus 4.6** | **0 / 12** | **12 / 12** |
+| **Opus 4.8** | **0 / 12** | **12 / 12** |
+| **Opus 5.0** | **7 / 12** | **11 / 12** |
+
+Given one shot, the two older models **never** produce working code against the
+unfamiliar crate API; XERJ passes every task. The frontier model (5.0) has enough
+recall to one-shot 7/12 — so its correctness lift is smaller (7→11) but still positive,
+and it does so at a fraction of the cost (§3).
+
+## 3. Efficiency — output tokens & cost (5 retry rounds allowed)
+
+| model | arm | solved | output tok | cost | mean rounds |
+|---|---|---:|---:|---:|---:|
+| Opus 4.6 | bare | 4/12 | 75,496 | $16.75 | 2.0 |
+| | native | 12/12 | 10,901 | $2.44 | 8.0 |
+| | **xerj** | **12/12** | **7,497** | **$2.01** | 5.2 |
+| Opus 4.8 | bare | 12/12 | 35,272 | $6.11 | 2.0 |
+| | native | 12/12 | 11,693 | $2.09 | 6.2 |
+| | **xerj** | 11/12 | **8,385** | **$1.81** | 4.5 |
+| Opus 5.0 | bare | 12/12 | 42,354 | $5.98 | 1.6 |
+| | native | 12/12 | 11,516 | $2.15 | 5.2 |
+| | **xerj** | **12/12** | **8,127** | **$1.94** | 3.4 |
+
+**Headline ratio (native → xerj output tokens), the identical-agent comparison:**
+
+| model | native/xerj out-tok | bare/xerj out-tok | cost native/xerj |
+|---|---:|---:|---:|
+| Opus 4.6 | **1.45×** | 10.07× | 1.21× |
+| Opus 4.8 | **1.39×** | 4.21× | 1.15× |
+| Opus 5.0 | **1.42×** | 5.21× | 1.11× |
+
+XERJ is **~1.4× fewer output tokens than grep across all three models** (consistent
+with the earlier case study's ~1.65×), in fewer rounds, at lower cost. Note Opus 4.6
+`bare` solved only **4/12 even with 5 retry rounds** while burning **$16.75** — the
+weaker the model, the more reference-coding matters, on both axes.
+
+## 4. How to read this honestly
+
+- The **correctness** benefit is real but **model-dependent**: dramatic for weaker
+ recall (4.6/4.8 pass@1 0/12→12/12), modest for the frontier model (5.0: 7→11).
+- The **efficiency** benefit (`native`→`xerj` ~1.4×, cost 1.1–1.2×) is **consistent
+ across all three models** and is the safest headline claim.
+- Do **not** claim a flat "0→12 for every model." The defensible statement is the
+ nuanced one above.
+
+## 5. The honest boundary — memorized code (measured, same harness)
+
+The value is **gated by memorization**. On the `fieldnorm-quant` task over **tantivy**
+(a popular public crate the model has trained on), from the case study:
+
+| arm | solved | med output tok | total $ |
+|---|---:|---:|---:|
+| bare (memory) | 3/3 | 10,213 | $0.98 |
+| **native** (grep) | 3/3 | 1,440 | **$0.60 — best** |
+| xerj (inject) | 3/3 | 8,504 | $1.43 — **worst** |
+
+All arms pass because the code is memorized (`bare` reproduced the exact 256-entry
+norm table from memory). Here **retrieval is the worst arm** — injecting a large
+reference backfires on bulk-data reproduction. Reference-coding wins on **private,
+internal, niche, or post-cutoff** code; it is overhead on popular public libraries the
+model already knows. That boundary is a feature of the honest claim, not a caveat to
+hide.
+
+## 6. Recognized benchmarks (SWE-bench Verified) — transparency
+
+The recognized benchmark that tests this capability is **SWE-bench Verified** (patch a
+real repo so its hidden tests pass — the natural fit for "index the repo and retrieve
+the code to change"). It **could not be run in this sandbox**, and this was verified,
+not assumed:
+
+- **No usable Docker daemon** — SWE-bench's official harness runs each repo's tests in
+ per-repo Docker images; the daemon is present but not runnable here.
+- **Python-version wall** — only Python 3.14 is available; SWE-bench instances pin
+ 2013–2020-era commits (e.g. `psf__requests-1142` @ 2013) whose `setup.py` will not
+ build under 3.14 / modern setuptools. Probed end-to-end on `flask-5014` and
+ `requests-1142`: both fail environment setup at baseline.
+
+We do **not** substitute a hand-rolled n≈2 scaffold and call it "the SWE-bench score" —
+that is precisely the kind of number that is not defensible. Instead: a **turnkey
+SWE-bench Verified harness** (with the XERJ index/retrieve arm) is provided so the
+**official** number reproduces in any Docker-capable environment — see
+[`tools/xerj-code/swebench/README.md`](../../tools/xerj-code/swebench/README.md).
+
+## 7. Reproduce
+
+```sh
+# server + corpus
+xerj --insecure --port 9450 --data-dir /tmp/xbench --embed-mode lexical & # background
+XERJ_URL=http://localhost:9450 XERJ_BIN=$PWD/engine/target/release/xerj \
+ bash .claude/skills/xerj-code/scripts/xc-index.sh novel-libs --fresh
+
+# soundness (must print ALL SOUND)
+cd .claude/skills/xerj-code/measure && python3 validate_sift.py
+
+# a model, both settings
+TASKS="tasks/sift-build.json tasks/sift-until.json tasks/spool-window.json \
+ tasks/trellis-order.json tasks/quill-codec.json tasks/weft-scan.json"
+ANTHROPIC_MODEL=claude-opus-4-8 XERJ_URL=http://localhost:9450 \
+ python3 csrun.py novel-libs $TASKS --arms bare,native,xerj --trials 2 \
+ --corpus-dir ~/.xerj-code/corpora/novel-libs --out results-opus48.json # efficiency
+ANTHROPIC_MODEL=claude-opus-4-8 XERJ_URL=http://localhost:9450 \
+ python3 csrun.py novel-libs $TASKS --arms bare,xerj --trials 2 --rounds 1 \
+ --out results-pass1-opus48.json # correctness
+```
+
+Raw result JSONs for this run: `/home/claude/ai/xerj-bench/results-*.json`.
+
+## 8. Limitations (stated, not buried)
+
+- n is modest (6 tasks × 2 trials/model); pass/fail on contamination-free API tasks is
+ fairly deterministic, but confidence intervals at this n are wide — treat the
+ *direction and magnitude* as the result, not the third significant figure.
+- Purpose-built crates guarantee non-memorization but are simpler than real repos; the
+ recognized-repo signal is exactly what the SWE-bench harness (§6) is for.
+- `bare` is memory + compile-error retries, not pure one-shot except in the pass@1
+ setting; both settings are reported so the reader picks the lens.
+- A `kv-oss` re-index during this session hit an engine catalog error
+ (`catalog read-back … disagrees with the sealed generation projection`) — filed as a
+ separate finding; the memorized boundary (§5) is cited from the prior verified run.
diff --git a/tools/xerj-code/swebench/README.md b/tools/xerj-code/swebench/README.md
new file mode 100644
index 000000000..4df0d2206
--- /dev/null
+++ b/tools/xerj-code/swebench/README.md
@@ -0,0 +1,92 @@
+# SWE-bench Verified × XERJ reference coding — turnkey harness
+
+Measures whether XERJ reference-coding improves an agent on **SWE-bench Verified**,
+the benchmark the AI labs score coding agents on. It is split so the **verdict is the
+official one**: this repo only *generates* patches; scoring is the standard
+`swebench.harness.run_evaluation` in Docker. The number you get is the real leaderboard
+number, for the subset you run.
+
+> **Why this isn't run in the XERJ sandbox.** SWE-bench's evaluation runs each repo's
+> tests in per-repo **Docker** images, and its instances pin **2013–2020-era Python**
+> commits. A sandbox without a Docker daemon, or with only a modern Python (3.14), can
+> neither evaluate nor even set up baselines (verified: `flask-5014`, `requests-1142`
+> both fail environment setup on 3.14). Run this on a **Docker-capable Linux box**.
+
+## Arms
+
+Identical agent, one difference — the clean test of reference-coding:
+
+- **`native`** — the repo is checked out; the agent must **grep** to find the code to change.
+- **`xerj`** — same, but XERJ has indexed the repo and **pre-retrieved** the relevant
+ passages into the prompt.
+
+(No `bare` arm: patching a real repo from pure memory is not a meaningful task. The
+`bare` floor lives in the contamination-free study, `measure/CASE_STUDY.md`.)
+
+## Prerequisites
+
+```sh
+pip install swebench datasets # official harness + dataset loader
+docker info # must succeed (daemon running)
+claude --version # authenticated Claude Code CLI
+xerj --version # XERJ release binary (set XERJ_BIN to its path)
+```
+
+Start a XERJ node the `xerj` arm will index into (a private port, throwaway data dir):
+
+```sh
+xerj --insecure --port 9200 --data-dir /tmp/swe-xerj --embed-mode lexical &
+export XERJ_URL=http://localhost:9200 XERJ_BIN=$(command -v xerj)
+```
+
+## 1. Generate predictions (both arms, per model)
+
+Start small — one lightweight repo — to confirm the loop end-to-end, then scale.
+
+```sh
+REPOS="psf/requests pytest-dev/pytest" # or drop --repos for the full 500
+
+for arm in native xerj; do
+ ANTHROPIC_MODEL=claude-opus-4-8 \
+ python3 gen_predictions.py --arm $arm --repos $REPOS --limit 20 \
+ --out preds-opus48-$arm.jsonl --tokens-out tokens-opus48-$arm.jsonl
+done
+```
+
+Repeat with `ANTHROPIC_MODEL=claude-opus-5`, `claude-opus-4-6`, … The model that
+actually ran is recorded from claude's JSON `modelUsage` in the `--tokens-out` log, so
+the selection is verifiable, not assumed.
+
+## 2. Score with the OFFICIAL harness (Docker)
+
+```sh
+for arm in native xerj; do
+ python3 -m swebench.harness.run_evaluation \
+ --dataset_name princeton-nlp/SWE-bench_Verified \
+ --predictions_path preds-opus48-$arm.jsonl \
+ --run_id opus48-$arm --max_workers 4
+done
+```
+
+Each run prints/writes a report with **resolved / total** — that is the SWE-bench
+resolve-rate for that arm+model, computed by the repos' own `FAIL_TO_PASS` /
+`PASS_TO_PASS` tests. The XERJ claim is the **native → xerj delta** in resolve-rate,
+alongside the **token/cost delta** from the `--tokens-out` logs.
+
+## 3. Read it honestly
+
+- Report **resolve-rate per arm** and the **native→xerj delta**, plus output-tokens and
+ \$/instance from the token logs. State **n** (instances actually evaluated) and which
+ repos — do not extrapolate a subset to the full 500.
+- On SWE-bench repos the model has largely trained on, the *resolve-rate* lift from
+ retrieval may be small (memorized code — see the boundary in
+ `../../measure/CASE_STUDY.md` and `demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md`);
+ the defensible signal there is usually **efficiency** (fewer grep turns / output
+ tokens for the same fix). Report whatever the data shows, wins and losses.
+- Keep the agent scaffold identical between arms (same `--max-turns`, same prompt
+ scaffolding) so the only variable is XERJ pre-retrieval.
+
+## Files
+
+- `gen_predictions.py` — generates the `native`/`xerj` prediction JSONLs (+ token logs).
+- scoring — the official `swebench.harness.run_evaluation` (not vendored here on purpose).
diff --git a/tools/xerj-code/swebench/gen_predictions.py b/tools/xerj-code/swebench/gen_predictions.py
new file mode 100644
index 000000000..21b0d232f
--- /dev/null
+++ b/tools/xerj-code/swebench/gen_predictions.py
@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""
+SWE-bench Verified — generate model patches for the `native` and `xerj` arms.
+
+This is the GENERATION half of a defensible SWE-bench run. It does NOT judge
+correctness itself: it writes standard SWE-bench prediction files, and you score
+them with the OFFICIAL `swebench.harness.run_evaluation` (Docker) — so the number
+is the real leaderboard number, not a home-grown verdict. See README.md.
+
+Two arms, identical agent, one difference:
+ native : the repo is checked out; the agent must FIND the code to change by grepping.
+ xerj : same, but XERJ has indexed the repo and PRE-RETRIEVED the relevant
+ passages, injected into the prompt. (This is the reference-coding arm.)
+(There is deliberately no `bare` arm: patching a real repo from pure memory is
+not a meaningful task.)
+
+Output: one predictions JSONL per (arm, model), lines of
+ {"instance_id", "model_name_or_path", "model_patch"}
+
+Requires: `datasets`, the `claude` CLI (authenticated), a running XERJ
+(`XERJ_URL`, default http://localhost:9200) with its binary on `XERJ_BIN`, git.
+Model is selected per run via ANTHROPIC_MODEL (verified from claude's JSON
+`modelUsage`).
+"""
+import argparse, json, os, subprocess, tempfile, shutil, sys, time, pathlib
+
+XERJ_URL = os.environ.get("XERJ_URL", "http://localhost:9200")
+XERJ_BIN = os.environ.get("XERJ_BIN", "xerj")
+XC = os.path.join(os.path.dirname(__file__), "..", "scripts", "xc.py")
+
+
+def sh(cmd, cwd=None, timeout=None, env=None):
+ return subprocess.run(cmd, cwd=cwd, timeout=timeout, capture_output=True,
+ text=True, env={**os.environ, **(env or {})})
+
+
+def clone_at(repo, base_commit, dest):
+ sh(["git", "clone", "--quiet", f"https://github.com/{repo}", dest])
+ sh(["git", "checkout", "--quiet", base_commit], cwd=dest)
+
+
+def retrieve_context(repo_dir, instance_id, problem, k=6, chars=8000):
+ """xerj arm: index the repo, retrieve passages relevant to the issue."""
+ prefix = "swe-" + instance_id.replace("/", "-")
+ idx = sh([XERJ_BIN, "autoindex", repo_dir, "--url", XERJ_URL,
+ "--prefix", prefix, "--no-graph"], timeout=1800)
+ # exit 3 == completed-with-junk == success
+ if idx.returncode not in (0, 3):
+ return None, f"index-failed:{idx.returncode}:{idx.stderr[:160]}"
+ # first ~3 lines of the issue make the best lexical query on the default embedder
+ query = " ".join(problem.split("\n")[:3])[:300]
+ r = sh(["python3", XC, prefix, query, "--k", str(k), "--chars", str(chars),
+ "--full"], env={"XERJ_URL": XERJ_URL})
+ if r.returncode != 0:
+ return None, f"retrieve-failed:{r.returncode}:{r.stderr[:160]}"
+ return r.stdout, None
+
+
+def run_agent(repo_dir, problem, ref_block, timeout, max_turns):
+ ref = ""
+ if ref_block:
+ ref = ("\n\nXERJ retrieved these passages from THIS repository — the code you "
+ "most likely need to change is here (cite file:line):\n\n" + ref_block)
+ prompt = (
+ "You are fixing a bug in the checked-out repository at the current working "
+ "directory. Here is the issue:\n\n" + problem +
+ "\n\nEdit the source files to fix it. Do NOT edit or add tests — a hidden "
+ "suite will judge your patch. Do not run the test suite. Make the smallest "
+ "correct change. When done, stop." + ref)
+ t0 = time.time()
+ p = subprocess.run(
+ ["claude", "-p", prompt, "--output-format", "json",
+ "--permission-mode", "bypassPermissions", "--max-turns", str(max_turns)],
+ cwd=repo_dir, capture_output=True, text=True, timeout=timeout)
+ el = time.time() - t0
+ usage, model = {}, None
+ try:
+ d = json.loads(p.stdout)
+ u = d.get("usage", {})
+ usage = {"in": u.get("input_tokens", 0), "out": u.get("output_tokens", 0),
+ "cost": d.get("total_cost_usd", 0.0)}
+ model = list(d.get("modelUsage", {}).keys())
+ except Exception:
+ pass
+ diff = sh(["git", "diff"], cwd=repo_dir).stdout
+ return diff, usage, model, el
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--arm", choices=["native", "xerj"], required=True)
+ ap.add_argument("--dataset", default="princeton-nlp/SWE-bench_Verified")
+ ap.add_argument("--repos", nargs="*", help="restrict to these repos (e.g. psf/requests)")
+ ap.add_argument("--limit", type=int, default=0, help="max instances (0=all)")
+ ap.add_argument("--instances", nargs="*", help="explicit instance_ids")
+ ap.add_argument("--max-turns", type=int, default=60)
+ ap.add_argument("--timeout", type=int, default=1200)
+ ap.add_argument("--out", required=True)
+ ap.add_argument("--tokens-out", help="optional per-instance token/cost log (JSONL)")
+ args = ap.parse_args()
+
+ from datasets import load_dataset
+ ds = load_dataset(args.dataset, split="test")
+ rows = list(ds)
+ if args.repos:
+ rows = [r for r in rows if r["repo"] in set(args.repos)]
+ if args.instances:
+ rows = [r for r in rows if r["instance_id"] in set(args.instances)]
+ if args.limit:
+ rows = rows[:args.limit]
+ model_name = os.environ.get("ANTHROPIC_MODEL", "default") + f"+xerj-{args.arm}"
+ print(f"generating {len(rows)} predictions | arm={args.arm} | model_env="
+ f"{os.environ.get('ANTHROPIC_MODEL','default')}", file=sys.stderr)
+
+ outf = open(args.out, "w")
+ tlog = open(args.tokens_out, "w") if args.tokens_out else None
+ for i, r in enumerate(rows, 1):
+ iid = r["instance_id"]
+ work = tempfile.mkdtemp(prefix="swe-")
+ repo_dir = os.path.join(work, "repo")
+ try:
+ clone_at(r["repo"], r["base_commit"], repo_dir)
+ ref, err = (None, None)
+ if args.arm == "xerj":
+ ref, err = retrieve_context(repo_dir, iid, r["problem_statement"])
+ if err:
+ print(f" [{i}/{len(rows)}] {iid} retrieve error: {err}", file=sys.stderr)
+ diff, usage, model, el = run_agent(
+ repo_dir, r["problem_statement"], ref, args.timeout, args.max_turns)
+ outf.write(json.dumps({"instance_id": iid,
+ "model_name_or_path": model_name,
+ "model_patch": diff}) + "\n")
+ outf.flush()
+ if tlog:
+ tlog.write(json.dumps({"instance_id": iid, "arm": args.arm,
+ "model": model, "usage": usage,
+ "elapsed_s": round(el, 1),
+ "patch_bytes": len(diff)}) + "\n")
+ tlog.flush()
+ print(f" [{i}/{len(rows)}] {iid} {model} out={usage.get('out')} "
+ f"patch={len(diff)}B {el:.0f}s", file=sys.stderr)
+ except subprocess.TimeoutExpired:
+ print(f" [{i}/{len(rows)}] {iid} TIMEOUT", file=sys.stderr)
+ outf.write(json.dumps({"instance_id": iid,
+ "model_name_or_path": model_name,
+ "model_patch": ""}) + "\n")
+ except Exception as e:
+ print(f" [{i}/{len(rows)}] {iid} ERROR {e}", file=sys.stderr)
+ finally:
+ shutil.rmtree(work, ignore_errors=True)
+ outf.close()
+ if tlog:
+ tlog.close()
+ print(f"wrote {args.out}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()