AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(bench): 3-model reference-coding benchmark + SWE-bench turnkey harness
xerj-orgf635ee51bd4b
diff --git a/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md b/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.mdnew file mode 100644index 000000000..288031a11--- /dev/null+++ b/demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md@@ -0,0 +1,172 @@+# Reference-coding impact — 3-model controlled benchmark (2026-08-18)++**Question:** what is the *measured* impact of XERJ reference-coding on real coding+tasks, across Opus 4.6 / 4.8 / 5.0 — on tasks a model **cannot pass without it**,+and on token/cost — reported so it is **defensible**, not marketing.++**One-line answer.** On unfamiliar-API tasks with **one shot**, memory alone cannot+produce working code (Opus 4.6 & 4.8: **0/12**); XERJ takes it to **12/12**. As a+model's recall improves the *correctness* gap shrinks (Opus 5.0 one-shots **7/12**),+but the **efficiency** win is universal and consistent: XERJ spends **~1.4× fewer+output tokens than a grep-driven agent** and **4–10× fewer than memory-with-retries**,+at lower cost, on every model. The value is **gated by memorization** — on code the+model already knows, retrieval is neutral-to-harmful (measured, §5).++---++## 1. Method — why this is trustworthy++Three arms of the **same** Claude Code, same tasks, differing only by the reference:++| arm | what it has | isolates |+|---|---|---|+| `bare` | memory only, no tools (retries on its own compile errors) | parametric recall |+| `native` | the reference repo on disk — **must grep to find the API** | full agentic Claude Code |+| `xerj` | **same agent, XERJ pre-retrieved the passage** | Claude Code + reference coding |++`native` vs `xerj` is the headline (identical agent; only difference is whether XERJ+pre-retrieved or the agent had to search). `bare` is the memory floor.++- **Objective, hidden verdict.** Generated code must **compile and pass a `cargo test`+ suite the model never sees** — not a similarity score, not an LLM judge.+- **Contamination-free tasks.** 6 tasks over purpose-built crates (`sift`, `spool`,+ `trellis`, `quill`, `weft`) — real code written for the study, so in **no** model's+ training set. Memory has to *guess* the API. Soundness was proven first:+ `validate_sift.py` → golden solution passes, no-seal variant fails the test, guessed+ API won't compile → **ALL SOUND**.+- **Real tokens & dollars** from `claude -p --output-format json`; the model that ran+ each call is verified from the JSON `modelUsage` field (not assumed).+- **Integrity.** During the run, two `bare`-arm codegen subprocesses tried to obtain+ the reference (`sift`, `weft`) via cross-session messages. Both were **refused** —+ feeding a `bare` arm the reference would contaminate it. Every `bare` `weft-scan`+ trial in the final data was checked: Opus 4.6/4.8 → genuine compile-errors; Opus 5.0's+ one `weft` pass predates the message and matches its overall 7/12 one-shot strength.+ **No contamination in the reported numbers.**++Matrix: 6 tasks × 3 arms × {5-round, pass@1} × Opus 4.6/4.8/5.0 × 2 trials.++---++## 2. Correctness — "cannot pass without it" (pass@1, one shot)++| model | `bare` (memory) | `xerj` (reference-coded) |+|---|---:|---:|+| **Opus 4.6** | **0 / 12** | **12 / 12** |+| **Opus 4.8** | **0 / 12** | **12 / 12** |+| **Opus 5.0** | **7 / 12** | **11 / 12** |++Given one shot, the two older models **never** produce working code against the+unfamiliar crate API; XERJ passes every task. The frontier model (5.0) has enough+recall to one-shot 7/12 — so its correctness lift is smaller (7→11) but still positive,+and it does so at a fraction of the cost (§3).++## 3. Efficiency — output tokens & cost (5 retry rounds allowed)++| model | arm | solved | output tok | cost | mean rounds |+|---|---|---:|---:|---:|---:|+| Opus 4.6 | bare | 4/12 | 75,496 | $16.75 | 2.0 |+| | native | 12/12 | 10,901 | $2.44 | 8.0 |+| | **xerj** | **12/12** | **7,497** | **$2.01** | 5.2 |+| Opus 4.8 | bare | 12/12 | 35,272 | $6.11 | 2.0 |+| | native | 12/12 | 11,693 | $2.09 | 6.2 |+| | **xerj** | 11/12 | **8,385** | **$1.81** | 4.5 |+| Opus 5.0 | bare | 12/12 | 42,354 | $5.98 | 1.6 |+| | native | 12/12 | 11,516 | $2.15 | 5.2 |+| | **xerj** | **12/12** | **8,127** | **$1.94** | 3.4 |++**Headline ratio (native → xerj output tokens), the identical-agent comparison:**++| model | native/xerj out-tok | bare/xerj out-tok | cost native/xerj |+|---|---:|---:|---:|+| Opus 4.6 | **1.45×** | 10.07× | 1.21× |+| Opus 4.8 | **1.39×** | 4.21× | 1.15× |+| Opus 5.0 | **1.42×** | 5.21× | 1.11× |++XERJ is **~1.4× fewer output tokens than grep across all three models** (consistent+with the earlier case study's ~1.65×), in fewer rounds, at lower cost. Note Opus 4.6+`bare` solved only **4/12 even with 5 retry rounds** while burning **$16.75** — the+weaker the model, the more reference-coding matters, on both axes.++## 4. How to read this honestly++- The **correctness** benefit is real but **model-dependent**: dramatic for weaker+ recall (4.6/4.8 pass@1 0/12→12/12), modest for the frontier model (5.0: 7→11).+- The **efficiency** benefit (`native`→`xerj` ~1.4×, cost 1.1–1.2×) is **consistent+ across all three models** and is the safest headline claim.+- Do **not** claim a flat "0→12 for every model." The defensible statement is the+ nuanced one above.++## 5. The honest boundary — memorized code (measured, same harness)++The value is **gated by memorization**. On the `fieldnorm-quant` task over **tantivy**+(a popular public crate the model has trained on), from the case study:++| arm | solved | med output tok | total $ |+|---|---:|---:|---:|+| bare (memory) | 3/3 | 10,213 | $0.98 |+| **native** (grep) | 3/3 | 1,440 | **$0.60 — best** |+| xerj (inject) | 3/3 | 8,504 | $1.43 — **worst** |++All arms pass because the code is memorized (`bare` reproduced the exact 256-entry+norm table from memory). Here **retrieval is the worst arm** — injecting a large+reference backfires on bulk-data reproduction. Reference-coding wins on **private,+internal, niche, or post-cutoff** code; it is overhead on popular public libraries the+model already knows. That boundary is a feature of the honest claim, not a caveat to+hide.++## 6. Recognized benchmarks (SWE-bench Verified) — transparency++The recognized benchmark that tests this capability is **SWE-bench Verified** (patch a+real repo so its hidden tests pass — the natural fit for "index the repo and retrieve+the code to change"). It **could not be run in this sandbox**, and this was verified,+not assumed:++- **No usable Docker daemon** — SWE-bench's official harness runs each repo's tests in+ per-repo Docker images; the daemon is present but not runnable here.+- **Python-version wall** — only Python 3.14 is available; SWE-bench instances pin+ 2013–2020-era commits (e.g. `psf__requests-1142` @ 2013) whose `setup.py` will not+ build under 3.14 / modern setuptools. Probed end-to-end on `flask-5014` and+ `requests-1142`: both fail environment setup at baseline.++We do **not** substitute a hand-rolled n≈2 scaffold and call it "the SWE-bench score" —+that is precisely the kind of number that is not defensible. Instead: a **turnkey+SWE-bench Verified harness** (with the XERJ index/retrieve arm) is provided so the+**official** number reproduces in any Docker-capable environment — see+[`tools/xerj-code/swebench/README.md`](../../tools/xerj-code/swebench/README.md).++## 7. Reproduce++```sh+# server + corpus+xerj --insecure --port 9450 --data-dir /tmp/xbench --embed-mode lexical & # background+XERJ_URL=http://localhost:9450 XERJ_BIN=$PWD/engine/target/release/xerj \+ bash .claude/skills/xerj-code/scripts/xc-index.sh novel-libs --fresh++# soundness (must print ALL SOUND)+cd .claude/skills/xerj-code/measure && python3 validate_sift.py++# a model, both settings+TASKS="tasks/sift-build.json tasks/sift-until.json tasks/spool-window.json \+ tasks/trellis-order.json tasks/quill-codec.json tasks/weft-scan.json"+ANTHROPIC_MODEL=claude-opus-4-8 XERJ_URL=http://localhost:9450 \+ python3 csrun.py novel-libs $TASKS --arms bare,native,xerj --trials 2 \+ --corpus-dir ~/.xerj-code/corpora/novel-libs --out results-opus48.json # efficiency+ANTHROPIC_MODEL=claude-opus-4-8 XERJ_URL=http://localhost:9450 \+ python3 csrun.py novel-libs $TASKS --arms bare,xerj --trials 2 --rounds 1 \+ --out results-pass1-opus48.json # correctness+```++Raw result JSONs for this run: `/home/claude/ai/xerj-bench/results-*.json`.++## 8. Limitations (stated, not buried)++- n is modest (6 tasks × 2 trials/model); pass/fail on contamination-free API tasks is+ fairly deterministic, but confidence intervals at this n are wide — treat the+ *direction and magnitude* as the result, not the third significant figure.+- Purpose-built crates guarantee non-memorization but are simpler than real repos; the+ recognized-repo signal is exactly what the SWE-bench harness (§6) is for.+- `bare` is memory + compile-error retries, not pure one-shot except in the pass@1+ setting; both settings are reported so the reader picks the lens.+- A `kv-oss` re-index during this session hit an engine catalog error+ (`catalog read-back … disagrees with the sealed generation projection`) — filed as a+ separate finding; the memorized boundary (§5) is cited from the prior verified run.diff --git a/tools/xerj-code/swebench/README.md b/tools/xerj-code/swebench/README.mdnew file mode 100644index 000000000..4df0d2206--- /dev/null+++ b/tools/xerj-code/swebench/README.md@@ -0,0 +1,92 @@+# SWE-bench Verified × XERJ reference coding — turnkey harness++Measures whether XERJ reference-coding improves an agent on **SWE-bench Verified**,+the benchmark the AI labs score coding agents on. It is split so the **verdict is the+official one**: this repo only *generates* patches; scoring is the standard+`swebench.harness.run_evaluation` in Docker. The number you get is the real leaderboard+number, for the subset you run.++> **Why this isn't run in the XERJ sandbox.** SWE-bench's evaluation runs each repo's+> tests in per-repo **Docker** images, and its instances pin **2013–2020-era Python**+> commits. A sandbox without a Docker daemon, or with only a modern Python (3.14), can+> neither evaluate nor even set up baselines (verified: `flask-5014`, `requests-1142`+> both fail environment setup on 3.14). Run this on a **Docker-capable Linux box**.++## Arms++Identical agent, one difference — the clean test of reference-coding:++- **`native`** — the repo is checked out; the agent must **grep** to find the code to change.+- **`xerj`** — same, but XERJ has indexed the repo and **pre-retrieved** the relevant+ passages into the prompt.++(No `bare` arm: patching a real repo from pure memory is not a meaningful task. The+`bare` floor lives in the contamination-free study, `measure/CASE_STUDY.md`.)++## Prerequisites++```sh+pip install swebench datasets # official harness + dataset loader+docker info # must succeed (daemon running)+claude --version # authenticated Claude Code CLI+xerj --version # XERJ release binary (set XERJ_BIN to its path)+```++Start a XERJ node the `xerj` arm will index into (a private port, throwaway data dir):++```sh+xerj --insecure --port 9200 --data-dir /tmp/swe-xerj --embed-mode lexical &+export XERJ_URL=http://localhost:9200 XERJ_BIN=$(command -v xerj)+```++## 1. Generate predictions (both arms, per model)++Start small — one lightweight repo — to confirm the loop end-to-end, then scale.++```sh+REPOS="psf/requests pytest-dev/pytest" # or drop --repos for the full 500++for arm in native xerj; do+ ANTHROPIC_MODEL=claude-opus-4-8 \+ python3 gen_predictions.py --arm $arm --repos $REPOS --limit 20 \+ --out preds-opus48-$arm.jsonl --tokens-out tokens-opus48-$arm.jsonl+done+```++Repeat with `ANTHROPIC_MODEL=claude-opus-5`, `claude-opus-4-6`, … The model that+actually ran is recorded from claude's JSON `modelUsage` in the `--tokens-out` log, so+the selection is verifiable, not assumed.++## 2. Score with the OFFICIAL harness (Docker)++```sh+for arm in native xerj; do+ python3 -m swebench.harness.run_evaluation \+ --dataset_name princeton-nlp/SWE-bench_Verified \+ --predictions_path preds-opus48-$arm.jsonl \+ --run_id opus48-$arm --max_workers 4+done+```++Each run prints/writes a report with **resolved / total** — that is the SWE-bench+resolve-rate for that arm+model, computed by the repos' own `FAIL_TO_PASS` /+`PASS_TO_PASS` tests. The XERJ claim is the **native → xerj delta** in resolve-rate,+alongside the **token/cost delta** from the `--tokens-out` logs.++## 3. Read it honestly++- Report **resolve-rate per arm** and the **native→xerj delta**, plus output-tokens and+ \$/instance from the token logs. State **n** (instances actually evaluated) and which+ repos — do not extrapolate a subset to the full 500.+- On SWE-bench repos the model has largely trained on, the *resolve-rate* lift from+ retrieval may be small (memorized code — see the boundary in+ `../../measure/CASE_STUDY.md` and `demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md`);+ the defensible signal there is usually **efficiency** (fewer grep turns / output+ tokens for the same fix). Report whatever the data shows, wins and losses.+- Keep the agent scaffold identical between arms (same `--max-turns`, same prompt+ scaffolding) so the only variable is XERJ pre-retrieval.++## Files++- `gen_predictions.py` — generates the `native`/`xerj` prediction JSONLs (+ token logs).+- scoring — the official `swebench.harness.run_evaluation` (not vendored here on purpose).diff --git a/tools/xerj-code/swebench/gen_predictions.py b/tools/xerj-code/swebench/gen_predictions.pynew file mode 100644index 000000000..21b0d232f--- /dev/null+++ b/tools/xerj-code/swebench/gen_predictions.py@@ -0,0 +1,158 @@+#!/usr/bin/env python3+"""+SWE-bench Verified — generate model patches for the `native` and `xerj` arms.++This is the GENERATION half of a defensible SWE-bench run. It does NOT judge+correctness itself: it writes standard SWE-bench prediction files, and you score+them with the OFFICIAL `swebench.harness.run_evaluation` (Docker) — so the number+is the real leaderboard number, not a home-grown verdict. See README.md.++Two arms, identical agent, one difference:+ native : the repo is checked out; the agent must FIND the code to change by grepping.+ xerj : same, but XERJ has indexed the repo and PRE-RETRIEVED the relevant+ passages, injected into the prompt. (This is the reference-coding arm.)+(There is deliberately no `bare` arm: patching a real repo from pure memory is+not a meaningful task.)++Output: one predictions JSONL per (arm, model), lines of+ {"instance_id", "model_name_or_path", "model_patch"}++Requires: `datasets`, the `claude` CLI (authenticated), a running XERJ+(`XERJ_URL`, default http://localhost:9200) with its binary on `XERJ_BIN`, git.+Model is selected per run via ANTHROPIC_MODEL (verified from claude's JSON+`modelUsage`).+"""+import argparse, json, os, subprocess, tempfile, shutil, sys, time, pathlib++XERJ_URL = os.environ.get("XERJ_URL", "http://localhost:9200")+XERJ_BIN = os.environ.get("XERJ_BIN", "xerj")+XC = os.path.join(os.path.dirname(__file__), "..", "scripts", "xc.py")+++def sh(cmd, cwd=None, timeout=None, env=None):+ return subprocess.run(cmd, cwd=cwd, timeout=timeout, capture_output=True,+ text=True, env={**os.environ, **(env or {})})+++def clone_at(repo, base_commit, dest):+ sh(["git", "clone", "--quiet", f"https://github.com/{repo}", dest])+ sh(["git", "checkout", "--quiet", base_commit], cwd=dest)+++def retrieve_context(repo_dir, instance_id, problem, k=6, chars=8000):+ """xerj arm: index the repo, retrieve passages relevant to the issue."""+ prefix = "swe-" + instance_id.replace("/", "-")+ idx = sh([XERJ_BIN, "autoindex", repo_dir, "--url", XERJ_URL,+ "--prefix", prefix, "--no-graph"], timeout=1800)+ # exit 3 == completed-with-junk == success+ if idx.returncode not in (0, 3):+ return None, f"index-failed:{idx.returncode}:{idx.stderr[:160]}"+ # first ~3 lines of the issue make the best lexical query on the default embedder+ query = " ".join(problem.split("\n")[:3])[:300]+ r = sh(["python3", XC, prefix, query, "--k", str(k), "--chars", str(chars),+ "--full"], env={"XERJ_URL": XERJ_URL})+ if r.returncode != 0:+ return None, f"retrieve-failed:{r.returncode}:{r.stderr[:160]}"+ return r.stdout, None+++def run_agent(repo_dir, problem, ref_block, timeout, max_turns):+ ref = ""+ if ref_block:+ ref = ("\n\nXERJ retrieved these passages from THIS repository — the code you "+ "most likely need to change is here (cite file:line):\n\n" + ref_block)+ prompt = (+ "You are fixing a bug in the checked-out repository at the current working "+ "directory. Here is the issue:\n\n" + problem ++ "\n\nEdit the source files to fix it. Do NOT edit or add tests — a hidden "+ "suite will judge your patch. Do not run the test suite. Make the smallest "+ "correct change. When done, stop." + ref)+ t0 = time.time()+ p = subprocess.run(+ ["claude", "-p", prompt, "--output-format", "json",+ "--permission-mode", "bypassPermissions", "--max-turns", str(max_turns)],+ cwd=repo_dir, capture_output=True, text=True, timeout=timeout)+ el = time.time() - t0+ usage, model = {}, None+ try:+ d = json.loads(p.stdout)+ u = d.get("usage", {})+ usage = {"in": u.get("input_tokens", 0), "out": u.get("output_tokens", 0),+ "cost": d.get("total_cost_usd", 0.0)}+ model = list(d.get("modelUsage", {}).keys())+ except Exception:+ pass+ diff = sh(["git", "diff"], cwd=repo_dir).stdout+ return diff, usage, model, el+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--arm", choices=["native", "xerj"], required=True)+ ap.add_argument("--dataset", default="princeton-nlp/SWE-bench_Verified")+ ap.add_argument("--repos", nargs="*", help="restrict to these repos (e.g. psf/requests)")+ ap.add_argument("--limit", type=int, default=0, help="max instances (0=all)")+ ap.add_argument("--instances", nargs="*", help="explicit instance_ids")+ ap.add_argument("--max-turns", type=int, default=60)+ ap.add_argument("--timeout", type=int, default=1200)+ ap.add_argument("--out", required=True)+ ap.add_argument("--tokens-out", help="optional per-instance token/cost log (JSONL)")+ args = ap.parse_args()++ from datasets import load_dataset+ ds = load_dataset(args.dataset, split="test")+ rows = list(ds)+ if args.repos:+ rows = [r for r in rows if r["repo"] in set(args.repos)]+ if args.instances:+ rows = [r for r in rows if r["instance_id"] in set(args.instances)]+ if args.limit:+ rows = rows[:args.limit]+ model_name = os.environ.get("ANTHROPIC_MODEL", "default") + f"+xerj-{args.arm}"+ print(f"generating {len(rows)} predictions | arm={args.arm} | model_env="+ f"{os.environ.get('ANTHROPIC_MODEL','default')}", file=sys.stderr)++ outf = open(args.out, "w")+ tlog = open(args.tokens_out, "w") if args.tokens_out else None+ for i, r in enumerate(rows, 1):+ iid = r["instance_id"]+ work = tempfile.mkdtemp(prefix="swe-")+ repo_dir = os.path.join(work, "repo")+ try:+ clone_at(r["repo"], r["base_commit"], repo_dir)+ ref, err = (None, None)+ if args.arm == "xerj":+ ref, err = retrieve_context(repo_dir, iid, r["problem_statement"])+ if err:+ print(f" [{i}/{len(rows)}] {iid} retrieve error: {err}", file=sys.stderr)+ diff, usage, model, el = run_agent(+ repo_dir, r["problem_statement"], ref, args.timeout, args.max_turns)+ outf.write(json.dumps({"instance_id": iid,+ "model_name_or_path": model_name,+ "model_patch": diff}) + "\n")+ outf.flush()+ if tlog:+ tlog.write(json.dumps({"instance_id": iid, "arm": args.arm,+ "model": model, "usage": usage,+ "elapsed_s": round(el, 1),+ "patch_bytes": len(diff)}) + "\n")+ tlog.flush()+ print(f" [{i}/{len(rows)}] {iid} {model} out={usage.get('out')} "+ f"patch={len(diff)}B {el:.0f}s", file=sys.stderr)+ except subprocess.TimeoutExpired:+ print(f" [{i}/{len(rows)}] {iid} TIMEOUT", file=sys.stderr)+ outf.write(json.dumps({"instance_id": iid,+ "model_name_or_path": model_name,+ "model_patch": ""}) + "\n")+ except Exception as e:+ print(f" [{i}/{len(rows)}] {iid} ERROR {e}", file=sys.stderr)+ finally:+ shutil.rmtree(work, ignore_errors=True)+ outf.close()+ if tlog:+ tlog.close()+ print(f"wrote {args.out}", file=sys.stderr)+++if __name__ == "__main__":+ main()