AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
XERJ.code case study — does reference retrieval beat native Claude Code?
Status: complete (2026-08-05). Six tracks, real tokens and dollars, across five languages (Rust, Python, JavaScript, C, Java): thirteen purpose-built unfamiliar-code libraries plus two memorised controls (tantivy, valkey+memcached). Every number produced by
csrun.py/csmulti.py+summarize.pyon this machine. Full method (what/which repos indexed, how indexed, how searched, how to reproduce) below. Raw data indata/results-*.json; captured generated programs ingenerated/.One-line answer: on the seven domains carrying a contract the model cannot recall (a seal, a generational handle, a lazy refill, a specific hash scheme, an exact tie-break), XERJ makes the same Claude Code solve 21/21 where bare solves 1/21, at 1.5× fewer output tokens and 1.3× lower cost than native Claude Code, and 6.5× cheaper than flailing from memory ($3.38 vs $21.9). The compiler can leak an API name but never a contract. On memorised public code, retrieval is overhead. The value is gated by memorization; the line is sharp.
The question, stated so it can be answered
The earlier results (RESULTS.md) established one thing cleanly and left the
central claim open:
- Established: on public code the model has memorised (memchr, regex, aho-corasick — three of the most-cloned Rust crates), the bare model is at ceiling. Retrieval adds context and changes nothing, because the model already knows the answer. A retrieval system cannot demonstrate value on a corpus its reader has memorised.
- Open: the case the tool is actually for — code the model has not memorised (private, recent, or idiosyncratic) — was never measured.
This study measures that case, in the currency the claim is made in: real
tokens and real dollars, from claude -p --output-format json, not the byte
proxies the earlier work was limited to. And it compares against the right
baseline — not a bare model, but native Claude Code with tools, which is what
a developer actually competes against.
Three arms — and why the two peers must be identical
The load-bearing insight (and a correction made mid-study): xerj is Claude Code plus a prompt plus a retrieved snippet. It is the same model and the same agent. So the honest comparison is not "a crippled model vs a full agent" — it is two runs of the identical agentic Claude Code that differ in exactly one thing: whether XERJ pre-retrieved the reference, or the agent had to grep for it.
| arm | modality | reference material | what it models |
|---|---|---|---|
| native | agentic Claude Code, full tools, cargo build self-verify |
the cloned repos mounted on disk | Claude Code as-is: grep the source, then write |
| xerj | the same agentic Claude Code, same tools, same self-verify | one retrieved passage injected; corpus not mounted | Claude Code + XERJ: the retrieval replaces the grep |
| bare (floor) | tool-free, memory only | none | a lower bound: what the model knows without any lookup |
The native vs xerj comparison is the headline — both are the same Claude
Code, so any difference is attributable to the retrieval, not the harness. bare
is a supplementary floor (no tools, no self-verify) that shows the raw knowledge
gap; it is not a peer of the other two and is never compared on cost as if it
were.
A discarded first design, kept as a lesson. The xerj arm was initially run tool-free and single-shot — no
cargo build, no self-verify — while native was fully agentic. That handicapped xerj: on BM25 it copied tantivy's internalScore/FieldNormReadertypes out of the injected file and, unable to compile, looped and failed, while native (which could compile and adapt) passed. That is a harness artifact, not a fact about retrieval: the same Claude Code cannot be worse for having more information in front of it. The arms were rebuilt so the two peers are byte-for-byte the same agent. The earlier run was discarded.
The honesty rails that remain:
- The verdict is objective and hidden. No arm sees the tests.
validate_*.pyproves, before any model runs, that the tests reject the wrong answers a model would actually produce (a forgotten finalizer, a textbook-instead-of-Lucene formula, guessed API names). A benchmark whose tests pass wrong code measures nothing. bareis genuinely tool-free.--disallowedToolsblocks Bash/Read/Grep etc. (An even earlier draft used--tools "", which does not disable tools — the "bare" model shelled out and vendored the dependency source. Discarded, and documented, because it is exactly the leak that turns a null into a false positive.)- xerj is not given the corpus to grep. It gets only the retrieved passage, because the whole point is that XERJ's index answers the query so the agent does not carry or crawl the whole corpus.
Cost model (why output tokens are the ones that matter)
Opus 5 list rates: input $5/MTok, output $25/MTok (5×), cached input
~$0.50/MTok. A retry lap spends output; a retrieved passage spends input. So the
trade is: pay cheap input once to avoid expensive output laps. Full arithmetic in
COSTS.md. This study reports output tokens, total input tokens, and
total_cost_usd (the honest bottom line, which already prices cache reads at
0.1×) for every arm.
Method — what was indexed, how, and how it was searched
Every arm that retrieves does so through the shipped xerj-code scripts against a
local XERJ instance. Nothing is mocked. This section is the full provenance.
The reference corpora
| corpus | what it is | source | indexed records |
|---|---|---|---|
sift-lib |
1 purpose-built Rust lib (sift) |
measure/reflib/sift |
3 |
novel-libs |
9 purpose-built Rust libs + the tantivy tree as bulk | measure/reflib/* + tantivy |
102,623 |
ml-libs |
4 purpose-built libs, one each in Python / JS / C / Java | measure/reflib-ml/* |
5 |
search-eco |
tantivy (real public Rust search engine) | github.com/quickwit-oss/tantivy | 102,606 |
kv-oss |
valkey + memcached (real public C servers) | github.com/valkey-io/valkey, github.com/memcached/memcached | 7,923 |
rust-text |
memchr + regex + aho-corasick (real public Rust) | github.com/BurntSushi/*, rust-lang/regex | 53,873 |
Two kinds of reference deliberately:
- Purpose-built libraries (
reflib/,reflib-ml/) — real code (each compiles and passes its own test suite) written for this study, so it cannot be in any training set. Each carries a runtime contract the compiler cannot warn about (a seal-before-read, a generational handle, a lazy refill, a promote-on-fetch). This is the only honest way to measure retrieval on "code the model has not memorised" without waiting for a genuinely private codebase. - Real public repos — tantivy, valkey, memcached, memchr/regex/aho-corasick. These are the memorised controls: the model has trained on them, so they show what retrieval is worth when the model already knows the answer (nothing — see Tracks 2–3).
How they were indexed
Two wrapper scripts, shipped at the time in tools/xerj-code/
(since removed — the xerj subcommands that replaced them are listed under
How you reproduce it):
xc-corpus.sh <name> <git-url>… # shallow clone (--depth 1) + record each repo's licence
xc-index.sh <name> # xerj autoindex <dir> --url <XERJ> --prefix xc-<name> --no-graph
xc-corpus.sh writes a corpus.json next to the clones — URLs, full commit
SHAs and licences, a few hundred bytes, no source. That file is the shareable
definition of a corpus: xc-corpus.sh --from <manifest> rebuilds it at the same
commits somewhere else, which is what makes two people's retrieval results
comparable. Pinned, licence-reviewed definitions for the corpora used here are
in tools/xerj-code/hub/.
xc-index.sh runs xerj autoindex — XERJ's zero-config folder ingester: it
sniffs file types, extracts per-file records (for source it pulls symbol/definition
metadata via tree-sitter), and bulk-loads them into the local engine under the
prefix xc-<name>. --no-graph skips the relationship-graph pass (this use needs
ranked passages, not edges). Indexing is fast: the 102k-record novel-libs tree
indexed in ~10s. A purpose-built lib is 1 record (one file); a real repo is
thousands.
How they were searched
xc.py <corpus> "<query>" hits XERJ's Elasticsearch-compatible _search and
returns ranked passages with file:line provenance:
- default ranking is BM25 (
multi_matchover the file body and its extracted definitions); a hybrid BM25+vector RRF mode exists but BM25 is the default and what these runs used. --no-symbolreturns the whole matching file (right for a small, self-contained reference library — the model gets the full API + doc example in one passage); without it,xc.pyreturns just the matching definition located from the file'ssymbols[].- the query is behavioural keywords (e.g. "generational handle plant pluck wither stale reuse"); the top hit is the correct library even when it is one file among 102k records (measured score 80–95, ranked above all of tantivy).
The xerj arm gets that passage injected into its prompt. The native arm is
handed the same corpus tree on disk and must grep/read to find the API itself —
that search cost is what the two arms differ by. See a worked query and its output
in tools/xerj-code/SKILL.md.
How you reproduce it
The retrieval half ran as follows at the time: start XERJ, then
xc-corpus.sh (or xc-corpus.sh --from tools/xerj-code/hub/<corpus>.json, which
pins the exact commits these runs used) → xc-index.sh → xc.py. It needs the
xerj binary, git and python3. To point it at your own code, index your
repo instead of a corpus — that is the real use case.
(As of #977 these are binary
subcommands — xerj corpus add --from, xerj corpus index, xerj code — and
no longer need python3; the commands above are what was actually run at the
time of this study.)
The measurement harness — the three-arm runner, the purpose-built reference
libraries, the hidden-test task specs — is not in this repository; only its
outputs are (data/results-*.json, generated/). What it does, if you want to
build the equivalent: run three arms of the same agent (memory only / grep-driven
/ retrieval-injected) over tasks whose verdict is objective — the code compiles
and passes a hidden test the model never sees — using the claude CLI's own
token and cost accounting (claude -p --output-format json, no API key), and
report medians. Validate each task first by injecting a known-correct and a
known-wrong solution; a benchmark whose tests pass wrong code measures nothing.
Track 1 — novel API (the mechanism, isolated)
Corpus: sift, a purpose-built, real, compiling Rust library (a
conservative-update count-min sketch) with an API the model cannot have seen:
furnish/absorb/seal/crest/fuse, and a two-phase contract (you must
seal() before any read; the compiler cannot warn you if you forget — only a
runtime panic does). Source in reflib/sift/; it passes its own test suite.
Why synthetic here is correct: this track isolates one variable — can the
model use an API it must retrieve because it cannot recall it? A real library
would confound "unfamiliar" with "partially memorised." The three tasks each
return a real sift::Sift, so a HashMap workaround cannot satisfy them, and each
probes a different unguessable facet:
sift-build— construction + the seal-before-read contract + the crest querysift-merge—fusemust happen beforeseal(an ordering the compiler cannot hint at)sift-until—absorbreturns the running estimate (a non-obvious semantic)
Task soundness (measured before any model ran): validate_sift.py — golden
solution passes; a version that forgets seal() fails (runtime, not compile); a
version using guessed new()/push() names does not compile. All three
discriminate.
Results — retrieval wins decisively
3 tasks (sift-build, sift-merge, sift-until) × 3 trials, corrected agentic
arms. Medians per arm; totals are the sum across all 9 runs.
| arm | solved | med out-tok | med in-tok | total $ |
|---|---|---|---|---|
| bare (memory) | 0/9 | 18,843 | 306,000 | $9.25 |
| native (grep the dep) | 9/9 | 1,220 | 123,000 | $1.93 |
| xerj (retrieval injected) | 9/9 | 756 | 100,000 | $1.54 |
- xerj vs bare: 24.9× fewer output tokens, 6.2× cheaper — and success vs total failure. Bare never solves a single sift task in 9 attempts, and it burns $9.25 doing so: it writes an entire count-min sketch from scratch (one round hit 27,928 output tokens) with guessed method names, fails to compile, and gives up. This is the "many loops wasting tokens" the premise describes, measured.
- xerj vs native: 1.6× fewer output tokens, 1.2× cheaper, and 4.4 vs 7.0 turns. Both are the same agentic Claude Code and both solve every task; xerj is leaner because the retrieval replaces the grep. The gap is modest here only because the sift reference is a single file — native's grep is cheap. It widens with corpus size (see Track 3's input-token blow-up).
The result is dead stable across trials: xerj sift-build ran 620 / 580 / 565 output tokens on its three trials. This is the regime the tool is built for — an API the model cannot recall — and it is the only track where retrieval wins.
Track 2 — search internals (the real-product anchor)
Corpus: search-eco — tantivy (Rust), cloned and indexed on a second XERJ
instance. This is the case XERJ's own developers live in: implement a search
internal so it is byte-compatible with an existing engine, using the ecosystem as
reference. It is also the track where native vs xerj is most informative:
native must grep a large multi-file corpus to find the reference; xerj retrieves
it in one call, and the token cost of that difference is the measurement.
Task — fieldnorm-quant: reproduce tantivy's field-norm quantization — the
exact 256-entry FIELD_NORMS_TABLE and the id_to_fieldnorm / fieldnorm_to_id
functions that encode field lengths into one byte. The table is a specific
non-linear curve (id 100 → 3096, id 200 → 16,777,240); it is not derivable by
formula, so it is the purest form of "knowledge the model cannot have and must
retrieve." Soundness checked: golden (the real table) passes; a plausible linear
guess fails.
Why not BM25 (a discarded task, kept as a lesson). The first Track-2 task was "implement BM25 exactly as Lucene/tantivy compute it." It turned out to be a bad measurement, and finding out why was itself instructive. BM25 has several real conventions that even the references disagree on — the
ln(1+x)idf form, and crucially the(k1+1)numerator factor, which Lucene ≥8 removed but tantivy keeps. Bothnativeandxerjtest-failed it: native read tantivy's code (which includes(k1+1)) and then reasoned itself out of it, citing the Lucene-8 change; the exact-1e-9 tolerance also turned tantivy'sScore=f32and lossy fieldnorm cache into precision noise. So BM25 measured "does the agent argue with the reference," not "does retrieval fill a knowledge gap." It was replaced withfieldnorm-quant, where there is nothing to argue with: you have the 256 numbers or you don't. The finding stands on its own — codegen against entangled, convention-divergent reference code is unreliable for reasons that have nothing to do with where the code came from.
Results — retrieval is neutral-to-harmful (the code is memorized)
fieldnorm-quant, 3 trials, corrected agentic arms.
| arm | solved | med out-tok | med in-tok | total $ |
|---|---|---|---|---|
| bare (memory) | 3/3 | 10,213 | 17,470 | $0.98 |
| native (grep + copy) | 3/3 | 1,440 | 121,042 | $0.60 |
| xerj (inject) | 3/3 | 8,504 | 149,217 | $1.43 |
All three arms pass — because tantivy is memorized. bare reproduced the
exact 256-entry FIELD_NORMS_TABLE from memory (id 100 → 3096, id 200 →
16,777,240), even writing the correct structural comment ("piecewise-exponential,
groups of 8, doubling step"). A table that looks impossible to guess is
memorized when its source is a popular public crate — the same wall the original
RESULTS.md hit on memchr/regex, now confirmed on a fourth corpus.
Here retrieval is the worst arm. native greps code.rs and copies the
table concisely; xerj has the whole file injected and re-emits the 256-value
table as output (sometimes retranscribing it), spending 5.9× native's output
tokens. Injecting a large reference backfires when the task is bulk-data
reproduction. (Caveat: this makes fieldnorm-quant partly a transcription
benchmark; the clean retrieval-value signal is Track 1.)
Track 3 — realism control (public, memorised code)
Corpus: kv-oss (valkey + memcached). Re-runs the existing RESP/memcached-KV
tasks — public protocols the model knows — now with real tokens and the native
arm. Expected to reproduce the earlier "bare is at ceiling" finding; included so
the contrast with Track 1 is measured on the same instrument, not asserted.
Results — pure memory wins; retrieval and grep are both overhead
resp-kv + memcached-kv, 3 trials each, corrected agentic arms.
| arm | solved | med out-tok | med in-tok | total $ |
|---|---|---|---|---|
| bare (memory) | 6/6 | 7,886 | 17,500 | $1.49 |
| native (grep) | 6/6 | 21,380 | 1,060,000 | $9.14 |
| xerj (inject) | 5/6 | 13,954 | 141,000 | $4.40 |
The ranking inverts from Track 1. On code the model has memorised, bare
is cheapest and most reliable — it just writes the protocol handler. native is
the worst: it grep-crawls valkey + memcached for code the model already knows,
averaging 28 turns and pulling ~1.06M input tokens of file content into
context — an 7.5× input blow-up over xerj, and the concrete cost of "grep the
corpus" scaling with corpus size. xerj sits in between and was also the only arm
to miss a solve (5/6): the injected snippet is unnecessary here and can distract.
xerj beats native 2.3× on cost here — but that is the wrong headline: both
lose to doing nothing (bare). When the model knows the answer, every form of
retrieval is a tax.
Track 4 — three more unfamiliar domains, buried in a large corpus
Track 1 (sift) showed retrieval wins on one unfamiliar API, but two questions stayed open: does it generalise past a single library, and does the native-vs-xerj gap widen when the reference is hard to find (sift was one file, so native's grep was cheap)? Track 4 answers both.
Three new purpose-built reference libraries, each real (compiles, passes its own tests), each in a different domain, each with an API the model cannot recall and a runtime contract the compiler cannot warn about:
grove— a bump arena with generational handles:sprout/plant/pluck/wither, where a freed handle is detectably stale and its slot is reused under a bumped generation. (contract: stale reads returnNone, never the new occupant.)weft— a table-driven byte scanner builtwarp→thread→weaveinto a separateLoomtype. (contract: builder and scanner are different types; you mustweavebefore you canscan; maximal-munch runs.)tally— fixed-point decimals with banker's rounding:of/add/quantize, result ofaddcarries the larger scale. (contract:quantizerounds half-to-even, not half-up.)
Each task returns the library's own type, so a hand-rolled workaround cannot
satisfy it; validate confirms golden passes and the guessed-API
(new/insert/build…) does not compile.
The corpus is large on purpose. All four novel libraries are indexed
alongside the entire tantivy tree (≈102k records, 560 files). So the
reference is a needle in a real haystack. xerj retrieves it directly (top hit,
score 80–95, ranked above all of tantivy); native is pointed at the same tree
and must search it — the realistic "grep your vendored deps" cost.
Results — the sift win generalises, cleanly
3 tasks (grove-compact, weft-scan, tally-round) × 2 trials.
| arm | solved | med out-tok | med in-tok | total $ |
|---|---|---|---|---|
| bare (memory) | 1/6 | 16,112 | 367,698 | $5.93 |
| native (search the tree) | 6/6 | 1,344 | ~150,000 | $1.24 |
| xerj (retrieval) | 6/6 | 892 | ~98,000 | $0.95 |
- Generality: across four unfamiliar domains now (streaming sketch, arena allocator, lexer, fixed-point decimal), bare essentially fails (1 of 15 solves combined with sift) while native and xerj each solve 15/15. The sift result was not a quirk of one library.
- xerj vs native: 1.5× fewer output tokens, 1.2× cheaper, and 4.7 vs 7.7 turns — the same shape as sift, reproduced on three fresh domains. xerj skips the search; native spends 3 extra turns finding the API.
- The reference was buried in ~102k records (the tantivy tree), and xerj still retrieved the right library as the top hit (score 80–95). Native found it too — grep stays precise when the API name is distinctive — but pulled ~1.5× more input tokens into context doing so (~150k vs ~98k). The corpus-size penalty on native is real but modest here; it explodes only when the search is unfocused (see Track 3's kv, where native hit 1.06M input tokens).
One honest wrinkle: the compile-loop can leak an API
weft bare passed once (1/2) — not from memory, but because after its first
compile error, rustc's own "did you mean thread?" suggestions leaked the real
method names, and it recovered on round two. It cost 15,075 output tokens to
xerj's 610 (25×). So the tool-free loop is a weak, expensive substitute for
retrieval on API names the compiler can suggest — but it cannot leak a runtime
contract (grove's generational staleness, tally's banker's rounding), which is
why bare fails those outright. Retrieval delivers names and contract together, in
one shot.
Track 5 — five more domains, and a sharper picture of when
Five more purpose-built libraries, five more domains, same three arms (2 trials):
spool (power-of-two ring buffer that returns what it evicts), cadence
(lazy-refill token bucket), quill (zig-zag + LEB128 varint codec),
trellis (topological sort with a smallest-index tie-break and a cycle
error), sieve (Bloom filter with Kirsch–Mitzenmacher double hashing and a
settle-before-sense seal). Each real, each with a hidden test; corpus is the
same ~102k-record tree.
| task | bare | native | xerj | med out-tok (bare/native/xerj) |
|---|---|---|---|---|
| cadence-run | 0/2 | 2/2 | 2/2 | 18,162 / 1,114 / 744 |
| trellis-order | 0/2 | 2/2 | 2/2 | 21,318 / 1,091 / 782 |
| sieve-member | 0/2 | 2/2 | 2/2 | 13,817 / 1,048 / 670 |
| spool-window | 2/2 | 2/2 | 2/2 | 15,974 / 935 / 656 |
| quill-codec | 2/2 | 2/2 | 2/2 | 1,356 / 1,242 / 999 |
| total | 4/10, $9.11 | 10/10, $1.82 | 10/10, $1.49 | xerj vs native: 1.5× / 1.2× |
The five cases map the boundary precisely, and that is their value:
- Three clean wins (
cadence,trellis,sieve). The contract is genuinely unrecallable — a lazy refill, a specific tie-break + cycle-node set, a specific double-hashing scheme. bare fails 0/2 on each, burning 14–21k output tokens reinventing the wheel; xerj takes it to 2/2 at ~700 tokens. Retrieval is the difference between working and broken code, at ~20× fewer tokens. - One expensive recovery (
spool). The spec necessarily says "the buffer reports an evicted value," which leaks the one non-obvious bit of the contract, andrustc's suggestions supply the method names. So bare does reach 2/2 — but over two rounds and 16k output tokens against xerj's 656 (24×). - One honest non-win (
quill). A zig-zag + LEB128 varint is protobuf's wire format; the model has it memorised and reproduces it in ~1,356 tokens without the reference. Retrieval saves nothing here — it is a known convention, and the study says so.
What the generated code actually looks like
sieve-member, one task, three arms — this is the whole argument in one frame.
bare (tool-free) cannot read the crate, so it guesses the API from compiler error messages and hedges with fallback traits — its own opening comment:
//! The vendored `sieve` crate source is not readable from this session ...
//! so the exact spelling of the constructor / record / finalize entry points has
//! to be inferred. The compiler told us one thing for sure: `Sieve` has no `new`
//! ... To avoid betting the whole module on a single spelling, the calls below
//! are written against the most likely names and backed by fallback traits.
It produced 282 lines of this, and it still failed the test — it guessed the API surface but not the Kirsch–Mitzenmacher hashing, so its membership answers were wrong. Cost: ~20k output tokens, two rounds.
xerj got the reference injected and wrote 8 lines — and its own comment shows it learned the seal contract from the passage, the exact thing bare could never recover:
pub fn build(bits: usize, k: u32, keys: &[u64]) -> sieve::Sieve {
let mut s = sieve::Sieve::mesh(bits, k);
for &key in keys { s.dust(key); }
// `sense` panics unless the sieve has been sealed first.
s.settle();
s
}
Cost: ~670 output tokens, one round, correct. (native reached the same 8-line
answer, at ~1,048 tokens, after searching the tree for the source.) The full
captured programs are in measure/generated/.
Track 6 — the same result in four more languages
Everything above is Rust. The mechanism is not language-specific, and this track
shows it. Four purpose-built reference libraries, one per language, each with an
unguessable API and a contract the compiler cannot hint at — verdict is the real
toolchain (python3 test.py, node test.js, cc *.c && ./prog,
javac *.java && java Test):
- Python —
warden(LRU cache):hold/stow/fetch/trail, where afetchpromotes to most-recently-used and overflow evicts the LRU key. - JavaScript —
garner(prefix trie):plant/sprig/seek/crop, whereseekis true only for a stored word, false for a mere prefix. - C —
arena(generational allocator):sprout/plant/pluck/wither, where a withered handle stays stale even after its slot is reused. - Java —
Ledger(append-only log):open/append/seal/replay, where you mustsealbefore you mayreplay.
Two tasks per language (8 tasks × 3 arms × 2 trials = 16 runs/arm). The harness is
csmulti.py; tasks in tasks-ml/; each task returns the library's own type or is
judged by its contract, and validate confirmed golden passes and the guessed-API
version does not build/run, in every language.
Results — retrieval is cheapest in every language, by 2–278×
| task | lang | bare | native | xerj (median output tokens shown) |
|---|---|---|---|---|
| warden-run | Python | 1/2, 19,972 | 2/2, 1,520 | 2/2, 139 |
| warden-evict | Python | 2/2, 10,769 | 2/2, 1,379 | 2/2, 342 |
| garner-seek | JS | 2/2, 4,300 | 2/2, 1,229 | 2/2, 229 |
| garner-crop | JS | 2/2, 4,421 | 2/2, 1,361 | 2/2, 1,106 |
| arena-build | C | 0/2, 28,404 | 2/2, 2,617 | 2/2, 2,159 |
| arena-reuse | C | 2/2, 9,056 | 2/2, 2,529 | 2/2, 820 |
| ledger-replay | Java | 2/2, 17,081 | 2/2, 1,328 | 2/2, 92 |
| ledger-checkpoint | Java | 0/2, 36,453 | 2/2, 1,274 | 2/2, 103 |
| total | 11/16, $11.18 | 16/16, $3.27 | 16/16, $1.58 |
Median output tokens per language — the token-savings headline, and it is consistent everywhere:
| language | bare | native | xerj | xerj vs native | xerj vs bare |
|---|---|---|---|---|---|
| Python | 14,752 | 1,440 | 214 | 6.7× | 69× |
| JavaScript | 4,300 | 1,253 | 646 | 1.9× | 6.7× |
| C | 18,792 | 2,617 | 988 | 2.6× | 19× |
| Java | 27,108 | 1,274 | 98 | 13× | 278× |
xerj and native both solve 16/16; xerj does it for 9,982 output tokens against
native's 26,477 (2.7×) and $1.58 against $3.27 (2.1×). bare solves 11/16 and
burns 26× the output tokens of xerj overall — and fails outright on the two
tasks whose contract is both forced and non-leaky (arena-build's opaque Arena*,
ledger-checkpoint's truncate-then-seal).
How bare behaves depends on the language, and it is honest to say so
- Forced + non-leaky contract → bare fails (
arena-build,ledger-checkpoint). - Forced but the error message leaks the step → bare recovers, expensively.
Java throws
IllegalStateException("replay before seal"); that message names the missingseal, so bare recoversledger-replay— over ~17,000 output tokens, against xerj's 92 (185×). The compiler/runtime is a costly, partial substitute for the reference; XERJ is neither. - Not forced (dynamic language, known algorithm) → bare reimplements. A trie
(
garner) is standard, so JS bare rebuilds it — correctly, but at 4,300 tokens to xerj's 229. The LRU contract (warden) is subtler, so Python bare failed one trial before inferring it.
In every one of those regimes, xerj wrote the least code and was correct.
The same task, three ways (Java, ledger-checkpoint)
bare cannot read Ledger.java, so it reinvents an append-log with checkpoint
semantics — 503 lines — and still fails the truncate-then-seal contract.
xerj, handed the reference, writes the whole thing in four lines:
public class Solution {
public static Ledger upTo(long[] vals, long seq) {
Ledger l = Ledger.open();
for (long v : vals) l.append(v);
l.checkpoint(seq); // keep 0..seq
l.seal(); // required before replay
return l;
}
}
103 output tokens, one shot, correct — where bare burned ~36,000 and failed. The
captured programs are in measure/generated/.
Verdict — reference-coding's value is gated by memorization, and the line is sharp
Put the regimes side by side (solve rate and total $ across each regime's runs):
| regime | bare | native | xerj | who wins |
|---|---|---|---|---|
| unrecallable contract — 7 domains, 21 runs (sift, grove, tally, weft, cadence, trellis, sieve) | 1/21, $21.9 | 21/21, $4.26 | 21/21, $3.38 | xerj |
| unfamiliar but spec-/compiler-recoverable (spool) | 2/2, $2.07 | 2/2, $0.36 | 2/2, $0.29 | xerj (native close) |
| memorised convention (quill varint) | 2/2, $0.33 | 2/2, $0.36 | 2/2, $0.31 | tie / bare |
| memorised bulk data (fieldnorm) | 3/3, $0.98 | 3/3, $0.60 | 3/3, $1.43 | native |
| memorised logic (kv) | 6/6, $1.49 | 6/6, $9.14 | 5/6, $4.40 | bare |
The answer to "is xerj-code better than native Claude Code on the same model?" is: yes, whenever the reference carries knowledge the model cannot recall — and the harder the recall, the bigger the win.
- The core result, across nine unfamiliar domains and 25 runs: native and
xerj each solve 25/25; bare solves 5/25, and four of those five are the
two soft cases (a memorised varint, and a ring buffer whose one non-obvious rule
the spec had to state). On the seven domains with a genuinely unrecallable
contract — a seal, a generational handle, a lazy refill, a specific hash
scheme, an exact tie-break — bare solves 1/21 (a lone
rustc-suggestion recovery) while burning $21.9 flailing; xerj takes the same Claude Code to 21/21 at $3.38, 1.5× fewer output tokens and 1.3× cheaper than native ($4.26), ~35% fewer turns, and 6.5× cheaper than bare. - The compiler is a partial, expensive stand-in for retrieval.
rustc's "did you mean" can leak method names (soweft/spoolbare sometimes recovers), but never a runtime contract, and only ever at 20–25× the token cost. XERJ delivers names and contract together in one shot. - The native gap grows when its search is unfocused: grep pulled 1.06M input tokens into context on the kv corpus, against retrieval's flat ~100–140k.
- It is not a Rust artefact. Track 6 reproduces the win in Python, JavaScript, C and Java: across 16 runs, xerj and native both solve 16/16, xerj at 2.7× fewer output tokens and 2.1× lower cost than native; bare solves 11/16 and spends up to 278× xerj's tokens (Java) recovering or reinventing. In a concise language the ratio is starkest — xerj wrote correct Java in a median of 98 output tokens.
- On memorised code — which is all popular public code, including a 256-value quantization table that looks unguessable — retrieval is overhead. Pure memory is cheapest; injecting a reference ranges from neutral to actively harmful (worst on bulk-data tasks, where the model re-emits what it was handed).
The product implication is exact: XERJ.code's market is the customer's own private, proprietary, or post-cutoff code — where the model cannot fall back on memory and native Claude Code must crawl an unindexed tree. On public library references it earns nothing, and the honest skill description must say so.
What is measured, and what is not
Measured, on this machine, 2026-08-05, real tokens and dollars from
claude -p --output-format json:
- 3 regimes × 3 arms × 3 trials each (sift/kv) or 3 trials (fieldnorm); the headline sift result is 9/9 vs 0/9, not a marginal effect.
- The two peers (native, xerj) are the byte-identical agentic Claude Code; the bare floor is tool-free memory.
Not measured / honest gaps:
- The native-vs-xerj gap stays ~1.5×, not 10×. Track 4 buried the reference
in ~102k records and the gap did not explode, because
grepon a distinctive API name is precise regardless of tree size — native found the file in ~3 extra turns and ~1.5× the input. The regime where retrieval should crush grep is behavioural / semantic search: when you know what you need but not the symbol name, so there is no keyword to grep. XERJ's shipped embedder is lexical (feature-hash), so this study cannot show that advantage; a neural embedder on a behaviourally-described task is the untested lever most likely to widen the gap. - A genuinely large private codebase (e.g. XERJ's own source) as the corpus, vs a purpose-built library. Track 4's libraries are unfamiliar by construction but small; a real private corpus is the honest end state.
- n=2–3 per cell. Effects this large (1/15 vs 15/15) do not need more; the memorised-track cost orderings are noisier and should be read as directional.
- One model, Rust only.
- xerj's 5/6 on kv: retrieval did not make the model more reliable on memorised code; on unfamiliar code it was 15/15.
Bugs this work found
- A real XERJ crash, fixed in PR #182: the timestamp parsers in
xerj-compress/src/field_codec.rssliced strings by byte index after a byte-length guard, so any multibyte-UTF-8 value ≥19 bytes (Lucene's accented test data) panicked the server thread mid-index. Fixed with checked slicing (v.get(a..b)?) across all three parsers, with regression tests that reproduce the panic. Found by feeding a real multilingual corpus toautoindex. - The
--tools ""leak (harness): it does not disable tools; the "bare" model shelled out and vendored the dependency. Caught and fixed before any reported number depended on it. - BM25 is an ambiguous codegen target (task design): Lucene ≥8 and tantivy
disagree on the
(k1+1)factor, so both agent arms "argued with" the reference. Replaced withfieldnorm-quant.