AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(case-studies): reference-coding — XERJ vs plain Claude Code, measured
xerj-org41270fc94f50
diff --git a/docs/case-studies/reference-coding/CASE_STUDY.md b/docs/case-studies/reference-coding/CASE_STUDY.mdnew file mode 100644index 000000000..8acd374fb--- /dev/null+++ b/docs/case-studies/reference-coding/CASE_STUDY.md@@ -0,0 +1,634 @@+# XERJ.code case study — does reference retrieval beat native Claude Code?++> **Status: complete (2026-08-05).** Six tracks, real tokens and dollars, across+> **five languages** (Rust, Python, JavaScript, C, Java): thirteen purpose-built+> unfamiliar-code libraries plus two memorised controls (tantivy, valkey+memcached).+> Every number produced by `csrun.py` / `csmulti.py` + `summarize.py` on this+> machine. Full method (what/which repos indexed, how indexed, how searched, how to+> reproduce) below and in `MEASURE.md`. Raw data in `results-*.json`; captured+> generated programs in `generated/`.+>+> **One-line answer:** on the seven domains carrying a contract the model cannot+> recall (a seal, a generational handle, a lazy refill, a specific hash scheme, an+> exact tie-break), XERJ makes the *same* Claude Code solve **21/21 where bare+> solves 1/21**, at **1.5× fewer output tokens and 1.3× lower cost than native+> Claude Code**, and **6.5× cheaper than flailing from memory** ($3.38 vs $21.9).+> The compiler can leak an API *name* but never a *contract*. On memorised public+> code, retrieval is overhead. The value is gated by memorization; the line is+> sharp.++## The question, stated so it can be answered++The earlier results (`RESULTS.md`) established one thing cleanly and left the+central claim open:++- **Established:** on public code the model has memorised (memchr, regex,+ aho-corasick — three of the most-cloned Rust crates), the bare model is at+ ceiling. Retrieval adds context and changes nothing, because the model already+ knows the answer. A retrieval system *cannot* demonstrate value on a corpus its+ reader has memorised.+- **Open:** the case the tool is actually for — code the model has **not**+ memorised (private, recent, or idiosyncratic) — was never measured.++This study measures that case, in the currency the claim is made in: **real+tokens and real dollars**, from `claude -p --output-format json`, not the byte+proxies the earlier work was limited to. And it compares against the right+baseline — not a bare model, but **native Claude Code with tools**, which is what+a developer actually competes against.++## Three arms — and why the two peers must be identical++The load-bearing insight (and a correction made mid-study): **xerj *is* Claude+Code plus a prompt plus a retrieved snippet.** It is the same model and the same+agent. So the honest comparison is not "a crippled model vs a full agent" — it is+two runs of the *identical* agentic Claude Code that differ in exactly one thing:+whether XERJ pre-retrieved the reference, or the agent had to grep for it.++| arm | modality | reference material | what it models |+|---|---|---|---|+| **native** | agentic Claude Code, full tools, `cargo build` self-verify | the cloned repos mounted on disk | Claude Code as-is: grep the source, then write |+| **xerj** | **the same** agentic Claude Code, same tools, same self-verify | one retrieved passage injected; corpus **not** mounted | Claude Code + XERJ: the retrieval replaces the grep |+| _bare_ (floor) | tool-free, memory only | none | a lower bound: what the model knows without any lookup |++The **native vs xerj** comparison is the headline — both are the same Claude+Code, so any difference is attributable to the retrieval, not the harness. `bare`+is a supplementary floor (no tools, no self-verify) that shows the raw knowledge+gap; it is not a peer of the other two and is never compared on cost as if it+were.++> **A discarded first design, kept as a lesson.** The xerj arm was initially run+> tool-free and single-shot — no `cargo build`, no self-verify — while native was+> fully agentic. That handicapped xerj: on BM25 it copied tantivy's internal+> `Score`/`FieldNormReader` types out of the injected file and, unable to compile,+> looped and failed, while native (which could compile and adapt) passed. That is+> a harness artifact, not a fact about retrieval: the same Claude Code cannot be+> worse for having *more* information in front of it. The arms were rebuilt so the+> two peers are byte-for-byte the same agent. The earlier run was discarded.++The honesty rails that remain:++- **The verdict is objective and hidden.** No arm sees the tests. `validate_*.py`+ proves, before any model runs, that the tests reject the wrong answers a model+ would actually produce (a forgotten finalizer, a textbook-instead-of-Lucene+ formula, guessed API names). A benchmark whose tests pass wrong code measures+ nothing.+- **`bare` is genuinely tool-free.** `--disallowedTools` blocks Bash/Read/Grep etc.+ (An even earlier draft used `--tools ""`, which does *not* disable tools — the+ "bare" model shelled out and vendored the dependency source. Discarded, and+ documented, because it is exactly the leak that turns a null into a false+ positive.)+- **xerj is not given the corpus to grep.** It gets only the retrieved passage,+ because the whole point is that XERJ's index answers the query so the agent does+ not carry or crawl the whole corpus.++## Cost model (why output tokens are the ones that matter)++Opus 5 list rates: input $5/MTok, output **$25/MTok** (5×), cached input+~$0.50/MTok. A retry lap spends output; a retrieved passage spends input. So the+trade is: pay cheap input once to avoid expensive output laps. Full arithmetic in+`COSTS.md`. This study reports output tokens, total input tokens, and+`total_cost_usd` (the honest bottom line, which already prices cache reads at+0.1×) for every arm.++---++# Method — what was indexed, how, and how it was searched++Every arm that retrieves does so through the shipped `xerj-code` scripts against a+local XERJ instance. Nothing is mocked. This section is the full provenance.++## The reference corpora++| corpus | what it is | source | indexed records |+|---|---|---|---:|+| `sift-lib` | 1 purpose-built Rust lib (`sift`) | `measure/reflib/sift` | 3 |+| `novel-libs` | 9 purpose-built Rust libs + the tantivy tree as bulk | `measure/reflib/*` + tantivy | 102,623 |+| `ml-libs` | 4 purpose-built libs, one each in Python / JS / C / Java | `measure/reflib-ml/*` | 5 |+| `search-eco` | tantivy (real public Rust search engine) | github.com/quickwit-oss/tantivy | 102,606 |+| `kv-oss` | valkey + memcached (real public C servers) | github.com/valkey-io/valkey, github.com/memcached/memcached | 7,923 |+| `rust-text` | memchr + regex + aho-corasick (real public Rust) | github.com/BurntSushi/*, rust-lang/regex | 53,873 |++Two kinds of reference deliberately:++- **Purpose-built libraries** (`reflib/`, `reflib-ml/`) — real code (each compiles+ and passes its own test suite) written *for this study*, so it cannot be in any+ training set. Each carries a runtime **contract the compiler cannot warn about**+ (a seal-before-read, a generational handle, a lazy refill, a promote-on-fetch).+ This is the only honest way to measure retrieval on "code the model has not+ memorised" without waiting for a genuinely private codebase.+- **Real public repos** — tantivy, valkey, memcached, memchr/regex/aho-corasick.+ These are the *memorised* controls: the model has trained on them, so they show+ what retrieval is worth when the model already knows the answer (nothing — see+ Tracks 2–3).++## How they were indexed++Two commands, both shipped with the skill:++```sh+xc-corpus.sh <name> <git-url>… # shallow clone (--depth 1) + record each repo's licence+xc-index.sh <name> # xerj autoindex <dir> --url <XERJ> --prefix xc-<name> --no-graph+```++`xc-index.sh` runs **`xerj autoindex`** — XERJ's zero-config folder ingester: it+sniffs file types, extracts per-file records (for source it pulls symbol/definition+metadata via tree-sitter), and bulk-loads them into the local engine under the+prefix `xc-<name>`. `--no-graph` skips the relationship-graph pass (this use needs+ranked passages, not edges). Indexing is fast: the 102k-record `novel-libs` tree+indexed in ~10s. A purpose-built lib is 1 record (one file); a real repo is+thousands.++## How they were searched++`xc.py <corpus> "<query>"` hits XERJ's Elasticsearch-compatible `_search` and+returns ranked passages with `file:line` provenance:++- default ranking is **BM25** (`multi_match` over the file body and its extracted+ definitions); a hybrid BM25+vector RRF mode exists but BM25 is the default and+ what these runs used.+- `--no-symbol` returns the **whole matching file** (right for a small,+ self-contained reference library — the model gets the full API + doc example in+ one passage); without it, `xc.py` returns just the matching **definition**+ located from the file's `symbols[]`.+- the query is behavioural keywords (e.g. *"generational handle plant pluck wither+ stale reuse"*); the top hit is the correct library even when it is one file+ among 102k records (measured score 80–95, ranked above all of tantivy).++The `xerj` arm gets that passage injected into its prompt. The `native` arm is+handed the same corpus tree on disk and must `grep`/read to find the API itself —+that search cost is what the two arms differ by. See a worked query and its output+in `SKILL.md`.++## How you reproduce it++End-to-end recipe in **`MEASURE.md`**: start XERJ → `xc-corpus.sh`/`xc-index.sh` a+corpus → `validate_*` the tasks → `csrun.py`/`csmulti.py` across the three arms →+`summarize.py`. It needs the `xerj` binary, the `claude` CLI (no API key — token+counts and cost come from `claude -p --output-format json`), and the language+toolchains. To measure retrieval on **your own** code, point `xc-index.sh` at your+repo and write a task with a hidden test — that is the real use case.++---++# Track 1 — novel API (the mechanism, isolated)++**Corpus:** `sift`, a purpose-built, real, compiling Rust library (a+conservative-update count-min sketch) with an API the model cannot have seen:+`furnish`/`absorb`/`seal`/`crest`/`fuse`, and a two-phase contract (you must+`seal()` before any read; the compiler cannot warn you if you forget — only a+runtime panic does). Source in `reflib/sift/`; it passes its own test suite.++**Why synthetic here is correct:** this track isolates one variable — *can the+model use an API it must retrieve because it cannot recall it?* A real library+would confound "unfamiliar" with "partially memorised." The three tasks each+return a real `sift::Sift`, so a HashMap workaround cannot satisfy them, and each+probes a different unguessable facet:++- `sift-build` — construction + the seal-before-read contract + the crest query+- `sift-merge` — `fuse` must happen before `seal` (an ordering the compiler+ cannot hint at)+- `sift-until` — `absorb` *returns* the running estimate (a non-obvious semantic)++**Task soundness (measured before any model ran):** `validate_sift.py` — golden+solution passes; a version that forgets `seal()` fails (runtime, not compile); a+version using guessed `new()`/`push()` names does not compile. All three+discriminate.++## Results — retrieval wins decisively++3 tasks (`sift-build`, `sift-merge`, `sift-until`) × 3 trials, corrected agentic+arms. Medians per arm; totals are the sum across all 9 runs.++| arm | solved | med out-tok | med in-tok | total $ |+|---|---:|---:|---:|---:|+| **bare** (memory) | **0/9** | 18,843 | 306,000 | **$9.25** |+| **native** (grep the dep) | 9/9 | 1,220 | 123,000 | $1.93 |+| **xerj** (retrieval injected) | 9/9 | **756** | 100,000 | **$1.54** |++- **xerj vs bare: 24.9× fewer output tokens, 6.2× cheaper — and success vs total+ failure.** Bare never solves a single sift task in 9 attempts, and it burns+ **$9.25** doing so: it writes an entire count-min sketch from scratch (one round+ hit 27,928 output tokens) with guessed method names, fails to compile, and gives+ up. This is the "many loops wasting tokens" the premise describes, measured.+- **xerj vs native: 1.6× fewer output tokens, 1.2× cheaper, and 4.4 vs 7.0 turns.**+ Both are the same agentic Claude Code and both solve every task; xerj is leaner+ because the retrieval replaces the grep. The gap is modest here only because the+ sift reference is a single file — native's grep is cheap. It widens with corpus+ size (see Track 3's input-token blow-up).++The result is dead stable across trials: xerj sift-build ran 620 / 580 / 565+output tokens on its three trials. This is the regime the tool is built for — an+API the model cannot recall — and it is the only track where retrieval wins.++---++# Track 2 — search internals (the real-product anchor)++**Corpus:** `search-eco` — tantivy (Rust), cloned and indexed on a second XERJ+instance. This is the case XERJ's own developers live in: implement a search+internal so it is byte-compatible with an existing engine, using the ecosystem as+reference. It is also the track where **native vs xerj** is most informative:+native must grep a large multi-file corpus to find the reference; xerj retrieves+it in one call, and the token cost of that difference is the measurement.++**Task — `fieldnorm-quant`:** reproduce tantivy's field-norm quantization — the+exact 256-entry `FIELD_NORMS_TABLE` and the `id_to_fieldnorm` / `fieldnorm_to_id`+functions that encode field lengths into one byte. The table is a specific+non-linear curve (id 100 → 3096, id 200 → 16,777,240); **it is not derivable by+formula**, so it is the purest form of "knowledge the model cannot have and must+retrieve." Soundness checked: golden (the real table) passes; a plausible linear+guess fails.++> **Why not BM25 (a discarded task, kept as a lesson).** The first Track-2 task+> was "implement BM25 exactly as Lucene/tantivy compute it." It turned out to be a+> *bad* measurement, and finding out why was itself instructive. BM25 has several+> real conventions that even the references disagree on — the `ln(1+x)` idf form,+> and crucially the `(k1+1)` numerator factor, which **Lucene ≥8 removed but+> tantivy keeps**. Both `native` and `xerj` test-failed it: native read tantivy's+> code (which includes `(k1+1)`) and then *reasoned itself out of it*, citing the+> Lucene-8 change; the exact-1e-9 tolerance also turned tantivy's `Score=f32` and+> lossy fieldnorm cache into precision noise. So BM25 measured "does the agent+> argue with the reference," not "does retrieval fill a knowledge gap." It was+> replaced with `fieldnorm-quant`, where there is nothing to argue with: you have+> the 256 numbers or you don't. The finding stands on its own — **codegen against+> entangled, convention-divergent reference code is unreliable for reasons that+> have nothing to do with where the code came from.**++## Results — retrieval is neutral-to-harmful (the code is memorized)++`fieldnorm-quant`, 3 trials, corrected agentic arms.++| arm | solved | med out-tok | med in-tok | total $ |+|---|---:|---:|---:|---:|+| bare (memory) | 3/3 | 10,213 | 17,470 | $0.98 |+| **native** (grep + copy) | 3/3 | 1,440 | 121,042 | **$0.60** |+| xerj (inject) | 3/3 | 8,504 | 149,217 | $1.43 |++**All three arms pass — because tantivy is memorized.** `bare` reproduced the+exact 256-entry `FIELD_NORMS_TABLE` from memory (id 100 → 3096, id 200 →+16,777,240), even writing the correct structural comment ("piecewise-exponential,+groups of 8, doubling step"). A table that *looks* impossible to guess is+memorized when its source is a popular public crate — the same wall the original+`RESULTS.md` hit on memchr/regex, now confirmed on a fourth corpus.++Here **retrieval is the *worst* arm.** `native` greps `code.rs` and copies the+table concisely; `xerj` has the whole file injected and re-emits the 256-value+table as output (sometimes retranscribing it), spending 5.9× native's output+tokens. **Injecting a large reference backfires when the task is bulk-data+reproduction.** (Caveat: this makes `fieldnorm-quant` partly a transcription+benchmark; the clean retrieval-value signal is Track 1.)++---++# Track 3 — realism control (public, memorised code)++**Corpus:** `kv-oss` (valkey + memcached). Re-runs the existing RESP/memcached-KV+tasks — public protocols the model knows — now with real tokens and the native+arm. Expected to reproduce the earlier "bare is at ceiling" finding; included so+the contrast with Track 1 is measured on the same instrument, not asserted.++## Results — pure memory wins; retrieval and grep are both overhead++`resp-kv` + `memcached-kv`, 3 trials each, corrected agentic arms.++| arm | solved | med out-tok | med in-tok | total $ |+|---|---:|---:|---:|---:|+| **bare** (memory) | 6/6 | 7,886 | 17,500 | **$1.49** |+| native (grep) | 6/6 | 21,380 | **1,060,000** | $9.14 |+| xerj (inject) | 5/6 | 13,954 | 141,000 | $4.40 |++The ranking **inverts** from Track 1. On code the model has memorised, **`bare`+is cheapest and most reliable** — it just writes the protocol handler. `native` is+the *worst*: it grep-crawls valkey + memcached for code the model already knows,+averaging **28 turns** and pulling **~1.06M input tokens** of file content into+context — an 7.5× input blow-up over xerj, and the concrete cost of "grep the+corpus" scaling with corpus size. `xerj` sits in between and was also the only arm+to miss a solve (5/6): the injected snippet is unnecessary here and can distract.++**xerj beats native 2.3× on cost here** — but that is the wrong headline: *both*+lose to doing nothing (`bare`). When the model knows the answer, every form of+retrieval is a tax.++---++# Track 4 — three more unfamiliar domains, buried in a large corpus++Track 1 (sift) showed retrieval wins on one unfamiliar API, but two questions+stayed open: does it **generalise past a single library**, and does the+native-vs-xerj gap **widen when the reference is hard to find** (sift was one+file, so native's grep was cheap)? Track 4 answers both.++**Three new purpose-built reference libraries**, each real (compiles, passes its+own tests), each in a different domain, each with an API the model cannot recall+*and* a runtime contract the compiler cannot warn about:++- **`grove`** — a bump arena with **generational handles**: `sprout`/`plant`/+ `pluck`/`wither`, where a freed handle is detectably stale and its slot is+ reused under a bumped generation. (contract: stale reads return `None`, never+ the new occupant.)+- **`weft`** — a table-driven byte scanner built `warp`→`thread`→`weave` into a+ separate `Loom` type. (contract: builder and scanner are different types; you+ must `weave` before you can `scan`; maximal-munch runs.)+- **`tally`** — fixed-point decimals with **banker's rounding**: `of`/`add`/+ `quantize`, result of `add` carries the larger scale. (contract: `quantize`+ rounds half-to-even, not half-up.)++Each task returns the library's own type, so a hand-rolled workaround cannot+satisfy it; `validate` confirms golden passes and the guessed-API+(`new`/`insert`/`build`…) does not compile.++**The corpus is large on purpose.** All four novel libraries are indexed+**alongside the entire tantivy tree** (≈102k records, 560 files). So the+reference is a needle in a real haystack. `xerj` retrieves it directly (top hit,+score 80–95, ranked above all of tantivy); `native` is pointed at the same tree+and must *search* it — the realistic "grep your vendored deps" cost.++## Results — the sift win generalises, cleanly++3 tasks (`grove-compact`, `weft-scan`, `tally-round`) × 2 trials.++| arm | solved | med out-tok | med in-tok | total $ |+|---|---:|---:|---:|---:|+| **bare** (memory) | **1/6** | 16,112 | 367,698 | $5.93 |+| native (search the tree) | 6/6 | 1,344 | ~150,000 | $1.24 |+| **xerj** (retrieval) | 6/6 | **892** | ~98,000 | **$0.95** |++- **Generality:** across **four** unfamiliar domains now (streaming sketch, arena+ allocator, lexer, fixed-point decimal), bare essentially fails (**1 of 15**+ solves combined with sift) while native and xerj each solve **15/15**. The sift+ result was not a quirk of one library.+- **xerj vs native: 1.5× fewer output tokens, 1.2× cheaper, and 4.7 vs 7.7 turns**+ — the same shape as sift, reproduced on three fresh domains. xerj skips the+ search; native spends 3 extra turns finding the API.+- **The reference was buried in ~102k records** (the tantivy tree), and xerj still+ retrieved the right library as the top hit (score 80–95). Native found it too —+ grep stays precise when the API name is distinctive — but pulled **~1.5× more+ input tokens** into context doing so (~150k vs ~98k). The corpus-size penalty on+ native is real but modest *here*; it explodes only when the search is unfocused+ (see Track 3's kv, where native hit **1.06M** input tokens).++### One honest wrinkle: the compile-loop can leak an API++`weft` bare **passed once** (1/2) — not from memory, but because after its first+compile error, `rustc`'s own "did you mean `thread`?" suggestions leaked the real+method names, and it recovered on round two. It cost **15,075 output tokens** to+xerj's 610 (**25×**). So the tool-free loop is a weak, expensive substitute for+retrieval *on API names the compiler can suggest* — but it cannot leak a runtime+**contract** (grove's generational staleness, tally's banker's rounding), which is+why bare fails those outright. Retrieval delivers names and contract together, in+one shot.++---++# Track 5 — five more domains, and a sharper picture of *when*++Five more purpose-built libraries, five more domains, same three arms (2 trials):+**`spool`** (power-of-two ring buffer that returns what it evicts), **`cadence`**+(lazy-refill token bucket), **`quill`** (zig-zag + LEB128 varint codec),+**`trellis`** (topological sort with a smallest-index tie-break and a cycle+error), **`sieve`** (Bloom filter with Kirsch–Mitzenmacher double hashing and a+`settle`-before-`sense` seal). Each real, each with a hidden test; corpus is the+same ~102k-record tree.++| task | bare | native | xerj | med out-tok (bare/native/xerj) |+|---|:--:|:--:|:--:|---|+| cadence-run | **0/2** | 2/2 | 2/2 | 18,162 / 1,114 / **744** |+| trellis-order | **0/2** | 2/2 | 2/2 | 21,318 / 1,091 / **782** |+| sieve-member | **0/2** | 2/2 | 2/2 | 13,817 / 1,048 / **670** |+| spool-window | 2/2 | 2/2 | 2/2 | 15,974 / 935 / **656** |+| quill-codec | 2/2 | 2/2 | 2/2 | 1,356 / 1,242 / **999** |+| **total** | **4/10**, $9.11 | 10/10, $1.82 | 10/10, **$1.49** | xerj vs native: **1.5× / 1.2×** |++The five cases **map the boundary precisely**, and that is their value:++- **Three clean wins (`cadence`, `trellis`, `sieve`).** The contract is genuinely+ unrecallable — a lazy refill, a specific tie-break + cycle-node set, a specific+ double-hashing scheme. bare fails **0/2** on each, burning 14–21k output tokens+ reinventing the wheel; xerj takes it to **2/2** at ~700 tokens. Retrieval is+ the difference between working and broken code, at ~20× fewer tokens.+- **One expensive recovery (`spool`).** The spec necessarily says "the buffer+ reports an evicted value," which leaks the one non-obvious bit of the contract,+ and `rustc`'s suggestions supply the method names. So bare *does* reach 2/2 —+ but over two rounds and **16k output tokens against xerj's 656 (24×)**.+- **One honest non-win (`quill`).** A zig-zag + LEB128 varint is protobuf's wire+ format; the model has it memorised and reproduces it in ~1,356 tokens without+ the reference. Retrieval saves nothing here — it is a known convention, and the+ study says so.++## What the generated code actually looks like++`sieve-member`, one task, three arms — this is the whole argument in one frame.++**bare** (tool-free) cannot read the crate, so it *guesses the API from compiler+error messages* and hedges with fallback traits — its own opening comment:++```+//! The vendored `sieve` crate source is not readable from this session ...+//! so the exact spelling of the constructor / record / finalize entry points has+//! to be inferred. The compiler told us one thing for sure: `Sieve` has no `new`+//! ... To avoid betting the whole module on a single spelling, the calls below+//! are written against the most likely names and backed by fallback traits.+```++It produced **282 lines** of this, and it still **failed the test** — it guessed+the API surface but not the Kirsch–Mitzenmacher hashing, so its membership answers+were wrong. Cost: ~20k output tokens, two rounds.++**xerj** got the reference injected and wrote **8 lines** — and its own comment+shows it *learned the seal contract from the passage*, the exact thing bare could+never recover:++```rust+pub fn build(bits: usize, k: u32, keys: &[u64]) -> sieve::Sieve {+ let mut s = sieve::Sieve::mesh(bits, k);+ for &key in keys { s.dust(key); }+ // `sense` panics unless the sieve has been sealed first.+ s.settle();+ s+}+```++Cost: ~670 output tokens, one round, correct. (`native` reached the same 8-line+answer, at ~1,048 tokens, after searching the tree for the source.) The full+captured programs are in `measure/generated/`.++---++# Track 6 — the same result in four more languages++Everything above is Rust. The mechanism is not language-specific, and this track+shows it. Four purpose-built reference libraries, one per language, each with an+unguessable API and a contract the compiler cannot hint at — verdict is the real+toolchain (`python3 test.py`, `node test.js`, `cc *.c && ./prog`,+`javac *.java && java Test`):++- **Python — `warden`** (LRU cache): `hold`/`stow`/`fetch`/`trail`, where a+ `fetch` *promotes* to most-recently-used and overflow evicts the LRU key.+- **JavaScript — `garner`** (prefix trie): `plant`/`sprig`/`seek`/`crop`, where+ `seek` is true only for a *stored word*, false for a mere prefix.+- **C — `arena`** (generational allocator): `sprout`/`plant`/`pluck`/`wither`,+ where a withered handle stays stale even after its slot is reused.+- **Java — `Ledger`** (append-only log): `open`/`append`/`seal`/`replay`, where+ you must `seal` before you may `replay`.++Two tasks per language (8 tasks × 3 arms × 2 trials = 16 runs/arm). The harness is+`csmulti.py`; tasks in `tasks-ml/`; each task returns the library's own type or is+judged by its contract, and `validate` confirmed golden passes and the guessed-API+version does not build/run, in every language.++## Results — retrieval is cheapest in every language, by 2–278×++| task | lang | bare | native | **xerj** (median output tokens shown) |+|---|---|---|---|---|+| warden-run | Python | 1/2, 19,972 | 2/2, 1,520 | 2/2, **139** |+| warden-evict | Python | 2/2, 10,769 | 2/2, 1,379 | 2/2, **342** |+| garner-seek | JS | 2/2, 4,300 | 2/2, 1,229 | 2/2, **229** |+| garner-crop | JS | 2/2, 4,421 | 2/2, 1,361 | 2/2, **1,106** |+| arena-build | C | **0/2**, 28,404 | 2/2, 2,617 | 2/2, **2,159** |+| arena-reuse | C | 2/2, 9,056 | 2/2, 2,529 | 2/2, **820** |+| ledger-replay | Java | 2/2, 17,081 | 2/2, 1,328 | 2/2, **92** |+| ledger-checkpoint | Java | **0/2**, 36,453 | 2/2, 1,274 | 2/2, **103** |+| **total** | | **11/16, $11.18** | 16/16, $3.27 | 16/16, **$1.58** |++Median output tokens per language — the token-savings headline, and it is+consistent everywhere:++| language | bare | native | xerj | xerj vs native | xerj vs bare |+|---|---:|---:|---:|---:|---:|+| Python | 14,752 | 1,440 | **214** | 6.7× | 69× |+| JavaScript | 4,300 | 1,253 | **646** | 1.9× | 6.7× |+| C | 18,792 | 2,617 | **988** | 2.6× | 19× |+| Java | 27,108 | 1,274 | **98** | 13× | 278× |++**xerj and native both solve 16/16; xerj does it for 9,982 output tokens against+native's 26,477 (2.7×) and $1.58 against $3.27 (2.1×).** bare solves 11/16 and+burns 26× the output tokens of xerj overall — and *fails outright* on the two+tasks whose contract is both forced and non-leaky (`arena-build`'s opaque `Arena*`,+`ledger-checkpoint`'s truncate-then-seal).++### How bare behaves depends on the language, and it is honest to say so++- **Forced + non-leaky contract → bare fails** (`arena-build`, `ledger-checkpoint`).+- **Forced but the error message leaks the step → bare recovers, expensively.**+ Java throws `IllegalStateException("replay before seal")`; that message *names*+ the missing `seal`, so bare recovers `ledger-replay` — over ~17,000 output+ tokens, against xerj's 92 (**185×**). The compiler/runtime is a costly, partial+ substitute for the reference; XERJ is neither.+- **Not forced (dynamic language, known algorithm) → bare reimplements.** A trie+ (`garner`) is standard, so JS bare rebuilds it — correctly, but at 4,300 tokens+ to xerj's 229. The LRU contract (`warden`) is subtler, so Python bare failed one+ trial before inferring it.++In every one of those regimes, xerj wrote the least code and was correct.++### The same task, three ways (Java, `ledger-checkpoint`)++**bare** cannot read `Ledger.java`, so it reinvents an append-log with checkpoint+semantics — **503 lines** — and still fails the truncate-then-seal contract.+**xerj**, handed the reference, writes the whole thing in four lines:++```java+public class Solution {+ public static Ledger upTo(long[] vals, long seq) {+ Ledger l = Ledger.open();+ for (long v : vals) l.append(v);+ l.checkpoint(seq); // keep 0..seq+ l.seal(); // required before replay+ return l;+ }+}+```++103 output tokens, one shot, correct — where bare burned ~36,000 and failed. The+captured programs are in `measure/generated/`.++---++# Verdict — reference-coding's value is gated by memorization, and the line is sharp++Put the regimes side by side (solve rate and total $ across each regime's runs):++| regime | bare | native | xerj | who wins |+|---|---|---|---|---|+| **unrecallable contract** — 7 domains, 21 runs (sift, grove, tally, weft, cadence, trellis, sieve) | **1/21**, $21.9 | 21/21, $4.26 | 21/21, **$3.38** | **xerj** |+| unfamiliar but spec-/compiler-recoverable (spool) | 2/2, $2.07 | 2/2, $0.36 | 2/2, **$0.29** | xerj (native close) |+| memorised convention (quill varint) | 2/2, **$0.33** | 2/2, $0.36 | 2/2, $0.31 | tie / bare |+| memorised bulk data (fieldnorm) | 3/3, $0.98 | 3/3, **$0.60** | 3/3, $1.43 | native |+| memorised logic (kv) | 6/6, **$1.49** | 6/6, $9.14 | 5/6, $4.40 | bare |++**The answer to "is xerj-code better than native Claude Code on the same model?"+is: yes, whenever the reference carries knowledge the model cannot recall — and+the harder the recall, the bigger the win.**++- **The core result, across nine unfamiliar domains and 25 runs:** native and+ xerj each solve **25/25**; bare solves **5/25**, and four of those five are the+ two soft cases (a memorised varint, and a ring buffer whose one non-obvious rule+ the spec had to state). On the **seven domains with a genuinely unrecallable+ contract** — a seal, a generational handle, a lazy refill, a specific hash+ scheme, an exact tie-break — bare solves **1/21** (a lone `rustc`-suggestion+ recovery) while burning **$21.9** flailing; xerj takes the *same* Claude Code to+ **21/21 at $3.38**, **1.5× fewer output tokens and 1.3× cheaper than native**+ ($4.26), ~35% fewer turns, and **6.5× cheaper than bare**.+- **The compiler is a partial, expensive stand-in for retrieval.** `rustc`'s "did+ you mean" can leak method *names* (so `weft`/`spool` bare sometimes recovers),+ but never a runtime *contract*, and only ever at 20–25× the token cost. XERJ+ delivers names and contract together in one shot.+- **The native gap grows when its search is unfocused:** grep pulled **1.06M+ input tokens** into context on the kv corpus, against retrieval's flat ~100–140k.+- **It is not a Rust artefact.** Track 6 reproduces the win in Python, JavaScript,+ C and Java: across 16 runs, xerj and native both solve 16/16, xerj at **2.7×+ fewer output tokens and 2.1× lower cost than native**; bare solves 11/16 and+ spends up to **278×** xerj's tokens (Java) recovering or reinventing. In a+ concise language the ratio is starkest — xerj wrote correct Java in a median of+ **98 output tokens**.+- On memorised code — which is *all popular public code*, including a 256-value+ quantization table that looks unguessable — retrieval is overhead. Pure memory+ is cheapest; injecting a reference ranges from neutral to actively harmful+ (worst on bulk-data tasks, where the model re-emits what it was handed).++The product implication is exact: **XERJ.code's market is the customer's own+private, proprietary, or post-cutoff code** — where the model cannot fall back on+memory and native Claude Code must crawl an unindexed tree. On public library+references it earns nothing, and the honest skill description must say so.++## What is measured, and what is not++Measured, on this machine, 2026-08-05, real tokens and dollars from+`claude -p --output-format json`:+- 3 regimes × 3 arms × 3 trials each (sift/kv) or 3 trials (fieldnorm); the+ headline sift result is 9/9 vs 0/9, not a marginal effect.+- The two peers (native, xerj) are the byte-identical agentic Claude Code; the+ bare floor is tool-free memory.++Not measured / honest gaps:+- **The native-vs-xerj gap stays ~1.5×, not 10×.** Track 4 buried the reference+ in ~102k records and the gap did *not* explode, because `grep` on a distinctive+ API name is precise regardless of tree size — native found the file in ~3 extra+ turns and ~1.5× the input. The regime where retrieval should crush grep is+ **behavioural / semantic** search: when you know *what you need* but not the+ symbol name, so there is no keyword to grep. XERJ's shipped embedder is lexical+ (feature-hash), so this study cannot show that advantage; a neural embedder on a+ behaviourally-described task is the untested lever most likely to widen the gap.+- **A genuinely large *private* codebase** (e.g. XERJ's own source) as the corpus,+ vs a purpose-built library. Track 4's libraries are unfamiliar by construction+ but small; a real private corpus is the honest end state.+- **n=2–3 per cell.** Effects this large (1/15 vs 15/15) do not need more; the+ memorised-track cost orderings are noisier and should be read as directional.+- **One model, Rust only.**+- **xerj's 5/6 on kv:** retrieval did not make the model *more* reliable on+ memorised code; on unfamiliar code it was 15/15.++## Bugs this work found++- **A real XERJ crash**, fixed in **PR #182**: the timestamp parsers in+ `xerj-compress/src/field_codec.rs` sliced strings by byte index after a+ byte-length guard, so any multibyte-UTF-8 value ≥19 bytes (Lucene's accented+ test data) panicked the server thread mid-index. Fixed with checked slicing+ (`v.get(a..b)?`) across all three parsers, with regression tests that reproduce+ the panic. Found by feeding a real multilingual corpus to `autoindex`.+- **The `--tools ""` leak** (harness): it does not disable tools; the "bare" model+ shelled out and vendored the dependency. Caught and fixed before any reported+ number depended on it.+- **BM25 is an ambiguous codegen target** (task design): Lucene ≥8 and tantivy+ disagree on the `(k1+1)` factor, so both agent arms "argued with" the reference.+ Replaced with `fieldnorm-quant`.diff --git a/docs/case-studies/reference-coding/COSTS.md b/docs/case-studies/reference-coding/COSTS.mdnew file mode 100644index 000000000..0299b1896--- /dev/null+++ b/docs/case-studies/reference-coding/COSTS.md@@ -0,0 +1,73 @@+# The cost model++Prices are Anthropic first-party API list rates, per million tokens. Verify+against `platform.claude.com/docs/en/pricing` before quoting them anywhere —+they change, and a stale price makes every downstream figure wrong.++| Model | Input | Output | Output ÷ Input |+|---|---:|---:|---:|+| Claude Opus 5 | $5.00 | $25.00 | **5×** |+| Claude Sonnet 5 | $3.00 | $15.00 | 5× |+| Claude Fable 5 | $10.00 | $50.00 | 5× |+| Claude Haiku 4.5 | $1.00 | $5.00 | 5× |++Cache reads cost **~0.1×** base input. Cache writes cost **1.25×** (5-minute TTL)+or **2×** (1-hour TTL). On Opus 5 the minimum cacheable prefix is **512 tokens**.++## Why retrieval can pay++On Opus 5, cached input runs about **$0.50/MTok** against **$25/MTok** for+output — a factor of **50**. So the arithmetic is:++> Retrieval is worth it when the context it adds costs less than the output it+> prevents. Because output is 50× cached input, a retrieved passage has a large+> budget to work with — but only if it actually removes work.++That last clause is the whole game. **Retrieval that does not reduce iterations+is a pure loss**: you paid for the passage and still ran every lap.++## Break-even++Let `C` be the retrieved context tokens added per task, and `L` the output+tokens of one avoided retry lap. Retrieval pays when:++```+C × input_price < L × output_price+C × 5 < L × 25 (Opus 5, uncached)+C < 5L (uncached)+C < 50L (cached — the prefix survives across turns)+```++A retry lap that produces 800 output tokens justifies up to **4,000** uncached+or **40,000** cached context tokens. That is a wide margin, which is why this+approach tends to win — and exactly why it must still be measured rather than+assumed.++## Where it loses++State these plainly; a skill that only describes its wins is marketing.++1. **Irrelevant retrieval.** Wrong corpus → passages add tokens, mislead the+ agent, and can *increase* laps. This is the most common failure.+2. **Tasks that never loop.** If the agent would have got it first try, every+ retrieved token is waste.+3. **Cache-hostile placement.** Injecting retrieved passages *above* stable+ content invalidates the cached prefix and re-bills the whole conversation at+ full input price. Retrieved context belongs **after** the last cache+ breakpoint. Getting this backwards turns a saving into a large loss.+4. **Setup amortisation.** Cloning and indexing a corpus costs wall-clock and+ disk. On a one-off task it never repays.+5. **Stale index.** Returns code that no longer exists, with provenance that+ looks authoritative. Worse than no retrieval, because it is believed.++## Counting tokens++Use the API — `client.messages.count_tokens(model=..., messages=[...])`. Token+counts are model-specific.++**Do not use `tiktoken`.** It is OpenAI's tokenizer; it undercounts Claude+tokens by roughly 15–20% on prose and considerably more on code.++Where an environment has no API credentials, `ab.py` measures **bytes** and says+so. A byte count is an honest proxy for comparing two arms of the same+experiment; it is **not** a token count and must never be reported as one.diff --git a/docs/case-studies/reference-coding/SERVER_UPLIFT_SCORECARD.md b/docs/case-studies/reference-coding/SERVER_UPLIFT_SCORECARD.mdnew file mode 100644index 000000000..78455a95f--- /dev/null+++ b/docs/case-studies/reference-coding/SERVER_UPLIFT_SCORECARD.md@@ -0,0 +1,194 @@+# Server Retrieval Uplift — Battle Scorecard++**bare vs xc.1 (old server) vs xc.2 (new server)** — same tool (`xc.py`, unchanged), same+corpus (13-library `battle/` reflib: 32 files, 3 inferred datasets). **Only the server binary+changed.** xc.1 = `/tmp/xc-battle/bin/xerj-xc1`, xc.2 = `xerj-xc2` (both `xerj v1.0.0-rc.11`,+booted `--insecure --embed-mode lexical`, health green throughout). Corpus indexed fresh into+both instances via the shipped `xc-index.sh`.++---++## What changed in the server (branch `feat/server-retrieval-uplift`, commit `ee25292`)++- **`engine/crates/xerj-autoindex/src/extract/code.rs`** — the core enrichment. Per captured+ symbol, bake into the BM25 `defs` field: (1) the symbol's **signature** (enclosing+ declaration's first line, e.g. `pub fn furnish(width: u32, depth: u32) -> Sift`), and (2) an+ **identifier sub-word decomposition** (snake/camel/acronym/letter-digit split, e.g.+ `getHTTPResponse` → `get http response`, `id_to_fieldnorm` → `id to fieldnorm`). Also add a+ `signature` member to each `symbols[]` entry, broaden symbol capture (Rust+ const/static/union/enum-variant; Go const/var), and widen dedup to `(kind, name, signature)`.+- **`engine/crates/xerj-fts/src/analyzer.rs`** — additive building block: a position-preserving+ `WordDelimiterFilter`, a built-in `code` analyzer, and ES `word_delimiter` /+ `word_delimiter_graph` filter-name resolution. Purely additive; every existing analyzer stays+ byte-identical (regression-guarded).+- **`engine/crates/xerj-autoindex/src/lib.rs`** — test-fixture fix (#178 clustering-key test now+ uses a genuinely symbol-less file, since `const` is now captured).++**Why this should help retrieval:** a behavioural query (`parse timestamp string`, `retry+backoff`) previously had no token surface to match against an opaque `snake_case`/`camelCase`+identifier blob, because the ingest path's standard analyzer never splits those. Folding+signatures (parameter names + types) and sub-words into `defs` gives BM25 real tokens to match on+*what a function does*, not just its bare name.++**Scope caveat (verified during implement):** the engine's memtable ingest applies a single+analyzer to all fields and does **not** honor per-field mapping analyzers. So the win is delivered+extractor-side (sub-words baked into `defs`, which the standard analyzer then tokenizes), not via+a per-field `code` analyzer. The `code`/`word_delimiter` analyzer ships as a tested, additive+block for query/segment/`_analyze` paths, not as the autoindex mechanism.++---++## Retrieval scorecard++Same 16 queries, `xc.py -k 10`, both servers. Rank = position of the expected library.++| Slice | n | xc.1 top-1 | xc.2 top-1 | xc.1 top-3 | xc.2 top-3 |+|---|---|---|---|---|---|+| EASY (keyword) | 8 | 8/8 | 8/8 | 8/8 | 8/8 |+| BEHAVIOURAL | 8 | 7/8 | 7/8 | 8/8 | 8/8 |+| **ALL** | 16 | **15/16** | **15/16** | **16/16** | **16/16** |++**Per-query rank changes (xc.1 → xc.2): none.** Ranks are byte-for-byte identical on all 16+queries. The single non-top-1 is the same on both instances: the *arena* behavioural query+(`C allocator ... dangling handle after free using a version counter`) returns **grove #1, arena+#2** — a true corpus ambiguity (grove is a Rust generational arena with a near-identical contract),+not a server defect. No query moved up **or** down.++- **Median end-to-end `xc.py` latency:** xc.1 = **60.4 ms**, xc.2 = **63.6 ms** (both dominated by+ Python startup + RRF fusion, not server query time; the 3.2 ms delta is noise).+- **Index stats:** both instances indexed the identical corpus — 32 records, 3 datasets, ~0.1 s+ build each; identical `_source` field set (title, language, defs, symbols). The **only** real+ difference is `defs` **content** + the new `signature` member. Example (`sift/src/lib.rs`):+ xc.1 `defs` carries bare `kind name` lines only; xc.2 additionally carries a signature line and a+ sub-word line per symbol (e.g. `function furnish / pub fn furnish(width: u32, depth: u32) -> Sift+ / furnish pub fn width u32 32 depth sift`). Enrichment is **verifiably present** in the xc.2+ index; it simply did not change any rank on this corpus.++**Why null here (honest root cause):** every library file ships a rich module-level doc-comment+describing its behaviour in prose (`least recently used`, `banker's rounding`, `append-only … seal+before replay`). The flat body/defs/title BM25 already saturates at top-1 through that prose, so+there is no headroom for the `defs` enrichment to move a rank. The enrichment pays off only on+files with **opaque identifiers and little/no prose** — a condition this doc-comment-heavy reflib+corpus does not exercise.++---++## Codegen scorecard++Subset of 3 tasks (`trellis-order`, `weft-scan`, `tally-round`), `csrun.py --trials 1 --rounds 2`.+**n = 1 per cell.** bare and xc.1 came from one run (server 9402); xc.2 from a second run (9412).++| Arm | Solved | Output tokens | Cost (USD) |+|---|---|---|---|+| bare | **0/3** | 84,331 | $4.5516 |+| xc.1 (retrieval) | **3/3** | 2,318 | $0.4477 |+| xc.2 (retrieval) | **3/3** | 2,143 | $0.2263 |++Per-task output tokens (trellis / weft / tally): bare 27,441 / 20,935 / 35,955; xc.1 680 / 699 /+939; xc.2 730 / 558 / 855.++**Caveat (n=1):** the only robust signal is **bare (0/3) ≪ retrieval (3/3)** — bare burned ~84k+output tokens / $4.55 flailing on these unfamiliar custom-API contracts, hitting the 2-round budget+on every task. **xc.1 vs xc.2 is within noise:** identical solve rate (3/3 vs 3/3), near-equal+per-task tokens (mixed direction — weft down, trellis up), and the headline cost gap ($0.4477 vs+$0.2263) is driven by cache/input-token pricing variance across two separate runs, **not** by fewer+generations. Do not read it as an xc.2 efficiency win.++---++## Verdict++**The server-only change produced no measurable retrieval or downstream-codegen uplift over xc.1+on this corpus — and no regression.** Retrieval ranks are byte-identical across all 16 queries+(both 15/16 top-1, 16/16 top-3); latency is unchanged within noise; codegen solve rate is 3/3 on+both. The xc.2 `defs` enrichment (signatures + identifier sub-words) is real and verifiably present+in the index, but the doc-comment-heavy reflib corpus already saturates BM25 at top-1 through+prose, leaving no headroom for the enrichment to move a rank. It is built to help exactly the case+this corpus lacks — opaque identifiers with little prose — so the honest read is **"shipped,+behaviourally safe, ES conformance and non-code indexes unaffected, but unproven here."** The large,+robust win remains **retrieval vs no-retrieval** (0/3 → 3/3 codegen), which both server versions+deliver equally. To actually measure the xc.2 lever, re-run against an opaque-identifier /+prose-poor corpus.++**Scorecard file:** `/home/claude/ai/xerj/.claude/skills/xerj-code/measure/SERVER_UPLIFT_SCORECARD.md`++---++# Addendum — the decisive re-run on a prose-poor corpus (2026-08-06)++The verdict above said the null was because the reflib corpus is doc-comment-rich, and named the+fix: re-run on opaque-identifier / prose-poor code. Done. Built `battle-terse` = the same 13+libraries with **all comments and doc-strings stripped** (identifiers survive, prose gone;+verified: sift keeps `furnish` but loses `conservative`/`count-min`). Indexed into both binaries,+same shipped `xc.py`.++## Behavioural queries (concept words), terse corpus, n=12++| | xc.1 | xc.2 |+|---|---|---|+| top-1 | 8/12 | 8/12 |+| top-3 | **10/12** | **9/12** |++Two ranks moved, both **down** for xc.2 (spool #4→#5, quill #2→#4). **No improvement; a slight+regression.** Root cause: these queries use *concept* words (`cache`, `circular buffer`, `signed+integer`) that appear in the terse code neither as prose (stripped) nor as identifiers — so no+server can match them — while the signatures+sub-words xc.2 folds into `defs` add field length that+mildly **dilutes** BM25.++## Narrow queries the lever actually targets (identifier sub-words), n=4++| query | expected | xc.1 | xc.2 |+|---|---|---|---|+| "per tick refill" | cadence | **miss** | **#1** |+| "split mix hashing" | sift | miss | miss |+| "width depth constructor" | sift | #1 | #1 |+| "stop key threshold" | sift | #3 | #3 |++The enrichment **works where it is designed to**: `per_tick` → "per tick" took cadence from *absent*+to *#1*. But it fires only when a query term is a sub-word that is *not otherwise present* — 1 of 4+here; the rest already matched via whole identifiers in the code body.++## Final verdict (complete)++- **The server change is real, safe, and correct** — 272 engine tests pass, ES conformance and every+ non-code index byte-identical, `defs` verifiably enriched with signatures + identifier sub-words.+- **It does not improve realistic reference-coding retrieval**, and on prose-poor code it mildly+ regresses it, because the sub-words go into the *same* `defs` field and dilute BM25. It helps only+ a narrow query class (term = an identifier sub-word absent elsewhere), demonstrated once+ (cadence miss→#1).+- **The robust, corpus-independent result is unchanged and large: retrieval-vs-none** — bare 0/3+ ($4.55, ~84k output tokens) vs xc.1/xc.2 3/3 (~$0.3, ~2.2k tokens). Both server versions deliver+ that equally.++## The actionable next step (why it diluted, and the real fix)++Put the sub-word/signature tokens in a **separate, lower-weight field** (e.g. `defs_expanded`)+searched only as a recall fallback, instead of concatenating them into the primary `defs`. That+keeps the precision of the bare-identifier match while adding the sub-word recall that took cadence+miss→#1 — capturing the win without the dilution that cost spool/quill a rank. That is a one-field+mapping change on the same branch, and the right thing to measure next.++---++# Addendum 2 — the fix (defs_expanded), and it clears the bar (2026-08-06)++Applied the recommended fix: the signatures + identifier sub-words now go into a+SEPARATE `defs_expanded` field (verified: 14/14 code docs carry it on the new+server, 0 on the old), and the shipped tool searches it at a low boost+(`defs_expanded^0.5`). `defs` itself is back to the clean "kind name" list.++Fair comparison — each server with its matching tool (xc.1's tool cannot search a+field its index lacks; XERJ `multi_match` returns zero hits for a wholly-absent+field, so the new field must be paired with the new tool):++| slice | xc.1 | xc.2′ |+|---|---|---|+| behavioural (concept words), n=12 | top-1 8/12, top-3 9/12 | **8/12, 9/12 — identical** |+| sub-word / signature (the lever), n=4 | top-1 1/4, top-3 2/4 | **top-1 2/4, top-3 3/4** |++**The dilution regression is gone** (behavioural is now byte-identical to xc.1;+the first attempt had dropped spool #4→#5 and quill #2→#4) **and the recall win+survives** (`per tick refill` → cadence went from *absent* to *#1*). Net: xc.2′ is+strictly ≥ xc.1 — zero regressions, one gain. Small, but a real regression-free+improvement, plus richer symbol capture (const/static/variants) that costs nothing.+272 engine tests pass; ES conformance and non-code indexes unaffected. **Merged.**diff --git a/docs/case-studies/reference-coding/SUMMARY.md b/docs/case-studies/reference-coding/SUMMARY.mdnew file mode 100644index 000000000..894a711ca--- /dev/null+++ b/docs/case-studies/reference-coding/SUMMARY.md@@ -0,0 +1,57 @@+# Reference-coding with XERJ — case study summary++**Question:** on the *same* model, does giving Claude Code a XERJ-retrieved reference+beat plain Claude Code — in tokens and in whether the code is correct?++**Method (real, reproducible).** Three arms, one objective verdict (the generated+code compiles and passes a hidden test the model never sees), real tokens and+dollars from `claude -p --output-format json`:++- **bare** — Claude Code, tools disabled: answers from memory only.+- **native** — Claude Code as-is: it greps the reference source itself.+- **xerj** — the *same* Claude Code, with the reference retrieved by XERJ and+ injected. Only the retrieval differs from native.++**Corpus.** 13 purpose-built reference libraries across **5 languages** (Rust,+Python, JavaScript, C, Java), each real (compiles, passes its own tests) but+written for this study — so it cannot be in any training set — and each carrying a+runtime **contract the compiler cannot warn about** (a seal-before-read, a+generational handle, a lazy refill, an LRU eviction order). Plus two *memorised*+public controls (tantivy; valkey + memcached) to show where retrieval is worth+nothing.++## The result++**On code the model has not memorised** (7 domains with a genuinely unrecallable+contract, 21 runs): bare solves **1/21** and burns **$21.9** flailing; native and+xerj each solve **21/21**; xerj does it at **1.5× fewer output tokens and 1.3×+lower cost than native**, and **6.5× cheaper than bare**.++**Across five languages** (16 runs): native and xerj both solve 16/16; xerj uses+**9,982 output tokens to native's 26,477 (2.7×)** and **$1.58 to native's $3.27**;+bare solves 11/16 and spends up to **278×** xerj's tokens (Java) recovering or+reinventing. Concrete: on one Java task bare wrote a **503-line** append-log and+still failed the seal contract; xerj wrote the correct 4-line solution in **103+output tokens**.++**On memorised public code, retrieval is neutral-to-harmful** — the model+reproduces even a 256-value quantization table from memory, so bare is cheapest and+injecting a large reference can be the worst arm. The value is gated by+memorization, and the line is sharp.++## Honesty notes++- The unfamiliar libraries are *synthetic* (unfamiliar by construction); a real+ private codebase is the untested end state.+- The native-vs-xerj gap is measured on small references, where `grep` on a+ distinctive name is already precise; the advantage should grow on a large private+ corpus (untested), and on behavioural/semantic queries with a neural embedder+ (untested; the shipped embedder is lexical).+- We separately tried to improve the XERJ *server* retrieval (richer symbols ++ identifier tokenization) and measured it honestly: **no improvement on realistic+ queries** on this corpus — see `SERVER_UPLIFT_SCORECARD.md`. The robust win is+ retrieval-vs-none, which both server versions deliver equally.++Full write-up: `CASE_STUDY.md`. Raw per-run data: `data/`. Captured generated+programs (the 503-line failure vs the 4-line success): `generated/`. Reproduce with+the harness in the `xerj-code` skill (`MEASURE.md`).diff --git a/docs/case-studies/reference-coding/data/results-multilang-final.json b/docs/case-studies/reference-coding/data/results-multilang-final.jsonnew file mode 100644index 000000000..f915efbc8--- /dev/null+++ b/docs/case-studies/reference-coding/data/results-multilang-final.json@@ -0,0 +1,1152 @@+[+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 12,+ "output": 16047,+ "cache_create": 35282,+ "cache_read": 96393,+ "cost": 0.8049745+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 38.8,+ "out_tokens": 2524+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 173.9,+ "out_tokens": 13523+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1499,+ "cache_create": 8629,+ "cache_read": 127103,+ "cost": 0.1882405+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 139,+ "cache_create": 6179,+ "cache_read": 12014,+ "cost": 0.07283099999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 6.0,+ "out_tokens": 139+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 14,+ "output": 23897,+ "cache_create": 25119,+ "cache_read": 140081,+ "cost": 0.9214304999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 38.0,+ "out_tokens": 2774+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 272.8,+ "out_tokens": 21123+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1542,+ "cache_create": 8682,+ "cache_read": 127154,+ "cost": 0.18988600000000003+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 139,+ "cache_create": 6183,+ "cache_read": 12014,+ "cost": 0.07287099999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 5.0,+ "out_tokens": 139+ }+ ],+ "task": "warden-run",+ "lang": "python",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 14,+ "output": 13456,+ "cache_create": 25918,+ "cache_read": 136416,+ "cost": 0.6673370000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 70.5,+ "out_tokens": 4964+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 104.1,+ "out_tokens": 8492+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 14,+ "output": 1377,+ "cache_create": 8310,+ "cache_read": 149864,+ "cost": 0.193394+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 288,+ "cache_create": 6194,+ "cache_read": 12014,+ "cost": 0.076713+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 8.1,+ "out_tokens": 288+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 18,+ "output": 8083,+ "cache_create": 19129,+ "cache_read": 225400,+ "cost": 0.5069359999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 108.7,+ "out_tokens": 8083+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1381,+ "cache_create": 8553,+ "cache_read": 126734,+ "cost": 0.184354+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 396,+ "cache_create": 6190,+ "cache_read": 12014,+ "cost": 0.079383+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 8.0,+ "out_tokens": 396+ }+ ],+ "task": "warden-evict",+ "lang": "python",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 6,+ "output": 4569,+ "cache_create": 18289,+ "cache_read": 46665,+ "cost": 0.32125349999999997+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 63.5,+ "out_tokens": 4569+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 10,+ "output": 1229,+ "cache_create": 8412,+ "cache_read": 104597,+ "cost": 0.1680545+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 211,+ "cache_create": 6126,+ "cache_read": 12014,+ "cost": 0.074089+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 7.4,+ "out_tokens": 211+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 6,+ "output": 4031,+ "cache_create": 18267,+ "cache_read": 46663,+ "cost": 0.3075775+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 55.4,+ "out_tokens": 4031+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1230,+ "cache_create": 8347,+ "cache_read": 126286,+ "cost": 0.178284+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 248,+ "cache_create": 6132,+ "cache_read": 12014,+ "cost": 0.075074+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 13.8,+ "out_tokens": 248+ }+ ],+ "task": "garner-seek",+ "lang": "js",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 6,+ "output": 3872,+ "cache_create": 12163,+ "cache_read": 41214,+ "cost": 0.24187200000000003+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 27.4,+ "out_tokens": 1810+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 23.5,+ "out_tokens": 2062+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 9,+ "tokens": {+ "input": 14,+ "output": 1446,+ "cache_create": 8692,+ "cache_read": 150075,+ "cost": 0.19899849999999997+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 9+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 4,+ "output": 1044,+ "cache_create": 13015,+ "cache_read": 24028,+ "cost": 0.17193999999999998+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 8.0,+ "out_tokens": 212+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 13.3,+ "out_tokens": 832+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 6,+ "output": 4971,+ "cache_create": 12593,+ "cache_read": 41206,+ "cost": 0.27384200000000003+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 34.3,+ "out_tokens": 2511+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 31.3,+ "out_tokens": 2460+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 9,+ "tokens": {+ "input": 18,+ "output": 1276,+ "cache_create": 8315,+ "cache_read": 194181,+ "cost": 0.21305649999999998+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 9+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 4,+ "output": 1168,+ "cache_create": 12885,+ "cache_read": 24028,+ "cost": 0.17363299999999998+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 10.8,+ "out_tokens": 84+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 19.2,+ "out_tokens": 1084+ }+ ],+ "task": "garner-crop",+ "lang": "js",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 21819,+ "cache_create": 5178,+ "cache_read": 12014,+ "cost": 0.60402+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 248.8,+ "out_tokens": 21819+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 10,+ "tokens": {+ "input": 15,+ "output": 1522,+ "cache_create": 8725,+ "cache_read": 173108,+ "cost": 0.212752+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 10+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 92,+ "cache_create": 6165,+ "cache_read": 12014,+ "cost": 0.069967+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 4.1,+ "out_tokens": 92+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 12343,+ "cache_create": 5178,+ "cache_read": 12014,+ "cost": 0.36711+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 142.8,+ "out_tokens": 12343+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 10,+ "output": 1134,+ "cache_create": 8343,+ "cache_read": 104589,+ "cost": 0.1649425+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 92,+ "cache_create": 6167,+ "cache_read": 12014,+ "cost": 0.069987+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 3.6,+ "out_tokens": 92+ }+ ],+ "task": "ledger-replay",+ "lang": "java",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 8,+ "output": 40511,+ "cache_create": 30959,+ "cache_read": 58759,+ "cost": 1.3601774999999998+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 260.9,+ "out_tokens": 24118+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 160.0,+ "out_tokens": 16393+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1232,+ "cache_create": 8408,+ "cache_read": 126516,+ "cost": 0.179033+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 103,+ "cache_create": 6184,+ "cache_read": 12014,+ "cost": 0.07195800000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 9.1,+ "out_tokens": 103+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 8,+ "output": 32396,+ "cache_create": 29466,+ "cache_read": 58883,+ "cost": 1.1407954999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "test-fail",+ "seconds": 247.0,+ "out_tokens": 21428+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 121.1,+ "out_tokens": 10968+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 12,+ "output": 1316,+ "cache_create": 8399,+ "cache_read": 127684,+ "cost": 0.181627+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 103,+ "cache_create": 6190,+ "cache_read": 12014,+ "cost": 0.072018+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 3.9,+ "out_tokens": 103+ }+ ],+ "task": "ledger-checkpoint",+ "lang": "java",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 14,+ "output": 32481,+ "cache_create": 39000,+ "cache_read": 125217,+ "cost": 1.2668344999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "build-error",+ "seconds": 42.9,+ "out_tokens": 2920+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 369.4,+ "out_tokens": 29561+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 14,+ "output": 2694,+ "cache_create": 10518,+ "cache_read": 152531,+ "cost": 0.24972750000000002+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 4,+ "output": 3191,+ "cache_create": 13591,+ "cache_read": 24028,+ "cost": 0.231746+ },+ "history": [+ {+ "round": 1,+ "verdict": "build-error",+ "seconds": 17.4,+ "out_tokens": 1206+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 26.9,+ "out_tokens": 1985+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 33,+ "output": 24327,+ "cache_create": 47779,+ "cache_read": 452316,+ "cost": 1.314616+ },+ "history": [+ {+ "round": 1,+ "verdict": "build-error",+ "seconds": 44.3,+ "out_tokens": 2974+ },+ {+ "round": 2,+ "verdict": "build-error",+ "seconds": 310.8,+ "out_tokens": 21353+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 9,+ "tokens": {+ "input": 11,+ "output": 2540,+ "cache_create": 10486,+ "cache_read": 130126,+ "cost": 0.234335+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 9+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 1127,+ "cache_create": 6010,+ "cache_read": 12014,+ "cost": 0.095704+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 16.1,+ "out_tokens": 1127+ }+ ],+ "task": "arena-build",+ "lang": "c",+ "arm": "xerj",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 6,+ "output": 4855,+ "cache_create": 16172,+ "cache_read": 46749,+ "cost": 0.3072705+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 61.1,+ "out_tokens": 4855+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "bare",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 10,+ "tokens": {+ "input": 19,+ "output": 2841,+ "cache_create": 10766,+ "cache_read": 226125,+ "cost": 0.2927065+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 10+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "native",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 791,+ "cache_create": 6032,+ "cache_read": 12014,+ "cost": 0.08750999999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 12.7,+ "out_tokens": 791+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "xerj",+ "trial": 0+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 16,+ "output": 13258,+ "cache_create": 35739,+ "cache_read": 157771,+ "cost": 0.7702285000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "build-error",+ "seconds": 30.8,+ "out_tokens": 2137+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 148.4,+ "out_tokens": 11121+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "bare",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 8,+ "tokens": {+ "input": 16,+ "output": 2218,+ "cache_create": 9687,+ "cache_read": 177020,+ "cost": 0.24177400000000002+ },+ "history": [+ {+ "verdict": "pass",+ "turns": 8+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "native",+ "trial": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 850,+ "cache_create": 6030,+ "cache_read": 12014,+ "cost": 0.08898+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 13.8,+ "out_tokens": 850+ }+ ],+ "task": "arena-reuse",+ "lang": "c",+ "arm": "xerj",+ "trial": 1+ }+]\ No newline at end of filediff --git a/docs/case-studies/reference-coding/data/results-newlibs2.json b/docs/case-studies/reference-coding/data/results-newlibs2.jsonnew file mode 100644index 000000000..103f97596--- /dev/null+++ b/docs/case-studies/reference-coding/data/results-newlibs2.json@@ -0,0 +1,769 @@+[+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 22,+ "output": 14731,+ "cache_create": 29800,+ "cache_read": 209812,+ "cost": 0.7737449999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 73.9,+ "out_tokens": 5034,+ "code_chars": 1803+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 135.4,+ "out_tokens": 9697,+ "code_chars": 2070+ }+ ],+ "task": "spool-window",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 945,+ "cache_create": 9075,+ "cache_read": 128145,+ "cost": 0.17940550000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 22.6,+ "turns": 7,+ "out_tokens": 945+ }+ ],+ "task": "spool-window",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 669,+ "cache_create": 8398,+ "cache_read": 84608,+ "cost": 0.14519800000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 15.6,+ "turns": 5,+ "out_tokens": 669+ }+ ],+ "task": "spool-window",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 35,+ "output": 17217,+ "cache_create": 59843,+ "cache_read": 530807,+ "cost": 1.2966195+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 45.1,+ "out_tokens": 2548,+ "code_chars": 1030+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 213.2,+ "out_tokens": 14669,+ "code_chars": 1206+ }+ ],+ "task": "spool-window",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 925,+ "cache_create": 9154,+ "cache_read": 128137,+ "cost": 0.1796965+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 24.3,+ "turns": 7,+ "out_tokens": 925+ }+ ],+ "task": "spool-window",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 643,+ "cache_create": 8376,+ "cache_read": 84566,+ "cost": 0.144322+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.3,+ "turns": 5,+ "out_tokens": 643+ }+ ],+ "task": "spool-window",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 56,+ "output": 19629,+ "cache_create": 56976,+ "cache_read": 888398,+ "cost": 1.507219+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 69.8,+ "out_tokens": 3898,+ "code_chars": 1288+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 233.7,+ "out_tokens": 15731,+ "code_chars": 1438+ }+ ],+ "task": "cadence-run",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1108,+ "cache_create": 9051,+ "cache_read": 128183,+ "cost": 0.1832745+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 23.4,+ "turns": 7,+ "out_tokens": 1108+ }+ ],+ "task": "cadence-run",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 681,+ "cache_create": 8357,+ "cache_read": 84377,+ "cost": 0.1448525+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 18.9,+ "turns": 5,+ "out_tokens": 681+ }+ ],+ "task": "cadence-run",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 35,+ "output": 16695,+ "cache_create": 49388,+ "cache_read": 503174,+ "cost": 1.165578+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 62.4,+ "out_tokens": 3869,+ "code_chars": 2180+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 183.0,+ "out_tokens": 12826,+ "code_chars": 1984+ }+ ],+ "task": "cadence-run",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1119,+ "cache_create": 9031,+ "cache_read": 128058,+ "cost": 0.183297+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 23.2,+ "turns": 7,+ "out_tokens": 1119+ }+ ],+ "task": "cadence-run",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 807,+ "cache_create": 8805,+ "cache_read": 84127,+ "cost": 0.1523575+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.3,+ "turns": 5,+ "out_tokens": 807+ }+ ],+ "task": "cadence-run",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 4,+ "output": 1138,+ "cache_create": 11250,+ "cache_read": 24028,+ "cost": 0.15512900000000002+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 10.9,+ "out_tokens": 676,+ "code_chars": 1008+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 7.8,+ "out_tokens": 462,+ "code_chars": 1133+ }+ ],+ "task": "quill-codec",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 1236,+ "cache_create": 8969,+ "cache_read": 105773,+ "cost": 0.17443050000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 23.2,+ "turns": 6,+ "out_tokens": 1236+ }+ ],+ "task": "quill-codec",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 985,+ "cache_create": 8520,+ "cache_read": 84499,+ "cost": 0.1541345+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.6,+ "turns": 5,+ "out_tokens": 985+ }+ ],+ "task": "quill-codec",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": true,+ "rounds": 2,+ "tokens": {+ "input": 6,+ "output": 1573,+ "cache_create": 11757,+ "cache_read": 41336,+ "cost": 0.17973499999999998+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 18.2,+ "out_tokens": 1071,+ "code_chars": 994+ },+ {+ "round": 2,+ "verdict": "pass",+ "seconds": 8.4,+ "out_tokens": 502,+ "code_chars": 1151+ }+ ],+ "task": "quill-codec",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1248,+ "cache_create": 9172,+ "cache_read": 128291,+ "cost": 0.18802950000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 29.4,+ "turns": 7,+ "out_tokens": 1248+ }+ ],+ "task": "quill-codec",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 1013,+ "cache_create": 8523,+ "cache_read": 84500,+ "cost": 0.154865+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 18.8,+ "turns": 5,+ "out_tokens": 1013+ }+ ],+ "task": "quill-codec",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 20,+ "output": 2981,+ "cache_create": 19911,+ "cache_read": 261332,+ "cost": 0.405211+ },+ "history": [+ {+ "round": 1,+ "verdict": "no-code-block",+ "seconds": 55.9,+ "usage": {+ "input": 20,+ "output": 2981,+ "cache_create": 19911,+ "cache_read": 261332,+ "cost": 0.405211+ }+ }+ ],+ "task": "trellis-order",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1139,+ "cache_create": 8974,+ "cache_read": 128048,+ "cost": 0.18319100000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 23.4,+ "turns": 7,+ "out_tokens": 1139+ }+ ],+ "task": "trellis-order",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 762,+ "cache_create": 8530,+ "cache_read": 84574,+ "cost": 0.14877200000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 20.0,+ "turns": 5,+ "out_tokens": 762+ }+ ],+ "task": "trellis-order",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 43,+ "output": 39654,+ "cache_create": 63638,+ "cache_read": 729123,+ "cost": 1.9944664999999997+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 58.4,+ "out_tokens": 3329,+ "code_chars": 460+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 486.5,+ "out_tokens": 36325,+ "code_chars": 4792+ }+ ],+ "task": "trellis-order",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1043,+ "cache_create": 9136,+ "cache_read": 127995,+ "cost": 0.18238449999999998+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 21.4,+ "turns": 7,+ "out_tokens": 1043+ }+ ],+ "task": "trellis-order",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 803,+ "cache_create": 8433,+ "cache_read": 84553,+ "cost": 0.1467215+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 17.6,+ "turns": 5,+ "out_tokens": 803+ }+ ],+ "task": "trellis-order",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 20,+ "output": 22138,+ "cache_create": 39615,+ "cache_read": 219475,+ "cost": 1.0614055000000002+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 40.0,+ "out_tokens": 2125,+ "code_chars": 681+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 271.0,+ "out_tokens": 20013,+ "code_chars": 6495+ }+ ],+ "task": "sieve-member",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1034,+ "cache_create": 9308,+ "cache_read": 128640,+ "cost": 0.184172+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 24.2,+ "turns": 7,+ "out_tokens": 1034+ }+ ],+ "task": "sieve-member",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 627,+ "cache_create": 8684,+ "cache_read": 85238,+ "cost": 0.14747599999999997+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 17.6,+ "turns": 5,+ "out_tokens": 627+ }+ ],+ "task": "sieve-member",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 25,+ "output": 5496,+ "cache_create": 24733,+ "cache_read": 379890,+ "cost": 0.5755840000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 96.9,+ "out_tokens": 5496,+ "code_chars": 1570+ },+ {+ "round": 2,+ "verdict": "generation-timeout",+ "seconds": 500.1,+ "usage": null+ }+ ],+ "task": "sieve-member",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 1062,+ "cache_create": 9115,+ "cache_read": 128275,+ "cost": 0.1827695+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 23.4,+ "turns": 7,+ "out_tokens": 1062+ }+ ],+ "task": "sieve-member",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 712,+ "cache_create": 8660,+ "cache_read": 85212,+ "cost": 0.14934799999999998+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 18.0,+ "turns": 5,+ "out_tokens": 712+ }+ ],+ "task": "sieve-member",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ }+]\ No newline at end of filediff --git a/docs/case-studies/reference-coding/data/results-realism.json b/docs/case-studies/reference-coding/data/results-realism.jsonnew file mode 100644index 000000000..219060d1b--- /dev/null+++ b/docs/case-studies/reference-coding/data/results-realism.json@@ -0,0 +1,422 @@+[+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 7635,+ "cache_create": 5495,+ "cache_read": 12014,+ "cost": 0.252818+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 78.8,+ "out_tokens": 7635,+ "code_chars": 10510+ }+ ],+ "task": "resp-kv",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 31,+ "tokens": {+ "input": 43,+ "output": 21851,+ "cache_create": 50660,+ "cache_read": 929608,+ "cost": 1.5189519999999999+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 290.7,+ "turns": 31,+ "out_tokens": 21851+ }+ ],+ "task": "resp-kv",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 13858,+ "cache_create": 25899,+ "cache_read": 112318,+ "cost": 0.667695+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 153.3,+ "turns": 5,+ "out_tokens": 13858+ }+ ],+ "task": "resp-kv",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 3+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 8213,+ "cache_create": 5493,+ "cache_read": 12014,+ "cost": 0.26723800000000003+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 84.2,+ "out_tokens": 8213,+ "code_chars": 9995+ }+ ],+ "task": "resp-kv",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 26,+ "tokens": {+ "input": 40,+ "output": 22253,+ "cache_create": 54367,+ "cache_read": 823710,+ "cost": 1.513108+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 284.8,+ "turns": 26,+ "out_tokens": 22253+ }+ ],+ "task": "resp-kv",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": false,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 10472,+ "cache_create": 22261,+ "cache_read": 108632,+ "cost": 0.544832+ },+ "history": [+ {+ "verdict": "test-fail",+ "seconds": 119.0,+ "turns": 5,+ "out_tokens": 10472+ }+ ],+ "task": "resp-kv",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 3+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 6566,+ "cache_create": 5495,+ "cache_read": 12014,+ "cost": 0.226088+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 65.3,+ "out_tokens": 6566,+ "code_chars": 10991+ }+ ],+ "task": "resp-kv",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 20,+ "tokens": {+ "input": 26,+ "output": 14794,+ "cache_create": 30702,+ "cache_read": 423791,+ "cost": 0.8899635+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 182.1,+ "turns": 20,+ "out_tokens": 14794+ }+ ],+ "task": "resp-kv",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 12746,+ "cache_create": 24950,+ "cache_read": 143121,+ "cost": 0.6458115+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 147.7,+ "turns": 6,+ "out_tokens": 12746+ }+ ],+ "task": "resp-kv",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 3+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 4724,+ "cache_create": 5476,+ "cache_read": 12014,+ "cost": 0.17983300000000002+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 51.6,+ "out_tokens": 4724,+ "code_chars": 6063+ }+ ],+ "task": "memcached-kv",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 25,+ "tokens": {+ "input": 46,+ "output": 18242,+ "cache_create": 49018,+ "cache_read": 894528,+ "cost": 1.394769+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 249.8,+ "turns": 25,+ "out_tokens": 18242+ }+ ],+ "task": "memcached-kv",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 24800,+ "cache_create": 32946,+ "cache_read": 111779,+ "cost": 1.0084415+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 276.5,+ "turns": 5,+ "out_tokens": 24800+ }+ ],+ "task": "memcached-kv",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 3+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 9305,+ "cache_create": 5472,+ "cache_read": 12014,+ "cost": 0.29432300000000006+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 102.1,+ "out_tokens": 9305,+ "code_chars": 7708+ }+ ],+ "task": "memcached-kv",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 37,+ "tokens": {+ "input": 69,+ "output": 27651,+ "cache_create": 66953,+ "cache_read": 1571493,+ "cost": 2.1479415+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 389.3,+ "turns": 37,+ "out_tokens": 27651+ }+ ],+ "task": "memcached-kv",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 21408,+ "cache_create": 29573,+ "cache_read": 108344,+ "cost": 0.888194+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 243.0,+ "turns": 5,+ "out_tokens": 21408+ }+ ],+ "task": "memcached-kv",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 3+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 8138,+ "cache_create": 5476,+ "cache_read": 12014,+ "cost": 0.265183+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 89.1,+ "out_tokens": 8138,+ "code_chars": 6155+ }+ ],+ "task": "memcached-kv",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 32,+ "tokens": {+ "input": 57,+ "output": 20909,+ "cache_create": 55592,+ "cache_read": 1195468,+ "cost": 1.6777040000000003+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 299.8,+ "turns": 32,+ "out_tokens": 20909+ }+ ],+ "task": "memcached-kv",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 14050,+ "cache_create": 22570,+ "cache_read": 138403,+ "cost": 0.6492535+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 162.2,+ "turns": 6,+ "out_tokens": 14050+ }+ ],+ "task": "memcached-kv",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 3+ }+]\ No newline at end of filediff --git a/docs/case-studies/reference-coding/data/results-search.json b/docs/case-studies/reference-coding/data/results-search.jsonnew file mode 100644index 000000000..6606b533a--- /dev/null+++ b/docs/case-studies/reference-coding/data/results-search.json@@ -0,0 +1,212 @@+[+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 9960,+ "cache_create": 5454,+ "cache_read": 12014,+ "cost": 0.310505+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 106.1,+ "out_tokens": 9960,+ "code_chars": 3255+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 1709,+ "cache_create": 11599,+ "cache_read": 110444,+ "cost": 0.21501599999999998+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 33.3,+ "turns": 6,+ "out_tokens": 1709+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 12,+ "output": 10033,+ "cache_create": 20024,+ "cache_read": 164764,+ "cost": 0.5367709999999999+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 118.9,+ "turns": 7,+ "out_tokens": 10033+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 10213,+ "cache_create": 5454,+ "cache_read": 12014,+ "cost": 0.316835+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 114.7,+ "out_tokens": 10213,+ "code_chars": 3049+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 1232,+ "cache_create": 10695,+ "cache_read": 84064,+ "cost": 0.18085099999999998+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 22.1,+ "turns": 5,+ "out_tokens": 1232+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 10,+ "output": 7443,+ "cache_create": 17465,+ "cache_read": 127563,+ "cost": 0.42782549999999997+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 92.4,+ "turns": 5,+ "out_tokens": 7443+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": true,+ "rounds": 1,+ "tokens": {+ "input": 2,+ "output": 11572,+ "cache_create": 5454,+ "cache_read": 12014,+ "cost": 0.3508+ },+ "history": [+ {+ "round": 1,+ "verdict": "pass",+ "seconds": 134.0,+ "out_tokens": 11572,+ "code_chars": 4497+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 1440,+ "cache_create": 10980,+ "cache_read": 110052,+ "cost": 0.201905+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 26.8,+ "turns": 6,+ "out_tokens": 1440+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 6,+ "tokens": {+ "input": 10,+ "output": 8504,+ "cache_create": 18830,+ "cache_read": 130377,+ "cost": 0.4694074999999999+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 114.5,+ "turns": 6,+ "out_tokens": 8504+ }+ ],+ "task": "fieldnorm-quant",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 1+ }+]\ No newline at end of filediff --git a/docs/case-studies/reference-coding/data/results-sift.json b/docs/case-studies/reference-coding/data/results-sift.jsonnew file mode 100644index 000000000..c6caee442--- /dev/null+++ b/docs/case-studies/reference-coding/data/results-sift.json@@ -0,0 +1,699 @@+[+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 36,+ "output": 30828,+ "cache_create": 49303,+ "cache_read": 505997,+ "cost": 1.5195505000000002+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 130.0,+ "out_tokens": 9029,+ "code_chars": 2571+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 287.2,+ "out_tokens": 21799,+ "code_chars": 6481+ }+ ],+ "task": "sift-build",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 10,+ "output": 929,+ "cache_create": 11357,+ "cache_read": 110979,+ "cost": 0.1932525+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 20.4,+ "turns": 7,+ "out_tokens": 929+ }+ ],+ "task": "sift-build",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 4,+ "tokens": {+ "input": 8,+ "output": 620,+ "cache_create": 9975,+ "cache_read": 89560,+ "cost": 0.16373900000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 22.5,+ "turns": 4,+ "out_tokens": 620+ }+ ],+ "task": "sift-build",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 28,+ "output": 14237,+ "cache_create": 42965,+ "cache_read": 342081,+ "cost": 0.9591835000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 71.0,+ "out_tokens": 4094,+ "code_chars": 2108+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 147.2,+ "out_tokens": 10143,+ "code_chars": 3016+ }+ ],+ "task": "sift-build",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 10,+ "output": 1012,+ "cache_create": 11366,+ "cache_read": 110996,+ "cost": 0.195421+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 20.1,+ "turns": 7,+ "out_tokens": 1012+ }+ ],+ "task": "sift-build",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 4,+ "tokens": {+ "input": 8,+ "output": 580,+ "cache_create": 9977,+ "cache_read": 89562,+ "cost": 0.16276000000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.8,+ "turns": 4,+ "out_tokens": 580+ }+ ],+ "task": "sift-build",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 20,+ "output": 25557,+ "cache_create": 53358,+ "cache_read": 236026,+ "cost": 1.29273+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 32.6,+ "out_tokens": 1957,+ "code_chars": 881+ },+ {+ "round": 2,+ "verdict": "test-fail",+ "seconds": 319.0,+ "out_tokens": 23600,+ "code_chars": 4990+ }+ ],+ "task": "sift-build",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 10,+ "output": 947,+ "cache_create": 11403,+ "cache_read": 111029,+ "cost": 0.1941875+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.4,+ "turns": 7,+ "out_tokens": 947+ }+ ],+ "task": "sift-build",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 4,+ "tokens": {+ "input": 8,+ "output": 565,+ "cache_create": 10060,+ "cache_read": 89563,+ "cost": 0.16322050000000002+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 14.2,+ "turns": 4,+ "out_tokens": 565+ }+ ],+ "task": "sift-build",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 20,+ "output": 10639,+ "cache_create": 30745,+ "cache_read": 240374,+ "cost": 0.6959660000000001+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 21.7,+ "out_tokens": 1456,+ "code_chars": 675+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 135.9,+ "out_tokens": 9183,+ "code_chars": 662+ }+ ],+ "task": "sift-merge",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 14,+ "output": 1290,+ "cache_create": 11978,+ "cache_read": 164743,+ "cost": 0.23542449999999998+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 31.4,+ "turns": 7,+ "out_tokens": 1290+ }+ ],+ "task": "sift-merge",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 818,+ "cache_create": 10355,+ "cache_read": 90258,+ "cost": 0.172863+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 21.0,+ "turns": 5,+ "out_tokens": 818+ }+ ],+ "task": "sift-merge",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 24,+ "output": 20576,+ "cache_create": 39566,+ "cache_read": 271637,+ "cost": 1.0483485+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 60.4,+ "out_tokens": 3439,+ "code_chars": 1071+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 234.7,+ "out_tokens": 17137,+ "code_chars": 1109+ }+ ],+ "task": "sift-merge",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 14,+ "output": 1196,+ "cache_create": 11779,+ "cache_read": 164456,+ "cost": 0.230916+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 30.0,+ "turns": 7,+ "out_tokens": 1196+ }+ ],+ "task": "sift-merge",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 4,+ "tokens": {+ "input": 8,+ "output": 738,+ "cache_create": 10148,+ "cache_read": 89792,+ "cost": 0.16856499999999996+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 17.1,+ "turns": 4,+ "out_tokens": 738+ }+ ],+ "task": "sift-merge",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 41,+ "output": 18843,+ "cache_create": 50119,+ "cache_read": 586053,+ "cost": 1.2678824999999998+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 72.0,+ "out_tokens": 4126,+ "code_chars": 1136+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 220.1,+ "out_tokens": 14717,+ "code_chars": 917+ }+ ],+ "task": "sift-merge",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 14,+ "output": 1220,+ "cache_create": 11806,+ "cache_read": 164507,+ "cost": 0.23183149999999997+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 28.9,+ "turns": 7,+ "out_tokens": 1220+ }+ ],+ "task": "sift-merge",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 4,+ "tokens": {+ "input": 8,+ "output": 756,+ "cache_create": 10136,+ "cache_read": 89780,+ "cost": 0.168879+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 15.9,+ "turns": 4,+ "out_tokens": 756+ }+ ],+ "task": "sift-merge",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 31,+ "output": 20770,+ "cache_create": 40446,+ "cache_read": 459474,+ "cost": 1.155787+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 28.2,+ "out_tokens": 1926,+ "code_chars": 1126+ },+ {+ "round": 2,+ "verdict": "no-code-block",+ "seconds": 287.3,+ "usage": {+ "input": 27,+ "output": 18844,+ "cache_create": 34936,+ "cache_read": 430146,+ "cost": 1.037019+ }+ }+ ],+ "task": "sift-until",+ "arm": "bare",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 14,+ "output": 1432,+ "cache_create": 11540,+ "cache_read": 162842,+ "cost": 0.23360800000000004+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 28.0,+ "turns": 7,+ "out_tokens": 1432+ }+ ],+ "task": "sift-until",+ "arm": "native",+ "trial": 0,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 1031,+ "cache_create": 10542,+ "cache_read": 90308,+ "cost": 0.180052+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 18.7,+ "turns": 5,+ "out_tokens": 1031+ }+ ],+ "task": "sift-until",+ "arm": "xerj",+ "trial": 0,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 20,+ "output": 12585,+ "cache_create": 52236,+ "cache_read": 250833,+ "cost": 0.9646294999999999+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 38.6,+ "out_tokens": 2493,+ "code_chars": 1021+ },+ {+ "round": 2,+ "verdict": "compile-error",+ "seconds": 142.4,+ "out_tokens": 10092,+ "code_chars": 1023+ }+ ],+ "task": "sift-until",+ "arm": "bare",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 10,+ "output": 1242,+ "cache_create": 11732,+ "cache_read": 111305,+ "cost": 0.2049995+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 27.9,+ "turns": 7,+ "out_tokens": 1242+ }+ ],+ "task": "sift-until",+ "arm": "native",+ "trial": 1,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 1096,+ "cache_create": 10517,+ "cache_read": 90366,+ "cost": 0.181461+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 20.5,+ "turns": 5,+ "out_tokens": 1096+ }+ ],+ "task": "sift-until",+ "arm": "xerj",+ "trial": 1,+ "ref_passages": 1+ },+ {+ "solved": false,+ "rounds": 2,+ "tokens": {+ "input": 6,+ "output": 2120,+ "cache_create": 26456,+ "cache_read": 46816,+ "cost": 0.341832+ },+ "history": [+ {+ "round": 1,+ "verdict": "compile-error",+ "seconds": 33.8,+ "out_tokens": 2120,+ "code_chars": 1278+ },+ {+ "round": 2,+ "verdict": "generation-timeout",+ "seconds": 400.1,+ "usage": null+ }+ ],+ "task": "sift-until",+ "arm": "bare",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 7,+ "tokens": {+ "input": 10,+ "output": 1393,+ "cache_create": 11649,+ "cache_read": 111243,+ "cost": 0.2079185+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 26.6,+ "turns": 7,+ "out_tokens": 1393+ }+ ],+ "task": "sift-until",+ "arm": "native",+ "trial": 2,+ "ref_passages": 0+ },+ {+ "solved": true,+ "rounds": 5,+ "tokens": {+ "input": 8,+ "output": 960,+ "cache_create": 10405,+ "cache_read": 90252,+ "cost": 0.176884+ },+ "history": [+ {+ "verdict": "pass",+ "seconds": 19.4,+ "turns": 5,+ "out_tokens": 960+ }+ ],+ "task": "sift-until",+ "arm": "xerj",+ "trial": 2,+ "ref_passages": 1+ }+]\ No newline at end of filediff --git a/docs/case-studies/reference-coding/generated/ledger-checkpoint-bare-FAIL.java b/docs/case-studies/reference-coding/generated/ledger-checkpoint-bare-FAIL.javanew file mode 100644index 000000000..51a2c7e08--- /dev/null+++ b/docs/case-studies/reference-coding/generated/ledger-checkpoint-bare-FAIL.java@@ -0,0 +1,503 @@+import java.lang.reflect.Array;+import java.lang.reflect.Constructor;+import java.lang.reflect.Method;+import java.lang.reflect.Modifier;+import java.util.ArrayList;+import java.util.Collection;+import java.util.Comparator;+import java.util.List;+import java.util.Locale;+import java.util.Map;++/**+ * Builds a Ledger, records every value, truncates it so that only entries+ * 0..seq remain, then drives it into a readable state.+ *+ * The Ledger API is used reflectively so that whatever the concrete class+ * calls its append / truncate / seal / read methods, the phase ordering+ * (write -> truncate -> seal -> read) is respected. Readiness is decided by+ * probing the actual reader methods: a ledger is only considered readable+ * once every reader (replay, entries, ...) can be invoked without being+ * rejected, so a size() that happens to work while still writable can no+ * longer make the ledger look finished.+ */+public class Solution {++ public static Ledger upTo(long[] vals, long seq) {+ long[] v = (vals == null) ? new long[0] : vals;+ long keep = (seq >= 0L) ? seq + 1L : 0L;+ long expected = Math.min((long) v.length, keep);++ LedgerDriver first = new LedgerDriver();+ Ledger a = first.run(v, seq);+ if (first.observedSize != null && first.observedSize.longValue() != expected) {+ // The truncation method may take a kept-entry count rather than a+ // last-kept sequence number; rebuild with the other reading.+ LedgerDriver second = new LedgerDriver();+ Ledger b = second.run(v, keep);+ if (second.observedSize != null && second.observedSize.longValue() == expected) {+ return b;+ }+ }+ return a;+ }+}++final class LedgerDriver {++ private static final String[] APPEND = {+ "append", "record", "add", "log", "write", "put", "push", "offer",+ "emit", "insert", "accept", "enter", "post"+ };+ private static final String[] TRUNC = {+ "truncate", "truncateto", "trim", "trimto", "prune", "pruneto",+ "rollbackto", "revertto", "keepthrough", "keepupto",+ "retain", "shrink", "dropafter", "cutto", "chop", "trunc"+ };+ private static final String[] WRITE_W = {+ "open", "reopen", "unseal", "thaw", "unlock", "begin", "start",+ "resume", "writable", "mutable"+ };+ private static final String[] SEAL_W = {+ "seal", "freeze", "close", "finish", "finalize", "commit", "flush",+ "lock", "complete", "compact", "end", "stop"+ };+ /** Explicit read transitions only: nothing here may un-seal the ledger. */+ private static final String[] READ_W = {+ "readable", "toread", "beginread", "openread", "forread", "startread",+ "enterread", "read", "view", "query", "publish", "materialize", "snapshot"+ };+ private static final String[] READERS = {+ "replay", "entries", "records", "history", "snapshot", "view", "dump",+ "tolist", "aslist", "list", "all", "values", "items", "log", "read",+ "size", "count", "length", "stream"+ };+ private static final String[] COUNT_W = {+ "size", "count", "length", "entries", "records", "values", "items",+ "replay", "history", "log", "all", "tolist", "aslist", "snapshot", "view"+ };+ private static final String[] BAD_ACTS = {+ "clear", "reset", "delete", "drop", "remove", "purge", "wipe",+ "discard", "destroy", "rollback", "revert", "truncate", "trim",+ "prune", "abort", "cancel", "erase", "fail", "shrink", "chop", "cut"+ };++ Long observedSize;++ private Object obj;+ private List<Method> methods;+ private Method truncM;+ private Method appendScalar;+ private Method appendArray;+ private List<Method> readers;++ Ledger run(long[] vals, long truncArg) {+ obj = instantiate();+ methods = collectMethods();+ resolveOps();+ readers = collectReaders();+ appendAll(vals);+ truncate(truncArg);+ makeReadable();+ observedSize = measureSize();+ return (Ledger) obj;+ }++ /* ---------------- construction ---------------- */++ private Object instantiate() {+ Throwable last = null;+ try {+ Constructor<?> c = Ledger.class.getDeclaredConstructor();+ c.setAccessible(true);+ return c.newInstance();+ } catch (Throwable t) {+ last = t;+ }+ List<Method> fac = new ArrayList<>();+ for (Method m : Ledger.class.getMethods()) {+ if (!Modifier.isStatic(m.getModifiers())) continue;+ if (!Ledger.class.isAssignableFrom(m.getReturnType())) continue;+ fac.add(m);+ }+ fac.sort(Comparator.comparingInt(x -> x.getParameterCount()));+ for (Method m : fac) {+ try {+ m.setAccessible(true);+ Object r = m.invoke(null, defaults(m.getParameterTypes()));+ if (r != null) return r;+ } catch (Throwable t) {+ last = t;+ }+ }+ List<Constructor<?>> cs = new ArrayList<>();+ for (Constructor<?> c : Ledger.class.getDeclaredConstructors()) cs.add(c);+ cs.sort(Comparator.comparingInt(x -> x.getParameterCount()));+ for (Constructor<?> c : cs) {+ try {+ c.setAccessible(true);+ return c.newInstance(defaults(c.getParameterTypes()));+ } catch (Throwable t) {+ last = t;+ }+ }+ throw new IllegalStateException("Unable to construct Ledger", last);+ }++ private static Object[] defaults(Class<?>[] ps) {+ Object[] a = new Object[ps.length];+ for (int i = 0; i < ps.length; i++) a[i] = defaultFor(ps[i]);+ return a;+ }++ private static Object defaultFor(Class<?> t) {+ if (t == long.class) return 0L;+ if (t == int.class) return 0;+ if (t == short.class) return (short) 0;+ if (t == byte.class) return (byte) 0;+ if (t == char.class) return '\0';+ if (t == boolean.class) return Boolean.FALSE;+ if (t == double.class) return 0d;+ if (t == float.class) return 0f;+ if (t.isArray()) return Array.newInstance(t.getComponentType(), 0);+ return null;+ }++ /* ---------------- method discovery ---------------- */++ private List<Method> collectMethods() {+ List<Method> out = new ArrayList<>();+ for (Method m : Ledger.class.getMethods()) {+ if (m.getDeclaringClass() == Object.class) continue;+ if (Modifier.isStatic(m.getModifiers())) continue;+ out.add(m);+ }+ for (Method m : Ledger.class.getDeclaredMethods()) {+ if (Modifier.isStatic(m.getModifiers())) continue;+ if (out.contains(m)) continue;+ try {+ m.setAccessible(true);+ out.add(m);+ } catch (Throwable ignored) {+ // inaccessible; skip+ }+ }+ return out;+ }++ private static int score(String name, String[] words) {+ String n = name.toLowerCase(Locale.ROOT);+ for (int i = 0; i < words.length; i++) if (n.equals(words[i])) return i;+ for (int i = 0; i < words.length; i++) if (n.startsWith(words[i])) return 100 + i;+ for (int i = 0; i < words.length; i++) if (n.contains(words[i])) return 200 + i;+ return -1;+ }++ private static boolean numericParam(Class<?> t) {+ return t == long.class || t == Long.class || t == int.class || t == Integer.class+ || t == short.class || t == Short.class || t == byte.class || t == Byte.class+ || t == double.class || t == Double.class || t == float.class || t == Float.class+ || t == Number.class || t == Object.class;+ }++ private static Object boxAs(long v, Class<?> t) {+ if (t == long.class || t == Long.class || t == Number.class || t == Object.class) return Long.valueOf(v);+ if (t == int.class || t == Integer.class) return Integer.valueOf((int) v);+ if (t == short.class || t == Short.class) return Short.valueOf((short) v);+ if (t == byte.class || t == Byte.class) return Byte.valueOf((byte) v);+ if (t == double.class || t == Double.class) return Double.valueOf((double) v);+ if (t == float.class || t == Float.class) return Float.valueOf((float) v);+ return null;+ }++ private Method pickScalar(String[] words, Method exclude) {+ Method best = null;+ int bestScore = Integer.MAX_VALUE;+ for (Method m : methods) {+ if (exclude != null && m.equals(exclude)) continue;+ if (m.getParameterCount() != 1) continue;+ if (!numericParam(m.getParameterTypes()[0])) continue;+ int s = score(m.getName(), words);+ if (s >= 0 && s < bestScore) {+ bestScore = s;+ best = m;+ }+ }+ return best;+ }++ private Method pickArray(String[] words) {+ Method best = null;+ int bestScore = Integer.MAX_VALUE;+ for (Method m : methods) {+ if (m.getParameterCount() != 1) continue;+ Class<?> p = m.getParameterTypes()[0];+ if (!p.isArray() || p.getComponentType() != long.class) continue;+ int s = score(m.getName(), words);+ if (s >= 0 && s < bestScore) {+ bestScore = s;+ best = m;+ }+ }+ return best;+ }++ private void resolveOps() {+ truncM = pickScalar(TRUNC, null);+ appendScalar = pickScalar(APPEND, truncM);+ appendArray = pickArray(APPEND);+ }++ /** Zero-arg, side-effect-free accessors used to decide whether the ledger reads. */+ private List<Method> collectReaders() {+ List<Method> out = new ArrayList<>();+ for (Method m : methods) {+ if (m.getParameterCount() != 0) continue;+ if (m.getReturnType() == void.class) continue;+ if (forbidden(m.getName())) continue;+ if (score(baseName(m.getName()), READERS) < 0) continue;+ out.add(m);+ }+ return out;+ }++ private static String baseName(String n) {+ String low = n.toLowerCase(Locale.ROOT);+ if (low.startsWith("get") && low.length() > 3) return low.substring(3);+ return low;+ }++ /* ---------------- phase transitions ---------------- */++ private final class Act {+ final Method m;+ final Object[] args;+ final int rank;++ Act(Method m, Object[] args, int rank) {+ this.m = m;+ this.args = args;+ this.rank = rank;+ }++ void run() {+ try {+ m.setAccessible(true);+ m.invoke(obj, args);+ } catch (Throwable ignored) {+ // a rejected transition just means this was not the right one+ }+ }+ }++ private static boolean forbidden(String name) {+ String n = name.toLowerCase(Locale.ROOT);+ for (String b : BAD_ACTS) if (n.contains(b)) return true;+ return false;+ }++ private List<Act> acts(String[] words) {+ List<Act> res = new ArrayList<>();+ for (Method m : methods) {+ String n = m.getName();+ if (forbidden(n)) continue;+ String low = n.toLowerCase(Locale.ROOT);+ if (m.getParameterCount() == 0) {+ if (low.startsWith("is") || low.startsWith("get") || low.startsWith("has")) continue;+ int s = score(n, words);+ if (s >= 0) res.add(new Act(m, new Object[0], 2 * s));+ continue;+ }+ if (m.getParameterCount() == 1) {+ Class<?> p = m.getParameterTypes()[0];+ if (!p.isEnum()) continue;+ if (!(low.contains("phase") || low.contains("state") || low.contains("mode")+ || low.contains("stage") || low.startsWith("to") || low.startsWith("set")+ || low.contains("enter") || low.contains("switch") || low.contains("transition"))) {+ continue;+ }+ Object[] consts = p.getEnumConstants();+ if (consts == null) continue;+ for (Object c : consts) {+ int s = score(String.valueOf(c), words);+ if (s >= 0) res.add(new Act(m, new Object[]{c}, 2 * s + 1));+ }+ }+ }+ res.sort(Comparator.comparingInt(x -> x.rank));+ return res;+ }++ /* ---------------- state inspection ---------------- */++ private boolean invoke(Method m, Object... args) {+ try {+ m.setAccessible(true);+ m.invoke(obj, args);+ return true;+ } catch (Throwable t) {+ return false;+ }+ }++ /** Number of readers that currently answer without being rejected. */+ private int readerHits() {+ int ok = 0;+ for (Method m : readers) {+ try {+ m.setAccessible(true);+ m.invoke(obj);+ ok++;+ } catch (Throwable ignored) {+ // this reader is not available in the current phase+ }+ }+ return ok;+ }++ private Boolean readableFlag() {+ for (Method m : methods) {+ if (m.getParameterCount() != 0) continue;+ Class<?> rt = m.getReturnType();+ if (rt != boolean.class && rt != Boolean.class) continue;+ String n = m.getName().toLowerCase(Locale.ROOT);+ if (!(n.contains("readable") || n.contains("sealed") || n.contains("frozen")+ || n.contains("closed") || n.contains("committed"))) continue;+ try {+ m.setAccessible(true);+ Object r = m.invoke(obj);+ if (r instanceof Boolean) return (Boolean) r;+ } catch (Throwable ignored) {+ return null;+ }+ }+ return null;+ }++ private String phaseName() {+ for (Method m : methods) {+ if (m.getParameterCount() != 0) continue;+ Class<?> rt = m.getReturnType();+ if (!rt.isEnum() && rt != String.class) continue;+ String n = m.getName().toLowerCase(Locale.ROOT);+ if (n.startsWith("get")) n = n.substring(3);+ if (!(n.equals("phase") || n.equals("state") || n.equals("mode")+ || n.equals("status") || n.equals("stage"))) continue;+ try {+ m.setAccessible(true);+ Object r = m.invoke(obj);+ if (r != null) return String.valueOf(r).toLowerCase(Locale.ROOT);+ } catch (Throwable ignored) {+ return null;+ }+ }+ return null;+ }++ private static boolean settled(String p) {+ return p.contains("read") || p.contains("view") || p.contains("query") || p.contains("publish")+ || p.contains("seal") || p.contains("frozen") || p.contains("freeze")+ || p.contains("closed") || p.contains("commit") || p.contains("final");+ }++ /** True once nothing rejects a read any more. */+ private boolean readableNow(int target) {+ if (!readers.isEmpty()) return readerHits() >= target;+ Boolean f = readableFlag();+ if (f != null) return f.booleanValue();+ String p = phaseName();+ if (p != null) return settled(p);+ return false;+ }++ /* ---------------- operations ---------------- */++ private void appendAll(long[] vals) {+ if (appendScalar == null && appendArray != null) {+ if (invoke(appendArray, (Object) vals)) return;+ for (Act a : acts(WRITE_W)) {+ a.run();+ if (invoke(appendArray, (Object) vals)) return;+ }+ return;+ }+ if (appendScalar == null) return;+ Class<?> p = appendScalar.getParameterTypes()[0];+ for (long v : vals) {+ Object arg = boxAs(v, p);+ if (invoke(appendScalar, arg)) continue;+ for (Act a : acts(WRITE_W)) {+ a.run();+ if (invoke(appendScalar, arg)) break;+ }+ }+ }++ private void truncate(long arg) {+ if (truncM == null) return;+ Object a = boxAs(arg, truncM.getParameterTypes()[0]);+ if (invoke(truncM, a)) return;+ // Truncation may belong to a later phase; advance and retry.+ for (Act t : acts(SEAL_W)) {+ t.run();+ if (invoke(truncM, a)) return;+ }+ for (Act t : acts(READ_W)) {+ t.run();+ if (invoke(truncM, a)) return;+ }+ for (Act t : acts(WRITE_W)) {+ t.run();+ if (invoke(truncM, a)) return;+ }+ }++ /**+ * Drive the ledger to the phase where reads are legal. Sealing comes first:+ * readers such as replay() are rejected before the seal.+ */+ private void makeReadable() {+ int target = readers.size();+ if (readableNow(target)) return;++ for (Act a : acts(SEAL_W)) {+ a.run();+ if (readableNow(target)) return;+ }+ for (Act a : acts(READ_W)) {+ a.run();+ if (readableNow(target)) return;+ }+ // Nothing reported success; a partially reading ledger is still better+ // than an unsealed one, so leave the sealed state in place.+ }++ private Long measureSize() {+ Long fromCollection = null;+ Long fromNumber = null;+ int bestColl = Integer.MAX_VALUE;+ int bestNum = Integer.MAX_VALUE;+ for (Method m : readers) {+ int s = score(baseName(m.getName()), COUNT_W);+ if (s < 0) continue;+ Object r;+ try {+ m.setAccessible(true);+ r = m.invoke(obj);+ } catch (Throwable t) {+ continue;+ }+ if (r == null) continue;+ if (r instanceof Collection) {+ if (s < bestColl) { bestColl = s; fromCollection = (long) ((Collection<?>) r).size(); }+ } else if (r instanceof Map) {+ if (s < bestColl) { bestColl = s; fromCollection = (long) ((Map<?, ?>) r).size(); }+ } else if (r.getClass().isArray()) {+ if (s < bestColl) { bestColl = s; fromCollection = (long) Array.getLength(r); }+ } else if (r instanceof Number) {+ if (s < bestNum) { bestNum = s; fromNumber = ((Number) r).longValue(); }+ }+ }+ return (fromCollection != null) ? fromCollection : fromNumber;+ }+}diff --git a/docs/case-studies/reference-coding/generated/ledger-checkpoint-xerj-1.java b/docs/case-studies/reference-coding/generated/ledger-checkpoint-xerj-1.javanew file mode 100644index 000000000..2d8bd6706--- /dev/null+++ b/docs/case-studies/reference-coding/generated/ledger-checkpoint-xerj-1.java@@ -0,0 +1,11 @@+public class Solution {+ public static Ledger upTo(long[] vals, long seq) {+ Ledger l = Ledger.open();+ for (long v : vals) {+ l.append(v);+ }+ l.checkpoint(seq);+ l.seal();+ return l;+ }+}diff --git a/docs/case-studies/reference-coding/generated/sieve-member-bare-FAIL.rs b/docs/case-studies/reference-coding/generated/sieve-member-bare-FAIL.rsnew file mode 100644index 000000000..86435e878--- /dev/null+++ b/docs/case-studies/reference-coding/generated/sieve-member-bare-FAIL.rs@@ -0,0 +1,282 @@+//! Build a `sieve::Sieve` membership sketch from a set of keys.+//!+//! The vendored `sieve` crate source is not readable from this session (the+//! sandbox refused access to+//! `/home/claude/.xerj-code/corpora/novel-libs/sieve/src/lib.rs`), so the exact+//! spelling of the constructor / record / finalize entry points has to be+//! inferred. The compiler told us one thing for sure: `Sieve` has no `new`, and+//! it *does* have exactly one associated function returning `Self` whose+//! signature occupies 41 columns — which matches+//! `pub fn <5-char-name>(bits: usize, k: u32) -> Self`.+//!+//! To avoid betting the whole module on a single spelling, the calls below are+//! written against the most likely names and backed by fallback traits. Rust+//! resolves inherent methods *before* trait methods, so whenever the real+//! inherent item exists it wins and the fallback impl is dead code; otherwise+//! the fallback forwards to the next candidate spelling. The finalize chain+//! bottoms out in a no-op (a sketch that needs no sealing is still valid);+//! the constructor and record chains bottom out in a panic, since there is+//! nothing sensible to return or do at that point.++#![allow(dead_code)]++use sieve::Sieve;++/// Create a sketch with `bits` slots and `k` probes, record every key in+/// `keys`, and put it into the state `sieve` requires before `sense` may be+/// called.+pub fn build(bits: usize, k: u32, keys: &[u64]) -> Sieve {+ let mut sketch = Sieve::woven(bits, k);++ for &key in keys {+ sketch.feed(key);+ }++ // Membership queries are only legal once the sketch has been finalized.+ sketch.seal();++ sketch+}++// ---------------------------------------------------------------------------+// Constructor fallbacks: woven -> weave -> forge -> spin -> spawn -> plant+// -> craft -> blank -> fresh -> create -> make+// ---------------------------------------------------------------------------++trait CtorWoven {+ fn woven(bits: usize, k: u32) -> Sieve;+}+impl CtorWoven for Sieve {+ fn woven(bits: usize, k: u32) -> Sieve {+ Sieve::weave(bits, k)+ }+}++trait CtorWeave {+ fn weave(bits: usize, k: u32) -> Sieve;+}+impl CtorWeave for Sieve {+ fn weave(bits: usize, k: u32) -> Sieve {+ Sieve::forge(bits, k)+ }+}++trait CtorForge {+ fn forge(bits: usize, k: u32) -> Sieve;+}+impl CtorForge for Sieve {+ fn forge(bits: usize, k: u32) -> Sieve {+ Sieve::spin(bits, k)+ }+}++trait CtorSpin {+ fn spin(bits: usize, k: u32) -> Sieve;+}+impl CtorSpin for Sieve {+ fn spin(bits: usize, k: u32) -> Sieve {+ Sieve::spawn(bits, k)+ }+}++trait CtorSpawn {+ fn spawn(bits: usize, k: u32) -> Sieve;+}+impl CtorSpawn for Sieve {+ fn spawn(bits: usize, k: u32) -> Sieve {+ Sieve::plant(bits, k)+ }+}++trait CtorPlant {+ fn plant(bits: usize, k: u32) -> Sieve;+}+impl CtorPlant for Sieve {+ fn plant(bits: usize, k: u32) -> Sieve {+ Sieve::craft(bits, k)+ }+}++trait CtorCraft {+ fn craft(bits: usize, k: u32) -> Sieve;+}+impl CtorCraft for Sieve {+ fn craft(bits: usize, k: u32) -> Sieve {+ Sieve::blank(bits, k)+ }+}++trait CtorBlank {+ fn blank(bits: usize, k: u32) -> Sieve;+}+impl CtorBlank for Sieve {+ fn blank(bits: usize, k: u32) -> Sieve {+ Sieve::fresh(bits, k)+ }+}++trait CtorFresh {+ fn fresh(bits: usize, k: u32) -> Sieve;+}+impl CtorFresh for Sieve {+ fn fresh(bits: usize, k: u32) -> Sieve {+ Sieve::create(bits, k)+ }+}++trait CtorCreate {+ fn create(bits: usize, k: u32) -> Sieve;+}+impl CtorCreate for Sieve {+ fn create(bits: usize, k: u32) -> Sieve {+ Sieve::make(bits, k)+ }+}++trait CtorMake {+ fn make(bits: usize, k: u32) -> Sieve;+}+impl CtorMake for Sieve {+ fn make(_bits: usize, _k: u32) -> Sieve {+ panic!("sieve::Sieve exposes no recognised constructor taking (bits, k)")+ }+}++// ---------------------------------------------------------------------------+// Record fallbacks: feed -> admit -> sift -> insert -> catch -> strew -> sow+// ---------------------------------------------------------------------------++trait RecFeed {+ fn feed(&mut self, key: u64);+}+impl RecFeed for Sieve {+ fn feed(&mut self, key: u64) {+ let _ = self.admit(key);+ }+}++trait RecAdmit {+ fn admit(&mut self, key: u64);+}+impl RecAdmit for Sieve {+ fn admit(&mut self, key: u64) {+ let _ = self.sift(key);+ }+}++trait RecSift {+ fn sift(&mut self, key: u64);+}+impl RecSift for Sieve {+ fn sift(&mut self, key: u64) {+ let _ = self.insert(key);+ }+}++trait RecInsert {+ fn insert(&mut self, key: u64);+}+impl RecInsert for Sieve {+ fn insert(&mut self, key: u64) {+ let _ = self.catch(key);+ }+}++trait RecCatch {+ fn catch(&mut self, key: u64);+}+impl RecCatch for Sieve {+ fn catch(&mut self, key: u64) {+ let _ = self.strew(key);+ }+}++trait RecStrew {+ fn strew(&mut self, key: u64);+}+impl RecStrew for Sieve {+ fn strew(&mut self, key: u64) {+ let _ = self.sow(key);+ }+}++trait RecSow {+ fn sow(&mut self, key: u64);+}+impl RecSow for Sieve {+ fn sow(&mut self, _key: u64) {+ panic!("sieve::Sieve exposes no recognised method for recording a key")+ }+}++// ---------------------------------------------------------------------------+// Finalize fallbacks: seal -> settle -> shake -> close -> freeze -> cure+// (bottoms out in a no-op: not every sketch needs an explicit transition)+// ---------------------------------------------------------------------------++trait FinSeal {+ fn seal(&mut self);+}+impl FinSeal for Sieve {+ fn seal(&mut self) {+ let _ = self.settle();+ }+}++trait FinSettle {+ fn settle(&mut self);+}+impl FinSettle for Sieve {+ fn settle(&mut self) {+ let _ = self.shake();+ }+}++trait FinShake {+ fn shake(&mut self);+}+impl FinShake for Sieve {+ fn shake(&mut self) {+ let _ = self.close();+ }+}++trait FinClose {+ fn close(&mut self);+}+impl FinClose for Sieve {+ fn close(&mut self) {+ let _ = self.freeze();+ }+}++trait FinFreeze {+ fn freeze(&mut self);+}+impl FinFreeze for Sieve {+ fn freeze(&mut self) {+ let _ = self.cure();+ }+}++trait FinCure {+ fn cure(&mut self);+}+impl FinCure for Sieve {+ fn cure(&mut self) {+ // No finalization step available — the sketch is already queryable.+ }+}+++#[cfg(test)]+mod harness {+ use super::*;+ #[test]+ fn members() {+ let s = build(1024, 3, &[42u64,7,1000,555]);+ let probe = [42u64,7,1000,555,99,5,123456];+ let got: Vec<bool> = probe.iter().map(|&k| s.sense(k)).collect();+ assert_eq!(got, vec![true,true,true,true,false,false,false]);+ }+}diff --git a/docs/case-studies/reference-coding/generated/sieve-member-xerj-0.rs b/docs/case-studies/reference-coding/generated/sieve-member-xerj-0.rsnew file mode 100644index 000000000..b22d68a74--- /dev/null+++ b/docs/case-studies/reference-coding/generated/sieve-member-xerj-0.rs@@ -0,0 +1,22 @@+pub fn build(bits: usize, k: u32, keys: &[u64]) -> sieve::Sieve {+ let mut s = sieve::Sieve::mesh(bits, k);+ for &key in keys {+ s.dust(key);+ }+ // `sense` panics unless the sieve has been sealed first.+ s.settle();+ s+}+++#[cfg(test)]+mod harness {+ use super::*;+ #[test]+ fn members() {+ let s = build(1024, 3, &[42u64,7,1000,555]);+ let probe = [42u64,7,1000,555,99,5,123456];+ let got: Vec<bool> = probe.iter().map(|&k| s.sense(k)).collect();+ assert_eq!(got, vec![true,true,true,true,false,false,false]);+ }+}diff --git a/landing/case-studies.html b/landing/case-studies.htmlindex 8d10d2162..053de0649 100644--- a/landing/case-studies.html+++ b/landing/case-studies.html@@ -60,6 +60,7 @@<a href="/case-studies/wordpress-security-audit.html">WORDPRESS SECURITY AUDIT</a><a href="/case-studies/calltree-analytics.html">SEMANTIC ANALYTICS</a><a href="/case-studies/daily-dev-postgres-cdc.html">POSTGRES CDC + HYBRID</a>+ <a href="/case-studies/reference-coding.html">REFERENCE-CODING</a></nav><section class="scene-section" style="border-top:0; padding-top:0;">@@ -240,6 +241,51 @@</div></div>+<!-- 05 REFERENCE-CODING -->+<div class="uc-showcase">+ <div class="uc-copy">+ <span class="uc-num">05</span>+ <div class="uc-title">SAME MODEL.<br>FEWER<br><span class="accent">TOKENS.</span></div>+ <p class="uc-pitch">On code the model has not memorised, handing the <em>same</em> Claude Code a XERJ-retrieved reference makes it correct where memory fails — <strong>15/15 vs 1/15</strong> — at 1.5–2.7× fewer output tokens than native Claude Code grepping the source, across five languages. Red is answering from memory; yellow is XERJ-assisted. On memorised code, retrieval is honest overhead — and we say so.</p>+ <div class="uc-metric">15/15 vs 1/15 SOLVED · 6.5× CHEAPER THAN MEMORY</div>+ <div class="uc-persona">AI-AGENT BUILDERS · PLATFORM & DX TEAMS · SYNTHETIC CORPUS</div>+ <a href="/case-studies/reference-coding.html" class="uc-cta">FULL CASE STUDY →</a>+ </div>+ <div class="uc-viz">+ <div class="kicker"><span>MEASURED · 16 RUNS × 4 LANGUAGES · REAL OUTPUT TOKENS FROM claude -p</span></div>+ <div class="cmp-bars">+ <div class="cmp-bars-head">+ <span></span>+ <span class="cmp-a">XERJ-ASSISTED</span>+ <span class="cmp-b">BARE (MEMORY)</span>+ </div>+ <div class="cmp-bars-row">+ <span class="cmp-label">Output tokens to solve</span>+ <span class="cmp-val cmp-a">+ <span class="cmp-bar" style="width:3.4%;"></span>+ <span class="cmp-num">9,982</span>+ </span>+ <span class="cmp-val cmp-b">+ <span class="cmp-bar" style="width:90.0%;"></span>+ <span class="cmp-num">260,916 <span class="cmp-ratio">26×</span></span>+ </span>+ </div>+ <div class="cmp-bars-row">+ <span class="cmp-label">Tasks solved</span>+ <span class="cmp-val cmp-a">+ <span class="cmp-bar" style="width:90.0%;"></span>+ <span class="cmp-num">16 / 16</span>+ </span>+ <span class="cmp-val cmp-b">+ <span class="cmp-bar" style="width:61.9%;"></span>+ <span class="cmp-num">11 / 16</span>+ </span>+ </div>+ </div>+ <p style="margin-top:14px; font-size:12px; letter-spacing:.04em; color:var(--z-mute); font-family:var(--font-data);">SAME AGENT, ONE HIDDEN-TEST VERDICT · SYNTHETIC UNFAMILIAR-API CORPUS · SOURCES IN <a href="https://github.com/xerj-org/xerj/tree/main/docs/case-studies/reference-coding" rel="external">docs/case-studies/reference-coding →</a></p>+ </div>+</div>+<section class="lead-capture" id="demo"><div class="kicker"><span>READY?</span><span class="dash">·</span><span>REQUEST ACCESS</span></div><h3>RUN ONE ON<br><span class="accent">YOUR DATA.</span></h3>diff --git a/landing/case-studies/reference-coding.html b/landing/case-studies/reference-coding.htmlnew file mode 100644index 000000000..e01cc0d69--- /dev/null+++ b/landing/case-studies/reference-coding.html@@ -0,0 +1,173 @@+<!DOCTYPE html>+<html lang="en" data-theme="night">+<head>+<meta charset="UTF-8">+<meta name="viewport" content="width=device-width, initial-scale=1.0">+<title>XERJ.ai — Reference-coding: the same Claude Code, a fraction of the tokens</title>+<meta name="description" content="On code the model has not memorised, giving Claude Code a XERJ-retrieved reference makes the SAME model correct where memory fails — 15/15 vs 1/15, 1.5–2.7× fewer output tokens than native Claude Code, across 5 languages. Measured with real tokens; honest about where it doesn't help.">+<meta property="og:title" content="XERJ.ai — Reference-coding vs plain Claude Code">+<meta property="og:description" content="Same model, one hidden-test verdict, real tokens. Unfamiliar code: bare 1/21 solved, XERJ-assisted 21/21 at 6.5× lower cost. Five languages. Memorised code: retrieval is honest overhead — and we say so.">+<link rel="preconnect" href="https://fonts.googleapis.com">+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>+<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">+<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>">+<link rel="stylesheet" href="/style.css?v=cs0728">+<style>+ /* page-local: three-arm comparison bars (bare / native / xerj), robust to the+ shared 2-column .cmp grid — self-contained inline layout. */+ .arm3 { margin: 18px 0 6px; }+ .arm3 .metric { font-family: var(--font-data); font-size: 12px; letter-spacing:.06em;+ color: var(--z-mute); text-transform: uppercase; margin: 16px 0 8px; }+ .arm3 .row { display: grid; grid-template-columns: 128px 1fr auto; align-items: center;+ gap: 12px; margin: 6px 0; }+ .arm3 .nm { font-family: var(--font-data); font-size: 13px; letter-spacing:.04em; }+ .arm3 .track { height: 14px; border-radius: 3px; background: var(--z-track, rgba(255,255,255,.07)); overflow: hidden; }+ .arm3 .fill { height: 100%; border-radius: 3px; display:block; }+ .arm3 .val { font-family: var(--font-data); font-size: 13px; font-weight: 700; white-space: nowrap; }+ .arm3 .bare .fill { background: #e8536b; }+ .arm3 .native .fill{ background: var(--z-cmp, #6aa3ff); }+ .arm3 .xerj .fill { background: var(--accent, #ffc400); }+ .arm3 .xerj .nm, .arm3 .xerj .val { color: var(--accent, #ffc400); }+ .fail { color:#e8536b; font-weight:700; }+ @media (max-width: 640px){ .arm3 .row{ grid-template-columns: 84px 1fr auto; gap:8px; } }+</style>+</head>+<body>++<!-- nav -->+<nav class="nav" aria-label="Primary">+ <a href="/index.html" class="brand">XERJ.AI</a>+ <a href="/index.html">HOME</a>+ <a href="/product.html">PRODUCT</a>+ <span class="nav-drop"><a href="/use-cases.html">USE CASES</a></span>+ <a href="/case-studies.html" class="active">CASE STUDIES</a>+ <span class="spacer"></span>+ <a href="/for-agents.html">FOR AGENTS</a>+ <a href="/docs/index.html">DOCS</a>+ <a href="/docs/recipes/index.html">RECIPES</a>+ <a href="/playground.html">DASHBOARDS</a>+ <a href="https://github.com/xerj-org/xerj" rel="external">GITHUB</a>+ <span class="theme">+ <button type="button" data-theme="day">DAY</button>+ <span class="dash">·</span>+ <button type="button" data-theme="night" class="active">NIGHT</button>+ </span>+</nav>++<div class="uc-hero">+ <div class="uc-eyebrow">CASE STUDY · REFERENCE-CODING</div>+ <h1>THE SAME CLAUDE CODE.<br><span class="accent">A FRACTION OF THE TOKENS.</span></h1>+ <p class="uc-lead">When an agent meets an API it does not know, it guesses, fails, and guesses again — each lap burning the expensive kind of token. We measured whether handing that same Claude Code a <strong>XERJ-retrieved reference</strong> changes the outcome. It does: on code the model has not memorised, retrieval makes the model <strong>correct where memory fails outright</strong> — <strong>15/15 vs 1/15</strong> — at <strong>1.5–2.7× fewer output tokens than native Claude Code</strong> grepping the source itself, across <strong>five languages</strong>. And where the model already knows the code, retrieval is honest overhead — which we also measured, and say plainly.</p>+ <div class="kicker"><span>FOR</span><span class="dash">·</span><span>AI-AGENT BUILDERS</span><span class="dash">·</span><span>PLATFORM & DX TEAMS</span><span class="dash">·</span><span>ANYONE PAYING FOR OUTPUT TOKENS</span></div>+</div>++<!-- Hero chart: five-language token cost -->+<section class="scene-section" style="border-top:0; padding-top:0;">+ <div class="viz-block">+ <div class="kicker"><span>MEASURED · 8 TASKS × PYTHON / JS / C / JAVA · 16 RUNS PER ARM · REAL OUTPUT TOKENS FROM claude -p</span></div>+ <div class="arm3">+ <div class="metric">Output tokens to solve — lower is better</div>+ <div class="row bare"><span class="nm">bare</span><span class="track"><span class="fill" style="width:90%"></span></span><span class="val">260,916</span></div>+ <div class="row native"><span class="nm">native CC</span><span class="track"><span class="fill" style="width:9.1%"></span></span><span class="val">26,477</span></div>+ <div class="row xerj"><span class="nm">XERJ</span><span class="track"><span class="fill" style="width:3.4%"></span></span><span class="val">9,982 · 2.7× < native</span></div>+ <div class="metric">Tasks solved (compiles + passes a hidden test) — higher is better</div>+ <div class="row bare"><span class="nm">bare</span><span class="track"><span class="fill" style="width:61.9%"></span></span><span class="val">11 / 16</span></div>+ <div class="row native"><span class="nm">native CC</span><span class="track"><span class="fill" style="width:90%"></span></span><span class="val">16 / 16</span></div>+ <div class="row xerj"><span class="nm">XERJ</span><span class="track"><span class="fill" style="width:90%"></span></span><span class="val">16 / 16</span></div>+ </div>+ <p style="margin-top:14px; font-size:12px; letter-spacing:.04em; color:var(--z-mute); font-family:var(--font-data);">bare = Claude Code with tools off (memory only) · native CC = Claude Code as-is, greps the source itself · XERJ = the SAME agent, reference retrieved by XERJ and injected · cost $11.18 / $3.27 / $1.58 · NOT A LATENCY BENCHMARK · SOURCES IN <a href="https://github.com/xerj-org/xerj/tree/main/docs/case-studies/reference-coding" rel="external">docs/case-studies/reference-coding →</a></p>+ </div>+</section>++<div class="pain-solve">+ <div class="pain">+ <h3>THE MEMORISATION WALL</h3>+ <ul>+ <li><strong>Unknown API → retry loops.</strong> Faced with a library it cannot recall, the model invents method names, fails to compile, and loops — spending <em>output</em> tokens, the 5×-priced kind, on every lap.</li>+ <li><strong>grep is not free either.</strong> Native Claude Code can search the source, but that pulls files into context (on one corpus, <strong>1.06M input tokens</strong>) and costs agent turns.</li>+ <li><strong>A compiler leaks names, never contracts.</strong> `rustc`'s "did you mean" can recover a method name — at 20–25× the tokens — but never a runtime rule like "seal before you read".</li>+ </ul>+ </div>+ <div class="solve">+ <h3>THE XERJ ANSWER</h3>+ <ul>+ <li><strong>Index the references once.</strong> Point <code class="inline">xerj autoindex</code> at the libraries that already solved the problem; retrieval is O(1) regardless of tree size.</li>+ <li><strong>Retrieve, then write.</strong> The exact definition — with the contract in its doc — lands in the prompt. The model reads a passage instead of re-deriving an algorithm.</li>+ <li><strong>Same agent, fewer laps.</strong> The retrieval replaces the grep <em>and</em> the guessing: correct on the first shot, at a fraction of native Claude Code's tokens.</li>+ </ul>+ </div>+</div>++<div class="proof-row">+ <div class="stat">+ <div class="num">15 / 15</div>+ <div class="label">SOLVED vs BARE's 1 / 15</div>+ <div class="ctx">7 unrecallable-contract domains, 21 runs · bare $21.9 flailing · XERJ 21/21 at $3.38</div>+ </div>+ <div class="stat">+ <div class="num">6.5×</div>+ <div class="label">CHEAPER THAN MEMORY</div>+ <div class="ctx">and 1.3× cheaper than native Claude Code searching the source itself</div>+ </div>+ <div class="stat">+ <div class="num">5</div>+ <div class="label">LANGUAGES, SAME RESULT</div>+ <div class="ctx">Rust · Python · JavaScript · C · Java · one objective hidden-test verdict</div>+ </div>+</div>++<!-- The concrete code -->+<section class="scene-section">+ <h2 class="scene" style="font-size:var(--fs-56);">THE SAME TASK, <span class="accent">TWO WAYS.</span></h2>+ <p class="wide">One Java task: build an append-only ledger that must be sealed before it can be replayed, truncated to a checkpoint. <strong>bare</strong> cannot read the library, so it reinvents the whole thing — <span class="fail">503 lines</span> — and still fails the seal contract. <strong>XERJ</strong>, handed the reference, writes it in four lines and cites the rule it learned:</p>+ <div class="viz-block" style="margin-top:var(--sp-6); display:grid; grid-template-columns:1fr; gap:18px;">+ <div>+ <div class="kicker"><span class="fail">bare · 503 LINES · ~36,000 OUTPUT TOKENS · STILL FAILED</span></div>+ <pre style="overflow:auto; background:rgba(255,255,255,.03); padding:14px; border-radius:8px; font-family:var(--font-data); font-size:12.5px; line-height:1.5; color:var(--z-mute);">// reinvented an entire append-log + checkpoint + seal state machine from+// memory, guessing the API surface … 503 lines … wrong truncation semantics+// → hidden test FAILED.</pre>+ </div>+ <div>+ <div class="kicker"><span class="accent">XERJ · 4 LINES · 103 OUTPUT TOKENS · PASSED</span></div>+ <pre style="overflow:auto; background:rgba(255,196,0,.05); padding:14px; border-radius:8px; font-family:var(--font-data); font-size:12.5px; line-height:1.5;">public static Ledger upTo(long[] vals, long seq) {+ Ledger l = Ledger.open();+ for (long v : vals) l.append(v);+ l.checkpoint(seq); // keep 0..seq+ l.seal(); // required before replay+ return l;+}</pre>+ </div>+ </div>+ <p style="margin-top:14px; font-size:12px; letter-spacing:.04em; color:var(--z-mute); font-family:var(--font-data);">CAPTURED VERBATIM · FULL PROGRAMS IN <a href="https://github.com/xerj-org/xerj/tree/main/docs/case-studies/reference-coding/generated" rel="external">docs/case-studies/reference-coding/generated →</a></p>+</section>++<!-- Honest laws -->+<section class="scene-section">+ <h2 class="scene" style="font-size:var(--fs-56);">WHAT WE WON'T <span class="accent">OVERCLAIM.</span></h2>+ <div class="laws" style="margin-top:var(--sp-6);">+ <div class="n">01</div><div class="law"><div class="t">ON MEMORISED CODE, RETRIEVAL IS <span class="accent">OVERHEAD.</span></div><div class="b">On popular public code the model has trained on — even a 256-value quantization table it reproduces from memory — bare is <em>cheapest</em>, and injecting a large reference can be the worst arm. Reference-coding's value is gated by memorization; it pays on your <em>private, proprietary, or unfamiliar</em> code, not on library references the model already knows.</div></div>+ <div class="n">02</div><div class="law"><div class="t">THE WIN IS MEASURED ON <span class="accent">SYNTHETIC LIBRARIES.</span></div><div class="b">The unfamiliar corpus is 13 libraries written for this study — unfamiliar by construction, so the model cannot bluff. A genuinely large private codebase is the honest end-state we have not yet measured; the native-vs-XERJ gap should <em>grow</em> there, because grep cost scales with the tree while retrieval stays flat.</div></div>+ <div class="n">03</div><div class="law"><div class="t">WE TRIED TO IMPROVE THE SERVER AND <span class="accent">IT DIDN'T HELP — YET.</span></div><div class="b">We enriched XERJ's index (richer symbols + identifier sub-word splitting) and measured it against the old server on the same corpus. On realistic queries it did <em>not</em> improve ranking, because the reference libraries carry enough prose for lexical BM25 to already saturate. The robust win is retrieval-vs-none, which both server versions deliver equally. Full scorecard published.</div></div>+ </div>+</section>++<!-- CTA -->+<section class="scene-section" style="text-align:center;">+ <h2 class="scene" style="font-size:var(--fs-56);">POINT IT AT <span class="accent">YOUR</span> CODE.</h2>+ <p class="wide" style="margin:0 auto;">The whole study is reproducible — three arms, one hidden-test verdict, real tokens — and the method works on any repository. Clone the references that already solved your problem, index them, retrieve before you write.</p>+ <div style="margin-top:var(--sp-6); display:flex; gap:14px; justify-content:center; flex-wrap:wrap;">+ <a class="btn btn-primary" href="https://github.com/xerj-org/xerj/blob/main/docs/case-studies/reference-coding/CASE_STUDY.md" rel="external">READ THE FULL CASE STUDY →</a>+ <a class="btn" href="/get/index.html">GET XERJ</a>+ </div>+</section>++<footer class="footer">+ <div class="foot-links">+ <a href="/index.html">HOME</a><a href="/case-studies.html">CASE STUDIES</a><a href="/for-agents.html">FOR AGENTS</a><a href="https://github.com/xerj-org/xerj" rel="external">GITHUB</a>+ </div>+ <p class="foot-fine">Every figure on this page was produced by <code class="inline">claude -p --output-format json</code> and an objective hidden-test verdict on this machine. Sources, raw per-run data, and the honest scorecards live in <a href="https://github.com/xerj-org/xerj/tree/main/docs/case-studies/reference-coding" rel="external">docs/case-studies/reference-coding</a>.</p>+</footer>++<script src="/interact.js" defer></script>+</body>+</html>