AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
site: decisions post rewritten in plain language, four charts, FiQA/AG News rows on the hub
xerj-orgfb0fe417241a
diff --git a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.mdindex 65d988042..19d3d6775 100644--- a/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md+++ b/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews/README.md@@ -34,8 +34,10 @@ HTTP, one connection, distinct items (no cache reuse — every item is new,which is the real workload). Run logs: `run-2026-09-20.txt`.Same-box wire comparison for the same workload class: the hosted Jev API-measured p50 1,230 ms per item on SciFact-scale shortlists (see-`../../../beir-hybrid/results/2026-09-20-rerank-full/`); the local vote on+measured p50 1,230 ms per query on the FiQA rerank run, its fastest of the+three datasets (see+`../../../beir-hybrid/results/2026-09-20-rerank-full-fiqa/`; SciFact and+NFCorpus measured 1.49 s and 1.48 s); the local vote on120k history answers `/v1/systemone` at p50 42.7 ms (300 distinct items,final binary). Different tools — the vote needs this labelled history, Jevneeds nothing — so this is a cost/latency/egress contrast, not a qualitydiff --git a/landing/benchmarks/decisions.html b/landing/benchmarks/decisions.htmlindex d86b29d49..30ee91d3d 100644--- a/landing/benchmarks/decisions.html+++ b/landing/benchmarks/decisions.html@@ -3,8 +3,8 @@<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0">-<title>XERJ.ai — Measuring the Jev judge, then speaking the System One wire</title>-<meta name="description" content="A rerank stage that measured 0.3822 nDCG@10, was root-caused to one request shape and fixed to 0.8299; a 300-query controlled run of Jev on XERJ shortlists; and a XERJ node that answers /v1/systemone locally — the pip-installed jev-reranker runs against it unmodified.">+<title>XERJ.ai — We measured the Jev judge. Then we answered its wire.</title>+<meta name="description" content="We broke our own Jev rerank stage in public (0.3822 nDCG@10), fixed it, then ran 1,271 judged queries on three public datasets. Then we made XERJ answer the System One wire locally. The pip client runs against it unmodified. Nothing leaves your machine."><link rel="preconnect" href="https://fonts.googleapis.com"><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin><link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">@@ -110,13 +110,19 @@pre.code .prompt { color:var(--z-accent); font-weight:700; user-select:none; }pre.code .out { color:var(--z-ink); }pre.code .cmt { opacity:.62; }++ /* bullet lists — same rule the hub carries locally */+ ul.deep { list-style:none; padding:0; max-width:920px; }+ ul.deep li { margin:14px 0; line-height:1.65; color:var(--z-mute); font-size:15px; }+ ul.deep li strong { color:var(--z-ink); }+ ul.deep code.inline { color:var(--z-ink); }</style><!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit --><link rel="canonical" href="https://xerj.org/benchmarks/decisions"><meta property="og:site_name" content="XERJ.ai"><meta property="og:type" content="website">-<meta property="og:title" content="XERJ.ai — Measuring the Jev judge, then speaking the System One wire">-<meta property="og:description" content="A rerank stage that measured 0.3822 nDCG@10, was root-caused to one request shape and fixed to 0.8299; a 300-query controlled run of Jev on XERJ shortlists; and a XERJ node that answers /v1/systemone locally — the pip-installed jev-reranker runs against it unmodified.">+<meta property="og:title" content="XERJ.ai — We measured the Jev judge. Then we answered its wire.">+<meta property="og:description" content="We broke our own Jev rerank stage in public (0.3822 nDCG@10), fixed it, then ran 1,271 judged queries on three public datasets. Then we made XERJ answer the System One wire locally. The pip client runs against it unmodified. Nothing leaves your machine."><meta property="og:url" content="https://xerj.org/benchmarks/decisions"><meta property="og:image" content="https://xerj.org/og/xerj-card.png"><meta property="og:image:width" content="1200">@@ -124,8 +130,8 @@<meta property="og:image:alt" content="XERJ.ai — the search engine built for AI agents"><meta property="og:locale" content="en_US"><meta name="twitter:card" content="summary_large_image">-<meta name="twitter:title" content="XERJ.ai — Measuring the Jev judge, then speaking the System One wire">-<meta name="twitter:description" content="A rerank stage that measured 0.3822 nDCG@10, was root-caused to one request shape and fixed to 0.8299; a 300-query controlled run of Jev on XERJ shortlists; and a XERJ node that answers /v1/systemone locally — the pip-installed jev-reranker runs against it unmodified.">+<meta name="twitter:title" content="XERJ.ai — We measured the Jev judge. Then we answered its wire.">+<meta name="twitter:description" content="We broke our own Jev rerank stage in public (0.3822 nDCG@10), fixed it, then ran 1,271 judged queries on three public datasets. Then we made XERJ answer the System One wire locally. The pip client runs against it unmodified. Nothing leaves your machine."><meta name="twitter:image" content="https://xerj.org/og/xerj-card.png"><script type="application/ld+json">{@@ -164,7 +170,7 @@"applicationSubCategory": "Search engine","operatingSystem": "Linux, macOS, Windows","url": "https://xerj.org/benchmarks/decisions",- "description": "A rerank stage that measured 0.3822 nDCG@10, was root-caused to one request shape and fixed to 0.8299; a 300-query controlled run of Jev on XERJ shortlists; and a XERJ node that answers /v1/systemone locally — the pip-installed jev-reranker runs against it unmodified.",+ "description": "We broke our own Jev rerank stage in public (0.3822 nDCG@10), fixed it, then ran 1,271 judged queries on three public datasets. Then we made XERJ answer the System One wire locally. The pip client runs against it unmodified. Nothing leaves your machine.","image": "https://xerj.org/og/xerj-card.png","license": "https://www.apache.org/licenses/LICENSE-2.0","isAccessibleForFree": true,@@ -218,27 +224,21 @@</nav><div class="uc-hero">- <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, SPOKEN · 2026-09-20</div>+ <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, ANSWERED · 2026-09-20</div><h1>MEASURE THE JUDGE.<br><span class="accent">ANSWER ITS WIRE.</span></h1>- <p class="uc-lead">Two experiments, run in that order. First: what does TypeSafe AI's Jev judge model actually do to- XERJ's ranking — measured, not assumed, including the day our own integration scored- <strong>0.3822 nDCG@10 against BM25's 0.7750</strong> and had to be autopsied. Second, the inverse: a XERJ node that- <em>answers</em> the System One wire locally, so the pip-installed- <code class="inline">jev-reranker</code> client runs against it <strong>unmodified</strong> — zero requests leaving the- machine. Every number below traces to a run whose runner and raw output are in the repo; the losses are published- next to the wins.</p>+ <p class="uc-lead">Two experiments, in that order. First we measured what TypeSafe AI's Jev judge model really does to our ranking. We include the day our own code scored <strong>0.3822 nDCG@10</strong> while plain BM25 scored 0.7750. That bug was ours. We found it, fixed it, and ran 1,271 judged queries on three public datasets. Then we flipped the question. A XERJ node now <em>answers</em> the System One wire itself. The pip-installed <code class="inline">jev-reranker</code> runs against it with no code changes. No request leaves your machine. Every number below comes from a run you can find in the repo. The losses sit next to the wins.</p></div><section class="scene-section" style="border-top:0; padding-top:0;"><div class="stat-band"><div class="stat"><div class="v">0.3822</div><div class="k">the stage, as it shipped</div>- <div class="s">nDCG@10 over BM25's 0.7750 on 40 judged SciFact queries — our defect, published with the root cause below</div></div>+ <div class="s">nDCG@10 against BM25's 0.7750 on 40 judged SciFact queries. Our defect. The root cause is below.</div></div><div class="stat"><div class="v">0.8299</div><div class="k">the same stage, fixed</div>- <div class="s">one request shape changed, same shortlists, same day — within provider wobble of the raw API's 0.8389</div></div>- <div class="stat"><div class="v">+0.084</div><div class="k">nDCG@10 over BM25, full run</div>- <div class="s">SciFact 300 queries, Jev median over 3 repeats: 0.6572 → 0.7410; NFCorpus: 0.3016 → 0.3312</div></div>- <div class="stat"><div class="v">0.806</div><div class="k">gate separation, zero egress</div>- <div class="s">mean noul 0.9014 spam vs 0.0956 ham — the unmodified pip client ranking off a local XERJ node</div></div>+ <div class="s">one request shape changed. Same shortlists, same day. The raw API scored 0.8389.</div></div>+ <div class="stat"><div class="v">+0.126</div><div class="k">biggest lift, FiQA</div>+ <div class="s">BM25 0.2382 to Jev 0.3638 over 648 judged queries. SciFact gained +0.084. NFCorpus gained +0.030.</div></div>+ <div class="stat"><div class="v">42.7 ms</div><div class="k">our answer to the same wire</div>+ <div class="s">p50 per item on 120,000 labelled examples, local, free, no egress. The hosted API measured 1,230 ms per query on the FiQA rerank run (§02). Different tool, different workload.</div></div></div></section>@@ -246,39 +246,30 @@<section class="scene-section" id="autopsy"><div class="kicker"><span class="accent">01</span><span class="dash">·</span><span>THE AUTOPSY · A STAGE THAT SHIPPED UNMEASURED</span></div><h2 class="scene" style="font-size:var(--fs-56);">FIRST, WE BROKE IT<br><span class="accent">IN PUBLIC.</span></h2>- <p class="wide">The <code class="inline">rerank</code> stage — hand BM25's top hits to a hosted relevance judge,- reorder by the probability that comes back — shipped with a caveat that said, honestly,- <em>quality not verified, no provider key</em>. Then TypeSafe granted this project an evaluation key. The first- measured run turned the caveat into a number, and the number was a disaster: <strong>0.3822 nDCG@10 where plain BM25- scored 0.7750</strong>. Adding the judge made ranking <em>worse</em> than not using it.</p>+ <p class="wide">The <code class="inline">rerank</code> stage takes BM25's top hits, sends them to a hosted relevance judge, and reorders by the probability that comes back. It shipped with a caveat that said: <em>quality not verified, no provider key</em>. Then TypeSafe gave this project an evaluation key. The first measured run put a number on the caveat. The number was a disaster. The judge made ranking <em>worse</em> than not using it.</p><div class="loss">- <strong>The defect was ours, not the model's.</strong> Isolated at the raw API, not guessed: the request builder put- every candidate into shared <code class="inline">state</code> and asked each question key an untargeted- <em>"Does this document contain information that answers the query?"</em> — so <strong>"this document" resolved to- nothing</strong>, and the model judged the whole pile once, echoing roughly that probability back per question.- Within every 30-document window the probabilities were near-uniform (spread ≈ 0.02) — query-level noise wearing a- ranking's clothes.+ <strong>The defect was ours. Not the model's.</strong> Our request builder put every candidate into one shared <code class="inline">state</code> blob. Then it asked each question an untargeted phrase: <em>"Does this document contain information that answers the query?"</em> The words "this document" pointed at nothing. So the model judged the whole pile once. It echoed roughly the same probability back for every question. Inside each 30-document window the probabilities were nearly flat. The spread was about 0.02. That is query-level noise dressed up as a ranking.</div><div class="vt-wrap"><table class="vt"><thead><tr><th>Same 40 judged SciFact queries · same BM25 shortlists · same day</th><th>nDCG@10</th><th>What differed</th></tr></thead><tbody>- <tr><td>BM25 alone — no judge</td><td class="n">0.7750</td><td class="t">baseline</td></tr>- <tr><td>V1 — candidates in shared <code class="inline">state</code>, untargeted question</td>- <td class="n l">0.3822</td><td>what shipped; "this document" bound to nothing</td></tr>- <tr><td>V2 — one noul per document, the document inside its own question</td>- <td class="n w">0.8389</td><td>raw API, no stage machinery around it</td></tr>- <tr><td>The fixed stage — V2 shape through the real <code class="inline">rerank</code> path</td>- <td class="n w">0.8299</td><td>the commit that landed; p50 399 ms / p95 515 ms per query</td></tr>+ <tr><td>BM25 alone, no judge</td><td class="n">0.7750</td><td class="t">baseline</td></tr>+ <tr><td>V1, candidates in shared <code class="inline">state</code>, untargeted question</td>+ <td class="n l">0.3822</td><td>what shipped. "This document" bound to nothing.</td></tr>+ <tr><td>V2, one noul per document, the document inside its own question</td>+ <td class="n w">0.8389</td><td>raw API, no stage code around it</td></tr>+ <tr><td>The fixed stage, V2 shape through the real <code class="inline">rerank</code> path</td>+ <td class="n w">0.8299</td><td>the commit that landed. p50 399 ms, p95 515 ms per query.</td></tr></tbody></table></div><div class="vchart"><div class="vc-legend">- <b class="xerj">BM25 — XERJ first stage</b>+ <b class="xerj">BM25, XERJ first stage</b><b class="bare">Jev through the pre-fix stage</b><b class="native">Jev through the fixed stage</b><b class="native">Jev, raw API</b>@@ -288,7 +279,7 @@<div class="vc-plot"><div class="vc-col xerj"><span class="vc-v">0.7750</span><i class="vc-bar" style="height:92.4%"></i></div><div class="vc-col bare"><span class="vc-v">0.3822</span><i class="vc-bar" style="height:45.6%"></i></div>- <div class="vc-col native"><span class="vc-v">0.8299</span><i class="vc-bar" style="height:99.0%"></i></div>+ <div class="vc-col native"><span class="vc-v">0.8299</span><i class="vc-bar" style="height:98.9%"></i></div><div class="vc-col native"><span class="vc-v">0.8389</span><i class="vc-bar" style="height:100%"></i></div></div><div class="vc-axis">@@ -300,17 +291,11 @@<div class="vc-title"><b>SciFact, first 40 judged queries</b> — pilot, 2026-09-20</div></div></div>- <p class="vc-note">The pilot's role was diagnosis: 40 queries were enough to see a 2× gap. The full run below is- every judged query, three repeats. The shape reference for the fix was read out of the installed- <code class="inline">jev-reranker</code> wheel (MIT) — listwise mode, the document referenced by backtick path- inside its own question — not recalled from memory.</p>+ <p class="vc-note">The pilot was for diagnosis. Forty queries were enough to see a 2× gap. The fix came from reading the installed <code class="inline">jev-reranker</code> wheel (MIT licence), not from memory: listwise mode, the document named by backtick path inside its own question.</p></div><ul class="deep">- <li><strong>The lesson generalises past this bug.</strong> A judge model answers the question it is asked —- literally, at the token level. An integration that scores well can still be asking a different question than the- one you think you're asking. Measure the integration itself, at the raw wire, before believing any number that- comes back.</li>+ <li><strong>The lesson travels.</strong> A judge model answers the question you actually sent, token by token. Your code can score well and still ask a different question than the one you think you asked. Measure the integration at the raw wire before you trust any number that comes back.</li></ul><p class="src">Raw artifacts: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot" rel="external">benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot</a>and <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">…/2026-09-20-rerank-full</a> ·@@ -320,48 +305,44 @@<!-- ============================ 02 · FULL RUN ============================ --><section class="scene-section" id="fullrun">- <div class="kicker"><span class="accent">02</span><span class="dash">·</span><span>THE FULL RUN · EVERY JUDGED QUERY, THREE REPEATS</span></div>- <h2 class="scene" style="font-size:var(--fs-56);">THEN, THE WHOLE<br><span class="accent">DATASET. TWICE.</span></h2>- <p class="wide">300 SciFact queries and 323 NFCorpus queries, BM25 top-30 shortlists fetched once (deterministic),- the fixed request shape sent <strong>three times per dataset with the query order shuffled</strong>, per-query medians- taken over the repeats. Zero failed requests. The repeats exist because the provider is not deterministic —- that finding is below, and it sets the resolution of everything in this section.</p>+ <div class="kicker"><span class="accent">02</span><span class="dash">·</span><span>THE FULL RUN · 1,271 JUDGED QUERIES, THREE PUBLIC DATASETS</span></div>+ <h2 class="scene" style="font-size:var(--fs-56);">THEN THE WHOLE<br><span class="accent">DATASETS. THREE TIMES.</span></h2>+ <p class="wide">Three BEIR datasets, all public, all from the standard distribution. SciFact has 300 judged queries. NFCorpus has 323. FiQA has 648. We fetched BM25 top-30 shortlists once. The shortlist step is deterministic. Then we sent the fixed request shape three times per dataset, with the query order shuffled each time, and took per-query medians. Zero failed requests. We repeated because the provider is not deterministic. That finding is below, and it sets the resolution of everything in this section.</p>++ <p class="wide">Before calling any model we checked our harness on FiQA. Our local BM25 scored <strong>0.2382 nDCG@10</strong>. BEIR's published BM25 figure for FiQA is <strong>0.236</strong>. Close enough to trust the scorer.</p><div class="vt-wrap"><table class="vt">- <thead><tr><th>nDCG@10, BEIR <code class="inline">test</code> split</th><th>SciFact · 300 q</th><th>NFCorpus · 323 q</th><th>Run by</th></tr></thead>+ <thead><tr><th>nDCG@10, BEIR <code class="inline">test</code> split</th><th>SciFact · 300 q</th><th>NFCorpus · 323 q</th><th>FiQA · 648 q</th><th>Run by</th></tr></thead><tbody><tr><td>XERJ BM25 (<code class="inline">multi_match</code>, no embedder)</td>- <td class="n">0.6572</td><td class="n">0.3016</td><td>us · any node</td></tr>- <tr><td>XERJ hybrid RRF (<code class="inline">--embed-mode neural</code>) — measured 2026-09-18, rc.74</td>- <td class="n">0.6993</td><td class="n w">0.3448</td><td>us · that run</td></tr>+ <td class="n">0.6572</td><td class="n">0.3016</td><td class="n">0.2382</td><td>us, any node</td></tr>+ <tr><td>XERJ hybrid RRF (<code class="inline">--embed-mode neural</code>), measured 2026-09-18, rc.74</td>+ <td class="n">0.6993</td><td class="n w">0.3448</td><td class="n t">not run</td><td>us, that run</td></tr><tr><td><strong>XERJ BM25 top-30, reordered by Jev</strong> (<code class="inline">jev-1.13.0</code>, pinned)</td>- <td class="n w">0.7410</td><td class="n">0.3312</td><td>us · this run, median of 3</td></tr>+ <td class="n w">0.7410</td><td class="n">0.3312</td><td class="n w">0.3638</td><td>us, this run, median of 3</td></tr><tr><td>Jev (TypeSafe AI), hosted reranker</td>- <td class="n">0.768</td><td class="n">0.358</td><td><strong>not run by us</strong> — published in the hev/jev-rerank README</td></tr>+ <td class="n">0.768</td><td class="n">0.358</td><td class="n t">no figure</td>+ <td><strong>not run by us</strong>, published in the hev/jev-rerank README</td></tr><tr><td>Voyage rerank-3, hosted reranker</td>- <td class="n">0.755</td><td class="n">0.357</td><td><strong>not run by us</strong> — same README</td></tr>+ <td class="n">0.755</td><td class="n">0.357</td><td class="n t">no figure</td>+ <td><strong>not run by us</strong>, same README</td></tr><tr><td>Cohere rerank-v3.5, hosted reranker</td>- <td class="n">0.745</td><td class="n">0.340</td><td><strong>not run by us</strong> — same README</td></tr>+ <td class="n">0.745</td><td class="n">0.340</td><td class="n t">no figure</td>+ <td><strong>not run by us</strong>, same README</td></tr></tbody></table></div><ul class="deep">- <li><strong>Read the Run by column before the numbers.</strong> The three hosted rows rerank that project's own- first-stage shortlists, not XERJ's — same dataset, same metric, nothing else controlled. Our Jev rows rerank- <em>XERJ's</em> BM25 top-30, which is the configuration an operator can actually run. The published 0.768 and our- 0.7410 are not a race; they are two different experiments that share a metric.</li>- <li><strong>Where the judge wins and where it doesn't.</strong> On SciFact it lifts the best measured XERJ arm- (0.6993 hybrid → 0.7410 reranked). On NFCorpus it does not: BM25+Jev 0.3312 sits <em>below</em> the local hybrid- 0.3448, and the losses concentrate in medical shorthand, where 30-candidate windows are mostly noise to a- general-purpose judge — and in the 25 queries where BM25 returned nothing at all, which no reorder can repair.</li>+ <li><strong>Read the Run-by column before the numbers.</strong> The three hosted rows rerank that project's own first-stage shortlists, not ours. Same dataset, same metric, nothing else controlled. Our Jev rows rerank XERJ's own BM25 top-30. That is the setup an operator can actually run. The published 0.768 and our 0.7410 are not a race. They are two different experiments that share a metric.</li>+ <li><strong>Where the judge wins big, and where it does not.</strong> FiQA is the biggest lift, +0.126. Finance questions live on phrases like "business expense on a business trip", and BM25's word overlap buries good documents deep. The judge reads them and pulls them up. On SciFact it beats our best local arm too, 0.6993 hybrid to 0.7410 reranked. On NFCorpus it loses to the local hybrid, 0.3312 against 0.3448. The losses sit in medical shorthand, where 30-candidate windows are mostly noise to a general judge, and in the 25 queries where BM25 returned nothing at all. No reorder can fix an empty list.</li></ul><div class="vchart"><div class="vc-legend">- <b class="xerj">XERJ BM25 — the shortlist</b>- <b class="native">BM25 top-30 + Jev — this run</b>+ <b class="xerj">XERJ BM25, the shortlist</b>+ <b class="native">BM25 top-30 + Jev, this run</b></div><div class="vc-groups"><div class="vc-group">@@ -382,27 +363,37 @@</div><div class="vc-axis"><span>BM25<span class="sub">no judge</span></span>- <span>+ Jev<span class="sub win">118 W · 74 L · 131 T</span></span>+ <span>+ Jev<span class="sub">118 W · 74 L · 131 T</span></span></div><div class="vc-title"><b>NFCorpus · 323 queries</b> — median of 3 repeats</div></div>+ <div class="vc-group">+ <div class="vc-plot">+ <div class="vc-col xerj"><span class="vc-v">0.2382</span><i class="vc-bar" style="height:65.5%"></i></div>+ <div class="vc-col native"><span class="vc-v">0.3638</span><i class="vc-bar" style="height:100%"></i></div>+ </div>+ <div class="vc-axis">+ <span>BM25<span class="sub">no judge</span></span>+ <span>+ Jev<span class="sub win">249 W · 57 L · 342 T</span></span>+ </div>+ <div class="vc-title"><b>FiQA · 648 queries</b> — median of 3 repeats, the biggest lift</div>+ </div></div>- <p class="vc-note">W / L / T counted per query against the same run's BM25 arm. Both datasets net positive; neither- is a landslide, and NFCorpus's per-query losses are in the table above, not hidden.</p>+ <p class="vc-note">W / L / T counted per query against the same run's BM25 arm. All three datasets net positive. NFCorpus is the weakest, and its per-query losses are in the table above, not hidden.</p></div><div class="vt-wrap"><table class="vt">- <thead><tr><th>The bill, and the clock</th><th>SciFact</th><th>NFCorpus</th></tr></thead>+ <thead><tr><th>The bill, and the clock</th><th>SciFact</th><th>NFCorpus</th><th>FiQA</th></tr></thead><tbody><tr><td>Judge input tokens (document text sent, 3 repeats)</td>- <td class="n">9,589,068</td><td class="n">7,618,170</td></tr>- <tr><td>Cost at $0.042 / Mtok, input-only</td>- <td class="n">$0.4027</td><td class="n">$0.3200</td></tr>+ <td class="n">9,589,068</td><td class="n">7,618,170</td><td class="n">15,178,758</td></tr>+ <tr><td>Cost at $0.042 / Mtok, input only</td>+ <td class="n">$0.4027</td><td class="n">$0.3200</td><td class="n">$0.6375</td></tr><tr><td>Wall clock per query, p50</td>- <td class="n">1.49 s</td><td class="n">1.48 s</td></tr>+ <td class="n">1.49 s</td><td class="n">1.48 s</td><td class="n">1.23 s</td></tr><tr><td>Failed requests</td>- <td class="n">0</td><td class="n">0</td></tr>+ <td class="n">0</td><td class="n">0</td><td class="n">0</td></tr></tbody></table></div>@@ -410,92 +401,91 @@<div class="wire"><div class="wp"><div class="wh"><span class="wn">FINDING 1</span><h3>The provider is not deterministic</h3>- <span class="wm">900 same-document pairs per dataset</span></div>- <p>The same request, three times: mean per-document probability drift- <strong>|Δ| 0.0077</strong> (SciFact) / <strong>0.0096</strong> (NFCorpus), max 0.08. Repeat-level nDCG means- spanned 0.7422–0.7474 and 0.3285–0.3316. <strong>Any comparison of Jev numbers tighter than ~±0.01 nDCG@10 is- measuring noise.</strong> Our repeats were medians for exactly this reason; a single-shot benchmark of a hosted- judge is a coin with a ±0.01 bias.</p>+ <span class="wm">same request, sent 3 times</span></div>+ <p>Mean per-document probability drift was <strong>0.0077</strong> on SciFact and <strong>0.0096</strong> on NFCorpus, with a max of 0.08. On FiQA the repeat means spanned 0.3622 to 0.3648. So: <strong>any comparison of Jev numbers closer than about ±0.01 nDCG@10 is measuring noise.</strong> We report medians for that reason. A single-shot benchmark of a hosted judge is a coin with a ±0.01 bias.</p></div><div class="wp">- <div class="wh"><span class="wn">FINDING 2</span><h3>The probabilities are not calibrated to BEIR relevance</h3>- <span class="wm">9,000 + 7,186 (probability, relevance) pairs</span></div>- <p><strong>Expected calibration error 0.1023 on SciFact, 0.1332 on NFCorpus.</strong> The reliability curve is- monotone — the model discriminates, it orders documents well — but every bin above 0.2 overstates relevance.- On SciFact, documents the model rated around <strong>0.85 were relevant 42% of the time</strong>; on NFCorpus,- bin 9 is 0.92 confidence / 0.55 actual. A Jev probability is a usable <em>ordering</em> signal and an unusable- <em>threshold</em> against graded human labels: setting <code class="inline">min_score</code> on the raw number- cuts documents the qrels call relevant. The curves are in the result JSONs, bin by bin.</p>+ <div class="wh"><span class="wn">FINDING 2</span><h3>The probabilities do not mean what they say</h3>+ <span class="wm">19,440 confidence/relevance pairs on FiQA</span></div>+ <p>A judge probability is not the chance a document is relevant. On FiQA the expected calibration error is <strong>0.3109</strong>. Documents the model rated near <strong>0.93 confidence were relevant 34% of the time</strong>. The curve still rises to the right, so the model ranks well. But every bin above 0.2 promises far more than it delivers. SciFact: ECE 0.1023, bin 0.85 confidence, 42% relevant. NFCorpus: ECE 0.1332, bin 0.92 confidence, 55% relevant. Use the ordering. Never set <code class="inline">min_score</code> on the raw number.</p>+ </div>+ </div>++ <div class="vchart">+ <div class="vc-legend">+ <b class="bare">what the judge said</b>+ <b class="xerj">what was true</b>+ </div>+ <div class="vc-groups">+ <div class="vc-group">+ <div class="vc-plot">+ <div class="vc-col bare"><span class="vc-v">0.75</span><i class="vc-bar" style="height:74.6%"></i></div>+ <div class="vc-col xerj"><span class="vc-v">0.04</span><i class="vc-bar" style="height:4.1%"></i></div>+ <div class="vc-col bare"><span class="vc-v">0.85</span><i class="vc-bar" style="height:84.5%"></i></div>+ <div class="vc-col xerj"><span class="vc-v">0.12</span><i class="vc-bar" style="height:12.4%"></i></div>+ <div class="vc-col bare"><span class="vc-v">0.93</span><i class="vc-bar" style="height:93.0%"></i></div>+ <div class="vc-col xerj"><span class="vc-v">0.34</span><i class="vc-bar" style="height:33.8%"></i></div>+ </div>+ <div class="vc-axis">+ <span>bin 7<span class="sub">said / was</span></span>+ <span></span>+ <span>bin 8<span class="sub">said / was</span></span>+ <span></span>+ <span>bin 9<span class="sub">said / was</span></span>+ <span></span>+ </div>+ <div class="vc-title"><b>FiQA, top three confidence bins</b> — mean confidence vs share actually relevant</div>+ </div></div>+ <p class="vc-note">19,440 (probability, relevance) pairs, 10 equal-width bins; the three highest are shown. The full curve, bin by bin, is in <code class="inline">fiqa-full-run.json</code>. The gap between the red bar and the gold bar is the whole finding: at the top of the scale the judge says "certain" and means "one in three".</p></div><ul class="deep">- <li><strong>About the calibration number you'll meet again below.</strong> The local history vote on this site- measured ECE 0.012 on Banking77. That is <em>not</em> a scoreboard win over the judge's 0.10–0.13: the vote was- in-domain classification against its own training labels; the judge was scored cross-domain against BEIR's graded- human relevance. Calibration is a property of the target, not of the interface. Both numbers are honest on their- own task, and neither redeems the other.</li>+ <li><strong>About the calibration number you will meet again below.</strong> The local history vote on this site measured ECE 0.012 on Banking77. That is not a scoreboard win over the judge's 0.10 to 0.31. The vote did in-domain classification against its own training labels. The judge was scored cross-domain against graded human relevance labels. Calibration belongs to the task, not to the interface. Both numbers are honest on their own task. Neither redeems the other.</li></ul>- <p class="src">Runner and raw JSON: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">benchmarks/beir-hybrid/results/2026-09-20-rerank-full</a>- (per-query BM25/median numbers, reliability curves, repeat spreads) · run 2026-09-20 on the wire-compat branch's- engine, scoring byte-identical to the hybrid benchmark's <code class="inline">eval.py</code> ·- hosted-reranker figures: <a href="https://github.com/hev/jev-rerank" rel="external">hev/jev-rerank</a> (MIT), a third party's, not TypeSafe's and not ours.</p>+ <p class="src">Runner and raw JSON: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">…/2026-09-20-rerank-full</a> (SciFact, NFCorpus) and+ <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa" rel="external">…/2026-09-20-rerank-full-fiqa</a> (FiQA, with dataset provenance) ·+ hybrid-row figures (0.6993 / 0.3448, 2026-09-18, rc.74): <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid" rel="external">benchmarks/beir-hybrid</a> (<code class="inline">results/scifact.txt</code>, <code class="inline">results/nfcorpus.txt</code>) ·+ run 2026-09-20 on the wire-compat branch engine, scoring byte-identical to the hybrid benchmark's <code class="inline">eval.py</code> ·+ FiQA fetched from the <a href="https://github.com/beir-cellar/beir" rel="external">standard BEIR distribution</a> ·+ hosted-reranker figures: <a href="https://github.com/hev/jev-rerank" rel="external">hev/jev-rerank</a> (MIT). A third party published them. Not TypeSafe's numbers, and not ours.</p></section><!-- ============================ 03 · THE WIRE ============================ --><section class="scene-section" id="wire"><div class="kicker"><span class="accent">03</span><span class="dash">·</span><span>THE WIRE · A XERJ NODE THAT ANSWERS /v1/systemone</span></div><h2 class="scene" style="font-size:var(--fs-56);">THEN WE ASKED THE<br><span class="accent">QUESTION BACKWARDS.</span></h2>- <p class="wide">System One is a documented interface for typed judgement: <code class="inline">POST /v1/systemone</code>- with a state, a model, and a map of questions, each answered <code class="inline">noul</code> (a yes/no with a 0–1- probability) or <code class="inline">choice</code> (N named options with probabilities). XERJ already had the retrieval- answer for exactly that shape of question: <strong>index the labelled history you already have, answer a new question- by a rank-weighted vote over its k nearest neighbours, and report the winning label's vote share as the- probability</strong> — sub-millisecond, BM25-only, no model. So the node learned to speak the wire.</p>+ <p class="wide">System One is a documented interface for typed judgement. You POST a state, a model, and a map of questions. Each question comes back <code class="inline">noul</code>, a yes/no with a 0 to 1 probability, or <code class="inline">choice</code>, N named options with probabilities. XERJ already had a fast answer for exactly that shape of question. <strong>Index the labelled history you already have. Answer a new question by a rank-weighted vote over its k nearest neighbours. Report the winning label's vote share as the probability.</strong> No model. No tokens. So the node learned to speak the wire.</p><div class="wire"><div class="wp"><div class="wh"><span class="wn">SURFACE 1</span><h3><code class="inline">POST /v1/systemone</code></h3><span class="wm">native REST · --port + 1</span></div>- <p>The documented request and response shape, answered by the vote. A client written for TypeSafe's API works- unmodified: <code class="inline">state</code> string or object, <code class="inline">instructions</code> string /- object / array, backtick paths (<code class="inline">`documents.doc_0`</code>) resolved against state before the- vote text is built, answers keyed exactly by the ids sent — everything the strict clients validate.</p>+ <p>The documented request and response shape, answered by the vote. A client written for TypeSafe's API works unmodified. <code class="inline">state</code> as string or object. <code class="inline">instructions</code> as string, object, or array. Backtick paths like <code class="inline">`documents.doc_0`</code> resolved against state. Answers keyed by exactly the ids sent. Everything strict clients validate is there.</p></div><div class="wp"><div class="wh"><span class="wn">SURFACE 2</span><h3><code class="inline">POST /_decide</code></h3><span class="wm">ES-compat port · the audit surface</span></div>- <p>The same vote without the wire costume: it names its index per request, and it returns the- <strong>evidence</strong> — each neighbour's id, label, the engine's own BM25 score, its 1/rank weight, and its- text — so any answer can be checked against the history that produced it. Abstains below- <code class="inline">decisions.min_confidence</code>.</p>+ <p>The same vote without the wire costume. It names its index per request, and it returns the <strong>evidence</strong>: each neighbour's id, label, the engine's own BM25 score, its 1/rank weight, and its text. Any answer can be checked against the history that produced it. Below <code class="inline">decisions.min_confidence</code> it abstains.</p></div><div class="wp neg"><div class="wh"><span class="wn">BREAK 1</span><h3><code class="inline">model</code> is echoed truthfully</h3><span class="wm">deliberate · test-pinned</span></div>- <p>The response says <code class="inline">xerj-history-vote-1</code>, never a Jev model name. Echoing- <code class="inline">jev-1.13.0</code> would claim these probabilities are the hosted model's; the requested name- rides at <code class="inline">decisions.requested_model</code>. This page is about compatibility, and- compatibility does not require lying about whose numbers they are.</p>+ <p>The response says <code class="inline">xerj-history-vote-1</code>. Never a Jev model name. Echoing <code class="inline">jev-1.13.0</code> would claim these probabilities are the hosted model's. The name you asked for rides at <code class="inline">decisions.requested_model</code>. Compatibility does not require lying about whose numbers they are.</p></div><div class="wp neg">- <div class="wh"><span class="wn">BREAK 2</span><h3>Zero support is a 422, never a fabricated 0.5</h3>+ <div class="wh"><span class="wn">BREAK 2</span><h3>Zero support is a 422, never a made-up 0.5</h3><span class="wm">deliberate · test-pinned</span></div>- <p>A question whose vote finds no labelled neighbour carries no information. The endpoint refuses it — HTTP 422,- naming the question ids — rather than inventing a middle probability. A silent 0.5 is exactly the- silent-fake defect class this project treats as a bug, wire compatibility or not.</p>+ <p>A question whose vote finds no labelled neighbour carries no information. The endpoint refuses it. HTTP 422, naming the question ids. A silent 0.5 is exactly the silent-fake defect class this project treats as a bug. Wire compatibility does not change that.</p></div></div><h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">The gate: the pip client, unmodified, on a local node</h3>- <p class="wide">No SDK of ours, no patches, no subclassing — <code class="inline">pip install jev-reranker</code>- (0.1.0, MIT) in a clean venv, its own listwise instruction template, its own splitting, its own strict response- validation, pointed at a XERJ node holding 4,000 labelled SMS messages. Two environment variables, and one of them- is a trap:</p>+ <p class="wide">No SDK of ours, no patches, no subclassing. <code class="inline">pip install jev-reranker</code> (0.1.0, MIT) in a clean venv, its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. Two environment variables, and one of them is a trap:</p><pre class="code"><span class="prompt">$</span> pip install jev-reranker-<span class="prompt">$</span> <span class="cmt"># TYPESAFE_ENDPOINT carries the FULL path — TYPESAFE_BASE_URL is silently ignored</span>-<span class="prompt">$</span> <span class="cmt"># by this client (reranker.py:238) and the call still goes to the real host.</span>+<span class="prompt">$</span> <span class="cmt"># TYPESAFE_ENDPOINT carries the FULL path. TYPESAFE_BASE_URL is silently ignored</span>+<span class="prompt">$</span> <span class="cmt"># by this client (reranker.py:238), and the call still goes to the real host.</span><span class="prompt">$</span> export TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone<span class="prompt">$</span> export TYPESAFE_API_KEY=local-only <span class="cmt"># the node ignores it; localhost only</span><span class="prompt">$</span> python3 gate_run.py@@ -509,36 +499,66 @@mean noul spam=0.9014 ham=0.0956 gap=0.8058GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre>- <p class="wide">The client's own validation <em>is</em> the wire gate — answers keyed exactly by the sent ids, every- noul numeric in 0..1, model a non-empty string, usage non-negative integers. Surviving it unmodified is the- compatibility claim; the spam/ham separation is the semantic one. <strong>Zero requests left the machine:</strong>- the vote is an ordinary search of an ordinary index, and the module adds no outbound client.</p>+ <p class="wide">The client's own validation <em>is</em> the wire gate. Answers keyed by the sent ids. Every noul numeric in 0..1. Model a non-empty string. Usage non-negative integers. Surviving it unmodified is the compatibility claim. The spam/ham separation is the semantic one. <strong>Zero requests left the machine.</strong> The vote is an ordinary search of an ordinary index, and the module adds no outbound client.</p><div class="loss">- <strong>The first gate attempt failed, and the failure is the most useful thing on this page.</strong> The wire puts- the query in <code class="inline">state</code>, and the vote text joins it — so query vocabulary participates in- retrieval. With a spam-bearing query ("winner claim prize"), <em>every</em> document retrieved spam neighbours and- the vote measured the query, not the documents: spam 1.000 vs ham 0.785, no separation. The fixed gate uses a- query whose every term is absent from the corpus ("triage inbox unsolicited correspondence"): 0.9014 vs 0.0956.- This is inherent to a retrieval vote, not a defect to fix — <strong>pick the query for the history you- have</strong>, and don't let a rerank-shaped habit leak vocabulary into a classification-shaped question.+ <strong>The first gate attempt failed. It is the most useful thing on this page.</strong> The wire puts the query in <code class="inline">state</code>, and the vote text joins it, so query vocabulary takes part in retrieval. With a spam-bearing query like "winner claim prize", <em>every</em> document's vote retrieved spam neighbours. The vote measured the query, not the documents: spam 1.000 against ham 0.785. No separation. The fixed gate uses a query whose every term is absent from the corpus: "triage inbox unsolicited correspondence". Result, 0.9014 against 0.0956. This is inherent to a retrieval vote, not a defect. <strong>Pick the query for the history you have.</strong> Do not let a rerank habit leak vocabulary into a classification question.+ </div>++ <h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">Speed: the Rust side of the same wire</h3>+ <p class="wide">A full news item against 120,000 labelled AG News examples is a heavy query. About 40 terms, and the match set is most of the index. This is the workload that exposed the waste in our own search loop. The old path built a hit list per term, folded them through hash maps, and sorted the whole match set. All that to return a size-10 page. We replaced it with a WAND-style pruned walk for the one shape every <code class="inline">match</code> query becomes. A document is now scored only when it can still make the page even at its best possible score. The method is the classic one from Broder et al., and the same idea tantivy uses, adapted to our postings format.</p>++ <div class="vt-wrap">+ <table class="vt">+ <thead><tr><th>AG News vote · 120,000 history docs · all 7,600 test items · identical inputs</th><th>ms / item</th><th>Accuracy</th><th>Decisions changed</th></tr></thead>+ <tbody>+ <tr><td>Engine before the change</td><td class="n">68.7</td><td class="n">0.9182</td><td class="t">—</td></tr>+ <tr><td>WAND first cut</td><td class="n">51.2</td><td class="n">0.9182</td><td class="n w">none</td></tr>+ <tr><td>WAND, allocation-free inner loop (shipped)</td><td class="n w">34.2</td><td class="n">0.9182</td><td class="n w">none</td></tr>+ </tbody>+ </table>+ </div>++ <p class="wide"><strong>2.0× faster end to end, and the 7,600 decisions are bit-identical</strong>: same labels, same confidence bands. 83.1% of items decided at confidence 0.8 or higher, and 96.7% of those right. Zero items had no neighbour. Ranking quality did not move because scores are computed in the same order and to the same values. We proved that three ways. The three BEIR runs reproduce to four decimals on the new engine, 0.6572, 0.3016, 0.2382. A 648-query A/B over a byte-identical copy of the FiQA index found zero order changes and zero score-bit changes. The full ES wire-conformance suite stayed at 1371 passed, 0 failed.</p>++ <div class="vchart">+ <div class="vc-legend">+ <b class="bare">hosted Jev API</b>+ <b class="xerj">XERJ node, this engine</b>+ </div>+ <div class="vc-groups">+ <div class="vc-group">+ <div class="vc-plot">+ <div class="vc-col bare"><span class="vc-v">1,230 ms</span><i class="vc-bar" style="height:100%"></i></div>+ <div class="vc-col xerj"><span class="vc-v">42.7 ms</span><i class="vc-bar" style="height:3.5%; min-height:8px"></i></div>+ <div class="vc-col xerj"><span class="vc-v">34.8 ms</span><i class="vc-bar" style="height:2.8%; min-height:8px"></i></div>+ </div>+ <div class="vc-axis">+ <span>hosted<span class="sub">paid, text leaves</span></span>+ <span>/v1/systemone<span class="sub win">local, free</span></span>+ <span>/_decide<span class="sub">local, evidence</span></span>+ </div>+ <div class="vc-title"><b>Same wire, p50 per call</b> — hosted: FiQA run · local: 300 distinct items, one box</div>+ </div>+ </div>+ <p class="vc-note">The hosted bar is the Jev API's measured p50 on the FiQA rerank run in §02, 1.23 s per query. Its fastest of the three datasets, so the bar is generous to the hosted side; SciFact and NFCorpus measured 1.49 s and 1.48 s. The local bars are the same wire answered by the vote over 120k AG News history on this engine. Different tools: the hosted judge needs no history, the vote needs this index. This is a speed, cost and egress comparison, not a quality race. No Jev number exists for AG News and we claim none.</p></div><h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">What the vote itself is worth</h3>- <p class="wide">Measured earlier on the same engine (<a href="/benchmarks/">the hub's ranking section</a>), on the- vote's home ground — labelled history, in-domain classification. <strong>No judge model was run on these datasets by- us, and none of these rows say anything about Jev.</strong></p>+ <p class="wide">Measured earlier on the same engine (<a href="/benchmarks/">the hub's ranking section</a>), on the vote's home ground: labelled history, in-domain classification. <strong>No judge model was run on these datasets by us, and none of these rows say anything about Jev.</strong></p><div class="vt-wrap"><table class="vt">- <thead><tr><th>Decisions from labelled history · k = 10 · 1,000 held-out items</th><th>Accuracy</th><th>Calibration error</th><th>Decided at confidence ≥ 0.8</th><th>ms / item</th></tr></thead>+ <thead><tr><th>Decisions from labelled history · k = 10 · held-out items</th><th>Accuracy</th><th>Calibration error</th><th>Decided at confidence ≥ 0.8</th><th>ms / item</th></tr></thead><tbody>- <tr><td>Banking77, 77-way intent routing — BM25, no model of any kind</td>+ <tr><td>Banking77, 77-way intent routing · 1,000 items — BM25, no model of any kind</td><td class="n">0.819</td><td class="n">0.089</td><td class="n">50.6% at 0.996</td><td class="n">2.2</td></tr><tr><td>Banking77 — MiniLM neighbours (<code class="inline">--embed-mode neural</code>)</td><td class="n w">0.933</td><td class="n w">0.012</td><td class="n w">86.9% at 0.979</td><td class="n">8.6</td></tr>- <tr><td>SMS spam, yes / no — BM25, no model of any kind</td>+ <tr><td>SMS spam, yes / no · 1,000 items — BM25, no model of any kind</td><td class="n w">0.983</td><td class="n">0.017</td><td class="n">95.4% at 0.992</td><td class="n w">0.9</td></tr>+ <tr><td>AG News, 4-way topic · 7,600 items — BM25, no model of any kind</td>+ <td class="n w">0.9182</td><td class="n">0.019</td><td class="n">83.1% at 0.9669</td><td class="n">34.2</td></tr></tbody></table></div>@@ -550,39 +570,35 @@ GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre>"neighbours":[{"_id":"1805","label":"spam",...,"weight":1.0}, ...]}</span></pre><ul class="deep">- <li><strong>Read this before quoting the compatibility claim.</strong> The node did not "become a language model."- It answers a typed judgement wire from a search over your own labelled history, says so in its model field,- refuses what it cannot know (zero support, <code class="inline">score</code>-type ordinal questions), and attaches- the neighbours that produced every answer. If you need judgement where you have <em>no</em> history — a new- policy, an unseen category — this is the wrong tool, and the hosted judge is the right one. The two ends of the- wire are complements, not competitors.</li>+ <li><strong>Read this before quoting the compatibility claim.</strong> The node did not "become a language model." It answers a typed judgement wire from a search over your own labelled history. It says so in its model field. It refuses what it cannot know: zero support, <code class="inline">score</code>-type ordinal questions. It attaches the neighbours behind every answer. If you need judgement where you have <em>no</em> history, a new policy, an unseen category, this is the wrong tool and the hosted judge is the right one. The two ends of the wire are complements, not competitors.</li>+ <li><strong>Latency scales with the match set, not with the corpus.</strong> AG News items are long and match nearly everything, so they cost tens of milliseconds. Short texts on the same engine cost far less: SMS is 0.9 ms per message. One shape at one scale, measured. No TB-scale claim.</li></ul>- <p class="src">Endpoint: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-api/src/systemone_api.rs" rel="external">engine/crates/xerj-api/src/systemone_api.rs</a> ·+ <p class="src">WAND engine change: <a href="https://github.com/xerj-org/xerj/commit/cf6a9989" rel="external">cf6a9989</a>, with the A/B logs ·+ AG News artifacts and provenance: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews" rel="external">benchmarks/decisions-as-retrieval/results/2026-09-20-agnews</a> ·+ endpoint: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-api/src/systemone_api.rs" rel="external">engine/crates/xerj-api/src/systemone_api.rs</a> ·14 end-to-end tests: <a href="https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-api/tests/systemone_http.rs" rel="external">tests/systemone_http.rs</a> ·operator doc: <a href="https://github.com/xerj-org/xerj/blob/main/docs/DECISIONS.md" rel="external">docs/DECISIONS.md</a> ·- gate harness and transcript: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/systemone-gate" rel="external">benchmarks/systemone-gate</a> ·- the vote's own numbers: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval" rel="external">benchmarks/decisions-as-retrieval</a>.</p>+ gate harness and transcript: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/systemone-gate" rel="external">benchmarks/systemone-gate</a>.</p></section><!-- ============================ 04 · ECONOMICS ============================ --><section class="scene-section" id="economics"><div class="kicker"><span class="accent">04</span><span class="dash">·</span><span>THE ECONOMICS · RENTED JUDGEMENT, OWNED DECISIONS</span></div><h2 class="scene" style="font-size:var(--fs-56);">JUDGEMENT IS RENTED.<br><span class="accent">DECISIONS ARE OWNED.</span></h2>- <p class="wide">The full run's bill, next to the vote's, for the same 300-query workload — different tools, priced- honestly side by side:</p>+ <p class="wide">The judge's bill on the 300-query SciFact run, next to the vote's numbers on its own measured workloads. Two tools, two workloads, priced side by side:</p><div class="vt-wrap"><table class="vt">- <thead><tr><th>SciFact-scale workload · 300 queries</th><th>Judge (hosted, this run)</th><th>Vote (local, this engine)</th></tr></thead>+ <thead><tr><th>The bills</th><th>Judge (hosted, SciFact run)</th><th>Vote (local, own workloads)</th></tr></thead><tbody><tr><td>Cash cost</td><td class="n">$0.4027 <span class="t">at $0.042/Mtok in</span></td><td class="n w">$0</td></tr><tr><td>Latency per item, p50</td>- <td class="n">1.49 s</td><td class="n w">0.9–2.2 ms</td></tr>+ <td class="n">1.49 s</td><td class="n w">34.8 ms <span class="t">(measured on 120k AG News, its heaviest workload)</span></td></tr><tr><td>Document text that leaves the machine</td><td class="n l">9.6 M tokens' worth <span class="t">(3 repeats)</span></td><td class="n w">none</td></tr><tr><td>Works with no labelled history</td>- <td class="n w">yes — that is its job</td><td class="n l">no — that is its limit</td></tr>+ <td class="n w">yes, that is its job</td><td class="n l">no, that is its limit</td></tr><tr><td>Calibration, on its own target</td><td class="n">ECE 0.10 <span class="t">cross-domain BEIR</span></td><td class="n">ECE 0.012 <span class="t">in-domain Banking77</span></td></tr></tbody>@@ -590,28 +606,12 @@ GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre></div><ul class="deep">- <li><strong>Both columns survive contact with the other's weakness.</strong> The judge is the only option where- there is no history — and every call is metered, slower, and ships document text to a third party. The vote is- free, sub-millisecond, and never leaves the node — and it cannot judge anything your history doesn't already- cover. At 300 queries a day the judge costs $0.40 and is irrelevant to your budget — use it freely. At- 300,000 messages a day it is 1.49 s each — <strong>124 hours of provider latency per day</strong>, however you- parallelise it — plus a bill of roughly <strong>$400 a day</strong> at this run's measured rate, plus every one- of those messages leaving the node. The vote answers each in under a millisecond, on the node, for nothing.</li>- <li><strong>Why the wire matters more than either number.</strong> Every System One client — the reranker, the- SDKs, whatever an agent is already holding — now has a second, local implementation of the interface it speaks.- Point <code class="inline">TYPESAFE_ENDPOINT</code> at a XERJ node when the questions have history and must stay- home; point it at the hosted API when they don't. Same client, same wire, no code change — that is the whole- bridge.</li>+ <li><strong>Both columns survive contact with the other's weakness.</strong> The judge is the only option where there is no history. And every call is metered, slower, and ships document text to a third party. The vote is free, tens of milliseconds, and never leaves the node. And it cannot judge anything your history does not already cover. At 300 queries a day the judge costs $0.40 and is irrelevant to your budget. Use it freely. At 300,000 messages a day, at this run's measured 1.49 s per call, that is <strong>over 120 hours of provider latency per day</strong>, however you parallelise it. It is roughly <strong>$400 a day</strong> at this run's measured rate. And every one of those messages leaves the node. The vote answers each in tens of milliseconds, on the node, for nothing.</li>+ <li><strong>Why the wire matters more than either number.</strong> Every System One client, the reranker, the SDKs, whatever an agent is already holding, now has a second local implementation of the interface it speaks. Point <code class="inline">TYPESAFE_ENDPOINT</code> at a XERJ node when the questions have history and must stay home. Point it at the hosted API when they do not. Same client. Same wire. No code change. That is the whole bridge.</li></ul><div class="loss" style="border-left-color:var(--z-accent);">- <strong>What we claim, and what we do not.</strong> Claimed: every number on this page, from the runs cited beside- it, on our node, our shortlists, the pinned model. Not claimed: anything about how Jev scores on Banking77 or SMS- (TypeSafe has published no such numbers and we ran none — which is why those tables carry no judge row); any- zero-shot ability for the local vote; that the published hosted-reranker figures and ours are comparable beyond- dataset and metric. TypeSafe AI granted the evaluation key that made §01–§02 possible — the measurements are of- their model, through their documented API, and this project thanks them for the access. The findings here are- ours, warts published.+ <strong>What we claim, and what we do not.</strong> Claimed: every number on this page, from the runs cited beside it, on our node, our shortlists, the pinned model. Not claimed: anything about how Jev scores on Banking77, SMS or AG News. TypeSafe has published no such numbers and we ran none, which is why those tables carry no judge row. Not claimed: any zero-shot ability for the local vote, or that the published hosted-reranker figures and ours compare beyond dataset and metric. TypeSafe AI granted the evaluation key that made §01 and §02 possible. The measurements are of their model, through their documented API, and this project thanks them for the access. The findings are ours, warts published.</div><h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">Reproduce the gate</h3>@@ -635,9 +635,11 @@ EOFGATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre><p class="src">Everything on this page: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot" rel="external">the pilot</a> ·<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">the full run</a> ·+ <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa" rel="external">the FiQA run</a> ·+ <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews" rel="external">the AG News vote</a> ·<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/systemone-gate" rel="external">the gate</a> ·<a href="https://github.com/xerj-org/xerj/blob/main/docs/research/typesafe-skill-2026-09/NOTES.md" rel="external">the research narrative</a> ·- measured 2026-09-19..20 · engine at the wire-compat branch, ES-YAML conformance 1371 passed / 0 failed ·+ measured 2026-09-19..20 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance 1371 passed / 0 failed ·related: <a href="/benchmarks/">the benchmarks hub</a> · <a href="/answers/rerank-search-results-calibrated-judge">the rerank stage, documented</a> ·<a href="/benchmarks/elasticsearch">the ES scorecard</a>.</p></section>diff --git a/landing/benchmarks/index.html b/landing/benchmarks/index.htmlindex 8edbd94ce..c3239f0b9 100644--- a/landing/benchmarks/index.html+++ b/landing/benchmarks/index.html@@ -400,24 +400,24 @@<div class="vt-wrap"><table class="vt">- <thead><tr><th>nDCG@10, BEIR <code class="inline">test</code> split</th><th>SciFact · 300 q · 5,183 docs</th><th>NFCorpus · 323 q · 3,633 docs</th><th>Run by</th></tr></thead>+ <thead><tr><th>nDCG@10, BEIR <code class="inline">test</code> split</th><th>SciFact · 300 q · 5,183 docs</th><th>NFCorpus · 323 q · 3,633 docs</th><th>FiQA · 648 q · 57,638 docs</th><th>Run by</th></tr></thead><tbody><tr><td>XERJ BM25 (<code class="inline">multi_match</code>, no embedder)</td>- <td class="n">0.6572</td><td class="n">0.3016</td><td>us · any node</td></tr>+ <td class="n">0.6572</td><td class="n">0.3016</td><td class="n">0.2382</td><td>us · any node</td></tr><tr><td>XERJ MiniLM vectors only (<code class="inline">semantic</code>)</td>- <td class="n">0.6764</td><td class="n">0.3291</td><td>us · <code class="inline">--embed-mode neural</code></td></tr>+ <td class="n">0.6764</td><td class="n">0.3291</td><td class="n t">not run</td><td>us · <code class="inline">--embed-mode neural</code></td></tr><tr><td>XERJ BM25 top-30, reordered by MiniLM</td>- <td class="n">0.6855</td><td class="n">0.3323</td><td>us · <code class="inline">--embed-mode neural</code></td></tr>+ <td class="n">0.6855</td><td class="n">0.3323</td><td class="n t">not run</td><td>us · <code class="inline">--embed-mode neural</code></td></tr><tr><td><strong>XERJ hybrid RRF</strong> (<code class="inline">hybrid</code>, server-side)</td>- <td class="n">0.6993</td><td class="n w">0.3448</td><td>us · <code class="inline">--embed-mode neural</code></td></tr>+ <td class="n">0.6993</td><td class="n w">0.3448</td><td class="n t">not run</td><td>us · <code class="inline">--embed-mode neural</code></td></tr><tr><td><strong>XERJ BM25 top-30, reordered by Jev</strong> (<code class="inline">jev-1.13.0</code>, pinned)</td>- <td class="n w">0.7410</td><td class="n">0.3312</td><td>us · median of 3 repeats — <a href="/benchmarks/decisions">the write-up</a></td></tr>+ <td class="n w">0.7410</td><td class="n">0.3312</td><td class="n w">0.3638</td><td>us · median of 3 repeats — <a href="/benchmarks/decisions">the write-up</a></td></tr><tr><td>Jev (TypeSafe AI), hosted reranker</td>- <td class="n">0.768</td><td class="n">0.358</td><td><strong>not run by us</strong> — published in the hev/jev-rerank README</td></tr>+ <td class="n">0.768</td><td class="n">0.358</td><td class="n t">no figure</td><td><strong>not run by us</strong> — published in the hev/jev-rerank README</td></tr><tr><td>Voyage rerank-3, hosted reranker</td>- <td class="n">0.755</td><td class="n">0.357</td><td><strong>not run by us</strong> — same README</td></tr>+ <td class="n">0.755</td><td class="n">0.357</td><td class="n t">no figure</td><td><strong>not run by us</strong> — same README</td></tr><tr><td>Cohere rerank-v3.5, hosted reranker</td>- <td class="n">0.745</td><td class="n">0.340</td><td><strong>not run by us</strong> — same README</td></tr>+ <td class="n">0.745</td><td class="n">0.340</td><td class="n t">no figure</td><td><strong>not run by us</strong> — same README</td></tr></tbody></table></div>@@ -439,9 +439,12 @@disaster we published.</strong> With an evaluation key granted by TypeSafe, the stage as it shipped scored<strong>0.3822 nDCG@10 against BM25's 0.7750</strong> on a 40-query pilot — our request asked the judge anuntargeted question, so it judged the pile once and echoed query-level noise. Root-caused, fixed- (<strong>0.8299</strong>, raw API 0.8389), then run in full: <strong>0.7410 / 0.3312</strong> over 300+323- queries, three repeats. The probabilities order documents well but are <strong>not calibrated to BEIR- relevance</strong> (ECE 0.10–0.13) — do not <code class="inline">min_score</code> the raw number.+ (<strong>0.8299</strong>, raw API 0.8389), then run in full on three datasets:+ <strong>0.7410 / 0.3312 / 0.3638</strong> over 1,271 judged queries, three repeats each. FiQA is the judge's+ biggest lift (+0.126 over BM25) and its worst calibration case. The probabilities order documents well but+ are <strong>not calibrated to BEIR relevance</strong>. ECE runs 0.10–0.31 across the three datasets, and+ on FiQA documents rated 0.93 were relevant 34% of the time. Do not <code class="inline">min_score</code>+ the raw number.<a href="/benchmarks/decisions">The full autopsy, run and gate</a>. The stage is also the one XERJ featurethat sends document text off the machine —<a href="/answers/rerank-search-results-calibrated-judge">read that before turning it on</a>.</li>@@ -449,20 +452,22 @@<div class="vt-wrap"><table class="vt">- <thead><tr><th>Decisions from labelled history · k = 10 · 1,000 held-out items</th><th>Accuracy</th><th>Calibration error</th><th>Decided at confidence ≥ 0.8</th><th>ms / item</th></tr></thead>+ <thead><tr><th>Decisions from labelled history · k = 10 · held-out items</th><th>Accuracy</th><th>Calibration error</th><th>Decided at confidence ≥ 0.8</th><th>ms / item</th></tr></thead><tbody>- <tr><td>Banking77, 77-way intent routing — BM25, no model of any kind</td>+ <tr><td>Banking77, 77-way intent routing · 1,000 items — BM25, no model of any kind</td><td class="n">0.819</td><td class="n">0.089</td><td class="n">50.6% at 0.996</td><td class="n">2.2</td></tr><tr><td>Banking77 — MiniLM neighbours (<code class="inline">--embed-mode neural</code>)</td><td class="n w">0.933</td><td class="n w">0.012</td><td class="n w">86.9% at 0.979</td><td class="n">8.6</td></tr><tr><td>Banking77 — hybrid RRF (<code class="inline">--embed-mode neural</code>)</td><td class="n w">0.937</td><td class="n">0.052</td><td class="n">79.2% at 0.989</td><td class="n">12.4</td></tr>- <tr><td>SMS spam, yes / no — BM25, no model of any kind</td>+ <tr><td>SMS spam, yes / no · 1,000 items — BM25, no model of any kind</td><td class="n w">0.983</td><td class="n">0.017</td><td class="n">95.4% at 0.992</td><td class="n w">0.9</td></tr><tr><td>SMS spam — MiniLM neighbours (<code class="inline">--embed-mode neural</code>)</td><td class="n">0.982</td><td class="n">0.009</td><td class="n">95.8% at 0.989</td><td class="n">74.5</td></tr><tr><td>SMS spam — hybrid RRF (<code class="inline">--embed-mode neural</code>)</td><td class="n w">0.989</td><td class="n">0.015</td><td class="n">95.7% at 0.997</td><td class="n">67.2</td></tr>+ <tr><td>AG News, 4-way topic · 7,600 items · 120,000-deep history — BM25, no model of any kind</td>+ <td class="n w">0.9182</td><td class="n">0.019</td><td class="n">83.1% at 0.9669</td><td class="n">34.2</td></tr></tbody></table></div>@@ -472,18 +477,26 @@new text by a rank-weighted vote over its ten nearest neighbours. On Banking77 the neural arm settled 86.9%of traffic at confidence 0.8 or higher and was right on 0.979 of it, with no tokens spent; only the restneeds a judge model or a person. Spam detection reached 0.983 with plain BM25, under a millisecond per- message. The vote now also <strong>speaks TypeSafe's System One wire</strong> —+ message. AG News shows the scale end: 92% of 7,600 four-way topic decisions right against a 120,000-deep+ history, 34.2 ms per item. The vote now also <strong>speaks TypeSafe's System One wire</strong> —<code class="inline">POST /v1/systemone</code> on the native port — so the pip-installed<code class="inline">jev-reranker</code> runs against a local XERJ node unmodified, zero egress:<a href="/benchmarks/decisions">the write-up</a>.</li>+ <li><strong>The engine under it got 2.0× faster with bit-identical results.</strong> A full news item is a+ ~40-clause disjunction whose match set is most of the index, and the generic search path paid full+ materialisation for it. A WAND-style pruned walk (Broder et al., the same idea tantivy uses) now scores a+ document only when it can still make the page. 68.7 to 34.2 ms per item over all 7,600 AG News items, same+ labels, same confidence bands; the three BEIR BM25 figures reproduce to four decimals and the ES-YAML+ conformance suite stayed at 1371 passed / 0 failed.+ <a href="https://github.com/xerj-org/xerj/commit/cf6a9989" rel="external">Commit cf6a9989</a>.</li><li><strong>What it is not.</strong> It is not zero-shot: a new category has no history to vote from, and thatcase needs a judge model. XERJ did not run any judge model on these datasets and claims nothing about how- one would score. Two datasets, one run each.</li>+ one would score. Three datasets, one run each.</li></ul><p class="src">Scripts, raw output and reproduce commands:<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid" rel="external">benchmarks/beir-hybrid</a> ·<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval" rel="external">benchmarks/decisions-as-retrieval</a> ·- run 2026-09-18 on xerj v1.0.0-rc.74 · written up in+ run 2026-09-18 on xerj v1.0.0-rc.74 · AG News 2026-09-20 on the wire-compat branch engine (cf6a9989) · written up in<a href="/answers/hybrid-search-quality-measured-ndcg">hybrid search quality</a> and<a href="/answers/classify-text-from-labelled-history-without-llm">decisions without a model call</a>.Hosted-reranker figures: <a href="https://github.com/hev/jev-rerank" rel="external">hev/jev-rerank</a> (MIT) ·diff --git a/landing/llms.txt b/landing/llms.txtindex 67bf13b21..cf1c03bb1 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -42,7 +42,7 @@ Then, from an unknown folder to an answered question: `xerj autoindex <folder>`## Caveats (read before you rely on a capability)> - **kNN is HNSW-served (approximate) when unfiltered, exact brute-force otherwise.** An unfiltered `knn` on a full-precision cosine field with ≥1,024 docs runs a beam search over a persisted HNSW graph and exact-rescores its candidates, so `_score`s match the exact path bit-for-bit. `num_candidates` is the beam width (floored at 800; ES's 1.5×k default applies when omitted). On the 50k × 128-d bench corpus: recall@10 1.00 on the official bench query, 100-probe mean 0.976 / min 0.90 (ES 8.13.4, same protocol: 0.937 / 0.70). Recall on the ANN path is **measured, not guaranteed**, and it can return fewer than k hits. Filtered or nested kNN, non-cosine similarity, SQ8-quantized fields, indexes under 1,024 docs, a stale graph, and `semantic` / `_memory` recall all use the exact scan (recall 1.00 by construction; latency scales with vectors scanned).-> - **`rerank` is the only SEARCH-TIME feature that sends document text off the node — and it needs an operator-configured provider.** Adding `"rerank": {…}` to a `_search` body POSTs the question and the text of up to `window` hits (default 30, max 300) to a third-party relevance judge (TypeSafe AI's Jev), which returns a calibrated 0–1 probability per document; that probability replaces `_score` and `rerank.min_score` prunes by it. Two other features send text off the node, both operator configuration and off by default: `[embedding] default_endpoint` / `--embed-mode proxy` (document text at ingest and query text at search time to an external embeddings API) and the WAL tap (`PUT /_xerj/wal_tap`, every write on tapped indices to an external `_bulk` endpoint); a default node contacts neither, and its other outbound connections (the one-time neural model download, cluster Raft to your own peers) carry no document text. `rerank` is opt-in per request, inert until the node's operator sets a key (`[rerank] api_key` or `TYPESAFE_API_KEY` — you cannot pass one in a request), and `[rerank] enabled = false` forbids it. **Do not add `rerank` on data that must stay local unless the user has agreed.** Only fields the response returns are sent, so `_source` filtering also limits what leaves; `rerank.fields` is an exhaustive allow-list. Failure policy is *degrade on deadline, surface on contract*: a slow provider → HTTP 200 with the engine's order and `_rerank.applied:false` (**always read `_rerank.applied`**); no key → `503`, disabled → `403`, rejected key / malformed body / unreachable → `502`, all `rerank_exception` with **no hits** — do not retry, repeat the search without `rerank`. Refused (400), never silently ignored: with `sort`, `search_after`, `collapse`, scroll (open and continuation), `size:0`, and on `_msearch`, search templates, `_async_search`, the native `/v1` search API and gRPC. When applied, every `_score` is a probability or `null` (hits with no verdict, counted in `_rerank.unjudged`, sort last) — never a BM25 value beside probabilities. **Ranking quality IS measured — including this project's own integration defect, published**: with an evaluation key granted by TypeSafe, the stage as it first shipped scored **0.3822 nDCG@10 where BM25 alone scored 0.7750** on a 40-query SciFact pilot, because the request never bound "this document" to a document and the judge scored the whole pile once. Fixed by putting each candidate inside its own noul: **0.8299** on the pilot, and **0.7410 SciFact · 0.3312 NFCorpus** on the full run (300 + 323 queries, BM25 top-30 shortlists, three shuffled repeats, medians; the BM25 arms scored 0.6572 / 0.3016). Two measured cautions: the provider is non-deterministic (comparisons tighter than ~±0.01 nDCG@10 are noise), and the probabilities **order documents well but are NOT calibrated to BEIR relevance** (ECE 0.10–0.13; documents rated ~0.85 were relevant 42% of the time) — never threshold the raw probability with `min_score`. Reference: [docs/RERANK.md](https://github.com/xerj-org/xerj/blob/main/docs/RERANK.md) · write-up with runs and gate: [benchmarks/decisions](https://xerj.org/benchmarks/decisions).+> - **`rerank` is the only SEARCH-TIME feature that sends document text off the node — and it needs an operator-configured provider.** Adding `"rerank": {…}` to a `_search` body POSTs the question and the text of up to `window` hits (default 30, max 300) to a third-party relevance judge (TypeSafe AI's Jev), which returns a calibrated 0–1 probability per document; that probability replaces `_score` and `rerank.min_score` prunes by it. Two other features send text off the node, both operator configuration and off by default: `[embedding] default_endpoint` / `--embed-mode proxy` (document text at ingest and query text at search time to an external embeddings API) and the WAL tap (`PUT /_xerj/wal_tap`, every write on tapped indices to an external `_bulk` endpoint); a default node contacts neither, and its other outbound connections (the one-time neural model download, cluster Raft to your own peers) carry no document text. `rerank` is opt-in per request, inert until the node's operator sets a key (`[rerank] api_key` or `TYPESAFE_API_KEY` — you cannot pass one in a request), and `[rerank] enabled = false` forbids it. **Do not add `rerank` on data that must stay local unless the user has agreed.** Only fields the response returns are sent, so `_source` filtering also limits what leaves; `rerank.fields` is an exhaustive allow-list. Failure policy is *degrade on deadline, surface on contract*: a slow provider → HTTP 200 with the engine's order and `_rerank.applied:false` (**always read `_rerank.applied`**); no key → `503`, disabled → `403`, rejected key / malformed body / unreachable → `502`, all `rerank_exception` with **no hits** — do not retry, repeat the search without `rerank`. Refused (400), never silently ignored: with `sort`, `search_after`, `collapse`, scroll (open and continuation), `size:0`, and on `_msearch`, search templates, `_async_search`, the native `/v1` search API and gRPC. When applied, every `_score` is a probability or `null` (hits with no verdict, counted in `_rerank.unjudged`, sort last) — never a BM25 value beside probabilities. **Ranking quality IS measured — including this project's own integration defect, published**: with an evaluation key granted by TypeSafe, the stage as it first shipped scored **0.3822 nDCG@10 where BM25 alone scored 0.7750** on a 40-query SciFact pilot, because the request never bound "this document" to a document and the judge scored the whole pile once. Fixed by putting each candidate inside its own noul: **0.8299** on the pilot, and **0.7410 SciFact · 0.3312 NFCorpus · 0.3638 FiQA** on the full run (1,271 judged queries across three public BEIR datasets, BM25 top-30 shortlists, three shuffled repeats, medians; the BM25 arms scored 0.6572 / 0.3016 / 0.2382 — FiQA against BEIR's published 0.236, which validates the harness). Two measured cautions: the provider is non-deterministic (comparisons tighter than ~±0.01 nDCG@10 are noise), and the probabilities **order documents well but are NOT calibrated to BEIR relevance** (ECE 0.10–0.31; worst on FiQA — documents rated ~0.93 confidence were relevant 34% of the time) — never threshold the raw probability with `min_score`. Reference: [docs/RERANK.md](https://github.com/xerj-org/xerj/blob/main/docs/RERANK.md) · write-up with runs and gate: [benchmarks/decisions](https://xerj.org/benchmarks/decisions).> - **Hybrid fusion is `rrf` or `linear` ONLY.** `fusion:"learned"` is NOT implemented and fails loudly — `400`, `hybrid fusion learned is not yet supported; use rrf or linear`. Do not depend on it.> - **The DEFAULT embedding mode is lexical, not neural.** With no extra flags, semantic search and memory recall use a deterministic feature-hashing embedder (384-d, L2-normalised → cosine): hybrid lexical+vector retrieval — word / sub-word overlap, not deep semantic understanding or synonym-level recall. For real neural semantics start the server with `--embed-mode neural` — a built-in in-process BERT encoder (all-MiniLM-L6-v2) that ships in the binary and auto-downloads its model (~90 MB) on first use — or `--embed-mode proxy` (any external OpenAI-compatible `/v1/embeddings`). Only describe output as neural when one of those is actually running.> * **`--embed-mode` is a server-start flag that applies AT INGEST.** Text is embedded as it is indexed, so the mode in force when a document was written is the mode that document keeps. Switching lexical → neural does **not** re-embed anything already indexed: you have to re-index every corpus you care about. **Decide before you index your first corpus.**@@ -72,7 +72,7 @@ Then, from an unknown folder to an answered question: `xerj autoindex <folder>`- `POST /_bulk` · `POST /{index}/_bulk` (NDJSON: action line + doc line).- `POST|GET /{index}/_search` — full query DSL, `semantic`, `knn`, `hybrid`, top-level `knn`, `aggs`, `sort`, `_source`, `highlight`, and the optional `rerank` second stage (see Caveats: sends text off the machine, needs a provider key) · `POST /_search` (all indices) · `POST /_msearch`.- `GET /_xerj/rerank` — is a rerank provider configured? Reports that a key is set and whether it came from `config` or `env`, never the key. Superuser only; without superuser rights a `rerank` search tells you the same thing (`503` no key, `403` disabled).-- `POST /v1/systemone` (native REST port, `--port + 1`) · `POST /_decide` (ES-compat port) — typed judgement (`noul`: 0–1 probability; `choice`: per-option probabilities) answered locally by a rank-weighted vote over a labelled-history index; nothing leaves the node. Needs `[decisions] index` in config (503 until set). `/v1/systemone` speaks the documented System One wire — `pip install jev-reranker` runs unmodified against it with `TYPESAFE_ENDPOINT=http://localhost:<port+1>/v1/systemone` (the full URL incl. path; that client ignores `TYPESAFE_BASE_URL`). Two deliberate breaks from the hosted API: `model` echoes `xerj-history-vote-1`, never a Jev model name; a question with zero labelled support is a 422 naming the ids, never a fabricated 0.5. `/_decide` returns the evidence — neighbours with labels, engine scores, weights. Reference: [docs/DECISIONS.md](https://github.com/xerj-org/xerj/blob/main/docs/DECISIONS.md).+- `POST /v1/systemone` (native REST port, `--port + 1`) · `POST /_decide` (ES-compat port) — typed judgement (`noul`: 0–1 probability; `choice`: per-option probabilities) answered locally by a rank-weighted vote over a labelled-history index; nothing leaves the node. Needs `[decisions] index` in config (503 until set). `/v1/systemone` speaks the documented System One wire — `pip install jev-reranker` runs unmodified against it with `TYPESAFE_ENDPOINT=http://localhost:<port+1>/v1/systemone` (the full URL incl. path; that client ignores `TYPESAFE_BASE_URL`). Two deliberate breaks from the hosted API: `model` echoes `xerj-history-vote-1`, never a Jev model name; a question with zero labelled support is a 422 naming the ids, never a fabricated 0.5. `/_decide` returns the evidence — neighbours with labels, engine scores, weights. Measured on 120k-item AG News history: 0.9182 accuracy over all 7,600 test items, p50 42.7 ms per item on `/v1/systemone`. Reference: [docs/DECISIONS.md](https://github.com/xerj-org/xerj/blob/main/docs/DECISIONS.md).- `POST|GET /{index}/_count` · `POST /{index}/_delete_by_query` · `POST /{index}/_update_by_query` · `POST /_mget` · `POST /{index}/_mget`.- `POST /_memory/{namespace}` (also `GET` list, `DELETE` namespace) · `POST /_memory/{namespace}/_recall` · `DELETE /_memory/{namespace}/{id}`.- `POST /autoindex-catalog/_search` — the self-describing catalog written by `xerj autoindex`. CLI equivalents: `xerj autoindex map [--json]`, `xerj autoindex status`.diff --git a/scripts/seo/pagedata.py b/scripts/seo/pagedata.pyindex fe9fd8efb..691646a04 100755--- a/scripts/seo/pagedata.py+++ b/scripts/seo/pagedata.py@@ -82,8 +82,8 @@ PAGES: dict[str, dict[str, str]] = {),"benchmarks/decisions.html": dict(label="Jev measured, wire spoken", kind="software",- title="XERJ.ai — Measuring the Jev judge, then speaking the System One wire",- description="A rerank stage that measured 0.3822 nDCG@10, was root-caused to one request shape and fixed to 0.8299; a 300-query controlled run of Jev on XERJ shortlists; and a XERJ node that answers /v1/systemone locally — the pip-installed jev-reranker runs against it unmodified.",+ title="XERJ.ai — We measured the Jev judge. Then we answered its wire.",+ description="We broke our own Jev rerank stage in public (0.3822 nDCG@10), fixed it, then ran 1,271 judged queries on three public datasets. Then we made XERJ answer the System One wire locally. The pip client runs against it unmodified. Nothing leaves your machine.",),"demo/index.html": dict(label="Real-data demo", kind="software",