AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(api): /v1/systemone votes on the payload, never the instruction prose
xerj-orgfed0fffa012a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 597b14f1e..f2ac79ef0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Fixed++- **`/v1/systemone` votes on the payload, never on the instruction prose.**+ A question's `instructions` used to be concatenated into the retrieval+ query, so a well-written instruction destroyed accuracy — measured on the+ SMS history (300 held-out, seed 7): 0.9867 empty vs 0.6700 criteria-rich, a+ 32-point swing from wording alone (#1000). And an object `state`+ contributed only its `query` field, so `{"message": …}` voted on the+ instruction at the spam base rate: accuracy 0.1733 vs 0.9667 for the same+ message as a string (#1001). The vote text is now exactly what the question+ points at: its backtick references (instructions data fields first, then+ state paths — both client shapes keep working), or, when it references+ nothing, the state's own string content (a string whole, an object by all+ string leaves). A reference that resolves to structure rather than text —+ the rubric object `jev-reranker` ≥ 0.1.2 ships with its relevance preset —+ is acknowledged and skipped, not embedded and not fatal. References that+ resolve to nowhere and payloads with no text are 422s naming the question.+ Also fixed the `no_support` error body, which interpolated `k` and the+ question ids into each other's slots.+ Fail-before: five new `systemone_http` tests, all failing on the old code+ (`[0.0, 0.0, 0.4545, 0.0]` across four instruction wordings of one state);+ a sixth pins the client's rubric shape, a 422 before the skip rule. After:+ the 300 held-out messages score 0.9767 identically across all eight+ wording/shape arms, and the pip-client acceptance gate passes through both+ client entry points (neutral-query gap 0.8542) and under a deliberately+ spammy query (gap 0.6087, threshold 0.3) — client 0.1.2, unmodified.+## [1.0.0-rc.76] - 2026-09-21The JEV-interface and honest-fixes release. The headline is a measured answerdiff --git a/benchmarks/systemone-gate/README.md b/benchmarks/systemone-gate/README.mdindex 224017095..58b056726 100644--- a/benchmarks/systemone-gate/README.md+++ b/benchmarks/systemone-gate/README.md@@ -1,15 +1,15 @@# The /v1/systemone acceptance gate — jev-reranker, unmodified, on a XERJ node-Run 2026-09-20, locally built `xerj-server` at the wire-compat commit,-`--insecure --port 9440` (native REST on 9441), throwaway data dir,-`[decisions] index = "sms"` (`gate-decisions.toml`: k=10, text/label fields,-`positive_label = "spam"`). Client: `pip install jev-reranker` (0.1.0, MIT) in a-clean venv — no patches, no subclassing, its own listwise instruction template,-its own splitting, its own strict response validation. The only configuration:+Run 2026-09-21, locally built `xerj-server` at the vote-text fix (#1000/#1001),+`--port 9490` (native REST on 9491), throwaway data dir, `[decisions]+index = "sms"` (`gate-decisions.toml`: k=10, text/label fields,+`positive_label = "spam"`). Client: `pip install jev-reranker` (0.1.2, MIT) —+no patches, no subclassing, its own listwise instruction template, its own+splitting, its own strict response validation. The only configuration:```sh-TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone-TYPESAFE_API_KEY=<any non-empty string> # the node ignores it; localhost only+TYPESAFE_ENDPOINT=http://localhost:9491/v1/systemone+TYPESAFE_API_KEY=<the node's admin key>````TYPESAFE_BASE_URL` is silently ignored by the client — the endpoint variable@@ -20,13 +20,25 @@ must carry the full path. (Verified in the wheel: `reranker.py:238`.)`gate_load_sms.py` indexes the 4,000-message SMS train split(`benchmarks/decisions-as-retrieval`'s split: shuffle seed 7, first 4,000) andfixes 10 held-out gate documents (5 spam, 5 ham, 40–140 chars) in-`gate-docs.json`. `gate_run.py` ranks them with `JevReranker().rerank(...)`.+`gate-docs.json`. `gate_run.py` ranks them through BOTH client entry points:++1. the constructor default (listwise, compact instruction — references+ `` `documents.doc_N` `` and `` `query` `` in backticks);+2. `relevance_rerank()` — the documented relevance preset, which since client+ 0.1.2 ships its rubric OBJECT in state and references it in backticks. The+ node acknowledges the rubric without embedding its prose in the vote+ (#1000's defect class); the two phases score identically, which is the+ proof.+3. the constructor default again, under a deliberately spammy query.+The query is `triage inbox unsolicited correspondence` — every term verified-absent from the corpus, because `state.query` joins every vote text and a-vocabulary-bearing query ("winner claim prize") retrieves spam neighbours for-EVERY document, which measures the query, not the documents. First gate-attempt failed exactly there (spam 1.000 vs ham 0.785); the transcript below-is the fixed run.+absent from the corpus. Since client 0.1.2 the question itself references+`` `query` `` in backticks, so the query is named payload and joins the vote+BY CLIENT DESIGN (the question asked is "does this document answer this+query"). For a classification history that is noise, which is why the gate+query is neutral — and phase 3 proves the separation survives a spammy query+("winner claim free prize congratulations") that the pre-#1000 node would+have collapsed on (the first gate attempt then: spam 1.000 vs ham 0.785).## Result (`gate-transcript.txt`)@@ -34,9 +46,10 @@ The client's own validation IS the wire gate: answers keyed exactly by thesent question ids, every answer `{type: "noul", noul}` numeric in 0..1,`model` a non-empty string (it resolved `xerj-history-vote-1` — the node'struthful echo, never a Jev name), `usage` non-negative integers. The semantic-gate: mean noul 0.9014 over spam vs 0.0956 over ham (gap 0.806, threshold-0.3). No request left the node: the vote is an ordinary search of the `sms`-index; the module adds no outbound client.+gate, threshold 0.3: neutral query gap **0.8542** (spam mean 0.9447 vs ham+0.0905, identical across both entry points), spammy query gap **0.6087**. No+request left the node: the vote is an ordinary search of the `sms` index;+the module adds no outbound client.## The response the client does not show@@ -44,12 +57,14 @@ index; the module adds no outbound client.engine scores, weights, and an abstain verdict below `decisions.min_confidence`:```sh-curl -s localhost:9440/_decide -H 'content-type: application/json' \+curl -s localhost:9490/_decide -H 'content-type: application/json' \+ -H "authorization: ApiKey $(cat <data_dir>/admin.key)" \-d '{"index":"sms","question":"URGENT! Your Mobile number has been awarded"}'# {"index":"sms","k":10,"label":"spam","confidence":1.0,"abstain":false,# "neighbours":[{"_id":"1805","label":"spam",...,"weight":1.0}, ...]}```Zero-support questions are a 422 naming the question ids, never a fabricated-0.5; `score` questions (no vote analogue) are a 422 naming the alternative.-Both behaviours are pinned by `engine/crates/xerj-api/tests/systemone_http.rs`.+0.5; `score` questions (no vote analogue) are a 422 naming the alternative;+a backtick reference that resolves to nothing is a 422 naming the path. All+pinned by `engine/crates/xerj-api/tests/systemone_http.rs`.diff --git a/benchmarks/systemone-gate/gate-transcript.txt b/benchmarks/systemone-gate/gate-transcript.txtindex 77c11a2c7..8a02f9e0a 100644--- a/benchmarks/systemone-gate/gate-transcript.txt+++ b/benchmarks/systemone-gate/gate-transcript.txt@@ -1,13 +1,39 @@+client: jev-reranker 0.1.2 (pip, unmodified)client resolved model: ['xerj-history-vote-1']- 0.5068 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600+-- constructor default (listwise), neutral query+ 0.7236 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 886001.0000 spam Please call our customer service representative on 0800 169 6031 betwe1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT- 0.1024 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca+ 0.1110 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca0.0000 ham If you're not in my car in an hour and a half I'm going apeshit0.3414 ham Awesome question with a cute answer: Someone asked a boy "how is ur li0.0000 ham Carlos says we can pick up from him later so yeah we're set- 0.0341 ham The greatest test of courage on earth is to bear defeat without losing-mean noul spam=0.9014 ham=0.0956 gap=0.8058+ 0.0000 ham The greatest test of courage on earth is to bear defeat without losing+ mean noul spam=0.9447 ham=0.0905 gap=0.8542+-- rerank_relevance() preset (rubric object), neutral query+ 0.7236 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600+ 1.0000 spam Please call our customer service representative on 0800 169 6031 betwe+ 1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C+ 1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.+ 1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT+ 0.1110 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca+ 0.0000 ham If you're not in my car in an hour and a half I'm going apeshit+ 0.3414 ham Awesome question with a cute answer: Someone asked a boy "how is ur li+ 0.0000 ham Carlos says we can pick up from him later so yeah we're set+ 0.0000 ham The greatest test of courage on earth is to bear defeat without losing+ mean noul spam=0.9447 ham=0.0905 gap=0.8542+-- constructor default, deliberately spammy query+ 1.0000 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600+ 1.0000 spam Please call our customer service representative on 0800 169 6031 betwe+ 1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C+ 1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.+ 1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT+ 0.2887 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca+ 0.3414 ham If you're not in my car in an hour and a half I'm going apeshit+ 0.7174 ham Awesome question with a cute answer: Someone asked a boy "how is ur li+ 0.1707 ham Carlos says we can pick up from him later so yeah we're set+ 0.4382 ham The greatest test of courage on earth is to bear defeat without losing+ mean noul spam=1.0000 ham=0.3913 gap=0.6087GATE PASSED: unmodified jev-reranker ranked from the XERJ nodediff --git a/benchmarks/systemone-gate/gate_run.py b/benchmarks/systemone-gate/gate_run.pyindex d9e29812a..d3b1152f8 100644--- a/benchmarks/systemone-gate/gate_run.py+++ b/benchmarks/systemone-gate/gate_run.py@@ -2,10 +2,10 @@XERJ node.TYPESAFE_ENDPOINT=http://localhost:<native-rest-port>/v1/systemone- TYPESAFE_API_KEY=<any non-empty string — the node ignores it, localhost only>+ TYPESAFE_API_KEY=<the node's admin key>-Ranks the fixed gate documents (gate-docs.json: 5 spam, 5 ham) with the-client's own listwise pipeline and asserts two things:+Ranks the fixed gate documents (gate-docs.json: 5 spam, 5 ham) through BOTH+client entry points and asserts two things per phase:1. WIRE: the call completes with no ResponseValidationError — answers keyedexactly, noul numeric in 0..1, model echoed as a non-empty string, usage@@ -13,39 +13,66 @@ client's own listwise pipeline and asserts two things:2. SEMANTICS: the local vote separates the classes — mean noul over spamdocuments strictly exceeds mean noul over ham documents by >= 0.3.-Run: python3 gate_run.py [native_url_base]+Phase 1 is the constructor default (listwise, compact instruction). Phase 2+is `rerank_relevance()` — the documented relevance preset, which since client+0.1.2 ships its rubric OBJECT in state and references it in backticks; the+node must acknowledge the rubric without embedding its prose in the vote.++The query stays neutral (`triage inbox unsolicited correspondence`, every+term absent from the corpus): the client's own question references `query`+in backticks, so the query is named payload and joins the vote by design.+For a classification history that adds noise — which is why the query is+neutral, and why phase 3 asserts survival of a deliberately spammy query+("winner claim free prize congratulations") under the same threshold.++Run: python3 gate_run.py"""-import json, os, sys+import json+from importlib.metadata import version++import jev_reranker+from jev_reranker import JevRerankerdocs = json.load(open("gate-docs.json"))texts = [d["text"] for d in docs]labels = [d["label"] for d in docs]-# The query rides in state.query and joins every vote text, so its terms must-# be NEUTRAL for the history: a spammy query ("winner claim prize") retrieves-# spam neighbours for every document and the vote says nothing about the doc.-# Every term here is absent from the SMS corpus (checked against sms.tsv), so-# the document drives the vote. That asymmetry — query vocabulary pollutes a-# classification history — is documented in docs/RERANK.md.QUERY = "triage inbox unsolicited correspondence"+SPAMMY = "winner claim free prize congratulations"-from jev_reranker import JevReranker-rr = JevReranker() # reads TYPESAFE_ENDPOINT / TYPESAFE_API_KEY from the env-result = rr.rerank(QUERY, texts, detail=True)+def phase(name, scores):+ print(f"-- {name}")+ for lab, txt, s in zip(labels, texts, scores):+ print(f" {s:.4f} {lab:4} {txt[:70]}")+ assert len(scores) == len(texts)+ assert all(isinstance(s, float) and 0.0 <= s <= 1.0 for s in scores), scores+ spam = [s for s, l in zip(scores, labels) if l == "spam"]+ ham = [s for s, l in zip(scores, labels) if l == "ham"]+ ms, mh = sum(spam) / len(spam), sum(ham) / len(ham)+ print(f" mean noul spam={ms:.4f} ham={mh:.4f} gap={ms - mh:.4f}")+ assert ms - mh >= 0.3, f"vote did not separate: {ms:.3f} vs {mh:.3f}"+-# {"results": [{"document_index", "score", "text", ...}, ...]} — order follows-# the client's ranking; re-key by the document index it reports.-by_index = {r["document_index"]: r["score"] for r in result["results"]}-scores = [by_index[i] for i in range(len(texts))]+def rank(query):+ result = rr.rerank(query, texts, detail=True)+ by_index = {r["document_index"]: r["score"] for r in result["results"]}+ return [by_index[i] for i in range(len(texts))], result+++print(f"client: jev-reranker {version('jev-reranker')} (pip, unmodified)")++rr = JevReranker() # reads TYPESAFE_ENDPOINT / TYPESAFE_API_KEY from the env+scores, result = rank(QUERY)print("client resolved model:", result["detail"]["resolved_models"])-for lab, txt, s in zip(labels, texts, scores):- print(f" {s:.4f} {lab:4} {txt[:70]}")--assert len(scores) == len(texts)-assert all(isinstance(s, float) and 0.0 <= s <= 1.0 for s in scores), scores-spam = [s for s, l in zip(scores, labels) if l == "spam"]-ham = [s for s, l in zip(scores, labels) if l == "ham"]-ms, mh = sum(spam) / len(spam), sum(ham) / len(ham)-print(f"mean noul spam={ms:.4f} ham={mh:.4f} gap={ms - mh:.4f}")-assert ms - mh >= 0.3, f"vote did not separate: {ms:.3f} vs {mh:.3f}"+phase("constructor default (listwise), neutral query", scores)++# The documented preset: rubric object in state, referenced in backticks.+relevance = rr.relevance_rerank(QUERY, texts, threshold=0.0, return_documents=False)+scores = [s for _, s in sorted(+ (r["document_index"], r["score"]) for r in relevance["results"])]+phase("rerank_relevance() preset (rubric object), neutral query", scores)++scores, _ = rank(SPAMMY)+phase("constructor default, deliberately spammy query", scores)+print("GATE PASSED: unmodified jev-reranker ranked from the XERJ node")diff --git a/docs/DECISIONS.md b/docs/DECISIONS.mdindex 7735c069b..bce4fd91c 100644--- a/docs/DECISIONS.md+++ b/docs/DECISIONS.md@@ -62,26 +62,48 @@ wrong one when you do not. Where there is no history — a new policy, a newcategory — use a judge model; that is what[docs/RERANK.md](./RERANK.md) is for.-**The query's vocabulary pollutes a classification history.** The wire puts-the query in `state`, and the vote text is the resolved question text plus-that query — so query terms participate in retrieval. For a rerank-shaped-request that is correct (relevance IS query-relative). For a classification-history it can swamp the document: against the SMS corpus, the query-"congratulations winner claim free prize" retrieves spam neighbours for *every*-document and the vote says nothing about the document (measured: spam mean-1.000 vs ham 0.785 — no separation). Keep the query neutral for the corpus-(every term of `triage inbox unsolicited correspondence` is absent from the SMS-corpus: spam 0.901 vs ham 0.096). This is inherent to a retrieval vote, not a-defect to fix: pick the query for the history you have.+**What the vote retrieves on — the payload, never the prose (#1000, #1001).**+A question's `instructions` are a question ABOUT the payload; their vocabulary+is not retrieval text. The vote text is exactly what the question points at:++- every `` `path` `` reference the instructions name — resolved against the+ instructions' own string fields first (the data-field pattern:+ `{"question": "Judge `document` …", "document": text}`, which+ `xerj-rerank`'s stage and the provider's docs use), then against the state+ by dotted path (`` `documents.doc_0` ``, the shape `jev-reranker` sends).+ A reference that resolves to structure rather than text — the rubric+ object `jev-reranker` ≥ 0.1.2 ships with its relevance preset — is+ acknowledged and skipped: judge rules are not retrieval vocabulary;+- a question that references nothing votes on the state alone — a string+ state whole, an object state by ALL its string leaves, whatever the keys+ are named;+- references that resolve to nothing, and a payload with no text at all, are+ 422s naming the question — never a confident vote on text the question+ never saw.++Two defects this design closes, both measured on the SMS history (4,000+train / 300 held-out, shuffle seed 7): instruction wording used to change the+answer — accuracy 0.9867 with an empty instruction against 0.6700 with a+criteria-rich one, a 32-point swing from wording alone — and an object state+used to contribute only its `query` field, so `{"message": …}` voted on the+instruction prose at the spam base rate (accuracy 0.1733 against 0.9667 for+the same message as a string). After the fix the same 300 held-out messages+score identically under every instruction wording and every state shape —+0.9767 across all eight arms (the absolute number sits ~1 point under the+issue runs' 0.9867 because the history was re-indexed fresh for the re-run;+the invariant is the identity, not the digit). One nuance the client itself+decides: since 0.1.2, `jev-reranker`'s own question references `` `query` ``+in backticks, so the query is named payload and joins the vote BY CLIENT+DESIGN — the acceptance gate still uses a neutral query for a classification+history, and survives a deliberately spammy one (gap 0.61 against 0.85,+threshold 0.3).## The wire, and the two deliberate breaksRequest: `{state, model, questions: {id: {type, instructions, criteria}}}`.`state` may be a string or an object; `instructions` may be a string, object-or array (all string leaves join); strings may reference state data with-backtick paths — `` `documents.doc_0` `` — resolved against `state` before the-vote text is built. That is the reference pattern of the API and the shape-`jev-reranker` sends.+or array. `criteria` descriptions are never retrieval text; the options they+name are `choice` answers, not query vocabulary.Response: `{model, answers, usage}` with `answers` keyed exactly by thequestion ids sent, each answer carrying only its documented fields (clientsdiff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rsindex 1e6ab5a7f..788a8d24b 100644--- a/engine/crates/xerj-api/src/systemone_api.rs+++ b/engine/crates/xerj-api/src/systemone_api.rs@@ -21,11 +21,20 @@//!//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.//! `state` may be a string or an object; question `instructions` may be a-//! string, object or array (all string leaves are joined). Strings may point-//! at state data with backtick paths — `` `documents.doc_0` `` — which are-//! resolved against `state` before the vote text is built. That is the-//! reference pattern of the System One API and the shape `jev-reranker`-//! sends in its listwise mode.+//! string, object or array. **The vote text is payload data, never+//! instruction prose (#1000, #1001).** A question votes on exactly what it+//! points at: every `` `path` `` reference its instructions name — resolved+//! against the instructions' own string fields first (the data-field+//! pattern: `{"question": "Judge `document` …", "document": text}`) and then+//! against the state by dotted path (`` `documents.doc_0` ``, the shape+//! `jev-reranker` sends) — or, when it references nothing, on the state+//! alone: a string state whole, an object state by all its string leaves.+//! A reference that resolves to structure rather than text (the rubric+//! object `jev-reranker` ≥ 0.1.2 ships with its relevance preset) is+//! acknowledged and skipped: judge rules are not retrieval vocabulary.+//! References that resolve to nothing, and a payload with no text at all,+//! are 422s naming the question — never a confident vote on text the+//! question never saw.//!//! Response: `{ model, answers, usage }` with `answers` keyed exactly by the//! question ids sent and each answer carrying only its documented fields —@@ -389,36 +398,90 @@ fn parse_question(id: &str, q: &Value, state_val: &Value) -> Result<Question, St))}};- // `instructions` is polymorphic on this wire: string, object or array.- // Every string leaf joins the vote text — an object's fields arrive in- // serde_json's BTreeMap order, which is stable but arbitrary; the text is- // retrieved by content, not by field order.- let mut text = String::new();- if let Some(instructions) = obj.get("instructions") {- join_strings(instructions, &mut text);+ // The vote text is RETRIEVAL TEXT, and retrieval text must be payload+ // data, never instruction prose (#1000). An instruction is a question+ // ABOUT the payload; its vocabulary retrieves neighbours on its own —+ // measured 0.9867 accuracy with an empty instruction against 0.6700 with+ // a criteria-rich one, the same 300 messages. So a question votes on+ // exactly what it POINTS AT:+ //+ // - every `` `path` `` reference its instructions name, resolved first+ // against the instructions' own string fields (the data-field pattern:+ // `{"question": "Judge `document` …", "document": text}`, which+ // xerj-rerank's stage and the provider's docs use) and then against+ // the state (the jev-reranker pattern: documents in state, referenced+ // by dotted path). The state does NOT ride along here — a question+ // that named its payload gets its payload, not its payload plus+ // whatever else the state holds;+ // - a reference that resolves to STRUCTURE, not text, contributes no+ // vote text. `jev-reranker` ≥ 0.1.2's `rerank_relevance()` preset+ // ships its rubric object in state and names it in backticks; the+ // rubric is judge rules, and embedding its string leaves is exactly+ // the #1000 defect (criteria-rich prose, the measured 0.6700 class).+ // Structure is acknowledged and skipped — text is text, rules are+ // rules;+ // - a question that references nothing votes on the state alone: a+ // string state whole, an object state by ALL its string leaves (#1001:+ // only `query` used to be read, so `{"message": …}` answered from the+ // instruction alone, at the spam base rate);+ // - a question whose references resolve NOWHERE is refused naming them —+ // a confident vote on text the question never saw is the silent-fake+ // defect class, in another coat;+ // - a vote text that is still empty is refused naming the question.+ //+ // `criteria` descriptions are never embedded: no benchmark measures+ // criteria text, and it is the richest vocabulary on the wire.+ let instructions = obj.get("instructions");+ let mut paths: Vec<String> = Vec::new();+ if let Some(instr) = instructions {+ collect_backtick_paths(instr, &mut paths);}- // `criteria` descriptions are NOT embedded: the vote is over text the- // history was labelled on, and no benchmark measures criteria text.- let resolved = resolve_backticks(&text, state_val);- // The query rides along from state — a plain-string state whole, an- // object state via its `query` field, exactly the two shapes the wire's- // own examples use. Other state fields are referenced (or not) by the- // instructions' backtick paths, which have already been resolved.- let mut vote_text = resolved;- match state_val {- Value::String(s) => {- vote_text.push(' ');- vote_text.push_str(s);- }- Value::Object(m) => {- if let Some(q) = m.get("query").and_then(Value::as_str) {- vote_text.push(' ');- vote_text.push_str(q);+ let mut vote_text = String::new();+ if paths.is_empty() {+ join_strings(state_val, &mut vote_text);+ } else {+ let mut unresolved: Vec<String> = Vec::new();+ for path in &paths {+ match resolve_reference(instructions, state_val, path) {+ Some(Value::String(text)) => {+ if !vote_text.is_empty() {+ vote_text.push(' ');+ }+ vote_text.push_str(text);+ }+ // Resolves, but to structure — an object, number, boolean.+ // Rules and figures carry no retrieval vocabulary: skipping+ // them is the difference between this and #1000, where the+ // prose joined the vote and wording moved accuracy 32 points.+ Some(_) => {}+ None => unresolved.push(path.clone()),}}- _ => {}+ if !unresolved.is_empty() {+ let listed = unresolved+ .iter()+ .map(|p| format!("`{p}`"))+ .collect::<Vec<_>>()+ .join(", ");+ return Err(format!(+ "question `{id}`: backtick reference{} {listed} resolve{} to nothing in \+ this request — the vote would run on text the question never named; \+ reference existing instructions fields or state paths, or drop the \+ backticks",+ if unresolved.len() == 1 { "" } else { "s" },+ if unresolved.len() == 1 { "s" } else { "" },+ ));+ }}let vote_text = clip(vote_text.trim(), MAX_VOTE_TEXT_CHARS);+ if vote_text.is_empty() {+ return Err(format!(+ "question `{id}`: no text to vote on — the payload resolved to nothing; put \+ text in `state` (a string, or an object with string fields) or reference it \+ in backticks from the instructions, remembering that only string-valued \+ references carry text"+ ));+ }Ok(Question { vote_text, kind })}@@ -482,43 +545,56 @@ fn join_strings(v: &Value, out: &mut String) {}}-/// Resolve `` `dotted.path` `` references against `state`. An unresolvable-/// path is left literally in place — visible in the vote text rather than-/// silently dropped.-fn resolve_backticks(text: &str, state: &Value) -> String {- if !text.contains('`') || state.is_null() {- return text.to_string();- }- let mut out = String::with_capacity(text.len());- let mut rest = text;- while let Some(start) = rest.find('`') {- out.push_str(&rest[..start]);- let after = &rest[start + 1..];- match after.find('`') {- Some(end) => {- let path = &after[..end];- match walk(state, path) {- Some(v) => out.push_str(v),- None => out.push_str(&rest[start..start + end + 2]),+/// Collect every `` `path` `` a question's instructions name, in order of+/// first appearance, deduplicated. The prose between backticks is ignored —+/// it is the question, not the payload (#1000). An unpaired backtick names+/// nothing.+fn collect_backtick_paths(v: &Value, out: &mut Vec<String>) {+ match v {+ Value::String(s) => {+ let mut rest = s.as_str();+ while let Some(start) = rest.find('`') {+ let after = &rest[start + 1..];+ match after.find('`') {+ Some(end) => {+ let path = &after[..end];+ if !path.is_empty() && !out.iter().any(|p| p == path) {+ out.push(path.to_string());+ }+ rest = &after[end + 1..];+ }+ None => break,}- rest = &after[end + 1..];- }- None => {- out.push_str(&rest[start..]);- return out;}}+ Value::Array(a) => a.iter().for_each(|x| collect_backtick_paths(x, out)),+ Value::Object(m) => m.values().for_each(|x| collect_backtick_paths(x, out)),+ _ => {}}- out.push_str(rest);- out}-fn walk<'a>(v: &'a Value, path: &str) -> Option<&'a str> {+/// Resolve one backtick reference to the value it names. The instructions'+/// own string fields win over same-named state paths — the data-field+/// pattern puts the payload next to the prose that names it — then the+/// state is walked by dotted path. `None` means the reference names+/// nothing in this request; a caller decides what a non-string resolution+/// means (here: structure, not vote text).+fn resolve_reference<'a>(+ instructions: Option<&'a Value>,+ state: &'a Value,+ path: &str,+) -> Option<&'a Value> {+ instructions+ .and_then(|i| walk(i, path))+ .or_else(|| walk(state, path))+}++fn walk<'a>(v: &'a Value, path: &str) -> Option<&'a Value> {let mut cur = v;for seg in path.split('.') {cur = cur.get(seg)?;}- cur.as_str()+ Some(cur)}fn clip(s: &str, max: usize) -> String {@@ -580,12 +656,12 @@ fn no_support(ids: &[String], cfg: &DecisionsConfig) -> axum::response::Responseerror, not a fabricated probability — add labelled examples to the history \index or use POST /_decide to inspect what is there",cfg.index,- cfg.k,if ids.len() == 1 { "" } else { "s" },ids.iter().map(|i| format!("`{i}`")).collect::<Vec<_>>()- .join(", ")+ .join(", "),+ cfg.k,),)}diff --git a/engine/crates/xerj-api/tests/systemone_http.rs b/engine/crates/xerj-api/tests/systemone_http.rsindex 734ad0f91..d2046b1c3 100644--- a/engine/crates/xerj-api/tests/systemone_http.rs+++ b/engine/crates/xerj-api/tests/systemone_http.rs@@ -263,6 +263,180 @@ async fn a_plain_string_state_with_object_instructions_also_votes() {assert_eq!(r["answers"]["q1"]["noul"], json!(1.0));}+/// #1000: instruction wording must not change the answer. The instruction is+/// a question ABOUT the payload; its vocabulary is not retrieval text. The+/// issue measured a 32-point accuracy swing on a real SMS history (0.9867+/// empty instruction vs 0.6700 criteria-rich) purely from wording — here the+/// same state is asked three times with three different instructions and all+/// three answers must be identical.+#[tokio::test]+async fn instruction_wording_does_not_change_the_answer() {+ let node = gate_node().await;+ // A ham-shaped state: its own terms retrieve the `false` doc and nothing+ // else, whatever the prose says.+ let state = json!("shipping delivery times");+ let rich = "Is this SMS message spam? prize claims, paid subscriptions, \+ marketing, premium-rate numbers, refund, money back";+ let mut answers = Vec::new();+ for instructions in [+ None,+ Some(json!("Is this message spam?")),+ Some(json!(rich)),+ // the criteria-block shape a Jev migrant sends+ Some(json!({"question": "Is this message spam?", "criteria": {+ "spam": "prize claims, paid subscriptions, marketing",+ "ham": "plain personal correspondence"}})),+ ] {+ let mut q = serde_json::Map::new();+ q.insert("type".into(), json!("noul"));+ if let Some(i) = instructions {+ q.insert("instructions".into(), i);+ }+ let (st, r) = node+ .systemone(json!({+ "state": state,+ "questions": {"d0": Value::Object(q)}+ }))+ .await;+ assert_eq!(st, StatusCode::OK, "{r}");+ answers.push(r["answers"]["d0"]["noul"].as_f64().unwrap_or(-1.0));+ }+ // All four answers identical, and the vote followed the STATE: a+ // shipping text over this history is `false` outright.+ assert!(+ answers.iter().all(|a| (a - answers[0]).abs() < 1e-9),+ "instruction wording changed the answer: {answers:?}"+ );+ assert!(+ (answers[0] - 0.0).abs() < 1e-9,+ "the state's own vote is `false`, got {answers:?}"+ );+}++/// #1001: an object state votes on ALL its string leaves. The issue measured+/// accuracy 0.1733 — exactly the spam base rate — for `{"message": …}` (every+/// item scored from identical instruction text) against 0.9667 for the same+/// message as a string or as `{"query": …}`. The key name must not decide+/// whether the payload is searched.+#[tokio::test]+async fn an_object_state_votes_on_all_its_string_leaves() {+ let node = gate_node().await;+ let (st, r) = node+ .systemone(json!({+ "state": {"message": "refund refund asked for money back twice charged"},+ "questions": {+ "is_spam": {"type": "noul", "instructions": "Is this SMS message spam?"}+ }+ }))+ .await;+ assert_eq!(st, StatusCode::OK, "{r}");+ assert_eq!(r["answers"]["is_spam"]["noul"], json!(1.0), "{r}");+}++/// A question whose backtick references resolve to nothing is REFUSED with+/// the paths named — not silently voted on whatever text is left. A confident+/// answer computed from a payload the question never saw is the #1001 defect+/// class in another coat.+#[tokio::test]+async fn a_backtick_reference_that_resolves_nowhere_is_named_not_guessed() {+ let node = gate_node().await;+ let (st, r) = node+ .systemone(json!({+ "state": {"message": "refund money back"},+ "questions": {+ "q1": {"type": "noul", "instructions": "Is `typo.path` relevant?"}+ }+ }))+ .await;+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");+ assert_eq!(r["error"]["type"], "illegal_argument", "{r}");+ let why = reason(&r);+ assert!(why.contains("`q1`"), "{why}");+ assert!(why.contains("`typo.path`"), "{why}");+}++/// A payload with no text at all is a 422 naming the question — not a vote+/// on the instruction prose over a history that never saw the payload.+#[tokio::test]+async fn a_state_with_no_text_is_refused_by_name() {+ let node = gate_node().await;+ let (st, r) = node+ .systemone(json!({+ "state": {"spam_count": 42, "flags": [true, false]},+ "questions": {"q1": {"type": "noul", "instructions": "classify"}}+ }))+ .await;+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");+ assert_eq!(r["error"]["type"], "illegal_argument", "{r}");+ let why = reason(&r);+ assert!(why.contains("`q1`"), "{why}");+ assert!(why.contains("state"), "{why}");+}++/// The data-field pattern — the question carries its payload inside+/// `instructions` and names it in backticks, the shape xerj-rerank's stage+/// sends and the TypeSafe docs prescribe. The referenced field IS the vote+/// text, and a state whose vocabulary would pollute the vote stays out of+/// it: the question selected its payload.+#[tokio::test]+async fn an_instructions_data_field_is_the_payload_and_the_state_stays_out() {+ let node = gate_node().await;+ let (st, r) = node+ .systemone(json!({+ "state": "shipping delivery times",+ "questions": {+ "d0": {"type": "noul", "instructions": {+ "question": "Judge `document` against the query in the state.",+ "document": "refund refund asked for money back"+ }}+ }+ }))+ .await;+ assert_eq!(st, StatusCode::OK, "{r}");+ // The refund document votes `true` outright — the misleading state string+ // never entered the query.+ assert_eq!(r["answers"]["d0"]["noul"], json!(1.0), "{r}");+ assert_eq!(r["decisions"]["evidence"]["d0"]["label"], "true", "{r}");+}++/// `jev-reranker` ≥ 0.1.2's `rerank_relevance()` preset ships its rubric+/// object in `state` and names it in backticks ("Apply all remaining+/// evaluation rules in `rubric`."). The rubric is judge rules, not payload:+/// it must be acknowledged without its string leaves joining the vote —+/// embedding them is exactly the #1000 defect (criteria-rich prose). The+/// vote runs on the text references: the document and the query. On the+/// pre-skip code this shape was a 422 ("`rubric` resolves to nothing").+#[tokio::test]+async fn a_structure_reference_like_the_clients_rubric_is_not_vote_text() {+ let node = gate_node().await;+ let (st, r) = node+ .systemone(json!({+ "state": {+ "query": "triage inbox unsolicited correspondence",+ "documents": {"doc_0": "shipping delivery times when does it arrive"},+ "rubric": {+ "instructions": "refund refund asked for money back prize winner claim",+ "criteria": {"true": "retain", "false": "discard"}+ }+ },+ "model": "jev-1.13.0",+ "questions": {+ "doc_0": {"type": "noul",+ "instructions": "Does `documents.doc_0` help answer `query` and \+ deserve a high position in its search results? Apply all \+ remaining evaluation rules in `rubric`.",+ "criteria": {"true": "retain", "false": "discard"}}+ }+ }))+ .await;+ assert_eq!(st, StatusCode::OK, "{r}");+ // The shipping document drives the vote to `false` outright — the rubric's+ // refund/spam vocabulary never entered the query, and the neutral query's+ // terms retrieve nothing over this history.+ assert_eq!(r["answers"]["doc_0"]["noul"], json!(0.0), "{r}");+ assert_eq!(r["decisions"]["evidence"]["doc_0"]["label"], "false", "{r}");+}+/// A `choice` question answers with a probability for EVERY criterion option —/// options with no neighbour weight appear as 0.0, not as missing keys — the/// winner is the argmax, and its confidence is its own probability.@@ -280,8 +454,9 @@ async fn choice_answers_carry_a_probability_for_every_option() {).await;- // state carries no `query` field, so the vote text is exactly the resolved- // ticket — the same string /_decide is asked below, making the arithmetic+ // The vote text is exactly what the question's `ticket` reference+ // resolves to — instruction prose is the question, never the query — so+ // it is the same string /_decide is asked below, making the arithmetic// cross-checkable between the two surfaces.let vote = "cancel my subscription";let (st, r) = nodediff --git a/landing/blog/does-xerj-beat-jev.html b/landing/blog/does-xerj-beat-jev.htmlindex f368e3c76..8dd35feec 100644--- a/landing/blog/does-xerj-beat-jev.html+++ b/landing/blog/does-xerj-beat-jev.html@@ -273,7 +273,7 @@<div class="uc-hero"><p class="blog-crumb"><a href="/blog/">← ALL POSTS · /BLOG</a></p>- <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, ANSWERED · RUNS 2026-09-19..20</div>+ <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, ANSWERED · RUNS 2026-09-19..21</div><h1>DOES XERJ BEAT JEV?<br><span class="accent">ON THE BILL, OUTRIGHT. ON FIQA, NO.</span></h1><p class="blog-byline">AUTHOR · XERJ ENGINEERING TEAM · PUBLISHED · 2026-09-21 · UPDATED · 2026-09-21</p>@@ -283,7 +283,7 @@<div class="vh"><span class="wn">THE VERDICT</span><span class="wm">split, measured, with the losses left in</span></div><p>Not outright — and not behind, either: a split verdict, measured over <strong>1,271 judged BEIR queries</strong> (3 shuffled repeats each). Where reading beats word-overlap, Jev's hosted judge wins rerank: our BM25 top-30 reordered by Jev scores <strong>0.7410 nDCG@10 on SciFact</strong> (versus 0.6993 for our best zero-API-token arm, hybrid RRF) and <strong>0.3638 on FiQA</strong> (versus 0.2382 BM25 — a +0.126 lift, the biggest we measured and the clearest case where hosted Jev still wins); our local hybrid keeps NFCorpus, 0.3448 to the rerank path's 0.3312, and that losing row stays in the table rather than being hidden.</p>- <p>On the bill XERJ wins outright: the FiQA rerank run cost <strong>15,178,758 judge input tokens — $0.6375</strong> at $0.042/Mtok, input only, p50 1.23 s per call — while XERJ answers the same wire (<code class="inline">/v1/systemone</code> and <code class="inline">/_decide</code>) locally with zero tokens, no model, and p50 42.7 ms on <code class="inline">/v1/systemone</code> (<code class="inline">/_decide</code>: 34.8 ms), and the unmodified pip-installed <code class="inline">jev-reranker</code> ran against a XERJ node and passed its own strict validation (spam 0.9014 versus ham 0.0956). AG News reaches 0.9182 accuracy on all 7,600 test items with no model and no tokens — no Jev row exists for AG News and none is claimed.</p>+ <p>On the bill XERJ wins outright: the FiQA rerank run cost <strong>15,178,758 judge input tokens — $0.6375</strong> at $0.042/Mtok, input only, p50 1.23 s per call — while XERJ answers the same wire (<code class="inline">/v1/systemone</code> and <code class="inline">/_decide</code>) locally with zero tokens, no model, and p50 42.7 ms on <code class="inline">/v1/systemone</code> (<code class="inline">/_decide</code>: 34.8 ms), and the unmodified pip-installed <code class="inline">jev-reranker</code> ran against a XERJ node and passed its own strict validation through both of its entry points (spam 0.9447 versus ham 0.0905, client 0.1.2, re-run 2026-09-21). AG News reaches 0.9182 accuracy on all 7,600 test items with no model and no tokens — no Jev row exists for AG News and none is claimed.</p><p>Hold the caveats: hosted Jev's published SciFact 0.768 and NFCorpus 0.358 are not run by us and are not a race with our rows (different experiments sharing a metric); provider non-determinism makes any gap within ±0.01 nDCG@10 noise; and Jev's FiQA probabilities are badly calibrated (ECE 0.3109 — documents rated 0.93 confidence were relevant 34% of the time; order by them, never threshold them with <code class="inline">min_score</code>). Verdict: use Jev where the judge wins, FiQA-style rerank; use XERJ where tokens, latency, and data locality decide.</p></section>@@ -544,7 +544,7 @@</div><h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">The gate: the pip client, unmodified, on a local node</h3>- <p class="wide">No SDK of ours, no patches, no subclassing. <code class="inline">pip install jev-reranker</code> (0.1.0, MIT) in a clean venv, its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. Two environment variables, and one of them is a trap:</p>+ <p class="wide">No SDK of ours, no patches, no subclassing. <code class="inline">pip install jev-reranker</code> (0.1.2, MIT), its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. The gate ranks through both of the client's entry points — the constructor default, and the documented relevance preset, which since 0.1.2 ships its rubric as an <em>object</em> the vote must acknowledge without reading it as retrieval text. Two environment variables, and one of them is a trap:</p><pre class="code"><span class="prompt">$</span> pip install jev-reranker<span class="prompt">$</span> <span class="cmt"># TYPESAFE_ENDPOINT carries the FULL path. TYPESAFE_BASE_URL is silently ignored</span>@@ -552,20 +552,21 @@<span class="prompt">$</span> export TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone<span class="prompt">$</span> export TYPESAFE_API_KEY=local-only <span class="cmt"># the node ignores it; localhost only</span><span class="prompt">$</span> python3 gate_run.py-<span class="out">client resolved model: ['xerj-history-vote-1']+<span class="out">client: jev-reranker 0.1.2 (pip, unmodified)+client resolved model: ['xerj-history-vote-1']1.0000 spam Please call our customer service representative on 0800 169 6031 betwe1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT0.0000 ham If you're not in my car in an hour and a half I'm going apeshit0.0000 ham Carlos says we can pick up from him later so yeah we're set- ... <span class="cmt"># 10 fixed docs, 5+5</span>-mean noul spam=0.9014 ham=0.0956 gap=0.8058+ ... <span class="cmt"># 10 fixed docs, 5+5, both entry points</span>+mean noul spam=0.9447 ham=0.0905 gap=0.8542GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre><p class="wide">The client's own validation <em>is</em> the wire gate. Answers keyed by the sent ids. Every noul numeric in 0..1. Model a non-empty string. Usage non-negative integers. Surviving it unmodified is the compatibility claim. The spam/ham separation is the semantic one. <strong>Zero requests left the machine.</strong> The vote is an ordinary search of an ordinary index, and the module adds no outbound client.</p><div class="loss">- <strong>The first gate attempt failed. It is the most useful thing on this page.</strong> The wire puts the query in <code class="inline">state</code>, and the vote text joins it, so query vocabulary takes part in retrieval. With a spam-bearing query like "winner claim prize", <em>every</em> document's vote retrieved spam neighbours. The vote measured the query, not the documents: spam 1.000 against ham 0.785. No separation. The fixed gate uses a query whose every term is absent from the corpus: "triage inbox unsolicited correspondence". Result, 0.9014 against 0.0956. This is inherent to a retrieval vote, not a defect. <strong>Pick the query for the history you have.</strong> Do not let a rerank habit leak vocabulary into a classification question.+ <strong>The first gate attempt failed. It is the most useful thing on this page.</strong> The wire puts the query in <code class="inline">state</code>, and the vote text joined it, so query vocabulary took part in retrieval. With a spam-bearing query like "winner claim prize", <em>every</em> document's vote retrieved spam neighbours. The vote measured the query, not the documents: spam 1.000 against ham 0.785. No separation. We first called that "inherent to a retrieval vote". <strong>It was not. It was two defects, and we fixed them.</strong> The vote text used to concatenate the instruction's prose into the retrieval query — wording alone moved accuracy 32 points (0.9867 with an empty instruction, 0.6700 with a criteria-rich one, same 300 held-out messages) — and an object state contributed only its <code class="inline">query</code> field, so <code class="inline">{"message": …}</code> voted on prose at the spam base rate (0.1733 against 0.9667 for the same message as a string). The vote text is now exactly what the question points at: its backtick references, or the state's own string content — never instruction prose, never state the question never named. After the fix, all eight wording/shape arms of the same 300 messages score identically (0.9767). One part <em>is</em> by design, and it is the client's choice, not ours: since 0.1.2 the client's own question references the query in backticks, so the query is named payload — the question being asked is "does this document answer this query". <strong>Pick the query for the history you have</strong> still holds: under the deliberately spammy query the gap narrows from 0.8542 to 0.6087, and the gate asserts both.</div><h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">Speed: the Rust side of the same wire</h3>@@ -698,7 +699,7 @@ EOF<span class="prompt">$</span> python3 -m venv ~/gate-venv && ~/gate-venv/bin/pip install jev-reranker<span class="prompt">$</span> TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone \TYPESAFE_API_KEY=local-only ~/gate-venv/bin/python benchmarks/systemone-gate/gate_run.py-<span class="out">mean noul spam=0.9014 ham=0.0956 gap=0.8058+<span class="out">mean noul spam=0.9447 ham=0.0905 gap=0.8542GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre><p class="src">Everything on this page: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot" rel="external">the pilot</a> ·<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">the full run</a> ·@@ -706,7 +707,7 @@ GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre><a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews" rel="external">the AG News vote</a> ·<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/systemone-gate" rel="external">the gate</a> ·<a href="https://github.com/xerj-org/xerj/blob/main/docs/research/typesafe-skill-2026-09/NOTES.md" rel="external">the research narrative</a> ·- measured 2026-09-19..20 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance <!-- snapshot:2026-09-20 -->1371 passed / 0 failed on that branch (canonical: 1366 / 1369 today) ·+ measured 2026-09-19..21 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance <!-- snapshot:2026-09-20 -->1371 passed / 0 failed on that branch (canonical: 1366 / 1369 today) ·related: <a href="/benchmarks/">the benchmarks hub</a> · <a href="/answers/rerank-search-results-calibrated-judge">the rerank stage, documented</a> ·<a href="/benchmarks/elasticsearch">the ES scorecard</a>.</p></section>