Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(api): /v1/systemone votes on the payload, never the instruction prose

xerj-orgfed0fffa012a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 597b14f1e..f2ac79ef0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Fixed
+
+- **`/v1/systemone` votes on the payload, never on the instruction prose.**
+ A question's `instructions` used to be concatenated into the retrieval
+ query, so a well-written instruction destroyed accuracy — measured on the
+ SMS history (300 held-out, seed 7): 0.9867 empty vs 0.6700 criteria-rich, a
+ 32-point swing from wording alone (#1000). And an object `state`
+ contributed only its `query` field, so `{"message": …}` voted on the
+ instruction at the spam base rate: accuracy 0.1733 vs 0.9667 for the same
+ message as a string (#1001). The vote text is now exactly what the question
+ points at: its backtick references (instructions data fields first, then
+ state paths — both client shapes keep working), or, when it references
+ nothing, the state's own string content (a string whole, an object by all
+ string leaves). A reference that resolves to structure rather than text —
+ the rubric object `jev-reranker` ≥ 0.1.2 ships with its relevance preset —
+ is acknowledged and skipped, not embedded and not fatal. References that
+ resolve to nowhere and payloads with no text are 422s naming the question.
+ Also fixed the `no_support` error body, which interpolated `k` and the
+ question ids into each other's slots.
+ Fail-before: five new `systemone_http` tests, all failing on the old code
+ (`[0.0, 0.0, 0.4545, 0.0]` across four instruction wordings of one state);
+ a sixth pins the client's rubric shape, a 422 before the skip rule. After:
+ the 300 held-out messages score 0.9767 identically across all eight
+ wording/shape arms, and the pip-client acceptance gate passes through both
+ client entry points (neutral-query gap 0.8542) and under a deliberately
+ spammy query (gap 0.6087, threshold 0.3) — client 0.1.2, unmodified.
+
## [1.0.0-rc.76] - 2026-09-21
The JEV-interface and honest-fixes release. The headline is a measured answer
diff --git a/benchmarks/systemone-gate/README.md b/benchmarks/systemone-gate/README.md
index 224017095..58b056726 100644
--- a/benchmarks/systemone-gate/README.md
+++ b/benchmarks/systemone-gate/README.md
@@ -1,15 +1,15 @@
# The /v1/systemone acceptance gate — jev-reranker, unmodified, on a XERJ node
-Run 2026-09-20, locally built `xerj-server` at the wire-compat commit,
-`--insecure --port 9440` (native REST on 9441), throwaway data dir,
-`[decisions] index = "sms"` (`gate-decisions.toml`: k=10, text/label fields,
-`positive_label = "spam"`). Client: `pip install jev-reranker` (0.1.0, MIT) in a
-clean venv — no patches, no subclassing, its own listwise instruction template,
-its own splitting, its own strict response validation. The only configuration:
+Run 2026-09-21, locally built `xerj-server` at the vote-text fix (#1000/#1001),
+`--port 9490` (native REST on 9491), throwaway data dir, `[decisions]
+index = "sms"` (`gate-decisions.toml`: k=10, text/label fields,
+`positive_label = "spam"`). Client: `pip install jev-reranker` (0.1.2, MIT) —
+no patches, no subclassing, its own listwise instruction template, its own
+splitting, its own strict response validation. The only configuration:
```sh
-TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone
-TYPESAFE_API_KEY=<any non-empty string> # the node ignores it; localhost only
+TYPESAFE_ENDPOINT=http://localhost:9491/v1/systemone
+TYPESAFE_API_KEY=<the node's admin key>
```
`TYPESAFE_BASE_URL` is silently ignored by the client — the endpoint variable
@@ -20,13 +20,25 @@ must carry the full path. (Verified in the wheel: `reranker.py:238`.)
`gate_load_sms.py` indexes the 4,000-message SMS train split
(`benchmarks/decisions-as-retrieval`'s split: shuffle seed 7, first 4,000) and
fixes 10 held-out gate documents (5 spam, 5 ham, 40–140 chars) in
-`gate-docs.json`. `gate_run.py` ranks them with `JevReranker().rerank(...)`.
+`gate-docs.json`. `gate_run.py` ranks them through BOTH client entry points:
+
+1. the constructor default (listwise, compact instruction — references
+ `` `documents.doc_N` `` and `` `query` `` in backticks);
+2. `relevance_rerank()` — the documented relevance preset, which since client
+ 0.1.2 ships its rubric OBJECT in state and references it in backticks. The
+ node acknowledges the rubric without embedding its prose in the vote
+ (#1000's defect class); the two phases score identically, which is the
+ proof.
+3. the constructor default again, under a deliberately spammy query.
+
The query is `triage inbox unsolicited correspondence` — every term verified
-absent from the corpus, because `state.query` joins every vote text and a
-vocabulary-bearing query ("winner claim prize") retrieves spam neighbours for
-EVERY document, which measures the query, not the documents. First gate
-attempt failed exactly there (spam 1.000 vs ham 0.785); the transcript below
-is the fixed run.
+absent from the corpus. Since client 0.1.2 the question itself references
+`` `query` `` in backticks, so the query is named payload and joins the vote
+BY CLIENT DESIGN (the question asked is "does this document answer this
+query"). For a classification history that is noise, which is why the gate
+query is neutral — and phase 3 proves the separation survives a spammy query
+("winner claim free prize congratulations") that the pre-#1000 node would
+have collapsed on (the first gate attempt then: spam 1.000 vs ham 0.785).
## Result (`gate-transcript.txt`)
@@ -34,9 +46,10 @@ The client's own validation IS the wire gate: answers keyed exactly by the
sent question ids, every answer `{type: "noul", noul}` numeric in 0..1,
`model` a non-empty string (it resolved `xerj-history-vote-1` — the node's
truthful echo, never a Jev name), `usage` non-negative integers. The semantic
-gate: mean noul 0.9014 over spam vs 0.0956 over ham (gap 0.806, threshold
-0.3). No request left the node: the vote is an ordinary search of the `sms`
-index; the module adds no outbound client.
+gate, threshold 0.3: neutral query gap **0.8542** (spam mean 0.9447 vs ham
+0.0905, identical across both entry points), spammy query gap **0.6087**. No
+request left the node: the vote is an ordinary search of the `sms` index;
+the module adds no outbound client.
## The response the client does not show
@@ -44,12 +57,14 @@ index; the module adds no outbound client.
engine scores, weights, and an abstain verdict below `decisions.min_confidence`:
```sh
-curl -s localhost:9440/_decide -H 'content-type: application/json' \
+curl -s localhost:9490/_decide -H 'content-type: application/json' \
+ -H "authorization: ApiKey $(cat <data_dir>/admin.key)" \
-d '{"index":"sms","question":"URGENT! Your Mobile number has been awarded"}'
# {"index":"sms","k":10,"label":"spam","confidence":1.0,"abstain":false,
# "neighbours":[{"_id":"1805","label":"spam",...,"weight":1.0}, ...]}
```
Zero-support questions are a 422 naming the question ids, never a fabricated
-0.5; `score` questions (no vote analogue) are a 422 naming the alternative.
-Both behaviours are pinned by `engine/crates/xerj-api/tests/systemone_http.rs`.
+0.5; `score` questions (no vote analogue) are a 422 naming the alternative;
+a backtick reference that resolves to nothing is a 422 naming the path. All
+pinned by `engine/crates/xerj-api/tests/systemone_http.rs`.
diff --git a/benchmarks/systemone-gate/gate-transcript.txt b/benchmarks/systemone-gate/gate-transcript.txt
index 77c11a2c7..8a02f9e0a 100644
--- a/benchmarks/systemone-gate/gate-transcript.txt
+++ b/benchmarks/systemone-gate/gate-transcript.txt
@@ -1,13 +1,39 @@
+client: jev-reranker 0.1.2 (pip, unmodified)
client resolved model: ['xerj-history-vote-1']
- 0.5068 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600
+-- constructor default (listwise), neutral query
+ 0.7236 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600
1.0000 spam Please call our customer service representative on 0800 169 6031 betwe
1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.
1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
- 0.1024 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca
+ 0.1110 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca
0.0000 ham If you're not in my car in an hour and a half I'm going apeshit
0.3414 ham Awesome question with a cute answer: Someone asked a boy "how is ur li
0.0000 ham Carlos says we can pick up from him later so yeah we're set
- 0.0341 ham The greatest test of courage on earth is to bear defeat without losing
-mean noul spam=0.9014 ham=0.0956 gap=0.8058
+ 0.0000 ham The greatest test of courage on earth is to bear defeat without losing
+ mean noul spam=0.9447 ham=0.0905 gap=0.8542
+-- rerank_relevance() preset (rubric object), neutral query
+ 0.7236 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600
+ 1.0000 spam Please call our customer service representative on 0800 169 6031 betwe
+ 1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
+ 1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.
+ 1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
+ 0.1110 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca
+ 0.0000 ham If you're not in my car in an hour and a half I'm going apeshit
+ 0.3414 ham Awesome question with a cute answer: Someone asked a boy "how is ur li
+ 0.0000 ham Carlos says we can pick up from him later so yeah we're set
+ 0.0000 ham The greatest test of courage on earth is to bear defeat without losing
+ mean noul spam=0.9447 ham=0.0905 gap=0.8542
+-- constructor default, deliberately spammy query
+ 1.0000 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600
+ 1.0000 spam Please call our customer service representative on 0800 169 6031 betwe
+ 1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
+ 1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.
+ 1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
+ 0.2887 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca
+ 0.3414 ham If you're not in my car in an hour and a half I'm going apeshit
+ 0.7174 ham Awesome question with a cute answer: Someone asked a boy "how is ur li
+ 0.1707 ham Carlos says we can pick up from him later so yeah we're set
+ 0.4382 ham The greatest test of courage on earth is to bear defeat without losing
+ mean noul spam=1.0000 ham=0.3913 gap=0.6087
GATE PASSED: unmodified jev-reranker ranked from the XERJ node
diff --git a/benchmarks/systemone-gate/gate_run.py b/benchmarks/systemone-gate/gate_run.py
index d9e29812a..d3b1152f8 100644
--- a/benchmarks/systemone-gate/gate_run.py
+++ b/benchmarks/systemone-gate/gate_run.py
@@ -2,10 +2,10 @@
XERJ node.
TYPESAFE_ENDPOINT=http://localhost:<native-rest-port>/v1/systemone
- TYPESAFE_API_KEY=<any non-empty string — the node ignores it, localhost only>
+ TYPESAFE_API_KEY=<the node's admin key>
-Ranks the fixed gate documents (gate-docs.json: 5 spam, 5 ham) with the
-client's own listwise pipeline and asserts two things:
+Ranks the fixed gate documents (gate-docs.json: 5 spam, 5 ham) through BOTH
+client entry points and asserts two things per phase:
1. WIRE: the call completes with no ResponseValidationError — answers keyed
exactly, noul numeric in 0..1, model echoed as a non-empty string, usage
@@ -13,39 +13,66 @@ client's own listwise pipeline and asserts two things:
2. SEMANTICS: the local vote separates the classes — mean noul over spam
documents strictly exceeds mean noul over ham documents by >= 0.3.
-Run: python3 gate_run.py [native_url_base]
+Phase 1 is the constructor default (listwise, compact instruction). Phase 2
+is `rerank_relevance()` — the documented relevance preset, which since client
+0.1.2 ships its rubric OBJECT in state and references it in backticks; the
+node must acknowledge the rubric without embedding its prose in the vote.
+
+The query stays neutral (`triage inbox unsolicited correspondence`, every
+term absent from the corpus): the client's own question references `query`
+in backticks, so the query is named payload and joins the vote by design.
+For a classification history that adds noise — which is why the query is
+neutral, and why phase 3 asserts survival of a deliberately spammy query
+("winner claim free prize congratulations") under the same threshold.
+
+Run: python3 gate_run.py
"""
-import json, os, sys
+import json
+from importlib.metadata import version
+
+import jev_reranker
+from jev_reranker import JevReranker
docs = json.load(open("gate-docs.json"))
texts = [d["text"] for d in docs]
labels = [d["label"] for d in docs]
-# The query rides in state.query and joins every vote text, so its terms must
-# be NEUTRAL for the history: a spammy query ("winner claim prize") retrieves
-# spam neighbours for every document and the vote says nothing about the doc.
-# Every term here is absent from the SMS corpus (checked against sms.tsv), so
-# the document drives the vote. That asymmetry — query vocabulary pollutes a
-# classification history — is documented in docs/RERANK.md.
QUERY = "triage inbox unsolicited correspondence"
+SPAMMY = "winner claim free prize congratulations"
-from jev_reranker import JevReranker
-rr = JevReranker() # reads TYPESAFE_ENDPOINT / TYPESAFE_API_KEY from the env
-result = rr.rerank(QUERY, texts, detail=True)
+def phase(name, scores):
+ print(f"-- {name}")
+ for lab, txt, s in zip(labels, texts, scores):
+ print(f" {s:.4f} {lab:4} {txt[:70]}")
+ assert len(scores) == len(texts)
+ assert all(isinstance(s, float) and 0.0 <= s <= 1.0 for s in scores), scores
+ spam = [s for s, l in zip(scores, labels) if l == "spam"]
+ ham = [s for s, l in zip(scores, labels) if l == "ham"]
+ ms, mh = sum(spam) / len(spam), sum(ham) / len(ham)
+ print(f" mean noul spam={ms:.4f} ham={mh:.4f} gap={ms - mh:.4f}")
+ assert ms - mh >= 0.3, f"vote did not separate: {ms:.3f} vs {mh:.3f}"
+
-# {"results": [{"document_index", "score", "text", ...}, ...]} — order follows
-# the client's ranking; re-key by the document index it reports.
-by_index = {r["document_index"]: r["score"] for r in result["results"]}
-scores = [by_index[i] for i in range(len(texts))]
+def rank(query):
+ result = rr.rerank(query, texts, detail=True)
+ by_index = {r["document_index"]: r["score"] for r in result["results"]}
+ return [by_index[i] for i in range(len(texts))], result
+
+
+print(f"client: jev-reranker {version('jev-reranker')} (pip, unmodified)")
+
+rr = JevReranker() # reads TYPESAFE_ENDPOINT / TYPESAFE_API_KEY from the env
+scores, result = rank(QUERY)
print("client resolved model:", result["detail"]["resolved_models"])
-for lab, txt, s in zip(labels, texts, scores):
- print(f" {s:.4f} {lab:4} {txt[:70]}")
-
-assert len(scores) == len(texts)
-assert all(isinstance(s, float) and 0.0 <= s <= 1.0 for s in scores), scores
-spam = [s for s, l in zip(scores, labels) if l == "spam"]
-ham = [s for s, l in zip(scores, labels) if l == "ham"]
-ms, mh = sum(spam) / len(spam), sum(ham) / len(ham)
-print(f"mean noul spam={ms:.4f} ham={mh:.4f} gap={ms - mh:.4f}")
-assert ms - mh >= 0.3, f"vote did not separate: {ms:.3f} vs {mh:.3f}"
+phase("constructor default (listwise), neutral query", scores)
+
+# The documented preset: rubric object in state, referenced in backticks.
+relevance = rr.relevance_rerank(QUERY, texts, threshold=0.0, return_documents=False)
+scores = [s for _, s in sorted(
+ (r["document_index"], r["score"]) for r in relevance["results"])]
+phase("rerank_relevance() preset (rubric object), neutral query", scores)
+
+scores, _ = rank(SPAMMY)
+phase("constructor default, deliberately spammy query", scores)
+
print("GATE PASSED: unmodified jev-reranker ranked from the XERJ node")
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 7735c069b..bce4fd91c 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -62,26 +62,48 @@ wrong one when you do not. Where there is no history — a new policy, a new
category — use a judge model; that is what
[docs/RERANK.md](./RERANK.md) is for.
-**The query's vocabulary pollutes a classification history.** The wire puts
-the query in `state`, and the vote text is the resolved question text plus
-that query — so query terms participate in retrieval. For a rerank-shaped
-request that is correct (relevance IS query-relative). For a classification
-history it can swamp the document: against the SMS corpus, the query
-"congratulations winner claim free prize" retrieves spam neighbours for *every*
-document and the vote says nothing about the document (measured: spam mean
-1.000 vs ham 0.785 — no separation). Keep the query neutral for the corpus
-(every term of `triage inbox unsolicited correspondence` is absent from the SMS
-corpus: spam 0.901 vs ham 0.096). This is inherent to a retrieval vote, not a
-defect to fix: pick the query for the history you have.
+**What the vote retrieves on — the payload, never the prose (#1000, #1001).**
+A question's `instructions` are a question ABOUT the payload; their vocabulary
+is not retrieval text. The vote text is exactly what the question points at:
+
+- every `` `path` `` reference the instructions name — resolved against the
+ instructions' own string fields first (the data-field pattern:
+ `{"question": "Judge `document` …", "document": text}`, which
+ `xerj-rerank`'s stage and the provider's docs use), then against the state
+ by dotted path (`` `documents.doc_0` ``, the shape `jev-reranker` sends).
+ A reference that resolves to structure rather than text — the rubric
+ object `jev-reranker` ≥ 0.1.2 ships with its relevance preset — is
+ acknowledged and skipped: judge rules are not retrieval vocabulary;
+- a question that references nothing votes on the state alone — a string
+ state whole, an object state by ALL its string leaves, whatever the keys
+ are named;
+- references that resolve to nothing, and a payload with no text at all, are
+ 422s naming the question — never a confident vote on text the question
+ never saw.
+
+Two defects this design closes, both measured on the SMS history (4,000
+train / 300 held-out, shuffle seed 7): instruction wording used to change the
+answer — accuracy 0.9867 with an empty instruction against 0.6700 with a
+criteria-rich one, a 32-point swing from wording alone — and an object state
+used to contribute only its `query` field, so `{"message": …}` voted on the
+instruction prose at the spam base rate (accuracy 0.1733 against 0.9667 for
+the same message as a string). After the fix the same 300 held-out messages
+score identically under every instruction wording and every state shape —
+0.9767 across all eight arms (the absolute number sits ~1 point under the
+issue runs' 0.9867 because the history was re-indexed fresh for the re-run;
+the invariant is the identity, not the digit). One nuance the client itself
+decides: since 0.1.2, `jev-reranker`'s own question references `` `query` ``
+in backticks, so the query is named payload and joins the vote BY CLIENT
+DESIGN — the acceptance gate still uses a neutral query for a classification
+history, and survives a deliberately spammy one (gap 0.61 against 0.85,
+threshold 0.3).
## The wire, and the two deliberate breaks
Request: `{state, model, questions: {id: {type, instructions, criteria}}}`.
`state` may be a string or an object; `instructions` may be a string, object
-or array (all string leaves join); strings may reference state data with
-backtick paths — `` `documents.doc_0` `` — resolved against `state` before the
-vote text is built. That is the reference pattern of the API and the shape
-`jev-reranker` sends.
+or array. `criteria` descriptions are never retrieval text; the options they
+name are `choice` answers, not query vocabulary.
Response: `{model, answers, usage}` with `answers` keyed exactly by the
question ids sent, each answer carrying only its documented fields (clients
diff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rs
index 1e6ab5a7f..788a8d24b 100644
--- a/engine/crates/xerj-api/src/systemone_api.rs
+++ b/engine/crates/xerj-api/src/systemone_api.rs
@@ -21,11 +21,20 @@
//!
//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.
//! `state` may be a string or an object; question `instructions` may be a
-//! string, object or array (all string leaves are joined). Strings may point
-//! at state data with backtick paths — `` `documents.doc_0` `` — which are
-//! resolved against `state` before the vote text is built. That is the
-//! reference pattern of the System One API and the shape `jev-reranker`
-//! sends in its listwise mode.
+//! string, object or array. **The vote text is payload data, never
+//! instruction prose (#1000, #1001).** A question votes on exactly what it
+//! points at: every `` `path` `` reference its instructions name — resolved
+//! against the instructions' own string fields first (the data-field
+//! pattern: `{"question": "Judge `document` …", "document": text}`) and then
+//! against the state by dotted path (`` `documents.doc_0` ``, the shape
+//! `jev-reranker` sends) — or, when it references nothing, on the state
+//! alone: a string state whole, an object state by all its string leaves.
+//! A reference that resolves to structure rather than text (the rubric
+//! object `jev-reranker` ≥ 0.1.2 ships with its relevance preset) is
+//! acknowledged and skipped: judge rules are not retrieval vocabulary.
+//! References that resolve to nothing, and a payload with no text at all,
+//! are 422s naming the question — never a confident vote on text the
+//! question never saw.
//!
//! Response: `{ model, answers, usage }` with `answers` keyed exactly by the
//! question ids sent and each answer carrying only its documented fields —
@@ -389,36 +398,90 @@ fn parse_question(id: &str, q: &Value, state_val: &Value) -> Result<Question, St
))
}
};
- // `instructions` is polymorphic on this wire: string, object or array.
- // Every string leaf joins the vote text — an object's fields arrive in
- // serde_json's BTreeMap order, which is stable but arbitrary; the text is
- // retrieved by content, not by field order.
- let mut text = String::new();
- if let Some(instructions) = obj.get("instructions") {
- join_strings(instructions, &mut text);
+ // The vote text is RETRIEVAL TEXT, and retrieval text must be payload
+ // data, never instruction prose (#1000). An instruction is a question
+ // ABOUT the payload; its vocabulary retrieves neighbours on its own —
+ // measured 0.9867 accuracy with an empty instruction against 0.6700 with
+ // a criteria-rich one, the same 300 messages. So a question votes on
+ // exactly what it POINTS AT:
+ //
+ // - every `` `path` `` reference its instructions name, resolved first
+ // against the instructions' own string fields (the data-field pattern:
+ // `{"question": "Judge `document` …", "document": text}`, which
+ // xerj-rerank's stage and the provider's docs use) and then against
+ // the state (the jev-reranker pattern: documents in state, referenced
+ // by dotted path). The state does NOT ride along here — a question
+ // that named its payload gets its payload, not its payload plus
+ // whatever else the state holds;
+ // - a reference that resolves to STRUCTURE, not text, contributes no
+ // vote text. `jev-reranker` ≥ 0.1.2's `rerank_relevance()` preset
+ // ships its rubric object in state and names it in backticks; the
+ // rubric is judge rules, and embedding its string leaves is exactly
+ // the #1000 defect (criteria-rich prose, the measured 0.6700 class).
+ // Structure is acknowledged and skipped — text is text, rules are
+ // rules;
+ // - a question that references nothing votes on the state alone: a
+ // string state whole, an object state by ALL its string leaves (#1001:
+ // only `query` used to be read, so `{"message": …}` answered from the
+ // instruction alone, at the spam base rate);
+ // - a question whose references resolve NOWHERE is refused naming them —
+ // a confident vote on text the question never saw is the silent-fake
+ // defect class, in another coat;
+ // - a vote text that is still empty is refused naming the question.
+ //
+ // `criteria` descriptions are never embedded: no benchmark measures
+ // criteria text, and it is the richest vocabulary on the wire.
+ let instructions = obj.get("instructions");
+ let mut paths: Vec<String> = Vec::new();
+ if let Some(instr) = instructions {
+ collect_backtick_paths(instr, &mut paths);
}
- // `criteria` descriptions are NOT embedded: the vote is over text the
- // history was labelled on, and no benchmark measures criteria text.
- let resolved = resolve_backticks(&text, state_val);
- // The query rides along from state — a plain-string state whole, an
- // object state via its `query` field, exactly the two shapes the wire's
- // own examples use. Other state fields are referenced (or not) by the
- // instructions' backtick paths, which have already been resolved.
- let mut vote_text = resolved;
- match state_val {
- Value::String(s) => {
- vote_text.push(' ');
- vote_text.push_str(s);
- }
- Value::Object(m) => {
- if let Some(q) = m.get("query").and_then(Value::as_str) {
- vote_text.push(' ');
- vote_text.push_str(q);
+ let mut vote_text = String::new();
+ if paths.is_empty() {
+ join_strings(state_val, &mut vote_text);
+ } else {
+ let mut unresolved: Vec<String> = Vec::new();
+ for path in &paths {
+ match resolve_reference(instructions, state_val, path) {
+ Some(Value::String(text)) => {
+ if !vote_text.is_empty() {
+ vote_text.push(' ');
+ }
+ vote_text.push_str(text);
+ }
+ // Resolves, but to structure — an object, number, boolean.
+ // Rules and figures carry no retrieval vocabulary: skipping
+ // them is the difference between this and #1000, where the
+ // prose joined the vote and wording moved accuracy 32 points.
+ Some(_) => {}
+ None => unresolved.push(path.clone()),
}
}
- _ => {}
+ if !unresolved.is_empty() {
+ let listed = unresolved
+ .iter()
+ .map(|p| format!("`{p}`"))
+ .collect::<Vec<_>>()
+ .join(", ");
+ return Err(format!(
+ "question `{id}`: backtick reference{} {listed} resolve{} to nothing in \
+ this request — the vote would run on text the question never named; \
+ reference existing instructions fields or state paths, or drop the \
+ backticks",
+ if unresolved.len() == 1 { "" } else { "s" },
+ if unresolved.len() == 1 { "s" } else { "" },
+ ));
+ }
}
let vote_text = clip(vote_text.trim(), MAX_VOTE_TEXT_CHARS);
+ if vote_text.is_empty() {
+ return Err(format!(
+ "question `{id}`: no text to vote on — the payload resolved to nothing; put \
+ text in `state` (a string, or an object with string fields) or reference it \
+ in backticks from the instructions, remembering that only string-valued \
+ references carry text"
+ ));
+ }
Ok(Question { vote_text, kind })
}
@@ -482,43 +545,56 @@ fn join_strings(v: &Value, out: &mut String) {
}
}
-/// Resolve `` `dotted.path` `` references against `state`. An unresolvable
-/// path is left literally in place — visible in the vote text rather than
-/// silently dropped.
-fn resolve_backticks(text: &str, state: &Value) -> String {
- if !text.contains('`') || state.is_null() {
- return text.to_string();
- }
- let mut out = String::with_capacity(text.len());
- let mut rest = text;
- while let Some(start) = rest.find('`') {
- out.push_str(&rest[..start]);
- let after = &rest[start + 1..];
- match after.find('`') {
- Some(end) => {
- let path = &after[..end];
- match walk(state, path) {
- Some(v) => out.push_str(v),
- None => out.push_str(&rest[start..start + end + 2]),
+/// Collect every `` `path` `` a question's instructions name, in order of
+/// first appearance, deduplicated. The prose between backticks is ignored —
+/// it is the question, not the payload (#1000). An unpaired backtick names
+/// nothing.
+fn collect_backtick_paths(v: &Value, out: &mut Vec<String>) {
+ match v {
+ Value::String(s) => {
+ let mut rest = s.as_str();
+ while let Some(start) = rest.find('`') {
+ let after = &rest[start + 1..];
+ match after.find('`') {
+ Some(end) => {
+ let path = &after[..end];
+ if !path.is_empty() && !out.iter().any(|p| p == path) {
+ out.push(path.to_string());
+ }
+ rest = &after[end + 1..];
+ }
+ None => break,
}
- rest = &after[end + 1..];
- }
- None => {
- out.push_str(&rest[start..]);
- return out;
}
}
+ Value::Array(a) => a.iter().for_each(|x| collect_backtick_paths(x, out)),
+ Value::Object(m) => m.values().for_each(|x| collect_backtick_paths(x, out)),
+ _ => {}
}
- out.push_str(rest);
- out
}
-fn walk<'a>(v: &'a Value, path: &str) -> Option<&'a str> {
+/// Resolve one backtick reference to the value it names. The instructions'
+/// own string fields win over same-named state paths — the data-field
+/// pattern puts the payload next to the prose that names it — then the
+/// state is walked by dotted path. `None` means the reference names
+/// nothing in this request; a caller decides what a non-string resolution
+/// means (here: structure, not vote text).
+fn resolve_reference<'a>(
+ instructions: Option<&'a Value>,
+ state: &'a Value,
+ path: &str,
+) -> Option<&'a Value> {
+ instructions
+ .and_then(|i| walk(i, path))
+ .or_else(|| walk(state, path))
+}
+
+fn walk<'a>(v: &'a Value, path: &str) -> Option<&'a Value> {
let mut cur = v;
for seg in path.split('.') {
cur = cur.get(seg)?;
}
- cur.as_str()
+ Some(cur)
}
fn clip(s: &str, max: usize) -> String {
@@ -580,12 +656,12 @@ fn no_support(ids: &[String], cfg: &DecisionsConfig) -> axum::response::Response
error, not a fabricated probability — add labelled examples to the history \
index or use POST /_decide to inspect what is there",
cfg.index,
- cfg.k,
if ids.len() == 1 { "" } else { "s" },
ids.iter()
.map(|i| format!("`{i}`"))
.collect::<Vec<_>>()
- .join(", ")
+ .join(", "),
+ cfg.k,
),
)
}
diff --git a/engine/crates/xerj-api/tests/systemone_http.rs b/engine/crates/xerj-api/tests/systemone_http.rs
index 734ad0f91..d2046b1c3 100644
--- a/engine/crates/xerj-api/tests/systemone_http.rs
+++ b/engine/crates/xerj-api/tests/systemone_http.rs
@@ -263,6 +263,180 @@ async fn a_plain_string_state_with_object_instructions_also_votes() {
assert_eq!(r["answers"]["q1"]["noul"], json!(1.0));
}
+/// #1000: instruction wording must not change the answer. The instruction is
+/// a question ABOUT the payload; its vocabulary is not retrieval text. The
+/// issue measured a 32-point accuracy swing on a real SMS history (0.9867
+/// empty instruction vs 0.6700 criteria-rich) purely from wording — here the
+/// same state is asked three times with three different instructions and all
+/// three answers must be identical.
+#[tokio::test]
+async fn instruction_wording_does_not_change_the_answer() {
+ let node = gate_node().await;
+ // A ham-shaped state: its own terms retrieve the `false` doc and nothing
+ // else, whatever the prose says.
+ let state = json!("shipping delivery times");
+ let rich = "Is this SMS message spam? prize claims, paid subscriptions, \
+ marketing, premium-rate numbers, refund, money back";
+ let mut answers = Vec::new();
+ for instructions in [
+ None,
+ Some(json!("Is this message spam?")),
+ Some(json!(rich)),
+ // the criteria-block shape a Jev migrant sends
+ Some(json!({"question": "Is this message spam?", "criteria": {
+ "spam": "prize claims, paid subscriptions, marketing",
+ "ham": "plain personal correspondence"}})),
+ ] {
+ let mut q = serde_json::Map::new();
+ q.insert("type".into(), json!("noul"));
+ if let Some(i) = instructions {
+ q.insert("instructions".into(), i);
+ }
+ let (st, r) = node
+ .systemone(json!({
+ "state": state,
+ "questions": {"d0": Value::Object(q)}
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+ answers.push(r["answers"]["d0"]["noul"].as_f64().unwrap_or(-1.0));
+ }
+ // All four answers identical, and the vote followed the STATE: a
+ // shipping text over this history is `false` outright.
+ assert!(
+ answers.iter().all(|a| (a - answers[0]).abs() < 1e-9),
+ "instruction wording changed the answer: {answers:?}"
+ );
+ assert!(
+ (answers[0] - 0.0).abs() < 1e-9,
+ "the state's own vote is `false`, got {answers:?}"
+ );
+}
+
+/// #1001: an object state votes on ALL its string leaves. The issue measured
+/// accuracy 0.1733 — exactly the spam base rate — for `{"message": …}` (every
+/// item scored from identical instruction text) against 0.9667 for the same
+/// message as a string or as `{"query": …}`. The key name must not decide
+/// whether the payload is searched.
+#[tokio::test]
+async fn an_object_state_votes_on_all_its_string_leaves() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": {"message": "refund refund asked for money back twice charged"},
+ "questions": {
+ "is_spam": {"type": "noul", "instructions": "Is this SMS message spam?"}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+ assert_eq!(r["answers"]["is_spam"]["noul"], json!(1.0), "{r}");
+}
+
+/// A question whose backtick references resolve to nothing is REFUSED with
+/// the paths named — not silently voted on whatever text is left. A confident
+/// answer computed from a payload the question never saw is the #1001 defect
+/// class in another coat.
+#[tokio::test]
+async fn a_backtick_reference_that_resolves_nowhere_is_named_not_guessed() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": {"message": "refund money back"},
+ "questions": {
+ "q1": {"type": "noul", "instructions": "Is `typo.path` relevant?"}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert_eq!(r["error"]["type"], "illegal_argument", "{r}");
+ let why = reason(&r);
+ assert!(why.contains("`q1`"), "{why}");
+ assert!(why.contains("`typo.path`"), "{why}");
+}
+
+/// A payload with no text at all is a 422 naming the question — not a vote
+/// on the instruction prose over a history that never saw the payload.
+#[tokio::test]
+async fn a_state_with_no_text_is_refused_by_name() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": {"spam_count": 42, "flags": [true, false]},
+ "questions": {"q1": {"type": "noul", "instructions": "classify"}}
+ }))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert_eq!(r["error"]["type"], "illegal_argument", "{r}");
+ let why = reason(&r);
+ assert!(why.contains("`q1`"), "{why}");
+ assert!(why.contains("state"), "{why}");
+}
+
+/// The data-field pattern — the question carries its payload inside
+/// `instructions` and names it in backticks, the shape xerj-rerank's stage
+/// sends and the TypeSafe docs prescribe. The referenced field IS the vote
+/// text, and a state whose vocabulary would pollute the vote stays out of
+/// it: the question selected its payload.
+#[tokio::test]
+async fn an_instructions_data_field_is_the_payload_and_the_state_stays_out() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": "shipping delivery times",
+ "questions": {
+ "d0": {"type": "noul", "instructions": {
+ "question": "Judge `document` against the query in the state.",
+ "document": "refund refund asked for money back"
+ }}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+ // The refund document votes `true` outright — the misleading state string
+ // never entered the query.
+ assert_eq!(r["answers"]["d0"]["noul"], json!(1.0), "{r}");
+ assert_eq!(r["decisions"]["evidence"]["d0"]["label"], "true", "{r}");
+}
+
+/// `jev-reranker` ≥ 0.1.2's `rerank_relevance()` preset ships its rubric
+/// object in `state` and names it in backticks ("Apply all remaining
+/// evaluation rules in `rubric`."). The rubric is judge rules, not payload:
+/// it must be acknowledged without its string leaves joining the vote —
+/// embedding them is exactly the #1000 defect (criteria-rich prose). The
+/// vote runs on the text references: the document and the query. On the
+/// pre-skip code this shape was a 422 ("`rubric` resolves to nothing").
+#[tokio::test]
+async fn a_structure_reference_like_the_clients_rubric_is_not_vote_text() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": {
+ "query": "triage inbox unsolicited correspondence",
+ "documents": {"doc_0": "shipping delivery times when does it arrive"},
+ "rubric": {
+ "instructions": "refund refund asked for money back prize winner claim",
+ "criteria": {"true": "retain", "false": "discard"}
+ }
+ },
+ "model": "jev-1.13.0",
+ "questions": {
+ "doc_0": {"type": "noul",
+ "instructions": "Does `documents.doc_0` help answer `query` and \
+ deserve a high position in its search results? Apply all \
+ remaining evaluation rules in `rubric`.",
+ "criteria": {"true": "retain", "false": "discard"}}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+ // The shipping document drives the vote to `false` outright — the rubric's
+ // refund/spam vocabulary never entered the query, and the neutral query's
+ // terms retrieve nothing over this history.
+ assert_eq!(r["answers"]["doc_0"]["noul"], json!(0.0), "{r}");
+ assert_eq!(r["decisions"]["evidence"]["doc_0"]["label"], "false", "{r}");
+}
+
/// A `choice` question answers with a probability for EVERY criterion option —
/// options with no neighbour weight appear as 0.0, not as missing keys — the
/// winner is the argmax, and its confidence is its own probability.
@@ -280,8 +454,9 @@ async fn choice_answers_carry_a_probability_for_every_option() {
)
.await;
- // state carries no `query` field, so the vote text is exactly the resolved
- // ticket — the same string /_decide is asked below, making the arithmetic
+ // The vote text is exactly what the question's `ticket` reference
+ // resolves to — instruction prose is the question, never the query — so
+ // it is the same string /_decide is asked below, making the arithmetic
// cross-checkable between the two surfaces.
let vote = "cancel my subscription";
let (st, r) = node
diff --git a/landing/blog/does-xerj-beat-jev.html b/landing/blog/does-xerj-beat-jev.html
index f368e3c76..8dd35feec 100644
--- a/landing/blog/does-xerj-beat-jev.html
+++ b/landing/blog/does-xerj-beat-jev.html
@@ -273,7 +273,7 @@
<div class="uc-hero">
<p class="blog-crumb"><a href="/blog/">&larr; ALL POSTS · /BLOG</a></p>
- <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, ANSWERED · RUNS 2026-09-19..20</div>
+ <div class="uc-eyebrow">BENCHMARKS · ONE JUDGE MODEL, MEASURED · ONE WIRE, ANSWERED · RUNS 2026-09-19..21</div>
<h1>DOES XERJ BEAT JEV?<br><span class="accent">ON THE BILL, OUTRIGHT. ON FIQA, NO.</span></h1>
<p class="blog-byline">AUTHOR · XERJ ENGINEERING TEAM · PUBLISHED · 2026-09-21 · UPDATED · 2026-09-21</p>
@@ -283,7 +283,7 @@
<div class="vh"><span class="wn">THE VERDICT</span>
<span class="wm">split, measured, with the losses left in</span></div>
<p>Not outright — and not behind, either: a split verdict, measured over <strong>1,271 judged BEIR queries</strong> (3 shuffled repeats each). Where reading beats word-overlap, Jev's hosted judge wins rerank: our BM25 top-30 reordered by Jev scores <strong>0.7410 nDCG@10 on SciFact</strong> (versus 0.6993 for our best zero-API-token arm, hybrid RRF) and <strong>0.3638 on FiQA</strong> (versus 0.2382 BM25 — a +0.126 lift, the biggest we measured and the clearest case where hosted Jev still wins); our local hybrid keeps NFCorpus, 0.3448 to the rerank path's 0.3312, and that losing row stays in the table rather than being hidden.</p>
- <p>On the bill XERJ wins outright: the FiQA rerank run cost <strong>15,178,758 judge input tokens — $0.6375</strong> at $0.042/Mtok, input only, p50 1.23 s per call — while XERJ answers the same wire (<code class="inline">/v1/systemone</code> and <code class="inline">/_decide</code>) locally with zero tokens, no model, and p50 42.7 ms on <code class="inline">/v1/systemone</code> (<code class="inline">/_decide</code>: 34.8 ms), and the unmodified pip-installed <code class="inline">jev-reranker</code> ran against a XERJ node and passed its own strict validation (spam 0.9014 versus ham 0.0956). AG News reaches 0.9182 accuracy on all 7,600 test items with no model and no tokens — no Jev row exists for AG News and none is claimed.</p>
+ <p>On the bill XERJ wins outright: the FiQA rerank run cost <strong>15,178,758 judge input tokens — $0.6375</strong> at $0.042/Mtok, input only, p50 1.23 s per call — while XERJ answers the same wire (<code class="inline">/v1/systemone</code> and <code class="inline">/_decide</code>) locally with zero tokens, no model, and p50 42.7 ms on <code class="inline">/v1/systemone</code> (<code class="inline">/_decide</code>: 34.8 ms), and the unmodified pip-installed <code class="inline">jev-reranker</code> ran against a XERJ node and passed its own strict validation through both of its entry points (spam 0.9447 versus ham 0.0905, client 0.1.2, re-run 2026-09-21). AG News reaches 0.9182 accuracy on all 7,600 test items with no model and no tokens — no Jev row exists for AG News and none is claimed.</p>
<p>Hold the caveats: hosted Jev's published SciFact 0.768 and NFCorpus 0.358 are not run by us and are not a race with our rows (different experiments sharing a metric); provider non-determinism makes any gap within ±0.01 nDCG@10 noise; and Jev's FiQA probabilities are badly calibrated (ECE 0.3109 — documents rated 0.93 confidence were relevant 34% of the time; order by them, never threshold them with <code class="inline">min_score</code>). Verdict: use Jev where the judge wins, FiQA-style rerank; use XERJ where tokens, latency, and data locality decide.</p>
</section>
@@ -544,7 +544,7 @@
</div>
<h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">The gate: the pip client, unmodified, on a local node</h3>
- <p class="wide">No SDK of ours, no patches, no subclassing. <code class="inline">pip install jev-reranker</code> (0.1.0, MIT) in a clean venv, its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. Two environment variables, and one of them is a trap:</p>
+ <p class="wide">No SDK of ours, no patches, no subclassing. <code class="inline">pip install jev-reranker</code> (0.1.2, MIT), its own listwise template, its own splitting, its own strict response validation, pointed at a XERJ node holding 4,000 labelled SMS messages. The gate ranks through both of the client's entry points — the constructor default, and the documented relevance preset, which since 0.1.2 ships its rubric as an <em>object</em> the vote must acknowledge without reading it as retrieval text. Two environment variables, and one of them is a trap:</p>
<pre class="code"><span class="prompt">$</span> pip install jev-reranker
<span class="prompt">$</span> <span class="cmt"># TYPESAFE_ENDPOINT carries the FULL path. TYPESAFE_BASE_URL is silently ignored</span>
@@ -552,20 +552,21 @@
<span class="prompt">$</span> export TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone
<span class="prompt">$</span> export TYPESAFE_API_KEY=local-only <span class="cmt"># the node ignores it; localhost only</span>
<span class="prompt">$</span> python3 gate_run.py
-<span class="out">client resolved model: ['xerj-history-vote-1']
+<span class="out">client: jev-reranker 0.1.2 (pip, unmodified)
+client resolved model: ['xerj-history-vote-1']
1.0000 spam Please call our customer service representative on 0800 169 6031 betwe
1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
0.0000 ham If you're not in my car in an hour and a half I'm going apeshit
0.0000 ham Carlos says we can pick up from him later so yeah we're set
- ... <span class="cmt"># 10 fixed docs, 5+5</span>
-mean noul spam=0.9014 ham=0.0956 gap=0.8058
+ ... <span class="cmt"># 10 fixed docs, 5+5, both entry points</span>
+mean noul spam=0.9447 ham=0.0905 gap=0.8542
GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre>
<p class="wide">The client's own validation <em>is</em> the wire gate. Answers keyed by the sent ids. Every noul numeric in 0..1. Model a non-empty string. Usage non-negative integers. Surviving it unmodified is the compatibility claim. The spam/ham separation is the semantic one. <strong>Zero requests left the machine.</strong> The vote is an ordinary search of an ordinary index, and the module adds no outbound client.</p>
<div class="loss">
- <strong>The first gate attempt failed. It is the most useful thing on this page.</strong> The wire puts the query in <code class="inline">state</code>, and the vote text joins it, so query vocabulary takes part in retrieval. With a spam-bearing query like "winner claim prize", <em>every</em> document's vote retrieved spam neighbours. The vote measured the query, not the documents: spam 1.000 against ham 0.785. No separation. The fixed gate uses a query whose every term is absent from the corpus: "triage inbox unsolicited correspondence". Result, 0.9014 against 0.0956. This is inherent to a retrieval vote, not a defect. <strong>Pick the query for the history you have.</strong> Do not let a rerank habit leak vocabulary into a classification question.
+ <strong>The first gate attempt failed. It is the most useful thing on this page.</strong> The wire puts the query in <code class="inline">state</code>, and the vote text joined it, so query vocabulary took part in retrieval. With a spam-bearing query like "winner claim prize", <em>every</em> document's vote retrieved spam neighbours. The vote measured the query, not the documents: spam 1.000 against ham 0.785. No separation. We first called that "inherent to a retrieval vote". <strong>It was not. It was two defects, and we fixed them.</strong> The vote text used to concatenate the instruction's prose into the retrieval query — wording alone moved accuracy 32 points (0.9867 with an empty instruction, 0.6700 with a criteria-rich one, same 300 held-out messages) — and an object state contributed only its <code class="inline">query</code> field, so <code class="inline">{"message": …}</code> voted on prose at the spam base rate (0.1733 against 0.9667 for the same message as a string). The vote text is now exactly what the question points at: its backtick references, or the state's own string content — never instruction prose, never state the question never named. After the fix, all eight wording/shape arms of the same 300 messages score identically (0.9767). One part <em>is</em> by design, and it is the client's choice, not ours: since 0.1.2 the client's own question references the query in backticks, so the query is named payload — the question being asked is "does this document answer this query". <strong>Pick the query for the history you have</strong> still holds: under the deliberately spammy query the gap narrows from 0.8542 to 0.6087, and the gate asserts both.
</div>
<h3 class="scene" style="font-size:var(--fs-24); margin-top:var(--sp-8);">Speed: the Rust side of the same wire</h3>
@@ -698,7 +699,7 @@ EOF
<span class="prompt">$</span> python3 -m venv ~/gate-venv &amp;&amp; ~/gate-venv/bin/pip install jev-reranker
<span class="prompt">$</span> TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone \
TYPESAFE_API_KEY=local-only ~/gate-venv/bin/python benchmarks/systemone-gate/gate_run.py
-<span class="out">mean noul spam=0.9014 ham=0.0956 gap=0.8058
+<span class="out">mean noul spam=0.9447 ham=0.0905 gap=0.8542
GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre>
<p class="src">Everything on this page: <a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot" rel="external">the pilot</a> ·
<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/beir-hybrid/results/2026-09-20-rerank-full" rel="external">the full run</a> ·
@@ -706,7 +707,7 @@ GATE PASSED: unmodified jev-reranker ranked from the XERJ node</span></pre>
<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/decisions-as-retrieval/results/2026-09-20-agnews" rel="external">the AG News vote</a> ·
<a href="https://github.com/xerj-org/xerj/tree/main/benchmarks/systemone-gate" rel="external">the gate</a> ·
<a href="https://github.com/xerj-org/xerj/blob/main/docs/research/typesafe-skill-2026-09/NOTES.md" rel="external">the research narrative</a> ·
- measured 2026-09-19..20 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance <!-- snapshot:2026-09-20 -->1371 passed / 0 failed on that branch (canonical: 1366 / 1369 today) ·
+ measured 2026-09-19..21 · engine at the wire-compat branch plus the WAND commit cf6a9989, ES-YAML conformance <!-- snapshot:2026-09-20 -->1371 passed / 0 failed on that branch (canonical: 1366 / 1369 today) ·
related: <a href="/benchmarks/">the benchmarks hub</a> · <a href="/answers/rerank-search-results-calibrated-judge">the rerank stage, documented</a> ·
<a href="/benchmarks/elasticsearch">the ES scorecard</a>.</p>
</section>