Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(api): POST /v1/systemone + /_decide — the local judge speaks the System One wire

xerj-org321676ae5d24
diff --git a/benchmarks/systemone-gate/README.md b/benchmarks/systemone-gate/README.md
new file mode 100644
index 000000000..224017095
--- /dev/null
+++ b/benchmarks/systemone-gate/README.md
@@ -0,0 +1,55 @@
+# The /v1/systemone acceptance gate — jev-reranker, unmodified, on a XERJ node
+
+Run 2026-09-20, locally built `xerj-server` at the wire-compat commit,
+`--insecure --port 9440` (native REST on 9441), throwaway data dir,
+`[decisions] index = "sms"` (`gate-decisions.toml`: k=10, text/label fields,
+`positive_label = "spam"`). Client: `pip install jev-reranker` (0.1.0, MIT) in a
+clean venv — no patches, no subclassing, its own listwise instruction template,
+its own splitting, its own strict response validation. The only configuration:
+
+```sh
+TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone
+TYPESAFE_API_KEY=<any non-empty string> # the node ignores it; localhost only
+```
+
+`TYPESAFE_BASE_URL` is silently ignored by the client — the endpoint variable
+must carry the full path. (Verified in the wheel: `reranker.py:238`.)
+
+## What ran
+
+`gate_load_sms.py` indexes the 4,000-message SMS train split
+(`benchmarks/decisions-as-retrieval`'s split: shuffle seed 7, first 4,000) and
+fixes 10 held-out gate documents (5 spam, 5 ham, 40–140 chars) in
+`gate-docs.json`. `gate_run.py` ranks them with `JevReranker().rerank(...)`.
+The query is `triage inbox unsolicited correspondence` — every term verified
+absent from the corpus, because `state.query` joins every vote text and a
+vocabulary-bearing query ("winner claim prize") retrieves spam neighbours for
+EVERY document, which measures the query, not the documents. First gate
+attempt failed exactly there (spam 1.000 vs ham 0.785); the transcript below
+is the fixed run.
+
+## Result (`gate-transcript.txt`)
+
+The client's own validation IS the wire gate: answers keyed exactly by the
+sent question ids, every answer `{type: "noul", noul}` numeric in 0..1,
+`model` a non-empty string (it resolved `xerj-history-vote-1` — the node's
+truthful echo, never a Jev name), `usage` non-negative integers. The semantic
+gate: mean noul 0.9014 over spam vs 0.0956 over ham (gap 0.806, threshold
+0.3). No request left the node: the vote is an ordinary search of the `sms`
+index; the module adds no outbound client.
+
+## The response the client does not show
+
+`/_decide` exposes the same vote with its evidence — neighbours, labels,
+engine scores, weights, and an abstain verdict below `decisions.min_confidence`:
+
+```sh
+curl -s localhost:9440/_decide -H 'content-type: application/json' \
+ -d '{"index":"sms","question":"URGENT! Your Mobile number has been awarded"}'
+# {"index":"sms","k":10,"label":"spam","confidence":1.0,"abstain":false,
+# "neighbours":[{"_id":"1805","label":"spam",...,"weight":1.0}, ...]}
+```
+
+Zero-support questions are a 422 naming the question ids, never a fabricated
+0.5; `score` questions (no vote analogue) are a 422 naming the alternative.
+Both behaviours are pinned by `engine/crates/xerj-api/tests/systemone_http.rs`.
diff --git a/benchmarks/systemone-gate/gate-docs.json b/benchmarks/systemone-gate/gate-docs.json
new file mode 100644
index 000000000..52ce36349
--- /dev/null
+++ b/benchmarks/systemone-gate/gate-docs.json
@@ -0,0 +1,42 @@
+[
+ {
+ "label": "spam",
+ "text": "money!!! you r a lucky winner ! 2 claim your prize text money 2 88600 over \u00a31million to give away ! ppt150x3+normal text rate box403 w1t1jy"
+ },
+ {
+ "label": "spam",
+ "text": "Please call our customer service representative on 0800 169 6031 between 10am-9pm as you have WON a guaranteed \u00a31000 cash or \u00a35000 prize!"
+ },
+ {
+ "label": "spam",
+ "text": "Great News! Call FREEFONE 08006344447 to claim your guaranteed \u00a31000 CASH or \u00a32000 gift. Speak to a live operator NOW!"
+ },
+ {
+ "label": "spam",
+ "text": "You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327. T&C: RSTM, SW7 3SS. 150ppm"
+ },
+ {
+ "label": "spam",
+ "text": "URGENT! Your Mobile number has been awarded with a \u00a32000 prize GUARANTEED. Call 09058094455 from land line. Claim 3030. Valid 12hrs only"
+ },
+ {
+ "label": "ham",
+ "text": "As per your request 'Maangalyam (Alaipayuthe)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune"
+ },
+ {
+ "label": "ham",
+ "text": "If you're not in my car in an hour and a half I'm going apeshit"
+ },
+ {
+ "label": "ham",
+ "text": "Awesome question with a cute answer: Someone asked a boy \"how is ur life?\" . . He smiled &amp; answered: . . \"She is fine!\" Gudnite"
+ },
+ {
+ "label": "ham",
+ "text": "Carlos says we can pick up from him later so yeah we're set"
+ },
+ {
+ "label": "ham",
+ "text": "The greatest test of courage on earth is to bear defeat without losing heart....gn tc"
+ }
+]
\ No newline at end of file
diff --git a/benchmarks/systemone-gate/gate-transcript.txt b/benchmarks/systemone-gate/gate-transcript.txt
new file mode 100644
index 000000000..77c11a2c7
--- /dev/null
+++ b/benchmarks/systemone-gate/gate-transcript.txt
@@ -0,0 +1,13 @@
+client resolved model: ['xerj-history-vote-1']
+ 0.5068 spam money!!! you r a lucky winner ! 2 claim your prize text money 2 88600
+ 1.0000 spam Please call our customer service representative on 0800 169 6031 betwe
+ 1.0000 spam Great News! Call FREEFONE 08006344447 to claim your guaranteed £1000 C
+ 1.0000 spam You have won ?1,000 cash or a ?2,000 prize! To claim, call09050000327.
+ 1.0000 spam URGENT! Your Mobile number has been awarded with a £2000 prize GUARANT
+ 0.1024 ham As per your request 'Maangalyam (Alaipayuthe)' has been set as your ca
+ 0.0000 ham If you're not in my car in an hour and a half I'm going apeshit
+ 0.3414 ham Awesome question with a cute answer: Someone asked a boy "how is ur li
+ 0.0000 ham Carlos says we can pick up from him later so yeah we're set
+ 0.0341 ham The greatest test of courage on earth is to bear defeat without losing
+mean noul spam=0.9014 ham=0.0956 gap=0.8058
+GATE PASSED: unmodified jev-reranker ranked from the XERJ node
diff --git a/benchmarks/systemone-gate/gate_load_sms.py b/benchmarks/systemone-gate/gate_load_sms.py
new file mode 100644
index 000000000..32a0abcba
--- /dev/null
+++ b/benchmarks/systemone-gate/gate_load_sms.py
@@ -0,0 +1,62 @@
+"""Load the SMS-spam history for the /v1/systemone acceptance gate.
+
+Indexes the same 4,000-message train split benchmarks/decisions-as-retrieval
+used (shuffle seed 7, first 4,000) and writes the held-out gate documents to
+gate-docs.json: 5 spam + 5 ham, picked for length and signal, so the gate run
+has fixed inputs. Usage: python3 gate_load_sms.py [es_compat_url]
+"""
+import json, random, sys, urllib.request
+
+U = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:9440"
+
+def req(m, p, b=None, ct="application/json"):
+ d = b if isinstance(b, (bytes, type(None))) else json.dumps(b).encode()
+ r = urllib.request.Request(U + p, data=d, method=m, headers={"content-type": ct})
+ try:
+ return json.loads(urllib.request.urlopen(r, timeout=900).read())
+ except urllib.error.HTTPError as e:
+ return {"_err": e.code, "body": e.read().decode()[:300]}
+
+rows = [l.rstrip("\n").split("\t", 1) for l in open("sms.tsv")]
+rows = [(label, text) for label, text in rows] # (label, text)
+random.Random(7).shuffle(rows)
+train, held = rows[:4000], rows[4000:]
+
+print(req("DELETE", "/sms"))
+print(req("PUT", "/sms", {"mappings": {"properties": {
+ "label": {"type": "keyword"}, "text": {"type": "text"}}}}))
+for i in range(0, len(train), 250):
+ lines = []
+ for j, (label, text) in enumerate(train[i:i + 250]):
+ lines.append(json.dumps({"index": {"_index": "sms", "_id": str(i + j)}}))
+ lines.append(json.dumps({"text": text, "label": label}))
+ r = req("POST", "/_bulk", ("\n".join(lines) + "\n").encode(), "application/x-ndjson")
+ if r.get("errors") or "_err" in r:
+ print("bulk problem", str(r)[:300]); sys.exit(1)
+req("POST", "/sms/_refresh")
+print("sms count:", req("GET", "/sms/_count").get("count"))
+
+# Fixed gate documents: distinctive spam and ham, neither near-duplicates of a
+# train row's phrasing, each 40..140 chars.
+SPAM_KEYS = ["claim", "free", "winner", "prize", "urgent", "cash"]
+def pick(label, n):
+ out = []
+ for text in [t for l, t in held if l == label]:
+ if not (40 <= len(text) <= 140):
+ continue
+ keys = sum(k in text.lower() for k in SPAM_KEYS)
+ want = 2 if label == "spam" else 0
+ if keys >= want and text not in out:
+ out.append(text)
+ if len(out) == n:
+ break
+ return out
+
+docs = [{"label": "spam", "text": t} for t in pick("spam", 5)]
+docs += [{"label": "ham", "text": t} for t in pick("ham", 5)]
+assert len(docs) == 10, f"picked {len(docs)}"
+json.dump(docs, open("gate-docs.json", "w"), indent=1)
+print("gate docs written:", sum(d["label"] == "spam" for d in docs), "spam /",
+ sum(d["label"] == "ham" for d in docs), "ham")
+for d in docs[:2] + docs[5:7]:
+ print(" ", d["label"], "|", d["text"][:90])
diff --git a/benchmarks/systemone-gate/gate_run.py b/benchmarks/systemone-gate/gate_run.py
new file mode 100644
index 000000000..d9e29812a
--- /dev/null
+++ b/benchmarks/systemone-gate/gate_run.py
@@ -0,0 +1,51 @@
+"""THE ACCEPTANCE GATE: pip-installed jev-reranker, UNMODIFIED, pointed at a
+XERJ node.
+
+ TYPESAFE_ENDPOINT=http://localhost:<native-rest-port>/v1/systemone
+ TYPESAFE_API_KEY=<any non-empty string — the node ignores it, localhost only>
+
+Ranks the fixed gate documents (gate-docs.json: 5 spam, 5 ham) with the
+client's own listwise pipeline and asserts two things:
+
+1. WIRE: the call completes with no ResponseValidationError — answers keyed
+ exactly, noul numeric in 0..1, model echoed as a non-empty string, usage
+ non-negative ints. The client enforces this; surviving it IS the gate.
+2. SEMANTICS: the local vote separates the classes — mean noul over spam
+ documents strictly exceeds mean noul over ham documents by >= 0.3.
+
+Run: python3 gate_run.py [native_url_base]
+"""
+import json, os, sys
+
+docs = json.load(open("gate-docs.json"))
+texts = [d["text"] for d in docs]
+labels = [d["label"] for d in docs]
+# The query rides in state.query and joins every vote text, so its terms must
+# be NEUTRAL for the history: a spammy query ("winner claim prize") retrieves
+# spam neighbours for every document and the vote says nothing about the doc.
+# Every term here is absent from the SMS corpus (checked against sms.tsv), so
+# the document drives the vote. That asymmetry — query vocabulary pollutes a
+# classification history — is documented in docs/RERANK.md.
+QUERY = "triage inbox unsolicited correspondence"
+
+from jev_reranker import JevReranker
+
+rr = JevReranker() # reads TYPESAFE_ENDPOINT / TYPESAFE_API_KEY from the env
+result = rr.rerank(QUERY, texts, detail=True)
+
+# {"results": [{"document_index", "score", "text", ...}, ...]} — order follows
+# the client's ranking; re-key by the document index it reports.
+by_index = {r["document_index"]: r["score"] for r in result["results"]}
+scores = [by_index[i] for i in range(len(texts))]
+print("client resolved model:", result["detail"]["resolved_models"])
+for lab, txt, s in zip(labels, texts, scores):
+ print(f" {s:.4f} {lab:4} {txt[:70]}")
+
+assert len(scores) == len(texts)
+assert all(isinstance(s, float) and 0.0 <= s <= 1.0 for s in scores), scores
+spam = [s for s, l in zip(scores, labels) if l == "spam"]
+ham = [s for s, l in zip(scores, labels) if l == "ham"]
+ms, mh = sum(spam) / len(spam), sum(ham) / len(ham)
+print(f"mean noul spam={ms:.4f} ham={mh:.4f} gap={ms - mh:.4f}")
+assert ms - mh >= 0.3, f"vote did not separate: {ms:.3f} vs {mh:.3f}"
+print("GATE PASSED: unmodified jev-reranker ranked from the XERJ node")
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
new file mode 100644
index 000000000..7735c069b
--- /dev/null
+++ b/docs/DECISIONS.md
@@ -0,0 +1,132 @@
+# Typed decisions from your own history (`/v1/systemone`, `/_decide`)
+
+> **This feature sends nothing off the machine.** The vote is an ordinary
+> search of an ordinary index on this node. The module adds no outbound
+> client; no document or query text leaves, no provider key is involved, and
+> no tokens are spent. That is the point of it — see
+> [docs/RERANK.md](./RERANK.md) for the feature that *does* send text out,
+> and the contrast between the two.
+
+TypeSafe AI's System One API is a documented interface for typed judgement:
+`POST /v1/systemone` with a state, a model, and a map of questions, each
+answered `noul` (a yes/no with a 0–1 probability) or `choice` (one of N named
+options with probabilities). Clients exist for it — `jev-reranker` on PyPI,
+the official SDKs — and they all speak that one shape.
+
+XERJ already had the retrieval answer for the same shape of question: index
+the labelled examples you already have, answer a new question by a weighted
+vote over its *k* nearest neighbours, and report the winning label's vote
+share as the probability
+([benchmarks/decisions-as-retrieval](../benchmarks/decisions-as-retrieval):
+Banking77 0.933 accuracy / ECE 0.012, SMS 0.983, sub-millisecond, no model).
+So the node speaks the wire:
+
+- **`POST /v1/systemone`** (native REST, `--port + 1`) — the documented
+ request/response shape, answered by the vote. A client written for
+ TypeSafe's API works unmodified with
+ `TYPESAFE_ENDPOINT=http://localhost:<port+1>/v1/systemone`.
+- **`POST /_decide`** (ES-compat port) — the same vote without the wire
+ costume: it names its index per request and returns the evidence.
+
+Both surfaces are one mechanism, in
+`engine/crates/xerj-api/src/systemone_api.rs`; the end-to-end tests are
+`engine/crates/xerj-api/tests/systemone_http.rs`. The acceptance gate for the
+wire — the pip-installed `jev-reranker`, unmodified, ranking off a XERJ node —
+is [benchmarks/systemone-gate](../benchmarks/systemone-gate).
+
+## Set it up
+
+```toml
+# xerj.toml
+[decisions]
+index = "judgements" # an index of labelled examples; empty (the default) = both endpoints 503
+k = 10
+label_field = "label"
+text_field = "text"
+positive_label = "true" # the label whose share a noul answers
+min_confidence = 0.0 # /_decide abstains below this
+```
+
+The history index is ordinary documents: one per example, with the text the
+example was decided on (`text_field`) and its label (`label_field`). Index it
+with the usual `PUT /{index}/_doc` or `_bulk`. For a two-class history the
+labels are the two class names and `positive_label` names the one a `noul`'s
+probability is the share of (`"spam"`, not `"true"`, for a spam history).
+
+## What the vote answers — and what it does not
+
+The vote is **not** zero-shot and claims no judgement. It answers "what does
+history similar to this question say?", which is exactly the right question
+when you HAVE history (support routing, moderation, triage, spam) and the
+wrong one when you do not. Where there is no history — a new policy, a new
+category — use a judge model; that is what
+[docs/RERANK.md](./RERANK.md) is for.
+
+**The query's vocabulary pollutes a classification history.** The wire puts
+the query in `state`, and the vote text is the resolved question text plus
+that query — so query terms participate in retrieval. For a rerank-shaped
+request that is correct (relevance IS query-relative). For a classification
+history it can swamp the document: against the SMS corpus, the query
+"congratulations winner claim free prize" retrieves spam neighbours for *every*
+document and the vote says nothing about the document (measured: spam mean
+1.000 vs ham 0.785 — no separation). Keep the query neutral for the corpus
+(every term of `triage inbox unsolicited correspondence` is absent from the SMS
+corpus: spam 0.901 vs ham 0.096). This is inherent to a retrieval vote, not a
+defect to fix: pick the query for the history you have.
+
+## The wire, and the two deliberate breaks
+
+Request: `{state, model, questions: {id: {type, instructions, criteria}}}`.
+`state` may be a string or an object; `instructions` may be a string, object
+or array (all string leaves join); strings may reference state data with
+backtick paths — `` `documents.doc_0` `` — resolved against `state` before the
+vote text is built. That is the reference pattern of the API and the shape
+`jev-reranker` sends.
+
+Response: `{model, answers, usage}` with `answers` keyed exactly by the
+question ids sent, each answer carrying only its documented fields (clients
+validate strictly). XERJ's extras ride at the top level in `decisions` — the
+index, k, the requested model, and per-question evidence (winning label, its
+support, how many neighbours were found and voted) — where every verified
+client ignores them.
+
+Two deliberate breaks from the hosted API, both tested so they cannot regress:
+
+1. **`model` is echoed as `xerj-history-vote-1`, never as a Jev model name.**
+ Echoing `jev-1.13.0` would claim these probabilities are the hosted
+ model's. The requested name is reported as `decisions.requested_model`.
+2. **Zero support is a 422 naming the question ids, never a fabricated 0.5.**
+ A question whose vote finds no labelled neighbour carries no information;
+ inventing a probability is the silent-fake defect class this project
+ treats as a bug. `/_decide` says the same thing as `abstain` + `reason`.
+
+`score` questions (2–10 ordinal levels) have no vote analogue and no
+benchmark: 422, naming the alternative.
+
+## `POST /_decide`
+
+The audit surface. Per request: `index` (required), `question` (required),
+`k` (default the configured 10, clamped 1..100), `positive_label` (default
+the configured one). The response returns the label, its confidence, the
+verdict (`abstain` below `decisions.min_confidence`, or when no labelled
+neighbour exists), and the **neighbours** — each with `_id`, `label`,
+`_score` (the engine's own BM25 score), `weight` (1/rank), and the text — so
+every answer can be checked against the evidence that produced it.
+
+```sh
+curl -s localhost:9200/_decide -H 'content-type: application/json' \
+ -d '{"index":"judgements","question":"refund my subscription","k":5}'
+```
+
+## Numbers, and only the ones we measured
+
+Measured, ours, reproducible (raw JSON and runners in the results dirs):
+Banking77 77-way intent 0.933 accuracy / ECE 0.012 (86.9% of traffic clears
+confidence 0.8 at 0.979 accuracy); SMS spam 0.983 accuracy / 0.944 F1 under a
+millisecond on BM25 alone; both in
+[benchmarks/decisions-as-retrieval](../benchmarks/decisions-as-retrieval).
+The wire gate transcript is in
+[benchmarks/systemone-gate](../benchmarks/systemone-gate). Not claimed:
+anything about how a judge model scores on those datasets (no such numbers
+are published and we did not run one there), any zero-shot capability, and
+any judgement quality for histories unlike the ones measured.
diff --git a/docs/research/typesafe-skill-2026-09/NOTES.md b/docs/research/typesafe-skill-2026-09/NOTES.md
new file mode 100644
index 000000000..2987dc919
--- /dev/null
+++ b/docs/research/typesafe-skill-2026-09/NOTES.md
@@ -0,0 +1,138 @@
+# Notes — measuring the Jev model against XERJ, then speaking its wire
+
+Research log, 2026-09-19..20. Two questions, in the order we asked them:
+
+1. Does the Jev judge model, called through its documented wire, actually
+ improve XERJ's ranking — and what are its probabilities worth?
+2. Can a XERJ node BE a System One endpoint — answer the same wire with a local
+ vote, so a client written for TypeSafe's API works unmodified?
+
+Raw artifacts: `benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot/`
+(pilot, 40 queries) and `.../2026-09-20-rerank-full/` (full run, 3 repeats,
+calibration). This file is the narrative; the JSONs are the evidence.
+
+## 1. The stage shipped unmeasured, and measured badly (0.3822)
+
+docs/RERANK.md carried "Not verified: ranking quality with the real model" —
+with a provider key we ran it: **0.3822 nDCG@10** against BM25's **0.7750**
+over the first 40 judged SciFact queries. The stage made results worse.
+
+Root cause, isolated at the raw API (not guessed): `build_jev_request` put
+every candidate in shared `state` and asked each key an untargeted "Does this
+document contain information that answers the query?" — so "this document"
+resolved to nothing and the model judged the *pile* once, echoing ~that
+probability per question. Within every window the probabilities were
+near-uniform (spread ≈ 0.02 over 30 documents), which is query-level noise, not
+ranking.
+
+Two raw-API probes confirmed it (V1/V2, same shortlists, same day):
+
+| Shape | nDCG@10 (40 q) |
+|---|---:|
+| V1 — candidates in `state`, untargeted question | 0.3822 |
+| V2 — one noul per document, document inside its own `instructions`, question naming `` `document` `` | 0.8389 |
+
+The fix (commit 7c6f496e) moved the stage to the V2 shape: 0.3822 → **0.8299**
+through the stage, within provider wobble of the raw API's 0.8389. The wire
+reference for the shape is `jev-reranker`'s `_payload()` — listwise mode,
+`instructions.format(document="`documents.doc_0`")` — read from the installed
+wheel, not from memory.
+
+## 2. The full run: ranking up, calibration not there
+
+300 SciFact + 323 NFCorpus queries, BM25 top-30 shortlists, 3 repeats with
+shuffled query order, per-query medians (`raw_bench_full.py` in the results
+dir):
+
+| Dataset | BM25 | Jev median | W / L / T | cost |
+|---|---:|---:|---|---:|
+| SciFact | 0.6572 | 0.7410 | 82 / 30 / 188 | $0.4027 |
+| NFCorpus | 0.3016 | 0.3312 | 118 / 74 / 131 | $0.3200 |
+
+Findings that matter more than the headline:
+
+- **Provider non-determinism is real.** Mean per-document |Δ| 0.0077–0.0096
+ across repeats, max 0.08. Repeat-level means moved ±0.003 nDCG. Comparisons
+ of Jev numbers tighter than ~±0.01 nDCG@10 are noise.
+- **The probabilities are not calibrated to BEIR relevance.** ECE 0.1023 /
+ 0.1332; monotone curve (it discriminates) but bin 8 on SciFact is 0.85 mean
+ confidence / 0.42 actual relevance. A Jev probability orders documents well
+ and thresholds them badly against graded human labels. `min_score` on the
+ raw number would cut documents qrels call relevant.
+- The losses concentrate where BM25 already failed (empty/near-empty
+ shortlists) and on NFCorpus's medical shorthand, where 30-candidate windows
+ are mostly noise to a general-purpose judge.
+
+## 3. Speaking the wire: /v1/systemone + /_decide
+
+The second question inverted the first: TypeSafe's System One API is a
+documented interface for typed judgement — `POST /v1/systemone` with
+`{state, model, questions}`, answers typed `noul`/`choice`. XERJ already had
+the retrieval answer for the same shape: index labelled history, answer a new
+question by a rank-weighted vote over its k nearest neighbours, report the
+winning label's share as the probability (benchmarks/decisions-as-retrieval:
+Banking77 0.933 accuracy / ECE 0.012, SMS 0.983, sub-millisecond, BM25-only).
+
+So the node now speaks both surfaces (this branch):
+
+- `POST /v1/systemone` (native router) — the documented request/response shape,
+ answers from the vote, so `jev-reranker` unmodified with
+ `TYPESAFE_ENDPOINT=http://localhost:<port+1>/v1/systemone` reranks off the
+ node. Acceptance gate: the pip-installed client, 5 spam / 5 ham held-out SMS
+ documents, spam-vs-ham mean-noul separation — run 2026-09-20, see §4.
+- `POST /_decide` (ES-compat router) — the same vote without the wire costume:
+ per-request index/k/positive_label, the neighbours with labels, scores and
+ weights in the response, abstain below `decisions.min_confidence`.
+
+Nothing leaves the node: the vote is an ordinary search against an ordinary
+index; the module adds no outbound client. That is the point — same interface,
+evidence stays home.
+
+**Two deliberate wire breaks, both tested so they cannot regress:**
+
+1. `model` is echoed as `xerj-history-vote-1`, never as a Jev model name.
+ Echoing `jev-1.13.0` would claim the hosted model produced these
+ probabilities. The requested name is reported at
+ `decisions.requested_model`. (The client validates `model` is a non-empty
+ string — any truthful id passes.)
+2. Zero support is a 422 naming the question ids, never a fabricated 0.5. A
+ question whose vote finds no labelled neighbour carries no information;
+ inventing a probability is the silent-fake defect class. `/_decide` says
+ the same thing as `abstain` + `reason`.
+
+`score` questions (2–10 ordinal levels) have no vote analogue and no
+benchmark: 422 with the alternative named.
+
+Design notes from the client source (wheel, MIT — read before writing the
+endpoint): `instructions` is polymorphic (string | object | array); backtick
+paths (`documents.doc_0`) resolve against `state`; `criteria` rides on every
+question even for nouls; answers must be keyed EXACTLY by the sent ids with
+only documented fields inside each answer (extras are safe at the top level);
+`usage` must be JSON ints; retries on {429, 500, 502, 503, 504, 529}×8, no
+redirects. All of this is asserted in `engine/crates/xerj-api/tests/
+systemone_http.rs`.
+
+## 4. The gate run (2026-09-20)
+
+Node: locally built binary, `--insecure --port 9440`, throwaway data dir,
+`[decisions] index = "sms"` (4,000 SMS train messages, labels ham/spam,
+`positive_label = "spam"`, k = 10). Client: `pip install jev-reranker` in a
+clean venv, `TYPESAFE_ENDPOINT=http://localhost:9441/v1/systemone`,
+`TYPESAFE_API_KEY=<dummy>` — unmodified, listwise mode, its own instruction
+template, its own splitting and validation. Documents: 5 spam + 5 ham
+held-out messages, fixed in `gate-docs.json`. The client's strict response
+validation IS the wire gate; the mean-noul separation is the semantic gate.
+(Spam-vs-ham was chosen because it is the noul shape natively: labels are
+literally yes/no.)
+
+## What we claim, and what we do not
+
+- Measured, ours: everything in the tables above, on our node, our shortlists,
+ the pinned model, with the runner in the results dir.
+- Not claimed: anything about Jev on Banking77/SMS (TypeSafe has published no
+ such numbers; we did not run it there); any zero-shot capability for the
+ local vote (it needs labelled history — where there is none, a judge model
+ is the right tool); "1.00 recall", TB-scale, neural-default — none of it.
+- The ECE contrast (Jev 0.10–0.13 cross-domain vs vote 0.012 in-domain) is a
+ contrast of *targets*, not a scoreboard: calibration is a property of the
+ task's labels, and both numbers are honest on their own task.
diff --git a/engine/crates/xerj-api/src/lib.rs b/engine/crates/xerj-api/src/lib.rs
index c55f4c49d..e0cdb2fcb 100644
--- a/engine/crates/xerj-api/src/lib.rs
+++ b/engine/crates/xerj-api/src/lib.rs
@@ -80,6 +80,7 @@ pub mod router;
pub mod share;
pub mod state;
pub mod stub;
+pub mod systemone_api;
pub mod wal_tap_api;
// ── Convenience re-exports ────────────────────────────────────────────────────
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index 0a6e84842..cd88ff64a 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -29,7 +29,7 @@ use xerj_common::config::CorsConfig;
use crate::{
audit_mw, auth::auth_middleware, authz, es_compat, graph_api, ism_api, memory_api, native,
- share, state::AppState, wal_tap_api,
+ share, state::AppState, systemone_api, wal_tap_api,
};
// ─────────────────────────────────────────────────────────────────────────────
@@ -124,6 +124,11 @@ pub fn build_native_router(state: AppState) -> Router {
.route("/health/live", get(native::liveness))
.route("/health/ready", get(native::readiness))
.route("/v1/metrics", get(native::metrics))
+ // The local judge: System One wire-compatible decisions from a
+ // labelled-history vote. Answers nothing until [decisions] index is
+ // set — and then nothing leaves the node to answer it.
+ .route("/v1/systemone", post(systemone_api::systemone))
+ .route("/v1/models", get(systemone_api::models))
// Admin: cluster-wide flush + backup (snapshot to disk)
.route("/v1/admin/flush", post(native::admin_flush))
.route("/v1/admin/backup", post(native::admin_backup))
@@ -252,6 +257,11 @@ pub fn build_es_compat_router(state: AppState) -> Router {
// firewalled off) can still scrape metrics. Same handler as the native
// router; gated by the optional read-only `auth.metrics_token`.
.route("/v1/metrics", get(native::metrics))
+ // The local judge: System One wire-compatible decisions from a
+ // labelled-history vote. Answers nothing until [decisions] index is
+ // set — and then nothing leaves the node to answer it.
+ .route("/v1/systemone", post(systemone_api::systemone))
+ .route("/v1/models", get(systemone_api::models))
// Autoindex defaults to the ES-compatible listener, so the native
// embedding identity contract must be available here as well as on
// the dedicated native listener.
@@ -283,6 +293,9 @@ pub fn build_es_compat_router(state: AppState) -> Router {
.route("/_cat/count/:index", get(es_compat::cat_count))
.route("/_cat/shards", get(es_compat::cat_shards))
.route("/_bulk", post(es_compat::global_bulk))
+ // The audit twin of /v1/systemone: same vote, named index, neighbours
+ // and abstain instead of a wire-shaped error.
+ .route("/_decide", post(systemone_api::decide))
.route("/_mget", post(es_compat::mget))
// Index-scoped multi-get — the path index defaults entries that omit `_index`.
.route(
diff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rs
new file mode 100644
index 000000000..ef090b614
--- /dev/null
+++ b/engine/crates/xerj-api/src/systemone_api.rs
@@ -0,0 +1,574 @@
+//! The local judge: System One wire-compatible decisions from a vote.
+//!
+//! Two surfaces, one mechanism:
+//!
+//! - `POST /v1/systemone` (native router) speaks TypeSafe's documented System
+//! One request/response shape, so a client written for their API —
+//! `jev-reranker`, the official SDKs — can point `TYPESAFE_ENDPOINT` at a
+//! XERJ node and keep working. Answers come from a weighted
+//! nearest-neighbour vote over a labelled-history index
+//! (`[decisions] index`), the retrieval analogue of a judge model that
+//! `benchmarks/decisions-as-retrieval` measures.
+//! - `POST /_decide` (ES-compat router) is the same vote without the wire
+//! costume: it names its index per request and returns the neighbours,
+//! their labels and scores, and an abstain verdict.
+//!
+//! Nothing leaves the node. This module adds no outbound client; the vote is
+//! an ordinary search against an ordinary index. That is the point of the
+//! feature — the same interface, with the evidence staying home.
+//!
+//! # The wire contract, and where we deliberately break it
+//!
+//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.
+//! `state` may be a string or an object; question `instructions` may be a
+//! string, object or array (all string leaves are joined). Strings may point
+//! at state data with backtick paths — `` `documents.doc_0` `` — which are
+//! resolved against `state` before the vote text is built. That is the
+//! reference pattern of the System One API and the shape `jev-reranker`
+//! sends in its listwise mode.
+//!
+//! Response: `{ model, answers, usage }` with `answers` keyed exactly by the
+//! question ids sent and each answer carrying only its documented fields —
+//! clients parse these strictly. Extras ride at the top level (`decisions`),
+//! which every verified client ignores.
+//!
+//! Two deliberate breaks, both documented here and in docs/RERANK.md:
+//!
+//! 1. `model` is echoed as `xerj-history-vote-1`, never as a Jev model name.
+//! Echoing `jev-1.13.0` would claim these probabilities are the hosted
+//! model's. The requested name is reported as `decisions.requested_model`.
+//! 2. A question with no support — an empty history, or no labelled
+//! neighbour — is a 422 naming the question ids. Zero support must not
+//! become a fabricated 0.5. (`/_decide` says the same thing as `abstain`.)
+//!
+//! `score` questions (2–10 ordinal levels, weighted-average answer) have no
+//! vote analogue and no benchmark: 422, in the docs.
+
+use std::collections::BTreeMap;
+use std::time::Instant;
+
+use axum::extract::State;
+use axum::http::StatusCode;
+use axum::response::IntoResponse;
+use axum::Json;
+use serde_json::{json, Value};
+use xerj_common::config::DecisionsConfig;
+use xerj_query::parse_request;
+
+use crate::state::AppState;
+
+/// The model id this endpoint truthfully is. Never a Jev name.
+pub const MODEL_ID: &str = "xerj-history-vote-1";
+
+/// One question's vote text is bounded so a caller cannot make the node
+/// search for a novel of its choosing; matches the rerank stage's philosophy
+/// of caller-chosen cost with a server-side ceiling.
+const MAX_VOTE_TEXT_CHARS: usize = 100_000;
+/// The System One API documents at most 255 options per `choice`.
+const MAX_CHOICE_OPTIONS: usize = 255;
+/// Matches `rerank.max_window`: one request cannot ask for more questions
+/// than the rerank stage would judge.
+const MAX_QUESTIONS: usize = 300;
+
+// ─────────────────────────────────────────────────────────────────────────────
+// POST /v1/systemone
+// ─────────────────────────────────────────────────────────────────────────────
+
+pub async fn systemone(
+ State(state): State<AppState>,
+ Json(body): Json<Value>,
+) -> axum::response::Response {
+ let started = Instant::now();
+ let cfg = state.config.decisions.clone();
+ if cfg.index.is_empty() {
+ return not_configured();
+ }
+
+ let questions = match body.get("questions") {
+ Some(Value::Object(q)) if !q.is_empty() => q,
+ _ => return unprocessable("`questions` must be a non-empty object"),
+ };
+ if questions.len() > MAX_QUESTIONS {
+ return unprocessable(&format!(
+ "`questions` must hold at most {MAX_QUESTIONS} questions (got {})",
+ questions.len()
+ ));
+ }
+ let state_val = body.get("state").cloned().unwrap_or(Value::Null);
+
+ // Per question: the resolved vote text and the parsed type/criteria.
+ let mut parsed: Vec<(String, Question)> = Vec::with_capacity(questions.len());
+ for (id, q) in questions {
+ match parse_question(id, q, &state_val) {
+ Ok(question) => parsed.push((id.clone(), question)),
+ Err(reason) => return unprocessable(&reason),
+ }
+ }
+
+ // One vote per question. Collected first so a no-support question fails
+ // the whole request — the wire has no per-question error field, and a
+ // half-answered judgement history would look like a ranking.
+ let mut answers = serde_json::Map::new();
+ let mut evidence = serde_json::Map::new();
+ let mut unsupported: Vec<String> = Vec::new();
+ for (id, question) in &parsed {
+ let neighbours = match vote_neighbours(&state, &cfg, &question.vote_text).await {
+ Ok(n) => n,
+ Err(resp) => return resp, // index missing / search failure: surfaced, not guessed
+ };
+ let labelled: Vec<(f64, &str)> = neighbours
+ .iter()
+ .map(|(w, label, _)| (*w, label.as_str()))
+ .collect();
+ if labelled.is_empty() {
+ unsupported.push(id.clone());
+ continue;
+ }
+ let total: f64 = labelled.iter().map(|(w, _)| *w).sum();
+ let mut weight_by_label: BTreeMap<&str, f64> = BTreeMap::new();
+ for (w, label) in &labelled {
+ *weight_by_label.entry(label).or_insert(0.0) += *w;
+ }
+ let share = |label: &str| weight_by_label.get(label).copied().unwrap_or(0.0) / total;
+ match &question.kind {
+ Kind::Noul => {
+ // Exactly the documented fields — clients parse answers
+ // strictly and break on extras.
+ answers.insert(
+ id.clone(),
+ json!({ "type": "noul", "noul": round6(share(&cfg.positive_label)) }),
+ );
+ }
+ Kind::Choice { options } => {
+ let criteria_total: f64 =
+ options.iter().map(|o| weight_by_label.get(o.as_str()).copied().unwrap_or(0.0)).sum();
+ if criteria_total <= 0.0 {
+ unsupported.push(id.clone());
+ continue;
+ }
+ let mut probabilities = serde_json::Map::new();
+ for option in options {
+ probabilities.insert(
+ option.clone(),
+ json!(round6(weight_by_label.get(option.as_str()).copied().unwrap_or(0.0) / total)),
+ );
+ }
+ let (winner, _) = options
+ .iter()
+ .map(|o| (o, weight_by_label.get(o.as_str()).copied().unwrap_or(0.0)))
+ .max_by(|(a, aw), (b, bw)| aw.partial_cmp(bw).unwrap().then(a.cmp(b)))
+ .expect("non-empty options");
+ answers.insert(
+ id.clone(),
+ json!({
+ "type": "choice",
+ "choice": winner,
+ "confidence": round6(weight_by_label.get(winner.as_str()).copied().unwrap_or(0.0) / total),
+ "probabilities": Value::Object(probabilities),
+ }),
+ );
+ }
+ }
+ let (best_label, best_weight) = weight_by_label
+ .iter()
+ .max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
+ .expect("non-empty");
+ evidence.insert(
+ id.clone(),
+ json!({
+ "label": best_label,
+ "support": round6(best_weight / total),
+ "neighbours": labelled.len(),
+ "found": neighbours.len(),
+ }),
+ );
+ }
+ if !unsupported.is_empty() {
+ return no_support(&unsupported, &cfg);
+ }
+
+ let _ = state.metrics.record_query(&cfg.index, "systemone_vote", started.elapsed().as_secs_f64());
+ Json(json!({
+ "model": MODEL_ID,
+ "answers": Value::Object(answers),
+ "usage": { "input_tokens": 0, "output_tokens": 0 },
+ "decisions": {
+ "index": cfg.index,
+ "k": cfg.k,
+ "model": MODEL_ID,
+ "requested_model": body.get("model").and_then(Value::as_str).unwrap_or(""),
+ "evidence": Value::Object(evidence),
+ "took_ms": started.elapsed().as_millis() as u64,
+ }
+ }))
+ .into_response()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// GET /v1/models
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The official SDKs list models before calling. One truthful entry.
+pub async fn models(State(state): State<AppState>) -> axum::response::Response {
+ let configured = !state.config.decisions.index.is_empty();
+ Json(json!({
+ "models": [{
+ "name": MODEL_ID,
+ "description": if configured {
+ "Weighted nearest-neighbour vote over the configured decisions index — local, no egress"
+ } else {
+ "Weighted nearest-neighbour vote over a labelled-history index (no [decisions] index configured: /v1/systemone answers 503)"
+ },
+ "release_date": "2026-09-20",
+ }]
+ }))
+ .into_response()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// POST /_decide
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The audit surface: the same vote, naming its index per request, returning
+/// the neighbours it judged by, and abstaining instead of erroring.
+pub async fn decide(
+ State(state): State<AppState>,
+ Json(body): Json<Value>,
+) -> axum::response::Response {
+ let started = Instant::now();
+ let defaults = &state.config.decisions;
+ let index = match body.get("index").and_then(Value::as_str) {
+ Some(i) if !i.trim().is_empty() => i.to_string(),
+ _ => return unprocessable("`index` must name the judgement-history index"),
+ };
+ let question = match body.get("question").and_then(Value::as_str) {
+ Some(q) if !q.trim().is_empty() => q.to_string(),
+ _ => return unprocessable("`question` must be a non-empty string"),
+ };
+ let k = body
+ .get("k")
+ .and_then(Value::as_u64)
+ .map(|v| v.clamp(1, 100) as usize)
+ .unwrap_or(defaults.k);
+ let positive = body
+ .get("positive_label")
+ .and_then(Value::as_str)
+ .unwrap_or(&defaults.positive_label)
+ .to_string();
+ let mut cfg = DecisionsConfig {
+ index: index.clone(),
+ k,
+ positive_label: positive.clone(),
+ ..defaults.clone()
+ };
+ cfg.label_field = defaults.label_field.clone();
+ cfg.text_field = defaults.text_field.clone();
+
+ let neighbours = match vote_neighbours(&state, &cfg, &clip(&question, MAX_VOTE_TEXT_CHARS)).await
+ {
+ Ok(n) => n,
+ Err(resp) => return resp,
+ };
+ let total: f64 = neighbours.iter().map(|(w, _, _)| *w).sum();
+ let mut weight_by_label: BTreeMap<String, f64> = BTreeMap::new();
+ for (w, label, _) in &neighbours {
+ *weight_by_label.entry(label.to_string()).or_insert(0.0) += *w;
+ }
+ let (label, confidence, abstain, reason) = match weight_by_label
+ .iter()
+ .max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
+ {
+ Some((l, w)) if total > 0.0 => {
+ let c = *w / total;
+ let abstain = c < defaults.min_confidence;
+ let reason = if abstain {
+ Some(format!(
+ "confidence {:.3} below decisions.min_confidence {:.3}",
+ c, defaults.min_confidence
+ ))
+ } else {
+ None
+ };
+ (Some(l.clone()), c, abstain, reason)
+ }
+ _ => (
+ None,
+ 0.0,
+ true,
+ Some("no labelled neighbour in the history index".to_string()),
+ ),
+ };
+
+ let neighbour_list: Vec<Value> = neighbours
+ .iter()
+ .map(|(w, label, hit)| {
+ json!({
+ "_id": hit.id,
+ "label": label,
+ "_score": hit.score,
+ "weight": round6(*w),
+ "text": hit.source.get(&cfg.text_field).cloned().unwrap_or(Value::Null),
+ })
+ })
+ .collect();
+ let _ = state
+ .metrics
+ .record_query(&index, "decide", started.elapsed().as_secs_f64());
+
+ let mut resp = json!({
+ "index": index,
+ "k": k,
+ "positive_label": positive,
+ "label": label,
+ "confidence": round6(confidence),
+ "abstain": abstain,
+ "neighbours": neighbour_list,
+ "took_ms": started.elapsed().as_millis() as u64,
+ });
+ if let Some(r) = reason {
+ resp["reason"] = json!(r);
+ }
+ Json(resp).into_response()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The vote
+// ─────────────────────────────────────────────────────────────────────────────
+
+struct Question {
+ vote_text: String,
+ kind: Kind,
+}
+
+enum Kind {
+ Noul,
+ Choice { options: Vec<String> },
+}
+
+fn parse_question(id: &str, q: &Value, state_val: &Value) -> Result<Question, String> {
+ let obj = q
+ .as_object()
+ .ok_or_else(|| format!("question `{id}` must be an object"))?;
+ let kind = match obj.get("type").and_then(Value::as_str) {
+ Some("noul") => Kind::Noul,
+ Some("choice") => {
+ let criteria = obj
+ .get("criteria")
+ .and_then(Value::as_object)
+ .ok_or_else(|| format!("question `{id}` is a choice and needs `criteria` naming its options"))?;
+ if criteria.is_empty() || criteria.len() > MAX_CHOICE_OPTIONS {
+ return Err(format!(
+ "question `{id}`: `criteria` must name 1..{MAX_CHOICE_OPTIONS} options (got {})",
+ criteria.len()
+ ));
+ }
+ Kind::Choice {
+ options: criteria.keys().cloned().collect(),
+ }
+ }
+ Some("score") => {
+ return Err(format!(
+ "question `{id}`: `score` questions (2-10 ordinal levels) have no vote analogue \
+ and are not served; split them into nouls or use /_decide"
+ ))
+ }
+ other => {
+ return Err(format!(
+ "question `{id}`: `type` must be \"noul\" or \"choice\" (got {})",
+ other.unwrap_or("<missing>")
+ ))
+ }
+ };
+ // `instructions` is polymorphic on this wire: string, object or array.
+ // Every string leaf joins the vote text — an object's fields arrive in
+ // serde_json's BTreeMap order, which is stable but arbitrary; the text is
+ // retrieved by content, not by field order.
+ let mut text = String::new();
+ if let Some(instructions) = obj.get("instructions") {
+ join_strings(instructions, &mut text);
+ }
+ // `criteria` descriptions are NOT embedded: the vote is over text the
+ // history was labelled on, and no benchmark measures criteria text.
+ let resolved = resolve_backticks(&text, state_val);
+ // The query rides along from state — a plain-string state whole, an
+ // object state via its `query` field, exactly the two shapes the wire's
+ // own examples use. Other state fields are referenced (or not) by the
+ // instructions' backtick paths, which have already been resolved.
+ let mut vote_text = resolved;
+ match state_val {
+ Value::String(s) => {
+ vote_text.push(' ');
+ vote_text.push_str(s);
+ }
+ Value::Object(m) => {
+ if let Some(q) = m.get("query").and_then(Value::as_str) {
+ vote_text.push(' ');
+ vote_text.push_str(q);
+ }
+ }
+ _ => {}
+ }
+ let vote_text = clip(vote_text.trim(), MAX_VOTE_TEXT_CHARS);
+ Ok(Question { vote_text, kind })
+}
+
+/// One weighted-neighbour record: (weight, label, hit). Unlabelled hits are
+/// dropped here — they carry no vote.
+async fn vote_neighbours(
+ state: &AppState,
+ cfg: &DecisionsConfig,
+ text: &str,
+) -> Result<Vec<(f64, String, xerj_query::executor::Hit)>, axum::response::Response> {
+ let idx = match state.engine.get_index(&cfg.index) {
+ Ok(i) => i,
+ Err(e) => return Err(index_error(&cfg.index, &e.to_string())),
+ };
+ // The text field is configuration, not a literal — build the match by
+ // hand so the field name is data, not syntax.
+ let mut match_on = serde_json::Map::new();
+ match_on.insert(cfg.text_field.clone(), json!(text));
+ let query_body = json!({
+ "query": { "match": Value::Object(match_on) },
+ "size": cfg.k,
+ "_source": true,
+ });
+ let search_req = match parse_request(&query_body) {
+ Ok(r) => r,
+ Err(e) => return Err(unprocessable(&format!("internal query would not parse: {e}"))),
+ };
+ let result = match idx.search(&search_req).await {
+ Ok(r) => r,
+ Err(e) => return Err(index_error(&cfg.index, &e.to_string())),
+ };
+ let mut out = Vec::with_capacity(result.hits.len());
+ for (i, hit) in result.hits.into_iter().enumerate() {
+ if let Some(label) = hit.source.get(&cfg.label_field).and_then(Value::as_str) {
+ out.push((1.0 / (i as f64 + 1.0), label.to_string(), hit));
+ }
+ }
+ Ok(out)
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Text helpers
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// Join every string leaf of a polymorphic `instructions` value.
+fn join_strings(v: &Value, out: &mut String) {
+ match v {
+ Value::String(s) => {
+ if !out.is_empty() {
+ out.push(' ');
+ }
+ out.push_str(s);
+ }
+ Value::Array(a) => a.iter().for_each(|x| join_strings(x, out)),
+ Value::Object(m) => m.values().for_each(|x| join_strings(x, out)),
+ _ => {}
+ }
+}
+
+/// Resolve `` `dotted.path` `` references against `state`. An unresolvable
+/// path is left literally in place — visible in the vote text rather than
+/// silently dropped.
+fn resolve_backticks(text: &str, state: &Value) -> String {
+ if !text.contains('`') || state.is_null() {
+ return text.to_string();
+ }
+ let mut out = String::with_capacity(text.len());
+ let mut rest = text;
+ while let Some(start) = rest.find('`') {
+ out.push_str(&rest[..start]);
+ let after = &rest[start + 1..];
+ match after.find('`') {
+ Some(end) => {
+ let path = &after[..end];
+ match walk(state, path) {
+ Some(v) => out.push_str(v),
+ None => out.push_str(&rest[start..start + end + 2]),
+ }
+ rest = &after[end + 1..];
+ }
+ None => {
+ out.push_str(&rest[start..]);
+ return out;
+ }
+ }
+ }
+ out.push_str(rest);
+ out
+}
+
+fn walk<'a>(v: &'a Value, path: &str) -> Option<&'a str> {
+ let mut cur = v;
+ for seg in path.split('.') {
+ cur = cur.get(seg)?;
+ }
+ cur.as_str()
+}
+
+fn clip(s: &str, max: usize) -> String {
+ if s.len() <= max {
+ return s.to_string();
+ }
+ let mut cut = max;
+ while !s.is_char_boundary(cut) {
+ cut -= 1;
+ }
+ s[..cut].to_string()
+}
+
+fn round6(x: f64) -> f64 {
+ (x * 1_000_000.0).round() / 1_000_000.0
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Error bodies — every one names its cause and never guesses an answer
+// ─────────────────────────────────────────────────────────────────────────────
+
+fn error_body(status: u16, kind: &str, reason: String) -> axum::response::Response {
+ (
+ StatusCode::from_u16(status).unwrap_or(StatusCode::UNPROCESSABLE_ENTITY),
+ Json(json!({ "error": { "type": kind, "reason": reason } })),
+ )
+ .into_response()
+}
+
+fn unprocessable(reason: &str) -> axum::response::Response {
+ error_body(422, "illegal_argument", reason.to_string())
+}
+
+fn not_configured() -> axum::response::Response {
+ error_body(
+ 503,
+ "not_configured",
+ "no decisions index is configured: set [decisions] index to an index of labelled \
+ examples (one document per example, with the label and text fields this node's \
+ decisions settings name), then POST /v1/systemone again"
+ .to_string(),
+ )
+}
+
+fn index_error(index: &str, cause: &str) -> axum::response::Response {
+ error_body(
+ 422,
+ "history_index_error",
+ format!("decisions index `{index}` could not be searched: {cause}"),
+ )
+}
+
+fn no_support(ids: &[String], cfg: &DecisionsConfig) -> axum::response::Response {
+ error_body(
+ 422,
+ "no_support",
+ format!(
+ "no labelled neighbour in `{}` for question{} {} (k={}); zero support is an \
+ error, not a fabricated probability — add labelled examples to the history \
+ index or use POST /_decide to inspect what is there",
+ cfg.index,
+ cfg.k,
+ if ids.len() == 1 { "" } else { "s" },
+ ids.iter().map(|i| format!("`{i}`")).collect::<Vec<_>>().join(", ")
+ ),
+ )
+}
diff --git a/engine/crates/xerj-api/tests/systemone_http.rs b/engine/crates/xerj-api/tests/systemone_http.rs
new file mode 100644
index 000000000..edaf81b21
--- /dev/null
+++ b/engine/crates/xerj-api/tests/systemone_http.rs
@@ -0,0 +1,534 @@
+//! End-to-end checks for the local judge — `POST /v1/systemone` (native
+//! router) and `POST /_decide` (ES-compat router) — over HTTP, against real
+//! in-process indexes of labelled history.
+//!
+//! THE VOTE IS RETRIEVAL, NOT A MODEL. It is a weighted nearest-neighbour vote
+//! over an index the test seeds, so every probability here is checkable
+//! arithmetic. Nothing in this file says anything about judge-model quality —
+//! that is measured in `benchmarks/beir-hybrid/results/`, not by these tests.
+//!
+//! The `systemone` tests speak the request/response shape TypeSafe documents
+//! and `jev-reranker` sends, because that client — unmodified — is the
+//! acceptance gate for this endpoint:
+//!
+//! ```text
+//! request { state: { query, documents: { doc_0: "...", ... } }, model,
+//! questions: { d0: { type: "noul",
+//! instructions: "Is `documents.doc_0` relevant?" } } }
+//! response { model, answers: { d0: { type: "noul", noul: 0..1 } }, usage }
+//! ```
+//!
+//! Two deliberate wire breaks are asserted here too, so they cannot regress
+//! into silence: `model` is echoed as this node's own id (never the requested
+//! Jev name), and a question with zero support is a 422 naming the question —
+//! never a fabricated 0.5.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The node
+// ─────────────────────────────────────────────────────────────────────────────
+
+struct Node {
+ /// The XERJ-native REST API (`/v1/...`): /v1/systemone, /v1/models.
+ native: axum::Router,
+ /// The ES-compat API: index creation, _search, /_decide.
+ app: axum::Router,
+ _dir: tempfile::TempDir,
+}
+
+/// A node whose `[decisions]` block names `index`. Empty string leaves the
+/// endpoints unconfigured (503), which is itself under test.
+async fn node_with_decisions(index: &str, positive_label: &str, min_confidence: f64) -> Node {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ config.decisions.index = index.to_string();
+ config.decisions.positive_label = positive_label.to_string();
+ config.decisions.min_confidence = min_confidence;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = xerj_api::state::AppState::new(config, engine, metrics);
+ Node {
+ native: xerj_api::router::build_native_router(state.clone()),
+ app: xerj_api::router::build_es_compat_router(state),
+ _dir: dir,
+ }
+}
+
+/// The refund-history node: two `true` docs about refunds, one `false` doc
+/// about shipping, so a refund question votes 1.0 and a shipping question 0.0
+/// whatever the internal ordering — every neighbour of "refund …" is `true`.
+async fn gate_node() -> Node {
+ let node = node_with_decisions("history", "true", 0.0).await;
+ seed(
+ &node,
+ "history",
+ &[
+ ("h1", "refund refund refund asked for money back", "true"),
+ ("h2", "refund policy says money back", "true"),
+ ("h3", "shipping delivery times", "false"),
+ ],
+ )
+ .await;
+ node
+}
+
+impl Node {
+ async fn call(&self, router: &axum::Router, method: &str, path: &str, body: Value) -> (StatusCode, Value) {
+ let response = router
+ .clone()
+ .oneshot(
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+ .expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ (
+ status,
+ serde_json::from_str(&String::from_utf8_lossy(&bytes)).unwrap_or(Value::Null),
+ )
+ }
+
+ async fn systemone(&self, body: Value) -> (StatusCode, Value) {
+ self.call(&self.native.clone(), "POST", "/v1/systemone", body).await
+ }
+ async fn models(&self) -> (StatusCode, Value) {
+ self.call(&self.native.clone(), "GET", "/v1/models", json!({})).await
+ }
+ async fn decide(&self, body: Value) -> (StatusCode, Value) {
+ self.call(&self.app.clone(), "POST", "/_decide", body).await
+ }
+}
+
+/// Create an index of `text`/`label` documents and refresh it.
+async fn seed(node: &Node, index: &str, docs: &[(&str, &str, &str)]) {
+ let (st, b) = node
+ .call(
+ &node.app.clone(),
+ "PUT",
+ &format!("/{index}"),
+ json!({"mappings": {"properties": {
+ "text": {"type": "text"},
+ "label": {"type": "keyword"}
+ }}}),
+ )
+ .await;
+ assert!(st.is_success(), "create {index}: {st} {b}");
+ for (id, text, label) in docs {
+ let (st, b) = node
+ .call(
+ &node.app.clone(),
+ "PUT",
+ &format!("/{index}/_doc/{id}"),
+ json!({"text": text, "label": label}),
+ )
+ .await;
+ assert!(st.is_success(), "index {index}/{id}: {st} {b}");
+ }
+ let (st, b) = node
+ .call(&node.app.clone(), "POST", &format!("/{index}/_refresh"), json!({}))
+ .await;
+ assert!(st.is_success(), "refresh {index}: {st} {b}");
+}
+
+fn reason(r: &Value) -> String {
+ r["error"]["reason"].as_str().unwrap_or_default().to_string()
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// GET /v1/models
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The SDKs list models before calling. One entry, and it is this node's own
+/// id — never a Jev model name, which would claim a hosted judge that is not
+/// answering.
+#[tokio::test]
+async fn models_lists_one_truthful_entry() {
+ let node = gate_node().await;
+ let (st, r) = node.models().await;
+ assert!(st.is_success(), "{st} {r}");
+ let models = r["models"].as_array().expect("models array");
+ assert_eq!(models.len(), 1);
+ assert_eq!(models[0]["name"], "xerj-history-vote-1");
+ assert_ne!(models[0]["name"], "jev-1.13.0");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// POST /v1/systemone — the wire
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The acceptance-gate shape exactly as `jev-reranker`'s `_payload()` builds
+/// it: object state, string instructions carrying backtick references into
+/// that state, ids `d0`/`d1`. The answer set must be keyed identically, each
+/// answer carrying only its documented fields.
+#[tokio::test]
+async fn answers_a_jev_reranker_shaped_request_with_backtick_references() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": {
+ "query": "complaint",
+ "documents": {
+ "doc_0": "refund refund asked for money back twice charged",
+ "doc_1": "where is my parcel shipping delivery"
+ }
+ },
+ "model": "jev-1.13.0",
+ "questions": {
+ "d0": {"type": "noul", "instructions": "Is `documents.doc_0` relevant?"},
+ "d1": {"type": "noul", "instructions": "Is `documents.doc_1` relevant?"}
+ }
+ }))
+ .await;
+
+ assert_eq!(st, StatusCode::OK, "{r}");
+ // Deliberate wire break #1: the model echo is truthful. The requested
+ // name rides at decisions.requested_model.
+ assert_eq!(r["model"], "xerj-history-vote-1");
+ assert_eq!(r["decisions"]["requested_model"], "jev-1.13.0");
+ // Answers keyed exactly by the ids sent — no more, no fewer.
+ let answers = r["answers"].as_object().expect("answers object");
+ assert_eq!(answers.len(), 2, "{answers:?}");
+ for id in ["d0", "d1"] {
+ let a = answers.get(id).unwrap_or(&Value::Null);
+ // Clients parse answers strictly: exactly the documented fields.
+ let keys: Vec<&str> = a.as_object().map(|o| o.keys().map(String::as_str).collect()).unwrap_or_default();
+ assert_eq!(keys, vec!["type", "noul"], "{id}: {a}");
+ assert_eq!(a["type"], "noul");
+ }
+ // doc_0 resolved to refund text → every neighbour is `true` → 1.0.
+ // doc_1 resolved to shipping text → the only neighbour is `false` → 0.0.
+ assert_eq!(r["answers"]["d0"]["noul"], json!(1.0));
+ assert_eq!(r["answers"]["d1"]["noul"], json!(0.0));
+ // usage must be JSON integers — the client's metering does dict.get, no
+ // coercion.
+ assert!(r["usage"]["input_tokens"].as_u64().is_some(), "{r}");
+ assert!(r["usage"]["output_tokens"].as_u64().is_some(), "{r}");
+ // Evidence rides at the top level, where every verified client ignores.
+ assert_eq!(r["decisions"]["index"], "history");
+ assert_eq!(r["decisions"]["evidence"]["d0"]["label"], "true");
+ assert_eq!(r["decisions"]["evidence"]["d0"]["support"], json!(1.0));
+}
+
+/// The other documented state shape: a plain string, with instructions as an
+/// object (the shape XERJ's own rerank client sends at providers).
+#[tokio::test]
+async fn a_plain_string_state_with_object_instructions_also_votes() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": "refund money back",
+ "model": "jev-latest",
+ "questions": {
+ "q1": {"type": "noul", "instructions": {
+ "question": "Is this relevant to the query?",
+ "document": "asked for a refund"
+ }}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+ assert_eq!(r["answers"]["q1"]["noul"], json!(1.0));
+}
+
+/// A `choice` question answers with a probability for EVERY criterion option —
+/// options with no neighbour weight appear as 0.0, not as missing keys — the
+/// winner is the argmax, and its confidence is its own probability.
+#[tokio::test]
+async fn choice_answers_carry_a_probability_for_every_option() {
+ let node = node_with_decisions("mixed", "billing", 0.0).await;
+ seed(
+ &node,
+ "mixed",
+ &[
+ ("m1", "cancel cancel cancel my subscription", "billing"),
+ ("m2", "cancel subscription please", "tech"),
+ ("m3", "subscription renewal", "tech"),
+ ],
+ )
+ .await;
+
+ // state carries no `query` field, so the vote text is exactly the resolved
+ // ticket — the same string /_decide is asked below, making the arithmetic
+ // cross-checkable between the two surfaces.
+ let vote = "cancel my subscription";
+ let (st, r) = node
+ .systemone(json!({
+ "state": {"ticket": vote},
+ "model": "jev-1.13.0",
+ "questions": {
+ "c1": {"type": "choice",
+ "instructions": "Route `ticket`.",
+ "criteria": {"billing": "about money", "tech": "about software"}}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::OK, "{r}");
+
+ let a = &r["answers"]["c1"];
+ assert_eq!(a["type"], "choice");
+ let probs = a["probabilities"].as_object().expect("probabilities");
+ let mut keys: Vec<&str> = probs.keys().map(String::as_str).collect();
+ keys.sort_unstable();
+ assert_eq!(keys, vec!["billing", "tech"], "{probs:?}");
+ let sum: f64 = probs.values().filter_map(Value::as_f64).sum();
+ assert!((sum - 1.0).abs() < 1e-4, "probabilities sum to {sum}: {probs:?}");
+ // The choice is the argmax of the reported probabilities, and the
+ // confidence is that option's own probability.
+ let (argmax, _) = probs
+ .iter()
+ .map(|(k, v)| (k, v.as_f64().unwrap_or(0.0)))
+ .max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
+ .expect("non-empty");
+ assert_eq!(a["choice"], json!(argmax));
+ assert_eq!(a["confidence"], probs[argmax]);
+ assert_eq!(r["decisions"]["evidence"]["c1"]["support"], a["confidence"]);
+
+ // The same vote through /_decide names its neighbours; the probabilities
+ // above must be the weighted shares of exactly those neighbours.
+ let (st2, d) = node.decide(json!({"index": "mixed", "question": vote})).await;
+ assert!(st2.is_success(), "{d}");
+ let total: f64 = d["neighbours"]
+ .as_array()
+ .expect("neighbours")
+ .iter()
+ .map(|n| n["weight"].as_f64().unwrap_or(0.0))
+ .sum();
+ assert!(total > 0.0, "{d}");
+ for (option, p) in probs {
+ let w: f64 = d["neighbours"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .filter(|n| &n["label"] == option)
+ .map(|n| n["weight"].as_f64().unwrap_or(0.0))
+ .sum();
+ let expected = (w / total * 1_000_000.0).round() / 1_000_000.0;
+ assert!(
+ (p.as_f64().unwrap_or(-1.0) - expected).abs() < 1e-6,
+ "option {option}: reported {}, weighted share {expected}",
+ p
+ );
+ }
+}
+
+/// Deliberate wire break #2: `score` has no vote analogue, and saying so with
+/// the alternative beats inventing one.
+#[tokio::test]
+async fn a_score_question_is_refused_with_an_alternative() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": "refund",
+ "questions": {"s1": {"type": "score", "instructions": "How relevant?"}}
+ }))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ let why = reason(&r);
+ assert!(why.contains("score"), "{why}");
+ assert!(why.contains("/_decide"), "{why}");
+}
+
+/// Deliberate wire break #3: zero support is an error naming the question ids.
+/// A fabricated 0.5 would rank a document by nothing.
+#[tokio::test]
+async fn zero_support_is_an_error_naming_the_question_not_a_half() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({
+ "state": "zzzqqq nothing matches this",
+ "questions": {
+ "d0": {"type": "noul", "instructions": "zzzqqq"},
+ "d1": {"type": "noul", "instructions": "wwwwww also nothing"}
+ }
+ }))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert_eq!(r["error"]["type"], "no_support");
+ let why = reason(&r);
+ assert!(why.contains("`d0`"), "{why}");
+ assert!(why.contains("`d1`"), "{why}");
+ assert!(!r.to_string().contains("0.5"), "no fabricated halves: {r}");
+ assert!(r["answers"].is_null(), "no half-answered judgement history: {r}");
+}
+
+/// Until `[decisions] index` names an index, the endpoint says 503 — not a
+/// vote over nothing.
+#[tokio::test]
+async fn systemone_503s_until_an_index_is_configured() {
+ let node = node_with_decisions("", "true", 0.0).await;
+ let (st, r) = node
+ .systemone(json!({"state": "refund", "questions": {
+ "d0": {"type": "noul", "instructions": "refund"}
+ }}))
+ .await;
+ assert_eq!(st, StatusCode::SERVICE_UNAVAILABLE, "{r}");
+ assert_eq!(r["error"]["type"], "not_configured");
+}
+
+/// A configured index that does not exist is a different failure from
+/// unconfigured, and is surfaced — not answered from an empty history.
+#[tokio::test]
+async fn a_configured_but_missing_index_is_surfaced_as_an_index_error() {
+ let node = node_with_decisions("nope", "true", 0.0).await;
+ let (st, r) = node
+ .systemone(json!({"state": "refund", "questions": {
+ "d0": {"type": "noul", "instructions": "refund"}
+ }}))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert_eq!(r["error"]["type"], "history_index_error");
+ assert!(reason(&r).contains("nope"), "{r}");
+}
+
+#[tokio::test]
+async fn questions_must_be_a_non_empty_object() {
+ let node = gate_node().await;
+ for bad in [json!({}), json!([]), json!("d0"), Value::Null] {
+ let (st, r) = node
+ .systemone(json!({"state": "refund", "questions": bad}))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{bad}: {r}");
+ }
+}
+
+#[tokio::test]
+async fn a_question_of_unknown_type_is_refused_by_id() {
+ let node = gate_node().await;
+ let (st, r) = node
+ .systemone(json!({"state": "refund", "questions": {
+ "d0": {"type": "noul", "instructions": "refund"},
+ "dx": {"type": "haiku", "instructions": "refund"}
+ }}))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ let why = reason(&r);
+ assert!(why.contains("`dx`"), "{why}");
+ assert!(why.contains("haiku"), "{why}");
+}
+
+/// One request cannot ask for more questions than the rerank stage would
+/// judge documents (300): the vote is a search per question.
+#[tokio::test]
+async fn more_than_300_questions_is_refused() {
+ let node = gate_node().await;
+ let mut questions = serde_json::Map::new();
+ for i in 0..301 {
+ questions.insert(format!("d{i}"), json!({"type": "noul", "instructions": "refund"}));
+ }
+ let (st, r) = node
+ .systemone(json!({"state": "refund", "questions": Value::Object(questions)}))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert!(reason(&r).contains("300"), "{r}");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// POST /_decide — the audit surface
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// /_decide shows its work: the neighbours, their labels, engine scores and
+/// vote weights, plus the label and confidence derived from them. k is
+/// per-request.
+#[tokio::test]
+async fn decide_returns_its_neighbours_and_honours_per_request_k() {
+ let node = gate_node().await;
+
+ let (st, r) = node
+ .decide(json!({"index": "history", "question": "refund money back", "k": 1}))
+ .await;
+ assert!(st.is_success(), "{r}");
+ assert_eq!(r["index"], "history");
+ assert_eq!(r["k"], 1);
+ assert_eq!(r["label"], "true");
+ assert!((r["confidence"].as_f64().unwrap_or(0.0) - 1.0).abs() < 1e-9, "{r}");
+ assert_eq!(r["abstain"], json!(false));
+ let neighbours = r["neighbours"].as_array().expect("neighbours");
+ assert_eq!(neighbours.len(), 1, "k=1: {neighbours:?}");
+ let n = &neighbours[0];
+ assert_eq!(n["_id"], "h1");
+ assert_eq!(n["label"], "true");
+ assert!(n["_score"].as_f64().is_some(), "engine score: {n}");
+ assert!(n["text"].as_str().is_some_and(|t| t.contains("refund")), "{n}");
+
+ // Full k: both refund docs are neighbours, weights reciprocal in rank.
+ let (_, r) = node
+ .decide(json!({"index": "history", "question": "refund money back"}))
+ .await;
+ let neighbours = r["neighbours"].as_array().expect("neighbours");
+ assert_eq!(neighbours.len(), 2, "{neighbours:?}");
+ let w0 = neighbours[0]["weight"].as_f64().unwrap_or(0.0);
+ let w1 = neighbours[1]["weight"].as_f64().unwrap_or(0.0);
+ assert!((w0 - 1.0).abs() < 1e-9, "rank 1 weight: {w0}");
+ assert!((w1 - 0.5).abs() < 1e-9, "rank 2 weight: {w1}");
+}
+
+/// An operator who raises `decisions.min_confidence` gets abstentions with the
+/// reason, and a question with no neighbours abstains rather than erroring —
+/// abstention lives on /_decide, errors on the wire surface.
+#[tokio::test]
+async fn decide_abstains_below_min_confidence_and_on_no_neighbours() {
+ let strict = node_with_decisions("history", "true", 0.99).await;
+ seed(
+ &strict,
+ "history",
+ &[
+ ("h1", "refund refund refund asked for money back", "true"),
+ ("h2", "refund policy says money back", "true"),
+ ("h3", "shipping delivery times", "false"),
+ ],
+ )
+ .await;
+
+ // Mixed neighbourhood: refund docs plus the shipping doc at rank 3 →
+ // confidence 1.5/1.8333 = 0.818 < 0.99 → abstain.
+ let (st, r) = strict
+ .decide(json!({"index": "history", "question": "refund shipping delivery"}))
+ .await;
+ assert!(st.is_success(), "{r}");
+ assert_eq!(r["abstain"], json!(true));
+ assert!(r["reason"].as_str().is_some_and(|s| s.contains("min_confidence")), "{r}");
+
+ // Nothing matches at all: abstain with the no-neighbour reason.
+ let (st, r) = strict
+ .decide(json!({"index": "history", "question": "zzzqqq nothing"}))
+ .await;
+ assert!(st.is_success(), "{r}");
+ assert_eq!(r["abstain"], json!(true));
+ assert!(r["label"].is_null(), "{r}");
+ assert!(r["reason"].as_str().is_some_and(|s| s.contains("no labelled neighbour")), "{r}");
+}
+
+#[tokio::test]
+async fn decide_validates_its_request_before_searching() {
+ let node = gate_node().await;
+ for (body, needle) in [
+ (json!({"question": "refund"}), "index"),
+ (json!({"index": "history"}), "question"),
+ (json!({"index": " ", "question": "refund"}), "index"),
+ ] {
+ let (st, r) = node.decide(body.clone()).await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{body}: {r}");
+ assert!(reason(&r).contains(needle), "{body}: {r}");
+ }
+ // An index that does not exist is an error naming it, not a 503 and not
+ // an abstain over an empty history.
+ let (st, r) = node
+ .decide(json!({"index": "nope", "question": "refund"}))
+ .await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{r}");
+ assert_eq!(r["error"]["type"], "history_index_error");
+}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index fdb3ed173..8ecc4ccbb 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1,6 +1,6 @@
//! xerj configuration system.
//!
-//! Configuration is intentionally minimal: **120 settings** versus
+//! Configuration is intentionally minimal: **126 settings** versus
//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
//! production-ready default. The format is TOML, loaded from a single file.
//!
@@ -97,9 +97,13 @@ pub struct Config {
pub wal_tap: WalTapConfig,
/// Second-stage reranking provider — 3 settings. Inert until a key is set.
pub rerank: RerankProviderConfig,
+ /// Typed decisions answered from a labelled-history index by
+ /// nearest-neighbour vote — 6 settings. Inert until `decisions.index`
+ /// names an index that exists.
+ pub decisions: DecisionsConfig,
}
-// 22 sub-configs, 120 leaf settings in total. Do not maintain that sum by hand
+// 23 sub-configs, 126 leaf settings in total. Do not maintain that sum by hand
// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
// serialised `Config::default()` and fails if this comment and the module
// header stop matching. `Default` is derived: every field is a sub-config that
@@ -463,7 +467,7 @@ impl Config {
}
// ═════════════════════════════════════════════════════════════════════════════
-// Sub-configs (120 user-facing settings total; counted by
+// Sub-configs (126 user-facing settings total; counted by
// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
@@ -2288,6 +2292,46 @@ impl RerankProviderConfig {
}
}
+
+/// Typed decisions (`/v1/systemone`, `/_decide`) answered locally by a
+/// weighted nearest-neighbour vote over a labelled-history index — the
+/// retrieval analogue of a judge model, with the evidence staying on the node.
+///
+/// **6 settings.**
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(default, deny_unknown_fields)]
+pub struct DecisionsConfig {
+ /// Index holding the judgement history: one document per labelled
+ /// example. Empty (the default) disables both endpoints with a 503 that
+ /// names this setting.
+ pub index: String,
+ /// How many nearest examples vote. The published Banking77 / SMS-spam
+ /// measurements used k = 10.
+ pub k: usize,
+ /// Document field holding each example's label.
+ pub label_field: String,
+ /// Document field holding each example's text.
+ pub text_field: String,
+ /// The label that means "true" for a `noul`. For a `choice` the labels
+ /// come from the question's own `criteria` keys.
+ pub positive_label: String,
+ /// `/_decide` abstains below this vote share (0 = never abstain; the
+ /// `/v1/systemone` wire has no abstain field and errors instead).
+ pub min_confidence: f64,
+}
+
+impl Default for DecisionsConfig {
+ fn default() -> Self {
+ Self {
+ index: String::new(),
+ k: 10,
+ label_field: "label".to_string(),
+ text_field: "text".to_string(),
+ positive_label: "true".to_string(),
+ min_confidence: 0.0,
+ }
+ }
+}
// ═════════════════════════════════════════════════════════════════════════════
// Tests
// ═════════════════════════════════════════════════════════════════════════════
@@ -3030,6 +3074,7 @@ mod tests {
("lifecycle", 1),
("wal_tap", 10),
("rerank", 3),
+ ("decisions", 6),
];
/// Count the settings by *counting them*.
@@ -3072,7 +3117,7 @@ mod tests {
"the section table must sum to the whole config"
);
assert_eq!(
- total, 120,
+ total, 126,
"the total settings count changed. It is quoted in this module's \
header, in xerj-common/src/lib.rs, in engine/README.md, in \
xerj.default.toml and in EXPECTED_SETTINGS in \
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index e6887a416..540b7b218 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -7,13 +7,13 @@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
-//! **120 settings**, each meaningful and production-tested. Every default
+//! **126 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
-//! - [`config`] — TOML-based configuration (120 settings)
+//! - [`config`] — TOML-based configuration (126 settings)
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rs
index ce82bc807..0d90a1b82 100644
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {
/// Settings in a default `Config`, measured by `count_settings`. Quoted in
/// `xerj-common/src/config.rs`; keep the two in step.
-const EXPECTED_SETTINGS: usize = 120;
+const EXPECTED_SETTINGS: usize = 126;
#[tokio::test]
async fn journey_zero_config() {
diff --git a/engine/crates/xerj-rerank/src/lib.rs b/engine/crates/xerj-rerank/src/lib.rs
index 5f89745f2..5702f6788 100644
--- a/engine/crates/xerj-rerank/src/lib.rs
+++ b/engine/crates/xerj-rerank/src/lib.rs
@@ -553,13 +553,6 @@ impl RerankConfig {
Ok(cfg)
}
- fn question(&self) -> &str {
- self.instructions.as_deref().unwrap_or(
- "Does this document contain information that answers the query? \
- Judge only whether it is relevant to the query, not whether it is \
- well written.",
- )
- }
}
/// Truncate on a char boundary. Byte slicing a `&str` here would panic on any
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index 402d6d9c1..8a87017d9 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -1,6 +1,6 @@
# xerj Default Configuration
# ============================================================
-# This file sets 55 of XERJ's 120 user-facing settings to their
+# This file sets 55 of XERJ's 126 user-facing settings to their
# production-ready defaults, and describes the rest in comments.
# Every value here IS the default, so copying this file changes
# nothing until you edit it (a test enforces that). The complete
@@ -602,3 +602,22 @@ access_log = false
# enabled = true
# api_key = "ts-..." # not a default ("")
# endpoint = "https://api.typesafe.ai/v1/systemone" # not a default ("")
+
+# ─────────────────────────────────────────────────────────────────────────
+# Decisions — typed answers from your own labelled history, no model call.
+#
+# Index the examples you already have (one document per example: the text and
+# its label), and POST /v1/systemone answers System One-shaped questions with
+# a weighted nearest-neighbour vote over that index — same interface as a
+# judge model, nothing leaves the node. POST /_decide is the same vote with
+# the evidence in the response: neighbours, labels, weights, and an abstain
+# below min_confidence. Empty `index` (the default) leaves both answering 503.
+# See docs/RERANK.md and benchmarks/decisions-as-retrieval.
+#
+# [decisions]
+# index = "judgements" # not a default ("") — an index of labelled examples
+# k = 10 # neighbours that vote
+# label_field = "label"
+# text_field = "text"
+# positive_label = "true" # the label a noul's probability is the share of
+# min_confidence = 0.0 # /_decide abstains below this