AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
systemone: calibration layer — p_cal beside every p_raw (#1063)
xerj-org615fd96e92d9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 86d0011a8..8a83184d1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **systemone: calibration layer — `p_cal` beside every `p_raw`, published+ reliability curve ([#1063](https://github.com/xerj-org/xerj/issues/1063)).** A ladder+ probability ranks before it odds — on the FiQA rerank baseline a hosted+ noul of 0.93 meant relevance 34% of the time (ECE 0.3109). The new+ `[decisions] calibration = isotonic | temperature` setting (default `none`,+ the section's 8th setting) fits a correction on a deterministic held-out+ fifth of the node's recorded outcomes — outcome documents carry the truth+ label, the `p` that was served for that label, and NO `source` field+ (flywheel answers are predictions; a fit on predictions would learn the+ model is right by construction), 10 labelled pairs minimum — and every+ probability-shipping path on the decide surface carries the calibrated+ probability BESIDE the raw one: `/_decide` answers `p_raw` (the positive+ label's share, the quantity a noul answers) plus `p_cal`;+ `/v1/systemone` carries both in `decisions.evidence` per noul question+ with the strictly-parsed answer objects untouched; choice per-option+ shares are a different quantity and are not calibrated (the block says+ `"scope": "noul"`). No calibration configured → no `p_cal` field at all;+ configured but unfitted → `p_cal: null` with a published, actionable+ reason; fitted → the fit's value, never a copy of `p_raw` pretending to+ be calibrated — and whenever a `p_cal` exists, a `calibration` block rides+ beside it with the method, the fit's ECE raw and calibrated, and the+ fitted-on/held-out counts, so no probability ships without an ECE beside+ it. `GET /_decide/_calibration` publishes the whole reliability curve+ (binned `p_raw` vs empirical frequency with bin counts, raw and+ calibrated), both ECEs, the fit's parameters (isotonic knots or the+ temperature), the pair counts, and the fitted documents' date range; it+ always refits, so it doubles as the operator's refresh (decide requests+ read a per-index cache refreshed at most once a minute). Isotonic is PAVA+ with linear interpolation and end-clamping; temperature is one scalar on+ the log-odds fitted by golden-section NLL search; both are implemented+ from first principles in `xerj_common::calibration` with method citations+ (retrieval skipped: the reference corpora are not present in this+ sandbox). Measured against the FiQA baseline's own retained curve+ (`benchmarks/decisions-calibration`, stdlib Python mirroring the Rust+ exactly — both reproduce the published raw 0.3109 and agree to the fourth+ decimal): raw held-out ECE 0.3831 → isotonic 0.0330 (gate ≤ 0.10 MET);+ temperature 0.3903, honestly failing — one scalar cannot map 0.93 → 0.34+ while keeping 0.05 → 0.0002. Scope of that measurement: bin-level+ held-out (the run retained 10 binned aggregates, not the 19,440 raw+ pairs); the pair-level re-run needs the model's raw scores and is queued+ as the rc.80 final form. `confidence` keeps its raw meaning and the+ abstain gate keeps applying to it.+- **`POST /_ask` + `xerj_plan` MCP tool: prompt in, validated query DSL out([#1056](https://github.com/xerj-org/xerj/issues/1056), PR[#1076](https://github.com/xerj-org/xerj/pull/1076)).** `POST /_ask`diff --git a/benchmarks/decisions-calibration/README.md b/benchmarks/decisions-calibration/README.mdnew file mode 100644index 000000000..054d939d9--- /dev/null+++ b/benchmarks/decisions-calibration/README.md@@ -0,0 +1,66 @@+# decisions-calibration — the #1063 gate measurement++Issue #1063's gate: **Jev FiQA rerank ECE 0.3109 → ≤ 0.10 on held-out**,+baseline `benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa`.+Measured here from the baseline's own retained curve, with no new model+calls, no network, no RNG.++## Result++| quantity | ECE | meets ≤ 0.10? |+|---|---|---|+| raw, all 19,440 pairs (reproduces the published number exactly) | **0.3109** | — |+| raw, held-out bins only | 0.3831 | no |+| **isotonic (PAVA), held-out** | **0.0330** | **yes** |+| temperature, held-out | 0.3903 | no |++`results/2026-09-30-calibration-gate.json` is the machine-readable run.++Reproduce: `python3 fiqa_gate.py` (stdlib only; writes nothing unless `--out`).++## Method, and its honest scope++The baseline run retained the reliability curve as **10 binned aggregates**+(`calibration.curve`: mean_conf, acc, n per bin; `per_query` holds nDCG only) —+not the 19,440 raw (p, y) pairs. So this measurement works at **bin+granularity**:++- pairs are reconstructed per bin — `n` pairs at `mean_conf`,+ `round(acc·n)` of them positive — the same reconstruction the Rust+ module's unit tests pin (`FIQA_BASELINE_CURVE` + `bin_pairs` in+ `engine/crates/xerj-common/src/calibration.rs`);+- the held-out split is **even bins fit (0,2,4,6,8), odd bins evaluate+ (1,3,5,7,9)** — deterministic (#940: no RNG, no clock), and bin-level+ held-out means whole probability *regions* the fit never saw, which is a+ stronger test of interpolation than a random pair split;+- the raw held-out ECE (0.3831) is *harder* than the published 0.3109,+ because the odd bins carry the pathological top bin (0.9297 conf → 0.3378+ acc). The gate was not met by evaluating on an easy split.++What this does NOT claim: a pair-level held-out measurement. The fit here+sees five points, not 15,000 — bin-level aggregates are already smoothed, and+smoothing flatters calibration fits. The pair-level re-run needs the model's+raw scores re-emitted per query (the baseline kept only nDCG); that costs API+calls and is queued as the **rc.80 final form** of this gate. The production+path (`/_decide/_calibration`) fits on raw pairs by construction, so the+serving behaviour is the stronger form already.++## Why temperature fails, and is still shipped++Temperature is one scalar on the log-odds: `p' = σ(logit(p)/T)`. The FiQA+curve needs 0.93 → 0.34 at the top while keeping 0.05 → 0.0002 at the bottom —+a steep monotone *remap*, not the uniform flattening a single T can express.+Its NLL optimum here is T = 1.1517 (barely softer than raw), and its held-out+ECE (0.3903) barely moves. Isotonic is the gate-meeting method; temperature+is shipped because it is the right tool when the history is thin (few knots+to overfit) or monotone-in-log-odds — and `/_decide/_calibration` publishes+both ECEs, so a node running either method shows its own number.++## Cross-check++`fiqa_gate.py` mirrors the Rust implementation exactly — same bin edges+(`floor(p·10)`, p=1 → bin 9), same PAVA with linear interpolation between+knots and end-clamping, same temperature bracket (ln T ∈ [ln(1/16), ln 16],+100 golden-section iterations, logit clamped at 1e-6). Agreement between the+two (0.0330 here; the Rust pinning test asserts ≤ 0.10 on the identical+split) is a cross-check on both.diff --git a/benchmarks/decisions-calibration/fiqa_gate.py b/benchmarks/decisions-calibration/fiqa_gate.pynew file mode 100644index 000000000..fdc4e3ead--- /dev/null+++ b/benchmarks/decisions-calibration/fiqa_gate.py@@ -0,0 +1,280 @@+#!/usr/bin/env python3+"""Measure the #1063 calibration gate against the FiQA rerank baseline.++Issue #1063's gate: Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out data.+The baseline run (benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa)+retained the reliability curve as 10 BINNED AGGREGATES, not the 19,440 raw+pairs (per_query holds nDCG only), so this measurement works at bin+granularity: each bin's (mean_conf, acc, n) is expanded into n pairs at+mean_conf, positives = round(acc*n), exactly the reconstruction the Rust+module's unit tests use (engine/crates/xerj-common/src/calibration.rs,+FIQA_BASELINE_CURVE + bin_pairs).++Held-out split, deterministic (#940 rule — no RNG, no clock): even-index bins+(0,2,4,6,8) fit the calibration, odd-index bins (1,3,5,7,9) evaluate it.+Held-out at bin level means whole probability REGIONS the fit never saw —+stronger than a random pair split on interpolation, weaker than the full+pair-level re-run (which needs the model's raw scores; that is the rc.80+final form, noted in the README).++This script mirrors the Rust implementation exactly — same bin edges+(b = floor(p*10), p=1 -> bin 9), same PAVA with linear interpolation between+knots and clamping at the ends, same temperature bracket (ln T in+[ln(1/16), ln 16], 100 golden-section iterations, logit clamped at 1e-6) — so+agreement between the two is a cross-check on both. Stdlib only, no network,+no API calls: every number comes from the existing baseline file.++Usage: python3 fiqa_gate.py [--baseline PATH] [--out PATH]+"""++import argparse+import json+import math+import sys+from pathlib import Path++DEFAULT_BASELINE = (+ Path(__file__).resolve().parents[1]+ / "beir-hybrid"+ / "results"+ / "2026-09-20-rerank-full-fiqa"+ / "fiqa-full-run.json"+)+GATE = 0.10+BINS = 10+LOGIT_EPS = 1e-6+T_BRACKET = (math.log(1.0 / 16.0), math.log(16.0))+GOLDEN_ITERS = 100+++# ── the algorithms, mirroring engine/crates/xerj-common/src/calibration.rs ──+++def bin_of(p, bins=BINS):+ """b = floor(p*bins), p=1 -> the last bin (the Rust module's clamp)."""+ b = int(p * bins)+ return min(b, bins - 1)+++def ece(pairs, bins=BINS):+ """ECE = sum_b (n_b/N)*|acc_b - conf_b| (Naeini et al. 2015)."""+ n = len(pairs)+ if n == 0:+ return 0.0+ sums = [[0, 0.0, 0.0] for _ in range(bins)] # count, sum p, sum y+ for p, y in pairs:+ s = sums[bin_of(p, bins)]+ s[0] += 1+ s[1] += p+ s[2] += y+ total = 0.0+ for c, sp, sy in sums:+ if c:+ total += (c / n) * abs(sy / c - sp / c)+ return total+++class Isotonic:+ """PAVA (pool-adjacent-violators) with the Rust module's knots:+ linear interpolation between block means, clamped at the ends+ (Barlow et al. 1972; Zadrozny & Elkan 2002 for calibration use)."""++ def __init__(self, pairs):+ pts = sorted(((p, y) for p, y in pairs), key=lambda t: (t[0], t[1]))+ # Pool tied p by weighted mean first.+ xs, ys, ws = [], [], []+ i = 0+ while i < len(pts):+ j = i+ p_sum = y_sum = 0.0+ while j < len(pts) and pts[j][0] == pts[i][0]:+ p_sum += pts[j][0]+ y_sum += pts[j][1]+ j += 1+ xs.append(p_sum / (j - i))+ ys.append(y_sum / (j - i))+ ws.append(float(j - i))+ i = j+ # Stack of blocks: [y_sum, n, first_index].+ blocks = []+ for x, y, w in zip(xs, ys, ws):+ blocks.append([y * w, w, len(blocks)])+ while len(blocks) >= 2 and (+ blocks[-2][0] / blocks[-2][1] > blocks[-1][0] / blocks[-1][1]+ ):+ y2, n2, f2 = blocks.pop()+ y1, n1, f1 = blocks.pop()+ blocks.append([y1 + y2, n1 + n2, f1])+ self.knots = []+ for y_sum, n, first in blocks:+ self.knots.append((xs[first], y_sum / n))++ def apply(self, p):+ k = self.knots+ if not k:+ return p+ if p <= k[0][0]:+ return k[0][1]+ if p >= k[-1][0]:+ return k[-1][1]+ lo, hi = 0, len(k) - 1+ while hi - lo > 1:+ mid = (lo + hi) // 2+ if k[mid][0] <= p:+ lo = mid+ else:+ hi = mid+ (x0, y0), (x1, y1) = k[lo], k[hi]+ if x1 == x0:+ return y0+ return y0 + (y1 - y0) * (p - x0) / (x1 - x0)+++def logit(p):+ p = min(max(p, LOGIT_EPS), 1.0 - LOGIT_EPS)+ return math.log(p / (1.0 - p))+++def sigmoid(z):+ if z >= 0:+ return 1.0 / (1.0 + math.exp(-z))+ e = math.exp(z)+ return e / (1.0 + e)+++def temperature_nll(pairs, ln_t):+ """NLL of the pairs under p' = sigmoid(logit(p)/T) (Platt 1999; Guo et+ al. 2017). Saturated pairs contribute the eps-clamped log-probability —+ the softplus form the Rust module uses."""+ t = math.exp(ln_t)+ nll = 0.0+ for p, y in pairs:+ q = min(max(sigmoid(logit(p) / t), LOGIT_EPS), 1.0 - LOGIT_EPS)+ nll += -(y * math.log(q) + (1.0 - y) * math.log(1.0 - q))+ return nll+++class Temperature:+ def __init__(self, pairs):+ lo, hi = T_BRACKET+ gr = (math.sqrt(5.0) - 1.0) / 2.0+ a, b = lo, hi+ c = b - gr * (b - a)+ d = a + gr * (b - a)+ fc, fd = temperature_nll(pairs, c), temperature_nll(pairs, d)+ for _ in range(GOLDEN_ITERS):+ if fc < fd:+ b, d, fd = d, c, fc+ c = b - gr * (b - a)+ fc = temperature_nll(pairs, c)+ else:+ a, c, fc = c, d, fd+ d = a + gr * (b - a)+ fd = temperature_nll(pairs, d)+ self.t = math.exp((a + b) / 2.0)++ def apply(self, p):+ return sigmoid(logit(p) / self.t)+++# ── the measurement ──────────────────────────────────────────────────────────+++def bin_pairs(bins):+ """Expand binned aggregates into pairs — the same reconstruction as the+ Rust module's bin_pairs test helper."""+ out = []+ for conf, acc, n in bins:+ positives = round(acc * n)+ out.extend((conf, 1.0 if i < positives else 0.0) for i in range(n))+ return out+++def main():+ ap = argparse.ArgumentParser(description=__doc__)+ ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE)+ ap.add_argument("--out", type=Path, default=None)+ args = ap.parse_args()++ run = json.loads(args.baseline.read_text())+ cal = run["calibration"]+ curve = cal["curve"]+ bins = [+ (curve[str(i)]["mean_conf"], curve[str(i)]["acc"], curve[str(i)]["n"])+ for i in range(cal["bins"])+ ]+ all_pairs = bin_pairs(bins)+ fit_pairs = bin_pairs(bins[::2]) # even bins+ held_pairs = bin_pairs(bins[1::2]) # odd bins — whole regions unseen++ raw_all = ece(all_pairs)+ raw_held = ece(held_pairs)++ iso = Isotonic(fit_pairs)+ iso_calibrated = [(iso.apply(p), y) for p, y in held_pairs]+ iso_held = ece(iso_calibrated)++ temp = Temperature(fit_pairs)+ temp_calibrated = [(temp.apply(p), y) for p, y in held_pairs]+ temp_held = ece(temp_calibrated)++ result = {+ "issue": 1063,+ "gate": "Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out",+ "gate_threshold": GATE,+ "baseline_run": str(args.baseline),+ "method_note": (+ "The baseline retained 10 binned aggregates (n_pairs=19440), not "+ "the raw pairs; pairs are reconstructed per bin at mean_conf with "+ "positives = round(acc*n), and the held-out split is at BIN "+ "granularity: fit on even bins, evaluate on odd bins (whole "+ "probability regions the fit never saw). The pair-level held-out "+ "re-run needs the model's raw scores and is the rc.80 final form."+ ),+ "pairs": {+ "total": len(all_pairs),+ "fitted_on": len(fit_pairs),+ "held_out": len(held_pairs),+ },+ "raw_ece": {+ "all_bins": round(raw_all, 4),+ "published": cal["ece"],+ "held_out_bins": round(raw_held, 4),+ },+ "isotonic": {+ "held_out_ece": round(iso_held, 4),+ "meets_gate": iso_held <= GATE,+ "knots": [[round(x, 4), round(y, 4)] for x, y in iso.knots],+ },+ "temperature": {+ "held_out_ece": round(temp_held, 4),+ "meets_gate": temp_held <= GATE,+ "t": round(temp.t, 4),+ },+ }++ print(json.dumps(result, indent=2))+ if args.out:+ args.out.parent.mkdir(parents=True, exist_ok=True)+ args.out.write_text(json.dumps(result, indent=2) + "\n")+ print(f"\nwrote {args.out}", file=sys.stderr)++ ok = raw_all <= cal["ece"] + 0.0005 and iso_held <= GATE+ if not ok:+ print(+ "GATE CHECK FAILED: reproduction "+ f"{raw_all:.4f} vs published {cal['ece']}, isotonic held-out "+ f"{iso_held:.4f} vs gate {GATE}",+ file=sys.stderr,+ )+ return 1+ print(+ f"\ngate met: raw {raw_held:.4f} held-out -> isotonic {iso_held:.4f} "+ f"(<= {GATE}); temperature {temp_held:.4f}",+ file=sys.stderr,+ )+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.json b/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.jsonnew file mode 100644index 000000000..71d897c15--- /dev/null+++ b/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.json@@ -0,0 +1,48 @@+{+ "issue": 1063,+ "gate": "Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out",+ "gate_threshold": 0.1,+ "baseline_run": "/workspace/.claude/worktrees/wf_6ba42604-1b6-1/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json",+ "method_note": "The baseline retained 10 binned aggregates (n_pairs=19440), not the raw pairs; pairs are reconstructed per bin at mean_conf with positives = round(acc*n), and the held-out split is at BIN granularity: fit on even bins, evaluate on odd bins (whole probability regions the fit never saw). The pair-level held-out re-run needs the model's raw scores and is the rc.80 final form.",+ "pairs": {+ "total": 19440,+ "fitted_on": 11446,+ "held_out": 7994+ },+ "raw_ece": {+ "all_bins": 0.3109,+ "published": 0.3109,+ "held_out_bins": 0.3831+ },+ "isotonic": {+ "held_out_ece": 0.033,+ "meets_gate": true,+ "knots": [+ [+ 0.0455,+ 0.0002+ ],+ [+ 0.2434,+ 0.0034+ ],+ [+ 0.4445,+ 0.0077+ ],+ [+ 0.6451,+ 0.0224+ ],+ [+ 0.8452,+ 0.1242+ ]+ ]+ },+ "temperature": {+ "held_out_ece": 0.3903,+ "meets_gate": false,+ "t": 1.1517+ }+}diff --git a/docs/DECISIONS.md b/docs/DECISIONS.mdindex d60b2f535..d5b8a69a0 100644--- a/docs/DECISIONS.md+++ b/docs/DECISIONS.md@@ -119,6 +119,65 @@ number or the config refuses to boot (0 would erase a correction from thevote, a negative value would invert it). The boosted weight is the weight thevote used, so `/_decide` shows it in each neighbour's `weight`.+## Calibration: `p_cal` beside every `p_raw` (#1063)++A ladder probability ranks before it odds. On the FiQA rerank baseline a+hosted noul of 0.93 meant relevance 34 % of the time — ECE 0.3109+([benchmarks/decisions-calibration](../benchmarks/decisions-calibration)).+With `[decisions] calibration` set, the node fits a correction on its own+recorded outcomes and ships the calibrated probability BESIDE the raw one:++- **`isotonic`** — non-decreasing regression of the empirical positive rate+ on the raw probability (PAVA), interpolated between knots. The method that+ meets the FiQA gate: held-out ECE 0.0330 against raw 0.3831 on the same+ bins (gate ≤ 0.10).+- **`temperature`** — one scalar on the log-odds, `p' = σ(logit(p)/T)`,+ fitted on negative log-likelihood. Rank-preserving by construction, and+ the right tool on a thin history; on the FiQA-shaped curve it barely moves+ (held-out 0.3903) because one scalar cannot map 0.93 → 0.34 while keeping+ 0.05 → 0.0002.++Both fit on a **deterministic held-out fifth** of the outcome pairs+(stable-sorted, every fifth held out — no RNG, no clock, so the same history+produces bit-identical `p_cal`), and the fit is cached per index and+refreshed at most once a minute; `GET /_decide/_calibration` always refits.++**Recording an outcome — the fit's data contract.** A history document is a+calibration pair when it carries the truth `label` (the configured+`label_field`), the `p` that was served **for that document's own label**,+and **no `source` field**:++```sh+curl -X PUT localhost:9200/judgements/_doc/outcome-42 -H 'content-type: application/json' \+ -d '{"text": "the message that was decided", "label": "false", "p": 0.9}'+```++That reads as "the system said 0.9 for its own label; the truth was `false`"+— the positive-label probability 1 − 0.9 against outcome 0. A document with+`source` is a flywheel answer, i.e. the model's own prediction, and is+excluded: a fit on predictions would learn the model is right by+construction. `human: true` corrections carrying the `p` they correct are the+best pairs there are — truth plus the number that was wrong. Ten labelled+pairs minimum, or the node says so instead of extrapolating.++**The honesty rules, all three on the wire.** No calibration configured → no+`p_cal` field at all (`calibration: {"configured": "none"}` on `/_decide`).+Configured but unfitted → `p_cal: null` with the reason published in the+`calibration` block beside it. Fitted → `p_cal` is the fit's value, and it is+never a copy of `p_raw` pretending to be calibrated. Whenever `p_cal`+exists, the `calibration` block rides with it — method, scope, the fit's ECE+raw and calibrated, and what it was fitted on — so **no probability ships+without an ECE beside it**. `p_raw` is always the honest raw share;+`confidence` keeps its raw meaning and the abstain gate keeps applying to it+(calibration changes what the number means, not the verdict).++**Scope.** The calibrated quantity is the positive label's probability — the+noul's answer, `/_decide`'s `p_raw`. Per-option `choice` shares are a+different quantity and are not calibrated by this fit (the block says+`"scope": "noul"`). The fit applies to every tier's noul through one seam —+history vote, local head, and (when that tier is built) a hosted passthrough+score — with no usable fit meaning "say so", never `p_cal = p_raw`.+## Set it up```toml@@ -131,6 +190,7 @@ text_field = "text"positive_label = "true" # the label whose share a noul answersmin_confidence = 0.0 # /_decide abstains below thishuman_weight = 2.0 # vote-weight multiplier for history docs carrying human: true+calibration = "none" # none | isotonic | temperature — p_cal beside every p_raw (#1063)```The history index is ordinary documents: one per example, with the text the@@ -222,11 +282,13 @@ tier is armed), `question` (required), `k` (default the configured 10,clamped 1..100), `positive_label` (default the configured one). The responsereturns the label, its confidence, the verdict (`abstain` below`decisions.min_confidence`, or when no labelled neighbour exists), the **tier**-and **source** that answered (the same tier name under both fields), and the-**neighbours** — each with `_id`, `label`, `_score` (the engine's own BM25-score), `weight` (1/rank, × `decisions.human_weight` for a `human: true`-document), and the text — so every answer can be checked against the evidence-that produced it. A local-tier answer has no neighbours (having none is what+and **source** that answered (the same tier name under both fields), the+**`p_raw`** — the positive label's vote share, the quantity a noul answers,+not the winner's confidence — with **`p_cal`** beside it when calibration is+configured (see above), and the **neighbours** — each with `_id`, `label`,+`_score` (the engine's own BM25 score), `weight` (1/rank, ×+`decisions.human_weight` for a `human: true` document), and the text — so+every answer can be checked against the evidence that produced it. A local-tier answer has no neighbours (having none is whattier 2 means) and carries `tier: "local"`, `source: "local"`,`model: "xerj-decide-local-1"`; when it did not abstain and the request namedan index, it is cached into that index (the flywheel above).@@ -236,6 +298,27 @@ curl -s localhost:9200/_decide -H 'content-type: application/json' \-d '{"index":"judgements","question":"refund my subscription","k":5}'```+## `GET /_decide/_calibration`++The publication (#1063): the reliability curve of a labelled history — every+occupied bin's `[lo, hi)`, count, mean probability and empirical positive+rate — for the raw probabilities and (when fitted) the calibrated ones, the+ECE of each, the fit's parameters (an isotonic fit's knots, or the+temperature), the pair counts (fitted on / held out / total), and the date+range of the fitted documents. Always refits, so it doubles as the+operator's refresh; the fit it publishes becomes the cached fit every decide+request reads.++```sh+curl -s 'localhost:9200/_decide/_calibration?index=judgements'+```++`?index=` names the history (default: the configured `[decisions] index`; a+request naming nothing on a node that configures nothing is a 422, not an+empty curve). The raw curve and raw ECE publish even when calibration is not+configured — the reliability of the raw probabilities is audit information+in its own right — with a `reason` naming the setting that would fit them.+## Numbers, and only the ones we measuredMeasured, ours, reproducible (raw JSON and runners in the results dirs):@@ -243,6 +326,12 @@ Banking77 77-way intent 0.933 accuracy / ECE 0.012 (86.9% of traffic clearsconfidence 0.8 at 0.979 accuracy); SMS spam 0.983 accuracy / 0.944 F1 under amillisecond on BM25 alone; both in[benchmarks/decisions-as-retrieval](../benchmarks/decisions-as-retrieval).+The calibration gate: isotonic held-out ECE 0.0330 against raw 0.3831 on the+FiQA rerank baseline's own curve (published raw 0.3109 reproduced exactly;+gate ≤ 0.10), measured at bin granularity from the retained aggregates —+the pair-level re-run needs the model's raw scores and is the rc.80 final+form — in+[benchmarks/decisions-calibration](../benchmarks/decisions-calibration).The wire gate transcript is in[benchmarks/systemone-gate](../benchmarks/systemone-gate). A third, smallerrun measures `/_decide` itself on an email-labelling history — 4 labels,diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rsindex b08bbf087..dc818c6e1 100644--- a/engine/crates/xerj-api/src/router.rs+++ b/engine/crates/xerj-api/src/router.rs@@ -300,6 +300,13 @@ pub fn build_es_compat_router(state: AppState) -> Router {// The audit twin of /v1/systemone: same vote, named index, neighbours// and abstain instead of a wire-shaped error..route("/_decide", post(systemone_api::decide))+ // The decide surface's published reliability (#1063): the binned+ // curve, the ECE raw and calibrated, the fit params. GET, read-only,+ // same posture as the surface it publishes.+ .route(+ "/_decide/_calibration",+ get(systemone_api::decide_calibration),+ )// #1056, ES-compat surface: same handler as /v1/ask; the index// rides in the body, exactly like /_decide and /_msearch..route("/_ask", post(ask_api::ask))diff --git a/engine/crates/xerj-api/src/state.rs b/engine/crates/xerj-api/src/state.rsindex 9e2e9e330..73fd7a885 100644--- a/engine/crates/xerj-api/src/state.rs+++ b/engine/crates/xerj-api/src/state.rs@@ -553,6 +553,12 @@ pub struct AppState {/// from the environment again. A test arms a node by replacing this/// field, so parallel tests never race on process-wide env state.pub decide: Arc<crate::systemone_api::DecideSettings>,+ /// The decide surface's calibration fits (#1063): one cached fit per+ /// decisions index, lazily computed by [`crate::systemone_api`] and+ /// refreshed by `GET /_decide/_calibration`. Held here, not in the+ /// handler, so every request path that ships a probability reads the same+ /// fit.+ pub decide_calibration: Arc<crate::systemone_api::CalibrationCache>,}impl AppState {@@ -592,9 +598,11 @@ impl AppState {&config.rerank.endpoint,));let decide = Arc::new(crate::systemone_api::DecideSettings::from_env());+ let decide_calibration = Arc::new(crate::systemone_api::CalibrationCache::default());Self {rerank,decide,+ decide_calibration,config: Arc::new(config),engine: Arc::new(engine),metrics: Arc::new(metrics),diff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rsindex c1d7db857..25a2b999b 100644--- a/engine/crates/xerj-api/src/systemone_api.rs+++ b/engine/crates/xerj-api/src/systemone_api.rs@@ -59,6 +59,24 @@//! in the vote — see [`neighbour_weight`]. The weight is read in the vote//! aggregation, so a correction outranks the cached answers it corrects.//!+//! # Calibration: `p_cal` beside every `p_raw` (#1063)+//!+//! A ladder probability ranks before it odds: on the FiQA rerank baseline a+//! hosted noul of 0.93 meant relevance 34 % of the time (ECE 0.3109). With+//! `[decisions] calibration = isotonic | temperature`, the node fits a+//! correction on a held-out fifth of its recorded outcomes ([`xerj_common::+//! calibration`]) and answers carry the calibrated probability BESIDE the raw+//! one — `p_raw` never replaced, `p_cal` absent when no calibration is+//! configured and `null` (with a published reason) when configured but+//! unfitted, never a copy of `p_raw` pretending to be calibrated. The fit+//! applies to every tier's probability through one seam, so the hosted tier,+//! when it is built, calibrates its passthrough score the same way. The+//! calibrated quantity is the positive label's probability (a noul, `/_decide`'s+//! `p_raw`); per-option `choice` shares are a different quantity and are not+//! calibrated by this fit — every response that ships a probability carries+//! the fit's ECE beside it, and `GET /_decide/_calibration` publishes the+//! whole reliability curve.+//!//! # The wire contract, and where we deliberately break it//!//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.@@ -104,10 +122,13 @@//! vote analogue and no benchmark: 422, in the docs.use std::collections::BTreeMap;+use std::collections::HashMap;#[cfg(feature = "decide-local")]use std::path::Path;#[cfg(feature = "decide-local")]use std::path::PathBuf;+use std::sync::Arc;+use std::time::Duration;use std::time::Instant;use axum::extract::State;@@ -115,6 +136,10 @@ use axum::http::StatusCode;use axum::response::IntoResponse;use axum::Json;use serde_json::{json, Value};+use xerj_common::calibration::CalibrationMethod;+use xerj_common::calibration::CalibrationPair;+use xerj_common::calibration::CalibrationReport;+use xerj_common::calibration::FittedCalibration;use xerj_common::config::DecisionsConfig;use xerj_query::parse_request;@@ -304,6 +329,291 @@ fn is_human_correction(source: &Value) -> bool {source.get("human") == Some(&Value::Bool(true))}+// ─────────────────────────────────────────────────────────────────────────────+// Calibration — p_cal beside every p_raw (#1063)+// ─────────────────────────────────────────────────────────────────────────────++/// How many history documents one fit reads, at most: the search page cap+/// (`index.max_result_window`, 10,000 by default). A history with more+/// labelled outcomes than this fits on the first page in engine order,+/// canonically re-sorted before the split — the histories this feature is+/// for (thousands of cached answers, hundreds of recorded outcomes) sit far+/// under it.+const MAX_FIT_DOCS: usize = 10_000;++/// Fewer labelled pairs than this and there is no fit to publish: the+/// response says so rather than extrapolate a curve from a handful of points.+/// Ten is the smallest set whose held-out fifth still evaluates.+const MIN_FIT_PAIRS: usize = 10;++/// A cached fit refreshes at most this often on demand, so a decide request+/// never pays a full index scan per call and an operator who indexes more+/// outcome documents sees the fit move within a minute without a restart.+/// `GET /_decide/_calibration` always refits, whatever this says.+const FIT_REFRESH: Duration = Duration::from_secs(60);++/// One per-index cache entry: the report (or the reason there is none) and+/// when it was computed. The clock stamps the CACHE, never the fit — the fit+/// is a pure function of the pair multiset, the #940 rule.+struct CachedFit {+ report: Result<Arc<CalibrationReport>, String>,+ fitted_at: Instant,+}++/// The node's calibration state: one cached fit per decisions index, held in+/// [`AppState::decide_calibration`] so every probability-shipping path reads+/// the same fit. Fitted lazily on first use, refreshed at most once a minute,+/// force-refreshed by [`decide_calibration`] (the publication endpoint).+#[derive(Default)]+pub struct CalibrationCache {+ entries: tokio::sync::Mutex<HashMap<String, CachedFit>>,+}++impl CalibrationCache {+ /// Publish a fit the caller already computed (the `_calibration`+ /// endpoint's own scan) as the index's cached fit.+ async fn refresh_from(&self, index: &str, report: Arc<CalibrationReport>) {+ self.entries.lock().await.insert(+ index.to_string(),+ CachedFit {+ report: Ok(report),+ fitted_at: Instant::now(),+ },+ );+ }+}++/// What calibration a request has available — the one lookup every+/// probability-emitting path on this surface shares.+enum CalibrationState {+ /// `[decisions] calibration` is `none`: `p_raw` ships alone and no+ /// `p_cal` exists at all.+ Off,+ /// Configured and fitted: the report every `p_cal` and ECE comes from.+ Fitted(Arc<CalibrationReport>),+ /// Configured but no usable fit — the string is why, published beside the+ /// `p_cal: null` it explains.+ Unfit(String),+}++/// Resolve the request's calibration state — at most one scan of the+/// decisions index, cached per `cfg.index`. `force` is the operator's+/// explicit refit. The scan and the fit are deterministic, so two requests+/// between refreshes serve bit-identical `p_cal`.+async fn calibration_state(+ state: &AppState,+ cfg: &DecisionsConfig,+ force: bool,+) -> CalibrationState {+ if cfg.calibration == CalibrationMethod::None {+ return CalibrationState::Off;+ }+ if cfg.index.is_empty() {+ return CalibrationState::Unfit(+ "no decisions index: calibration is fitted on labelled history — set [decisions] \+ index, or name one per request on /_decide"+ .to_string(),+ );+ }+ let mut entries = state.decide_calibration.entries.lock().await;+ let fresh = match entries.get(&cfg.index) {+ Some(cached) => !force && cached.fitted_at.elapsed() <= FIT_REFRESH,+ None => false,+ };+ if fresh {+ let cached = entries.get(&cfg.index).expect("checked above");+ return match &cached.report {+ Ok(report) => CalibrationState::Fitted(report.clone()),+ Err(reason) => CalibrationState::Unfit(reason.clone()),+ };+ }+ let (pairs, ts_range) = match scan_calibration_pairs(state, cfg).await {+ Ok(found) => found,+ Err(reason) => {+ entries.insert(+ cfg.index.clone(),+ CachedFit {+ report: Err(reason.clone()),+ fitted_at: Instant::now(),+ },+ );+ return CalibrationState::Unfit(reason);+ }+ };+ let outcome = fit_pairs(cfg, &pairs, ts_range);+ let state_out = match &outcome {+ Ok(report) => CalibrationState::Fitted(report.clone()),+ Err(reason) => CalibrationState::Unfit(reason.clone()),+ };+ entries.insert(+ cfg.index.clone(),+ CachedFit {+ report: outcome,+ fitted_at: Instant::now(),+ },+ );+ state_out+}++/// Fit the configured method on scanned pairs, or say why there is no fit to+/// ship. Pure plumbing over [`xerj_common::calibration::fit_calibration`].+fn fit_pairs(+ cfg: &DecisionsConfig,+ pairs: &[CalibrationPair],+ ts_range: Option<(String, String)>,+) -> Result<Arc<CalibrationReport>, String> {+ if pairs.len() < MIN_FIT_PAIRS {+ return Err(unfit_reason(cfg, pairs.len()));+ }+ let mut report = xerj_common::calibration::fit_calibration(cfg.calibration, pairs)+ .ok_or_else(|| "[decisions] calibration is none; nothing to fit".to_string())?;+ report.ts_range = ts_range;+ Ok(Arc::new(report))+}++/// The published reason a configured calibration has no fit — names what a+/// fit needs and how to record it.+fn unfit_reason(cfg: &DecisionsConfig, pairs: usize) -> String {+ format!(+ "only {pairs} labelled calibration pair{} in `{}` — a fit needs at least {MIN_FIT_PAIRS} \+ (#1063): record outcomes by indexing documents carrying the truth label, the p that \+ was served, and no `source` field (a `source` marks a cached answer, whose label is \+ the model's own prediction, and a fit on predictions would learn the model is right \+ by construction)",+ if pairs == 1 { "" } else { "s" },+ cfg.index+ )+}++/// One history document's calibration pair, or `None` when the document is+/// not a labelled outcome. Pure, and unit-tested: this function IS the+/// data contract of the fit.+///+/// A document is an OUTCOME — not a prediction — when it names a truth label+/// in the configured `label_field`, carries the probability that was served+/// in `p`, and has NO `source` field: `source` marks a flywheel answer+/// (#1061), whose label is the model's own guess, and calibrating on guesses+/// would teach the layer the model is right by construction. Record an+/// outcome with the ordinary indexing path:+/// `{"text": …, "label": <the truth>, "p": <what was served>}` — `human:+/// true` optional, the vote already weights it. The pair is the positive+/// label's probability — `p` when the label is the positive one, `1 − p`+/// when it is not — against the outcome `label == positive_label`.+fn pair_from_doc(source: &Value, cfg: &DecisionsConfig) -> Option<CalibrationPair> {+ if source.get("source").is_some() {+ return None; // a cached answer: a prediction, not an outcome+ }+ let label = source.get(&cfg.label_field).and_then(Value::as_str)?;+ let p = source.get("p").and_then(Value::as_f64)?;+ if !p.is_finite() || !(0.0..=1.0).contains(&p) {+ return None; // a p that is not a probability is not calibration data+ }+ let positive = label == cfg.positive_label;+ Some(CalibrationPair {+ p: if positive { p } else { 1.0 - p },+ y: positive as u8 as f64,+ })+}++/// Scan the decisions index for labelled outcome documents and return them+/// as calibration pairs, with the RFC 3339 range of their `ts` fields.+async fn scan_calibration_pairs(+ state: &AppState,+ cfg: &DecisionsConfig,+) -> Result<(Vec<CalibrationPair>, Option<(String, String)>), String> {+ let idx = state.engine.get_index(&cfg.index).map_err(|e| {+ format!(+ "decisions index `{}` could not be searched for calibration pairs: {e}",+ cfg.index+ )+ })?;+ let query_body = json!({+ "query": { "match_all": {} },+ "size": MAX_FIT_DOCS,+ "_source": true,+ });+ let search_req =+ parse_request(&query_body).map_err(|e| format!("internal query would not parse: {e}"))?;+ let result = idx+ .search(&search_req)+ .await+ .map_err(|e| format!("decisions index `{}` could not be searched: {e}", cfg.index))?;+ let mut pairs = Vec::with_capacity(result.hits.len());+ let mut oldest: Option<String> = None;+ let mut newest: Option<String> = None;+ for hit in &result.hits {+ if let Some(pair) = pair_from_doc(&hit.source, cfg) {+ pairs.push(pair);+ // The published range spans the FITTED documents — the ones that+ // became pairs — not every cached answer in the index.+ if let Some(ts) = hit.source.get("ts").and_then(Value::as_str) {+ // RFC 3339 sorts lexicographically, so min/max by string is+ // the range.+ oldest = Some(match oldest {+ Some(o) if o.as_str() <= ts => o,+ _ => ts.to_string(),+ });+ newest = Some(match newest {+ Some(n) if n.as_str() >= ts => n,+ _ => ts.to_string(),+ });+ }+ }+ }+ Ok((pairs, oldest.zip(newest)))+}++/// `p_cal` for one `p_raw` under the request's calibration state: absent+/// when calibration is off, `null` when configured but unfit (the reason+/// rides in the calibration block), and the fit's value otherwise — never a+/// copy of `p_raw` pretending to be calibrated.+fn p_cal_value(state: &CalibrationState, p_raw: f64) -> Option<Value> {+ match state {+ CalibrationState::Off => None,+ CalibrationState::Unfit(_) => Some(Value::Null),+ CalibrationState::Fitted(report) => Some(json!(round6(report.fit.apply(p_raw)))),+ }+}++/// The calibration block a response carries whenever calibration is+/// configured: the method, its scope, the fit's held-out ECE raw and+/// calibrated, and what it was fitted on. This block is the "ECE beside+/// every probability" — the surface's honesty contract (#1063).+fn calibration_block(state: &CalibrationState) -> Option<Value> {+ match state {+ CalibrationState::Off => None,+ CalibrationState::Fitted(report) => Some(json!({+ "method": report.method.as_str(),+ "scope": "noul",+ "ece": {+ "raw": round6(report.ece_raw),+ "calibrated": round6(report.ece_calibrated),+ },+ "fitted_on": report.fitted_on,+ "held_out": report.held_out,+ "labelled_pairs": report.labelled_pairs,+ })),+ CalibrationState::Unfit(reason) => Some(json!({+ "reason": reason,+ })),+ }+}++/// The fit's parameters, published by `/_decide/_calibration`: an isotonic+/// fit's knots, or the temperature.+fn fit_params_json(fit: &FittedCalibration) -> Value {+ match fit {+ FittedCalibration::Isotonic(iso) => json!({+ "knots": iso+ .knots()+ .iter()+ .map(|(x, y)| json!([round6(*x), round6(*y)]))+ .collect::<Vec<_>>(),+ }),+ FittedCalibration::Temperature(t) => json!({ "temperature": round6(t.t) }),+ }+}+// ─────────────────────────────────────────────────────────────────────────────// DecideSettings — the node's ladder configuration// ─────────────────────────────────────────────────────────────────────────────@@ -518,6 +828,11 @@ pub async fn systemone(if cfg.index.is_empty() && !local_enabled {return not_configured();}+ // One calibration lookup per request (#1063): every tier's noul+ // probability below reads the same fit, and the answers' strictly-parsed+ // wire objects stay untouched — p_raw/p_cal ride in `decisions.evidence`,+ // the extras channel established clients ignore.+ let calibration = calibration_state(&state, &cfg, false).await;let questions = match body.get("questions") {Some(Value::Object(q)) if !q.is_empty() => q,@@ -632,17 +947,26 @@ pub async fn systemone(.iter().max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap()).expect("non-empty");- evidence.insert(- id.clone(),- json!({- "tier": DecideTier::History.as_str(),- "source": DecideTier::History.as_str(),- "label": best_label,- "support": round6(best_weight / total),- "neighbours": labelled.len(),- "found": neighbours.len(),- }),- );+ let mut evidence_value = json!({+ "tier": DecideTier::History.as_str(),+ "source": DecideTier::History.as_str(),+ "label": best_label,+ "support": round6(best_weight / total),+ "neighbours": labelled.len(),+ "found": neighbours.len(),+ });+ // The calibrated quantity is the noul's — the positive+ // label's share — so only a noul question carries p_raw/p_cal+ // here. A choice's per-option shares are a different+ // quantity; the decisions.calibration block names the scope.+ if matches!(question.kind, Kind::Noul) {+ let p_raw = share(&cfg.positive_label);+ evidence_value["p_raw"] = json!(round6(p_raw));+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {+ evidence_value["p_cal"] = p_cal;+ }+ }+ evidence.insert(id.clone(), evidence_value);}Err(failure) => {match resolve_tier(&failure, local_enabled) {@@ -753,17 +1077,25 @@ pub async fn systemone(.zip(&scores).max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap()).expect("labels are non-empty");- evidence.insert(- id.clone(),- json!({- "tier": DecideTier::Local.as_str(),- "source": DecideTier::Local.as_str(),- "model": LOCAL_MODEL_ID,- "label": best_label,- "support": round6(*best_score as f64),- "hypotheses": Value::Object(by_label),- }),- );+ let mut evidence_value = json!({+ "tier": DecideTier::Local.as_str(),+ "source": DecideTier::Local.as_str(),+ "model": LOCAL_MODEL_ID,+ "label": best_label,+ "support": round6(*best_score as f64),+ "hypotheses": Value::Object(by_label),+ });+ // The head's noul is `scores[0]` — the positive hypothesis's+ // probability — so it calibrates through the same seam as the+ // vote's (and, when that tier exists, a hosted score's).+ if matches!(question.kind, Kind::Noul) {+ let p_raw = scores[0] as f64;+ evidence_value["p_raw"] = json!(round6(p_raw));+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {+ evidence_value["p_cal"] = p_cal;+ }+ }+ evidence.insert(id.clone(), evidence_value);cached.push(CachedAnswer {text: question.vote_text.clone(),label: best_label.clone(),@@ -806,6 +1138,12 @@ pub async fn systemone("took_ms".into(),json!(started.elapsed().as_millis() as u64),);+ // The ECE beside every probability (#1063): present whenever calibration+ // is configured — fitted (method, ECE, fitted-on) or unfit (reason) —+ // and absent when the node never promised a p_cal.+ if let Some(block) = calibration_block(&calibration) {+ decisions.insert("calibration".into(), block);+ }if local_answered > 0 {decisions.insert("local_model".into(), json!(LOCAL_MODEL_ID));decisions.insert("local_answered".into(), json!(local_answered));@@ -898,6 +1236,11 @@ pub async fn decide(cfg.label_field = defaults.label_field.clone();cfg.text_field = defaults.text_field.clone();+ // One calibration lookup per request (#1063), against the index THIS+ // request named — the fit is cached per index, so the audit surface and+ // the wire surface read the same fit for the same history.+ let calibration = calibration_state(&state, &cfg, false).await;+let neighbours = if cfg.index.is_empty() {Err(HistorySupport::NotConfigured)} else {@@ -935,6 +1278,16 @@ pub async fn decide(for (w, label, _) in &neighbours {*weight_by_label.entry(label.to_string()).or_insert(0.0) += *w;}+ // The calibrated quantity (#1063): the positive label's share — the same+ // quantity a noul answers on the wire — NOT the winner's share that+ // `confidence` reports. `confidence` keeps its raw meaning and the+ // abstain gate keeps applying to it: calibration changes what the number+ // means, not the verdict.+ let p_raw = if total > 0.0 {+ weight_by_label.get(&positive).copied().unwrap_or(0.0) / total+ } else {+ 0.0+ };let (label, confidence, abstain, reason) = match weight_by_label.iter().max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())@@ -988,6 +1341,16 @@ pub async fn decide("source": DecideTier::History.as_str(),"took_ms": started.elapsed().as_millis() as u64,});+ // p_cal beside p_raw, never replacing it (#1063): p_raw is always the+ // honest raw share, p_cal exists only when calibration is configured+ // (null, with the reason in the block, when configured but unfitted),+ // and the calibration block is the ECE that rides beside both.+ resp["p_raw"] = json!(round6(p_raw));+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {+ resp["p_cal"] = p_cal;+ }+ resp["calibration"] =+ calibration_block(&calibration).unwrap_or(json!({ "configured": "none" }));if let Some(r) = reason {resp["reason"] = json!(r);}@@ -1026,6 +1389,11 @@ async fn decide_local(Err(e) => return local_unavailable(&e, local.model_dir()),};let scores = &scores[0];+ // The head's positive-label probability is `scores[0]` — the same+ // quantity a noul answers, through the same calibration seam as every+ // other tier's (#1063).+ let p_raw = scores[0] as f64;+ let calibration = calibration_state(state, cfg, false).await;let (label, confidence) = if scores[0] >= scores[1] {(labels[0].clone(), scores[0] as f64)} else {@@ -1062,6 +1430,14 @@ async fn decide_local("model": LOCAL_MODEL_ID,"took_ms": started.elapsed().as_millis() as u64,});+ // The audit surface always names its calibration state, and p_cal rides+ // beside p_raw exactly as on the history tier (#1063).+ resp["p_raw"] = json!(round6(p_raw));+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {+ resp["p_cal"] = p_cal;+ }+ resp["calibration"] =+ calibration_block(&calibration).unwrap_or(json!({ "configured": "none" }));if abstain {resp["reason"] = json!(format!("confidence {:.3} below decisions.min_confidence {:.3}",@@ -1071,6 +1447,104 @@ async fn decide_local(Json(resp).into_response()}+// ─────────────────────────────────────────────────────────────────────────────+// GET /_decide/_calibration+// ─────────────────────────────────────────────────────────────────────────────++/// Publish the calibration (#1063): the reliability curve of the labelled+/// history — binned `p_raw` against empirical frequency, with bin counts —+/// the ECE raw and calibrated, the fit's parameters, what it was fitted on,+/// and the date range of the fitted documents. Always refits, so this is the+/// operator's refresh as well as the publication; the fit it produces becomes+/// the cached fit every decide request reads.+///+/// `?index=` names the history (default: the configured `[decisions] index`).+/// The curve and raw ECE publish even when calibration is not configured —+/// the reliability of the raw probabilities is audit information in its own+/// right — with a `reason` naming the setting that would fit them.+pub async fn decide_calibration(+ State(state): State<AppState>,+ axum::extract::Query(params): axum::extract::Query<HashMap<String, String>>,+) -> axum::response::Response {+ let defaults = &state.config.decisions;+ let index = params+ .get("index")+ .map(String::as_str)+ .map(str::trim)+ .filter(|s| !s.is_empty())+ .unwrap_or(defaults.index.as_str())+ .to_string();+ if index.is_empty() {+ return unprocessable(+ "name an index with ?index=… or set [decisions] index — calibration is fitted \+ on a labelled history",+ );+ }+ let cfg = DecisionsConfig {+ index: index.clone(),+ ..defaults.clone()+ };+ let (pairs, ts_range) = match scan_calibration_pairs(&state, &cfg).await {+ Ok(found) => found,+ Err(reason) => return index_error(&index, &reason),+ };+ let curve_raw = xerj_common::calibration::reliability_curve(+ &pairs,+ xerj_common::calibration::CALIBRATION_BINS,+ );+ let ece_raw = xerj_common::calibration::ece_from_curve(&curve_raw);++ let mut resp = serde_json::Map::new();+ resp.insert("index".into(), json!(index));+ resp.insert("configured".into(), json!(cfg.calibration.as_str()));+ resp.insert("labelled_pairs".into(), json!(pairs.len()));+ resp.insert(+ "bins".into(),+ json!(xerj_common::calibration::CALIBRATION_BINS),+ );+ resp.insert(+ "curve".into(),+ json!({ "raw": curve_raw, "calibrated": Value::Null }),+ );+ resp.insert(+ "ece".into(),+ json!({ "raw": round6(ece_raw), "calibrated": Value::Null }),+ );+ if let Some((lo, hi)) = &ts_range {+ resp.insert("ts_range".into(), json!([lo, hi]));+ }+ match fit_pairs(&cfg, &pairs, ts_range) {+ Ok(report) => {+ resp.insert("method".into(), json!(report.method.as_str()));+ resp.insert("scope".into(), json!("noul"));+ resp.insert("fitted_on".into(), json!(report.fitted_on));+ resp.insert("held_out".into(), json!(report.held_out));+ resp.insert("params".into(), fit_params_json(&report.fit));+ resp.insert(+ "ece".into(),+ json!({+ "raw": round6(report.ece_raw),+ "calibrated": round6(report.ece_calibrated),+ }),+ );+ resp.insert(+ "curve".into(),+ json!({+ "raw": report.curve_raw,+ "calibrated": report.curve_calibrated,+ }),+ );+ // The publication is the refresh: every decide request after+ // this reads the fit just published.+ state.decide_calibration.refresh_from(&index, report).await;+ }+ Err(reason) => {+ resp.insert("reason".into(), json!(reason));+ }+ }+ Json(Value::Object(resp)).into_response()+}+// ─────────────────────────────────────────────────────────────────────────────// The vote// ─────────────────────────────────────────────────────────────────────────────@@ -1657,4 +2131,124 @@ mod tests {assert_eq!(DecideTier::Local.as_str(), "local");assert_eq!(DecideTier::Hosted.as_str(), "hosted");}++ // ─────────────────────────────────────────────────────────────────────────+ // Calibration (#1063)+ // ─────────────────────────────────────────────────────────────────────────++ /// The fit's data contract: a labelled outcome document is a pair, the+ /// positive label's probability against the outcome; a flywheel answer+ /// (`source`) is NOT, because its label is the model's own prediction —+ /// calibrating on predictions would learn the model is right by+ /// construction.+ #[test]+ fn calibration_pairs_come_from_outcomes_not_predictions() {+ let cfg = DecisionsConfig::default(); // label_field "label", positive "true"+ // An outcome the system got wrong: it said 0.9, the truth is "false".+ // The positive-label probability is 1 − 0.9, the outcome 0.+ let doc = json!({"text": "…", "label": "false", "p": 0.9});+ let pair = pair_from_doc(&doc, &cfg).expect("outcome pair");+ assert!(+ (pair.p - 0.1).abs() < 1e-12,+ "p_pos = 1 − p when label ≠ positive"+ );+ assert_eq!(pair.y, 0.0);+ // An outcome it got right: p of the positive label, outcome 1.+ let doc = json!({"text": "…", "label": "true", "p": 0.8});+ let pair = pair_from_doc(&doc, &cfg).expect("outcome pair");+ assert!((pair.p - 0.8).abs() < 1e-12);+ assert_eq!(pair.y, 1.0);+ // A human correction carrying the p it corrects: still an outcome,+ // and the best kind — truth plus the number that was wrong.+ let doc = json!({"text": "…", "label": "true", "p": 0.3, "human": true});+ assert!(pair_from_doc(&doc, &cfg).is_some());+ // A flywheel answer: a prediction, never a pair.+ let doc = json!({"text": "…", "label": "true", "p": 0.8, "source": "local", "ts": "…"});+ assert!(+ pair_from_doc(&doc, &cfg).is_none(),+ "a `source` document is the model's own answer"+ );+ // No p, no pair — a seeded example has a truth but no raw+ // probability to calibrate.+ assert!(pair_from_doc(&json!({"text": "…", "label": "true"}), &cfg).is_none());+ // No label, no pair.+ assert!(pair_from_doc(&json!({"text": "…", "p": 0.8}), &cfg).is_none());+ // A p that is not a probability is not calibration data.+ assert!(pair_from_doc(&json!({"label": "true", "p": 1.4}), &cfg).is_none());+ assert!(pair_from_doc(&json!({"label": "true", "p": -0.1}), &cfg).is_none());+ assert!(pair_from_doc(&json!({"label": "true"}), &cfg).is_none());+ }++ /// The three states on the wire: off ships no p_cal at all, unfit ships+ /// null with the reason published in the block, fitted ships the fit's+ /// value — and never a copy of p_raw pretending to be calibrated.+ #[test]+ fn p_cal_is_absent_null_or_calibrated_never_a_copy() {+ // Off: nothing.+ assert!(p_cal_value(&CalibrationState::Off, 0.7).is_none());+ assert!(calibration_block(&CalibrationState::Off).is_none());+ // Unfit: null, with the reason riding in the block.+ let unfit = CalibrationState::Unfit("only 3 labelled pairs".into());+ assert_eq!(p_cal_value(&unfit, 0.7), Some(Value::Null));+ let block = calibration_block(&unfit).expect("block");+ assert_eq!(block["reason"], "only 3 labelled pairs");+ // Fitted: the fit's value and its ECE, not p_raw echoed back. The+ // identity fit (one knot per distinct p, already monotone) would echo+ // — so use a fit that learned the opposite of the raw number.+ let report = Arc::new(+ xerj_common::calibration::fit_calibration(+ xerj_common::calibration::CalibrationMethod::Isotonic,+ &[+ CalibrationPair { p: 0.9, y: 0.0 },+ CalibrationPair { p: 0.9, y: 0.0 },+ CalibrationPair { p: 0.9, y: 0.0 },+ CalibrationPair { p: 0.9, y: 0.0 },+ CalibrationPair { p: 0.9, y: 0.0 },+ CalibrationPair { p: 0.2, y: 0.0 },+ CalibrationPair { p: 0.2, y: 0.0 },+ CalibrationPair { p: 0.2, y: 0.0 },+ CalibrationPair { p: 0.2, y: 0.0 },+ CalibrationPair { p: 0.2, y: 0.0 },+ ],+ )+ .expect("fit"),+ );+ let fitted = CalibrationState::Fitted(report);+ let p_cal = p_cal_value(&fitted, 0.9).expect("a value");+ let p_cal = p_cal.as_f64().expect("a number");+ assert!(+ (p_cal - 0.9).abs() > 1e-9,+ "a fit that learned 0.9 means 0 must not echo 0.9 back (got {p_cal})"+ );+ assert!(+ (p_cal - 0.0).abs() < 1e-9,+ "five negatives at 0.9 → p_cal 0"+ );+ let block = calibration_block(&fitted).expect("block");+ assert_eq!(block["method"], "isotonic");+ assert_eq!(block["scope"], "noul");+ assert!(+ block["ece"]["calibrated"].is_number(),+ "the ECE rides beside: {block}"+ );+ assert!(block["ece"]["raw"].is_number());+ assert_eq!(block["fitted_on"], 8);+ assert_eq!(block["held_out"], 2);+ }++ /// The unfit reason names what a fit needs and how to record it — an+ /// operator who sees `p_cal: null` can act on the string beside it.+ #[test]+ fn the_unfit_reason_names_the_recording_path() {+ let reason = unfit_reason(&DecisionsConfig::default(), 3);+ assert!(reason.contains("3 labelled calibration pairs"), "{reason}");+ assert!(+ reason.contains("10"),+ "the minimum is in the message: {reason}"+ );+ assert!(+ reason.contains("source"),+ "the flywheel exclusion is explained: {reason}"+ );+ }}diff --git a/engine/crates/xerj-api/tests/decide_calibration.rs b/engine/crates/xerj-api/tests/decide_calibration.rsnew file mode 100644index 000000000..9255944d3--- /dev/null+++ b/engine/crates/xerj-api/tests/decide_calibration.rs@@ -0,0 +1,508 @@+//! End-to-end checks for the decide surface's calibration layer (issue+//! #1063): `p_cal` beside every `p_raw`, the fit's ECE riding beside both,+//! `GET /_decide/_calibration` publishing the reliability curve, and the+//! honesty rules — no `p_cal` at all when calibration is not configured,+//! `null` with a published reason when configured but unfitted, and never a+//! copy of `p_raw` pretending to be calibrated.+//!+//! The history is built so the arithmetic is exact and the answer visibly+//! moves: twenty outcome documents whose raw positive-label probability is+//! 0.25, half of them true — isotonic pools the one distinct raw probability+//! into a single knot at the base rate 0.5, so every `p_cal` is 0.5 whatever+//! the vote's share was. (0.25/0.75 are the exactly-representable pair:+//! 1 − 0.75 = 0.25 in f64, where 1 − 0.8 = 0.19999999999999996 — two raw+//! probabilities, no pooling. Ask how the first run of this file failed.) The strictly-parsed `/v1/systemone`+//! answer objects stay untouched (the wire-compatibility #1072 established):+//! `p_raw`/`p_cal` ride in `decisions.evidence`, never in the answer.++use axum::body::Body;+use axum::http::{Request, StatusCode};+use serde_json::{json, Value};+use tower::ServiceExt;++use xerj_api::state::AppState;+use xerj_common::calibration::CalibrationMethod;++// ─────────────────────────────────────────────────────────────────────────────+// The node+// ─────────────────────────────────────────────────────────────────────────────++struct Node {+ native: axum::Router,+ app: axum::Router,+ _dir: tempfile::TempDir,+}++/// A node whose `[decisions]` block names `index`, with the calibration+/// method overrideable — the layer's one new setting.+async fn node_with(index: &str, calibration: CalibrationMethod) -> Node {+ let dir = tempfile::tempdir().expect("tempdir");+ let mut config = xerj_common::config::Config::default();+ config.server.data_dir = dir.path().to_string_lossy().into_owned();+ config.storage.wal_sync = xerj_common::config::WalSync::Async;+ config.decisions.index = index.to_string();+ config.decisions.calibration = calibration;+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");+ let state = AppState::new(config, engine, metrics);+ Node {+ native: xerj_api::router::build_native_router(state.clone()),+ app: xerj_api::router::build_es_compat_router(state),+ _dir: dir,+ }+}++impl Node {+ async fn call(+ &self,+ router: &axum::Router,+ method: &str,+ path: &str,+ body: Value,+ ) -> (StatusCode, Value) {+ let response = router+ .clone()+ .oneshot(+ Request::builder()+ .method(method)+ .uri(path)+ .header("content-type", "application/json")+ .body(Body::from(body.to_string()))+ .expect("request"),+ )+ .await+ .expect("response");+ let status = response.status();+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .expect("body");+ (+ status,+ serde_json::from_str(&String::from_utf8_lossy(&bytes)).unwrap_or(Value::Null),+ )+ }++ async fn systemone(&self, body: Value) -> (StatusCode, Value) {+ self.call(&self.native.clone(), "POST", "/v1/systemone", body)+ .await+ }+ async fn decide(&self, body: Value) -> (StatusCode, Value) {+ self.call(&self.app.clone(), "POST", "/_decide", body).await+ }+ async fn calibration(&self, query: &str) -> (StatusCode, Value) {+ self.call(&self.app.clone(), "GET", query, json!({})).await+ }+ async fn put_doc(&self, index: &str, id: &str, doc: Value) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "PUT",+ &format!("/{index}/_doc/{id}"),+ doc,+ )+ .await;+ assert!(st.is_success(), "index {index}/{id}: {st} {b}");+ }+ async fn create_index(&self, index: &str) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "PUT",+ &format!("/{index}"),+ json!({"mappings": {"properties": {+ "text": {"type": "text"},+ "label": {"type": "keyword"}+ }}}),+ )+ .await;+ assert!(st.is_success(), "create {index}: {st} {b}");+ }+ async fn refresh(&self, index: &str) {+ let (st, b) = self+ .call(+ &self.app.clone(),+ "POST",+ &format!("/{index}/_refresh"),+ json!({}),+ )+ .await;+ assert!(st.is_success(), "refresh {index}: {st} {b}");+ }+}++/// Seed the outcome history: `per_side` `refund` texts labelled positive+/// (`true`, `p: 0.25`) and `per_side` `outage` texts labelled negative+/// (`false`, `p: 0.75`). Each document's `p` is the probability the system+/// served FOR ITS OWN LABEL, so both sides convert to the same positive-label+/// probability: the positives are (0.25, 1), the negatives (1 − 0.75 = 0.25,+/// 0) — exactly, because 0.75 and 0.25 are exactly representable (1 − 0.8 is+/// not, and the two-tenths the first draft used landed on two DIFFERENT raw+/// probabilities: 0.19999999999999996 and 0.2). Every pair therefore sits at+/// raw 0.25 against a 0.5 base rate — isotonic pools the one distinct raw+/// probability into a single knot at 0.5, so every `p_cal` on this history is+/// exactly 0.5, visibly not a copy of any vote share. The outage texts share+/// no query term with the refund texts, so a refund-shaped question retrieves+/// only refund documents and votes 1.0.+async fn seed_outcomes(node: &Node, index: &str, per_side: usize) {+ node.create_index(index).await;+ for i in 0..per_side {+ node.put_doc(+ index,+ &format!("refund-{i}"),+ json!({+ "text": format!("refund request number {i} for the annual subscription"),+ "label": "true",+ "p": 0.25,+ "ts": "2026-09-28T10:00:00.000Z",+ }),+ )+ .await;+ node.put_doc(+ index,+ &format!("outage-{i}"),+ json!({+ "text": format!("outage report number {i} for the weekend window"),+ "label": "false",+ "p": 0.75,+ "ts": "2026-09-29T10:00:00.000Z",+ }),+ )+ .await;+ }+ node.refresh(index).await;+}++// ─────────────────────────────────────────────────────────────────────────────+// /_decide+// ─────────────────────────────────────────────────────────────────────────────++/// The core contract: `p_raw` beside `p_cal`, the calibration block naming+/// the method and its ECE, and `p_cal` moved by the fit — 0.5 on this+/// history, whatever the vote's share was — never a copy of `p_raw`.+#[tokio::test]+async fn decide_ships_p_cal_beside_p_raw_with_the_fit_ece() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription", "k": 10}))+ .await;+ assert!(st.is_success(), "{st} {body}");+ // The vote: all ten neighbours are `refund` docs, so the positive share+ // is 1.0 — a confident raw probability.+ assert_eq!(body["p_raw"], 1.0, "p_raw is the positive label's share");+ assert_eq!(+ body["p_cal"], 0.5,+ "the fit pooled the one distinct raw probability to the base rate: {body}"+ );+ assert_eq!(body["confidence"], 1.0, "confidence keeps its raw meaning");+ let cal = &body["calibration"];+ assert_eq!(cal["method"], "isotonic", "{cal}");+ assert_eq!(cal["scope"], "noul");+ assert!(cal["ece"]["raw"].is_number(), "the ECE rides beside: {cal}");+ assert!(cal["ece"]["calibrated"].is_number(), "{cal}");+ assert_eq!(cal["labelled_pairs"], 20);+ assert_eq!(cal["fitted_on"], 16, "the deterministic 80/20: {cal}");+ assert_eq!(cal["held_out"], 4);+}++/// An uncalibrated node ships `p_raw` alone: no `p_cal` field at all, and a+/// calibration block that says so — never a p_cal copying p_raw.+#[tokio::test]+async fn without_calibration_there_is_no_p_cal_at_all() {+ let node = node_with("judgements", CalibrationMethod::None).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert_eq!(body["p_raw"], 1.0);+ assert!(+ body.get("p_cal").is_none(),+ "no calibration configured → no p_cal field: {body}"+ );+ assert_eq!(body["calibration"], json!({"configured": "none"}));+}++/// Configured but unfitted: `p_cal` is `null`, and the reason rides beside+/// it in the calibration block — the operator can act on the string.+#[tokio::test]+async fn a_configured_but_unfitted_node_ships_null_with_the_reason() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ seed_outcomes(&node, "judgements", 2).await; // 4 pairs < the 10-pair minimum++ let (st, body) = node+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert!(+ body["p_raw"].is_number(),+ "p_raw ships whatever the four-document vote said: {body}"+ );+ assert!(+ body["p_cal"].is_null(),+ "configured but unfitted → null, not a copy: {body}"+ );+ let reason = body["calibration"]["reason"]+ .as_str()+ .expect("the reason is published beside the null");+ assert!(reason.contains("4 labelled calibration pairs"), "{reason}");+ assert!(+ reason.contains("source"),+ "the flywheel exclusion is explained: {reason}"+ );+}++/// Temperature calibrates the same quantity through the same seam. On this+/// history every fit pair sits at p 0.2 with a 0.5 base rate, so the NLL+/// optimum is "as flat as the bracket allows": T pins at its ceiling (16) and+/// the saturated p_raw 1.0 pulls most of the way back toward the base rate —+/// σ(logit(1)/16) ≈ 0.703. Rank preserved, confidence honestly deflated,+/// never all the way to a pretend 0.5.+#[tokio::test]+async fn temperature_calibrates_the_same_quantity() {+ let node = node_with("judgements", CalibrationMethod::Temperature).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert_eq!(body["p_raw"], 1.0);+ let p_cal = body["p_cal"].as_f64().expect("a number");+ assert!(+ (p_cal - 0.703_386).abs() < 0.005,+ "T pinned at its ceiling pulls σ(logit(1)/16) back toward the base rate: {p_cal}"+ );+ assert!(+ p_cal < body["p_raw"].as_f64().expect("p_raw"),+ "the saturated raw probability deflates: {p_cal}"+ );+ assert_eq!(body["calibration"]["method"], "temperature");+}++// ─────────────────────────────────────────────────────────────────────────────+// /v1/systemone — the strictly-parsed wire stays untouched+// ─────────────────────────────────────────────────────────────────────────────++/// `p_raw`/`p_cal` ride in `decisions.evidence` (the extras channel), the+/// answer object keeps exactly its documented fields, a noul calibrates, and+/// a choice's per-option shares — a different quantity — do not.+#[tokio::test]+async fn systemone_calibrates_nouls_in_evidence_and_leaves_the_wire_alone() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .systemone(json!({+ "state": {"documents": {+ "doc_0": "refund request for the annual subscription",+ "doc_1": "outage report for the weekend window"+ }},+ "questions": {+ "n": {"type": "noul", "instructions": "Judge `documents.doc_0` for a refund request."},+ "c": {"type": "choice", "instructions": "Route `documents.doc_1`.", "criteria": {"true": {}, "false": {}}}+ }+ }))+ .await;+ assert!(st.is_success(), "{st} {body}");++ // The wire objects: exactly the documented fields, no p_raw/p_cal inside.+ let noul = &body["answers"]["n"];+ assert_eq!(+ noul.as_object().map(|o| o.len()),+ Some(2),+ "type + noul only: {noul}"+ );+ let choice = &body["answers"]["c"];+ assert!(+ choice.get("p_raw").is_none() && choice.get("p_cal").is_none(),+ "the answer object is strictly parsed and stays untouched: {choice}"+ );++ // The evidence: the noul calibrates, the choice does not.+ let evidence = &body["decisions"]["evidence"];+ assert_eq!(evidence["n"]["p_raw"], 1.0, "the noul's p_raw: {evidence}");+ assert_eq!(+ evidence["n"]["p_cal"], 0.5,+ "the pooled base rate: {evidence}"+ );+ assert!(+ evidence["c"].get("p_raw").is_none() && evidence["c"].get("p_cal").is_none(),+ "a choice's per-option shares are a different quantity, out of this fit's scope: {evidence}"+ );++ // The block with the ECE rides at the decisions level.+ let cal = &body["decisions"]["calibration"];+ assert_eq!(cal["method"], "isotonic");+ assert_eq!(cal["scope"], "noul");+ assert!(cal["ece"]["calibrated"].is_number(), "{cal}");+ assert_eq!(cal["labelled_pairs"], 20);+}++/// An uncalibrated wire node carries no calibration block at all and no+/// p_cal in any evidence.+#[tokio::test]+async fn systemone_without_calibration_carries_no_block() {+ let node = node_with("judgements", CalibrationMethod::None).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .systemone(json!({+ "state": "refund request for the annual subscription",+ "questions": {"n": {"type": "noul", "instructions": "Is this a refund request?"}}+ }))+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert!(+ body["decisions"].get("calibration").is_none(),+ "no calibration configured → no block: {body}"+ );+ assert_eq!(body["decisions"]["evidence"]["n"]["p_raw"], 1.0);+ assert!(+ body["decisions"]["evidence"]["n"].get("p_cal").is_none(),+ "and no p_cal: {body}"+ );+}++// ─────────────────────────────────────────────────────────────────────────────+// GET /_decide/_calibration+// ─────────────────────────────────────────────────────────────────────────────++/// The publication: the reliability curve (binned p_raw against empirical+/// frequency, with bin counts), the ECE raw and calibrated, the fit's+/// parameters, and what it was fitted on. On this history every pair sits at+/// raw 0.2, so the raw curve is one bin of 20 at rate 0.5.+#[tokio::test]+async fn calibration_publishes_the_reliability_curve() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .calibration("/_decide/_calibration?index=judgements")+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert_eq!(body["index"], "judgements");+ assert_eq!(body["configured"], "isotonic");+ assert_eq!(body["method"], "isotonic");+ assert_eq!(body["scope"], "noul");+ assert_eq!(body["labelled_pairs"], 20);+ assert_eq!(body["fitted_on"], 16);+ assert_eq!(body["held_out"], 4);+ assert_eq!(body["bins"], 10);++ let raw_curve = body["curve"]["raw"].as_array().expect("the raw curve");+ assert_eq!(+ raw_curve.len(),+ 1,+ "one occupied bin (all pairs at 0.25): {raw_curve:?}"+ );+ let bin = &raw_curve[0];+ assert_eq!(bin["n"], 20, "bin counts publish");+ assert_eq!(bin["lo"], 0.2);+ assert_eq!(bin["hi"], 0.3); // the bin edge: (b+1)/10 in f64+ assert!((bin["mean_p"].as_f64().unwrap() - 0.25).abs() < 1e-12);+ assert!((bin["positive_rate"].as_f64().unwrap() - 0.5).abs() < 1e-12);+ assert!(+ body["curve"]["calibrated"].is_array(),+ "the calibrated curve publishes too: {body}"+ );+ assert!(body["ece"]["raw"].is_number() && body["ece"]["calibrated"].is_number());+ let knots = body["params"]["knots"].as_array().expect("the fit's knots");+ assert_eq!(+ knots.len(),+ 1,+ "one distinct raw probability → one knot: {knots:?}"+ );+ assert_eq!(knots[0][0], 0.25);+ assert_eq!(knots[0][1], 0.5);+ assert_eq!(+ body["ts_range"],+ json!(["2026-09-28T10:00:00.000Z", "2026-09-29T10:00:00.000Z"]),+ "the fitted documents' date range publishes"+ );+}++/// The endpoint also publishes WITHOUT a configured calibration — the raw+/// reliability of the history is audit information in its own right — with a+/// reason naming the setting that would fit it.+#[tokio::test]+async fn calibration_publishes_the_raw_curve_even_when_not_configured() {+ let node = node_with("judgements", CalibrationMethod::None).await;+ seed_outcomes(&node, "judgements", 10).await;++ let (st, body) = node+ .calibration("/_decide/_calibration?index=judgements")+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert_eq!(body["configured"], "none");+ assert_eq!(body["labelled_pairs"], 20);+ assert!(+ body["curve"]["raw"].is_array(),+ "the raw curve still publishes: {body}"+ );+ assert!(body["ece"]["raw"].is_number(), "{body}");+ let reason = body["reason"]+ .as_str()+ .expect("the reason names the setting");+ assert!(reason.contains("calibration"), "{reason}");+}++/// Cached flywheel answers (`source`) are predictions, not outcomes: they do+/// not feed the fit, whatever else they do to the vote.+#[tokio::test]+async fn cached_answers_do_not_feed_the_fit() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ seed_outcomes(&node, "judgements", 10).await;+ // Five cached local-tier answers — confident, and self-consistent by+ // construction. If they fed the fit they would drag it off the recorded+ // outcomes' base rate.+ for i in 0..5 {+ node.put_doc(+ "judgements",+ &format!("cached-{i}"),+ json!({+ "text": format!("cached answer number {i}"),+ "label": "true",+ "p": 0.99,+ "source": "local",+ "ts": "2026-09-30T10:00:00.000Z",+ }),+ )+ .await;+ }+ node.refresh("judgements").await;++ let (st, body) = node+ .calibration("/_decide/_calibration?index=judgements")+ .await;+ assert!(st.is_success(), "{st} {body}");+ assert_eq!(+ body["labelled_pairs"], 20,+ "the five `source` documents are predictions, not outcomes: {body}"+ );+ assert_eq!(body["fitted_on"], 16);+}++/// No index to read: the endpoint refuses naming what it needs, instead of+/// publishing an empty curve as if it were a measurement.+#[tokio::test]+async fn calibration_with_no_index_refuses() {+ // A node with no [decisions] index and nothing named per request.+ let node = node_with("", CalibrationMethod::Isotonic).await;+ let (st, body) = node.calibration("/_decide/_calibration").await;+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{st} {body}");+ let reason = body["error"]["reason"].as_str().expect("the refusal");+ assert!(reason.contains("index"), "{reason}");+}++/// A named index that does not exist is the vote's own error shape — the+/// calibration surface never invents a fit from an unreadable history.+#[tokio::test]+async fn calibration_on_a_missing_index_errors() {+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;+ let (st, body) = node.calibration("/_decide/_calibration?index=nope").await;+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{st} {body}");+ assert_eq!(body["error"]["type"], "history_index_error", "{body}");+}diff --git a/engine/crates/xerj-common/src/calibration.rs b/engine/crates/xerj-common/src/calibration.rsnew file mode 100644index 000000000..fbbe95ec7--- /dev/null+++ b/engine/crates/xerj-common/src/calibration.rs@@ -0,0 +1,834 @@+//! Probability calibration for the decide surface — isotonic (PAVA) and+//! temperature (Platt) scaling, the reliability curve, and ECE (#1063).+//!+//! A probability from the decide ladder — the history vote's share, the local+//! head's softmax, and (when that tier exists) a hosted passthrough score — is+//! a *ranking* signal before it is an *odds*. The FiQA rerank measurement made+//! that concrete: provider noul probabilities with mean confidence 0.93 on a+//! bin whose actual relevance rate was 0.34, ECE 0.3109+//! (`benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa`). Ordering+//! worth +0.126 nDCG; a number that must never gate. This module is the+//! correction layer: one calibrated probability (`p_cal`) beside every raw one+//! (`p_raw`), never replacing it, and one ECE implementation reused by every+//! path that ships a probability.+//!+//! # The two methods, and their citations+//!+//! - **Isotonic** — non-decreasing regression of the empirical positive rate+//! on the raw probability, by the pool-adjacent-violators algorithm (PAVA:+//! Barlow, Bartholomew, Bremner & Brunk, *Statistical Inference under Order+//! Restrictions*, Wiley 1972; the stack-of-blocks formulation is de Leeuw,+//! Hornik & Mair, "Isotonic Optimization in R: Pool-Adjacent-Violators+//! (PAVA) and Active Set Methods", JSS 2009). Its use for classifier scores+//! is Zadrozny & Elkan, "Transforming classifier scores into accurate+//! multiclass probability estimates", KDD 2002. Fitted values are joined by+//! linear interpolation and clamped at the ends (constant extrapolation) —+//! the same out-of-sample behaviour as scikit-learn's `IsotonicRegression`.+//! - **Temperature** — one scalar on the log-odds, `p' = σ(logit(p)/T)`,+//! fitted by minimising held-out negative log-likelihood: Platt,+//! "Probabilistic Outputs for Support Vector Machines", 1999 (the+//! two-parameter original; the one-parameter scaling on the model's own+//! log-odds is the form Guo, Pleiss, Sun & Weinberger, "On Calibration of+//! Modern Neural Networks", ICML 2017, validate). Temperature is+//! rank-preserving by construction: `σ(logit(·)/T)` is strictly increasing+//! for any `T > 0`.+//!+//! # ECE+//!+//! Expected calibration error, the statistic the FiQA baseline is quoted in:+//! `ECE = Σ_b (n_b/N)·|acc_b − conf_b|` over equal-width bins of the+//! probability range (Naeini, Cooper & Hausknecht, "Obtaining Well Calibrated+//! Probabilities Using Bayesian Binning", AAAI 2015). Implemented once, in+//! [`ece_from_curve`], and reused for raw and calibrated probabilities alike.+//!+//! # Determinism (#940's lesson)+//!+//! Every fit here is a pure function of the pair multiset: no clock, no RNG,+//! no map-iteration order. The held-out split is a fixed rule — pairs+//! stable-sorted by `(p, y)`, every fifth held out — so the same history+//! produces bit-identical `p_cal` on every request, and the published+//! reliability curve is reproducible. Nothing in this module reads the+//! environment, the filesystem, or the system clock.++use serde::{Deserialize, Serialize};++/// Equal-width bins of the probability range used for reliability curves and+/// ECE — the count the published FiQA curve uses (10).+pub const CALIBRATION_BINS: usize = 10;++/// One labelled pair: a raw probability `p ∈ [0,1]` and its outcome `y ∈+/// {0,1}` (stored as `f64` so weighted/aggregate points — a bin's mean+/// confidence and positive rate — go through the same arithmetic).+#[derive(Debug, Clone, Copy, PartialEq)]+pub struct CalibrationPair {+ pub p: f64,+ pub y: f64,+}++impl CalibrationPair {+ pub fn new(p: f64, y: bool) -> Self {+ Self {+ p,+ y: if y { 1.0 } else { 0.0 },+ }+ }+}++/// One bin of the reliability curve: the raw-probability interval `[lo, hi)`,+/// how many pairs fell in it, their mean raw probability (`mean_p`), and their+/// empirical positive rate (`positive_rate`). Serialize is derived so the+/// curve publishes verbatim.+#[derive(Debug, Clone, Copy, PartialEq, Serialize)]+pub struct ReliabilityBin {+ /// Inclusive lower edge of the bin.+ pub lo: f64,+ /// Exclusive upper edge; the last bin's edge is inclusive of 1.0.+ pub hi: f64,+ /// Pairs in this bin.+ pub n: usize,+ /// Mean raw (or, where the caller calibrated first, calibrated)+ /// probability of those pairs.+ pub mean_p: f64,+ /// Fraction of those pairs whose outcome was positive.+ pub positive_rate: f64,+}++/// Bin the pairs into `bins` equal-width bins of `[0,1]` and return the+/// non-empty ones, in bin order. Pure: the bin of a pair is a function of its+/// `p` alone (`min(floor(p·bins), bins−1)`), so the curve is deterministic+/// for a pair multiset regardless of input order.+pub fn reliability_curve(pairs: &[CalibrationPair], bins: usize) -> Vec<ReliabilityBin> {+ debug_assert!(bins > 0);+ let bins = bins.max(1);+ // (count, Σp, Σy) per bin — usize counts, f64 sums, no allocation for+ // empty bins beyond the three slots.+ let mut count = vec![0usize; bins];+ let mut sum_p = vec![0f64; bins];+ let mut sum_y = vec![0f64; bins];+ for pair in pairs {+ // A p outside [0,1] is the caller's bug, not a bin; clamp it into the+ // range rather than panicking on the index math below.+ let p = pair.p.clamp(0.0, 1.0);+ let b = ((p * bins as f64) as usize).min(bins - 1);+ count[b] += 1;+ sum_p[b] += p;+ sum_y[b] += pair.y;+ }+ (0..bins)+ .filter(|&b| count[b] > 0)+ .map(|b| {+ let n = count[b];+ ReliabilityBin {+ lo: b as f64 / bins as f64,+ hi: (b + 1) as f64 / bins as f64,+ n,+ mean_p: sum_p[b] / n as f64,+ positive_rate: sum_y[b] / n as f64,+ }+ })+ .collect()+}++/// ECE from a reliability curve: `Σ_b (n_b/N)·|positive_rate_b − mean_p_b|`.+/// The one implementation every probability-shipping path reuses — raw and+/// calibrated curves, the endpoint publication, and the tests that pin the+/// FiQA arithmetic. An empty curve has no probabilities to be wrong about: 0.+pub fn ece_from_curve(curve: &[ReliabilityBin]) -> f64 {+ let total: usize = curve.iter().map(|b| b.n).sum();+ if total == 0 {+ return 0.0;+ }+ curve+ .iter()+ .map(|b| (b.n as f64 / total as f64) * (b.positive_rate - b.mean_p).abs())+ .sum()+}++/// ECE of raw probabilities against their outcomes, binned — convenience over+/// [`reliability_curve`] + [`ece_from_curve`].+pub fn ece(pairs: &[CalibrationPair], bins: usize) -> f64 {+ ece_from_curve(&reliability_curve(pairs, bins))+}++// ─────────────────────────────────────────────────────────────────────────────+// Isotonic — PAVA+// ─────────────────────────────────────────────────────────────────────────────++/// An isotonic calibration fitted by PAVA: non-decreasing knots+/// `(x = p_raw, y = fitted probability)`, applied by linear interpolation+/// between neighbours and clamped at the ends.+///+/// Monotone by construction — the fit cannot invent a region where a higher+/// raw probability means a lower calibrated one — and it is the method that+/// meets the FiQA held-out gate (see+/// `benchmarks/decisions-calibration/`); temperature is the cheaper,+/// one-parameter alternative offered for the same job.+#[derive(Debug, Clone, PartialEq)]+pub struct IsotonicCalibration {+ /// Strictly increasing raw probabilities (tied `p` pooled into one+ /// weighted point at fit time).+ xs: Vec<f64>,+ /// The PAVA-fitted value at each `x` — non-decreasing.+ ys: Vec<f64>,+}++impl IsotonicCalibration {+ /// Fit on the pairs. Pure: ties in `p` are pooled by weighted mean and+ /// the PAVA pass is a deterministic stack algorithm, so the fit depends+ /// only on the pair multiset.+ pub fn fit(pairs: &[CalibrationPair]) -> Self {+ // Pool tied p: isotonic regression estimates a FUNCTION of p, so all+ // pairs at one p must share one fitted value — the weighted mean of+ // their outcomes. (A pair at the same p with outcomes 0 and 1 is not+ // a violation to pool; it is one point at the base rate.)+ let mut pooled: Vec<(f64 /*x*/, f64 /*Σy*/, f64 /*n*/)> = Vec::new();+ let mut sorted: Vec<CalibrationPair> = pairs.to_vec();+ sorted.sort_by(|a, b| {+ a.p.partial_cmp(&b.p)+ .unwrap()+ .then(a.y.partial_cmp(&b.y).unwrap())+ });+ for pair in sorted {+ match pooled.last_mut() {+ Some((x, sum_y, n)) if *x == pair.p => {+ *sum_y += pair.y;+ *n += 1.0;+ }+ _ => pooled.push((pair.p, pair.y, 1.0)),+ }+ }+ // PAVA (de Leeuw–Hornik–Mair's stack of blocks): walk the pooled+ // points in x order; while the previous block's weighted mean exceeds+ // the current one's, pool them. Each block is (Σy, n) over a+ // contiguous run of points; its mean is the fitted value for every+ // point in it.+ let mut blocks: Vec<(f64, f64, usize /*first pooled index*/)> = Vec::new();+ for (i, (_, sum_y, n)) in pooled.iter().enumerate() {+ blocks.push((*sum_y, *n, i));+ while blocks.len() >= 2 {+ let (prev, cur) = (blocks.len() - 2, blocks.len() - 1);+ let prev_mean = blocks[prev].0 / blocks[prev].1;+ let cur_mean = blocks[cur].0 / blocks[cur].1;+ if prev_mean > cur_mean {+ let (sy, sn, _) = blocks.pop().expect("cur block");+ let b = blocks.last_mut().expect("prev block");+ b.0 += sy;+ b.1 += sn;+ } else {+ break;+ }+ }+ }+ // Expand blocks back to pooled points: every point in a block carries+ // the block's mean, and those (x, mean) pairs are the interpolation+ // knots.+ let mut ys = vec![0f64; pooled.len()];+ for (bi, (sum_y, n, first)) in blocks.iter().enumerate() {+ let mean = sum_y / n;+ let last = blocks.get(bi + 1).map(|b| b.2).unwrap_or(pooled.len());+ for y in &mut ys[*first..last] {+ *y = mean;+ }+ }+ Self {+ xs: pooled.iter().map(|(x, _, _)| *x).collect(),+ ys,+ }+ }++ /// The fitted knots, `(p_raw, p_cal)` — the parameters the+ /// `/_decide/_calibration` endpoint publishes.+ pub fn knots(&self) -> Vec<(f64, f64)> {+ self.xs+ .iter()+ .copied()+ .zip(self.ys.iter().copied())+ .collect()+ }++ /// Linear interpolation between knots; constant (clamped) outside their+ /// range. Monotone non-decreasing for any input, including inputs the fit+ /// never saw above or below its support.+ pub fn apply(&self, p: f64) -> f64 {+ match self.xs.first() {+ None => return p.clamp(0.0, 1.0), // empty fit: identity, caller gates on fit size+ Some(&first_x) if p <= first_x => return self.ys[0],+ _ => {}+ }+ if p >= *self.xs.last().expect("non-empty") {+ return *self.ys.last().expect("non-empty");+ }+ // Binary search for the knot interval containing p.+ let i = self.xs.partition_point(|&x| x < p);+ let (x0, y0) = (self.xs[i - 1], self.ys[i - 1]);+ let (x1, y1) = (self.xs[i], self.ys[i]);+ if x1 == x0 {+ return y0;+ }+ y0 + (p - x0) * (y1 - y0) / (x1 - x0)+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// Temperature — one scalar on the log-odds+// ─────────────────────────────────────────────────────────────────────────────++/// Probabilities at the rails have no finite log-odds; clamp before the+/// transform so 0 and 1 are representable (Platt's own practice).+const LOGIT_EPS: f64 = 1e-6;++fn logit(p: f64) -> f64 {+ let p = p.clamp(LOGIT_EPS, 1.0 - LOGIT_EPS);+ (p / (1.0 - p)).ln()+}++fn sigmoid(z: f64) -> f64 {+ if z >= 0.0 {+ 1.0 / (1.0 + (-z).exp())+ } else {+ let e = z.exp();+ e / (1.0 + e)+ }+}++/// ln(1 + e^x), the numerically stable softplus — the two log terms of the+/// Bernoulli NLL.+fn softplus(x: f64) -> f64 {+ if x > 20.0 {+ x // e^-x underflows; ln(1+e^x) ≈ x to f64 precision+ } else if x < -20.0 {+ (-x).exp() // ≈ e^x, and ln(1+e^x) ≈ e^x+ } else {+ x.exp().ln_1p()+ }+}++/// A temperature calibration: `p_cal = σ(logit(p_raw)/T)`. `T > 1` flattens+/// confident probabilities toward the base rate, `T < 1` sharpens them; any+/// `T > 0` preserves rank exactly.+#[derive(Debug, Clone, Copy, PartialEq)]+pub struct TemperatureCalibration {+ pub t: f64,+}++/// The search bracket for `T`, on the log scale: a temperature outside+/// `[1/16, 16]` says the raw probabilities are unusable as odds in either+/// direction, which the caller reports rather than ships. A function, not a+/// const, because `f64::ln` is not const-callable.+fn t_bracket() -> (f64, f64) {+ ((1.0f64 / 16.0).ln(), 16.0f64.ln())+}++impl TemperatureCalibration {+ /// Fit `T` by minimising the pairs' negative log-likelihood — Platt+ /// (1999) / Guo et al. (2017)'s held-out form — by golden-section search+ /// on `ln T` over [`T_BRACKET`]. Deterministic: a fixed bracket and a+ /// fixed 100 halvings of it, no starting point, no tolerance decision, so+ /// the same pairs always fit the same `T`.+ pub fn fit(pairs: &[CalibrationPair]) -> Self {+ let zs: Vec<f64> = pairs.iter().map(|p| logit(p.p)).collect();+ let nll = |ln_t: f64| -> f64 {+ let t = ln_t.exp();+ pairs+ .iter()+ .zip(&zs)+ .map(|(pair, &z)| {+ let a = z / t;+ pair.y * softplus(-a) + (1.0 - pair.y) * softplus(a)+ })+ .sum()+ };+ // Golden-section minimisation (the ratio φ−1 = 0.618…).+ let phi = (5f64.sqrt() - 1.0) / 2.0;+ let (mut lo, mut hi) = t_bracket();+ let mut c = hi - phi * (hi - lo);+ let mut d = lo + phi * (hi - lo);+ for _ in 0..100 {+ if nll(c) < nll(d) {+ hi = d;+ } else {+ lo = c;+ }+ c = hi - phi * (hi - lo);+ d = lo + phi * (hi - lo);+ }+ Self {+ t: ((lo + hi) / 2.0)+ .exp()+ .clamp(t_bracket().0.exp(), t_bracket().1.exp()),+ }+ }++ pub fn apply(&self, p: f64) -> f64 {+ sigmoid(logit(p) / self.t)+ }+}++// ─────────────────────────────────────────────────────────────────────────────+// The fitted layer — method choice, held-out split, one report+// ─────────────────────────────────────────────────────────────────────────────++/// The configured calibration method — the `[decisions] calibration` setting's+/// type. `None` is the default: an uncalibrated node ships `p_raw` alone and+/// no `p_cal` at all, never a copy of `p_raw` pretending to be calibrated.+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "lowercase")]+pub enum CalibrationMethod {+ #[default]+ None,+ Isotonic,+ Temperature,+}++impl CalibrationMethod {+ /// The wire name — also the config-file value.+ pub fn as_str(self) -> &'static str {+ match self {+ CalibrationMethod::None => "none",+ CalibrationMethod::Isotonic => "isotonic",+ CalibrationMethod::Temperature => "temperature",+ }+ }+}++/// The fitted calibration, whichever method produced it. One `apply`, so the+/// caller never branches on the method to turn a raw probability into a+/// calibrated one — history-vote shares, local-head probabilities, and (when+/// the hosted tier is built) a passthrough score all go through this.+#[derive(Debug, Clone, PartialEq)]+pub enum FittedCalibration {+ Isotonic(IsotonicCalibration),+ Temperature(TemperatureCalibration),+}++impl FittedCalibration {+ /// The method's wire name (`isotonic` / `temperature`).+ pub fn method(&self) -> &'static str {+ match self {+ FittedCalibration::Isotonic(_) => "isotonic",+ FittedCalibration::Temperature(_) => "temperature",+ }+ }++ /// Calibrate one raw probability. Monotone non-decreasing for both+ /// methods, so a ranking by `p_raw` and by `p_cal` can disagree only on+ /// ties.+ pub fn apply(&self, p: f64) -> f64 {+ match self {+ FittedCalibration::Isotonic(f) => f.apply(p),+ FittedCalibration::Temperature(f) => f.apply(p),+ }+ }+}++/// The deterministic held-out split: every fifth pair, in `(p, y)` order, is+/// held out — 20 % of the history, stratified across the probability range+/// (so the held-out set spans the same `p_raw` support the fit sees, which is+/// what makes an isotonic fit evaluable), with no clock and no RNG. The same+/// multiset always splits identically (#940's lesson: a fit that moves+/// between requests is a defect, not a measurement).+pub fn held_out_split(pairs: &[CalibrationPair]) -> (Vec<CalibrationPair>, Vec<CalibrationPair>) {+ let mut sorted: Vec<CalibrationPair> = pairs.to_vec();+ sorted.sort_by(|a, b| {+ a.p.partial_cmp(&b.p)+ .unwrap()+ .then(a.y.partial_cmp(&b.y).unwrap())+ });+ let mut fit = Vec::with_capacity(sorted.len() * 4 / 5);+ let mut held = Vec::with_capacity(sorted.len() / 5 + 1);+ for (i, pair) in sorted.into_iter().enumerate() {+ if i % 5 == 4 {+ held.push(pair);+ } else {+ fit.push(pair);+ }+ }+ (fit, held)+}++/// The full fitted report `/_decide/_calibration` publishes: what was fitted+/// on, how it evaluates, and the reliability curves of the raw and calibrated+/// probabilities over every labelled pair.+#[derive(Debug, Clone, PartialEq)]+pub struct CalibrationReport {+ /// The method that produced this fit.+ pub method: CalibrationMethod,+ /// The fit itself, for applying to new probabilities.+ pub fit: FittedCalibration,+ /// Pairs the fit was fitted on (80 %).+ pub fitted_on: usize,+ /// Pairs the ECEs below evaluate on (20 %, held out of the fit).+ pub held_out: usize,+ /// Every labelled pair the report saw (fit + held out).+ pub labelled_pairs: usize,+ /// Held-out ECE of the raw probabilities — what the node would ship+ /// without this layer.+ pub ece_raw: f64,+ /// Held-out ECE after calibration — the number shipped beside every+ /// `p_cal`.+ pub ece_calibrated: f64,+ /// Reliability curve of the raw probabilities over all pairs.+ pub curve_raw: Vec<ReliabilityBin>,+ /// Reliability curve of the calibrated probabilities over all pairs.+ pub curve_calibrated: Vec<ReliabilityBin>,+ /// The RFC 3339 range of the fitted documents' `ts` fields, when they+ /// carry one — publication metadata stamped by the caller that owns the+ /// documents, never an input to the fit itself (which stays a pure+ /// function of the pairs).+ pub ts_range: Option<(String, String)>,+}++/// Fit `method` on a held-out split of `pairs` and evaluate it: fit on 80 %,+/// publish raw and calibrated ECE on the 20 % held out, and the reliability+/// curves over all pairs. Pure — the whole report is a function of the pair+/// multiset. `None` fits nothing and reports nothing; the caller decides what+/// an unconfigured node ships (no `p_cal` at all).+pub fn fit_calibration(+ method: CalibrationMethod,+ pairs: &[CalibrationPair],+) -> Option<CalibrationReport> {+ if method == CalibrationMethod::None {+ return None;+ }+ let (fit_pairs, held_pairs) = held_out_split(pairs);+ let fit = match method {+ CalibrationMethod::Isotonic => {+ FittedCalibration::Isotonic(IsotonicCalibration::fit(&fit_pairs))+ }+ CalibrationMethod::Temperature => {+ FittedCalibration::Temperature(TemperatureCalibration::fit(&fit_pairs))+ }+ CalibrationMethod::None => return None,+ };+ let raw_pairs: Vec<CalibrationPair> = pairs.to_vec();+ let calibrated_pairs: Vec<CalibrationPair> = pairs+ .iter()+ .map(|p| CalibrationPair {+ p: fit.apply(p.p),+ y: p.y,+ })+ .collect();+ let held_raw: Vec<CalibrationPair> = held_pairs.clone();+ let held_calibrated: Vec<CalibrationPair> = held_pairs+ .iter()+ .map(|p| CalibrationPair {+ p: fit.apply(p.p),+ y: p.y,+ })+ .collect();+ Some(CalibrationReport {+ method,+ fit,+ fitted_on: fit_pairs.len(),+ held_out: held_pairs.len(),+ labelled_pairs: pairs.len(),+ ece_raw: ece(&held_raw, CALIBRATION_BINS),+ ece_calibrated: ece(&held_calibrated, CALIBRATION_BINS),+ curve_raw: reliability_curve(&raw_pairs, CALIBRATION_BINS),+ curve_calibrated: reliability_curve(&calibrated_pairs, CALIBRATION_BINS),+ ts_range: None,+ })+}++// ─────────────────────────────────────────────────────────────────────────────+// Tests — the algorithms are the contract, so they are pinned directly+// ─────────────────────────────────────────────────────────────────────────────++#[cfg(test)]+mod tests {+ use super::*;++ fn pairs(pts: &[(f64, f64)]) -> Vec<CalibrationPair> {+ pts.iter().map(|&(p, y)| CalibrationPair { p, y }).collect()+ }++ /// The FiQA baseline's own reliability curve, verbatim from+ /// `benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json`+ /// (`calibration.curve`, bins 0–9): `(mean_conf, acc, n)`. 19,440+ /// (probability, relevance) pairs, published ECE 0.3109.+ const FIQA_BASELINE_CURVE: [(f64, f64, usize); 10] = [+ (0.0455, 0.0002, 5572),+ (0.1418, 0.0003, 3005),+ (0.2434, 0.0034, 2061),+ (0.3438, 0.0033, 1529),+ (0.4445, 0.0077, 1423),+ (0.5442, 0.0071, 1271),+ (0.6451, 0.0224, 1118),+ (0.7455, 0.0408, 1224),+ (0.8452, 0.1242, 1272),+ (0.9297, 0.3378, 965),+ ];++ /// Expand binned aggregates into pairs at the bin's mean confidence, the+ /// bin's count split by its positive rate — the reconstruction every+ /// FiQA number in this module works from (the run retained bins, not+ /// pairs).+ fn bin_pairs<'a>(+ bins: impl IntoIterator<Item = &'a (f64, f64, usize)>,+ ) -> Vec<CalibrationPair> {+ let mut out = Vec::new();+ for &(conf, acc, n) in bins {+ let positives = (acc * n as f64).round() as usize;+ for i in 0..n {+ out.push(CalibrationPair {+ p: conf,+ y: if i < positives { 1.0 } else { 0.0 },+ });+ }+ }+ out+ }++ fn fiqa_baseline_pairs() -> Vec<CalibrationPair> {+ bin_pairs(FIQA_BASELINE_CURVE.iter())+ }++ /// PAVA's defining property on the classic textbook input: the raw points+ /// dip mid-range, and the fit pools them into the weighted mean of the+ /// violating run.+ #[test]+ fn pava_pools_the_violating_run() {+ // (x, y) with equal weights: 0→0, 1→1, 2→0, 3→1 dips at x=2; the+ // block {1,2} violates and pools to 0.5 each.+ let pts = pairs(&[(0.0, 0.0), (0.3, 1.0), (0.6, 0.0), (0.9, 1.0)]);+ let fit = IsotonicCalibration::fit(&pts);+ let knots = fit.knots();+ assert_eq!(knots[1].1, 0.5, "the violating run pools: {knots:?}");+ assert_eq!(knots[2].1, 0.5, "both members of the run share one value");+ assert_eq!(knots[0].1, 0.0, "an undisrupted point is untouched");+ assert_eq!(knots[3].1, 1.0);+ }++ /// Isotonic output is monotone non-decreasing in the input, on data whose+ /// empirical rates are noisy — the property that makes p_cal a+ /// probability, not a re-scoring.+ #[test]+ fn isotonic_output_is_monotone() {+ let pts = pairs(&[+ (0.05, 0.0),+ (0.1, 1.0),+ (0.15, 0.0),+ (0.2, 0.0),+ (0.5, 1.0),+ (0.55, 0.0),+ (0.9, 1.0),+ (0.95, 0.0),+ ]);+ let fit = IsotonicCalibration::fit(&pts);+ let mut prev = 0.0f64;+ for p in [0.0, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99, 1.0] {+ let cal = fit.apply(p);+ assert!(+ cal >= prev - 1e-12,+ "apply({p}) = {cal} < apply of a smaller input {prev}"+ );+ assert!((0.0..=1.0).contains(&cal), "p_cal is a probability: {cal}");+ prev = cal;+ }+ }++ /// Interpolation between knots, clamped outside them: a fit on [0.2, 0.8]+ /// is constant below 0.2 and above 0.8, linear between.+ #[test]+ fn isotonic_interpolates_and_clamps() {+ let pts = pairs(&[(0.2, 0.0), (0.8, 1.0)]);+ let fit = IsotonicCalibration::fit(&pts);+ assert_eq!(fit.apply(0.0), 0.0, "below the lowest knot: clamped");+ assert_eq!(fit.apply(1.0), 1.0, "above the highest knot: clamped");+ assert!((fit.apply(0.5) - 0.5).abs() < 1e-12, "linear between knots");+ }++ /// Tied raw probabilities pool to their base rate — one p, one fitted+ /// value, whatever outcomes share it.+ #[test]+ fn tied_probabilities_pool() {+ let pts = pairs(&[(0.7, 1.0), (0.7, 0.0), (0.7, 0.0), (0.7, 1.0), (0.7, 0.0)]);+ let fit = IsotonicCalibration::fit(&pts);+ assert_eq!(fit.knots().len(), 1, "one distinct p → one knot");+ assert!((fit.apply(0.7) - 0.4).abs() < 1e-12, "the base rate 2/5");+ }++ /// Temperature preserves rank exactly and only moves the numbers: with+ /// T fit on data that is over-confident, every probability shrinks toward+ /// the middle without any two swapping order.+ #[test]+ fn temperature_shifts_but_preserves_rank() {+ let pts = pairs(&[+ (0.95, 0.0),+ (0.9, 0.0),+ (0.9, 1.0),+ (0.8, 0.0),+ (0.6, 0.0),+ (0.95, 1.0),+ (0.99, 1.0),+ (0.85, 0.0),+ (0.92, 0.0),+ (0.7, 0.0),+ ]);+ let fit = TemperatureCalibration::fit(&pts);+ assert!(fit.t > 1.0, "over-confident data fits T > 1: {}", fit.t);+ let raw = [0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 0.99];+ let cal: Vec<f64> = raw.iter().map(|p| fit.apply(*p)).collect();+ let mut sorted_cal = cal.clone();+ sorted_cal.sort_by(|a, b| a.partial_cmp(b).unwrap());+ assert_eq!(cal, sorted_cal, "rank preserved");+ assert!(cal[5] < 0.95, "high confidence shrinks: {}", cal[5]);+ assert!((fit.apply(0.5) - 0.5).abs() < 1e-9, "the midpoint is fixed");+ }++ /// Under-confident data fits T < 1 (sharpening) — the bracket's other+ /// half.+ #[test]+ fn under_confident_data_sharpens() {+ let pts = pairs(&[+ (0.6, 1.0),+ (0.62, 1.0),+ (0.58, 0.0),+ (0.61, 1.0),+ (0.4, 0.0),+ (0.39, 0.0),+ (0.41, 0.0),+ (0.6, 1.0),+ (0.42, 0.0),+ (0.59, 0.0),+ ]);+ let fit = TemperatureCalibration::fit(&pts);+ assert!(fit.t < 1.0, "under-confident data fits T < 1: {}", fit.t);+ assert!(fit.apply(0.61) > 0.61, "sharpened upward");+ assert!(fit.apply(0.41) < 0.41, "sharpened downward");+ }++ /// ECE identities: a perfectly calibrated set is 0, a constant confident+ /// wrong set is its full error, and the published FiQA ECE 0.3109 is+ /// reproduced by the shared implementation from the baseline's own curve+ /// — the arithmetic this module shares with the measurement that+ /// motivated it.+ #[test]+ fn ece_identities_and_the_fiqa_arithmetic() {+ // Perfect: every bin's mean equals its rate.+ let perfect: Vec<CalibrationPair> = (0..10)+ .flat_map(|b| {+ let p = (b as f64 + 0.5) / 10.0;+ let positives = (p * 10.0).round() as usize;+ (0..10).map(move |i| CalibrationPair {+ p,+ y: if i < positives { 1.0 } else { 0.0 },+ })+ })+ .collect();+ assert!(ece(&perfect, 10) < 0.11, "the discretisation is coarse");+ // Constant and confidently wrong: mean 0.9 against a 0.1 rate,+ // |0.9 − 0.1| = 0.8 exactly.+ let wrong: Vec<CalibrationPair> = (0..10)+ .map(|i| CalibrationPair {+ p: 0.9,+ y: (i == 0) as u8 as f64,+ })+ .collect();+ assert!((ece(&wrong, 10) - 0.8).abs() < 1e-12);+ // Empty: no probabilities, no error.+ assert_eq!(ece(&[], 10), 0.0);+ // The published FiQA ECE, recomputed from the baseline's own curve+ // (`fiqa-full-run.json`, calibration.curve) as pairs-at-the-bin-mean.+ let pairs = fiqa_baseline_pairs();+ assert_eq!(pairs.len(), 19_440);+ assert!(+ (ece(&pairs, 10) - 0.3109).abs() < 0.0005,+ "the FiQA ECE, from this module: {}",+ ece(&pairs, 10)+ );+ }++ /// The split is exactly 80/20, stratified across the probability range,+ /// and deterministic — the same multiset in a different order splits+ /// identically.+ #[test]+ fn held_out_split_is_deterministic_and_stratified() {+ let pts: Vec<CalibrationPair> = (0..100)+ .map(|i| CalibrationPair {+ p: (i % 10) as f64 / 10.0,+ y: (i % 3 == 0) as u8 as f64,+ })+ .collect();+ let (fit, held) = held_out_split(&pts);+ assert_eq!(fit.len(), 80);+ assert_eq!(held.len(), 20);+ // Stratified: every distinct p appears in both halves.+ for p in [0.0, 0.1, 0.5, 0.9] {+ assert!(fit.iter().any(|c| c.p == p), "p={p} in the fit set");+ assert!(held.iter().any(|c| c.p == p), "p={p} in the held-out set");+ }+ // Deterministic under input reordering: the same multiset splits to+ // the same two multisets (compare sorted, since the stable sort may+ // order equal (p, y) pairs differently in the two runs).+ let mut shuffled = pts.clone();+ shuffled.reverse();+ let (fit2, held2) = held_out_split(&shuffled);+ let canon = |mut v: Vec<CalibrationPair>| {+ v.sort_by(|a, b| {+ a.p.partial_cmp(&b.p)+ .unwrap()+ .then(a.y.partial_cmp(&b.y).unwrap())+ });+ v+ };+ assert_eq!(canon(fit), canon(fit2));+ assert_eq!(canon(held), canon(held2));+ }++ /// The issue's gate, on a deterministic bin-level reconstruction of the+ /// FiQA baseline: isotonic fitted on the EVEN bins, evaluated held-out on+ /// the ODD bins — whole probability regions the fit never saw, which is+ /// the honest form of held-out this data supports (the run retained bins,+ /// not pairs). This pins the same numbers the results file in+ /// `benchmarks/decisions-calibration/` publishes, so a regression in the+ /// fit code fails a test rather than a release gate.+ #[test]+ fn isotonic_meets_the_fiqa_gate_on_the_baseline_curve() {+ let fiqa = FIQA_BASELINE_CURVE;+ let fit_pairs: Vec<CalibrationPair> = bin_pairs(fiqa.iter().step_by(2));+ let held_pairs: Vec<CalibrationPair> = bin_pairs(fiqa.iter().skip(1).step_by(2));+ // Raw held-out ECE is the same order as the published 0.3109 —+ // the odd bins are representative, not a gift.+ let raw_held = ece(&held_pairs, 10);+ assert!(+ (raw_held - 0.3831).abs() < 0.0005,+ "raw held-out ECE {raw_held} (benchmarks/decisions-calibration)"+ );+ // Isotonic on the even bins, applied to the odd bins' means.+ let fit = IsotonicCalibration::fit(&fit_pairs);+ let calibrated: Vec<CalibrationPair> = held_pairs+ .iter()+ .map(|p| CalibrationPair {+ p: fit.apply(p.p),+ y: p.y,+ })+ .collect();+ let cal_held = ece(&calibrated, 10);+ // The exact value is pinned to the Python measurement+ // (benchmarks/decisions-calibration/fiqa_gate.py) — two independent+ // implementations of PAVA + ECE agreeing on the same split is a+ // cross-check on both.+ assert!(+ (cal_held - 0.0330).abs() < 0.0005,+ "held-out calibrated ECE {cal_held} (benchmarks/decisions-calibration says 0.0330)"+ );+ assert!(cal_held <= 0.10, "the #1063 gate");+ }++ /// An empty or single-point fit is degenerate but total: no panic, and+ /// apply stays in [0,1].+ #[test]+ fn degenerate_fits_are_total() {+ let empty = IsotonicCalibration::fit(&[]);+ assert_eq!(empty.apply(0.7), 0.7, "empty fit: identity");+ let single = IsotonicCalibration::fit(&[CalibrationPair { p: 0.6, y: 1.0 }]);+ assert!((single.apply(0.6) - 1.0).abs() < 1e-12);+ let temp = TemperatureCalibration::fit(&[CalibrationPair { p: 0.6, y: 1.0 }]);+ assert!(temp.t.is_finite() && temp.t > 0.0);+ assert!((0.0..=1.0).contains(&temp.apply(0.6)));+ // The rails do not produce infinities.+ assert!((0.0..=1.0).contains(&temp.apply(0.0)));+ assert!((0.0..=1.0).contains(&temp.apply(1.0)));+ }+}diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rsindex 06c114846..f5499211b 100644--- a/engine/crates/xerj-common/src/config.rs+++ b/engine/crates/xerj-common/src/config.rs@@ -1,6 +1,6 @@//! xerj configuration system.//!-//! Configuration is intentionally minimal: **129 settings** versus+//! Configuration is intentionally minimal: **130 settings** versus//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible//! production-ready default. The format is TOML, loaded from a single file.//!@@ -39,6 +39,7 @@use serde::{Deserialize, Serialize};use std::path::Path;+use crate::calibration::CalibrationMethod;use crate::error::XerjError;// ═════════════════════════════════════════════════════════════════════════════@@ -98,12 +99,12 @@ pub struct Config {/// Second-stage reranking provider — 3 settings. Inert until a key is set.pub rerank: RerankProviderConfig,/// Typed decisions answered from a labelled-history index by- /// nearest-neighbour vote — 7 settings. Inert until `decisions.index`+ /// nearest-neighbour vote — 8 settings. Inert until `decisions.index`/// names an index that exists.pub decisions: DecisionsConfig,}-// 23 sub-configs, 129 leaf settings in total. Do not maintain that sum by hand+// 23 sub-configs, 130 leaf settings in total. Do not maintain that sum by hand// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a// serialised `Config::default()` and fails if this comment and the module// header stop matching. `Default` is derived: every field is a sub-config that@@ -484,7 +485,7 @@ impl Config {}// ═════════════════════════════════════════════════════════════════════════════-// Sub-configs (129 user-facing settings total; counted by+// Sub-configs (130 user-facing settings total; counted by// `journey_zero_config`, not by hand)// ═════════════════════════════════════════════════════════════════════════════@@ -2342,7 +2343,7 @@ impl RerankProviderConfig {/// weighted nearest-neighbour vote over a labelled-history index — the/// retrieval analogue of a judge model, with the evidence staying on the node.///-/// **6 settings.**+/// **8 settings.**#[derive(Debug, Clone, Serialize, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct DecisionsConfig {@@ -2368,6 +2369,12 @@ pub struct DecisionsConfig {/// (issue #1061). 2.0 — the issue's ≥ 2x floor for corrections — is the/// default; 1.0 makes corrections count exactly like any other example.pub human_weight: f64,+ /// Probability calibration for the decide surface (issue #1063): fit a+ /// correction from raw probabilities to calibrated ones on a held-out+ /// 20 % of the labelled history, then ship `p_cal` beside every `p_raw`.+ /// `isotonic` (PAVA) or `temperature` (one scalar on the log-odds);+ /// `none` (the default) ships `p_raw` alone and no `p_cal` at all.+ pub calibration: CalibrationMethod,}impl Default for DecisionsConfig {@@ -2380,6 +2387,7 @@ impl Default for DecisionsConfig {positive_label: "true".to_string(),min_confidence: 0.0,human_weight: 2.0,+ calibration: CalibrationMethod::None,}}}@@ -3190,7 +3198,7 @@ mod tests {("lifecycle", 1),("wal_tap", 10),("rerank", 3),- ("decisions", 7),+ ("decisions", 8),];/// Count the settings by *counting them*.@@ -3233,7 +3241,7 @@ mod tests {"the section table must sum to the whole config");assert_eq!(- total, 129,+ total, 130,"the total settings count changed. It is quoted in this module's \header, in xerj-common/src/lib.rs, in engine/README.md, in \xerj.default.toml and in EXPECTED_SETTINGS in \@@ -3262,6 +3270,38 @@ mod tests {assert!(cfg.grpc_h2c_exposed_off_loopback());}+ /// `decisions.calibration` parses the issue's two methods and refuses a+ /// typo by naming the valid ones — the same refusal shape as every other+ /// enum setting, so an operator hears about `calibration = "isotonic2"`+ /// at boot, not from a response that silently shipped no p_cal.+ #[test]+ fn decisions_calibration_parses_the_two_methods_and_refuses_a_typo() {+ for (value, expected) in [+ ("isotonic", CalibrationMethod::Isotonic),+ ("temperature", CalibrationMethod::Temperature),+ ] {+ let cfg = Config::from_toml_str(&format!(+ "[decisions]\nindex = \"judgements\"\ncalibration = \"{value}\"\n"+ ))+ .expect("parses");+ assert_eq!(cfg.decisions.calibration, expected);+ }+ // The default is none: an uncalibrated node, not a hidden isotonic.+ assert_eq!(+ DecisionsConfig::default().calibration,+ CalibrationMethod::None+ );+ let err = Config::from_toml_str(+ "[decisions]\nindex = \"judgements\"\ncalibration = \"isotonic2\"\n",+ )+ .expect_err("typo must be refused");+ let msg = err.to_string();+ assert!(+ msg.contains("calibration") && msg.contains("isotonic"),+ "the error names the field and the valid values: {msg}"+ );+ }+#[test]fn tls_on_bound_to_loopback_is_fine() {for bind in ["127.0.0.1", "127.0.0.5", "::1", "::ffff:127.0.0.1"] {diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rsindex bfd20a0e3..103374065 100644--- a/engine/crates/xerj-common/src/lib.rs+++ b/engine/crates/xerj-common/src/lib.rs@@ -7,16 +7,18 @@//! ## Design philosophy//!//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes-//! **129 settings**, each meaningful and production-tested. Every default+//! **130 settings**, each meaningful and production-tested. Every default//! is chosen so that a fresh deployment with zero configuration changes performs//! well for the majority of workloads.//!//! ## Modules//!-//! - [`config`] — TOML-based configuration (129 settings)+//! - [`config`] — TOML-based configuration (130 settings)//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`//! - [`error`] — Unified error type ([`XerjError`])//! - [`types`] — Core domain types (documents, fields, IDs)+//! - [`calibration`] — probability calibration for the decide surface:+//! isotonic (PAVA) and temperature (Platt) scaling, reliability curves, ECE//! - [`field_coercion`] — ES-faithful ingest-time coercion/enforcement for//! numeric and boolean fields (the one predicate every write path shares)//! - [`schema`] — Index schema management and mapping evolution@@ -28,6 +30,7 @@//! - [`xccode`] — the reference-coding semantics shared by `xerj code`,//! `xerj corpus`, and the `xerj_code_search` MCP tool (issue #977)+pub mod calibration;pub mod config;pub mod error;pub mod feedback;diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rsindex 6e4d98051..3481e4ae1 100644--- a/engine/crates/xerj-engine/tests/product_experience.rs+++ b/engine/crates/xerj-engine/tests/product_experience.rs@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {/// Settings in a default `Config`, measured by `count_settings`. Quoted in/// `xerj-common/src/config.rs`; keep the two in step.-const EXPECTED_SETTINGS: usize = 129;+const EXPECTED_SETTINGS: usize = 130;#[tokio::test]async fn journey_zero_config() {diff --git a/engine/xerj.default.toml b/engine/xerj.default.tomlindex d0c5821e0..e8f49ec05 100644--- a/engine/xerj.default.toml+++ b/engine/xerj.default.toml@@ -1,6 +1,6 @@# xerj Default Configuration# ============================================================-# This file sets 55 of XERJ's 129 user-facing settings to their+# This file sets 55 of XERJ's 130 user-facing settings to their# production-ready defaults, and describes the rest in comments.# Every value here IS the default, so copying this file changes# nothing until you edit it (a test enforces that). The complete@@ -656,3 +656,4 @@ access_log = false# positive_label = "true" # the label a noul's probability is the share of# min_confidence = 0.0 # /_decide abstains below this# human_weight = 2.0 # vote-weight multiplier for history docs carrying human: true+# calibration = "none" # none | isotonic | temperature — p_cal beside every p_raw (#1063)