Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

systemone: calibration layer — p_cal beside every p_raw (#1063)

xerj-org615fd96e92d9
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 86d0011a8..8a83184d1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **systemone: calibration layer — `p_cal` beside every `p_raw`, published
+ reliability curve ([#1063](https://github.com/xerj-org/xerj/issues/1063)).** A ladder
+ probability ranks before it odds — on the FiQA rerank baseline a hosted
+ noul of 0.93 meant relevance 34% of the time (ECE 0.3109). The new
+ `[decisions] calibration = isotonic | temperature` setting (default `none`,
+ the section's 8th setting) fits a correction on a deterministic held-out
+ fifth of the node's recorded outcomes — outcome documents carry the truth
+ label, the `p` that was served for that label, and NO `source` field
+ (flywheel answers are predictions; a fit on predictions would learn the
+ model is right by construction), 10 labelled pairs minimum — and every
+ probability-shipping path on the decide surface carries the calibrated
+ probability BESIDE the raw one: `/_decide` answers `p_raw` (the positive
+ label's share, the quantity a noul answers) plus `p_cal`;
+ `/v1/systemone` carries both in `decisions.evidence` per noul question
+ with the strictly-parsed answer objects untouched; choice per-option
+ shares are a different quantity and are not calibrated (the block says
+ `"scope": "noul"`). No calibration configured → no `p_cal` field at all;
+ configured but unfitted → `p_cal: null` with a published, actionable
+ reason; fitted → the fit's value, never a copy of `p_raw` pretending to
+ be calibrated — and whenever a `p_cal` exists, a `calibration` block rides
+ beside it with the method, the fit's ECE raw and calibrated, and the
+ fitted-on/held-out counts, so no probability ships without an ECE beside
+ it. `GET /_decide/_calibration` publishes the whole reliability curve
+ (binned `p_raw` vs empirical frequency with bin counts, raw and
+ calibrated), both ECEs, the fit's parameters (isotonic knots or the
+ temperature), the pair counts, and the fitted documents' date range; it
+ always refits, so it doubles as the operator's refresh (decide requests
+ read a per-index cache refreshed at most once a minute). Isotonic is PAVA
+ with linear interpolation and end-clamping; temperature is one scalar on
+ the log-odds fitted by golden-section NLL search; both are implemented
+ from first principles in `xerj_common::calibration` with method citations
+ (retrieval skipped: the reference corpora are not present in this
+ sandbox). Measured against the FiQA baseline's own retained curve
+ (`benchmarks/decisions-calibration`, stdlib Python mirroring the Rust
+ exactly — both reproduce the published raw 0.3109 and agree to the fourth
+ decimal): raw held-out ECE 0.3831 → isotonic 0.0330 (gate ≤ 0.10 MET);
+ temperature 0.3903, honestly failing — one scalar cannot map 0.93 → 0.34
+ while keeping 0.05 → 0.0002. Scope of that measurement: bin-level
+ held-out (the run retained 10 binned aggregates, not the 19,440 raw
+ pairs); the pair-level re-run needs the model's raw scores and is queued
+ as the rc.80 final form. `confidence` keeps its raw meaning and the
+ abstain gate keeps applying to it.
+
- **`POST /_ask` + `xerj_plan` MCP tool: prompt in, validated query DSL out
([#1056](https://github.com/xerj-org/xerj/issues/1056), PR
[#1076](https://github.com/xerj-org/xerj/pull/1076)).** `POST /_ask`
diff --git a/benchmarks/decisions-calibration/README.md b/benchmarks/decisions-calibration/README.md
new file mode 100644
index 000000000..054d939d9
--- /dev/null
+++ b/benchmarks/decisions-calibration/README.md
@@ -0,0 +1,66 @@
+# decisions-calibration — the #1063 gate measurement
+
+Issue #1063's gate: **Jev FiQA rerank ECE 0.3109 → ≤ 0.10 on held-out**,
+baseline `benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa`.
+Measured here from the baseline's own retained curve, with no new model
+calls, no network, no RNG.
+
+## Result
+
+| quantity | ECE | meets ≤ 0.10? |
+|---|---|---|
+| raw, all 19,440 pairs (reproduces the published number exactly) | **0.3109** | — |
+| raw, held-out bins only | 0.3831 | no |
+| **isotonic (PAVA), held-out** | **0.0330** | **yes** |
+| temperature, held-out | 0.3903 | no |
+
+`results/2026-09-30-calibration-gate.json` is the machine-readable run.
+
+Reproduce: `python3 fiqa_gate.py` (stdlib only; writes nothing unless `--out`).
+
+## Method, and its honest scope
+
+The baseline run retained the reliability curve as **10 binned aggregates**
+(`calibration.curve`: mean_conf, acc, n per bin; `per_query` holds nDCG only) —
+not the 19,440 raw (p, y) pairs. So this measurement works at **bin
+granularity**:
+
+- pairs are reconstructed per bin — `n` pairs at `mean_conf`,
+ `round(acc·n)` of them positive — the same reconstruction the Rust
+ module's unit tests pin (`FIQA_BASELINE_CURVE` + `bin_pairs` in
+ `engine/crates/xerj-common/src/calibration.rs`);
+- the held-out split is **even bins fit (0,2,4,6,8), odd bins evaluate
+ (1,3,5,7,9)** — deterministic (#940: no RNG, no clock), and bin-level
+ held-out means whole probability *regions* the fit never saw, which is a
+ stronger test of interpolation than a random pair split;
+- the raw held-out ECE (0.3831) is *harder* than the published 0.3109,
+ because the odd bins carry the pathological top bin (0.9297 conf → 0.3378
+ acc). The gate was not met by evaluating on an easy split.
+
+What this does NOT claim: a pair-level held-out measurement. The fit here
+sees five points, not 15,000 — bin-level aggregates are already smoothed, and
+smoothing flatters calibration fits. The pair-level re-run needs the model's
+raw scores re-emitted per query (the baseline kept only nDCG); that costs API
+calls and is queued as the **rc.80 final form** of this gate. The production
+path (`/_decide/_calibration`) fits on raw pairs by construction, so the
+serving behaviour is the stronger form already.
+
+## Why temperature fails, and is still shipped
+
+Temperature is one scalar on the log-odds: `p' = σ(logit(p)/T)`. The FiQA
+curve needs 0.93 → 0.34 at the top while keeping 0.05 → 0.0002 at the bottom —
+a steep monotone *remap*, not the uniform flattening a single T can express.
+Its NLL optimum here is T = 1.1517 (barely softer than raw), and its held-out
+ECE (0.3903) barely moves. Isotonic is the gate-meeting method; temperature
+is shipped because it is the right tool when the history is thin (few knots
+to overfit) or monotone-in-log-odds — and `/_decide/_calibration` publishes
+both ECEs, so a node running either method shows its own number.
+
+## Cross-check
+
+`fiqa_gate.py` mirrors the Rust implementation exactly — same bin edges
+(`floor(p·10)`, p=1 → bin 9), same PAVA with linear interpolation between
+knots and end-clamping, same temperature bracket (ln T ∈ [ln(1/16), ln 16],
+100 golden-section iterations, logit clamped at 1e-6). Agreement between the
+two (0.0330 here; the Rust pinning test asserts ≤ 0.10 on the identical
+split) is a cross-check on both.
diff --git a/benchmarks/decisions-calibration/fiqa_gate.py b/benchmarks/decisions-calibration/fiqa_gate.py
new file mode 100644
index 000000000..fdc4e3ead
--- /dev/null
+++ b/benchmarks/decisions-calibration/fiqa_gate.py
@@ -0,0 +1,280 @@
+#!/usr/bin/env python3
+"""Measure the #1063 calibration gate against the FiQA rerank baseline.
+
+Issue #1063's gate: Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out data.
+The baseline run (benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa)
+retained the reliability curve as 10 BINNED AGGREGATES, not the 19,440 raw
+pairs (per_query holds nDCG only), so this measurement works at bin
+granularity: each bin's (mean_conf, acc, n) is expanded into n pairs at
+mean_conf, positives = round(acc*n), exactly the reconstruction the Rust
+module's unit tests use (engine/crates/xerj-common/src/calibration.rs,
+FIQA_BASELINE_CURVE + bin_pairs).
+
+Held-out split, deterministic (#940 rule — no RNG, no clock): even-index bins
+(0,2,4,6,8) fit the calibration, odd-index bins (1,3,5,7,9) evaluate it.
+Held-out at bin level means whole probability REGIONS the fit never saw —
+stronger than a random pair split on interpolation, weaker than the full
+pair-level re-run (which needs the model's raw scores; that is the rc.80
+final form, noted in the README).
+
+This script mirrors the Rust implementation exactly — same bin edges
+(b = floor(p*10), p=1 -> bin 9), same PAVA with linear interpolation between
+knots and clamping at the ends, same temperature bracket (ln T in
+[ln(1/16), ln 16], 100 golden-section iterations, logit clamped at 1e-6) — so
+agreement between the two is a cross-check on both. Stdlib only, no network,
+no API calls: every number comes from the existing baseline file.
+
+Usage: python3 fiqa_gate.py [--baseline PATH] [--out PATH]
+"""
+
+import argparse
+import json
+import math
+import sys
+from pathlib import Path
+
+DEFAULT_BASELINE = (
+ Path(__file__).resolve().parents[1]
+ / "beir-hybrid"
+ / "results"
+ / "2026-09-20-rerank-full-fiqa"
+ / "fiqa-full-run.json"
+)
+GATE = 0.10
+BINS = 10
+LOGIT_EPS = 1e-6
+T_BRACKET = (math.log(1.0 / 16.0), math.log(16.0))
+GOLDEN_ITERS = 100
+
+
+# ── the algorithms, mirroring engine/crates/xerj-common/src/calibration.rs ──
+
+
+def bin_of(p, bins=BINS):
+ """b = floor(p*bins), p=1 -> the last bin (the Rust module's clamp)."""
+ b = int(p * bins)
+ return min(b, bins - 1)
+
+
+def ece(pairs, bins=BINS):
+ """ECE = sum_b (n_b/N)*|acc_b - conf_b| (Naeini et al. 2015)."""
+ n = len(pairs)
+ if n == 0:
+ return 0.0
+ sums = [[0, 0.0, 0.0] for _ in range(bins)] # count, sum p, sum y
+ for p, y in pairs:
+ s = sums[bin_of(p, bins)]
+ s[0] += 1
+ s[1] += p
+ s[2] += y
+ total = 0.0
+ for c, sp, sy in sums:
+ if c:
+ total += (c / n) * abs(sy / c - sp / c)
+ return total
+
+
+class Isotonic:
+ """PAVA (pool-adjacent-violators) with the Rust module's knots:
+ linear interpolation between block means, clamped at the ends
+ (Barlow et al. 1972; Zadrozny & Elkan 2002 for calibration use)."""
+
+ def __init__(self, pairs):
+ pts = sorted(((p, y) for p, y in pairs), key=lambda t: (t[0], t[1]))
+ # Pool tied p by weighted mean first.
+ xs, ys, ws = [], [], []
+ i = 0
+ while i < len(pts):
+ j = i
+ p_sum = y_sum = 0.0
+ while j < len(pts) and pts[j][0] == pts[i][0]:
+ p_sum += pts[j][0]
+ y_sum += pts[j][1]
+ j += 1
+ xs.append(p_sum / (j - i))
+ ys.append(y_sum / (j - i))
+ ws.append(float(j - i))
+ i = j
+ # Stack of blocks: [y_sum, n, first_index].
+ blocks = []
+ for x, y, w in zip(xs, ys, ws):
+ blocks.append([y * w, w, len(blocks)])
+ while len(blocks) >= 2 and (
+ blocks[-2][0] / blocks[-2][1] > blocks[-1][0] / blocks[-1][1]
+ ):
+ y2, n2, f2 = blocks.pop()
+ y1, n1, f1 = blocks.pop()
+ blocks.append([y1 + y2, n1 + n2, f1])
+ self.knots = []
+ for y_sum, n, first in blocks:
+ self.knots.append((xs[first], y_sum / n))
+
+ def apply(self, p):
+ k = self.knots
+ if not k:
+ return p
+ if p <= k[0][0]:
+ return k[0][1]
+ if p >= k[-1][0]:
+ return k[-1][1]
+ lo, hi = 0, len(k) - 1
+ while hi - lo > 1:
+ mid = (lo + hi) // 2
+ if k[mid][0] <= p:
+ lo = mid
+ else:
+ hi = mid
+ (x0, y0), (x1, y1) = k[lo], k[hi]
+ if x1 == x0:
+ return y0
+ return y0 + (y1 - y0) * (p - x0) / (x1 - x0)
+
+
+def logit(p):
+ p = min(max(p, LOGIT_EPS), 1.0 - LOGIT_EPS)
+ return math.log(p / (1.0 - p))
+
+
+def sigmoid(z):
+ if z >= 0:
+ return 1.0 / (1.0 + math.exp(-z))
+ e = math.exp(z)
+ return e / (1.0 + e)
+
+
+def temperature_nll(pairs, ln_t):
+ """NLL of the pairs under p' = sigmoid(logit(p)/T) (Platt 1999; Guo et
+ al. 2017). Saturated pairs contribute the eps-clamped log-probability —
+ the softplus form the Rust module uses."""
+ t = math.exp(ln_t)
+ nll = 0.0
+ for p, y in pairs:
+ q = min(max(sigmoid(logit(p) / t), LOGIT_EPS), 1.0 - LOGIT_EPS)
+ nll += -(y * math.log(q) + (1.0 - y) * math.log(1.0 - q))
+ return nll
+
+
+class Temperature:
+ def __init__(self, pairs):
+ lo, hi = T_BRACKET
+ gr = (math.sqrt(5.0) - 1.0) / 2.0
+ a, b = lo, hi
+ c = b - gr * (b - a)
+ d = a + gr * (b - a)
+ fc, fd = temperature_nll(pairs, c), temperature_nll(pairs, d)
+ for _ in range(GOLDEN_ITERS):
+ if fc < fd:
+ b, d, fd = d, c, fc
+ c = b - gr * (b - a)
+ fc = temperature_nll(pairs, c)
+ else:
+ a, c, fc = c, d, fd
+ d = a + gr * (b - a)
+ fd = temperature_nll(pairs, d)
+ self.t = math.exp((a + b) / 2.0)
+
+ def apply(self, p):
+ return sigmoid(logit(p) / self.t)
+
+
+# ── the measurement ──────────────────────────────────────────────────────────
+
+
+def bin_pairs(bins):
+ """Expand binned aggregates into pairs — the same reconstruction as the
+ Rust module's bin_pairs test helper."""
+ out = []
+ for conf, acc, n in bins:
+ positives = round(acc * n)
+ out.extend((conf, 1.0 if i < positives else 0.0) for i in range(n))
+ return out
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__)
+ ap.add_argument("--baseline", type=Path, default=DEFAULT_BASELINE)
+ ap.add_argument("--out", type=Path, default=None)
+ args = ap.parse_args()
+
+ run = json.loads(args.baseline.read_text())
+ cal = run["calibration"]
+ curve = cal["curve"]
+ bins = [
+ (curve[str(i)]["mean_conf"], curve[str(i)]["acc"], curve[str(i)]["n"])
+ for i in range(cal["bins"])
+ ]
+ all_pairs = bin_pairs(bins)
+ fit_pairs = bin_pairs(bins[::2]) # even bins
+ held_pairs = bin_pairs(bins[1::2]) # odd bins — whole regions unseen
+
+ raw_all = ece(all_pairs)
+ raw_held = ece(held_pairs)
+
+ iso = Isotonic(fit_pairs)
+ iso_calibrated = [(iso.apply(p), y) for p, y in held_pairs]
+ iso_held = ece(iso_calibrated)
+
+ temp = Temperature(fit_pairs)
+ temp_calibrated = [(temp.apply(p), y) for p, y in held_pairs]
+ temp_held = ece(temp_calibrated)
+
+ result = {
+ "issue": 1063,
+ "gate": "Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out",
+ "gate_threshold": GATE,
+ "baseline_run": str(args.baseline),
+ "method_note": (
+ "The baseline retained 10 binned aggregates (n_pairs=19440), not "
+ "the raw pairs; pairs are reconstructed per bin at mean_conf with "
+ "positives = round(acc*n), and the held-out split is at BIN "
+ "granularity: fit on even bins, evaluate on odd bins (whole "
+ "probability regions the fit never saw). The pair-level held-out "
+ "re-run needs the model's raw scores and is the rc.80 final form."
+ ),
+ "pairs": {
+ "total": len(all_pairs),
+ "fitted_on": len(fit_pairs),
+ "held_out": len(held_pairs),
+ },
+ "raw_ece": {
+ "all_bins": round(raw_all, 4),
+ "published": cal["ece"],
+ "held_out_bins": round(raw_held, 4),
+ },
+ "isotonic": {
+ "held_out_ece": round(iso_held, 4),
+ "meets_gate": iso_held <= GATE,
+ "knots": [[round(x, 4), round(y, 4)] for x, y in iso.knots],
+ },
+ "temperature": {
+ "held_out_ece": round(temp_held, 4),
+ "meets_gate": temp_held <= GATE,
+ "t": round(temp.t, 4),
+ },
+ }
+
+ print(json.dumps(result, indent=2))
+ if args.out:
+ args.out.parent.mkdir(parents=True, exist_ok=True)
+ args.out.write_text(json.dumps(result, indent=2) + "\n")
+ print(f"\nwrote {args.out}", file=sys.stderr)
+
+ ok = raw_all <= cal["ece"] + 0.0005 and iso_held <= GATE
+ if not ok:
+ print(
+ "GATE CHECK FAILED: reproduction "
+ f"{raw_all:.4f} vs published {cal['ece']}, isotonic held-out "
+ f"{iso_held:.4f} vs gate {GATE}",
+ file=sys.stderr,
+ )
+ return 1
+ print(
+ f"\ngate met: raw {raw_held:.4f} held-out -> isotonic {iso_held:.4f} "
+ f"(<= {GATE}); temperature {temp_held:.4f}",
+ file=sys.stderr,
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.json b/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.json
new file mode 100644
index 000000000..71d897c15
--- /dev/null
+++ b/benchmarks/decisions-calibration/results/2026-09-30-calibration-gate.json
@@ -0,0 +1,48 @@
+{
+ "issue": 1063,
+ "gate": "Jev FiQA rerank ECE 0.3109 -> <= 0.10 on held-out",
+ "gate_threshold": 0.1,
+ "baseline_run": "/workspace/.claude/worktrees/wf_6ba42604-1b6-1/benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json",
+ "method_note": "The baseline retained 10 binned aggregates (n_pairs=19440), not the raw pairs; pairs are reconstructed per bin at mean_conf with positives = round(acc*n), and the held-out split is at BIN granularity: fit on even bins, evaluate on odd bins (whole probability regions the fit never saw). The pair-level held-out re-run needs the model's raw scores and is the rc.80 final form.",
+ "pairs": {
+ "total": 19440,
+ "fitted_on": 11446,
+ "held_out": 7994
+ },
+ "raw_ece": {
+ "all_bins": 0.3109,
+ "published": 0.3109,
+ "held_out_bins": 0.3831
+ },
+ "isotonic": {
+ "held_out_ece": 0.033,
+ "meets_gate": true,
+ "knots": [
+ [
+ 0.0455,
+ 0.0002
+ ],
+ [
+ 0.2434,
+ 0.0034
+ ],
+ [
+ 0.4445,
+ 0.0077
+ ],
+ [
+ 0.6451,
+ 0.0224
+ ],
+ [
+ 0.8452,
+ 0.1242
+ ]
+ ]
+ },
+ "temperature": {
+ "held_out_ece": 0.3903,
+ "meets_gate": false,
+ "t": 1.1517
+ }
+}
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index d60b2f535..d5b8a69a0 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -119,6 +119,65 @@ number or the config refuses to boot (0 would erase a correction from the
vote, a negative value would invert it). The boosted weight is the weight the
vote used, so `/_decide` shows it in each neighbour's `weight`.
+## Calibration: `p_cal` beside every `p_raw` (#1063)
+
+A ladder probability ranks before it odds. On the FiQA rerank baseline a
+hosted noul of 0.93 meant relevance 34 % of the time — ECE 0.3109
+([benchmarks/decisions-calibration](../benchmarks/decisions-calibration)).
+With `[decisions] calibration` set, the node fits a correction on its own
+recorded outcomes and ships the calibrated probability BESIDE the raw one:
+
+- **`isotonic`** — non-decreasing regression of the empirical positive rate
+ on the raw probability (PAVA), interpolated between knots. The method that
+ meets the FiQA gate: held-out ECE 0.0330 against raw 0.3831 on the same
+ bins (gate ≤ 0.10).
+- **`temperature`** — one scalar on the log-odds, `p' = σ(logit(p)/T)`,
+ fitted on negative log-likelihood. Rank-preserving by construction, and
+ the right tool on a thin history; on the FiQA-shaped curve it barely moves
+ (held-out 0.3903) because one scalar cannot map 0.93 → 0.34 while keeping
+ 0.05 → 0.0002.
+
+Both fit on a **deterministic held-out fifth** of the outcome pairs
+(stable-sorted, every fifth held out — no RNG, no clock, so the same history
+produces bit-identical `p_cal`), and the fit is cached per index and
+refreshed at most once a minute; `GET /_decide/_calibration` always refits.
+
+**Recording an outcome — the fit's data contract.** A history document is a
+calibration pair when it carries the truth `label` (the configured
+`label_field`), the `p` that was served **for that document's own label**,
+and **no `source` field**:
+
+```sh
+curl -X PUT localhost:9200/judgements/_doc/outcome-42 -H 'content-type: application/json' \
+ -d '{"text": "the message that was decided", "label": "false", "p": 0.9}'
+```
+
+That reads as "the system said 0.9 for its own label; the truth was `false`"
+— the positive-label probability 1 − 0.9 against outcome 0. A document with
+`source` is a flywheel answer, i.e. the model's own prediction, and is
+excluded: a fit on predictions would learn the model is right by
+construction. `human: true` corrections carrying the `p` they correct are the
+best pairs there are — truth plus the number that was wrong. Ten labelled
+pairs minimum, or the node says so instead of extrapolating.
+
+**The honesty rules, all three on the wire.** No calibration configured → no
+`p_cal` field at all (`calibration: {"configured": "none"}` on `/_decide`).
+Configured but unfitted → `p_cal: null` with the reason published in the
+`calibration` block beside it. Fitted → `p_cal` is the fit's value, and it is
+never a copy of `p_raw` pretending to be calibrated. Whenever `p_cal`
+exists, the `calibration` block rides with it — method, scope, the fit's ECE
+raw and calibrated, and what it was fitted on — so **no probability ships
+without an ECE beside it**. `p_raw` is always the honest raw share;
+`confidence` keeps its raw meaning and the abstain gate keeps applying to it
+(calibration changes what the number means, not the verdict).
+
+**Scope.** The calibrated quantity is the positive label's probability — the
+noul's answer, `/_decide`'s `p_raw`. Per-option `choice` shares are a
+different quantity and are not calibrated by this fit (the block says
+`"scope": "noul"`). The fit applies to every tier's noul through one seam —
+history vote, local head, and (when that tier is built) a hosted passthrough
+score — with no usable fit meaning "say so", never `p_cal = p_raw`.
+
## Set it up
```toml
@@ -131,6 +190,7 @@ text_field = "text"
positive_label = "true" # the label whose share a noul answers
min_confidence = 0.0 # /_decide abstains below this
human_weight = 2.0 # vote-weight multiplier for history docs carrying human: true
+calibration = "none" # none | isotonic | temperature — p_cal beside every p_raw (#1063)
```
The history index is ordinary documents: one per example, with the text the
@@ -222,11 +282,13 @@ tier is armed), `question` (required), `k` (default the configured 10,
clamped 1..100), `positive_label` (default the configured one). The response
returns the label, its confidence, the verdict (`abstain` below
`decisions.min_confidence`, or when no labelled neighbour exists), the **tier**
-and **source** that answered (the same tier name under both fields), and the
-**neighbours** — each with `_id`, `label`, `_score` (the engine's own BM25
-score), `weight` (1/rank, × `decisions.human_weight` for a `human: true`
-document), and the text — so every answer can be checked against the evidence
-that produced it. A local-tier answer has no neighbours (having none is what
+and **source** that answered (the same tier name under both fields), the
+**`p_raw`** — the positive label's vote share, the quantity a noul answers,
+not the winner's confidence — with **`p_cal`** beside it when calibration is
+configured (see above), and the **neighbours** — each with `_id`, `label`,
+`_score` (the engine's own BM25 score), `weight` (1/rank, ×
+`decisions.human_weight` for a `human: true` document), and the text — so
+every answer can be checked against the evidence that produced it. A local-tier answer has no neighbours (having none is what
tier 2 means) and carries `tier: "local"`, `source: "local"`,
`model: "xerj-decide-local-1"`; when it did not abstain and the request named
an index, it is cached into that index (the flywheel above).
@@ -236,6 +298,27 @@ curl -s localhost:9200/_decide -H 'content-type: application/json' \
-d '{"index":"judgements","question":"refund my subscription","k":5}'
```
+## `GET /_decide/_calibration`
+
+The publication (#1063): the reliability curve of a labelled history — every
+occupied bin's `[lo, hi)`, count, mean probability and empirical positive
+rate — for the raw probabilities and (when fitted) the calibrated ones, the
+ECE of each, the fit's parameters (an isotonic fit's knots, or the
+temperature), the pair counts (fitted on / held out / total), and the date
+range of the fitted documents. Always refits, so it doubles as the
+operator's refresh; the fit it publishes becomes the cached fit every decide
+request reads.
+
+```sh
+curl -s 'localhost:9200/_decide/_calibration?index=judgements'
+```
+
+`?index=` names the history (default: the configured `[decisions] index`; a
+request naming nothing on a node that configures nothing is a 422, not an
+empty curve). The raw curve and raw ECE publish even when calibration is not
+configured — the reliability of the raw probabilities is audit information
+in its own right — with a `reason` naming the setting that would fit them.
+
## Numbers, and only the ones we measured
Measured, ours, reproducible (raw JSON and runners in the results dirs):
@@ -243,6 +326,12 @@ Banking77 77-way intent 0.933 accuracy / ECE 0.012 (86.9% of traffic clears
confidence 0.8 at 0.979 accuracy); SMS spam 0.983 accuracy / 0.944 F1 under a
millisecond on BM25 alone; both in
[benchmarks/decisions-as-retrieval](../benchmarks/decisions-as-retrieval).
+The calibration gate: isotonic held-out ECE 0.0330 against raw 0.3831 on the
+FiQA rerank baseline's own curve (published raw 0.3109 reproduced exactly;
+gate ≤ 0.10), measured at bin granularity from the retained aggregates —
+the pair-level re-run needs the model's raw scores and is the rc.80 final
+form — in
+[benchmarks/decisions-calibration](../benchmarks/decisions-calibration).
The wire gate transcript is in
[benchmarks/systemone-gate](../benchmarks/systemone-gate). A third, smaller
run measures `/_decide` itself on an email-labelling history — 4 labels,
diff --git a/engine/crates/xerj-api/src/router.rs b/engine/crates/xerj-api/src/router.rs
index b08bbf087..dc818c6e1 100644
--- a/engine/crates/xerj-api/src/router.rs
+++ b/engine/crates/xerj-api/src/router.rs
@@ -300,6 +300,13 @@ pub fn build_es_compat_router(state: AppState) -> Router {
// The audit twin of /v1/systemone: same vote, named index, neighbours
// and abstain instead of a wire-shaped error.
.route("/_decide", post(systemone_api::decide))
+ // The decide surface's published reliability (#1063): the binned
+ // curve, the ECE raw and calibrated, the fit params. GET, read-only,
+ // same posture as the surface it publishes.
+ .route(
+ "/_decide/_calibration",
+ get(systemone_api::decide_calibration),
+ )
// #1056, ES-compat surface: same handler as /v1/ask; the index
// rides in the body, exactly like /_decide and /_msearch.
.route("/_ask", post(ask_api::ask))
diff --git a/engine/crates/xerj-api/src/state.rs b/engine/crates/xerj-api/src/state.rs
index 9e2e9e330..73fd7a885 100644
--- a/engine/crates/xerj-api/src/state.rs
+++ b/engine/crates/xerj-api/src/state.rs
@@ -553,6 +553,12 @@ pub struct AppState {
/// from the environment again. A test arms a node by replacing this
/// field, so parallel tests never race on process-wide env state.
pub decide: Arc<crate::systemone_api::DecideSettings>,
+ /// The decide surface's calibration fits (#1063): one cached fit per
+ /// decisions index, lazily computed by [`crate::systemone_api`] and
+ /// refreshed by `GET /_decide/_calibration`. Held here, not in the
+ /// handler, so every request path that ships a probability reads the same
+ /// fit.
+ pub decide_calibration: Arc<crate::systemone_api::CalibrationCache>,
}
impl AppState {
@@ -592,9 +598,11 @@ impl AppState {
&config.rerank.endpoint,
));
let decide = Arc::new(crate::systemone_api::DecideSettings::from_env());
+ let decide_calibration = Arc::new(crate::systemone_api::CalibrationCache::default());
Self {
rerank,
decide,
+ decide_calibration,
config: Arc::new(config),
engine: Arc::new(engine),
metrics: Arc::new(metrics),
diff --git a/engine/crates/xerj-api/src/systemone_api.rs b/engine/crates/xerj-api/src/systemone_api.rs
index c1d7db857..25a2b999b 100644
--- a/engine/crates/xerj-api/src/systemone_api.rs
+++ b/engine/crates/xerj-api/src/systemone_api.rs
@@ -59,6 +59,24 @@
//! in the vote — see [`neighbour_weight`]. The weight is read in the vote
//! aggregation, so a correction outranks the cached answers it corrects.
//!
+//! # Calibration: `p_cal` beside every `p_raw` (#1063)
+//!
+//! A ladder probability ranks before it odds: on the FiQA rerank baseline a
+//! hosted noul of 0.93 meant relevance 34 % of the time (ECE 0.3109). With
+//! `[decisions] calibration = isotonic | temperature`, the node fits a
+//! correction on a held-out fifth of its recorded outcomes ([`xerj_common::
+//! calibration`]) and answers carry the calibrated probability BESIDE the raw
+//! one — `p_raw` never replaced, `p_cal` absent when no calibration is
+//! configured and `null` (with a published reason) when configured but
+//! unfitted, never a copy of `p_raw` pretending to be calibrated. The fit
+//! applies to every tier's probability through one seam, so the hosted tier,
+//! when it is built, calibrates its passthrough score the same way. The
+//! calibrated quantity is the positive label's probability (a noul, `/_decide`'s
+//! `p_raw`); per-option `choice` shares are a different quantity and are not
+//! calibrated by this fit — every response that ships a probability carries
+//! the fit's ECE beside it, and `GET /_decide/_calibration` publishes the
+//! whole reliability curve.
+//!
//! # The wire contract, and where we deliberately break it
//!
//! Request: `{ state, model, questions: { id: { type, instructions, criteria } } }`.
@@ -104,10 +122,13 @@
//! vote analogue and no benchmark: 422, in the docs.
use std::collections::BTreeMap;
+use std::collections::HashMap;
#[cfg(feature = "decide-local")]
use std::path::Path;
#[cfg(feature = "decide-local")]
use std::path::PathBuf;
+use std::sync::Arc;
+use std::time::Duration;
use std::time::Instant;
use axum::extract::State;
@@ -115,6 +136,10 @@ use axum::http::StatusCode;
use axum::response::IntoResponse;
use axum::Json;
use serde_json::{json, Value};
+use xerj_common::calibration::CalibrationMethod;
+use xerj_common::calibration::CalibrationPair;
+use xerj_common::calibration::CalibrationReport;
+use xerj_common::calibration::FittedCalibration;
use xerj_common::config::DecisionsConfig;
use xerj_query::parse_request;
@@ -304,6 +329,291 @@ fn is_human_correction(source: &Value) -> bool {
source.get("human") == Some(&Value::Bool(true))
}
+// ─────────────────────────────────────────────────────────────────────────────
+// Calibration — p_cal beside every p_raw (#1063)
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// How many history documents one fit reads, at most: the search page cap
+/// (`index.max_result_window`, 10,000 by default). A history with more
+/// labelled outcomes than this fits on the first page in engine order,
+/// canonically re-sorted before the split — the histories this feature is
+/// for (thousands of cached answers, hundreds of recorded outcomes) sit far
+/// under it.
+const MAX_FIT_DOCS: usize = 10_000;
+
+/// Fewer labelled pairs than this and there is no fit to publish: the
+/// response says so rather than extrapolate a curve from a handful of points.
+/// Ten is the smallest set whose held-out fifth still evaluates.
+const MIN_FIT_PAIRS: usize = 10;
+
+/// A cached fit refreshes at most this often on demand, so a decide request
+/// never pays a full index scan per call and an operator who indexes more
+/// outcome documents sees the fit move within a minute without a restart.
+/// `GET /_decide/_calibration` always refits, whatever this says.
+const FIT_REFRESH: Duration = Duration::from_secs(60);
+
+/// One per-index cache entry: the report (or the reason there is none) and
+/// when it was computed. The clock stamps the CACHE, never the fit — the fit
+/// is a pure function of the pair multiset, the #940 rule.
+struct CachedFit {
+ report: Result<Arc<CalibrationReport>, String>,
+ fitted_at: Instant,
+}
+
+/// The node's calibration state: one cached fit per decisions index, held in
+/// [`AppState::decide_calibration`] so every probability-shipping path reads
+/// the same fit. Fitted lazily on first use, refreshed at most once a minute,
+/// force-refreshed by [`decide_calibration`] (the publication endpoint).
+#[derive(Default)]
+pub struct CalibrationCache {
+ entries: tokio::sync::Mutex<HashMap<String, CachedFit>>,
+}
+
+impl CalibrationCache {
+ /// Publish a fit the caller already computed (the `_calibration`
+ /// endpoint's own scan) as the index's cached fit.
+ async fn refresh_from(&self, index: &str, report: Arc<CalibrationReport>) {
+ self.entries.lock().await.insert(
+ index.to_string(),
+ CachedFit {
+ report: Ok(report),
+ fitted_at: Instant::now(),
+ },
+ );
+ }
+}
+
+/// What calibration a request has available — the one lookup every
+/// probability-emitting path on this surface shares.
+enum CalibrationState {
+ /// `[decisions] calibration` is `none`: `p_raw` ships alone and no
+ /// `p_cal` exists at all.
+ Off,
+ /// Configured and fitted: the report every `p_cal` and ECE comes from.
+ Fitted(Arc<CalibrationReport>),
+ /// Configured but no usable fit — the string is why, published beside the
+ /// `p_cal: null` it explains.
+ Unfit(String),
+}
+
+/// Resolve the request's calibration state — at most one scan of the
+/// decisions index, cached per `cfg.index`. `force` is the operator's
+/// explicit refit. The scan and the fit are deterministic, so two requests
+/// between refreshes serve bit-identical `p_cal`.
+async fn calibration_state(
+ state: &AppState,
+ cfg: &DecisionsConfig,
+ force: bool,
+) -> CalibrationState {
+ if cfg.calibration == CalibrationMethod::None {
+ return CalibrationState::Off;
+ }
+ if cfg.index.is_empty() {
+ return CalibrationState::Unfit(
+ "no decisions index: calibration is fitted on labelled history — set [decisions] \
+ index, or name one per request on /_decide"
+ .to_string(),
+ );
+ }
+ let mut entries = state.decide_calibration.entries.lock().await;
+ let fresh = match entries.get(&cfg.index) {
+ Some(cached) => !force && cached.fitted_at.elapsed() <= FIT_REFRESH,
+ None => false,
+ };
+ if fresh {
+ let cached = entries.get(&cfg.index).expect("checked above");
+ return match &cached.report {
+ Ok(report) => CalibrationState::Fitted(report.clone()),
+ Err(reason) => CalibrationState::Unfit(reason.clone()),
+ };
+ }
+ let (pairs, ts_range) = match scan_calibration_pairs(state, cfg).await {
+ Ok(found) => found,
+ Err(reason) => {
+ entries.insert(
+ cfg.index.clone(),
+ CachedFit {
+ report: Err(reason.clone()),
+ fitted_at: Instant::now(),
+ },
+ );
+ return CalibrationState::Unfit(reason);
+ }
+ };
+ let outcome = fit_pairs(cfg, &pairs, ts_range);
+ let state_out = match &outcome {
+ Ok(report) => CalibrationState::Fitted(report.clone()),
+ Err(reason) => CalibrationState::Unfit(reason.clone()),
+ };
+ entries.insert(
+ cfg.index.clone(),
+ CachedFit {
+ report: outcome,
+ fitted_at: Instant::now(),
+ },
+ );
+ state_out
+}
+
+/// Fit the configured method on scanned pairs, or say why there is no fit to
+/// ship. Pure plumbing over [`xerj_common::calibration::fit_calibration`].
+fn fit_pairs(
+ cfg: &DecisionsConfig,
+ pairs: &[CalibrationPair],
+ ts_range: Option<(String, String)>,
+) -> Result<Arc<CalibrationReport>, String> {
+ if pairs.len() < MIN_FIT_PAIRS {
+ return Err(unfit_reason(cfg, pairs.len()));
+ }
+ let mut report = xerj_common::calibration::fit_calibration(cfg.calibration, pairs)
+ .ok_or_else(|| "[decisions] calibration is none; nothing to fit".to_string())?;
+ report.ts_range = ts_range;
+ Ok(Arc::new(report))
+}
+
+/// The published reason a configured calibration has no fit — names what a
+/// fit needs and how to record it.
+fn unfit_reason(cfg: &DecisionsConfig, pairs: usize) -> String {
+ format!(
+ "only {pairs} labelled calibration pair{} in `{}` — a fit needs at least {MIN_FIT_PAIRS} \
+ (#1063): record outcomes by indexing documents carrying the truth label, the p that \
+ was served, and no `source` field (a `source` marks a cached answer, whose label is \
+ the model's own prediction, and a fit on predictions would learn the model is right \
+ by construction)",
+ if pairs == 1 { "" } else { "s" },
+ cfg.index
+ )
+}
+
+/// One history document's calibration pair, or `None` when the document is
+/// not a labelled outcome. Pure, and unit-tested: this function IS the
+/// data contract of the fit.
+///
+/// A document is an OUTCOME — not a prediction — when it names a truth label
+/// in the configured `label_field`, carries the probability that was served
+/// in `p`, and has NO `source` field: `source` marks a flywheel answer
+/// (#1061), whose label is the model's own guess, and calibrating on guesses
+/// would teach the layer the model is right by construction. Record an
+/// outcome with the ordinary indexing path:
+/// `{"text": …, "label": <the truth>, "p": <what was served>}` — `human:
+/// true` optional, the vote already weights it. The pair is the positive
+/// label's probability — `p` when the label is the positive one, `1 − p`
+/// when it is not — against the outcome `label == positive_label`.
+fn pair_from_doc(source: &Value, cfg: &DecisionsConfig) -> Option<CalibrationPair> {
+ if source.get("source").is_some() {
+ return None; // a cached answer: a prediction, not an outcome
+ }
+ let label = source.get(&cfg.label_field).and_then(Value::as_str)?;
+ let p = source.get("p").and_then(Value::as_f64)?;
+ if !p.is_finite() || !(0.0..=1.0).contains(&p) {
+ return None; // a p that is not a probability is not calibration data
+ }
+ let positive = label == cfg.positive_label;
+ Some(CalibrationPair {
+ p: if positive { p } else { 1.0 - p },
+ y: positive as u8 as f64,
+ })
+}
+
+/// Scan the decisions index for labelled outcome documents and return them
+/// as calibration pairs, with the RFC 3339 range of their `ts` fields.
+async fn scan_calibration_pairs(
+ state: &AppState,
+ cfg: &DecisionsConfig,
+) -> Result<(Vec<CalibrationPair>, Option<(String, String)>), String> {
+ let idx = state.engine.get_index(&cfg.index).map_err(|e| {
+ format!(
+ "decisions index `{}` could not be searched for calibration pairs: {e}",
+ cfg.index
+ )
+ })?;
+ let query_body = json!({
+ "query": { "match_all": {} },
+ "size": MAX_FIT_DOCS,
+ "_source": true,
+ });
+ let search_req =
+ parse_request(&query_body).map_err(|e| format!("internal query would not parse: {e}"))?;
+ let result = idx
+ .search(&search_req)
+ .await
+ .map_err(|e| format!("decisions index `{}` could not be searched: {e}", cfg.index))?;
+ let mut pairs = Vec::with_capacity(result.hits.len());
+ let mut oldest: Option<String> = None;
+ let mut newest: Option<String> = None;
+ for hit in &result.hits {
+ if let Some(pair) = pair_from_doc(&hit.source, cfg) {
+ pairs.push(pair);
+ // The published range spans the FITTED documents — the ones that
+ // became pairs — not every cached answer in the index.
+ if let Some(ts) = hit.source.get("ts").and_then(Value::as_str) {
+ // RFC 3339 sorts lexicographically, so min/max by string is
+ // the range.
+ oldest = Some(match oldest {
+ Some(o) if o.as_str() <= ts => o,
+ _ => ts.to_string(),
+ });
+ newest = Some(match newest {
+ Some(n) if n.as_str() >= ts => n,
+ _ => ts.to_string(),
+ });
+ }
+ }
+ }
+ Ok((pairs, oldest.zip(newest)))
+}
+
+/// `p_cal` for one `p_raw` under the request's calibration state: absent
+/// when calibration is off, `null` when configured but unfit (the reason
+/// rides in the calibration block), and the fit's value otherwise — never a
+/// copy of `p_raw` pretending to be calibrated.
+fn p_cal_value(state: &CalibrationState, p_raw: f64) -> Option<Value> {
+ match state {
+ CalibrationState::Off => None,
+ CalibrationState::Unfit(_) => Some(Value::Null),
+ CalibrationState::Fitted(report) => Some(json!(round6(report.fit.apply(p_raw)))),
+ }
+}
+
+/// The calibration block a response carries whenever calibration is
+/// configured: the method, its scope, the fit's held-out ECE raw and
+/// calibrated, and what it was fitted on. This block is the "ECE beside
+/// every probability" — the surface's honesty contract (#1063).
+fn calibration_block(state: &CalibrationState) -> Option<Value> {
+ match state {
+ CalibrationState::Off => None,
+ CalibrationState::Fitted(report) => Some(json!({
+ "method": report.method.as_str(),
+ "scope": "noul",
+ "ece": {
+ "raw": round6(report.ece_raw),
+ "calibrated": round6(report.ece_calibrated),
+ },
+ "fitted_on": report.fitted_on,
+ "held_out": report.held_out,
+ "labelled_pairs": report.labelled_pairs,
+ })),
+ CalibrationState::Unfit(reason) => Some(json!({
+ "reason": reason,
+ })),
+ }
+}
+
+/// The fit's parameters, published by `/_decide/_calibration`: an isotonic
+/// fit's knots, or the temperature.
+fn fit_params_json(fit: &FittedCalibration) -> Value {
+ match fit {
+ FittedCalibration::Isotonic(iso) => json!({
+ "knots": iso
+ .knots()
+ .iter()
+ .map(|(x, y)| json!([round6(*x), round6(*y)]))
+ .collect::<Vec<_>>(),
+ }),
+ FittedCalibration::Temperature(t) => json!({ "temperature": round6(t.t) }),
+ }
+}
+
// ─────────────────────────────────────────────────────────────────────────────
// DecideSettings — the node's ladder configuration
// ─────────────────────────────────────────────────────────────────────────────
@@ -518,6 +828,11 @@ pub async fn systemone(
if cfg.index.is_empty() && !local_enabled {
return not_configured();
}
+ // One calibration lookup per request (#1063): every tier's noul
+ // probability below reads the same fit, and the answers' strictly-parsed
+ // wire objects stay untouched — p_raw/p_cal ride in `decisions.evidence`,
+ // the extras channel established clients ignore.
+ let calibration = calibration_state(&state, &cfg, false).await;
let questions = match body.get("questions") {
Some(Value::Object(q)) if !q.is_empty() => q,
@@ -632,17 +947,26 @@ pub async fn systemone(
.iter()
.max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
.expect("non-empty");
- evidence.insert(
- id.clone(),
- json!({
- "tier": DecideTier::History.as_str(),
- "source": DecideTier::History.as_str(),
- "label": best_label,
- "support": round6(best_weight / total),
- "neighbours": labelled.len(),
- "found": neighbours.len(),
- }),
- );
+ let mut evidence_value = json!({
+ "tier": DecideTier::History.as_str(),
+ "source": DecideTier::History.as_str(),
+ "label": best_label,
+ "support": round6(best_weight / total),
+ "neighbours": labelled.len(),
+ "found": neighbours.len(),
+ });
+ // The calibrated quantity is the noul's — the positive
+ // label's share — so only a noul question carries p_raw/p_cal
+ // here. A choice's per-option shares are a different
+ // quantity; the decisions.calibration block names the scope.
+ if matches!(question.kind, Kind::Noul) {
+ let p_raw = share(&cfg.positive_label);
+ evidence_value["p_raw"] = json!(round6(p_raw));
+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {
+ evidence_value["p_cal"] = p_cal;
+ }
+ }
+ evidence.insert(id.clone(), evidence_value);
}
Err(failure) => {
match resolve_tier(&failure, local_enabled) {
@@ -753,17 +1077,25 @@ pub async fn systemone(
.zip(&scores)
.max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
.expect("labels are non-empty");
- evidence.insert(
- id.clone(),
- json!({
- "tier": DecideTier::Local.as_str(),
- "source": DecideTier::Local.as_str(),
- "model": LOCAL_MODEL_ID,
- "label": best_label,
- "support": round6(*best_score as f64),
- "hypotheses": Value::Object(by_label),
- }),
- );
+ let mut evidence_value = json!({
+ "tier": DecideTier::Local.as_str(),
+ "source": DecideTier::Local.as_str(),
+ "model": LOCAL_MODEL_ID,
+ "label": best_label,
+ "support": round6(*best_score as f64),
+ "hypotheses": Value::Object(by_label),
+ });
+ // The head's noul is `scores[0]` — the positive hypothesis's
+ // probability — so it calibrates through the same seam as the
+ // vote's (and, when that tier exists, a hosted score's).
+ if matches!(question.kind, Kind::Noul) {
+ let p_raw = scores[0] as f64;
+ evidence_value["p_raw"] = json!(round6(p_raw));
+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {
+ evidence_value["p_cal"] = p_cal;
+ }
+ }
+ evidence.insert(id.clone(), evidence_value);
cached.push(CachedAnswer {
text: question.vote_text.clone(),
label: best_label.clone(),
@@ -806,6 +1138,12 @@ pub async fn systemone(
"took_ms".into(),
json!(started.elapsed().as_millis() as u64),
);
+ // The ECE beside every probability (#1063): present whenever calibration
+ // is configured — fitted (method, ECE, fitted-on) or unfit (reason) —
+ // and absent when the node never promised a p_cal.
+ if let Some(block) = calibration_block(&calibration) {
+ decisions.insert("calibration".into(), block);
+ }
if local_answered > 0 {
decisions.insert("local_model".into(), json!(LOCAL_MODEL_ID));
decisions.insert("local_answered".into(), json!(local_answered));
@@ -898,6 +1236,11 @@ pub async fn decide(
cfg.label_field = defaults.label_field.clone();
cfg.text_field = defaults.text_field.clone();
+ // One calibration lookup per request (#1063), against the index THIS
+ // request named — the fit is cached per index, so the audit surface and
+ // the wire surface read the same fit for the same history.
+ let calibration = calibration_state(&state, &cfg, false).await;
+
let neighbours = if cfg.index.is_empty() {
Err(HistorySupport::NotConfigured)
} else {
@@ -935,6 +1278,16 @@ pub async fn decide(
for (w, label, _) in &neighbours {
*weight_by_label.entry(label.to_string()).or_insert(0.0) += *w;
}
+ // The calibrated quantity (#1063): the positive label's share — the same
+ // quantity a noul answers on the wire — NOT the winner's share that
+ // `confidence` reports. `confidence` keeps its raw meaning and the
+ // abstain gate keeps applying to it: calibration changes what the number
+ // means, not the verdict.
+ let p_raw = if total > 0.0 {
+ weight_by_label.get(&positive).copied().unwrap_or(0.0) / total
+ } else {
+ 0.0
+ };
let (label, confidence, abstain, reason) = match weight_by_label
.iter()
.max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap())
@@ -988,6 +1341,16 @@ pub async fn decide(
"source": DecideTier::History.as_str(),
"took_ms": started.elapsed().as_millis() as u64,
});
+ // p_cal beside p_raw, never replacing it (#1063): p_raw is always the
+ // honest raw share, p_cal exists only when calibration is configured
+ // (null, with the reason in the block, when configured but unfitted),
+ // and the calibration block is the ECE that rides beside both.
+ resp["p_raw"] = json!(round6(p_raw));
+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {
+ resp["p_cal"] = p_cal;
+ }
+ resp["calibration"] =
+ calibration_block(&calibration).unwrap_or(json!({ "configured": "none" }));
if let Some(r) = reason {
resp["reason"] = json!(r);
}
@@ -1026,6 +1389,11 @@ async fn decide_local(
Err(e) => return local_unavailable(&e, local.model_dir()),
};
let scores = &scores[0];
+ // The head's positive-label probability is `scores[0]` — the same
+ // quantity a noul answers, through the same calibration seam as every
+ // other tier's (#1063).
+ let p_raw = scores[0] as f64;
+ let calibration = calibration_state(state, cfg, false).await;
let (label, confidence) = if scores[0] >= scores[1] {
(labels[0].clone(), scores[0] as f64)
} else {
@@ -1062,6 +1430,14 @@ async fn decide_local(
"model": LOCAL_MODEL_ID,
"took_ms": started.elapsed().as_millis() as u64,
});
+ // The audit surface always names its calibration state, and p_cal rides
+ // beside p_raw exactly as on the history tier (#1063).
+ resp["p_raw"] = json!(round6(p_raw));
+ if let Some(p_cal) = p_cal_value(&calibration, p_raw) {
+ resp["p_cal"] = p_cal;
+ }
+ resp["calibration"] =
+ calibration_block(&calibration).unwrap_or(json!({ "configured": "none" }));
if abstain {
resp["reason"] = json!(format!(
"confidence {:.3} below decisions.min_confidence {:.3}",
@@ -1071,6 +1447,104 @@ async fn decide_local(
Json(resp).into_response()
}
+// ─────────────────────────────────────────────────────────────────────────────
+// GET /_decide/_calibration
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// Publish the calibration (#1063): the reliability curve of the labelled
+/// history — binned `p_raw` against empirical frequency, with bin counts —
+/// the ECE raw and calibrated, the fit's parameters, what it was fitted on,
+/// and the date range of the fitted documents. Always refits, so this is the
+/// operator's refresh as well as the publication; the fit it produces becomes
+/// the cached fit every decide request reads.
+///
+/// `?index=` names the history (default: the configured `[decisions] index`).
+/// The curve and raw ECE publish even when calibration is not configured —
+/// the reliability of the raw probabilities is audit information in its own
+/// right — with a `reason` naming the setting that would fit them.
+pub async fn decide_calibration(
+ State(state): State<AppState>,
+ axum::extract::Query(params): axum::extract::Query<HashMap<String, String>>,
+) -> axum::response::Response {
+ let defaults = &state.config.decisions;
+ let index = params
+ .get("index")
+ .map(String::as_str)
+ .map(str::trim)
+ .filter(|s| !s.is_empty())
+ .unwrap_or(defaults.index.as_str())
+ .to_string();
+ if index.is_empty() {
+ return unprocessable(
+ "name an index with ?index=… or set [decisions] index — calibration is fitted \
+ on a labelled history",
+ );
+ }
+ let cfg = DecisionsConfig {
+ index: index.clone(),
+ ..defaults.clone()
+ };
+ let (pairs, ts_range) = match scan_calibration_pairs(&state, &cfg).await {
+ Ok(found) => found,
+ Err(reason) => return index_error(&index, &reason),
+ };
+ let curve_raw = xerj_common::calibration::reliability_curve(
+ &pairs,
+ xerj_common::calibration::CALIBRATION_BINS,
+ );
+ let ece_raw = xerj_common::calibration::ece_from_curve(&curve_raw);
+
+ let mut resp = serde_json::Map::new();
+ resp.insert("index".into(), json!(index));
+ resp.insert("configured".into(), json!(cfg.calibration.as_str()));
+ resp.insert("labelled_pairs".into(), json!(pairs.len()));
+ resp.insert(
+ "bins".into(),
+ json!(xerj_common::calibration::CALIBRATION_BINS),
+ );
+ resp.insert(
+ "curve".into(),
+ json!({ "raw": curve_raw, "calibrated": Value::Null }),
+ );
+ resp.insert(
+ "ece".into(),
+ json!({ "raw": round6(ece_raw), "calibrated": Value::Null }),
+ );
+ if let Some((lo, hi)) = &ts_range {
+ resp.insert("ts_range".into(), json!([lo, hi]));
+ }
+ match fit_pairs(&cfg, &pairs, ts_range) {
+ Ok(report) => {
+ resp.insert("method".into(), json!(report.method.as_str()));
+ resp.insert("scope".into(), json!("noul"));
+ resp.insert("fitted_on".into(), json!(report.fitted_on));
+ resp.insert("held_out".into(), json!(report.held_out));
+ resp.insert("params".into(), fit_params_json(&report.fit));
+ resp.insert(
+ "ece".into(),
+ json!({
+ "raw": round6(report.ece_raw),
+ "calibrated": round6(report.ece_calibrated),
+ }),
+ );
+ resp.insert(
+ "curve".into(),
+ json!({
+ "raw": report.curve_raw,
+ "calibrated": report.curve_calibrated,
+ }),
+ );
+ // The publication is the refresh: every decide request after
+ // this reads the fit just published.
+ state.decide_calibration.refresh_from(&index, report).await;
+ }
+ Err(reason) => {
+ resp.insert("reason".into(), json!(reason));
+ }
+ }
+ Json(Value::Object(resp)).into_response()
+}
+
// ─────────────────────────────────────────────────────────────────────────────
// The vote
// ─────────────────────────────────────────────────────────────────────────────
@@ -1657,4 +2131,124 @@ mod tests {
assert_eq!(DecideTier::Local.as_str(), "local");
assert_eq!(DecideTier::Hosted.as_str(), "hosted");
}
+
+ // ─────────────────────────────────────────────────────────────────────────
+ // Calibration (#1063)
+ // ─────────────────────────────────────────────────────────────────────────
+
+ /// The fit's data contract: a labelled outcome document is a pair, the
+ /// positive label's probability against the outcome; a flywheel answer
+ /// (`source`) is NOT, because its label is the model's own prediction —
+ /// calibrating on predictions would learn the model is right by
+ /// construction.
+ #[test]
+ fn calibration_pairs_come_from_outcomes_not_predictions() {
+ let cfg = DecisionsConfig::default(); // label_field "label", positive "true"
+ // An outcome the system got wrong: it said 0.9, the truth is "false".
+ // The positive-label probability is 1 − 0.9, the outcome 0.
+ let doc = json!({"text": "…", "label": "false", "p": 0.9});
+ let pair = pair_from_doc(&doc, &cfg).expect("outcome pair");
+ assert!(
+ (pair.p - 0.1).abs() < 1e-12,
+ "p_pos = 1 − p when label ≠ positive"
+ );
+ assert_eq!(pair.y, 0.0);
+ // An outcome it got right: p of the positive label, outcome 1.
+ let doc = json!({"text": "…", "label": "true", "p": 0.8});
+ let pair = pair_from_doc(&doc, &cfg).expect("outcome pair");
+ assert!((pair.p - 0.8).abs() < 1e-12);
+ assert_eq!(pair.y, 1.0);
+ // A human correction carrying the p it corrects: still an outcome,
+ // and the best kind — truth plus the number that was wrong.
+ let doc = json!({"text": "…", "label": "true", "p": 0.3, "human": true});
+ assert!(pair_from_doc(&doc, &cfg).is_some());
+ // A flywheel answer: a prediction, never a pair.
+ let doc = json!({"text": "…", "label": "true", "p": 0.8, "source": "local", "ts": "…"});
+ assert!(
+ pair_from_doc(&doc, &cfg).is_none(),
+ "a `source` document is the model's own answer"
+ );
+ // No p, no pair — a seeded example has a truth but no raw
+ // probability to calibrate.
+ assert!(pair_from_doc(&json!({"text": "…", "label": "true"}), &cfg).is_none());
+ // No label, no pair.
+ assert!(pair_from_doc(&json!({"text": "…", "p": 0.8}), &cfg).is_none());
+ // A p that is not a probability is not calibration data.
+ assert!(pair_from_doc(&json!({"label": "true", "p": 1.4}), &cfg).is_none());
+ assert!(pair_from_doc(&json!({"label": "true", "p": -0.1}), &cfg).is_none());
+ assert!(pair_from_doc(&json!({"label": "true"}), &cfg).is_none());
+ }
+
+ /// The three states on the wire: off ships no p_cal at all, unfit ships
+ /// null with the reason published in the block, fitted ships the fit's
+ /// value — and never a copy of p_raw pretending to be calibrated.
+ #[test]
+ fn p_cal_is_absent_null_or_calibrated_never_a_copy() {
+ // Off: nothing.
+ assert!(p_cal_value(&CalibrationState::Off, 0.7).is_none());
+ assert!(calibration_block(&CalibrationState::Off).is_none());
+ // Unfit: null, with the reason riding in the block.
+ let unfit = CalibrationState::Unfit("only 3 labelled pairs".into());
+ assert_eq!(p_cal_value(&unfit, 0.7), Some(Value::Null));
+ let block = calibration_block(&unfit).expect("block");
+ assert_eq!(block["reason"], "only 3 labelled pairs");
+ // Fitted: the fit's value and its ECE, not p_raw echoed back. The
+ // identity fit (one knot per distinct p, already monotone) would echo
+ // — so use a fit that learned the opposite of the raw number.
+ let report = Arc::new(
+ xerj_common::calibration::fit_calibration(
+ xerj_common::calibration::CalibrationMethod::Isotonic,
+ &[
+ CalibrationPair { p: 0.9, y: 0.0 },
+ CalibrationPair { p: 0.9, y: 0.0 },
+ CalibrationPair { p: 0.9, y: 0.0 },
+ CalibrationPair { p: 0.9, y: 0.0 },
+ CalibrationPair { p: 0.9, y: 0.0 },
+ CalibrationPair { p: 0.2, y: 0.0 },
+ CalibrationPair { p: 0.2, y: 0.0 },
+ CalibrationPair { p: 0.2, y: 0.0 },
+ CalibrationPair { p: 0.2, y: 0.0 },
+ CalibrationPair { p: 0.2, y: 0.0 },
+ ],
+ )
+ .expect("fit"),
+ );
+ let fitted = CalibrationState::Fitted(report);
+ let p_cal = p_cal_value(&fitted, 0.9).expect("a value");
+ let p_cal = p_cal.as_f64().expect("a number");
+ assert!(
+ (p_cal - 0.9).abs() > 1e-9,
+ "a fit that learned 0.9 means 0 must not echo 0.9 back (got {p_cal})"
+ );
+ assert!(
+ (p_cal - 0.0).abs() < 1e-9,
+ "five negatives at 0.9 → p_cal 0"
+ );
+ let block = calibration_block(&fitted).expect("block");
+ assert_eq!(block["method"], "isotonic");
+ assert_eq!(block["scope"], "noul");
+ assert!(
+ block["ece"]["calibrated"].is_number(),
+ "the ECE rides beside: {block}"
+ );
+ assert!(block["ece"]["raw"].is_number());
+ assert_eq!(block["fitted_on"], 8);
+ assert_eq!(block["held_out"], 2);
+ }
+
+ /// The unfit reason names what a fit needs and how to record it — an
+ /// operator who sees `p_cal: null` can act on the string beside it.
+ #[test]
+ fn the_unfit_reason_names_the_recording_path() {
+ let reason = unfit_reason(&DecisionsConfig::default(), 3);
+ assert!(reason.contains("3 labelled calibration pairs"), "{reason}");
+ assert!(
+ reason.contains("10"),
+ "the minimum is in the message: {reason}"
+ );
+ assert!(
+ reason.contains("source"),
+ "the flywheel exclusion is explained: {reason}"
+ );
+ }
}
diff --git a/engine/crates/xerj-api/tests/decide_calibration.rs b/engine/crates/xerj-api/tests/decide_calibration.rs
new file mode 100644
index 000000000..9255944d3
--- /dev/null
+++ b/engine/crates/xerj-api/tests/decide_calibration.rs
@@ -0,0 +1,508 @@
+//! End-to-end checks for the decide surface's calibration layer (issue
+//! #1063): `p_cal` beside every `p_raw`, the fit's ECE riding beside both,
+//! `GET /_decide/_calibration` publishing the reliability curve, and the
+//! honesty rules — no `p_cal` at all when calibration is not configured,
+//! `null` with a published reason when configured but unfitted, and never a
+//! copy of `p_raw` pretending to be calibrated.
+//!
+//! The history is built so the arithmetic is exact and the answer visibly
+//! moves: twenty outcome documents whose raw positive-label probability is
+//! 0.25, half of them true — isotonic pools the one distinct raw probability
+//! into a single knot at the base rate 0.5, so every `p_cal` is 0.5 whatever
+//! the vote's share was. (0.25/0.75 are the exactly-representable pair:
+//! 1 − 0.75 = 0.25 in f64, where 1 − 0.8 = 0.19999999999999996 — two raw
+//! probabilities, no pooling. Ask how the first run of this file failed.) The strictly-parsed `/v1/systemone`
+//! answer objects stay untouched (the wire-compatibility #1072 established):
+//! `p_raw`/`p_cal` ride in `decisions.evidence`, never in the answer.
+
+use axum::body::Body;
+use axum::http::{Request, StatusCode};
+use serde_json::{json, Value};
+use tower::ServiceExt;
+
+use xerj_api::state::AppState;
+use xerj_common::calibration::CalibrationMethod;
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The node
+// ─────────────────────────────────────────────────────────────────────────────
+
+struct Node {
+ native: axum::Router,
+ app: axum::Router,
+ _dir: tempfile::TempDir,
+}
+
+/// A node whose `[decisions]` block names `index`, with the calibration
+/// method overrideable — the layer's one new setting.
+async fn node_with(index: &str, calibration: CalibrationMethod) -> Node {
+ let dir = tempfile::tempdir().expect("tempdir");
+ let mut config = xerj_common::config::Config::default();
+ config.server.data_dir = dir.path().to_string_lossy().into_owned();
+ config.storage.wal_sync = xerj_common::config::WalSync::Async;
+ config.decisions.index = index.to_string();
+ config.decisions.calibration = calibration;
+ let metrics = xerj_common::metrics::Metrics::new().expect("metrics");
+ let engine = xerj_engine::Engine::new(config.clone()).expect("engine");
+ let state = AppState::new(config, engine, metrics);
+ Node {
+ native: xerj_api::router::build_native_router(state.clone()),
+ app: xerj_api::router::build_es_compat_router(state),
+ _dir: dir,
+ }
+}
+
+impl Node {
+ async fn call(
+ &self,
+ router: &axum::Router,
+ method: &str,
+ path: &str,
+ body: Value,
+ ) -> (StatusCode, Value) {
+ let response = router
+ .clone()
+ .oneshot(
+ Request::builder()
+ .method(method)
+ .uri(path)
+ .header("content-type", "application/json")
+ .body(Body::from(body.to_string()))
+ .expect("request"),
+ )
+ .await
+ .expect("response");
+ let status = response.status();
+ let bytes = axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .expect("body");
+ (
+ status,
+ serde_json::from_str(&String::from_utf8_lossy(&bytes)).unwrap_or(Value::Null),
+ )
+ }
+
+ async fn systemone(&self, body: Value) -> (StatusCode, Value) {
+ self.call(&self.native.clone(), "POST", "/v1/systemone", body)
+ .await
+ }
+ async fn decide(&self, body: Value) -> (StatusCode, Value) {
+ self.call(&self.app.clone(), "POST", "/_decide", body).await
+ }
+ async fn calibration(&self, query: &str) -> (StatusCode, Value) {
+ self.call(&self.app.clone(), "GET", query, json!({})).await
+ }
+ async fn put_doc(&self, index: &str, id: &str, doc: Value) {
+ let (st, b) = self
+ .call(
+ &self.app.clone(),
+ "PUT",
+ &format!("/{index}/_doc/{id}"),
+ doc,
+ )
+ .await;
+ assert!(st.is_success(), "index {index}/{id}: {st} {b}");
+ }
+ async fn create_index(&self, index: &str) {
+ let (st, b) = self
+ .call(
+ &self.app.clone(),
+ "PUT",
+ &format!("/{index}"),
+ json!({"mappings": {"properties": {
+ "text": {"type": "text"},
+ "label": {"type": "keyword"}
+ }}}),
+ )
+ .await;
+ assert!(st.is_success(), "create {index}: {st} {b}");
+ }
+ async fn refresh(&self, index: &str) {
+ let (st, b) = self
+ .call(
+ &self.app.clone(),
+ "POST",
+ &format!("/{index}/_refresh"),
+ json!({}),
+ )
+ .await;
+ assert!(st.is_success(), "refresh {index}: {st} {b}");
+ }
+}
+
+/// Seed the outcome history: `per_side` `refund` texts labelled positive
+/// (`true`, `p: 0.25`) and `per_side` `outage` texts labelled negative
+/// (`false`, `p: 0.75`). Each document's `p` is the probability the system
+/// served FOR ITS OWN LABEL, so both sides convert to the same positive-label
+/// probability: the positives are (0.25, 1), the negatives (1 − 0.75 = 0.25,
+/// 0) — exactly, because 0.75 and 0.25 are exactly representable (1 − 0.8 is
+/// not, and the two-tenths the first draft used landed on two DIFFERENT raw
+/// probabilities: 0.19999999999999996 and 0.2). Every pair therefore sits at
+/// raw 0.25 against a 0.5 base rate — isotonic pools the one distinct raw
+/// probability into a single knot at 0.5, so every `p_cal` on this history is
+/// exactly 0.5, visibly not a copy of any vote share. The outage texts share
+/// no query term with the refund texts, so a refund-shaped question retrieves
+/// only refund documents and votes 1.0.
+async fn seed_outcomes(node: &Node, index: &str, per_side: usize) {
+ node.create_index(index).await;
+ for i in 0..per_side {
+ node.put_doc(
+ index,
+ &format!("refund-{i}"),
+ json!({
+ "text": format!("refund request number {i} for the annual subscription"),
+ "label": "true",
+ "p": 0.25,
+ "ts": "2026-09-28T10:00:00.000Z",
+ }),
+ )
+ .await;
+ node.put_doc(
+ index,
+ &format!("outage-{i}"),
+ json!({
+ "text": format!("outage report number {i} for the weekend window"),
+ "label": "false",
+ "p": 0.75,
+ "ts": "2026-09-29T10:00:00.000Z",
+ }),
+ )
+ .await;
+ }
+ node.refresh(index).await;
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// /_decide
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The core contract: `p_raw` beside `p_cal`, the calibration block naming
+/// the method and its ECE, and `p_cal` moved by the fit — 0.5 on this
+/// history, whatever the vote's share was — never a copy of `p_raw`.
+#[tokio::test]
+async fn decide_ships_p_cal_beside_p_raw_with_the_fit_ece() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription", "k": 10}))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ // The vote: all ten neighbours are `refund` docs, so the positive share
+ // is 1.0 — a confident raw probability.
+ assert_eq!(body["p_raw"], 1.0, "p_raw is the positive label's share");
+ assert_eq!(
+ body["p_cal"], 0.5,
+ "the fit pooled the one distinct raw probability to the base rate: {body}"
+ );
+ assert_eq!(body["confidence"], 1.0, "confidence keeps its raw meaning");
+ let cal = &body["calibration"];
+ assert_eq!(cal["method"], "isotonic", "{cal}");
+ assert_eq!(cal["scope"], "noul");
+ assert!(cal["ece"]["raw"].is_number(), "the ECE rides beside: {cal}");
+ assert!(cal["ece"]["calibrated"].is_number(), "{cal}");
+ assert_eq!(cal["labelled_pairs"], 20);
+ assert_eq!(cal["fitted_on"], 16, "the deterministic 80/20: {cal}");
+ assert_eq!(cal["held_out"], 4);
+}
+
+/// An uncalibrated node ships `p_raw` alone: no `p_cal` field at all, and a
+/// calibration block that says so — never a p_cal copying p_raw.
+#[tokio::test]
+async fn without_calibration_there_is_no_p_cal_at_all() {
+ let node = node_with("judgements", CalibrationMethod::None).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert_eq!(body["p_raw"], 1.0);
+ assert!(
+ body.get("p_cal").is_none(),
+ "no calibration configured → no p_cal field: {body}"
+ );
+ assert_eq!(body["calibration"], json!({"configured": "none"}));
+}
+
+/// Configured but unfitted: `p_cal` is `null`, and the reason rides beside
+/// it in the calibration block — the operator can act on the string.
+#[tokio::test]
+async fn a_configured_but_unfitted_node_ships_null_with_the_reason() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ seed_outcomes(&node, "judgements", 2).await; // 4 pairs < the 10-pair minimum
+
+ let (st, body) = node
+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert!(
+ body["p_raw"].is_number(),
+ "p_raw ships whatever the four-document vote said: {body}"
+ );
+ assert!(
+ body["p_cal"].is_null(),
+ "configured but unfitted → null, not a copy: {body}"
+ );
+ let reason = body["calibration"]["reason"]
+ .as_str()
+ .expect("the reason is published beside the null");
+ assert!(reason.contains("4 labelled calibration pairs"), "{reason}");
+ assert!(
+ reason.contains("source"),
+ "the flywheel exclusion is explained: {reason}"
+ );
+}
+
+/// Temperature calibrates the same quantity through the same seam. On this
+/// history every fit pair sits at p 0.2 with a 0.5 base rate, so the NLL
+/// optimum is "as flat as the bracket allows": T pins at its ceiling (16) and
+/// the saturated p_raw 1.0 pulls most of the way back toward the base rate —
+/// σ(logit(1)/16) ≈ 0.703. Rank preserved, confidence honestly deflated,
+/// never all the way to a pretend 0.5.
+#[tokio::test]
+async fn temperature_calibrates_the_same_quantity() {
+ let node = node_with("judgements", CalibrationMethod::Temperature).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .decide(json!({"index": "judgements", "question": "refund request for the annual subscription"}))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert_eq!(body["p_raw"], 1.0);
+ let p_cal = body["p_cal"].as_f64().expect("a number");
+ assert!(
+ (p_cal - 0.703_386).abs() < 0.005,
+ "T pinned at its ceiling pulls σ(logit(1)/16) back toward the base rate: {p_cal}"
+ );
+ assert!(
+ p_cal < body["p_raw"].as_f64().expect("p_raw"),
+ "the saturated raw probability deflates: {p_cal}"
+ );
+ assert_eq!(body["calibration"]["method"], "temperature");
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// /v1/systemone — the strictly-parsed wire stays untouched
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// `p_raw`/`p_cal` ride in `decisions.evidence` (the extras channel), the
+/// answer object keeps exactly its documented fields, a noul calibrates, and
+/// a choice's per-option shares — a different quantity — do not.
+#[tokio::test]
+async fn systemone_calibrates_nouls_in_evidence_and_leaves_the_wire_alone() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .systemone(json!({
+ "state": {"documents": {
+ "doc_0": "refund request for the annual subscription",
+ "doc_1": "outage report for the weekend window"
+ }},
+ "questions": {
+ "n": {"type": "noul", "instructions": "Judge `documents.doc_0` for a refund request."},
+ "c": {"type": "choice", "instructions": "Route `documents.doc_1`.", "criteria": {"true": {}, "false": {}}}
+ }
+ }))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+
+ // The wire objects: exactly the documented fields, no p_raw/p_cal inside.
+ let noul = &body["answers"]["n"];
+ assert_eq!(
+ noul.as_object().map(|o| o.len()),
+ Some(2),
+ "type + noul only: {noul}"
+ );
+ let choice = &body["answers"]["c"];
+ assert!(
+ choice.get("p_raw").is_none() && choice.get("p_cal").is_none(),
+ "the answer object is strictly parsed and stays untouched: {choice}"
+ );
+
+ // The evidence: the noul calibrates, the choice does not.
+ let evidence = &body["decisions"]["evidence"];
+ assert_eq!(evidence["n"]["p_raw"], 1.0, "the noul's p_raw: {evidence}");
+ assert_eq!(
+ evidence["n"]["p_cal"], 0.5,
+ "the pooled base rate: {evidence}"
+ );
+ assert!(
+ evidence["c"].get("p_raw").is_none() && evidence["c"].get("p_cal").is_none(),
+ "a choice's per-option shares are a different quantity, out of this fit's scope: {evidence}"
+ );
+
+ // The block with the ECE rides at the decisions level.
+ let cal = &body["decisions"]["calibration"];
+ assert_eq!(cal["method"], "isotonic");
+ assert_eq!(cal["scope"], "noul");
+ assert!(cal["ece"]["calibrated"].is_number(), "{cal}");
+ assert_eq!(cal["labelled_pairs"], 20);
+}
+
+/// An uncalibrated wire node carries no calibration block at all and no
+/// p_cal in any evidence.
+#[tokio::test]
+async fn systemone_without_calibration_carries_no_block() {
+ let node = node_with("judgements", CalibrationMethod::None).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .systemone(json!({
+ "state": "refund request for the annual subscription",
+ "questions": {"n": {"type": "noul", "instructions": "Is this a refund request?"}}
+ }))
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert!(
+ body["decisions"].get("calibration").is_none(),
+ "no calibration configured → no block: {body}"
+ );
+ assert_eq!(body["decisions"]["evidence"]["n"]["p_raw"], 1.0);
+ assert!(
+ body["decisions"]["evidence"]["n"].get("p_cal").is_none(),
+ "and no p_cal: {body}"
+ );
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// GET /_decide/_calibration
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The publication: the reliability curve (binned p_raw against empirical
+/// frequency, with bin counts), the ECE raw and calibrated, the fit's
+/// parameters, and what it was fitted on. On this history every pair sits at
+/// raw 0.2, so the raw curve is one bin of 20 at rate 0.5.
+#[tokio::test]
+async fn calibration_publishes_the_reliability_curve() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .calibration("/_decide/_calibration?index=judgements")
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert_eq!(body["index"], "judgements");
+ assert_eq!(body["configured"], "isotonic");
+ assert_eq!(body["method"], "isotonic");
+ assert_eq!(body["scope"], "noul");
+ assert_eq!(body["labelled_pairs"], 20);
+ assert_eq!(body["fitted_on"], 16);
+ assert_eq!(body["held_out"], 4);
+ assert_eq!(body["bins"], 10);
+
+ let raw_curve = body["curve"]["raw"].as_array().expect("the raw curve");
+ assert_eq!(
+ raw_curve.len(),
+ 1,
+ "one occupied bin (all pairs at 0.25): {raw_curve:?}"
+ );
+ let bin = &raw_curve[0];
+ assert_eq!(bin["n"], 20, "bin counts publish");
+ assert_eq!(bin["lo"], 0.2);
+ assert_eq!(bin["hi"], 0.3); // the bin edge: (b+1)/10 in f64
+ assert!((bin["mean_p"].as_f64().unwrap() - 0.25).abs() < 1e-12);
+ assert!((bin["positive_rate"].as_f64().unwrap() - 0.5).abs() < 1e-12);
+ assert!(
+ body["curve"]["calibrated"].is_array(),
+ "the calibrated curve publishes too: {body}"
+ );
+ assert!(body["ece"]["raw"].is_number() && body["ece"]["calibrated"].is_number());
+ let knots = body["params"]["knots"].as_array().expect("the fit's knots");
+ assert_eq!(
+ knots.len(),
+ 1,
+ "one distinct raw probability → one knot: {knots:?}"
+ );
+ assert_eq!(knots[0][0], 0.25);
+ assert_eq!(knots[0][1], 0.5);
+ assert_eq!(
+ body["ts_range"],
+ json!(["2026-09-28T10:00:00.000Z", "2026-09-29T10:00:00.000Z"]),
+ "the fitted documents' date range publishes"
+ );
+}
+
+/// The endpoint also publishes WITHOUT a configured calibration — the raw
+/// reliability of the history is audit information in its own right — with a
+/// reason naming the setting that would fit it.
+#[tokio::test]
+async fn calibration_publishes_the_raw_curve_even_when_not_configured() {
+ let node = node_with("judgements", CalibrationMethod::None).await;
+ seed_outcomes(&node, "judgements", 10).await;
+
+ let (st, body) = node
+ .calibration("/_decide/_calibration?index=judgements")
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert_eq!(body["configured"], "none");
+ assert_eq!(body["labelled_pairs"], 20);
+ assert!(
+ body["curve"]["raw"].is_array(),
+ "the raw curve still publishes: {body}"
+ );
+ assert!(body["ece"]["raw"].is_number(), "{body}");
+ let reason = body["reason"]
+ .as_str()
+ .expect("the reason names the setting");
+ assert!(reason.contains("calibration"), "{reason}");
+}
+
+/// Cached flywheel answers (`source`) are predictions, not outcomes: they do
+/// not feed the fit, whatever else they do to the vote.
+#[tokio::test]
+async fn cached_answers_do_not_feed_the_fit() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ seed_outcomes(&node, "judgements", 10).await;
+ // Five cached local-tier answers — confident, and self-consistent by
+ // construction. If they fed the fit they would drag it off the recorded
+ // outcomes' base rate.
+ for i in 0..5 {
+ node.put_doc(
+ "judgements",
+ &format!("cached-{i}"),
+ json!({
+ "text": format!("cached answer number {i}"),
+ "label": "true",
+ "p": 0.99,
+ "source": "local",
+ "ts": "2026-09-30T10:00:00.000Z",
+ }),
+ )
+ .await;
+ }
+ node.refresh("judgements").await;
+
+ let (st, body) = node
+ .calibration("/_decide/_calibration?index=judgements")
+ .await;
+ assert!(st.is_success(), "{st} {body}");
+ assert_eq!(
+ body["labelled_pairs"], 20,
+ "the five `source` documents are predictions, not outcomes: {body}"
+ );
+ assert_eq!(body["fitted_on"], 16);
+}
+
+/// No index to read: the endpoint refuses naming what it needs, instead of
+/// publishing an empty curve as if it were a measurement.
+#[tokio::test]
+async fn calibration_with_no_index_refuses() {
+ // A node with no [decisions] index and nothing named per request.
+ let node = node_with("", CalibrationMethod::Isotonic).await;
+ let (st, body) = node.calibration("/_decide/_calibration").await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{st} {body}");
+ let reason = body["error"]["reason"].as_str().expect("the refusal");
+ assert!(reason.contains("index"), "{reason}");
+}
+
+/// A named index that does not exist is the vote's own error shape — the
+/// calibration surface never invents a fit from an unreadable history.
+#[tokio::test]
+async fn calibration_on_a_missing_index_errors() {
+ let node = node_with("judgements", CalibrationMethod::Isotonic).await;
+ let (st, body) = node.calibration("/_decide/_calibration?index=nope").await;
+ assert_eq!(st, StatusCode::UNPROCESSABLE_ENTITY, "{st} {body}");
+ assert_eq!(body["error"]["type"], "history_index_error", "{body}");
+}
diff --git a/engine/crates/xerj-common/src/calibration.rs b/engine/crates/xerj-common/src/calibration.rs
new file mode 100644
index 000000000..fbbe95ec7
--- /dev/null
+++ b/engine/crates/xerj-common/src/calibration.rs
@@ -0,0 +1,834 @@
+//! Probability calibration for the decide surface — isotonic (PAVA) and
+//! temperature (Platt) scaling, the reliability curve, and ECE (#1063).
+//!
+//! A probability from the decide ladder — the history vote's share, the local
+//! head's softmax, and (when that tier exists) a hosted passthrough score — is
+//! a *ranking* signal before it is an *odds*. The FiQA rerank measurement made
+//! that concrete: provider noul probabilities with mean confidence 0.93 on a
+//! bin whose actual relevance rate was 0.34, ECE 0.3109
+//! (`benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa`). Ordering
+//! worth +0.126 nDCG; a number that must never gate. This module is the
+//! correction layer: one calibrated probability (`p_cal`) beside every raw one
+//! (`p_raw`), never replacing it, and one ECE implementation reused by every
+//! path that ships a probability.
+//!
+//! # The two methods, and their citations
+//!
+//! - **Isotonic** — non-decreasing regression of the empirical positive rate
+//! on the raw probability, by the pool-adjacent-violators algorithm (PAVA:
+//! Barlow, Bartholomew, Bremner & Brunk, *Statistical Inference under Order
+//! Restrictions*, Wiley 1972; the stack-of-blocks formulation is de Leeuw,
+//! Hornik & Mair, "Isotonic Optimization in R: Pool-Adjacent-Violators
+//! (PAVA) and Active Set Methods", JSS 2009). Its use for classifier scores
+//! is Zadrozny & Elkan, "Transforming classifier scores into accurate
+//! multiclass probability estimates", KDD 2002. Fitted values are joined by
+//! linear interpolation and clamped at the ends (constant extrapolation) —
+//! the same out-of-sample behaviour as scikit-learn's `IsotonicRegression`.
+//! - **Temperature** — one scalar on the log-odds, `p' = σ(logit(p)/T)`,
+//! fitted by minimising held-out negative log-likelihood: Platt,
+//! "Probabilistic Outputs for Support Vector Machines", 1999 (the
+//! two-parameter original; the one-parameter scaling on the model's own
+//! log-odds is the form Guo, Pleiss, Sun & Weinberger, "On Calibration of
+//! Modern Neural Networks", ICML 2017, validate). Temperature is
+//! rank-preserving by construction: `σ(logit(·)/T)` is strictly increasing
+//! for any `T > 0`.
+//!
+//! # ECE
+//!
+//! Expected calibration error, the statistic the FiQA baseline is quoted in:
+//! `ECE = Σ_b (n_b/N)·|acc_b − conf_b|` over equal-width bins of the
+//! probability range (Naeini, Cooper & Hausknecht, "Obtaining Well Calibrated
+//! Probabilities Using Bayesian Binning", AAAI 2015). Implemented once, in
+//! [`ece_from_curve`], and reused for raw and calibrated probabilities alike.
+//!
+//! # Determinism (#940's lesson)
+//!
+//! Every fit here is a pure function of the pair multiset: no clock, no RNG,
+//! no map-iteration order. The held-out split is a fixed rule — pairs
+//! stable-sorted by `(p, y)`, every fifth held out — so the same history
+//! produces bit-identical `p_cal` on every request, and the published
+//! reliability curve is reproducible. Nothing in this module reads the
+//! environment, the filesystem, or the system clock.
+
+use serde::{Deserialize, Serialize};
+
+/// Equal-width bins of the probability range used for reliability curves and
+/// ECE — the count the published FiQA curve uses (10).
+pub const CALIBRATION_BINS: usize = 10;
+
+/// One labelled pair: a raw probability `p ∈ [0,1]` and its outcome `y ∈
+/// {0,1}` (stored as `f64` so weighted/aggregate points — a bin's mean
+/// confidence and positive rate — go through the same arithmetic).
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct CalibrationPair {
+ pub p: f64,
+ pub y: f64,
+}
+
+impl CalibrationPair {
+ pub fn new(p: f64, y: bool) -> Self {
+ Self {
+ p,
+ y: if y { 1.0 } else { 0.0 },
+ }
+ }
+}
+
+/// One bin of the reliability curve: the raw-probability interval `[lo, hi)`,
+/// how many pairs fell in it, their mean raw probability (`mean_p`), and their
+/// empirical positive rate (`positive_rate`). Serialize is derived so the
+/// curve publishes verbatim.
+#[derive(Debug, Clone, Copy, PartialEq, Serialize)]
+pub struct ReliabilityBin {
+ /// Inclusive lower edge of the bin.
+ pub lo: f64,
+ /// Exclusive upper edge; the last bin's edge is inclusive of 1.0.
+ pub hi: f64,
+ /// Pairs in this bin.
+ pub n: usize,
+ /// Mean raw (or, where the caller calibrated first, calibrated)
+ /// probability of those pairs.
+ pub mean_p: f64,
+ /// Fraction of those pairs whose outcome was positive.
+ pub positive_rate: f64,
+}
+
+/// Bin the pairs into `bins` equal-width bins of `[0,1]` and return the
+/// non-empty ones, in bin order. Pure: the bin of a pair is a function of its
+/// `p` alone (`min(floor(p·bins), bins−1)`), so the curve is deterministic
+/// for a pair multiset regardless of input order.
+pub fn reliability_curve(pairs: &[CalibrationPair], bins: usize) -> Vec<ReliabilityBin> {
+ debug_assert!(bins > 0);
+ let bins = bins.max(1);
+ // (count, Σp, Σy) per bin — usize counts, f64 sums, no allocation for
+ // empty bins beyond the three slots.
+ let mut count = vec![0usize; bins];
+ let mut sum_p = vec![0f64; bins];
+ let mut sum_y = vec![0f64; bins];
+ for pair in pairs {
+ // A p outside [0,1] is the caller's bug, not a bin; clamp it into the
+ // range rather than panicking on the index math below.
+ let p = pair.p.clamp(0.0, 1.0);
+ let b = ((p * bins as f64) as usize).min(bins - 1);
+ count[b] += 1;
+ sum_p[b] += p;
+ sum_y[b] += pair.y;
+ }
+ (0..bins)
+ .filter(|&b| count[b] > 0)
+ .map(|b| {
+ let n = count[b];
+ ReliabilityBin {
+ lo: b as f64 / bins as f64,
+ hi: (b + 1) as f64 / bins as f64,
+ n,
+ mean_p: sum_p[b] / n as f64,
+ positive_rate: sum_y[b] / n as f64,
+ }
+ })
+ .collect()
+}
+
+/// ECE from a reliability curve: `Σ_b (n_b/N)·|positive_rate_b − mean_p_b|`.
+/// The one implementation every probability-shipping path reuses — raw and
+/// calibrated curves, the endpoint publication, and the tests that pin the
+/// FiQA arithmetic. An empty curve has no probabilities to be wrong about: 0.
+pub fn ece_from_curve(curve: &[ReliabilityBin]) -> f64 {
+ let total: usize = curve.iter().map(|b| b.n).sum();
+ if total == 0 {
+ return 0.0;
+ }
+ curve
+ .iter()
+ .map(|b| (b.n as f64 / total as f64) * (b.positive_rate - b.mean_p).abs())
+ .sum()
+}
+
+/// ECE of raw probabilities against their outcomes, binned — convenience over
+/// [`reliability_curve`] + [`ece_from_curve`].
+pub fn ece(pairs: &[CalibrationPair], bins: usize) -> f64 {
+ ece_from_curve(&reliability_curve(pairs, bins))
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Isotonic — PAVA
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// An isotonic calibration fitted by PAVA: non-decreasing knots
+/// `(x = p_raw, y = fitted probability)`, applied by linear interpolation
+/// between neighbours and clamped at the ends.
+///
+/// Monotone by construction — the fit cannot invent a region where a higher
+/// raw probability means a lower calibrated one — and it is the method that
+/// meets the FiQA held-out gate (see
+/// `benchmarks/decisions-calibration/`); temperature is the cheaper,
+/// one-parameter alternative offered for the same job.
+#[derive(Debug, Clone, PartialEq)]
+pub struct IsotonicCalibration {
+ /// Strictly increasing raw probabilities (tied `p` pooled into one
+ /// weighted point at fit time).
+ xs: Vec<f64>,
+ /// The PAVA-fitted value at each `x` — non-decreasing.
+ ys: Vec<f64>,
+}
+
+impl IsotonicCalibration {
+ /// Fit on the pairs. Pure: ties in `p` are pooled by weighted mean and
+ /// the PAVA pass is a deterministic stack algorithm, so the fit depends
+ /// only on the pair multiset.
+ pub fn fit(pairs: &[CalibrationPair]) -> Self {
+ // Pool tied p: isotonic regression estimates a FUNCTION of p, so all
+ // pairs at one p must share one fitted value — the weighted mean of
+ // their outcomes. (A pair at the same p with outcomes 0 and 1 is not
+ // a violation to pool; it is one point at the base rate.)
+ let mut pooled: Vec<(f64 /*x*/, f64 /*Σy*/, f64 /*n*/)> = Vec::new();
+ let mut sorted: Vec<CalibrationPair> = pairs.to_vec();
+ sorted.sort_by(|a, b| {
+ a.p.partial_cmp(&b.p)
+ .unwrap()
+ .then(a.y.partial_cmp(&b.y).unwrap())
+ });
+ for pair in sorted {
+ match pooled.last_mut() {
+ Some((x, sum_y, n)) if *x == pair.p => {
+ *sum_y += pair.y;
+ *n += 1.0;
+ }
+ _ => pooled.push((pair.p, pair.y, 1.0)),
+ }
+ }
+ // PAVA (de Leeuw–Hornik–Mair's stack of blocks): walk the pooled
+ // points in x order; while the previous block's weighted mean exceeds
+ // the current one's, pool them. Each block is (Σy, n) over a
+ // contiguous run of points; its mean is the fitted value for every
+ // point in it.
+ let mut blocks: Vec<(f64, f64, usize /*first pooled index*/)> = Vec::new();
+ for (i, (_, sum_y, n)) in pooled.iter().enumerate() {
+ blocks.push((*sum_y, *n, i));
+ while blocks.len() >= 2 {
+ let (prev, cur) = (blocks.len() - 2, blocks.len() - 1);
+ let prev_mean = blocks[prev].0 / blocks[prev].1;
+ let cur_mean = blocks[cur].0 / blocks[cur].1;
+ if prev_mean > cur_mean {
+ let (sy, sn, _) = blocks.pop().expect("cur block");
+ let b = blocks.last_mut().expect("prev block");
+ b.0 += sy;
+ b.1 += sn;
+ } else {
+ break;
+ }
+ }
+ }
+ // Expand blocks back to pooled points: every point in a block carries
+ // the block's mean, and those (x, mean) pairs are the interpolation
+ // knots.
+ let mut ys = vec![0f64; pooled.len()];
+ for (bi, (sum_y, n, first)) in blocks.iter().enumerate() {
+ let mean = sum_y / n;
+ let last = blocks.get(bi + 1).map(|b| b.2).unwrap_or(pooled.len());
+ for y in &mut ys[*first..last] {
+ *y = mean;
+ }
+ }
+ Self {
+ xs: pooled.iter().map(|(x, _, _)| *x).collect(),
+ ys,
+ }
+ }
+
+ /// The fitted knots, `(p_raw, p_cal)` — the parameters the
+ /// `/_decide/_calibration` endpoint publishes.
+ pub fn knots(&self) -> Vec<(f64, f64)> {
+ self.xs
+ .iter()
+ .copied()
+ .zip(self.ys.iter().copied())
+ .collect()
+ }
+
+ /// Linear interpolation between knots; constant (clamped) outside their
+ /// range. Monotone non-decreasing for any input, including inputs the fit
+ /// never saw above or below its support.
+ pub fn apply(&self, p: f64) -> f64 {
+ match self.xs.first() {
+ None => return p.clamp(0.0, 1.0), // empty fit: identity, caller gates on fit size
+ Some(&first_x) if p <= first_x => return self.ys[0],
+ _ => {}
+ }
+ if p >= *self.xs.last().expect("non-empty") {
+ return *self.ys.last().expect("non-empty");
+ }
+ // Binary search for the knot interval containing p.
+ let i = self.xs.partition_point(|&x| x < p);
+ let (x0, y0) = (self.xs[i - 1], self.ys[i - 1]);
+ let (x1, y1) = (self.xs[i], self.ys[i]);
+ if x1 == x0 {
+ return y0;
+ }
+ y0 + (p - x0) * (y1 - y0) / (x1 - x0)
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Temperature — one scalar on the log-odds
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// Probabilities at the rails have no finite log-odds; clamp before the
+/// transform so 0 and 1 are representable (Platt's own practice).
+const LOGIT_EPS: f64 = 1e-6;
+
+fn logit(p: f64) -> f64 {
+ let p = p.clamp(LOGIT_EPS, 1.0 - LOGIT_EPS);
+ (p / (1.0 - p)).ln()
+}
+
+fn sigmoid(z: f64) -> f64 {
+ if z >= 0.0 {
+ 1.0 / (1.0 + (-z).exp())
+ } else {
+ let e = z.exp();
+ e / (1.0 + e)
+ }
+}
+
+/// ln(1 + e^x), the numerically stable softplus — the two log terms of the
+/// Bernoulli NLL.
+fn softplus(x: f64) -> f64 {
+ if x > 20.0 {
+ x // e^-x underflows; ln(1+e^x) ≈ x to f64 precision
+ } else if x < -20.0 {
+ (-x).exp() // ≈ e^x, and ln(1+e^x) ≈ e^x
+ } else {
+ x.exp().ln_1p()
+ }
+}
+
+/// A temperature calibration: `p_cal = σ(logit(p_raw)/T)`. `T > 1` flattens
+/// confident probabilities toward the base rate, `T < 1` sharpens them; any
+/// `T > 0` preserves rank exactly.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct TemperatureCalibration {
+ pub t: f64,
+}
+
+/// The search bracket for `T`, on the log scale: a temperature outside
+/// `[1/16, 16]` says the raw probabilities are unusable as odds in either
+/// direction, which the caller reports rather than ships. A function, not a
+/// const, because `f64::ln` is not const-callable.
+fn t_bracket() -> (f64, f64) {
+ ((1.0f64 / 16.0).ln(), 16.0f64.ln())
+}
+
+impl TemperatureCalibration {
+ /// Fit `T` by minimising the pairs' negative log-likelihood — Platt
+ /// (1999) / Guo et al. (2017)'s held-out form — by golden-section search
+ /// on `ln T` over [`T_BRACKET`]. Deterministic: a fixed bracket and a
+ /// fixed 100 halvings of it, no starting point, no tolerance decision, so
+ /// the same pairs always fit the same `T`.
+ pub fn fit(pairs: &[CalibrationPair]) -> Self {
+ let zs: Vec<f64> = pairs.iter().map(|p| logit(p.p)).collect();
+ let nll = |ln_t: f64| -> f64 {
+ let t = ln_t.exp();
+ pairs
+ .iter()
+ .zip(&zs)
+ .map(|(pair, &z)| {
+ let a = z / t;
+ pair.y * softplus(-a) + (1.0 - pair.y) * softplus(a)
+ })
+ .sum()
+ };
+ // Golden-section minimisation (the ratio φ−1 = 0.618…).
+ let phi = (5f64.sqrt() - 1.0) / 2.0;
+ let (mut lo, mut hi) = t_bracket();
+ let mut c = hi - phi * (hi - lo);
+ let mut d = lo + phi * (hi - lo);
+ for _ in 0..100 {
+ if nll(c) < nll(d) {
+ hi = d;
+ } else {
+ lo = c;
+ }
+ c = hi - phi * (hi - lo);
+ d = lo + phi * (hi - lo);
+ }
+ Self {
+ t: ((lo + hi) / 2.0)
+ .exp()
+ .clamp(t_bracket().0.exp(), t_bracket().1.exp()),
+ }
+ }
+
+ pub fn apply(&self, p: f64) -> f64 {
+ sigmoid(logit(p) / self.t)
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The fitted layer — method choice, held-out split, one report
+// ─────────────────────────────────────────────────────────────────────────────
+
+/// The configured calibration method — the `[decisions] calibration` setting's
+/// type. `None` is the default: an uncalibrated node ships `p_raw` alone and
+/// no `p_cal` at all, never a copy of `p_raw` pretending to be calibrated.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub enum CalibrationMethod {
+ #[default]
+ None,
+ Isotonic,
+ Temperature,
+}
+
+impl CalibrationMethod {
+ /// The wire name — also the config-file value.
+ pub fn as_str(self) -> &'static str {
+ match self {
+ CalibrationMethod::None => "none",
+ CalibrationMethod::Isotonic => "isotonic",
+ CalibrationMethod::Temperature => "temperature",
+ }
+ }
+}
+
+/// The fitted calibration, whichever method produced it. One `apply`, so the
+/// caller never branches on the method to turn a raw probability into a
+/// calibrated one — history-vote shares, local-head probabilities, and (when
+/// the hosted tier is built) a passthrough score all go through this.
+#[derive(Debug, Clone, PartialEq)]
+pub enum FittedCalibration {
+ Isotonic(IsotonicCalibration),
+ Temperature(TemperatureCalibration),
+}
+
+impl FittedCalibration {
+ /// The method's wire name (`isotonic` / `temperature`).
+ pub fn method(&self) -> &'static str {
+ match self {
+ FittedCalibration::Isotonic(_) => "isotonic",
+ FittedCalibration::Temperature(_) => "temperature",
+ }
+ }
+
+ /// Calibrate one raw probability. Monotone non-decreasing for both
+ /// methods, so a ranking by `p_raw` and by `p_cal` can disagree only on
+ /// ties.
+ pub fn apply(&self, p: f64) -> f64 {
+ match self {
+ FittedCalibration::Isotonic(f) => f.apply(p),
+ FittedCalibration::Temperature(f) => f.apply(p),
+ }
+ }
+}
+
+/// The deterministic held-out split: every fifth pair, in `(p, y)` order, is
+/// held out — 20 % of the history, stratified across the probability range
+/// (so the held-out set spans the same `p_raw` support the fit sees, which is
+/// what makes an isotonic fit evaluable), with no clock and no RNG. The same
+/// multiset always splits identically (#940's lesson: a fit that moves
+/// between requests is a defect, not a measurement).
+pub fn held_out_split(pairs: &[CalibrationPair]) -> (Vec<CalibrationPair>, Vec<CalibrationPair>) {
+ let mut sorted: Vec<CalibrationPair> = pairs.to_vec();
+ sorted.sort_by(|a, b| {
+ a.p.partial_cmp(&b.p)
+ .unwrap()
+ .then(a.y.partial_cmp(&b.y).unwrap())
+ });
+ let mut fit = Vec::with_capacity(sorted.len() * 4 / 5);
+ let mut held = Vec::with_capacity(sorted.len() / 5 + 1);
+ for (i, pair) in sorted.into_iter().enumerate() {
+ if i % 5 == 4 {
+ held.push(pair);
+ } else {
+ fit.push(pair);
+ }
+ }
+ (fit, held)
+}
+
+/// The full fitted report `/_decide/_calibration` publishes: what was fitted
+/// on, how it evaluates, and the reliability curves of the raw and calibrated
+/// probabilities over every labelled pair.
+#[derive(Debug, Clone, PartialEq)]
+pub struct CalibrationReport {
+ /// The method that produced this fit.
+ pub method: CalibrationMethod,
+ /// The fit itself, for applying to new probabilities.
+ pub fit: FittedCalibration,
+ /// Pairs the fit was fitted on (80 %).
+ pub fitted_on: usize,
+ /// Pairs the ECEs below evaluate on (20 %, held out of the fit).
+ pub held_out: usize,
+ /// Every labelled pair the report saw (fit + held out).
+ pub labelled_pairs: usize,
+ /// Held-out ECE of the raw probabilities — what the node would ship
+ /// without this layer.
+ pub ece_raw: f64,
+ /// Held-out ECE after calibration — the number shipped beside every
+ /// `p_cal`.
+ pub ece_calibrated: f64,
+ /// Reliability curve of the raw probabilities over all pairs.
+ pub curve_raw: Vec<ReliabilityBin>,
+ /// Reliability curve of the calibrated probabilities over all pairs.
+ pub curve_calibrated: Vec<ReliabilityBin>,
+ /// The RFC 3339 range of the fitted documents' `ts` fields, when they
+ /// carry one — publication metadata stamped by the caller that owns the
+ /// documents, never an input to the fit itself (which stays a pure
+ /// function of the pairs).
+ pub ts_range: Option<(String, String)>,
+}
+
+/// Fit `method` on a held-out split of `pairs` and evaluate it: fit on 80 %,
+/// publish raw and calibrated ECE on the 20 % held out, and the reliability
+/// curves over all pairs. Pure — the whole report is a function of the pair
+/// multiset. `None` fits nothing and reports nothing; the caller decides what
+/// an unconfigured node ships (no `p_cal` at all).
+pub fn fit_calibration(
+ method: CalibrationMethod,
+ pairs: &[CalibrationPair],
+) -> Option<CalibrationReport> {
+ if method == CalibrationMethod::None {
+ return None;
+ }
+ let (fit_pairs, held_pairs) = held_out_split(pairs);
+ let fit = match method {
+ CalibrationMethod::Isotonic => {
+ FittedCalibration::Isotonic(IsotonicCalibration::fit(&fit_pairs))
+ }
+ CalibrationMethod::Temperature => {
+ FittedCalibration::Temperature(TemperatureCalibration::fit(&fit_pairs))
+ }
+ CalibrationMethod::None => return None,
+ };
+ let raw_pairs: Vec<CalibrationPair> = pairs.to_vec();
+ let calibrated_pairs: Vec<CalibrationPair> = pairs
+ .iter()
+ .map(|p| CalibrationPair {
+ p: fit.apply(p.p),
+ y: p.y,
+ })
+ .collect();
+ let held_raw: Vec<CalibrationPair> = held_pairs.clone();
+ let held_calibrated: Vec<CalibrationPair> = held_pairs
+ .iter()
+ .map(|p| CalibrationPair {
+ p: fit.apply(p.p),
+ y: p.y,
+ })
+ .collect();
+ Some(CalibrationReport {
+ method,
+ fit,
+ fitted_on: fit_pairs.len(),
+ held_out: held_pairs.len(),
+ labelled_pairs: pairs.len(),
+ ece_raw: ece(&held_raw, CALIBRATION_BINS),
+ ece_calibrated: ece(&held_calibrated, CALIBRATION_BINS),
+ curve_raw: reliability_curve(&raw_pairs, CALIBRATION_BINS),
+ curve_calibrated: reliability_curve(&calibrated_pairs, CALIBRATION_BINS),
+ ts_range: None,
+ })
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Tests — the algorithms are the contract, so they are pinned directly
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn pairs(pts: &[(f64, f64)]) -> Vec<CalibrationPair> {
+ pts.iter().map(|&(p, y)| CalibrationPair { p, y }).collect()
+ }
+
+ /// The FiQA baseline's own reliability curve, verbatim from
+ /// `benchmarks/beir-hybrid/results/2026-09-20-rerank-full-fiqa/fiqa-full-run.json`
+ /// (`calibration.curve`, bins 0–9): `(mean_conf, acc, n)`. 19,440
+ /// (probability, relevance) pairs, published ECE 0.3109.
+ const FIQA_BASELINE_CURVE: [(f64, f64, usize); 10] = [
+ (0.0455, 0.0002, 5572),
+ (0.1418, 0.0003, 3005),
+ (0.2434, 0.0034, 2061),
+ (0.3438, 0.0033, 1529),
+ (0.4445, 0.0077, 1423),
+ (0.5442, 0.0071, 1271),
+ (0.6451, 0.0224, 1118),
+ (0.7455, 0.0408, 1224),
+ (0.8452, 0.1242, 1272),
+ (0.9297, 0.3378, 965),
+ ];
+
+ /// Expand binned aggregates into pairs at the bin's mean confidence, the
+ /// bin's count split by its positive rate — the reconstruction every
+ /// FiQA number in this module works from (the run retained bins, not
+ /// pairs).
+ fn bin_pairs<'a>(
+ bins: impl IntoIterator<Item = &'a (f64, f64, usize)>,
+ ) -> Vec<CalibrationPair> {
+ let mut out = Vec::new();
+ for &(conf, acc, n) in bins {
+ let positives = (acc * n as f64).round() as usize;
+ for i in 0..n {
+ out.push(CalibrationPair {
+ p: conf,
+ y: if i < positives { 1.0 } else { 0.0 },
+ });
+ }
+ }
+ out
+ }
+
+ fn fiqa_baseline_pairs() -> Vec<CalibrationPair> {
+ bin_pairs(FIQA_BASELINE_CURVE.iter())
+ }
+
+ /// PAVA's defining property on the classic textbook input: the raw points
+ /// dip mid-range, and the fit pools them into the weighted mean of the
+ /// violating run.
+ #[test]
+ fn pava_pools_the_violating_run() {
+ // (x, y) with equal weights: 0→0, 1→1, 2→0, 3→1 dips at x=2; the
+ // block {1,2} violates and pools to 0.5 each.
+ let pts = pairs(&[(0.0, 0.0), (0.3, 1.0), (0.6, 0.0), (0.9, 1.0)]);
+ let fit = IsotonicCalibration::fit(&pts);
+ let knots = fit.knots();
+ assert_eq!(knots[1].1, 0.5, "the violating run pools: {knots:?}");
+ assert_eq!(knots[2].1, 0.5, "both members of the run share one value");
+ assert_eq!(knots[0].1, 0.0, "an undisrupted point is untouched");
+ assert_eq!(knots[3].1, 1.0);
+ }
+
+ /// Isotonic output is monotone non-decreasing in the input, on data whose
+ /// empirical rates are noisy — the property that makes p_cal a
+ /// probability, not a re-scoring.
+ #[test]
+ fn isotonic_output_is_monotone() {
+ let pts = pairs(&[
+ (0.05, 0.0),
+ (0.1, 1.0),
+ (0.15, 0.0),
+ (0.2, 0.0),
+ (0.5, 1.0),
+ (0.55, 0.0),
+ (0.9, 1.0),
+ (0.95, 0.0),
+ ]);
+ let fit = IsotonicCalibration::fit(&pts);
+ let mut prev = 0.0f64;
+ for p in [0.0, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99, 1.0] {
+ let cal = fit.apply(p);
+ assert!(
+ cal >= prev - 1e-12,
+ "apply({p}) = {cal} < apply of a smaller input {prev}"
+ );
+ assert!((0.0..=1.0).contains(&cal), "p_cal is a probability: {cal}");
+ prev = cal;
+ }
+ }
+
+ /// Interpolation between knots, clamped outside them: a fit on [0.2, 0.8]
+ /// is constant below 0.2 and above 0.8, linear between.
+ #[test]
+ fn isotonic_interpolates_and_clamps() {
+ let pts = pairs(&[(0.2, 0.0), (0.8, 1.0)]);
+ let fit = IsotonicCalibration::fit(&pts);
+ assert_eq!(fit.apply(0.0), 0.0, "below the lowest knot: clamped");
+ assert_eq!(fit.apply(1.0), 1.0, "above the highest knot: clamped");
+ assert!((fit.apply(0.5) - 0.5).abs() < 1e-12, "linear between knots");
+ }
+
+ /// Tied raw probabilities pool to their base rate — one p, one fitted
+ /// value, whatever outcomes share it.
+ #[test]
+ fn tied_probabilities_pool() {
+ let pts = pairs(&[(0.7, 1.0), (0.7, 0.0), (0.7, 0.0), (0.7, 1.0), (0.7, 0.0)]);
+ let fit = IsotonicCalibration::fit(&pts);
+ assert_eq!(fit.knots().len(), 1, "one distinct p → one knot");
+ assert!((fit.apply(0.7) - 0.4).abs() < 1e-12, "the base rate 2/5");
+ }
+
+ /// Temperature preserves rank exactly and only moves the numbers: with
+ /// T fit on data that is over-confident, every probability shrinks toward
+ /// the middle without any two swapping order.
+ #[test]
+ fn temperature_shifts_but_preserves_rank() {
+ let pts = pairs(&[
+ (0.95, 0.0),
+ (0.9, 0.0),
+ (0.9, 1.0),
+ (0.8, 0.0),
+ (0.6, 0.0),
+ (0.95, 1.0),
+ (0.99, 1.0),
+ (0.85, 0.0),
+ (0.92, 0.0),
+ (0.7, 0.0),
+ ]);
+ let fit = TemperatureCalibration::fit(&pts);
+ assert!(fit.t > 1.0, "over-confident data fits T > 1: {}", fit.t);
+ let raw = [0.1, 0.3, 0.5, 0.7, 0.85, 0.95, 0.99];
+ let cal: Vec<f64> = raw.iter().map(|p| fit.apply(*p)).collect();
+ let mut sorted_cal = cal.clone();
+ sorted_cal.sort_by(|a, b| a.partial_cmp(b).unwrap());
+ assert_eq!(cal, sorted_cal, "rank preserved");
+ assert!(cal[5] < 0.95, "high confidence shrinks: {}", cal[5]);
+ assert!((fit.apply(0.5) - 0.5).abs() < 1e-9, "the midpoint is fixed");
+ }
+
+ /// Under-confident data fits T < 1 (sharpening) — the bracket's other
+ /// half.
+ #[test]
+ fn under_confident_data_sharpens() {
+ let pts = pairs(&[
+ (0.6, 1.0),
+ (0.62, 1.0),
+ (0.58, 0.0),
+ (0.61, 1.0),
+ (0.4, 0.0),
+ (0.39, 0.0),
+ (0.41, 0.0),
+ (0.6, 1.0),
+ (0.42, 0.0),
+ (0.59, 0.0),
+ ]);
+ let fit = TemperatureCalibration::fit(&pts);
+ assert!(fit.t < 1.0, "under-confident data fits T < 1: {}", fit.t);
+ assert!(fit.apply(0.61) > 0.61, "sharpened upward");
+ assert!(fit.apply(0.41) < 0.41, "sharpened downward");
+ }
+
+ /// ECE identities: a perfectly calibrated set is 0, a constant confident
+ /// wrong set is its full error, and the published FiQA ECE 0.3109 is
+ /// reproduced by the shared implementation from the baseline's own curve
+ /// — the arithmetic this module shares with the measurement that
+ /// motivated it.
+ #[test]
+ fn ece_identities_and_the_fiqa_arithmetic() {
+ // Perfect: every bin's mean equals its rate.
+ let perfect: Vec<CalibrationPair> = (0..10)
+ .flat_map(|b| {
+ let p = (b as f64 + 0.5) / 10.0;
+ let positives = (p * 10.0).round() as usize;
+ (0..10).map(move |i| CalibrationPair {
+ p,
+ y: if i < positives { 1.0 } else { 0.0 },
+ })
+ })
+ .collect();
+ assert!(ece(&perfect, 10) < 0.11, "the discretisation is coarse");
+ // Constant and confidently wrong: mean 0.9 against a 0.1 rate,
+ // |0.9 − 0.1| = 0.8 exactly.
+ let wrong: Vec<CalibrationPair> = (0..10)
+ .map(|i| CalibrationPair {
+ p: 0.9,
+ y: (i == 0) as u8 as f64,
+ })
+ .collect();
+ assert!((ece(&wrong, 10) - 0.8).abs() < 1e-12);
+ // Empty: no probabilities, no error.
+ assert_eq!(ece(&[], 10), 0.0);
+ // The published FiQA ECE, recomputed from the baseline's own curve
+ // (`fiqa-full-run.json`, calibration.curve) as pairs-at-the-bin-mean.
+ let pairs = fiqa_baseline_pairs();
+ assert_eq!(pairs.len(), 19_440);
+ assert!(
+ (ece(&pairs, 10) - 0.3109).abs() < 0.0005,
+ "the FiQA ECE, from this module: {}",
+ ece(&pairs, 10)
+ );
+ }
+
+ /// The split is exactly 80/20, stratified across the probability range,
+ /// and deterministic — the same multiset in a different order splits
+ /// identically.
+ #[test]
+ fn held_out_split_is_deterministic_and_stratified() {
+ let pts: Vec<CalibrationPair> = (0..100)
+ .map(|i| CalibrationPair {
+ p: (i % 10) as f64 / 10.0,
+ y: (i % 3 == 0) as u8 as f64,
+ })
+ .collect();
+ let (fit, held) = held_out_split(&pts);
+ assert_eq!(fit.len(), 80);
+ assert_eq!(held.len(), 20);
+ // Stratified: every distinct p appears in both halves.
+ for p in [0.0, 0.1, 0.5, 0.9] {
+ assert!(fit.iter().any(|c| c.p == p), "p={p} in the fit set");
+ assert!(held.iter().any(|c| c.p == p), "p={p} in the held-out set");
+ }
+ // Deterministic under input reordering: the same multiset splits to
+ // the same two multisets (compare sorted, since the stable sort may
+ // order equal (p, y) pairs differently in the two runs).
+ let mut shuffled = pts.clone();
+ shuffled.reverse();
+ let (fit2, held2) = held_out_split(&shuffled);
+ let canon = |mut v: Vec<CalibrationPair>| {
+ v.sort_by(|a, b| {
+ a.p.partial_cmp(&b.p)
+ .unwrap()
+ .then(a.y.partial_cmp(&b.y).unwrap())
+ });
+ v
+ };
+ assert_eq!(canon(fit), canon(fit2));
+ assert_eq!(canon(held), canon(held2));
+ }
+
+ /// The issue's gate, on a deterministic bin-level reconstruction of the
+ /// FiQA baseline: isotonic fitted on the EVEN bins, evaluated held-out on
+ /// the ODD bins — whole probability regions the fit never saw, which is
+ /// the honest form of held-out this data supports (the run retained bins,
+ /// not pairs). This pins the same numbers the results file in
+ /// `benchmarks/decisions-calibration/` publishes, so a regression in the
+ /// fit code fails a test rather than a release gate.
+ #[test]
+ fn isotonic_meets_the_fiqa_gate_on_the_baseline_curve() {
+ let fiqa = FIQA_BASELINE_CURVE;
+ let fit_pairs: Vec<CalibrationPair> = bin_pairs(fiqa.iter().step_by(2));
+ let held_pairs: Vec<CalibrationPair> = bin_pairs(fiqa.iter().skip(1).step_by(2));
+ // Raw held-out ECE is the same order as the published 0.3109 —
+ // the odd bins are representative, not a gift.
+ let raw_held = ece(&held_pairs, 10);
+ assert!(
+ (raw_held - 0.3831).abs() < 0.0005,
+ "raw held-out ECE {raw_held} (benchmarks/decisions-calibration)"
+ );
+ // Isotonic on the even bins, applied to the odd bins' means.
+ let fit = IsotonicCalibration::fit(&fit_pairs);
+ let calibrated: Vec<CalibrationPair> = held_pairs
+ .iter()
+ .map(|p| CalibrationPair {
+ p: fit.apply(p.p),
+ y: p.y,
+ })
+ .collect();
+ let cal_held = ece(&calibrated, 10);
+ // The exact value is pinned to the Python measurement
+ // (benchmarks/decisions-calibration/fiqa_gate.py) — two independent
+ // implementations of PAVA + ECE agreeing on the same split is a
+ // cross-check on both.
+ assert!(
+ (cal_held - 0.0330).abs() < 0.0005,
+ "held-out calibrated ECE {cal_held} (benchmarks/decisions-calibration says 0.0330)"
+ );
+ assert!(cal_held <= 0.10, "the #1063 gate");
+ }
+
+ /// An empty or single-point fit is degenerate but total: no panic, and
+ /// apply stays in [0,1].
+ #[test]
+ fn degenerate_fits_are_total() {
+ let empty = IsotonicCalibration::fit(&[]);
+ assert_eq!(empty.apply(0.7), 0.7, "empty fit: identity");
+ let single = IsotonicCalibration::fit(&[CalibrationPair { p: 0.6, y: 1.0 }]);
+ assert!((single.apply(0.6) - 1.0).abs() < 1e-12);
+ let temp = TemperatureCalibration::fit(&[CalibrationPair { p: 0.6, y: 1.0 }]);
+ assert!(temp.t.is_finite() && temp.t > 0.0);
+ assert!((0.0..=1.0).contains(&temp.apply(0.6)));
+ // The rails do not produce infinities.
+ assert!((0.0..=1.0).contains(&temp.apply(0.0)));
+ assert!((0.0..=1.0).contains(&temp.apply(1.0)));
+ }
+}
diff --git a/engine/crates/xerj-common/src/config.rs b/engine/crates/xerj-common/src/config.rs
index 06c114846..f5499211b 100644
--- a/engine/crates/xerj-common/src/config.rs
+++ b/engine/crates/xerj-common/src/config.rs
@@ -1,6 +1,6 @@
//! xerj configuration system.
//!
-//! Configuration is intentionally minimal: **129 settings** versus
+//! Configuration is intentionally minimal: **130 settings** versus
//! Elasticsearch's 3000+. Every option is named, documented, and has a sensible
//! production-ready default. The format is TOML, loaded from a single file.
//!
@@ -39,6 +39,7 @@
use serde::{Deserialize, Serialize};
use std::path::Path;
+use crate::calibration::CalibrationMethod;
use crate::error::XerjError;
// ═════════════════════════════════════════════════════════════════════════════
@@ -98,12 +99,12 @@ pub struct Config {
/// Second-stage reranking provider — 3 settings. Inert until a key is set.
pub rerank: RerankProviderConfig,
/// Typed decisions answered from a labelled-history index by
- /// nearest-neighbour vote — 7 settings. Inert until `decisions.index`
+ /// nearest-neighbour vote — 8 settings. Inert until `decisions.index`
/// names an index that exists.
pub decisions: DecisionsConfig,
}
-// 23 sub-configs, 129 leaf settings in total. Do not maintain that sum by hand
+// 23 sub-configs, 130 leaf settings in total. Do not maintain that sum by hand
// — `journey_zero_config` in xerj-engine/tests/product_experience.rs counts a
// serialised `Config::default()` and fails if this comment and the module
// header stop matching. `Default` is derived: every field is a sub-config that
@@ -484,7 +485,7 @@ impl Config {
}
// ═════════════════════════════════════════════════════════════════════════════
-// Sub-configs (129 user-facing settings total; counted by
+// Sub-configs (130 user-facing settings total; counted by
// `journey_zero_config`, not by hand)
// ═════════════════════════════════════════════════════════════════════════════
@@ -2342,7 +2343,7 @@ impl RerankProviderConfig {
/// weighted nearest-neighbour vote over a labelled-history index — the
/// retrieval analogue of a judge model, with the evidence staying on the node.
///
-/// **6 settings.**
+/// **8 settings.**
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct DecisionsConfig {
@@ -2368,6 +2369,12 @@ pub struct DecisionsConfig {
/// (issue #1061). 2.0 — the issue's ≥ 2x floor for corrections — is the
/// default; 1.0 makes corrections count exactly like any other example.
pub human_weight: f64,
+ /// Probability calibration for the decide surface (issue #1063): fit a
+ /// correction from raw probabilities to calibrated ones on a held-out
+ /// 20 % of the labelled history, then ship `p_cal` beside every `p_raw`.
+ /// `isotonic` (PAVA) or `temperature` (one scalar on the log-odds);
+ /// `none` (the default) ships `p_raw` alone and no `p_cal` at all.
+ pub calibration: CalibrationMethod,
}
impl Default for DecisionsConfig {
@@ -2380,6 +2387,7 @@ impl Default for DecisionsConfig {
positive_label: "true".to_string(),
min_confidence: 0.0,
human_weight: 2.0,
+ calibration: CalibrationMethod::None,
}
}
}
@@ -3190,7 +3198,7 @@ mod tests {
("lifecycle", 1),
("wal_tap", 10),
("rerank", 3),
- ("decisions", 7),
+ ("decisions", 8),
];
/// Count the settings by *counting them*.
@@ -3233,7 +3241,7 @@ mod tests {
"the section table must sum to the whole config"
);
assert_eq!(
- total, 129,
+ total, 130,
"the total settings count changed. It is quoted in this module's \
header, in xerj-common/src/lib.rs, in engine/README.md, in \
xerj.default.toml and in EXPECTED_SETTINGS in \
@@ -3262,6 +3270,38 @@ mod tests {
assert!(cfg.grpc_h2c_exposed_off_loopback());
}
+ /// `decisions.calibration` parses the issue's two methods and refuses a
+ /// typo by naming the valid ones — the same refusal shape as every other
+ /// enum setting, so an operator hears about `calibration = "isotonic2"`
+ /// at boot, not from a response that silently shipped no p_cal.
+ #[test]
+ fn decisions_calibration_parses_the_two_methods_and_refuses_a_typo() {
+ for (value, expected) in [
+ ("isotonic", CalibrationMethod::Isotonic),
+ ("temperature", CalibrationMethod::Temperature),
+ ] {
+ let cfg = Config::from_toml_str(&format!(
+ "[decisions]\nindex = \"judgements\"\ncalibration = \"{value}\"\n"
+ ))
+ .expect("parses");
+ assert_eq!(cfg.decisions.calibration, expected);
+ }
+ // The default is none: an uncalibrated node, not a hidden isotonic.
+ assert_eq!(
+ DecisionsConfig::default().calibration,
+ CalibrationMethod::None
+ );
+ let err = Config::from_toml_str(
+ "[decisions]\nindex = \"judgements\"\ncalibration = \"isotonic2\"\n",
+ )
+ .expect_err("typo must be refused");
+ let msg = err.to_string();
+ assert!(
+ msg.contains("calibration") && msg.contains("isotonic"),
+ "the error names the field and the valid values: {msg}"
+ );
+ }
+
#[test]
fn tls_on_bound_to_loopback_is_fine() {
for bind in ["127.0.0.1", "127.0.0.5", "::1", "::ffff:127.0.0.1"] {
diff --git a/engine/crates/xerj-common/src/lib.rs b/engine/crates/xerj-common/src/lib.rs
index bfd20a0e3..103374065 100644
--- a/engine/crates/xerj-common/src/lib.rs
+++ b/engine/crates/xerj-common/src/lib.rs
@@ -7,16 +7,18 @@
//! ## Design philosophy
//!
//! Unlike Elasticsearch's 3000+ configuration knobs, xerj deliberately exposes
-//! **129 settings**, each meaningful and production-tested. Every default
+//! **130 settings**, each meaningful and production-tested. Every default
//! is chosen so that a fresh deployment with zero configuration changes performs
//! well for the majority of workloads.
//!
//! ## Modules
//!
-//! - [`config`] — TOML-based configuration (129 settings)
+//! - [`config`] — TOML-based configuration (130 settings)
//! - [`feedback`] — the bug/UX-report invitation shared by every `--help`
//! - [`error`] — Unified error type ([`XerjError`])
//! - [`types`] — Core domain types (documents, fields, IDs)
+//! - [`calibration`] — probability calibration for the decide surface:
+//! isotonic (PAVA) and temperature (Platt) scaling, reliability curves, ECE
//! - [`field_coercion`] — ES-faithful ingest-time coercion/enforcement for
//! numeric and boolean fields (the one predicate every write path shares)
//! - [`schema`] — Index schema management and mapping evolution
@@ -28,6 +30,7 @@
//! - [`xccode`] — the reference-coding semantics shared by `xerj code`,
//! `xerj corpus`, and the `xerj_code_search` MCP tool (issue #977)
+pub mod calibration;
pub mod config;
pub mod error;
pub mod feedback;
diff --git a/engine/crates/xerj-engine/tests/product_experience.rs b/engine/crates/xerj-engine/tests/product_experience.rs
index 6e4d98051..3481e4ae1 100644
--- a/engine/crates/xerj-engine/tests/product_experience.rs
+++ b/engine/crates/xerj-engine/tests/product_experience.rs
@@ -753,7 +753,7 @@ fn count_settings(value: &serde_json::Value) -> usize {
/// Settings in a default `Config`, measured by `count_settings`. Quoted in
/// `xerj-common/src/config.rs`; keep the two in step.
-const EXPECTED_SETTINGS: usize = 129;
+const EXPECTED_SETTINGS: usize = 130;
#[tokio::test]
async fn journey_zero_config() {
diff --git a/engine/xerj.default.toml b/engine/xerj.default.toml
index d0c5821e0..e8f49ec05 100644
--- a/engine/xerj.default.toml
+++ b/engine/xerj.default.toml
@@ -1,6 +1,6 @@
# xerj Default Configuration
# ============================================================
-# This file sets 55 of XERJ's 129 user-facing settings to their
+# This file sets 55 of XERJ's 130 user-facing settings to their
# production-ready defaults, and describes the rest in comments.
# Every value here IS the default, so copying this file changes
# nothing until you edit it (a test enforces that). The complete
@@ -656,3 +656,4 @@ access_log = false
# positive_label = "true" # the label a noul's probability is the share of
# min_confidence = 0.0 # /_decide abstains below this
# human_weight = 2.0 # vote-weight multiplier for history docs carrying human: true
+# calibration = "none" # none | isotonic | temperature — p_cal beside every p_raw (#1063)