Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(eval): selective-prediction report for metamorphic variants
Charanraj24fb4419f6e2d6
diff --git a/research/eval/README.md b/research/eval/README.mdindex 3ba5f01a..870be7c2 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -392,3 +392,33 @@ For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`.**Semantic agreement and distribution stability are different properties.**A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.++### Selective prediction: does disagreement predict errors?++Robustness agreement is not a correctness measure, but on labelled cases it can be tested as a *signal* of correctness: are the baseline answers that change under a transform more often wrong? When gold indices are given, the report adds a `selective_prediction` section computed per labelled case from the baseline winner `w`:++| Signal | Definition |+|---|---|+| `confidence` | Baseline probability of `w` (the harness's maximum probability) |+| `agreement_<kind>` | Share of the `<kind>` variants whose canonical argmax is also `w` |+| `support_<kind>` | Mean probability the `<kind>` variants give `w` |+| `support_all` | Mean probability of `w` over the baseline and every variant |++`<kind>` is each transform present (`option_order`, `label_rename`). For every signal, `auroc` is the probability that a random correct case scores above a random wrong one (ties count half; `null` unless both correct and wrong cases exist). For the continuous signals, `accuracy_at_coverage` is the accuracy of the highest-scoring fraction of cases at 50, 70, 80 and 90% coverage; `agreement_*` is left out there because with one variant per kind it is binary, and the cut would depend on tie order. `n_labelled` and `n_wrong` give the sample size; unlabelled runs report `n_labelled: 0` only.++This measures whether a signal ranks errors below correct answers; it does not set a threshold. Consistently wrong predictions stay invisible to every `agreement_*` and `support_*` signal. With one variant per kind, `agreement_*` is coarse; the `support_*` signals use the full probability vectors.++#### Measured++MASSIVE `en`, `--per-lang 300 --n-opts 20`, seed 13, laya 0.3.21, CPU (65 wrong answers). Differences vs `confidence` are bootstrap 95% intervals over cases (2,000 resamples):++| Signal | AUROC | vs `confidence` |+|---|---:|---:|+| `confidence` | 0.827 | |+| `support_all` | 0.877 | +0.050 (+0.010 to +0.096) |+| `support_label_rename` | 0.866 | +0.039 (−0.018 to +0.094) |+| `support_option_order` | 0.805 | −0.023 (−0.066 to +0.018) |+| `agreement_label_rename` | 0.744 | −0.084 (−0.155 to −0.010) |+| `agreement_option_order` | 0.678 | |++With one variant per transform, only `support_all` clearly ranks errors below correct answers better than `confidence`; the binary `agreement_*` signals rank them worse. Keeping the top 70% of cases gives 93.3% accuracy by `support_all` and 91.0% by `confidence`.diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.pyindex d54befdf..d0f5145b 100644--- a/research/eval/metamorphic.py+++ b/research/eval/metamorphic.py@@ -235,6 +235,70 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_sizreturn {"report": _report(records), "cases": records}+def auroc(scores, labels):+ """Chance that a random correct case scores above a random wrong one.++ Ties count half. Returns None unless both correct and wrong cases exist.+ """+ pairs = sorted(zip(scores, labels), key=lambda pair: pair[0])+ ranks, start = [0.0] * len(pairs), 0+ while start < len(pairs):+ end = start+ while end + 1 < len(pairs) and pairs[end + 1][0] == pairs[start][0]:+ end += 1+ for i in range(start, end + 1):+ ranks[i] = (start + end) / 2 + 1+ start = end + 1+ positives = sum(1 for _, label in pairs if label)+ negatives = len(pairs) - positives+ if not positives or not negatives:+ return None+ rank_sum = sum(rank for rank, (_, label) in zip(ranks, pairs) if label)+ return (rank_sum - positives * (positives + 1) / 2) / (positives * negatives)+++def selective_prediction(records, coverages=(0.5, 0.7, 0.8, 0.9)):+ """Whether disagreement under the transforms predicts which baseline answers are wrong.++ Per labelled case, with w the baseline winner:+ confidence baseline probability of w+ agreement_<kind> 1 if the <kind> variant(s) also pick w, else the share that do+ support_<kind> mean probability the <kind> variant(s) give w+ support_all mean probability of w over the baseline and every variant+ Reports AUROC per signal and, for the continuous signals, the accuracy of the+ most-trusted fraction of cases at each coverage (ties broken by case order).+ """+ rows = []+ for record in records:+ baseline, variants = record["variants"][0], record["variants"][1:]+ if baseline.get("correct") is None:+ continue+ winner = baseline["pred_index"]+ row = {"correct": bool(baseline["correct"]), "confidence": baseline["probabilities"][winner]}+ for kind in dict.fromkeys(v["kind"] for v in variants):+ same = [v for v in variants if v["kind"] == kind]+ row["agreement_" + kind] = sum(v["pred_index"] == winner for v in same) / len(same)+ row["support_" + kind] = sum(v["probabilities"][winner] for v in same) / len(same)+ row["support_all"] = sum(v["probabilities"][winner] for v in record["variants"]) / len(record["variants"])+ rows.append(row)+ wrong = sum(not r["correct"] for r in rows)+ if not rows:+ return {"n_labelled": 0}+ signals = [key for key in rows[0] if key != "correct" and all(key in r for r in rows)]+ labels = [r["correct"] for r in rows]+ result = {"n_labelled": len(rows), "n_wrong": wrong,+ "auroc": {key: auroc([r[key] for r in rows], labels) for key in signals},+ "accuracy_at_coverage": {}}+ for key in signals:+ if key.startswith("agreement_"):+ continue # near-binary: the coverage cut would mostly depend on tie order+ ranked = sorted(range(len(rows)), key=lambda i: -rows[i][key])+ result["accuracy_at_coverage"][key] = {+ str(c): sum(rows[i]["correct"] for i in ranked[:max(1, round(len(rows) * c))])+ / max(1, round(len(rows) * c)) for c in coverages}+ return result++def _report(records):groups = {"option_order": [], "label_rename": []}for record in records:@@ -254,6 +318,7 @@ def _report(records):ece=harness.ece([v["confidence"] for v in labelled],[v["correct"] for v in labelled]))report["quality"][kind] = quality+ report["selective_prediction"] = selective_prediction(records)return reportdiff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.pyindex f9df757c..778db5ad 100644--- a/research/eval/test_metamorphic.py+++ b/research/eval/test_metamorphic.py@@ -246,5 +246,54 @@ class EvaluationTests(unittest.TestCase):self.assertEqual(payload["config"]["temperature"], [1.0] * 3)+class SelectivePredictionTests(unittest.TestCase):+ def test_auroc_known_values(self):+ self.assertEqual(m.auroc([0.9, 0.8, 0.2, 0.1], [True, True, False, False]), 1)+ self.assertEqual(m.auroc([0.1, 0.2, 0.8, 0.9], [True, True, False, False]), 0)+ self.assertEqual(m.auroc([0.5, 0.5, 0.5, 0.5], [True, False, True, False]), 0.5)+ self.assertAlmostEqual(m.auroc([0.9, 0.4, 0.6, 0.1], [True, True, False, False]), 0.75)+ self.assertIsNone(m.auroc([0.9, 0.8], [True, True]))+ self.assertIsNone(m.auroc([], []))++ def test_disagreement_flags_confident_errors(self):+ # "stable" cases follow the descriptions and are right at 0.7. "fragile"+ # cases follow the label tokens and are wrong at 0.9; renaming moves their+ # winner. Confidence ranks the errors on top; agreement under renaming+ # ranks them at the bottom. Reordering keeps keys with their descriptions,+ # so it cannot tell the two apart here.+ stable = ({"utterance": "stable"}, deepcopy(CASE[1]))+ fragile = ({"utterance": "fragile"}, deepcopy(CASE[1]))+ label_weights = {"billing": 0.9, "technical": 0.05, "sales": 0.05, "A": 0.2, "B": 0.6, "C": 0.2}++ def score(cases):+ out = []+ for state, questions in cases:+ if state["utterance"] == "stable":+ out.extend(semantic_scorer([(state, questions)]))+ else:+ out.append([label_weights[k] for k in questions["intent"]["criteria"]])+ return out++ report = m.evaluate([stable, stable, fragile, fragile], score, [0, 0, 1, 1])["report"]+ sp = report["selective_prediction"]+ self.assertEqual((sp["n_labelled"], sp["n_wrong"]), (4, 2))+ self.assertEqual(sp["auroc"]["confidence"], 0)+ self.assertEqual(sp["auroc"]["agreement_label_rename"], 1)+ self.assertEqual(sp["auroc"]["support_label_rename"], 1)+ self.assertEqual(sp["auroc"]["agreement_option_order"], 0.5)+ self.assertEqual(sp["auroc"]["support_all"], 1)+ self.assertEqual(sp["accuracy_at_coverage"]["confidence"]["0.5"], 0)+ self.assertEqual(sp["accuracy_at_coverage"]["support_all"]["0.5"], 1)+ self.assertNotIn("agreement_label_rename", sp["accuracy_at_coverage"])++ def test_unlabelled_or_one_class(self):+ report = m.evaluate([CASE], semantic_scorer)["report"]+ self.assertEqual(report["selective_prediction"], {"n_labelled": 0})+ sp = m.evaluate([CASE, CASE], semantic_scorer, [0, 0])["report"]["selective_prediction"]+ self.assertEqual((sp["n_labelled"], sp["n_wrong"]), (2, 0))+ self.assertIsNone(sp["auroc"]["confidence"])+ self.assertEqual(sp["accuracy_at_coverage"]["confidence"]["0.5"], 1)++if __name__ == "__main__":unittest.main()