Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(eval): selective-prediction report for metamorphic variants

Charanraj24fb4419f6e2d6
diff --git a/research/eval/README.md b/research/eval/README.md
index 3ba5f01a..870be7c2 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -392,3 +392,33 @@ For offline tests, pass `agent=None, score=fake_scorer` to `evaluate_variants`.
**Semantic agreement and distribution stability are different properties.**
A shift from `[0.91, 0.06, 0.03]` to `[0.88, 0.08, 0.04]` preserves the decision while showing nonzero drift. Switching the winner is reported as disagreement, regardless of whether confidence rises or falls. No metric here automatically classifies either observation as a bug; acceptable variation depends on the use case, and the report deliberately defines no universal pass/fail threshold.
+
+### Selective prediction: does disagreement predict errors?
+
+Robustness agreement is not a correctness measure, but on labelled cases it can be tested as a *signal* of correctness: are the baseline answers that change under a transform more often wrong? When gold indices are given, the report adds a `selective_prediction` section computed per labelled case from the baseline winner `w`:
+
+| Signal | Definition |
+|---|---|
+| `confidence` | Baseline probability of `w` (the harness's maximum probability) |
+| `agreement_<kind>` | Share of the `<kind>` variants whose canonical argmax is also `w` |
+| `support_<kind>` | Mean probability the `<kind>` variants give `w` |
+| `support_all` | Mean probability of `w` over the baseline and every variant |
+
+`<kind>` is each transform present (`option_order`, `label_rename`). For every signal, `auroc` is the probability that a random correct case scores above a random wrong one (ties count half; `null` unless both correct and wrong cases exist). For the continuous signals, `accuracy_at_coverage` is the accuracy of the highest-scoring fraction of cases at 50, 70, 80 and 90% coverage; `agreement_*` is left out there because with one variant per kind it is binary, and the cut would depend on tie order. `n_labelled` and `n_wrong` give the sample size; unlabelled runs report `n_labelled: 0` only.
+
+This measures whether a signal ranks errors below correct answers; it does not set a threshold. Consistently wrong predictions stay invisible to every `agreement_*` and `support_*` signal. With one variant per kind, `agreement_*` is coarse; the `support_*` signals use the full probability vectors.
+
+#### Measured
+
+MASSIVE `en`, `--per-lang 300 --n-opts 20`, seed 13, laya 0.3.21, CPU (65 wrong answers). Differences vs `confidence` are bootstrap 95% intervals over cases (2,000 resamples):
+
+| Signal | AUROC | vs `confidence` |
+|---|---:|---:|
+| `confidence` | 0.827 | |
+| `support_all` | 0.877 | +0.050 (+0.010 to +0.096) |
+| `support_label_rename` | 0.866 | +0.039 (−0.018 to +0.094) |
+| `support_option_order` | 0.805 | −0.023 (−0.066 to +0.018) |
+| `agreement_label_rename` | 0.744 | −0.084 (−0.155 to −0.010) |
+| `agreement_option_order` | 0.678 | |
+
+With one variant per transform, only `support_all` clearly ranks errors below correct answers better than `confidence`; the binary `agreement_*` signals rank them worse. Keeping the top 70% of cases gives 93.3% accuracy by `support_all` and 91.0% by `confidence`.
diff --git a/research/eval/metamorphic.py b/research/eval/metamorphic.py
index d54befdf..d0f5145b 100644
--- a/research/eval/metamorphic.py
+++ b/research/eval/metamorphic.py
@@ -235,6 +235,70 @@ def _evaluate_generated(cases, generated_by_case, score, gold_indices, batch_siz
return {"report": _report(records), "cases": records}
+def auroc(scores, labels):
+ """Chance that a random correct case scores above a random wrong one.
+
+ Ties count half. Returns None unless both correct and wrong cases exist.
+ """
+ pairs = sorted(zip(scores, labels), key=lambda pair: pair[0])
+ ranks, start = [0.0] * len(pairs), 0
+ while start < len(pairs):
+ end = start
+ while end + 1 < len(pairs) and pairs[end + 1][0] == pairs[start][0]:
+ end += 1
+ for i in range(start, end + 1):
+ ranks[i] = (start + end) / 2 + 1
+ start = end + 1
+ positives = sum(1 for _, label in pairs if label)
+ negatives = len(pairs) - positives
+ if not positives or not negatives:
+ return None
+ rank_sum = sum(rank for rank, (_, label) in zip(ranks, pairs) if label)
+ return (rank_sum - positives * (positives + 1) / 2) / (positives * negatives)
+
+
+def selective_prediction(records, coverages=(0.5, 0.7, 0.8, 0.9)):
+ """Whether disagreement under the transforms predicts which baseline answers are wrong.
+
+ Per labelled case, with w the baseline winner:
+ confidence baseline probability of w
+ agreement_<kind> 1 if the <kind> variant(s) also pick w, else the share that do
+ support_<kind> mean probability the <kind> variant(s) give w
+ support_all mean probability of w over the baseline and every variant
+ Reports AUROC per signal and, for the continuous signals, the accuracy of the
+ most-trusted fraction of cases at each coverage (ties broken by case order).
+ """
+ rows = []
+ for record in records:
+ baseline, variants = record["variants"][0], record["variants"][1:]
+ if baseline.get("correct") is None:
+ continue
+ winner = baseline["pred_index"]
+ row = {"correct": bool(baseline["correct"]), "confidence": baseline["probabilities"][winner]}
+ for kind in dict.fromkeys(v["kind"] for v in variants):
+ same = [v for v in variants if v["kind"] == kind]
+ row["agreement_" + kind] = sum(v["pred_index"] == winner for v in same) / len(same)
+ row["support_" + kind] = sum(v["probabilities"][winner] for v in same) / len(same)
+ row["support_all"] = sum(v["probabilities"][winner] for v in record["variants"]) / len(record["variants"])
+ rows.append(row)
+ wrong = sum(not r["correct"] for r in rows)
+ if not rows:
+ return {"n_labelled": 0}
+ signals = [key for key in rows[0] if key != "correct" and all(key in r for r in rows)]
+ labels = [r["correct"] for r in rows]
+ result = {"n_labelled": len(rows), "n_wrong": wrong,
+ "auroc": {key: auroc([r[key] for r in rows], labels) for key in signals},
+ "accuracy_at_coverage": {}}
+ for key in signals:
+ if key.startswith("agreement_"):
+ continue # near-binary: the coverage cut would mostly depend on tie order
+ ranked = sorted(range(len(rows)), key=lambda i: -rows[i][key])
+ result["accuracy_at_coverage"][key] = {
+ str(c): sum(rows[i]["correct"] for i in ranked[:max(1, round(len(rows) * c))])
+ / max(1, round(len(rows) * c)) for c in coverages}
+ return result
+
+
def _report(records):
groups = {"option_order": [], "label_rename": []}
for record in records:
@@ -254,6 +318,7 @@ def _report(records):
ece=harness.ece([v["confidence"] for v in labelled],
[v["correct"] for v in labelled]))
report["quality"][kind] = quality
+ report["selective_prediction"] = selective_prediction(records)
return report
diff --git a/research/eval/test_metamorphic.py b/research/eval/test_metamorphic.py
index f9df757c..778db5ad 100644
--- a/research/eval/test_metamorphic.py
+++ b/research/eval/test_metamorphic.py
@@ -246,5 +246,54 @@ class EvaluationTests(unittest.TestCase):
self.assertEqual(payload["config"]["temperature"], [1.0] * 3)
+class SelectivePredictionTests(unittest.TestCase):
+ def test_auroc_known_values(self):
+ self.assertEqual(m.auroc([0.9, 0.8, 0.2, 0.1], [True, True, False, False]), 1)
+ self.assertEqual(m.auroc([0.1, 0.2, 0.8, 0.9], [True, True, False, False]), 0)
+ self.assertEqual(m.auroc([0.5, 0.5, 0.5, 0.5], [True, False, True, False]), 0.5)
+ self.assertAlmostEqual(m.auroc([0.9, 0.4, 0.6, 0.1], [True, True, False, False]), 0.75)
+ self.assertIsNone(m.auroc([0.9, 0.8], [True, True]))
+ self.assertIsNone(m.auroc([], []))
+
+ def test_disagreement_flags_confident_errors(self):
+ # "stable" cases follow the descriptions and are right at 0.7. "fragile"
+ # cases follow the label tokens and are wrong at 0.9; renaming moves their
+ # winner. Confidence ranks the errors on top; agreement under renaming
+ # ranks them at the bottom. Reordering keeps keys with their descriptions,
+ # so it cannot tell the two apart here.
+ stable = ({"utterance": "stable"}, deepcopy(CASE[1]))
+ fragile = ({"utterance": "fragile"}, deepcopy(CASE[1]))
+ label_weights = {"billing": 0.9, "technical": 0.05, "sales": 0.05, "A": 0.2, "B": 0.6, "C": 0.2}
+
+ def score(cases):
+ out = []
+ for state, questions in cases:
+ if state["utterance"] == "stable":
+ out.extend(semantic_scorer([(state, questions)]))
+ else:
+ out.append([label_weights[k] for k in questions["intent"]["criteria"]])
+ return out
+
+ report = m.evaluate([stable, stable, fragile, fragile], score, [0, 0, 1, 1])["report"]
+ sp = report["selective_prediction"]
+ self.assertEqual((sp["n_labelled"], sp["n_wrong"]), (4, 2))
+ self.assertEqual(sp["auroc"]["confidence"], 0)
+ self.assertEqual(sp["auroc"]["agreement_label_rename"], 1)
+ self.assertEqual(sp["auroc"]["support_label_rename"], 1)
+ self.assertEqual(sp["auroc"]["agreement_option_order"], 0.5)
+ self.assertEqual(sp["auroc"]["support_all"], 1)
+ self.assertEqual(sp["accuracy_at_coverage"]["confidence"]["0.5"], 0)
+ self.assertEqual(sp["accuracy_at_coverage"]["support_all"]["0.5"], 1)
+ self.assertNotIn("agreement_label_rename", sp["accuracy_at_coverage"])
+
+ def test_unlabelled_or_one_class(self):
+ report = m.evaluate([CASE], semantic_scorer)["report"]
+ self.assertEqual(report["selective_prediction"], {"n_labelled": 0})
+ sp = m.evaluate([CASE, CASE], semantic_scorer, [0, 0])["report"]["selective_prediction"]
+ self.assertEqual((sp["n_labelled"], sp["n_wrong"]), (2, 0))
+ self.assertIsNone(sp["auroc"]["confidence"])
+ self.assertEqual(sp["accuracy_at_coverage"]["confidence"]["0.5"], 1)
+
+
if __name__ == "__main__":
unittest.main()