Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(evals): a coverage cut must not split a group of tied confidences

Adeshina Lasisi4f8645b35a56
diff --git a/docs/evals.md b/docs/evals.md
index 28e15cff..b53405af 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -133,12 +133,48 @@ Each metric is computed per answer where it applies and aggregated over the data
| `score_within_<tol>` | `score` | fraction within an absolute tolerance |
| `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on `answer["answer_confidence"]`, the calibrated probability Laya reports on every answer type |
| `brier` | any answer with a confidence and a known label | Brier score of confidence as P(correct), `mean((confidence - correct)**2)`; lower is better |
-| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve (mean selective risk over every coverage); lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |
-| `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the most-confident 50% / 80% of answers -- what abstaining on the least-confident tail buys |
+| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve, averaged over the distinct confidence levels; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |
+| `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the answers a confidence threshold at the 50% / 80% coverage point accepts -- what abstaining on the least-confident tail buys. A threshold cannot split a group of equal confidences, so this can cover more than the named fraction; see [coverage cuts](#coverage-cuts-and-ties) |
| `mean_confidence` | any answer with a confidence | mean reported `answer["answer_confidence"]` |
| `latency_p50_ms`, `latency_p95_ms` | per request | wall time each request waited, informational -- see [batching](#batching-and-timing) |
| `cost_per_decision_p50_ms`, `cost_per_decision_p95_ms` | per decision | a call's wall time divided by the rows it carried, informational |
+### Coverage cuts and ties
+
+Both coverage metrics cut on a confidence **threshold**, and a threshold accepts every answer at its
+own confidence. So a cut never splits a group of answers that share one: when `coverage * n` falls
+inside such a group, every member of the group is accepted. The number of answers behind the figure
+is therefore the group's upper edge rather than the named fraction -- `selective_accuracy@50` over a
+slice whose confidences are all equal is that slice's own accuracy, not the better half of it. The
+count the gate prints (`n=` in a rule's failure message) is the slice's size, not the accepted size,
+so a very wide group is not visible from the message alone.
+
+Ties are the normal case rather than a corner: a fitted temperature can leave a bucket reporting a
+point mass, which `laya.common.answer_confidence` records of the shipped `choice:11+`, and a real
+checkpoint produced a six-row group at exactly 1.0 out of twelve answers. Cutting at a row index
+instead made both metrics depend on the order the dataset arrived in -- the same rows, shuffled,
+moved `selective_accuracy@50` between 0.000 and 1.000.
+
+`aurc` integrates one risk value per distinct level, weighted by the answers that level spans, so it
+remains an area under the risk--coverage curve rather than an average of unevenly sized points.
+
+Two consequences worth planning for:
+
+* **A number can move in either direction, by more than a reordering could.** Where a group
+ straddles the cut, the threshold reading differs from every row-index reading of the same data:
+ measured, up to 0.48 for `selective_accuracy@50` and 0.09 for `aurc`. A gate that was passing may
+ fail, and one that was failing may pass; the previous verdict depended on row order.
+* **Regenerate committed baselines.** `config.coverage_metric_definition` records which definition
+ produced a report, and a relative rule (`max_drop` / `max_increase`) on `aurc` or
+ `selective_accuracy@*` refuses a baseline recorded under a different one. Without that refusal a
+ stale baseline hides a real regression: a slice recorded at 0.033 under the old definition reads
+ 0.517 under this one, so a candidate that genuinely dropped 0.217 would clear a `max_drop` of
+ 0.05. `ece` and `brier` do not cut and stay comparable.
+
+With no ties in the data there is one level per answer, and both metrics are exactly what they have
+always been -- bit-identical, not merely close.
+
+
Add `ScoreWithin(0.25)` to the evaluator list for a tolerance metric; the default set is
`choice_accuracy`, `noul_accuracy`, `score_mae`, `mean_confidence`, plus `ece`. From the CLI the
same thing is one flag: `laya-evals run data.jsonl --score-within 0.25` reports `score_within_0.25`
@@ -263,6 +299,7 @@ is reviewable on its own:
| `dataset_sha256` | the sha256 of the dataset bytes that were parsed |
| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset |
| `laya_version` | the `laya` that computed the numbers |
+| `coverage_metric_definition` | which definition of `aurc` / `selective_accuracy@*` produced this report (see [coverage cuts](#coverage-cuts-and-ties)). A relative gate rule on either refuses a baseline recorded under a different one, rather than subtracting numbers that do not mean the same thing |
| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |
| `gate_policy` | the optional slice gate policy applied by `run --gate-policy` |
| `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |
diff --git a/laya/_eval_policy.py b/laya/_eval_policy.py
index 1742d1b3..26f08e8f 100644
--- a/laya/_eval_policy.py
+++ b/laya/_eval_policy.py
@@ -106,6 +106,35 @@ def _value(report: evals.EvalReport, dimension: str, value: str,
return float(result)
+def _stale_coverage_baseline(baseline: Any, metric: str) -> Optional[str]:
+ """Why a relative rule on a coverage metric must not be computed against `baseline`, or None.
+
+ A `max_drop` / `max_increase` rule subtracts the baseline's number from this run's. That is
+ only meaningful if both were measured by the same definition of a coverage cut. Version 2 cuts
+ on a confidence threshold, so a cut never splits a group of equal confidences; version 1 cut at
+ a row index and depended on the order the dataset arrived in. The two disagree by as much as
+ 0.48 on the same rows, so the subtraction is not merely noisy -- it is unsafe in the PASS
+ direction, letting a real regression through.
+
+ The absence of the key IS the old definition: every report written since it existed carries it.
+ That is why this refuses instead of following `comparable_to`'s "a key missing on either side is
+ unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a
+ committed baseline. A definition change is not a patch release.
+
+ `ece` and `brier` are untouched by the cut, so a rule on either keeps comparing as before.
+ """
+ if not evals.is_coverage_metric(metric):
+ return None
+ config = baseline.get("config") if isinstance(baseline, dict) else None
+ recorded = config.get("coverage_metric_definition") if isinstance(config, dict) else None
+ if recorded == evals.COVERAGE_METRIC_DEFINITION:
+ return None
+ return ("baseline reports coverage-metric definition %s, this run uses %s -- regenerate the "
+ "baseline before gating %r on a relative limit" %
+ ("1 (unrecorded)" if recorded is None else recorded,
+ evals.COVERAGE_METRIC_DEFINITION, metric))
+
+
def check_policy(report: evals.EvalReport, policy: Dict[str, Any],
baseline: Optional[Dict[str, Any]] = None) -> List[str]:
"""Return actionable failures. Missing measurements never count as a pass."""
@@ -147,6 +176,12 @@ def check_policy(report: evals.EvalReport, policy: Dict[str, Any],
comparator = next(name for name in _LIMITS if name in rule)
limit = rule[comparator]
if comparator in _RELATIVE:
+ # the raw baseline document, not `base_report`: that is rebuilt as an
+ # `EvalReport` and carries only what `_identity_of` kept
+ stale = _stale_coverage_baseline(baseline, metric)
+ if stale is not None:
+ failures.append("%s: %s" % (label, stale))
+ continue
previous = _value(base_report, dimension, value, metric)
previous_count = _count(base_report, dimension, value, metric)
if previous is None:
diff --git a/laya/evals.py b/laya/evals.py
index 163575a9..57987094 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -232,10 +232,26 @@ def _library_version() -> Optional[str]:
return getattr(laya, "__version__", None)
+#: Which definition of the coverage metrics (`aurc`, `selective_accuracy@*`) produced a report.
+#: Bumped when a cut's meaning changes, which is not a schema change -- the keys and their types
+#: are identical, only the arithmetic behind two of them moves. Version 2 cuts on a confidence
+#: THRESHOLD, so a cut never splits a group of equal confidences; version 1 (every report written
+#: before this existed, and recognised by the key's ABSENCE) cut at a row index and therefore
+#: depended on the order the dataset happened to arrive in.
+#:
+#: A relative gate rule subtracts a baseline's number from this run's. Across definitions that
+#: subtraction is meaningless, and it is unsafe in the PASS direction: a baseline recorded at 0.0333
+#: by version 1 -- a value version 2 reports as 0.5167 for the same rows -- lets a candidate that
+#: truly regressed by 0.2167 clear a `max_drop` of 0.05. `_eval_policy` refuses that comparison
+#: rather than performing the arithmetic; see `check_policy`.
+COVERAGE_METRIC_DEFINITION = 2
+
+
def _run_identity(dataset: "Dataset") -> Dict[str, Any]:
"""The `config` keys this module can fill in on its own, with no knowledge of the CLI."""
identity: Dict[str, Any] = {"schema": REPORT_SCHEMA,
- "questions_sha256": questions_fingerprint(dataset)}
+ "questions_sha256": questions_fingerprint(dataset),
+ "coverage_metric_definition": COVERAGE_METRIC_DEFINITION}
version = _library_version()
if version is not None:
identity["laya_version"] = version
@@ -359,18 +375,78 @@ def brier(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[fl
return float(np.mean((c - y) ** 2))
-def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):
- """(coverage, risk) over the most-confident-first ordering; coverage k/n, risk = error@top-k."""
- order = np.argsort(-np.asarray(confidences, dtype=float), kind="mergesort") # desc, stable
- y = np.asarray(corrects, dtype=bool)[order].astype(float)
+def _threshold_accepts(sorted_desc, k: int) -> int:
+ """How many rows a confidence threshold placed at `sorted_desc[k - 1]` accepts.
+
+ The end of the tie group the cut landed in, which is what makes a coverage number independent
+ of the order a dataset happened to arrive in. Never fewer than `k`: a threshold accepts at
+ least the rows the coverage asked for.
+
+ NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN is
+ false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever rows
+ came first" -- the exact order-dependence this exists to remove, for the input a custom runner
+ is most likely to hand over by accident. NaNs sort last and form one group, so a cut inside
+ them accepts all of them.
+ """
+ cut = sorted_desc[k - 1]
+ if cut != cut: # NaN
+ return len(sorted_desc)
+ return max(k, int(np.count_nonzero(sorted_desc >= cut)))
+
+
+def _levels(confidences: Sequence[float], corrects: Sequence[bool]):
+ """`(accepted, risk, n)` at each distinct confidence level, most confident first.
+
+ One row per LEVEL, not per answer. A coverage cut is bought with a confidence threshold, and a
+ threshold accepts every answer at its own confidence -- `calibrate._select_abstention_threshold`
+ already says exactly this about the error it judges ("a tie group's errors must all be counted
+ before the level is judged"), and it is the same rule here.
+
+ Cutting at a row index instead reports whichever tie members the dataset happened to list
+ first. A stable sort makes that reproducible, not order-independent, and ties are the normal
+ case rather than a corner: `answer_confidence`'s own docstring records that the shipped
+ `choice:11+` bucket "returns a point mass at 1.0". Permuting a JSONL of 8 rows, all tied, 4
+ correct, moved `aurc` from 0.1827 to 0.8173 and `selective_accuracy@50` from 1.000 to 0.000.
+
+ `accepted[i]` is how many answers a threshold at level `i` accepts, which is what lets `aurc`
+ weight a level by the answers it spans rather than counting a 1-answer level and a 999-answer
+ level alike.
+ """
+ # `atleast_1d`: a 0-dimensional input (a bare scalar) answered before this change, and indexing
+ # it with an argsort result raises.
+ c = np.atleast_1d(np.asarray(confidences, dtype=float))
+ y = np.atleast_1d(np.asarray(corrects, dtype=bool).astype(float))
n = len(y)
- k = np.arange(1, n + 1)
- risk = 1.0 - np.cumsum(y) / k
- return k / n, risk
+ if n == 0:
+ return np.empty(0), np.empty(0), 0
+ order = np.argsort(-c, kind="mergesort") # desc, stable; NaN sorts last
+ c, y = c[order], y[order]
+ # The last answer of each tie group: where the next confidence differs, plus the final one.
+ #
+ # `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every NaN
+ # confidence becomes a level of its own, the running total walks them in dataset order, and the
+ # order-dependence this exists to remove survives for exactly the input a custom runner is most
+ # likely to hand over by accident. All NaNs are one group, ranked last.
+ nan_pair = np.isnan(c[1:]) & np.isnan(c[:-1])
+ last = np.flatnonzero(np.concatenate(((c[1:] != c[:-1]) & ~nan_pair, [True])))
+ accepted = last.astype(float) + 1.0
+ risk = 1.0 - np.cumsum(y)[last] / accepted
+ return accepted, risk, n
+
+
+def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):
+ """`(coverage, risk)` at each distinct confidence level.
+
+ `_levels` with coverage as the x axis, for callers that want the curve rather than its area.
+ """
+ accepted, risk, n = _levels(confidences, corrects)
+ if n == 0:
+ return np.empty(0), np.empty(0)
+ return accepted / n, risk
def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[float]:
- """Area Under the Risk-Coverage curve (mean selective risk over every coverage). Lower is better.
+ """Area Under the Risk-Coverage curve: selective risk integrated over coverage. Lower is better.
A classifier whose confidence perfectly ranks right from wrong drives AURC toward the overall
error rate's area under the ideal curve; a confidence no better than random leaves it at the
@@ -378,19 +454,42 @@ def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flo
"""
if not confidences:
return None
- _coverage, risk = _risk_coverage(confidences, corrects)
- return float(np.mean(risk))
+ accepted, risk, n = _levels(confidences, corrects)
+ if n == 0:
+ return None
+ # Each level's risk weighted by the ANSWERS it spans, which is what keeps this an area rather
+ # than an average of unevenly sized points: with 999 answers at one confidence and 1 at another,
+ # an unweighted mean of the two risks reports 5e-4 where the area is 1e-6.
+ #
+ # `sum(risk * rows) / n`, not `sum(risk * width)`: with no ties every level holds one answer, so
+ # this reduces to the same pairwise sum over the same denominator and is bit-identical to the
+ # per-answer mean it replaces. Dividing inside the sum instead loses that in the last bits.
+ rows_per_level = np.diff(np.concatenate(([0.0], accepted)))
+ return float(np.sum(risk * rows_per_level) / n)
def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],
coverage: float) -> Optional[float]:
- """Accuracy over the most-confident `coverage` fraction of answers (0 < coverage <= 1)."""
+ """Accuracy over the most-confident `coverage` fraction of answers (0 < coverage <= 1).
+
+ The cut is a confidence threshold, so it cannot land inside a group of equal confidences: when
+ `coverage * n` falls within one, every member of that group is included and the reported
+ coverage is the group's upper edge rather than the fraction asked for. That is the only
+ order-independent reading -- see `_risk_coverage` -- and it is what makes this comparable
+ across two permutations of the same dataset. Without ties, `k` is unchanged.
+ """
if not confidences or not 0.0 < coverage <= 1.0:
return None
import math as _math
- order = np.argsort(-np.asarray(confidences, dtype=float), kind="mergesort")
- y = np.asarray(corrects, dtype=bool)[order].astype(float)
+ c = np.atleast_1d(np.asarray(confidences, dtype=float))
+ order = np.argsort(-c, kind="mergesort")
+ c = c[order]
+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order].astype(float)
k = max(1, int(_math.ceil(coverage * len(y))))
+ # Extend through the tie group the cut fell in -- see `_threshold_accepts`, which both this
+ # and the risk-coverage curve go through, so the two cannot disagree about what a cut
+ # accepts, and neither falls back to dataset order when the cut lands on a NaN.
+ k = _threshold_accepts(c, k)
return float(np.mean(y[:k]))
@@ -398,6 +497,15 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],
SELECTIVE_COVERAGES = (0.5, 0.8)
+def is_coverage_metric(name: str) -> bool:
+ """True for a metric whose value depends on where a coverage cut falls.
+
+ `ece` and `brier` read the same `(confidence, correct)` pairs but do not cut, so a change to
+ what a cut accepts cannot move them and a baseline that predates it stays comparable.
+ """
+ return name == "aurc" or name.startswith("selective_accuracy@")
+
+
def is_confidence_metric(name: str) -> bool:
"""True for a dataset-level metric computed from `(confidence, correct)` pairs, not per-answer
`scores` -- ECE and the selective-classification metrics. Policy counting treats these alike."""
diff --git a/tests/test_evals.py b/tests/test_evals.py
index 24218036..ea9a80f5 100644
--- a/tests/test_evals.py
+++ b/tests/test_evals.py
@@ -4,6 +4,9 @@ Run: python -m pytest tests/test_evals.py -q
"""
import json
import re
+import warnings
+
+import numpy as np
import pytest
@@ -118,6 +121,295 @@ def test_selective_metrics_on_known_inputs():
assert not is_confidence_metric("choice_accuracy")
+def test_coverage_cut_does_not_split_a_tie_group():
+ """A coverage cut is a confidence threshold, so permuting the dataset cannot move it.
+
+ Before this was fixed, a stable sort made the result reproducible but not order-independent:
+ `y[:k]` took whichever tie members the dataset listed first. Ties are the normal case, not a
+ corner -- `common.answer_confidence`'s own docstring records that the shipped `choice:11+`
+ bucket "returns a point mass at 1.0".
+ """
+ conf = [1.0] * 8
+ correct_first = [True, True, True, True, False, False, False, False]
+ wrong_first = [False, False, False, False, True, True, True, True]
+ interleaved = [True, False, True, False, True, False, True, False]
+ # 8 rows, 4 correct, every confidence identical: the only honest answer is the group's own
+ # accuracy, whatever order the rows arrive in. Before: 1.000 / 0.000 / 0.500 for sel@50.
+ for y in (correct_first, wrong_first, interleaved):
+ assert selective_accuracy(conf, y, 0.5) == pytest.approx(0.5)
+ assert selective_accuracy(conf, y, 0.8) == pytest.approx(0.5)
+ assert aurc(conf, y) == pytest.approx(0.5)
+ # `brier` and `ece` do not cut, so a change to what a cut accepts cannot move them. Asserted on
+ # DISTINCT, deliberately unsorted confidences and by comparing two orderings to each other: on
+ # the all-tied data above, any confidence-sorting rewrite is the identity, so a mutant that
+ # permuted the rows by confidence passed every one of these.
+ jumbled_conf = [0.3, 0.95, 0.6, 0.1, 0.8]
+ jumbled_corr = [False, True, True, False, False]
+ order = [3, 0, 4, 2, 1]
+ other_conf = [jumbled_conf[i] for i in order]
+ other_corr = [jumbled_corr[i] for i in order]
+ assert brier(jumbled_conf, jumbled_corr) == pytest.approx(
+ brier(other_conf, other_corr), abs=1e-12)
+ assert ece(jumbled_conf, jumbled_corr) == pytest.approx(
+ ece(other_conf, other_corr), abs=1e-12)
+ # and the values themselves, so a rewrite that made both sides equally wrong is still caught
+ assert brier(jumbled_conf, jumbled_corr) == pytest.approx(
+ sum((c - y) ** 2 for c, y in zip(jumbled_conf, jumbled_corr)) / 5, abs=1e-12)
+
+
+def test_coverage_cut_extends_through_the_tie_it_lands_in():
+ """The threshold is the confidence at the cut, and it accepts every row at that confidence."""
+ # two groups: 0.9 (both correct) and 0.5 (two correct of four). The FIRST member of the 0.5
+ # group is wrong on purpose: with a correct one there, accepting a row below the threshold
+ # cannot change the mean, and an implementation that over-accepts passes unnoticed.
+ conf = [0.9, 0.9, 0.5, 0.5, 0.5, 0.5]
+ corr = [True, True, False, True, True, False]
+ # coverage 1/3 -> k = ceil(2.0) = 2, the cut is 0.9, and nothing at 0.5 is accepted. If one
+ # were, this would be 2/3 rather than 1.0.
+ assert selective_accuracy(conf, corr, 1 / 3) == pytest.approx(1.0)
+ # coverage 0.5 -> k = ceil(3.0) = 3, which lands inside the 0.5 group, so the threshold is
+ # 0.5 and all six rows are accepted: 4 correct of 6. The figure therefore covers the group's
+ # upper edge (6 answers) rather than the 3 asked for -- the only order-independent reading.
+ assert selective_accuracy(conf, corr, 0.5) == pytest.approx(4 / 6)
+ assert selective_accuracy(conf, corr, 0.8) == pytest.approx(4 / 6)
+ # aurc integrates one risk per distinct level over the ROWS that level spans: 0.9 -> 2 rows
+ # accepted, 0 errors, risk 0; 0.5 -> 6 accepted, 2 errors, risk 2/6, spanning the other 4 rows.
+ # Weighted, not averaged: an unweighted mean of the two risks would weight a 2-row level the
+ # same as a 4-row one, which is how a 999-row level and a 1-row level came to count alike.
+ assert aurc(conf, corr) == pytest.approx((0.0 * 2 + (2 / 6) * 4) / 6)
+
+
+def test_coverage_metrics_are_unchanged_without_ties():
+ """Distinct confidences must produce exactly what they always did, to the last bit.
+
+ The fix is only allowed to move a number a permutation of the same dataset could already
+ move. With no ties there is one level per row, so every point of the risk-coverage curve --
+ and `k` itself -- is what it was before.
+ """
+ conf, corr = [0.9, 0.8, 0.2, 0.1], [True, True, False, False]
+ # the hand-computed values from `test_selective_metrics_on_known_inputs`, restated as floats
+ # so this fails if the definition drifts rather than if that test is edited
+ assert aurc(conf, corr) == pytest.approx((0 + 0 + 1 / 3 + 1 / 2) / 4, abs=1e-12)
+ assert selective_accuracy(conf, corr, 0.5) == pytest.approx(1.0, abs=1e-12)
+ assert selective_accuracy(conf, corr, 0.75) == pytest.approx(2 / 3, abs=1e-12)
+ assert selective_accuracy(conf, corr, 1.0) == pytest.approx(0.5, abs=1e-12)
+ # a single row, and a coverage that rounds below one row, still report that row
+ assert selective_accuracy([0.4], [True], 0.01) == pytest.approx(1.0)
+ assert aurc([0.4], [False]) == pytest.approx(1.0)
+ # `coverage * n` is rarely an integer, and the fraction rounds UP so the answer covers at
+ # least the fraction asked for. Five rows at 0.5 means three, not two -- every case above
+ # happens to land on a whole row, so nothing here could see a floor in place of the ceiling.
+ five, corr5 = [0.9, 0.8, 0.7, 0.6, 0.5], [True, True, False, True, True]
+ assert selective_accuracy(five, corr5, 0.5) == pytest.approx(2 / 3) # ceil(2.5) = 3 rows
+ # 0.3 -> ceil(1.5) = 2 rows. The second row is WRONG, so a floor (1 row) would report 1.0.
+ five_mixed = [0.9, 0.8, 0.7, 0.6, 0.5]
+ corr_mixed = [True, False, True, True, True]
+ assert selective_accuracy(five_mixed, corr_mixed, 0.3) == pytest.approx(0.5)
+ assert selective_accuracy(five, corr5, 0.7) == pytest.approx(3 / 4) # ceil(3.5) = 4 rows
+
+
+def test_non_finite_confidences_are_order_independent_too():
+ """A NaN confidence must not reintroduce the order-dependence this change removes.
+
+ Confidences come from whatever a runner returns, so NaN and infinities are reachable --
+ `_eval_policy._finite_number` exists because this codebase already expects that. `NaN != NaN`,
+ so a boundary test written on `!=` alone would make every NaN row its own level and walk them
+ in dataset order: the headline defect, surviving for exactly the input a custom runner is most
+ likely to hand over by accident. All NaNs are one group, ranked last.
+
+ These VALUES differ from what the old definition reported. They have to: the old ones were
+ whichever rows the dataset listed first. `docs/evals.md` says so.
+ """
+ nan, inf = float("nan"), float("inf")
+ with warnings.catch_warnings():
+ warnings.simplefilter("error") # a RuntimeWarning here fails the test
+ # all NaN: one group, so every coverage reports the group's own accuracy
+ for y in ([True, False, True], [True, True, False], [False, True, True]):
+ assert selective_accuracy([nan] * 3, y, 0.5) == pytest.approx(2 / 3)
+ assert aurc([nan] * 3, y) == pytest.approx(1 / 3)
+ # a NaN among finite values: the finite ones rank above it, the NaN group below
+ assert selective_accuracy([0.9, nan, 0.5], [True, False, True], 0.5) == pytest.approx(1.0)
+ # infinities are ordinary numbers to a comparison, and keep ranking normally
+ assert selective_accuracy([inf, 0.5, -inf], [True, True, False], 0.5) == pytest.approx(1.0)
+ assert aurc([0.9, 0.7, nan, 0.2], [True, False, True, True]) == pytest.approx(
+ (0 * 1 + 0.5 * 1 + (1 - 2 / 3) * 1 + (1 - 3 / 4) * 1) / 4, abs=1e-12)
+
+ # and the whole report, through `evaluate()`, with only the NaN rows reordered
+ def report_for(rows):
+ examples = [Example("s%d" % i, Q, {"intent": "a"}) for i, _ in enumerate(rows)]
+ answers = {"s%d" % i: {"intent": dict(choice_answer("a" if ok else "b", conf),
+ answer_confidence=conf)}
+ for i, (conf, ok) in enumerate(rows)}
+ rep = evaluate(StubRunner(answers), Dataset(examples), evaluators=[ChoiceAccuracy()])
+ return {n: rep.overall[n] for n in ("aurc", "selective_accuracy@50",
+ "selective_accuracy@80")}
+
+ finite = [(0.9, True), (0.8, True), (0.7, True), (0.6, True)]
+ block = [(nan, True), (nan, False), (nan, True), (nan, False), (nan, True), (nan, False)]
+ assert report_for(finite + block) == report_for(finite + block[::-1])
+
+
+def test_confidences_group_on_exact_equality():
+ """Two confidences that differ by one ULP are two levels, not one.
+
+ A threshold comparison is exact, so grouping has to be too. Nothing else in this file puts two
+ confidences closer together than 0.1, which let a tolerance window or an `isclose` boundary
+ pass every check while changing which answers a cut accepts. The shipped agents round
+ confidence to 4 decimals, so this is reachable only from a custom runner -- which is also true
+ of every NaN case above.
+ """
+ one = 1.0
+ just_below = 1.0 - 2 ** -52 # the next double below 1.0
+ assert just_below != one and round(just_below, 4) == round(one, 4)
+ # two levels: the correct answer ranks first and the 50% cut takes only it
+ assert selective_accuracy([one, just_below], [True, False], 0.5) == pytest.approx(1.0)
+ assert aurc([one, just_below], [True, False]) == pytest.approx((0.0 * 1 + 0.5 * 1) / 2)
+ # one level, were they treated as equal, would report the pair's own accuracy instead
+ assert selective_accuracy([one, one], [True, False], 0.5) == pytest.approx(0.5)
+
+
+def test_corrects_are_read_as_booleans():
+ """`corrects` is a correctness flag, not a count.
+
+ `_levels` reads it through `dtype=bool`; reading it as a float lets a value outside {0, 1}
+ produce a NEGATIVE risk, and `brier` already reads the same argument as a float, so the two
+ readings were unpinned and mutually inconsistent.
+ """
+ assert aurc([0.9, 0.8], [2, 0]) == pytest.approx(aurc([0.9, 0.8], [True, False]))
+ assert aurc([0.9, 0.8], [2, 0]) >= 0.0
+ assert selective_accuracy([0.9, 0.8], [2, 0], 1.0) == pytest.approx(0.5)
+
+
+def test_a_0d_confidence_still_answers():
+ """A bare scalar used to answer, and indexing it with an argsort result raises.
+
+ Reachable only from the library entry point, and `_aggregate` always builds lists -- but an
+ `IndexError` escaping `laya-evals` exits 1, which is the code that means "the model regressed".
+ """
+ assert selective_accuracy(np.float64(0.5), np.True_, 0.5) == pytest.approx(1.0)
+ assert aurc(np.float64(0.5), np.False_) == pytest.approx(1.0)
+
+
+def test_report_metrics_survive_a_permuted_dataset():
+ """The gate reads these through `evaluate()`, so permuting a JSONL must not move them.
+
+ `_eval_policy.check_policy` gates a slice on whatever metric a policy names, and
+ `evals.is_confidence_metric` recognises `aurc` and `selective_accuracy@*` by name -- so a
+ coverage metric that depended on row order meant a shuffled dataset could pass or fail a
+ release gate on nothing at all.
+ """
+ examples = [Example("s%d" % i, Q, {"intent": "a"}) for i in range(1, 9)]
+ # every answer at the same confidence, half of them wrong: the tie case, through the report
+ answers = {"s%d" % i: {"intent": choice_answer("a" if i <= 4 else "b", 1.0)}
+ for i in range(1, 9)}
+ names = ("aurc", "selective_accuracy@50", "selective_accuracy@80", "ece", "brier")
+
+ def metrics(order):
+ report = evaluate(StubRunner(answers), Dataset([examples[i] for i in order]),
+ evaluators=[ChoiceAccuracy()])
+ return {n: report.overall[n] for n in names}
+
+ forward = metrics(range(8))
+ assert metrics(range(7, -1, -1)) == forward
+ assert metrics([4, 0, 5, 1, 6, 2, 7, 3]) == forward
+ # and the values are the tie group's own accuracy, not whichever half came first
+ assert forward["selective_accuracy@50"] == pytest.approx(0.5)
+ assert forward["aurc"] == pytest.approx(0.5)
+
+
+def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmp_path):
+ """A `max_drop` across definitions is unsafe in the PASS direction, so it must not be computed.
+
+ The two definitions disagree by as much as 0.48 on the same answers. A slice recorded at 0.033
+ by the old one reads 0.517 under this one, so subtracting them lets a candidate that genuinely
+ dropped 0.217 clear a `max_drop` of 0.05 -- a real regression shipping because a baseline is
+ stale. `comparable_to` cannot see it: the dataset bytes and the question schema are identical,
+ and its "a key missing on either side is unknown" rule exists so a patch release cannot
+ invalidate a baseline. A definition change is not a patch release, so this refuses on the
+ key's ABSENCE.
+ """
+ from laya import _eval_policy
+
+ policy = _eval_policy.load_policy(_write_gate_policy(
+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",
+ "min_count": 5, "max_drop": 0.05}))
+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))
+
+ stale = _coverage_gate_report(0.033333, None)
+ failures = _eval_policy.check_policy(candidate, policy, stale)
+ assert any("regenerate the baseline" in f for f in failures), failures
+ assert any("definition 1 (unrecorded)" in f for f in failures), failures
+ # an explicit version 1 is refused the same way
+ assert any("regenerate the baseline" in f for f in
+ _eval_policy.check_policy(candidate, policy, _coverage_gate_report(0.033333, 1)))
+
+ # regenerated under this definition, the same gate CATCHES the regression it was hiding
+ fresh = _coverage_gate_report(0.516667, evals.COVERAGE_METRIC_DEFINITION)
+ failures = _eval_policy.check_policy(candidate, policy, fresh)
+ assert any("max_drop" in f and "exceeded" in f for f in failures), failures
+ assert not any("regenerate" in f for f in failures), failures
+
+
+def test_the_refusal_is_limited_to_metrics_a_cut_can_move(tmp_path):
+ """`ece` and `brier` do not cut, so a baseline that predates the change stays comparable.
+
+ Sweeping them into the refusal would break every committed baseline for no reason.
+ """
+ from laya import _eval_policy
+
+ candidate = EvalReport(**_coverage_gate_report(0.10, evals.COVERAGE_METRIC_DEFINITION,
+ metric="ece"))
+ stale = _coverage_gate_report(0.11, None, metric="ece")
+ policy = _eval_policy.load_policy(_write_gate_policy(
+ tmp_path, {"slice": {"language": "zh"}, "metric": "ece",
+ "min_count": 5, "max_drop": 0.05}))
+ assert _eval_policy.check_policy(candidate, policy, stale) == []
+ assert evals.is_coverage_metric("aurc")
+ assert evals.is_coverage_metric("selective_accuracy@80")
+ assert not evals.is_coverage_metric("ece")
+ assert not evals.is_coverage_metric("brier")
+ assert not evals.is_coverage_metric("choice_accuracy")
+
+
+def test_an_absolute_rule_needs_no_baseline_and_is_unaffected(tmp_path):
+ """Only the relative comparison is across definitions; a `min` reads this run alone."""
+ from laya import _eval_policy
+
+ policy = _eval_policy.load_policy(_write_gate_policy(
+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",
+ "min_count": 5, "min": 0.2}))
+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))
+ assert _eval_policy.check_policy(candidate, policy, None) == []
+
+
+def test_every_report_records_which_coverage_definition_produced_it():
+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}),
+ Dataset([Example("s1", Q, {"intent": "a"})]))
+ assert report.config["coverage_metric_definition"] == evals.COVERAGE_METRIC_DEFINITION
+ assert evals.COVERAGE_METRIC_DEFINITION == 2
+
+
+def test_aurc_is_bit_identical_to_the_per_answer_mean_without_ties():
+ """Not `approx`: the claim is bit-identity, and only an exact comparison can hold it.
+
+ With no ties every level holds one answer, so weighting by rows and dividing once at the end
+ reduces to the same pairwise sum over the same denominator. Weighting by the coverage width
+ instead -- mathematically the same area -- loses the last bits on about half of all datasets,
+ which would quietly move every committed number that has no tie in it.
+ """
+ import random
+
+ random.seed(5)
+ for _ in range(200):
+ n = random.randint(1, 40)
+ conf = random.sample([i / 10000 for i in range(1, 9999)], n)
+ corr = [random.random() < 0.6 for _ in range(n)]
+ order = np.argsort(-np.asarray(conf, dtype=float), kind="mergesort")
+ y = np.asarray(corr, dtype=bool)[order].astype(float)
+ per_answer = 1.0 - np.cumsum(y) / np.arange(1, n + 1)
+ assert aurc(conf, corr) == float(np.mean(per_answer))
+
+
def test_selective_metrics_reach_the_report():
# choice answers with a spread of confidence and correctness so the pairs exist
ds = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"}),
@@ -1751,6 +2043,22 @@ def _slice_gate_report(en_correct, zh_correct):
"cases": cases}
+def _coverage_gate_report(value, definition, metric="selective_accuracy@50"):
+ """A slice-gate report carrying a coverage metric and a definition stamp.
+
+ `definition=None` is a report written before `coverage_metric_definition` existed, which is
+ exactly what every committed baseline is.
+ """
+ cases = [{"language": "zh", "scores": {}, "confidence": 0.8, "correct": True}
+ for _ in range(20)]
+ config = {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "a" * 64,
+ "questions_sha256": "b" * 64}
+ if definition is not None:
+ config["coverage_metric_definition"] = definition
+ return {"config": config, "overall": {metric: value},
+ "slices": {"language": {"zh": {metric: value}}}, "cases": cases}
+
+
def _write_gate_policy(tmp_path, rule):
path = tmp_path / "gates.json"
path.write_text(json.dumps({"version": 1, "rules": [rule]}), encoding="utf-8")