Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(evals): a coverage cut must not split a group of tied confidences
Adeshina Lasisi4f8645b35a56
diff --git a/docs/evals.md b/docs/evals.mdindex 28e15cff..b53405af 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -133,12 +133,48 @@ Each metric is computed per answer where it applies and aggregated over the data| `score_within_<tol>` | `score` | fraction within an absolute tolerance || `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on `answer["answer_confidence"]`, the calibrated probability Laya reports on every answer type || `brier` | any answer with a confidence and a known label | Brier score of confidence as P(correct), `mean((confidence - correct)**2)`; lower is better |-| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve (mean selective risk over every coverage); lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |-| `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the most-confident 50% / 80% of answers -- what abstaining on the least-confident tail buys |+| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve, averaged over the distinct confidence levels; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |+| `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the answers a confidence threshold at the 50% / 80% coverage point accepts -- what abstaining on the least-confident tail buys. A threshold cannot split a group of equal confidences, so this can cover more than the named fraction; see [coverage cuts](#coverage-cuts-and-ties) || `mean_confidence` | any answer with a confidence | mean reported `answer["answer_confidence"]` || `latency_p50_ms`, `latency_p95_ms` | per request | wall time each request waited, informational -- see [batching](#batching-and-timing) || `cost_per_decision_p50_ms`, `cost_per_decision_p95_ms` | per decision | a call's wall time divided by the rows it carried, informational |+### Coverage cuts and ties++Both coverage metrics cut on a confidence **threshold**, and a threshold accepts every answer at its+own confidence. So a cut never splits a group of answers that share one: when `coverage * n` falls+inside such a group, every member of the group is accepted. The number of answers behind the figure+is therefore the group's upper edge rather than the named fraction -- `selective_accuracy@50` over a+slice whose confidences are all equal is that slice's own accuracy, not the better half of it. The+count the gate prints (`n=` in a rule's failure message) is the slice's size, not the accepted size,+so a very wide group is not visible from the message alone.++Ties are the normal case rather than a corner: a fitted temperature can leave a bucket reporting a+point mass, which `laya.common.answer_confidence` records of the shipped `choice:11+`, and a real+checkpoint produced a six-row group at exactly 1.0 out of twelve answers. Cutting at a row index+instead made both metrics depend on the order the dataset arrived in -- the same rows, shuffled,+moved `selective_accuracy@50` between 0.000 and 1.000.++`aurc` integrates one risk value per distinct level, weighted by the answers that level spans, so it+remains an area under the risk--coverage curve rather than an average of unevenly sized points.++Two consequences worth planning for:++* **A number can move in either direction, by more than a reordering could.** Where a group+ straddles the cut, the threshold reading differs from every row-index reading of the same data:+ measured, up to 0.48 for `selective_accuracy@50` and 0.09 for `aurc`. A gate that was passing may+ fail, and one that was failing may pass; the previous verdict depended on row order.+* **Regenerate committed baselines.** `config.coverage_metric_definition` records which definition+ produced a report, and a relative rule (`max_drop` / `max_increase`) on `aurc` or+ `selective_accuracy@*` refuses a baseline recorded under a different one. Without that refusal a+ stale baseline hides a real regression: a slice recorded at 0.033 under the old definition reads+ 0.517 under this one, so a candidate that genuinely dropped 0.217 would clear a `max_drop` of+ 0.05. `ece` and `brier` do not cut and stay comparable.++With no ties in the data there is one level per answer, and both metrics are exactly what they have+always been -- bit-identical, not merely close.++Add `ScoreWithin(0.25)` to the evaluator list for a tolerance metric; the default set is`choice_accuracy`, `noul_accuracy`, `score_mae`, `mean_confidence`, plus `ece`. From the CLI thesame thing is one flag: `laya-evals run data.jsonl --score-within 0.25` reports `score_within_0.25`@@ -263,6 +299,7 @@ is reviewable on its own:| `dataset_sha256` | the sha256 of the dataset bytes that were parsed || `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset || `laya_version` | the `laya` that computed the numbers |+| `coverage_metric_definition` | which definition of `aurc` / `selective_accuracy@*` produced this report (see [coverage cuts](#coverage-cuts-and-ties)). A relative gate rule on either refuses a baseline recorded under a different one, rather than subtracting numbers that do not mean the same thing || `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` || `gate_policy` | the optional slice gate policy applied by `run --gate-policy` || `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |diff --git a/laya/_eval_policy.py b/laya/_eval_policy.pyindex 1742d1b3..26f08e8f 100644--- a/laya/_eval_policy.py+++ b/laya/_eval_policy.py@@ -106,6 +106,35 @@ def _value(report: evals.EvalReport, dimension: str, value: str,return float(result)+def _stale_coverage_baseline(baseline: Any, metric: str) -> Optional[str]:+ """Why a relative rule on a coverage metric must not be computed against `baseline`, or None.++ A `max_drop` / `max_increase` rule subtracts the baseline's number from this run's. That is+ only meaningful if both were measured by the same definition of a coverage cut. Version 2 cuts+ on a confidence threshold, so a cut never splits a group of equal confidences; version 1 cut at+ a row index and depended on the order the dataset arrived in. The two disagree by as much as+ 0.48 on the same rows, so the subtraction is not merely noisy -- it is unsafe in the PASS+ direction, letting a real regression through.++ The absence of the key IS the old definition: every report written since it existed carries it.+ That is why this refuses instead of following `comparable_to`'s "a key missing on either side is+ unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a+ committed baseline. A definition change is not a patch release.++ `ece` and `brier` are untouched by the cut, so a rule on either keeps comparing as before.+ """+ if not evals.is_coverage_metric(metric):+ return None+ config = baseline.get("config") if isinstance(baseline, dict) else None+ recorded = config.get("coverage_metric_definition") if isinstance(config, dict) else None+ if recorded == evals.COVERAGE_METRIC_DEFINITION:+ return None+ return ("baseline reports coverage-metric definition %s, this run uses %s -- regenerate the "+ "baseline before gating %r on a relative limit" %+ ("1 (unrecorded)" if recorded is None else recorded,+ evals.COVERAGE_METRIC_DEFINITION, metric))++def check_policy(report: evals.EvalReport, policy: Dict[str, Any],baseline: Optional[Dict[str, Any]] = None) -> List[str]:"""Return actionable failures. Missing measurements never count as a pass."""@@ -147,6 +176,12 @@ def check_policy(report: evals.EvalReport, policy: Dict[str, Any],comparator = next(name for name in _LIMITS if name in rule)limit = rule[comparator]if comparator in _RELATIVE:+ # the raw baseline document, not `base_report`: that is rebuilt as an+ # `EvalReport` and carries only what `_identity_of` kept+ stale = _stale_coverage_baseline(baseline, metric)+ if stale is not None:+ failures.append("%s: %s" % (label, stale))+ continueprevious = _value(base_report, dimension, value, metric)previous_count = _count(base_report, dimension, value, metric)if previous is None:diff --git a/laya/evals.py b/laya/evals.pyindex 163575a9..57987094 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -232,10 +232,26 @@ def _library_version() -> Optional[str]:return getattr(laya, "__version__", None)+#: Which definition of the coverage metrics (`aurc`, `selective_accuracy@*`) produced a report.+#: Bumped when a cut's meaning changes, which is not a schema change -- the keys and their types+#: are identical, only the arithmetic behind two of them moves. Version 2 cuts on a confidence+#: THRESHOLD, so a cut never splits a group of equal confidences; version 1 (every report written+#: before this existed, and recognised by the key's ABSENCE) cut at a row index and therefore+#: depended on the order the dataset happened to arrive in.+#:+#: A relative gate rule subtracts a baseline's number from this run's. Across definitions that+#: subtraction is meaningless, and it is unsafe in the PASS direction: a baseline recorded at 0.0333+#: by version 1 -- a value version 2 reports as 0.5167 for the same rows -- lets a candidate that+#: truly regressed by 0.2167 clear a `max_drop` of 0.05. `_eval_policy` refuses that comparison+#: rather than performing the arithmetic; see `check_policy`.+COVERAGE_METRIC_DEFINITION = 2++def _run_identity(dataset: "Dataset") -> Dict[str, Any]:"""The `config` keys this module can fill in on its own, with no knowledge of the CLI."""identity: Dict[str, Any] = {"schema": REPORT_SCHEMA,- "questions_sha256": questions_fingerprint(dataset)}+ "questions_sha256": questions_fingerprint(dataset),+ "coverage_metric_definition": COVERAGE_METRIC_DEFINITION}version = _library_version()if version is not None:identity["laya_version"] = version@@ -359,18 +375,78 @@ def brier(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flreturn float(np.mean((c - y) ** 2))-def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):- """(coverage, risk) over the most-confident-first ordering; coverage k/n, risk = error@top-k."""- order = np.argsort(-np.asarray(confidences, dtype=float), kind="mergesort") # desc, stable- y = np.asarray(corrects, dtype=bool)[order].astype(float)+def _threshold_accepts(sorted_desc, k: int) -> int:+ """How many rows a confidence threshold placed at `sorted_desc[k - 1]` accepts.++ The end of the tie group the cut landed in, which is what makes a coverage number independent+ of the order a dataset happened to arrive in. Never fewer than `k`: a threshold accepts at+ least the rows the coverage asked for.++ NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN is+ false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever rows+ came first" -- the exact order-dependence this exists to remove, for the input a custom runner+ is most likely to hand over by accident. NaNs sort last and form one group, so a cut inside+ them accepts all of them.+ """+ cut = sorted_desc[k - 1]+ if cut != cut: # NaN+ return len(sorted_desc)+ return max(k, int(np.count_nonzero(sorted_desc >= cut)))+++def _levels(confidences: Sequence[float], corrects: Sequence[bool]):+ """`(accepted, risk, n)` at each distinct confidence level, most confident first.++ One row per LEVEL, not per answer. A coverage cut is bought with a confidence threshold, and a+ threshold accepts every answer at its own confidence -- `calibrate._select_abstention_threshold`+ already says exactly this about the error it judges ("a tie group's errors must all be counted+ before the level is judged"), and it is the same rule here.++ Cutting at a row index instead reports whichever tie members the dataset happened to list+ first. A stable sort makes that reproducible, not order-independent, and ties are the normal+ case rather than a corner: `answer_confidence`'s own docstring records that the shipped+ `choice:11+` bucket "returns a point mass at 1.0". Permuting a JSONL of 8 rows, all tied, 4+ correct, moved `aurc` from 0.1827 to 0.8173 and `selective_accuracy@50` from 1.000 to 0.000.++ `accepted[i]` is how many answers a threshold at level `i` accepts, which is what lets `aurc`+ weight a level by the answers it spans rather than counting a 1-answer level and a 999-answer+ level alike.+ """+ # `atleast_1d`: a 0-dimensional input (a bare scalar) answered before this change, and indexing+ # it with an argsort result raises.+ c = np.atleast_1d(np.asarray(confidences, dtype=float))+ y = np.atleast_1d(np.asarray(corrects, dtype=bool).astype(float))n = len(y)- k = np.arange(1, n + 1)- risk = 1.0 - np.cumsum(y) / k- return k / n, risk+ if n == 0:+ return np.empty(0), np.empty(0), 0+ order = np.argsort(-c, kind="mergesort") # desc, stable; NaN sorts last+ c, y = c[order], y[order]+ # The last answer of each tie group: where the next confidence differs, plus the final one.+ #+ # `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every NaN+ # confidence becomes a level of its own, the running total walks them in dataset order, and the+ # order-dependence this exists to remove survives for exactly the input a custom runner is most+ # likely to hand over by accident. All NaNs are one group, ranked last.+ nan_pair = np.isnan(c[1:]) & np.isnan(c[:-1])+ last = np.flatnonzero(np.concatenate(((c[1:] != c[:-1]) & ~nan_pair, [True])))+ accepted = last.astype(float) + 1.0+ risk = 1.0 - np.cumsum(y)[last] / accepted+ return accepted, risk, n+++def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):+ """`(coverage, risk)` at each distinct confidence level.++ `_levels` with coverage as the x axis, for callers that want the curve rather than its area.+ """+ accepted, risk, n = _levels(confidences, corrects)+ if n == 0:+ return np.empty(0), np.empty(0)+ return accepted / n, riskdef aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[float]:- """Area Under the Risk-Coverage curve (mean selective risk over every coverage). Lower is better.+ """Area Under the Risk-Coverage curve: selective risk integrated over coverage. Lower is better.A classifier whose confidence perfectly ranks right from wrong drives AURC toward the overallerror rate's area under the ideal curve; a confidence no better than random leaves it at the@@ -378,19 +454,42 @@ def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flo"""if not confidences:return None- _coverage, risk = _risk_coverage(confidences, corrects)- return float(np.mean(risk))+ accepted, risk, n = _levels(confidences, corrects)+ if n == 0:+ return None+ # Each level's risk weighted by the ANSWERS it spans, which is what keeps this an area rather+ # than an average of unevenly sized points: with 999 answers at one confidence and 1 at another,+ # an unweighted mean of the two risks reports 5e-4 where the area is 1e-6.+ #+ # `sum(risk * rows) / n`, not `sum(risk * width)`: with no ties every level holds one answer, so+ # this reduces to the same pairwise sum over the same denominator and is bit-identical to the+ # per-answer mean it replaces. Dividing inside the sum instead loses that in the last bits.+ rows_per_level = np.diff(np.concatenate(([0.0], accepted)))+ return float(np.sum(risk * rows_per_level) / n)def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],coverage: float) -> Optional[float]:- """Accuracy over the most-confident `coverage` fraction of answers (0 < coverage <= 1)."""+ """Accuracy over the most-confident `coverage` fraction of answers (0 < coverage <= 1).++ The cut is a confidence threshold, so it cannot land inside a group of equal confidences: when+ `coverage * n` falls within one, every member of that group is included and the reported+ coverage is the group's upper edge rather than the fraction asked for. That is the only+ order-independent reading -- see `_risk_coverage` -- and it is what makes this comparable+ across two permutations of the same dataset. Without ties, `k` is unchanged.+ """if not confidences or not 0.0 < coverage <= 1.0:return Noneimport math as _math- order = np.argsort(-np.asarray(confidences, dtype=float), kind="mergesort")- y = np.asarray(corrects, dtype=bool)[order].astype(float)+ c = np.atleast_1d(np.asarray(confidences, dtype=float))+ order = np.argsort(-c, kind="mergesort")+ c = c[order]+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order].astype(float)k = max(1, int(_math.ceil(coverage * len(y))))+ # Extend through the tie group the cut fell in -- see `_threshold_accepts`, which both this+ # and the risk-coverage curve go through, so the two cannot disagree about what a cut+ # accepts, and neither falls back to dataset order when the cut lands on a NaN.+ k = _threshold_accepts(c, k)return float(np.mean(y[:k]))@@ -398,6 +497,15 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],SELECTIVE_COVERAGES = (0.5, 0.8)+def is_coverage_metric(name: str) -> bool:+ """True for a metric whose value depends on where a coverage cut falls.++ `ece` and `brier` read the same `(confidence, correct)` pairs but do not cut, so a change to+ what a cut accepts cannot move them and a baseline that predates it stays comparable.+ """+ return name == "aurc" or name.startswith("selective_accuracy@")++def is_confidence_metric(name: str) -> bool:"""True for a dataset-level metric computed from `(confidence, correct)` pairs, not per-answer`scores` -- ECE and the selective-classification metrics. Policy counting treats these alike."""diff --git a/tests/test_evals.py b/tests/test_evals.pyindex 24218036..ea9a80f5 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -4,6 +4,9 @@ Run: python -m pytest tests/test_evals.py -q"""import jsonimport re+import warnings++import numpy as npimport pytest@@ -118,6 +121,295 @@ def test_selective_metrics_on_known_inputs():assert not is_confidence_metric("choice_accuracy")+def test_coverage_cut_does_not_split_a_tie_group():+ """A coverage cut is a confidence threshold, so permuting the dataset cannot move it.++ Before this was fixed, a stable sort made the result reproducible but not order-independent:+ `y[:k]` took whichever tie members the dataset listed first. Ties are the normal case, not a+ corner -- `common.answer_confidence`'s own docstring records that the shipped `choice:11+`+ bucket "returns a point mass at 1.0".+ """+ conf = [1.0] * 8+ correct_first = [True, True, True, True, False, False, False, False]+ wrong_first = [False, False, False, False, True, True, True, True]+ interleaved = [True, False, True, False, True, False, True, False]+ # 8 rows, 4 correct, every confidence identical: the only honest answer is the group's own+ # accuracy, whatever order the rows arrive in. Before: 1.000 / 0.000 / 0.500 for sel@50.+ for y in (correct_first, wrong_first, interleaved):+ assert selective_accuracy(conf, y, 0.5) == pytest.approx(0.5)+ assert selective_accuracy(conf, y, 0.8) == pytest.approx(0.5)+ assert aurc(conf, y) == pytest.approx(0.5)+ # `brier` and `ece` do not cut, so a change to what a cut accepts cannot move them. Asserted on+ # DISTINCT, deliberately unsorted confidences and by comparing two orderings to each other: on+ # the all-tied data above, any confidence-sorting rewrite is the identity, so a mutant that+ # permuted the rows by confidence passed every one of these.+ jumbled_conf = [0.3, 0.95, 0.6, 0.1, 0.8]+ jumbled_corr = [False, True, True, False, False]+ order = [3, 0, 4, 2, 1]+ other_conf = [jumbled_conf[i] for i in order]+ other_corr = [jumbled_corr[i] for i in order]+ assert brier(jumbled_conf, jumbled_corr) == pytest.approx(+ brier(other_conf, other_corr), abs=1e-12)+ assert ece(jumbled_conf, jumbled_corr) == pytest.approx(+ ece(other_conf, other_corr), abs=1e-12)+ # and the values themselves, so a rewrite that made both sides equally wrong is still caught+ assert brier(jumbled_conf, jumbled_corr) == pytest.approx(+ sum((c - y) ** 2 for c, y in zip(jumbled_conf, jumbled_corr)) / 5, abs=1e-12)+++def test_coverage_cut_extends_through_the_tie_it_lands_in():+ """The threshold is the confidence at the cut, and it accepts every row at that confidence."""+ # two groups: 0.9 (both correct) and 0.5 (two correct of four). The FIRST member of the 0.5+ # group is wrong on purpose: with a correct one there, accepting a row below the threshold+ # cannot change the mean, and an implementation that over-accepts passes unnoticed.+ conf = [0.9, 0.9, 0.5, 0.5, 0.5, 0.5]+ corr = [True, True, False, True, True, False]+ # coverage 1/3 -> k = ceil(2.0) = 2, the cut is 0.9, and nothing at 0.5 is accepted. If one+ # were, this would be 2/3 rather than 1.0.+ assert selective_accuracy(conf, corr, 1 / 3) == pytest.approx(1.0)+ # coverage 0.5 -> k = ceil(3.0) = 3, which lands inside the 0.5 group, so the threshold is+ # 0.5 and all six rows are accepted: 4 correct of 6. The figure therefore covers the group's+ # upper edge (6 answers) rather than the 3 asked for -- the only order-independent reading.+ assert selective_accuracy(conf, corr, 0.5) == pytest.approx(4 / 6)+ assert selective_accuracy(conf, corr, 0.8) == pytest.approx(4 / 6)+ # aurc integrates one risk per distinct level over the ROWS that level spans: 0.9 -> 2 rows+ # accepted, 0 errors, risk 0; 0.5 -> 6 accepted, 2 errors, risk 2/6, spanning the other 4 rows.+ # Weighted, not averaged: an unweighted mean of the two risks would weight a 2-row level the+ # same as a 4-row one, which is how a 999-row level and a 1-row level came to count alike.+ assert aurc(conf, corr) == pytest.approx((0.0 * 2 + (2 / 6) * 4) / 6)+++def test_coverage_metrics_are_unchanged_without_ties():+ """Distinct confidences must produce exactly what they always did, to the last bit.++ The fix is only allowed to move a number a permutation of the same dataset could already+ move. With no ties there is one level per row, so every point of the risk-coverage curve --+ and `k` itself -- is what it was before.+ """+ conf, corr = [0.9, 0.8, 0.2, 0.1], [True, True, False, False]+ # the hand-computed values from `test_selective_metrics_on_known_inputs`, restated as floats+ # so this fails if the definition drifts rather than if that test is edited+ assert aurc(conf, corr) == pytest.approx((0 + 0 + 1 / 3 + 1 / 2) / 4, abs=1e-12)+ assert selective_accuracy(conf, corr, 0.5) == pytest.approx(1.0, abs=1e-12)+ assert selective_accuracy(conf, corr, 0.75) == pytest.approx(2 / 3, abs=1e-12)+ assert selective_accuracy(conf, corr, 1.0) == pytest.approx(0.5, abs=1e-12)+ # a single row, and a coverage that rounds below one row, still report that row+ assert selective_accuracy([0.4], [True], 0.01) == pytest.approx(1.0)+ assert aurc([0.4], [False]) == pytest.approx(1.0)+ # `coverage * n` is rarely an integer, and the fraction rounds UP so the answer covers at+ # least the fraction asked for. Five rows at 0.5 means three, not two -- every case above+ # happens to land on a whole row, so nothing here could see a floor in place of the ceiling.+ five, corr5 = [0.9, 0.8, 0.7, 0.6, 0.5], [True, True, False, True, True]+ assert selective_accuracy(five, corr5, 0.5) == pytest.approx(2 / 3) # ceil(2.5) = 3 rows+ # 0.3 -> ceil(1.5) = 2 rows. The second row is WRONG, so a floor (1 row) would report 1.0.+ five_mixed = [0.9, 0.8, 0.7, 0.6, 0.5]+ corr_mixed = [True, False, True, True, True]+ assert selective_accuracy(five_mixed, corr_mixed, 0.3) == pytest.approx(0.5)+ assert selective_accuracy(five, corr5, 0.7) == pytest.approx(3 / 4) # ceil(3.5) = 4 rows+++def test_non_finite_confidences_are_order_independent_too():+ """A NaN confidence must not reintroduce the order-dependence this change removes.++ Confidences come from whatever a runner returns, so NaN and infinities are reachable --+ `_eval_policy._finite_number` exists because this codebase already expects that. `NaN != NaN`,+ so a boundary test written on `!=` alone would make every NaN row its own level and walk them+ in dataset order: the headline defect, surviving for exactly the input a custom runner is most+ likely to hand over by accident. All NaNs are one group, ranked last.++ These VALUES differ from what the old definition reported. They have to: the old ones were+ whichever rows the dataset listed first. `docs/evals.md` says so.+ """+ nan, inf = float("nan"), float("inf")+ with warnings.catch_warnings():+ warnings.simplefilter("error") # a RuntimeWarning here fails the test+ # all NaN: one group, so every coverage reports the group's own accuracy+ for y in ([True, False, True], [True, True, False], [False, True, True]):+ assert selective_accuracy([nan] * 3, y, 0.5) == pytest.approx(2 / 3)+ assert aurc([nan] * 3, y) == pytest.approx(1 / 3)+ # a NaN among finite values: the finite ones rank above it, the NaN group below+ assert selective_accuracy([0.9, nan, 0.5], [True, False, True], 0.5) == pytest.approx(1.0)+ # infinities are ordinary numbers to a comparison, and keep ranking normally+ assert selective_accuracy([inf, 0.5, -inf], [True, True, False], 0.5) == pytest.approx(1.0)+ assert aurc([0.9, 0.7, nan, 0.2], [True, False, True, True]) == pytest.approx(+ (0 * 1 + 0.5 * 1 + (1 - 2 / 3) * 1 + (1 - 3 / 4) * 1) / 4, abs=1e-12)++ # and the whole report, through `evaluate()`, with only the NaN rows reordered+ def report_for(rows):+ examples = [Example("s%d" % i, Q, {"intent": "a"}) for i, _ in enumerate(rows)]+ answers = {"s%d" % i: {"intent": dict(choice_answer("a" if ok else "b", conf),+ answer_confidence=conf)}+ for i, (conf, ok) in enumerate(rows)}+ rep = evaluate(StubRunner(answers), Dataset(examples), evaluators=[ChoiceAccuracy()])+ return {n: rep.overall[n] for n in ("aurc", "selective_accuracy@50",+ "selective_accuracy@80")}++ finite = [(0.9, True), (0.8, True), (0.7, True), (0.6, True)]+ block = [(nan, True), (nan, False), (nan, True), (nan, False), (nan, True), (nan, False)]+ assert report_for(finite + block) == report_for(finite + block[::-1])+++def test_confidences_group_on_exact_equality():+ """Two confidences that differ by one ULP are two levels, not one.++ A threshold comparison is exact, so grouping has to be too. Nothing else in this file puts two+ confidences closer together than 0.1, which let a tolerance window or an `isclose` boundary+ pass every check while changing which answers a cut accepts. The shipped agents round+ confidence to 4 decimals, so this is reachable only from a custom runner -- which is also true+ of every NaN case above.+ """+ one = 1.0+ just_below = 1.0 - 2 ** -52 # the next double below 1.0+ assert just_below != one and round(just_below, 4) == round(one, 4)+ # two levels: the correct answer ranks first and the 50% cut takes only it+ assert selective_accuracy([one, just_below], [True, False], 0.5) == pytest.approx(1.0)+ assert aurc([one, just_below], [True, False]) == pytest.approx((0.0 * 1 + 0.5 * 1) / 2)+ # one level, were they treated as equal, would report the pair's own accuracy instead+ assert selective_accuracy([one, one], [True, False], 0.5) == pytest.approx(0.5)+++def test_corrects_are_read_as_booleans():+ """`corrects` is a correctness flag, not a count.++ `_levels` reads it through `dtype=bool`; reading it as a float lets a value outside {0, 1}+ produce a NEGATIVE risk, and `brier` already reads the same argument as a float, so the two+ readings were unpinned and mutually inconsistent.+ """+ assert aurc([0.9, 0.8], [2, 0]) == pytest.approx(aurc([0.9, 0.8], [True, False]))+ assert aurc([0.9, 0.8], [2, 0]) >= 0.0+ assert selective_accuracy([0.9, 0.8], [2, 0], 1.0) == pytest.approx(0.5)+++def test_a_0d_confidence_still_answers():+ """A bare scalar used to answer, and indexing it with an argsort result raises.++ Reachable only from the library entry point, and `_aggregate` always builds lists -- but an+ `IndexError` escaping `laya-evals` exits 1, which is the code that means "the model regressed".+ """+ assert selective_accuracy(np.float64(0.5), np.True_, 0.5) == pytest.approx(1.0)+ assert aurc(np.float64(0.5), np.False_) == pytest.approx(1.0)+++def test_report_metrics_survive_a_permuted_dataset():+ """The gate reads these through `evaluate()`, so permuting a JSONL must not move them.++ `_eval_policy.check_policy` gates a slice on whatever metric a policy names, and+ `evals.is_confidence_metric` recognises `aurc` and `selective_accuracy@*` by name -- so a+ coverage metric that depended on row order meant a shuffled dataset could pass or fail a+ release gate on nothing at all.+ """+ examples = [Example("s%d" % i, Q, {"intent": "a"}) for i in range(1, 9)]+ # every answer at the same confidence, half of them wrong: the tie case, through the report+ answers = {"s%d" % i: {"intent": choice_answer("a" if i <= 4 else "b", 1.0)}+ for i in range(1, 9)}+ names = ("aurc", "selective_accuracy@50", "selective_accuracy@80", "ece", "brier")++ def metrics(order):+ report = evaluate(StubRunner(answers), Dataset([examples[i] for i in order]),+ evaluators=[ChoiceAccuracy()])+ return {n: report.overall[n] for n in names}++ forward = metrics(range(8))+ assert metrics(range(7, -1, -1)) == forward+ assert metrics([4, 0, 5, 1, 6, 2, 7, 3]) == forward+ # and the values are the tie group's own accuracy, not whichever half came first+ assert forward["selective_accuracy@50"] == pytest.approx(0.5)+ assert forward["aurc"] == pytest.approx(0.5)+++def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmp_path):+ """A `max_drop` across definitions is unsafe in the PASS direction, so it must not be computed.++ The two definitions disagree by as much as 0.48 on the same answers. A slice recorded at 0.033+ by the old one reads 0.517 under this one, so subtracting them lets a candidate that genuinely+ dropped 0.217 clear a `max_drop` of 0.05 -- a real regression shipping because a baseline is+ stale. `comparable_to` cannot see it: the dataset bytes and the question schema are identical,+ and its "a key missing on either side is unknown" rule exists so a patch release cannot+ invalidate a baseline. A definition change is not a patch release, so this refuses on the+ key's ABSENCE.+ """+ from laya import _eval_policy++ policy = _eval_policy.load_policy(_write_gate_policy(+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",+ "min_count": 5, "max_drop": 0.05}))+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))++ stale = _coverage_gate_report(0.033333, None)+ failures = _eval_policy.check_policy(candidate, policy, stale)+ assert any("regenerate the baseline" in f for f in failures), failures+ assert any("definition 1 (unrecorded)" in f for f in failures), failures+ # an explicit version 1 is refused the same way+ assert any("regenerate the baseline" in f for f in+ _eval_policy.check_policy(candidate, policy, _coverage_gate_report(0.033333, 1)))++ # regenerated under this definition, the same gate CATCHES the regression it was hiding+ fresh = _coverage_gate_report(0.516667, evals.COVERAGE_METRIC_DEFINITION)+ failures = _eval_policy.check_policy(candidate, policy, fresh)+ assert any("max_drop" in f and "exceeded" in f for f in failures), failures+ assert not any("regenerate" in f for f in failures), failures+++def test_the_refusal_is_limited_to_metrics_a_cut_can_move(tmp_path):+ """`ece` and `brier` do not cut, so a baseline that predates the change stays comparable.++ Sweeping them into the refusal would break every committed baseline for no reason.+ """+ from laya import _eval_policy++ candidate = EvalReport(**_coverage_gate_report(0.10, evals.COVERAGE_METRIC_DEFINITION,+ metric="ece"))+ stale = _coverage_gate_report(0.11, None, metric="ece")+ policy = _eval_policy.load_policy(_write_gate_policy(+ tmp_path, {"slice": {"language": "zh"}, "metric": "ece",+ "min_count": 5, "max_drop": 0.05}))+ assert _eval_policy.check_policy(candidate, policy, stale) == []+ assert evals.is_coverage_metric("aurc")+ assert evals.is_coverage_metric("selective_accuracy@80")+ assert not evals.is_coverage_metric("ece")+ assert not evals.is_coverage_metric("brier")+ assert not evals.is_coverage_metric("choice_accuracy")+++def test_an_absolute_rule_needs_no_baseline_and_is_unaffected(tmp_path):+ """Only the relative comparison is across definitions; a `min` reads this run alone."""+ from laya import _eval_policy++ policy = _eval_policy.load_policy(_write_gate_policy(+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",+ "min_count": 5, "min": 0.2}))+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))+ assert _eval_policy.check_policy(candidate, policy, None) == []+++def test_every_report_records_which_coverage_definition_produced_it():+ report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}),+ Dataset([Example("s1", Q, {"intent": "a"})]))+ assert report.config["coverage_metric_definition"] == evals.COVERAGE_METRIC_DEFINITION+ assert evals.COVERAGE_METRIC_DEFINITION == 2+++def test_aurc_is_bit_identical_to_the_per_answer_mean_without_ties():+ """Not `approx`: the claim is bit-identity, and only an exact comparison can hold it.++ With no ties every level holds one answer, so weighting by rows and dividing once at the end+ reduces to the same pairwise sum over the same denominator. Weighting by the coverage width+ instead -- mathematically the same area -- loses the last bits on about half of all datasets,+ which would quietly move every committed number that has no tie in it.+ """+ import random++ random.seed(5)+ for _ in range(200):+ n = random.randint(1, 40)+ conf = random.sample([i / 10000 for i in range(1, 9999)], n)+ corr = [random.random() < 0.6 for _ in range(n)]+ order = np.argsort(-np.asarray(conf, dtype=float), kind="mergesort")+ y = np.asarray(corr, dtype=bool)[order].astype(float)+ per_answer = 1.0 - np.cumsum(y) / np.arange(1, n + 1)+ assert aurc(conf, corr) == float(np.mean(per_answer))++def test_selective_metrics_reach_the_report():# choice answers with a spread of confidence and correctness so the pairs existds = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"}),@@ -1751,6 +2043,22 @@ def _slice_gate_report(en_correct, zh_correct):"cases": cases}+def _coverage_gate_report(value, definition, metric="selective_accuracy@50"):+ """A slice-gate report carrying a coverage metric and a definition stamp.++ `definition=None` is a report written before `coverage_metric_definition` existed, which is+ exactly what every committed baseline is.+ """+ cases = [{"language": "zh", "scores": {}, "confidence": 0.8, "correct": True}+ for _ in range(20)]+ config = {"schema": evals.REPORT_SCHEMA, "dataset_sha256": "a" * 64,+ "questions_sha256": "b" * 64}+ if definition is not None:+ config["coverage_metric_definition"] = definition+ return {"config": config, "overall": {metric: value},+ "slices": {"language": {"zh": {metric: value}}}, "cases": cases}++def _write_gate_policy(tmp_path, rule):path = tmp_path / "gates.json"path.write_text(json.dumps({"version": 1, "rules": [rule]}), encoding="utf-8")