Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(evals): refuse a cross-definition coverage comparison on either side
Adeshina Lasisid0f8bccb98ba
diff --git a/docs/evals.md b/docs/evals.mdindex b53405af..8a65d0ec 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -133,7 +133,7 @@ Each metric is computed per answer where it applies and aggregated over the data| `score_within_<tol>` | `score` | fraction within an absolute tolerance || `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on `answer["answer_confidence"]`, the calibrated probability Laya reports on every answer type || `brier` | any answer with a confidence and a known label | Brier score of confidence as P(correct), `mean((confidence - correct)**2)`; lower is better |-| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve, averaged over the distinct confidence levels; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |+| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve: one risk value per distinct confidence level, each weighted by the answers that level spans; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated || `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the answers a confidence threshold at the 50% / 80% coverage point accepts -- what abstaining on the least-confident tail buys. A threshold cannot split a group of equal confidences, so this can cover more than the named fraction; see [coverage cuts](#coverage-cuts-and-ties) || `mean_confidence` | any answer with a confidence | mean reported `answer["answer_confidence"]` || `latency_p50_ms`, `latency_p95_ms` | per request | wall time each request waited, informational -- see [batching](#batching-and-timing) |@@ -162,11 +162,18 @@ Two consequences worth planning for:* **A number can move in either direction, by more than a reordering could.** Where a groupstraddles the cut, the threshold reading differs from every row-index reading of the same data:- measured, up to 0.48 for `selective_accuracy@50` and 0.09 for `aurc`. A gate that was passing may- fail, and one that was failing may pass; the previous verdict depended on row order.+ measured over 400,001 tie-shaped datasets, up to 0.500 for `selective_accuracy@50` and 0.351+ for `aurc`. On the twelve-answer shape above -- six correct, all at confidence 1.0 -- `aurc` moves+ 0.327 (0.173 to 0.500). A gate that was passing may fail, and one that was failing may pass; the+ previous verdict depended on row order, including for an absolute `min` or `max` limit, which is+ refused by nothing because it reads a single run.* **Regenerate committed baselines.** `config.coverage_metric_definition` records which definition- produced a report, and a relative rule (`max_drop` / `max_increase`) on `aurc` or- `selective_accuracy@*` refuses a baseline recorded under a different one. Without that refusal a+ produced a report. Comparing a coverage metric across two definitions is refused on **both** the+ gates that subtract a baseline -- `--baseline --tolerance` (`EvalReport.compare`) and a relative+ rule (`max_drop` / `max_increase`) under `--gate-policy` -- and on **either** side being stale,+ not just the baseline: a candidate produced by an older `laya` carries a row-order artifact that+ can read *better* than the truth, so gating it against a correctly regenerated baseline would+ pass a regression that the correctly scored report fails. Without that refusal astale baseline hides a real regression: a slice recorded at 0.033 under the old definition reads0.517 under this one, so a candidate that genuinely dropped 0.217 would clear a `max_drop` of0.05. `ece` and `brier` do not cut and stay comparable.diff --git a/laya/_eval_policy.py b/laya/_eval_policy.pyindex 26f08e8f..0ce5c736 100644--- a/laya/_eval_policy.py+++ b/laya/_eval_policy.py@@ -106,33 +106,21 @@ def _value(report: evals.EvalReport, dimension: str, value: str,return float(result)-def _stale_coverage_baseline(baseline: Any, metric: str) -> Optional[str]:- """Why a relative rule on a coverage metric must not be computed against `baseline`, or None.-- A `max_drop` / `max_increase` rule subtracts the baseline's number from this run's. That is- only meaningful if both were measured by the same definition of a coverage cut. Version 2 cuts- on a confidence threshold, so a cut never splits a group of equal confidences; version 1 cut at- a row index and depended on the order the dataset arrived in. The two disagree by as much as- 0.48 on the same rows, so the subtraction is not merely noisy -- it is unsafe in the PASS- direction, letting a real regression through.-- The absence of the key IS the old definition: every report written since it existed carries it.- That is why this refuses instead of following `comparable_to`'s "a key missing on either side is- unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a- committed baseline. A definition change is not a patch release.-- `ece` and `brier` are untouched by the cut, so a rule on either keeps comparing as before.+def _incomparable_coverage_definitions(report: evals.EvalReport, baseline: Any,+ metric: str) -> Optional[str]:+ """Why a relative rule on `metric` must not subtract these two reports, or None.++ Delegates to `evals.coverage_definition_conflict` so that this gate and `EvalReport.compare`+ cannot drift apart about what "comparable" means -- they are the same rule, not two copies of+ it. Both the candidate and the baseline are checked: see that function for why a stale+ candidate is the more dangerous of the two."""if not evals.is_coverage_metric(metric):return None- config = baseline.get("config") if isinstance(baseline, dict) else None- recorded = config.get("coverage_metric_definition") if isinstance(config, dict) else None- if recorded == evals.COVERAGE_METRIC_DEFINITION:+ conflict = evals.coverage_definition_conflict(report.config, baseline)+ if conflict is None:return None- return ("baseline reports coverage-metric definition %s, this run uses %s -- regenerate the "- "baseline before gating %r on a relative limit" %- ("1 (unrecorded)" if recorded is None else recorded,- evals.COVERAGE_METRIC_DEFINITION, metric))+ return "%s on a relative limit" % conflictdef check_policy(report: evals.EvalReport, policy: Dict[str, Any],@@ -178,7 +166,7 @@ def check_policy(report: evals.EvalReport, policy: Dict[str, Any],if comparator in _RELATIVE:# the raw baseline document, not `base_report`: that is rebuilt as an# `EvalReport` and carries only what `_identity_of` kept- stale = _stale_coverage_baseline(baseline, metric)+ stale = _incomparable_coverage_definitions(report, baseline, metric)if stale is not None:failures.append("%s: %s" % (label, stale))continuediff --git a/laya/evals.py b/laya/evals.pyindex 57987094..f3c0c0e8 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -375,27 +375,41 @@ def brier(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flreturn float(np.mean((c - y) ** 2))-def _threshold_accepts(sorted_desc, k: int) -> int:- """How many rows a confidence threshold placed at `sorted_desc[k - 1]` accepts.+def _accepted_at_cut(sorted_desc, k: int) -> int:+ """How many answers a confidence threshold placed at `sorted_desc[k - 1]` accepts.The end of the tie group the cut landed in, which is what makes a coverage number independent- of the order a dataset happened to arrive in. Never fewer than `k`: a threshold accepts at- least the rows the coverage asked for.-- NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN is- false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever rows- came first" -- the exact order-dependence this exists to remove, for the input a custom runner- is most likely to hand over by accident. NaNs sort last and form one group, so a cut inside- them accepts all of them.+ of the order a dataset happened to arrive in.++ This is the single-cut answer; `_levels` computes the same group boundaries for *every* level,+ which is what `aurc` integrates over. Two routes to one rule, so they are not merely asserted+ to agree -- `test_a_single_cut_agrees_with_the_risk_coverage_curve` sweeps them against each+ other, including over NaN and infinities, so an edit to either side fails a test rather than+ diverging quietly. Computing one cut from the full scan instead cost 19% to 58% on+ `selective_accuracy` for no change in any value.++ NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN+ is false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever+ rows came first" -- the exact order-dependence this exists to remove, for the input a custom+ runner is most likely to hand over by accident. NaNs sort last and form one group, so a cut+ inside them accepts all of them.++ The NaN branch is kept although `searchsorted` happens to answer the same thing -- numpy ranks+ NaN last for a binary search too, verified over 63,697 NaN cuts without a disagreement. Leaving+ it implicit would make a correct answer depend on an undocumented ordering convention, so the+ branch stays and its mutant is reported as equivalent rather than as a test gap."""cut = sorted_desc[k - 1]if cut != cut: # NaNreturn len(sorted_desc)- return max(k, int(np.count_nonzero(sorted_desc >= cut)))+ # `sorted_desc` descends, so its negation ascends and a binary search finds the end of the tie+ # group in O(log n) rather than a full pass per call. NaNs rank above every number in numpy's+ # sort order, so they sit at the end of the negated array and a finite cut never counts them.+ return int(np.searchsorted(-sorted_desc, -cut, side="right"))def _levels(confidences: Sequence[float], corrects: Sequence[bool]):- """`(accepted, risk, n)` at each distinct confidence level, most confident first.+ """`(accepted, hits, n)` at each distinct confidence level, most confident first.One row per LEVEL, not per answer. A coverage cut is bought with a confidence threshold, and athreshold accepts every answer at its own confidence -- `calibrate._select_abstention_threshold`@@ -410,28 +424,41 @@ def _levels(confidences: Sequence[float], corrects: Sequence[bool]):`accepted[i]` is how many answers a threshold at level `i` accepts, which is what lets `aurc`weight a level by the answers it spans rather than counting a 1-answer level and a 999-answer- level alike.+ level alike. `hits[i]` is how many of those are correct.++ Counts, not a risk already divided: `corrects` is 0.0/1.0, so every partial sum is an exact+ integer in float64 and `hits / accepted` reproduces `np.mean` of the accepted slice bit for+ bit. Returning `1 - hits / accepted` and recovering accuracy as `1 - risk` instead costs the+ last bits of 157,289 values in a 1.2-million-value sweep -- harmless to a gate, but it would+ make this change move numbers on datasets with no tie in them, which it is not allowed to do."""# `atleast_1d`: a 0-dimensional input (a bare scalar) answered before this change, and indexing# it with an argsort result raises.c = np.atleast_1d(np.asarray(confidences, dtype=float))- y = np.atleast_1d(np.asarray(corrects, dtype=bool).astype(float))+ # `corrects` stays boolean here: `cumsum` of a boolean array counts in int64, which is exact,+ # and skipping the float conversion saves a pass over the whole column.+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))n = len(y)if n == 0:return np.empty(0), np.empty(0), 0order = np.argsort(-c, kind="mergesort") # desc, stable; NaN sorts lastc, y = c[order], y[order]# The last answer of each tie group: where the next confidence differs, plus the final one.+ boundary = c[1:] != c[:-1]+ # NaN needs `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every+ # NaN confidence becomes a level of its own, the running total walks them in dataset order, and+ # the order-dependence this exists to remove survives for exactly the input a custom runner is+ # most likely to hand over by accident. All NaNs are one group, ranked last.#- # `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every NaN- # confidence becomes a level of its own, the running total walks them in dataset order, and the- # order-dependence this exists to remove survives for exactly the input a custom runner is most- # likely to hand over by accident. All NaNs are one group, ranked last.- nan_pair = np.isnan(c[1:]) & np.isnan(c[:-1])- last = np.flatnonzero(np.concatenate(((c[1:] != c[:-1]) & ~nan_pair, [True])))+ # Guarded by one scalar test rather than paid for always: NaN sorts to the end, so if the last+ # confidence is not NaN then none are, and two full `isnan` passes over the column can be+ # skipped. That is the common case, and the two passes were a third of this function's cost at+ # a million rows.+ if np.isnan(c[-1]):+ boundary &= ~(np.isnan(c[1:]) & np.isnan(c[:-1]))+ last = np.flatnonzero(np.concatenate((boundary, [True])))accepted = last.astype(float) + 1.0- risk = 1.0 - np.cumsum(y)[last] / accepted- return accepted, risk, n+ return accepted, np.cumsum(y)[last], ndef _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):@@ -439,10 +466,10 @@ def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):`_levels` with coverage as the x axis, for callers that want the curve rather than its area."""- accepted, risk, n = _levels(confidences, corrects)+ accepted, hits, n = _levels(confidences, corrects)if n == 0:return np.empty(0), np.empty(0)- return accepted / n, risk+ return accepted / n, 1.0 - hits / accepteddef aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[float]:@@ -454,9 +481,10 @@ def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flo"""if not confidences:return None- accepted, risk, n = _levels(confidences, corrects)+ accepted, hits, n = _levels(confidences, corrects)if n == 0:return None+ risk = 1.0 - hits / accepted# Each level's risk weighted by the ANSWERS it spans, which is what keeps this an area rather# than an average of unevenly sized points: with 999 answers at one confidence and 1 at another,# an unweighted mean of the two risks reports 5e-4 where the area is 1e-6.@@ -477,6 +505,12 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],coverage is the group's upper edge rather than the fraction asked for. That is the onlyorder-independent reading -- see `_risk_coverage` -- and it is what makes this comparableacross two permutations of the same dataset. Without ties, `k` is unchanged.++ The group this cut lands in comes from `_accepted_at_cut`, which answers for one cut what+ `_levels` answers for every level. The two are swept against each other by+ `test_a_single_cut_agrees_with_the_risk_coverage_curve`, so they cannot drift apart silently;+ deriving this from the full scan instead was measured at 19% to 58% slower here for no change+ in any value."""if not confidences or not 0.0 < coverage <= 1.0:return None@@ -484,13 +518,14 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],c = np.atleast_1d(np.asarray(confidences, dtype=float))order = np.argsort(-c, kind="mergesort")c = c[order]- y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order].astype(float)+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order]k = max(1, int(_math.ceil(coverage * len(y))))- # Extend through the tie group the cut fell in -- see `_threshold_accepts`, which both this- # and the risk-coverage curve go through, so the two cannot disagree about what a cut- # accepts, and neither falls back to dataset order when the cut lands on a NaN.- k = _threshold_accepts(c, k)- return float(np.mean(y[:k]))+ # Extend through the tie group the cut fell in, so the cut is a confidence threshold rather+ # than a row index and cannot report whichever tie members arrived first.+ accepted = _accepted_at_cut(c, k)+ # `count_nonzero / accepted` rather than `mean`: both are exact for a boolean column, and this+ # skips the float conversion of the accepted slice.+ return float(np.count_nonzero(y[:accepted]) / accepted)#: Coverage points reported as `selective_accuracy@NN`.@@ -506,6 +541,62 @@ def is_coverage_metric(name: str) -> bool:return name == "aurc" or name.startswith("selective_accuracy@")+def coverage_definition_conflict(candidate_config: Any, baseline: Any) -> Optional[str]:+ """Why a coverage metric must not be compared across these two documents, or None.++ Comparing `aurc` or `selective_accuracy@*` between two reports subtracts one number from the+ other, which is only meaningful if the same definition of a coverage cut produced both.+ Definition 2 cuts on a confidence threshold, so a cut never splits a group of equal+ confidences; definition 1 cut at a row index and depended on the order the dataset arrived in.+ Measured over 400,001 tie-shaped datasets the two disagree by up to 0.35 on `aurc` and 0.50 on+ `selective_accuracy@50`, so the subtraction is not merely noisy -- it is unsafe in the PASS+ direction, letting a real regression through.++ **Both sides are checked, not just the baseline.** A stale CANDIDATE is the more dangerous+ case and the easier one to arrive at by accident: a report produced by a pinned older `laya`+ in a container, or an archived artifact re-checked later, carries definition-1 numbers whose+ row-order artifact can read BETTER than the truth. Gating that against a correctly regenerated+ baseline passes a regression that the correctly scored report fails.++ The absence of the key IS definition 1: every report written since it existed carries it. That+ is why this refuses instead of following `comparable_to`'s "a key missing on either side is+ unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a+ committed baseline. A definition change is not a patch release.++ `ece` and `brier` read the same pairs but do not cut, so they stay comparable; callers gate+ this behind `is_coverage_metric`.+ """+ here = _recorded_coverage_definition(candidate_config)+ there = _recorded_coverage_definition(+ baseline.get("config") if isinstance(baseline, dict) else None)+ if here == COVERAGE_METRIC_DEFINITION and there == COVERAGE_METRIC_DEFINITION:+ return None+ stale = []+ if here != COVERAGE_METRIC_DEFINITION:+ stale.append("this report says %s" % _shown_definition(here))+ if there != COVERAGE_METRIC_DEFINITION:+ stale.append("the baseline says %s" % _shown_definition(there))+ return ("%s, but a coverage cut here is definition %s -- regenerate it before comparing a "+ "coverage metric" % (" and ".join(stale), COVERAGE_METRIC_DEFINITION))+++def _recorded_coverage_definition(config: Any) -> Any:+ return config.get("coverage_metric_definition") if isinstance(config, dict) else None+++def _shown_definition(recorded: Any) -> str:+ """A recorded definition, rendered for a message, with attacker-supplied content bounded.++ The value comes from a report file, which on a pull-request gate is content the author+ controls, and the message it lands in is printed to a CI log. A five-megabyte string in that+ field produced a five-megabyte failure line; 40 characters is enough to diagnose a real one.+ """+ if recorded is None:+ return "1 (unrecorded)"+ shown = repr(recorded)+ return shown if len(shown) <= 40 else shown[:40] + "... (truncated)"++def is_confidence_metric(name: str) -> bool:"""True for a dataset-level metric computed from `(confidence, correct)` pairs, not per-answer`scores` -- ECE and the selective-classification metrics. Policy counting treats these alike."""@@ -609,6 +700,19 @@ class EvalReport:if metric.endswith("_ms") and metric not in tolerances:continueallowed = float(tolerances.get(metric, 0.0))+ # A coverage metric is only comparable when both documents were produced by the same+ # definition of a cut. This path is the one `docs/evals.md` leads with, and+ # `--gate-policy` is opt-in, so without this check the default gate performs exactly+ # the cross-definition subtraction the policy gate refuses.+ if is_coverage_metric(metric):+ conflict = coverage_definition_conflict(self.config, baseline)+ if conflict is not None:+ deltas[metric] = {+ "baseline": float(base_value),+ "value": float(self.overall.get(metric, float("nan"))),+ "diff": float("nan"), "tolerance": allowed, "incomparable": conflict}+ ok = False+ continueif metric not in self.overall:deltas[metric] = {"baseline": float(base_value), "value": float("nan"),"diff": float("nan"), "tolerance": allowed, "missing": True}diff --git a/laya/evals_cli.py b/laya/evals_cli.pyindex 67328e18..7ab6ca16 100644--- a/laya/evals_cli.py+++ b/laya/evals_cli.py@@ -256,6 +256,12 @@ def _print_deltas(deltas: Dict[str, Dict[str, Any]]) -> None:if delta.get("missing"):print("%-18s baseline=%.4f missing from the report" % (metric, delta["baseline"]))continue+ if delta.get("incomparable"):+ # The diff is NaN by construction here, and "diff=+nan" on its own is not a+ # diagnosis: print the reason the comparison was refused, not just its symptom.+ print("%-18s baseline=%.4f value=%.4f not compared: %s"+ % (metric, delta["baseline"], delta["value"], delta["incomparable"]))+ continueprint("%-18s baseline=%.4f value=%.4f diff=%+.4f (tol %.4f)"% (metric, delta["baseline"], delta["value"], delta["diff"], delta["tolerance"]))diff --git a/tests/test_evals.py b/tests/test_evals.pyindex ea9a80f5..914b3a9b 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -320,7 +320,7 @@ def test_report_metrics_survive_a_permuted_dataset():def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmp_path):"""A `max_drop` across definitions is unsafe in the PASS direction, so it must not be computed.- The two definitions disagree by as much as 0.48 on the same answers. A slice recorded at 0.033+ The two definitions disagree by up to 0.500 on the same answers. A slice recorded at 0.033by the old one reads 0.517 under this one, so subtracting them lets a candidate that genuinelydropped 0.217 clear a `max_drop` of 0.05 -- a real regression shipping because a baseline isstale. `comparable_to` cannot see it: the dataset bytes and the question schema are identical,@@ -337,10 +337,10 @@ def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmpstale = _coverage_gate_report(0.033333, None)failures = _eval_policy.check_policy(candidate, policy, stale)- assert any("regenerate the baseline" in f for f in failures), failures- assert any("definition 1 (unrecorded)" in f for f in failures), failures+ assert any("regenerate it before comparing" in f for f in failures), failures+ assert any("the baseline says 1 (unrecorded)" in f for f in failures), failures# an explicit version 1 is refused the same way- assert any("regenerate the baseline" in f for f in+ assert any("regenerate it before comparing" in f for f in_eval_policy.check_policy(candidate, policy, _coverage_gate_report(0.033333, 1)))# regenerated under this definition, the same gate CATCHES the regression it was hiding@@ -382,6 +382,81 @@ def test_an_absolute_rule_needs_no_baseline_and_is_unaffected(tmp_path):assert _eval_policy.check_policy(candidate, policy, None) == []+def test_a_stale_candidate_is_refused_as_well_as_a_stale_baseline(tmp_path):+ """The dangerous direction: an old-definition CANDIDATE can read BETTER than the truth.++ A report scored by the old definition carries a row-order artifact. Here the same slice reads+ 1.000 under definition 1 (its correct rows happened to be listed first) where definition 2 says+ 0.500, so subtracting it from a correctly regenerated 0.500 baseline shows no drop at all and+ the gate passes -- while the honestly scored candidate fails. Checking only the baseline leaves+ this open, and it is the easier of the two to reach by accident: a CI artifact produced by a+ pinned older `laya`, or an archived report re-checked after the baseline was regenerated.+ """+ from laya import _eval_policy++ policy = _eval_policy.load_policy(_write_gate_policy(+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",+ "min_count": 5, "max_drop": 0.05}))+ fresh_baseline = _coverage_gate_report(0.50, evals.COVERAGE_METRIC_DEFINITION)+ stale_candidate = EvalReport(**_coverage_gate_report(1.0, None))++ failures = _eval_policy.check_policy(stale_candidate, policy, fresh_baseline)+ assert any("this report says 1 (unrecorded)" in f for f in failures), failures+ # Proof the refusal is what stops it: the subtraction itself sees 1.0 against 0.50, which is a+ # GAIN, so without the refusal there is no `max_drop` failure to catch the regression.+ assert not any("max_drop" in f for f in failures), failures+ # both sides stale is refused too, and says so about both+ both = _eval_policy.check_policy(stale_candidate, policy, _coverage_gate_report(0.50, None))+ assert any("this report says" in f and "the baseline says" in f for f in both), both+++def test_compare_refuses_a_coverage_metric_across_definitions():+ """`EvalReport.compare` is the gate this project's docs lead with, and it also subtracts.++ `--gate-policy` is opt-in; `--baseline --tolerance` is the documented default. A refusal that+ covered only the policy gate would leave the usual path performing exactly the cross-definition+ subtraction the policy gate exists to refuse.+ """+ report = EvalReport(**_coverage_gate_report(0.50, evals.COVERAGE_METRIC_DEFINITION))+ stale = _coverage_gate_report(0.47, None)++ ok, deltas = report.compare(stale, {"selective_accuracy@50": 0.10})+ assert not ok, deltas+ assert "incomparable" in deltas["selective_accuracy@50"], deltas+ # Without the refusal this passes: 0.50 against 0.47 is a 0.03 move inside a 0.10 tolerance,+ # while the regenerated baseline for the same answers is 0.03 and the real move is 0.47.+ fresh = _coverage_gate_report(0.47, evals.COVERAGE_METRIC_DEFINITION)+ ok_same_definition, deltas_same = report.compare(fresh, {"selective_accuracy@50": 0.10})+ assert ok_same_definition, deltas_same+ assert "incomparable" not in deltas_same["selective_accuracy@50"]+ # `ece` does not cut, so an unstamped baseline stays comparable+ ece = EvalReport(**_coverage_gate_report(0.10, evals.COVERAGE_METRIC_DEFINITION, metric="ece"))+ ok_ece, deltas_ece = ece.compare(_coverage_gate_report(0.11, None, metric="ece"),+ {"ece": 0.05})+ assert ok_ece, deltas_ece+++def test_the_refusal_bounds_a_hostile_recorded_definition(tmp_path):+ """The stamp is read from a report file, which on a pull-request gate the author writes.++ The refusal names the value it found, and that message is printed to a CI log, so the value is+ attacker-controlled content on its way into a log. Five megabytes in the field produced a+ five-megabyte failure line before this was bounded.+ """+ from laya import _eval_policy++ policy = _eval_policy.load_policy(_write_gate_policy(+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",+ "min_count": 5, "max_drop": 0.05}))+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))++ failures = _eval_policy.check_policy(candidate, policy,+ _coverage_gate_report(0.03, "x" * 5_000_000))+ assert failures+ assert max(len(f) for f in failures) < 400, max(len(f) for f in failures)+ assert any("truncated" in f for f in failures), failures++def test_every_report_records_which_coverage_definition_produced_it():report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}),Dataset([Example("s1", Q, {"intent": "a"})]))@@ -410,6 +485,74 @@ def test_aurc_is_bit_identical_to_the_per_answer_mean_without_ties():assert aurc(conf, corr) == float(np.mean(per_answer))+def test_a_single_cut_agrees_with_the_risk_coverage_curve():+ """The two routes to one rule must not drift apart, so a test holds them together.++ `selective_accuracy` answers for one cut (`_accepted_at_cut`, a binary search) what `_levels`+ answers for every level (a full scan, which `aurc` integrates). Deriving the single cut from+ the full scan was measured 19% to 58% slower for no change in any value, so both exist -- and+ a comment asserting they agree would be enforced by nothing. This is the enforcement: an edit+ to either side fails here.++ Exact equality, not `approx`: both routes divide one exact integer count by another, over the+ same accepted set, so any difference at all is a real divergence rather than rounding.+ """+ import math+ import random++ rng = random.Random(13)+ pool = [0.0, -0.0, 1.0, 0.5, 0.25, -1.0, 2.0,+ float("nan"), float("inf"), -float("inf"), 1.0 - 2.0 ** -52]+ for _ in range(4000):+ n = rng.randint(1, 14)+ conf = [rng.choice(pool) for _ in range(n)]+ corr = [rng.random() < 0.5 for _ in range(n)]+ accepted, hits, levels_n = evals._levels(conf, corr)+ assert levels_n == n+ for coverage in (0.01, 0.25, 0.5, 0.8, 1.0):+ k = max(1, int(math.ceil(coverage * n)))+ level = int(np.searchsorted(accepted, k, side="left"))+ # the two routes must agree on how many answers the cut accepts ...+ #+ # Sorted the way `_levels` sorts: `argsort(-c)` ranks NaN LAST, where+ # `np.sort(c)[::-1]` would rank it first and feed the lookup a different array.+ column = np.asarray(conf, dtype=float)+ sorted_desc = column[np.argsort(-column, kind="mergesort")]+ assert evals._accepted_at_cut(sorted_desc, k) == int(accepted[level]), (+ n, k, conf)+ # ... and therefore on the accuracy over them+ assert selective_accuracy(conf, corr, coverage) == float(hits[level] / accepted[level]), (+ n, coverage, conf, corr)+++def test_selective_accuracy_is_bit_identical_to_the_accepted_slice_mean():+ """Not `approx`: the claim is bit-identity with what this function answered before.++ `selective_accuracy` now reads the same levels `aurc` reads, so it had to be shown that routing+ it through them does not move a value. It returns `hits / accepted` -- the exact count of+ correct answers over the exact count accepted -- because `corrects` is 0.0/1.0 and every+ partial sum is therefore an exact integer in float64.++ Recovering the same number as `1 - risk` instead, from a risk the levels had already divided,+ round-trips through a subtraction and differs in the last bits on about one value in eight. A+ gate would not notice; a user diffing two reports of the same no-tie data would, and this+ change is not allowed to move a number that no tie could move.+ """+ import random++ random.seed(11)+ for _ in range(200):+ n = random.randint(1, 40)+ conf = random.sample([i / 10000 for i in range(1, 9999)], n) # distinct: no ties+ corr = [random.random() < 0.6 for _ in range(n)]+ order = np.argsort(-np.asarray(conf, dtype=float), kind="mergesort")+ y = np.asarray(corr, dtype=bool)[order].astype(float)+ for coverage in (0.1, 0.25, 0.5, 0.8, 1.0):+ k = max(1, int(np.ceil(coverage * n)))+ assert selective_accuracy(conf, corr, coverage) == float(np.mean(y[:k])), (+ n, coverage, conf, corr)++def test_selective_metrics_reach_the_report():# choice answers with a spread of confidence and correctness so the pairs existds = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"}),diff --git a/tests/test_evals_api.py b/tests/test_evals_api.pyindex 3a031d4b..2a1c8bc7 100644--- a/tests/test_evals_api.py+++ b/tests/test_evals_api.py@@ -44,10 +44,16 @@ check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "s# --------------------------------------------------------------- exports / callablesfor name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression","REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint",- "brier", "aurc", "selective_accuracy", "is_confidence_metric"):+ "brier", "aurc", "selective_accuracy", "is_confidence_metric",+ # `_eval_policy` imports these three, so they are a contract between two modules in+ # this package, not internals -- a rename would break the release gate, not just a+ # caller.+ "is_coverage_metric", "coverage_definition_conflict",+ "COVERAGE_METRIC_DEFINITION"):check_true("laya.evals.%s exists" % name, hasattr(evals, name))check("REPORT_SCHEMA", evals.REPORT_SCHEMA, "laya-evals-report/1")+check("the coverage-cut definition this code computes", evals.COVERAGE_METRIC_DEFINITION, 2)check_true("EvalReport.comparable_to is part of the report contract",callable(getattr(evals.EvalReport, "comparable_to", None)))check("EvalReport fields are unchanged", [f.name for f in dataclasses.fields(evals.EvalReport)],