Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(evals): refuse a cross-definition coverage comparison on either side

Adeshina Lasisid0f8bccb98ba
diff --git a/docs/evals.md b/docs/evals.md
index b53405af..8a65d0ec 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -133,7 +133,7 @@ Each metric is computed per answer where it applies and aggregated over the data
| `score_within_<tol>` | `score` | fraction within an absolute tolerance |
| `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on `answer["answer_confidence"]`, the calibrated probability Laya reports on every answer type |
| `brier` | any answer with a confidence and a known label | Brier score of confidence as P(correct), `mean((confidence - correct)**2)`; lower is better |
-| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve, averaged over the distinct confidence levels; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |
+| `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve: one risk value per distinct confidence level, each weighted by the answers that level spans; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated |
| `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the answers a confidence threshold at the 50% / 80% coverage point accepts -- what abstaining on the least-confident tail buys. A threshold cannot split a group of equal confidences, so this can cover more than the named fraction; see [coverage cuts](#coverage-cuts-and-ties) |
| `mean_confidence` | any answer with a confidence | mean reported `answer["answer_confidence"]` |
| `latency_p50_ms`, `latency_p95_ms` | per request | wall time each request waited, informational -- see [batching](#batching-and-timing) |
@@ -162,11 +162,18 @@ Two consequences worth planning for:
* **A number can move in either direction, by more than a reordering could.** Where a group
straddles the cut, the threshold reading differs from every row-index reading of the same data:
- measured, up to 0.48 for `selective_accuracy@50` and 0.09 for `aurc`. A gate that was passing may
- fail, and one that was failing may pass; the previous verdict depended on row order.
+ measured over 400,001 tie-shaped datasets, up to 0.500 for `selective_accuracy@50` and 0.351
+ for `aurc`. On the twelve-answer shape above -- six correct, all at confidence 1.0 -- `aurc` moves
+ 0.327 (0.173 to 0.500). A gate that was passing may fail, and one that was failing may pass; the
+ previous verdict depended on row order, including for an absolute `min` or `max` limit, which is
+ refused by nothing because it reads a single run.
* **Regenerate committed baselines.** `config.coverage_metric_definition` records which definition
- produced a report, and a relative rule (`max_drop` / `max_increase`) on `aurc` or
- `selective_accuracy@*` refuses a baseline recorded under a different one. Without that refusal a
+ produced a report. Comparing a coverage metric across two definitions is refused on **both** the
+ gates that subtract a baseline -- `--baseline --tolerance` (`EvalReport.compare`) and a relative
+ rule (`max_drop` / `max_increase`) under `--gate-policy` -- and on **either** side being stale,
+ not just the baseline: a candidate produced by an older `laya` carries a row-order artifact that
+ can read *better* than the truth, so gating it against a correctly regenerated baseline would
+ pass a regression that the correctly scored report fails. Without that refusal a
stale baseline hides a real regression: a slice recorded at 0.033 under the old definition reads
0.517 under this one, so a candidate that genuinely dropped 0.217 would clear a `max_drop` of
0.05. `ece` and `brier` do not cut and stay comparable.
diff --git a/laya/_eval_policy.py b/laya/_eval_policy.py
index 26f08e8f..0ce5c736 100644
--- a/laya/_eval_policy.py
+++ b/laya/_eval_policy.py
@@ -106,33 +106,21 @@ def _value(report: evals.EvalReport, dimension: str, value: str,
return float(result)
-def _stale_coverage_baseline(baseline: Any, metric: str) -> Optional[str]:
- """Why a relative rule on a coverage metric must not be computed against `baseline`, or None.
-
- A `max_drop` / `max_increase` rule subtracts the baseline's number from this run's. That is
- only meaningful if both were measured by the same definition of a coverage cut. Version 2 cuts
- on a confidence threshold, so a cut never splits a group of equal confidences; version 1 cut at
- a row index and depended on the order the dataset arrived in. The two disagree by as much as
- 0.48 on the same rows, so the subtraction is not merely noisy -- it is unsafe in the PASS
- direction, letting a real regression through.
-
- The absence of the key IS the old definition: every report written since it existed carries it.
- That is why this refuses instead of following `comparable_to`'s "a key missing on either side is
- unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a
- committed baseline. A definition change is not a patch release.
-
- `ece` and `brier` are untouched by the cut, so a rule on either keeps comparing as before.
+def _incomparable_coverage_definitions(report: evals.EvalReport, baseline: Any,
+ metric: str) -> Optional[str]:
+ """Why a relative rule on `metric` must not subtract these two reports, or None.
+
+ Delegates to `evals.coverage_definition_conflict` so that this gate and `EvalReport.compare`
+ cannot drift apart about what "comparable" means -- they are the same rule, not two copies of
+ it. Both the candidate and the baseline are checked: see that function for why a stale
+ candidate is the more dangerous of the two.
"""
if not evals.is_coverage_metric(metric):
return None
- config = baseline.get("config") if isinstance(baseline, dict) else None
- recorded = config.get("coverage_metric_definition") if isinstance(config, dict) else None
- if recorded == evals.COVERAGE_METRIC_DEFINITION:
+ conflict = evals.coverage_definition_conflict(report.config, baseline)
+ if conflict is None:
return None
- return ("baseline reports coverage-metric definition %s, this run uses %s -- regenerate the "
- "baseline before gating %r on a relative limit" %
- ("1 (unrecorded)" if recorded is None else recorded,
- evals.COVERAGE_METRIC_DEFINITION, metric))
+ return "%s on a relative limit" % conflict
def check_policy(report: evals.EvalReport, policy: Dict[str, Any],
@@ -178,7 +166,7 @@ def check_policy(report: evals.EvalReport, policy: Dict[str, Any],
if comparator in _RELATIVE:
# the raw baseline document, not `base_report`: that is rebuilt as an
# `EvalReport` and carries only what `_identity_of` kept
- stale = _stale_coverage_baseline(baseline, metric)
+ stale = _incomparable_coverage_definitions(report, baseline, metric)
if stale is not None:
failures.append("%s: %s" % (label, stale))
continue
diff --git a/laya/evals.py b/laya/evals.py
index 57987094..f3c0c0e8 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -375,27 +375,41 @@ def brier(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[fl
return float(np.mean((c - y) ** 2))
-def _threshold_accepts(sorted_desc, k: int) -> int:
- """How many rows a confidence threshold placed at `sorted_desc[k - 1]` accepts.
+def _accepted_at_cut(sorted_desc, k: int) -> int:
+ """How many answers a confidence threshold placed at `sorted_desc[k - 1]` accepts.
The end of the tie group the cut landed in, which is what makes a coverage number independent
- of the order a dataset happened to arrive in. Never fewer than `k`: a threshold accepts at
- least the rows the coverage asked for.
-
- NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN is
- false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever rows
- came first" -- the exact order-dependence this exists to remove, for the input a custom runner
- is most likely to hand over by accident. NaNs sort last and form one group, so a cut inside
- them accepts all of them.
+ of the order a dataset happened to arrive in.
+
+ This is the single-cut answer; `_levels` computes the same group boundaries for *every* level,
+ which is what `aurc` integrates over. Two routes to one rule, so they are not merely asserted
+ to agree -- `test_a_single_cut_agrees_with_the_risk_coverage_curve` sweeps them against each
+ other, including over NaN and infinities, so an edit to either side fails a test rather than
+ diverging quietly. Computing one cut from the full scan instead cost 19% to 58% on
+ `selective_accuracy` for no change in any value.
+
+ NaN is handled explicitly rather than left to the comparison. Every comparison against a NaN
+ is false, so `sorted_desc >= nan` accepts nothing and the cut would fall back to "whichever
+ rows came first" -- the exact order-dependence this exists to remove, for the input a custom
+ runner is most likely to hand over by accident. NaNs sort last and form one group, so a cut
+ inside them accepts all of them.
+
+ The NaN branch is kept although `searchsorted` happens to answer the same thing -- numpy ranks
+ NaN last for a binary search too, verified over 63,697 NaN cuts without a disagreement. Leaving
+ it implicit would make a correct answer depend on an undocumented ordering convention, so the
+ branch stays and its mutant is reported as equivalent rather than as a test gap.
"""
cut = sorted_desc[k - 1]
if cut != cut: # NaN
return len(sorted_desc)
- return max(k, int(np.count_nonzero(sorted_desc >= cut)))
+ # `sorted_desc` descends, so its negation ascends and a binary search finds the end of the tie
+ # group in O(log n) rather than a full pass per call. NaNs rank above every number in numpy's
+ # sort order, so they sit at the end of the negated array and a finite cut never counts them.
+ return int(np.searchsorted(-sorted_desc, -cut, side="right"))
def _levels(confidences: Sequence[float], corrects: Sequence[bool]):
- """`(accepted, risk, n)` at each distinct confidence level, most confident first.
+ """`(accepted, hits, n)` at each distinct confidence level, most confident first.
One row per LEVEL, not per answer. A coverage cut is bought with a confidence threshold, and a
threshold accepts every answer at its own confidence -- `calibrate._select_abstention_threshold`
@@ -410,28 +424,41 @@ def _levels(confidences: Sequence[float], corrects: Sequence[bool]):
`accepted[i]` is how many answers a threshold at level `i` accepts, which is what lets `aurc`
weight a level by the answers it spans rather than counting a 1-answer level and a 999-answer
- level alike.
+ level alike. `hits[i]` is how many of those are correct.
+
+ Counts, not a risk already divided: `corrects` is 0.0/1.0, so every partial sum is an exact
+ integer in float64 and `hits / accepted` reproduces `np.mean` of the accepted slice bit for
+ bit. Returning `1 - hits / accepted` and recovering accuracy as `1 - risk` instead costs the
+ last bits of 157,289 values in a 1.2-million-value sweep -- harmless to a gate, but it would
+ make this change move numbers on datasets with no tie in them, which it is not allowed to do.
"""
# `atleast_1d`: a 0-dimensional input (a bare scalar) answered before this change, and indexing
# it with an argsort result raises.
c = np.atleast_1d(np.asarray(confidences, dtype=float))
- y = np.atleast_1d(np.asarray(corrects, dtype=bool).astype(float))
+ # `corrects` stays boolean here: `cumsum` of a boolean array counts in int64, which is exact,
+ # and skipping the float conversion saves a pass over the whole column.
+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))
n = len(y)
if n == 0:
return np.empty(0), np.empty(0), 0
order = np.argsort(-c, kind="mergesort") # desc, stable; NaN sorts last
c, y = c[order], y[order]
# The last answer of each tie group: where the next confidence differs, plus the final one.
+ boundary = c[1:] != c[:-1]
+ # NaN needs `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every
+ # NaN confidence becomes a level of its own, the running total walks them in dataset order, and
+ # the order-dependence this exists to remove survives for exactly the input a custom runner is
+ # most likely to hand over by accident. All NaNs are one group, ranked last.
#
- # `isnan` on both sides as well as `!=`, because NaN != NaN is true: without it every NaN
- # confidence becomes a level of its own, the running total walks them in dataset order, and the
- # order-dependence this exists to remove survives for exactly the input a custom runner is most
- # likely to hand over by accident. All NaNs are one group, ranked last.
- nan_pair = np.isnan(c[1:]) & np.isnan(c[:-1])
- last = np.flatnonzero(np.concatenate(((c[1:] != c[:-1]) & ~nan_pair, [True])))
+ # Guarded by one scalar test rather than paid for always: NaN sorts to the end, so if the last
+ # confidence is not NaN then none are, and two full `isnan` passes over the column can be
+ # skipped. That is the common case, and the two passes were a third of this function's cost at
+ # a million rows.
+ if np.isnan(c[-1]):
+ boundary &= ~(np.isnan(c[1:]) & np.isnan(c[:-1]))
+ last = np.flatnonzero(np.concatenate((boundary, [True])))
accepted = last.astype(float) + 1.0
- risk = 1.0 - np.cumsum(y)[last] / accepted
- return accepted, risk, n
+ return accepted, np.cumsum(y)[last], n
def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):
@@ -439,10 +466,10 @@ def _risk_coverage(confidences: Sequence[float], corrects: Sequence[bool]):
`_levels` with coverage as the x axis, for callers that want the curve rather than its area.
"""
- accepted, risk, n = _levels(confidences, corrects)
+ accepted, hits, n = _levels(confidences, corrects)
if n == 0:
return np.empty(0), np.empty(0)
- return accepted / n, risk
+ return accepted / n, 1.0 - hits / accepted
def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[float]:
@@ -454,9 +481,10 @@ def aurc(confidences: Sequence[float], corrects: Sequence[bool]) -> Optional[flo
"""
if not confidences:
return None
- accepted, risk, n = _levels(confidences, corrects)
+ accepted, hits, n = _levels(confidences, corrects)
if n == 0:
return None
+ risk = 1.0 - hits / accepted
# Each level's risk weighted by the ANSWERS it spans, which is what keeps this an area rather
# than an average of unevenly sized points: with 999 answers at one confidence and 1 at another,
# an unweighted mean of the two risks reports 5e-4 where the area is 1e-6.
@@ -477,6 +505,12 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],
coverage is the group's upper edge rather than the fraction asked for. That is the only
order-independent reading -- see `_risk_coverage` -- and it is what makes this comparable
across two permutations of the same dataset. Without ties, `k` is unchanged.
+
+ The group this cut lands in comes from `_accepted_at_cut`, which answers for one cut what
+ `_levels` answers for every level. The two are swept against each other by
+ `test_a_single_cut_agrees_with_the_risk_coverage_curve`, so they cannot drift apart silently;
+ deriving this from the full scan instead was measured at 19% to 58% slower here for no change
+ in any value.
"""
if not confidences or not 0.0 < coverage <= 1.0:
return None
@@ -484,13 +518,14 @@ def selective_accuracy(confidences: Sequence[float], corrects: Sequence[bool],
c = np.atleast_1d(np.asarray(confidences, dtype=float))
order = np.argsort(-c, kind="mergesort")
c = c[order]
- y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order].astype(float)
+ y = np.atleast_1d(np.asarray(corrects, dtype=bool))[order]
k = max(1, int(_math.ceil(coverage * len(y))))
- # Extend through the tie group the cut fell in -- see `_threshold_accepts`, which both this
- # and the risk-coverage curve go through, so the two cannot disagree about what a cut
- # accepts, and neither falls back to dataset order when the cut lands on a NaN.
- k = _threshold_accepts(c, k)
- return float(np.mean(y[:k]))
+ # Extend through the tie group the cut fell in, so the cut is a confidence threshold rather
+ # than a row index and cannot report whichever tie members arrived first.
+ accepted = _accepted_at_cut(c, k)
+ # `count_nonzero / accepted` rather than `mean`: both are exact for a boolean column, and this
+ # skips the float conversion of the accepted slice.
+ return float(np.count_nonzero(y[:accepted]) / accepted)
#: Coverage points reported as `selective_accuracy@NN`.
@@ -506,6 +541,62 @@ def is_coverage_metric(name: str) -> bool:
return name == "aurc" or name.startswith("selective_accuracy@")
+def coverage_definition_conflict(candidate_config: Any, baseline: Any) -> Optional[str]:
+ """Why a coverage metric must not be compared across these two documents, or None.
+
+ Comparing `aurc` or `selective_accuracy@*` between two reports subtracts one number from the
+ other, which is only meaningful if the same definition of a coverage cut produced both.
+ Definition 2 cuts on a confidence threshold, so a cut never splits a group of equal
+ confidences; definition 1 cut at a row index and depended on the order the dataset arrived in.
+ Measured over 400,001 tie-shaped datasets the two disagree by up to 0.35 on `aurc` and 0.50 on
+ `selective_accuracy@50`, so the subtraction is not merely noisy -- it is unsafe in the PASS
+ direction, letting a real regression through.
+
+ **Both sides are checked, not just the baseline.** A stale CANDIDATE is the more dangerous
+ case and the easier one to arrive at by accident: a report produced by a pinned older `laya`
+ in a container, or an archived artifact re-checked later, carries definition-1 numbers whose
+ row-order artifact can read BETTER than the truth. Gating that against a correctly regenerated
+ baseline passes a regression that the correctly scored report fails.
+
+ The absence of the key IS definition 1: every report written since it existed carries it. That
+ is why this refuses instead of following `comparable_to`'s "a key missing on either side is
+ unknown rather than a conflict" rule, which exists so a patch release cannot invalidate a
+ committed baseline. A definition change is not a patch release.
+
+ `ece` and `brier` read the same pairs but do not cut, so they stay comparable; callers gate
+ this behind `is_coverage_metric`.
+ """
+ here = _recorded_coverage_definition(candidate_config)
+ there = _recorded_coverage_definition(
+ baseline.get("config") if isinstance(baseline, dict) else None)
+ if here == COVERAGE_METRIC_DEFINITION and there == COVERAGE_METRIC_DEFINITION:
+ return None
+ stale = []
+ if here != COVERAGE_METRIC_DEFINITION:
+ stale.append("this report says %s" % _shown_definition(here))
+ if there != COVERAGE_METRIC_DEFINITION:
+ stale.append("the baseline says %s" % _shown_definition(there))
+ return ("%s, but a coverage cut here is definition %s -- regenerate it before comparing a "
+ "coverage metric" % (" and ".join(stale), COVERAGE_METRIC_DEFINITION))
+
+
+def _recorded_coverage_definition(config: Any) -> Any:
+ return config.get("coverage_metric_definition") if isinstance(config, dict) else None
+
+
+def _shown_definition(recorded: Any) -> str:
+ """A recorded definition, rendered for a message, with attacker-supplied content bounded.
+
+ The value comes from a report file, which on a pull-request gate is content the author
+ controls, and the message it lands in is printed to a CI log. A five-megabyte string in that
+ field produced a five-megabyte failure line; 40 characters is enough to diagnose a real one.
+ """
+ if recorded is None:
+ return "1 (unrecorded)"
+ shown = repr(recorded)
+ return shown if len(shown) <= 40 else shown[:40] + "... (truncated)"
+
+
def is_confidence_metric(name: str) -> bool:
"""True for a dataset-level metric computed from `(confidence, correct)` pairs, not per-answer
`scores` -- ECE and the selective-classification metrics. Policy counting treats these alike."""
@@ -609,6 +700,19 @@ class EvalReport:
if metric.endswith("_ms") and metric not in tolerances:
continue
allowed = float(tolerances.get(metric, 0.0))
+ # A coverage metric is only comparable when both documents were produced by the same
+ # definition of a cut. This path is the one `docs/evals.md` leads with, and
+ # `--gate-policy` is opt-in, so without this check the default gate performs exactly
+ # the cross-definition subtraction the policy gate refuses.
+ if is_coverage_metric(metric):
+ conflict = coverage_definition_conflict(self.config, baseline)
+ if conflict is not None:
+ deltas[metric] = {
+ "baseline": float(base_value),
+ "value": float(self.overall.get(metric, float("nan"))),
+ "diff": float("nan"), "tolerance": allowed, "incomparable": conflict}
+ ok = False
+ continue
if metric not in self.overall:
deltas[metric] = {"baseline": float(base_value), "value": float("nan"),
"diff": float("nan"), "tolerance": allowed, "missing": True}
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index 67328e18..7ab6ca16 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -256,6 +256,12 @@ def _print_deltas(deltas: Dict[str, Dict[str, Any]]) -> None:
if delta.get("missing"):
print("%-18s baseline=%.4f missing from the report" % (metric, delta["baseline"]))
continue
+ if delta.get("incomparable"):
+ # The diff is NaN by construction here, and "diff=+nan" on its own is not a
+ # diagnosis: print the reason the comparison was refused, not just its symptom.
+ print("%-18s baseline=%.4f value=%.4f not compared: %s"
+ % (metric, delta["baseline"], delta["value"], delta["incomparable"]))
+ continue
print("%-18s baseline=%.4f value=%.4f diff=%+.4f (tol %.4f)"
% (metric, delta["baseline"], delta["value"], delta["diff"], delta["tolerance"]))
diff --git a/tests/test_evals.py b/tests/test_evals.py
index ea9a80f5..914b3a9b 100644
--- a/tests/test_evals.py
+++ b/tests/test_evals.py
@@ -320,7 +320,7 @@ def test_report_metrics_survive_a_permuted_dataset():
def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmp_path):
"""A `max_drop` across definitions is unsafe in the PASS direction, so it must not be computed.
- The two definitions disagree by as much as 0.48 on the same answers. A slice recorded at 0.033
+ The two definitions disagree by up to 0.500 on the same answers. A slice recorded at 0.033
by the old one reads 0.517 under this one, so subtracting them lets a candidate that genuinely
dropped 0.217 clear a `max_drop` of 0.05 -- a real regression shipping because a baseline is
stale. `comparable_to` cannot see it: the dataset bytes and the question schema are identical,
@@ -337,10 +337,10 @@ def test_a_relative_rule_refuses_a_baseline_from_the_old_coverage_definition(tmp
stale = _coverage_gate_report(0.033333, None)
failures = _eval_policy.check_policy(candidate, policy, stale)
- assert any("regenerate the baseline" in f for f in failures), failures
- assert any("definition 1 (unrecorded)" in f for f in failures), failures
+ assert any("regenerate it before comparing" in f for f in failures), failures
+ assert any("the baseline says 1 (unrecorded)" in f for f in failures), failures
# an explicit version 1 is refused the same way
- assert any("regenerate the baseline" in f for f in
+ assert any("regenerate it before comparing" in f for f in
_eval_policy.check_policy(candidate, policy, _coverage_gate_report(0.033333, 1)))
# regenerated under this definition, the same gate CATCHES the regression it was hiding
@@ -382,6 +382,81 @@ def test_an_absolute_rule_needs_no_baseline_and_is_unaffected(tmp_path):
assert _eval_policy.check_policy(candidate, policy, None) == []
+def test_a_stale_candidate_is_refused_as_well_as_a_stale_baseline(tmp_path):
+ """The dangerous direction: an old-definition CANDIDATE can read BETTER than the truth.
+
+ A report scored by the old definition carries a row-order artifact. Here the same slice reads
+ 1.000 under definition 1 (its correct rows happened to be listed first) where definition 2 says
+ 0.500, so subtracting it from a correctly regenerated 0.500 baseline shows no drop at all and
+ the gate passes -- while the honestly scored candidate fails. Checking only the baseline leaves
+ this open, and it is the easier of the two to reach by accident: a CI artifact produced by a
+ pinned older `laya`, or an archived report re-checked after the baseline was regenerated.
+ """
+ from laya import _eval_policy
+
+ policy = _eval_policy.load_policy(_write_gate_policy(
+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",
+ "min_count": 5, "max_drop": 0.05}))
+ fresh_baseline = _coverage_gate_report(0.50, evals.COVERAGE_METRIC_DEFINITION)
+ stale_candidate = EvalReport(**_coverage_gate_report(1.0, None))
+
+ failures = _eval_policy.check_policy(stale_candidate, policy, fresh_baseline)
+ assert any("this report says 1 (unrecorded)" in f for f in failures), failures
+ # Proof the refusal is what stops it: the subtraction itself sees 1.0 against 0.50, which is a
+ # GAIN, so without the refusal there is no `max_drop` failure to catch the regression.
+ assert not any("max_drop" in f for f in failures), failures
+ # both sides stale is refused too, and says so about both
+ both = _eval_policy.check_policy(stale_candidate, policy, _coverage_gate_report(0.50, None))
+ assert any("this report says" in f and "the baseline says" in f for f in both), both
+
+
+def test_compare_refuses_a_coverage_metric_across_definitions():
+ """`EvalReport.compare` is the gate this project's docs lead with, and it also subtracts.
+
+ `--gate-policy` is opt-in; `--baseline --tolerance` is the documented default. A refusal that
+ covered only the policy gate would leave the usual path performing exactly the cross-definition
+ subtraction the policy gate exists to refuse.
+ """
+ report = EvalReport(**_coverage_gate_report(0.50, evals.COVERAGE_METRIC_DEFINITION))
+ stale = _coverage_gate_report(0.47, None)
+
+ ok, deltas = report.compare(stale, {"selective_accuracy@50": 0.10})
+ assert not ok, deltas
+ assert "incomparable" in deltas["selective_accuracy@50"], deltas
+ # Without the refusal this passes: 0.50 against 0.47 is a 0.03 move inside a 0.10 tolerance,
+ # while the regenerated baseline for the same answers is 0.03 and the real move is 0.47.
+ fresh = _coverage_gate_report(0.47, evals.COVERAGE_METRIC_DEFINITION)
+ ok_same_definition, deltas_same = report.compare(fresh, {"selective_accuracy@50": 0.10})
+ assert ok_same_definition, deltas_same
+ assert "incomparable" not in deltas_same["selective_accuracy@50"]
+ # `ece` does not cut, so an unstamped baseline stays comparable
+ ece = EvalReport(**_coverage_gate_report(0.10, evals.COVERAGE_METRIC_DEFINITION, metric="ece"))
+ ok_ece, deltas_ece = ece.compare(_coverage_gate_report(0.11, None, metric="ece"),
+ {"ece": 0.05})
+ assert ok_ece, deltas_ece
+
+
+def test_the_refusal_bounds_a_hostile_recorded_definition(tmp_path):
+ """The stamp is read from a report file, which on a pull-request gate the author writes.
+
+ The refusal names the value it found, and that message is printed to a CI log, so the value is
+ attacker-controlled content on its way into a log. Five megabytes in the field produced a
+ five-megabyte failure line before this was bounded.
+ """
+ from laya import _eval_policy
+
+ policy = _eval_policy.load_policy(_write_gate_policy(
+ tmp_path, {"slice": {"language": "zh"}, "metric": "selective_accuracy@50",
+ "min_count": 5, "max_drop": 0.05}))
+ candidate = EvalReport(**_coverage_gate_report(0.30, evals.COVERAGE_METRIC_DEFINITION))
+
+ failures = _eval_policy.check_policy(candidate, policy,
+ _coverage_gate_report(0.03, "x" * 5_000_000))
+ assert failures
+ assert max(len(f) for f in failures) < 400, max(len(f) for f in failures)
+ assert any("truncated" in f for f in failures), failures
+
+
def test_every_report_records_which_coverage_definition_produced_it():
report = evaluate(StubRunner({"s1": {"intent": choice_answer("a")}}),
Dataset([Example("s1", Q, {"intent": "a"})]))
@@ -410,6 +485,74 @@ def test_aurc_is_bit_identical_to_the_per_answer_mean_without_ties():
assert aurc(conf, corr) == float(np.mean(per_answer))
+def test_a_single_cut_agrees_with_the_risk_coverage_curve():
+ """The two routes to one rule must not drift apart, so a test holds them together.
+
+ `selective_accuracy` answers for one cut (`_accepted_at_cut`, a binary search) what `_levels`
+ answers for every level (a full scan, which `aurc` integrates). Deriving the single cut from
+ the full scan was measured 19% to 58% slower for no change in any value, so both exist -- and
+ a comment asserting they agree would be enforced by nothing. This is the enforcement: an edit
+ to either side fails here.
+
+ Exact equality, not `approx`: both routes divide one exact integer count by another, over the
+ same accepted set, so any difference at all is a real divergence rather than rounding.
+ """
+ import math
+ import random
+
+ rng = random.Random(13)
+ pool = [0.0, -0.0, 1.0, 0.5, 0.25, -1.0, 2.0,
+ float("nan"), float("inf"), -float("inf"), 1.0 - 2.0 ** -52]
+ for _ in range(4000):
+ n = rng.randint(1, 14)
+ conf = [rng.choice(pool) for _ in range(n)]
+ corr = [rng.random() < 0.5 for _ in range(n)]
+ accepted, hits, levels_n = evals._levels(conf, corr)
+ assert levels_n == n
+ for coverage in (0.01, 0.25, 0.5, 0.8, 1.0):
+ k = max(1, int(math.ceil(coverage * n)))
+ level = int(np.searchsorted(accepted, k, side="left"))
+ # the two routes must agree on how many answers the cut accepts ...
+ #
+ # Sorted the way `_levels` sorts: `argsort(-c)` ranks NaN LAST, where
+ # `np.sort(c)[::-1]` would rank it first and feed the lookup a different array.
+ column = np.asarray(conf, dtype=float)
+ sorted_desc = column[np.argsort(-column, kind="mergesort")]
+ assert evals._accepted_at_cut(sorted_desc, k) == int(accepted[level]), (
+ n, k, conf)
+ # ... and therefore on the accuracy over them
+ assert selective_accuracy(conf, corr, coverage) == float(hits[level] / accepted[level]), (
+ n, coverage, conf, corr)
+
+
+def test_selective_accuracy_is_bit_identical_to_the_accepted_slice_mean():
+ """Not `approx`: the claim is bit-identity with what this function answered before.
+
+ `selective_accuracy` now reads the same levels `aurc` reads, so it had to be shown that routing
+ it through them does not move a value. It returns `hits / accepted` -- the exact count of
+ correct answers over the exact count accepted -- because `corrects` is 0.0/1.0 and every
+ partial sum is therefore an exact integer in float64.
+
+ Recovering the same number as `1 - risk` instead, from a risk the levels had already divided,
+ round-trips through a subtraction and differs in the last bits on about one value in eight. A
+ gate would not notice; a user diffing two reports of the same no-tie data would, and this
+ change is not allowed to move a number that no tie could move.
+ """
+ import random
+
+ random.seed(11)
+ for _ in range(200):
+ n = random.randint(1, 40)
+ conf = random.sample([i / 10000 for i in range(1, 9999)], n) # distinct: no ties
+ corr = [random.random() < 0.6 for _ in range(n)]
+ order = np.argsort(-np.asarray(conf, dtype=float), kind="mergesort")
+ y = np.asarray(corr, dtype=bool)[order].astype(float)
+ for coverage in (0.1, 0.25, 0.5, 0.8, 1.0):
+ k = max(1, int(np.ceil(coverage * n)))
+ assert selective_accuracy(conf, corr, coverage) == float(np.mean(y[:k])), (
+ n, coverage, conf, corr)
+
+
def test_selective_metrics_reach_the_report():
# choice answers with a spread of confidence and correctness so the pairs exist
ds = Dataset([Example("s1", Q, {"intent": "a"}), Example("s2", Q, {"intent": "a"}),
diff --git a/tests/test_evals_api.py b/tests/test_evals_api.py
index 3a031d4b..2a1c8bc7 100644
--- a/tests/test_evals_api.py
+++ b/tests/test_evals_api.py
@@ -44,10 +44,16 @@ check_true("ScoreWithin names its tolerance", evals.ScoreWithin(0.25).name == "s
# --------------------------------------------------------------- exports / callables
for name in ("Dataset", "Example", "EvalError", "EvalReport", "evaluate", "ece", "assert_regression",
"REPORT_SCHEMA", "questions_fingerprint", "file_fingerprint",
- "brier", "aurc", "selective_accuracy", "is_confidence_metric"):
+ "brier", "aurc", "selective_accuracy", "is_confidence_metric",
+ # `_eval_policy` imports these three, so they are a contract between two modules in
+ # this package, not internals -- a rename would break the release gate, not just a
+ # caller.
+ "is_coverage_metric", "coverage_definition_conflict",
+ "COVERAGE_METRIC_DEFINITION"):
check_true("laya.evals.%s exists" % name, hasattr(evals, name))
check("REPORT_SCHEMA", evals.REPORT_SCHEMA, "laya-evals-report/1")
+check("the coverage-cut definition this code computes", evals.COVERAGE_METRIC_DEFINITION, 2)
check_true("EvalReport.comparable_to is part of the report contract",
callable(getattr(evals.EvalReport, "comparable_to", None)))
check("EvalReport fields are unchanged", [f.name for f in dataclasses.fields(evals.EvalReport)],