Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(evals): the --min-confidence gate flags, it does not overwrite the answer

aashish25444af3c15d691
diff --git a/docs/evals.md b/docs/evals.md
index 8a65d0ec..3bc6d8ee 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -229,21 +229,26 @@ raising `TypeError` halfway through a long run.
`--min-confidence T` forwards core's opt-in abstention threshold (#361) to every call the run
makes, so `Router` and `ONNXAgent` mark answers whose `answer_confidence` falls below `T` with
-`low_confidence: True` before the harness sees them. Unlike grouping, this changes the answers
-that score: the same run at `T=0` and `T=0.7` is a different experiment, and a `precision@coverage`
-sweep is a series of these, not a single baseline drifting.
+`low_confidence: True` and `abstention: "abstained"` before the harness sees them. The gate is a
+*reporting* control, not a scoring one: `apply_confidence_gate` leaves
+`answer["choice"] / ["noul"] / ["score"]` as the raw argmax and `_aggregate` reads only those
+keys, so every accuracy, calibration and coverage number -- `ece`, `brier`, `aurc`,
+`selective_accuracy@NN` -- is the same at `T=0` and `T=0.7`. What changes is the report's
+`config.timing.min_confidence` and `config.timing.min_confidence_sent`, and any caller who acts
+on the flag downstream of the harness.
The accepted range is core's `laya.confidence.check_min_confidence` -- `[0.0, 1.0]`, finite, not
a bool -- rather than a copy here, so a value the gate itself would reject fails as a usage error
-(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for a
-`precision@coverage` sweep, and a check that dropped it would hide the sweep's own floor.
+(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for an
+abstention sweep and the value `flag_low_confidence` treats as a no-op, so a check that dropped
+it would hide which arm actually ran.
A runner whose `predict` or (for a batched run) whose `predict_batch` predates the gate is
-**refused with a named `EvalError`**, not scored without the threshold. Silently dropping a
-scoring control is the class of lie this harness exists to prevent: the report would publish a
-`precision@coverage` figure for a policy that never ran. `config.timing` records both the ask and
-the fact: `min_confidence` is the threshold that was requested, `min_confidence_sent` says whether
-any call this run made actually carried it.
+**refused with a named `EvalError`**, not run without the threshold. Silently dropping it would
+let `report.config["timing"]["min_confidence"]` name a threshold the harness never applied -- the
+class of lie this harness exists to prevent, even though the metric numbers stay identical.
+`config.timing` records both the ask and the fact: `min_confidence` is the threshold that was
+requested, `min_confidence_sent` says whether any call this run made actually carried it.
## Slices
diff --git a/laya/evals.py b/laya/evals.py
index f3c0c0e8..ff9bc556 100644
--- a/laya/evals.py
+++ b/laya/evals.py
@@ -831,12 +831,14 @@ def _takes_min_confidence(runner: Any, fn_name: str = "predict_batch") -> bool:
"""Whether `runner`'s entry point accepts an abstention threshold on the call.
The same signature check `_takes_sort_by_length` makes, applied to whichever entry point the
- harness is about to call: `min_confidence` changes the answer (an abstention overwrites a
- low-confidence choice), so it is a scoring control, not an optimisation. A runner that predates
- the gate (#361) must still be scoreable -- silently dropping the threshold and reporting the
- same run would give a `precision@coverage` number for a policy that never ran -- so when the
- guard is false the harness raises rather than lies. The CLI catches the raise into a
- pre-flight message before any checkpoint loads.
+ harness is about to call. The gate is a *reporting* control, not a scoring one: it writes
+ `low_confidence: True` and `abstention: "abstained"` on answers whose `answer_confidence`
+ falls below the threshold, and leaves `answer["choice"] / ["noul"] / ["score"]` as the raw
+ argmax -- so every metric in `_aggregate` reads the same numbers at every threshold. What
+ changes is `report.config["timing"]["min_confidence"]` and `["min_confidence_sent"]`: those
+ keys claim the gate ran. A runner that predates it cannot honour that claim, so when the
+ guard is false the harness raises rather than publish a threshold it never applied. The CLI
+ catches the raise into a pre-flight message before any checkpoint loads.
"""
fn = getattr(runner, fn_name, None)
if fn is None:
@@ -867,13 +869,16 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evalua
runner that predates the knob is unaffected by a run that does not ask. Nothing about the
scored answers changes -- the results come back in chunk order either way.
- `min_confidence` is the abstention threshold `Router` and `ONNXAgent` apply to
- `answer_confidence` (#361): answers below it come back abstained, so the run scores the
- policy at that threshold, not the raw argmax. Unlike `sort_by_length` this changes the
- answers, so a runner whose batch entry point (or whose single ``predict``, on the fallback
- path) predates the gate is refused rather than silently scored without it -- the report
- would otherwise publish a `precision@coverage` figure for an abstention policy that never
- ran.
+ `min_confidence` is the opt-in abstention threshold `Router` and `ONNXAgent` apply to
+ `answer_confidence` (#361). `laya.confidence.apply_confidence_gate` marks every answer below
+ it with `low_confidence: True` and `abstention: "abstained"`, and the raw argmax stays on
+ `answer["choice"] / ["noul"] / ["score"]` -- so `_aggregate` publishes the same accuracy,
+ calibration and coverage numbers at every threshold. Like `sort_by_length`, nothing about the
+ scored answers changes. Unlike `sort_by_length`, the report itself names the threshold under
+ `report.config["timing"]["min_confidence"]` and asserts it was sent under
+ `["min_confidence_sent"]`, so a runner whose batch entry point (or whose single ``predict``,
+ on the fallback path) predates the gate is refused: the harness will not publish a
+ `min_confidence` it did not apply.
`on_error` is ``"fail"`` (re-raise a runner error) or ``"skip"`` (record it and continue),
the latter for evaluating a flaky fleet without aborting the whole run.
@@ -889,11 +894,14 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evalua
if on_error not in ("fail", "skip"):
raise EvalError("on_error must be 'fail' or 'skip', got %r" % on_error)
# `min_confidence` is validated here -- and, unlike `sort_by_length`, a run that asks for it
- # on a runner that does not accept it is refused -- because an abstention threshold changes
- # which answers score as correct. Silently dropping it would publish a `precision@coverage`
- # number for a policy that never ran, which is exactly the class of lie a baseline report is
- # supposed to prevent. `laya.confidence.check_min_confidence` is the same validator the Router
- # uses, so the accepted range cannot drift from what the gate itself enforces.
+ # on a runner that does not accept it is refused. The gate does not change which answers score
+ # as correct: `apply_confidence_gate` writes `low_confidence` and `abstention` state fields
+ # while leaving `answer["choice"] / ["noul"] / ["score"]` as the raw argmax, so `_aggregate`
+ # publishes the same numbers at every threshold. What would be a lie is
+ # `report.config["timing"]["min_confidence"]`: it names the threshold as if the run applied
+ # it. A runner that predates the gate (#361) cannot honour that claim, so the guard raises.
+ # `laya.confidence.check_min_confidence` is the same validator the Router uses, so the
+ # accepted range cannot drift from what the abstention gate itself enforces.
try:
mc = check_min_confidence(min_confidence) if min_confidence is not None else None
except ValueError as exc:
@@ -915,7 +923,7 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evalua
"evaluate(min_confidence=%r) refused: this runner's %s does not accept the "
"abstention threshold. Either use a Router/ONNXAgent that gates on "
"answer_confidence (#361), or drop the threshold -- the report would "
- "otherwise score a policy that never ran." % (min_confidence, target))
+ "otherwise name a `min_confidence` it never applied." % (min_confidence, target))
evaluators = list(evaluators) if evaluators is not None else default_evaluators()
cases: List[Dict[str, Any]] = []
waits: List[float] = [] # what each request actually waited: its chunk's whole call
diff --git a/laya/evals_cli.py b/laya/evals_cli.py
index 66897dc1..5f28d7f3 100644
--- a/laya/evals_cli.py
+++ b/laya/evals_cli.py
@@ -192,12 +192,16 @@ def _build_parser() -> argparse.ArgumentParser:
"examples into the same forward pass so each pads to a shorter maximum; "
"scores the same answers, in the same order")
run.add_argument("--min-confidence", dest="min_confidence", type=float, metavar="THRESHOLD",
- help="abstention threshold on `answer_confidence` (#361): answers below it "
- "come back abstained, so the run scores the policy at that threshold "
- "rather than the raw argmax. Accepted range is core's -- "
- "`laya.confidence.check_min_confidence` -- not a copy of it here, and a "
- "runner that predates the gate is refused with a named error rather "
- "than silently scored without it")
+ help="abstention threshold on `answer_confidence` (#361): every answer below "
+ "it is returned flagged with `low_confidence: True` and "
+ "`abstention: \"abstained\"`. The gate writes those state fields and "
+ "leaves `answer[\"choice\"] / [\"noul\"] / [\"score\"]` as the raw argmax, "
+ "so the metrics are identical at every threshold -- what changes is the "
+ "report's config, which names the threshold and asserts it was sent. "
+ "Accepted range is core's -- `laya.confidence.check_min_confidence` -- "
+ "not a copy of it here, and a runner that predates the gate is refused "
+ "with a named error rather than publishing a `min_confidence` it never "
+ "applied")
run.add_argument("--on-error", choices=("fail", "skip"), default="fail",
help="'fail' (the default) stops the run when a runner call raises; 'skip' "
"lists every row it could not score under the report's config.errored "
diff --git a/tests/test_evals.py b/tests/test_evals.py
index 59cbaea2..5c5bb5d7 100644
--- a/tests/test_evals.py
+++ b/tests/test_evals.py
@@ -905,10 +905,14 @@ def test_a_pass_through_wrapper_is_given_the_knob():
# ------------------------------------------------------- abstention gate (#361 knob)
#
# `Router.predict` / `Router.predict_batch` and `ONNXAgent.predict` gate on `answer_confidence`
-# and mark answers below `min_confidence` with `low_confidence: True` (#361). That is a scoring
-# control: an answer below the threshold is not the same decision as one above. A `laya-evals run`
-# that could not pass it through had no way to measure `precision@coverage` at any threshold
-# except by wrapping a Router by hand -- the exact class of thing this harness exists to be.
+# and mark answers below `min_confidence` with `low_confidence: True` and `abstention:
+# "abstained"` (#361). The gate is a *reporting* control: `apply_confidence_gate` leaves
+# `answer["choice"] / ["noul"] / ["score"]` as the raw argmax, so every metric in `_aggregate`
+# reads the same numbers at every threshold. What the harness forwards is the *claim* the
+# report makes: `report.config["timing"]["min_confidence"]` names the threshold and
+# `["min_confidence_sent"]` asserts the call carried it. A `laya-evals run` that could not pass
+# the kwarg through had no way to publish that claim on a real Router -- the exact class of
+# thing this harness exists to be.
UNSET = object() # no caller sends it, so `mc is UNSET` distinguishes "absent" from 0.0 or None
@@ -991,8 +995,9 @@ def test_min_confidence_zero_is_still_a_threshold():
def test_a_runner_without_the_gate_is_refused_not_silently_scored():
- """Silently dropping a scoring control would report `precision@coverage` for a policy that
- never ran. `RequestsRunner`'s `predict_batch(requests, batch_size=None)` is exactly what a
+ """Silently dropping the kwarg would let `report.config["timing"]["min_confidence"]` name a
+ threshold the harness never applied -- a lying report even though the metric numbers stay
+ identical. `RequestsRunner`'s `predict_batch(requests, batch_size=None)` is exactly what a
pre-#361 runner looks like, so a run that asks for a threshold on it must fail loudly."""
with pytest.raises(EvalError) as exc:
evaluate(RequestsRunner(ANSWERS3), _three(), evaluators=[ChoiceAccuracy()],
@@ -2070,7 +2075,7 @@ def test_docs_and_the_cli_name_the_same_flags():
taught - registered)
# The other half of parity: a registered flag nobody documents is unreachable in practice. The
# page that teaches `--batch-size` has to teach the grouping that makes a bounded pass cheaper,
- # and the abstention threshold that changes which answers score at all.
+ # and the abstention threshold the report names even though the gate flags rather than rescores.
assert {"--batch-size", "--sort-by-length", "--min-confidence"} <= taught, \
"the evals page teaches the batch size but not the grouping or abstention knobs"
assert "--score-within" in quickstart, "the tolerance metric has to be reachable from the quickstart"
diff --git a/tests/test_evals_api.py b/tests/test_evals_api.py
index 2a1c8bc7..82815b97 100644
--- a/tests/test_evals_api.py
+++ b/tests/test_evals_api.py
@@ -84,8 +84,10 @@ check_true("evaluate: the abstention knob sits after the batch size it may apply
# The two batch entry points the CLI wires up: both have to take the knob by the same name, or a
# `--sort-by-length` run reports `sort_by_length_sent: false` for the surface it ships. The same
# is true for the abstention threshold; a `--min-confidence` run that silently dropped the
-# argument would publish a `precision@coverage` figure for a policy that never ran, so the CLI
-# runner shapes have to accept the kwarg under the same name the guard on `evaluate` checks.
+# argument would publish `report.config["timing"]["min_confidence"]` naming a threshold that was
+# never applied -- the metrics stay identical (the gate flags, it does not overwrite the argmax),
+# but the report itself would lie -- so the CLI runner shapes have to accept the kwarg under the
+# same name the guard on `evaluate` checks.
from laya import evals_cli # noqa: E402
for label, fn in (("RouterRunner.predict_batch", evals_cli.RouterRunner.predict_batch),