Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(evals): the --min-confidence gate flags, it does not overwrite the answer
aashish25444af3c15d691
diff --git a/docs/evals.md b/docs/evals.mdindex 8a65d0ec..3bc6d8ee 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -229,21 +229,26 @@ raising `TypeError` halfway through a long run.`--min-confidence T` forwards core's opt-in abstention threshold (#361) to every call the runmakes, so `Router` and `ONNXAgent` mark answers whose `answer_confidence` falls below `T` with-`low_confidence: True` before the harness sees them. Unlike grouping, this changes the answers-that score: the same run at `T=0` and `T=0.7` is a different experiment, and a `precision@coverage`-sweep is a series of these, not a single baseline drifting.+`low_confidence: True` and `abstention: "abstained"` before the harness sees them. The gate is a+*reporting* control, not a scoring one: `apply_confidence_gate` leaves+`answer["choice"] / ["noul"] / ["score"]` as the raw argmax and `_aggregate` reads only those+keys, so every accuracy, calibration and coverage number -- `ece`, `brier`, `aurc`,+`selective_accuracy@NN` -- is the same at `T=0` and `T=0.7`. What changes is the report's+`config.timing.min_confidence` and `config.timing.min_confidence_sent`, and any caller who acts+on the flag downstream of the harness.The accepted range is core's `laya.confidence.check_min_confidence` -- `[0.0, 1.0]`, finite, nota bool -- rather than a copy here, so a value the gate itself would reject fails as a usage error-(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for a-`precision@coverage` sweep, and a check that dropped it would hide the sweep's own floor.+(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for an+abstention sweep and the value `flag_low_confidence` treats as a no-op, so a check that dropped+it would hide which arm actually ran.A runner whose `predict` or (for a batched run) whose `predict_batch` predates the gate is-**refused with a named `EvalError`**, not scored without the threshold. Silently dropping a-scoring control is the class of lie this harness exists to prevent: the report would publish a-`precision@coverage` figure for a policy that never ran. `config.timing` records both the ask and-the fact: `min_confidence` is the threshold that was requested, `min_confidence_sent` says whether-any call this run made actually carried it.+**refused with a named `EvalError`**, not run without the threshold. Silently dropping it would+let `report.config["timing"]["min_confidence"]` name a threshold the harness never applied -- the+class of lie this harness exists to prevent, even though the metric numbers stay identical.+`config.timing` records both the ask and the fact: `min_confidence` is the threshold that was+requested, `min_confidence_sent` says whether any call this run made actually carried it.## Slicesdiff --git a/laya/evals.py b/laya/evals.pyindex f3c0c0e8..ff9bc556 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -831,12 +831,14 @@ def _takes_min_confidence(runner: Any, fn_name: str = "predict_batch") -> bool:"""Whether `runner`'s entry point accepts an abstention threshold on the call.The same signature check `_takes_sort_by_length` makes, applied to whichever entry point the- harness is about to call: `min_confidence` changes the answer (an abstention overwrites a- low-confidence choice), so it is a scoring control, not an optimisation. A runner that predates- the gate (#361) must still be scoreable -- silently dropping the threshold and reporting the- same run would give a `precision@coverage` number for a policy that never ran -- so when the- guard is false the harness raises rather than lies. The CLI catches the raise into a- pre-flight message before any checkpoint loads.+ harness is about to call. The gate is a *reporting* control, not a scoring one: it writes+ `low_confidence: True` and `abstention: "abstained"` on answers whose `answer_confidence`+ falls below the threshold, and leaves `answer["choice"] / ["noul"] / ["score"]` as the raw+ argmax -- so every metric in `_aggregate` reads the same numbers at every threshold. What+ changes is `report.config["timing"]["min_confidence"]` and `["min_confidence_sent"]`: those+ keys claim the gate ran. A runner that predates it cannot honour that claim, so when the+ guard is false the harness raises rather than publish a threshold it never applied. The CLI+ catches the raise into a pre-flight message before any checkpoint loads."""fn = getattr(runner, fn_name, None)if fn is None:@@ -867,13 +869,16 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evaluarunner that predates the knob is unaffected by a run that does not ask. Nothing about thescored answers changes -- the results come back in chunk order either way.- `min_confidence` is the abstention threshold `Router` and `ONNXAgent` apply to- `answer_confidence` (#361): answers below it come back abstained, so the run scores the- policy at that threshold, not the raw argmax. Unlike `sort_by_length` this changes the- answers, so a runner whose batch entry point (or whose single ``predict``, on the fallback- path) predates the gate is refused rather than silently scored without it -- the report- would otherwise publish a `precision@coverage` figure for an abstention policy that never- ran.+ `min_confidence` is the opt-in abstention threshold `Router` and `ONNXAgent` apply to+ `answer_confidence` (#361). `laya.confidence.apply_confidence_gate` marks every answer below+ it with `low_confidence: True` and `abstention: "abstained"`, and the raw argmax stays on+ `answer["choice"] / ["noul"] / ["score"]` -- so `_aggregate` publishes the same accuracy,+ calibration and coverage numbers at every threshold. Like `sort_by_length`, nothing about the+ scored answers changes. Unlike `sort_by_length`, the report itself names the threshold under+ `report.config["timing"]["min_confidence"]` and asserts it was sent under+ `["min_confidence_sent"]`, so a runner whose batch entry point (or whose single ``predict``,+ on the fallback path) predates the gate is refused: the harness will not publish a+ `min_confidence` it did not apply.`on_error` is ``"fail"`` (re-raise a runner error) or ``"skip"`` (record it and continue),the latter for evaluating a flaky fleet without aborting the whole run.@@ -889,11 +894,14 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evaluaif on_error not in ("fail", "skip"):raise EvalError("on_error must be 'fail' or 'skip', got %r" % on_error)# `min_confidence` is validated here -- and, unlike `sort_by_length`, a run that asks for it- # on a runner that does not accept it is refused -- because an abstention threshold changes- # which answers score as correct. Silently dropping it would publish a `precision@coverage`- # number for a policy that never ran, which is exactly the class of lie a baseline report is- # supposed to prevent. `laya.confidence.check_min_confidence` is the same validator the Router- # uses, so the accepted range cannot drift from what the gate itself enforces.+ # on a runner that does not accept it is refused. The gate does not change which answers score+ # as correct: `apply_confidence_gate` writes `low_confidence` and `abstention` state fields+ # while leaving `answer["choice"] / ["noul"] / ["score"]` as the raw argmax, so `_aggregate`+ # publishes the same numbers at every threshold. What would be a lie is+ # `report.config["timing"]["min_confidence"]`: it names the threshold as if the run applied+ # it. A runner that predates the gate (#361) cannot honour that claim, so the guard raises.+ # `laya.confidence.check_min_confidence` is the same validator the Router uses, so the+ # accepted range cannot drift from what the abstention gate itself enforces.try:mc = check_min_confidence(min_confidence) if min_confidence is not None else Noneexcept ValueError as exc:@@ -915,7 +923,7 @@ def evaluate(runner: Any, dataset: Dataset, evaluators: Optional[Sequence[Evalua"evaluate(min_confidence=%r) refused: this runner's %s does not accept the ""abstention threshold. Either use a Router/ONNXAgent that gates on ""answer_confidence (#361), or drop the threshold -- the report would "- "otherwise score a policy that never ran." % (min_confidence, target))+ "otherwise name a `min_confidence` it never applied." % (min_confidence, target))evaluators = list(evaluators) if evaluators is not None else default_evaluators()cases: List[Dict[str, Any]] = []waits: List[float] = [] # what each request actually waited: its chunk's whole calldiff --git a/laya/evals_cli.py b/laya/evals_cli.pyindex 66897dc1..5f28d7f3 100644--- a/laya/evals_cli.py+++ b/laya/evals_cli.py@@ -192,12 +192,16 @@ def _build_parser() -> argparse.ArgumentParser:"examples into the same forward pass so each pads to a shorter maximum; ""scores the same answers, in the same order")run.add_argument("--min-confidence", dest="min_confidence", type=float, metavar="THRESHOLD",- help="abstention threshold on `answer_confidence` (#361): answers below it "- "come back abstained, so the run scores the policy at that threshold "- "rather than the raw argmax. Accepted range is core's -- "- "`laya.confidence.check_min_confidence` -- not a copy of it here, and a "- "runner that predates the gate is refused with a named error rather "- "than silently scored without it")+ help="abstention threshold on `answer_confidence` (#361): every answer below "+ "it is returned flagged with `low_confidence: True` and "+ "`abstention: \"abstained\"`. The gate writes those state fields and "+ "leaves `answer[\"choice\"] / [\"noul\"] / [\"score\"]` as the raw argmax, "+ "so the metrics are identical at every threshold -- what changes is the "+ "report's config, which names the threshold and asserts it was sent. "+ "Accepted range is core's -- `laya.confidence.check_min_confidence` -- "+ "not a copy of it here, and a runner that predates the gate is refused "+ "with a named error rather than publishing a `min_confidence` it never "+ "applied")run.add_argument("--on-error", choices=("fail", "skip"), default="fail",help="'fail' (the default) stops the run when a runner call raises; 'skip' ""lists every row it could not score under the report's config.errored "diff --git a/tests/test_evals.py b/tests/test_evals.pyindex 59cbaea2..5c5bb5d7 100644--- a/tests/test_evals.py+++ b/tests/test_evals.py@@ -905,10 +905,14 @@ def test_a_pass_through_wrapper_is_given_the_knob():# ------------------------------------------------------- abstention gate (#361 knob)## `Router.predict` / `Router.predict_batch` and `ONNXAgent.predict` gate on `answer_confidence`-# and mark answers below `min_confidence` with `low_confidence: True` (#361). That is a scoring-# control: an answer below the threshold is not the same decision as one above. A `laya-evals run`-# that could not pass it through had no way to measure `precision@coverage` at any threshold-# except by wrapping a Router by hand -- the exact class of thing this harness exists to be.+# and mark answers below `min_confidence` with `low_confidence: True` and `abstention:+# "abstained"` (#361). The gate is a *reporting* control: `apply_confidence_gate` leaves+# `answer["choice"] / ["noul"] / ["score"]` as the raw argmax, so every metric in `_aggregate`+# reads the same numbers at every threshold. What the harness forwards is the *claim* the+# report makes: `report.config["timing"]["min_confidence"]` names the threshold and+# `["min_confidence_sent"]` asserts the call carried it. A `laya-evals run` that could not pass+# the kwarg through had no way to publish that claim on a real Router -- the exact class of+# thing this harness exists to be.UNSET = object() # no caller sends it, so `mc is UNSET` distinguishes "absent" from 0.0 or None@@ -991,8 +995,9 @@ def test_min_confidence_zero_is_still_a_threshold():def test_a_runner_without_the_gate_is_refused_not_silently_scored():- """Silently dropping a scoring control would report `precision@coverage` for a policy that- never ran. `RequestsRunner`'s `predict_batch(requests, batch_size=None)` is exactly what a+ """Silently dropping the kwarg would let `report.config["timing"]["min_confidence"]` name a+ threshold the harness never applied -- a lying report even though the metric numbers stay+ identical. `RequestsRunner`'s `predict_batch(requests, batch_size=None)` is exactly what apre-#361 runner looks like, so a run that asks for a threshold on it must fail loudly."""with pytest.raises(EvalError) as exc:evaluate(RequestsRunner(ANSWERS3), _three(), evaluators=[ChoiceAccuracy()],@@ -2070,7 +2075,7 @@ def test_docs_and_the_cli_name_the_same_flags():taught - registered)# The other half of parity: a registered flag nobody documents is unreachable in practice. The# page that teaches `--batch-size` has to teach the grouping that makes a bounded pass cheaper,- # and the abstention threshold that changes which answers score at all.+ # and the abstention threshold the report names even though the gate flags rather than rescores.assert {"--batch-size", "--sort-by-length", "--min-confidence"} <= taught, \"the evals page teaches the batch size but not the grouping or abstention knobs"assert "--score-within" in quickstart, "the tolerance metric has to be reachable from the quickstart"diff --git a/tests/test_evals_api.py b/tests/test_evals_api.pyindex 2a1c8bc7..82815b97 100644--- a/tests/test_evals_api.py+++ b/tests/test_evals_api.py@@ -84,8 +84,10 @@ check_true("evaluate: the abstention knob sits after the batch size it may apply# The two batch entry points the CLI wires up: both have to take the knob by the same name, or a# `--sort-by-length` run reports `sort_by_length_sent: false` for the surface it ships. The same# is true for the abstention threshold; a `--min-confidence` run that silently dropped the-# argument would publish a `precision@coverage` figure for a policy that never ran, so the CLI-# runner shapes have to accept the kwarg under the same name the guard on `evaluate` checks.+# argument would publish `report.config["timing"]["min_confidence"]` naming a threshold that was+# never applied -- the metrics stay identical (the gate flags, it does not overwrite the argmax),+# but the report itself would lie -- so the CLI runner shapes have to accept the kwarg under the+# same name the guard on `evaluate` checks.from laya import evals_cli # noqa: E402for label, fn in (("RouterRunner.predict_batch", evals_cli.RouterRunner.predict_batch),