Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(confidence): clear the stale low_confidence marker at min_confidence 0.0
Aashish2863500ac6fc
diff --git a/docs/evals.md b/docs/evals.mdindex dcba75b8..53a8c79b 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -386,8 +386,8 @@ on the flag downstream of the harness.The accepted range is core's `laya.confidence.check_min_confidence` -- `[0.0, 1.0]`, finite, nota bool -- rather than a copy here, so a value the gate itself would reject fails as a usage error(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for an-abstention sweep and the value `flag_low_confidence` treats as a no-op, so a check that dropped-it would hide which arm actually ran.+abstention sweep and the threshold at which `flag_low_confidence` marks no answer, so a check that+dropped it would hide which arm actually ran.A runner whose `predict` or (for a batched run) whose `predict_batch` predates the gate is**refused with a named `EvalError`**, not run without the threshold. Silently dropping it woulddiff --git a/laya/confidence.py b/laya/confidence.pyindex f70f1e79..c7530d13 100644--- a/laya/confidence.py+++ b/laya/confidence.py@@ -210,10 +210,12 @@ def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) ->`min_confidence` is either a float (one threshold for every answer) or a per-bucket mapping(#394), in which case each answer is gated at the threshold of its own option-count bucket via:func:`resolve_min_confidence`.++ Exactly ``0.0`` runs this same loop rather than short-circuiting out of it. Nothing falls below+ it, so it marks no answer -- and the pass is what clears a marker an earlier, stricter threshold+ left on a reused payload, which is the state #910 was filed about."""is_map = isinstance(min_confidence, dict)- if not is_map and min_confidence == 0.0:- returnfor res in results:answers = res.get("answers") if isinstance(res, dict) else Noneif not isinstance(answers, dict):@@ -272,10 +274,10 @@ def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: OptionaThe flag itself stays :func:`flag_low_confidence`'s -- this delegates rather thanre-implementing the rule, so the boolean and the reported state cannot drift apart.- A `min_confidence` of exactly ``0.0`` *was* set, so states are reported, and- :func:`flag_low_confidence` treats ``0.0`` as a no-op because nothing can fall below it. Every- answer carrying a usable confidence therefore reads ``passed``, and the threshold echo is what- distinguishes that from a real pass at a real threshold.+ A `min_confidence` of exactly ``0.0`` *was* set, so states are reported. Nothing can fall below+ it, so :func:`flag_low_confidence` marks no answer at it and clears whatever a stricter threshold+ left on a reused payload. Every answer carrying a usable confidence therefore reads ``passed``,+ and the threshold echo is what distinguishes that from a real pass at a real threshold."""if min_confidence is None:return@@ -288,8 +290,6 @@ def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optionafor a in answers.values():if not isinstance(a, dict):continue- if not is_map and min_confidence == 0.0:- a.pop("low_confidence", None)if a.get("low_confidence"):a["abstention"] = GATE_ABSTAINEDelif _gate_confidence(a) is None:diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex 50e40904..4a0e60ff 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -281,6 +281,26 @@ clean_res = [{"answers": {"q": {"answer_confidence": 0.95}}}]flag_low_confidence(clean_res, 0.5)check_true("stale/unflagged answer stays clean", "low_confidence" not in clean_res[0]["answers"]["q"])+# `0.0` belongs to this case as much as `0.5` does -- it is the most permissive threshold there is --+# but the function used to short-circuit on it before the clearing pass, so a stricter threshold's+# marker survived a re-evaluation at `0.0`.+zero_res = [{"answers": {"intent": {"choice": "refund", "answer_confidence": 0.8}}}]+zero_ans = zero_res[0]["answers"]["intent"]+flag_low_confidence(zero_res, 0.9)+check_true("stale/flag 0.9 sets flag before the 0.0 pass", zero_ans.get("low_confidence") is True)+flag_low_confidence(zero_res, 0.0)+check_true("stale/flag 0.0 clears flag", "low_confidence" not in zero_ans)++# One rule, one implementation: the boolean the flag writes and the state the gate reports cannot+# disagree about the same threshold change, which is the claim `apply_confidence_gate`'s docstring+# makes about delegating rather than re-implementing.+gate_both = [{"answers": {"intent": {"choice": "refund", "answer_confidence": 0.8}}}]+apply_confidence_gate(gate_both, 0.9)+apply_confidence_gate(gate_both, 0.0)+check_true("stale/flag and gate agree after 0.9 -> 0.0",+ ("low_confidence" in zero_ans) == ("low_confidence" in gate_both[0]["answers"]["intent"]),+ repr((zero_ans, gate_both[0]["answers"]["intent"])))+# 2. apply_confidence_gate clears stale flag and updates abstention state on re-evaluationgate_reuse = [{"answers": {