Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(confidence): clear the stale low_confidence marker at min_confidence 0.0

Aashish2863500ac6fc
diff --git a/docs/evals.md b/docs/evals.md
index dcba75b8..53a8c79b 100644
--- a/docs/evals.md
+++ b/docs/evals.md
@@ -386,8 +386,8 @@ on the flag downstream of the harness.
The accepted range is core's `laya.confidence.check_min_confidence` -- `[0.0, 1.0]`, finite, not
a bool -- rather than a copy here, so a value the gate itself would reject fails as a usage error
(exit 2) before any checkpoint loads. `0.0` is a legal ask: it is the control arm for an
-abstention sweep and the value `flag_low_confidence` treats as a no-op, so a check that dropped
-it would hide which arm actually ran.
+abstention sweep and the threshold at which `flag_low_confidence` marks no answer, so a check that
+dropped it would hide which arm actually ran.
A runner whose `predict` or (for a batched run) whose `predict_batch` predates the gate is
**refused with a named `EvalError`**, not run without the threshold. Silently dropping it would
diff --git a/laya/confidence.py b/laya/confidence.py
index f70f1e79..c7530d13 100644
--- a/laya/confidence.py
+++ b/laya/confidence.py
@@ -210,10 +210,12 @@ def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) ->
`min_confidence` is either a float (one threshold for every answer) or a per-bucket mapping
(#394), in which case each answer is gated at the threshold of its own option-count bucket via
:func:`resolve_min_confidence`.
+
+ Exactly ``0.0`` runs this same loop rather than short-circuiting out of it. Nothing falls below
+ it, so it marks no answer -- and the pass is what clears a marker an earlier, stricter threshold
+ left on a reused payload, which is the state #910 was filed about.
"""
is_map = isinstance(min_confidence, dict)
- if not is_map and min_confidence == 0.0:
- return
for res in results:
answers = res.get("answers") if isinstance(res, dict) else None
if not isinstance(answers, dict):
@@ -272,10 +274,10 @@ def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optiona
The flag itself stays :func:`flag_low_confidence`'s -- this delegates rather than
re-implementing the rule, so the boolean and the reported state cannot drift apart.
- A `min_confidence` of exactly ``0.0`` *was* set, so states are reported, and
- :func:`flag_low_confidence` treats ``0.0`` as a no-op because nothing can fall below it. Every
- answer carrying a usable confidence therefore reads ``passed``, and the threshold echo is what
- distinguishes that from a real pass at a real threshold.
+ A `min_confidence` of exactly ``0.0`` *was* set, so states are reported. Nothing can fall below
+ it, so :func:`flag_low_confidence` marks no answer at it and clears whatever a stricter threshold
+ left on a reused payload. Every answer carrying a usable confidence therefore reads ``passed``,
+ and the threshold echo is what distinguishes that from a real pass at a real threshold.
"""
if min_confidence is None:
return
@@ -288,8 +290,6 @@ def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optiona
for a in answers.values():
if not isinstance(a, dict):
continue
- if not is_map and min_confidence == 0.0:
- a.pop("low_confidence", None)
if a.get("low_confidence"):
a["abstention"] = GATE_ABSTAINED
elif _gate_confidence(a) is None:
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index 50e40904..4a0e60ff 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -281,6 +281,26 @@ clean_res = [{"answers": {"q": {"answer_confidence": 0.95}}}]
flag_low_confidence(clean_res, 0.5)
check_true("stale/unflagged answer stays clean", "low_confidence" not in clean_res[0]["answers"]["q"])
+# `0.0` belongs to this case as much as `0.5` does -- it is the most permissive threshold there is --
+# but the function used to short-circuit on it before the clearing pass, so a stricter threshold's
+# marker survived a re-evaluation at `0.0`.
+zero_res = [{"answers": {"intent": {"choice": "refund", "answer_confidence": 0.8}}}]
+zero_ans = zero_res[0]["answers"]["intent"]
+flag_low_confidence(zero_res, 0.9)
+check_true("stale/flag 0.9 sets flag before the 0.0 pass", zero_ans.get("low_confidence") is True)
+flag_low_confidence(zero_res, 0.0)
+check_true("stale/flag 0.0 clears flag", "low_confidence" not in zero_ans)
+
+# One rule, one implementation: the boolean the flag writes and the state the gate reports cannot
+# disagree about the same threshold change, which is the claim `apply_confidence_gate`'s docstring
+# makes about delegating rather than re-implementing.
+gate_both = [{"answers": {"intent": {"choice": "refund", "answer_confidence": 0.8}}}]
+apply_confidence_gate(gate_both, 0.9)
+apply_confidence_gate(gate_both, 0.0)
+check_true("stale/flag and gate agree after 0.9 -> 0.0",
+ ("low_confidence" in zero_ans) == ("low_confidence" in gate_both[0]["answers"]["intent"]),
+ repr((zero_ans, gate_both[0]["answers"]["intent"])))
+
# 2. apply_confidence_gate clears stale flag and updates abstention state on re-evaluation
gate_reuse = [{
"answers": {