Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(confidence): report the abstention gate's state, only where a gate ran

Bruce-Yii6bbdae05cbb5
diff --cc README.md
index 97a7307f,328a6194..9274451c
--- a/README.md
+++ b/README.md
@@@ -97,7 -97,7 +97,7 @@@ The shipped checkpoints work zero-shot
## What's new in 0.3.21
* **ONNX catches up with PyTorch.** `ONNXAgent` gains `predict_batch` (with `sort_by_length`), `predict_long` and `decide_batch`, `scripts/export_onnx.py --quantize` writes a per-channel INT8 copy for CPU, and `laya-evals run --onnx` scores an export with the same gates as the torch path.
- * **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them. Every answer also reports `abstention` — `not_configured`, `passed`, `abstained` or `unevaluated` — so a caller can tell an ungated run from a gated one that cleared.
-* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them.
++* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them. When a threshold is set, every answer also reports `abstention` — `passed`, `abstained` or `unevaluated` — so a caller can tell a gate that cleared from a gate that never ran; with no threshold, nothing is added at all.
* **Batch everywhere.** `decide_batch`, `Router.predict_long`, `laya --batch FILE`, the MCP `laya_predict_batch` / `laya_route_batch` / `laya_decide` tools, and LangChain `batch()` / `abatch()` all run on shared forward passes. New `LayaDecision` (LangChain), LlamaIndex selectors (`laya[llamaindex]`) and CrewAI routing (`laya[crewai]`).
* **Per-request token budget.** `max_len` / `head_max_len` now reach every surface: `laya-serve` (capped by `LAYA_MAX_TOKEN_BUDGET`), `Router.predict_batch` requests, the CLI (`--questions`, `--max-len`, `--head-max-len`), MCP tools and LangChain nodes.
* **Operations.** `LAYA_MAX_LOADED`, `LAYA_REVISION` and per-checkpoint SHA-256 maps; `/health` reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries `Retry-After`; `compile=True` no longer recompiles for every request shape.
@@@ -719,17 -815,6 +815,18 @@@ else
The threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.
- A gate is a policy, and a policy whose application you cannot observe is not one. `low_confidence` is written only when the gate fires, so its absence cannot tell you "you passed no `min_confidence`" apart from "a gate ran and this answer cleared it" — you could neither compute an abstention rate nor prove the gate was in effect. Every answer therefore also reports `abstention`, one of four values, present whether or not a gate was configured:
++A gate is a policy, and a policy whose application you cannot observe is not one. `low_confidence` is written only when the gate fires, so its absence cannot tell you "a gate ran and this answer cleared it" apart from "no gate ran at all" — you could neither compute an abstention rate nor prove the gate was in effect. **When you pass `min_confidence`,** every answer therefore also reports `abstention`:
+
+| `abstention` | meaning |
+|---|---|
- | `not_configured` | no `min_confidence` was passed, so there is no gate |
+| `passed` | a gate ran and this answer's `answer_confidence` cleared it |
+| `abstained` | a gate ran and this answer's `answer_confidence` fell below it |
+| `unevaluated` | a gate ran and this answer carried no usable confidence, so the gate could not decide |
+
+`abstention_threshold` echoes the threshold whenever a gate ran, so a log can be re-split by the gate that produced it instead of by whatever the caller happened to remember passing. `unevaluated` is the case a boolean cannot express — reporting it as a pass would be as wrong as reporting it as a flag, so a NaN, a missing confidence or a `bool` lands in its own state. The raw answer, probabilities and confidence are untouched either way.
+
++Left without `min_confidence`, the response is byte-for-byte what it was before any of this: no `abstention`, no `abstention_threshold`, no flag. That is deliberate — a field on every answer that only some calls populate is a schema change for callers who never asked to be gated. The presence of `abstention` is how you tell a gated run from an ungated one; a `min_confidence` of exactly `0.0` still counts as set, so it reports states, and `flag_low_confidence` treats `0.0` as a no-op because nothing can fall below it.
++
---
## Prediction Hooks
diff --cc docs/reference/helpers.md
index 56751064,7f53485e..eeda50a0
--- a/docs/reference/helpers.md
+++ b/docs/reference/helpers.md
@@@ -44,12 -44,10 +44,14 @@@
::: laya.confidence.flag_low_confidence
+::: laya.confidence.apply_confidence_gate
+
+::: laya.confidence.GATE_STATES
+
## Calibration and training
+ ::: laya.common.answer_confidence
+
::: laya.common.confidence_from_probs
::: laya.common.ece_score
diff --cc laya/agent.py
index c7d914bc,c98e6859..866fae43
--- a/laya/agent.py
+++ b/laya/agent.py
@@@ -28,11 -35,14 +35,14 @@@ from .common import
_reuse_question_tokens,
_disable_question_token_reuse,
encode_text,
+ render_criterion,
render_options,
+ resolve_lang_temperatures,
serialize_state,
temp_bucket,
+ unpermute_probs,
)
-from .confidence import check_min_confidence, flag_low_confidence
+from .confidence import apply_confidence_gate, check_min_confidence
from .hooks import (
HookRegistry, PredictContext, _as_sequence, aggregate_usage, compose_hooks, dispatch,
normalise_hooks, validate_timeout,
@@@ -1273,14 -1488,16 +1487,20 @@@ class Agent(HookRegistry)
usage: Dict[str, Any] = {}
for r in results:
for key, val in r["usage"].items():
- usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
+ prev = usage.get(key)
+ if isinstance(val, (int, float)):
+ usage[key] = (prev if isinstance(prev, (int, float)) else 0) + val
+ elif isinstance(val, dict) and isinstance(prev, dict):
+ usage[key] = {**prev, **val}
+ else:
+ usage[key] = val
usage["output_tokens"] = 0
usage["windows"] = len(results)
- return {"model": "laya-rl-agent", "answers": answers, "usage": usage}
+ result = {"model": "laya-rl-agent", "answers": answers, "usage": usage}
+ # `predict_long` accepts no `min_confidence` -- the window loop has no single confidence to
+ # gate on -- so every answer reports that it ran ungated rather than reporting nothing.
+ apply_confidence_gate([result], None)
+ return result
@torch.no_grad()
def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]], lang: Optional[str] = None,
diff --cc laya/confidence.py
index 495d2037,5a20657a..43b67ee5
--- a/laya/confidence.py
+++ b/laya/confidence.py
@@@ -7,23 -7,46 +7,54 @@@ import mat
from typing import Any, Dict, List, Optional
+ def answer_confidence_value(answer: Dict[str, Any]) -> Optional[float]:
+ """The `answer_confidence` of `answer`, or None if it did not report a usable one.
+
+ `answer_confidence` is `max(p)` -- the probability mass on the answer being reported. It is
+ the quantity temperature scaling fits, the quantity every calibration figure in this
+ repository is computed on, and the quantity the `min_confidence` gate is defined against
+ (#361). That makes it the right number for a threshold a caller compares against; it is
+ not on its own a claim that the number is right. Reading it as "about c of the answers
+ returned at c are correct" holds only after temperatures have been fitted and validated on
+ held-out data for that checkpoint and question shape -- the shipped checkpoints are
+ over-confident as shipped, and `laya-multilingual` ships with no fitted temperatures at all
+ (README, Calibration). `common.answer_confidence` states the quantity; this states when it
+ carries that property.
+
+ `confidence` is a different quantity on a different scale -- normalized entropy, which
+ `common.confidence_from_probs` calls "not calibrated" and #394 says does not transfer across
+ option counts. So this deliberately does **not** fall back to it: reporting the entropy
+ number under the `answer_confidence` name would be worse than reporting nothing, because the
+ name is what a caller filters on.
+
+ A `bool` is not a confidence and a NaN is not a decision, so both give None rather than 0.0.
+ """
+ conf = answer.get("answer_confidence")
+ if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):
+ return float(conf)
+ return None
+
+
def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:
- """The confidence the abstention gate reads for `answer`, or None if there is not a usable one.
+ """The number the abstention gate compares against `min_confidence`, or None if there is none.
- `answer_confidence` is the calibrated max(p) confidence and is what the gate must read;
- `confidence` is the fallback, and for `choice` and `score` it is normalized entropy on a
- different scale, which must not be compared against the same threshold. A bool, a NaN, an
- infinity or a missing field is not a number the gate can decide on.
+ `answer_confidence` first, falling back to the entropy `confidence` so an answer that carries
+ only the older field is still gated rather than silently passed.
+
- Shared by :func:`flag_low_confidence` and :func:`apply_confidence_gate` so the flag and the
- reported state cannot disagree about which quantity was read.
++ None means *no usable number*, never an unusable one. An earlier version returned whatever
++ `confidence` held when the validity test failed, so an answer with no `answer_confidence` and a
++ NaN `confidence` returned NaN. `flag_low_confidence` was indifferent to that -- `NaN < x` is
++ False and `None is not None` is also False, so it flagged either way -- but a caller that
++ *reads* the number could not tell "nothing to gate on" from "a gate ran on a NaN", and
++ :func:`apply_confidence_gate` reports those two differently on purpose.
"""
- conf = answer.get("answer_confidence")
- if conf is None:
- conf = answer.get("confidence")
- if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):
+ conf = answer_confidence_value(answer)
- if conf is None:
- conf = answer.get("confidence")
- if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):
- return float(conf)
- return conf
++ if conf is not None:
+ return conf
++ fallback = answer.get("confidence")
++ if isinstance(fallback, (int, float)) and not isinstance(fallback, bool) and math.isfinite(fallback):
++ return float(fallback)
+ return None
def check_min_confidence(v: Any) -> float:
@@@ -55,60 -79,3 +87,67 @@@ def flag_low_confidence(results: List[D
conf = _gate_confidence(a)
if conf is not None and conf < min_confidence:
a["low_confidence"] = True
+
+
- #: The confidence gate's state, as reported on every answer by :func:`apply_confidence_gate`.
- #:
- #: `low_confidence` is written only when the gate fires, so its absence cannot tell a caller "no
- #: ``min_confidence`` was passed" apart from "a gate ran and this answer cleared it". These four
- #: values make the state a thing you read rather than a thing you infer from a missing field.
- GATE_NOT_CONFIGURED = "not_configured" # no threshold was passed, so there is no gate to apply
- GATE_PASSED = "passed" # a gate ran and this answer's confidence cleared it
- GATE_ABSTAINED = "abstained" # a gate ran and this answer's confidence fell below it
- GATE_UNEVALUATED = "unevaluated" # a gate ran and this answer carried no usable confidence
++#: The states :func:`apply_confidence_gate` reports, and the only ones. There is deliberately no
++#: "no gate was configured" member: when `min_confidence` is not set the function writes nothing at
++#: all, so a caller tells "this run had no gate" from "this answer cleared the gate" by whether
++#: `abstention` is present. A sentinel for the unconfigured case would put that same information
++#: back into the payload for every caller, which is the thing the gate is meant to avoid.
++GATE_PASSED = "passed" # a gate ran and this answer's confidence cleared it
++GATE_ABSTAINED = "abstained" # a gate ran and this answer's confidence fell below it
++GATE_UNEVALUATED = "unevaluated" # a gate ran and this answer carried no usable confidence
+
- GATE_STATES = (GATE_NOT_CONFIGURED, GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)
++GATE_STATES = (GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)
+
+
+def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optional[float] = None) -> None:
- """Report the confidence gate's state on every answer, whether or not a gate was configured.
++ """Report the confidence gate's state, on the answers a gate was actually applied to.
+
- A gate is a policy, and a policy whose application cannot be observed is not one. This writes
- ``abstention`` -- one of :data:`GATE_STATES` -- onto every answer, plus ``abstention_threshold``
- whenever a gate actually ran. It is what lets a caller answer three questions it otherwise
++ A gate is a policy, and a policy whose application cannot be observed is not one. With
++ `min_confidence` set, this writes ``abstention`` -- one of :data:`GATE_STATES` -- onto every
++ answer, plus ``abstention_threshold``, so a caller can answer three questions it otherwise
+ cannot:
+
- * what fraction of decisions abstained, without guessing which of "no gate" and "gate
- cleared it" produced a missing ``low_confidence``;
- * whether the gate was in effect at all on this run;
- * what threshold produced these results, which ``flag_low_confidence`` consumes and drops --
- without it a batch that used different thresholds per request class cannot be re-split.
++ * what fraction of decisions abstained, rather than inferring it from whether
++ ``low_confidence`` happened to be set;
++ * how many answers the gate could not decide on, which a boolean cannot express at all;
++ * what threshold produced these results -- ``flag_low_confidence`` consumes the threshold and
++ drops it, so without this a batch run with per-class thresholds cannot be re-split.
+
+ ``GATE_UNEVALUATED`` is the case a boolean cannot express: the gate ran and the answer carried
- no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as
- reporting it as a flag.
-
- The flag itself stays :func:`flag_low_confidence`'s -- this delegates to it rather than
- re-implementing the rule, so the boolean and the state cannot drift apart. Call this
- unconditionally, once per call, in place of a `if min_confidence is not None:` guard: that
- guard is what leaves a path reporting nothing at all.
++ no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie
++ as reporting it as a flag.
++
++ **With `min_confidence` unset, this writes nothing.** No ``abstention``, no
++ ``abstention_threshold``, no flag. That is the whole contract: an ungated call returns exactly
++ the payload it returned before, and the presence of the field -- not a fourth value read out
++ of it -- is what tells a caller the gate ran. Call it unconditionally, once per call, in place
++ of an `if min_confidence is not None:` guard: that guard is what leaves a path reporting
++ nothing at all, which is the state this function exists to distinguish.
++
++ The flag itself stays :func:`flag_low_confidence`'s -- this delegates rather than
++ re-implementing the rule, so the boolean and the reported state cannot drift apart.
++
++ A `min_confidence` of exactly ``0.0`` *was* set, so states are reported, and
++ :func:`flag_low_confidence` treats ``0.0`` as a no-op because nothing can fall below it. Every
++ answer carrying a usable confidence therefore reads ``passed``, and the threshold echo is what
++ distinguishes that from a real pass at a real threshold.
+ """
- if min_confidence is not None:
- flag_low_confidence(results, min_confidence)
++ if min_confidence is None:
++ return
++ flag_low_confidence(results, min_confidence)
+ for res in results:
+ answers = res.get("answers") if isinstance(res, dict) else None
+ if not isinstance(answers, dict):
+ continue
+ for a in answers.values():
+ if not isinstance(a, dict):
+ continue
- if min_confidence is None:
- a["abstention"] = GATE_NOT_CONFIGURED
- continue
+ if a.get("low_confidence"):
+ a["abstention"] = GATE_ABSTAINED
+ elif _gate_confidence(a) is None:
+ a["abstention"] = GATE_UNEVALUATED
+ else:
+ a["abstention"] = GATE_PASSED
+ a["abstention_threshold"] = float(min_confidence)
diff --cc laya/onnx_agent.py
index 0095bb12,8a67f065..c5675121
--- a/laya/onnx_agent.py
+++ b/laya/onnx_agent.py
@@@ -511,13 -545,16 +544,19 @@@ class ONNXAgent(HookRegistry)
usage: Dict[str, Any] = {}
for r in results:
for key, val in r["usage"].items():
- usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
+ prev = usage.get(key)
+ if isinstance(val, (int, float)):
+ usage[key] = (prev if isinstance(prev, (int, float)) else 0) + val
+ elif isinstance(val, dict) and isinstance(prev, dict):
+ usage[key] = {**prev, **val}
+ else:
+ usage[key] = val
usage["output_tokens"] = 0
usage["windows"] = len(results)
- return {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}
+ result = {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}
+ # As in `Agent.predict_long`: no `min_confidence` here, so say so on every answer.
+ apply_confidence_gate([result], None)
+ return result
def _infer(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
max_len: Optional[int] = None, head_max_len: Optional[int] = None,
diff --cc laya/structured.py
index fe0bd34e,9e75dcec..e4cd512a
--- a/laya/structured.py
+++ b/laya/structured.py
@@@ -20,7 -20,7 +20,7 @@@ from collections.abc import Sequence a
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
- from .confidence import apply_confidence_gate, check_min_confidence
-from .confidence import answer_confidence_value, check_min_confidence, flag_low_confidence
++from .confidence import answer_confidence_value, apply_confidence_gate, check_min_confidence
MAX_PROPERTIES = 32
MAX_OPTIONS = 32
diff --cc tests/test_confidence.py
index ae86d594,02dc784e..d4c210bf
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@@ -15,6 -15,6 +15,7 @@@ untouched, so nothing a caller gates o
No weights are loaded: the confidence helpers are pure.
"""
++import copy
import math
import os
import sys
@@@ -145,170 -145,6 +146,183 @@@ check_true("flag/q_exact unflagged", "l
check("flag/answer_confidence intact", ans["q_low"]["answer_confidence"], 0.45)
check("flag/raw choice intact", ans["q_high"]["choice"], "a")
+# --------------------------------------------------------------- the gate's state is reported
+# `low_confidence` is written only when the gate fires, so its absence cannot tell a caller
- # "no min_confidence was passed" apart from "a gate ran and this answer cleared it". An operator
- # with 10,000 logged decisions cannot compute an abstention rate, cannot prove the gate was in
- # effect, and cannot re-split a batch that used different thresholds per request class -- the
- # threshold is consumed and dropped. `docs/staged-adoption.md` asks a shadow record to carry
- # "errors and any fallback or review decision"; there was no field to put that in.
++# "a gate ran and this answer cleared it" from "no gate ran at all". An operator with 10,000
++# logged decisions cannot compute an abstention rate, cannot tell whether a run was gated, and
++# cannot re-split a batch that used different thresholds per request class -- the threshold is
++# consumed and dropped. `docs/staged-adoption.md` asks a shadow record to carry "errors and any
++# fallback or review decision"; there was no field to put that in.
+#
- # The contract: the gate's state is a value on every answer, so it is never inferred from a
- # missing field.
++# The contract: where a gate ran, its state is an explicit value, so it is never inferred from a
++# missing field. Where no gate was configured, nothing is written at all -- an ungated call returns
++# the payload it always returned, rather than growing a field every existing caller would then have
++# to read. The two cases are told apart by the presence of `abstention`, which is why the vocabulary
++# has no "not configured" member to read out of the field.
+from laya.confidence import ( # noqa: E402
+ GATE_ABSTAINED,
- GATE_NOT_CONFIGURED,
+ GATE_PASSED,
+ GATE_STATES,
+ GATE_UNEVALUATED,
+ apply_confidence_gate,
+)
+
+check("gate/the vocabulary is closed",
- list(GATE_STATES), ["not_configured", "passed", "abstained", "unevaluated"])
++ list(GATE_STATES), ["passed", "abstained", "unevaluated"])
+
+fresh = lambda: [{ # noqa: E731
+ "answers": {
+ "q_high": {"type": "choice", "choice": "a", "answer_confidence": 0.92},
+ "q_low": {"type": "choice", "choice": "b", "answer_confidence": 0.45},
+ }
+}]
+
- # No gate: every answer says so, and nothing is flagged.
++# No gate: the payload is untouched. Not a sentinel, not a flag -- byte-for-byte what the caller
++# got before this helper existed, so no existing caller has to learn a field to keep working.
+off = fresh()
++before = copy.deepcopy(off)
+apply_confidence_gate(off, None)
++check("gate/an ungated call writes nothing at all", off, before)
+for qid, a in off[0]["answers"].items():
- check("gate/not_configured %s" % qid, a["abstention"], GATE_NOT_CONFIGURED)
- check_true("gate/not_configured carries no threshold %s" % qid, "abstention_threshold" not in a)
- check_true("gate/not_configured is not a flag %s" % qid, "low_confidence" not in a)
++ check_true("gate/ungated %s reports no state" % qid, "abstention" not in a)
++ check_true("gate/ungated %s carries no threshold" % qid, "abstention_threshold" not in a)
++ check_true("gate/ungated %s is not a flag" % qid, "low_confidence" not in a)
+
+# A gate that ran: the answer that cleared it says `passed`, which is the state the boolean
+# could not express. This is the whole point -- before, this answer was byte-identical to the
+# ungated one above.
+on = fresh()
+apply_confidence_gate(on, 0.80)
+check("gate/cleared answer says passed", on[0]["answers"]["q_high"]["abstention"], GATE_PASSED)
+check("gate/cleared answer is still unflagged", "low_confidence" in on[0]["answers"]["q_high"], False)
+check("gate/cleared answer echoes the threshold", on[0]["answers"]["q_high"]["abstention_threshold"], 0.80)
+check("gate/below-threshold answer says abstained", on[0]["answers"]["q_low"]["abstention"], GATE_ABSTAINED)
+check("gate/below-threshold answer is flagged", on[0]["answers"]["q_low"]["low_confidence"], True)
+check("gate/below-threshold echoes the threshold", on[0]["answers"]["q_low"]["abstention_threshold"], 0.80)
+check_true("gate/the raw answer is untouched",
+ on[0]["answers"]["q_high"]["choice"] == "a" and on[0]["answers"]["q_low"]["choice"] == "b")
+
+# The two states a caller used to be unable to tell apart are now told apart by a value, and the
+# distinction survives a round trip through JSON, which is how a shadow record reaches a log.
+import json as _json # noqa: E402
+
+round_tripped = _json.loads(_json.dumps(on[0]["answers"]))
+check("gate/survives a JSON round trip", round_tripped["q_high"]["abstention"], GATE_PASSED)
+
+# One rule, one implementation: the flag stays `flag_low_confidence`'s, not a second copy of it.
+# `apply_confidence_gate(results, 0.0)` is a gate that ran and nothing can fail, which is why
+# `flag_low_confidence`'s 0.0 no-op above is still correct and still asserted.
+zero = fresh()
+apply_confidence_gate(zero, 0.0)
+check("gate/0.0 reports passed", zero[0]["answers"]["q_low"]["abstention"], GATE_PASSED)
+check_true("gate/0.0 still flags nothing", "low_confidence" not in zero[0]["answers"]["q_low"])
+
+# A non-finite or missing confidence is skipped by `flag_low_confidence`, so it is not an
+# abstention. It is also not a pass: the gate ran and could not decide, which is the state Argo
+# Rollouts ships as `Inconclusive` and the one collapsing it into success would be the same lie
+# `low_confidence`'s absence already tells.
+odd = [{"answers": {
+ "q_nan": {"type": "choice", "answer_confidence": float("nan")},
+ "q_none": {"type": "choice"},
+ "q_inf": {"type": "choice", "answer_confidence": float("inf")},
+ "q_bool": {"type": "choice", "answer_confidence": True},
++ # The fallback path, which is where the number used to leak. With no usable
++ # `answer_confidence` the gate reads the entropy `confidence`, and an entropy NaN has to come
++ # back as "nothing to gate on" -- not as the NaN itself, which is not None and would
++ # therefore read as a pass. `flag_low_confidence` never noticed (NaN < x is False, and
++ # None is not None is also False), so nothing else pinned this down.
++ "q_nan_entropy": {"type": "choice", "confidence": float("nan")},
++ "q_bool_entropy": {"type": "choice", "confidence": True},
+ "q_fine": {"type": "choice", "answer_confidence": 0.99},
+}}]
+apply_confidence_gate(odd, 0.80)
+for qid, a in odd[0]["answers"].items():
+ if qid == "q_fine":
+ check("gate/a usable confidence is evaluated %s" % qid, a["abstention"], GATE_PASSED)
+ else:
+ check("gate/unusable confidence is unevaluated %s" % qid, a["abstention"], GATE_UNEVALUATED)
+ check_true("gate/unusable confidence is not flagged %s" % qid, "low_confidence" not in a)
+
+# `answer_confidence` is the quantity the gate reads, so a `noul`-style entropy `confidence`
+# below threshold does not abstain while `answer_confidence` above it does. The state follows
+# the same field the flag follows.
+pref = [{"answers": {
+ "q_a": {"type": "noul", "answer_confidence": 0.91, "confidence": 0.10},
+ "q_b": {"type": "choice", "answer_confidence": 0.10, "confidence": 0.99},
+}}]
+apply_confidence_gate(pref, 0.80)
+check("gate/gate reads answer_confidence (a)", pref[0]["answers"]["q_a"]["abstention"], GATE_PASSED)
+check("gate/gate reads answer_confidence (b)", pref[0]["answers"]["q_b"]["abstention"], GATE_ABSTAINED)
+
+# The guards the six call sites rely on: a result that is not a dict, answers that are not a
+# dict, and answers that are not dicts. Every call site passes a list it has already partly
+apply_confidence_gate([None, {"answers": None}, {"answers": [1, 2]}], 0.5)
+apply_confidence_gate([], None)
+PASS.append("gate/non-dict results are skipped without raising")
+
+# --------------------------------------------------------------- the operator's three questions
+# End to end through a real call site, weight-free. `decide` forwards to any runner with a
+# `predict`, so this exercises `laya/structured.py`'s own gate call site rather than the helper
+# in isolation -- the six call sites are where the state used to be skipped entirely.
+from laya.structured import decide # noqa: E402
+
+_GATE_Q = {"type": "choice", "instructions": "Pick one.",
+ "criteria": {"a": "alpha", "b": "beta"}}
+CONFIDENCES = {"high": 0.95, "low": 0.40}
+QUESTIONS = {qid: dict(_GATE_Q) for qid in CONFIDENCES}
+
+
+class _StubRunner:
+ def __init__(self, conf):
+ self.conf = conf
+
+ def predict(self, state, questions, **kwargs):
+ return {"model": "stub", "answers": {
+ qid: {"type": "choice", "choice": "a", "answer_confidence": self.conf[qid]}
+ for qid in questions}}
+
+
+stub = _StubRunner(CONFIDENCES)
+
+# 1. Was the gate in effect at all? Before, both answers were byte-identical to the gated pair
- # below, so the answer was "no way to tell".
++# below, so the answer was "no way to tell". Now the ungated run simply has no `abstention`
++# key, and the gated one does -- which is how a caller tells them apart.
+ungated = decide(stub, "state", questions=QUESTIONS)
+for qid in CONFIDENCES:
- check("workflow/ungated %s says so" % qid, ungated[qid]["abstention"], GATE_NOT_CONFIGURED)
++ check_true("workflow/ungated %s reports no state" % qid, "abstention" not in ungated[qid])
+
+gated = decide(stub, "state", questions=QUESTIONS, min_confidence=0.80)
+check("workflow/the cleared answer reports passed", gated["high"]["abstention"], GATE_PASSED)
+check("workflow/the low answer reports abstained", gated["low"]["abstention"], GATE_ABSTAINED)
+
+# 2. What fraction abstained, over whatever decisions the operator kept? Computable from the
+# artifact, with no re-run and no out-of-band record of which threshold was used.
+log = [gated, decide(stub, "other", questions=QUESTIONS, min_confidence=0.80)]
+answers = [a for d in log for a in d.values()]
+abstained = sum(a["abstention"] == GATE_ABSTAINED for a in answers)
+check("workflow/abstention rate over the log", abstained, 2)
+check("workflow/…out of", len(answers), 4)
+check_true("workflow/…and no answer is left unlabelled", all("abstention" in a for a in answers))
+
+# 3. What threshold produced these? `flag_low_confidence` consumes it and drops it, so before
+# this a batch that gated different request classes differently could not be re-split.
+check("workflow/the threshold is recoverable", gated["low"]["abstention_threshold"], 0.80)
+check_true("workflow/…and a per-class mix is re-splittable",
+ {a["abstention_threshold"] for a in gated.values()} == {0.80})
+
+# The schema projection is unchanged -- an abstained field is still `null` -- so the flat
+# `values` mapping still cannot say why. `return_details=True` is where the state is readable,
+# and it was unreadable before.
+SCHEMA = {"type": "object",
+ "properties": {"high": {"type": "string", "enum": ["a", "b"]},
+ "low": {"type": "string", "enum": ["a", "b"]}}}
+projected = decide(stub, "state", schema=SCHEMA, min_confidence=0.80)
+check("workflow/the schema projection is unchanged", projected["low"], None)
+check("workflow/…and still reports the cleared field", projected["high"], "a")
+detailed = decide(stub, "state", schema=SCHEMA, min_confidence=0.80, return_details=True)
+check("workflow/return_details carries the state",
+ detailed.answers["low"]["abstention"], GATE_ABSTAINED)
+
# --------------------------------------------------------------- exported
import laya # noqa: E402
@@@ -321,21 -157,6 +335,20 @@@ check_true("export/check_min_confidenc
check_true("export/flag_low_confidence is importable from laya", hasattr(laya, "flag_low_confidence"))
check_true("export/flag_low_confidence is in __all__", "flag_low_confidence" in laya.__all__)
check_true("export/flag_low_confidence is in dir()", "flag_low_confidence" in dir(laya))
+check_true("export/apply_confidence_gate is importable from laya", hasattr(laya, "apply_confidence_gate"))
+check_true("export/apply_confidence_gate is in __all__", "apply_confidence_gate" in laya.__all__)
+check_true("export/apply_confidence_gate is in dir()", "apply_confidence_gate" in dir(laya))
+check_true("export/GATE_STATES is importable from laya", hasattr(laya, "GATE_STATES"))
+check_true("export/GATE_STATES is in __all__", "GATE_STATES" in laya.__all__)
+check_true("export/GATE_STATES is in dir()", "GATE_STATES" in dir(laya))
- # The four values are reached through GATE_STATES rather than exported one by one: iterating the
++# The three values are reached through GATE_STATES rather than exported one by one: iterating the
+# vocabulary is the whole use, and every `laya.__all__` name has to earn a documented entry
+# (`tests/test_packaging.py` enforces that).
+import laya.confidence as _confidence # noqa: E402
+
+check("export/GATE_STATES is the vocabulary a caller iterates", list(GATE_STATES),
- [_confidence.GATE_NOT_CONFIGURED, _confidence.GATE_PASSED,
- _confidence.GATE_ABSTAINED, _confidence.GATE_UNEVALUATED])
++ [_confidence.GATE_PASSED, _confidence.GATE_ABSTAINED, _confidence.GATE_UNEVALUATED])
+check("export/laya re-exports the same tuple", laya.GATE_STATES, _confidence.GATE_STATES)
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
diff --cc tests/test_predict_long.py
index d9924639,19397de2..5666c4bb
--- a/tests/test_predict_long.py
+++ b/tests/test_predict_long.py
@@@ -206,11 -257,7 +257,11 @@@ stub = _LongStub(
out = _router(stub).predict_long({"body": "y" * 300}, Q, model="english",
window=64, stride=32, batch_size=8, lang="de")
check("router/routing key attached", out["routing"]["model"], "english")
-check("router/answers come from the scan", out["answers"], {"scanned": {"noul": 0.5}})
+# `predict_long` takes no `min_confidence` -- a window scan has no single confidence to gate on --
- # so every scanned answer says it ran ungated instead of saying nothing. That is the point: an
- # operator reading a long-document result can now see there was no gate on it.
++# so the gate writes nothing and the scan's answers come back exactly as produced. A long-document
++# result is not a gated decision, and now it does not claim to be one.
+check("router/answers come from the scan", out["answers"],
- {"scanned": {"noul": 0.5, "abstention": "not_configured"}})
++ {"scanned": {"noul": 0.5}})
check("router/window forwarded", stub.calls[0]["window"], 64)
check("router/stride forwarded", stub.calls[0]["stride"], 32)
check("router/batch_size forwarded", stub.calls[0]["batch_size"], 8)
@@@ -259,8 -306,7 +310,8 @@@ class _Recorder
recorder_stub = _LongStub()
_router(recorder_stub).predict_long({"body": "y" * 300}, Q, model="english",
on_predict_end=_Recorder().on_predict_end)
-check("router/per-call end hook sees the scan", ended["answers"], {"scanned": {"noul": 0.5}})
+check("router/per-call end hook sees the scan", ended["answers"],
- {"scanned": {"noul": 0.5, "abstention": "not_configured"}})
++ {"scanned": {"noul": 0.5}})
# 8. an agent with no predict_long is a named caller error, not a bare AttributeError
class _NoScan:
diff --cc tests/test_structured.py
index 5a80d331,e570a77e..ccda8890
--- a/tests/test_structured.py
+++ b/tests/test_structured.py
@@@ -163,10 -172,8 +172,10 @@@ check("decide/details routing", details
runner = FakeRunner({"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})
out = decide(runner, "s", questions={"a": {"type": "noul", "instructions": "?"}})
- # The pass-through is the raw answer, and it now carries the gate's state: no `min_confidence`
- # was passed, so every answer reports `not_configured` rather than reporting nothing.
++# The pass-through is the raw answer, and no `min_confidence` was passed, so it comes back exactly
++# as the runner produced it -- the gate adds nothing to a call it was not asked to gate.
check("decide/questions pass-through returns answers", out,
- {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}})
+ {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})
check_raises("decide/requires schema or questions", ValueError, lambda: decide(runner, "s"))
check_raises("decide/rejects both", ValueError, lambda: decide(runner, "s", schema=SCHEMA, questions={}))
@@@ -282,8 -348,7 +350,8 @@@ check("batch/details routing", details[
# questions= is the raw-answers pass-through, exactly like decide()
raw = decide_batch(FakeBatchRunner({"a": {"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}}),
["a"], questions={"x": {"type": "noul", "instructions": "?"}})
-check("batch/questions pass-through", raw, [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}])
+check("batch/questions pass-through", raw,
- [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}}])
++ [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}])
class FakeRouterLike: