Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(confidence): report the abstention gate's state on every answer
Bruce-Yii02aaed4ca19b
diff --git a/README.md b/README.mdindex 4aadb0c4..97a7307f 100644--- a/README.md+++ b/README.md@@ -97,7 +97,7 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o## What's new in 0.3.21* **ONNX catches up with PyTorch.** `ONNXAgent` gains `predict_batch` (with `sort_by_length`), `predict_long` and `decide_batch`, `scripts/export_onnx.py --quantize` writes a per-channel INT8 copy for CPU, and `laya-evals run --onnx` scores an export with the same gates as the torch path.-* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them.+* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them. Every answer also reports `abstention` — `not_configured`, `passed`, `abstained` or `unevaluated` — so a caller can tell an ungated run from a gated one that cleared.* **Batch everywhere.** `decide_batch`, `Router.predict_long`, `laya --batch FILE`, the MCP `laya_predict_batch` / `laya_route_batch` / `laya_decide` tools, and LangChain `batch()` / `abatch()` all run on shared forward passes. New `LayaDecision` (LangChain), LlamaIndex selectors (`laya[llamaindex]`) and CrewAI routing (`laya[crewai]`).* **Per-request token budget.** `max_len` / `head_max_len` now reach every surface: `laya-serve` (capped by `LAYA_MAX_TOKEN_BUDGET`), `Router.predict_batch` requests, the CLI (`--questions`, `--max-len`, `--head-max-len`), MCP tools and LangChain nodes.* **Operations.** `LAYA_MAX_LOADED`, `LAYA_REVISION` and per-checkpoint SHA-256 maps; `/health` reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries `Retry-After`; `compile=True` no longer recompiles for every request shape.@@ -719,6 +719,17 @@ else:The threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.+A gate is a policy, and a policy whose application you cannot observe is not one. `low_confidence` is written only when the gate fires, so its absence cannot tell you "you passed no `min_confidence`" apart from "a gate ran and this answer cleared it" — you could neither compute an abstention rate nor prove the gate was in effect. Every answer therefore also reports `abstention`, one of four values, present whether or not a gate was configured:++| `abstention` | meaning |+|---|---|+| `not_configured` | no `min_confidence` was passed, so there is no gate |+| `passed` | a gate ran and this answer's `answer_confidence` cleared it |+| `abstained` | a gate ran and this answer's `answer_confidence` fell below it |+| `unevaluated` | a gate ran and this answer carried no usable confidence, so the gate could not decide |++`abstention_threshold` echoes the threshold whenever a gate ran, so a log can be re-split by the gate that produced it instead of by whatever the caller happened to remember passing. `unevaluated` is the case a boolean cannot express — reporting it as a pass would be as wrong as reporting it as a flag, so a NaN, a missing confidence or a `bool` lands in its own state. The raw answer, probabilities and confidence are untouched either way.+---## Prediction Hooksdiff --git a/docs/reference/helpers.md b/docs/reference/helpers.mdindex 972c520b..56751064 100644--- a/docs/reference/helpers.md+++ b/docs/reference/helpers.md@@ -44,6 +44,10 @@::: laya.confidence.flag_low_confidence+::: laya.confidence.apply_confidence_gate++::: laya.confidence.GATE_STATES+## Calibration and training::: laya.common.confidence_from_probsdiff --git a/laya/__init__.py b/laya/__init__.pyindex fda54c7c..a5f78587 100644--- a/laya/__init__.py+++ b/laya/__init__.py@@ -30,6 +30,12 @@ _LAZY_ATTRS = {"confidence_from_probs": (".common", "confidence_from_probs"),"check_min_confidence": (".confidence", "check_min_confidence"),"flag_low_confidence": (".confidence", "flag_low_confidence"),+ "apply_confidence_gate": (".confidence", "apply_confidence_gate"),+ "GATE_NOT_CONFIGURED": (".confidence", "GATE_NOT_CONFIGURED"),+ "GATE_PASSED": (".confidence", "GATE_PASSED"),+ "GATE_ABSTAINED": (".confidence", "GATE_ABSTAINED"),+ "GATE_UNEVALUATED": (".confidence", "GATE_UNEVALUATED"),+ "GATE_STATES": (".confidence", "GATE_STATES"),"render_options": (".common", "render_options"),"QTYPES": (".common", "QTYPES"),"QTYPE_NAMES": (".common", "QTYPE_NAMES"),@@ -90,6 +96,12 @@ __all__ = ["confidence_from_probs","check_min_confidence","flag_low_confidence",+ "apply_confidence_gate",+ "GATE_NOT_CONFIGURED",+ "GATE_PASSED",+ "GATE_ABSTAINED",+ "GATE_UNEVALUATED",+ "GATE_STATES","render_options","QTYPES","QTYPE_NAMES",diff --git a/laya/agent.py b/laya/agent.pyindex 5ca3861c..c7d914bc 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -32,7 +32,7 @@ from .common import (serialize_state,temp_bucket,)-from .confidence import check_min_confidence, flag_low_confidence+from .confidence import apply_confidence_gate, check_min_confidencefrom .hooks import (HookRegistry, PredictContext, _as_sequence, aggregate_usage, compose_hooks, dispatch,normalise_hooks, validate_timeout,@@ -1076,8 +1076,7 @@ class Agent(HookRegistry):ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0if ctx.results is not None:ctx.usage = aggregate_usage(ctx.results)- if mc is not None:- flag_low_confidence(ctx.results, mc)+ apply_confidence_gate(ctx.results, mc)try:dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)except BaseException as hook_exc:@@ -1277,7 +1276,11 @@ class Agent(HookRegistry):usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else valusage["output_tokens"] = 0usage["windows"] = len(results)- return {"model": "laya-rl-agent", "answers": answers, "usage": usage}+ result = {"model": "laya-rl-agent", "answers": answers, "usage": usage}+ # `predict_long` accepts no `min_confidence` -- the window loop has no single confidence to+ # gate on -- so every answer reports that it ran ungated rather than reporting nothing.+ apply_confidence_gate([result], None)+ return result@torch.no_grad()def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]], lang: Optional[str] = None,diff --git a/laya/confidence.py b/laya/confidence.pyindex 253329bf..495d2037 100644--- a/laya/confidence.py+++ b/laya/confidence.py@@ -4,7 +4,26 @@ Pure Python: safe to import without PyTorch so that Router and structureddecisions stay lightweight and free of torch import overhead."""import math-from typing import Any, Dict, List+from typing import Any, Dict, List, Optional+++def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:+ """The confidence the abstention gate reads for `answer`, or None if there is not a usable one.++ `answer_confidence` is the calibrated max(p) confidence and is what the gate must read;+ `confidence` is the fallback, and for `choice` and `score` it is normalized entropy on a+ different scale, which must not be compared against the same threshold. A bool, a NaN, an+ infinity or a missing field is not a number the gate can decide on.++ Shared by :func:`flag_low_confidence` and :func:`apply_confidence_gate` so the flag and the+ reported state cannot disagree about which quantity was read.+ """+ conf = answer.get("answer_confidence")+ if conf is None:+ conf = answer.get("confidence")+ if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):+ return conf+ return Nonedef check_min_confidence(v: Any) -> float:@@ -33,8 +52,63 @@ def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) ->for a in answers.values():if not isinstance(a, dict):continue- conf = a.get("answer_confidence")- if conf is None:- conf = a.get("confidence")- if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf) and conf < min_confidence:+ conf = _gate_confidence(a)+ if conf is not None and conf < min_confidence:a["low_confidence"] = True+++#: The confidence gate's state, as reported on every answer by :func:`apply_confidence_gate`.+#:+#: `low_confidence` is written only when the gate fires, so its absence cannot tell a caller "no+#: ``min_confidence`` was passed" apart from "a gate ran and this answer cleared it". These four+#: values make the state a thing you read rather than a thing you infer from a missing field.+GATE_NOT_CONFIGURED = "not_configured" # no threshold was passed, so there is no gate to apply+GATE_PASSED = "passed" # a gate ran and this answer's confidence cleared it+GATE_ABSTAINED = "abstained" # a gate ran and this answer's confidence fell below it+GATE_UNEVALUATED = "unevaluated" # a gate ran and this answer carried no usable confidence++GATE_STATES = (GATE_NOT_CONFIGURED, GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)+++def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optional[float] = None) -> None:+ """Report the confidence gate's state on every answer, whether or not a gate was configured.++ A gate is a policy, and a policy whose application cannot be observed is not one. This writes+ ``abstention`` -- one of :data:`GATE_STATES` -- onto every answer, plus ``abstention_threshold``+ whenever a gate actually ran. It is what lets a caller answer three questions it otherwise+ cannot:++ * what fraction of decisions abstained, without guessing which of "no gate" and "gate+ cleared it" produced a missing ``low_confidence``;+ * whether the gate was in effect at all on this run;+ * what threshold produced these results, which ``flag_low_confidence`` consumes and drops --+ without it a batch that used different thresholds per request class cannot be re-split.++ ``GATE_UNEVALUATED`` is the case a boolean cannot express: the gate ran and the answer carried+ no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as+ reporting it as a flag.++ The flag itself stays :func:`flag_low_confidence`'s -- this delegates to it rather than+ re-implementing the rule, so the boolean and the state cannot drift apart. Call this+ unconditionally, once per call, in place of a `if min_confidence is not None:` guard: that+ guard is what leaves a path reporting nothing at all.+ """+ if min_confidence is not None:+ flag_low_confidence(results, min_confidence)+ for res in results:+ answers = res.get("answers") if isinstance(res, dict) else None+ if not isinstance(answers, dict):+ continue+ for a in answers.values():+ if not isinstance(a, dict):+ continue+ if min_confidence is None:+ a["abstention"] = GATE_NOT_CONFIGURED+ continue+ if a.get("low_confidence"):+ a["abstention"] = GATE_ABSTAINED+ elif _gate_confidence(a) is None:+ a["abstention"] = GATE_UNEVALUATED+ else:+ a["abstention"] = GATE_PASSED+ a["abstention_threshold"] = float(min_confidence)diff --git a/laya/onnx_agent.py b/laya/onnx_agent.pyindex c31778a9..0095bb12 100644--- a/laya/onnx_agent.py+++ b/laya/onnx_agent.py@@ -27,7 +27,7 @@ from laya.common import (TEMP_MAX,clamp_temperature,)-from laya.confidence import check_min_confidence, flag_low_confidence+from laya.confidence import apply_confidence_gate, check_min_confidenceclass ONNXAgent(HookRegistry):@@ -356,8 +356,7 @@ class ONNXAgent(HookRegistry):ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0if ctx.results is not None:ctx.usage = aggregate_usage(ctx.results)- if mc is not None:- flag_low_confidence(ctx.results, mc)+ apply_confidence_gate(ctx.results, mc)try:dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)except BaseException as hook_exc:@@ -515,7 +514,10 @@ class ONNXAgent(HookRegistry):usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else valusage["output_tokens"] = 0usage["windows"] = len(results)- return {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}+ result = {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}+ # As in `Agent.predict_long`: no `min_confidence` here, so say so on every answer.+ apply_confidence_gate([result], None)+ return resultdef _infer(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],max_len: Optional[int] = None, head_max_len: Optional[int] = None,diff --git a/laya/router.py b/laya/router.pyindex 9a2f3cc8..97c58526 100644--- a/laya/router.py+++ b/laya/router.py@@ -36,7 +36,7 @@ import timefrom collections.abc import Sequence as SequenceABCfrom typing import Any, Dict, List, Optional, Sequence, Union-from .confidence import check_min_confidence, flag_low_confidence+from .confidence import apply_confidence_gate, check_min_confidencefrom .hooks import (HookRegistry, PredictContext, aggregate_usage, compose_hooks, dispatch, normalise_hooks,validate_timeout,@@ -803,8 +803,7 @@ class Router(HookRegistry):ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0if ctx.results is not None:ctx.usage = aggregate_usage(ctx.results)- if mc is not None:- flag_low_confidence(ctx.results, mc)+ apply_confidence_gate(ctx.results, mc)try:dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)except BaseException as hook_exc:@@ -1153,14 +1152,14 @@ class Router(HookRegistry):# had already run. Each is still ended, so a hook that opens something in start# (a span, an in-flight count) always sees the matching end.for ctx in started:- if mc is not None and ctx.results:- flag_low_confidence(ctx.results, mc)+ if ctx.results:+ apply_confidence_gate(ctx.results, mc)self._end_contexts(active, started, raise_errors, timeout, error=exc)raisefor ctx in started:- if mc is not None and ctx.results:- flag_low_confidence(ctx.results, mc)+ if ctx.results:+ apply_confidence_gate(ctx.results, mc)self._end_contexts(active, started, raise_errors, timeout)for i, ctx in zip(indices, started):results[i] = ctx.results[0]diff --git a/laya/structured.py b/laya/structured.pyindex 30e5d3ba..fe0bd34e 100644--- a/laya/structured.py+++ b/laya/structured.py@@ -20,7 +20,7 @@ from collections.abc import Sequence as SequenceABCfrom dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Optional, Sequence, Tuple-from .confidence import check_min_confidence, flag_low_confidence+from .confidence import apply_confidence_gate, check_min_confidenceMAX_PROPERTIES = 32MAX_OPTIONS = 32@@ -280,8 +280,8 @@ def decide(runner, state: Any, schema: Any = None, *, questions: Optional[Dict[selse:raise- if mc is not None and isinstance(result, dict):- flag_low_confidence([result], mc)+ if isinstance(result, dict):+ apply_confidence_gate([result], mc)answers = result.get("answers", {}) or {}values = _project(answers, fields) if fields is not None else dict(answers)@@ -342,10 +342,9 @@ def decide_batch(runner, states: Sequence[Any], schema: Any = None, *,# Agent convention: a list of states evaluated against one question set.results = predict_batch(list(states), questions, **predict_kwargs)- if mc is not None:- # Flagged here rather than passed down, so a runner whose predict_batch predates the- # keyword still gets the same projection.- flag_low_confidence([r for r in results if isinstance(r, dict)], mc)+ # Applied here rather than passed down, so a runner whose predict_batch predates the+ # keyword still gets the same projection and the same reported state.+ apply_confidence_gate([r for r in results if isinstance(r, dict)], mc)def _one(r: Dict[str, Any]) -> Any:answers = r.get("answers", {}) or {}diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex 02dc784e..f0d99911 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -145,6 +145,170 @@ check_true("flag/q_exact unflagged", "low_confidence" not in ans["q_exact"])check("flag/answer_confidence intact", ans["q_low"]["answer_confidence"], 0.45)check("flag/raw choice intact", ans["q_high"]["choice"], "a")+# --------------------------------------------------------------- the gate's state is reported+# `low_confidence` is written only when the gate fires, so its absence cannot tell a caller+# "no min_confidence was passed" apart from "a gate ran and this answer cleared it". An operator+# with 10,000 logged decisions cannot compute an abstention rate, cannot prove the gate was in+# effect, and cannot re-split a batch that used different thresholds per request class -- the+# threshold is consumed and dropped. `docs/staged-adoption.md` asks a shadow record to carry+# "errors and any fallback or review decision"; there was no field to put that in.+#+# The contract: the gate's state is a value on every answer, so it is never inferred from a+# missing field.+from laya.confidence import ( # noqa: E402+ GATE_ABSTAINED,+ GATE_NOT_CONFIGURED,+ GATE_PASSED,+ GATE_STATES,+ GATE_UNEVALUATED,+ apply_confidence_gate,+)++check("gate/the vocabulary is closed",+ list(GATE_STATES), ["not_configured", "passed", "abstained", "unevaluated"])++fresh = lambda: [{ # noqa: E731+ "answers": {+ "q_high": {"type": "choice", "choice": "a", "answer_confidence": 0.92},+ "q_low": {"type": "choice", "choice": "b", "answer_confidence": 0.45},+ }+}]++# No gate: every answer says so, and nothing is flagged.+off = fresh()+apply_confidence_gate(off, None)+for qid, a in off[0]["answers"].items():+ check("gate/not_configured %s" % qid, a["abstention"], GATE_NOT_CONFIGURED)+ check_true("gate/not_configured carries no threshold %s" % qid, "abstention_threshold" not in a)+ check_true("gate/not_configured is not a flag %s" % qid, "low_confidence" not in a)++# A gate that ran: the answer that cleared it says `passed`, which is the state the boolean+# could not express. This is the whole point -- before, this answer was byte-identical to the+# ungated one above.+on = fresh()+apply_confidence_gate(on, 0.80)+check("gate/cleared answer says passed", on[0]["answers"]["q_high"]["abstention"], GATE_PASSED)+check("gate/cleared answer is still unflagged", "low_confidence" in on[0]["answers"]["q_high"], False)+check("gate/cleared answer echoes the threshold", on[0]["answers"]["q_high"]["abstention_threshold"], 0.80)+check("gate/below-threshold answer says abstained", on[0]["answers"]["q_low"]["abstention"], GATE_ABSTAINED)+check("gate/below-threshold answer is flagged", on[0]["answers"]["q_low"]["low_confidence"], True)+check("gate/below-threshold echoes the threshold", on[0]["answers"]["q_low"]["abstention_threshold"], 0.80)+check_true("gate/the raw answer is untouched",+ on[0]["answers"]["q_high"]["choice"] == "a" and on[0]["answers"]["q_low"]["choice"] == "b")++# The two states a caller used to be unable to tell apart are now told apart by a value, and the+# distinction survives a round trip through JSON, which is how a shadow record reaches a log.+import json as _json # noqa: E402++round_tripped = _json.loads(_json.dumps(on[0]["answers"]))+check("gate/survives a JSON round trip", round_tripped["q_high"]["abstention"], GATE_PASSED)++# One rule, one implementation: the flag stays `flag_low_confidence`'s, not a second copy of it.+# `apply_confidence_gate(results, 0.0)` is a gate that ran and nothing can fail, which is why+# `flag_low_confidence`'s 0.0 no-op above is still correct and still asserted.+zero = fresh()+apply_confidence_gate(zero, 0.0)+check("gate/0.0 reports passed", zero[0]["answers"]["q_low"]["abstention"], GATE_PASSED)+check_true("gate/0.0 still flags nothing", "low_confidence" not in zero[0]["answers"]["q_low"])++# A non-finite or missing confidence is skipped by `flag_low_confidence`, so it is not an+# abstention. It is also not a pass: the gate ran and could not decide, which is the state Argo+# Rollouts ships as `Inconclusive` and the one collapsing it into success would be the same lie+# `low_confidence`'s absence already tells.+odd = [{"answers": {+ "q_nan": {"type": "choice", "answer_confidence": float("nan")},+ "q_none": {"type": "choice"},+ "q_inf": {"type": "choice", "answer_confidence": float("inf")},+ "q_bool": {"type": "choice", "answer_confidence": True},+ "q_fine": {"type": "choice", "answer_confidence": 0.99},+}}]+apply_confidence_gate(odd, 0.80)+for qid, a in odd[0]["answers"].items():+ if qid == "q_fine":+ check("gate/a usable confidence is evaluated %s" % qid, a["abstention"], GATE_PASSED)+ else:+ check("gate/unusable confidence is unevaluated %s" % qid, a["abstention"], GATE_UNEVALUATED)+ check_true("gate/unusable confidence is not flagged %s" % qid, "low_confidence" not in a)++# `answer_confidence` is the quantity the gate reads, so a `noul`-style entropy `confidence`+# below threshold does not abstain while `answer_confidence` above it does. The state follows+# the same field the flag follows.+pref = [{"answers": {+ "q_a": {"type": "noul", "answer_confidence": 0.91, "confidence": 0.10},+ "q_b": {"type": "choice", "answer_confidence": 0.10, "confidence": 0.99},+}}]+apply_confidence_gate(pref, 0.80)+check("gate/gate reads answer_confidence (a)", pref[0]["answers"]["q_a"]["abstention"], GATE_PASSED)+check("gate/gate reads answer_confidence (b)", pref[0]["answers"]["q_b"]["abstention"], GATE_ABSTAINED)++# The guards the six call sites rely on: a result that is not a dict, answers that are not a+# dict, and answers that are not dicts. Every call site passes a list it has already partly+apply_confidence_gate([None, {"answers": None}, {"answers": [1, 2]}], 0.5)+apply_confidence_gate([], None)+PASS.append("gate/non-dict results are skipped without raising")++# --------------------------------------------------------------- the operator's three questions+# End to end through a real call site, weight-free. `decide` forwards to any runner with a+# `predict`, so this exercises `laya/structured.py`'s own gate call site rather than the helper+# in isolation -- the six call sites are where the state used to be skipped entirely.+from laya.structured import decide # noqa: E402++_GATE_Q = {"type": "choice", "instructions": "Pick one.",+ "criteria": {"a": "alpha", "b": "beta"}}+CONFIDENCES = {"high": 0.95, "low": 0.40}+QUESTIONS = {qid: dict(_GATE_Q) for qid in CONFIDENCES}+++class _StubRunner:+ def __init__(self, conf):+ self.conf = conf++ def predict(self, state, questions, **kwargs):+ return {"model": "stub", "answers": {+ qid: {"type": "choice", "choice": "a", "answer_confidence": self.conf[qid]}+ for qid in questions}}+++stub = _StubRunner(CONFIDENCES)++# 1. Was the gate in effect at all? Before, both answers were byte-identical to the gated pair+# below, so the answer was "no way to tell".+ungated = decide(stub, "state", questions=QUESTIONS)+for qid in CONFIDENCES:+ check("workflow/ungated %s says so" % qid, ungated[qid]["abstention"], GATE_NOT_CONFIGURED)++gated = decide(stub, "state", questions=QUESTIONS, min_confidence=0.80)+check("workflow/the cleared answer reports passed", gated["high"]["abstention"], GATE_PASSED)+check("workflow/the low answer reports abstained", gated["low"]["abstention"], GATE_ABSTAINED)++# 2. What fraction abstained, over whatever decisions the operator kept? Computable from the+# artifact, with no re-run and no out-of-band record of which threshold was used.+log = [gated, decide(stub, "other", questions=QUESTIONS, min_confidence=0.80)]+answers = [a for d in log for a in d.values()]+abstained = sum(a["abstention"] == GATE_ABSTAINED for a in answers)+check("workflow/abstention rate over the log", abstained, 2)+check("workflow/…out of", len(answers), 4)+check_true("workflow/…and no answer is left unlabelled", all("abstention" in a for a in answers))++# 3. What threshold produced these? `flag_low_confidence` consumes it and drops it, so before+# this a batch that gated different request classes differently could not be re-split.+check("workflow/the threshold is recoverable", gated["low"]["abstention_threshold"], 0.80)+check_true("workflow/…and a per-class mix is re-splittable",+ {a["abstention_threshold"] for a in gated.values()} == {0.80})++# The schema projection is unchanged -- an abstained field is still `null` -- so the flat+# `values` mapping still cannot say why. `return_details=True` is where the state is readable,+# and it was unreadable before.+SCHEMA = {"type": "object",+ "properties": {"high": {"type": "string", "enum": ["a", "b"]},+ "low": {"type": "string", "enum": ["a", "b"]}}}+projected = decide(stub, "state", schema=SCHEMA, min_confidence=0.80)+check("workflow/the schema projection is unchanged", projected["low"], None)+check("workflow/…and still reports the cleared field", projected["high"], "a")+detailed = decide(stub, "state", schema=SCHEMA, min_confidence=0.80, return_details=True)+check("workflow/return_details carries the state",+ detailed.answers["low"]["abstention"], GATE_ABSTAINED)+# --------------------------------------------------------------- exportedimport laya # noqa: E402@@ -157,6 +321,11 @@ check_true("export/check_min_confidence is in dir()", "check_min_confidence" incheck_true("export/flag_low_confidence is importable from laya", hasattr(laya, "flag_low_confidence"))check_true("export/flag_low_confidence is in __all__", "flag_low_confidence" in laya.__all__)check_true("export/flag_low_confidence is in dir()", "flag_low_confidence" in dir(laya))+check_true("export/apply_confidence_gate is importable from laya", hasattr(laya, "apply_confidence_gate"))+check_true("export/apply_confidence_gate is in __all__", "apply_confidence_gate" in laya.__all__)+check_true("export/apply_confidence_gate is in dir()", "apply_confidence_gate" in dir(laya))+for _name in ("GATE_NOT_CONFIGURED", "GATE_PASSED", "GATE_ABSTAINED", "GATE_UNEVALUATED", "GATE_STATES"):+ check_true("export/%s is importable from laya" % _name, hasattr(laya, _name))print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:diff --git a/tests/test_predict_long.py b/tests/test_predict_long.pyindex 67d54d69..d9924639 100644--- a/tests/test_predict_long.py+++ b/tests/test_predict_long.py@@ -206,7 +206,11 @@ stub = _LongStub()out = _router(stub).predict_long({"body": "y" * 300}, Q, model="english",window=64, stride=32, batch_size=8, lang="de")check("router/routing key attached", out["routing"]["model"], "english")-check("router/answers come from the scan", out["answers"], {"scanned": {"noul": 0.5}})+# `predict_long` takes no `min_confidence` -- a window scan has no single confidence to gate on --+# so every scanned answer says it ran ungated instead of saying nothing. That is the point: an+# operator reading a long-document result can now see there was no gate on it.+check("router/answers come from the scan", out["answers"],+ {"scanned": {"noul": 0.5, "abstention": "not_configured"}})check("router/window forwarded", stub.calls[0]["window"], 64)check("router/stride forwarded", stub.calls[0]["stride"], 32)check("router/batch_size forwarded", stub.calls[0]["batch_size"], 8)@@ -255,7 +259,8 @@ class _Recorder:recorder_stub = _LongStub()_router(recorder_stub).predict_long({"body": "y" * 300}, Q, model="english",on_predict_end=_Recorder().on_predict_end)-check("router/per-call end hook sees the scan", ended["answers"], {"scanned": {"noul": 0.5}})+check("router/per-call end hook sees the scan", ended["answers"],+ {"scanned": {"noul": 0.5, "abstention": "not_configured"}})# 8. an agent with no predict_long is a named caller error, not a bare AttributeErrorclass _NoScan:diff --git a/tests/test_structured.py b/tests/test_structured.pyindex fcf4ae2c..5a80d331 100644--- a/tests/test_structured.py+++ b/tests/test_structured.py@@ -163,8 +163,10 @@ check("decide/details routing", details.routing, {"model": "english"})runner = FakeRunner({"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})out = decide(runner, "s", questions={"a": {"type": "noul", "instructions": "?"}})+# The pass-through is the raw answer, and it now carries the gate's state: no `min_confidence`+# was passed, so every answer reports `not_configured` rather than reporting nothing.check("decide/questions pass-through returns answers", out,- {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})+ {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}})check_raises("decide/requires schema or questions", ValueError, lambda: decide(runner, "s"))check_raises("decide/rejects both", ValueError, lambda: decide(runner, "s", schema=SCHEMA, questions={}))@@ -280,7 +282,8 @@ check("batch/details routing", details[0].routing, {"model": "english"})# questions= is the raw-answers pass-through, exactly like decide()raw = decide_batch(FakeBatchRunner({"a": {"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}}),["a"], questions={"x": {"type": "noul", "instructions": "?"}})-check("batch/questions pass-through", raw, [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}])+check("batch/questions pass-through", raw,+ [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}}])class FakeRouterLike: