Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(confidence): report the abstention gate's state on every answer

Bruce-Yii02aaed4ca19b
diff --git a/README.md b/README.md
index 4aadb0c4..97a7307f 100644
--- a/README.md
+++ b/README.md
@@ -97,7 +97,7 @@ The shipped checkpoints work zero-shot, but fine-tuning on decisions from your o
## What's new in 0.3.21
* **ONNX catches up with PyTorch.** `ONNXAgent` gains `predict_batch` (with `sort_by_length`), `predict_long` and `decide_batch`, `scripts/export_onnx.py --quantize` writes a per-channel INT8 copy for CPU, and `laya-evals run --onnx` scores an export with the same gates as the torch path.
-* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them.
+* **Opt-in abstention.** `min_confidence=` on `predict`, `predict_batch`, `decide` and `decide_batch` flags answers below a threshold on `answer_confidence` with `low_confidence: True`, and `decide` returns `None` for them. Every answer also reports `abstention` — `not_configured`, `passed`, `abstained` or `unevaluated` — so a caller can tell an ungated run from a gated one that cleared.
* **Batch everywhere.** `decide_batch`, `Router.predict_long`, `laya --batch FILE`, the MCP `laya_predict_batch` / `laya_route_batch` / `laya_decide` tools, and LangChain `batch()` / `abatch()` all run on shared forward passes. New `LayaDecision` (LangChain), LlamaIndex selectors (`laya[llamaindex]`) and CrewAI routing (`laya[crewai]`).
* **Per-request token budget.** `max_len` / `head_max_len` now reach every surface: `laya-serve` (capped by `LAYA_MAX_TOKEN_BUDGET`), `Router.predict_batch` requests, the CLI (`--questions`, `--max-len`, `--head-max-len`), MCP tools and LangChain nodes.
* **Operations.** `LAYA_MAX_LOADED`, `LAYA_REVISION` and per-checkpoint SHA-256 maps; `/health` reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries `Retry-After`; `compile=True` no longer recompiles for every request shape.
@@ -719,6 +719,17 @@ else:
The threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.
+A gate is a policy, and a policy whose application you cannot observe is not one. `low_confidence` is written only when the gate fires, so its absence cannot tell you "you passed no `min_confidence`" apart from "a gate ran and this answer cleared it" — you could neither compute an abstention rate nor prove the gate was in effect. Every answer therefore also reports `abstention`, one of four values, present whether or not a gate was configured:
+
+| `abstention` | meaning |
+|---|---|
+| `not_configured` | no `min_confidence` was passed, so there is no gate |
+| `passed` | a gate ran and this answer's `answer_confidence` cleared it |
+| `abstained` | a gate ran and this answer's `answer_confidence` fell below it |
+| `unevaluated` | a gate ran and this answer carried no usable confidence, so the gate could not decide |
+
+`abstention_threshold` echoes the threshold whenever a gate ran, so a log can be re-split by the gate that produced it instead of by whatever the caller happened to remember passing. `unevaluated` is the case a boolean cannot express — reporting it as a pass would be as wrong as reporting it as a flag, so a NaN, a missing confidence or a `bool` lands in its own state. The raw answer, probabilities and confidence are untouched either way.
+
---
## Prediction Hooks
diff --git a/docs/reference/helpers.md b/docs/reference/helpers.md
index 972c520b..56751064 100644
--- a/docs/reference/helpers.md
+++ b/docs/reference/helpers.md
@@ -44,6 +44,10 @@
::: laya.confidence.flag_low_confidence
+::: laya.confidence.apply_confidence_gate
+
+::: laya.confidence.GATE_STATES
+
## Calibration and training
::: laya.common.confidence_from_probs
diff --git a/laya/__init__.py b/laya/__init__.py
index fda54c7c..a5f78587 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -30,6 +30,12 @@ _LAZY_ATTRS = {
"confidence_from_probs": (".common", "confidence_from_probs"),
"check_min_confidence": (".confidence", "check_min_confidence"),
"flag_low_confidence": (".confidence", "flag_low_confidence"),
+ "apply_confidence_gate": (".confidence", "apply_confidence_gate"),
+ "GATE_NOT_CONFIGURED": (".confidence", "GATE_NOT_CONFIGURED"),
+ "GATE_PASSED": (".confidence", "GATE_PASSED"),
+ "GATE_ABSTAINED": (".confidence", "GATE_ABSTAINED"),
+ "GATE_UNEVALUATED": (".confidence", "GATE_UNEVALUATED"),
+ "GATE_STATES": (".confidence", "GATE_STATES"),
"render_options": (".common", "render_options"),
"QTYPES": (".common", "QTYPES"),
"QTYPE_NAMES": (".common", "QTYPE_NAMES"),
@@ -90,6 +96,12 @@ __all__ = [
"confidence_from_probs",
"check_min_confidence",
"flag_low_confidence",
+ "apply_confidence_gate",
+ "GATE_NOT_CONFIGURED",
+ "GATE_PASSED",
+ "GATE_ABSTAINED",
+ "GATE_UNEVALUATED",
+ "GATE_STATES",
"render_options",
"QTYPES",
"QTYPE_NAMES",
diff --git a/laya/agent.py b/laya/agent.py
index 5ca3861c..c7d914bc 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -32,7 +32,7 @@ from .common import (
serialize_state,
temp_bucket,
)
-from .confidence import check_min_confidence, flag_low_confidence
+from .confidence import apply_confidence_gate, check_min_confidence
from .hooks import (
HookRegistry, PredictContext, _as_sequence, aggregate_usage, compose_hooks, dispatch,
normalise_hooks, validate_timeout,
@@ -1076,8 +1076,7 @@ class Agent(HookRegistry):
ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
if ctx.results is not None:
ctx.usage = aggregate_usage(ctx.results)
- if mc is not None:
- flag_low_confidence(ctx.results, mc)
+ apply_confidence_gate(ctx.results, mc)
try:
dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)
except BaseException as hook_exc:
@@ -1277,7 +1276,11 @@ class Agent(HookRegistry):
usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
usage["output_tokens"] = 0
usage["windows"] = len(results)
- return {"model": "laya-rl-agent", "answers": answers, "usage": usage}
+ result = {"model": "laya-rl-agent", "answers": answers, "usage": usage}
+ # `predict_long` accepts no `min_confidence` -- the window loop has no single confidence to
+ # gate on -- so every answer reports that it ran ungated rather than reporting nothing.
+ apply_confidence_gate([result], None)
+ return result
@torch.no_grad()
def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]], lang: Optional[str] = None,
diff --git a/laya/confidence.py b/laya/confidence.py
index 253329bf..495d2037 100644
--- a/laya/confidence.py
+++ b/laya/confidence.py
@@ -4,7 +4,26 @@ Pure Python: safe to import without PyTorch so that Router and structured
decisions stay lightweight and free of torch import overhead.
"""
import math
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional
+
+
+def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:
+ """The confidence the abstention gate reads for `answer`, or None if there is not a usable one.
+
+ `answer_confidence` is the calibrated max(p) confidence and is what the gate must read;
+ `confidence` is the fallback, and for `choice` and `score` it is normalized entropy on a
+ different scale, which must not be compared against the same threshold. A bool, a NaN, an
+ infinity or a missing field is not a number the gate can decide on.
+
+ Shared by :func:`flag_low_confidence` and :func:`apply_confidence_gate` so the flag and the
+ reported state cannot disagree about which quantity was read.
+ """
+ conf = answer.get("answer_confidence")
+ if conf is None:
+ conf = answer.get("confidence")
+ if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):
+ return conf
+ return None
def check_min_confidence(v: Any) -> float:
@@ -33,8 +52,63 @@ def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) ->
for a in answers.values():
if not isinstance(a, dict):
continue
- conf = a.get("answer_confidence")
- if conf is None:
- conf = a.get("confidence")
- if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf) and conf < min_confidence:
+ conf = _gate_confidence(a)
+ if conf is not None and conf < min_confidence:
a["low_confidence"] = True
+
+
+#: The confidence gate's state, as reported on every answer by :func:`apply_confidence_gate`.
+#:
+#: `low_confidence` is written only when the gate fires, so its absence cannot tell a caller "no
+#: ``min_confidence`` was passed" apart from "a gate ran and this answer cleared it". These four
+#: values make the state a thing you read rather than a thing you infer from a missing field.
+GATE_NOT_CONFIGURED = "not_configured" # no threshold was passed, so there is no gate to apply
+GATE_PASSED = "passed" # a gate ran and this answer's confidence cleared it
+GATE_ABSTAINED = "abstained" # a gate ran and this answer's confidence fell below it
+GATE_UNEVALUATED = "unevaluated" # a gate ran and this answer carried no usable confidence
+
+GATE_STATES = (GATE_NOT_CONFIGURED, GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)
+
+
+def apply_confidence_gate(results: List[Dict[str, Any]], min_confidence: Optional[float] = None) -> None:
+ """Report the confidence gate's state on every answer, whether or not a gate was configured.
+
+ A gate is a policy, and a policy whose application cannot be observed is not one. This writes
+ ``abstention`` -- one of :data:`GATE_STATES` -- onto every answer, plus ``abstention_threshold``
+ whenever a gate actually ran. It is what lets a caller answer three questions it otherwise
+ cannot:
+
+ * what fraction of decisions abstained, without guessing which of "no gate" and "gate
+ cleared it" produced a missing ``low_confidence``;
+ * whether the gate was in effect at all on this run;
+ * what threshold produced these results, which ``flag_low_confidence`` consumes and drops --
+ without it a batch that used different thresholds per request class cannot be re-split.
+
+ ``GATE_UNEVALUATED`` is the case a boolean cannot express: the gate ran and the answer carried
+ no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as
+ reporting it as a flag.
+
+ The flag itself stays :func:`flag_low_confidence`'s -- this delegates to it rather than
+ re-implementing the rule, so the boolean and the state cannot drift apart. Call this
+ unconditionally, once per call, in place of a `if min_confidence is not None:` guard: that
+ guard is what leaves a path reporting nothing at all.
+ """
+ if min_confidence is not None:
+ flag_low_confidence(results, min_confidence)
+ for res in results:
+ answers = res.get("answers") if isinstance(res, dict) else None
+ if not isinstance(answers, dict):
+ continue
+ for a in answers.values():
+ if not isinstance(a, dict):
+ continue
+ if min_confidence is None:
+ a["abstention"] = GATE_NOT_CONFIGURED
+ continue
+ if a.get("low_confidence"):
+ a["abstention"] = GATE_ABSTAINED
+ elif _gate_confidence(a) is None:
+ a["abstention"] = GATE_UNEVALUATED
+ else:
+ a["abstention"] = GATE_PASSED
+ a["abstention_threshold"] = float(min_confidence)
diff --git a/laya/onnx_agent.py b/laya/onnx_agent.py
index c31778a9..0095bb12 100644
--- a/laya/onnx_agent.py
+++ b/laya/onnx_agent.py
@@ -27,7 +27,7 @@ from laya.common import (
TEMP_MAX,
clamp_temperature,
)
-from laya.confidence import check_min_confidence, flag_low_confidence
+from laya.confidence import apply_confidence_gate, check_min_confidence
class ONNXAgent(HookRegistry):
@@ -356,8 +356,7 @@ class ONNXAgent(HookRegistry):
ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
if ctx.results is not None:
ctx.usage = aggregate_usage(ctx.results)
- if mc is not None:
- flag_low_confidence(ctx.results, mc)
+ apply_confidence_gate(ctx.results, mc)
try:
dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)
except BaseException as hook_exc:
@@ -515,7 +514,10 @@ class ONNXAgent(HookRegistry):
usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
usage["output_tokens"] = 0
usage["windows"] = len(results)
- return {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}
+ result = {"model": "laya-rl-agent-onnx", "answers": answers, "usage": usage}
+ # As in `Agent.predict_long`: no `min_confidence` here, so say so on every answer.
+ apply_confidence_gate([result], None)
+ return result
def _infer(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
max_len: Optional[int] = None, head_max_len: Optional[int] = None,
diff --git a/laya/router.py b/laya/router.py
index 9a2f3cc8..97c58526 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -36,7 +36,7 @@ import time
from collections.abc import Sequence as SequenceABC
from typing import Any, Dict, List, Optional, Sequence, Union
-from .confidence import check_min_confidence, flag_low_confidence
+from .confidence import apply_confidence_gate, check_min_confidence
from .hooks import (
HookRegistry, PredictContext, aggregate_usage, compose_hooks, dispatch, normalise_hooks,
validate_timeout,
@@ -803,8 +803,7 @@ class Router(HookRegistry):
ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
if ctx.results is not None:
ctx.usage = aggregate_usage(ctx.results)
- if mc is not None:
- flag_low_confidence(ctx.results, mc)
+ apply_confidence_gate(ctx.results, mc)
try:
dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock, timeout=timeout)
except BaseException as hook_exc:
@@ -1153,14 +1152,14 @@ class Router(HookRegistry):
# had already run. Each is still ended, so a hook that opens something in start
# (a span, an in-flight count) always sees the matching end.
for ctx in started:
- if mc is not None and ctx.results:
- flag_low_confidence(ctx.results, mc)
+ if ctx.results:
+ apply_confidence_gate(ctx.results, mc)
self._end_contexts(active, started, raise_errors, timeout, error=exc)
raise
for ctx in started:
- if mc is not None and ctx.results:
- flag_low_confidence(ctx.results, mc)
+ if ctx.results:
+ apply_confidence_gate(ctx.results, mc)
self._end_contexts(active, started, raise_errors, timeout)
for i, ctx in zip(indices, started):
results[i] = ctx.results[0]
diff --git a/laya/structured.py b/laya/structured.py
index 30e5d3ba..fe0bd34e 100644
--- a/laya/structured.py
+++ b/laya/structured.py
@@ -20,7 +20,7 @@ from collections.abc import Sequence as SequenceABC
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
-from .confidence import check_min_confidence, flag_low_confidence
+from .confidence import apply_confidence_gate, check_min_confidence
MAX_PROPERTIES = 32
MAX_OPTIONS = 32
@@ -280,8 +280,8 @@ def decide(runner, state: Any, schema: Any = None, *, questions: Optional[Dict[s
else:
raise
- if mc is not None and isinstance(result, dict):
- flag_low_confidence([result], mc)
+ if isinstance(result, dict):
+ apply_confidence_gate([result], mc)
answers = result.get("answers", {}) or {}
values = _project(answers, fields) if fields is not None else dict(answers)
@@ -342,10 +342,9 @@ def decide_batch(runner, states: Sequence[Any], schema: Any = None, *,
# Agent convention: a list of states evaluated against one question set.
results = predict_batch(list(states), questions, **predict_kwargs)
- if mc is not None:
- # Flagged here rather than passed down, so a runner whose predict_batch predates the
- # keyword still gets the same projection.
- flag_low_confidence([r for r in results if isinstance(r, dict)], mc)
+ # Applied here rather than passed down, so a runner whose predict_batch predates the
+ # keyword still gets the same projection and the same reported state.
+ apply_confidence_gate([r for r in results if isinstance(r, dict)], mc)
def _one(r: Dict[str, Any]) -> Any:
answers = r.get("answers", {}) or {}
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index 02dc784e..f0d99911 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -145,6 +145,170 @@ check_true("flag/q_exact unflagged", "low_confidence" not in ans["q_exact"])
check("flag/answer_confidence intact", ans["q_low"]["answer_confidence"], 0.45)
check("flag/raw choice intact", ans["q_high"]["choice"], "a")
+# --------------------------------------------------------------- the gate's state is reported
+# `low_confidence` is written only when the gate fires, so its absence cannot tell a caller
+# "no min_confidence was passed" apart from "a gate ran and this answer cleared it". An operator
+# with 10,000 logged decisions cannot compute an abstention rate, cannot prove the gate was in
+# effect, and cannot re-split a batch that used different thresholds per request class -- the
+# threshold is consumed and dropped. `docs/staged-adoption.md` asks a shadow record to carry
+# "errors and any fallback or review decision"; there was no field to put that in.
+#
+# The contract: the gate's state is a value on every answer, so it is never inferred from a
+# missing field.
+from laya.confidence import ( # noqa: E402
+ GATE_ABSTAINED,
+ GATE_NOT_CONFIGURED,
+ GATE_PASSED,
+ GATE_STATES,
+ GATE_UNEVALUATED,
+ apply_confidence_gate,
+)
+
+check("gate/the vocabulary is closed",
+ list(GATE_STATES), ["not_configured", "passed", "abstained", "unevaluated"])
+
+fresh = lambda: [{ # noqa: E731
+ "answers": {
+ "q_high": {"type": "choice", "choice": "a", "answer_confidence": 0.92},
+ "q_low": {"type": "choice", "choice": "b", "answer_confidence": 0.45},
+ }
+}]
+
+# No gate: every answer says so, and nothing is flagged.
+off = fresh()
+apply_confidence_gate(off, None)
+for qid, a in off[0]["answers"].items():
+ check("gate/not_configured %s" % qid, a["abstention"], GATE_NOT_CONFIGURED)
+ check_true("gate/not_configured carries no threshold %s" % qid, "abstention_threshold" not in a)
+ check_true("gate/not_configured is not a flag %s" % qid, "low_confidence" not in a)
+
+# A gate that ran: the answer that cleared it says `passed`, which is the state the boolean
+# could not express. This is the whole point -- before, this answer was byte-identical to the
+# ungated one above.
+on = fresh()
+apply_confidence_gate(on, 0.80)
+check("gate/cleared answer says passed", on[0]["answers"]["q_high"]["abstention"], GATE_PASSED)
+check("gate/cleared answer is still unflagged", "low_confidence" in on[0]["answers"]["q_high"], False)
+check("gate/cleared answer echoes the threshold", on[0]["answers"]["q_high"]["abstention_threshold"], 0.80)
+check("gate/below-threshold answer says abstained", on[0]["answers"]["q_low"]["abstention"], GATE_ABSTAINED)
+check("gate/below-threshold answer is flagged", on[0]["answers"]["q_low"]["low_confidence"], True)
+check("gate/below-threshold echoes the threshold", on[0]["answers"]["q_low"]["abstention_threshold"], 0.80)
+check_true("gate/the raw answer is untouched",
+ on[0]["answers"]["q_high"]["choice"] == "a" and on[0]["answers"]["q_low"]["choice"] == "b")
+
+# The two states a caller used to be unable to tell apart are now told apart by a value, and the
+# distinction survives a round trip through JSON, which is how a shadow record reaches a log.
+import json as _json # noqa: E402
+
+round_tripped = _json.loads(_json.dumps(on[0]["answers"]))
+check("gate/survives a JSON round trip", round_tripped["q_high"]["abstention"], GATE_PASSED)
+
+# One rule, one implementation: the flag stays `flag_low_confidence`'s, not a second copy of it.
+# `apply_confidence_gate(results, 0.0)` is a gate that ran and nothing can fail, which is why
+# `flag_low_confidence`'s 0.0 no-op above is still correct and still asserted.
+zero = fresh()
+apply_confidence_gate(zero, 0.0)
+check("gate/0.0 reports passed", zero[0]["answers"]["q_low"]["abstention"], GATE_PASSED)
+check_true("gate/0.0 still flags nothing", "low_confidence" not in zero[0]["answers"]["q_low"])
+
+# A non-finite or missing confidence is skipped by `flag_low_confidence`, so it is not an
+# abstention. It is also not a pass: the gate ran and could not decide, which is the state Argo
+# Rollouts ships as `Inconclusive` and the one collapsing it into success would be the same lie
+# `low_confidence`'s absence already tells.
+odd = [{"answers": {
+ "q_nan": {"type": "choice", "answer_confidence": float("nan")},
+ "q_none": {"type": "choice"},
+ "q_inf": {"type": "choice", "answer_confidence": float("inf")},
+ "q_bool": {"type": "choice", "answer_confidence": True},
+ "q_fine": {"type": "choice", "answer_confidence": 0.99},
+}}]
+apply_confidence_gate(odd, 0.80)
+for qid, a in odd[0]["answers"].items():
+ if qid == "q_fine":
+ check("gate/a usable confidence is evaluated %s" % qid, a["abstention"], GATE_PASSED)
+ else:
+ check("gate/unusable confidence is unevaluated %s" % qid, a["abstention"], GATE_UNEVALUATED)
+ check_true("gate/unusable confidence is not flagged %s" % qid, "low_confidence" not in a)
+
+# `answer_confidence` is the quantity the gate reads, so a `noul`-style entropy `confidence`
+# below threshold does not abstain while `answer_confidence` above it does. The state follows
+# the same field the flag follows.
+pref = [{"answers": {
+ "q_a": {"type": "noul", "answer_confidence": 0.91, "confidence": 0.10},
+ "q_b": {"type": "choice", "answer_confidence": 0.10, "confidence": 0.99},
+}}]
+apply_confidence_gate(pref, 0.80)
+check("gate/gate reads answer_confidence (a)", pref[0]["answers"]["q_a"]["abstention"], GATE_PASSED)
+check("gate/gate reads answer_confidence (b)", pref[0]["answers"]["q_b"]["abstention"], GATE_ABSTAINED)
+
+# The guards the six call sites rely on: a result that is not a dict, answers that are not a
+# dict, and answers that are not dicts. Every call site passes a list it has already partly
+apply_confidence_gate([None, {"answers": None}, {"answers": [1, 2]}], 0.5)
+apply_confidence_gate([], None)
+PASS.append("gate/non-dict results are skipped without raising")
+
+# --------------------------------------------------------------- the operator's three questions
+# End to end through a real call site, weight-free. `decide` forwards to any runner with a
+# `predict`, so this exercises `laya/structured.py`'s own gate call site rather than the helper
+# in isolation -- the six call sites are where the state used to be skipped entirely.
+from laya.structured import decide # noqa: E402
+
+_GATE_Q = {"type": "choice", "instructions": "Pick one.",
+ "criteria": {"a": "alpha", "b": "beta"}}
+CONFIDENCES = {"high": 0.95, "low": 0.40}
+QUESTIONS = {qid: dict(_GATE_Q) for qid in CONFIDENCES}
+
+
+class _StubRunner:
+ def __init__(self, conf):
+ self.conf = conf
+
+ def predict(self, state, questions, **kwargs):
+ return {"model": "stub", "answers": {
+ qid: {"type": "choice", "choice": "a", "answer_confidence": self.conf[qid]}
+ for qid in questions}}
+
+
+stub = _StubRunner(CONFIDENCES)
+
+# 1. Was the gate in effect at all? Before, both answers were byte-identical to the gated pair
+# below, so the answer was "no way to tell".
+ungated = decide(stub, "state", questions=QUESTIONS)
+for qid in CONFIDENCES:
+ check("workflow/ungated %s says so" % qid, ungated[qid]["abstention"], GATE_NOT_CONFIGURED)
+
+gated = decide(stub, "state", questions=QUESTIONS, min_confidence=0.80)
+check("workflow/the cleared answer reports passed", gated["high"]["abstention"], GATE_PASSED)
+check("workflow/the low answer reports abstained", gated["low"]["abstention"], GATE_ABSTAINED)
+
+# 2. What fraction abstained, over whatever decisions the operator kept? Computable from the
+# artifact, with no re-run and no out-of-band record of which threshold was used.
+log = [gated, decide(stub, "other", questions=QUESTIONS, min_confidence=0.80)]
+answers = [a for d in log for a in d.values()]
+abstained = sum(a["abstention"] == GATE_ABSTAINED for a in answers)
+check("workflow/abstention rate over the log", abstained, 2)
+check("workflow/…out of", len(answers), 4)
+check_true("workflow/…and no answer is left unlabelled", all("abstention" in a for a in answers))
+
+# 3. What threshold produced these? `flag_low_confidence` consumes it and drops it, so before
+# this a batch that gated different request classes differently could not be re-split.
+check("workflow/the threshold is recoverable", gated["low"]["abstention_threshold"], 0.80)
+check_true("workflow/…and a per-class mix is re-splittable",
+ {a["abstention_threshold"] for a in gated.values()} == {0.80})
+
+# The schema projection is unchanged -- an abstained field is still `null` -- so the flat
+# `values` mapping still cannot say why. `return_details=True` is where the state is readable,
+# and it was unreadable before.
+SCHEMA = {"type": "object",
+ "properties": {"high": {"type": "string", "enum": ["a", "b"]},
+ "low": {"type": "string", "enum": ["a", "b"]}}}
+projected = decide(stub, "state", schema=SCHEMA, min_confidence=0.80)
+check("workflow/the schema projection is unchanged", projected["low"], None)
+check("workflow/…and still reports the cleared field", projected["high"], "a")
+detailed = decide(stub, "state", schema=SCHEMA, min_confidence=0.80, return_details=True)
+check("workflow/return_details carries the state",
+ detailed.answers["low"]["abstention"], GATE_ABSTAINED)
+
# --------------------------------------------------------------- exported
import laya # noqa: E402
@@ -157,6 +321,11 @@ check_true("export/check_min_confidence is in dir()", "check_min_confidence" in
check_true("export/flag_low_confidence is importable from laya", hasattr(laya, "flag_low_confidence"))
check_true("export/flag_low_confidence is in __all__", "flag_low_confidence" in laya.__all__)
check_true("export/flag_low_confidence is in dir()", "flag_low_confidence" in dir(laya))
+check_true("export/apply_confidence_gate is importable from laya", hasattr(laya, "apply_confidence_gate"))
+check_true("export/apply_confidence_gate is in __all__", "apply_confidence_gate" in laya.__all__)
+check_true("export/apply_confidence_gate is in dir()", "apply_confidence_gate" in dir(laya))
+for _name in ("GATE_NOT_CONFIGURED", "GATE_PASSED", "GATE_ABSTAINED", "GATE_UNEVALUATED", "GATE_STATES"):
+ check_true("export/%s is importable from laya" % _name, hasattr(laya, _name))
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
diff --git a/tests/test_predict_long.py b/tests/test_predict_long.py
index 67d54d69..d9924639 100644
--- a/tests/test_predict_long.py
+++ b/tests/test_predict_long.py
@@ -206,7 +206,11 @@ stub = _LongStub()
out = _router(stub).predict_long({"body": "y" * 300}, Q, model="english",
window=64, stride=32, batch_size=8, lang="de")
check("router/routing key attached", out["routing"]["model"], "english")
-check("router/answers come from the scan", out["answers"], {"scanned": {"noul": 0.5}})
+# `predict_long` takes no `min_confidence` -- a window scan has no single confidence to gate on --
+# so every scanned answer says it ran ungated instead of saying nothing. That is the point: an
+# operator reading a long-document result can now see there was no gate on it.
+check("router/answers come from the scan", out["answers"],
+ {"scanned": {"noul": 0.5, "abstention": "not_configured"}})
check("router/window forwarded", stub.calls[0]["window"], 64)
check("router/stride forwarded", stub.calls[0]["stride"], 32)
check("router/batch_size forwarded", stub.calls[0]["batch_size"], 8)
@@ -255,7 +259,8 @@ class _Recorder:
recorder_stub = _LongStub()
_router(recorder_stub).predict_long({"body": "y" * 300}, Q, model="english",
on_predict_end=_Recorder().on_predict_end)
-check("router/per-call end hook sees the scan", ended["answers"], {"scanned": {"noul": 0.5}})
+check("router/per-call end hook sees the scan", ended["answers"],
+ {"scanned": {"noul": 0.5, "abstention": "not_configured"}})
# 8. an agent with no predict_long is a named caller error, not a bare AttributeError
class _NoScan:
diff --git a/tests/test_structured.py b/tests/test_structured.py
index fcf4ae2c..5a80d331 100644
--- a/tests/test_structured.py
+++ b/tests/test_structured.py
@@ -163,8 +163,10 @@ check("decide/details routing", details.routing, {"model": "english"})
runner = FakeRunner({"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})
out = decide(runner, "s", questions={"a": {"type": "noul", "instructions": "?"}})
+# The pass-through is the raw answer, and it now carries the gate's state: no `min_confidence`
+# was passed, so every answer reports `not_configured` rather than reporting nothing.
check("decide/questions pass-through returns answers", out,
- {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9}})
+ {"a": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}})
check_raises("decide/requires schema or questions", ValueError, lambda: decide(runner, "s"))
check_raises("decide/rejects both", ValueError, lambda: decide(runner, "s", schema=SCHEMA, questions={}))
@@ -280,7 +282,8 @@ check("batch/details routing", details[0].routing, {"model": "english"})
# questions= is the raw-answers pass-through, exactly like decide()
raw = decide_batch(FakeBatchRunner({"a": {"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}}),
["a"], questions={"x": {"type": "noul", "instructions": "?"}})
-check("batch/questions pass-through", raw, [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9}}])
+check("batch/questions pass-through", raw,
+ [{"x": {"type": "noul", "noul": 0.9, "confidence": 0.9, "abstention": "not_configured"}}])
class FakeRouterLike: