Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(structured): stop calling answer_confidence "calibrated"
Bruce-Yiiacb2befe1388
diff --git a/docs/structured.md b/docs/structured.mdindex 28ba0e84..c650fa2b 100644--- a/docs/structured.md+++ b/docs/structured.md@@ -146,32 +146,40 @@ with per-field confidence, probabilities, the raw answers, and the usage and rou```pythonresult = agent.decide(state, schema=Ticket, return_details=True)-result.values["department"] # "billing"-result.answer_confidence["department"] # 0.94 calibrated max(p): gate on this one-result.confidence["department"] # 0.71 normalized entropy, which depends on label count-result.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}-result.usage # {"input_tokens": 42, "output_tokens": 0}-result.routing # the Router decision, when a Router answered+result.values["department"] # "billing"+result.answer_confidence["department"] # 0.94 max(p): the quantity min_confidence gates on+result.confidence["department"] # 0.71 normalized entropy, which depends on label count+result.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}+result.usage # {"input_tokens": 42, "output_tokens": 0}+result.routing # the Router decision, when a Router answered````confidence` and `answer_confidence` are different quantities, and the names follow the definitions.-`answer_confidence` is the calibrated `max(p)`: it is what temperature scaling fits, what the-reported ECE measures, and what `min_confidence` is defined against. `confidence` is normalized-entropy, which depends on how many options the question had -- `tests/test_confidence.py` pins that-a two-option distribution comes back as 0.90 on a `noul` and 0.53 on an equivalent `choice` -- so it-does not compare against a threshold, and it is not what the calibration figures describe. A field-that reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.+`answer_confidence` is `max(p)`, the probability mass on the answer being reported. It is what+temperature scaling fits, what every calibration figure in this repository is computed on, and what+`min_confidence` is compared against — which is the reason to gate on it rather than on+`confidence`. `confidence` is normalized entropy, which depends on how many options the question+had: `tests/test_confidence.py` pins that a two-option distribution comes back as 0.90 on a `noul`+and 0.53 on an equivalent `choice`, so it does not compare against a threshold. A field that+reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.-Gate on the calibrated one:+Gate on the same quantity the gate uses:```pythonif result.answer_confidence["department"] < 0.6:result.values["department"] = "human-review"```-Both numbers are reported because both are sometimes what you want: `confidence` answers "how-concentrated is this distribution", `answer_confidence` answers "how much should I trust this-answer", and only the second is comparable across question shapes.+`answer_confidence` being the right number to *filter* on is not the same as it being a trustworthy+probability. Reading it as "about c of the answers returned at c are correct" holds only after+temperatures have been fitted and validated on held-out data for that checkpoint and question shape.+The shipped checkpoints are over-confident as shipped and `laya-multilingual` ships with no fitted+temperatures at all — see the README's+[Calibration](https://github.com/NandhaKishorM/laya#calibration) and+[Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sections, and the+[fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb)+for the fitting loop. Fit before you rely on the level; report it because it is the quantity the+gate and the eval harness both use.## How it maps internallydiff --git a/laya/confidence.py b/laya/confidence.pyindex c96e5ff0..5a20657a 100644--- a/laya/confidence.py+++ b/laya/confidence.py@@ -7,16 +7,25 @@ import mathfrom typing import Any, Dict, List, Optional-def calibrated_confidence(answer: Dict[str, Any]) -> Optional[float]:- """The calibrated `answer_confidence` of `answer`, or None if it did not report one.+def answer_confidence_value(answer: Dict[str, Any]) -> Optional[float]:+ """The `answer_confidence` of `answer`, or None if it did not report a usable one.- `answer_confidence` is the calibrated max(p) quantity: it is what temperature scaling fits,- what the reported ECE measures, and what the `min_confidence` gate is defined against (#361).- `confidence`, by contrast, is normalized entropy on a scale that depends on the number of- options -- `common.confidence_from_probs` calls it "not calibrated", and #394 is an open issue- saying a confidence threshold does not transfer across option counts. So this deliberately- does **not** fall back to `confidence`: reporting the entropy number under the calibrated name- would be worse than reporting nothing, because the name is what a caller filters on.+ `answer_confidence` is `max(p)` -- the probability mass on the answer being reported. It is+ the quantity temperature scaling fits, the quantity every calibration figure in this+ repository is computed on, and the quantity the `min_confidence` gate is defined against+ (#361). That makes it the right number for a threshold a caller compares against; it is+ not on its own a claim that the number is right. Reading it as "about c of the answers+ returned at c are correct" holds only after temperatures have been fitted and validated on+ held-out data for that checkpoint and question shape -- the shipped checkpoints are+ over-confident as shipped, and `laya-multilingual` ships with no fitted temperatures at all+ (README, Calibration). `common.answer_confidence` states the quantity; this states when it+ carries that property.++ `confidence` is a different quantity on a different scale -- normalized entropy, which+ `common.confidence_from_probs` calls "not calibrated" and #394 says does not transfer across+ option counts. So this deliberately does **not** fall back to it: reporting the entropy+ number under the `answer_confidence` name would be worse than reporting nothing, because the+ name is what a caller filters on.A `bool` is not a confidence and a NaN is not a decision, so both give None rather than 0.0."""@@ -29,10 +38,10 @@ def calibrated_confidence(answer: Dict[str, Any]) -> Optional[float]:def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:"""The number the abstention gate compares against `min_confidence`, or None if there is none.- The calibrated `answer_confidence` first, falling back to the entropy `confidence` so an- answer that carries only the older field is still gated rather than silently passed.+ `answer_confidence` first, falling back to the entropy `confidence` so an answer that carries+ only the older field is still gated rather than silently passed."""- conf = calibrated_confidence(answer)+ conf = answer_confidence_value(answer)if conf is None:conf = answer.get("confidence")if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):@@ -53,8 +62,9 @@ def check_min_confidence(v: Any) -> float:def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) -> None:"""Opt-in abstention marker (#361): flag answers whose confidence falls below `min_confidence`.- Reads `answer_confidence` (the calibrated max(p) confidence, invariant to label count k),- falling back to `confidence` if `answer_confidence` is absent.+ Reads `answer_confidence` (`max(p)`, the quantity the calibration figures describe and the one+ that does not drift with the number of options), falling back to `confidence` if+ `answer_confidence` is absent.The raw answer and confidence stay intact; `low_confidence: True` is added."""if min_confidence == 0.0:diff --git a/laya/structured.py b/laya/structured.pyindex 5f296d9d..d72547de 100644--- a/laya/structured.py+++ b/laya/structured.py@@ -20,7 +20,7 @@ from collections.abc import Sequence as SequenceABCfrom dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Optional, Sequence, Tuple-from .confidence import calibrated_confidence, check_min_confidence, flag_low_confidence+from .confidence import answer_confidence_value, check_min_confidence, flag_low_confidenceMAX_PROPERTIES = 32MAX_OPTIONS = 32@@ -38,12 +38,21 @@ class DecisionResult:`values` is the schema-shaped output. `confidence` and `probabilities` are keyed by field,and `answers` is Laya's raw answer per field.- `answer_confidence` is the calibrated max(p) confidence per field -- the quantity temperature- scaling fits, the reported ECE measures, and the `min_confidence` gate is defined against --- under its own name. `confidence` keeps the normalized-entropy value it has always had, because- that is not a calibrated quantity and a caller filtering this artifact to decide what to- escalate has to be able to tell the two apart. A field that reported no usable calibrated- confidence maps to `None`, which is distinct from a reported `0.0`.+ `answer_confidence` is `max(p)` per field -- the probability mass on the answer being+ reported, under its own name. That makes it the same decision quantity `min_confidence`+ compares against and the calibration and eval stack measures, which is the reason to report+ it: a caller filtering this artifact to decide what to automate has to be filtering on the+ number the gate actually used.++ It is not a claim that the number is right. Reading it as "about c of the answers returned at+ c are correct" holds only after temperatures are fitted and validated on held-out data for+ that checkpoint and question shape; the shipped checkpoints are over-confident as shipped and+ `laya-multilingual` ships with no fitted temperatures at all. See `common.answer_confidence`+ and the README's Calibration section.++ `confidence` keeps the normalized-entropy value it has always had, because that is a+ different quantity on a scale that depends on the label count. A field that reported no+ usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`."""values: Dict[str, Any]@@ -247,8 +256,8 @@ def _details(values: Dict[str, Any], answers: Dict[str, Any], result: Dict[str,for name, answer in answers.items():confidence[name] = float(answer.get("confidence", 0.0))# Read through the module that owns the definition, so this and the `min_confidence` gate- # cannot disagree about what "calibrated" means.- answer_confidence[name] = calibrated_confidence(answer)+ # cannot disagree about which quantity is being reported.+ answer_confidence[name] = answer_confidence_value(answer)if answer.get("type") == "noul":p = float(answer.get("noul", 0.0))probabilities[name] = {"false": round(1.0 - p, 4), "true": round(p, 4)}diff --git a/tests/test_structured.py b/tests/test_structured.pyindex b51b2a69..1ddbd713 100644--- a/tests/test_structured.py+++ b/tests/test_structured.py@@ -202,43 +202,50 @@ check_true("decide/details answers flag set", det.answers["urgency"].get("low_cocheck("decide/details high conf value kept", det.values["department"], "billing")check_true("decide/details high conf flag unset", det.answers["department"].get("low_confidence") is not True)-# --------------------------------------------------------------------- calibrated confidence+# --------------------------------------------------------------------- answer_confidence# The decision: "which of my structured decisions are safe to automate?" is answered by filtering-# the details artifact, so the number that filter reads has to be the calibrated one.+# the details artifact, so the number that filter reads has to be the one the gate uses.#-# `docs/structured.md` calls this bridge one that returns "typed values with calibrated-# confidence". `DecisionResult.confidence` is built from the answer's `confidence`, and-# `laya/common.py:confidence_from_probs` documents that field as "not calibrated: it is not what-# temperature scaling fits and not what the reported ECE measures. See `answer_confidence`".-# Laya's own gate is defined against `answer_confidence` (#361/#456), and #394 is an open issue+# `DecisionResult.confidence` is built from the answer's `confidence` field, which is normalized+# entropy: `laya/common.py` calls it "not calibrated: it is not what temperature scaling fits and+# not what the reported ECE measures", `tests/test_confidence.py` pins that a two-option+# distribution reads 0.90 on a `noul` and 0.53 on an equivalent `choice`, and #394 is an open issue# saying a confidence threshold does not transfer across option counts -- which is the entropy-# definition's failure mode specifically.+# definition's failure mode, since `log(k)` is in the denominator.#-# So a caller filtering `details.confidence` to decide what to escalate filters on a quantity the-# repo documents as neither temperature-scaled nor ECE-measured, under the name of the one that is.+# So a caller filtering `details.confidence` to decide what to escalate filters on a different+# quantity from the one `min_confidence` compares against, and on one that moves with the shape of+# the question rather than with how right the answer is.+#+# Terminology, deliberately careful: `answer_confidence` is `max(p)`, the quantity calibration+# fits and every calibration figure is computed on, and the quantity the gate is defined against.+# That makes it the right number to *filter* on. It is NOT a claim that the number is right --+# "about c of the answers returned at c are correct" holds only after temperatures are fitted and+# validated for that checkpoint and question shape, and the shipped checkpoints are over-confident+# as shipped (README, Calibration).CAL = {"department": {"type": "choice", "choice": "billing", "confidence": 0.30, "answer_confidence": 0.95},"urgency": {"type": "score", "score": 2.0, "confidence": 0.90, "answer_confidence": 0.40,"probabilities": {"0": 0.1, "1": 0.5, "2": 0.4}, "legend": {}}}cal_det = decide(FakeRunner(CAL), "s", schema=SCHEMA, return_details=True)# The two numbers differ on the same field, which is the whole point: the entropy value ranks-# `urgency` above `department` and the calibrated one reverses it.+# `urgency` above `department` and max(p) reverses it.check_true("details/the two confidences really do disagree",cal_det.confidence["department"] < cal_det.confidence["urgency"])-check_true("details/…and the calibrated one reverses that order",+check_true("details/…and answer_confidence reverses that order",cal_det.answer_confidence["department"] > cal_det.answer_confidence["urgency"])-check("details/calibrated is reported per field", cal_det.answer_confidence["department"], 0.95)+check("details/answer_confidence is reported per field", cal_det.answer_confidence["department"], 0.95)-# The name means what it says, so a caller that sorts by it sorts by the quantity Laya's-# calibration figures describe.+# The name means what it says, so a caller that sorts by it sorts on the same quantity the gate+# and the eval harness use.check("details/entropy field is left exactly as it was", cal_det.confidence["department"], 0.30)-# A field with no usable confidence is `None`, not 0.0. `confidence` defaults to 0.0, so today an-# absent confidence and a genuinely zero one are the same value, and a caller filtering on "below-# 0.4, escalate" escalates both without being able to tell why.+# A field with no usable answer_confidence is `None`, not 0.0. `confidence` defaults to 0.0, so+# today an absent confidence and a genuinely zero one are the same value, and a caller filtering on+# "below 0.4, escalate" escalates both without being able to tell why.NO_CONF = {"department": {"type": "choice", "choice": "billing", "confidence": 0.0}}no_det = decide(FakeRunner(NO_CONF), "s", schema=SCHEMA, return_details=True)-check("details/absent calibrated confidence is None, not 0.0", no_det.answer_confidence["department"], None)+check("details/absent answer_confidence is None, not 0.0", no_det.answer_confidence["department"], None)check_true("details/…and stays distinct from a reported zero",no_det.answer_confidence["department"] is not cal_det.answer_confidence["urgency"])