Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(structured): stop calling answer_confidence "calibrated"

Bruce-Yiiacb2befe1388
diff --git a/docs/structured.md b/docs/structured.md
index 28ba0e84..c650fa2b 100644
--- a/docs/structured.md
+++ b/docs/structured.md
@@ -146,32 +146,40 @@ with per-field confidence, probabilities, the raw answers, and the usage and rou
```python
result = agent.decide(state, schema=Ticket, return_details=True)
-result.values["department"] # "billing"
-result.answer_confidence["department"] # 0.94 calibrated max(p): gate on this one
-result.confidence["department"] # 0.71 normalized entropy, which depends on label count
-result.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}
-result.usage # {"input_tokens": 42, "output_tokens": 0}
-result.routing # the Router decision, when a Router answered
+result.values["department"] # "billing"
+result.answer_confidence["department"] # 0.94 max(p): the quantity min_confidence gates on
+result.confidence["department"] # 0.71 normalized entropy, which depends on label count
+result.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}
+result.usage # {"input_tokens": 42, "output_tokens": 0}
+result.routing # the Router decision, when a Router answered
```
`confidence` and `answer_confidence` are different quantities, and the names follow the definitions.
-`answer_confidence` is the calibrated `max(p)`: it is what temperature scaling fits, what the
-reported ECE measures, and what `min_confidence` is defined against. `confidence` is normalized
-entropy, which depends on how many options the question had -- `tests/test_confidence.py` pins that
-a two-option distribution comes back as 0.90 on a `noul` and 0.53 on an equivalent `choice` -- so it
-does not compare against a threshold, and it is not what the calibration figures describe. A field
-that reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.
+`answer_confidence` is `max(p)`, the probability mass on the answer being reported. It is what
+temperature scaling fits, what every calibration figure in this repository is computed on, and what
+`min_confidence` is compared against — which is the reason to gate on it rather than on
+`confidence`. `confidence` is normalized entropy, which depends on how many options the question
+had: `tests/test_confidence.py` pins that a two-option distribution comes back as 0.90 on a `noul`
+and 0.53 on an equivalent `choice`, so it does not compare against a threshold. A field that
+reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.
-Gate on the calibrated one:
+Gate on the same quantity the gate uses:
```python
if result.answer_confidence["department"] < 0.6:
result.values["department"] = "human-review"
```
-Both numbers are reported because both are sometimes what you want: `confidence` answers "how
-concentrated is this distribution", `answer_confidence` answers "how much should I trust this
-answer", and only the second is comparable across question shapes.
+`answer_confidence` being the right number to *filter* on is not the same as it being a trustworthy
+probability. Reading it as "about c of the answers returned at c are correct" holds only after
+temperatures have been fitted and validated on held-out data for that checkpoint and question shape.
+The shipped checkpoints are over-confident as shipped and `laya-multilingual` ships with no fitted
+temperatures at all — see the README's
+[Calibration](https://github.com/NandhaKishorM/laya#calibration) and
+[Honest limits](https://github.com/NandhaKishorM/laya#honest-limits) sections, and the
+[fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb)
+for the fitting loop. Fit before you rely on the level; report it because it is the quantity the
+gate and the eval harness both use.
## How it maps internally
diff --git a/laya/confidence.py b/laya/confidence.py
index c96e5ff0..5a20657a 100644
--- a/laya/confidence.py
+++ b/laya/confidence.py
@@ -7,16 +7,25 @@ import math
from typing import Any, Dict, List, Optional
-def calibrated_confidence(answer: Dict[str, Any]) -> Optional[float]:
- """The calibrated `answer_confidence` of `answer`, or None if it did not report one.
+def answer_confidence_value(answer: Dict[str, Any]) -> Optional[float]:
+ """The `answer_confidence` of `answer`, or None if it did not report a usable one.
- `answer_confidence` is the calibrated max(p) quantity: it is what temperature scaling fits,
- what the reported ECE measures, and what the `min_confidence` gate is defined against (#361).
- `confidence`, by contrast, is normalized entropy on a scale that depends on the number of
- options -- `common.confidence_from_probs` calls it "not calibrated", and #394 is an open issue
- saying a confidence threshold does not transfer across option counts. So this deliberately
- does **not** fall back to `confidence`: reporting the entropy number under the calibrated name
- would be worse than reporting nothing, because the name is what a caller filters on.
+ `answer_confidence` is `max(p)` -- the probability mass on the answer being reported. It is
+ the quantity temperature scaling fits, the quantity every calibration figure in this
+ repository is computed on, and the quantity the `min_confidence` gate is defined against
+ (#361). That makes it the right number for a threshold a caller compares against; it is
+ not on its own a claim that the number is right. Reading it as "about c of the answers
+ returned at c are correct" holds only after temperatures have been fitted and validated on
+ held-out data for that checkpoint and question shape -- the shipped checkpoints are
+ over-confident as shipped, and `laya-multilingual` ships with no fitted temperatures at all
+ (README, Calibration). `common.answer_confidence` states the quantity; this states when it
+ carries that property.
+
+ `confidence` is a different quantity on a different scale -- normalized entropy, which
+ `common.confidence_from_probs` calls "not calibrated" and #394 says does not transfer across
+ option counts. So this deliberately does **not** fall back to it: reporting the entropy
+ number under the `answer_confidence` name would be worse than reporting nothing, because the
+ name is what a caller filters on.
A `bool` is not a confidence and a NaN is not a decision, so both give None rather than 0.0.
"""
@@ -29,10 +38,10 @@ def calibrated_confidence(answer: Dict[str, Any]) -> Optional[float]:
def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:
"""The number the abstention gate compares against `min_confidence`, or None if there is none.
- The calibrated `answer_confidence` first, falling back to the entropy `confidence` so an
- answer that carries only the older field is still gated rather than silently passed.
+ `answer_confidence` first, falling back to the entropy `confidence` so an answer that carries
+ only the older field is still gated rather than silently passed.
"""
- conf = calibrated_confidence(answer)
+ conf = answer_confidence_value(answer)
if conf is None:
conf = answer.get("confidence")
if isinstance(conf, (int, float)) and not isinstance(conf, bool) and math.isfinite(conf):
@@ -53,8 +62,9 @@ def check_min_confidence(v: Any) -> float:
def flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) -> None:
"""Opt-in abstention marker (#361): flag answers whose confidence falls below `min_confidence`.
- Reads `answer_confidence` (the calibrated max(p) confidence, invariant to label count k),
- falling back to `confidence` if `answer_confidence` is absent.
+ Reads `answer_confidence` (`max(p)`, the quantity the calibration figures describe and the one
+ that does not drift with the number of options), falling back to `confidence` if
+ `answer_confidence` is absent.
The raw answer and confidence stay intact; `low_confidence: True` is added.
"""
if min_confidence == 0.0:
diff --git a/laya/structured.py b/laya/structured.py
index 5f296d9d..d72547de 100644
--- a/laya/structured.py
+++ b/laya/structured.py
@@ -20,7 +20,7 @@ from collections.abc import Sequence as SequenceABC
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Sequence, Tuple
-from .confidence import calibrated_confidence, check_min_confidence, flag_low_confidence
+from .confidence import answer_confidence_value, check_min_confidence, flag_low_confidence
MAX_PROPERTIES = 32
MAX_OPTIONS = 32
@@ -38,12 +38,21 @@ class DecisionResult:
`values` is the schema-shaped output. `confidence` and `probabilities` are keyed by field,
and `answers` is Laya's raw answer per field.
- `answer_confidence` is the calibrated max(p) confidence per field -- the quantity temperature
- scaling fits, the reported ECE measures, and the `min_confidence` gate is defined against --
- under its own name. `confidence` keeps the normalized-entropy value it has always had, because
- that is not a calibrated quantity and a caller filtering this artifact to decide what to
- escalate has to be able to tell the two apart. A field that reported no usable calibrated
- confidence maps to `None`, which is distinct from a reported `0.0`.
+ `answer_confidence` is `max(p)` per field -- the probability mass on the answer being
+ reported, under its own name. That makes it the same decision quantity `min_confidence`
+ compares against and the calibration and eval stack measures, which is the reason to report
+ it: a caller filtering this artifact to decide what to automate has to be filtering on the
+ number the gate actually used.
+
+ It is not a claim that the number is right. Reading it as "about c of the answers returned at
+ c are correct" holds only after temperatures are fitted and validated on held-out data for
+ that checkpoint and question shape; the shipped checkpoints are over-confident as shipped and
+ `laya-multilingual` ships with no fitted temperatures at all. See `common.answer_confidence`
+ and the README's Calibration section.
+
+ `confidence` keeps the normalized-entropy value it has always had, because that is a
+ different quantity on a scale that depends on the label count. A field that reported no
+ usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.
"""
values: Dict[str, Any]
@@ -247,8 +256,8 @@ def _details(values: Dict[str, Any], answers: Dict[str, Any], result: Dict[str,
for name, answer in answers.items():
confidence[name] = float(answer.get("confidence", 0.0))
# Read through the module that owns the definition, so this and the `min_confidence` gate
- # cannot disagree about what "calibrated" means.
- answer_confidence[name] = calibrated_confidence(answer)
+ # cannot disagree about which quantity is being reported.
+ answer_confidence[name] = answer_confidence_value(answer)
if answer.get("type") == "noul":
p = float(answer.get("noul", 0.0))
probabilities[name] = {"false": round(1.0 - p, 4), "true": round(p, 4)}
diff --git a/tests/test_structured.py b/tests/test_structured.py
index b51b2a69..1ddbd713 100644
--- a/tests/test_structured.py
+++ b/tests/test_structured.py
@@ -202,43 +202,50 @@ check_true("decide/details answers flag set", det.answers["urgency"].get("low_co
check("decide/details high conf value kept", det.values["department"], "billing")
check_true("decide/details high conf flag unset", det.answers["department"].get("low_confidence") is not True)
-# --------------------------------------------------------------------- calibrated confidence
+# --------------------------------------------------------------------- answer_confidence
# The decision: "which of my structured decisions are safe to automate?" is answered by filtering
-# the details artifact, so the number that filter reads has to be the calibrated one.
+# the details artifact, so the number that filter reads has to be the one the gate uses.
#
-# `docs/structured.md` calls this bridge one that returns "typed values with calibrated
-# confidence". `DecisionResult.confidence` is built from the answer's `confidence`, and
-# `laya/common.py:confidence_from_probs` documents that field as "not calibrated: it is not what
-# temperature scaling fits and not what the reported ECE measures. See `answer_confidence`".
-# Laya's own gate is defined against `answer_confidence` (#361/#456), and #394 is an open issue
+# `DecisionResult.confidence` is built from the answer's `confidence` field, which is normalized
+# entropy: `laya/common.py` calls it "not calibrated: it is not what temperature scaling fits and
+# not what the reported ECE measures", `tests/test_confidence.py` pins that a two-option
+# distribution reads 0.90 on a `noul` and 0.53 on an equivalent `choice`, and #394 is an open issue
# saying a confidence threshold does not transfer across option counts -- which is the entropy
-# definition's failure mode specifically.
+# definition's failure mode, since `log(k)` is in the denominator.
#
-# So a caller filtering `details.confidence` to decide what to escalate filters on a quantity the
-# repo documents as neither temperature-scaled nor ECE-measured, under the name of the one that is.
+# So a caller filtering `details.confidence` to decide what to escalate filters on a different
+# quantity from the one `min_confidence` compares against, and on one that moves with the shape of
+# the question rather than with how right the answer is.
+#
+# Terminology, deliberately careful: `answer_confidence` is `max(p)`, the quantity calibration
+# fits and every calibration figure is computed on, and the quantity the gate is defined against.
+# That makes it the right number to *filter* on. It is NOT a claim that the number is right --
+# "about c of the answers returned at c are correct" holds only after temperatures are fitted and
+# validated for that checkpoint and question shape, and the shipped checkpoints are over-confident
+# as shipped (README, Calibration).
CAL = {"department": {"type": "choice", "choice": "billing", "confidence": 0.30, "answer_confidence": 0.95},
"urgency": {"type": "score", "score": 2.0, "confidence": 0.90, "answer_confidence": 0.40,
"probabilities": {"0": 0.1, "1": 0.5, "2": 0.4}, "legend": {}}}
cal_det = decide(FakeRunner(CAL), "s", schema=SCHEMA, return_details=True)
# The two numbers differ on the same field, which is the whole point: the entropy value ranks
-# `urgency` above `department` and the calibrated one reverses it.
+# `urgency` above `department` and max(p) reverses it.
check_true("details/the two confidences really do disagree",
cal_det.confidence["department"] < cal_det.confidence["urgency"])
-check_true("details/…and the calibrated one reverses that order",
+check_true("details/…and answer_confidence reverses that order",
cal_det.answer_confidence["department"] > cal_det.answer_confidence["urgency"])
-check("details/calibrated is reported per field", cal_det.answer_confidence["department"], 0.95)
+check("details/answer_confidence is reported per field", cal_det.answer_confidence["department"], 0.95)
-# The name means what it says, so a caller that sorts by it sorts by the quantity Laya's
-# calibration figures describe.
+# The name means what it says, so a caller that sorts by it sorts on the same quantity the gate
+# and the eval harness use.
check("details/entropy field is left exactly as it was", cal_det.confidence["department"], 0.30)
-# A field with no usable confidence is `None`, not 0.0. `confidence` defaults to 0.0, so today an
-# absent confidence and a genuinely zero one are the same value, and a caller filtering on "below
-# 0.4, escalate" escalates both without being able to tell why.
+# A field with no usable answer_confidence is `None`, not 0.0. `confidence` defaults to 0.0, so
+# today an absent confidence and a genuinely zero one are the same value, and a caller filtering on
+# "below 0.4, escalate" escalates both without being able to tell why.
NO_CONF = {"department": {"type": "choice", "choice": "billing", "confidence": 0.0}}
no_det = decide(FakeRunner(NO_CONF), "s", schema=SCHEMA, return_details=True)
-check("details/absent calibrated confidence is None, not 0.0", no_det.answer_confidence["department"], None)
+check("details/absent answer_confidence is None, not 0.0", no_det.answer_confidence["department"], None)
check_true("details/…and stays distinct from a reported zero",
no_det.answer_confidence["department"] is not cal_det.answer_confidence["urgency"])