Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(calibrate): report a temperature fit the clamp moved (#1245)

Nandakishor389f30c0dfaa
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.md
index 5424e6c0..90d0ce39 100644
--- a/docs/questions-and-answers.md
+++ b/docs/questions-and-answers.md
@@ -80,10 +80,13 @@ exactly the keys `false` and `true`, and the polarity is unchanged: `noul` is st
"labels": {"false": "automatic", "true": "escalate"}}
```
-Labels are not a way to fix a question the model gets wrong. `noul` follows its own option
-labels, most strongly on the English checkpoint, so a label pair that reads as a decision
-("approve" / "reject") can pull the answer toward the label rather than the state. Validate any
-relabelling on your own data before relying on it.
+Labels move the answer, in both directions. `noul` follows its own option labels, most strongly
+on the English checkpoint, so a label pair that reads as a decision ("approve" / "reject") can
+pull the answer toward the label rather than the state. That same pull is the one workaround for
+the wordings in [#1244](https://github.com/NandhaKishorM/laya/issues/1244) that answer without
+reading the state at all: naming the two options restores the question, where rewording it does
+not reliably. So labels are worth reaching for when an answer ignores the state, and worth
+suspecting when an answer tracks the label. Validate any relabelling on your own data either way.
### `score` — an ordered level
@@ -236,6 +239,10 @@ That mirrors what the library does for a `choice` question written as a list of
return the cancellation label for a state that says *not* to cancel, with high confidence, on
both checkpoints. See [#377](https://github.com/NandhaKishorM/laya/issues/377).
- **`noul` can follow its labels rather than the state**, so validate any relabelling.
+- **A `noul` instruction asking whether something is "positive" can answer no for every state**,
+ at confidence 1.0, on both PyTorch checkpoints. Setting `labels` to the two words restores it;
+ capitalising the word helps in some wordings and not in others. See
+ [#1244](https://github.com/NandhaKishorM/laya/issues/1244).
- **More options is not free.** Past roughly 20 the model degrades quickly; use the shortlist
helper to reduce a large label space before asking, or split it into a coarse and a fine
question.
diff --git a/laya/calibrate.py b/laya/calibrate.py
index 3607b347..fd7e9d1d 100644
--- a/laya/calibrate.py
+++ b/laya/calibrate.py
@@ -14,7 +14,9 @@ Two sample floors, on purpose:
Fitted temperatures go through `clamp_temperature` and therefore `[TEMP_MIN, TEMP_MAX]`
from `laya.common`, the same guard checkpoint load uses. There is no second pair of
-bounds in this module.
+bounds in this module. A clamp that moved a fit is reported in `fit_temperature_map`'s
+`temperature_clamped`, since the clamped value alone cannot say whether the optimum was
+the bound or merely beyond it.
"""
import warnings
from numbers import Integral
@@ -24,6 +26,7 @@ import numpy as np
import torch
from .common import (
+ QTYPE_NAMES,
QTYPES,
TEMP_MAX,
TEMP_MIN,
@@ -130,19 +133,15 @@ def _pairs_by_width(pairs: Sequence) -> List[Tuple[torch.Tensor, torch.Tensor]]:
torch.from_numpy(np.stack([t for _t, t in groups[w]]))) for w in sorted(groups)]
-def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:
- """Fit one scalar T by NLL + LBFGS on log T.
+def _fit_unclamped_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:
+ """The optimised scale before `clamp_temperature`, for callers that report the clamp.
- The result is `clamp_temperature` of the optimised scale, so it lies in
- `[TEMP_MIN, TEMP_MAX]` (or is the neutral 1.0 when the value is not a number).
- The search is bounded to `LOG_T_MARGIN` e-folds outside that band, which covers any
- minimum the answer can use; past it the float32 scale drifts toward `exp()` underflow,
- where the objective divides by zero and every gradient becomes NaN, and
- `clamp_temperature` reads that NaN back as the neutral 1.0.
- Returns 1.0 when fewer than `min_n` pairs are given. `min_n` defaults to
- `MIN_BUCKET_N` (the per-bucket floor). Type-level fits pass `MIN_TYPE_N`, which
- is lower, so a dataset that fills no bucket still gets a scalar instead of
- staying at 1.0.
+ `fit_one_temperature` is this value clamped, and that is the value anything installs.
+ Only the clamp reporting in `_fit_groups` reads the raw scale, so that a bucket whose
+ optimum sits outside `[TEMP_MIN, TEMP_MAX]` can be told apart from one whose optimum
+ really is a bound. The search band is `LOG_T_MARGIN` e-folds wider than the answer
+ band, so a raw value can be far outside it; a NaN from the underflow case described in
+ `fit_one_temperature` reaches the caller as a NaN rather than as 1.0.
"""
if min_n is None:
min_n = MIN_BUCKET_N
@@ -176,8 +175,28 @@ def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:
return loss
opt.step(closure)
- fitted = float(log_t.detach().clamp(min=log_lo, max=log_hi).exp().item())
- return clamp_temperature(fitted, TEMP_MIN, TEMP_MAX)
+ return float(log_t.detach().clamp(min=log_lo, max=log_hi).exp().item())
+
+
+def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:
+ """Fit one scalar T by NLL + LBFGS on log T.
+
+ The result is `clamp_temperature` of the optimised scale, so it lies in
+ `[TEMP_MIN, TEMP_MAX]` (or is the neutral 1.0 when the value is not a number).
+ The search is bounded to `LOG_T_MARGIN` e-folds outside that band, which covers any
+ minimum the answer can use; past it the float32 scale drifts toward `exp()` underflow,
+ where the objective divides by zero and every gradient becomes NaN, and
+ `clamp_temperature` reads that NaN back as the neutral 1.0.
+ Returns 1.0 when fewer than `min_n` pairs are given. `min_n` defaults to
+ `MIN_BUCKET_N` (the per-bucket floor). Type-level fits pass `MIN_TYPE_N`, which
+ is lower, so a dataset that fills no bucket still gets a scalar instead of
+ staying at 1.0.
+
+ A clamped result is indistinguishable from an optimum that really is a bound, because
+ both are the bound. `fit_temperature_map`'s `temperature_clamped` is where the
+ difference is reported; it carries the unclamped scale beside the value installed.
+ """
+ return clamp_temperature(_fit_unclamped_temperature(pairs, min_n), TEMP_MIN, TEMP_MAX)
def _iter_records(records: Iterable) -> List[Tuple[int, np.ndarray, np.ndarray, int]]:
@@ -273,20 +292,40 @@ def _ece_split(recs, seed):
return fit_recs, eval_recs, excluded
+def _clamp_entry(raw: float, applied: float) -> Optional[Dict[str, Any]]:
+ """Describe a fit the clamp moved, or None when it left the fit alone.
+
+ A NaN raw scale is the underflow case, not a clamp: `clamp_temperature` answers the
+ neutral 1.0 for it, which is a different fact and is not reported here.
+ """
+ if not np.isfinite(raw) or TEMP_MIN <= raw <= TEMP_MAX:
+ return None
+ return {"fitted": raw, "applied": applied, "bound": "min" if raw < TEMP_MIN else "max"}
+
+
def _fit_groups(by_type, by_bucket):
temperature = [1.0] * N_QTYPES
+ clamped: Dict[str, Dict[str, Any]] = {}
for qt in range(N_QTYPES):
pairs = by_type[qt]
# MIN_TYPE_N, not MIN_BUCKET_N: the scalar has to exist when every bucket is small.
if len(pairs) >= MIN_TYPE_N:
- temperature[qt] = fit_one_temperature(pairs, min_n=MIN_TYPE_N)
+ raw = _fit_unclamped_temperature(pairs, min_n=MIN_TYPE_N)
+ temperature[qt] = clamp_temperature(raw, TEMP_MIN, TEMP_MAX)
+ entry = _clamp_entry(raw, temperature[qt])
+ if entry is not None:
+ clamped[QTYPE_NAMES[qt]] = entry
temperature_by_options = {}
for key, pairs in by_bucket.items():
if len(pairs) < MIN_BUCKET_N:
continue
- temperature_by_options[key] = fit_one_temperature(pairs)
- return temperature, temperature_by_options
+ raw = _fit_unclamped_temperature(pairs)
+ temperature_by_options[key] = clamp_temperature(raw, TEMP_MIN, TEMP_MAX)
+ entry = _clamp_entry(raw, temperature_by_options[key])
+ if entry is not None:
+ clamped[key] = entry
+ return temperature, temperature_by_options, clamped
def fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: int = 0) -> Dict[str, Any]:
@@ -307,6 +346,16 @@ def fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: int
value is a `ValueError` naming `seed` on either path, not numpy's cast error on one path
and silence on the other. `seed=None` stays numpy's entropy seed: accepted, and the split
is then not reproducible.
+ `temperature_clamped` names every fit the clamp moved, on either path: a bucket key as
+ `temperature_by_options` spells it, or a question-type name for one of the `temperature`
+ scalars. Each entry carries `fitted` (the unclamped optimum), `applied` (the value in
+ this result) and `bound`, `"min"` or `"max"`. It is empty when nothing was clamped, and
+ it reports a fact a caller cannot otherwise recover: a value of exactly `TEMP_MIN` or
+ `TEMP_MAX` is what both a clamped fit and a genuine fit on the bound look like, so
+ without this an optimum of 0.387 installs as 0.5 and reads as a finished fit. The
+ clamped value is still the one to install, because `load_calibration` applies the same
+ bounds at checkpoint load; what the entry adds is that the data wanted to go further.
+
`report["n"]` is the number of records passed in; `report["n_eval"]` is the held-out
count the ECE rests on. A bucket that would drop below `MIN_BUCKET_N` after the
holdout is fit on all of its records, left out of the eval set, and named in
@@ -328,11 +377,12 @@ def fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: int
excluded = []
by_type, by_bucket = all_by_type, all_buckets
- temperature, temperature_by_options = _fit_groups(by_type, by_bucket)
+ temperature, temperature_by_options, clamped = _fit_groups(by_type, by_bucket)
out: Dict[str, Any] = {
"temperature": temperature,
"temperature_by_options": temperature_by_options,
"n_by_bucket": n_by_bucket,
+ "temperature_clamped": clamped,
}
if compute_ece:
report = _ece_report(eval_recs, temperature, temperature_by_options)
diff --git a/tests/test_calibrate.py b/tests/test_calibrate.py
index ae6ca20d..4be9071b 100644
--- a/tests/test_calibrate.py
+++ b/tests/test_calibrate.py
@@ -27,8 +27,10 @@ from laya.calibrate import ( # noqa: E402
MIN_BUCKET_N,
MIN_TYPE_N,
MIN_BINNING_BUCKET_N,
+ _clamp_entry,
_ece_report,
_ece_split,
+ _fit_unclamped_temperature,
_iter_records,
_pairs_by_width,
_softmax,
@@ -41,7 +43,7 @@ from laya.calibrate import ( # noqa: E402
fit_temperatures,
records_from_labeled,
)
-from laya.common import QTYPES, TEMP_MAX, TEMP_MIN, ece_score, temp_bucket # noqa: E402
+from laya.common import QTYPES, TEMP_MAX, TEMP_MIN, clamp_temperature, ece_score, temp_bucket # noqa: E402
import laya.calibrate as _calibrate # noqa: E402
import laya.agent as _agent_module # noqa: E402
@@ -297,6 +299,60 @@ check_true("weight/doubling a width pulls the fit to that width",
"even=%r heavy=%r wide=%r" % (t_even, t_heavy, t_wide))
+# ------------------------------------------------- the clamp is reported, not only applied
+# A clamped fit and a fit whose optimum really is the bound are the same number, so the
+# number cannot say which happened. `temperature_clamped` carries the unclamped scale
+# beside it. The value to install is still the clamped one, so these checks also pin that
+# the report changes nothing about `temperature` or `temperature_by_options`.
+lo_recs = [(QTYPES["choice"], z, t, 2) for z, t in lo_pairs]
+hi_recs = [(QTYPES["choice"], z, t, 2) for z, t in hi_pairs]
+mid_recs = [(QTYPES["choice"], z, t, len(z)) for z, t in mid]
+lo_map, hi_map, mid_map = (fit_temperature_map(r) for r in (lo_recs, hi_recs, mid_recs))
+
+check_true("clamp-report/the key is always present",
+ all("temperature_clamped" in m for m in (lo_map, hi_map, mid_map, med, fitted)))
+check("clamp-report/an interior fit is not reported", mid_map["temperature_clamped"], {})
+check_true("clamp-report/interior fit really was interior",
+ TEMP_MIN < mid_map["temperature"][QTYPES["choice"]] < TEMP_MAX,
+ "T=%r" % mid_map["temperature"][QTYPES["choice"]])
+
+for _name, _m, _bound, _applied in (("low", lo_map, "min", TEMP_MIN), ("high", hi_map, "max", TEMP_MAX)):
+ _rep = _m["temperature_clamped"]
+ # Both the bucket and the type scalar are fit from the same rows here, so both are named,
+ # the bucket by its `temperature_by_options` key and the type by its question-type name.
+ check("clamp-report/%s names the bucket and the type" % _name,
+ sorted(_rep), ["choice", "choice:2"])
+ check("clamp-report/%s bound" % _name, sorted({e["bound"] for e in _rep.values()}), [_bound])
+ check("clamp-report/%s applied is the clamped value" % _name,
+ sorted({e["applied"] for e in _rep.values()}), [_applied])
+ check("clamp-report/%s applied is what the result installs" % _name,
+ _rep.get("choice:2", {}).get("applied"), _m["temperature_by_options"]["choice:2"])
+ check_true("clamp-report/%s fitted is outside the band" % _name,
+ all(not TEMP_MIN <= e["fitted"] <= TEMP_MAX for e in _rep.values()),
+ "%r" % _rep)
+ check_true("clamp-report/%s fitted is on the right side" % _name,
+ all((e["fitted"] < TEMP_MIN) is (_bound == "min") for e in _rep.values()),
+ "%r" % _rep)
+
+# The clamped value stays the public answer: the report is additive, not a new return path.
+check("clamp-report/fit_one_temperature is unchanged at the floor",
+ fit_one_temperature(lo_pairs), lo_map["temperature_by_options"]["choice:2"])
+check("clamp-report/fit_one_temperature is unchanged at the ceiling",
+ fit_one_temperature(hi_pairs), hi_map["temperature_by_options"]["choice:2"])
+check_true("clamp-report/the unclamped fit is the clamped one's source",
+ abs(clamp_temperature(_fit_unclamped_temperature(lo_pairs), TEMP_MIN, TEMP_MAX)
+ - fit_one_temperature(lo_pairs)) < 1e-12)
+# The underflow case is a NaN, which `clamp_temperature` reports as the neutral 1.0. That is a
+# different fact from a clamp and must not be dressed up as one.
+check("clamp-report/a NaN fit is not a clamp", _clamp_entry(float("nan"), 1.0), None)
+check("clamp-report/an infinite fit is not a clamp", _clamp_entry(float("inf"), TEMP_MAX), None)
+check("clamp-report/a fit on the bound itself is not a clamp", _clamp_entry(TEMP_MIN, TEMP_MIN), None)
+# The search bound lives with the search now; the prose in `fit_one_temperature` must not be
+# what satisfies this.
+check_true("clamp-report/the search bound is in the fitting function",
+ "LOG_T_MARGIN" in inspect.getsource(_fit_unclamped_temperature))
+
+
# --------------------------------------------------------------- ECE on a held-out split, not the fit rows
report = fitted["report"]
check_true("ece/after < before", report["ece_after"] < report["ece_before"],