Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(calibrate): report a temperature fit the clamp moved (#1245)
Nandakishor389f30c0dfaa
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.mdindex 5424e6c0..90d0ce39 100644--- a/docs/questions-and-answers.md+++ b/docs/questions-and-answers.md@@ -80,10 +80,13 @@ exactly the keys `false` and `true`, and the polarity is unchanged: `noul` is st"labels": {"false": "automatic", "true": "escalate"}}```-Labels are not a way to fix a question the model gets wrong. `noul` follows its own option-labels, most strongly on the English checkpoint, so a label pair that reads as a decision-("approve" / "reject") can pull the answer toward the label rather than the state. Validate any-relabelling on your own data before relying on it.+Labels move the answer, in both directions. `noul` follows its own option labels, most strongly+on the English checkpoint, so a label pair that reads as a decision ("approve" / "reject") can+pull the answer toward the label rather than the state. That same pull is the one workaround for+the wordings in [#1244](https://github.com/NandhaKishorM/laya/issues/1244) that answer without+reading the state at all: naming the two options restores the question, where rewording it does+not reliably. So labels are worth reaching for when an answer ignores the state, and worth+suspecting when an answer tracks the label. Validate any relabelling on your own data either way.### `score` — an ordered level@@ -236,6 +239,10 @@ That mirrors what the library does for a `choice` question written as a list ofreturn the cancellation label for a state that says *not* to cancel, with high confidence, onboth checkpoints. See [#377](https://github.com/NandhaKishorM/laya/issues/377).- **`noul` can follow its labels rather than the state**, so validate any relabelling.+- **A `noul` instruction asking whether something is "positive" can answer no for every state**,+ at confidence 1.0, on both PyTorch checkpoints. Setting `labels` to the two words restores it;+ capitalising the word helps in some wordings and not in others. See+ [#1244](https://github.com/NandhaKishorM/laya/issues/1244).- **More options is not free.** Past roughly 20 the model degrades quickly; use the shortlisthelper to reduce a large label space before asking, or split it into a coarse and a finequestion.diff --git a/laya/calibrate.py b/laya/calibrate.pyindex 3607b347..fd7e9d1d 100644--- a/laya/calibrate.py+++ b/laya/calibrate.py@@ -14,7 +14,9 @@ Two sample floors, on purpose:Fitted temperatures go through `clamp_temperature` and therefore `[TEMP_MIN, TEMP_MAX]`from `laya.common`, the same guard checkpoint load uses. There is no second pair of-bounds in this module.+bounds in this module. A clamp that moved a fit is reported in `fit_temperature_map`'s+`temperature_clamped`, since the clamped value alone cannot say whether the optimum was+the bound or merely beyond it."""import warningsfrom numbers import Integral@@ -24,6 +26,7 @@ import numpy as npimport torchfrom .common import (+ QTYPE_NAMES,QTYPES,TEMP_MAX,TEMP_MIN,@@ -130,19 +133,15 @@ def _pairs_by_width(pairs: Sequence) -> List[Tuple[torch.Tensor, torch.Tensor]]:torch.from_numpy(np.stack([t for _t, t in groups[w]]))) for w in sorted(groups)]-def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:- """Fit one scalar T by NLL + LBFGS on log T.+def _fit_unclamped_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:+ """The optimised scale before `clamp_temperature`, for callers that report the clamp.- The result is `clamp_temperature` of the optimised scale, so it lies in- `[TEMP_MIN, TEMP_MAX]` (or is the neutral 1.0 when the value is not a number).- The search is bounded to `LOG_T_MARGIN` e-folds outside that band, which covers any- minimum the answer can use; past it the float32 scale drifts toward `exp()` underflow,- where the objective divides by zero and every gradient becomes NaN, and- `clamp_temperature` reads that NaN back as the neutral 1.0.- Returns 1.0 when fewer than `min_n` pairs are given. `min_n` defaults to- `MIN_BUCKET_N` (the per-bucket floor). Type-level fits pass `MIN_TYPE_N`, which- is lower, so a dataset that fills no bucket still gets a scalar instead of- staying at 1.0.+ `fit_one_temperature` is this value clamped, and that is the value anything installs.+ Only the clamp reporting in `_fit_groups` reads the raw scale, so that a bucket whose+ optimum sits outside `[TEMP_MIN, TEMP_MAX]` can be told apart from one whose optimum+ really is a bound. The search band is `LOG_T_MARGIN` e-folds wider than the answer+ band, so a raw value can be far outside it; a NaN from the underflow case described in+ `fit_one_temperature` reaches the caller as a NaN rather than as 1.0."""if min_n is None:min_n = MIN_BUCKET_N@@ -176,8 +175,28 @@ def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:return lossopt.step(closure)- fitted = float(log_t.detach().clamp(min=log_lo, max=log_hi).exp().item())- return clamp_temperature(fitted, TEMP_MIN, TEMP_MAX)+ return float(log_t.detach().clamp(min=log_lo, max=log_hi).exp().item())+++def fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float:+ """Fit one scalar T by NLL + LBFGS on log T.++ The result is `clamp_temperature` of the optimised scale, so it lies in+ `[TEMP_MIN, TEMP_MAX]` (or is the neutral 1.0 when the value is not a number).+ The search is bounded to `LOG_T_MARGIN` e-folds outside that band, which covers any+ minimum the answer can use; past it the float32 scale drifts toward `exp()` underflow,+ where the objective divides by zero and every gradient becomes NaN, and+ `clamp_temperature` reads that NaN back as the neutral 1.0.+ Returns 1.0 when fewer than `min_n` pairs are given. `min_n` defaults to+ `MIN_BUCKET_N` (the per-bucket floor). Type-level fits pass `MIN_TYPE_N`, which+ is lower, so a dataset that fills no bucket still gets a scalar instead of+ staying at 1.0.++ A clamped result is indistinguishable from an optimum that really is a bound, because+ both are the bound. `fit_temperature_map`'s `temperature_clamped` is where the+ difference is reported; it carries the unclamped scale beside the value installed.+ """+ return clamp_temperature(_fit_unclamped_temperature(pairs, min_n), TEMP_MIN, TEMP_MAX)def _iter_records(records: Iterable) -> List[Tuple[int, np.ndarray, np.ndarray, int]]:@@ -273,20 +292,40 @@ def _ece_split(recs, seed):return fit_recs, eval_recs, excluded+def _clamp_entry(raw: float, applied: float) -> Optional[Dict[str, Any]]:+ """Describe a fit the clamp moved, or None when it left the fit alone.++ A NaN raw scale is the underflow case, not a clamp: `clamp_temperature` answers the+ neutral 1.0 for it, which is a different fact and is not reported here.+ """+ if not np.isfinite(raw) or TEMP_MIN <= raw <= TEMP_MAX:+ return None+ return {"fitted": raw, "applied": applied, "bound": "min" if raw < TEMP_MIN else "max"}++def _fit_groups(by_type, by_bucket):temperature = [1.0] * N_QTYPES+ clamped: Dict[str, Dict[str, Any]] = {}for qt in range(N_QTYPES):pairs = by_type[qt]# MIN_TYPE_N, not MIN_BUCKET_N: the scalar has to exist when every bucket is small.if len(pairs) >= MIN_TYPE_N:- temperature[qt] = fit_one_temperature(pairs, min_n=MIN_TYPE_N)+ raw = _fit_unclamped_temperature(pairs, min_n=MIN_TYPE_N)+ temperature[qt] = clamp_temperature(raw, TEMP_MIN, TEMP_MAX)+ entry = _clamp_entry(raw, temperature[qt])+ if entry is not None:+ clamped[QTYPE_NAMES[qt]] = entrytemperature_by_options = {}for key, pairs in by_bucket.items():if len(pairs) < MIN_BUCKET_N:continue- temperature_by_options[key] = fit_one_temperature(pairs)- return temperature, temperature_by_options+ raw = _fit_unclamped_temperature(pairs)+ temperature_by_options[key] = clamp_temperature(raw, TEMP_MIN, TEMP_MAX)+ entry = _clamp_entry(raw, temperature_by_options[key])+ if entry is not None:+ clamped[key] = entry+ return temperature, temperature_by_options, clampeddef fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: int = 0) -> Dict[str, Any]:@@ -307,6 +346,16 @@ def fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: intvalue is a `ValueError` naming `seed` on either path, not numpy's cast error on one pathand silence on the other. `seed=None` stays numpy's entropy seed: accepted, and the splitis then not reproducible.+ `temperature_clamped` names every fit the clamp moved, on either path: a bucket key as+ `temperature_by_options` spells it, or a question-type name for one of the `temperature`+ scalars. Each entry carries `fitted` (the unclamped optimum), `applied` (the value in+ this result) and `bound`, `"min"` or `"max"`. It is empty when nothing was clamped, and+ it reports a fact a caller cannot otherwise recover: a value of exactly `TEMP_MIN` or+ `TEMP_MAX` is what both a clamped fit and a genuine fit on the bound look like, so+ without this an optimum of 0.387 installs as 0.5 and reads as a finished fit. The+ clamped value is still the one to install, because `load_calibration` applies the same+ bounds at checkpoint load; what the entry adds is that the data wanted to go further.+`report["n"]` is the number of records passed in; `report["n_eval"]` is the held-outcount the ECE rests on. A bucket that would drop below `MIN_BUCKET_N` after theholdout is fit on all of its records, left out of the eval set, and named in@@ -328,11 +377,12 @@ def fit_temperature_map(records: Iterable, compute_ece: bool = False, seed: intexcluded = []by_type, by_bucket = all_by_type, all_buckets- temperature, temperature_by_options = _fit_groups(by_type, by_bucket)+ temperature, temperature_by_options, clamped = _fit_groups(by_type, by_bucket)out: Dict[str, Any] = {"temperature": temperature,"temperature_by_options": temperature_by_options,"n_by_bucket": n_by_bucket,+ "temperature_clamped": clamped,}if compute_ece:report = _ece_report(eval_recs, temperature, temperature_by_options)diff --git a/tests/test_calibrate.py b/tests/test_calibrate.pyindex ae6ca20d..4be9071b 100644--- a/tests/test_calibrate.py+++ b/tests/test_calibrate.py@@ -27,8 +27,10 @@ from laya.calibrate import ( # noqa: E402MIN_BUCKET_N,MIN_TYPE_N,MIN_BINNING_BUCKET_N,+ _clamp_entry,_ece_report,_ece_split,+ _fit_unclamped_temperature,_iter_records,_pairs_by_width,_softmax,@@ -41,7 +43,7 @@ from laya.calibrate import ( # noqa: E402fit_temperatures,records_from_labeled,)-from laya.common import QTYPES, TEMP_MAX, TEMP_MIN, ece_score, temp_bucket # noqa: E402+from laya.common import QTYPES, TEMP_MAX, TEMP_MIN, clamp_temperature, ece_score, temp_bucket # noqa: E402import laya.calibrate as _calibrate # noqa: E402import laya.agent as _agent_module # noqa: E402@@ -297,6 +299,60 @@ check_true("weight/doubling a width pulls the fit to that width","even=%r heavy=%r wide=%r" % (t_even, t_heavy, t_wide))+# ------------------------------------------------- the clamp is reported, not only applied+# A clamped fit and a fit whose optimum really is the bound are the same number, so the+# number cannot say which happened. `temperature_clamped` carries the unclamped scale+# beside it. The value to install is still the clamped one, so these checks also pin that+# the report changes nothing about `temperature` or `temperature_by_options`.+lo_recs = [(QTYPES["choice"], z, t, 2) for z, t in lo_pairs]+hi_recs = [(QTYPES["choice"], z, t, 2) for z, t in hi_pairs]+mid_recs = [(QTYPES["choice"], z, t, len(z)) for z, t in mid]+lo_map, hi_map, mid_map = (fit_temperature_map(r) for r in (lo_recs, hi_recs, mid_recs))++check_true("clamp-report/the key is always present",+ all("temperature_clamped" in m for m in (lo_map, hi_map, mid_map, med, fitted)))+check("clamp-report/an interior fit is not reported", mid_map["temperature_clamped"], {})+check_true("clamp-report/interior fit really was interior",+ TEMP_MIN < mid_map["temperature"][QTYPES["choice"]] < TEMP_MAX,+ "T=%r" % mid_map["temperature"][QTYPES["choice"]])++for _name, _m, _bound, _applied in (("low", lo_map, "min", TEMP_MIN), ("high", hi_map, "max", TEMP_MAX)):+ _rep = _m["temperature_clamped"]+ # Both the bucket and the type scalar are fit from the same rows here, so both are named,+ # the bucket by its `temperature_by_options` key and the type by its question-type name.+ check("clamp-report/%s names the bucket and the type" % _name,+ sorted(_rep), ["choice", "choice:2"])+ check("clamp-report/%s bound" % _name, sorted({e["bound"] for e in _rep.values()}), [_bound])+ check("clamp-report/%s applied is the clamped value" % _name,+ sorted({e["applied"] for e in _rep.values()}), [_applied])+ check("clamp-report/%s applied is what the result installs" % _name,+ _rep.get("choice:2", {}).get("applied"), _m["temperature_by_options"]["choice:2"])+ check_true("clamp-report/%s fitted is outside the band" % _name,+ all(not TEMP_MIN <= e["fitted"] <= TEMP_MAX for e in _rep.values()),+ "%r" % _rep)+ check_true("clamp-report/%s fitted is on the right side" % _name,+ all((e["fitted"] < TEMP_MIN) is (_bound == "min") for e in _rep.values()),+ "%r" % _rep)++# The clamped value stays the public answer: the report is additive, not a new return path.+check("clamp-report/fit_one_temperature is unchanged at the floor",+ fit_one_temperature(lo_pairs), lo_map["temperature_by_options"]["choice:2"])+check("clamp-report/fit_one_temperature is unchanged at the ceiling",+ fit_one_temperature(hi_pairs), hi_map["temperature_by_options"]["choice:2"])+check_true("clamp-report/the unclamped fit is the clamped one's source",+ abs(clamp_temperature(_fit_unclamped_temperature(lo_pairs), TEMP_MIN, TEMP_MAX)+ - fit_one_temperature(lo_pairs)) < 1e-12)+# The underflow case is a NaN, which `clamp_temperature` reports as the neutral 1.0. That is a+# different fact from a clamp and must not be dressed up as one.+check("clamp-report/a NaN fit is not a clamp", _clamp_entry(float("nan"), 1.0), None)+check("clamp-report/an infinite fit is not a clamp", _clamp_entry(float("inf"), TEMP_MAX), None)+check("clamp-report/a fit on the bound itself is not a clamp", _clamp_entry(TEMP_MIN, TEMP_MIN), None)+# The search bound lives with the search now; the prose in `fit_one_temperature` must not be+# what satisfies this.+check_true("clamp-report/the search bound is in the fitting function",+ "LOG_T_MARGIN" in inspect.getsource(_fit_unclamped_temperature))++# --------------------------------------------------------------- ECE on a held-out split, not the fit rowsreport = fitted["report"]check_true("ece/after < before", report["ece_after"] < report["ece_before"],