Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(evals): the calibration column must name its fallback, and the shipped level is not calibrated
Aashish1241bfb2d0f9
diff --git a/docs/evals.md b/docs/evals.mdindex 8a65d0ec..4174d79c 100644--- a/docs/evals.md+++ b/docs/evals.md@@ -131,14 +131,39 @@ Each metric is computed per answer where it applies and aggregated over the data| `noul_accuracy` | `noul` | fraction whose boolean (probability >= 0.5) matches || `score_mae` | `score` | mean absolute error || `score_within_<tol>` | `score` | fraction within an absolute tolerance |-| `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on `answer["answer_confidence"]`, the calibrated probability Laya reports on every answer type |+| `ece` | any answer with a confidence | expected calibration error, 15 bins, computed on the column `laya.evals._answer_confidence` reads -- `answer["answer_confidence"]` where the answer carries it, and a fallback where it does not; see [which confidence a metric reads](#which-confidence-a-metric-reads) || `brier` | any answer with a confidence and a known label | Brier score of confidence as P(correct), `mean((confidence - correct)**2)`; lower is better || `aurc` | any answer with a confidence and a known label | area under the risk--coverage curve: one risk value per distinct confidence level, each weighted by the answers that level spans; lower is better, and rewards a confidence that *ranks* right from wrong rather than just being calibrated || `selective_accuracy@50`, `selective_accuracy@80` | any answer with a confidence and a known label | accuracy over the answers a confidence threshold at the 50% / 80% coverage point accepts -- what abstaining on the least-confident tail buys. A threshold cannot split a group of equal confidences, so this can cover more than the named fraction; see [coverage cuts](#coverage-cuts-and-ties) |-| `mean_confidence` | any answer with a confidence | mean reported `answer["answer_confidence"]` |+| `mean_confidence` | any answer with a confidence | mean of the same column -- `answer["answer_confidence"]` where the answer carries it || `latency_p50_ms`, `latency_p95_ms` | per request | wall time each request waited, informational -- see [batching](#batching-and-timing) || `cost_per_decision_p50_ms`, `cost_per_decision_p95_ms` | per decision | a call's wall time divided by the rows it carried, informational |+### Which confidence a metric reads++Every row above that takes a confidence (`ece`, `brier`, `aurc`, both `selective_accuracy@*`,+`mean_confidence`) gets its column from `laya.evals._answer_confidence`, which prefers+`answer["answer_confidence"]` -- the probability of the answer being reported, the quantity temperature+scaling fits. It is not a claim that the number is right as shipped: both base checkpoints are+over-confident and `laya-multilingual` ships no fitted temperatures at all -- see the README's+[Calibration](https://github.com/NandhaKishorM/laya#calibration) -- which is what `ece` measures+rather than assumes.++An answer that carries no `answer_confidence` falls through, in order, to `confidence`, then+`max(p, 1 - p)` for a `noul`, then `max(probabilities)`. Those are different quantities. On `choice`+and `score` the `confidence` field is normalized entropy, which moves with the option count (#394)+rather than with how right the answer is; `max(probabilities)` is the mass on the top option, equal to+the reported answer's probability only when the reported answer is the argmax.++The shape that actually arrives without the field is the strict Jev wire contract: `LAYA_JEV_STRICT`+drops `answer_confidence` from every answer (`laya/serve.py::_project_jev_strict`, and the flag's row+in [the HTTP API page](http-api.md)), so a report run over recorded strict responses calibrates the+entropy number. Measured on three labelled rows -- `choice` carrying `answer_confidence` 0.95 against+an entropy `confidence` of 0.7887, `score` 0.90 against 0.6410, `noul` 0.87 -- the same dataset scores+`mean_confidence` 0.9067 and `ece` 0.0900 over the full payloads and 0.7666 and 0.1707 over the strict+projection. Two reports are comparable only when their answers carry the same field;+`tests/test_evals.py` pins both paths so a change to that fallback order has to be made deliberately.+### Coverage cuts and tiesBoth coverage metrics cut on a confidence **threshold**, and a threshold accepts every answer at itsdiff --git a/laya/evals.py b/laya/evals.pyindex f3c0c0e8..ab622ab2 100644--- a/laya/evals.py+++ b/laya/evals.py@@ -323,11 +323,27 @@ def default_evaluators() -> List[Evaluator]:def _answer_confidence(answer: Dict[str, Any]) -> Optional[float]:- """The calibrated confidence Laya reports: `answer_confidence`, not the entropy score.-- `answer["confidence"]` is entropy-based for choice and score, so calibration metrics must use- `answer_confidence`, which Laya reports on every answer type. The other keys are fallbacks for- a stripped-down result.+ """The column the calibration metrics read: `answer_confidence` when the answer carries one.++ `answer_confidence` is the probability of the answer being reported -- the quantity temperature+ scaling fits and the quantity this repository's calibration figures are computed on. It is not+ *calibrated* as shipped: both base checkpoints are over-confident and `laya-multilingual` ships no+ fitted temperatures at all (README, Calibration), which is why `ece` measures this column instead+ of assuming it.++ An answer carrying no `answer_confidence` falls through three more reads, in order: the+ `confidence` field, then `max(p, 1 - p)` for a `noul`, then `max(probabilities)`. Those are not+ the same quantity, and the first is the one this function otherwise exists to avoid: `confidence`+ is normalized entropy on `choice` and `score`, which moves with the option count (#394) rather+ than with how right the answer is, and `laya.confidence.answer_confidence_value` refuses to fall+ back to it for the abstention gate on exactly that reasoning. `max(probabilities)` is the mass on+ the top option, which is the reported answer's probability only when the answer *is* the argmax.++ The stripped-down shape is not hypothetical. `LAYA_JEV_STRICT` projects the served response onto+ the Jev wire contract, which carries no `answer_confidence` (`laya/serve.py`, and the flag's row+ in docs/http-api.md), so a report run over recorded strict responses calibrates the entropy+ number and is not comparable to one run over full payloads. `tests/test_evals.py` pins both paths+ and the gap between them; `docs/evals.md` says so where an operator reads the metrics."""confidence = answer.get("answer_confidence")if isinstance(confidence, (int, float)):