Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): report Jev's confidence formula as x_jev_confidence (#302)
kmitulc932543ca501
diff --git a/docs/http-api.md b/docs/http-api.mdindex 8c2c9a6f..a93644ab 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -164,12 +164,12 @@ hide that. The detail is the same `unknown model` text core raises, plus the rem"queue": {"type": "choice", "choice": "billing","probabilities": {"billing": 0.9519, "tech": 0.0327, "other": 0.0154},"confidence": 0.797, "answer_confidence": 0.9519,- "action": {"act_probability": 1.0}},+ "action": {"act_probability": 1.0}, "x_jev_confidence": 0.9278},"urgency": {"type": "score", "score": 1.6994,"legend": {"0": "calm", "1": "firm", "2": "angry", "3": "furious"},"probabilities": {"0": 0.0249, "1": 0.4136, "2": 0.3985, "3": 0.1629},"confidence": 0.1925, "answer_confidence": 0.4136,- "action": {"act_probability": 1.0}}+ "action": {"act_probability": 1.0}, "x_jev_confidence": 0.2507}},"usage": {"input_tokens": 83, "output_tokens": 0, "state_tokens": 12,"state_tokens_dropped": 0, "truncated": false, "truncated_questions": []},@@ -187,8 +187,8 @@ name of the decision head, and the checkpoint that answered is in `routing`.| answer type | keys ||---|---|-| `choice` | `choice` (the argmax option), `probabilities` per option |-| `score` | `score` (expected level index, may fall between levels), `probabilities` keyed `"0".. "k-1"`, `legend` mapping index to the level text |+| `choice` | `choice` (the argmax option), `probabilities` per option, `x_jev_confidence` |+| `score` | `score` (expected level index, may fall between levels), `probabilities` keyed `"0".. "k-1"`, `legend` mapping index to the level text, `x_jev_confidence` || `noul` | `noul`, the probability of the yes option || all | `type` (the discriminator, echoing the question's own type), `confidence`, `answer_confidence`, and `action.act_probability` || gate | `abstention`, `abstention_threshold` and `low_confidence`, written by the abstention gate -- see below |@@ -240,7 +240,7 @@ the question ids. A `lang_guess` leaves no key of its own -- the hint it acted o`reason`. The `model` and `task` branches report `workflow` as `null` too, because they answerbefore the question ids are read.-### Confidence: two numbers, not interchangeable+### Confidence: three numbers, not interchangeable- `answer_confidence` is the probability mass on the reported answer (`max(p)`). It is thequantity temperature scaling fits and the one this repo's ECE figures are computed on, so it@@ -248,10 +248,14 @@ before the question ids are read.but only for a checkpoint whose temperature fit has been validated on your traffic.- `confidence` means something different per type: normalized entropy `1 - H(p)/log(k)` on`choice` and `score`, and `max(p_yes, p_no)` on `noul` (where it equals `answer_confidence`).+- `x_jev_confidence` is the `confidence` Jev would report for the same probabilities, on `choice`+ and `score` answers only (a Jev `noul` answer has no confidence). It uses TypeSafe's own+ formulas: `(p_max - 1/n)/(1 - 1/n)` for `choice`, and for `score` one minus the probability-weighted+ distance from the most likely level, scaled by the same distance for a uniform distribution.-Never compare the two against one threshold. Also note the difference when porting from Jev:-TypeSafe defines confidence as `(n*p_max - 1)/(n - 1)`, so a threshold carried over from a Jev-deployment gates differently on Laya's entropy value.+Never compare them against one threshold. When porting from Jev, a threshold carried over from a+Jev deployment gates differently on Laya's entropy `confidence` (#302); read it against+`x_jev_confidence` instead. The `x_` prefix keeps the field clear of any name Jev adds later.### Strict Jev contract: `LAYA_JEV_STRICT`@@ -267,7 +271,8 @@ answering, on both `/v1/systemone` and `/v1/systemone/batch`:- a `score` answer keeps `type`, `score`, `probabilities`, `confidence` and `legend`;- a `noul` answer keeps `type` and `noul`;- `usage` keeps `input_tokens` and `output_tokens`; the truncation facts and the collapsed-- options ceiling are not sent.+ options ceiling are not sent;+- `x_jev_confidence` is not sent, like the other additions.The projection keeps only the contracted keys and recomputes nothing: every value is the one theresult already carries, so the probabilities and scores a strict client reads are identical todiff --git a/laya/confidence.py b/laya/confidence.pyindex de251a9c..7f59f55b 100644--- a/laya/confidence.py+++ b/laya/confidence.py@@ -35,6 +35,47 @@ def answer_confidence_value(answer: Dict[str, Any]) -> Optional[float]:return None+def jev_confidence(answer: Dict[str, Any]) -> Optional[float]:+ """The `confidence` Jev would report for this answer's `probabilities`, or None (#302).++ Laya's `confidence` is normalized entropy, and Jev's is a different formula, so a threshold+ carried over from Jev selects about twice as much on Laya. These are TypeSafe's own formulas+ (`system-one-adapter-python`, `_utils/confidence_metrics.py`):++ choice (max(p) - 1/n) / (1 - 1/n)+ score max(0, 1 - sum(p_i * |i - mode|) / D), D the same sum for a uniform distribution++ Both are 1.0 for a single option, and a zero total is read as uniform, as the adapter does.+ A noul answer has no Jev `confidence`, and malformed probabilities give None rather than a+ guess, so a caller filtering on this number never filters on a made-up one.+ """+ kind = answer.get("type")+ probabilities = answer.get("probabilities")+ if kind not in ("choice", "score") or not isinstance(probabilities, dict) or not probabilities:+ return None+ try:+ if kind == "score":+ p = [float(probabilities[key]) for key in sorted(probabilities, key=int)]+ else:+ p = [float(value) for value in probabilities.values()]+ except (TypeError, ValueError):+ return None+ if any(isinstance(v, bool) for v in probabilities.values()) or not all(math.isfinite(v) for v in p):+ return None+ n = len(p)+ if n == 1:+ return 1.0+ total = sum(p)+ p = [v / total for v in p] if total else [1.0 / n] * n+ if kind == "choice":+ return (max(p) - 1.0 / n) / (1.0 - 1.0 / n)+ mode = max(range(n), key=p.__getitem__)+ spread = sum(v * abs(i - mode) for i, v in enumerate(p))+ center = (n - 1) / 2+ uniform_spread = sum(abs(i - center) for i in range(n)) / n+ return max(0.0, 1.0 - spread / uniform_spread)++def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:"""The number the abstention gate compares against `min_confidence`, or None if there is none.diff --git a/laya/serve.py b/laya/serve.pyindex 8706b9fd..606e50bc 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -8,6 +8,12 @@ working unchanged. All this module adds is the HTTP surface Laya itself does notship: a ``POST /v1/systemone`` route and its ``POST /v1/systemone/batch`` sibling,an optional bearer check, and a health probe.+Each ``choice`` and ``score`` answer also carries ``x_jev_confidence``: the+``confidence`` Jev's own formula gives for the same probabilities. Laya's+``confidence`` is normalized entropy and keeps that meaning; a client porting a+threshold from Jev reads ``x_jev_confidence`` instead (#302). ``LAYA_JEV_STRICT``+leaves it out with the other additions.+Configuration is entirely via environment variables so the same entry pointserves a laptop dev run and a systemd unit:@@ -202,6 +208,23 @@ def _project_jev_strict(result: Dict[str, Any]) -> Dict[str, Any]:"output_tokens": usage.get("output_tokens", 0)}}+def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:+ """`result` with `x_jev_confidence` on every choice and score answer (#302).++ The `x_` prefix keeps it clear of any field Jev may add later. An answer whose+ probabilities give no usable number is left without the field rather than given a guess.+ The result is copied, not edited, so a router that reuses its answer dicts is unaffected.+ """+ from .confidence import jev_confidence+ if not isinstance(result.get("answers"), dict):+ return result+ answers: Dict[str, Any] = {}+ for qid, answer in result["answers"].items():+ conf = jev_confidence(answer) if isinstance(answer, dict) else None+ answers[qid] = answer if conf is None else {**answer, "x_jev_confidence": round(conf, 4)}+ return {**result, "answers": answers}++def _published_model_ids() -> Dict[str, str]:"""Public Hugging Face ids accepted so a client can name a checkpoint.@@ -1175,6 +1198,8 @@ def create_app(router: Optional[Any] = None):infer_ms = (time.perf_counter() - t0) * 1000.0if _env_bool("LAYA_JEV_STRICT", False):result = _project_jev_strict(result)+ else:+ result = _add_jev_confidence(result)from fastapi.responses import JSONResponsereturn JSONResponse(content=result,@@ -1315,6 +1340,8 @@ def create_app(router: Optional[Any] = None):infer_ms = (time.perf_counter() - t0) * 1000.0if _env_bool("LAYA_JEV_STRICT", False):batch_res["results"] = [_project_jev_strict(item) for item in batch_res["results"]]+ else:+ batch_res["results"] = [_add_jev_confidence(item) for item in batch_res["results"]]from fastapi.responses import JSONResponsereturn JSONResponse(content=batch_res,diff --git a/tests/test_confidence.py b/tests/test_confidence.pyindex b831841e..8285e6ac 100644--- a/tests/test_confidence.py+++ b/tests/test_confidence.py@@ -2241,6 +2241,61 @@ for _fn in (test_presets_page_drops_the_three_set_claim,else:PASS.append("docs-presets/%s" % _fn.__name__)+# --- jev_confidence: Jev's `confidence` formula, reported beside Laya's own (#302) ----------+# The vectors are TypeSafe's own (system-one-adapter-python, tests/utils/test_confidence_metrics.py)+# plus the two answers in the jevcompat spec's example response.+from laya.confidence import jev_confidence # noqa: E402+++def _choice(*p):+ return {"type": "choice", "probabilities": {"o%d" % i: v for i, v in enumerate(p)}}+++def _score(*p):+ return {"type": "score", "probabilities": {str(i): v for i, v in enumerate(p)}}+++for _name, _answer, _want in (+ ("choice/uniform-2", _choice(0.5, 0.5), 0.0),+ ("choice/zero-total-is-uniform", _choice(0.2, 0.2), 0.0),+ ("choice/0.82-0.18", _choice(0.82, 0.18), 0.64),+ ("choice/single-option", _choice(1.0), 1.0),+ ("choice/jevcompat-example", _choice(0.88, 0.12), 0.76),+ ("score/uniform-5", _score(*[0.2] * 5), 0.0),+ ("score/unnormalized-uniform-5", _score(*[0.04] * 5), 0.0),+ ("score/peaked-5", _score(0.01, 0.02, 0.07, 0.3, 0.6), 0.55),+ ("score/single-level", _score(1.0), 1.0),+ ("score/jevcompat-example", _score(0.0, 0.95, 0.05), 0.925),+):+ _got = jev_confidence(_answer)+ check_true("jev_confidence/%s" % _name, _got is not None and abs(_got - _want) < 1e-9,+ "got %r, want %r" % (_got, _want))++# Score levels are read in numeric order, not string order: "10" sorts before "2" as text.+_levels = {str(i): 0.0 for i in range(11)}+_levels["10"] = 1.0+check("jev_confidence/score-levels-in-numeric-order",+ jev_confidence({"type": "score", "probabilities": _levels}), 1.0)++# The same distribution gives the two numbers #302 measured apart: entropy is not Jev's formula.+_p = np.array([0.7, 0.1, 0.1, 0.1])+check_true("jev_confidence/differs-from-entropy-confidence",+ abs(jev_confidence(_choice(*_p)) - 0.6) < 1e-9 and confidence_from_probs(_p, 4) < 0.35,+ "jev %r, entropy %r" % (jev_confidence(_choice(*_p)), confidence_from_probs(_p, 4)))++# No number rather than a made-up one: a noul has no Jev confidence, and malformed input is refused.+for _name, _answer in (+ ("noul", {"type": "noul", "noul": 0.9}),+ ("unknown-type", {"type": "weird", "probabilities": {"a": 1.0}}),+ ("missing-probabilities", {"type": "choice"}),+ ("empty-probabilities", {"type": "choice", "probabilities": {}}),+ ("non-numeric-probability", _choice(0.5, "high")),+ ("bool-probability", _choice(True, False)),+ ("nan-probability", _choice(float("nan"), 0.5)),+ ("non-integer-score-level", {"type": "score", "probabilities": {"low": 0.5, "high": 0.5}}),+):+ check("jev_confidence/none-for-%s" % _name, jev_confidence(_answer), None)+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL", f)diff --git a/tests/test_serve.py b/tests/test_serve.pyindex c1c10b6d..0850cd66 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -29,6 +29,7 @@ from laya.serve import ( # noqa: E402DEFAULT_MAX_TOKEN_BUDGET,MAX_BODY_BYTES,MAX_STATE_CHARS,+ _add_jev_confidence,_apply_thread_limit,_check_request_limits,_env_bool,@@ -254,6 +255,39 @@ def test_jev_strict_leaves_an_unknown_answer_shape_unchanged(monkeypatch):assert "routing" not in body+def test_choice_and_score_answers_carry_jev_confidence(monkeypatch):+ """Each choice and score answer reports Jev's `confidence` beside Laya's own (#302)."""+ client, _ = _strict_client(monkeypatch, None)+ answers = client.post("/v1/systemone", json=REQ).json()["answers"]+ # choice: (max(p) - 1/n) / (1 - 1/n) over billing 0.9519 / tech 0.0327 / other 0.0154+ assert answers["queue"]["x_jev_confidence"] == 0.9278+ # score: 1 - sum(p_i * |i - mode|) / D with mode 1 and D = 2/3, so 1 - 0.35 / (2/3)+ assert answers["urgency"]["x_jev_confidence"] == 0.475+ # Laya's own numbers are untouched, and a noul answer has no Jev confidence to report+ assert answers["queue"]["confidence"] == 0.797+ assert answers["urgency"]["confidence"] == 0.1925+ assert "x_jev_confidence" not in answers["threat"]+ # the router's own answer dicts are not edited, so a reused result cannot leak the field+ assert "x_jev_confidence" not in FULL_ANSWERS["queue"]+++def test_batch_answers_carry_jev_confidence(monkeypatch):+ client, _ = _strict_client(monkeypatch, None)+ data = client.post("/v1/systemone/batch", json={"states": ["one", "two"],+ "questions": REQ["questions"]}).json()+ assert [item["answers"]["queue"]["x_jev_confidence"] for item in data["results"]] == [0.9278, 0.9278]+++def test_jev_strict_leaves_out_jev_confidence(monkeypatch):+ """Strict mode serves only the contracted keys, so the `x_` addition is not among them."""+ client, _ = _strict_client(monkeypatch, "1")+ single = client.post("/v1/systemone", json=REQ).json()+ batch = client.post("/v1/systemone/batch", json={"states": ["one"],+ "questions": REQ["questions"]}).json()+ for answers in (single["answers"], batch["results"][0]["answers"]):+ assert all("x_jev_confidence" not in answer for answer in answers.values())++def test_known_model_is_honoured(monkeypatch):client, fake = _client(monkeypatch)client.post("/v1/systemone", json={**REQ, "model": "multilingual"})@@ -3106,6 +3140,19 @@ def _gate_written_keys():return sorted(written)+def _serve_added_keys():+ """Every key the server adds onto an answer the agent built, by answer type (#302).++ Read off the server's own `_add_jev_confidence` run on one answer of each type, rather than+ transcribed, so a key it starts or stops adding moves this list with it.+ """+ probe = {"choice": {"type": "choice", "probabilities": {"a": 0.6, "b": 0.4}},+ "score": {"type": "score", "probabilities": {"0": 0.6, "1": 0.4}},+ "noul": {"type": "noul", "noul": 0.6}}+ served = _add_jev_confidence({"answers": probe})["answers"]+ return {qtype: sorted(set(served[qtype]) - set(probe[qtype])) for qtype in probe}++def _md_answer_table(section):"""The `| answer type | keys |` table as rows: which keys the page says an answer carries.@@ -3144,9 +3191,10 @@ def test_http_api_page_documents_the_gate_report_on_an_answer():read a response page that said the answer had been "marked" without saying what the mark is called,what state the answer is in, or what threshold produced it.- So the table is compared against both writers: the `answers[qid] = {...}` literals of each agent,- and the `a[...] = ...` assignments of the gate. Both directions, and against the printed sample too- -- which must carry no gate report, because the request the page prints sets no threshold.+ So the table is compared against every writer: the `answers[qid] = {...}` literals of each agent,+ the `a[...] = ...` assignments of the gate, and the keys the server adds (#302). Both directions,+ and against the printed sample too -- which must carry no gate report, because the request the+ page prints sets no threshold."""page = open(os.path.join(ROOT, "docs", "http-api.md"), encoding="utf-8").read()section = page[page.index("### Response"):page.index("### Confidence")]@@ -3167,11 +3215,12 @@ def test_http_api_page_documents_the_gate_report_on_an_answer():# The row's first column is the discriminator every answer stamps, so it is documented by the row# that documents the type rather than as a key inside the cell.discriminator = {"type"}+ served = _serve_added_keys()for qtype, built in torch_site.items():documented = rows[qtype] | rows["all"] | rows["gate"] | discriminator- assert documented == set(built) | set(gate), (- "the %s rows say an answer carries %s, the agents build %s and the gate adds %s" % (- qtype, sorted(documented), built, gate))+ assert documented == set(built) | set(gate) | set(served[qtype]), (+ "the %s rows say an answer carries %s, the agents build %s, the gate adds %s and the "+ "server adds %s" % (qtype, sorted(documented), built, gate, served[qtype]))# And the sample is one answer to the request the page itself prints, which sends no threshold:# so its answers show the always-on keys and no gate report at all.