Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): report Jev's confidence formula as x_jev_confidence (#302)

kmitulc932543ca501
diff --git a/docs/http-api.md b/docs/http-api.md
index 8c2c9a6f..a93644ab 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -164,12 +164,12 @@ hide that. The detail is the same `unknown model` text core raises, plus the rem
"queue": {"type": "choice", "choice": "billing",
"probabilities": {"billing": 0.9519, "tech": 0.0327, "other": 0.0154},
"confidence": 0.797, "answer_confidence": 0.9519,
- "action": {"act_probability": 1.0}},
+ "action": {"act_probability": 1.0}, "x_jev_confidence": 0.9278},
"urgency": {"type": "score", "score": 1.6994,
"legend": {"0": "calm", "1": "firm", "2": "angry", "3": "furious"},
"probabilities": {"0": 0.0249, "1": 0.4136, "2": 0.3985, "3": 0.1629},
"confidence": 0.1925, "answer_confidence": 0.4136,
- "action": {"act_probability": 1.0}}
+ "action": {"act_probability": 1.0}, "x_jev_confidence": 0.2507}
},
"usage": {"input_tokens": 83, "output_tokens": 0, "state_tokens": 12,
"state_tokens_dropped": 0, "truncated": false, "truncated_questions": []},
@@ -187,8 +187,8 @@ name of the decision head, and the checkpoint that answered is in `routing`.
| answer type | keys |
|---|---|
-| `choice` | `choice` (the argmax option), `probabilities` per option |
-| `score` | `score` (expected level index, may fall between levels), `probabilities` keyed `"0".. "k-1"`, `legend` mapping index to the level text |
+| `choice` | `choice` (the argmax option), `probabilities` per option, `x_jev_confidence` |
+| `score` | `score` (expected level index, may fall between levels), `probabilities` keyed `"0".. "k-1"`, `legend` mapping index to the level text, `x_jev_confidence` |
| `noul` | `noul`, the probability of the yes option |
| all | `type` (the discriminator, echoing the question's own type), `confidence`, `answer_confidence`, and `action.act_probability` |
| gate | `abstention`, `abstention_threshold` and `low_confidence`, written by the abstention gate -- see below |
@@ -240,7 +240,7 @@ the question ids. A `lang_guess` leaves no key of its own -- the hint it acted o
`reason`. The `model` and `task` branches report `workflow` as `null` too, because they answer
before the question ids are read.
-### Confidence: two numbers, not interchangeable
+### Confidence: three numbers, not interchangeable
- `answer_confidence` is the probability mass on the reported answer (`max(p)`). It is the
quantity temperature scaling fits and the one this repo's ECE figures are computed on, so it
@@ -248,10 +248,14 @@ before the question ids are read.
but only for a checkpoint whose temperature fit has been validated on your traffic.
- `confidence` means something different per type: normalized entropy `1 - H(p)/log(k)` on
`choice` and `score`, and `max(p_yes, p_no)` on `noul` (where it equals `answer_confidence`).
+- `x_jev_confidence` is the `confidence` Jev would report for the same probabilities, on `choice`
+ and `score` answers only (a Jev `noul` answer has no confidence). It uses TypeSafe's own
+ formulas: `(p_max - 1/n)/(1 - 1/n)` for `choice`, and for `score` one minus the probability-weighted
+ distance from the most likely level, scaled by the same distance for a uniform distribution.
-Never compare the two against one threshold. Also note the difference when porting from Jev:
-TypeSafe defines confidence as `(n*p_max - 1)/(n - 1)`, so a threshold carried over from a Jev
-deployment gates differently on Laya's entropy value.
+Never compare them against one threshold. When porting from Jev, a threshold carried over from a
+Jev deployment gates differently on Laya's entropy `confidence` (#302); read it against
+`x_jev_confidence` instead. The `x_` prefix keeps the field clear of any name Jev adds later.
### Strict Jev contract: `LAYA_JEV_STRICT`
@@ -267,7 +271,8 @@ answering, on both `/v1/systemone` and `/v1/systemone/batch`:
- a `score` answer keeps `type`, `score`, `probabilities`, `confidence` and `legend`;
- a `noul` answer keeps `type` and `noul`;
- `usage` keeps `input_tokens` and `output_tokens`; the truncation facts and the collapsed-
- options ceiling are not sent.
+ options ceiling are not sent;
+- `x_jev_confidence` is not sent, like the other additions.
The projection keeps only the contracted keys and recomputes nothing: every value is the one the
result already carries, so the probabilities and scores a strict client reads are identical to
diff --git a/laya/confidence.py b/laya/confidence.py
index de251a9c..7f59f55b 100644
--- a/laya/confidence.py
+++ b/laya/confidence.py
@@ -35,6 +35,47 @@ def answer_confidence_value(answer: Dict[str, Any]) -> Optional[float]:
return None
+def jev_confidence(answer: Dict[str, Any]) -> Optional[float]:
+ """The `confidence` Jev would report for this answer's `probabilities`, or None (#302).
+
+ Laya's `confidence` is normalized entropy, and Jev's is a different formula, so a threshold
+ carried over from Jev selects about twice as much on Laya. These are TypeSafe's own formulas
+ (`system-one-adapter-python`, `_utils/confidence_metrics.py`):
+
+ choice (max(p) - 1/n) / (1 - 1/n)
+ score max(0, 1 - sum(p_i * |i - mode|) / D), D the same sum for a uniform distribution
+
+ Both are 1.0 for a single option, and a zero total is read as uniform, as the adapter does.
+ A noul answer has no Jev `confidence`, and malformed probabilities give None rather than a
+ guess, so a caller filtering on this number never filters on a made-up one.
+ """
+ kind = answer.get("type")
+ probabilities = answer.get("probabilities")
+ if kind not in ("choice", "score") or not isinstance(probabilities, dict) or not probabilities:
+ return None
+ try:
+ if kind == "score":
+ p = [float(probabilities[key]) for key in sorted(probabilities, key=int)]
+ else:
+ p = [float(value) for value in probabilities.values()]
+ except (TypeError, ValueError):
+ return None
+ if any(isinstance(v, bool) for v in probabilities.values()) or not all(math.isfinite(v) for v in p):
+ return None
+ n = len(p)
+ if n == 1:
+ return 1.0
+ total = sum(p)
+ p = [v / total for v in p] if total else [1.0 / n] * n
+ if kind == "choice":
+ return (max(p) - 1.0 / n) / (1.0 - 1.0 / n)
+ mode = max(range(n), key=p.__getitem__)
+ spread = sum(v * abs(i - mode) for i, v in enumerate(p))
+ center = (n - 1) / 2
+ uniform_spread = sum(abs(i - center) for i in range(n)) / n
+ return max(0.0, 1.0 - spread / uniform_spread)
+
+
def _gate_confidence(answer: Dict[str, Any]) -> Optional[float]:
"""The number the abstention gate compares against `min_confidence`, or None if there is none.
diff --git a/laya/serve.py b/laya/serve.py
index 8706b9fd..606e50bc 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -8,6 +8,12 @@ working unchanged. All this module adds is the HTTP surface Laya itself does not
ship: a ``POST /v1/systemone`` route and its ``POST /v1/systemone/batch`` sibling,
an optional bearer check, and a health probe.
+Each ``choice`` and ``score`` answer also carries ``x_jev_confidence``: the
+``confidence`` Jev's own formula gives for the same probabilities. Laya's
+``confidence`` is normalized entropy and keeps that meaning; a client porting a
+threshold from Jev reads ``x_jev_confidence`` instead (#302). ``LAYA_JEV_STRICT``
+leaves it out with the other additions.
+
Configuration is entirely via environment variables so the same entry point
serves a laptop dev run and a systemd unit:
@@ -202,6 +208,23 @@ def _project_jev_strict(result: Dict[str, Any]) -> Dict[str, Any]:
"output_tokens": usage.get("output_tokens", 0)}}
+def _add_jev_confidence(result: Dict[str, Any]) -> Dict[str, Any]:
+ """`result` with `x_jev_confidence` on every choice and score answer (#302).
+
+ The `x_` prefix keeps it clear of any field Jev may add later. An answer whose
+ probabilities give no usable number is left without the field rather than given a guess.
+ The result is copied, not edited, so a router that reuses its answer dicts is unaffected.
+ """
+ from .confidence import jev_confidence
+ if not isinstance(result.get("answers"), dict):
+ return result
+ answers: Dict[str, Any] = {}
+ for qid, answer in result["answers"].items():
+ conf = jev_confidence(answer) if isinstance(answer, dict) else None
+ answers[qid] = answer if conf is None else {**answer, "x_jev_confidence": round(conf, 4)}
+ return {**result, "answers": answers}
+
+
def _published_model_ids() -> Dict[str, str]:
"""Public Hugging Face ids accepted so a client can name a checkpoint.
@@ -1175,6 +1198,8 @@ def create_app(router: Optional[Any] = None):
infer_ms = (time.perf_counter() - t0) * 1000.0
if _env_bool("LAYA_JEV_STRICT", False):
result = _project_jev_strict(result)
+ else:
+ result = _add_jev_confidence(result)
from fastapi.responses import JSONResponse
return JSONResponse(
content=result,
@@ -1315,6 +1340,8 @@ def create_app(router: Optional[Any] = None):
infer_ms = (time.perf_counter() - t0) * 1000.0
if _env_bool("LAYA_JEV_STRICT", False):
batch_res["results"] = [_project_jev_strict(item) for item in batch_res["results"]]
+ else:
+ batch_res["results"] = [_add_jev_confidence(item) for item in batch_res["results"]]
from fastapi.responses import JSONResponse
return JSONResponse(
content=batch_res,
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index b831841e..8285e6ac 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -2241,6 +2241,61 @@ for _fn in (test_presets_page_drops_the_three_set_claim,
else:
PASS.append("docs-presets/%s" % _fn.__name__)
+# --- jev_confidence: Jev's `confidence` formula, reported beside Laya's own (#302) ----------
+# The vectors are TypeSafe's own (system-one-adapter-python, tests/utils/test_confidence_metrics.py)
+# plus the two answers in the jevcompat spec's example response.
+from laya.confidence import jev_confidence # noqa: E402
+
+
+def _choice(*p):
+ return {"type": "choice", "probabilities": {"o%d" % i: v for i, v in enumerate(p)}}
+
+
+def _score(*p):
+ return {"type": "score", "probabilities": {str(i): v for i, v in enumerate(p)}}
+
+
+for _name, _answer, _want in (
+ ("choice/uniform-2", _choice(0.5, 0.5), 0.0),
+ ("choice/zero-total-is-uniform", _choice(0.2, 0.2), 0.0),
+ ("choice/0.82-0.18", _choice(0.82, 0.18), 0.64),
+ ("choice/single-option", _choice(1.0), 1.0),
+ ("choice/jevcompat-example", _choice(0.88, 0.12), 0.76),
+ ("score/uniform-5", _score(*[0.2] * 5), 0.0),
+ ("score/unnormalized-uniform-5", _score(*[0.04] * 5), 0.0),
+ ("score/peaked-5", _score(0.01, 0.02, 0.07, 0.3, 0.6), 0.55),
+ ("score/single-level", _score(1.0), 1.0),
+ ("score/jevcompat-example", _score(0.0, 0.95, 0.05), 0.925),
+):
+ _got = jev_confidence(_answer)
+ check_true("jev_confidence/%s" % _name, _got is not None and abs(_got - _want) < 1e-9,
+ "got %r, want %r" % (_got, _want))
+
+# Score levels are read in numeric order, not string order: "10" sorts before "2" as text.
+_levels = {str(i): 0.0 for i in range(11)}
+_levels["10"] = 1.0
+check("jev_confidence/score-levels-in-numeric-order",
+ jev_confidence({"type": "score", "probabilities": _levels}), 1.0)
+
+# The same distribution gives the two numbers #302 measured apart: entropy is not Jev's formula.
+_p = np.array([0.7, 0.1, 0.1, 0.1])
+check_true("jev_confidence/differs-from-entropy-confidence",
+ abs(jev_confidence(_choice(*_p)) - 0.6) < 1e-9 and confidence_from_probs(_p, 4) < 0.35,
+ "jev %r, entropy %r" % (jev_confidence(_choice(*_p)), confidence_from_probs(_p, 4)))
+
+# No number rather than a made-up one: a noul has no Jev confidence, and malformed input is refused.
+for _name, _answer in (
+ ("noul", {"type": "noul", "noul": 0.9}),
+ ("unknown-type", {"type": "weird", "probabilities": {"a": 1.0}}),
+ ("missing-probabilities", {"type": "choice"}),
+ ("empty-probabilities", {"type": "choice", "probabilities": {}}),
+ ("non-numeric-probability", _choice(0.5, "high")),
+ ("bool-probability", _choice(True, False)),
+ ("nan-probability", _choice(float("nan"), 0.5)),
+ ("non-integer-score-level", {"type": "score", "probabilities": {"low": 0.5, "high": 0.5}}),
+):
+ check("jev_confidence/none-for-%s" % _name, jev_confidence(_answer), None)
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL", f)
diff --git a/tests/test_serve.py b/tests/test_serve.py
index c1c10b6d..0850cd66 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -29,6 +29,7 @@ from laya.serve import ( # noqa: E402
DEFAULT_MAX_TOKEN_BUDGET,
MAX_BODY_BYTES,
MAX_STATE_CHARS,
+ _add_jev_confidence,
_apply_thread_limit,
_check_request_limits,
_env_bool,
@@ -254,6 +255,39 @@ def test_jev_strict_leaves_an_unknown_answer_shape_unchanged(monkeypatch):
assert "routing" not in body
+def test_choice_and_score_answers_carry_jev_confidence(monkeypatch):
+ """Each choice and score answer reports Jev's `confidence` beside Laya's own (#302)."""
+ client, _ = _strict_client(monkeypatch, None)
+ answers = client.post("/v1/systemone", json=REQ).json()["answers"]
+ # choice: (max(p) - 1/n) / (1 - 1/n) over billing 0.9519 / tech 0.0327 / other 0.0154
+ assert answers["queue"]["x_jev_confidence"] == 0.9278
+ # score: 1 - sum(p_i * |i - mode|) / D with mode 1 and D = 2/3, so 1 - 0.35 / (2/3)
+ assert answers["urgency"]["x_jev_confidence"] == 0.475
+ # Laya's own numbers are untouched, and a noul answer has no Jev confidence to report
+ assert answers["queue"]["confidence"] == 0.797
+ assert answers["urgency"]["confidence"] == 0.1925
+ assert "x_jev_confidence" not in answers["threat"]
+ # the router's own answer dicts are not edited, so a reused result cannot leak the field
+ assert "x_jev_confidence" not in FULL_ANSWERS["queue"]
+
+
+def test_batch_answers_carry_jev_confidence(monkeypatch):
+ client, _ = _strict_client(monkeypatch, None)
+ data = client.post("/v1/systemone/batch", json={"states": ["one", "two"],
+ "questions": REQ["questions"]}).json()
+ assert [item["answers"]["queue"]["x_jev_confidence"] for item in data["results"]] == [0.9278, 0.9278]
+
+
+def test_jev_strict_leaves_out_jev_confidence(monkeypatch):
+ """Strict mode serves only the contracted keys, so the `x_` addition is not among them."""
+ client, _ = _strict_client(monkeypatch, "1")
+ single = client.post("/v1/systemone", json=REQ).json()
+ batch = client.post("/v1/systemone/batch", json={"states": ["one"],
+ "questions": REQ["questions"]}).json()
+ for answers in (single["answers"], batch["results"][0]["answers"]):
+ assert all("x_jev_confidence" not in answer for answer in answers.values())
+
+
def test_known_model_is_honoured(monkeypatch):
client, fake = _client(monkeypatch)
client.post("/v1/systemone", json={**REQ, "model": "multilingual"})
@@ -3106,6 +3140,19 @@ def _gate_written_keys():
return sorted(written)
+def _serve_added_keys():
+ """Every key the server adds onto an answer the agent built, by answer type (#302).
+
+ Read off the server's own `_add_jev_confidence` run on one answer of each type, rather than
+ transcribed, so a key it starts or stops adding moves this list with it.
+ """
+ probe = {"choice": {"type": "choice", "probabilities": {"a": 0.6, "b": 0.4}},
+ "score": {"type": "score", "probabilities": {"0": 0.6, "1": 0.4}},
+ "noul": {"type": "noul", "noul": 0.6}}
+ served = _add_jev_confidence({"answers": probe})["answers"]
+ return {qtype: sorted(set(served[qtype]) - set(probe[qtype])) for qtype in probe}
+
+
def _md_answer_table(section):
"""The `| answer type | keys |` table as rows: which keys the page says an answer carries.
@@ -3144,9 +3191,10 @@ def test_http_api_page_documents_the_gate_report_on_an_answer():
read a response page that said the answer had been "marked" without saying what the mark is called,
what state the answer is in, or what threshold produced it.
- So the table is compared against both writers: the `answers[qid] = {...}` literals of each agent,
- and the `a[...] = ...` assignments of the gate. Both directions, and against the printed sample too
- -- which must carry no gate report, because the request the page prints sets no threshold.
+ So the table is compared against every writer: the `answers[qid] = {...}` literals of each agent,
+ the `a[...] = ...` assignments of the gate, and the keys the server adds (#302). Both directions,
+ and against the printed sample too -- which must carry no gate report, because the request the
+ page prints sets no threshold.
"""
page = open(os.path.join(ROOT, "docs", "http-api.md"), encoding="utf-8").read()
section = page[page.index("### Response"):page.index("### Confidence")]
@@ -3167,11 +3215,12 @@ def test_http_api_page_documents_the_gate_report_on_an_answer():
# The row's first column is the discriminator every answer stamps, so it is documented by the row
# that documents the type rather than as a key inside the cell.
discriminator = {"type"}
+ served = _serve_added_keys()
for qtype, built in torch_site.items():
documented = rows[qtype] | rows["all"] | rows["gate"] | discriminator
- assert documented == set(built) | set(gate), (
- "the %s rows say an answer carries %s, the agents build %s and the gate adds %s" % (
- qtype, sorted(documented), built, gate))
+ assert documented == set(built) | set(gate) | set(served[qtype]), (
+ "the %s rows say an answer carries %s, the agents build %s, the gate adds %s and the "
+ "server adds %s" % (qtype, sorted(documented), built, gate, served[qtype]))
# And the sample is one answer to the request the page itself prints, which sends no threshold:
# so its answers show the always-on keys and no gate report at all.