Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(api): clarify independent probability rounding

monsterlag1478a56f1ec1
diff --git a/docs/http-api.md b/docs/http-api.md
index 6e3a22f2..c9d805b8 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -190,6 +190,11 @@ The sample is one answer this server gave, verbatim: the request above, the cach
checkpoint on CPU. `answers` and `usage` are the keys Jev clients decode; `model` is the constant
name of the decision head, and the checkpoint that answered is in `routing`.
+`probabilities` on `choice` and `score` answers are rounded independently to four decimal places
+and need not sum exactly to 1, just as in the [in-process API](questions-and-answers.md#probability-precision).
+For strict unit-sum validation, allow a rounding tolerance that accounts for the number of options,
+or normalize a copy of the distribution.
+
| answer type | keys |
|---|---|
| `choice` | `choice` (the argmax option), `probabilities` per option, `x_jev_confidence` |
@@ -312,7 +317,7 @@ The answer keeps the Jev shape, so the official SDK parses it:
- `probabilities` names every option, in criteria order. The finalists carry the final round's
probabilities and an option eliminated in an earlier round carries `0.0`, so `choice` is still
- the argmax and the values still sum to 1;
+ the argmax, but the rounded values need not sum exactly to 1;
- `x_tournament` lists the `finalists` and the number of `rounds`, so a caller can tell an
eliminated option from one the final round scored. Under `LAYA_JEV_STRICT` it is not sent;
- `confidence` and `answer_confidence` are the final round's, over the finalists;
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.md
index 90d0ce39..c3ccd583 100644
--- a/docs/questions-and-answers.md
+++ b/docs/questions-and-answers.md
@@ -112,6 +112,14 @@ disagreement with the model. If you want the label, take the argmax of `probabil
`answer_confidence`'s counterpart from the legend — do not round `score` and assume it is the
label. `legend` exists so you never have to guess which index means what.
+## Probability precision
+
+On `choice` and `score` answers, `probabilities` are rounded independently to four decimal places.
+They need not sum exactly to 1: three equally likely options return `0.3333` each, totalling
+`0.9999`. The returned values are not renormalized after rounding. If your consumer requires a
+unit sum, allow a rounding tolerance that accounts for the number of options, or normalize a copy
+of the distribution before validating it.
+
## Reading confidence
Every answer carries two confidence numbers, and they measure different things.
diff --git a/tests/test_confidence.py b/tests/test_confidence.py
index 949db982..c38730b5 100644
--- a/tests/test_confidence.py
+++ b/tests/test_confidence.py
@@ -2566,6 +2566,38 @@ for _qid in ("q", "r", "s"):
check_true("jev_public/served noul carries no field",
"x_jev_confidence" not in _served["n"], _served["n"].keys())
+# --- probability precision: both decoders round independently, not to a unit sum (#555) ------
+from laya.agent import Agent # noqa: E402
+from laya.onnx_agent import ONNXAgent # noqa: E402
+
+for _cls in (Agent, ONNXAgent):
+ _agent = _cls.__new__(_cls) # no checkpoint, model or ONNX session
+ _agent.temperature = [1.0, 1.0, 1.0]
+ _agent.temperature_by_options = {}
+ for _k, _value, _total in ((3, 0.3333, 0.9999), (6, 0.1667, 1.0002)):
+ for _qtype in ("choice", "score"):
+ _labels = ["o%d" % i for i in range(_k)]
+ _crit = dict.fromkeys(_labels) if _qtype == "choice" else _labels
+ _answer = _agent._decode_answers(
+ np.zeros((1, _k)), np.array([[1.0, 0.0]]), [{"markers": list(range(_k))}],
+ ["q"], {"q": {"t": _qtype, "crit": _crit}}, 0)["q"]
+ _probs = list(_answer["probabilities"].values())
+ _name = "rounding/%s/%s/k=%d" % (_cls.__name__, _qtype, _k)
+ check(_name + "/four decimal places", _probs, [_value] * _k)
+ check_true(_name + "/non-unit sum", close(sum(_probs), _total), repr(_probs))
+
+for _page in ("questions-and-answers.md", "http-api.md"):
+ _text = _prose(_read_page(os.path.join(DOC_ROOT, "docs", _page)))
+ check_true("rounding/docs/%s explains precision" % _page,
+ bool(_re.search(r"`probabilities`.*?rounded independently to four decimal places", _text)))
+ check_true("rounding/docs/%s qualifies the unit sum" % _page,
+ "need not sum exactly to 1" in _text)
+
+_tournament = _prose(_read_page(os.path.join(DOC_ROOT, "docs", "http-api.md")).split(
+ "### Large choices:", 1)[1].split("## Limits", 1)[0])
+check_true("rounding/docs/tournament does not promise an exact unit sum",
+ "need not sum exactly to 1" in _tournament)
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL", f)