Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs: attribute `confidence` to each question type on both confidence pages

Aashish6411090f17dd
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.md
index 69937629..456568a5 100644
--- a/docs/questions-and-answers.md
+++ b/docs/questions-and-answers.md
@@ -136,9 +136,12 @@ if ans["answer_confidence"] >= THRESHOLD:
...
```
-`confidence` is normalized entropy: high when the distribution is peaked, low when it is spread
-out, regardless of whether the top answer is correct. It is a useful signal and it is **not** on
-the same scale, so the two must not be gated against one number:
+`confidence` is not one formula. On a `choice` or a `score` it is normalized entropy — high when the
+distribution is peaked, low when it is spread out, regardless of whether the top answer is correct,
+and its scale moves with the option count. On a `noul` it is `max(p_true, 1 - p_true)`, the
+probability of the side being reported, which is why `urgent` below reads 0.8727 rather than the 0.45
+entropy gives over two options. Either way it is **not** on `answer_confidence`'s scale, so the two
+must not be gated against one number:
```python
# the same three answers, and the two numbers are not the same
diff --git a/docs/structured.md b/docs/structured.md
index 970053a4..f4a9ce14 100644
--- a/docs/structured.md
+++ b/docs/structured.md
@@ -156,7 +156,7 @@ with per-field confidence, probabilities, the raw answers, and the usage and rou
result = agent.decide(state, schema=Ticket, return_details=True)
result.values["department"] # "billing"
result.answer_confidence["department"] # 0.94 max(p): the quantity min_confidence gates on
-result.confidence["department"] # 0.71 normalized entropy, which depends on label count
+result.confidence["department"] # 0.79 1 - H(p)/log(k) on this 3-option `choice` field
result.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}
result.usage # {"input_tokens": ..., "output_tokens": 0,
# "state_tokens": ..., "state_tokens_dropped": ...,
@@ -177,10 +177,15 @@ to the code that builds it.
`answer_confidence` is `max(p)`, the probability mass on the answer being reported. It is what
temperature scaling fits, what every calibration figure in this repository is computed on, and what
`min_confidence` is compared against — which is the reason to gate on it rather than on
-`confidence`. `confidence` is normalized entropy, which depends on how many options the question
-had: `tests/test_confidence.py` pins that a two-option distribution comes back as 0.90 on a `noul`
-and 0.53 on an equivalent `choice`, so it does not compare against a threshold. A field that
-reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.
+`confidence`. `confidence` is not one formula. On a `choice` or a `score` it is normalized entropy,
+`1 - H(p) / log(k)`, whose scale moves with the number of options the question had. On a `noul` it is
+`max(p_true, 1 - p_true)`, the probability of the side being reported, which over two options is
+`max(p)`. `tests/test_confidence.py` pins a two-option distribution as 0.90 on a `noul` against 0.53
+on an equivalent `choice`; 0.53 is what entropy gives at `k = 2`, so the 0.90 is the second formula and
+not the first at a smaller option count. The schema above mixes both — `department` is a `choice`,
+`needs_human` a `noul` — so one `DecisionResult.confidence` dict can hold two scales at once, which is
+why neither reading compares against a threshold you have not measured per type. A field that reported
+no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.
Gate on the same quantity the gate uses: