Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs: attribute `confidence` to each question type on both confidence pages
Aashish6411090f17dd
diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.mdindex 69937629..456568a5 100644--- a/docs/questions-and-answers.md+++ b/docs/questions-and-answers.md@@ -136,9 +136,12 @@ if ans["answer_confidence"] >= THRESHOLD:...```-`confidence` is normalized entropy: high when the distribution is peaked, low when it is spread-out, regardless of whether the top answer is correct. It is a useful signal and it is **not** on-the same scale, so the two must not be gated against one number:+`confidence` is not one formula. On a `choice` or a `score` it is normalized entropy — high when the+distribution is peaked, low when it is spread out, regardless of whether the top answer is correct,+and its scale moves with the option count. On a `noul` it is `max(p_true, 1 - p_true)`, the+probability of the side being reported, which is why `urgent` below reads 0.8727 rather than the 0.45+entropy gives over two options. Either way it is **not** on `answer_confidence`'s scale, so the two+must not be gated against one number:```python# the same three answers, and the two numbers are not the samediff --git a/docs/structured.md b/docs/structured.mdindex 970053a4..f4a9ce14 100644--- a/docs/structured.md+++ b/docs/structured.md@@ -156,7 +156,7 @@ with per-field confidence, probabilities, the raw answers, and the usage and rouresult = agent.decide(state, schema=Ticket, return_details=True)result.values["department"] # "billing"result.answer_confidence["department"] # 0.94 max(p): the quantity min_confidence gates on-result.confidence["department"] # 0.71 normalized entropy, which depends on label count+result.confidence["department"] # 0.79 1 - H(p)/log(k) on this 3-option `choice` fieldresult.probabilities["department"] # {"billing": 0.94, "support": 0.06, "sales": 0.0}result.usage # {"input_tokens": ..., "output_tokens": 0,# "state_tokens": ..., "state_tokens_dropped": ...,@@ -177,10 +177,15 @@ to the code that builds it.`answer_confidence` is `max(p)`, the probability mass on the answer being reported. It is whattemperature scaling fits, what every calibration figure in this repository is computed on, and what`min_confidence` is compared against — which is the reason to gate on it rather than on-`confidence`. `confidence` is normalized entropy, which depends on how many options the question-had: `tests/test_confidence.py` pins that a two-option distribution comes back as 0.90 on a `noul`-and 0.53 on an equivalent `choice`, so it does not compare against a threshold. A field that-reported no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.+`confidence`. `confidence` is not one formula. On a `choice` or a `score` it is normalized entropy,+`1 - H(p) / log(k)`, whose scale moves with the number of options the question had. On a `noul` it is+`max(p_true, 1 - p_true)`, the probability of the side being reported, which over two options is+`max(p)`. `tests/test_confidence.py` pins a two-option distribution as 0.90 on a `noul` against 0.53+on an equivalent `choice`; 0.53 is what entropy gives at `k = 2`, so the 0.90 is the second formula and+not the first at a smaller option count. The schema above mixes both — `department` is a `choice`,+`needs_human` a `noul` — so one `DecisionResult.confidence` dict can hold two scales at once, which is+why neither reading compares against a threshold you have not measured per type. A field that reported+no usable `answer_confidence` maps to `None`, which is not the same as a reported `0.0`.Gate on the same quantity the gate uses: