Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(examples): example 30 must call score confidence 1 - H/log(k)

Aashishc531bb5a89b9
diff --git a/examples/30_custom_schema_design.py b/examples/30_custom_schema_design.py
index bef96e15..12ab8c4c 100644
--- a/examples/30_custom_schema_design.py
+++ b/examples/30_custom_schema_design.py
@@ -1,9 +1,12 @@
"""Example 30 -- writing your own question schema.
The presets are fixed schemas; this builds a four-question bug-triage schema from scratch
-as ordinary Python, runs it on two contrasting reports, then takes the design apart.
+as ordinary Python, runs it on two contrasting reports, then takes the design apart --
+including a recomputation of the score field's `confidence` from its own probabilities.
"""
+import math
+
from _common import banner, describe, device_line, heading, load, top
banner("30", "Designing a question schema", """
@@ -70,6 +73,23 @@ PROBE = ("label typo", {
"else works fine.",
})
+
+def entropy_confidence(probs):
+ """1 - H/log(k) over a reported distribution: `confidence`'s definition, recomputed by hand.
+
+ Same formula as `laya.common.confidence_from_probs`, and the published probabilities are
+ rounded to 4 dp while the agent computed the field from unrounded ones, so the recomputation
+ is compared against the reported field rather than substituted for it. A wide-but-ordered
+ distribution therefore reads as a LOW confidence here, which is the point the score guidance
+ makes below.
+ """
+ k = len(probs)
+ if k < 2:
+ return 1.0
+ ent = -sum(p * math.log(max(p, 1e-12)) for p in probs)
+ return max(0.0, min(1.0, 1.0 - ent / math.log(k)))
+
+
agent = load("english")
device_line(agent)
@@ -121,8 +141,20 @@ print(" versus UI %.2f / 3. A choice question (\"severe: yes/no\") would throw
% ui["severity"]["score"])
print(" away. Use a score when the levels are ordered and you will threshold or compare")
print(" them, and a choice when the labels are unordered. Read the score itself: `confidence`")
-print(" on a score is normalised entropy, so a wide-but-ordered distribution looks")
-print(" unconfident even when the expected level is informative.")
+print(" on a score is 1 - H/log(k), one MINUS the normalised entropy, so a wide-but-ordered")
+print(" distribution looks unconfident even when the expected level is informative. Both")
+print(" records here:")
+for label, ans in (("crash", crash), ("UI", ui)):
+ probs = list(ans["severity"]["probabilities"].values())
+ reported = ans["severity"]["confidence"]
+ recomputed = entropy_confidence(probs)
+ print(" %-6s k=%d entropy confidence: reported %.4f, recomputed %.4f (%.6f apart)"
+ % (label, len(probs), reported, recomputed, abs(reported - recomputed)))
+print(" The recomputation matches the reported field to the rounding (probabilities are")
+print(" published at 4 dp while the agent computed the field from unrounded logits), and each")
+print(" number is well below the corresponding `answer_confidence`, which is max(p) on the")
+print(" same distribution: the expected severity level is informative, the entropy scale is")
+print(" the wider, more conservative of the two.")
heading("4. every question shares one forward pass")
tokens = runs[0][2]["usage"]["input_tokens"]