Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): allow custom noul labels (#163)
Wasiq Nauman158398a7440e
diff --git a/README.md b/README.mdindex 677a4869..234cf61f 100644--- a/README.md+++ b/README.md@@ -479,6 +479,31 @@ triage = agent.predict({"message": "My payment failed twice"}, laya.triage_quest| **`score`** | Expected level on ordinal rubric, distribution, confidence | Frustration level, ticket urgency, harm severity || **`noul`** | Calibrated probability P(true) from 0.0 to 1.0 | Phishing detection, spam filtering, jailbreak detection, churn risk |+`noul` always scores two semantic slots in `[false, true]` order and returns the probability of+the second slot. For compatibility, those slots are shown to the model as `false` and `true` by+default. The optional `labels` mapping overrides only that model-facing text without changing the+returned meaning:++```python+question = {+ "type": "noul",+ "instructions": "Is this review positive?",+ "criteria": {+ "false": "the review is negative",+ "true": "the review is positive",+ },+ "labels": {+ "false": "B",+ "true": "A",+ },+}+```++The `labels` mapping is optional. It must contain exactly the string keys `false` and `true`,+whose values must be distinct non-empty strings. Mapping order does not matter, and the returned+`noul` value is still P(true). Label sensitivity varies by checkpoint and state, so validate any+override on your own data rather than treating `A`/`B` as a universal fix.+---## Benchmarks@@ -610,6 +635,10 @@ failure; it does not establish calibrated confidence.against a 0.318 random baseline and a 0.461 majority-class baseline. The 0.766 figure comesfrom the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base tospecialise, not a zero-shot decision engine.+* **Avoid boolean-word labels in `choice` questions.** Choice keys are rendered verbatim, and the+ current checkpoints can follow labels such as `true`/`false` or `yes`/`no` instead of the option+ descriptions. Use semantic labels or opaque labels such as `A`/`B`, and validate them on the+ checkpoint and states you serve.* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget (`max_len - head_max_len`):* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).@@ -647,7 +676,16 @@ result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.* Ordinal `score` questions are the weakest primitive (SST-5 0.372).-* **`noul` can follow its option labels instead of the state, most strongly on `laya` (English).** `noul` renders its two options as `false:` / `true:`, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Until a retrained checkpoint lands, check `noul` answers on your own data. If they look stuck, ask the same question as a two-option `choice` with neutral keys and your yes/no wording as the descriptions:+* **`noul` can follow its option labels instead of the state, most strongly on `laya` (English).** `noul` renders its two options as `false:` / `true:` by default, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Until a retrained checkpoint lands, check `noul` answers on your own data. You can override the model-facing pair while keeping the `noul` result as P(true):++ ```python+ {"type": "noul", "instructions": "Is this review positive?",+ "criteria": {"true": "yes, the review is positive", "false": "no, the review is negative"},+ "labels": {"true": "A", "false": "B"}}+ ```++ Label sensitivity varies by checkpoint and state, so validate the override on your own data. A+ two-option `choice` with neutral keys remains another workaround:```python{"type": "choice", "instructions": "Is this review positive?",diff --git a/laya/agent.py b/laya/agent.pyindex 21892ddc..e5e33728 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -18,6 +18,7 @@ from .common import (clamp_temperature,collate_items,confidence_from_probs,+ _resolve_noul_labels,render_options,temp_bucket,)@@ -385,6 +386,13 @@ class Agent:elif crit is not None and not isinstance(crit, dict):raise ValueError("question %r: a noul question takes 'criteria' as a dict with optional ""'true'/'false' descriptions, or omits it" % (qid,))+ if "labels" in qdef:+ if t != "noul":+ raise ValueError("question %r: 'labels' is only supported for noul questions" % (qid,))+ try:+ _resolve_noul_labels(qdef["labels"])+ except ValueError as e:+ raise ValueError("question %r: %s" % (qid, e)) from e@staticmethoddef _to_internal(qdef: Dict) -> Dict:@@ -403,7 +411,10 @@ class Agent:# on the English checkpoint one German question answered noul=0.1652 as a dict# and noul=0.2650 as the identical plain string.ins = json.dumps(ins, ensure_ascii=False)- return {"t": t, "ins": ins, "crit": crit}+ q = {"t": t, "ins": ins, "crit": crit}+ if "labels" in qdef:+ q["labels"] = qdef["labels"]+ return q@torch.no_grad()def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:@@ -414,7 +425,13 @@ class Agent:questions: Dictionary mapping question_id -> question definition.- choice: {"type": "choice", "instructions": "...", "criteria": {"optA": "...", ...}}- score: {"type": "score", "instructions": "...", "criteria": ["lvl0", "lvl1", ...]}- - noul: {"type": "noul", "instructions": "...", "criteria": {"true": "...", "false": "..."}}+ - noul: {"type": "noul", "instructions": "...",+ "criteria": {"false": "...", "true": "..."},+ "labels": {"false": "B", "true": "A"}}++ Noul criteria and labels are optional. Labels only control the text shown to the+ model; their keys retain false/true semantics, and the returned `noul` value is+ always P(true). Labels default to false/true for compatibility.Returns:Dictionary with answers, probabilities, calibrated confidence, and token usage.diff --git a/laya/common.py b/laya/common.pyindex ab2a303d..d73e248b 100644--- a/laya/common.py+++ b/laya/common.py@@ -11,6 +11,7 @@ from torch.utils.checkpoint import checkpointQTYPES = {"choice": 0, "score": 1, "noul": 2}QTYPE_NAMES = {v: k for k, v in QTYPES.items()}+_DEFAULT_NOUL_LABELS = {"false": "false", "true": "true"}def serialize_state(state: Union[str, dict, list]) -> str:@@ -31,19 +32,38 @@ def render_criterion(value) -> str:return json.dumps(value, ensure_ascii=False, separators=(", ", ": "), default=str)+def _resolve_noul_labels(labels=None):+ if labels is None:+ labels = _DEFAULT_NOUL_LABELS+ if not isinstance(labels, dict) or set(labels) != {"false", "true"}:+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")+ false_label, true_label = labels["false"], labels["true"]+ if not isinstance(false_label, str) or not isinstance(true_label, str):+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")+ false_label, true_label = false_label.strip(), true_label.strip()+ if not false_label or not true_label or false_label == true_label:+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")+ return false_label, true_label++def render_options(q: Dict) -> List[str]:- """Render option texts in label-index order. Noul is always [false, true]."""+ """Render option texts in label-index order. Noul semantic order is always [false, true]."""t, crit = q["t"], q.get("crit")+ if t != "noul" and "labels" in q:+ raise ValueError("labels is only supported for noul questions")if t == "choice":# only None/"" mean "no description"; 0 and False are legitimate criterion valuesreturn [k if v is None or v == "" else "%s: %s" % (k, render_criterion(v)) for k, v in crit.items()]if t == "score":return ["level %d: %s" % (i, render_criterion(c)) for i, c in enumerate(crit)]crit = crit or {}+ false_label, true_label = _resolve_noul_labels(q.get("labels"))false_crit, true_crit = crit.get("false"), crit.get("true")return [- "false: " + (render_criterion(false_crit) if false_crit not in (None, "") else "no, the statement does not hold"),- "true: " + (render_criterion(true_crit) if true_crit not in (None, "") else "yes, the statement holds"),+ false_label + ": "+ + (render_criterion(false_crit) if false_crit not in (None, "") else "no, the statement does not hold"),+ true_label + ": "+ + (render_criterion(true_crit) if true_crit not in (None, "") else "yes, the statement holds"),]diff --git a/tests/test_criteria.py b/tests/test_criteria.pyindex 583de24b..791da443 100644--- a/tests/test_criteria.py+++ b/tests/test_criteria.py@@ -36,6 +36,14 @@ def check_true(name, cond, detail=""):FAIL.append("%s %s" % (name, detail))+def check_raises(name, fn, message):+ try:+ fn()+ FAIL.append("%s: should have raised ValueError" % name)+ except ValueError as e:+ check(name, str(e), message)++# --------------------------------------------------------------- render_criterioncheck("criterion/str passes through", render_criterion("phishing or scam"), "phishing or scam")check("criterion/dict -> json", render_criterion({"desc": "phishing"}), '{"desc": "phishing"}')@@ -86,17 +94,50 @@ check("ece/zero confidence has its proper weight",ece_score(np.array([0.0, 1.0]), np.array([1.0, 1.0])), 0.5)-# --------------------------------------------------------------- unchanged behaviour+# --------------------------------------------------------------- noul labelscheck("noul/default false text", render_options({"t": "noul", "ins": "x", "crit": None})[0],"false: no, the statement does not hold")check("noul/default true text", render_options({"t": "noul", "ins": "x", "crit": None})[1],"true: yes, the statement holds")+check("noul/explicit None labels use defaults",+ render_options({"t": "noul", "ins": "x", "crit": None, "labels": None}),+ ["false: no, the statement does not hold", "true: yes, the statement holds"])check("noul/string criteria still work",render_options({"t": "noul", "ins": "x", "crit": {"true": "yes it is", "false": "no"}}),["false: no", "true: yes it is"])+custom_labels = {"true": " A ", "false": " B "}+custom_question = {"t": "noul", "ins": "x", "crit": None, "labels": custom_labels}+check("noul/custom labels preserve false-then-true semantics", render_options(custom_question),+ ["B: no, the statement does not hold", "A: yes, the statement holds"])+check("noul/custom label input is not mutated", custom_labels, {"true": " A ", "false": " B "})++_label_error = "noul labels must map exactly 'false' and 'true' to distinct non-empty strings"+for name, labels in [+ ("not a dict", ["negative", "positive"]),+ ("missing true", {"false": "negative"}),+ ("extra key", {"false": "negative", "true": "positive", "other": "x"}),+ ("blank value", {"false": " ", "true": "positive"}),+ ("duplicate values", {"false": "same", "true": "same"}),+ ("non-string value", {"false": 0, "true": "positive"}),+]:+ check_raises("noul/invalid labels " + name,+ lambda labels=labels: render_options({"t": "noul", "ins": "x", "labels": labels}),+ _label_error)++for qtype, crit in [("choice", {"a": None, "b": None}), ("score", ["low", "high"])]:+ check_raises("%s/rejects labels" % qtype,+ lambda qtype=qtype, crit=crit: render_options(+ {"t": qtype, "ins": "x", "crit": crit,+ "labels": {"false": "B", "true": "A"}}),+ "labels is only supported for noul questions")++# --------------------------------------------------------------- unchanged choice and score behaviourcheck("choice/string criteria still work",render_options({"t": "choice", "ins": "x", "crit": {"a": "first", "b": None}}),["a: first", "b"])+check("choice/boolean-word labels are not rewritten",+ render_options({"t": "choice", "ins": "x", "crit": {"true": "yes", "false": "no"}}),+ ["true: yes", "false: no"])check("score/string criteria still work",render_options({"t": "score", "ins": "x", "crit": ["low", "high"]}),["level 0: low", "level 1: high"])@@ -113,6 +154,27 @@ parsed = json.loads(render_options({"t": "noul", "ins": "x", "crit": {"true": {"a": 1}, "false": {"b": 2}}})[1].split("true: ", 1)[1])check("emitted json round-trips", parsed, {"a": 1})+# public labels reach the renderer through Agent._to_internal without changing caller data+from laya.agent import Agent # noqa: E402++public_labels = {"true": "A", "false": "B"}+public_question = {"type": "noul", "instructions": "Is this true?", "labels": public_labels}+internal = Agent._to_internal(public_question)+check("agent/forwards noul labels", internal["labels"], public_labels)+check("agent/forwarded labels reach renderer", render_options(internal),+ ["B: no, the statement does not hold", "A: yes, the statement holds"])+check("agent/leaves public question unchanged", public_question,+ {"type": "noul", "instructions": "Is this true?", "labels": {"true": "A", "false": "B"}})++boolean_criteria = {True: "yes", False: "no"}+boolean_question = {"type": "noul", "instructions": "Is this true?", "criteria": boolean_criteria,+ "labels": {"false": "B", "true": "A"}}+boolean_internal = Agent._to_internal(boolean_question)+check("agent/custom labels keep boolean criteria normalization", boolean_internal["crit"],+ {"true": "yes", "false": "no"})+check("agent/boolean criteria render with custom labels", render_options(boolean_internal), ["B: no", "A: yes"])+check("agent/leaves boolean criteria unchanged", boolean_criteria, {True: "yes", False: "no"})+# --------------------------------------------------------------- CPU-fallback warning (#9 follow-up)# The warning must fire only when a fallback actually happened -- not merely because the machine@@ -207,8 +269,18 @@ for label, qdef in [("score with an empty list", {"type": "score", "instructions": "How urgent?", "criteria": []}),("score with a dict of levels", {"type": "score", "instructions": "How urgent?","criteria": {"low": "no pressure", "high": "blocking"}}),+ ("choice with labels", {"type": "choice", "instructions": "Which team?",+ "criteria": ["billing", "tech"],+ "labels": {"false": "B", "true": "A"}}),+ ("score with labels", {"type": "score", "instructions": "How urgent?",+ "criteria": ["low", "high"],+ "labels": {"false": "B", "true": "A"}}),("noul with list criteria", {"type": "noul", "instructions": "Is it spam?", "criteria": ["a", "b"]}),("noul with string criteria", {"type": "noul", "instructions": "Is it spam?", "criteria": "spam?"}),+ ("noul with incomplete labels", {"type": "noul", "instructions": "Is it spam?",+ "labels": {"true": "A"}}),+ ("noul with duplicate labels", {"type": "noul", "instructions": "Is it spam?",+ "labels": {"false": "A", "true": "A"}}),("unknown type", {"type": "bool", "instructions": "Is it spam?"}),("missing type", {"instructions": "Is it spam?"}),("no instructions", {"type": "noul"}),@@ -256,6 +328,9 @@ GOOD = {"noul": {"type": "noul", "instructions": "Does the sender want a reply?"},"noul with criteria": {"type": "noul", "instructions": "Is it phishing?","criteria": {"true": "phishing", "false": "legitimate"}},+ "noul with labels": {"type": "noul", "instructions": "Is it phishing?",+ "criteria": {"true": "phishing", "false": "legitimate"},+ "labels": {"false": "B", "true": "A"}},"non-string instructions": {"type": "noul", "instructions": {"asks": "for a refund"}},}out = agent.system_one(STATE, GOOD)@@ -275,6 +350,9 @@ check_true("good/noul is a probability", 0.0 <= out["answers"]["noul"]["noul"] <check_true("good/noul with criteria is a probability",0.0 <= out["answers"]["noul with criteria"]["noul"] <= 1.0,str(out["answers"]["noul with criteria"]))+check_true("good/noul with labels is a probability",+ 0.0 <= out["answers"]["noul with labels"]["noul"] <= 1.0,+ str(out["answers"]["noul with labels"]))check("good/usage has no output tokens", out["usage"]["output_tokens"], 0)check_true("good/usage counted input tokens", out["usage"]["input_tokens"] > 0, str(out["usage"]))diff --git a/tests/test_local_e2e.py b/tests/test_local_e2e.pyindex 5f685314..20556bb5 100644--- a/tests/test_local_e2e.py+++ b/tests/test_local_e2e.py@@ -1,8 +1,9 @@"""End-to-end local test: real weights, real forward passes.-Covers the two things routing is supposed to buy us:+Covers three integration properties:1. non-English input reaches a checkpoint that can actually read it2. the shipped application presets still behave on English+ 3. a noul label override preserves false/true polarity on the English checkpointRun: python3 tests/test_local_e2e.py [model_root]Defaults to ~/laya_models, expecting laya/, laya-multilingual/, laya-typed-decisions/.@@ -75,6 +76,31 @@ CATS = {"billing": "invoices, payments, refunds", "technical": "bugs, outages, i"sales": "pricing, demos, new purchases", "hr": "hiring, leave, payroll"}QD = {"dept": {"type": "choice", "instructions": "Which team should handle `message`?", "criteria": CATS},"refund": {"type": "noul", "instructions": "Does the customer ask for money back?"}}+REVIEWS = [+ ("positive", {"body": "This product is excellent quality - six months in and not a single problem."}, True),+ ("negative", {"body": "It arrived broken and nobody answers when I contact support."}, False),+]+REVIEW_Q = {+ "plain": {"type": "noul", "instructions": "Is this review positive?",+ "labels": {"false": "B", "true": "A"}},+ "rich": {+ "type": "noul",+ "instructions": "Is this review positive?",+ "criteria": {"true": "the review is positive", "false": "the review is negative"},+ "labels": {"false": "B", "true": "A"},+ },+}+++def check_noul_label_override(agent, checkpoint):+ for polarity, state, want_true in REVIEWS:+ answers = agent.predict(state, REVIEW_Q)["answers"]+ for qid in REVIEW_Q:+ probability = answers[qid]["noul"]+ ok("noul labels/%s/%s/%s" % (checkpoint, polarity, qid),+ (probability > 0.5) == want_true, "P(true)=%.4f" % probability)++BILLING = [("english", "I was charged twice for invoice 4411, please refund it today."),("german", "Ich wurde zweimal fuer Rechnung 4411 belastet, bitte erstatten Sie den Betrag."),@@ -101,6 +127,7 @@ head("3. English checkpoint on the same non-English inputs (why routing matters)ml_only = {l: t for l, t in BILLING if l in ("hindi", "japanese", "chinese", "russian")}del mlen = laya.load(LOCAL["english"], device=DEVICE)+check_noul_label_override(en, "english")en_correct = 0for label, text in ml_only.items():a = en.predict({"message": text}, QD)["answers"]