Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): allow custom noul labels (#163)

Wasiq Nauman158398a7440e
diff --git a/README.md b/README.md
index 677a4869..234cf61f 100644
--- a/README.md
+++ b/README.md
@@ -479,6 +479,31 @@ triage = agent.predict({"message": "My payment failed twice"}, laya.triage_quest
| **`score`** | Expected level on ordinal rubric, distribution, confidence | Frustration level, ticket urgency, harm severity |
| **`noul`** | Calibrated probability P(true) from 0.0 to 1.0 | Phishing detection, spam filtering, jailbreak detection, churn risk |
+`noul` always scores two semantic slots in `[false, true]` order and returns the probability of
+the second slot. For compatibility, those slots are shown to the model as `false` and `true` by
+default. The optional `labels` mapping overrides only that model-facing text without changing the
+returned meaning:
+
+```python
+question = {
+ "type": "noul",
+ "instructions": "Is this review positive?",
+ "criteria": {
+ "false": "the review is negative",
+ "true": "the review is positive",
+ },
+ "labels": {
+ "false": "B",
+ "true": "A",
+ },
+}
+```
+
+The `labels` mapping is optional. It must contain exactly the string keys `false` and `true`,
+whose values must be distinct non-empty strings. Mapping order does not matter, and the returned
+`noul` value is still P(true). Label sensitivity varies by checkpoint and state, so validate any
+override on your own data rather than treating `A`/`B` as a universal fix.
+
---
## Benchmarks
@@ -610,6 +635,10 @@ failure; it does not establish calibrated confidence.
against a 0.318 random baseline and a 0.461 majority-class baseline. The 0.766 figure comes
from the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to
specialise, not a zero-shot decision engine.
+* **Avoid boolean-word labels in `choice` questions.** Choice keys are rendered verbatim, and the
+ current checkpoints can follow labels such as `true`/`false` or `yes`/`no` instead of the option
+ descriptions. Use semantic labels or opaque labels such as `A`/`B`, and validate them on the
+ checkpoint and states you serve.
* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget (`max_len - head_max_len`):
* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
@@ -647,7 +676,16 @@ result["shortlist"]["intent"]["labels"] # the top 20 labels sent to the model
[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.
* Ordinal `score` questions are the weakest primitive (SST-5 0.372).
-* **`noul` can follow its option labels instead of the state, most strongly on `laya` (English).** `noul` renders its two options as `false:` / `true:`, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Until a retrained checkpoint lands, check `noul` answers on your own data. If they look stuck, ask the same question as a two-option `choice` with neutral keys and your yes/no wording as the descriptions:
+* **`noul` can follow its option labels instead of the state, most strongly on `laya` (English).** `noul` renders its two options as `false:` / `true:` by default, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Until a retrained checkpoint lands, check `noul` answers on your own data. You can override the model-facing pair while keeping the `noul` result as P(true):
+
+ ```python
+ {"type": "noul", "instructions": "Is this review positive?",
+ "criteria": {"true": "yes, the review is positive", "false": "no, the review is negative"},
+ "labels": {"true": "A", "false": "B"}}
+ ```
+
+ Label sensitivity varies by checkpoint and state, so validate the override on your own data. A
+ two-option `choice` with neutral keys remains another workaround:
```python
{"type": "choice", "instructions": "Is this review positive?",
diff --git a/laya/agent.py b/laya/agent.py
index 21892ddc..e5e33728 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -18,6 +18,7 @@ from .common import (
clamp_temperature,
collate_items,
confidence_from_probs,
+ _resolve_noul_labels,
render_options,
temp_bucket,
)
@@ -385,6 +386,13 @@ class Agent:
elif crit is not None and not isinstance(crit, dict):
raise ValueError("question %r: a noul question takes 'criteria' as a dict with optional "
"'true'/'false' descriptions, or omits it" % (qid,))
+ if "labels" in qdef:
+ if t != "noul":
+ raise ValueError("question %r: 'labels' is only supported for noul questions" % (qid,))
+ try:
+ _resolve_noul_labels(qdef["labels"])
+ except ValueError as e:
+ raise ValueError("question %r: %s" % (qid, e)) from e
@staticmethod
def _to_internal(qdef: Dict) -> Dict:
@@ -403,7 +411,10 @@ class Agent:
# on the English checkpoint one German question answered noul=0.1652 as a dict
# and noul=0.2650 as the identical plain string.
ins = json.dumps(ins, ensure_ascii=False)
- return {"t": t, "ins": ins, "crit": crit}
+ q = {"t": t, "ins": ins, "crit": crit}
+ if "labels" in qdef:
+ q["labels"] = qdef["labels"]
+ return q
@torch.no_grad()
def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
@@ -414,7 +425,13 @@ class Agent:
questions: Dictionary mapping question_id -> question definition.
- choice: {"type": "choice", "instructions": "...", "criteria": {"optA": "...", ...}}
- score: {"type": "score", "instructions": "...", "criteria": ["lvl0", "lvl1", ...]}
- - noul: {"type": "noul", "instructions": "...", "criteria": {"true": "...", "false": "..."}}
+ - noul: {"type": "noul", "instructions": "...",
+ "criteria": {"false": "...", "true": "..."},
+ "labels": {"false": "B", "true": "A"}}
+
+ Noul criteria and labels are optional. Labels only control the text shown to the
+ model; their keys retain false/true semantics, and the returned `noul` value is
+ always P(true). Labels default to false/true for compatibility.
Returns:
Dictionary with answers, probabilities, calibrated confidence, and token usage.
diff --git a/laya/common.py b/laya/common.py
index ab2a303d..d73e248b 100644
--- a/laya/common.py
+++ b/laya/common.py
@@ -11,6 +11,7 @@ from torch.utils.checkpoint import checkpoint
QTYPES = {"choice": 0, "score": 1, "noul": 2}
QTYPE_NAMES = {v: k for k, v in QTYPES.items()}
+_DEFAULT_NOUL_LABELS = {"false": "false", "true": "true"}
def serialize_state(state: Union[str, dict, list]) -> str:
@@ -31,19 +32,38 @@ def render_criterion(value) -> str:
return json.dumps(value, ensure_ascii=False, separators=(", ", ": "), default=str)
+def _resolve_noul_labels(labels=None):
+ if labels is None:
+ labels = _DEFAULT_NOUL_LABELS
+ if not isinstance(labels, dict) or set(labels) != {"false", "true"}:
+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")
+ false_label, true_label = labels["false"], labels["true"]
+ if not isinstance(false_label, str) or not isinstance(true_label, str):
+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")
+ false_label, true_label = false_label.strip(), true_label.strip()
+ if not false_label or not true_label or false_label == true_label:
+ raise ValueError("noul labels must map exactly 'false' and 'true' to distinct non-empty strings")
+ return false_label, true_label
+
+
def render_options(q: Dict) -> List[str]:
- """Render option texts in label-index order. Noul is always [false, true]."""
+ """Render option texts in label-index order. Noul semantic order is always [false, true]."""
t, crit = q["t"], q.get("crit")
+ if t != "noul" and "labels" in q:
+ raise ValueError("labels is only supported for noul questions")
if t == "choice":
# only None/"" mean "no description"; 0 and False are legitimate criterion values
return [k if v is None or v == "" else "%s: %s" % (k, render_criterion(v)) for k, v in crit.items()]
if t == "score":
return ["level %d: %s" % (i, render_criterion(c)) for i, c in enumerate(crit)]
crit = crit or {}
+ false_label, true_label = _resolve_noul_labels(q.get("labels"))
false_crit, true_crit = crit.get("false"), crit.get("true")
return [
- "false: " + (render_criterion(false_crit) if false_crit not in (None, "") else "no, the statement does not hold"),
- "true: " + (render_criterion(true_crit) if true_crit not in (None, "") else "yes, the statement holds"),
+ false_label + ": "
+ + (render_criterion(false_crit) if false_crit not in (None, "") else "no, the statement does not hold"),
+ true_label + ": "
+ + (render_criterion(true_crit) if true_crit not in (None, "") else "yes, the statement holds"),
]
diff --git a/tests/test_criteria.py b/tests/test_criteria.py
index 583de24b..791da443 100644
--- a/tests/test_criteria.py
+++ b/tests/test_criteria.py
@@ -36,6 +36,14 @@ def check_true(name, cond, detail=""):
FAIL.append("%s %s" % (name, detail))
+def check_raises(name, fn, message):
+ try:
+ fn()
+ FAIL.append("%s: should have raised ValueError" % name)
+ except ValueError as e:
+ check(name, str(e), message)
+
+
# --------------------------------------------------------------- render_criterion
check("criterion/str passes through", render_criterion("phishing or scam"), "phishing or scam")
check("criterion/dict -> json", render_criterion({"desc": "phishing"}), '{"desc": "phishing"}')
@@ -86,17 +94,50 @@ check("ece/zero confidence has its proper weight",
ece_score(np.array([0.0, 1.0]), np.array([1.0, 1.0])), 0.5)
-# --------------------------------------------------------------- unchanged behaviour
+# --------------------------------------------------------------- noul labels
check("noul/default false text", render_options({"t": "noul", "ins": "x", "crit": None})[0],
"false: no, the statement does not hold")
check("noul/default true text", render_options({"t": "noul", "ins": "x", "crit": None})[1],
"true: yes, the statement holds")
+check("noul/explicit None labels use defaults",
+ render_options({"t": "noul", "ins": "x", "crit": None, "labels": None}),
+ ["false: no, the statement does not hold", "true: yes, the statement holds"])
check("noul/string criteria still work",
render_options({"t": "noul", "ins": "x", "crit": {"true": "yes it is", "false": "no"}}),
["false: no", "true: yes it is"])
+custom_labels = {"true": " A ", "false": " B "}
+custom_question = {"t": "noul", "ins": "x", "crit": None, "labels": custom_labels}
+check("noul/custom labels preserve false-then-true semantics", render_options(custom_question),
+ ["B: no, the statement does not hold", "A: yes, the statement holds"])
+check("noul/custom label input is not mutated", custom_labels, {"true": " A ", "false": " B "})
+
+_label_error = "noul labels must map exactly 'false' and 'true' to distinct non-empty strings"
+for name, labels in [
+ ("not a dict", ["negative", "positive"]),
+ ("missing true", {"false": "negative"}),
+ ("extra key", {"false": "negative", "true": "positive", "other": "x"}),
+ ("blank value", {"false": " ", "true": "positive"}),
+ ("duplicate values", {"false": "same", "true": "same"}),
+ ("non-string value", {"false": 0, "true": "positive"}),
+]:
+ check_raises("noul/invalid labels " + name,
+ lambda labels=labels: render_options({"t": "noul", "ins": "x", "labels": labels}),
+ _label_error)
+
+for qtype, crit in [("choice", {"a": None, "b": None}), ("score", ["low", "high"])]:
+ check_raises("%s/rejects labels" % qtype,
+ lambda qtype=qtype, crit=crit: render_options(
+ {"t": qtype, "ins": "x", "crit": crit,
+ "labels": {"false": "B", "true": "A"}}),
+ "labels is only supported for noul questions")
+
+# --------------------------------------------------------------- unchanged choice and score behaviour
check("choice/string criteria still work",
render_options({"t": "choice", "ins": "x", "crit": {"a": "first", "b": None}}),
["a: first", "b"])
+check("choice/boolean-word labels are not rewritten",
+ render_options({"t": "choice", "ins": "x", "crit": {"true": "yes", "false": "no"}}),
+ ["true: yes", "false: no"])
check("score/string criteria still work",
render_options({"t": "score", "ins": "x", "crit": ["low", "high"]}),
["level 0: low", "level 1: high"])
@@ -113,6 +154,27 @@ parsed = json.loads(render_options(
{"t": "noul", "ins": "x", "crit": {"true": {"a": 1}, "false": {"b": 2}}})[1].split("true: ", 1)[1])
check("emitted json round-trips", parsed, {"a": 1})
+# public labels reach the renderer through Agent._to_internal without changing caller data
+from laya.agent import Agent # noqa: E402
+
+public_labels = {"true": "A", "false": "B"}
+public_question = {"type": "noul", "instructions": "Is this true?", "labels": public_labels}
+internal = Agent._to_internal(public_question)
+check("agent/forwards noul labels", internal["labels"], public_labels)
+check("agent/forwarded labels reach renderer", render_options(internal),
+ ["B: no, the statement does not hold", "A: yes, the statement holds"])
+check("agent/leaves public question unchanged", public_question,
+ {"type": "noul", "instructions": "Is this true?", "labels": {"true": "A", "false": "B"}})
+
+boolean_criteria = {True: "yes", False: "no"}
+boolean_question = {"type": "noul", "instructions": "Is this true?", "criteria": boolean_criteria,
+ "labels": {"false": "B", "true": "A"}}
+boolean_internal = Agent._to_internal(boolean_question)
+check("agent/custom labels keep boolean criteria normalization", boolean_internal["crit"],
+ {"true": "yes", "false": "no"})
+check("agent/boolean criteria render with custom labels", render_options(boolean_internal), ["B: no", "A: yes"])
+check("agent/leaves boolean criteria unchanged", boolean_criteria, {True: "yes", False: "no"})
+
# --------------------------------------------------------------- CPU-fallback warning (#9 follow-up)
# The warning must fire only when a fallback actually happened -- not merely because the machine
@@ -207,8 +269,18 @@ for label, qdef in [
("score with an empty list", {"type": "score", "instructions": "How urgent?", "criteria": []}),
("score with a dict of levels", {"type": "score", "instructions": "How urgent?",
"criteria": {"low": "no pressure", "high": "blocking"}}),
+ ("choice with labels", {"type": "choice", "instructions": "Which team?",
+ "criteria": ["billing", "tech"],
+ "labels": {"false": "B", "true": "A"}}),
+ ("score with labels", {"type": "score", "instructions": "How urgent?",
+ "criteria": ["low", "high"],
+ "labels": {"false": "B", "true": "A"}}),
("noul with list criteria", {"type": "noul", "instructions": "Is it spam?", "criteria": ["a", "b"]}),
("noul with string criteria", {"type": "noul", "instructions": "Is it spam?", "criteria": "spam?"}),
+ ("noul with incomplete labels", {"type": "noul", "instructions": "Is it spam?",
+ "labels": {"true": "A"}}),
+ ("noul with duplicate labels", {"type": "noul", "instructions": "Is it spam?",
+ "labels": {"false": "A", "true": "A"}}),
("unknown type", {"type": "bool", "instructions": "Is it spam?"}),
("missing type", {"instructions": "Is it spam?"}),
("no instructions", {"type": "noul"}),
@@ -256,6 +328,9 @@ GOOD = {
"noul": {"type": "noul", "instructions": "Does the sender want a reply?"},
"noul with criteria": {"type": "noul", "instructions": "Is it phishing?",
"criteria": {"true": "phishing", "false": "legitimate"}},
+ "noul with labels": {"type": "noul", "instructions": "Is it phishing?",
+ "criteria": {"true": "phishing", "false": "legitimate"},
+ "labels": {"false": "B", "true": "A"}},
"non-string instructions": {"type": "noul", "instructions": {"asks": "for a refund"}},
}
out = agent.system_one(STATE, GOOD)
@@ -275,6 +350,9 @@ check_true("good/noul is a probability", 0.0 <= out["answers"]["noul"]["noul"] <
check_true("good/noul with criteria is a probability",
0.0 <= out["answers"]["noul with criteria"]["noul"] <= 1.0,
str(out["answers"]["noul with criteria"]))
+check_true("good/noul with labels is a probability",
+ 0.0 <= out["answers"]["noul with labels"]["noul"] <= 1.0,
+ str(out["answers"]["noul with labels"]))
check("good/usage has no output tokens", out["usage"]["output_tokens"], 0)
check_true("good/usage counted input tokens", out["usage"]["input_tokens"] > 0, str(out["usage"]))
diff --git a/tests/test_local_e2e.py b/tests/test_local_e2e.py
index 5f685314..20556bb5 100644
--- a/tests/test_local_e2e.py
+++ b/tests/test_local_e2e.py
@@ -1,8 +1,9 @@
"""End-to-end local test: real weights, real forward passes.
-Covers the two things routing is supposed to buy us:
+Covers three integration properties:
1. non-English input reaches a checkpoint that can actually read it
2. the shipped application presets still behave on English
+ 3. a noul label override preserves false/true polarity on the English checkpoint
Run: python3 tests/test_local_e2e.py [model_root]
Defaults to ~/laya_models, expecting laya/, laya-multilingual/, laya-typed-decisions/.
@@ -75,6 +76,31 @@ CATS = {"billing": "invoices, payments, refunds", "technical": "bugs, outages, i
"sales": "pricing, demos, new purchases", "hr": "hiring, leave, payroll"}
QD = {"dept": {"type": "choice", "instructions": "Which team should handle `message`?", "criteria": CATS},
"refund": {"type": "noul", "instructions": "Does the customer ask for money back?"}}
+REVIEWS = [
+ ("positive", {"body": "This product is excellent quality - six months in and not a single problem."}, True),
+ ("negative", {"body": "It arrived broken and nobody answers when I contact support."}, False),
+]
+REVIEW_Q = {
+ "plain": {"type": "noul", "instructions": "Is this review positive?",
+ "labels": {"false": "B", "true": "A"}},
+ "rich": {
+ "type": "noul",
+ "instructions": "Is this review positive?",
+ "criteria": {"true": "the review is positive", "false": "the review is negative"},
+ "labels": {"false": "B", "true": "A"},
+ },
+}
+
+
+def check_noul_label_override(agent, checkpoint):
+ for polarity, state, want_true in REVIEWS:
+ answers = agent.predict(state, REVIEW_Q)["answers"]
+ for qid in REVIEW_Q:
+ probability = answers[qid]["noul"]
+ ok("noul labels/%s/%s/%s" % (checkpoint, polarity, qid),
+ (probability > 0.5) == want_true, "P(true)=%.4f" % probability)
+
+
BILLING = [
("english", "I was charged twice for invoice 4411, please refund it today."),
("german", "Ich wurde zweimal fuer Rechnung 4411 belastet, bitte erstatten Sie den Betrag."),
@@ -101,6 +127,7 @@ head("3. English checkpoint on the same non-English inputs (why routing matters)
ml_only = {l: t for l, t in BILLING if l in ("hindi", "japanese", "chinese", "russian")}
del ml
en = laya.load(LOCAL["english"], device=DEVICE)
+check_noul_label_override(en, "english")
en_correct = 0
for label, text in ml_only.items():
a = en.predict({"message": text}, QD)["answers"]