Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
docs(common): correct the direction claim, and make the README check real
Parswanadhb19118bade31
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex d4f38072..d154ac2b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -102,6 +102,7 @@ jobs:python tests/test_batch.pypython tests/test_predict_long.pypython tests/test_attention_dynamic_shapes.py+ python tests/test_state_budget.pypython tests/test_hooks.pypython tests/test_hooks_api.pypython tests/test_structured.pydiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex c5356a86..95765b4c 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -43,6 +43,7 @@ jobs:python tests/test_criteria.pypython tests/test_batch.pypython tests/test_predict_long.py+ python tests/test_state_budget.pypython tests/test_hooks.pypython tests/test_hooks_api.pypython tests/test_onnx_lang_parity.pydiff --git a/README.md b/README.mdindex 070d7fcc..51ecf948 100644--- a/README.md+++ b/README.md@@ -1118,10 +1118,21 @@ failure; it does not establish calibrated confidence.The positive control passed on both checkpoints. These are narrow cancellation examples, notevidence that every negated state fails. Validate the exact checkpoint and wording you serve;using semantic keys alone does not avoid this failure.-* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length:** the renderer fills it only as far as the question and its option descriptions need, and `build_sequence` sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The cap-based figure `max_len - head_max_len` is therefore a **lower bound** on the room a request really has, and at the shipped defaults it understates that room substantially:- * `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped question renders a **48-token** head, leaving **463 tokens for state**, not 320.+* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length.** The renderer fills the option prompt budget only as far as the question and its option descriptions need, and `build_sequence` then sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The real room therefore depends on the question, not on the cap:+ * `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped department question below renders a **48-token** head, leaving **463 tokens for state**, not 320.* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`). The shipped question renders a **45-token** head, leaving **978 tokens for state**, not 768 (mmBERT-base encoder supports up to 8,192 with RoPE).- Because the head grows with the question and its option descriptions, the room a given request has is `max_len - head_len - 1` for *that* request. Sizing a state to the cap-based figure is safe but wasteful: it truncates earlier than the library would and drops state the model would otherwise have read. The cap-based figure is never an overestimate, since `head_len <= head_max_len + 3`.+ `max_len - head_max_len` is **not a safe figure in either direction**, so size a state to `max_len - head_len - 1` for the question you are actually asking:+ * For a question that does not fill the cap it **understates** the room, as above — conservative, but it discards state the model would have read.+ * For a high-cardinality question it **overstates** it. `per = max(4, (head_max_len - 16) // k)` floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly `head_max_len / 4` options the head grows *beyond* the cap. A state sized to the cap-based figure is then **truncated**, not merely conservative:++ | | `k` options | head | real room | cap-based figure |+ |---|---|---|---|---|+ | `laya`, `max_len=512` | 77 | 319 | **192** | 320 |+ | `laya`, `max_len=512` | 100 | 411 | **100** | 320 |+ | `laya-multilingual`, `max_len=1024` | 100 | 411 | **612** | 768 |++ The crossover is at `k ≈ head_max_len / 4` and the gap widens from there; `MAX_CHOICE_OPTIONS = 100` in `laya/serve.py` puts the overstating regime within reach over HTTP.+At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.diff --git a/tests/test_state_budget.py b/tests/test_state_budget.pyindex 109a2375..97156f42 100644--- a/tests/test_state_budget.py+++ b/tests/test_state_budget.py@@ -2,23 +2,30 @@`head_max_len` is a cap. `build_sequence` fills the option prompt budget only as far as thequestion and its option descriptions need, then sizes the state as `max_len - head_len - 1`-from the head it actually built. The README used to describe the remaining state budget as-`max_len - head_max_len`, which is a lower bound rather than the real figure: with the shipped-question it understates the room by 143 tokens on `laya` (463 vs 320) and 210 on-`laya-multilingual` (978 vs 768).--This suite pins the relationship, so a future change to either the renderer or the README-cannot silently desynchronise them. The last two checks are the ones that would have caught-the documentation defect; they use the real checkpoints when present and skip otherwise.+from the head it actually built.++`max_len - head_max_len` is wrong in *both* directions, which is what the README now says:++* a question that does not fill the cap -> the cap-based figure **understates** the room+ (conservative, but it discards state the model would have read);+* a high-cardinality question -> it **overstates** it. `per = max(4, (head_max_len - 16) // k)`+ floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly+ `head_max_len / 4` options the head grows beyond the cap, and a state sized to the cap-based+ figure is **truncated** rather than merely conservative.++This suite pins the relationship, and — when the checkpoints are present — reads the figures+back out of `README.md` and checks them against a live measurement, so the renderer, the+documentation and this file cannot drift apart silently."""import os+import reimport syssys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))from laya.common import build_sequence # noqa: E402-PASS, FAIL = [], []+PASS, FAIL, SKIPPED = [], [], []# The same 4-option question the README's token-budget section walks through.Q = {"t": "choice", "ins": "Which department should handle this request?",@@ -28,6 +35,25 @@ Q = {"t": "choice", "ins": "Which department should handle this request?","other": "everything else"}}+class _Skip(Exception):+ pass+++def skip(reason):+ """A skip that is visible in both runners.++ Under pytest this raises `pytest.skip`, so `-rs` reports it. As a plain script it raises+ `_Skip`, which the runner prints and counts -- a bare `return` passes vacuously and no+ runner flag reveals it.+ """+ print(" SKIP: " + reason)+ SKIPPED.append(reason)+ if __name__ != "__main__":+ import pytest+ pytest.skip(reason)+ raise _Skip(reason)++def check(name, got, want):if got == want:PASS.append(name)@@ -47,110 +73,165 @@ class _FakeTok:return {"input_ids": ids}def id_of(self, word):- """The single id this tokenizer assigns to a one-word string."""return self(" " + word)["input_ids"][0]-def _head_block(tok, q, head_max_len):+def _head_len(tok, q, head_max_len):"""Length of the built prompt the state is appended to, excluding the trailing [SEP]."""seq, _ = build_sequence(tok, "", q, max_len=10 ** 6, head_max_len=head_max_len)return len(seq) - 1def _kept_state(tok, state, q, max_len, head_max_len):- head = _head_block(tok, q, head_max_len)+ head = _head_len(tok, q, head_max_len)seq, _ = build_sequence(tok, state, q, max_len=max_len, head_max_len=head_max_len)return seq[head:-1]-def test_head_never_exceeds_the_cap_by_more_than_the_delimiters():- """head_len <= head_max_len + 3 (CLS plus a [SEP] on each side of the marker block)."""- tok = _FakeTok()- for head_max_len in (32, 64, 192, 256, 512):- head = _head_block(tok, Q, head_max_len)- assert head <= head_max_len + 3, (- "head=%d exceeds cap %d (+3)" % (head, head_max_len))+def _many_options(k, desc="a short description of this option"):+ return {"t": "choice", "ins": "Which intent is this?",+ "crit": {"intent_%d" % i: desc for i in range(k)}}+++def _models_dir():+ return os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")+++def _real_tok(name):+ """The checkpoint's tokenizer, or None. Callers decide whether absence is a skip."""+ path = os.path.join(_models_dir(), name, "tokenizer")+ if not os.path.isdir(path):+ return None+ from transformers import AutoTokenizer+ return AutoTokenizer.from_pretrained(path)+# --------------------------------------------------------------------------- invariant checks+def test_state_budget_is_sized_from_the_actual_head():"""A state longer than the room is cut to exactly `max_len - head_len - 1`."""tok = _FakeTok()max_len, head_max_len = 512, 192- head = _head_block(tok, Q, head_max_len)+ head = _head_len(tok, Q, head_max_len)assert head < head_max_len, "fixture assumes a question that does not fill the cap"room = max_len - head - 1state = " ".join("w%d" % i for i in range(room + 200))kept = _kept_state(tok, state, Q, max_len, head_max_len)check("kept state length is max_len - head_len - 1", len(kept), room)- check("room exceeds the cap-based figure", room > max_len - head_max_len, True)--def test_cap_based_figure_understates_the_room():- """A state of exactly `max_len - head_max_len` tokens is NOT truncated.- This is the assertion the README's old wording contradicts: it told a caller that- `max_len - head_max_len` was all the state they had.- """+def test_dropping_the_minus_one_is_caught():+ """The `-1` is load-bearing: one token more than the room must lose its tail."""tok = _FakeTok()max_len, head_max_len = 512, 192- documented = max_len - head_max_len # 320 under the old wording- head = _head_block(tok, Q, head_max_len)- real_room = max_len - head - 1+ head = _head_len(tok, Q, head_max_len)+ marker = "TAILMARKER"+ exact = max_len - head - 1 # the true room- assert real_room > documented, (- "fixture assumes the cap-based figure understates the room")+ state = " ".join("w%d" % i for i in range(exact - 1)) + " " + marker+ kept = _kept_state(tok, state, Q, max_len, head_max_len)+ check("a state of exactly the room is kept whole", len(kept), exact)+ assert tok.id_of(marker) in kept, "the tail marker must survive an exactly-sized state"++ over = " ".join("w%d" % i for i in range(exact)) + " " + marker+ kept_over = _kept_state(tok, over, Q, max_len, head_max_len)+ assert tok.id_of(marker) not in kept_over, (+ "a state one token over the room must lose its tail, so the room is max_len-head_len-1 "+ "and not max_len-head_len")+++def test_cap_based_figure_understates_a_short_question():+ """A state of exactly `max_len - head_max_len` tokens survives for a short question."""+ tok = _FakeTok()+ max_len, head_max_len = 512, 192+ documented = max_len - head_max_len+ head = _head_len(tok, Q, head_max_len)+ assert max_len - head - 1 > documented, "fixture assumes the cap-based figure understates"marker = "UNIQUETAILMARKER"state = " ".join("w%d" % i for i in range(documented - 3)) + " " + markerkept = _kept_state(tok, state, Q, max_len, head_max_len)assert tok.id_of(marker) in kept, (- "a %d-token state (the cap-based figure) must survive a %d budget; "- "it was truncated, so the room is smaller than the README claims"- % (documented, max_len))- check("full documented-size state survives", len(kept), documented - 2)+ "a %d-token state must survive a %d budget for this question" % (documented, max_len))-def test_shipped_checkpoints_match_the_documented_room():- """With each checkpoint's own tokenizer, the state room is the figure the README quotes.+def test_cap_based_figure_overstates_a_high_cardinality_question():+ """Past ~`head_max_len / 4` options the head exceeds the cap, so the cap-based figure lies.- Skipped when the checkpoints are not present (CI runs without weights).+ This is the direction that matters: a caller sizing a state to `max_len - head_max_len`+ is truncated, not merely conservative."""- lab = os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")- cases = [("english", 512, 192, 48, 463), ("multilingual", 1024, 256, 45, 978)]- try:- from transformers import AutoTokenizer- except Exception:- print(" (skipped: transformers unavailable)")- return+ tok = _FakeTok()+ max_len, head_max_len = 512, 192+ head = _head_len(tok, _many_options(100), head_max_len)+ assert head > head_max_len, (+ "a 100-option question must build a head beyond the %d cap, got %d" % (head_max_len, head))+ check("cap-based figure overstates the room at k=100",+ max_len - head_max_len > max_len - head - 1, True)+++# --------------------------------------------------------------------- documentation agreement++def _readme_quotes():+ """The figures README.md states, parsed out of the token-budget bullet."""+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+ with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:+ text = fh.read()+ heads = {int(m) for m in re.findall(r"\*\*(\d+)-token\*\* head", text)}+ rooms = {int(m) for m in re.findall(r"\*\*(\d+) tokens for state\*\*", text)}+ return heads, rooms+++def test_readme_quotes_the_measured_figures():+ """README's head/room figures equal a live measurement, for each shipped checkpoint.++ This is what makes the suite stop the README and the renderer desynchronising; without it+ the file only guards the renderer.+ """+ heads, rooms = _readme_quotes()+ assert heads and rooms, "could not parse the head/room figures out of README.md"+checked = 0- for name, max_len, head_max_len, want_head, want_room in cases:- path = os.path.join(lab, name, "tokenizer")- if not os.path.isdir(path):- print(" (skipped %s: no tokenizer at %s)" % (name, path))+ for name, max_len, head_max_len in (("english", 512, 192), ("multilingual", 1024, 256)):+ try:+ tok = _real_tok(name)+ except Exception as e: # transformers absent or unloadable+ skip("%s: tokenizer unavailable (%s)" % (name, e))+ continue+ if tok is None:+ skip("%s: no tokenizer under %s" % (name, _models_dir()))continue- tok = AutoTokenizer.from_pretrained(path)- head = _head_block(tok, Q, head_max_len)- check("%s head length" % name, head, want_head)- check("%s state room" % name, max_len - head - 1, want_room)+ head = _head_len(tok, Q, head_max_len)+ room = max_len - head - 1+ assert head in heads, (+ "README does not quote a %d-token head for %s (quotes %s)" % (head, name, sorted(heads)))+ assert room in rooms, (+ "README does not quote a %d-token state room for %s (quotes %s)"+ % (room, name, sorted(rooms)))checked += 1if not checked:- print(" (skipped: no checkpoints under %s)" % lab)+ skip("no checkpoints under %s" % _models_dir())if __name__ == "__main__":- for fn in (test_head_never_exceeds_the_cap_by_more_than_the_delimiters,- test_state_budget_is_sized_from_the_actual_head,- test_cap_based_figure_understates_the_room,- test_shipped_checkpoints_match_the_documented_room):+ for fn in (test_state_budget_is_sized_from_the_actual_head,+ test_dropping_the_minus_one_is_caught,+ test_cap_based_figure_understates_a_short_question,+ test_cap_based_figure_overstates_a_high_cardinality_question,+ test_readme_quotes_the_measured_figures):try:fn()except AssertionError as e:FAIL.append("%s: %s" % (fn.__name__, e))+ except _Skip:+ passelse:PASS.append(fn.__name__)- print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+ print("\n%d passed, %d failed, %d skipped" % (len(PASS), len(FAIL), len(SKIPPED)))for f in FAIL:print(" FAIL " + f)+ for s in SKIPPED:+ print(" SKIP " + s)sys.exit(1 if FAIL else 0)