Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(common): the state budget is max_len - head_len - 1, not max_len - head_max_len

Parswanadh33cd51f6f058
diff --git a/README.md b/README.md
index 4aadb0c4..070d7fcc 100644
--- a/README.md
+++ b/README.md
@@ -1118,9 +1118,10 @@ failure; it does not establish calibrated confidence.
The positive control passed on both checkpoints. These are narrow cancellation examples, not
evidence that every negated state fails. Validate the exact checkpoint and wording you serve;
using semantic keys alone does not avoid this failure.
-* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget (`max_len - head_max_len`):
- * `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).
- * `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
+* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length:** the renderer fills it only as far as the question and its option descriptions need, and `build_sequence` sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The cap-based figure `max_len - head_max_len` is therefore a **lower bound** on the room a request really has, and at the shipped defaults it understates that room substantially:
+ * `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped question renders a **48-token** head, leaving **463 tokens for state**, not 320.
+ * `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`). The shipped question renders a **45-token** head, leaving **978 tokens for state**, not 768 (mmBERT-base encoder supports up to 8,192 with RoPE).
+ Because the head grows with the question and its option descriptions, the room a given request has is `max_len - head_len - 1` for *that* request. Sizing a state to the cap-based figure is safe but wasteful: it truncates earlier than the library would and drops state the model would otherwise have read. The cap-based figure is never an overestimate, since `head_len <= head_max_len + 3`.
At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.
2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
diff --git a/tests/test_state_budget.py b/tests/test_state_budget.py
new file mode 100644
index 00000000..109a2375
--- /dev/null
+++ b/tests/test_state_budget.py
@@ -0,0 +1,156 @@
+"""Regression: the state budget is sized from the head actually built, not from `head_max_len`.
+
+`head_max_len` is a cap. `build_sequence` fills the option prompt budget only as far as the
+question and its option descriptions need, then sizes the state as `max_len - head_len - 1`
+from the head it actually built. The README used to describe the remaining state budget as
+`max_len - head_max_len`, which is a lower bound rather than the real figure: with the shipped
+question it understates the room by 143 tokens on `laya` (463 vs 320) and 210 on
+`laya-multilingual` (978 vs 768).
+
+This suite pins the relationship, so a future change to either the renderer or the README
+cannot silently desynchronise them. The last two checks are the ones that would have caught
+the documentation defect; they use the real checkpoints when present and skip otherwise.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from laya.common import build_sequence # noqa: E402
+
+PASS, FAIL = [], []
+
+# The same 4-option question the README's token-budget section walks through.
+Q = {"t": "choice", "ins": "Which department should handle this request?",
+ "crit": {"billing": "invoices, payments, refunds",
+ "technical": "bugs, outages, system errors",
+ "sales": "pricing, new contracts, plan upgrades",
+ "other": "everything else"}}
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s: got %r, want %r" % (name, got, want))
+
+
+class _FakeTok:
+ """Deterministic stand-in: one id per word, so head length is predictable."""
+ cls_token_id, sep_token_id, mask_token_id, pad_token_id = 0, 1, 4, 2
+ mask_token = "[MASK]"
+
+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
+ ids = [10 + (len(w) % 90) for w in text.split() if w]
+ if truncation and max_length:
+ ids = ids[:max_length]
+ return {"input_ids": ids}
+
+ def id_of(self, word):
+ """The single id this tokenizer assigns to a one-word string."""
+ return self(" " + word)["input_ids"][0]
+
+
+def _head_block(tok, q, head_max_len):
+ """Length of the built prompt the state is appended to, excluding the trailing [SEP]."""
+ seq, _ = build_sequence(tok, "", q, max_len=10 ** 6, head_max_len=head_max_len)
+ return len(seq) - 1
+
+
+def _kept_state(tok, state, q, max_len, head_max_len):
+ head = _head_block(tok, q, head_max_len)
+ seq, _ = build_sequence(tok, state, q, max_len=max_len, head_max_len=head_max_len)
+ return seq[head:-1]
+
+
+def test_head_never_exceeds_the_cap_by_more_than_the_delimiters():
+ """head_len <= head_max_len + 3 (CLS plus a [SEP] on each side of the marker block)."""
+ tok = _FakeTok()
+ for head_max_len in (32, 64, 192, 256, 512):
+ head = _head_block(tok, Q, head_max_len)
+ assert head <= head_max_len + 3, (
+ "head=%d exceeds cap %d (+3)" % (head, head_max_len))
+
+
+def test_state_budget_is_sized_from_the_actual_head():
+ """A state longer than the room is cut to exactly `max_len - head_len - 1`."""
+ tok = _FakeTok()
+ max_len, head_max_len = 512, 192
+ head = _head_block(tok, Q, head_max_len)
+ assert head < head_max_len, "fixture assumes a question that does not fill the cap"
+
+ room = max_len - head - 1
+ state = " ".join("w%d" % i for i in range(room + 200))
+ kept = _kept_state(tok, state, Q, max_len, head_max_len)
+ check("kept state length is max_len - head_len - 1", len(kept), room)
+ check("room exceeds the cap-based figure", room > max_len - head_max_len, True)
+
+
+def test_cap_based_figure_understates_the_room():
+ """A state of exactly `max_len - head_max_len` tokens is NOT truncated.
+
+ This is the assertion the README's old wording contradicts: it told a caller that
+ `max_len - head_max_len` was all the state they had.
+ """
+ tok = _FakeTok()
+ max_len, head_max_len = 512, 192
+ documented = max_len - head_max_len # 320 under the old wording
+ head = _head_block(tok, Q, head_max_len)
+ real_room = max_len - head - 1
+
+ assert real_room > documented, (
+ "fixture assumes the cap-based figure understates the room")
+
+ marker = "UNIQUETAILMARKER"
+ state = " ".join("w%d" % i for i in range(documented - 3)) + " " + marker
+ kept = _kept_state(tok, state, Q, max_len, head_max_len)
+ assert tok.id_of(marker) in kept, (
+ "a %d-token state (the cap-based figure) must survive a %d budget; "
+ "it was truncated, so the room is smaller than the README claims"
+ % (documented, max_len))
+ check("full documented-size state survives", len(kept), documented - 2)
+
+
+def test_shipped_checkpoints_match_the_documented_room():
+ """With each checkpoint's own tokenizer, the state room is the figure the README quotes.
+
+ Skipped when the checkpoints are not present (CI runs without weights).
+ """
+ lab = os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")
+ cases = [("english", 512, 192, 48, 463), ("multilingual", 1024, 256, 45, 978)]
+ try:
+ from transformers import AutoTokenizer
+ except Exception:
+ print(" (skipped: transformers unavailable)")
+ return
+ checked = 0
+ for name, max_len, head_max_len, want_head, want_room in cases:
+ path = os.path.join(lab, name, "tokenizer")
+ if not os.path.isdir(path):
+ print(" (skipped %s: no tokenizer at %s)" % (name, path))
+ continue
+ tok = AutoTokenizer.from_pretrained(path)
+ head = _head_block(tok, Q, head_max_len)
+ check("%s head length" % name, head, want_head)
+ check("%s state room" % name, max_len - head - 1, want_room)
+ checked += 1
+ if not checked:
+ print(" (skipped: no checkpoints under %s)" % lab)
+
+
+if __name__ == "__main__":
+ for fn in (test_head_never_exceeds_the_cap_by_more_than_the_delimiters,
+ test_state_budget_is_sized_from_the_actual_head,
+ test_cap_based_figure_understates_the_room,
+ test_shipped_checkpoints_match_the_documented_room):
+ try:
+ fn()
+ except AssertionError as e:
+ FAIL.append("%s: %s" % (fn.__name__, e))
+ else:
+ PASS.append(fn.__name__)
+
+ print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+ for f in FAIL:
+ print(" FAIL " + f)
+ sys.exit(1 if FAIL else 0)