Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(common): correct the direction claim, and make the README check real

Parswanadhb19118bade31
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d4f38072..d154ac2b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -102,6 +102,7 @@ jobs:
python tests/test_batch.py
python tests/test_predict_long.py
python tests/test_attention_dynamic_shapes.py
+ python tests/test_state_budget.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_structured.py
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index c5356a86..95765b4c 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -43,6 +43,7 @@ jobs:
python tests/test_criteria.py
python tests/test_batch.py
python tests/test_predict_long.py
+ python tests/test_state_budget.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_onnx_lang_parity.py
diff --git a/README.md b/README.md
index 070d7fcc..51ecf948 100644
--- a/README.md
+++ b/README.md
@@ -1118,10 +1118,21 @@ failure; it does not establish calibrated confidence.
The positive control passed on both checkpoints. These are narrow cancellation examples, not
evidence that every negated state fails. Validate the exact checkpoint and wording you serve;
using semantic keys alone does not avoid this failure.
-* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length:** the renderer fills it only as far as the question and its option descriptions need, and `build_sequence` sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The cap-based figure `max_len - head_max_len` is therefore a **lower bound** on the room a request really has, and at the shipped defaults it understates that room substantially:
- * `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped question renders a **48-token** head, leaving **463 tokens for state**, not 320.
+* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length.** The renderer fills the option prompt budget only as far as the question and its option descriptions need, and `build_sequence` then sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The real room therefore depends on the question, not on the cap:
+ * `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped department question below renders a **48-token** head, leaving **463 tokens for state**, not 320.
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`). The shipped question renders a **45-token** head, leaving **978 tokens for state**, not 768 (mmBERT-base encoder supports up to 8,192 with RoPE).
- Because the head grows with the question and its option descriptions, the room a given request has is `max_len - head_len - 1` for *that* request. Sizing a state to the cap-based figure is safe but wasteful: it truncates earlier than the library would and drops state the model would otherwise have read. The cap-based figure is never an overestimate, since `head_len <= head_max_len + 3`.
+ `max_len - head_max_len` is **not a safe figure in either direction**, so size a state to `max_len - head_len - 1` for the question you are actually asking:
+ * For a question that does not fill the cap it **understates** the room, as above — conservative, but it discards state the model would have read.
+ * For a high-cardinality question it **overstates** it. `per = max(4, (head_max_len - 16) // k)` floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly `head_max_len / 4` options the head grows *beyond* the cap. A state sized to the cap-based figure is then **truncated**, not merely conservative:
+
+ | | `k` options | head | real room | cap-based figure |
+ |---|---|---|---|---|
+ | `laya`, `max_len=512` | 77 | 319 | **192** | 320 |
+ | `laya`, `max_len=512` | 100 | 411 | **100** | 320 |
+ | `laya-multilingual`, `max_len=1024` | 100 | 411 | **612** | 768 |
+
+ The crossover is at `k ≈ head_max_len / 4` and the gap widens from there; `MAX_CHOICE_OPTIONS = 100` in `laya/serve.py` puts the overstating regime within reach over HTTP.
+
At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.
2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
diff --git a/tests/test_state_budget.py b/tests/test_state_budget.py
index 109a2375..97156f42 100644
--- a/tests/test_state_budget.py
+++ b/tests/test_state_budget.py
@@ -2,23 +2,30 @@
`head_max_len` is a cap. `build_sequence` fills the option prompt budget only as far as the
question and its option descriptions need, then sizes the state as `max_len - head_len - 1`
-from the head it actually built. The README used to describe the remaining state budget as
-`max_len - head_max_len`, which is a lower bound rather than the real figure: with the shipped
-question it understates the room by 143 tokens on `laya` (463 vs 320) and 210 on
-`laya-multilingual` (978 vs 768).
-
-This suite pins the relationship, so a future change to either the renderer or the README
-cannot silently desynchronise them. The last two checks are the ones that would have caught
-the documentation defect; they use the real checkpoints when present and skip otherwise.
+from the head it actually built.
+
+`max_len - head_max_len` is wrong in *both* directions, which is what the README now says:
+
+* a question that does not fill the cap -> the cap-based figure **understates** the room
+ (conservative, but it discards state the model would have read);
+* a high-cardinality question -> it **overstates** it. `per = max(4, (head_max_len - 16) // k)`
+ floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly
+ `head_max_len / 4` options the head grows beyond the cap, and a state sized to the cap-based
+ figure is **truncated** rather than merely conservative.
+
+This suite pins the relationship, and — when the checkpoints are present — reads the figures
+back out of `README.md` and checks them against a live measurement, so the renderer, the
+documentation and this file cannot drift apart silently.
"""
import os
+import re
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from laya.common import build_sequence # noqa: E402
-PASS, FAIL = [], []
+PASS, FAIL, SKIPPED = [], [], []
# The same 4-option question the README's token-budget section walks through.
Q = {"t": "choice", "ins": "Which department should handle this request?",
@@ -28,6 +35,25 @@ Q = {"t": "choice", "ins": "Which department should handle this request?",
"other": "everything else"}}
+class _Skip(Exception):
+ pass
+
+
+def skip(reason):
+ """A skip that is visible in both runners.
+
+ Under pytest this raises `pytest.skip`, so `-rs` reports it. As a plain script it raises
+ `_Skip`, which the runner prints and counts -- a bare `return` passes vacuously and no
+ runner flag reveals it.
+ """
+ print(" SKIP: " + reason)
+ SKIPPED.append(reason)
+ if __name__ != "__main__":
+ import pytest
+ pytest.skip(reason)
+ raise _Skip(reason)
+
+
def check(name, got, want):
if got == want:
PASS.append(name)
@@ -47,110 +73,165 @@ class _FakeTok:
return {"input_ids": ids}
def id_of(self, word):
- """The single id this tokenizer assigns to a one-word string."""
return self(" " + word)["input_ids"][0]
-def _head_block(tok, q, head_max_len):
+def _head_len(tok, q, head_max_len):
"""Length of the built prompt the state is appended to, excluding the trailing [SEP]."""
seq, _ = build_sequence(tok, "", q, max_len=10 ** 6, head_max_len=head_max_len)
return len(seq) - 1
def _kept_state(tok, state, q, max_len, head_max_len):
- head = _head_block(tok, q, head_max_len)
+ head = _head_len(tok, q, head_max_len)
seq, _ = build_sequence(tok, state, q, max_len=max_len, head_max_len=head_max_len)
return seq[head:-1]
-def test_head_never_exceeds_the_cap_by_more_than_the_delimiters():
- """head_len <= head_max_len + 3 (CLS plus a [SEP] on each side of the marker block)."""
- tok = _FakeTok()
- for head_max_len in (32, 64, 192, 256, 512):
- head = _head_block(tok, Q, head_max_len)
- assert head <= head_max_len + 3, (
- "head=%d exceeds cap %d (+3)" % (head, head_max_len))
+def _many_options(k, desc="a short description of this option"):
+ return {"t": "choice", "ins": "Which intent is this?",
+ "crit": {"intent_%d" % i: desc for i in range(k)}}
+
+
+def _models_dir():
+ return os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")
+
+
+def _real_tok(name):
+ """The checkpoint's tokenizer, or None. Callers decide whether absence is a skip."""
+ path = os.path.join(_models_dir(), name, "tokenizer")
+ if not os.path.isdir(path):
+ return None
+ from transformers import AutoTokenizer
+ return AutoTokenizer.from_pretrained(path)
+# --------------------------------------------------------------------------- invariant checks
+
def test_state_budget_is_sized_from_the_actual_head():
"""A state longer than the room is cut to exactly `max_len - head_len - 1`."""
tok = _FakeTok()
max_len, head_max_len = 512, 192
- head = _head_block(tok, Q, head_max_len)
+ head = _head_len(tok, Q, head_max_len)
assert head < head_max_len, "fixture assumes a question that does not fill the cap"
room = max_len - head - 1
state = " ".join("w%d" % i for i in range(room + 200))
kept = _kept_state(tok, state, Q, max_len, head_max_len)
check("kept state length is max_len - head_len - 1", len(kept), room)
- check("room exceeds the cap-based figure", room > max_len - head_max_len, True)
-
-def test_cap_based_figure_understates_the_room():
- """A state of exactly `max_len - head_max_len` tokens is NOT truncated.
- This is the assertion the README's old wording contradicts: it told a caller that
- `max_len - head_max_len` was all the state they had.
- """
+def test_dropping_the_minus_one_is_caught():
+ """The `-1` is load-bearing: one token more than the room must lose its tail."""
tok = _FakeTok()
max_len, head_max_len = 512, 192
- documented = max_len - head_max_len # 320 under the old wording
- head = _head_block(tok, Q, head_max_len)
- real_room = max_len - head - 1
+ head = _head_len(tok, Q, head_max_len)
+ marker = "TAILMARKER"
+ exact = max_len - head - 1 # the true room
- assert real_room > documented, (
- "fixture assumes the cap-based figure understates the room")
+ state = " ".join("w%d" % i for i in range(exact - 1)) + " " + marker
+ kept = _kept_state(tok, state, Q, max_len, head_max_len)
+ check("a state of exactly the room is kept whole", len(kept), exact)
+ assert tok.id_of(marker) in kept, "the tail marker must survive an exactly-sized state"
+
+ over = " ".join("w%d" % i for i in range(exact)) + " " + marker
+ kept_over = _kept_state(tok, over, Q, max_len, head_max_len)
+ assert tok.id_of(marker) not in kept_over, (
+ "a state one token over the room must lose its tail, so the room is max_len-head_len-1 "
+ "and not max_len-head_len")
+
+
+def test_cap_based_figure_understates_a_short_question():
+ """A state of exactly `max_len - head_max_len` tokens survives for a short question."""
+ tok = _FakeTok()
+ max_len, head_max_len = 512, 192
+ documented = max_len - head_max_len
+ head = _head_len(tok, Q, head_max_len)
+ assert max_len - head - 1 > documented, "fixture assumes the cap-based figure understates"
marker = "UNIQUETAILMARKER"
state = " ".join("w%d" % i for i in range(documented - 3)) + " " + marker
kept = _kept_state(tok, state, Q, max_len, head_max_len)
assert tok.id_of(marker) in kept, (
- "a %d-token state (the cap-based figure) must survive a %d budget; "
- "it was truncated, so the room is smaller than the README claims"
- % (documented, max_len))
- check("full documented-size state survives", len(kept), documented - 2)
+ "a %d-token state must survive a %d budget for this question" % (documented, max_len))
-def test_shipped_checkpoints_match_the_documented_room():
- """With each checkpoint's own tokenizer, the state room is the figure the README quotes.
+def test_cap_based_figure_overstates_a_high_cardinality_question():
+ """Past ~`head_max_len / 4` options the head exceeds the cap, so the cap-based figure lies.
- Skipped when the checkpoints are not present (CI runs without weights).
+ This is the direction that matters: a caller sizing a state to `max_len - head_max_len`
+ is truncated, not merely conservative.
"""
- lab = os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")
- cases = [("english", 512, 192, 48, 463), ("multilingual", 1024, 256, 45, 978)]
- try:
- from transformers import AutoTokenizer
- except Exception:
- print(" (skipped: transformers unavailable)")
- return
+ tok = _FakeTok()
+ max_len, head_max_len = 512, 192
+ head = _head_len(tok, _many_options(100), head_max_len)
+ assert head > head_max_len, (
+ "a 100-option question must build a head beyond the %d cap, got %d" % (head_max_len, head))
+ check("cap-based figure overstates the room at k=100",
+ max_len - head_max_len > max_len - head - 1, True)
+
+
+# --------------------------------------------------------------------- documentation agreement
+
+def _readme_quotes():
+ """The figures README.md states, parsed out of the token-budget bullet."""
+ root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
+ text = fh.read()
+ heads = {int(m) for m in re.findall(r"\*\*(\d+)-token\*\* head", text)}
+ rooms = {int(m) for m in re.findall(r"\*\*(\d+) tokens for state\*\*", text)}
+ return heads, rooms
+
+
+def test_readme_quotes_the_measured_figures():
+ """README's head/room figures equal a live measurement, for each shipped checkpoint.
+
+ This is what makes the suite stop the README and the renderer desynchronising; without it
+ the file only guards the renderer.
+ """
+ heads, rooms = _readme_quotes()
+ assert heads and rooms, "could not parse the head/room figures out of README.md"
+
checked = 0
- for name, max_len, head_max_len, want_head, want_room in cases:
- path = os.path.join(lab, name, "tokenizer")
- if not os.path.isdir(path):
- print(" (skipped %s: no tokenizer at %s)" % (name, path))
+ for name, max_len, head_max_len in (("english", 512, 192), ("multilingual", 1024, 256)):
+ try:
+ tok = _real_tok(name)
+ except Exception as e: # transformers absent or unloadable
+ skip("%s: tokenizer unavailable (%s)" % (name, e))
+ continue
+ if tok is None:
+ skip("%s: no tokenizer under %s" % (name, _models_dir()))
continue
- tok = AutoTokenizer.from_pretrained(path)
- head = _head_block(tok, Q, head_max_len)
- check("%s head length" % name, head, want_head)
- check("%s state room" % name, max_len - head - 1, want_room)
+ head = _head_len(tok, Q, head_max_len)
+ room = max_len - head - 1
+ assert head in heads, (
+ "README does not quote a %d-token head for %s (quotes %s)" % (head, name, sorted(heads)))
+ assert room in rooms, (
+ "README does not quote a %d-token state room for %s (quotes %s)"
+ % (room, name, sorted(rooms)))
checked += 1
if not checked:
- print(" (skipped: no checkpoints under %s)" % lab)
+ skip("no checkpoints under %s" % _models_dir())
if __name__ == "__main__":
- for fn in (test_head_never_exceeds_the_cap_by_more_than_the_delimiters,
- test_state_budget_is_sized_from_the_actual_head,
- test_cap_based_figure_understates_the_room,
- test_shipped_checkpoints_match_the_documented_room):
+ for fn in (test_state_budget_is_sized_from_the_actual_head,
+ test_dropping_the_minus_one_is_caught,
+ test_cap_based_figure_understates_a_short_question,
+ test_cap_based_figure_overstates_a_high_cardinality_question,
+ test_readme_quotes_the_measured_figures):
try:
fn()
except AssertionError as e:
FAIL.append("%s: %s" % (fn.__name__, e))
+ except _Skip:
+ pass
else:
PASS.append(fn.__name__)
- print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+ print("\n%d passed, %d failed, %d skipped" % (len(PASS), len(FAIL), len(SKIPPED)))
for f in FAIL:
print(" FAIL " + f)
+ for s in SKIPPED:
+ print(" SKIP " + s)
sys.exit(1 if FAIL else 0)