Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(hooks): size the taught token-budget pattern on the whole call
Aashish462736f9b964
diff --git a/docs/hooks/examples.md b/docs/hooks/examples.mdindex 25b1efb6..1fdeaf38 100644--- a/docs/hooks/examples.md+++ b/docs/hooks/examples.md@@ -341,20 +341,34 @@ hooks.clear_default_hooks()## Token budget-Shape the token budget for one call, from a hook or a per-call argument.+Shape the token budget for one call, from a hook or a per-call argument. A hook's value replaces+the budget in force, so it has to read that budget first: size on the widest question of the call,+stay above the token floor the core applies to the options, and widen `max_len` with `head_max_len`+so the state keeps a window.```pythondef widen(ctx):- k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})- if k >= 50:- ctx.head_max_len = 16 + 4 * k+ k = max((len(q.get("criteria", {}) or {}) for q in ctx.questions.values()), default=0)+ if k < 50:+ return+ cfg = getattr(ctx.agent, "cfg", None) or {}+ head = ctx.head_max_len if ctx.head_max_len is not None else cfg.get("head_max_len", 192)+ window = ctx.max_len if ctx.max_len is not None else cfg.get("max_len", 512)+ need = 16 + 8 * k+ if need > head:+ ctx.head_max_len = need+ ctx.max_len = max(window, need + 8 + 64)agent = laya.load("convaiinnovations/laya", on_predict_start=widen)# or per call-agent.system_one(state, questions, head_max_len=324, max_len=1024)+agent.system_one(state, questions, head_max_len=512, max_len=1024)```+[Token-budget shaping](patterns.md#token-budget-shaping) has the arithmetic behind each line, and+[`predict_shortlist`](../reference/helpers.md) is the option when a label set cannot fit even a+widened window.+## Async hooksWrap an async hook in `AsyncHook`; each coroutine runs to completion in the sync core, whether thediff --git a/docs/hooks/patterns.md b/docs/hooks/patterns.mdindex 5a228b14..0ceb57d8 100644--- a/docs/hooks/patterns.md+++ b/docs/hooks/patterns.md@@ -265,20 +265,44 @@ in tests so one test cannot leak a hook into the next.### Token-budget shaping-A start hook can raise the token budget for one call, for example when a question has many-options and the default head budget would collapse the labels. This does not touch the shared-agent config, so concurrent calls are unaffected.+A start hook can raise the token budget for one call, for example when a question has many options+and the default head budget would collapse the labels. Four details decide whether the hook helps+or quietly makes the call worse:++* A start hook's `ctx.head_max_len` *replaces* the budget for the call. What is in force before it+ is the caller's own per-call value, or the checkpoint default in `ctx.agent.cfg` -- so compare+ against that, and writing a plain number can lower the budget a caller already set.+* One call answers every question it carries, so size on the widest of them rather than on+ whichever happens to come first.+* Once the options no longer fit the head, `laya/common.py` gives each of them+ `max(4, (head_max_len - 16) // k)` tokens. `16 + 4 * k` therefore lands exactly on that floor:+ every label is still cut down to the tokens it shares with the others, which is the collapse the+ hook was written to avoid. `16 + 8 * k` leaves them distinguishable.+* The state gets `max_len - head_max_len - 8` tokens, so a widened head has to widen `max_len`+ with it or the state loses its window.```pythondef widen_for_high_cardinality(ctx):- k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})- if k >= 50:- ctx.head_max_len = max(ctx.head_max_len or 192, 16 + 4 * k)+ k = max((len(q.get("criteria", {}) or {}) for q in ctx.questions.values()), default=0)+ if k < 50:+ return+ cfg = getattr(ctx.agent, "cfg", None) or {}+ head = ctx.head_max_len if ctx.head_max_len is not None else cfg.get("head_max_len", 192)+ window = ctx.max_len if ctx.max_len is not None else cfg.get("max_len", 512)+ need = 16 + 8 * k # 8 tokens per label, not the core's floor of 4+ if need > head: # only ever widen, never lower+ ctx.head_max_len = need+ ctx.max_len = max(window, need + 8 + 64) # 8 reserved, then room for the stateagent = laya.load("convaiinnovations/laya", on_predict_start=widen_for_high_cardinality)```-The same knobs are available per call: `agent.system_one(state, questions, head_max_len=324)`.+This does not touch the shared agent config, so concurrent calls are unaffected. The same knobs are+available per call: `agent.system_one(state, questions, head_max_len=512, max_len=1024)`.++Widening is not free: a longer window means a larger tensor, and the checkpoints were trained at+512 (`laya`) and 1,024 tokens. Past that, narrowing the candidates with+[`predict_shortlist`](../reference/helpers.md) beats stretching the budget.## Anti-patternsdiff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 7ec975c5..635b3e21 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -353,6 +353,104 @@ check("patterns.md/enrich stays aligned with states",enrich(ctx)check("patterns.md/enrich still guards the recursion", len(ctx.results), len(BATCH))+# ------------------------------------------------ taught token-budget bodies size the whole call+#+# A start hook's ctx.head_max_len / ctx.max_len REPLACE the budgets in force (laya/agent.py), and+# what is in force is the caller's per-call value or the checkpoint default in `agent.cfg`. Once a+# question's options overflow the head, laya/common.py keeps each of them+# `max(4, (head_max_len - 16) // k)` tokens -- so `16 + 4 * k` buys exactly the floor it is meant to+# escape -- and the state is left with `max_len - head_max_len - 8` tokens, so widening the head+# without widening the window truncates the state instead of the labels.++WIDE_K = 60+WIDE_Q = {"type": "choice", "instructions": "Which of these intents is it?",+ "criteria": {"intent_%02d" % i: "intent number %02d about the account" % i+ for i in range(WIDE_K)}}+SMALL_Q = {"type": "choice", "instructions": "Which team?",+ "criteria": {"billing": "invoices and payments", "support": "product help"}}+CFG192 = {"head_max_len": 192, "max_len": 512}+CFG256 = {"head_max_len": 256, "max_len": 1024}+++def budget_ctx(questions, cfg=CFG192, **in_force):+ """A start-hook context for one call, with the budgets the caller left in place."""+ return PredictContext(states=["My invoice was charged twice."], questions=questions,+ model="english", agent=type("Ag", (), {"cfg": dict(cfg)})(), **in_force)+++def run_body(hook, ctx):+ """The name of what the body raised, or None -- a start hook that raises aborts the call."""+ try:+ hook(ctx)+ return None+ except Exception as exc:+ return exc.__class__.__name__+++BUDGET = [("docs/hooks/patterns.md", "widen_for_high_cardinality", "### Token-budget shaping"),+ ("docs/hooks/examples.md", "widen", "## Token budget")]+for path, name, anchor in BUDGET:+ hook = taught(path, name, after=anchor)+ check_true("%s/%s sizes on every question, not the first" % (path, name),+ "next(iter(ctx.questions" not in hook.__taught_body__,+ "the body still measures one question: %r" % hook.__taught_body__)++ call = budget_ctx({"first": SMALL_Q, "second": WIDE_Q})+ check("%s/%s/a call that needs no widening does not raise" % (path, name),+ run_body(hook, call), None)+ check_true("%s/%s/sizes on the widest question of the call" % (path, name),+ call.head_max_len is not None and (call.head_max_len - 16) // WIDE_K > 4,+ "wrote head=%s for a %d-option question: %s"+ % (call.head_max_len, WIDE_K,+ "it wrote no budget at all" if call.head_max_len is None+ else "each label keeps %d tokens" % max(4, (call.head_max_len - 16) // WIDE_K)))+ check_true("%s/%s/leaves the state a window" % (path, name),+ call.max_len is not None and call.max_len - call.head_max_len - 8 >= 64,+ "head=%s window=%s: %s"+ % (call.head_max_len, call.max_len,+ "the state keeps no window of its own"+ if call.max_len is None or call.head_max_len is None+ else "the state keeps %d tokens" % (call.max_len - call.head_max_len - 8)))++ empty = budget_ctx({})+ check("%s/%s/a call with no questions neither raises nor rewrites" % (path, name),+ (run_body(hook, empty), empty.head_max_len, empty.max_len), (None, None, None))+ small = budget_ctx({"q": SMALL_Q})+ run_body(hook, small)+ check("%s/%s/a narrow question leaves the budget alone" % (path, name),+ (small.head_max_len, small.max_len), (None, None))+ caller = budget_ctx({"q": WIDE_Q}, head_max_len=324, max_len=2048)+ run_body(hook, caller)+ check_true("%s/%s/never lowers the caller's own per-call budget" % (path, name),+ caller.head_max_len is not None and caller.head_max_len >= 324+ and caller.max_len is not None and caller.max_len >= 2048,+ "wrote (%s, %s) over a caller's (324, 2048)"+ % (caller.head_max_len, caller.max_len))+ wider = budget_ctx({"q": WIDE_Q}, head_max_len=640, max_len=4096)+ run_body(hook, wider)+ check_true("%s/%s/a caller who already went wider keeps their budget" % (path, name),+ (wider.head_max_len, wider.max_len) == (640, 4096),+ "wrote (%s, %s) over a caller's (640, 4096)"+ % (wider.head_max_len, wider.max_len))+ checkpoint = budget_ctx({"q": WIDE_Q}, cfg=CFG256)+ run_body(hook, checkpoint)+ check_true("%s/%s/never lowers the checkpoint default" % (path, name),+ checkpoint.head_max_len is None or checkpoint.head_max_len >= 256,+ "wrote head=%s where the checkpoint declares 256" % checkpoint.head_max_len)++# The two pages teach one behaviour, on the same calls.+_widest, _raised = budget_ctx({"first": SMALL_Q, "second": WIDE_Q}), budget_ctx({})+_first_budget = taught(BUDGET[0][0], BUDGET[0][1], after=BUDGET[0][2])+_first_budget(_widest)+run_body(_first_budget, _raised)+for path, name, anchor in BUDGET[1:]:+ other = taught(path, name, after=anchor)+ second, empty = budget_ctx({"first": SMALL_Q, "second": WIDE_Q}), budget_ctx({})+ check("%s/%s/sizes like the pattern page" % (path, name),+ (run_body(other, second), second.head_max_len, second.max_len,+ run_body(other, empty), empty.head_max_len),+ (None, _widest.head_max_len, _widest.max_len, None, _raised.head_max_len))+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL: