Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(hooks): size the taught token-budget pattern on the whole call

Aashish462736f9b964
diff --git a/docs/hooks/examples.md b/docs/hooks/examples.md
index 25b1efb6..1fdeaf38 100644
--- a/docs/hooks/examples.md
+++ b/docs/hooks/examples.md
@@ -341,20 +341,34 @@ hooks.clear_default_hooks()
## Token budget
-Shape the token budget for one call, from a hook or a per-call argument.
+Shape the token budget for one call, from a hook or a per-call argument. A hook's value replaces
+the budget in force, so it has to read that budget first: size on the widest question of the call,
+stay above the token floor the core applies to the options, and widen `max_len` with `head_max_len`
+so the state keeps a window.
```python
def widen(ctx):
- k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})
- if k >= 50:
- ctx.head_max_len = 16 + 4 * k
+ k = max((len(q.get("criteria", {}) or {}) for q in ctx.questions.values()), default=0)
+ if k < 50:
+ return
+ cfg = getattr(ctx.agent, "cfg", None) or {}
+ head = ctx.head_max_len if ctx.head_max_len is not None else cfg.get("head_max_len", 192)
+ window = ctx.max_len if ctx.max_len is not None else cfg.get("max_len", 512)
+ need = 16 + 8 * k
+ if need > head:
+ ctx.head_max_len = need
+ ctx.max_len = max(window, need + 8 + 64)
agent = laya.load("convaiinnovations/laya", on_predict_start=widen)
# or per call
-agent.system_one(state, questions, head_max_len=324, max_len=1024)
+agent.system_one(state, questions, head_max_len=512, max_len=1024)
```
+[Token-budget shaping](patterns.md#token-budget-shaping) has the arithmetic behind each line, and
+[`predict_shortlist`](../reference/helpers.md) is the option when a label set cannot fit even a
+widened window.
+
## Async hooks
Wrap an async hook in `AsyncHook`; each coroutine runs to completion in the sync core, whether the
diff --git a/docs/hooks/patterns.md b/docs/hooks/patterns.md
index 5a228b14..0ceb57d8 100644
--- a/docs/hooks/patterns.md
+++ b/docs/hooks/patterns.md
@@ -265,20 +265,44 @@ in tests so one test cannot leak a hook into the next.
### Token-budget shaping
-A start hook can raise the token budget for one call, for example when a question has many
-options and the default head budget would collapse the labels. This does not touch the shared
-agent config, so concurrent calls are unaffected.
+A start hook can raise the token budget for one call, for example when a question has many options
+and the default head budget would collapse the labels. Four details decide whether the hook helps
+or quietly makes the call worse:
+
+* A start hook's `ctx.head_max_len` *replaces* the budget for the call. What is in force before it
+ is the caller's own per-call value, or the checkpoint default in `ctx.agent.cfg` -- so compare
+ against that, and writing a plain number can lower the budget a caller already set.
+* One call answers every question it carries, so size on the widest of them rather than on
+ whichever happens to come first.
+* Once the options no longer fit the head, `laya/common.py` gives each of them
+ `max(4, (head_max_len - 16) // k)` tokens. `16 + 4 * k` therefore lands exactly on that floor:
+ every label is still cut down to the tokens it shares with the others, which is the collapse the
+ hook was written to avoid. `16 + 8 * k` leaves them distinguishable.
+* The state gets `max_len - head_max_len - 8` tokens, so a widened head has to widen `max_len`
+ with it or the state loses its window.
```python
def widen_for_high_cardinality(ctx):
- k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})
- if k >= 50:
- ctx.head_max_len = max(ctx.head_max_len or 192, 16 + 4 * k)
+ k = max((len(q.get("criteria", {}) or {}) for q in ctx.questions.values()), default=0)
+ if k < 50:
+ return
+ cfg = getattr(ctx.agent, "cfg", None) or {}
+ head = ctx.head_max_len if ctx.head_max_len is not None else cfg.get("head_max_len", 192)
+ window = ctx.max_len if ctx.max_len is not None else cfg.get("max_len", 512)
+ need = 16 + 8 * k # 8 tokens per label, not the core's floor of 4
+ if need > head: # only ever widen, never lower
+ ctx.head_max_len = need
+ ctx.max_len = max(window, need + 8 + 64) # 8 reserved, then room for the state
agent = laya.load("convaiinnovations/laya", on_predict_start=widen_for_high_cardinality)
```
-The same knobs are available per call: `agent.system_one(state, questions, head_max_len=324)`.
+This does not touch the shared agent config, so concurrent calls are unaffected. The same knobs are
+available per call: `agent.system_one(state, questions, head_max_len=512, max_len=1024)`.
+
+Widening is not free: a longer window means a larger tensor, and the checkpoints were trained at
+512 (`laya`) and 1,024 tokens. Past that, narrowing the candidates with
+[`predict_shortlist`](../reference/helpers.md) beats stretching the budget.
## Anti-patterns
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index 7ec975c5..635b3e21 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -353,6 +353,104 @@ check("patterns.md/enrich stays aligned with states",
enrich(ctx)
check("patterns.md/enrich still guards the recursion", len(ctx.results), len(BATCH))
+# ------------------------------------------------ taught token-budget bodies size the whole call
+#
+# A start hook's ctx.head_max_len / ctx.max_len REPLACE the budgets in force (laya/agent.py), and
+# what is in force is the caller's per-call value or the checkpoint default in `agent.cfg`. Once a
+# question's options overflow the head, laya/common.py keeps each of them
+# `max(4, (head_max_len - 16) // k)` tokens -- so `16 + 4 * k` buys exactly the floor it is meant to
+# escape -- and the state is left with `max_len - head_max_len - 8` tokens, so widening the head
+# without widening the window truncates the state instead of the labels.
+
+WIDE_K = 60
+WIDE_Q = {"type": "choice", "instructions": "Which of these intents is it?",
+ "criteria": {"intent_%02d" % i: "intent number %02d about the account" % i
+ for i in range(WIDE_K)}}
+SMALL_Q = {"type": "choice", "instructions": "Which team?",
+ "criteria": {"billing": "invoices and payments", "support": "product help"}}
+CFG192 = {"head_max_len": 192, "max_len": 512}
+CFG256 = {"head_max_len": 256, "max_len": 1024}
+
+
+def budget_ctx(questions, cfg=CFG192, **in_force):
+ """A start-hook context for one call, with the budgets the caller left in place."""
+ return PredictContext(states=["My invoice was charged twice."], questions=questions,
+ model="english", agent=type("Ag", (), {"cfg": dict(cfg)})(), **in_force)
+
+
+def run_body(hook, ctx):
+ """The name of what the body raised, or None -- a start hook that raises aborts the call."""
+ try:
+ hook(ctx)
+ return None
+ except Exception as exc:
+ return exc.__class__.__name__
+
+
+BUDGET = [("docs/hooks/patterns.md", "widen_for_high_cardinality", "### Token-budget shaping"),
+ ("docs/hooks/examples.md", "widen", "## Token budget")]
+for path, name, anchor in BUDGET:
+ hook = taught(path, name, after=anchor)
+ check_true("%s/%s sizes on every question, not the first" % (path, name),
+ "next(iter(ctx.questions" not in hook.__taught_body__,
+ "the body still measures one question: %r" % hook.__taught_body__)
+
+ call = budget_ctx({"first": SMALL_Q, "second": WIDE_Q})
+ check("%s/%s/a call that needs no widening does not raise" % (path, name),
+ run_body(hook, call), None)
+ check_true("%s/%s/sizes on the widest question of the call" % (path, name),
+ call.head_max_len is not None and (call.head_max_len - 16) // WIDE_K > 4,
+ "wrote head=%s for a %d-option question: %s"
+ % (call.head_max_len, WIDE_K,
+ "it wrote no budget at all" if call.head_max_len is None
+ else "each label keeps %d tokens" % max(4, (call.head_max_len - 16) // WIDE_K)))
+ check_true("%s/%s/leaves the state a window" % (path, name),
+ call.max_len is not None and call.max_len - call.head_max_len - 8 >= 64,
+ "head=%s window=%s: %s"
+ % (call.head_max_len, call.max_len,
+ "the state keeps no window of its own"
+ if call.max_len is None or call.head_max_len is None
+ else "the state keeps %d tokens" % (call.max_len - call.head_max_len - 8)))
+
+ empty = budget_ctx({})
+ check("%s/%s/a call with no questions neither raises nor rewrites" % (path, name),
+ (run_body(hook, empty), empty.head_max_len, empty.max_len), (None, None, None))
+ small = budget_ctx({"q": SMALL_Q})
+ run_body(hook, small)
+ check("%s/%s/a narrow question leaves the budget alone" % (path, name),
+ (small.head_max_len, small.max_len), (None, None))
+ caller = budget_ctx({"q": WIDE_Q}, head_max_len=324, max_len=2048)
+ run_body(hook, caller)
+ check_true("%s/%s/never lowers the caller's own per-call budget" % (path, name),
+ caller.head_max_len is not None and caller.head_max_len >= 324
+ and caller.max_len is not None and caller.max_len >= 2048,
+ "wrote (%s, %s) over a caller's (324, 2048)"
+ % (caller.head_max_len, caller.max_len))
+ wider = budget_ctx({"q": WIDE_Q}, head_max_len=640, max_len=4096)
+ run_body(hook, wider)
+ check_true("%s/%s/a caller who already went wider keeps their budget" % (path, name),
+ (wider.head_max_len, wider.max_len) == (640, 4096),
+ "wrote (%s, %s) over a caller's (640, 4096)"
+ % (wider.head_max_len, wider.max_len))
+ checkpoint = budget_ctx({"q": WIDE_Q}, cfg=CFG256)
+ run_body(hook, checkpoint)
+ check_true("%s/%s/never lowers the checkpoint default" % (path, name),
+ checkpoint.head_max_len is None or checkpoint.head_max_len >= 256,
+ "wrote head=%s where the checkpoint declares 256" % checkpoint.head_max_len)
+
+# The two pages teach one behaviour, on the same calls.
+_widest, _raised = budget_ctx({"first": SMALL_Q, "second": WIDE_Q}), budget_ctx({})
+_first_budget = taught(BUDGET[0][0], BUDGET[0][1], after=BUDGET[0][2])
+_first_budget(_widest)
+run_body(_first_budget, _raised)
+for path, name, anchor in BUDGET[1:]:
+ other = taught(path, name, after=anchor)
+ second, empty = budget_ctx({"first": SMALL_Q, "second": WIDE_Q}), budget_ctx({})
+ check("%s/%s/sizes like the pattern page" % (path, name),
+ (run_body(other, second), second.head_max_len, second.max_len,
+ run_body(other, empty), empty.head_max_len),
+ (None, _widest.head_max_len, _widest.max_len, None, _raised.head_max_len))
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL: