Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

docs(agent): predict_long must teach the window default's 64-token floor

Aashishe5a7f2b68890
diff --git a/laya/agent.py b/laya/agent.py
index 7714f1fc..7bc16a56 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -1643,9 +1643,11 @@ class Agent(HookRegistry):
above describe this method's windows, not the text the model read
Args:
- window: state tokens per window. Defaults to the checkpoint's state budget
- (`max_len - head_max_len - 8`), and either way is capped at the room the
- questions leave for the state inside `max_len` -- the smallest room of them,
+ window: state tokens per window. Defaults to the checkpoint's state budget,
+ `max(64, max_len - head_max_len - 8)` -- the 64 is a floor, so widening
+ `head_max_len` stops shrinking the default once the budget reaches it -- and
+ either way is capped at the room the questions leave for the state inside
+ `max_len` -- the smallest room of them,
because one list of windows is scored for every question. A wider window is
re-truncated on the way to the model, so it is clamped instead, with a
`RuntimeWarning` when the caller is the one who asked for it. Options are what
diff --git a/laya/common.py b/laya/common.py
index 45b5e2a4..2957e4e2 100644
--- a/laya/common.py
+++ b/laya/common.py
@@ -379,7 +379,9 @@ def window_budget(tok, questions, max_len: int = 512, head_max_len: int = 192,
windows are one list of states scored for every question in shared forward passes -- a window
sized for the roomiest question would be cut short for the tightest one, and the offsets
reported on its answers would mean something different per question. With no questions there is
- nothing to fit, so the caller's window (or the checkpoint default) stands.
+ nothing to fit, so the caller's window (or the checkpoint default) stands. That default is
+ `max(64, max_len - head_max_len - 8)`: the state budget the config leaves, with a 64-token
+ floor under it, so a widened `head_max_len` stops shrinking it there.
`questions` are internal question dicts, as `Agent._to_internal` returns them.
diff --git a/laya/onnx_agent.py b/laya/onnx_agent.py
index 34c94194..9d0fcc3f 100644
--- a/laya/onnx_agent.py
+++ b/laya/onnx_agent.py
@@ -427,8 +427,9 @@ class ONNXAgent(HookRegistry):
state: Text string, JSON dict, or conversation turn list.
questions: Question definitions, exactly as accepted by `system_one`.
window: State tokens per window. Defaults to the per-question state budget
- (`max_len - head_max_len - 8`). Smaller windows isolate a localized signal
- better at the cost of more windows, as in `Agent.predict_long`.
+ (`max(64, max_len - head_max_len - 8)`; the 64 is a floor). Smaller windows
+ isolate a localized signal better at the cost of more windows, as in
+ `Agent.predict_long`.
stride: Token step between windows. Defaults to `window // 2` (50% overlap).
aggregate: "auto" (the per-type rules above) is the only mode for now.
batch_size: Cap on windows per session run, to bound peak memory on very long states.
diff --git a/laya/router.py b/laya/router.py
index 9f2ff818..bbcdc3e2 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -1378,9 +1378,9 @@ class Router(HookRegistry):
Args:
window: state tokens per window. Defaults to the routed checkpoint's budget
- (`max_len - head_max_len - 8`), capped at the room the questions leave for the
- state so no window is truncated again on the way in; a smaller window isolates
- a localized span.
+ (`max(64, max_len - head_max_len - 8)`; the 64 is a floor), capped at the room
+ the questions leave for the state so no window is truncated again on the way
+ in; a smaller window isolates a localized span.
stride: token step between windows; defaults to half the effective window (50%
overlap). A stride past that window is a `ValueError`, since the tokens between
windows would reach no model.