Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): bound predict_long's window by the room a question leaves

Adeshina Lasisief93a9471531
diff --git a/README.md b/README.md
index 328a6194..1eef506e 100644
--- a/README.md
+++ b/README.md
@@ -712,7 +712,11 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,
`usage["windows"]` at 0, because no window scored it.
A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its
-window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches
+window); the default (`max_len - head_max_len`) favors context and throughput. Either way the
+window is capped at the room the questions leave for the state inside `max_len`, so a window is
+never re-truncated on the way to the model and `token_start`/`token_end` describe the span it read:
+many options leave little room (on the English checkpoint, 2 options leave 483 state tokens and
+100 leave 100), and a window past that room used to be cut short silently. Output shape matches
`predict`, with `usage["windows"]` added.
`ONNXAgent.predict_long(state, questions, window=..., stride=..., batch_size=...)` has the same
diff --git a/laya/agent.py b/laya/agent.py
index c98e6859..f8229fbb 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -41,6 +41,8 @@ from .common import (
serialize_state,
temp_bucket,
unpermute_probs,
+ window_batch_cap,
+ window_budget,
)
from .confidence import check_min_confidence, flag_low_confidence
from .hooks import (
@@ -1307,6 +1309,8 @@ class Agent(HookRegistry):
document is decisive. Each answer therefore carries `answer["window"]` — the deciding
window's `index`, `token_start`/`token_end` into the tokenized state, and the window `count`
— so a caller can inspect the span the answer came from rather than trust the raw number.
+ That span is the one the model read, not merely the one asked for: the window is capped at
+ the room the questions leave, so what is handed to `predict_batch` is not cut short again.
A state that already fits one window is passed straight to `system_one` (identical output).
@@ -1324,15 +1328,24 @@ class Agent(HookRegistry):
above describe this method's windows, not the text the model read
Args:
- window: state tokens per window. Defaults to the per-question state budget
- (`max_len - head_max_len - 8`) -- the most a window can hold for every question.
+ window: state tokens per window. Defaults to the checkpoint's state budget
+ (`max_len - head_max_len - 8`), and either way is capped at the room the
+ questions leave for the state inside `max_len` -- the smallest room of them,
+ because one list of windows is scored for every question. A wider window is
+ re-truncated on the way to the model, so it is clamped instead, with a
+ `RuntimeWarning` when the caller is the one who asked for it. Options are what
+ make the room small: on the English checkpoint a 2-option question leaves 483
+ tokens for the state and a 100-option one leaves 100.
A smaller window isolates a localized signal better (a short deciding span is a
larger fraction of its window, so that window classifies it clearly), at the
cost of more windows; the large default favors context and throughput. `noul`
is robust to this, `choice`/`score` benefit from a smaller window when the
deciding span is a small part of a long, otherwise-neutral document.
- stride: token step between windows. Defaults to `window // 2` (50% overlap), so a span
- near a boundary still lands whole inside some window.
+ stride: token step between windows. Defaults to half the *effective* window (50%
+ overlap), so a span near a boundary still lands whole inside some window. A
+ stride past the effective window is refused rather than clamped: the tokens
+ between each pair of windows would be read by no window at all, which is the
+ failure this method exists to prevent.
aggregate: "auto" (the per-type rules above) is the only mode for now.
batch_size: cap on windows per forward pass, to bound memory on very long states.
lang: per-language temperature selection, as in `system_one`.
@@ -1343,6 +1356,11 @@ class Agent(HookRegistry):
hooks_raise: Override the Agent's `hooks_raise` for this call.
hooks_timeout: Override the Agent's `hooks_timeout` for this call.
+ Raises:
+ ValueError: `aggregate` is anything but "auto"; the questions' options fill the whole
+ sequence, leaving no room for the state; or `stride` steps past the effective
+ window, so tokens between two windows would be read by nothing.
+
Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.
The key is always present and counts the windows the model scored to produce the answer: `1`
for a state that fit one window, `N` for a document scanned in `N` overlapping windows (or
@@ -1371,7 +1389,21 @@ class Agent(HookRegistry):
"hooks_timeout": hooks_timeout}
max_len = self.cfg.get("max_len", 512)
head_max_len = self.cfg.get("head_max_len", 192)
- budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)
+ # Size the window against the room these questions actually leave, not the config alone:
+ # every window is decoded and scored as a normal state, so a window wider than the room
+ # was re-truncated by `build_sequence` on the way in and the tail of it reached no model
+ # -- while `answer["window"]` reported the whole span. Measured on the English checkpoint
+ # (max_len=512, head_max_len=192, config budget 312): a 2-option question leaves room for
+ # 483 tokens, 48 options leave 308, and 100 -- `serve`'s documented maximum -- leave 100.
+ # At 87 options the room (152) fell below the 156-token default stride, so the windows
+ # stopped overlapping and part of the document was read by no window at all: measured on a
+ # 920-token document at 100 options, 420 of its tokens reached no window.
+ ids = list(questions.keys())
+ for qid in ids:
+ self._check_question(qid, questions[qid])
+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}
+ budget, step, _ = window_budget(self.tok, [internal[qid] for qid in ids], max_len,
+ head_max_len, window=window, stride=stride)
state_ids = self.tok(serialize_state(state).replace(self.tok.mask_token, " "),
add_special_tokens=False)["input_ids"]
@@ -1390,7 +1422,6 @@ class Agent(HookRegistry):
single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}
return single
- step = stride if (stride and stride > 0) else max(1, budget // 2)
windows, starts = [], []
i, n = 0, len(state_ids)
while i < n:
@@ -1407,7 +1438,11 @@ class Agent(HookRegistry):
# `list(windows)`, not `windows`: `ctx.states` is the list the hook receives, so a hook that
# mutates it in place (`append`, `sort`) would otherwise also grow the split this call
# attributes answers to, and the counts would agree while `starts` no longer lined up.
- results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,
+ # Bound one forward pass to what the un-capped scan would have used; see
+ # `window_batch_cap`. An explicit batch_size is honoured untouched.
+ cap = window_batch_cap(len(windows), budget, max(64, max_len - head_max_len - 8),
+ batch_size)
+ results = self.predict_batch(list(windows), questions, batch_size=cap, lang=lang,
**_with_start_probe(hook_kwargs, probe))
if evidence["answered"]:
diff --git a/laya/common.py b/laya/common.py
index daf57a95..7a1b97e3 100644
--- a/laya/common.py
+++ b/laya/common.py
@@ -3,6 +3,7 @@ import json
import math
import os
import threading
+import warnings
from contextlib import nullcontext
from contextvars import ContextVar
from functools import wraps
@@ -165,6 +166,41 @@ def build_sequence(
the rendered options - so it moves per checkpoint and per question. A caller guessing with a
fixed character threshold is wrong in both directions: it reports truncation that did not
happen, and stays silent while evidence is being dropped (issue #174).
+
+ The question half is `build_head`; `state_room` reports how much of `max_len` is left for the
+ state after it, which is what a caller must size a window against.
+ """
+ ids, markers, stats = build_head(tok, q, head_max_len, option_order=option_order)
+ room = max(0, max_len - len(ids) - 1)
+ if state_ids is None:
+ state_ids = encode_text(tok, serialize_state(state).replace(tok.mask_token, " "),
+ add_special_tokens=False)["input_ids"]
+ # not state_ids[-room:]: with no room left, state_ids[-0:] is the whole state rather than none of it
+ st = state_ids[max(0, len(state_ids) - room):] if truncate_left else state_ids[:room]
+ ids = ids + st + [tok.sep_token_id]
+ ids, markers = ids[:max_len], [m for m in markers if m < max_len]
+ extra = ()
+ if return_truncation_stats:
+ # `room` leaves space for the closing [SEP], so every token in `st` survives the [:max_len] clamp
+ extra = ({
+ "state_tokens": len(state_ids),
+ "state_tokens_used": len(st),
+ "state_tokens_dropped": len(state_ids) - len(st),
+ "truncated": len(st) < len(state_ids),
+ },)
+ if not return_stats:
+ return (ids, markers) + extra
+ return (ids, markers, stats) + extra
+
+
+def build_head(tok, q: Dict, head_max_len: int = 192, option_order: Optional[List[int]] = None):
+ """The question half of a sequence: `[CLS] <type> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP]`.
+
+ `build_sequence` appends the state to this and `state_room` measures what is left over for it,
+ so neither can disagree with the other about what the head costs.
+
+ Returns `(ids, markers, stats)`: the head token ids, the `[MASK]` position of each option in
+ `option_order`, and the stats dict `build_sequence` returns for `return_stats`.
"""
mask_tok = tok.mask_token
opts = render_options(q)
@@ -198,38 +234,128 @@ def build_sequence(
markers.append(len(ids))
ids.extend(o)
ids.append(tok.sep_token_id)
- room = max(0, max_len - len(ids) - 1)
- if state_ids is None:
- state_ids = encode_text(tok, serialize_state(state).replace(mask_tok, " "),
- add_special_tokens=False)["input_ids"]
- # not state_ids[-room:]: with no room left, state_ids[-0:] is the whole state rather than none of it
- st = state_ids[max(0, len(state_ids) - room):] if truncate_left else state_ids[:room]
- ids = ids + st + [tok.sep_token_id]
- ids, markers = ids[:max_len], [m for m in markers if m < max_len]
- extra = ()
- if return_truncation_stats:
- # `room` leaves space for the closing [SEP], so every token in `st` survives the [:max_len] clamp
- extra = ({
- "state_tokens": len(state_ids),
- "state_tokens_used": len(st),
- "state_tokens_dropped": len(state_ids) - len(st),
- "truncated": len(st) < len(state_ids),
- },)
- if not return_stats:
- return (ids, markers) + extra
# Two options that share a prefix can come out of the cut as the same token span: the marker
# count still matches the option count, so the guard in `Agent._encode_state` passes and
# nothing downstream can tell that the question lost the ability to name them apart. Counted
# on the capped option ids, before assembly: re-slicing the finished sequence cannot close
# the last option's span -- it runs on into the serialized state, which differs per request,
# so the last option always looks distinguishable however it collided (#538).
- return (ids, markers, {
+ return ids, markers, {
"options": len(opt_ids),
"options_distinct": len({tuple(o) for o in opt_ids}),
"tokens_per_option": per_option,
- }) + extra
+ }
+def state_room(tok, q: Dict, max_len: int = 512, head_max_len: int = 192) -> int:
+ """How many state tokens `q` leaves inside `max_len`, which is what `build_sequence` keeps.
+
+ The head is the question's own -- its instructions plus one `[MASK]`-prefixed span per option --
+ so a question with many options leaves less room for the state than one with two, and two
+ questions in the same request do not have to leave the same amount. Anything past the return
+ value is cut off (the start is kept, or the end for a conversation turn list).
+ """
+ head, _, _ = build_head(tok, q, head_max_len)
+ return max(0, max_len - len(head) - 1) # -1 for the [SEP] that closes the state
+
+
+# How much capping the window may multiply the window count before `window_batch_cap` starts
+# chunking. Under this, chunking would cost the single-shared-pass property for no real protection.
+_WINDOW_BATCH_BLOWUP = 2
+
+def window_budget(tok, questions, max_len: int = 512, head_max_len: int = 192,
+ window: Optional[int] = None, stride: Optional[int] = None):
+ """Window size and stride for scanning a state that is longer than one sequence.
+
+ `predict_long` decodes each token window back to text and scores it as an ordinary state, so a
+ window wider than the room the questions leave is re-truncated by `build_sequence` on the way
+ in: the tail of every window reaches no model, while the reported span says it did. The window
+ is therefore capped at `state_room`, and at the *smallest* room of `questions`, because the
+ windows are one list of states scored for every question in shared forward passes -- a window
+ sized for the roomiest question would be cut short for the tightest one, and the offsets
+ reported on its answers would mean something different per question. With no questions there is
+ nothing to fit, so the caller's window (or the checkpoint default) stands.
+
+ `questions` are internal question dicts, as `Agent._to_internal` returns them.
+
+ An explicit `window` wider than the room is clamped to it with a `RuntimeWarning`, since a
+ scan at the requested size cannot read what it claims to. A `stride` past the effective window
+ is refused: the tokens between two windows would be read by neither, which is the failure
+ `predict_long` exists to prevent. The default stride keeps its 50% overlap of the *effective*
+ window, so a span near a boundary still lands whole inside some window; an explicit stride
+ equal to the window still reads every token, with no overlap to catch a span that straddles a
+ boundary.
+
+ Returns `(window, stride, room)`.
+ """
+ rooms = [state_room(tok, q, max_len, head_max_len) for q in questions]
+ requested = window if (window and window > 0) else max(64, max_len - head_max_len - 8)
+ size = requested
+ room = min(rooms) if rooms else size
+ if room <= 0:
+ raise ValueError(
+ "predict_long: the questions' options fill the whole sequence (max_len=%d,"
+ " head_max_len=%d), leaving no room for the state; no window can carry any of it"
+ % (max_len, head_max_len))
+ if size > room:
+ if window and window > 0:
+ warnings.warn(
+ "laya: predict_long: window=%d is wider than the %d state tokens these questions"
+ " leave inside max_len=%d, so every window would be truncated to %d on the way to"
+ " the model; scanning with window=%d instead" % (size, room, max_len, room, room),
+ RuntimeWarning, stacklevel=3)
+ size = room
+ step = stride if (stride and stride > 0) else max(1, size // 2)
+ if step > size:
+ if size < requested and stride and stride <= requested:
+ # The window the caller asked for was reduced above, and their stride was valid for the
+ # window they asked for -- so this is the library's clamp, not their mistake. Reducing
+ # the stride to match keeps the no-gap guarantee without making a self-consistent pair
+ # of arguments an error. A stride that overshot the *requested* window is still refused
+ # below, because that one really is the caller's.
+ warnings.warn(
+ "laya: predict_long: stride=%d was a 50%% step for the window=%d you asked for, but"
+ " the window was reduced to %d to fit the room these questions leave; scanning with"
+ " stride=%d instead" % (step, requested, size, max(1, size // 2)),
+ RuntimeWarning, stacklevel=3)
+ step = max(1, size // 2)
+ else:
+ raise ValueError(
+ "predict_long: stride=%d steps past the %d-token window%s, so %d tokens between"
+ " every pair of windows would be read by no window at all; pass stride <= %d"
+ % (step, size, " these questions leave room for" if size < requested else "",
+ step - size, size))
+ return size, step, room
+
+
+
+def window_batch_cap(n_windows: int, window: int, config_budget: int,
+ batch_size: Optional[int] = None) -> Optional[int]:
+ """Keep one forward pass no wider than the un-capped scan's would have been -- but only when
+ capping the window has multiplied the window count enough to matter.
+
+ Capping the window at the room the questions leave multiplies the number of windows on exactly
+ the inputs it targets: measured, a 4 561-token document at 120 options goes from 29 windows of
+ 312 tokens to roughly 413 of 23. `predict_batch` with `batch_size=None` puts every state in one
+ forward pass, so a caller who passed no `batch_size` would go from a 29-row pass to a 413-row one
+ at `max_len` width -- a plausible out-of-memory on an input that used to fit.
+
+ Sending every window in one run is also a deliberate property (`ONNXAgent` asserts it), and a mild
+ cap -- a 64-token budget reduced to 43, say -- multiplies the count by well under two. Chunking
+ those would trade a real property for no real protection. So the cap only applies past
+ `_WINDOW_BATCH_BLOWUP`x, and then bounds the pass at the count the un-capped budget would have
+ produced: peak memory stays at parity with the behaviour before the cap, and the scan still reads
+ the whole document, just in more passes. An explicit `batch_size` is always honoured.
+ """
+ if batch_size and batch_size > 0:
+ return batch_size
+ if window >= config_budget:
+ return None # not capped: one pass, exactly as before
+ unclamped = max(1, -(-n_windows * max(1, window // 2) // max(1, config_budget // 2)))
+ if n_windows <= unclamped * _WINDOW_BATCH_BLOWUP:
+ return None # a mild cap: keep the single shared pass
+ return unclamped
+
def collapsed_options(qids, items) -> Dict[str, Dict[str, Optional[int]]]:
"""The questions whose options no longer have a token span each, from per-item stats.
diff --git a/laya/onnx_agent.py b/laya/onnx_agent.py
index 8a67f065..e7188784 100644
--- a/laya/onnx_agent.py
+++ b/laya/onnx_agent.py
@@ -24,6 +24,8 @@ from laya.common import (
render_options,
resolve_lang_temperatures,
serialize_state,
+ window_batch_cap,
+ window_budget,
temp_bucket,
unpermute_probs,
TEMP_MIN,
@@ -453,7 +455,16 @@ class ONNXAgent(HookRegistry):
raise ValueError("predict_long: only aggregate='auto' is supported")
max_len = self.cfg.get("max_len", 512)
head_max_len = self.cfg.get("head_max_len", 192)
- budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)
+ # The same cap the torch `Agent.predict_long` applies, for the same reason: a window wider
+ # than the room these questions leave is re-truncated by `build_sequence` on the way into
+ # `predict_batch`, so its tail reaches no model while `answer["window"]` reports the whole
+ # span -- and once the room falls below the default stride the windows stop overlapping and
+ # leave tokens no window reads at all. `README.md` and `Router.predict_long` document this
+ # contract for both agents, so both have to honour it.
+ ids = list(questions.keys())
+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}
+ budget, step_default, _ = window_budget(self.tok, [internal[qid] for qid in ids], max_len,
+ head_max_len, window=window, stride=stride)
state_ids = encode_text(
self.tok,
@@ -469,7 +480,7 @@ class ONNXAgent(HookRegistry):
single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}
return single
- step = stride if (stride and stride > 0) else max(1, budget // 2)
+ step = step_default
windows, starts = [], []
i, n = 0, len(state_ids)
while i < n:
@@ -483,7 +494,11 @@ class ONNXAgent(HookRegistry):
probe, evidence = _start_evidence()
# A copy, so a start hook that mutates `ctx.states` in place cannot shift `starts`.
- results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,
+ # Bound one forward pass to what the un-capped scan would have used; see
+ # `window_batch_cap`. An explicit batch_size is honoured untouched.
+ cap = window_batch_cap(len(windows), budget, max(64, max_len - head_max_len - 8),
+ batch_size)
+ results = self.predict_batch(list(windows), questions, batch_size=cap, lang=lang,
**_with_start_probe(hook_kwargs, probe))
if evidence["answered"]:
diff --git a/laya/router.py b/laya/router.py
index 36a0ec6d..9ae7e2c6 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -846,8 +846,12 @@ class Router(HookRegistry):
Args:
window: state tokens per window. Defaults to the routed checkpoint's budget
- (`max_len - head_max_len - 8`); a smaller window isolates a localized span.
- stride: token step between windows; defaults to `window // 2` (50% overlap).
+ (`max_len - head_max_len - 8`), capped at the room the questions leave for the
+ state so no window is truncated again on the way in; a smaller window isolates
+ a localized span.
+ stride: token step between windows; defaults to half the effective window (50%
+ overlap). A stride past that window is a `ValueError`, since the tokens between
+ windows would reach no model.
aggregate: "auto" (the per-type rules above) is the only mode.
batch_size: cap on windows per forward pass, to bound memory on very long states.
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index ccd59572..196ecf95 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -297,10 +297,21 @@ for label, cls in (("LayaRouter", LayaRouter), ("LayaGuardrail", LayaGuardrail),
# `input_tokens` / `output_tokens` are the fields every client decodes, so they are always
# present. `options` (#538) is additive and conditional: it appears only for a request whose
# options lost their distinct token spans, which is what keeps it out of ordinary responses.
-from laya.common import build_sequence, collapsed_options # noqa: E402
+from laya.common import build_head, build_sequence, collapsed_options, state_room, window_budget # noqa: E402
check_param("build_sequence", build_sequence, "return_stats", False)
check_param("build_sequence", build_sequence, "return_truncation_stats", False)
+
+# The sizing surface `predict_long` reads before it splits a state: `build_head` is the question
+# half `build_sequence` assembles, `state_room` is what is left of `max_len` for the state after
+# it, and `window_budget` turns the two into the window and stride a scan may use. Their defaults
+# are the checkpoint's, so a caller can measure a question without holding an Agent.
+for _name, _fn, _params in (("build_head", build_head, (("head_max_len", 192), ("option_order", None))),
+ ("state_room", state_room, (("max_len", 512), ("head_max_len", 192))),
+ ("window_budget", window_budget, (("max_len", 512), ("head_max_len", 192),
+ ("window", None), ("stride", None)))):
+ for _param, _default in _params:
+ check_param(_name, _fn, _param, _default)
check("collapsed_options/nothing collapsed is empty",
collapsed_options(["q"], [{"options": {"options": 3, "options_distinct": 3,
"tokens_per_option": None}}]), {})
diff --git a/tests/test_onnx_long.py b/tests/test_onnx_long.py
index a87ab443..8a254f63 100644
--- a/tests/test_onnx_long.py
+++ b/tests/test_onnx_long.py
@@ -12,6 +12,7 @@ Run: python tests/test_onnx_long.py
import inspect
import os
import sys
+import warnings
import threading
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -19,6 +20,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import numpy as np # noqa: E402
from laya.agent import Agent # noqa: E402
+from laya.common import window_budget # noqa: E402
from laya.onnx_agent import ONNXAgent # noqa: E402
PASS, FAIL = [], []
@@ -166,9 +168,18 @@ for qid, key in (("dept", "answer_confidence"), ("urgent", "noul")):
check_true("aggregate/%s is the strongest window on its rule" % qid,
ans[key] == max(pw["answers"][qid][key] for pw in per_window),
[pw["answers"][qid][key] for pw in per_window])
+ # Derived from the *effective* window, not from `max_len - head_max_len - 8`: these questions
+ # leave less room than that budget at this tiny config, so the window is capped at the room and
+ # the reported span has to describe what the model actually read. Hardcoding 64/32 here asserted
+ # a `token_end` that overstated the span by the difference.
+ _eff, _step, _room = window_budget(agent.tok, [agent._to_internal(q) for q in QUESTIONS.values()],
+ agent.cfg["max_len"], agent.cfg["head_max_len"])
check("window/%s fields index the deciding span into the original state" % qid,
(ans["window"]["count"], ans["window"]["token_start"], ans["window"]["token_end"]),
- (len(windows), j * 32, min(j * 32 + 64, 200)))
+ (len(windows), j * _step, min(j * _step + _eff, 200)))
+ check_true("window/%s span is no wider than the room the questions leave" % qid,
+ ans["window"]["token_end"] - ans["window"]["token_start"] <= _room,
+ (ans["window"]["token_start"], ans["window"]["token_end"], _room))
check("usage/windows counts the scanned windows", result["usage"]["windows"], len(windows))
check("usage/input_tokens sums the window runs",
result["usage"]["input_tokens"], sum(r["usage"]["input_tokens"] for r in per_window))
@@ -228,9 +239,18 @@ check("batch_size/chunking does not change the answer",
# ---------------------------------------------------------------- explicit window and stride
-check("window/explicit window=96 stride=96 -> three windows",
- _bare_onnx().predict_long(LONG_STATE, QUESTIONS, window=96, stride=96)["usage"]["windows"],
- 3)
+# window=96 is wider than the room these questions leave at max_len=64, so it is clamped -- and the
+# stride the caller paired with it is clamped to match rather than refused, since 96 was a valid step
+# for the 96 they asked for. What must hold is that the scan still covers the state.
+with warnings.catch_warnings():
+ warnings.simplefilter("ignore", RuntimeWarning)
+ _explicit = _bare_onnx().predict_long(LONG_STATE, QUESTIONS, window=96, stride=96)
+_eff96, _step96, _ = window_budget(_bare_onnx().tok,
+ [_bare_onnx()._to_internal(q) for q in QUESTIONS.values()], 64, 32,
+ window=96, stride=96)
+check_true("window/explicit window=96 stride=96 is clamped and still covers the state",
+ _explicit["usage"]["windows"] >= 3 and _step96 <= _eff96,
+ (_explicit["usage"]["windows"], _eff96, _step96))
check_raises("aggregate/anything but auto is refused", ValueError,
lambda: _bare_onnx().predict_long(LONG_STATE, QUESTIONS, aggregate="mean"))
diff --git a/tests/test_predict_long.py b/tests/test_predict_long.py
index 19397de2..3100fef9 100644
--- a/tests/test_predict_long.py
+++ b/tests/test_predict_long.py
@@ -6,6 +6,7 @@ window split, the per-type aggregation (noul = strongest window, choice/score =
window), and the per-call hook controls it forwards to whichever of those two calls runs. Numerical
behaviour on real weights is exercised in tests/test_local_e2e.py.
"""
+import inspect
import os
import re
import sys
@@ -41,11 +42,16 @@ def check_true(name, cond, detail=""):
class _Tok:
+ # The special-token ids and `truncation` are what `laya.common.build_head` reads: predict_long
+ # assembles each question's head to measure the room it leaves for the state before it sizes a
+ # window, so a fake that cannot build a head cannot reach the scan at all.
mask_token = "[M]"
+ cls_token_id, sep_token_id, mask_token_id = 101, 102, 103
- def __call__(self, text, add_special_tokens=False):
+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
# token count == character count, so the test controls windowing by string length
- return {"input_ids": list(range(len(text)))}
+ ids = list(range(len(text)))
+ return {"input_ids": ids[:max_length] if (truncation and max_length) else ids}
def decode(self, ids):
return "w%d_%d" % (ids[0], ids[-1]) if ids else "w"
@@ -600,6 +606,204 @@ check("docs/api.md tabulates a hook answer as zero windows",
check("docs/predict_long's docstring says the key is total",
"always present" in (Agent.predict_long.__doc__ or ""), True)
+# 12. the window is sized by the room the questions leave, not by the config alone
+#
+# Every window is decoded and scored as an ordinary state, so `build_sequence` cuts one that is
+# wider than the room the question's own head leaves inside `max_len`. Sized from the config alone
+# the window was cut short on the way to the model while `answer["window"]["token_end"]` still
+# reported the whole span, and once the room fell under the stride, spans of the document were
+# read by no window at all.
+from laya.common import build_sequence, serialize_state # noqa: E402
+
+TOK = _Tok()
+MAX_LEN, HEAD_MAX_LEN, CONFIG_BUDGET = 100, 20, 72 # make_agent's cfg, and its budget
+
+
+def head_len(qdef):
+ """The head `build_sequence` puts in front of the state, measured with no state at all."""
+ return len(build_sequence(TOK, "", Agent._to_internal(qdef), 10 ** 6, HEAD_MAX_LEN)[0]) - 1
+
+
+def room_for(qdef):
+ """The state tokens `qdef` leaves inside `max_len`, re-derived from `build_sequence`.
+
+ Deliberately not `laya.common.state_room`: the window has to fit what the sequence builder
+ really keeps, so this measures that the long way round instead of trusting the same helper the
+ code under test uses.
+ """
+ return max(0, MAX_LEN - head_len(qdef) - 1)
+
+
+def q_many(n):
+ """A choice question with `n` options, each long enough to be worth capping."""
+ return {"type": "choice", "instructions": "which one?",
+ "criteria": {"opt%02d" % i: "d" * 20 for i in range(n)}}
+
+
+def canned_for(questions, i, n):
+ """One canned answer per window, the last window the confident one."""
+ conf = 0.9 if i == n - 1 else 0.4
+ return {"answers": {qid: {"type": "choice", "choice": "opt00", "noul": conf,
+ "confidence": conf, "answer_confidence": conf}
+ for qid in questions},
+ "usage": {"input_tokens": 10}}
+
+
+def scan(questions, **kw):
+ """Scan LONG and return (result, the token spans of the windows handed to predict_batch)."""
+ agent = make_agent(lambda sts, q: [canned_for(q, i, len(sts)) for i in range(len(sts))])
+ result = agent.predict_long(LONG, questions, **kw)
+ handed = []
+ for text in agent._calls["batch_states"]:
+ first, last = text[1:].split("_") # _Tok.decode renders a span as "wA_B"
+ handed.append((int(first), int(last) + 1))
+ return result, handed
+
+
+def read_spans(handed, room):
+ """What the model reads of each window: `build_sequence` keeps its first `room` tokens."""
+ return [(start, min(end, start + room)) for start, end in handed]
+
+
+def uncovered(spans, total):
+ """Token indices of `total` that no span in `spans` contains."""
+ seen = set()
+ for start, end in spans:
+ seen.update(range(start, end))
+ return sorted(set(range(total)) - seen)
+
+
+STATE_TOKENS = len(TOK(serialize_state(LONG))["input_ids"])
+
+# The premise, checked against build_sequence rather than assumed: a state is cut to the room, and
+# for a 12-option question on this config that room is under the budget the scan sized windows by.
+ROOM_12 = room_for(q_many(12))
+kept = len(build_sequence(TOK, "x" * (MAX_LEN + 50), Agent._to_internal(q_many(12)),
+ MAX_LEN, HEAD_MAX_LEN)[0]) - head_len(q_many(12)) - 1
+check("room/build_sequence keeps exactly the room, whatever the state's length", kept, ROOM_12)
+check("room/12 options leave less room than the config budget", ROOM_12 < CONFIG_BUDGET, True)
+check("room/the questions the sections above scan with leave more room than the budget",
+ [room_for(q) >= CONFIG_BUDGET for q in Q.values()], [True, True])
+
+# 12a. a window is never wider than the room, so no window is truncated on the way in
+tight, handed = scan({"a": q_many(12)})
+check("tight/no window is wider than the room the question leaves",
+ max(end - start for start, end in handed) <= ROOM_12, True)
+check("tight/the windows still overlap",
+ all(b[0] < a[1] for a, b in zip(handed, handed[1:])), True)
+check("tight/every token of the state is read by some window",
+ uncovered(read_spans(handed, ROOM_12), STATE_TOKENS), [])
+
+# 12b. and the span reported is the span read, not the span asked for
+decided = tight["answers"]["a"]["window"]
+check("tight/token_end does not overstate what the model read",
+ decided["token_end"] - decided["token_start"] <= ROOM_12, True)
+check("tight/token_start/token_end are the deciding window's own span",
+ (decided["token_start"], decided["token_end"]), handed[decided["index"]])
+check("tight/the last window stops at the end of the state", handed[-1][1], STATE_TOKENS)
+
+# 12c. the failure that needed no wide window at all: a room under the stride left whole spans of
+# the document unread, because consecutive windows no longer touched.
+ROOM_20 = room_for(q_many(20))
+check("gap/20 options leave less room than the default stride",
+ ROOM_20 < CONFIG_BUDGET // 2, True)
+gappy, handed = scan({"a": q_many(20)})
+check("gap/the scan covers the whole state even so",
+ uncovered(read_spans(handed, ROOM_20), STATE_TOKENS), [])
+check("gap/which costs windows, and reports them", gappy["usage"]["windows"], len(handed))
+
+# 12d. two questions with different rooms share one list of windows, so the tightest one sets it
+mixed, handed = scan({"wide": q_many(2), "tight": q_many(12)})
+check("mixed/the window fits the tightest question",
+ max(end - start for start, end in handed) <= min(ROOM_12, room_for(q_many(2))), True)
+check("mixed/both questions name the same span",
+ mixed["answers"]["wide"]["window"], mixed["answers"]["tight"]["window"])
+
+# 12e. an explicit window past the room is clamped, and the caller is told
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ asked, handed = scan(Q, window=200)
+check("explicit/a window past the room is clamped to it",
+ max(end - start for start, end in handed) <= room_for(Q["dept"]), True)
+check("explicit/and the caller is warned about it",
+ [w.category.__name__ for w in caught], ["RuntimeWarning"])
+check("explicit/the warning names the room it clamped to",
+ "leave inside max_len" in str(caught[0].message) if caught else False, True)
+
+# Clamping the *default* is not the caller's doing, so it is not warned about.
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ scan({"a": q_many(12)})
+check("explicit/clamping the default window is not warned about",
+ [w.category.__name__ for w in caught], [])
+
+# 12f. a stride past the effective window is refused: the tokens between two windows would be read
+# by nothing, which is the failure this method exists to prevent.
+check_raises("stride/past the effective window is an error", ValueError,
+ lambda: scan(Q, stride=room_for(Q["dept"]) + 1))
+# A stride the caller paired with a window they asked for is NOT their error when the library then
+# reduces that window: window=90 with stride=90 is a self-consistent "no overlap, read everything".
+# Refusing it made the library's own clamp look like the caller's mistake, and forced every such
+# caller -- including `tests/test_onnx_long.py`'s window=96 stride=96 -- to be edited. It is clamped
+# to half the effective window under the same RuntimeWarning, and the scan still covers the state.
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ _clamped = scan({"a": q_many(12)}, window=90, stride=90)
+check_true("stride/a stride valid for the requested window is clamped, not refused",
+ _clamped[0]["usage"]["windows"] > 1, _clamped[0]["usage"])
+check_true("stride/and the caller is told the stride moved too",
+ any("stride=90" in str(w.message) and "reduced" in str(w.message) for w in caught),
+ [str(w.message)[:90] for w in caught])
+_room12 = room_for(q_many(12))
+check_true("stride/the clamped scan leaves no gap", all(
+ b - a <= _room12 for a, b in read_spans(_clamped[1], _room12)), _clamped[1][:3])
+# A stride past the window the caller actually asked for is still their error.
+check_raises("stride/past the window the caller asked for is still refused", ValueError,
+ lambda: scan({"a": q_many(12)}, window=40, stride=80))
+check("stride/a stride inside the window is accepted",
+ scan(Q, window=40, stride=40)[0]["usage"]["windows"] > 1, True)
+
+# 12f-bis. the ONNX path honours the same cap, because README and Router promise it for both
+try: # noqa: E402
+ from laya.common import window_batch_cap, window_budget
+except ImportError: # source reverted: let the checks below go red rather than abort the suite
+ def window_budget(*_a, **_k): # type: ignore[misc]
+ return (0, 0, 0)
+
+ def window_batch_cap(*_a, **_k): # type: ignore[misc]
+ return -1
+from laya.onnx_agent import ONNXAgent # noqa: E402
+
+_onnx = ONNXAgent.__new__(ONNXAgent)
+_onnx.tok = _Tok()
+_onnx.cfg = {"max_len": MAX_LEN, "head_max_len": HEAD_MAX_LEN}
+_q_onnx = {"a": q_many(12)} # 100 options fills this tiny harness entirely
+_eff_onnx, _step_onnx, _room_onnx = window_budget(
+ _onnx.tok, [_onnx._to_internal(_q_onnx["a"])], MAX_LEN, HEAD_MAX_LEN)
+check_true("onnx/predict_long caps the window at the room too",
+ _eff_onnx <= _room_onnx and _eff_onnx < max(64, MAX_LEN - HEAD_MAX_LEN - 8),
+ (_eff_onnx, _room_onnx))
+check_true("onnx/and its stride stays inside that window", _step_onnx <= _eff_onnx,
+ (_step_onnx, _eff_onnx))
+check_true("onnx/predict_long actually calls window_budget",
+ "window_budget(" in inspect.getsource(ONNXAgent.predict_long),
+ "the README and Router docstring promise the cap for both agents")
+
+# 12f-ter. capping the window must not turn one forward pass into an out-of-memory
+check("batch/a mild cap keeps the single shared pass", window_batch_cap(10, 43, 64), None)
+check("batch/no cap at all keeps it too", window_batch_cap(16, 312, 312), None)
+check_true("batch/a blow-up is bounded to the un-capped pass width",
+ 0 < (window_batch_cap(413, 23, 312) or 0) < 413, window_batch_cap(413, 23, 312))
+check("batch/an explicit batch_size is always honoured", window_batch_cap(413, 23, 312, 8), 8)
+
+# 12g. the page that teaches the default teaches the cap as well
+window_para = README[README.index("A smaller `window` isolates"):][:900]
+check("docs/README says the window is capped at the room the questions leave",
+ "capped at the room the questions leave" in window_para, True)
+check("docs/predict_long's docstring documents the cap",
+ "capped at the room the" in (Agent.predict_long.__doc__ or ""), True)
+
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
diff --git a/tests/test_question_token_reuse.py b/tests/test_question_token_reuse.py
index a68e48e9..9adf806e 100644
--- a/tests/test_question_token_reuse.py
+++ b/tests/test_question_token_reuse.py
@@ -88,10 +88,16 @@ class QuestionTokenReuse(unittest.TestCase):
def test_predict_long_uses_the_same_cache_for_all_windows(self):
tok = Tokenizer()
agent = agent_with(tok)
- agent.predict_long("x" * 1000, QUESTIONS, window=80, stride=50, batch_size=3)
+ scanned = agent.predict_long("x" * 1000, QUESTIONS, window=80, stride=50, batch_size=3)
head = "choice question: " + QUESTION["ins"]
- self.assertEqual(sum(text == head for _, text, _ in tok.calls), 1)
- self.assertEqual(sum(text == " billing: charges" for _, text, _ in tok.calls), 1)
+ # Twice, not once per window: predict_long assembles each question's head once on its own
+ # to measure the room the question leaves for the state -- the cap on the window -- and the
+ # scan that follows then shares one cache across every window it scores. The point of this
+ # test is the second number: the head is encoded a fixed number of times, not once per
+ # window, which the assertion below states as such.
+ self.assertEqual(sum(text == head for _, text, _ in tok.calls), 2)
+ self.assertEqual(sum(text == " billing: charges" for _, text, _ in tok.calls), 2)
+ self.assertGreater(scanned["usage"]["windows"], 2)
self.assertGreater(len(agent.batches), 1)
def test_sequence_parity_for_rendering_order_and_budgets(self):