Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): bound predict_long's window by the room a question leaves
Adeshina Lasisief93a9471531
diff --git a/README.md b/README.mdindex 328a6194..1eef506e 100644--- a/README.md+++ b/README.md@@ -712,7 +712,11 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,`usage["windows"]` at 0, because no window scored it.A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its-window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches+window); the default (`max_len - head_max_len`) favors context and throughput. Either way the+window is capped at the room the questions leave for the state inside `max_len`, so a window is+never re-truncated on the way to the model and `token_start`/`token_end` describe the span it read:+many options leave little room (on the English checkpoint, 2 options leave 483 state tokens and+100 leave 100), and a window past that room used to be cut short silently. Output shape matches`predict`, with `usage["windows"]` added.`ONNXAgent.predict_long(state, questions, window=..., stride=..., batch_size=...)` has the samediff --git a/laya/agent.py b/laya/agent.pyindex c98e6859..f8229fbb 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -41,6 +41,8 @@ from .common import (serialize_state,temp_bucket,unpermute_probs,+ window_batch_cap,+ window_budget,)from .confidence import check_min_confidence, flag_low_confidencefrom .hooks import (@@ -1307,6 +1309,8 @@ class Agent(HookRegistry):document is decisive. Each answer therefore carries `answer["window"]` — the decidingwindow's `index`, `token_start`/`token_end` into the tokenized state, and the window `count`— so a caller can inspect the span the answer came from rather than trust the raw number.+ That span is the one the model read, not merely the one asked for: the window is capped at+ the room the questions leave, so what is handed to `predict_batch` is not cut short again.A state that already fits one window is passed straight to `system_one` (identical output).@@ -1324,15 +1328,24 @@ class Agent(HookRegistry):above describe this method's windows, not the text the model readArgs:- window: state tokens per window. Defaults to the per-question state budget- (`max_len - head_max_len - 8`) -- the most a window can hold for every question.+ window: state tokens per window. Defaults to the checkpoint's state budget+ (`max_len - head_max_len - 8`), and either way is capped at the room the+ questions leave for the state inside `max_len` -- the smallest room of them,+ because one list of windows is scored for every question. A wider window is+ re-truncated on the way to the model, so it is clamped instead, with a+ `RuntimeWarning` when the caller is the one who asked for it. Options are what+ make the room small: on the English checkpoint a 2-option question leaves 483+ tokens for the state and a 100-option one leaves 100.A smaller window isolates a localized signal better (a short deciding span is alarger fraction of its window, so that window classifies it clearly), at thecost of more windows; the large default favors context and throughput. `noul`is robust to this, `choice`/`score` benefit from a smaller window when thedeciding span is a small part of a long, otherwise-neutral document.- stride: token step between windows. Defaults to `window // 2` (50% overlap), so a span- near a boundary still lands whole inside some window.+ stride: token step between windows. Defaults to half the *effective* window (50%+ overlap), so a span near a boundary still lands whole inside some window. A+ stride past the effective window is refused rather than clamped: the tokens+ between each pair of windows would be read by no window at all, which is the+ failure this method exists to prevent.aggregate: "auto" (the per-type rules above) is the only mode for now.batch_size: cap on windows per forward pass, to bound memory on very long states.lang: per-language temperature selection, as in `system_one`.@@ -1343,6 +1356,11 @@ class Agent(HookRegistry):hooks_raise: Override the Agent's `hooks_raise` for this call.hooks_timeout: Override the Agent's `hooks_timeout` for this call.+ Raises:+ ValueError: `aggregate` is anything but "auto"; the questions' options fill the whole+ sequence, leaving no room for the state; or `stride` steps past the effective+ window, so tokens between two windows would be read by nothing.+Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.The key is always present and counts the windows the model scored to produce the answer: `1`for a state that fit one window, `N` for a document scanned in `N` overlapping windows (or@@ -1371,7 +1389,21 @@ class Agent(HookRegistry):"hooks_timeout": hooks_timeout}max_len = self.cfg.get("max_len", 512)head_max_len = self.cfg.get("head_max_len", 192)- budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)+ # Size the window against the room these questions actually leave, not the config alone:+ # every window is decoded and scored as a normal state, so a window wider than the room+ # was re-truncated by `build_sequence` on the way in and the tail of it reached no model+ # -- while `answer["window"]` reported the whole span. Measured on the English checkpoint+ # (max_len=512, head_max_len=192, config budget 312): a 2-option question leaves room for+ # 483 tokens, 48 options leave 308, and 100 -- `serve`'s documented maximum -- leave 100.+ # At 87 options the room (152) fell below the 156-token default stride, so the windows+ # stopped overlapping and part of the document was read by no window at all: measured on a+ # 920-token document at 100 options, 420 of its tokens reached no window.+ ids = list(questions.keys())+ for qid in ids:+ self._check_question(qid, questions[qid])+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}+ budget, step, _ = window_budget(self.tok, [internal[qid] for qid in ids], max_len,+ head_max_len, window=window, stride=stride)state_ids = self.tok(serialize_state(state).replace(self.tok.mask_token, " "),add_special_tokens=False)["input_ids"]@@ -1390,7 +1422,6 @@ class Agent(HookRegistry):single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}return single- step = stride if (stride and stride > 0) else max(1, budget // 2)windows, starts = [], []i, n = 0, len(state_ids)while i < n:@@ -1407,7 +1438,11 @@ class Agent(HookRegistry):# `list(windows)`, not `windows`: `ctx.states` is the list the hook receives, so a hook that# mutates it in place (`append`, `sort`) would otherwise also grow the split this call# attributes answers to, and the counts would agree while `starts` no longer lined up.- results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,+ # Bound one forward pass to what the un-capped scan would have used; see+ # `window_batch_cap`. An explicit batch_size is honoured untouched.+ cap = window_batch_cap(len(windows), budget, max(64, max_len - head_max_len - 8),+ batch_size)+ results = self.predict_batch(list(windows), questions, batch_size=cap, lang=lang,**_with_start_probe(hook_kwargs, probe))if evidence["answered"]:diff --git a/laya/common.py b/laya/common.pyindex daf57a95..7a1b97e3 100644--- a/laya/common.py+++ b/laya/common.py@@ -3,6 +3,7 @@ import jsonimport mathimport osimport threading+import warningsfrom contextlib import nullcontextfrom contextvars import ContextVarfrom functools import wraps@@ -165,6 +166,41 @@ def build_sequence(the rendered options - so it moves per checkpoint and per question. A caller guessing with afixed character threshold is wrong in both directions: it reports truncation that did nothappen, and stays silent while evidence is being dropped (issue #174).++ The question half is `build_head`; `state_room` reports how much of `max_len` is left for the+ state after it, which is what a caller must size a window against.+ """+ ids, markers, stats = build_head(tok, q, head_max_len, option_order=option_order)+ room = max(0, max_len - len(ids) - 1)+ if state_ids is None:+ state_ids = encode_text(tok, serialize_state(state).replace(tok.mask_token, " "),+ add_special_tokens=False)["input_ids"]+ # not state_ids[-room:]: with no room left, state_ids[-0:] is the whole state rather than none of it+ st = state_ids[max(0, len(state_ids) - room):] if truncate_left else state_ids[:room]+ ids = ids + st + [tok.sep_token_id]+ ids, markers = ids[:max_len], [m for m in markers if m < max_len]+ extra = ()+ if return_truncation_stats:+ # `room` leaves space for the closing [SEP], so every token in `st` survives the [:max_len] clamp+ extra = ({+ "state_tokens": len(state_ids),+ "state_tokens_used": len(st),+ "state_tokens_dropped": len(state_ids) - len(st),+ "truncated": len(st) < len(state_ids),+ },)+ if not return_stats:+ return (ids, markers) + extra+ return (ids, markers, stats) + extra+++def build_head(tok, q: Dict, head_max_len: int = 192, option_order: Optional[List[int]] = None):+ """The question half of a sequence: `[CLS] <type> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP]`.++ `build_sequence` appends the state to this and `state_room` measures what is left over for it,+ so neither can disagree with the other about what the head costs.++ Returns `(ids, markers, stats)`: the head token ids, the `[MASK]` position of each option in+ `option_order`, and the stats dict `build_sequence` returns for `return_stats`."""mask_tok = tok.mask_tokenopts = render_options(q)@@ -198,38 +234,128 @@ def build_sequence(markers.append(len(ids))ids.extend(o)ids.append(tok.sep_token_id)- room = max(0, max_len - len(ids) - 1)- if state_ids is None:- state_ids = encode_text(tok, serialize_state(state).replace(mask_tok, " "),- add_special_tokens=False)["input_ids"]- # not state_ids[-room:]: with no room left, state_ids[-0:] is the whole state rather than none of it- st = state_ids[max(0, len(state_ids) - room):] if truncate_left else state_ids[:room]- ids = ids + st + [tok.sep_token_id]- ids, markers = ids[:max_len], [m for m in markers if m < max_len]- extra = ()- if return_truncation_stats:- # `room` leaves space for the closing [SEP], so every token in `st` survives the [:max_len] clamp- extra = ({- "state_tokens": len(state_ids),- "state_tokens_used": len(st),- "state_tokens_dropped": len(state_ids) - len(st),- "truncated": len(st) < len(state_ids),- },)- if not return_stats:- return (ids, markers) + extra# Two options that share a prefix can come out of the cut as the same token span: the marker# count still matches the option count, so the guard in `Agent._encode_state` passes and# nothing downstream can tell that the question lost the ability to name them apart. Counted# on the capped option ids, before assembly: re-slicing the finished sequence cannot close# the last option's span -- it runs on into the serialized state, which differs per request,# so the last option always looks distinguishable however it collided (#538).- return (ids, markers, {+ return ids, markers, {"options": len(opt_ids),"options_distinct": len({tuple(o) for o in opt_ids}),"tokens_per_option": per_option,- }) + extra+ }+def state_room(tok, q: Dict, max_len: int = 512, head_max_len: int = 192) -> int:+ """How many state tokens `q` leaves inside `max_len`, which is what `build_sequence` keeps.++ The head is the question's own -- its instructions plus one `[MASK]`-prefixed span per option --+ so a question with many options leaves less room for the state than one with two, and two+ questions in the same request do not have to leave the same amount. Anything past the return+ value is cut off (the start is kept, or the end for a conversation turn list).+ """+ head, _, _ = build_head(tok, q, head_max_len)+ return max(0, max_len - len(head) - 1) # -1 for the [SEP] that closes the state+++# How much capping the window may multiply the window count before `window_batch_cap` starts+# chunking. Under this, chunking would cost the single-shared-pass property for no real protection.+_WINDOW_BATCH_BLOWUP = 2++def window_budget(tok, questions, max_len: int = 512, head_max_len: int = 192,+ window: Optional[int] = None, stride: Optional[int] = None):+ """Window size and stride for scanning a state that is longer than one sequence.++ `predict_long` decodes each token window back to text and scores it as an ordinary state, so a+ window wider than the room the questions leave is re-truncated by `build_sequence` on the way+ in: the tail of every window reaches no model, while the reported span says it did. The window+ is therefore capped at `state_room`, and at the *smallest* room of `questions`, because the+ windows are one list of states scored for every question in shared forward passes -- a window+ sized for the roomiest question would be cut short for the tightest one, and the offsets+ reported on its answers would mean something different per question. With no questions there is+ nothing to fit, so the caller's window (or the checkpoint default) stands.++ `questions` are internal question dicts, as `Agent._to_internal` returns them.++ An explicit `window` wider than the room is clamped to it with a `RuntimeWarning`, since a+ scan at the requested size cannot read what it claims to. A `stride` past the effective window+ is refused: the tokens between two windows would be read by neither, which is the failure+ `predict_long` exists to prevent. The default stride keeps its 50% overlap of the *effective*+ window, so a span near a boundary still lands whole inside some window; an explicit stride+ equal to the window still reads every token, with no overlap to catch a span that straddles a+ boundary.++ Returns `(window, stride, room)`.+ """+ rooms = [state_room(tok, q, max_len, head_max_len) for q in questions]+ requested = window if (window and window > 0) else max(64, max_len - head_max_len - 8)+ size = requested+ room = min(rooms) if rooms else size+ if room <= 0:+ raise ValueError(+ "predict_long: the questions' options fill the whole sequence (max_len=%d,"+ " head_max_len=%d), leaving no room for the state; no window can carry any of it"+ % (max_len, head_max_len))+ if size > room:+ if window and window > 0:+ warnings.warn(+ "laya: predict_long: window=%d is wider than the %d state tokens these questions"+ " leave inside max_len=%d, so every window would be truncated to %d on the way to"+ " the model; scanning with window=%d instead" % (size, room, max_len, room, room),+ RuntimeWarning, stacklevel=3)+ size = room+ step = stride if (stride and stride > 0) else max(1, size // 2)+ if step > size:+ if size < requested and stride and stride <= requested:+ # The window the caller asked for was reduced above, and their stride was valid for the+ # window they asked for -- so this is the library's clamp, not their mistake. Reducing+ # the stride to match keeps the no-gap guarantee without making a self-consistent pair+ # of arguments an error. A stride that overshot the *requested* window is still refused+ # below, because that one really is the caller's.+ warnings.warn(+ "laya: predict_long: stride=%d was a 50%% step for the window=%d you asked for, but"+ " the window was reduced to %d to fit the room these questions leave; scanning with"+ " stride=%d instead" % (step, requested, size, max(1, size // 2)),+ RuntimeWarning, stacklevel=3)+ step = max(1, size // 2)+ else:+ raise ValueError(+ "predict_long: stride=%d steps past the %d-token window%s, so %d tokens between"+ " every pair of windows would be read by no window at all; pass stride <= %d"+ % (step, size, " these questions leave room for" if size < requested else "",+ step - size, size))+ return size, step, room++++def window_batch_cap(n_windows: int, window: int, config_budget: int,+ batch_size: Optional[int] = None) -> Optional[int]:+ """Keep one forward pass no wider than the un-capped scan's would have been -- but only when+ capping the window has multiplied the window count enough to matter.++ Capping the window at the room the questions leave multiplies the number of windows on exactly+ the inputs it targets: measured, a 4 561-token document at 120 options goes from 29 windows of+ 312 tokens to roughly 413 of 23. `predict_batch` with `batch_size=None` puts every state in one+ forward pass, so a caller who passed no `batch_size` would go from a 29-row pass to a 413-row one+ at `max_len` width -- a plausible out-of-memory on an input that used to fit.++ Sending every window in one run is also a deliberate property (`ONNXAgent` asserts it), and a mild+ cap -- a 64-token budget reduced to 43, say -- multiplies the count by well under two. Chunking+ those would trade a real property for no real protection. So the cap only applies past+ `_WINDOW_BATCH_BLOWUP`x, and then bounds the pass at the count the un-capped budget would have+ produced: peak memory stays at parity with the behaviour before the cap, and the scan still reads+ the whole document, just in more passes. An explicit `batch_size` is always honoured.+ """+ if batch_size and batch_size > 0:+ return batch_size+ if window >= config_budget:+ return None # not capped: one pass, exactly as before+ unclamped = max(1, -(-n_windows * max(1, window // 2) // max(1, config_budget // 2)))+ if n_windows <= unclamped * _WINDOW_BATCH_BLOWUP:+ return None # a mild cap: keep the single shared pass+ return unclamped+def collapsed_options(qids, items) -> Dict[str, Dict[str, Optional[int]]]:"""The questions whose options no longer have a token span each, from per-item stats.diff --git a/laya/onnx_agent.py b/laya/onnx_agent.pyindex 8a67f065..e7188784 100644--- a/laya/onnx_agent.py+++ b/laya/onnx_agent.py@@ -24,6 +24,8 @@ from laya.common import (render_options,resolve_lang_temperatures,serialize_state,+ window_batch_cap,+ window_budget,temp_bucket,unpermute_probs,TEMP_MIN,@@ -453,7 +455,16 @@ class ONNXAgent(HookRegistry):raise ValueError("predict_long: only aggregate='auto' is supported")max_len = self.cfg.get("max_len", 512)head_max_len = self.cfg.get("head_max_len", 192)- budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)+ # The same cap the torch `Agent.predict_long` applies, for the same reason: a window wider+ # than the room these questions leave is re-truncated by `build_sequence` on the way into+ # `predict_batch`, so its tail reaches no model while `answer["window"]` reports the whole+ # span -- and once the room falls below the default stride the windows stop overlapping and+ # leave tokens no window reads at all. `README.md` and `Router.predict_long` document this+ # contract for both agents, so both have to honour it.+ ids = list(questions.keys())+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}+ budget, step_default, _ = window_budget(self.tok, [internal[qid] for qid in ids], max_len,+ head_max_len, window=window, stride=stride)state_ids = encode_text(self.tok,@@ -469,7 +480,7 @@ class ONNXAgent(HookRegistry):single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}return single- step = stride if (stride and stride > 0) else max(1, budget // 2)+ step = step_defaultwindows, starts = [], []i, n = 0, len(state_ids)while i < n:@@ -483,7 +494,11 @@ class ONNXAgent(HookRegistry):probe, evidence = _start_evidence()# A copy, so a start hook that mutates `ctx.states` in place cannot shift `starts`.- results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,+ # Bound one forward pass to what the un-capped scan would have used; see+ # `window_batch_cap`. An explicit batch_size is honoured untouched.+ cap = window_batch_cap(len(windows), budget, max(64, max_len - head_max_len - 8),+ batch_size)+ results = self.predict_batch(list(windows), questions, batch_size=cap, lang=lang,**_with_start_probe(hook_kwargs, probe))if evidence["answered"]:diff --git a/laya/router.py b/laya/router.pyindex 36a0ec6d..9ae7e2c6 100644--- a/laya/router.py+++ b/laya/router.py@@ -846,8 +846,12 @@ class Router(HookRegistry):Args:window: state tokens per window. Defaults to the routed checkpoint's budget- (`max_len - head_max_len - 8`); a smaller window isolates a localized span.- stride: token step between windows; defaults to `window // 2` (50% overlap).+ (`max_len - head_max_len - 8`), capped at the room the questions leave for the+ state so no window is truncated again on the way in; a smaller window isolates+ a localized span.+ stride: token step between windows; defaults to half the effective window (50%+ overlap). A stride past that window is a `ValueError`, since the tokens between+ windows would reach no model.aggregate: "auto" (the per-type rules above) is the only mode.batch_size: cap on windows per forward pass, to bound memory on very long states.diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex ccd59572..196ecf95 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -297,10 +297,21 @@ for label, cls in (("LayaRouter", LayaRouter), ("LayaGuardrail", LayaGuardrail),# `input_tokens` / `output_tokens` are the fields every client decodes, so they are always# present. `options` (#538) is additive and conditional: it appears only for a request whose# options lost their distinct token spans, which is what keeps it out of ordinary responses.-from laya.common import build_sequence, collapsed_options # noqa: E402+from laya.common import build_head, build_sequence, collapsed_options, state_room, window_budget # noqa: E402check_param("build_sequence", build_sequence, "return_stats", False)check_param("build_sequence", build_sequence, "return_truncation_stats", False)++# The sizing surface `predict_long` reads before it splits a state: `build_head` is the question+# half `build_sequence` assembles, `state_room` is what is left of `max_len` for the state after+# it, and `window_budget` turns the two into the window and stride a scan may use. Their defaults+# are the checkpoint's, so a caller can measure a question without holding an Agent.+for _name, _fn, _params in (("build_head", build_head, (("head_max_len", 192), ("option_order", None))),+ ("state_room", state_room, (("max_len", 512), ("head_max_len", 192))),+ ("window_budget", window_budget, (("max_len", 512), ("head_max_len", 192),+ ("window", None), ("stride", None)))):+ for _param, _default in _params:+ check_param(_name, _fn, _param, _default)check("collapsed_options/nothing collapsed is empty",collapsed_options(["q"], [{"options": {"options": 3, "options_distinct": 3,"tokens_per_option": None}}]), {})diff --git a/tests/test_onnx_long.py b/tests/test_onnx_long.pyindex a87ab443..8a254f63 100644--- a/tests/test_onnx_long.py+++ b/tests/test_onnx_long.py@@ -12,6 +12,7 @@ Run: python tests/test_onnx_long.pyimport inspectimport osimport sys+import warningsimport threadingsys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))@@ -19,6 +20,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))import numpy as np # noqa: E402from laya.agent import Agent # noqa: E402+from laya.common import window_budget # noqa: E402from laya.onnx_agent import ONNXAgent # noqa: E402PASS, FAIL = [], []@@ -166,9 +168,18 @@ for qid, key in (("dept", "answer_confidence"), ("urgent", "noul")):check_true("aggregate/%s is the strongest window on its rule" % qid,ans[key] == max(pw["answers"][qid][key] for pw in per_window),[pw["answers"][qid][key] for pw in per_window])+ # Derived from the *effective* window, not from `max_len - head_max_len - 8`: these questions+ # leave less room than that budget at this tiny config, so the window is capped at the room and+ # the reported span has to describe what the model actually read. Hardcoding 64/32 here asserted+ # a `token_end` that overstated the span by the difference.+ _eff, _step, _room = window_budget(agent.tok, [agent._to_internal(q) for q in QUESTIONS.values()],+ agent.cfg["max_len"], agent.cfg["head_max_len"])check("window/%s fields index the deciding span into the original state" % qid,(ans["window"]["count"], ans["window"]["token_start"], ans["window"]["token_end"]),- (len(windows), j * 32, min(j * 32 + 64, 200)))+ (len(windows), j * _step, min(j * _step + _eff, 200)))+ check_true("window/%s span is no wider than the room the questions leave" % qid,+ ans["window"]["token_end"] - ans["window"]["token_start"] <= _room,+ (ans["window"]["token_start"], ans["window"]["token_end"], _room))check("usage/windows counts the scanned windows", result["usage"]["windows"], len(windows))check("usage/input_tokens sums the window runs",result["usage"]["input_tokens"], sum(r["usage"]["input_tokens"] for r in per_window))@@ -228,9 +239,18 @@ check("batch_size/chunking does not change the answer",# ---------------------------------------------------------------- explicit window and stride-check("window/explicit window=96 stride=96 -> three windows",- _bare_onnx().predict_long(LONG_STATE, QUESTIONS, window=96, stride=96)["usage"]["windows"],- 3)+# window=96 is wider than the room these questions leave at max_len=64, so it is clamped -- and the+# stride the caller paired with it is clamped to match rather than refused, since 96 was a valid step+# for the 96 they asked for. What must hold is that the scan still covers the state.+with warnings.catch_warnings():+ warnings.simplefilter("ignore", RuntimeWarning)+ _explicit = _bare_onnx().predict_long(LONG_STATE, QUESTIONS, window=96, stride=96)+_eff96, _step96, _ = window_budget(_bare_onnx().tok,+ [_bare_onnx()._to_internal(q) for q in QUESTIONS.values()], 64, 32,+ window=96, stride=96)+check_true("window/explicit window=96 stride=96 is clamped and still covers the state",+ _explicit["usage"]["windows"] >= 3 and _step96 <= _eff96,+ (_explicit["usage"]["windows"], _eff96, _step96))check_raises("aggregate/anything but auto is refused", ValueError,lambda: _bare_onnx().predict_long(LONG_STATE, QUESTIONS, aggregate="mean"))diff --git a/tests/test_predict_long.py b/tests/test_predict_long.pyindex 19397de2..3100fef9 100644--- a/tests/test_predict_long.py+++ b/tests/test_predict_long.py@@ -6,6 +6,7 @@ window split, the per-type aggregation (noul = strongest window, choice/score =window), and the per-call hook controls it forwards to whichever of those two calls runs. Numericalbehaviour on real weights is exercised in tests/test_local_e2e.py."""+import inspectimport osimport reimport sys@@ -41,11 +42,16 @@ def check_true(name, cond, detail=""):class _Tok:+ # The special-token ids and `truncation` are what `laya.common.build_head` reads: predict_long+ # assembles each question's head to measure the room it leaves for the state before it sizes a+ # window, so a fake that cannot build a head cannot reach the scan at all.mask_token = "[M]"+ cls_token_id, sep_token_id, mask_token_id = 101, 102, 103- def __call__(self, text, add_special_tokens=False):+ def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):# token count == character count, so the test controls windowing by string length- return {"input_ids": list(range(len(text)))}+ ids = list(range(len(text)))+ return {"input_ids": ids[:max_length] if (truncation and max_length) else ids}def decode(self, ids):return "w%d_%d" % (ids[0], ids[-1]) if ids else "w"@@ -600,6 +606,204 @@ check("docs/api.md tabulates a hook answer as zero windows",check("docs/predict_long's docstring says the key is total","always present" in (Agent.predict_long.__doc__ or ""), True)+# 12. the window is sized by the room the questions leave, not by the config alone+#+# Every window is decoded and scored as an ordinary state, so `build_sequence` cuts one that is+# wider than the room the question's own head leaves inside `max_len`. Sized from the config alone+# the window was cut short on the way to the model while `answer["window"]["token_end"]` still+# reported the whole span, and once the room fell under the stride, spans of the document were+# read by no window at all.+from laya.common import build_sequence, serialize_state # noqa: E402++TOK = _Tok()+MAX_LEN, HEAD_MAX_LEN, CONFIG_BUDGET = 100, 20, 72 # make_agent's cfg, and its budget+++def head_len(qdef):+ """The head `build_sequence` puts in front of the state, measured with no state at all."""+ return len(build_sequence(TOK, "", Agent._to_internal(qdef), 10 ** 6, HEAD_MAX_LEN)[0]) - 1+++def room_for(qdef):+ """The state tokens `qdef` leaves inside `max_len`, re-derived from `build_sequence`.++ Deliberately not `laya.common.state_room`: the window has to fit what the sequence builder+ really keeps, so this measures that the long way round instead of trusting the same helper the+ code under test uses.+ """+ return max(0, MAX_LEN - head_len(qdef) - 1)+++def q_many(n):+ """A choice question with `n` options, each long enough to be worth capping."""+ return {"type": "choice", "instructions": "which one?",+ "criteria": {"opt%02d" % i: "d" * 20 for i in range(n)}}+++def canned_for(questions, i, n):+ """One canned answer per window, the last window the confident one."""+ conf = 0.9 if i == n - 1 else 0.4+ return {"answers": {qid: {"type": "choice", "choice": "opt00", "noul": conf,+ "confidence": conf, "answer_confidence": conf}+ for qid in questions},+ "usage": {"input_tokens": 10}}+++def scan(questions, **kw):+ """Scan LONG and return (result, the token spans of the windows handed to predict_batch)."""+ agent = make_agent(lambda sts, q: [canned_for(q, i, len(sts)) for i in range(len(sts))])+ result = agent.predict_long(LONG, questions, **kw)+ handed = []+ for text in agent._calls["batch_states"]:+ first, last = text[1:].split("_") # _Tok.decode renders a span as "wA_B"+ handed.append((int(first), int(last) + 1))+ return result, handed+++def read_spans(handed, room):+ """What the model reads of each window: `build_sequence` keeps its first `room` tokens."""+ return [(start, min(end, start + room)) for start, end in handed]+++def uncovered(spans, total):+ """Token indices of `total` that no span in `spans` contains."""+ seen = set()+ for start, end in spans:+ seen.update(range(start, end))+ return sorted(set(range(total)) - seen)+++STATE_TOKENS = len(TOK(serialize_state(LONG))["input_ids"])++# The premise, checked against build_sequence rather than assumed: a state is cut to the room, and+# for a 12-option question on this config that room is under the budget the scan sized windows by.+ROOM_12 = room_for(q_many(12))+kept = len(build_sequence(TOK, "x" * (MAX_LEN + 50), Agent._to_internal(q_many(12)),+ MAX_LEN, HEAD_MAX_LEN)[0]) - head_len(q_many(12)) - 1+check("room/build_sequence keeps exactly the room, whatever the state's length", kept, ROOM_12)+check("room/12 options leave less room than the config budget", ROOM_12 < CONFIG_BUDGET, True)+check("room/the questions the sections above scan with leave more room than the budget",+ [room_for(q) >= CONFIG_BUDGET for q in Q.values()], [True, True])++# 12a. a window is never wider than the room, so no window is truncated on the way in+tight, handed = scan({"a": q_many(12)})+check("tight/no window is wider than the room the question leaves",+ max(end - start for start, end in handed) <= ROOM_12, True)+check("tight/the windows still overlap",+ all(b[0] < a[1] for a, b in zip(handed, handed[1:])), True)+check("tight/every token of the state is read by some window",+ uncovered(read_spans(handed, ROOM_12), STATE_TOKENS), [])++# 12b. and the span reported is the span read, not the span asked for+decided = tight["answers"]["a"]["window"]+check("tight/token_end does not overstate what the model read",+ decided["token_end"] - decided["token_start"] <= ROOM_12, True)+check("tight/token_start/token_end are the deciding window's own span",+ (decided["token_start"], decided["token_end"]), handed[decided["index"]])+check("tight/the last window stops at the end of the state", handed[-1][1], STATE_TOKENS)++# 12c. the failure that needed no wide window at all: a room under the stride left whole spans of+# the document unread, because consecutive windows no longer touched.+ROOM_20 = room_for(q_many(20))+check("gap/20 options leave less room than the default stride",+ ROOM_20 < CONFIG_BUDGET // 2, True)+gappy, handed = scan({"a": q_many(20)})+check("gap/the scan covers the whole state even so",+ uncovered(read_spans(handed, ROOM_20), STATE_TOKENS), [])+check("gap/which costs windows, and reports them", gappy["usage"]["windows"], len(handed))++# 12d. two questions with different rooms share one list of windows, so the tightest one sets it+mixed, handed = scan({"wide": q_many(2), "tight": q_many(12)})+check("mixed/the window fits the tightest question",+ max(end - start for start, end in handed) <= min(ROOM_12, room_for(q_many(2))), True)+check("mixed/both questions name the same span",+ mixed["answers"]["wide"]["window"], mixed["answers"]["tight"]["window"])++# 12e. an explicit window past the room is clamped, and the caller is told+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ asked, handed = scan(Q, window=200)+check("explicit/a window past the room is clamped to it",+ max(end - start for start, end in handed) <= room_for(Q["dept"]), True)+check("explicit/and the caller is warned about it",+ [w.category.__name__ for w in caught], ["RuntimeWarning"])+check("explicit/the warning names the room it clamped to",+ "leave inside max_len" in str(caught[0].message) if caught else False, True)++# Clamping the *default* is not the caller's doing, so it is not warned about.+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ scan({"a": q_many(12)})+check("explicit/clamping the default window is not warned about",+ [w.category.__name__ for w in caught], [])++# 12f. a stride past the effective window is refused: the tokens between two windows would be read+# by nothing, which is the failure this method exists to prevent.+check_raises("stride/past the effective window is an error", ValueError,+ lambda: scan(Q, stride=room_for(Q["dept"]) + 1))+# A stride the caller paired with a window they asked for is NOT their error when the library then+# reduces that window: window=90 with stride=90 is a self-consistent "no overlap, read everything".+# Refusing it made the library's own clamp look like the caller's mistake, and forced every such+# caller -- including `tests/test_onnx_long.py`'s window=96 stride=96 -- to be edited. It is clamped+# to half the effective window under the same RuntimeWarning, and the scan still covers the state.+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ _clamped = scan({"a": q_many(12)}, window=90, stride=90)+check_true("stride/a stride valid for the requested window is clamped, not refused",+ _clamped[0]["usage"]["windows"] > 1, _clamped[0]["usage"])+check_true("stride/and the caller is told the stride moved too",+ any("stride=90" in str(w.message) and "reduced" in str(w.message) for w in caught),+ [str(w.message)[:90] for w in caught])+_room12 = room_for(q_many(12))+check_true("stride/the clamped scan leaves no gap", all(+ b - a <= _room12 for a, b in read_spans(_clamped[1], _room12)), _clamped[1][:3])+# A stride past the window the caller actually asked for is still their error.+check_raises("stride/past the window the caller asked for is still refused", ValueError,+ lambda: scan({"a": q_many(12)}, window=40, stride=80))+check("stride/a stride inside the window is accepted",+ scan(Q, window=40, stride=40)[0]["usage"]["windows"] > 1, True)++# 12f-bis. the ONNX path honours the same cap, because README and Router promise it for both+try: # noqa: E402+ from laya.common import window_batch_cap, window_budget+except ImportError: # source reverted: let the checks below go red rather than abort the suite+ def window_budget(*_a, **_k): # type: ignore[misc]+ return (0, 0, 0)++ def window_batch_cap(*_a, **_k): # type: ignore[misc]+ return -1+from laya.onnx_agent import ONNXAgent # noqa: E402++_onnx = ONNXAgent.__new__(ONNXAgent)+_onnx.tok = _Tok()+_onnx.cfg = {"max_len": MAX_LEN, "head_max_len": HEAD_MAX_LEN}+_q_onnx = {"a": q_many(12)} # 100 options fills this tiny harness entirely+_eff_onnx, _step_onnx, _room_onnx = window_budget(+ _onnx.tok, [_onnx._to_internal(_q_onnx["a"])], MAX_LEN, HEAD_MAX_LEN)+check_true("onnx/predict_long caps the window at the room too",+ _eff_onnx <= _room_onnx and _eff_onnx < max(64, MAX_LEN - HEAD_MAX_LEN - 8),+ (_eff_onnx, _room_onnx))+check_true("onnx/and its stride stays inside that window", _step_onnx <= _eff_onnx,+ (_step_onnx, _eff_onnx))+check_true("onnx/predict_long actually calls window_budget",+ "window_budget(" in inspect.getsource(ONNXAgent.predict_long),+ "the README and Router docstring promise the cap for both agents")++# 12f-ter. capping the window must not turn one forward pass into an out-of-memory+check("batch/a mild cap keeps the single shared pass", window_batch_cap(10, 43, 64), None)+check("batch/no cap at all keeps it too", window_batch_cap(16, 312, 312), None)+check_true("batch/a blow-up is bounded to the un-capped pass width",+ 0 < (window_batch_cap(413, 23, 312) or 0) < 413, window_batch_cap(413, 23, 312))+check("batch/an explicit batch_size is always honoured", window_batch_cap(413, 23, 312, 8), 8)++# 12g. the page that teaches the default teaches the cap as well+window_para = README[README.index("A smaller `window` isolates"):][:900]+check("docs/README says the window is capped at the room the questions leave",+ "capped at the room the questions leave" in window_para, True)+check("docs/predict_long's docstring documents the cap",+ "capped at the room the" in (Agent.predict_long.__doc__ or ""), True)++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL " + f)diff --git a/tests/test_question_token_reuse.py b/tests/test_question_token_reuse.pyindex a68e48e9..9adf806e 100644--- a/tests/test_question_token_reuse.py+++ b/tests/test_question_token_reuse.py@@ -88,10 +88,16 @@ class QuestionTokenReuse(unittest.TestCase):def test_predict_long_uses_the_same_cache_for_all_windows(self):tok = Tokenizer()agent = agent_with(tok)- agent.predict_long("x" * 1000, QUESTIONS, window=80, stride=50, batch_size=3)+ scanned = agent.predict_long("x" * 1000, QUESTIONS, window=80, stride=50, batch_size=3)head = "choice question: " + QUESTION["ins"]- self.assertEqual(sum(text == head for _, text, _ in tok.calls), 1)- self.assertEqual(sum(text == " billing: charges" for _, text, _ in tok.calls), 1)+ # Twice, not once per window: predict_long assembles each question's head once on its own+ # to measure the room the question leaves for the state -- the cap on the window -- and the+ # scan that follows then shares one cache across every window it scores. The point of this+ # test is the second number: the head is encoded a fixed number of times, not once per+ # window, which the assertion below states as such.+ self.assertEqual(sum(text == head for _, text, _ in tok.calls), 2)+ self.assertEqual(sum(text == " billing: charges" for _, text, _ in tok.calls), 2)+ self.assertGreater(scanned["usage"]["windows"], 2)self.assertGreater(len(agent.batches), 1)def test_sequence_parity_for_rendering_order_and_budgets(self):