Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): take a hook's answer from ctx.results, not from the result count

Aashishbd18c4714f60
diff --git a/README.md b/README.md
index e1d8a8b7..c87cb1f8 100644
--- a/README.md
+++ b/README.md
@@ -578,12 +578,14 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,
document lets the neutral majority out-vote the one window that saw the deciding span.
- A state that already fits one window is passed straight to `system_one` (identical output, plus
`usage["windows"] = 1`). The key is total: `1` single window, `N` scanned windows, `0` a hook
- answered a multi-window document before the model read any of it.
+ answered the document before the model read any of it.
- Hooks wrap the inference that answers the state, so on a scanned document `on_predict_start`
fires once with `ctx.states` holding the decoded windows, not the state you passed in (it was
- tokenized to produce them). A start hook that answers with `ctx.skip(...)` is answering the
- whole document: its result comes back with no `answer["window"]` and `usage["windows"]` at 0,
- because no window scored it.
+ tokenized to produce them). A start hook may rewrite those window texts, and what is scored is
+ what it leaves behind, but it may not add, drop or reorder windows: the answer attributes itself
+ to a span of your document by index, so a call that runs inference over any other number of
+ states raises. A hook that means to answer the document calls `ctx.skip(...)` instead: its result
+ comes back with no `answer["window"]` and `usage["windows"]` at 0, because no window scored it.
A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its
window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
index e6d904d2..0f5d83d9 100644
--- a/docs/hooks/api.md
+++ b/docs/hooks/api.md
@@ -242,12 +242,15 @@ agent.predict(...) # alias of system_one
- On `predict_long` the hooks wrap the inference that answers the state, which for a document
needing several windows is the single shared `predict_batch` over them: `on_predict_start` fires
once, and `ctx.states` holds the decoded window texts in scan order rather than the caller's
- state, which was tokenized to produce them. A start hook that answers with `ctx.skip(...)` is
- answering the document, so its one result is returned as-is with no `window` key and
+ state, which was tokenized to produce them. A start hook may rewrite the text of those windows,
+ and what is scored is what it leaves behind, but it may not change how many there are or their
+ order -- the answer attributes itself to a span of the caller's document by index, so a call that
+ runs inference over any other number of states raises. A hook that means to answer the document
+ calls `ctx.skip(...)` instead: its one result is returned as-is with no `window` key and
`usage["windows"]` at `0` -- no window scored it. `usage["windows"]` is total over the three
- paths: `0` here, `1` for a state that fit a single window (a hook that answered it is answering
- that one window, so the count is the same as the model's), and `N` for a document scanned in `N`
- overlapping windows.
+ paths: `0` for a hook answer on either path, so a cached answer never reads as a window the model
+ read, `1` for a state that fit a single window and was scored, and `N` for a document scanned in
+ `N` overlapping windows.
### Router
diff --git a/laya/agent.py b/laya/agent.py
index c7407a31..c026f589 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -29,8 +29,8 @@ from .common import (
temp_bucket,
)
from .hooks import (
- HookRegistry, PredictContext, aggregate_usage, compose_hooks, dispatch, normalise_hooks,
- validate_timeout,
+ HookRegistry, PredictContext, _as_sequence, aggregate_usage, compose_hooks, dispatch,
+ normalise_hooks, validate_timeout,
)
from .revisions import resolve_revision, snapshot_revision, verify_digests
@@ -216,6 +216,34 @@ def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:
return amp_dtype(checkpoint_default)
+def _start_evidence():
+ """A recorder for `predict_long`: what the start-hook chain did, rather than what it cost.
+
+ Why a recorder at all: the result count cannot tell a hook that answered the document from a
+ hook that replaced the window list. Both return fewer results than there were windows, and the
+ two need opposite readings -- the first scored no window, the second scored the ones the hook
+ left behind, and booked tokens for them. `ctx.results` already distinguishes them, and only the
+ engine's own context sees it, so `predict_long` appends this probe to the call's start hooks.
+
+ Returns the probe and the dict it fills: `answered` is whether a hook replaced the call before
+ inference, `states` is the number of states that reached it.
+ """
+ evidence = {"answered": False, "states": None}
+
+ def probe(ctx):
+ evidence["answered"] = ctx.results is not None
+ evidence["states"] = len(ctx.states)
+
+ return probe, evidence
+
+
+def _with_start_probe(hook_kwargs, probe):
+ """`hook_kwargs` with `probe` appended after the caller's own start hooks."""
+ kwargs = dict(hook_kwargs)
+ kwargs["on_predict_start"] = list(_as_sequence(hook_kwargs.get("on_predict_start"))) + [probe]
+ return kwargs
+
+
class Agent(HookRegistry):
"""System 1 decision model runtime: fast, non-autoregressive, calibrated decisions."""
@@ -986,9 +1014,11 @@ class Agent(HookRegistry):
The hooks wrap the inference that answers the state, which for a document needing several
windows is the one shared `predict_batch` over them: `on_predict_start` fires once, and
`ctx.states` holds the decoded window texts in scan order -- not the caller's `state`, which
- was tokenized to produce them. A start hook that rewrites `ctx.states` therefore rewrites
- windows, and one that calls `ctx.skip(...)` is answering the document rather than a window,
- so its payload is returned without window attribution.
+ was tokenized to produce them. A start hook may rewrite the text of those windows, and what
+ is scored is what it leaves behind. It may not change how many there are, or their order:
+ the per-window attribution names spans of the caller's document by index, so a different
+ count raises. A hook that means to answer the document itself calls `ctx.skip(...)`, whose
+ single payload is returned without window attribution.
Args:
window: state tokens per window. Defaults to the per-question state budget
@@ -1011,9 +1041,10 @@ class Agent(HookRegistry):
hooks_timeout: Override the Agent's `hooks_timeout` for this call.
Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.
- The key is always present and counts the windows the answer was read from: `1` for a state
- that fit one window, `N` for a document scanned in `N` overlapping windows, and `0` when a
- start hook answered a multi-window document before any window was scored.
+ The key is always present and counts the windows the model scored to produce the answer: `1`
+ for a state that fit one window, `N` for a document scanned in `N` overlapping windows, and
+ `0` when a start hook answered before any window was scored -- on either path, so a cached
+ answer never reads as a window the model read.
"""
if aggregate != "auto":
raise ValueError("predict_long: only aggregate='auto' is supported")
@@ -1032,8 +1063,13 @@ class Agent(HookRegistry):
# shortest, most common inputs. The result is copied first: a start hook that answers with
# `ctx.skip(...)` hands back its own payload dict, and it may be a cached object.
if len(state_ids) <= budget:
- single = dict(self.system_one(state, questions, lang=lang, **hook_kwargs))
- single["usage"] = {**(single.get("usage") or {}), "windows": 1}
+ probe, evidence = _start_evidence()
+ single = dict(self.system_one(state, questions, lang=lang,
+ **_with_start_probe(hook_kwargs, probe)))
+ # Why 0 for a hook answer here: the state did fit one window, but no window was scored,
+ # which is the same fact the multi-window path reports as 0. Reading 1 would make a
+ # cached answer and a served answer agree on how much of the input the model saw.
+ single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}
return single
step = stride if (stride and stride > 0) else max(1, budget // 2)
@@ -1049,19 +1085,22 @@ class Agent(HookRegistry):
break
i += step
- results = self.predict_batch(windows, questions, batch_size=batch_size, lang=lang,
- **hook_kwargs)
-
- if len(results) != len(windows):
- # Only a hook can change the count: `ctx.skip(...)` runs before the split, so a hook
- # answering the whole document has no way to know how many windows it will produce and
- # returns one result. Aggregating over that payload would pick between answers that were
- # never scored and name a deciding window that decided nothing, so pass the document
- # answer through unattributed and say which windows were scored: none of them.
+ probe, evidence = _start_evidence()
+ # `list(windows)`, not `windows`: `ctx.states` is the list the hook receives, so a hook that
+ # mutates it in place (`append`, `sort`) would otherwise also grow the split this call
+ # attributes answers to, and the counts would agree while `starts` no longer lined up.
+ results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,
+ **_with_start_probe(hook_kwargs, probe))
+
+ if evidence["answered"]:
+ # The hook replaced the call before any window was scored. Aggregating over its payload
+ # would pick between answers that were never scored and name a deciding window that
+ # decided nothing, so pass the document answer through unattributed and report the
+ # count as what it is: none of them.
if len(results) != 1:
raise ValueError(
- "predict_long: %d windows scored, %d results returned; a start hook that answers"
- " a long state returns one result for the document" % (len(windows), len(results)))
+ "predict_long: a start hook answered this state with %d results; ctx.skip()"
+ " takes one result for the document, not one per window" % len(results))
warnings.warn("laya: predict_long: a hook answered the state before it was scanned, so "
"no window decided the result and none is reported", RuntimeWarning,
stacklevel=2)
@@ -1070,6 +1109,19 @@ class Agent(HookRegistry):
document["usage"]["windows"] = 0
return document
+ if len(results) != len(windows):
+ # A hook can also rewrite `ctx.states` and let inference run on what it left behind,
+ # which moves the count without answering anything. Those results cannot be attributed
+ # to windows of the caller's document -- the states that produced them are the hook's
+ # -- so refuse rather than name a span that was never read.
+ survived = ("" if evidence["states"] is None else
+ " (%d states reached inference)" % evidence["states"])
+ raise ValueError(
+ "predict_long: the state was split into %d windows and the call returned %d"
+ " results%s; a start hook may rewrite the text of a window but not their number"
+ " or their order -- call ctx.skip([result]) to answer the document without scoring"
+ " it" % (len(windows), len(results), survived))
+
ids = list(questions.keys())
internal = {qid: self._to_internal(questions[qid]) for qid in ids}
answers = {}
diff --git a/tests/test_predict_long.py b/tests/test_predict_long.py
index 59d280da..ab4471dd 100644
--- a/tests/test_predict_long.py
+++ b/tests/test_predict_long.py
@@ -9,10 +9,13 @@ behaviour on real weights is exercised in tests/test_local_e2e.py.
import os
import re
import sys
+import threading
import warnings
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+import numpy as np # noqa: E402
+
from laya.agent import Agent # noqa: E402
PASS, FAIL = [], []
@@ -33,6 +36,10 @@ def check_raises(name, exc, fn):
FAIL.append("%s: did not raise %s" % (name, exc.__name__))
+def check_true(name, cond, detail=""):
+ (PASS if cond else FAIL).append(name if cond else "%s: %s" % (name, detail))
+
+
class _Tok:
mask_token = "[M]"
@@ -67,6 +74,49 @@ def make_agent(batch_result_fn):
return a
+def make_real_agent():
+ """`predict_long` on the real `predict_batch`/`system_one`, with only the three composed
+ helpers stubbed -- the harness `tests/test_hooks.py` uses, so no weights are involved.
+
+ Why both agents: the stubbed `predict_batch` above is the right instrument for what
+ `predict_long` *forwards*, but it never runs a hook, so anything it says about a hook answering
+ rests on a result count alone. Here the hook chain really runs and every forward pass is
+ counted, which is the difference between "a hook answered the document" and "a hook replaced
+ the window list" -- both change the count.
+ """
+ a = Agent.__new__(Agent)
+ a.cfg = {"max_len": 100, "head_max_len": 20} # budget = max(64, 100-20-8) = 72
+ a.tok = type("Tok", (_Tok,), {"pad_token_id": 0})()
+ a._to_internal = staticmethod(Agent._to_internal).__func__
+ a.model_id = "convaiinnovations/laya"
+ a.hooks = []
+ a.hooks_raise = True
+ a.hooks_timeout = None
+ a._hooks_lock = threading.Lock()
+ a._forward_calls, a._encoded = [], []
+
+ def _encode_state(state, ids, internal, **overrides):
+ a._encoded.append(state)
+ return [{"ids": [1, 2, 3], "markers": [0, 1], "qtype": 2} for _ in ids]
+
+ def _forward(b):
+ n = b["input_ids"].shape[0]
+ a._forward_calls.append(n)
+ return np.zeros((n, 2), dtype=np.float32), np.full((n, 2), 0.5, dtype=np.float32)
+
+ def _decode_answers(logits, act, items, ids, internal, row, **kw):
+ conf = 0.4 + 0.05 * row # so the last window decides
+ return {"dept": {"type": "choice", "choice": "a", "confidence": conf,
+ "answer_confidence": conf},
+ "flag": {"type": "noul", "noul": conf, "confidence": conf,
+ "answer_confidence": conf}}
+
+ a._encode_state = _encode_state
+ a._forward = _forward
+ a._decode_answers = _decode_answers
+ return a
+
+
Q = {"dept": {"type": "choice", "instructions": "?", "criteria": {"a": "x", "b": "y"}},
"flag": {"type": "noul", "instructions": "?"}}
@@ -129,15 +179,25 @@ check_raises("aggregate/rejects unknown mode", ValueError,
HOOK_KEYS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")
LONG = {"body": "y" * 300}
sentinel = object()
+
+
+def sentinel_start(ctx):
+ pass
+
+
a = make_agent(canned)
-res = a.predict_long(LONG, Q, hooks=[sentinel], hooks_raise=False, hooks_timeout=2.5)
+res = a.predict_long(LONG, Q, hooks=[sentinel], on_predict_start=sentinel_start,
+ hooks_raise=False, hooks_timeout=2.5)
got = a._calls["batch_kwargs"]
check("hooks/forwarded to the scan", sorted(got), sorted(HOOK_KEYS))
check("hooks/the hook list reaches it", got.get("hooks"), [sentinel])
check("hooks/hooks_raise reaches it", got.get("hooks_raise"), False)
check("hooks/hooks_timeout reaches it", got.get("hooks_timeout"), 2.5)
-check("hooks/start/end default to None",
- [got.get("on_predict_start"), got.get("on_predict_end")], [None, None])
+started = got.get("on_predict_start")
+started = started if isinstance(started, list) else ([] if started is None else [started])
+check("hooks/the caller's start hook still runs first",
+ bool(started) and started[0] is sentinel_start, True)
+check("hooks/end stays absent when the caller sets none", got.get("on_predict_end"), None)
# 5. the same controls reach the fits-in-one-window path
a = make_agent(lambda s, q: [])
@@ -161,29 +221,123 @@ plain = make_agent(canned).predict_long(LONG, Q)
with_hooks = make_agent(canned).predict_long(LONG, Q, hooks=[sentinel], hooks_timeout=9)
check("hooks/forwarding is decision-neutral", with_hooks, plain)
-# 8. a start hook that answers the document is not attributed to a window that was never scored
-def one_document_answer(states, q):
- return [{"answers": {"dept": {"type": "choice", "choice": "a", "answer_confidence": 0.7},
- "flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},
- "usage": {"input_tokens": 3, "output_tokens": 0}}]
+# 8. a start hook that answers the document, on the engine's own evidence
+DOC = {"model": "laya-rl-agent",
+ "answers": {"dept": {"type": "choice", "choice": "b", "answer_confidence": 0.9},
+ "flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},
+ "usage": {"input_tokens": 3, "output_tokens": 0}}
+
+# The control first: how many windows this state really splits into, measured from the forward
+# passes it costs rather than from the key under test. Two questions, one row each per window.
+scan = make_real_agent()
+base = scan.predict_long(LONG, Q)
+nwin = sum(scan._forward_calls) // 2
+check("scan/more than one window", nwin > 1, True)
+check("scan/the scan reports the windows it read", base["usage"]["windows"], nwin)
+check("scan/the deciding window is the last",
+ base["answers"]["dept"]["window"]["index"], nwin - 1)
+
+# The instrument for sections 8-8c: a hook that answers must return, and a hook that replaces the
+# window list must raise. Letting either unexpected case propagate would abort the suite and hide
+# every check after it, so each call is captured and its type asserted by a named check.
+def _attempt(fn):
+ """Call `fn`, returning (value, exception)."""
+ try:
+ return fn(), None
+ except BaseException as exc: # noqa: BLE001
+ return None, exc
+
+
+def _kind(exc):
+ return exc.__class__.__name__ if exc else None
-a = make_agent(one_document_answer)
+def _narrow(keep):
+ """A start hook that truncates the window list and lets inference run on what is left."""
+ def hook(ctx):
+ ctx.states = list(ctx.states)[:keep]
+ return hook
+
+
+a = make_real_agent()
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
- res = a.predict_long(LONG, Q)
-nwin = len(a._calls["batch_states"])
-check("skip/the state was split into several windows", nwin > 1, True)
-check("skip/no window was scored", res["usage"]["windows"], 0)
-check("skip/the payload tokens are kept", res["usage"]["input_tokens"], 3)
+ res, skip_exc = _attempt(
+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))
+check("skip/one payload for the document is never refused as a count mismatch", _kind(skip_exc), None)
+check("skip/no forward pass ran", a._forward_calls, [])
+usage, answers = (res or {}).get("usage", {}), (res or {}).get("answers", {})
+check("skip/no window was scored", usage.get("windows", "<absent>"), 0)
+check("skip/the payload tokens are kept", usage.get("input_tokens", "<absent>"), 3)
+check("skip/the hook's answer is returned", answers.get("dept", {}).get("choice"), "b")
check("skip/no deciding window is claimed",
- sorted(k for v in res["answers"].values() for k in v if k == "window"), [])
-check("skip/the hook's answer is returned", res["answers"]["dept"]["choice"], "a")
+ sorted(k for v in answers.values() for k in v if k == "window"), [])
check("skip/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])
-# 9. a payload that matches neither the document nor the windows is an error, not a guess
+# 8b. a start hook that replaces the window list is the other reading of the same count, and it is
+# not a hook answer: inference ran on the states the hook left behind.
+a = make_real_agent()
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ _, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))
+narrow_err = str(narrow_exc) if narrow_exc else "<no error>"
+check("narrow/rejects a list shortened to one state", _kind(narrow_exc), "ValueError")
+# Why this check sits next to the rejection: the forward is the proof that this was a rewrite and
+# not an answer, and it is what made the two cases indistinguishable before.
+check("narrow/a forward did run, so nothing was answered before inference",
+ a._forward_calls, [2])
+check("narrow/the rewrite is not called a hook answer",
+ [w.category.__name__ for w in caught], [])
+check_true("narrow/the error names the split it could not attribute",
+ ("split into %d windows" % nwin) in narrow_err, narrow_err)
+check_true("narrow/the error offers ctx.skip as the way to answer",
+ "ctx.skip" in narrow_err, narrow_err)
+a = make_real_agent()
+check_raises("narrow/rejects a list shortened to two states", ValueError,
+ lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))
+# the other direction, so the rule is the count rather than a hook that dropped something. The
+# error must name the caller's split: `predict_long` hands the hook a copy, so an in-place
+# `append` grows the hook's list and leaves the attributed windows alone.
+a = make_real_agent()
+_, added_exc = _attempt(
+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.append("invented")))
+added_err = str(added_exc) if added_exc else "<no error>"
+check("grow/rejects a list with a state added", _kind(added_exc), "ValueError")
+check_true("grow/the count is the caller's split, not the hook's list",
+ ("split into %d windows" % nwin) in added_err, added_err)
+
+# 8c. rewriting the windows themselves stays supported: the count holds, so the attribution does
+a = make_real_agent()
+res, exc = _attempt(
+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.extend(())))
+check("rewrite/a no-op on the list scans as usual", _kind(exc), None)
+check("rewrite/a no-op on the list keeps the count",
+ ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+a = make_real_agent()
+res, exc = _attempt(lambda: a.predict_long(
+ LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [s.upper() for s in ctx.states])))
+check("rewrite/the hook's text is what was scored", (a._encoded or [""])[0].isupper(), True)
+check("rewrite/the same count is the same scan", _kind(exc), None)
+check("rewrite/the count is unchanged", ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+check("rewrite/the deciding window is still named",
+ ((res or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)
+
+# 8d. the fits-in-one-window path answers to a hook too, so 0 keeps meaning "no window scored this"
+a = make_real_agent()
+short_hooked, exc = _attempt(
+ lambda: a.predict_long({"body": "x" * 50}, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))
+check("short skip/no forward pass", a._forward_calls, [])
+check("short skip/a hook answer is not counted as a window read",
+ ((short_hooked or {}).get("usage") or {}).get("windows", "<absent>"), 0)
+a = make_real_agent()
+short_plain, exc = _attempt(lambda: a.predict_long({"body": "x" * 50}, Q))
+check("short skip/the same state without a hook is one window",
+ ((short_plain or {}).get("usage") or {}).get("windows", "<absent>"), 1)
+
+# 9. a batch call that returns the wrong count for its own states is an error, not a guess
a = make_agent(lambda s, q: [{"answers": {}, "usage": {}}] * 2)
-check_raises("skip/rejects a count matching nothing", ValueError, lambda: a.predict_long(LONG, Q))
+check_raises("count/rejects a result count matching nothing", ValueError,
+ lambda: a.predict_long(LONG, Q))
# 10. the count is total, so one question works on every result predict_long can return
short_all = make_agent(lambda s, q: []).predict_long({"body": "x" * 50}, Q)
@@ -191,12 +345,16 @@ scan_agent = make_agent(canned)
scanned_all = scan_agent.predict_long(LONG, Q)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
- hook_all = make_agent(one_document_answer).predict_long(LONG, Q)
+ hook_all, hook_exc = _attempt(
+ lambda: make_real_agent().predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))
+check("total/a hook answer returns rather than being refused", _kind(hook_exc), None)
check("total/no path leaves the key out",
- [r["usage"].get("windows", "<absent>") for r in (short_all, scanned_all, hook_all)],
+ [(r or {}).get("usage", {}).get("windows", "<absent>")
+ for r in (short_all, scanned_all, hook_all)],
[1, len(scan_agent._calls["batch_states"]), 0])
check("total/0 separates a hook answer from a single-window one",
- (hook_all["usage"].get("windows", "<absent>"), short_all["usage"].get("windows", "<absent>")),
+ ((hook_all or {}).get("usage", {}).get("windows", "<absent>"),
+ short_all["usage"].get("windows", "<absent>")),
(0, 1))
@@ -208,6 +366,18 @@ check("docs/README pins the single-window value", 'usage["windows"] = 1' in bull
check("docs/README states the key is total", "The key is total" in bullet, True)
check("docs/README documents all three counts", sorted(set(re.findall(r"`([0-9N])`", bullet))),
["0", "1", "N"])
+# The two rules this branch added, pinned where they are taught rather than in the code comment
+hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:800]
+check("docs/README says a hook may not change the window count or order",
+ "reorder windows" in hooks_bullet, True)
+check("docs/README names ctx.skip as the way to answer", "ctx.skip(...)" in hooks_bullet, True)
+API = open(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+ "docs", "hooks", "api.md"), encoding="utf-8").read()
+api_para = API[API.index("On `predict_long` the hooks wrap"):][:1200]
+check("docs/api.md matches: count and order are fixed",
+ "or their" in api_para and "number of states raises" in api_para, True)
+check("docs/api.md matches: the hook answer reads as zero windows",
+ 'usage["windows"]` at `0' in api_para, True)
check("docs/predict_long's docstring says the key is total",
"always present" in (Agent.predict_long.__doc__ or ""), True)