Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): aggregate a rewritten scan without naming a window for it
Aashish9ae2318f19f9
diff --git a/README.md b/README.mdindex c87cb1f8..1da97ee2 100644--- a/README.md+++ b/README.md@@ -581,11 +581,12 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,answered the document before the model read any of it.- Hooks wrap the inference that answers the state, so on a scanned document `on_predict_start`fires once with `ctx.states` holding the decoded windows, not the state you passed in (it was- tokenized to produce them). A start hook may rewrite those window texts, and what is scored is- what it leaves behind, but it may not add, drop or reorder windows: the answer attributes itself- to a span of your document by index, so a call that runs inference over any other number of- states raises. A hook that means to answer the document calls `ctx.skip(...)` instead: its result- comes back with no `answer["window"]` and `usage["windows"]` at 0, because no window scored it.+ tokenized to produce them). A start hook may replace that list: the answers are aggregated over+ whatever reached inference, and `usage["windows"]` counts those states. What a rewritten scan+ costs is the attribution — `answer["window"]` names a span of *your* document, so it is only+ reported when the scan reached inference unchanged. A hook that means to answer the document+ calls `ctx.skip(...)` instead: its result comes back with no `answer["window"]` and+ `usage["windows"]` at 0, because no window scored it.A smaller `window` isolates a short deciding span better (it becomes a larger fraction of itswindow); the default (`max_len - head_max_len`) favors context and throughput. Output shape matchesdiff --git a/docs/hooks/api.md b/docs/hooks/api.mdindex 0f5d83d9..fb580bee 100644--- a/docs/hooks/api.md+++ b/docs/hooks/api.md@@ -242,15 +242,18 @@ agent.predict(...) # alias of system_one- On `predict_long` the hooks wrap the inference that answers the state, which for a documentneeding several windows is the single shared `predict_batch` over them: `on_predict_start` firesonce, and `ctx.states` holds the decoded window texts in scan order rather than the caller's- state, which was tokenized to produce them. A start hook may rewrite the text of those windows,- and what is scored is what it leaves behind, but it may not change how many there are or their- order -- the answer attributes itself to a span of the caller's document by index, so a call that- runs inference over any other number of states raises. A hook that means to answer the document- calls `ctx.skip(...)` instead: its one result is returned as-is with no `window` key and- `usage["windows"]` at `0` -- no window scored it. `usage["windows"]` is total over the three- paths: `0` for a hook answer on either path, so a cached answer never reads as a window the model- read, `1` for a state that fit a single window and was scored, and `N` for a document scanned in- `N` overlapping windows.+ state, which was tokenized to produce them. `states` is mutable from a start hook, so the scan+ that reaches inference need not be the split `predict_long` computed. What changes is what the+ answer can claim:++ | what the start hook did | `usage["windows"]` | `answer["window"]` |+ |---|---|---|+ | answered with `ctx.skip([result])` | `0` | absent -- no window scored it |+ | left the scan as it was built | `N` | present -- `index`, `token_start`/`token_end` name the deciding span |+ | replaced the scan, in any way | the states that were scored | absent -- the offsets describe `predict_long`'s windows, not the text that was scored |++ `usage["windows"]` is total over every path, including the one where a state already fit a single+ window, so a cached answer never reads as a window the model read.### Routerdiff --git a/laya/agent.py b/laya/agent.pyindex c026f589..66e7e60c 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -217,22 +217,28 @@ def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:def _start_evidence():- """A recorder for `predict_long`: what the start-hook chain did, rather than what it cost.+ """A recorder for `predict_long`: what the start-hook chain left for inference to run on.- Why a recorder at all: the result count cannot tell a hook that answered the document from a- hook that replaced the window list. Both return fewer results than there were windows, and the- two need opposite readings -- the first scored no window, the second scored the ones the hook- left behind, and booked tokens for them. `ctx.results` already distinguishes them, and only the- engine's own context sees it, so `predict_long` appends this probe to the call's start hooks.+ Why a recorder at all: `ctx.skip()` only assigns `ctx.results` (see `laya.hooks.PredictContext`),+ so outside `predict_batch` the only account of what the hooks did is the context they were+ handed. The result count cannot tell a hook that answered the document from one that replaced+ the window list -- both return fewer results than there were windows, and the two need opposite+ readings: the first scored no window, the second scored the ones it left behind and booked+ tokens for them.++ The probe belongs last in the chain because `compose_hooks` orders defaults, then installed,+ then per-call hooks, and `dispatch` calls them in order, so a probe appended after the caller's+ own start hooks observes exactly what `predict_batch` is about to act on.Returns the probe and the dict it fills: `answered` is whether a hook replaced the call before- inference, `states` is the number of states that reached it.+ inference, `states` is a snapshot of the states that reached it (`None` if the probe never ran,+ which means `predict_batch` was replaced and no hook chain was dispatched)."""evidence = {"answered": False, "states": None}def probe(ctx):evidence["answered"] = ctx.results is not None- evidence["states"] = len(ctx.states)+ evidence["states"] = list(ctx.states)return probe, evidence@@ -1014,11 +1020,15 @@ class Agent(HookRegistry):The hooks wrap the inference that answers the state, which for a document needing severalwindows is the one shared `predict_batch` over them: `on_predict_start` fires once, and`ctx.states` holds the decoded window texts in scan order -- not the caller's `state`, which- was tokenized to produce them. A start hook may rewrite the text of those windows, and what- is scored is what it leaves behind. It may not change how many there are, or their order:- the per-window attribution names spans of the caller's document by index, so a different- count raises. A hook that means to answer the document itself calls `ctx.skip(...)`, whose- single payload is returned without window attribution.+ was tokenized to produce them. Three outcomes follow from what the chain leaves behind:++ * `ctx.skip([result])` answers the document: the payload comes back with no window+ attribution and `usage["windows"]` at 0, because nothing was scored+ * a scan left as this method built it: every window is scored, each answer carries+ `answer["window"]`, and `usage["windows"]` is the window count+ * a rewritten scan (`ctx.states` replaced, in any way): the answers are aggregated over+ the states that were scored, but no answer carries `answer["window"]` -- the offsets+ above describe this method's windows, not the text the model readArgs:window: state tokens per window. Defaults to the per-question state budget@@ -1042,9 +1052,10 @@ class Agent(HookRegistry):Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.The key is always present and counts the windows the model scored to produce the answer: `1`- for a state that fit one window, `N` for a document scanned in `N` overlapping windows, and- `0` when a start hook answered before any window was scored -- on either path, so a cached- answer never reads as a window the model read.+ for a state that fit one window, `N` for a document scanned in `N` overlapping windows (or+ the `N` a start hook rewrote them to), and `0` when a start hook answered the document, or+ left no states to score, before any window was read -- on either path, so a cached answer+ never reads as a window the model read."""if aggregate != "auto":raise ValueError("predict_long: only aggregate='auto' is supported")@@ -1109,18 +1120,30 @@ class Agent(HookRegistry):document["usage"]["windows"] = 0return document- if len(results) != len(windows):- # A hook can also rewrite `ctx.states` and let inference run on what it left behind,- # which moves the count without answering anything. Those results cannot be attributed- # to windows of the caller's document -- the states that produced them are the hook's- # -- so refuse rather than name a span that was never read.- survived = ("" if evidence["states"] is None else- " (%d states reached inference)" % evidence["states"])+ # The contract on `ctx.states` is that a start hook may replace it (see `docs/hooks/api.md`),+ # so a scan that comes back different from the split above is a supported outcome, not a+ # failure to report: the states that were scored are the hook's, while `starts` describes+ # this method's windows. Aggregate what came back and name nothing.+ rewritten = evidence["states"] is not None and evidence["states"] != windows++ if len(results) != len(windows) and not rewritten:+ # The observer saw the scan leave the hook chain and it is the one computed above, so+ # nothing here explains a count that is not the split. That is `predict_batch`+ # disagreeing with its own input -- a bug, or a replacement that never dispatched hooks.+ seen = ("the scan that reached inference was the split made here"+ if evidence["states"] is not None+ else "no start hook chain ran, so nothing rewrote the scan")raise ValueError("predict_long: the state was split into %d windows and the call returned %d"- " results%s; a start hook may rewrite the text of a window but not their number"- " or their order -- call ctx.skip([result]) to answer the document without scoring"- " it" % (len(windows), len(results), survived))+ " results, and %s" % (len(windows), len(results), seen))++ if not results:+ # A hook that left no states scored nothing, which is what 0 already means here; the+ # questions go unanswered rather than being aggregated over an empty list.+ warnings.warn("laya: predict_long: a start hook left no states to score, so the call"+ " aggregated nothing and returns no answers", RuntimeWarning, stacklevel=2)+ return {"model": "laya-rl-agent", "answers": {},+ "usage": {**aggregate_usage(results), "windows": 0}}ids = list(questions.keys())internal = {qid: self._to_internal(questions[qid]) for qid in ids}@@ -1139,9 +1162,10 @@ class Agent(HookRegistry):ans = per[best]# Name the window that decided, so a caller can check the deciding span itself. The# probability here is that window's, NOT a document-level calibrated number.- ans["window"] = {"index": best, "token_start": starts[best],- "token_end": min(starts[best] + budget, len(state_ids)),- "count": len(windows)}+ if not rewritten:+ ans["window"] = {"index": best, "token_start": starts[best],+ "token_end": min(starts[best] + budget, len(state_ids)),+ "count": len(results)}answers[qid] = ans# Aggregate usage generically so fields predict_batch may grow later (e.g. the fallback# counters from #351) are propagated, not silently dropped: sum numeric fields across@@ -1151,7 +1175,7 @@ class Agent(HookRegistry):for key, val in r["usage"].items():usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else valusage["output_tokens"] = 0- usage["windows"] = len(windows)+ usage["windows"] = len(results)return {"model": "laya-rl-agent", "answers": answers, "usage": usage}@torch.no_grad()diff --git a/tests/test_predict_long.py b/tests/test_predict_long.pyindex ab4471dd..2dfce408 100644--- a/tests/test_predict_long.py+++ b/tests/test_predict_long.py@@ -227,19 +227,10 @@ DOC = {"model": "laya-rl-agent","flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},"usage": {"input_tokens": 3, "output_tokens": 0}}-# The control first: how many windows this state really splits into, measured from the forward-# passes it costs rather than from the key under test. Two questions, one row each per window.-scan = make_real_agent()-base = scan.predict_long(LONG, Q)-nwin = sum(scan._forward_calls) // 2-check("scan/more than one window", nwin > 1, True)-check("scan/the scan reports the windows it read", base["usage"]["windows"], nwin)-check("scan/the deciding window is the last",- base["answers"]["dept"]["window"]["index"], nwin - 1)--# The instrument for sections 8-8c: a hook that answers must return, and a hook that replaces the-# window list must raise. Letting either unexpected case propagate would abort the suite and hide-# every check after it, so each call is captured and its type asserted by a named check.+# The instrument for sections 8-8e: a predict_long call is captured rather than let propagate, so+# a case that ends in the wrong shape -- an answer that should have come back raising, or a+# rejection that should not have happened at all -- fails a named check instead of aborting the run+# and hiding every check after it.def _attempt(fn):"""Call `fn`, returning (value, exception)."""try:@@ -252,6 +243,19 @@ def _kind(exc):return exc.__class__.__name__ if exc else None+# The control first: how many windows this state really splits into, measured from the forward+# passes it costs rather than from the key under test. Two questions, one row each per window.+scan = make_real_agent()+base, base_exc = _attempt(lambda: scan.predict_long(LONG, Q))+check("scan/an unhooked scan returns", _kind(base_exc), None)+nwin = sum(scan._forward_calls) // 2+check("scan/more than one window", nwin > 1, True)+check("scan/the scan reports the windows it read",+ ((base or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+check("scan/the deciding window is the last",+ ((base or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)++def _narrow(keep):"""A start hook that truncates the window list and lets inference run on what is left."""def hook(ctx):@@ -274,53 +278,70 @@ check("skip/no deciding window is claimed",sorted(k for v in answers.values() for k in v if k == "window"), [])check("skip/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])-# 8b. a start hook that replaces the window list is the other reading of the same count, and it is-# not a hook answer: inference ran on the states the hook left behind.+# 8b. a start hook that replaces the scan is the middle row: inference ran on the hook's states, so+# the answers aggregate, the count is theirs, and no span of the caller's document is named -- the+# offsets computed above describe windows that were not scored.a = make_real_agent()with warnings.catch_warnings(record=True) as caught:warnings.simplefilter("always")- _, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))-narrow_err = str(narrow_exc) if narrow_exc else "<no error>"-check("narrow/rejects a list shortened to one state", _kind(narrow_exc), "ValueError")-# Why this check sits next to the rejection: the forward is the proof that this was a rewrite and-# not an answer, and it is what made the two cases indistinguishable before.-check("narrow/a forward did run, so nothing was answered before inference",- a._forward_calls, [2])-check("narrow/the rewrite is not called a hook answer",+ narrow, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))+check("narrow/one state left behind is a scan, not a rejected count", _kind(narrow_exc), None)+# Why the forward sits next to the count: it is the proof that tokens were spent, which is exactly+# what made this case read as a hook answer before.+check("narrow/inference ran on the state that was left", a._forward_calls, [2])+check("narrow/the count is the state scored, not the split made",+ ((narrow or {}).get("usage") or {}).get("windows", "<absent>"), 1)+check("narrow/the tokens that forward booked are kept",+ ((narrow or {}).get("usage") or {}).get("input_tokens", 0) > 0, True)+check("narrow/no span of the caller's document is named",+ sorted(k for v in (narrow or {}).get("answers", {}).values() for k in v if k == "window"), [])+check("narrow/the answer is still aggregated",+ (narrow or {}).get("answers", {}).get("dept", {}).get("type"), "choice")+check("narrow/a rewrite is not warned about as a hook answer",[w.category.__name__ for w in caught], [])-check_true("narrow/the error names the split it could not attribute",- ("split into %d windows" % nwin) in narrow_err, narrow_err)-check_true("narrow/the error offers ctx.skip as the way to answer",- "ctx.skip" in narrow_err, narrow_err)a = make_real_agent()-check_raises("narrow/rejects a list shortened to two states", ValueError,- lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))-# the other direction, so the rule is the count rather than a hook that dropped something. The-# error must name the caller's split: `predict_long` hands the hook a copy, so an in-place-# `append` grows the hook's list and leaves the attributed windows alone.+two, exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))+check("narrow/two states left behind are counted as two",+ ((two or {}).get("usage") or {}).get("windows", "<absent>"), 2)+check("narrow/and two states is what the forward saw", a._forward_calls, [4])+# the other direction, and in place: `predict_long` hands the hook a copy of its split, so the+# state the hook invented is scored and counted without moving the windows computed above.a = make_real_agent()-_, added_exc = _attempt(+grown, exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.append("invented")))-added_err = str(added_exc) if added_exc else "<no error>"-check("grow/rejects a list with a state added", _kind(added_exc), "ValueError")-check_true("grow/the count is the caller's split, not the hook's list",- ("split into %d windows" % nwin) in added_err, added_err)+check("grow/the hook's added state is scored",+ ((grown or {}).get("usage") or {}).get("windows", "<absent>"), nwin + 1)+check("grow/the caller's split is the one that was extended", a._forward_calls, [2 * (nwin + 1)])+check("grow/and no span is named",+ sorted(k for v in (grown or {}).get("answers", {}).values() for k in v if k == "window"), [])-# 8c. rewriting the windows themselves stays supported: the count holds, so the attribution does+# 8c. an unchanged scan keeps the attribution; any rewrite drops it, even one of the same lengtha = make_real_agent()-res, exc = _attempt(+noop, exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.extend(())))-check("rewrite/a no-op on the list scans as usual", _kind(exc), None)-check("rewrite/a no-op on the list keeps the count",- ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+check("rewrite/a no-op on the list changes nothing", _kind(exc), None)+check("rewrite/a no-op keeps the count", ((noop or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+check("rewrite/a no-op keeps the deciding window",+ ((noop or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)a = make_real_agent()-res, exc = _attempt(lambda: a.predict_long(+upper, exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [s.upper() for s in ctx.states])))check("rewrite/the hook's text is what was scored", (a._encoded or [""])[0].isupper(), True)-check("rewrite/the same count is the same scan", _kind(exc), None)-check("rewrite/the count is unchanged", ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)-check("rewrite/the deciding window is still named",- ((res or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)+check("rewrite/the same length is still the same count",+ ((upper or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+check("rewrite/a text rewrite names no span",+ sorted(k for v in (upper or {}).get("answers", {}).values() for k in v if k == "window"), [])++# 8e. a hook that leaves nothing scores nothing: 0 windows and no answers, not a max() over []+a = make_real_agent()+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ empty, exc = _attempt(+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [])))+check("empty/nothing scored returns instead of raising", _kind(exc), None)+check("empty/no answer is invented", (empty or {"answers": "?"}).get("answers", "?"), {})+check("empty/the count says so", ((empty or {}).get("usage") or {}).get("windows", "<absent>"), 0)+check("empty/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])# 8d. the fits-in-one-window path answers to a hook too, so 0 keeps meaning "no window scored this"a = make_real_agent()@@ -367,17 +388,17 @@ check("docs/README states the key is total", "The key is total" in bullet, True)check("docs/README documents all three counts", sorted(set(re.findall(r"`([0-9N])`", bullet))),["0", "1", "N"])# The two rules this branch added, pinned where they are taught rather than in the code comment-hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:800]-check("docs/README says a hook may not change the window count or order",- "reorder windows" in hooks_bullet, True)+hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:900]+check("docs/README says attribution survives only an unchanged scan",+ "reported when the scan reached inference unchanged" in hooks_bullet, True)check("docs/README names ctx.skip as the way to answer", "ctx.skip(...)" in hooks_bullet, True)API = open(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),"docs", "hooks", "api.md"), encoding="utf-8").read()-api_para = API[API.index("On `predict_long` the hooks wrap"):][:1200]-check("docs/api.md matches: count and order are fixed",- "or their" in api_para and "number of states raises" in api_para, True)-check("docs/api.md matches: the hook answer reads as zero windows",- 'usage["windows"]` at `0' in api_para, True)+api_para = API[API.index("On `predict_long` the hooks wrap"):][:1600]+check("docs/api.md tabulates the rewritten scan as its own case",+ "| replaced the scan, in any way |" in api_para, True)+check("docs/api.md tabulates a hook answer as zero windows",+ "| answered with `ctx.skip([result])` | `0` |" in api_para, True)check("docs/predict_long's docstring says the key is total","always present" in (Agent.predict_long.__doc__ or ""), True)