Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(agent): aggregate a rewritten scan without naming a window for it

Aashish9ae2318f19f9
diff --git a/README.md b/README.md
index c87cb1f8..1da97ee2 100644
--- a/README.md
+++ b/README.md
@@ -581,11 +581,12 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,
answered the document before the model read any of it.
- Hooks wrap the inference that answers the state, so on a scanned document `on_predict_start`
fires once with `ctx.states` holding the decoded windows, not the state you passed in (it was
- tokenized to produce them). A start hook may rewrite those window texts, and what is scored is
- what it leaves behind, but it may not add, drop or reorder windows: the answer attributes itself
- to a span of your document by index, so a call that runs inference over any other number of
- states raises. A hook that means to answer the document calls `ctx.skip(...)` instead: its result
- comes back with no `answer["window"]` and `usage["windows"]` at 0, because no window scored it.
+ tokenized to produce them). A start hook may replace that list: the answers are aggregated over
+ whatever reached inference, and `usage["windows"]` counts those states. What a rewritten scan
+ costs is the attribution — `answer["window"]` names a span of *your* document, so it is only
+ reported when the scan reached inference unchanged. A hook that means to answer the document
+ calls `ctx.skip(...)` instead: its result comes back with no `answer["window"]` and
+ `usage["windows"]` at 0, because no window scored it.
A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its
window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
index 0f5d83d9..fb580bee 100644
--- a/docs/hooks/api.md
+++ b/docs/hooks/api.md
@@ -242,15 +242,18 @@ agent.predict(...) # alias of system_one
- On `predict_long` the hooks wrap the inference that answers the state, which for a document
needing several windows is the single shared `predict_batch` over them: `on_predict_start` fires
once, and `ctx.states` holds the decoded window texts in scan order rather than the caller's
- state, which was tokenized to produce them. A start hook may rewrite the text of those windows,
- and what is scored is what it leaves behind, but it may not change how many there are or their
- order -- the answer attributes itself to a span of the caller's document by index, so a call that
- runs inference over any other number of states raises. A hook that means to answer the document
- calls `ctx.skip(...)` instead: its one result is returned as-is with no `window` key and
- `usage["windows"]` at `0` -- no window scored it. `usage["windows"]` is total over the three
- paths: `0` for a hook answer on either path, so a cached answer never reads as a window the model
- read, `1` for a state that fit a single window and was scored, and `N` for a document scanned in
- `N` overlapping windows.
+ state, which was tokenized to produce them. `states` is mutable from a start hook, so the scan
+ that reaches inference need not be the split `predict_long` computed. What changes is what the
+ answer can claim:
+
+ | what the start hook did | `usage["windows"]` | `answer["window"]` |
+ |---|---|---|
+ | answered with `ctx.skip([result])` | `0` | absent -- no window scored it |
+ | left the scan as it was built | `N` | present -- `index`, `token_start`/`token_end` name the deciding span |
+ | replaced the scan, in any way | the states that were scored | absent -- the offsets describe `predict_long`'s windows, not the text that was scored |
+
+ `usage["windows"]` is total over every path, including the one where a state already fit a single
+ window, so a cached answer never reads as a window the model read.
### Router
diff --git a/laya/agent.py b/laya/agent.py
index c026f589..66e7e60c 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -217,22 +217,28 @@ def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:
def _start_evidence():
- """A recorder for `predict_long`: what the start-hook chain did, rather than what it cost.
+ """A recorder for `predict_long`: what the start-hook chain left for inference to run on.
- Why a recorder at all: the result count cannot tell a hook that answered the document from a
- hook that replaced the window list. Both return fewer results than there were windows, and the
- two need opposite readings -- the first scored no window, the second scored the ones the hook
- left behind, and booked tokens for them. `ctx.results` already distinguishes them, and only the
- engine's own context sees it, so `predict_long` appends this probe to the call's start hooks.
+ Why a recorder at all: `ctx.skip()` only assigns `ctx.results` (see `laya.hooks.PredictContext`),
+ so outside `predict_batch` the only account of what the hooks did is the context they were
+ handed. The result count cannot tell a hook that answered the document from one that replaced
+ the window list -- both return fewer results than there were windows, and the two need opposite
+ readings: the first scored no window, the second scored the ones it left behind and booked
+ tokens for them.
+
+ The probe belongs last in the chain because `compose_hooks` orders defaults, then installed,
+ then per-call hooks, and `dispatch` calls them in order, so a probe appended after the caller's
+ own start hooks observes exactly what `predict_batch` is about to act on.
Returns the probe and the dict it fills: `answered` is whether a hook replaced the call before
- inference, `states` is the number of states that reached it.
+ inference, `states` is a snapshot of the states that reached it (`None` if the probe never ran,
+ which means `predict_batch` was replaced and no hook chain was dispatched).
"""
evidence = {"answered": False, "states": None}
def probe(ctx):
evidence["answered"] = ctx.results is not None
- evidence["states"] = len(ctx.states)
+ evidence["states"] = list(ctx.states)
return probe, evidence
@@ -1014,11 +1020,15 @@ class Agent(HookRegistry):
The hooks wrap the inference that answers the state, which for a document needing several
windows is the one shared `predict_batch` over them: `on_predict_start` fires once, and
`ctx.states` holds the decoded window texts in scan order -- not the caller's `state`, which
- was tokenized to produce them. A start hook may rewrite the text of those windows, and what
- is scored is what it leaves behind. It may not change how many there are, or their order:
- the per-window attribution names spans of the caller's document by index, so a different
- count raises. A hook that means to answer the document itself calls `ctx.skip(...)`, whose
- single payload is returned without window attribution.
+ was tokenized to produce them. Three outcomes follow from what the chain leaves behind:
+
+ * `ctx.skip([result])` answers the document: the payload comes back with no window
+ attribution and `usage["windows"]` at 0, because nothing was scored
+ * a scan left as this method built it: every window is scored, each answer carries
+ `answer["window"]`, and `usage["windows"]` is the window count
+ * a rewritten scan (`ctx.states` replaced, in any way): the answers are aggregated over
+ the states that were scored, but no answer carries `answer["window"]` -- the offsets
+ above describe this method's windows, not the text the model read
Args:
window: state tokens per window. Defaults to the per-question state budget
@@ -1042,9 +1052,10 @@ class Agent(HookRegistry):
Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.
The key is always present and counts the windows the model scored to produce the answer: `1`
- for a state that fit one window, `N` for a document scanned in `N` overlapping windows, and
- `0` when a start hook answered before any window was scored -- on either path, so a cached
- answer never reads as a window the model read.
+ for a state that fit one window, `N` for a document scanned in `N` overlapping windows (or
+ the `N` a start hook rewrote them to), and `0` when a start hook answered the document, or
+ left no states to score, before any window was read -- on either path, so a cached answer
+ never reads as a window the model read.
"""
if aggregate != "auto":
raise ValueError("predict_long: only aggregate='auto' is supported")
@@ -1109,18 +1120,30 @@ class Agent(HookRegistry):
document["usage"]["windows"] = 0
return document
- if len(results) != len(windows):
- # A hook can also rewrite `ctx.states` and let inference run on what it left behind,
- # which moves the count without answering anything. Those results cannot be attributed
- # to windows of the caller's document -- the states that produced them are the hook's
- # -- so refuse rather than name a span that was never read.
- survived = ("" if evidence["states"] is None else
- " (%d states reached inference)" % evidence["states"])
+ # The contract on `ctx.states` is that a start hook may replace it (see `docs/hooks/api.md`),
+ # so a scan that comes back different from the split above is a supported outcome, not a
+ # failure to report: the states that were scored are the hook's, while `starts` describes
+ # this method's windows. Aggregate what came back and name nothing.
+ rewritten = evidence["states"] is not None and evidence["states"] != windows
+
+ if len(results) != len(windows) and not rewritten:
+ # The observer saw the scan leave the hook chain and it is the one computed above, so
+ # nothing here explains a count that is not the split. That is `predict_batch`
+ # disagreeing with its own input -- a bug, or a replacement that never dispatched hooks.
+ seen = ("the scan that reached inference was the split made here"
+ if evidence["states"] is not None
+ else "no start hook chain ran, so nothing rewrote the scan")
raise ValueError(
"predict_long: the state was split into %d windows and the call returned %d"
- " results%s; a start hook may rewrite the text of a window but not their number"
- " or their order -- call ctx.skip([result]) to answer the document without scoring"
- " it" % (len(windows), len(results), survived))
+ " results, and %s" % (len(windows), len(results), seen))
+
+ if not results:
+ # A hook that left no states scored nothing, which is what 0 already means here; the
+ # questions go unanswered rather than being aggregated over an empty list.
+ warnings.warn("laya: predict_long: a start hook left no states to score, so the call"
+ " aggregated nothing and returns no answers", RuntimeWarning, stacklevel=2)
+ return {"model": "laya-rl-agent", "answers": {},
+ "usage": {**aggregate_usage(results), "windows": 0}}
ids = list(questions.keys())
internal = {qid: self._to_internal(questions[qid]) for qid in ids}
@@ -1139,9 +1162,10 @@ class Agent(HookRegistry):
ans = per[best]
# Name the window that decided, so a caller can check the deciding span itself. The
# probability here is that window's, NOT a document-level calibrated number.
- ans["window"] = {"index": best, "token_start": starts[best],
- "token_end": min(starts[best] + budget, len(state_ids)),
- "count": len(windows)}
+ if not rewritten:
+ ans["window"] = {"index": best, "token_start": starts[best],
+ "token_end": min(starts[best] + budget, len(state_ids)),
+ "count": len(results)}
answers[qid] = ans
# Aggregate usage generically so fields predict_batch may grow later (e.g. the fallback
# counters from #351) are propagated, not silently dropped: sum numeric fields across
@@ -1151,7 +1175,7 @@ class Agent(HookRegistry):
for key, val in r["usage"].items():
usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
usage["output_tokens"] = 0
- usage["windows"] = len(windows)
+ usage["windows"] = len(results)
return {"model": "laya-rl-agent", "answers": answers, "usage": usage}
@torch.no_grad()
diff --git a/tests/test_predict_long.py b/tests/test_predict_long.py
index ab4471dd..2dfce408 100644
--- a/tests/test_predict_long.py
+++ b/tests/test_predict_long.py
@@ -227,19 +227,10 @@ DOC = {"model": "laya-rl-agent",
"flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},
"usage": {"input_tokens": 3, "output_tokens": 0}}
-# The control first: how many windows this state really splits into, measured from the forward
-# passes it costs rather than from the key under test. Two questions, one row each per window.
-scan = make_real_agent()
-base = scan.predict_long(LONG, Q)
-nwin = sum(scan._forward_calls) // 2
-check("scan/more than one window", nwin > 1, True)
-check("scan/the scan reports the windows it read", base["usage"]["windows"], nwin)
-check("scan/the deciding window is the last",
- base["answers"]["dept"]["window"]["index"], nwin - 1)
-
-# The instrument for sections 8-8c: a hook that answers must return, and a hook that replaces the
-# window list must raise. Letting either unexpected case propagate would abort the suite and hide
-# every check after it, so each call is captured and its type asserted by a named check.
+# The instrument for sections 8-8e: a predict_long call is captured rather than let propagate, so
+# a case that ends in the wrong shape -- an answer that should have come back raising, or a
+# rejection that should not have happened at all -- fails a named check instead of aborting the run
+# and hiding every check after it.
def _attempt(fn):
"""Call `fn`, returning (value, exception)."""
try:
@@ -252,6 +243,19 @@ def _kind(exc):
return exc.__class__.__name__ if exc else None
+# The control first: how many windows this state really splits into, measured from the forward
+# passes it costs rather than from the key under test. Two questions, one row each per window.
+scan = make_real_agent()
+base, base_exc = _attempt(lambda: scan.predict_long(LONG, Q))
+check("scan/an unhooked scan returns", _kind(base_exc), None)
+nwin = sum(scan._forward_calls) // 2
+check("scan/more than one window", nwin > 1, True)
+check("scan/the scan reports the windows it read",
+ ((base or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+check("scan/the deciding window is the last",
+ ((base or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)
+
+
def _narrow(keep):
"""A start hook that truncates the window list and lets inference run on what is left."""
def hook(ctx):
@@ -274,53 +278,70 @@ check("skip/no deciding window is claimed",
sorted(k for v in answers.values() for k in v if k == "window"), [])
check("skip/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])
-# 8b. a start hook that replaces the window list is the other reading of the same count, and it is
-# not a hook answer: inference ran on the states the hook left behind.
+# 8b. a start hook that replaces the scan is the middle row: inference ran on the hook's states, so
+# the answers aggregate, the count is theirs, and no span of the caller's document is named -- the
+# offsets computed above describe windows that were not scored.
a = make_real_agent()
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
- _, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))
-narrow_err = str(narrow_exc) if narrow_exc else "<no error>"
-check("narrow/rejects a list shortened to one state", _kind(narrow_exc), "ValueError")
-# Why this check sits next to the rejection: the forward is the proof that this was a rewrite and
-# not an answer, and it is what made the two cases indistinguishable before.
-check("narrow/a forward did run, so nothing was answered before inference",
- a._forward_calls, [2])
-check("narrow/the rewrite is not called a hook answer",
+ narrow, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))
+check("narrow/one state left behind is a scan, not a rejected count", _kind(narrow_exc), None)
+# Why the forward sits next to the count: it is the proof that tokens were spent, which is exactly
+# what made this case read as a hook answer before.
+check("narrow/inference ran on the state that was left", a._forward_calls, [2])
+check("narrow/the count is the state scored, not the split made",
+ ((narrow or {}).get("usage") or {}).get("windows", "<absent>"), 1)
+check("narrow/the tokens that forward booked are kept",
+ ((narrow or {}).get("usage") or {}).get("input_tokens", 0) > 0, True)
+check("narrow/no span of the caller's document is named",
+ sorted(k for v in (narrow or {}).get("answers", {}).values() for k in v if k == "window"), [])
+check("narrow/the answer is still aggregated",
+ (narrow or {}).get("answers", {}).get("dept", {}).get("type"), "choice")
+check("narrow/a rewrite is not warned about as a hook answer",
[w.category.__name__ for w in caught], [])
-check_true("narrow/the error names the split it could not attribute",
- ("split into %d windows" % nwin) in narrow_err, narrow_err)
-check_true("narrow/the error offers ctx.skip as the way to answer",
- "ctx.skip" in narrow_err, narrow_err)
a = make_real_agent()
-check_raises("narrow/rejects a list shortened to two states", ValueError,
- lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))
-# the other direction, so the rule is the count rather than a hook that dropped something. The
-# error must name the caller's split: `predict_long` hands the hook a copy, so an in-place
-# `append` grows the hook's list and leaves the attributed windows alone.
+two, exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))
+check("narrow/two states left behind are counted as two",
+ ((two or {}).get("usage") or {}).get("windows", "<absent>"), 2)
+check("narrow/and two states is what the forward saw", a._forward_calls, [4])
+# the other direction, and in place: `predict_long` hands the hook a copy of its split, so the
+# state the hook invented is scored and counted without moving the windows computed above.
a = make_real_agent()
-_, added_exc = _attempt(
+grown, exc = _attempt(
lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.append("invented")))
-added_err = str(added_exc) if added_exc else "<no error>"
-check("grow/rejects a list with a state added", _kind(added_exc), "ValueError")
-check_true("grow/the count is the caller's split, not the hook's list",
- ("split into %d windows" % nwin) in added_err, added_err)
+check("grow/the hook's added state is scored",
+ ((grown or {}).get("usage") or {}).get("windows", "<absent>"), nwin + 1)
+check("grow/the caller's split is the one that was extended", a._forward_calls, [2 * (nwin + 1)])
+check("grow/and no span is named",
+ sorted(k for v in (grown or {}).get("answers", {}).values() for k in v if k == "window"), [])
-# 8c. rewriting the windows themselves stays supported: the count holds, so the attribution does
+# 8c. an unchanged scan keeps the attribution; any rewrite drops it, even one of the same length
a = make_real_agent()
-res, exc = _attempt(
+noop, exc = _attempt(
lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.extend(())))
-check("rewrite/a no-op on the list scans as usual", _kind(exc), None)
-check("rewrite/a no-op on the list keeps the count",
- ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+check("rewrite/a no-op on the list changes nothing", _kind(exc), None)
+check("rewrite/a no-op keeps the count", ((noop or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+check("rewrite/a no-op keeps the deciding window",
+ ((noop or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)
a = make_real_agent()
-res, exc = _attempt(lambda: a.predict_long(
+upper, exc = _attempt(lambda: a.predict_long(
LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [s.upper() for s in ctx.states])))
check("rewrite/the hook's text is what was scored", (a._encoded or [""])[0].isupper(), True)
-check("rewrite/the same count is the same scan", _kind(exc), None)
-check("rewrite/the count is unchanged", ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
-check("rewrite/the deciding window is still named",
- ((res or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)
+check("rewrite/the same length is still the same count",
+ ((upper or {}).get("usage") or {}).get("windows", "<absent>"), nwin)
+check("rewrite/a text rewrite names no span",
+ sorted(k for v in (upper or {}).get("answers", {}).values() for k in v if k == "window"), [])
+
+# 8e. a hook that leaves nothing scores nothing: 0 windows and no answers, not a max() over []
+a = make_real_agent()
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ empty, exc = _attempt(
+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [])))
+check("empty/nothing scored returns instead of raising", _kind(exc), None)
+check("empty/no answer is invented", (empty or {"answers": "?"}).get("answers", "?"), {})
+check("empty/the count says so", ((empty or {}).get("usage") or {}).get("windows", "<absent>"), 0)
+check("empty/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])
# 8d. the fits-in-one-window path answers to a hook too, so 0 keeps meaning "no window scored this"
a = make_real_agent()
@@ -367,17 +388,17 @@ check("docs/README states the key is total", "The key is total" in bullet, True)
check("docs/README documents all three counts", sorted(set(re.findall(r"`([0-9N])`", bullet))),
["0", "1", "N"])
# The two rules this branch added, pinned where they are taught rather than in the code comment
-hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:800]
-check("docs/README says a hook may not change the window count or order",
- "reorder windows" in hooks_bullet, True)
+hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:900]
+check("docs/README says attribution survives only an unchanged scan",
+ "reported when the scan reached inference unchanged" in hooks_bullet, True)
check("docs/README names ctx.skip as the way to answer", "ctx.skip(...)" in hooks_bullet, True)
API = open(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"docs", "hooks", "api.md"), encoding="utf-8").read()
-api_para = API[API.index("On `predict_long` the hooks wrap"):][:1200]
-check("docs/api.md matches: count and order are fixed",
- "or their" in api_para and "number of states raises" in api_para, True)
-check("docs/api.md matches: the hook answer reads as zero windows",
- 'usage["windows"]` at `0' in api_para, True)
+api_para = API[API.index("On `predict_long` the hooks wrap"):][:1600]
+check("docs/api.md tabulates the rewritten scan as its own case",
+ "| replaced the scan, in any way |" in api_para, True)
+check("docs/api.md tabulates a hook answer as zero windows",
+ "| answered with `ctx.skip([result])` | `0` |" in api_para, True)
check("docs/predict_long's docstring says the key is total",
"always present" in (Agent.predict_long.__doc__ or ""), True)