Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(agent): predict_long — scan a state past the context window and aggregate per question

AlKor139acde82c2d05
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ce48ff4f..5acbe56e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -82,6 +82,7 @@ jobs:
python tests/test_router.py
python tests/test_criteria.py
python tests/test_batch.py
+ python tests/test_predict_long.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_structured.py
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 208613a3..1df23c1e 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -40,6 +40,7 @@ jobs:
python tests/test_router.py
python tests/test_criteria.py
python tests/test_batch.py
+ python tests/test_predict_long.py
python tests/test_email.py
python tests/test_download.py
python tests/test_shortlist.py
diff --git a/README.md b/README.md
index 507e230e..232d0ddf 100644
--- a/README.md
+++ b/README.md
@@ -541,6 +541,38 @@ batched (measured ~9–10×). On CPU, increasing batch size alone may not speed
length grouping can help by reducing the padded work in a mixed-length workload. See the
[CPU measurements and reproduction commands](research/README.md#length-batching).
+### Long documents: `predict_long`
+
+`predict`/`system_one` truncate a state that exceeds `max_len` to a single window (the first, or
+for a conversation list the last), silently dropping the rest. `predict_long` scans the whole state
+in overlapping windows, scores them in shared forward passes (via `predict_batch`), and aggregates
+per question:
+
+```python
+result = agent.predict_long(state, questions) # windows the state, one result back
+result = agent.predict_long(state, questions, window=256) # smaller window isolates a localized span
+```
+
+- `noul` takes the strongest window (the statement holds if any window supports it).
+- `choice` / `score` take the most-confident window — averaging over a long, mostly-neutral
+ document lets the neutral majority out-vote the one window that saw the deciding span.
+- A state that already fits one window is passed straight to `system_one` (identical output).
+
+A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its
+window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches
+`predict`, with `usage["windows"]` added.
+
+The returned probability is the deciding window's, **not a calibrated number for the whole
+document** — a `noul` max drifts up with the window count even with no signal, and `choice` can land
+on a confidently-neutral window when nothing is decisive. Each answer carries `answer["window"]`
+(the deciding window's `index`, `token_start`/`token_end`, and `count`) so you can check the span
+the answer actually came from:
+
+```python
+r = agent.predict_long(state, questions)
+r["answers"]["refund"]["window"] # {'index': 13, 'token_start': 4680, 'token_end': 5432, 'count': 14}
+```
+
---
## GPU Fast Path (TileLang)
diff --git a/laya/agent.py b/laya/agent.py
index b0fb78a7..f36515da 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -841,6 +841,108 @@ class Agent(HookRegistry):
raise
return ctx.results
+ def predict_long(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
+ window: Optional[int] = None, stride: Optional[int] = None,
+ aggregate: str = "auto", batch_size: Optional[int] = None,
+ lang: Optional[str] = None) -> Dict[str, Any]:
+ """Evaluate questions over a state longer than the context window, scanning it in
+ overlapping windows and aggregating per question.
+
+ `system_one`/`predict` truncate a state that exceeds `max_len` to a single window (the
+ first, or for a conversation list the last), silently dropping the rest. `predict_long`
+ tokenizes the state once, splits it into overlapping token windows, scores every window in
+ shared forward passes (via `predict_batch`), and combines the per-window answers:
+
+ * noul -> P(true) is the max over windows (the statement holds if any window supports it)
+ * choice-> the answer from the single most-confident window, so a localized signal isn't
+ out-voted by the many neutral windows a long document is mostly made of
+ (averaging drowns it -- the neutral majority dominates)
+ * score -> the level from the most-confident window, likewise
+
+ The returned probability/confidence is the deciding window's, **not a calibrated number for
+ the whole document**: a `noul` max over many windows drifts up with the window count even
+ with no signal, and `choice` can land on a confidently-neutral window when nothing in the
+ document is decisive. Each answer therefore carries `answer["window"]` — the deciding
+ window's `index`, `token_start`/`token_end` into the tokenized state, and the window `count`
+ — so a caller can inspect the span the answer came from rather than trust the raw number.
+
+ A state that already fits one window is passed straight to `system_one` (identical output).
+
+ Args:
+ window: state tokens per window. Defaults to the per-question state budget
+ (`max_len - head_max_len - 8`) -- the most a window can hold for every question.
+ A smaller window isolates a localized signal better (a short deciding span is a
+ larger fraction of its window, so that window classifies it clearly), at the
+ cost of more windows; the large default favors context and throughput. `noul`
+ is robust to this, `choice`/`score` benefit from a smaller window when the
+ deciding span is a small part of a long, otherwise-neutral document.
+ stride: token step between windows. Defaults to `window // 2` (50% overlap), so a span
+ near a boundary still lands whole inside some window.
+ aggregate: "auto" (the per-type rules above) is the only mode for now.
+ batch_size: cap on windows per forward pass, to bound memory on very long states.
+ lang: per-language temperature selection, as in `system_one`.
+
+ Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.
+ """
+ if aggregate != "auto":
+ raise ValueError("predict_long: only aggregate='auto' is supported")
+ max_len = self.cfg.get("max_len", 512)
+ head_max_len = self.cfg.get("head_max_len", 192)
+ budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)
+
+ state_ids = self.tok(serialize_state(state).replace(self.tok.mask_token, " "),
+ add_special_tokens=False)["input_ids"]
+ # Fits in one window: identical to a plain call, no windowing overhead.
+ if len(state_ids) <= budget:
+ return self.system_one(state, questions, lang=lang)
+
+ step = stride if (stride and stride > 0) else max(1, budget // 2)
+ windows, starts = [], []
+ i, n = 0, len(state_ids)
+ while i < n:
+ # Decode each token window back to text so predict_batch re-tokenizes it as a normal
+ # state. For BPE tokenizers the re-tokenized boundaries can shift by a token or two vs
+ # this split; harmless for aggregation since the 50% default overlap absorbs it.
+ windows.append(self.tok.decode(state_ids[i:i + budget]))
+ starts.append(i)
+ if i + budget >= n:
+ break
+ i += step
+
+ results = self.predict_batch(windows, questions, batch_size=batch_size, lang=lang)
+
+ ids = list(questions.keys())
+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}
+ answers = {}
+ for qid in ids:
+ per = [r["answers"][qid] for r in results]
+ if internal[qid]["t"] == "noul":
+ # Evidence anywhere: the strongest window decides. Its own P(true) and confidence
+ # (and act) are carried through, so the fields stay mutually consistent.
+ best = max(range(len(per)), key=lambda j: float(per[j]["noul"]))
+ else:
+ # choice / score: the most-confident window wins. Averaging over a long, mostly
+ # neutral document lets the neutral majority out-vote the one window that saw the
+ # deciding span; the single most-confident window preserves a localized signal.
+ best = max(range(len(per)), key=lambda j: float(per[j]["answer_confidence"]))
+ ans = per[best]
+ # Name the window that decided, so a caller can check the deciding span itself. The
+ # probability here is that window's, NOT a document-level calibrated number.
+ ans["window"] = {"index": best, "token_start": starts[best],
+ "token_end": min(starts[best] + budget, len(state_ids)),
+ "count": len(windows)}
+ answers[qid] = ans
+ # Aggregate usage generically so fields predict_batch may grow later (e.g. the fallback
+ # counters from #351) are propagated, not silently dropped: sum numeric fields across
+ # windows, carry any non-numeric field through, then record the window count.
+ usage: Dict[str, Any] = {}
+ for r in results:
+ for key, val in r["usage"].items():
+ usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val
+ usage["output_tokens"] = 0
+ usage["windows"] = len(windows)
+ return {"model": "laya-rl-agent", "answers": answers, "usage": usage}
+
@torch.no_grad()
def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]], lang: Optional[str] = None,
hooks=None, on_predict_start=None, on_predict_end=None,
diff --git a/tests/test_predict_long.py b/tests/test_predict_long.py
new file mode 100644
index 00000000..e8d70688
--- /dev/null
+++ b/tests/test_predict_long.py
@@ -0,0 +1,114 @@
+"""predict_long: scan a state longer than the window and aggregate per question.
+
+Weight-free. The real forward path is stubbed (predict_batch returns canned per-window answers),
+so this checks only predict_long's own logic: the fits-in-one-window short-circuit, the overlapping
+window split, and the per-type aggregation (noul = strongest window, choice/score = most-confident
+window). Numerical behaviour on real weights is exercised in tests/test_local_e2e.py.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from laya.agent import Agent # noqa: E402
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ (PASS if got == want else FAIL).append(name if got == want else "%s: got %r want %r" % (name, got, want))
+
+
+def check_raises(name, exc, fn):
+ try:
+ fn()
+ except exc:
+ PASS.append(name)
+ except Exception as e: # noqa: BLE001
+ FAIL.append("%s: raised %r not %s" % (name, e, exc.__name__))
+ else:
+ FAIL.append("%s: did not raise %s" % (name, exc.__name__))
+
+
+class _Tok:
+ mask_token = "[M]"
+
+ def __call__(self, text, add_special_tokens=False):
+ # token count == character count, so the test controls windowing by string length
+ return {"input_ids": list(range(len(text)))}
+
+ def decode(self, ids):
+ return "w%d_%d" % (ids[0], ids[-1]) if ids else "w"
+
+
+def make_agent(batch_result_fn):
+ a = Agent.__new__(Agent)
+ a.cfg = {"max_len": 100, "head_max_len": 20} # budget = max(64, 100-20-8) = 72
+ a.tok = _Tok()
+ a._to_internal = staticmethod(Agent._to_internal).__func__
+ a._calls = {"system_one": 0, "batch_states": None}
+
+ def _system_one(state, questions, lang=None):
+ a._calls["system_one"] += 1
+ return {"model": "laya-rl-agent", "answers": {"_via": "system_one"}, "usage": {"input_tokens": 1}}
+
+ def _predict_batch(states, questions, batch_size=None, lang=None):
+ a._calls["batch_states"] = list(states)
+ return batch_result_fn(list(states), questions)
+
+ a.system_one = _system_one
+ a.predict_batch = _predict_batch
+ return a
+
+
+Q = {"dept": {"type": "choice", "instructions": "?", "criteria": {"a": "x", "b": "y"}},
+ "flag": {"type": "noul", "instructions": "?"}}
+
+# 1. fits in one window -> delegates to system_one, no windowing
+a = make_agent(lambda s, q: [])
+short = a.predict_long({"body": "x" * 50}, Q) # 50 tokens <= budget 72
+check("short/delegates to system_one", short["answers"], {"_via": "system_one"})
+check("short/no predict_batch call", a._calls["batch_states"], None)
+
+# 2. long state -> overlapping windows, aggregated per question
+def canned(states, q):
+ # one canned answer per window; the 3rd window is the confident/positive one
+ out = []
+ for i, _ in enumerate(states):
+ conf = 0.9 if i == 2 else 0.4
+ ptrue = 0.95 if i == 2 else 0.1
+ out.append({"answers": {
+ "dept": {"type": "choice", "choice": "b" if i == 2 else "a",
+ "probabilities": {"a": 1 - conf, "b": conf}, "confidence": conf,
+ "answer_confidence": conf, "action": {"act_probability": 1.0}},
+ "flag": {"type": "noul", "noul": ptrue, "confidence": max(ptrue, 1 - ptrue),
+ "answer_confidence": max(ptrue, 1 - ptrue), "action": {"act_probability": 1.0}},
+ }, "usage": {"input_tokens": 10}})
+ return out
+
+
+a = make_agent(canned)
+# 300 tokens, budget 72, stride 36 -> several overlapping windows, last covers the tail
+res = a.predict_long({"body": "y" * 300}, Q)
+nwin = len(a._calls["batch_states"])
+check("long/windows recorded in usage", res["usage"]["windows"], nwin)
+check("long/more than one window", nwin > 1, True)
+check("long/overlap: stride is half the budget", a._calls["batch_states"][1], "w36_107")
+check("long/choice = most-confident window", res["answers"]["dept"]["choice"], "b")
+check("long/noul = strongest window", res["answers"]["flag"]["noul"], 0.95)
+check("long/usage sums window tokens", res["usage"]["input_tokens"], 10 * nwin)
+# the deciding window is named on each answer (window index 2 is the confident/positive one)
+check("long/choice names the deciding window", res["answers"]["dept"]["window"]["index"], 2)
+check("long/noul names the deciding window", res["answers"]["flag"]["window"]["index"], 2)
+check("long/window start is the 3rd overlap offset", res["answers"]["dept"]["window"]["token_start"], 72)
+check("long/window carries the count", res["answers"]["flag"]["window"]["count"], nwin)
+
+# 3. only aggregate="auto" is supported
+a = make_agent(canned)
+check_raises("aggregate/rejects unknown mode", ValueError,
+ lambda: a.predict_long({"body": "y" * 300}, Q, aggregate="mean"))
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL " + f)
+sys.exit(1 if FAIL else 0)