Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(agent): predict_long — scan a state past the context window and aggregate per question
AlKor139acde82c2d05
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex ce48ff4f..5acbe56e 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -82,6 +82,7 @@ jobs:python tests/test_router.pypython tests/test_criteria.pypython tests/test_batch.py+ python tests/test_predict_long.pypython tests/test_hooks.pypython tests/test_hooks_api.pypython tests/test_structured.pydiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 208613a3..1df23c1e 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -40,6 +40,7 @@ jobs:python tests/test_router.pypython tests/test_criteria.pypython tests/test_batch.py+ python tests/test_predict_long.pypython tests/test_email.pypython tests/test_download.pypython tests/test_shortlist.pydiff --git a/README.md b/README.mdindex 507e230e..232d0ddf 100644--- a/README.md+++ b/README.md@@ -541,6 +541,38 @@ batched (measured ~9–10×). On CPU, increasing batch size alone may not speedlength grouping can help by reducing the padded work in a mixed-length workload. See the[CPU measurements and reproduction commands](research/README.md#length-batching).+### Long documents: `predict_long`++`predict`/`system_one` truncate a state that exceeds `max_len` to a single window (the first, or+for a conversation list the last), silently dropping the rest. `predict_long` scans the whole state+in overlapping windows, scores them in shared forward passes (via `predict_batch`), and aggregates+per question:++```python+result = agent.predict_long(state, questions) # windows the state, one result back+result = agent.predict_long(state, questions, window=256) # smaller window isolates a localized span+```++- `noul` takes the strongest window (the statement holds if any window supports it).+- `choice` / `score` take the most-confident window — averaging over a long, mostly-neutral+ document lets the neutral majority out-vote the one window that saw the deciding span.+- A state that already fits one window is passed straight to `system_one` (identical output).++A smaller `window` isolates a short deciding span better (it becomes a larger fraction of its+window); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches+`predict`, with `usage["windows"]` added.++The returned probability is the deciding window's, **not a calibrated number for the whole+document** — a `noul` max drifts up with the window count even with no signal, and `choice` can land+on a confidently-neutral window when nothing is decisive. Each answer carries `answer["window"]`+(the deciding window's `index`, `token_start`/`token_end`, and `count`) so you can check the span+the answer actually came from:++```python+r = agent.predict_long(state, questions)+r["answers"]["refund"]["window"] # {'index': 13, 'token_start': 4680, 'token_end': 5432, 'count': 14}+```+---## GPU Fast Path (TileLang)diff --git a/laya/agent.py b/laya/agent.pyindex b0fb78a7..f36515da 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -841,6 +841,108 @@ class Agent(HookRegistry):raisereturn ctx.results+ def predict_long(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],+ window: Optional[int] = None, stride: Optional[int] = None,+ aggregate: str = "auto", batch_size: Optional[int] = None,+ lang: Optional[str] = None) -> Dict[str, Any]:+ """Evaluate questions over a state longer than the context window, scanning it in+ overlapping windows and aggregating per question.++ `system_one`/`predict` truncate a state that exceeds `max_len` to a single window (the+ first, or for a conversation list the last), silently dropping the rest. `predict_long`+ tokenizes the state once, splits it into overlapping token windows, scores every window in+ shared forward passes (via `predict_batch`), and combines the per-window answers:++ * noul -> P(true) is the max over windows (the statement holds if any window supports it)+ * choice-> the answer from the single most-confident window, so a localized signal isn't+ out-voted by the many neutral windows a long document is mostly made of+ (averaging drowns it -- the neutral majority dominates)+ * score -> the level from the most-confident window, likewise++ The returned probability/confidence is the deciding window's, **not a calibrated number for+ the whole document**: a `noul` max over many windows drifts up with the window count even+ with no signal, and `choice` can land on a confidently-neutral window when nothing in the+ document is decisive. Each answer therefore carries `answer["window"]` — the deciding+ window's `index`, `token_start`/`token_end` into the tokenized state, and the window `count`+ — so a caller can inspect the span the answer came from rather than trust the raw number.++ A state that already fits one window is passed straight to `system_one` (identical output).++ Args:+ window: state tokens per window. Defaults to the per-question state budget+ (`max_len - head_max_len - 8`) -- the most a window can hold for every question.+ A smaller window isolates a localized signal better (a short deciding span is a+ larger fraction of its window, so that window classifies it clearly), at the+ cost of more windows; the large default favors context and throughput. `noul`+ is robust to this, `choice`/`score` benefit from a smaller window when the+ deciding span is a small part of a long, otherwise-neutral document.+ stride: token step between windows. Defaults to `window // 2` (50% overlap), so a span+ near a boundary still lands whole inside some window.+ aggregate: "auto" (the per-type rules above) is the only mode for now.+ batch_size: cap on windows per forward pass, to bound memory on very long states.+ lang: per-language temperature selection, as in `system_one`.++ Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.+ """+ if aggregate != "auto":+ raise ValueError("predict_long: only aggregate='auto' is supported")+ max_len = self.cfg.get("max_len", 512)+ head_max_len = self.cfg.get("head_max_len", 192)+ budget = window if (window and window > 0) else max(64, max_len - head_max_len - 8)++ state_ids = self.tok(serialize_state(state).replace(self.tok.mask_token, " "),+ add_special_tokens=False)["input_ids"]+ # Fits in one window: identical to a plain call, no windowing overhead.+ if len(state_ids) <= budget:+ return self.system_one(state, questions, lang=lang)++ step = stride if (stride and stride > 0) else max(1, budget // 2)+ windows, starts = [], []+ i, n = 0, len(state_ids)+ while i < n:+ # Decode each token window back to text so predict_batch re-tokenizes it as a normal+ # state. For BPE tokenizers the re-tokenized boundaries can shift by a token or two vs+ # this split; harmless for aggregation since the 50% default overlap absorbs it.+ windows.append(self.tok.decode(state_ids[i:i + budget]))+ starts.append(i)+ if i + budget >= n:+ break+ i += step++ results = self.predict_batch(windows, questions, batch_size=batch_size, lang=lang)++ ids = list(questions.keys())+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}+ answers = {}+ for qid in ids:+ per = [r["answers"][qid] for r in results]+ if internal[qid]["t"] == "noul":+ # Evidence anywhere: the strongest window decides. Its own P(true) and confidence+ # (and act) are carried through, so the fields stay mutually consistent.+ best = max(range(len(per)), key=lambda j: float(per[j]["noul"]))+ else:+ # choice / score: the most-confident window wins. Averaging over a long, mostly+ # neutral document lets the neutral majority out-vote the one window that saw the+ # deciding span; the single most-confident window preserves a localized signal.+ best = max(range(len(per)), key=lambda j: float(per[j]["answer_confidence"]))+ ans = per[best]+ # Name the window that decided, so a caller can check the deciding span itself. The+ # probability here is that window's, NOT a document-level calibrated number.+ ans["window"] = {"index": best, "token_start": starts[best],+ "token_end": min(starts[best] + budget, len(state_ids)),+ "count": len(windows)}+ answers[qid] = ans+ # Aggregate usage generically so fields predict_batch may grow later (e.g. the fallback+ # counters from #351) are propagated, not silently dropped: sum numeric fields across+ # windows, carry any non-numeric field through, then record the window count.+ usage: Dict[str, Any] = {}+ for r in results:+ for key, val in r["usage"].items():+ usage[key] = (usage.get(key, 0) + val) if isinstance(val, (int, float)) else val+ usage["output_tokens"] = 0+ usage["windows"] = len(windows)+ return {"model": "laya-rl-agent", "answers": answers, "usage": usage}+@torch.no_grad()def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]], lang: Optional[str] = None,hooks=None, on_predict_start=None, on_predict_end=None,diff --git a/tests/test_predict_long.py b/tests/test_predict_long.pynew file mode 100644index 00000000..e8d70688--- /dev/null+++ b/tests/test_predict_long.py@@ -0,0 +1,114 @@+"""predict_long: scan a state longer than the window and aggregate per question.++Weight-free. The real forward path is stubbed (predict_batch returns canned per-window answers),+so this checks only predict_long's own logic: the fits-in-one-window short-circuit, the overlapping+window split, and the per-type aggregation (noul = strongest window, choice/score = most-confident+window). Numerical behaviour on real weights is exercised in tests/test_local_e2e.py.+"""+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))++from laya.agent import Agent # noqa: E402++PASS, FAIL = [], []+++def check(name, got, want):+ (PASS if got == want else FAIL).append(name if got == want else "%s: got %r want %r" % (name, got, want))+++def check_raises(name, exc, fn):+ try:+ fn()+ except exc:+ PASS.append(name)+ except Exception as e: # noqa: BLE001+ FAIL.append("%s: raised %r not %s" % (name, e, exc.__name__))+ else:+ FAIL.append("%s: did not raise %s" % (name, exc.__name__))+++class _Tok:+ mask_token = "[M]"++ def __call__(self, text, add_special_tokens=False):+ # token count == character count, so the test controls windowing by string length+ return {"input_ids": list(range(len(text)))}++ def decode(self, ids):+ return "w%d_%d" % (ids[0], ids[-1]) if ids else "w"+++def make_agent(batch_result_fn):+ a = Agent.__new__(Agent)+ a.cfg = {"max_len": 100, "head_max_len": 20} # budget = max(64, 100-20-8) = 72+ a.tok = _Tok()+ a._to_internal = staticmethod(Agent._to_internal).__func__+ a._calls = {"system_one": 0, "batch_states": None}++ def _system_one(state, questions, lang=None):+ a._calls["system_one"] += 1+ return {"model": "laya-rl-agent", "answers": {"_via": "system_one"}, "usage": {"input_tokens": 1}}++ def _predict_batch(states, questions, batch_size=None, lang=None):+ a._calls["batch_states"] = list(states)+ return batch_result_fn(list(states), questions)++ a.system_one = _system_one+ a.predict_batch = _predict_batch+ return a+++Q = {"dept": {"type": "choice", "instructions": "?", "criteria": {"a": "x", "b": "y"}},+ "flag": {"type": "noul", "instructions": "?"}}++# 1. fits in one window -> delegates to system_one, no windowing+a = make_agent(lambda s, q: [])+short = a.predict_long({"body": "x" * 50}, Q) # 50 tokens <= budget 72+check("short/delegates to system_one", short["answers"], {"_via": "system_one"})+check("short/no predict_batch call", a._calls["batch_states"], None)++# 2. long state -> overlapping windows, aggregated per question+def canned(states, q):+ # one canned answer per window; the 3rd window is the confident/positive one+ out = []+ for i, _ in enumerate(states):+ conf = 0.9 if i == 2 else 0.4+ ptrue = 0.95 if i == 2 else 0.1+ out.append({"answers": {+ "dept": {"type": "choice", "choice": "b" if i == 2 else "a",+ "probabilities": {"a": 1 - conf, "b": conf}, "confidence": conf,+ "answer_confidence": conf, "action": {"act_probability": 1.0}},+ "flag": {"type": "noul", "noul": ptrue, "confidence": max(ptrue, 1 - ptrue),+ "answer_confidence": max(ptrue, 1 - ptrue), "action": {"act_probability": 1.0}},+ }, "usage": {"input_tokens": 10}})+ return out+++a = make_agent(canned)+# 300 tokens, budget 72, stride 36 -> several overlapping windows, last covers the tail+res = a.predict_long({"body": "y" * 300}, Q)+nwin = len(a._calls["batch_states"])+check("long/windows recorded in usage", res["usage"]["windows"], nwin)+check("long/more than one window", nwin > 1, True)+check("long/overlap: stride is half the budget", a._calls["batch_states"][1], "w36_107")+check("long/choice = most-confident window", res["answers"]["dept"]["choice"], "b")+check("long/noul = strongest window", res["answers"]["flag"]["noul"], 0.95)+check("long/usage sums window tokens", res["usage"]["input_tokens"], 10 * nwin)+# the deciding window is named on each answer (window index 2 is the confident/positive one)+check("long/choice names the deciding window", res["answers"]["dept"]["window"]["index"], 2)+check("long/noul names the deciding window", res["answers"]["flag"]["window"]["index"], 2)+check("long/window start is the 3rd overlap offset", res["answers"]["dept"]["window"]["token_start"], 72)+check("long/window carries the count", res["answers"]["flag"]["window"]["count"], nwin)++# 3. only aggregate="auto" is supported+a = make_agent(canned)+check_raises("aggregate/rejects unknown mode", ValueError,+ lambda: a.predict_long({"body": "y" * 300}, Q, aggregate="mean"))++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))+for f in FAIL:+ print(" FAIL " + f)+sys.exit(1 if FAIL else 0)