Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(agent): take a hook's answer from ctx.results, not from the result count
Aashishbd18c4714f60
diff --git a/README.md b/README.mdindex e1d8a8b7..c87cb1f8 100644--- a/README.md+++ b/README.md@@ -578,12 +578,14 @@ result = agent.predict_long(state, questions, hooks=[AuditLog()]) # the scan,document lets the neutral majority out-vote the one window that saw the deciding span.- A state that already fits one window is passed straight to `system_one` (identical output, plus`usage["windows"] = 1`). The key is total: `1` single window, `N` scanned windows, `0` a hook- answered a multi-window document before the model read any of it.+ answered the document before the model read any of it.- Hooks wrap the inference that answers the state, so on a scanned document `on_predict_start`fires once with `ctx.states` holding the decoded windows, not the state you passed in (it was- tokenized to produce them). A start hook that answers with `ctx.skip(...)` is answering the- whole document: its result comes back with no `answer["window"]` and `usage["windows"]` at 0,- because no window scored it.+ tokenized to produce them). A start hook may rewrite those window texts, and what is scored is+ what it leaves behind, but it may not add, drop or reorder windows: the answer attributes itself+ to a span of your document by index, so a call that runs inference over any other number of+ states raises. A hook that means to answer the document calls `ctx.skip(...)` instead: its result+ comes back with no `answer["window"]` and `usage["windows"]` at 0, because no window scored it.A smaller `window` isolates a short deciding span better (it becomes a larger fraction of itswindow); the default (`max_len - head_max_len`) favors context and throughput. Output shape matchesdiff --git a/docs/hooks/api.md b/docs/hooks/api.mdindex e6d904d2..0f5d83d9 100644--- a/docs/hooks/api.md+++ b/docs/hooks/api.md@@ -242,12 +242,15 @@ agent.predict(...) # alias of system_one- On `predict_long` the hooks wrap the inference that answers the state, which for a documentneeding several windows is the single shared `predict_batch` over them: `on_predict_start` firesonce, and `ctx.states` holds the decoded window texts in scan order rather than the caller's- state, which was tokenized to produce them. A start hook that answers with `ctx.skip(...)` is- answering the document, so its one result is returned as-is with no `window` key and+ state, which was tokenized to produce them. A start hook may rewrite the text of those windows,+ and what is scored is what it leaves behind, but it may not change how many there are or their+ order -- the answer attributes itself to a span of the caller's document by index, so a call that+ runs inference over any other number of states raises. A hook that means to answer the document+ calls `ctx.skip(...)` instead: its one result is returned as-is with no `window` key and`usage["windows"]` at `0` -- no window scored it. `usage["windows"]` is total over the three- paths: `0` here, `1` for a state that fit a single window (a hook that answered it is answering- that one window, so the count is the same as the model's), and `N` for a document scanned in `N`- overlapping windows.+ paths: `0` for a hook answer on either path, so a cached answer never reads as a window the model+ read, `1` for a state that fit a single window and was scored, and `N` for a document scanned in+ `N` overlapping windows.### Routerdiff --git a/laya/agent.py b/laya/agent.pyindex c7407a31..c026f589 100644--- a/laya/agent.py+++ b/laya/agent.py@@ -29,8 +29,8 @@ from .common import (temp_bucket,)from .hooks import (- HookRegistry, PredictContext, aggregate_usage, compose_hooks, dispatch, normalise_hooks,- validate_timeout,+ HookRegistry, PredictContext, _as_sequence, aggregate_usage, compose_hooks, dispatch,+ normalise_hooks, validate_timeout,)from .revisions import resolve_revision, snapshot_revision, verify_digests@@ -216,6 +216,34 @@ def _cuda_amp_dtype(checkpoint_default: Optional[str]) -> torch.dtype:return amp_dtype(checkpoint_default)+def _start_evidence():+ """A recorder for `predict_long`: what the start-hook chain did, rather than what it cost.++ Why a recorder at all: the result count cannot tell a hook that answered the document from a+ hook that replaced the window list. Both return fewer results than there were windows, and the+ two need opposite readings -- the first scored no window, the second scored the ones the hook+ left behind, and booked tokens for them. `ctx.results` already distinguishes them, and only the+ engine's own context sees it, so `predict_long` appends this probe to the call's start hooks.++ Returns the probe and the dict it fills: `answered` is whether a hook replaced the call before+ inference, `states` is the number of states that reached it.+ """+ evidence = {"answered": False, "states": None}++ def probe(ctx):+ evidence["answered"] = ctx.results is not None+ evidence["states"] = len(ctx.states)++ return probe, evidence+++def _with_start_probe(hook_kwargs, probe):+ """`hook_kwargs` with `probe` appended after the caller's own start hooks."""+ kwargs = dict(hook_kwargs)+ kwargs["on_predict_start"] = list(_as_sequence(hook_kwargs.get("on_predict_start"))) + [probe]+ return kwargs++class Agent(HookRegistry):"""System 1 decision model runtime: fast, non-autoregressive, calibrated decisions."""@@ -986,9 +1014,11 @@ class Agent(HookRegistry):The hooks wrap the inference that answers the state, which for a document needing severalwindows is the one shared `predict_batch` over them: `on_predict_start` fires once, and`ctx.states` holds the decoded window texts in scan order -- not the caller's `state`, which- was tokenized to produce them. A start hook that rewrites `ctx.states` therefore rewrites- windows, and one that calls `ctx.skip(...)` is answering the document rather than a window,- so its payload is returned without window attribution.+ was tokenized to produce them. A start hook may rewrite the text of those windows, and what+ is scored is what it leaves behind. It may not change how many there are, or their order:+ the per-window attribution names spans of the caller's document by index, so a different+ count raises. A hook that means to answer the document itself calls `ctx.skip(...)`, whose+ single payload is returned without window attribution.Args:window: state tokens per window. Defaults to the per-question state budget@@ -1011,9 +1041,10 @@ class Agent(HookRegistry):hooks_timeout: Override the Agent's `hooks_timeout` for this call.Returns a single result dict, the same shape as `system_one`, with `usage["windows"]` added.- The key is always present and counts the windows the answer was read from: `1` for a state- that fit one window, `N` for a document scanned in `N` overlapping windows, and `0` when a- start hook answered a multi-window document before any window was scored.+ The key is always present and counts the windows the model scored to produce the answer: `1`+ for a state that fit one window, `N` for a document scanned in `N` overlapping windows, and+ `0` when a start hook answered before any window was scored -- on either path, so a cached+ answer never reads as a window the model read."""if aggregate != "auto":raise ValueError("predict_long: only aggregate='auto' is supported")@@ -1032,8 +1063,13 @@ class Agent(HookRegistry):# shortest, most common inputs. The result is copied first: a start hook that answers with# `ctx.skip(...)` hands back its own payload dict, and it may be a cached object.if len(state_ids) <= budget:- single = dict(self.system_one(state, questions, lang=lang, **hook_kwargs))- single["usage"] = {**(single.get("usage") or {}), "windows": 1}+ probe, evidence = _start_evidence()+ single = dict(self.system_one(state, questions, lang=lang,+ **_with_start_probe(hook_kwargs, probe)))+ # Why 0 for a hook answer here: the state did fit one window, but no window was scored,+ # which is the same fact the multi-window path reports as 0. Reading 1 would make a+ # cached answer and a served answer agree on how much of the input the model saw.+ single["usage"] = {**(single.get("usage") or {}), "windows": 0 if evidence["answered"] else 1}return singlestep = stride if (stride and stride > 0) else max(1, budget // 2)@@ -1049,19 +1085,22 @@ class Agent(HookRegistry):breaki += step- results = self.predict_batch(windows, questions, batch_size=batch_size, lang=lang,- **hook_kwargs)-- if len(results) != len(windows):- # Only a hook can change the count: `ctx.skip(...)` runs before the split, so a hook- # answering the whole document has no way to know how many windows it will produce and- # returns one result. Aggregating over that payload would pick between answers that were- # never scored and name a deciding window that decided nothing, so pass the document- # answer through unattributed and say which windows were scored: none of them.+ probe, evidence = _start_evidence()+ # `list(windows)`, not `windows`: `ctx.states` is the list the hook receives, so a hook that+ # mutates it in place (`append`, `sort`) would otherwise also grow the split this call+ # attributes answers to, and the counts would agree while `starts` no longer lined up.+ results = self.predict_batch(list(windows), questions, batch_size=batch_size, lang=lang,+ **_with_start_probe(hook_kwargs, probe))++ if evidence["answered"]:+ # The hook replaced the call before any window was scored. Aggregating over its payload+ # would pick between answers that were never scored and name a deciding window that+ # decided nothing, so pass the document answer through unattributed and report the+ # count as what it is: none of them.if len(results) != 1:raise ValueError(- "predict_long: %d windows scored, %d results returned; a start hook that answers"- " a long state returns one result for the document" % (len(windows), len(results)))+ "predict_long: a start hook answered this state with %d results; ctx.skip()"+ " takes one result for the document, not one per window" % len(results))warnings.warn("laya: predict_long: a hook answered the state before it was scanned, so ""no window decided the result and none is reported", RuntimeWarning,stacklevel=2)@@ -1070,6 +1109,19 @@ class Agent(HookRegistry):document["usage"]["windows"] = 0return document+ if len(results) != len(windows):+ # A hook can also rewrite `ctx.states` and let inference run on what it left behind,+ # which moves the count without answering anything. Those results cannot be attributed+ # to windows of the caller's document -- the states that produced them are the hook's+ # -- so refuse rather than name a span that was never read.+ survived = ("" if evidence["states"] is None else+ " (%d states reached inference)" % evidence["states"])+ raise ValueError(+ "predict_long: the state was split into %d windows and the call returned %d"+ " results%s; a start hook may rewrite the text of a window but not their number"+ " or their order -- call ctx.skip([result]) to answer the document without scoring"+ " it" % (len(windows), len(results), survived))+ids = list(questions.keys())internal = {qid: self._to_internal(questions[qid]) for qid in ids}answers = {}diff --git a/tests/test_predict_long.py b/tests/test_predict_long.pyindex 59d280da..ab4471dd 100644--- a/tests/test_predict_long.py+++ b/tests/test_predict_long.py@@ -9,10 +9,13 @@ behaviour on real weights is exercised in tests/test_local_e2e.py.import osimport reimport sys+import threadingimport warningssys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+import numpy as np # noqa: E402+from laya.agent import Agent # noqa: E402PASS, FAIL = [], []@@ -33,6 +36,10 @@ def check_raises(name, exc, fn):FAIL.append("%s: did not raise %s" % (name, exc.__name__))+def check_true(name, cond, detail=""):+ (PASS if cond else FAIL).append(name if cond else "%s: %s" % (name, detail))++class _Tok:mask_token = "[M]"@@ -67,6 +74,49 @@ def make_agent(batch_result_fn):return a+def make_real_agent():+ """`predict_long` on the real `predict_batch`/`system_one`, with only the three composed+ helpers stubbed -- the harness `tests/test_hooks.py` uses, so no weights are involved.++ Why both agents: the stubbed `predict_batch` above is the right instrument for what+ `predict_long` *forwards*, but it never runs a hook, so anything it says about a hook answering+ rests on a result count alone. Here the hook chain really runs and every forward pass is+ counted, which is the difference between "a hook answered the document" and "a hook replaced+ the window list" -- both change the count.+ """+ a = Agent.__new__(Agent)+ a.cfg = {"max_len": 100, "head_max_len": 20} # budget = max(64, 100-20-8) = 72+ a.tok = type("Tok", (_Tok,), {"pad_token_id": 0})()+ a._to_internal = staticmethod(Agent._to_internal).__func__+ a.model_id = "convaiinnovations/laya"+ a.hooks = []+ a.hooks_raise = True+ a.hooks_timeout = None+ a._hooks_lock = threading.Lock()+ a._forward_calls, a._encoded = [], []++ def _encode_state(state, ids, internal, **overrides):+ a._encoded.append(state)+ return [{"ids": [1, 2, 3], "markers": [0, 1], "qtype": 2} for _ in ids]++ def _forward(b):+ n = b["input_ids"].shape[0]+ a._forward_calls.append(n)+ return np.zeros((n, 2), dtype=np.float32), np.full((n, 2), 0.5, dtype=np.float32)++ def _decode_answers(logits, act, items, ids, internal, row, **kw):+ conf = 0.4 + 0.05 * row # so the last window decides+ return {"dept": {"type": "choice", "choice": "a", "confidence": conf,+ "answer_confidence": conf},+ "flag": {"type": "noul", "noul": conf, "confidence": conf,+ "answer_confidence": conf}}++ a._encode_state = _encode_state+ a._forward = _forward+ a._decode_answers = _decode_answers+ return a++Q = {"dept": {"type": "choice", "instructions": "?", "criteria": {"a": "x", "b": "y"}},"flag": {"type": "noul", "instructions": "?"}}@@ -129,15 +179,25 @@ check_raises("aggregate/rejects unknown mode", ValueError,HOOK_KEYS = ("hooks", "on_predict_start", "on_predict_end", "hooks_raise", "hooks_timeout")LONG = {"body": "y" * 300}sentinel = object()+++def sentinel_start(ctx):+ pass++a = make_agent(canned)-res = a.predict_long(LONG, Q, hooks=[sentinel], hooks_raise=False, hooks_timeout=2.5)+res = a.predict_long(LONG, Q, hooks=[sentinel], on_predict_start=sentinel_start,+ hooks_raise=False, hooks_timeout=2.5)got = a._calls["batch_kwargs"]check("hooks/forwarded to the scan", sorted(got), sorted(HOOK_KEYS))check("hooks/the hook list reaches it", got.get("hooks"), [sentinel])check("hooks/hooks_raise reaches it", got.get("hooks_raise"), False)check("hooks/hooks_timeout reaches it", got.get("hooks_timeout"), 2.5)-check("hooks/start/end default to None",- [got.get("on_predict_start"), got.get("on_predict_end")], [None, None])+started = got.get("on_predict_start")+started = started if isinstance(started, list) else ([] if started is None else [started])+check("hooks/the caller's start hook still runs first",+ bool(started) and started[0] is sentinel_start, True)+check("hooks/end stays absent when the caller sets none", got.get("on_predict_end"), None)# 5. the same controls reach the fits-in-one-window patha = make_agent(lambda s, q: [])@@ -161,29 +221,123 @@ plain = make_agent(canned).predict_long(LONG, Q)with_hooks = make_agent(canned).predict_long(LONG, Q, hooks=[sentinel], hooks_timeout=9)check("hooks/forwarding is decision-neutral", with_hooks, plain)-# 8. a start hook that answers the document is not attributed to a window that was never scored-def one_document_answer(states, q):- return [{"answers": {"dept": {"type": "choice", "choice": "a", "answer_confidence": 0.7},- "flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},- "usage": {"input_tokens": 3, "output_tokens": 0}}]+# 8. a start hook that answers the document, on the engine's own evidence+DOC = {"model": "laya-rl-agent",+ "answers": {"dept": {"type": "choice", "choice": "b", "answer_confidence": 0.9},+ "flag": {"type": "noul", "noul": 0.2, "answer_confidence": 0.8}},+ "usage": {"input_tokens": 3, "output_tokens": 0}}++# The control first: how many windows this state really splits into, measured from the forward+# passes it costs rather than from the key under test. Two questions, one row each per window.+scan = make_real_agent()+base = scan.predict_long(LONG, Q)+nwin = sum(scan._forward_calls) // 2+check("scan/more than one window", nwin > 1, True)+check("scan/the scan reports the windows it read", base["usage"]["windows"], nwin)+check("scan/the deciding window is the last",+ base["answers"]["dept"]["window"]["index"], nwin - 1)++# The instrument for sections 8-8c: a hook that answers must return, and a hook that replaces the+# window list must raise. Letting either unexpected case propagate would abort the suite and hide+# every check after it, so each call is captured and its type asserted by a named check.+def _attempt(fn):+ """Call `fn`, returning (value, exception)."""+ try:+ return fn(), None+ except BaseException as exc: # noqa: BLE001+ return None, exc+++def _kind(exc):+ return exc.__class__.__name__ if exc else None-a = make_agent(one_document_answer)+def _narrow(keep):+ """A start hook that truncates the window list and lets inference run on what is left."""+ def hook(ctx):+ ctx.states = list(ctx.states)[:keep]+ return hook+++a = make_real_agent()with warnings.catch_warnings(record=True) as caught:warnings.simplefilter("always")- res = a.predict_long(LONG, Q)-nwin = len(a._calls["batch_states"])-check("skip/the state was split into several windows", nwin > 1, True)-check("skip/no window was scored", res["usage"]["windows"], 0)-check("skip/the payload tokens are kept", res["usage"]["input_tokens"], 3)+ res, skip_exc = _attempt(+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))+check("skip/one payload for the document is never refused as a count mismatch", _kind(skip_exc), None)+check("skip/no forward pass ran", a._forward_calls, [])+usage, answers = (res or {}).get("usage", {}), (res or {}).get("answers", {})+check("skip/no window was scored", usage.get("windows", "<absent>"), 0)+check("skip/the payload tokens are kept", usage.get("input_tokens", "<absent>"), 3)+check("skip/the hook's answer is returned", answers.get("dept", {}).get("choice"), "b")check("skip/no deciding window is claimed",- sorted(k for v in res["answers"].values() for k in v if k == "window"), [])-check("skip/the hook's answer is returned", res["answers"]["dept"]["choice"], "a")+ sorted(k for v in answers.values() for k in v if k == "window"), [])check("skip/the caller is told", [w.category.__name__ for w in caught], ["RuntimeWarning"])-# 9. a payload that matches neither the document nor the windows is an error, not a guess+# 8b. a start hook that replaces the window list is the other reading of the same count, and it is+# not a hook answer: inference ran on the states the hook left behind.+a = make_real_agent()+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ _, narrow_exc = _attempt(lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(1)))+narrow_err = str(narrow_exc) if narrow_exc else "<no error>"+check("narrow/rejects a list shortened to one state", _kind(narrow_exc), "ValueError")+# Why this check sits next to the rejection: the forward is the proof that this was a rewrite and+# not an answer, and it is what made the two cases indistinguishable before.+check("narrow/a forward did run, so nothing was answered before inference",+ a._forward_calls, [2])+check("narrow/the rewrite is not called a hook answer",+ [w.category.__name__ for w in caught], [])+check_true("narrow/the error names the split it could not attribute",+ ("split into %d windows" % nwin) in narrow_err, narrow_err)+check_true("narrow/the error offers ctx.skip as the way to answer",+ "ctx.skip" in narrow_err, narrow_err)+a = make_real_agent()+check_raises("narrow/rejects a list shortened to two states", ValueError,+ lambda: a.predict_long(LONG, Q, on_predict_start=_narrow(2)))+# the other direction, so the rule is the count rather than a hook that dropped something. The+# error must name the caller's split: `predict_long` hands the hook a copy, so an in-place+# `append` grows the hook's list and leaves the attributed windows alone.+a = make_real_agent()+_, added_exc = _attempt(+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.append("invented")))+added_err = str(added_exc) if added_exc else "<no error>"+check("grow/rejects a list with a state added", _kind(added_exc), "ValueError")+check_true("grow/the count is the caller's split, not the hook's list",+ ("split into %d windows" % nwin) in added_err, added_err)++# 8c. rewriting the windows themselves stays supported: the count holds, so the attribution does+a = make_real_agent()+res, exc = _attempt(+ lambda: a.predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.states.extend(())))+check("rewrite/a no-op on the list scans as usual", _kind(exc), None)+check("rewrite/a no-op on the list keeps the count",+ ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+a = make_real_agent()+res, exc = _attempt(lambda: a.predict_long(+ LONG, Q, on_predict_start=lambda ctx: setattr(ctx, "states", [s.upper() for s in ctx.states])))+check("rewrite/the hook's text is what was scored", (a._encoded or [""])[0].isupper(), True)+check("rewrite/the same count is the same scan", _kind(exc), None)+check("rewrite/the count is unchanged", ((res or {}).get("usage") or {}).get("windows", "<absent>"), nwin)+check("rewrite/the deciding window is still named",+ ((res or {}).get("answers") or {}).get("dept", {}).get("window", {}).get("index"), nwin - 1)++# 8d. the fits-in-one-window path answers to a hook too, so 0 keeps meaning "no window scored this"+a = make_real_agent()+short_hooked, exc = _attempt(+ lambda: a.predict_long({"body": "x" * 50}, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))+check("short skip/no forward pass", a._forward_calls, [])+check("short skip/a hook answer is not counted as a window read",+ ((short_hooked or {}).get("usage") or {}).get("windows", "<absent>"), 0)+a = make_real_agent()+short_plain, exc = _attempt(lambda: a.predict_long({"body": "x" * 50}, Q))+check("short skip/the same state without a hook is one window",+ ((short_plain or {}).get("usage") or {}).get("windows", "<absent>"), 1)++# 9. a batch call that returns the wrong count for its own states is an error, not a guessa = make_agent(lambda s, q: [{"answers": {}, "usage": {}}] * 2)-check_raises("skip/rejects a count matching nothing", ValueError, lambda: a.predict_long(LONG, Q))+check_raises("count/rejects a result count matching nothing", ValueError,+ lambda: a.predict_long(LONG, Q))# 10. the count is total, so one question works on every result predict_long can returnshort_all = make_agent(lambda s, q: []).predict_long({"body": "x" * 50}, Q)@@ -191,12 +345,16 @@ scan_agent = make_agent(canned)scanned_all = scan_agent.predict_long(LONG, Q)with warnings.catch_warnings():warnings.simplefilter("ignore")- hook_all = make_agent(one_document_answer).predict_long(LONG, Q)+ hook_all, hook_exc = _attempt(+ lambda: make_real_agent().predict_long(LONG, Q, on_predict_start=lambda ctx: ctx.skip([DOC])))+check("total/a hook answer returns rather than being refused", _kind(hook_exc), None)check("total/no path leaves the key out",- [r["usage"].get("windows", "<absent>") for r in (short_all, scanned_all, hook_all)],+ [(r or {}).get("usage", {}).get("windows", "<absent>")+ for r in (short_all, scanned_all, hook_all)],[1, len(scan_agent._calls["batch_states"]), 0])check("total/0 separates a hook answer from a single-window one",- (hook_all["usage"].get("windows", "<absent>"), short_all["usage"].get("windows", "<absent>")),+ ((hook_all or {}).get("usage", {}).get("windows", "<absent>"),+ short_all["usage"].get("windows", "<absent>")),(0, 1))@@ -208,6 +366,18 @@ check("docs/README pins the single-window value", 'usage["windows"] = 1' in bullcheck("docs/README states the key is total", "The key is total" in bullet, True)check("docs/README documents all three counts", sorted(set(re.findall(r"`([0-9N])`", bullet))),["0", "1", "N"])+# The two rules this branch added, pinned where they are taught rather than in the code comment+hooks_bullet = README[README.index("Hooks wrap the inference that answers the state"):][:800]+check("docs/README says a hook may not change the window count or order",+ "reorder windows" in hooks_bullet, True)+check("docs/README names ctx.skip as the way to answer", "ctx.skip(...)" in hooks_bullet, True)+API = open(os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),+ "docs", "hooks", "api.md"), encoding="utf-8").read()+api_para = API[API.index("On `predict_long` the hooks wrap"):][:1200]+check("docs/api.md matches: count and order are fixed",+ "or their" in api_para and "number of states raises" in api_para, True)+check("docs/api.md matches: the hook answer reads as zero windows",+ 'usage["windows"]` at `0' in api_para, True)check("docs/predict_long's docstring says the key is total","always present" in (Agent.predict_long.__doc__ or ""), True)