Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(router): run Router predict hooks for every request in predict_batch (#289)
Hemanth Rayuduaca1920b0cf0
diff --git a/README.md b/README.mdindex 7d0ce4cd..c86ef176 100644--- a/README.md+++ b/README.md@@ -277,6 +277,8 @@ Each item can independently set `model`, `task`, `lang`, or `lang_guess`. Use `rRequests are validated before model loading. Different requests may use different question schemas; requests sharing both a checkpoint and question schema are passed together to `Agent.predict_batch()`.+[Prediction hooks](#prediction-hooks) installed on the `Router` run once per request, as they do for `predict()`, so a redaction hook rewrites every state before the model sees it. Requests that share a checkpoint run all their start hooks before their shared forward pass; see [`docs/hooks/lifecycle.md`](docs/hooks/lifecycle.md#routerpredict_batch).+You can also bound the Agent-level forward-pass batch size:```pythonresults = router.predict_batch(requests, batch_size=8)diff --git a/docs/hooks/lifecycle.md b/docs/hooks/lifecycle.mdindex b1745f70..76358b2c 100644--- a/docs/hooks/lifecycle.md+++ b/docs/hooks/lifecycle.md@@ -6,6 +6,7 @@ read the [Router](#routerpredict) one; it is the superset.- [Agent.predict_batch](#agentpredict_batch)- [Agent.system_one / predict](#agentsystem_one-predict)- [Router.predict](#routerpredict)+- [Router.predict_batch](#routerpredict_batch)- [Model lifecycle](#model-lifecycle)- [Caching with skip](#caching-with-skip)- [Empty inputs](#empty-inputs)@@ -109,6 +110,46 @@ Key points:an attached Agent with its own hooks runs those too, which is expected.- A Router-level `ctx.skip()` still adds `routing`, so the return shape is stable.+## Router.predict_batch++Each result is what `predict` returns for that request, so Router-level predict hooks run per+request here too: every request gets its own `PredictContext`, `run_id` and `elapsed_ms`.++```+Router.predict_batch(requests, batch_size=...)+ │+ ├─ route_batch(requests) ──► on_route, once per request (no checkpoint loaded yet)+ │+ └─ for each checkpoint, in order of first appearance:+ │+ ├─ load(checkpoint) ──► on_load / on_evict+ ├─ for each request of this checkpoint, in input order:+ │ ctx = PredictContext(states=[state], questions, decision, model, agent, router)+ │ on_predict_start a hook may redact, rewrite, set a token budget or skip+ ├─ group the requests left to infer by (questions, ctx.max_len, ctx.head_max_len)+ │ agent.predict_batch(states, questions, ...) ──► one shared forward pass per group+ │ result["routing"] = decision; ctx.results = [result]+ ├─ (any failure) ──► on_error for every started request without a result,+ │ on_predict_end for every started request, re-raise+ └─ on_predict_end, once per request of this checkpoint, in input order+```++Key points:++- Every start hook of a checkpoint's requests runs before any of their end hooks, because they+ share forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicate+ state within the same checkpoint group; it can across calls.+- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its own+ request only: requests are grouped for the forward pass after their start hooks have run.+ Mutating a questions dict in place changes it for every request that shares that dict, and for+ the caller, as it would with `predict`.+- Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook+ raises; the first such error is raised after all of them have run.+- If the batch fails, a request that did not get a result is reported as failed (`on_error`, with+ `ctx.error` set to the exception that failed the batch), because the caller gets no result for+ it. Requests of checkpoint groups that already finished have ended with their results, as the+ earlier calls of `[router.predict(...) for ...]` would have.+## Model lifecycle`on_load` fires when a checkpoint is built; `on_evict` when one is freed. Both run **after** thediff --git a/laya/router.py b/laya/router.pyindex d3cd785b..837d6fe0 100644--- a/laya/router.py+++ b/laya/router.py@@ -637,6 +637,14 @@ class Router(HookRegistry):Requests may independently specify ``model``, ``task``, ``lang`` or``lang_guess`` and may use different question schemas.+ Router-level predict hooks run per request, as ``predict`` runs them: each request+ gets its own ``PredictContext``, so ``on_predict_start`` can replace that request's+ state, questions or token budget, or ``ctx.skip(...)`` it, and ``on_predict_end``+ sees and may replace its result. Requests are grouped for the forward pass after+ their start hooks have run. If the batch fails, every request whose start hook ran+ and that has no result gets ``on_error``, then every started request gets+ ``on_predict_end``, before the exception propagates.+Args:requests: Sequence of request dictionaries. Every item requires ``state`` and``questions`` and may include ``model``, ``task``, ``lang`` or@@ -655,57 +663,98 @@ class Router(HookRegistry):# workload to at most one load per routed checkpoint for this call.groups: Dict[str, List[int]] = {}for i, decision in enumerate(decisions):- groups.setdefault(decision.model, []).append(i)+ # Indexed, not `.model`: an on_route hook may replace the decision with a plain dict,+ # which `predict` accepts too.+ groups.setdefault(decision["model"], []).append(i)results: List[Optional[Dict[str, Any]]] = [None] * len(requests)+ active = list(self.hooks)+ raise_errors = self.hooks_raisefor model_name, indices in groups.items():agent = self.load(model_name)-- # Agent.predict_batch evaluates one shared question schema over many states.- # Preserve Router's heterogeneous-request API by splitting each checkpoint- # group again whenever the question schema differs. The signature must be- # order-sensitive at every nesting level: options are positional in- # render_options, so two equal schemas that arrive with different key- # orders render differently and must not share a group -- grouping them- # would make the second caller's batched answers differ from its own- # single-request answers. (dict == and == on items() tuples both fall- # back to order-insensitive dict equality on the nested criteria.)- question_groups: List[Dict[str, Any]] = []- for i in indices:- questions = requests[i]["questions"]- schema = _question_schema(questions)+ started: List[PredictContext] = []+ try:+ # One context per request, built and started the way `predict` does it, so a+ # start hook sees -- and can redact, rewrite or skip -- each request before it+ # joins a shared forward pass.+ for i in indices:+ ctx = PredictContext(states=[requests[i]["state"]], questions=requests[i]["questions"],+ decision=dict(decisions[i]), model=model_name, agent=agent,+ router=self)+ started.append(ctx)+ dispatch(active, "on_predict_start", ctx, raise_errors=raise_errors, lock=self._hooks_lock)++ # Agent.predict_batch evaluates one shared question schema and token budget over+ # many states. Preserve Router's heterogeneous-request API by splitting each+ # checkpoint group again on what the start hooks left, so a rewritten question+ # set or `ctx.max_len` only applies to its own request.+ question_groups: List[Dict[str, Any]] = []+ for i, ctx in zip(indices, started):+ if ctx.results is not None:+ # A cache hit short-circuits inference; keep the `routing` key predict adds.+ for result in ctx.results:+ if isinstance(result, dict):+ result.setdefault("routing", dict(decisions[i]))+ continue+ # Pass token-budget overrides only when set, as `predict` does, so an+ # Agent-like object that does not accept them still works.+ overrides = {key: value for key, value in (("max_len", ctx.max_len),+ ("head_max_len", ctx.head_max_len))+ if value is not None}+ # Order-sensitive at every nesting level (#166): options are positional, so two+ # equal schemas with different key orders must not share a group.+ schema = _question_schema(ctx.questions)+ for group in question_groups:+ if group["schema"] == schema and group["overrides"] == overrides:+ group["items"].append((i, ctx))+ break+ else:+ question_groups.append({+ "questions": ctx.questions,+ "schema": schema,+ "overrides": overrides,+ "items": [(i, ctx)],+ })for group in question_groups:- if group["schema"] == schema:- group["indices"].append(i)- break- else:- question_groups.append({- "questions": questions,- "schema": schema,- "indices": [i],- })-- for group in question_groups:- group_indices = group["indices"]- states = [requests[i]["state"] for i in group_indices]-- batch_results = agent.predict_batch(- states,- group["questions"],- batch_size=batch_size,- )-- if len(batch_results) != len(group_indices):- raise RuntimeError(- "internal error: Agent.predict_batch returned %d results for %d states"- % (len(batch_results), len(group_indices))+ items = group["items"]+ batch_results = agent.predict_batch(+ [ctx.states[0] for _, ctx in items],+ group["questions"],+ batch_size=batch_size,+ **group["overrides"],)- for i, result in zip(group_indices, batch_results):- result["routing"] = dict(decisions[i])- results[i] = result+ if len(batch_results) != len(items):+ raise RuntimeError(+ "internal error: Agent.predict_batch returned %d results for %d states"+ % (len(batch_results), len(items))+ )++ for (i, ctx), result in zip(items, batch_results):+ result["routing"] = dict(decisions[i])+ ctx.results = [result]+ except BaseException as exc:+ # Every started request is ended, so a hook that opens something in start (a+ # span, an in-flight count) always sees the matching end. A request that already+ # has its result keeps it; the rest failed with the batch.+ for ctx in started:+ if ctx.results is None:+ ctx.error = exc+ try:+ dispatch(active, "on_error", ctx, raise_errors=raise_errors, lock=self._hooks_lock)+ except BaseException as hook_exc:+ exc.__context__ = hook_exc+ try:+ self._end_contexts(active, started, raise_errors)+ except BaseException as hook_exc:+ exc.__context__ = hook_exc+ raise++ self._end_contexts(active, started, raise_errors)+ for i, ctx in zip(indices, started):+ results[i] = ctx.results[0]# Every input index is assigned exactly once by construction. Keep this assertion local# so a future refactor cannot silently return a partially-filled batch.@@ -716,5 +765,30 @@ class Router(HookRegistry):predict_many = predict_batch+ def _end_contexts(self, active: List[Any], contexts: List[PredictContext], raise_errors: bool) -> None:+ """Finish each request of a batch the way `predict`'s `finally` finishes one.++ Every context gets its `on_predict_end` even if an earlier one's end hook raises; the+ first such failure is raised afterwards. On a context that already failed, a raising+ end hook is chained onto its error instead, as in `predict`. Timing and usage are set on+ all of them first, so one request's `elapsed_ms` never includes another's end hooks.+ """+ now = time.perf_counter()+ for ctx in contexts:+ ctx.elapsed_ms = (now - ctx.started_at) * 1000.0+ if ctx.results is not None:+ ctx.usage = aggregate_usage(ctx.results)+ first_error: Optional[BaseException] = None+ for ctx in contexts:+ try:+ dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock)+ except BaseException as hook_exc:+ if ctx.error is not None:+ ctx.error.__context__ = hook_exc+ elif first_error is None:+ first_error = hook_exc+ if first_error is not None:+ raise first_error+def __repr__(self):return "Router(loaded=%s, max_loaded=%d, default=%r)" % (self.loaded, self.max_loaded, self.default)diff --git a/tests/test_hooks.py b/tests/test_hooks.pyindex c1f57fe0..5ce598cc 100644--- a/tests/test_hooks.py+++ b/tests/test_hooks.py@@ -555,6 +555,256 @@ r.predict("hello", QUESTIONS, hooks=[pcr])check("router/per-call hooks apply to on_route", len(pcr.decisions), 1)+# --------------------------------------------------------------- Router.predict_batch+# `predict_batch` promises each result is what `predict` returns for that request, so Router-level+# predict hooks must run per request there too. It used to skip them: a redaction hook never ran,+# and the raw state reached the model.+class BatchFake:+ """Records every `predict_batch` call and answers with the state it was given."""++ def __init__(self):+ self.calls = []++ def predict_batch(self, states, questions, batch_size=None, **overrides):+ self.calls.append((list(states), questions, overrides))+ if "boom" in states:+ raise RuntimeError("inference failed")+ return [{"model": "fake", "answers": {"seen": s},+ "usage": {"input_tokens": len(s), "output_tokens": 0}} for s in states]++ def system_one(self, state, questions, **overrides):+ return self.predict_batch([state], questions, **overrides)[0]+++def batch_router(**kwargs):+ r = Router(**kwargs)+ en, ml = BatchFake(), BatchFake()+ r.attach("english", en)+ r.attach("multilingual", ml)+ return r, en, ml+++def req(state, model="english", questions=QUESTIONS):+ return {"state": state, "questions": questions, "model": model}+++def redact_start(ctx):+ ctx.states = [s.replace("secret", "[redacted]") for s in ctx.states]+++r, en, ml = batch_router(on_predict_start=redact_start)+out = r.predict_batch([req("a secret"), req("b secret", "multilingual"), req("c secret")])+check("router_batch/start hook rewrite reaches the agent",+ [c[0] for c in en.calls + ml.calls], [["a [redacted]", "c [redacted]"], ["b [redacted]"]])+check("router_batch/results follow the rewrite", [o["answers"]["seen"] for o in out],+ ["a [redacted]", "b [redacted]", "c [redacted]"])++batch_events = []+batch_results_at_start = {}+++class BatchTrace:+ def on_predict_start(self, ctx):+ batch_events.append(("start", ctx.states[0], ctx))+ batch_results_at_start[ctx.states[0]] = ctx.results++ def on_predict_end(self, ctx):+ batch_events.append(("end", ctx.states[0], ctx))++ def on_error(self, ctx):+ batch_events.append(("error", ctx.states[0], ctx))+++r, en, ml = batch_router(hooks=[BatchTrace()])+out = r.predict_batch([req("one"), req("two", "multilingual"), req("three")])+check("router_batch/one start and one end per request, per checkpoint group",+ [(e[0], e[1]) for e in batch_events],+ [("start", "one"), ("start", "three"), ("end", "one"), ("end", "three"),+ ("start", "two"), ("end", "two")])+starts = {e[1]: e[2] for e in batch_events if e[0] == "start"}+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}+check("router_batch/every request starts and ends", (sorted(starts), sorted(ends)),+ (["one", "three", "two"], ["one", "three", "two"]))+if len(starts) == len(ends) == 3:+ check("router_batch/start sees its own state", starts["two"].states, ["two"])+ check("router_batch/start sees its own questions", starts["two"].questions, QUESTIONS)+ check("router_batch/start sees its own decision", starts["two"].decision["model"], "multilingual")+ check("router_batch/context model is the routed checkpoint", starts["two"].model, "multilingual")+ check_true("router_batch/context agent is the routed agent", starts["two"].agent is ml)+ check_true("router_batch/context router is set", starts["two"].router is r)+ check("router_batch/start has no results", batch_results_at_start, {"one": None, "two": None, "three": None})+ check_true("router_batch/start and end share one context per request", ends["two"] is starts["two"])+ check("router_batch/end sees its own result", ends["two"].results[0]["answers"]["seen"], "two")+ check("router_batch/end result carries routing", ends["two"].results[0]["routing"]["model"], "multilingual")+ check("router_batch/usage is per request", ends["three"].usage, {"input_tokens": 5, "output_tokens": 0})+ check_true("router_batch/elapsed_ms set", ends["one"].elapsed_ms is not None)+ check("router_batch/run_id differs per request", len({c.run_id for c in starts.values()}), 3)+check("router_batch/still one agent call per checkpoint", [c[0] for c in en.calls + ml.calls],+ [["one", "three"], ["two"]])+++class PlainDictPin:+ """An on_route hook replacing the decision with a plain dict, which `predict` accepts."""++ def on_route(self, ctx):+ ctx.decision = {**ctx.decision, "model": "multilingual"}+++r, en, ml = batch_router(hooks=[PlainDictPin()])+try:+ out = r.predict_batch([req("pinned")])+ got = ([c[0] for c in ml.calls], out[0]["routing"]["model"])+except AttributeError as e:+ got = repr(e)+check("router_batch/an on_route hook may replace the decision with a plain dict", got, ([["pinned"]], "multilingual"))++group_ctxs = []+elapsed_when_first_end_ran = []+++def remember_start(ctx):+ group_ctxs.append(ctx)+++def first_end(ctx):+ if not elapsed_when_first_end_ran:+ elapsed_when_first_end_ran.append([c.elapsed_ms is not None for c in group_ctxs])+++r, en, ml = batch_router(on_predict_start=remember_start, on_predict_end=first_end)+r.predict_batch([req("a"), req("b"), req("c")])+check("router_batch/elapsed_ms is set for the whole group before any end hook runs",+ elapsed_when_first_end_ran, [[True, True, True]])+++def replace_result(ctx):+ ctx.results = [{"replaced": ctx.states[0]}]+++r, en, ml = batch_router(on_predict_end=replace_result)+check("router_batch/an end hook's replacement is what is returned",+ r.predict_batch([req("x"), req("y", "multilingual")]), [{"replaced": "x"}, {"replaced": "y"}])++cached_x = {"model": "cached", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}+r, en, ml = batch_router(on_predict_start=lambda c: c.skip([dict(cached_x)]) if c.states == ["x"] else None)+out = r.predict_batch([req("x"), req("y")])+check("router_batch/skip keeps that state from the agent", [c[0] for c in en.calls], [["y"]])+check("router_batch/skip returns the cached payload", out[0]["model"], "cached")+check("router_batch/skip still adds routing", out[0].get("routing", {}).get("model"), "english")+check("router_batch/the other request is inferred", out[1]["answers"]["seen"], "y")+++def annotate_end(ctx):+ ctx.results[0]["audited"] = ctx.decision["model"]+++mixed = [req("a secret"), req("b secret", "multilingual"), req("c", "english")]+r, en, ml = batch_router(on_predict_start=redact_start, on_predict_end=annotate_end)+batched = r.predict_batch(mixed)+one_by_one = [r.predict(m["state"], m["questions"], model=m["model"]) for m in mixed]+check("router_batch/identical to one predict call per request", batched, one_by_one)+++def budget_start(ctx):+ if ctx.states == ["long"]:+ ctx.max_len = 1024+++r, en, ml = batch_router(on_predict_start=budget_start)+r.predict_batch([req("short"), req("long"), req("short again")])+check("router_batch/hook-set token budget reaches the agent for that request only",+ [(c[0], c[2]) for c in en.calls], [(["short", "short again"], {}), (["long"], {"max_len": 1024})])++Q_OTHER = {"other": {"type": "noul", "instructions": "Other?"}}+r, en, ml = batch_router(on_predict_start=lambda c: setattr(c, "questions", Q_OTHER) if c.states == ["b"] else None)+r.predict_batch([req("a"), req("b"), req("c")])+check("router_batch/rewritten questions are batched on their own",+ [(c[0], c[1]) for c in en.calls], [(["a", "c"], QUESTIONS), (["b"], Q_OTHER)])+++class StrictBatchFake(BatchFake):+ """An agent-like object whose predict_batch takes no token-budget kwargs."""++ def predict_batch(self, states, questions, batch_size=None):+ return BatchFake.predict_batch(self, states, questions, batch_size)+++r = Router(on_predict_start=lambda c: None)+strict = StrictBatchFake()+r.attach("english", strict)+r.predict_batch([req("a"), req("b")], batch_size=8)+check("router_batch/default passes no override kwargs", [c[0] for c in strict.calls], [["a", "b"]])++batch_events = []+r, en, ml = batch_router(hooks=[BatchTrace()])+check_raises("router_batch/inference failure propagates", RuntimeError,+ lambda: r.predict_batch([req("ok", "multilingual"), req("boom"), req("also")]))+check("router_batch/failure pairs every started request with on_error then on_predict_end",+ [(e[0], e[1]) for e in batch_events],+ [("start", "ok"), ("end", "ok"), ("start", "boom"), ("start", "also"),+ ("error", "boom"), ("error", "also"), ("end", "boom"), ("end", "also")])+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}+check_true("router_batch/end of a failed request sees the error",+ all(isinstance(getattr(ends.get(s), "error", None), RuntimeError) for s in ("boom", "also")))+check_true("router_batch/an earlier group that finished ends without an error",+ "ok" in ends and ends["ok"].error is None)++batch_events = []+r, en, ml = batch_router(hooks=[BatchTrace()],+ on_predict_start=lambda c: c.skip([dict(cached_x)]) if c.states == ["cached"] else None)+check_raises("router_batch/inference failure with a cache hit in the group propagates", RuntimeError,+ lambda: r.predict_batch([req("cached"), req("boom")]))+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}+check("router_batch/only requests without a result get on_error",+ [e[1] for e in batch_events if e[0] == "error"], ["boom"])+check_true("router_batch/a cache hit in a failed group keeps its result and no error",+ "cached" in ends and ends["cached"].error is None and ends["cached"].results is not None)++batch_events = []+++def fail_end_on_a(ctx):+ if ctx.states == ["a"]:+ raise ValueError("end hook failed")+++r, en, ml = batch_router(on_predict_end=fail_end_on_a, hooks=[BatchTrace()])+check_raises("router_batch/end hook failure propagates", ValueError,+ lambda: r.predict_batch([req("a"), req("b")]))+check("router_batch/end hook failure still ends the other requests",+ [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "a"), ("end", "b")])++batch_events = []+r, en, ml = batch_router(hooks=[BatchTrace()])+check_raises("router_batch/inference failure propagates before a later group", RuntimeError,+ lambda: r.predict_batch([req("boom"), req("later", "multilingual")]))+check("router_batch/a later group never starts", ("start", "later") in [(e[0], e[1]) for e in batch_events], False)+check("router_batch/a later group runs no inference", ml.calls, [])++batch_events = []+++def fail_on_b(ctx):+ if ctx.states == ["b"]:+ raise ValueError("start hook failed")+++r, en, ml = batch_router(hooks=[BatchTrace()], on_predict_start=fail_on_b)+check_raises("router_batch/start hook failure propagates", ValueError,+ lambda: r.predict_batch([req("a"), req("b"), req("c")]))+check("router_batch/start hook failure still ends every started request",+ [(e[0], e[1]) for e in batch_events],+ [("start", "a"), ("start", "b"), ("error", "a"), ("error", "b"), ("end", "a"), ("end", "b")])+check("router_batch/start hook failure runs no inference", en.calls, [])++r, en, ml = batch_router(on_predict_start=fail_on_b, hooks_raise=False)+with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ out = r.predict_batch([req("a"), req("b")])+check("router_batch/hooks_raise=False keeps the batch going", [o["answers"]["seen"] for o in out], ["a", "b"])+check_true("router_batch/hooks_raise=False warns", any("start hook failed" in str(w.message) for w in caught))++# --------------------------------------------------------------- hooks_concurrent storager = Router()check("router/hooks_concurrent default True", r.hooks_concurrent, True)