Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(router): predict_batch hook follow-ups to #289
hemanthrayuduu93152a8a714a
diff --git a/docs/hooks/api.md b/docs/hooks/api.mdindex 8817e666..a1cb47e8 100644--- a/docs/hooks/api.md+++ b/docs/hooks/api.md@@ -18,8 +18,10 @@ from laya.hooks import HOOK_EVENTS, normalise_hooks, dispatch, aggregate_usage## PredictContext-A `PredictContext` is created once per public call and passed to every hook of that call. It is-**mutable**: hooks may rewrite `states`, `questions` and `results`, and `on_route` may rewrite+A `PredictContext` is created once per public call and passed to every hook of that call.+`Router.predict_batch` is the one exception: it creates one per request, as a `Router.predict`+call for that request would, so its Router-level hooks fire once per request with one state+each. It is **mutable**: hooks may rewrite `states`, `questions` and `results`, and `on_route` may rewrite`decision`. It uses identity equality (`eq=False`), so a context is hashable and two contexts arenever equal.@@ -44,7 +46,7 @@ class PredictContext:| field | type | set when | mutable | meaning ||---|---|---|---|---|-| `states` | `list` | always | yes (start) | the states for this call. `system_one`/`Router.predict` pass one; `predict_batch` passes many. A start hook may replace the list. |+| `states` | `list` | always | yes (start) | the states for this call. `system_one`/`Router.predict` pass one; `Agent.predict_batch` passes many; `Router.predict_batch` passes one per request. A start hook may replace the list. || `questions` | `dict` | always | yes (start) | the questions. A start hook may replace the dict. || `run_id` | `str` | always | no | a unique id shared by every hook of this call. Use it to correlate events and spans. || `results` | `list \| None` | end (and on a skip) | yes (end) | per-state result dicts, each shaped like `system_one`'s return. `None` until inference finishes. |diff --git a/docs/hooks/examples.md b/docs/hooks/examples.mdindex b4fef349..266ed36c 100644--- a/docs/hooks/examples.md+++ b/docs/hooks/examples.md@@ -263,7 +263,9 @@ router.predict(## Batch-Hooks fire once per `predict_batch` call, with `ctx.states` holding every state.+Hooks fire once per `Agent.predict_batch` call, with `ctx.states` holding every state.+`Router.predict_batch` runs its Router-level hooks once per request instead, each with one state+and its own `run_id`, so the same hook there writes one record per call of the hook.```pythondef audit_batch(ctx):diff --git a/docs/hooks/lifecycle.md b/docs/hooks/lifecycle.mdindex 76358b2c..692358a2 100644--- a/docs/hooks/lifecycle.md+++ b/docs/hooks/lifecycle.md@@ -129,9 +129,10 @@ Router.predict_batch(requests, batch_size=...)├─ group the requests left to infer by (questions, ctx.max_len, ctx.head_max_len)│ agent.predict_batch(states, questions, ...) ──► one shared forward pass per group│ result["routing"] = decision; ctx.results = [result]- ├─ (any failure) ──► on_error for every started request without a result,- │ on_predict_end for every started request, re-raise- └─ on_predict_end, once per request of this checkpoint, in input order+ ├─ ctx.usage, once per request+ ├─ (any failure) ──► for every started request, in reverse input order:+ │ ctx.error = exc, on_error, on_predict_end; then re-raise+ └─ on_predict_end, once per request of this checkpoint, in reverse input order```Key points:@@ -139,16 +140,26 @@ Key points:- Every start hook of a checkpoint's requests runs before any of their end hooks, because theyshare forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicatestate within the same checkpoint group; it can across calls.+- For the same reason the requests end in reverse of the order they started, so a hook that sets+ something in start and resets it in end (a `contextvars` value, an OpenTelemetry+ `context.attach` / `detach`) unwinds to the value it found.- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its ownrequest only: requests are grouped for the forward pass after their start hooks have run.- Mutating a questions dict in place changes it for every request that shares that dict, and for- the caller, as it would with `predict`.-- Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook+ Mutating a shared questions dict in place is different, and not what `predict` does: nothing of+ the group is inferred until all of its start hooks have run, so the change reaches every request+ that shares the dict, including those whose start hooks ran earlier, and the caller too. Assign+ a new dict to `ctx.questions` instead.+- The checkpoint name is resolved before grouping, so an `on_route` hook that pins a request by+ an alias (`"ml"`) shares that checkpoint's forward pass, and `ctx.model` is the resolved name.+- Every started request gets exactly one `on_predict_end`, even when another request's end hookraises; the first such error is raised after all of them have run.-- If the batch fails, a request that did not get a result is reported as failed (`on_error`, with- `ctx.error` set to the exception that failed the batch), because the caller gets no result for- it. Requests of checkpoint groups that already finished have ended with their results, as the- earlier calls of `[router.predict(...) for ...]` would have.+- If a checkpoint group fails, every one of its started requests fails with the exception: each+ gets `on_error` with `ctx.error` set to it, then `on_predict_end`. That includes a cache hit and+ a request whose question group had already run, because the caller gets the exception and no+ result for any of them, and it means `ctx.error` may be another request's failure (a start hook+ that raises for one request fails its group). Requests of checkpoint groups that already+ finished have ended with their results, as the earlier calls of `[router.predict(...) for ...]`+ would have.## Model lifecyclediff --git a/docs/hooks/tracing.md b/docs/hooks/tracing.mdindex d01c4434..bfd597a6 100644--- a/docs/hooks/tracing.md+++ b/docs/hooks/tracing.md@@ -14,7 +14,8 @@ start, end and error events (and any spans it opens) without keeping its own boo## run_id- A `uuid4().hex` string, created once per public call (`predict_batch`, `system_one`,- `Router.predict`, `ONNXAgent.system_one`).+ `Router.predict`, `ONNXAgent.system_one`). `Router.predict_batch` creates one per request+ instead, the `run_id` a `Router.predict` call for that request would have had.- Shared by every hook of that call, including `on_error` and `on_predict_end`.- Not global and not persisted: it identifies a call within the process. Put it in your logs andoutbound payloads to correlate across systems.diff --git a/laya/router.py b/laya/router.pyindex 7dd999eb..a9f8fa6e 100644--- a/laya/router.py+++ b/laya/router.py@@ -719,9 +719,10 @@ class Router(HookRegistry):gets its own ``PredictContext``, so ``on_predict_start`` can replace that request'sstate, questions or token budget, or ``ctx.skip(...)`` it, and ``on_predict_end``sees and may replace its result. Requests are grouped for the forward pass after- their start hooks have run. If the batch fails, every request whose start hook ran- and that has no result gets ``on_error``, then every started request gets- ``on_predict_end``, before the exception propagates.+ their start hooks have run, and a checkpoint group's requests end in reverse of the+ order they started. If a checkpoint group fails, every request of it whose start hook+ ran fails with the exception, a cache hit included: each gets ``on_error`` and then+ ``on_predict_end`` before the exception propagates.Args:requests: Sequence of request dictionaries. Every item requires ``state`` and@@ -743,10 +744,14 @@ class Router(HookRegistry):groups: Dict[str, List[int]] = {}for i, decision in enumerate(decisions):# Indexed, not `.model`: an on_route hook may replace the decision with a plain dict,- # which `predict` accepts too.- groups.setdefault(decision["model"], []).append(i)-- results: List[Optional[Dict[str, Any]]] = [None] * len(requests)+ # which `predict` accepts too. Normalised, because such a hook may name the checkpoint+ # by an alias ("ml"), and that request must share its checkpoint's forward pass.+ groups.setdefault(normalise_name(decision["model"]), []).append(i)++ # Counted rather than marked with None: an end hook may leave a None result, which+ # `predict` returns as it is.+ results: List[Any] = [None] * len(requests)+ answered = 0# `compose_hooks`, not `list(self.hooks)`: this is the composition `predict` uses at its# own dispatch site, and it is what merges in `set_default_hooks`. Reading the instance# list alone silently dropped every process-wide default from the batched path while@@ -855,61 +860,69 @@ class Router(HookRegistry):for (i, ctx), result in zip(items, batch_results):result["routing"] = dict(decisions[i])ctx.results = [result]++ # Usage is summed here, not while ending, so a malformed usage block (a cached+ # payload's, say) fails the group like any other error instead of escaping from+ # the end loop before a single end hook has run.+ usages = [aggregate_usage(ctx.results) for ctx in started]+ for ctx, usage in zip(started, usages):+ ctx.usage = usageexcept BaseException as exc:- # Every started request is ended, so a hook that opens something in start (a- # span, an in-flight count) always sees the matching end. A request that already- # has its result keeps it; the rest failed with the batch.- for ctx in started:- if ctx.results is None:- ctx.error = exc- try:- dispatch(active, "on_error", ctx, raise_errors=raise_errors,- lock=self._hooks_lock, timeout=timeout)- except BaseException as hook_exc:- exc.__context__ = hook_exc- try:- self._end_contexts(active, started, raise_errors, timeout)- except BaseException as hook_exc:- exc.__context__ = hook_exc+ # The caller gets this exception and no result, so every started request of the+ # group failed with it, including a cache hit or a request whose question group+ # had already run. Each is still ended, so a hook that opens something in start+ # (a span, an in-flight count) always sees the matching end.+ self._end_contexts(active, started, raise_errors, timeout, error=exc)raiseself._end_contexts(active, started, raise_errors, timeout)for i, ctx in zip(indices, started):results[i] = ctx.results[0]+ answered += len(indices)+ # Drop this group's references before the next load() can evict its checkpoint:+ # eviction's gc.collect() and empty_cache() only free an Agent nothing still holds,+ # and every context of the group holds it as `ctx.agent`.+ agent = ctx = started = question_groups = group = items = None# Every input index is assigned exactly once by construction. Keep this assertion local# so a future refactor cannot silently return a partially-filled batch.- if any(result is None for result in results):+ if answered != len(requests):raise RuntimeError("internal error: batch execution did not produce every result")- return [result for result in results if result is not None]+ return resultspredict_many = predict_batchdef _end_contexts(self, active: List[Any], contexts: List[PredictContext], raise_errors: bool,- timeout: Optional[float] = None) -> None:- """Finish each request of a batch the way `predict`'s `finally` finishes one.-- Every context gets its `on_predict_end` even if an earlier one's end hook raises; the- first such failure is raised afterwards. On a context that already failed, a raising- end hook is chained onto its error instead, as in `predict`. Timing and usage are set on- all of them first, so one request's `elapsed_ms` never includes another's end hooks.+ timeout: Optional[float] = None, error: Optional[BaseException] = None) -> None:+ """Finish each request of a batch the way `predict` finishes one.++ `error` is the exception that failed the batch, if one did: every context then fails+ with it and gets `on_error` before its `on_predict_end`, as in `predict`. Contexts end+ in reverse of the order they started, because all of them started before any ends: a+ hook that sets something in start and resets it in end (a contextvar, a tracing+ context) must unwind the last one first. Every context gets its `on_predict_end` even if+ another's hooks raise; the first such failure is raised afterwards. On a failed context,+ a raising hook is chained onto its error instead, as in `predict`. `elapsed_ms` is set+ on all of them first, so one request's never includes another's error or end hooks."""now = time.perf_counter()for ctx in contexts:ctx.elapsed_ms = (now - ctx.started_at) * 1000.0- if ctx.results is not None:- ctx.usage = aggregate_usage(ctx.results)+ if error is not None:+ ctx.error = errorfirst_error: Optional[BaseException] = None- for ctx in contexts:- try:- dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors,- lock=self._hooks_lock, timeout=timeout)- except BaseException as hook_exc:- if ctx.error is not None:- ctx.error.__context__ = hook_exc- elif first_error is None:- first_error = hook_exc+ for ctx in reversed(contexts):+ events = ("on_error", "on_predict_end") if ctx.error is not None else ("on_predict_end",)+ for event in events:+ try:+ dispatch(active, event, ctx, raise_errors=raise_errors,+ lock=self._hooks_lock, timeout=timeout)+ except BaseException as hook_exc:+ if ctx.error is not None:+ ctx.error.__context__ = hook_exc+ elif first_error is None:+ first_error = hook_excif first_error is not None:raise first_errordiff --git a/tests/test_hooks.py b/tests/test_hooks.pyindex 755e69d1..82137343 100644--- a/tests/test_hooks.py+++ b/tests/test_hooks.py@@ -6,6 +6,7 @@ with `_encode_state` / `_forward` / `_decode_answers` stubbed). The Router pathRun: python tests/test_hooks.py"""+import contextvarsimport osimport sysimport threading@@ -617,9 +618,9 @@ class BatchTrace:r, en, ml = batch_router(hooks=[BatchTrace()])out = r.predict_batch([req("one"), req("two", "multilingual"), req("three")])-check("router_batch/one start and one end per request, per checkpoint group",+check("router_batch/one start and one end per request, per checkpoint group, ends in reverse",[(e[0], e[1]) for e in batch_events],- [("start", "one"), ("start", "three"), ("end", "one"), ("end", "three"),+ [("start", "one"), ("start", "three"), ("end", "three"), ("end", "one"),("start", "two"), ("end", "two")])starts = {e[1]: e[2] for e in batch_events if e[0] == "start"}ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}@@ -742,7 +743,7 @@ check_raises("router_batch/inference failure propagates", RuntimeError,check("router_batch/failure pairs every started request with on_error then on_predict_end",[(e[0], e[1]) for e in batch_events],[("start", "ok"), ("end", "ok"), ("start", "boom"), ("start", "also"),- ("error", "boom"), ("error", "also"), ("end", "boom"), ("end", "also")])+ ("error", "also"), ("end", "also"), ("error", "boom"), ("end", "boom")])ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}check_true("router_batch/end of a failed request sees the error",all(isinstance(getattr(ends.get(s), "error", None), RuntimeError) for s in ("boom", "also")))@@ -755,10 +756,33 @@ r, en, ml = batch_router(hooks=[BatchTrace()],check_raises("router_batch/inference failure with a cache hit in the group propagates", RuntimeError,lambda: r.predict_batch([req("cached"), req("boom")]))ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}-check("router_batch/only requests without a result get on_error",- [e[1] for e in batch_events if e[0] == "error"], ["boom"])-check_true("router_batch/a cache hit in a failed group keeps its result and no error",- "cached" in ends and ends["cached"].error is None and ends["cached"].results is not None)+# The caller gets the exception and no result at all, so a cache hit of the failed group was not+# delivered either: reporting it as a success would record a decision nobody received.+check("router_batch/every request of a failed group gets on_error, a cache hit included",+ sorted(e[1] for e in batch_events if e[0] == "error"), ["boom", "cached"])+check_true("router_batch/a cache hit in a failed group ends with the batch's error",+ "cached" in ends and isinstance(ends["cached"].error, RuntimeError))++batch_events = []+r, en, ml = batch_router(hooks=[BatchTrace()],+ on_predict_start=lambda c: setattr(c, "questions", Q_OTHER) if c.states == ["boom"] else None)+check_raises("router_batch/a later question group's failure propagates", RuntimeError,+ lambda: r.predict_batch([req("inferred"), req("boom")]))+check("router_batch/the earlier question group did run", [c[0] for c in en.calls], [["inferred"], ["boom"]])+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}+check_true("router_batch/a request inferred before its group failed ends with the batch's error",+ "inferred" in ends and isinstance(ends["inferred"].error, RuntimeError))++batch_events = []+bad_usage = {"model": "cached", "answers": {}, "usage": {"input_tokens": "n/a", "output_tokens": 0}}+r, en, ml = batch_router(hooks=[BatchTrace()],+ on_predict_start=lambda c: c.skip([dict(bad_usage)]) if c.states == ["bad"] else None)+check_raises("router_batch/a malformed usage block fails the batch", ValueError,+ lambda: r.predict_batch([req("a"), req("bad"), req("c")]))+check("router_batch/a malformed usage block still ends every started request",+ sorted(e[1] for e in batch_events if e[0] == "end"), ["a", "bad", "c"])+check("router_batch/a malformed usage block fails every request of its group",+ sorted(e[1] for e in batch_events if e[0] == "error"), ["a", "bad", "c"])batch_events = []@@ -772,7 +796,7 @@ r, en, ml = batch_router(on_predict_end=fail_end_on_a, hooks=[BatchTrace()])check_raises("router_batch/end hook failure propagates", ValueError,lambda: r.predict_batch([req("a"), req("b")]))check("router_batch/end hook failure still ends the other requests",- [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "a"), ("end", "b")])+ [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "b"), ("end", "a")])batch_events = []r, en, ml = batch_router(hooks=[BatchTrace()])@@ -794,9 +818,79 @@ check_raises("router_batch/start hook failure propagates", ValueError,lambda: r.predict_batch([req("a"), req("b"), req("c")]))check("router_batch/start hook failure still ends every started request",[(e[0], e[1]) for e in batch_events],- [("start", "a"), ("start", "b"), ("error", "a"), ("error", "b"), ("end", "a"), ("end", "b")])+ [("start", "a"), ("start", "b"), ("error", "b"), ("end", "b"), ("error", "a"), ("end", "a")])check("router_batch/start hook failure runs no inference", en.calls, [])+# A hook that sets something in start and resets it in end (a contextvar, an OpenTelemetry+# context attach/detach, a logging MDC) is only correct if the requests unwind in reverse: all+# of a group's starts run before any of its ends, so ending in input order resets the wrong layer.+_batch_cv = contextvars.ContextVar("batch_cv", default=None)+_cv_tokens = {}+++class CvStack:+ def on_predict_start(self, ctx):+ _cv_tokens[ctx] = _batch_cv.set(ctx.states[0])++ def on_predict_end(self, ctx):+ _batch_cv.reset(_cv_tokens.pop(ctx))+++r, en, ml = batch_router(hooks=[CvStack()])+r.predict_batch([req("a"), req("b"), req("c")])+check("router_batch/set-in-start, reset-in-end hooks unwind to the caller's value", _batch_cv.get(), None)+_cv_tokens.clear()+r, en, ml = batch_router(hooks=[CvStack()])+check_raises("router_batch/unwinding a failed batch propagates", RuntimeError,+ lambda: r.predict_batch([req("a"), req("boom"), req("c")]))+check("router_batch/a failed batch also unwinds to the caller's value", _batch_cv.get(), None)+_cv_tokens.clear()++failed_ctxs = []+elapsed_when_first_error_ran = []+++def remember_failed_start(ctx):+ failed_ctxs.append(ctx)+++class FirstError:+ def on_error(self, ctx):+ if not elapsed_when_first_error_ran:+ elapsed_when_first_error_ran.append([c.elapsed_ms is not None for c in failed_ctxs])+++r, en, ml = batch_router(hooks=[FirstError()], on_predict_start=remember_failed_start)+check_raises("router_batch/a failed group propagates", RuntimeError,+ lambda: r.predict_batch([req("boom"), req("b"), req("c")]))+check("router_batch/elapsed_ms is set for a failed group before any on_error runs",+ elapsed_when_first_error_ran, [[True, True, True]])++r, en, ml = batch_router(on_predict_end=lambda c: setattr(c, "results", [None]) if c.states == ["x"] else None)+try:+ got = r.predict_batch([req("x"), req("y")])+ got = (got[0], got[1]["answers"]["seen"])+except Exception as e: # noqa: BLE001+ got = repr(e)+check("router_batch/a None result an end hook leaves is returned, as predict returns it", got, (None, "y"))+++class AliasPin:+ """An on_route hook pinning a request with an alias, as a plain dict."""++ def on_route(self, ctx):+ if ctx.states == ["alias"]:+ ctx.decision = {**ctx.decision, "model": "ml"}+++alias_models = []+r, en, ml = batch_router(hooks=[AliasPin()], on_predict_start=lambda c: alias_models.append(c.model))+r.predict_batch([req("alias"), req("full", "multilingual")])+check("router_batch/an aliased pin shares its checkpoint's forward pass", [c[0] for c in ml.calls],+ [["alias", "full"]])+check("router_batch/an aliased pin reports the resolved checkpoint", alias_models,+ ["multilingual", "multilingual"])+r, en, ml = batch_router(on_predict_start=fail_on_b, hooks_raise=False)with warnings.catch_warnings(record=True) as caught:warnings.simplefilter("always")diff --git a/tests/test_router_batch.py b/tests/test_router_batch.pyindex 8677a9a6..20664121 100644--- a/tests/test_router_batch.py+++ b/tests/test_router_batch.py@@ -111,6 +111,37 @@ def test_inference_exception_propagates_and_cache_remains_consistent(fake_agent)assert router.predict("after failure", Q, lang="ar")["answers"]["seen"] == "after failure"+def test_evicted_checkpoint_is_unreferenced_when_it_is_evicted(monkeypatch):+ # With max_loaded=1, loading the second group's checkpoint evicts the first. Eviction's+ # gc.collect() / empty_cache() only give its memory back if predict_batch no longer holds+ # it at that moment -- directly, or through the previous group's contexts.+ import gc+ import weakref++ import laya.agent++ refs = {}+ evicted_alive = []++ class Agent:+ def __init__(self, repo, *, device, token, subfolder):+ self.checkpoint = subfolder or "english"+ refs[self.checkpoint] = weakref.ref(self)++ def predict_batch(self, states, questions, batch_size=None):+ return [{"answers": {"seen": state}, "usage": {}} for state in states]++ class CheckFreed:+ def on_evict(self, ctx):+ gc.collect()+ evicted_alive.append((ctx.model, refs[ctx.model]() is not None))++ monkeypatch.setattr(laya.agent, "Agent", Agent)+ router = Router(max_loaded=1, hooks=[CheckFreed()])+ router.predict_batch([request("english text"), request("مرحبا")])+ assert evicted_alive == [("english", False)]++def test_warm_cache_and_repeated_batches(fake_agent):built, _ = fake_agentrouter = Router(max_loaded=2)