Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(router): run Router predict hooks for every request in predict_batch (#289)

Hemanth Rayuduaca1920b0cf0
diff --git a/README.md b/README.md
index 7d0ce4cd..c86ef176 100644
--- a/README.md
+++ b/README.md
@@ -277,6 +277,8 @@ Each item can independently set `model`, `task`, `lang`, or `lang_guess`. Use `r
Requests are validated before model loading. Different requests may use different question schemas; requests sharing both a checkpoint and question schema are passed together to `Agent.predict_batch()`.
+[Prediction hooks](#prediction-hooks) installed on the `Router` run once per request, as they do for `predict()`, so a redaction hook rewrites every state before the model sees it. Requests that share a checkpoint run all their start hooks before their shared forward pass; see [`docs/hooks/lifecycle.md`](docs/hooks/lifecycle.md#routerpredict_batch).
+
You can also bound the Agent-level forward-pass batch size:
```python
results = router.predict_batch(requests, batch_size=8)
diff --git a/docs/hooks/lifecycle.md b/docs/hooks/lifecycle.md
index b1745f70..76358b2c 100644
--- a/docs/hooks/lifecycle.md
+++ b/docs/hooks/lifecycle.md
@@ -6,6 +6,7 @@ read the [Router](#routerpredict) one; it is the superset.
- [Agent.predict_batch](#agentpredict_batch)
- [Agent.system_one / predict](#agentsystem_one-predict)
- [Router.predict](#routerpredict)
+- [Router.predict_batch](#routerpredict_batch)
- [Model lifecycle](#model-lifecycle)
- [Caching with skip](#caching-with-skip)
- [Empty inputs](#empty-inputs)
@@ -109,6 +110,46 @@ Key points:
an attached Agent with its own hooks runs those too, which is expected.
- A Router-level `ctx.skip()` still adds `routing`, so the return shape is stable.
+## Router.predict_batch
+
+Each result is what `predict` returns for that request, so Router-level predict hooks run per
+request here too: every request gets its own `PredictContext`, `run_id` and `elapsed_ms`.
+
+```
+Router.predict_batch(requests, batch_size=...)
+ │
+ ├─ route_batch(requests) ──► on_route, once per request (no checkpoint loaded yet)
+ │
+ └─ for each checkpoint, in order of first appearance:
+ │
+ ├─ load(checkpoint) ──► on_load / on_evict
+ ├─ for each request of this checkpoint, in input order:
+ │ ctx = PredictContext(states=[state], questions, decision, model, agent, router)
+ │ on_predict_start a hook may redact, rewrite, set a token budget or skip
+ ├─ group the requests left to infer by (questions, ctx.max_len, ctx.head_max_len)
+ │ agent.predict_batch(states, questions, ...) ──► one shared forward pass per group
+ │ result["routing"] = decision; ctx.results = [result]
+ ├─ (any failure) ──► on_error for every started request without a result,
+ │ on_predict_end for every started request, re-raise
+ └─ on_predict_end, once per request of this checkpoint, in input order
+```
+
+Key points:
+
+- Every start hook of a checkpoint's requests runs before any of their end hooks, because they
+ share forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicate
+ state within the same checkpoint group; it can across calls.
+- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its own
+ request only: requests are grouped for the forward pass after their start hooks have run.
+ Mutating a questions dict in place changes it for every request that shares that dict, and for
+ the caller, as it would with `predict`.
+- Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook
+ raises; the first such error is raised after all of them have run.
+- If the batch fails, a request that did not get a result is reported as failed (`on_error`, with
+ `ctx.error` set to the exception that failed the batch), because the caller gets no result for
+ it. Requests of checkpoint groups that already finished have ended with their results, as the
+ earlier calls of `[router.predict(...) for ...]` would have.
+
## Model lifecycle
`on_load` fires when a checkpoint is built; `on_evict` when one is freed. Both run **after** the
diff --git a/laya/router.py b/laya/router.py
index d3cd785b..837d6fe0 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -637,6 +637,14 @@ class Router(HookRegistry):
Requests may independently specify ``model``, ``task``, ``lang`` or
``lang_guess`` and may use different question schemas.
+ Router-level predict hooks run per request, as ``predict`` runs them: each request
+ gets its own ``PredictContext``, so ``on_predict_start`` can replace that request's
+ state, questions or token budget, or ``ctx.skip(...)`` it, and ``on_predict_end``
+ sees and may replace its result. Requests are grouped for the forward pass after
+ their start hooks have run. If the batch fails, every request whose start hook ran
+ and that has no result gets ``on_error``, then every started request gets
+ ``on_predict_end``, before the exception propagates.
+
Args:
requests: Sequence of request dictionaries. Every item requires ``state`` and
``questions`` and may include ``model``, ``task``, ``lang`` or
@@ -655,57 +663,98 @@ class Router(HookRegistry):
# workload to at most one load per routed checkpoint for this call.
groups: Dict[str, List[int]] = {}
for i, decision in enumerate(decisions):
- groups.setdefault(decision.model, []).append(i)
+ # Indexed, not `.model`: an on_route hook may replace the decision with a plain dict,
+ # which `predict` accepts too.
+ groups.setdefault(decision["model"], []).append(i)
results: List[Optional[Dict[str, Any]]] = [None] * len(requests)
+ active = list(self.hooks)
+ raise_errors = self.hooks_raise
for model_name, indices in groups.items():
agent = self.load(model_name)
-
- # Agent.predict_batch evaluates one shared question schema over many states.
- # Preserve Router's heterogeneous-request API by splitting each checkpoint
- # group again whenever the question schema differs. The signature must be
- # order-sensitive at every nesting level: options are positional in
- # render_options, so two equal schemas that arrive with different key
- # orders render differently and must not share a group -- grouping them
- # would make the second caller's batched answers differ from its own
- # single-request answers. (dict == and == on items() tuples both fall
- # back to order-insensitive dict equality on the nested criteria.)
- question_groups: List[Dict[str, Any]] = []
- for i in indices:
- questions = requests[i]["questions"]
- schema = _question_schema(questions)
+ started: List[PredictContext] = []
+ try:
+ # One context per request, built and started the way `predict` does it, so a
+ # start hook sees -- and can redact, rewrite or skip -- each request before it
+ # joins a shared forward pass.
+ for i in indices:
+ ctx = PredictContext(states=[requests[i]["state"]], questions=requests[i]["questions"],
+ decision=dict(decisions[i]), model=model_name, agent=agent,
+ router=self)
+ started.append(ctx)
+ dispatch(active, "on_predict_start", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+
+ # Agent.predict_batch evaluates one shared question schema and token budget over
+ # many states. Preserve Router's heterogeneous-request API by splitting each
+ # checkpoint group again on what the start hooks left, so a rewritten question
+ # set or `ctx.max_len` only applies to its own request.
+ question_groups: List[Dict[str, Any]] = []
+ for i, ctx in zip(indices, started):
+ if ctx.results is not None:
+ # A cache hit short-circuits inference; keep the `routing` key predict adds.
+ for result in ctx.results:
+ if isinstance(result, dict):
+ result.setdefault("routing", dict(decisions[i]))
+ continue
+ # Pass token-budget overrides only when set, as `predict` does, so an
+ # Agent-like object that does not accept them still works.
+ overrides = {key: value for key, value in (("max_len", ctx.max_len),
+ ("head_max_len", ctx.head_max_len))
+ if value is not None}
+ # Order-sensitive at every nesting level (#166): options are positional, so two
+ # equal schemas with different key orders must not share a group.
+ schema = _question_schema(ctx.questions)
+ for group in question_groups:
+ if group["schema"] == schema and group["overrides"] == overrides:
+ group["items"].append((i, ctx))
+ break
+ else:
+ question_groups.append({
+ "questions": ctx.questions,
+ "schema": schema,
+ "overrides": overrides,
+ "items": [(i, ctx)],
+ })
for group in question_groups:
- if group["schema"] == schema:
- group["indices"].append(i)
- break
- else:
- question_groups.append({
- "questions": questions,
- "schema": schema,
- "indices": [i],
- })
-
- for group in question_groups:
- group_indices = group["indices"]
- states = [requests[i]["state"] for i in group_indices]
-
- batch_results = agent.predict_batch(
- states,
- group["questions"],
- batch_size=batch_size,
- )
-
- if len(batch_results) != len(group_indices):
- raise RuntimeError(
- "internal error: Agent.predict_batch returned %d results for %d states"
- % (len(batch_results), len(group_indices))
+ items = group["items"]
+ batch_results = agent.predict_batch(
+ [ctx.states[0] for _, ctx in items],
+ group["questions"],
+ batch_size=batch_size,
+ **group["overrides"],
)
- for i, result in zip(group_indices, batch_results):
- result["routing"] = dict(decisions[i])
- results[i] = result
+ if len(batch_results) != len(items):
+ raise RuntimeError(
+ "internal error: Agent.predict_batch returned %d results for %d states"
+ % (len(batch_results), len(items))
+ )
+
+ for (i, ctx), result in zip(items, batch_results):
+ result["routing"] = dict(decisions[i])
+ ctx.results = [result]
+ except BaseException as exc:
+ # Every started request is ended, so a hook that opens something in start (a
+ # span, an in-flight count) always sees the matching end. A request that already
+ # has its result keeps it; the rest failed with the batch.
+ for ctx in started:
+ if ctx.results is None:
+ ctx.error = exc
+ try:
+ dispatch(active, "on_error", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ exc.__context__ = hook_exc
+ try:
+ self._end_contexts(active, started, raise_errors)
+ except BaseException as hook_exc:
+ exc.__context__ = hook_exc
+ raise
+
+ self._end_contexts(active, started, raise_errors)
+ for i, ctx in zip(indices, started):
+ results[i] = ctx.results[0]
# Every input index is assigned exactly once by construction. Keep this assertion local
# so a future refactor cannot silently return a partially-filled batch.
@@ -716,5 +765,30 @@ class Router(HookRegistry):
predict_many = predict_batch
+ def _end_contexts(self, active: List[Any], contexts: List[PredictContext], raise_errors: bool) -> None:
+ """Finish each request of a batch the way `predict`'s `finally` finishes one.
+
+ Every context gets its `on_predict_end` even if an earlier one's end hook raises; the
+ first such failure is raised afterwards. On a context that already failed, a raising
+ end hook is chained onto its error instead, as in `predict`. Timing and usage are set on
+ all of them first, so one request's `elapsed_ms` never includes another's end hooks.
+ """
+ now = time.perf_counter()
+ for ctx in contexts:
+ ctx.elapsed_ms = (now - ctx.started_at) * 1000.0
+ if ctx.results is not None:
+ ctx.usage = aggregate_usage(ctx.results)
+ first_error: Optional[BaseException] = None
+ for ctx in contexts:
+ try:
+ dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ if ctx.error is not None:
+ ctx.error.__context__ = hook_exc
+ elif first_error is None:
+ first_error = hook_exc
+ if first_error is not None:
+ raise first_error
+
def __repr__(self):
return "Router(loaded=%s, max_loaded=%d, default=%r)" % (self.loaded, self.max_loaded, self.default)
diff --git a/tests/test_hooks.py b/tests/test_hooks.py
index c1f57fe0..5ce598cc 100644
--- a/tests/test_hooks.py
+++ b/tests/test_hooks.py
@@ -555,6 +555,256 @@ r.predict("hello", QUESTIONS, hooks=[pcr])
check("router/per-call hooks apply to on_route", len(pcr.decisions), 1)
+# --------------------------------------------------------------- Router.predict_batch
+# `predict_batch` promises each result is what `predict` returns for that request, so Router-level
+# predict hooks must run per request there too. It used to skip them: a redaction hook never ran,
+# and the raw state reached the model.
+class BatchFake:
+ """Records every `predict_batch` call and answers with the state it was given."""
+
+ def __init__(self):
+ self.calls = []
+
+ def predict_batch(self, states, questions, batch_size=None, **overrides):
+ self.calls.append((list(states), questions, overrides))
+ if "boom" in states:
+ raise RuntimeError("inference failed")
+ return [{"model": "fake", "answers": {"seen": s},
+ "usage": {"input_tokens": len(s), "output_tokens": 0}} for s in states]
+
+ def system_one(self, state, questions, **overrides):
+ return self.predict_batch([state], questions, **overrides)[0]
+
+
+def batch_router(**kwargs):
+ r = Router(**kwargs)
+ en, ml = BatchFake(), BatchFake()
+ r.attach("english", en)
+ r.attach("multilingual", ml)
+ return r, en, ml
+
+
+def req(state, model="english", questions=QUESTIONS):
+ return {"state": state, "questions": questions, "model": model}
+
+
+def redact_start(ctx):
+ ctx.states = [s.replace("secret", "[redacted]") for s in ctx.states]
+
+
+r, en, ml = batch_router(on_predict_start=redact_start)
+out = r.predict_batch([req("a secret"), req("b secret", "multilingual"), req("c secret")])
+check("router_batch/start hook rewrite reaches the agent",
+ [c[0] for c in en.calls + ml.calls], [["a [redacted]", "c [redacted]"], ["b [redacted]"]])
+check("router_batch/results follow the rewrite", [o["answers"]["seen"] for o in out],
+ ["a [redacted]", "b [redacted]", "c [redacted]"])
+
+batch_events = []
+batch_results_at_start = {}
+
+
+class BatchTrace:
+ def on_predict_start(self, ctx):
+ batch_events.append(("start", ctx.states[0], ctx))
+ batch_results_at_start[ctx.states[0]] = ctx.results
+
+ def on_predict_end(self, ctx):
+ batch_events.append(("end", ctx.states[0], ctx))
+
+ def on_error(self, ctx):
+ batch_events.append(("error", ctx.states[0], ctx))
+
+
+r, en, ml = batch_router(hooks=[BatchTrace()])
+out = r.predict_batch([req("one"), req("two", "multilingual"), req("three")])
+check("router_batch/one start and one end per request, per checkpoint group",
+ [(e[0], e[1]) for e in batch_events],
+ [("start", "one"), ("start", "three"), ("end", "one"), ("end", "three"),
+ ("start", "two"), ("end", "two")])
+starts = {e[1]: e[2] for e in batch_events if e[0] == "start"}
+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
+check("router_batch/every request starts and ends", (sorted(starts), sorted(ends)),
+ (["one", "three", "two"], ["one", "three", "two"]))
+if len(starts) == len(ends) == 3:
+ check("router_batch/start sees its own state", starts["two"].states, ["two"])
+ check("router_batch/start sees its own questions", starts["two"].questions, QUESTIONS)
+ check("router_batch/start sees its own decision", starts["two"].decision["model"], "multilingual")
+ check("router_batch/context model is the routed checkpoint", starts["two"].model, "multilingual")
+ check_true("router_batch/context agent is the routed agent", starts["two"].agent is ml)
+ check_true("router_batch/context router is set", starts["two"].router is r)
+ check("router_batch/start has no results", batch_results_at_start, {"one": None, "two": None, "three": None})
+ check_true("router_batch/start and end share one context per request", ends["two"] is starts["two"])
+ check("router_batch/end sees its own result", ends["two"].results[0]["answers"]["seen"], "two")
+ check("router_batch/end result carries routing", ends["two"].results[0]["routing"]["model"], "multilingual")
+ check("router_batch/usage is per request", ends["three"].usage, {"input_tokens": 5, "output_tokens": 0})
+ check_true("router_batch/elapsed_ms set", ends["one"].elapsed_ms is not None)
+ check("router_batch/run_id differs per request", len({c.run_id for c in starts.values()}), 3)
+check("router_batch/still one agent call per checkpoint", [c[0] for c in en.calls + ml.calls],
+ [["one", "three"], ["two"]])
+
+
+class PlainDictPin:
+ """An on_route hook replacing the decision with a plain dict, which `predict` accepts."""
+
+ def on_route(self, ctx):
+ ctx.decision = {**ctx.decision, "model": "multilingual"}
+
+
+r, en, ml = batch_router(hooks=[PlainDictPin()])
+try:
+ out = r.predict_batch([req("pinned")])
+ got = ([c[0] for c in ml.calls], out[0]["routing"]["model"])
+except AttributeError as e:
+ got = repr(e)
+check("router_batch/an on_route hook may replace the decision with a plain dict", got, ([["pinned"]], "multilingual"))
+
+group_ctxs = []
+elapsed_when_first_end_ran = []
+
+
+def remember_start(ctx):
+ group_ctxs.append(ctx)
+
+
+def first_end(ctx):
+ if not elapsed_when_first_end_ran:
+ elapsed_when_first_end_ran.append([c.elapsed_ms is not None for c in group_ctxs])
+
+
+r, en, ml = batch_router(on_predict_start=remember_start, on_predict_end=first_end)
+r.predict_batch([req("a"), req("b"), req("c")])
+check("router_batch/elapsed_ms is set for the whole group before any end hook runs",
+ elapsed_when_first_end_ran, [[True, True, True]])
+
+
+def replace_result(ctx):
+ ctx.results = [{"replaced": ctx.states[0]}]
+
+
+r, en, ml = batch_router(on_predict_end=replace_result)
+check("router_batch/an end hook's replacement is what is returned",
+ r.predict_batch([req("x"), req("y", "multilingual")]), [{"replaced": "x"}, {"replaced": "y"}])
+
+cached_x = {"model": "cached", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}
+r, en, ml = batch_router(on_predict_start=lambda c: c.skip([dict(cached_x)]) if c.states == ["x"] else None)
+out = r.predict_batch([req("x"), req("y")])
+check("router_batch/skip keeps that state from the agent", [c[0] for c in en.calls], [["y"]])
+check("router_batch/skip returns the cached payload", out[0]["model"], "cached")
+check("router_batch/skip still adds routing", out[0].get("routing", {}).get("model"), "english")
+check("router_batch/the other request is inferred", out[1]["answers"]["seen"], "y")
+
+
+def annotate_end(ctx):
+ ctx.results[0]["audited"] = ctx.decision["model"]
+
+
+mixed = [req("a secret"), req("b secret", "multilingual"), req("c", "english")]
+r, en, ml = batch_router(on_predict_start=redact_start, on_predict_end=annotate_end)
+batched = r.predict_batch(mixed)
+one_by_one = [r.predict(m["state"], m["questions"], model=m["model"]) for m in mixed]
+check("router_batch/identical to one predict call per request", batched, one_by_one)
+
+
+def budget_start(ctx):
+ if ctx.states == ["long"]:
+ ctx.max_len = 1024
+
+
+r, en, ml = batch_router(on_predict_start=budget_start)
+r.predict_batch([req("short"), req("long"), req("short again")])
+check("router_batch/hook-set token budget reaches the agent for that request only",
+ [(c[0], c[2]) for c in en.calls], [(["short", "short again"], {}), (["long"], {"max_len": 1024})])
+
+Q_OTHER = {"other": {"type": "noul", "instructions": "Other?"}}
+r, en, ml = batch_router(on_predict_start=lambda c: setattr(c, "questions", Q_OTHER) if c.states == ["b"] else None)
+r.predict_batch([req("a"), req("b"), req("c")])
+check("router_batch/rewritten questions are batched on their own",
+ [(c[0], c[1]) for c in en.calls], [(["a", "c"], QUESTIONS), (["b"], Q_OTHER)])
+
+
+class StrictBatchFake(BatchFake):
+ """An agent-like object whose predict_batch takes no token-budget kwargs."""
+
+ def predict_batch(self, states, questions, batch_size=None):
+ return BatchFake.predict_batch(self, states, questions, batch_size)
+
+
+r = Router(on_predict_start=lambda c: None)
+strict = StrictBatchFake()
+r.attach("english", strict)
+r.predict_batch([req("a"), req("b")], batch_size=8)
+check("router_batch/default passes no override kwargs", [c[0] for c in strict.calls], [["a", "b"]])
+
+batch_events = []
+r, en, ml = batch_router(hooks=[BatchTrace()])
+check_raises("router_batch/inference failure propagates", RuntimeError,
+ lambda: r.predict_batch([req("ok", "multilingual"), req("boom"), req("also")]))
+check("router_batch/failure pairs every started request with on_error then on_predict_end",
+ [(e[0], e[1]) for e in batch_events],
+ [("start", "ok"), ("end", "ok"), ("start", "boom"), ("start", "also"),
+ ("error", "boom"), ("error", "also"), ("end", "boom"), ("end", "also")])
+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
+check_true("router_batch/end of a failed request sees the error",
+ all(isinstance(getattr(ends.get(s), "error", None), RuntimeError) for s in ("boom", "also")))
+check_true("router_batch/an earlier group that finished ends without an error",
+ "ok" in ends and ends["ok"].error is None)
+
+batch_events = []
+r, en, ml = batch_router(hooks=[BatchTrace()],
+ on_predict_start=lambda c: c.skip([dict(cached_x)]) if c.states == ["cached"] else None)
+check_raises("router_batch/inference failure with a cache hit in the group propagates", RuntimeError,
+ lambda: r.predict_batch([req("cached"), req("boom")]))
+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
+check("router_batch/only requests without a result get on_error",
+ [e[1] for e in batch_events if e[0] == "error"], ["boom"])
+check_true("router_batch/a cache hit in a failed group keeps its result and no error",
+ "cached" in ends and ends["cached"].error is None and ends["cached"].results is not None)
+
+batch_events = []
+
+
+def fail_end_on_a(ctx):
+ if ctx.states == ["a"]:
+ raise ValueError("end hook failed")
+
+
+r, en, ml = batch_router(on_predict_end=fail_end_on_a, hooks=[BatchTrace()])
+check_raises("router_batch/end hook failure propagates", ValueError,
+ lambda: r.predict_batch([req("a"), req("b")]))
+check("router_batch/end hook failure still ends the other requests",
+ [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "a"), ("end", "b")])
+
+batch_events = []
+r, en, ml = batch_router(hooks=[BatchTrace()])
+check_raises("router_batch/inference failure propagates before a later group", RuntimeError,
+ lambda: r.predict_batch([req("boom"), req("later", "multilingual")]))
+check("router_batch/a later group never starts", ("start", "later") in [(e[0], e[1]) for e in batch_events], False)
+check("router_batch/a later group runs no inference", ml.calls, [])
+
+batch_events = []
+
+
+def fail_on_b(ctx):
+ if ctx.states == ["b"]:
+ raise ValueError("start hook failed")
+
+
+r, en, ml = batch_router(hooks=[BatchTrace()], on_predict_start=fail_on_b)
+check_raises("router_batch/start hook failure propagates", ValueError,
+ lambda: r.predict_batch([req("a"), req("b"), req("c")]))
+check("router_batch/start hook failure still ends every started request",
+ [(e[0], e[1]) for e in batch_events],
+ [("start", "a"), ("start", "b"), ("error", "a"), ("error", "b"), ("end", "a"), ("end", "b")])
+check("router_batch/start hook failure runs no inference", en.calls, [])
+
+r, en, ml = batch_router(on_predict_start=fail_on_b, hooks_raise=False)
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ out = r.predict_batch([req("a"), req("b")])
+check("router_batch/hooks_raise=False keeps the batch going", [o["answers"]["seen"] for o in out], ["a", "b"])
+check_true("router_batch/hooks_raise=False warns", any("start hook failed" in str(w.message) for w in caught))
+
+
# --------------------------------------------------------------- hooks_concurrent storage
r = Router()
check("router/hooks_concurrent default True", r.hooks_concurrent, True)