Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(router): predict_batch hook follow-ups to #289

hemanthrayuduu93152a8a714a
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
index 8817e666..a1cb47e8 100644
--- a/docs/hooks/api.md
+++ b/docs/hooks/api.md
@@ -18,8 +18,10 @@ from laya.hooks import HOOK_EVENTS, normalise_hooks, dispatch, aggregate_usage
## PredictContext
-A `PredictContext` is created once per public call and passed to every hook of that call. It is
-**mutable**: hooks may rewrite `states`, `questions` and `results`, and `on_route` may rewrite
+A `PredictContext` is created once per public call and passed to every hook of that call.
+`Router.predict_batch` is the one exception: it creates one per request, as a `Router.predict`
+call for that request would, so its Router-level hooks fire once per request with one state
+each. It is **mutable**: hooks may rewrite `states`, `questions` and `results`, and `on_route` may rewrite
`decision`. It uses identity equality (`eq=False`), so a context is hashable and two contexts are
never equal.
@@ -44,7 +46,7 @@ class PredictContext:
| field | type | set when | mutable | meaning |
|---|---|---|---|---|
-| `states` | `list` | always | yes (start) | the states for this call. `system_one`/`Router.predict` pass one; `predict_batch` passes many. A start hook may replace the list. |
+| `states` | `list` | always | yes (start) | the states for this call. `system_one`/`Router.predict` pass one; `Agent.predict_batch` passes many; `Router.predict_batch` passes one per request. A start hook may replace the list. |
| `questions` | `dict` | always | yes (start) | the questions. A start hook may replace the dict. |
| `run_id` | `str` | always | no | a unique id shared by every hook of this call. Use it to correlate events and spans. |
| `results` | `list \| None` | end (and on a skip) | yes (end) | per-state result dicts, each shaped like `system_one`'s return. `None` until inference finishes. |
diff --git a/docs/hooks/examples.md b/docs/hooks/examples.md
index b4fef349..266ed36c 100644
--- a/docs/hooks/examples.md
+++ b/docs/hooks/examples.md
@@ -263,7 +263,9 @@ router.predict(
## Batch
-Hooks fire once per `predict_batch` call, with `ctx.states` holding every state.
+Hooks fire once per `Agent.predict_batch` call, with `ctx.states` holding every state.
+`Router.predict_batch` runs its Router-level hooks once per request instead, each with one state
+and its own `run_id`, so the same hook there writes one record per call of the hook.
```python
def audit_batch(ctx):
diff --git a/docs/hooks/lifecycle.md b/docs/hooks/lifecycle.md
index 76358b2c..692358a2 100644
--- a/docs/hooks/lifecycle.md
+++ b/docs/hooks/lifecycle.md
@@ -129,9 +129,10 @@ Router.predict_batch(requests, batch_size=...)
├─ group the requests left to infer by (questions, ctx.max_len, ctx.head_max_len)
│ agent.predict_batch(states, questions, ...) ──► one shared forward pass per group
│ result["routing"] = decision; ctx.results = [result]
- ├─ (any failure) ──► on_error for every started request without a result,
- │ on_predict_end for every started request, re-raise
- └─ on_predict_end, once per request of this checkpoint, in input order
+ ├─ ctx.usage, once per request
+ ├─ (any failure) ──► for every started request, in reverse input order:
+ │ ctx.error = exc, on_error, on_predict_end; then re-raise
+ └─ on_predict_end, once per request of this checkpoint, in reverse input order
```
Key points:
@@ -139,16 +140,26 @@ Key points:
- Every start hook of a checkpoint's requests runs before any of their end hooks, because they
share forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicate
state within the same checkpoint group; it can across calls.
+- For the same reason the requests end in reverse of the order they started, so a hook that sets
+ something in start and resets it in end (a `contextvars` value, an OpenTelemetry
+ `context.attach` / `detach`) unwinds to the value it found.
- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its own
request only: requests are grouped for the forward pass after their start hooks have run.
- Mutating a questions dict in place changes it for every request that shares that dict, and for
- the caller, as it would with `predict`.
-- Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook
+ Mutating a shared questions dict in place is different, and not what `predict` does: nothing of
+ the group is inferred until all of its start hooks have run, so the change reaches every request
+ that shares the dict, including those whose start hooks ran earlier, and the caller too. Assign
+ a new dict to `ctx.questions` instead.
+- The checkpoint name is resolved before grouping, so an `on_route` hook that pins a request by
+ an alias (`"ml"`) shares that checkpoint's forward pass, and `ctx.model` is the resolved name.
+- Every started request gets exactly one `on_predict_end`, even when another request's end hook
raises; the first such error is raised after all of them have run.
-- If the batch fails, a request that did not get a result is reported as failed (`on_error`, with
- `ctx.error` set to the exception that failed the batch), because the caller gets no result for
- it. Requests of checkpoint groups that already finished have ended with their results, as the
- earlier calls of `[router.predict(...) for ...]` would have.
+- If a checkpoint group fails, every one of its started requests fails with the exception: each
+ gets `on_error` with `ctx.error` set to it, then `on_predict_end`. That includes a cache hit and
+ a request whose question group had already run, because the caller gets the exception and no
+ result for any of them, and it means `ctx.error` may be another request's failure (a start hook
+ that raises for one request fails its group). Requests of checkpoint groups that already
+ finished have ended with their results, as the earlier calls of `[router.predict(...) for ...]`
+ would have.
## Model lifecycle
diff --git a/docs/hooks/tracing.md b/docs/hooks/tracing.md
index d01c4434..bfd597a6 100644
--- a/docs/hooks/tracing.md
+++ b/docs/hooks/tracing.md
@@ -14,7 +14,8 @@ start, end and error events (and any spans it opens) without keeping its own boo
## run_id
- A `uuid4().hex` string, created once per public call (`predict_batch`, `system_one`,
- `Router.predict`, `ONNXAgent.system_one`).
+ `Router.predict`, `ONNXAgent.system_one`). `Router.predict_batch` creates one per request
+ instead, the `run_id` a `Router.predict` call for that request would have had.
- Shared by every hook of that call, including `on_error` and `on_predict_end`.
- Not global and not persisted: it identifies a call within the process. Put it in your logs and
outbound payloads to correlate across systems.
diff --git a/laya/router.py b/laya/router.py
index 7dd999eb..a9f8fa6e 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -719,9 +719,10 @@ class Router(HookRegistry):
gets its own ``PredictContext``, so ``on_predict_start`` can replace that request's
state, questions or token budget, or ``ctx.skip(...)`` it, and ``on_predict_end``
sees and may replace its result. Requests are grouped for the forward pass after
- their start hooks have run. If the batch fails, every request whose start hook ran
- and that has no result gets ``on_error``, then every started request gets
- ``on_predict_end``, before the exception propagates.
+ their start hooks have run, and a checkpoint group's requests end in reverse of the
+ order they started. If a checkpoint group fails, every request of it whose start hook
+ ran fails with the exception, a cache hit included: each gets ``on_error`` and then
+ ``on_predict_end`` before the exception propagates.
Args:
requests: Sequence of request dictionaries. Every item requires ``state`` and
@@ -743,10 +744,14 @@ class Router(HookRegistry):
groups: Dict[str, List[int]] = {}
for i, decision in enumerate(decisions):
# Indexed, not `.model`: an on_route hook may replace the decision with a plain dict,
- # which `predict` accepts too.
- groups.setdefault(decision["model"], []).append(i)
-
- results: List[Optional[Dict[str, Any]]] = [None] * len(requests)
+ # which `predict` accepts too. Normalised, because such a hook may name the checkpoint
+ # by an alias ("ml"), and that request must share its checkpoint's forward pass.
+ groups.setdefault(normalise_name(decision["model"]), []).append(i)
+
+ # Counted rather than marked with None: an end hook may leave a None result, which
+ # `predict` returns as it is.
+ results: List[Any] = [None] * len(requests)
+ answered = 0
# `compose_hooks`, not `list(self.hooks)`: this is the composition `predict` uses at its
# own dispatch site, and it is what merges in `set_default_hooks`. Reading the instance
# list alone silently dropped every process-wide default from the batched path while
@@ -855,61 +860,69 @@ class Router(HookRegistry):
for (i, ctx), result in zip(items, batch_results):
result["routing"] = dict(decisions[i])
ctx.results = [result]
+
+ # Usage is summed here, not while ending, so a malformed usage block (a cached
+ # payload's, say) fails the group like any other error instead of escaping from
+ # the end loop before a single end hook has run.
+ usages = [aggregate_usage(ctx.results) for ctx in started]
+ for ctx, usage in zip(started, usages):
+ ctx.usage = usage
except BaseException as exc:
- # Every started request is ended, so a hook that opens something in start (a
- # span, an in-flight count) always sees the matching end. A request that already
- # has its result keeps it; the rest failed with the batch.
- for ctx in started:
- if ctx.results is None:
- ctx.error = exc
- try:
- dispatch(active, "on_error", ctx, raise_errors=raise_errors,
- lock=self._hooks_lock, timeout=timeout)
- except BaseException as hook_exc:
- exc.__context__ = hook_exc
- try:
- self._end_contexts(active, started, raise_errors, timeout)
- except BaseException as hook_exc:
- exc.__context__ = hook_exc
+ # The caller gets this exception and no result, so every started request of the
+ # group failed with it, including a cache hit or a request whose question group
+ # had already run. Each is still ended, so a hook that opens something in start
+ # (a span, an in-flight count) always sees the matching end.
+ self._end_contexts(active, started, raise_errors, timeout, error=exc)
raise
self._end_contexts(active, started, raise_errors, timeout)
for i, ctx in zip(indices, started):
results[i] = ctx.results[0]
+ answered += len(indices)
+ # Drop this group's references before the next load() can evict its checkpoint:
+ # eviction's gc.collect() and empty_cache() only free an Agent nothing still holds,
+ # and every context of the group holds it as `ctx.agent`.
+ agent = ctx = started = question_groups = group = items = None
# Every input index is assigned exactly once by construction. Keep this assertion local
# so a future refactor cannot silently return a partially-filled batch.
- if any(result is None for result in results):
+ if answered != len(requests):
raise RuntimeError("internal error: batch execution did not produce every result")
- return [result for result in results if result is not None]
+ return results
predict_many = predict_batch
def _end_contexts(self, active: List[Any], contexts: List[PredictContext], raise_errors: bool,
- timeout: Optional[float] = None) -> None:
- """Finish each request of a batch the way `predict`'s `finally` finishes one.
-
- Every context gets its `on_predict_end` even if an earlier one's end hook raises; the
- first such failure is raised afterwards. On a context that already failed, a raising
- end hook is chained onto its error instead, as in `predict`. Timing and usage are set on
- all of them first, so one request's `elapsed_ms` never includes another's end hooks.
+ timeout: Optional[float] = None, error: Optional[BaseException] = None) -> None:
+ """Finish each request of a batch the way `predict` finishes one.
+
+ `error` is the exception that failed the batch, if one did: every context then fails
+ with it and gets `on_error` before its `on_predict_end`, as in `predict`. Contexts end
+ in reverse of the order they started, because all of them started before any ends: a
+ hook that sets something in start and resets it in end (a contextvar, a tracing
+ context) must unwind the last one first. Every context gets its `on_predict_end` even if
+ another's hooks raise; the first such failure is raised afterwards. On a failed context,
+ a raising hook is chained onto its error instead, as in `predict`. `elapsed_ms` is set
+ on all of them first, so one request's never includes another's error or end hooks.
"""
now = time.perf_counter()
for ctx in contexts:
ctx.elapsed_ms = (now - ctx.started_at) * 1000.0
- if ctx.results is not None:
- ctx.usage = aggregate_usage(ctx.results)
+ if error is not None:
+ ctx.error = error
first_error: Optional[BaseException] = None
- for ctx in contexts:
- try:
- dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors,
- lock=self._hooks_lock, timeout=timeout)
- except BaseException as hook_exc:
- if ctx.error is not None:
- ctx.error.__context__ = hook_exc
- elif first_error is None:
- first_error = hook_exc
+ for ctx in reversed(contexts):
+ events = ("on_error", "on_predict_end") if ctx.error is not None else ("on_predict_end",)
+ for event in events:
+ try:
+ dispatch(active, event, ctx, raise_errors=raise_errors,
+ lock=self._hooks_lock, timeout=timeout)
+ except BaseException as hook_exc:
+ if ctx.error is not None:
+ ctx.error.__context__ = hook_exc
+ elif first_error is None:
+ first_error = hook_exc
if first_error is not None:
raise first_error
diff --git a/tests/test_hooks.py b/tests/test_hooks.py
index 755e69d1..82137343 100644
--- a/tests/test_hooks.py
+++ b/tests/test_hooks.py
@@ -6,6 +6,7 @@ with `_encode_state` / `_forward` / `_decode_answers` stubbed). The Router path
Run: python tests/test_hooks.py
"""
+import contextvars
import os
import sys
import threading
@@ -617,9 +618,9 @@ class BatchTrace:
r, en, ml = batch_router(hooks=[BatchTrace()])
out = r.predict_batch([req("one"), req("two", "multilingual"), req("three")])
-check("router_batch/one start and one end per request, per checkpoint group",
+check("router_batch/one start and one end per request, per checkpoint group, ends in reverse",
[(e[0], e[1]) for e in batch_events],
- [("start", "one"), ("start", "three"), ("end", "one"), ("end", "three"),
+ [("start", "one"), ("start", "three"), ("end", "three"), ("end", "one"),
("start", "two"), ("end", "two")])
starts = {e[1]: e[2] for e in batch_events if e[0] == "start"}
ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
@@ -742,7 +743,7 @@ check_raises("router_batch/inference failure propagates", RuntimeError,
check("router_batch/failure pairs every started request with on_error then on_predict_end",
[(e[0], e[1]) for e in batch_events],
[("start", "ok"), ("end", "ok"), ("start", "boom"), ("start", "also"),
- ("error", "boom"), ("error", "also"), ("end", "boom"), ("end", "also")])
+ ("error", "also"), ("end", "also"), ("error", "boom"), ("end", "boom")])
ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
check_true("router_batch/end of a failed request sees the error",
all(isinstance(getattr(ends.get(s), "error", None), RuntimeError) for s in ("boom", "also")))
@@ -755,10 +756,33 @@ r, en, ml = batch_router(hooks=[BatchTrace()],
check_raises("router_batch/inference failure with a cache hit in the group propagates", RuntimeError,
lambda: r.predict_batch([req("cached"), req("boom")]))
ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
-check("router_batch/only requests without a result get on_error",
- [e[1] for e in batch_events if e[0] == "error"], ["boom"])
-check_true("router_batch/a cache hit in a failed group keeps its result and no error",
- "cached" in ends and ends["cached"].error is None and ends["cached"].results is not None)
+# The caller gets the exception and no result at all, so a cache hit of the failed group was not
+# delivered either: reporting it as a success would record a decision nobody received.
+check("router_batch/every request of a failed group gets on_error, a cache hit included",
+ sorted(e[1] for e in batch_events if e[0] == "error"), ["boom", "cached"])
+check_true("router_batch/a cache hit in a failed group ends with the batch's error",
+ "cached" in ends and isinstance(ends["cached"].error, RuntimeError))
+
+batch_events = []
+r, en, ml = batch_router(hooks=[BatchTrace()],
+ on_predict_start=lambda c: setattr(c, "questions", Q_OTHER) if c.states == ["boom"] else None)
+check_raises("router_batch/a later question group's failure propagates", RuntimeError,
+ lambda: r.predict_batch([req("inferred"), req("boom")]))
+check("router_batch/the earlier question group did run", [c[0] for c in en.calls], [["inferred"], ["boom"]])
+ends = {e[1]: e[2] for e in batch_events if e[0] == "end"}
+check_true("router_batch/a request inferred before its group failed ends with the batch's error",
+ "inferred" in ends and isinstance(ends["inferred"].error, RuntimeError))
+
+batch_events = []
+bad_usage = {"model": "cached", "answers": {}, "usage": {"input_tokens": "n/a", "output_tokens": 0}}
+r, en, ml = batch_router(hooks=[BatchTrace()],
+ on_predict_start=lambda c: c.skip([dict(bad_usage)]) if c.states == ["bad"] else None)
+check_raises("router_batch/a malformed usage block fails the batch", ValueError,
+ lambda: r.predict_batch([req("a"), req("bad"), req("c")]))
+check("router_batch/a malformed usage block still ends every started request",
+ sorted(e[1] for e in batch_events if e[0] == "end"), ["a", "bad", "c"])
+check("router_batch/a malformed usage block fails every request of its group",
+ sorted(e[1] for e in batch_events if e[0] == "error"), ["a", "bad", "c"])
batch_events = []
@@ -772,7 +796,7 @@ r, en, ml = batch_router(on_predict_end=fail_end_on_a, hooks=[BatchTrace()])
check_raises("router_batch/end hook failure propagates", ValueError,
lambda: r.predict_batch([req("a"), req("b")]))
check("router_batch/end hook failure still ends the other requests",
- [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "a"), ("end", "b")])
+ [(e[0], e[1]) for e in batch_events if e[0] == "end"], [("end", "b"), ("end", "a")])
batch_events = []
r, en, ml = batch_router(hooks=[BatchTrace()])
@@ -794,9 +818,79 @@ check_raises("router_batch/start hook failure propagates", ValueError,
lambda: r.predict_batch([req("a"), req("b"), req("c")]))
check("router_batch/start hook failure still ends every started request",
[(e[0], e[1]) for e in batch_events],
- [("start", "a"), ("start", "b"), ("error", "a"), ("error", "b"), ("end", "a"), ("end", "b")])
+ [("start", "a"), ("start", "b"), ("error", "b"), ("end", "b"), ("error", "a"), ("end", "a")])
check("router_batch/start hook failure runs no inference", en.calls, [])
+# A hook that sets something in start and resets it in end (a contextvar, an OpenTelemetry
+# context attach/detach, a logging MDC) is only correct if the requests unwind in reverse: all
+# of a group's starts run before any of its ends, so ending in input order resets the wrong layer.
+_batch_cv = contextvars.ContextVar("batch_cv", default=None)
+_cv_tokens = {}
+
+
+class CvStack:
+ def on_predict_start(self, ctx):
+ _cv_tokens[ctx] = _batch_cv.set(ctx.states[0])
+
+ def on_predict_end(self, ctx):
+ _batch_cv.reset(_cv_tokens.pop(ctx))
+
+
+r, en, ml = batch_router(hooks=[CvStack()])
+r.predict_batch([req("a"), req("b"), req("c")])
+check("router_batch/set-in-start, reset-in-end hooks unwind to the caller's value", _batch_cv.get(), None)
+_cv_tokens.clear()
+r, en, ml = batch_router(hooks=[CvStack()])
+check_raises("router_batch/unwinding a failed batch propagates", RuntimeError,
+ lambda: r.predict_batch([req("a"), req("boom"), req("c")]))
+check("router_batch/a failed batch also unwinds to the caller's value", _batch_cv.get(), None)
+_cv_tokens.clear()
+
+failed_ctxs = []
+elapsed_when_first_error_ran = []
+
+
+def remember_failed_start(ctx):
+ failed_ctxs.append(ctx)
+
+
+class FirstError:
+ def on_error(self, ctx):
+ if not elapsed_when_first_error_ran:
+ elapsed_when_first_error_ran.append([c.elapsed_ms is not None for c in failed_ctxs])
+
+
+r, en, ml = batch_router(hooks=[FirstError()], on_predict_start=remember_failed_start)
+check_raises("router_batch/a failed group propagates", RuntimeError,
+ lambda: r.predict_batch([req("boom"), req("b"), req("c")]))
+check("router_batch/elapsed_ms is set for a failed group before any on_error runs",
+ elapsed_when_first_error_ran, [[True, True, True]])
+
+r, en, ml = batch_router(on_predict_end=lambda c: setattr(c, "results", [None]) if c.states == ["x"] else None)
+try:
+ got = r.predict_batch([req("x"), req("y")])
+ got = (got[0], got[1]["answers"]["seen"])
+except Exception as e: # noqa: BLE001
+ got = repr(e)
+check("router_batch/a None result an end hook leaves is returned, as predict returns it", got, (None, "y"))
+
+
+class AliasPin:
+ """An on_route hook pinning a request with an alias, as a plain dict."""
+
+ def on_route(self, ctx):
+ if ctx.states == ["alias"]:
+ ctx.decision = {**ctx.decision, "model": "ml"}
+
+
+alias_models = []
+r, en, ml = batch_router(hooks=[AliasPin()], on_predict_start=lambda c: alias_models.append(c.model))
+r.predict_batch([req("alias"), req("full", "multilingual")])
+check("router_batch/an aliased pin shares its checkpoint's forward pass", [c[0] for c in ml.calls],
+ [["alias", "full"]])
+check("router_batch/an aliased pin reports the resolved checkpoint", alias_models,
+ ["multilingual", "multilingual"])
+
r, en, ml = batch_router(on_predict_start=fail_on_b, hooks_raise=False)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
diff --git a/tests/test_router_batch.py b/tests/test_router_batch.py
index 8677a9a6..20664121 100644
--- a/tests/test_router_batch.py
+++ b/tests/test_router_batch.py
@@ -111,6 +111,37 @@ def test_inference_exception_propagates_and_cache_remains_consistent(fake_agent)
assert router.predict("after failure", Q, lang="ar")["answers"]["seen"] == "after failure"
+def test_evicted_checkpoint_is_unreferenced_when_it_is_evicted(monkeypatch):
+ # With max_loaded=1, loading the second group's checkpoint evicts the first. Eviction's
+ # gc.collect() / empty_cache() only give its memory back if predict_batch no longer holds
+ # it at that moment -- directly, or through the previous group's contexts.
+ import gc
+ import weakref
+
+ import laya.agent
+
+ refs = {}
+ evicted_alive = []
+
+ class Agent:
+ def __init__(self, repo, *, device, token, subfolder):
+ self.checkpoint = subfolder or "english"
+ refs[self.checkpoint] = weakref.ref(self)
+
+ def predict_batch(self, states, questions, batch_size=None):
+ return [{"answers": {"seen": state}, "usage": {}} for state in states]
+
+ class CheckFreed:
+ def on_evict(self, ctx):
+ gc.collect()
+ evicted_alive.append((ctx.model, refs[ctx.model]() is not None))
+
+ monkeypatch.setattr(laya.agent, "Agent", Agent)
+ router = Router(max_loaded=1, hooks=[CheckFreed()])
+ router.predict_batch([request("english text"), request("مرحبا")])
+ assert evicted_alive == [("english", False)]
+
+
def test_warm_cache_and_repeated_batches(fake_agent):
built, _ = fake_agent
router = Router(max_loaded=2)