Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(hooks): opt-in prediction hooks to observe or shape every decision (#270)

Emiliano G.O.b668a1b6bcd7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 52ae7c16..3c996a08 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -59,6 +59,8 @@ jobs:
python tests/test_router.py
python tests/test_criteria.py
python tests/test_batch.py
+ python tests/test_hooks.py
+ python tests/test_hooks_api.py
python tests/test_download.py
python tests/test_shortlist.py
python tests/test_decision_model.py
diff --git a/README.md b/README.md
index 286815eb..a0869a4a 100644
--- a/README.md
+++ b/README.md
@@ -444,6 +444,29 @@ else:
---
+## Prediction Hooks
+
+Hooks observe or shape every decision without forking: audit logging, PII redaction before
+inference, caching, metrics, confidence gating, routing overrides, and forwarding to an
+external service. They are opt-in, and unset hooks are a no-op.
+
+```python
+import laya
+
+def log(ctx):
+ print(ctx.model, ctx.results[0]["answers"], ctx.elapsed_ms)
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=log)
+agent.system_one("I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}})
+```
+
+A hook is a plain callable, or an object implementing any of `on_predict_start`,
+`on_predict_end`, `on_route`, `on_load`, `on_evict`, `on_error`. A start hook can rewrite the
+state/questions or `ctx.skip(...)` a cached answer; an end hook can rewrite the result. See
+[**`docs/hooks/`**](docs/hooks/index.md) and [`examples/hooks/`](examples/hooks/).
+
+---
+
## Built-in Workflow Presets
Laya provides pre-tuned question schemas for immediate production use:
diff --git a/docs/hooks/api.md b/docs/hooks/api.md
new file mode 100644
index 00000000..ac5b3a84
--- /dev/null
+++ b/docs/hooks/api.md
@@ -0,0 +1,283 @@
+# API reference
+
+Everything on this page is importable from `laya` (the common names) or `laya.hooks` (the
+whole surface).
+
+```python
+from laya import PredictContext, PredictHook, Hook
+from laya.hooks import HOOK_EVENTS, normalise_hooks, dispatch, aggregate_usage
+```
+
+- [PredictContext](#predictcontext)
+- [Hook protocol](#hook-protocol)
+- [Convenience types](#convenience-types)
+- [Configuration surface](#configuration-surface)
+- [Event payloads](#event-payloads)
+- [Validation](#validation)
+- [Advanced helpers](#advanced-helpers)
+
+## PredictContext
+
+A `PredictContext` is created once per public call and passed to every hook of that call. It is
+**mutable**: hooks may rewrite `states`, `questions` and `results`, and `on_route` may rewrite
+`decision`. It uses identity equality (`eq=False`), so a context is hashable and two contexts are
+never equal.
+
+```python
+@dataclass(eq=False)
+class PredictContext:
+ states: List[Any]
+ questions: Dict[str, Any]
+ run_id: str = <uuid4 hex>
+ results: Optional[List[Dict[str, Any]]] = None
+ decision: Optional[Dict[str, Any]] = None
+ model: Optional[str] = None
+ agent: Any = None
+ router: Any = None
+ max_len: Optional[int] = None
+ head_max_len: Optional[int] = None
+ usage: Optional[Dict[str, int]] = None
+ started_at: float = <perf_counter()>
+ elapsed_ms: Optional[float] = None
+ error: Optional[BaseException] = None
+```
+
+| field | type | set when | mutable | meaning |
+|---|---|---|---|---|
+| `states` | `list` | always | yes (start) | the states for this call. `system_one`/`Router.predict` pass one; `predict_batch` passes many. A start hook may replace the list. |
+| `questions` | `dict` | always | yes (start) | the questions. A start hook may replace the dict. |
+| `run_id` | `str` | always | no | a unique id shared by every hook of this call. Use it to correlate events and spans. |
+| `results` | `list \| None` | end (and on a skip) | yes (end) | per-state result dicts, each shaped like `system_one`'s return. `None` until inference finishes. |
+| `decision` | `dict \| None` | Router only | yes (route) | the `RouteDecision` (a `dict`) that selected the checkpoint. |
+| `model` | `str \| None` | always | no | the checkpoint id: `Agent.model_id` for an Agent, the resolved alias (for example `"english"`) for a Router. |
+| `agent` | `Agent \| ONNXAgent \| None` | predict events | no | the runtime answering the call. |
+| `router` | `Router \| None` | Router events | no | the Router, when one is in play. |
+| `max_len` | `int \| None` | always | yes (start) | per-call token budget for the encoder. `None` uses the agent config. |
+| `head_max_len` | `int \| None` | always | yes (start) | per-call token budget for the question head. `None` uses the agent config. |
+| `usage` | `dict \| None` | end | yes (end) | `{"input_tokens", "output_tokens"}`, summed over the states of the call. |
+| `started_at` | `float` | always | no | `time.perf_counter()` when the call began. |
+| `elapsed_ms` | `float \| None` | end | no | wall time for the whole call, milliseconds. |
+| `error` | `BaseException \| None` | failure path | no | the exception, set before `on_error` and `on_predict_end`. |
+
+### `PredictContext.skip(results)`
+
+Short-circuits inference. Called from `on_predict_start`, it sets `ctx.results` so the forward
+pass is skipped; `on_predict_end` still runs and the supplied results are returned.
+
+```python
+def cache_read(ctx):
+ hit = CACHE.get(key(ctx.states[0], ctx.questions))
+ if hit is not None:
+ ctx.skip([hit]) # list of per-state results, same shape as predict_batch's return
+```
+
+On the `Router`, a skipped payload gets a `routing` key added (without overwriting one it
+already has), so `Router.predict` keeps its documented return shape.
+
+## Hook protocol
+
+`Hook` is a `typing.Protocol`. Implement any subset of the methods; the rest are skipped.
+
+```python
+class Hook(Protocol):
+ def on_predict_start(self, ctx: PredictContext) -> None: ...
+ def on_predict_end(self, ctx: PredictContext) -> None: ...
+ def on_route(self, ctx: PredictContext) -> None: ...
+ def on_load(self, ctx: PredictContext) -> None: ...
+ def on_evict(self, ctx: PredictContext) -> None: ...
+ def on_error(self, ctx: PredictContext) -> None: ...
+```
+
+| event | where | runs | can change |
+|---|---|---|---|
+| `on_predict_start` | Agent, Router | before tokenization/forward | `states`, `questions`, or `skip()` |
+| `on_predict_end` | Agent, Router | after results exist, success or failure | `results` |
+| `on_route` | Router | after detection, before loading | `decision` |
+| `on_load` | Router | after a checkpoint is built | nothing (observe) |
+| `on_evict` | Router | after a checkpoint is freed | nothing (observe) |
+| `on_error` | Agent, Router | when a predict call fails | nothing (observe) |
+
+A hook is free to define extra attributes and methods; only the six event names are consulted.
+If a hook defines one of the six as a non-callable, configuration fails fast (see
+[Validation](#validation)).
+
+## Convenience types
+
+```python
+PredictHook = Callable[[PredictContext], None]
+```
+
+`PredictHook` is the type of a plain callable used with `on_predict_start=` / `on_predict_end=`.
+Pass a single callable or a sequence of them; each is wrapped into a minimal hook.
+
+## Runtime registration
+
+Every runtime mixes in `HookRegistry`, so hooks can be added, removed or scoped after
+construction. Mutation is thread-safe; a call reads a snapshot of the list, so adding or
+removing a hook never disturbs a call in flight.
+
+```python
+agent.add_hook(tracer) # one hook or a sequence; returns self for chaining
+agent.remove_hook(tracer) # by identity; True if it was installed
+
+with agent.hooks_installed(debug): # installed for the block, removed on exit
+ agent.system_one(state, questions)
+```
+
+`add_hook` accepts the same objects as `hooks=` (not plain callables). `hooks_installed` takes
+any number of hook objects or sequences and restores the previous list on exit, including when
+the block raises.
+
+## Configuration surface
+
+Every entry point accepts the same five hook parameters. `hooks` takes an object or a sequence
+of objects; `on_predict_start` / `on_predict_end` take a callable or a sequence.
+
+| parameter | type | default | meaning |
+|---|---|---|---|
+| `hooks` | `Hook \| Sequence[Hook] \| None` | `None` | lifecycle hooks (any of the six events). |
+| `on_predict_start` | `PredictHook \| Sequence[PredictHook] \| None` | `None` | convenience callables for one event. |
+| `on_predict_end` | `PredictHook \| Sequence[PredictHook] \| None` | `None` | convenience callables for one event. |
+| `hooks_raise` | `bool` | `True` | `True`: a hook exception propagates. `False`: warn and continue. |
+| `hooks_concurrent` | `bool` | `True` | `False`: dispatch hooks under a lock, one at a time. |
+
+### Agent
+
+```python
+Agent(
+ model_id_or_path="convaiinnovations/laya",
+ device=None, token=None, subfolder=None, fast=False, compile=False,
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise=True, hooks_concurrent=True,
+)
+
+load(..., hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise=True, hooks_concurrent=True)
+
+agent.predict_batch(states, questions, batch_size=None,
+ hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
+ max_len=None, head_max_len=None)
+
+agent.system_one(state, questions,
+ hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
+ max_len=None, head_max_len=None)
+
+agent.predict(...) # alias of system_one
+```
+
+- `hooks_raise` on a per-call method defaults to `None`, meaning "use the instance value".
+- `hooks_concurrent` is instance-level only.
+
+### Router
+
+```python
+Router(
+ models=None, device=None, token=None, max_loaded=2, default="english",
+ auto_task_detection=False, standalone_repos=False, preload=False, lang_guess=None,
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise=True, hooks_concurrent=True,
+)
+
+router.route(state, questions=None, model=None, task=None, lang=None, lang_guess=None,
+ hooks=None, hooks_raise=None)
+
+router.predict(state, questions, model=None, task=None, lang=None, lang_guess=None,
+ hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
+ max_len=None, head_max_len=None)
+
+router.system_one(...) # alias of predict
+router.load(name) # builds on first use; fires on_load
+router.preload(names=None) # builds several; fires on_load per build
+router.unload(name=None) # frees one or all; fires on_evict
+router.attach(name, agent) # registers an existing agent; does not fire on_load
+router.loaded # list of resident checkpoint names
+```
+
+- Per-call `hooks=` on `route` and `predict` apply to the whole call, including `on_route`.
+- `route()` is public: calling it dispatches `on_route` with the installed hooks plus any
+ per-call `hooks`.
+
+### ONNXAgent
+
+```python
+ONNXAgent(model_id_or_path, onnx_path="laya.onnx", subfolder=None,
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise=True, hooks_concurrent=True)
+
+onnx_agent.system_one(state, questions,
+ hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
+ max_len=None, head_max_len=None)
+
+onnx_agent.predict(...) # alias of system_one
+```
+
+`ONNXAgent` has no Router, so it exposes only the predict-level events.
+
+## Event payloads
+
+Which fields are populated, per event and runtime:
+
+| event | runtime | `states` | `questions` | `decision` | `model` | `agent` | `router` | `results` | `usage` | `elapsed_ms` | `error` |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| `on_predict_start` | Agent | ✓ | ✓ | – | ✓ | ✓ | – | – | – | – | – |
+| `on_predict_start` | Router | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | – | – | – | – |
+| `on_predict_end` | Agent | ✓ | ✓ | – | ✓ | ✓ | – | ✓ | on success | ✓ | on failure |
+| `on_predict_end` | Router | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | on success | ✓ | on failure |
+| `on_error` | both | ✓ | ✓ | ✓ (Router) | ✓ | ✓ | ✓ (Router) | – | – | – | ✓ |
+| `on_route` | Router | ✓ | ✓ | ✓ | – | – | ✓ | – | – | – | – |
+| `on_load` | Router | `[]` | `{}` | – | ✓ | ✓ | ✓ | – | – | – | – |
+| `on_evict` | Router | `[]` | `{}` | – | ✓ | – | ✓ | – | – | – | – |
+
+Timing details:
+
+- `on_predict_end` sees `results` on the success path. On the failure path `results` is `None`
+ unless a start hook had set them via `skip()`, and `usage` is therefore `None` too (it is
+ derived from `results`); `elapsed_ms` is always set.
+- `on_error` runs before the `finally` block that computes `elapsed_ms` and `usage`, so both are
+ `None` there. Read timing and usage from `on_predict_end` instead.
+- `run_id` is always populated.
+
+## Validation
+
+Configuration is validated when hooks are normalised, which happens at construction for
+installed hooks and at call time for per-call hooks. These raise `TypeError`:
+
+| case | message |
+|---|---|
+| a class is passed instead of an instance | `hooks entries must be instances, not classes; ...` |
+| an object implements none of the six events | `hooks entries must implement at least one of ...` |
+| an event attribute is not callable | `hooks entry X.on_predict_start must be callable, got int` |
+| `on_predict_start=` / `on_predict_end=` is not callable | `on_predict_start must be callable, got int` |
+
+`hooks=` does not accept plain callables, because a bare callable does not say *which* event it
+is for. Use `on_predict_start=` / `on_predict_end=` for those.
+
+## Advanced helpers
+
+These are used internally and are stable, but most users do not need them.
+
+```python
+HOOK_EVENTS # tuple of the six event names, in dispatch order
+normalise_hooks(hooks=None, on_predict_start=None, on_predict_end=None) -> list
+dispatch(hooks, event, ctx, *, raise_errors=True, lock=None) -> None
+aggregate_usage(results) -> {"input_tokens": int, "output_tokens": int}
+```
+
+`normalise_hooks` flattens a `hooks` object/sequence and the two callables into one ordered list.
+`dispatch` calls `event` on every hook that implements it, applying the raise policy and lock.
+`aggregate_usage` sums per-state usage blocks.
+
+```python
+from laya.hooks import normalise_hooks, dispatch, PredictContext
+
+hooks = normalise_hooks(on_predict_start=[log, redact])
+ctx = PredictContext(states=["..."], questions={...})
+dispatch(hooks, "on_predict_start", ctx)
+```
+
+## See also
+
+- [Lifecycle](lifecycle.md): when each event runs, with flowcharts.
+- [Errors](errors.md): the failure matrix and chaining rules.
+- [Patterns and anti-patterns](patterns.md): how to structure hooks well.
+- [Examples](examples.md): recipes for every use case.
diff --git a/docs/hooks/errors.md b/docs/hooks/errors.md
new file mode 100644
index 00000000..90e40005
--- /dev/null
+++ b/docs/hooks/errors.md
@@ -0,0 +1,142 @@
+# Error handling
+
+Hooks run inside the request they observe, so how their failures are handled matters. This page
+is the exact policy.
+
+- [The two policies](#the-two-policies)
+- [What runs when something fails](#what-runs-when-something-fails)
+- [Exception chaining](#exception-chaining)
+- [BaseException](#baseexception)
+- [Configuration errors](#configuration-errors)
+- [Warnings](#warnings)
+- [Choosing a policy](#choosing-a-policy)
+
+## The two policies
+
+`hooks_raise` controls what happens when a hook raises.
+
+| `hooks_raise` | behaviour |
+|---|---|
+| `True` (default) | the hook exception propagates out of the call. |
+| `False` | the hook is skipped with a `RuntimeWarning` and the call continues. |
+
+It is set per instance and can be overridden per call (`hooks_raise=` on `predict_batch`,
+`system_one`, `Router.route`, `Router.predict`). Per-call `None` means "use the instance value".
+
+```python
+# strict: a broken audit hook fails the request
+laya.load("convaiinnovations/laya", on_predict_end=audit, hooks_raise=True)
+
+# lenient: telemetry must never take down a served request
+laya.load("convaiinnovations/laya", on_predict_end=metrics, hooks_raise=False)
+```
+
+`dispatch` catches `Exception`. Anything that is not an `Exception` (see
+[BaseException](#baseexception)) is never swallowed, even with `hooks_raise=False`.
+
+## What runs when something fails
+
+The predict lifecycle is wrapped in `try / except / finally`, so cleanup hooks run on failure.
+
+```
+try:
+ on_predict_start
+ inference
+except BaseException as exc:
+ ctx.error = exc
+ on_error (best effort; cannot mask exc)
+ raise
+finally:
+ elapsed_ms, usage
+ on_predict_end (best effort; cannot mask exc on the failure path)
+```
+
+Failure matrix, per event and runtime:
+
+| event | runtime | if it raises |
+|---|---|---|
+| `on_predict_start` | Agent / Router | `hooks_raise=True`: `on_error` and `on_predict_end` still run, then the exception propagates. `False`: warn and continue (mutations made before the raise remain). |
+| inference | Agent / Router | `ctx.error` set, `on_error` runs, `on_predict_end` runs, exception propagates. |
+| `on_error` | Agent / Router | never masks the original exception; chained as `__context__`. |
+| `on_predict_end` (success path) | Agent / Router | `hooks_raise=True`: propagates (the result is computed but the call fails). `False`: warn. |
+| `on_predict_end` (failure path) | Agent / Router | never masks the original exception; chained as `__context__`. |
+| `on_route` | Router | propagates directly; there is no predict context yet. |
+| `on_load` | Router | propagates directly; the checkpoint stays built and resident. |
+| `on_evict` | Router | propagates directly; the checkpoint is already freed. |
+
+Consequences worth knowing:
+
+- A failing `on_load` leaves the model cached, so the next `load` returns it without firing
+ `on_load` again.
+- A failing `on_predict_end` on the success path means the caller gets an exception instead of a
+ result, even though inference succeeded. Use `hooks_raise=False` for end hooks that are pure
+ side effects.
+
+## Exception chaining
+
+When a hook fails while another exception is already propagating, the original exception is
+re-raised and the hook's exception is attached as `__context__`. The root cause is never lost.
+
+```python
+class BadTelemetry:
+ def on_error(self, ctx):
+ raise RuntimeError("telemetry down")
+
+try:
+ agent.system_one(state, questions, hooks=[BadTelemetry()])
+except RuntimeError as exc:
+ assert exc.__context__ is not None # the telemetry failure
+```
+
+The same rule applies to a failing `on_predict_end` on the failure path.
+
+## BaseException
+
+`dispatch` catches `Exception`, not `BaseException`, so `KeyboardInterrupt` and `SystemExit`
+always propagate. They still trigger the `except BaseException` branch of the predict lifecycle,
+which means `on_error` and `on_predict_end` run before the process unwinds. Keep those hooks
+fast and non-blocking if you care about interrupt latency.
+
+## Configuration errors
+
+Bad configuration fails fast with `TypeError`, before any inference:
+
+| case | raised at | example |
+|---|---|---|
+| class instead of instance | construction | `hooks=[MyHook]` |
+| no lifecycle method | construction | `hooks=[object()]` |
+| non-callable event | construction | `on_predict_start = 5` |
+| non-callable convenience hook | construction | `on_predict_start=123` |
+| plain callable in `hooks=` | construction | `hooks=[lambda ctx: None]` |
+
+Per-call hooks are validated when the call is made, so a bad per-call hook raises `TypeError`
+from `predict`/`system_one` rather than at construction.
+
+## Warnings
+
+With `hooks_raise=False`, each failing hook emits one `RuntimeWarning` naming the hook type and
+event:
+
+```
+laya: hook Metrics.on_predict_end failed: connection reset
+```
+
+The warning is emitted once per failure, not once per hook definition, so a flaky hook under
+load can be noisy. Aggregate or rate-limit inside the hook if that matters.
+
+## Choosing a policy
+
+| hook kind | recommended | why |
+|---|---|---|
+| policy / guardrail / redaction | `hooks_raise=True` | a policy that silently fails is a security hole. |
+| audit / logging | `hooks_raise=True` in tests, often `False` in production | losing an audit record should be loud, but not necessarily fatal. |
+| metrics / tracing | `hooks_raise=False` | observability must not fail the request. |
+| cache read/write | `hooks_raise=True` | a broken cache should surface, not silently miss. |
+
+You can mix: install a guardrail with the instance default and give a telemetry hook its own
+try/except, or use a separate `hooks_raise` per call.
+
+## See also
+
+- [Lifecycle](lifecycle.md): the `try/except/finally` shape in context.
+- [Patterns and anti-patterns](patterns.md): common mistakes with error handling.
diff --git a/docs/hooks/examples.md b/docs/hooks/examples.md
new file mode 100644
index 00000000..242a44e5
--- /dev/null
+++ b/docs/hooks/examples.md
@@ -0,0 +1,340 @@
+# Examples
+
+Copy-paste recipes. Every snippet is self-contained apart from the helpers it names
+(`ship`, `CACHE`, and so on), which you supply.
+
+- [Quick start](#quick-start)
+- [Audit](#audit)
+- [Redact PII](#redact-pii)
+- [Cache](#cache)
+- [Metrics](#metrics)
+- [Guardrail](#guardrail)
+- [Confidence gate](#confidence-gate)
+- [Routing pin](#routing-pin)
+- [Lifecycle](#lifecycle)
+- [Composition](#composition)
+- [Per-call hooks](#per-call-hooks)
+- [Batch](#batch)
+- [HTTP server](#http-server)
+- [ONNXAgent](#onnxagent)
+- [Runtime registration](#runtime-registration)
+- [Token budget](#token-budget)
+- [Testing hooks](#testing-hooks)
+
+## Quick start
+
+```python
+import laya
+
+def log(ctx):
+ print(ctx.model, ctx.results[0]["answers"])
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=log)
+agent.system_one("I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}})
+```
+
+## Audit
+
+The browser-use use case: capture every decision and ship it to an external service.
+
+```python
+import json, sys
+import laya
+
+def audit(ctx):
+ record = {
+ "run_id": ctx.run_id,
+ "model": ctx.model,
+ "routing": ctx.results[0].get("routing") if ctx.results else None,
+ "answers": ctx.results[0]["answers"] if ctx.results else None,
+ "usage": ctx.usage,
+ "elapsed_ms": round(ctx.elapsed_ms or 0.0, 3),
+ }
+ print(json.dumps(record), file=sys.stderr)
+ # ship_to_service(record)
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=audit)
+```
+
+A full runnable version is in [`examples/hooks/audit.py`](../../examples/hooks/audit.py).
+
+## Redact PII
+
+```python
+import re
+import laya
+
+EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
+PHONE = re.compile(r"\+?\d[\d ()-]{7,}\d")
+
+def scrub(value):
+ if isinstance(value, str):
+ return PHONE.sub("[phone]", EMAIL.sub("[email]", value))
+ if isinstance(value, dict):
+ return {k: scrub(v) for k, v in value.items()}
+ if isinstance(value, list):
+ return [scrub(v) for v in value]
+ return value
+
+def redact(ctx):
+ ctx.states = [scrub(s) for s in ctx.states]
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=redact)
+```
+
+See [`examples/hooks/redact.py`](../../examples/hooks/redact.py).
+
+## Cache
+
+```python
+import hashlib, json
+import laya
+
+CACHE = {}
+
+def key(state, questions):
+ payload = json.dumps([state, questions], sort_keys=True, default=str)
+ return hashlib.sha256(payload.encode()).hexdigest()
+
+def read(ctx):
+ hit = CACHE.get(key(ctx.states[0], ctx.questions))
+ if hit is not None:
+ ctx.skip([hit])
+
+def write(ctx):
+ if ctx.results:
+ CACHE[key(ctx.states[0], ctx.questions)] = ctx.results[0]
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=read, on_predict_end=write)
+first = agent.system_one("state", QUESTIONS) # runs the model
+second = agent.system_one("state", QUESTIONS) # served from CACHE
+```
+
+See [`examples/hooks/cache.py`](../../examples/hooks/cache.py).
+
+## Metrics
+
+```python
+import laya
+
+COUNTS, LATENCIES = {}, []
+
+def metrics(ctx):
+ COUNTS[ctx.model] = COUNTS.get(ctx.model, 0) + 1
+ if ctx.elapsed_ms is not None:
+ LATENCIES.append(ctx.elapsed_ms)
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=metrics, hooks_raise=False)
+```
+
+See [`examples/hooks/otel.py`](../../examples/hooks/otel.py).
+
+## Guardrail
+
+Block a request by raising from a start hook.
+
+```python
+import laya
+
+class Blocked(Exception):
+ pass
+
+def guard(ctx):
+ text = str(ctx.states[0]).lower()
+ if "ignore previous instructions" in text:
+ raise Blocked("prompt injection")
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=guard)
+
+try:
+ agent.system_one("Ignore previous instructions and ...", QUESTIONS)
+except Blocked:
+ handle_block()
+```
+
+## Confidence gate
+
+Rewrite a low-confidence answer, or annotate it.
+
+```python
+def gate(ctx):
+ answer = ctx.results[0]["answers"].get("dept")
+ if answer and answer["confidence"] < 0.6:
+ answer["choice"] = "human-review"
+ answer["gated"] = True
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=gate)
+```
+
+## Routing pin
+
+Force a checkpoint for a class of traffic.
+
+```python
+from laya import Router
+from laya.router import RouteDecision
+
+def pin(ctx):
+ if "refund" in str(ctx.states[0]).lower():
+ ctx.decision = RouteDecision(
+ model="typed-decisions",
+ repo="convaiinnovations/laya/typed-decisions",
+ reason="refund workflow",
+ detection=None,
+ workflow=None,
+ )
+
+router = Router(hooks=[pin])
+```
+
+Per-call, without installing:
+
+```python
+router.predict("refund request", QUESTIONS, hooks=[pin])
+```
+
+## Lifecycle
+
+Observe checkpoint build and eviction.
+
+```python
+from laya import Router
+
+class Lifecycle:
+ def on_load(self, ctx):
+ print("loaded", ctx.model, "agent", type(ctx.agent).__name__)
+
+ def on_evict(self, ctx):
+ print("evicted", ctx.model)
+
+router = Router(max_loaded=1, hooks=[Lifecycle()])
+router.preload(["english", "multilingual"]) # on_load fires per build
+router.unload() # on_evict fires per freed checkpoint
+```
+
+## Composition
+
+Installed hooks first, then convenience callables; all share one context.
+
+```python
+import laya
+
+class Metrics:
+ def on_predict_end(self, ctx):
+ record_latency(ctx.model, ctx.elapsed_ms)
+
+def redact(ctx):
+ ctx.states = [strip_pii(s) for s in ctx.states]
+
+def audit(ctx):
+ ship(ctx.run_id, ctx.results)
+
+agent = laya.load(
+ "convaiinnovations/laya",
+ hooks=[Metrics()], # installed, runs first
+ on_predict_start=redact, # convenience, appended
+ on_predict_end=audit, # convenience, appended
+ hooks_raise=True,
+)
+```
+
+## Per-call hooks
+
+Override or extend hooks for a single call.
+
+```python
+agent.system_one(
+ state,
+ questions,
+ on_predict_end=lambda ctx: debug_dump(ctx),
+ hooks_raise=False,
+)
+
+router.predict(
+ state,
+ questions,
+ hooks=[pin], # applies to on_route too
+ on_predict_end=audit,
+)
+```
+
+## Batch
+
+Hooks fire once per `predict_batch` call, with `ctx.states` holding every state.
+
+```python
+def audit_batch(ctx):
+ for state, result in zip(ctx.states, ctx.results):
+ ship_one(ctx.run_id, state, result)
+
+results = agent.predict_batch([state_a, state_b, state_c], questions, on_predict_end=audit_batch)
+```
+
+## HTTP server
+
+Router hooks fire for `laya.serve` automatically, because the server calls `Router.predict`.
+
+```python
+from laya import Router
+from laya.serve import create_app
+
+router = Router(hooks=[Metrics()], on_predict_end=audit, hooks_raise=False)
+app = create_app(router=router)
+```
+
+## ONNXAgent
+
+`ONNXAgent` exposes the predict-level events only.
+
+```python
+from laya.onnx_agent import ONNXAgent
+
+agent = ONNXAgent("convaiinnovations/laya", onnx_path="laya.onnx", on_predict_end=audit)
+agent.system_one(state, questions)
+```
+
+## Runtime registration
+
+Attach, detach or scope hooks after construction.
+
+```python
+agent.add_hook(Metrics()) # attach at runtime
+agent.remove_hook(Metrics()) # by identity
+
+with agent.hooks_installed(DebugDump()):
+ agent.system_one(state, questions) # DebugDump only here
+```
+
+## Token budget
+
+Shape the token budget for one call, from a hook or a per-call argument.
+
+```python
+def widen(ctx):
+ k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})
+ if k >= 50:
+ ctx.head_max_len = 16 + 4 * k
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=widen)
+
+# or per call
+agent.system_one(state, questions, head_max_len=324, max_len=1024)
+```
+
+## Testing hooks
+
+Assert what a hook saw without a model: drive `predict_batch` with the encode/forward/decode
+helpers stubbed, as [`tests/test_hooks.py`](../../tests/test_hooks.py) does.
+
+```python
+seen = []
+agent.predict_batch(["s0"], questions, on_predict_end=lambda ctx: seen.append(ctx.results))
+assert len(seen) == 1
+```
+
+The API surface is pinned by [`tests/test_hooks_api.py`](../../tests/test_hooks_api.py).
+
+## See also
+
+- [Tracing](tracing.md): `run_id`, spans, OpenTelemetry.
+- [Patterns and anti-patterns](patterns.md): the reasoning behind these recipes.
diff --git a/docs/hooks/index.md b/docs/hooks/index.md
new file mode 100644
index 00000000..1f5f75fe
--- /dev/null
+++ b/docs/hooks/index.md
@@ -0,0 +1,127 @@
+# Prediction Hooks
+
+Hooks let you observe or shape every decision Laya makes, without forking it.
+
+They are the extension seam for the things every real deployment needs: audit logging, PII
+redaction before inference, caching, metrics, confidence gating, routing overrides, and
+forwarding a decision to an external service. They are **opt-in**: with no hooks configured the
+behaviour of `Agent`, `Router` and `ONNXAgent` is unchanged.
+
+This folder is the full reference. Start here, then dive into the page you need:
+
+| page | what is in it |
+|---|---|
+| [API reference](api.md) | every class, field, parameter and default |
+| [Lifecycle](lifecycle.md) | exactly when each hook runs, with flowcharts |
+| [Errors](errors.md) | `hooks_raise`, `on_error`, exception chaining, failure matrix |
+| [Patterns and anti-patterns](patterns.md) | what to do, what to avoid, and why |
+| [Examples](examples.md) | copy-paste recipes for every use case |
+| [Tracing](tracing.md) | `run_id`, span correlation, OpenTelemetry |
+
+## Quick start
+
+```python
+import laya
+
+def log(ctx):
+ print(ctx.model, ctx.results[0]["answers"], ctx.elapsed_ms)
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=log)
+agent.system_one("I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}})
+```
+
+An object can implement any subset of the lifecycle events:
+
+```python
+class Audit:
+ def on_predict_start(self, ctx):
+ print("start", ctx.run_id)
+
+ def on_predict_end(self, ctx):
+ print("end", ctx.run_id, ctx.usage, ctx.elapsed_ms)
+
+ def on_error(self, ctx):
+ print("failed", ctx.run_id, ctx.error)
+
+laya.load("convaiinnovations/laya", hooks=[Audit()])
+```
+
+Hooks can also be added later or scoped to a block:
+
+```python
+agent.add_hook(tracer) # attach at runtime
+with agent.hooks_installed(debug): # installed for the block, removed on exit
+ agent.system_one(state, questions)
+```
+
+See [runtime registration](api.md#runtime-registration).
+
+## The mental model
+
+There are three ideas.
+
+1. **A hook is a callable or an object.** A plain function is convenient for one event; an
+ object is convenient for several. Both are passed to `hooks=` / `on_predict_start=` /
+ `on_predict_end=`.
+
+2. **Every hook of one call shares one mutable `PredictContext`.** It carries the states,
+ questions, results, routing decision, model name, usage, timing and any error. Because it is
+ mutable, a hook can *shape* the call, not only watch it: redact the state, rewrite the
+ questions, replace the result, or skip inference with a cached answer.
+
+3. **There are two scopes.** `Agent` hooks wrap a forward pass; `Router` hooks wrap routing plus
+ inference and can also see model lifecycle (`on_route`, `on_load`, `on_evict`). This mirrors
+ the "run hooks" vs "agent hooks" split in other agent frameworks.
+
+```
+ Router.predict(state, questions)
+ ┌──────────────────────────────────────────────────────────────────────────┐
+ │ route() │
+ │ ├─ detect language / workflow │
+ │ └─ on_route ctx.decision (a hook may replace it) │
+ │ │
+ │ load(decision.model) │
+ │ ├─ build checkpoint on first use ──► on_load ctx.model, ctx.agent │
+ │ └─ evict LRU checkpoint ───────────► on_evict ctx.model │
+ │ │
+ │ on_predict_start ctx.states, ctx.questions, ctx.decision │
+ │ │ │
+ │ ├── ctx.skip(results)? ──► skip the forward pass │
+ │ │ │
+ │ └── Agent.system_one(...) ──► Agent-level hooks run here │
+ │ on_predict_start ─► forward ─► on_predict_end │
+ │ │
+ │ result["routing"] = decision │
+ │ on_predict_end ctx.results, ctx.usage, ctx.elapsed_ms │
+ └──────────────────────────────────────────────────────────────────────────┘
+ any failure on the way ──► on_error, then on_predict_end
+```
+
+## Scope at a glance
+
+| | `Agent` / `ONNXAgent` | `Router` |
+|---|---|---|
+| `on_predict_start` | yes | yes |
+| `on_predict_end` | yes | yes |
+| `on_error` | yes | yes |
+| `on_route` | no | yes |
+| `on_load` | no | yes |
+| `on_evict` | no | yes |
+
+`laya.serve` and the MCP server call `Router.predict`, so Router hooks fire for them
+automatically. `Agent` hooks fire whenever the Router runs an attached or built agent.
+
+## Compatibility
+
+- No hooks configured means no behavioural change. The unset path is regression-tested.
+- All hook parameters are keyword arguments with defaults, so existing calls keep working.
+- `laya/hooks.py` is pure Python: `import laya` does not pull in torch because of it.
+- Hooks are synchronous. Keep them fast and non-blocking; see
+ [errors](errors.md) and [patterns](patterns.md) for the consequences on `laya.serve`.
+
+## See also
+
+- [`examples/hooks/`](../../examples/hooks/): runnable audit, redact, cache and metrics hooks.
+- [`tests/test_hooks.py`](../../tests/test_hooks.py): the behaviour spec.
+- [`tests/test_hooks_api.py`](../../tests/test_hooks_api.py): the API-stability guard.
+- [`laya/hooks.py`](../../laya/hooks.py): the implementation.
diff --git a/docs/hooks/lifecycle.md b/docs/hooks/lifecycle.md
new file mode 100644
index 00000000..b1745f70
--- /dev/null
+++ b/docs/hooks/lifecycle.md
@@ -0,0 +1,200 @@
+# Lifecycle
+
+This page is the precise order of events for each entry point. If you only read one diagram,
+read the [Router](#routerpredict) one; it is the superset.
+
+- [Agent.predict_batch](#agentpredict_batch)
+- [Agent.system_one / predict](#agentsystem_one-predict)
+- [Router.predict](#routerpredict)
+- [Model lifecycle](#model-lifecycle)
+- [Caching with skip](#caching-with-skip)
+- [Empty inputs](#empty-inputs)
+- [Ordering rules](#ordering-rules)
+- [Concurrency](#concurrency)
+
+## Agent.predict_batch
+
+`predict_batch` is the single implementation; `system_one` and `predict` call it with one state.
+
+```
+predict_batch(states, questions, batch_size=..., hooks=..., ...)
+ │
+ ├─ active = installed hooks + per-call hooks (installed first)
+ ├─ ctx = PredictContext(states, questions, model=self.model_id, agent=self)
+ │
+ ├─ try:
+ │ │
+ │ ├─ on_predict_start ─────────────────────────────┐
+ │ │ a hook may: │
+ │ │ • rewrite ctx.states / ctx.questions │
+ │ │ • set ctx.max_len / ctx.head_max_len │
+ │ │ • ctx.skip(results) ─────────────┐ │
+ │ │ • raise (aborts; see errors) │ │
+ │ │ │ │
+ │ ├─ if ctx.results is not None: ◄─────────┘ │ cache hit
+ │ │ skip tokenization and forward │
+ │ ├─ else: │
+ │ │ validate states is a list │
+ │ │ for each batch chunk: │
+ │ │ _encode_state ─► collate ─► _forward │
+ │ │ _decode_answers │
+ │ │ ctx.results = [...] │
+ │ │ │
+ │ └─ (any failure here) ──► except BaseException: │
+ │ ctx.error = exc │
+ │ on_error │
+ │ re-raise │
+ │ │
+ │ finally: │
+ │ ctx.elapsed_ms = now - ctx.started_at │
+ │ if ctx.results: ctx.usage = aggregate_usage(...)│
+ │ on_predict_end ─────────────────────────────────┘
+ │
+ └─ return ctx.results
+```
+
+The same `ctx` object flows through start, error and end, so `run_id` correlates them and an end
+hook can read `ctx.error`.
+
+## Agent.system_one / predict
+
+```
+system_one(state, questions, hooks=..., ...)
+ └─ predict_batch([state], questions, hooks=..., ...)[0]
+```
+
+So `system_one` inherits every hook and the same lifecycle, with `ctx.states == [state]`.
+
+## Router.predict
+
+```
+Router.predict(state, questions, model=..., hooks=..., on_predict_start=..., on_predict_end=...)
+ │
+ ├─ active = installed hooks + per-call hooks
+ │
+ ├─ route(state, questions, ..., hooks=per-call, hooks_raise=...)
+ │ │
+ │ ├─ _route(...) detect script / language / workflow
+ │ ├─ on_route ──► ctx.decision a hook may replace the decision
+ │ └─ return ctx.decision
+ │
+ ├─ load(decision["model"])
+ │ │
+ │ ├─ already resident? ──► return it
+ │ ├─ else build Agent(...) ──► on_load (after the Router lock is released)
+ │ └─ evict LRU checkpoints ──► on_evict (after the Router lock is released)
+ │
+ ├─ ctx = PredictContext(states=[state], questions, decision, model=decision.model,
+ │ agent=agent, router=self)
+ ├─ try:
+ │ ├─ on_predict_start
+ │ ├─ if ctx.results is None:
+ │ │ result = agent.system_one(ctx.states[0], ctx.questions)
+ │ │ └─ the Agent's own hooks run here (start / forward / end)
+ │ │ result["routing"] = decision
+ │ │ ctx.results = [result]
+ │ └─ else:
+ │ for each cached result: result.setdefault("routing", decision)
+ │ └─ (any failure) ──► except: on_error, re-raise
+ │ └─ finally: elapsed_ms, usage, on_predict_end
+ │
+ └─ return ctx.results[0]
+```
+
+Key points:
+
+- `on_route` runs before the model is loaded, so a hook can pin a checkpoint and avoid loading
+ another one.
+- Router-level predict hooks wrap the whole call. They are **not** forwarded into the Agent;
+ an attached Agent with its own hooks runs those too, which is expected.
+- A Router-level `ctx.skip()` still adds `routing`, so the return shape is stable.
+
+## Model lifecycle
+
+`on_load` fires when a checkpoint is built; `on_evict` when one is freed. Both run **after** the
+Router's internal lock is released, so a hook may safely call back into the Router.
+
+```
+load("multilingual")
+ │
+ ├─ [lock]
+ │ build Agent(...) (seconds: download + weights)
+ │ register in _agents / _order
+ │ evict LRU if over max_loaded ──► evicted = ["english"]
+ ├─ [unlock]
+ ├─ on_evict("english")
+ └─ on_load("multilingual")
+
+unload("english")
+ ├─ [lock] remove from _agents / _order
+ ├─ [unlock]
+ └─ on_evict("english")
+```
+
+`attach(name, agent)` registers an existing agent and does **not** fire `on_load`, because no
+checkpoint was built.
+
+## Caching with skip
+
+```
+on_predict_start
+ ├─ cache hit? ctx.skip([cached_result])
+ │ └─ forward pass skipped
+ │ └─ on_predict_end still runs
+ │ └─ Router adds `routing` if missing
+ └─ cache miss? nothing
+ └─ forward pass runs
+ └─ on_predict_end can store the result
+```
+
+See [`examples/hooks/cache.py`](../../examples/hooks/cache.py) for a working cache.
+
+## Empty inputs
+
+Hooks still fire so audit sees every call:
+
+| input | `ctx.results` at end |
+|---|---|
+| `predict_batch([])` | `[]` |
+| `predict_batch(states, {})` (no questions) | one empty-answer payload per state |
+| `system_one(state, {})` | a single empty-answer payload |
+
+No tokenization or forward pass happens in these cases, but `on_predict_start` and
+`on_predict_end` run.
+
+## Ordering rules
+
+1. Installed hooks run before per-call hooks, always.
+2. Within a list, hooks run in list order.
+3. For one event, every hook that implements it runs, in that order, before the next event.
+4. `on_error` runs before `on_predict_end` on the failure path.
+5. `on_evict` runs before `on_load` when a single `load` both evicts and builds.
+
+```
+installed: [A, B] per-call: [C]
+on_predict_start: A, B, C
+on_predict_end: A, B, C
+```
+
+## Concurrency
+
+`Agent` and `Router` are safe to call from many threads. Each call creates its own
+`PredictContext`, so contexts never leak across requests. The only shared state is the hook
+objects themselves, so a hook that is not thread-safe must either guard its own state or be
+installed with `hooks_concurrent=False`.
+
+```
+hooks_concurrent=True (default) hooks_concurrent=False
+ thread 1 ─┐ thread 1 ─┐
+ thread 2 ─┼─ hooks run in parallel thread 2 ─┼─ one hook at a time
+ thread 3 ─┘ thread 3 ─┘ (RLock)
+```
+
+`hooks_concurrent=False` serialises each hook invocation, not whole calls: two calls can still
+interleave between events. It uses a re-entrant lock, so a hook may call back into the same
+`Agent`/`Router` without deadlocking.
+
+## See also
+
+- [Errors](errors.md): what happens when a hook raises, per event.
+- [Patterns and anti-patterns](patterns.md): how to use the lifecycle well.
diff --git a/docs/hooks/patterns.md b/docs/hooks/patterns.md
new file mode 100644
index 00000000..d3942852
--- /dev/null
+++ b/docs/hooks/patterns.md
@@ -0,0 +1,398 @@
+# Patterns and anti-patterns
+
+Hooks are a small seam, and it is easy to use them well or badly. This page collects the shapes
+that hold up in production and the ones that bite.
+
+- [Patterns](#patterns)
+ - [Audit log](#audit-log)
+ - [PII redaction](#pii-redaction)
+ - [Caching](#caching)
+ - [Metrics](#metrics)
+ - [Guardrails](#guardrails)
+ - [Confidence gating](#confidence-gating)
+ - [Routing override](#routing-override)
+ - [Model lifecycle](#model-lifecycle)
+ - [Multi-tenant context](#multi-tenant-context)
+ - [Composition](#composition)
+ - [Scoped instrumentation](#scoped-instrumentation)
+ - [Token-budget shaping](#token-budget-shaping)
+- [Anti-patterns](#anti-patterns)
+ - [Blocking work](#blocking-work)
+ - [Raising from end hooks for control flow](#raising-from-end-hooks-for-control-flow)
+ - [Shared mutable state without a lock](#shared-mutable-state-without-a-lock)
+ - [Silent failure](#silent-failure)
+ - [Retaining contexts](#retaining-contexts)
+ - [Redacting too late](#redacting-too-late)
+ - [Per-question logic in a per-call hook](#per-question-logic-in-a-per-call-hook)
+ - [Recursive predict](#recursive-predict)
+ - [Plain callables in `hooks=`](#plain-callables-in-hooks)
+ - [Assuming results exist in end hooks](#assuming-results-exist-in-end-hooks)
+ - [Order-dependent hooks](#order-dependent-hooks)
+
+## Patterns
+
+### Audit log
+
+Record every decision with enough to reconstruct it: the state, the questions, the answers, the
+model, the routing decision, usage and latency.
+
+```python
+import json
+
+def audit(ctx):
+ json.dump({
+ "run_id": ctx.run_id,
+ "model": ctx.model,
+ "routing": ctx.results[0].get("routing") if ctx.results else None,
+ "answers": ctx.results[0]["answers"] if ctx.results else None,
+ "usage": ctx.usage,
+ "elapsed_ms": round(ctx.elapsed_ms or 0.0, 3),
+ }, sys.stdout)
+ sys.stdout.write("\n")
+
+laya.load("convaiinnovations/laya", on_predict_end=audit)
+```
+
+Make it lenient if losing a log line must not fail a request: `hooks_raise=False`. Make it
+strict if the audit trail is a compliance requirement.
+
+### PII redaction
+
+Redaction must happen in `on_predict_start`, before tokenization, or the model has already seen
+the data.
+
+```python
+import re
+EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
+
+def redact(ctx):
+ ctx.states = [
+ EMAIL.sub("[email]", s) if isinstance(s, str) else s
+ for s in ctx.states
+ ]
+
+laya.load("convaiinnovations/laya", on_predict_start=redact)
+```
+
+A redaction hook is a policy hook: keep `hooks_raise=True`, because a silently broken redactor
+is a data leak.
+
+### Caching
+
+A start hook checks the cache and calls `ctx.skip(...)`; an end hook fills it. The forward pass
+is skipped on a hit.
+
+```python
+import hashlib, json
+
+CACHE = {}
+
+def key(state, questions):
+ return hashlib.sha256(json.dumps([state, questions], sort_keys=True, default=str).encode()).hexdigest()
+
+def read(ctx):
+ hit = CACHE.get(key(ctx.states[0], ctx.questions))
+ if hit is not None:
+ ctx.skip([hit])
+
+def write(ctx):
+ if ctx.results:
+ CACHE[key(ctx.states[0], ctx.questions)] = ctx.results[0]
+
+laya.load("convaiinnovations/laya", on_predict_start=read, on_predict_end=write)
+```
+
+Guard the cache with a lock when serving concurrently. On the Router the cached payload still
+gets a `routing` key, so the return shape is unchanged.
+
+### Metrics
+
+Counters and histograms from `ctx.model`, `ctx.usage` and `ctx.elapsed_ms`. Keep it lenient.
+
+```python
+COUNTS, LATENCIES = {}, []
+
+def metrics(ctx):
+ COUNTS[ctx.model] = COUNTS.get(ctx.model, 0) + 1
+ if ctx.elapsed_ms is not None:
+ LATENCIES.append(ctx.elapsed_ms)
+
+laya.load("convaiinnovations/laya", on_predict_end=metrics, hooks_raise=False)
+```
+
+### Guardrails
+
+A policy hook raises to block a request. `hooks_raise=True` (the default) lets the block reach
+the caller; `on_error` and `on_predict_end` still run, so the audit trail records it.
+
+```python
+class Blocked(Exception):
+ pass
+
+def guard(ctx):
+ if "ssn" in str(ctx.states[0]).lower():
+ raise Blocked("possible PII in state")
+
+laya.load("convaiinnovations/laya", on_predict_start=guard)
+```
+
+### Confidence gating
+
+An end hook rewrites a low-confidence answer to a safe fallback, or annotates it for downstream
+logic. This is a result mutation, not a rejection.
+
+```python
+def gate(ctx):
+ answer = ctx.results[0]["answers"].get("dept")
+ if answer and answer["confidence"] < 0.6:
+ answer["choice"] = "human-review"
+ answer["gated"] = True
+
+laya.load("convaiinnovations/laya", on_predict_end=gate)
+```
+
+### Routing override
+
+`on_route` may replace `ctx.decision` to pin a checkpoint for a class of traffic.
+
+```python
+from laya.router import RouteDecision
+
+def pin(ctx):
+ if "refund" in str(ctx.states[0]).lower():
+ ctx.decision = RouteDecision(
+ model="typed-decisions",
+ repo="convaiinnovations/laya/typed-decisions",
+ reason="refund workflow",
+ detection=None,
+ workflow=None,
+ )
+
+Router(hooks=[pin])
+```
+
+### Model lifecycle
+
+`on_load` and `on_evict` observe checkpoints. Use them for warmup logs, memory accounting, or
+eviction alerts. They run outside the Router lock, so a hook may call back into the Router.
+
+```python
+class Lifecycle:
+ def on_load(self, ctx):
+ print("loaded", ctx.model)
+
+ def on_evict(self, ctx):
+ print("evicted", ctx.model)
+
+Router(hooks=[Lifecycle()])
+```
+
+### Multi-tenant context
+
+Thread a tenant id through by capturing it in the hook closure, or by reading it from a
+context-local. Do not store per-request state on the hook object without a lock.
+
+```python
+def make_audit(tenant):
+ def audit(ctx):
+ ship(tenant, ctx.run_id, ctx.results)
+ return audit
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=make_audit("acme"))
+```
+
+### Composition
+
+Several hooks of different kinds compose naturally; installed hooks run first, in order.
+
+```python
+agent = laya.load(
+ "convaiinnovations/laya",
+ hooks=[Metrics(), Guardrail()], # metrics first, then policy
+ on_predict_start=redact, # convenience callables appended after hooks
+ hooks_raise=True, # policy failures are fatal
+)
+```
+
+Keep the ordering deliberate and documented, because a later hook sees the mutations of an
+earlier one.
+
+### Scoped instrumentation
+
+Attach a tracer or debug hook only for the code that needs it, instead of reconstructing the
+agent. `hooks_installed` restores the previous list on exit, even if the block raises.
+
+```python
+with agent.hooks_installed(DebugDump()):
+ agent.system_one(state, questions) # DebugDump only here
+```
+
+`add_hook`/`remove_hook` do the same without a block, for a tracer that lives as long as the
+process.
+
+### Token-budget shaping
+
+A start hook can raise the token budget for one call, for example when a question has many
+options and the default head budget would collapse the labels. This does not touch the shared
+agent config, so concurrent calls are unaffected.
+
+```python
+def widen_for_high_cardinality(ctx):
+ k = len(next(iter(ctx.questions.values())).get("criteria", {}) or {})
+ if k >= 50:
+ ctx.head_max_len = max(ctx.head_max_len or 192, 16 + 4 * k)
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=widen_for_high_cardinality)
+```
+
+The same knobs are available per call: `agent.system_one(state, questions, head_max_len=324)`.
+
+## Anti-patterns
+
+### Blocking work
+
+Hooks run on the calling thread, and `laya.serve` uses a single inference worker. A hook that
+sleeps, waits on a network round-trip, or calls `input()` stalls every other request behind it.
+
+```python
+# bad: blocks the whole server
+def audit(ctx):
+ requests.post("https://slow.example/decisions", json=..., timeout=30)
+
+# better: enqueue, let a background worker ship it
+def audit(ctx):
+ QUEUE.put_nowait(record(ctx))
+```
+
+If you must do slow work, set `hooks_concurrent=False` to at least keep the hook itself from
+overlapping, and run `laya.serve` behind a queue.
+
+### Raising from end hooks for control flow
+
+`on_predict_end` runs after inference. Raising there throws away a computed result and, on the
+success path, surfaces to the caller. Use a start hook to block before paying for inference, or
+rewrite `ctx.results` to change the answer.
+
+### Shared mutable state without a lock
+
+The same hook instance runs on many threads. `self.counter += 1` races.
+
+```python
+# bad
+class Count:
+ def __init__(self): self.n = 0
+ def on_predict_end(self, ctx): self.n += 1
+
+# good
+import threading
+class Count:
+ def __init__(self):
+ self.n = 0
+ self._lock = threading.Lock()
+ def on_predict_end(self, ctx):
+ with self._lock:
+ self.n += 1
+```
+
+### Silent failure
+
+`hooks_raise=False` warns once per failure, but a hook that catches everything itself hides
+real problems.
+
+```python
+# bad: no one will ever know the audit trail stopped
+def audit(ctx):
+ try:
+ ship(record(ctx))
+ except Exception:
+ pass
+```
+
+If a hook is optional, let `hooks_raise=False` handle it and watch the warnings. If it is not,
+let it raise.
+
+### Retaining contexts
+
+A hook that appends `ctx` to a list keeps the whole state, questions, results and agent alive.
+
+```python
+# bad: unbounded memory growth
+SEEN = []
+def audit(ctx):
+ SEEN.append(ctx)
+
+# good: keep only what you need
+SEEN = []
+def audit(ctx):
+ SEEN.append((ctx.run_id, ctx.model, ctx.elapsed_ms))
+```
+
+### Redacting too late
+
+By `on_predict_end` the model has already tokenized the state. Redact in `on_predict_start`.
+
+### Per-question logic in a per-call hook
+
+There is one `PredictContext` per call, and one forward pass answers every question. There are no
+per-question events. Iterate the answers inside `on_predict_end`.
+
+```python
+def flag(ctx):
+ for qid, answer in ctx.results[0]["answers"].items():
+ if answer.get("confidence", 1.0) < 0.5:
+ alert(qid, ctx.run_id)
+```
+
+### Recursive predict
+
+A hook that calls `agent.predict`/`system_one` runs the hooks again. Without a depth guard this
+recurses.
+
+```python
+# bad
+def enrich(ctx):
+ ctx.results = [agent.predict(ctx.states[0], EXTRA_QUESTIONS)]
+
+# good: guard, or use a separate agent with no hooks
+def enrich(ctx):
+ if getattr(ctx, "_enriched", False):
+ return
+ ctx._enriched = True
+ ctx.results = [enricher.predict(ctx.states[0], EXTRA_QUESTIONS)]
+```
+
+### Plain callables in `hooks=`
+
+`hooks=` takes hook objects; a bare callable does not say which event it is for, so it is
+rejected. Use `on_predict_start=` / `on_predict_end=`.
+
+```python
+# bad: TypeError
+laya.load("convaiinnovations/laya", hooks=[lambda ctx: None])
+
+# good
+laya.load("convaiinnovations/laya", on_predict_end=lambda ctx: None)
+```
+
+### Assuming results exist in end hooks
+
+On the failure path `ctx.results` is `None` unless a start hook set it. Always check.
+
+```python
+def audit(ctx):
+ if ctx.results is None:
+ log_failure(ctx.run_id, ctx.error)
+ return
+ log_success(ctx.run_id, ctx.results)
+```
+
+### Order-dependent hooks
+
+A hook that reads a mutation from another hook is fragile unless the order is pinned. Installed
+hooks run in list order, then convenience callables; document any coupling, or merge the coupled
+hooks into one object.
+
+## See also
+
+- [Errors](errors.md): the failure matrix behind several of these anti-patterns.
+- [Examples](examples.md): fuller versions of the patterns above.
diff --git a/docs/hooks/tracing.md b/docs/hooks/tracing.md
new file mode 100644
index 00000000..d01c4434
--- /dev/null
+++ b/docs/hooks/tracing.md
@@ -0,0 +1,173 @@
+# Tracing
+
+Every hook of one call receives the same `PredictContext.run_id`, so a tracer can correlate the
+start, end and error events (and any spans it opens) without keeping its own bookkeeping.
+
+- [run_id](#run_id)
+- [A minimal tracer](#a-minimal-tracer)
+- [Span lifetime](#span-lifetime)
+- [Error spans](#error-spans)
+- [OpenTelemetry](#opentelemetry)
+- [Distributed propagation](#distributed-propagation)
+- [Nested calls](#nested-calls)
+
+## run_id
+
+- A `uuid4().hex` string, created once per public call (`predict_batch`, `system_one`,
+ `Router.predict`, `ONNXAgent.system_one`).
+- Shared by every hook of that call, including `on_error` and `on_predict_end`.
+- Not global and not persisted: it identifies a call within the process. Put it in your logs and
+ outbound payloads to correlate across systems.
+- Distinct per call, so two calls never collide.
+
+```
+call A: run_id=1a2b... start ──┐
+ ├─ end
+ error ─┘
+call B: run_id=9f8e... start ─── end
+```
+
+## A minimal tracer
+
+Keep a map from `run_id` to the span you opened at start, and close it at end or error.
+
+```python
+import time
+
+class Tracer:
+ def __init__(self):
+ self.spans = {}
+
+ def on_predict_start(self, ctx):
+ self.spans[ctx.run_id] = {
+ "model": ctx.model,
+ "started_at": ctx.started_at,
+ }
+
+ def on_predict_end(self, ctx):
+ span = self.spans.pop(ctx.run_id, None)
+ if span is None:
+ return
+ emit_span(
+ name="laya.predict",
+ run_id=ctx.run_id,
+ model=ctx.model,
+ duration_ms=ctx.elapsed_ms,
+ usage=ctx.usage,
+ ok=ctx.error is None,
+ )
+
+ def on_error(self, ctx):
+ # on_error runs before on_predict_end; leaving the span for on_predict_end is fine,
+ # or close it here if you prefer.
+ pass
+
+agent = laya.load("convaiinnovations/laya", hooks=[Tracer()])
+```
+
+Because `on_predict_end` always runs, it is the natural place to close a span, and it can see
+`ctx.error` on the failure path.
+
+## Span lifetime
+
+```
+on_predict_start ──► open span (run_id, model, started_at)
+ │
+ ├─ inference
+ │
+on_error ──► record ctx.error on the span
+ │
+on_predict_end ──► close span (elapsed_ms, usage, ok)
+```
+
+## Error spans
+
+`on_predict_end` runs on the failure path with `ctx.error` set, so one close site handles both:
+
+```python
+def on_predict_end(self, ctx):
+ span = self.spans.pop(ctx.run_id, None)
+ if span is None:
+ return
+ if ctx.error is not None:
+ span["status"] = "error"
+ span["error_type"] = type(ctx.error).__name__
+ span["error_message"] = str(ctx.error)
+ span["duration_ms"] = ctx.elapsed_ms
+ emit(span)
+```
+
+If you only implement `on_error`, remember it fires before `on_predict_end`; do not close the
+span in both or you will double-count.
+
+## OpenTelemetry
+
+The example [`examples/hooks/otel.py`](../../examples/hooks/otel.py) records counters and a
+histogram. For real spans, drive the OTel API from the tracer. Hooks are synchronous, so use the
+synchronous exporter (or enqueue and export from a worker).
+
+```python
+from opentelemetry import trace
+
+tracer = trace.get_tracer("laya")
+
+class OTelHooks:
+ def __init__(self):
+ self.spans = {}
+
+ def on_predict_start(self, ctx):
+ span = tracer.start_span("laya.predict", attributes={"laya.run_id": ctx.run_id, "laya.model": ctx.model})
+ self.spans[ctx.run_id] = span
+
+ def on_predict_end(self, ctx):
+ span = self.spans.pop(ctx.run_id, None)
+ if span is None:
+ return
+ if ctx.usage:
+ span.set_attribute("laya.input_tokens", ctx.usage["input_tokens"])
+ if ctx.error is not None:
+ span.record_exception(ctx.error)
+ span.set_status(trace.Status(trace.StatusCode.ERROR))
+ span.end()
+
+laya.load("convaiinnovations/laya", hooks=[OTelHooks()], hooks_raise=False)
+```
+
+Set `hooks_raise=False` so a tracer outage never fails a request.
+
+## Distributed propagation
+
+`run_id` is a plain string, so include it in whatever leaves the process: log lines, the payload
+sent to an audit service, or an HTTP header if a decision triggers a downstream call.
+
+```python
+def audit(ctx):
+ requests.post(
+ "https://audit.example/decisions",
+ json=record(ctx),
+ headers={"X-Laya-Run-Id": ctx.run_id},
+ timeout=2,
+ )
+```
+
+Remember that a blocking call like this stalls the calling thread; enqueue it instead when
+serving concurrently. See [patterns](patterns.md#blocking-work).
+
+## Nested calls
+
+A hook that calls `predict` again starts a new call with a new `run_id`. The parent and child are
+independent unless you link them yourself. Capture the parent id and pass it along:
+
+```python
+def enrich(ctx):
+ child = enricher.predict(ctx.states[0], EXTRA_QUESTIONS)
+ record_child_span(parent_run_id=ctx.run_id, child_run_id=child.get("run_id"))
+```
+
+Guard against recursion (see [anti-patterns](patterns.md#recursive-predict)); the easiest guard
+is a separate `enricher` agent with no hooks.
+
+## See also
+
+- [API reference](api.md): the full `PredictContext`.
+- [Patterns and anti-patterns](patterns.md): non-blocking tracing.
diff --git a/examples/hooks/README.md b/examples/hooks/README.md
new file mode 100644
index 00000000..d454bae7
--- /dev/null
+++ b/examples/hooks/README.md
@@ -0,0 +1,10 @@
+# Prediction hook examples
+
+Small, runnable hooks. See [`docs/hooks/index.md`](../../docs/hooks/index.md) for the full reference.
+
+- [`audit.py`](audit.py): log every decision and optionally ship it to an external service.
+- [`redact.py`](redact.py): strip emails/phones from the state before inference.
+- [`cache.py`](cache.py): cache decisions and skip the forward pass on a hit.
+- [`otel.py`](otel.py): counters and a latency histogram per decision.
+
+Each example loads a real checkpoint, so the first run downloads it.
diff --git a/examples/hooks/audit.py b/examples/hooks/audit.py
new file mode 100644
index 00000000..6b7b1e8b
--- /dev/null
+++ b/examples/hooks/audit.py
@@ -0,0 +1,43 @@
+"""Audit every decision: log it, and optionally ship it to an external service.
+
+Run from the repository root:
+
+ python examples/hooks/audit.py
+"""
+import json
+
+import laya
+
+
+def on_predict_end(ctx):
+ record = {
+ "model": ctx.model,
+ "answers": ctx.results[0]["answers"] if ctx.results else None,
+ "routing": ctx.results[0].get("routing") if ctx.results else None,
+ "usage": ctx.usage,
+ "elapsed_ms": round(ctx.elapsed_ms or 0.0, 2),
+ }
+ print(json.dumps(record, indent=2))
+ # Ship it to an external service if you want:
+ # import requests
+ # requests.post("https://example.invalid/decisions", json=record, timeout=2)
+
+
+QUESTIONS = {
+ "dept": {
+ "type": "choice",
+ "instructions": "Which team should handle this?",
+ "criteria": {"billing": "invoices and payments", "support": "product help"},
+ }
+}
+
+
+# Direct Agent use.
+agent = laya.load("convaiinnovations/laya", on_predict_end=on_predict_end)
+agent.system_one("I was charged twice for the same invoice.", QUESTIONS)
+
+# Router use: the hook also sees ctx.decision (which checkpoint was chosen).
+from laya import Router # noqa: E402
+
+router = Router(on_predict_end=on_predict_end)
+router.predict("I was charged twice for the same invoice.", QUESTIONS)
diff --git a/examples/hooks/cache.py b/examples/hooks/cache.py
new file mode 100644
index 00000000..3c3a0190
--- /dev/null
+++ b/examples/hooks/cache.py
@@ -0,0 +1,41 @@
+"""Cache decisions by state + questions, and skip inference on a hit.
+
+A start hook calls `ctx.skip(results)`; the engine skips the forward pass and still runs
+the end hooks.
+
+ python examples/hooks/cache.py
+"""
+import hashlib
+import json
+
+import laya
+
+CACHE = {}
+
+
+def cache_key(state, questions):
+ payload = json.dumps({"state": state, "questions": questions}, sort_keys=True, default=str)
+ return hashlib.sha256(payload.encode()).hexdigest()
+
+
+def cache_read(ctx):
+ hit = CACHE.get(cache_key(ctx.states[0], ctx.questions))
+ if hit is not None:
+ ctx.skip([hit])
+
+
+def cache_write(ctx):
+ if ctx.results:
+ CACHE[cache_key(ctx.states[0], ctx.questions)] = ctx.results[0]
+
+
+agent = laya.load("convaiinnovations/laya",
+ on_predict_start=cache_read, on_predict_end=cache_write)
+
+STATE = "I was charged twice for the same invoice."
+QUESTIONS = {"urgent": {"type": "noul", "instructions": "Is this urgent?"}}
+
+first = agent.system_one(STATE, QUESTIONS) # runs the model, fills the cache
+second = agent.system_one(STATE, QUESTIONS) # served from CACHE, no forward pass
+print("cache entries:", len(CACHE))
+print("same answer:", first["answers"] == second["answers"])
diff --git a/examples/hooks/otel.py b/examples/hooks/otel.py
new file mode 100644
index 00000000..cd383e56
--- /dev/null
+++ b/examples/hooks/otel.py
@@ -0,0 +1,28 @@
+"""Emit a counter and a latency histogram for every decision.
+
+Works without any dependency; uncomment the OpenTelemetry lines if it is installed.
+
+ python examples/hooks/otel.py
+"""
+import laya
+
+COUNTS = {}
+LATENCIES = []
+
+
+def on_predict_end(ctx):
+ model = ctx.model or "auto"
+ COUNTS[model] = COUNTS.get(model, 0) + 1
+ if ctx.elapsed_ms is not None:
+ LATENCIES.append(ctx.elapsed_ms)
+ # With OpenTelemetry installed (`pip install opentelemetry-api`):
+ # from opentelemetry import metrics
+ # meter = metrics.get_meter("laya")
+ # meter.create_counter("laya.decisions").add(1, {"model": model})
+ # meter.create_histogram("laya.decision.latency_ms").record(ctx.elapsed_ms or 0.0)
+
+
+agent = laya.load("convaiinnovations/laya", on_predict_end=on_predict_end)
+agent.system_one("I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}})
+print("counts:", COUNTS)
+print("latency samples:", len(LATENCIES))
diff --git a/examples/hooks/redact.py b/examples/hooks/redact.py
new file mode 100644
index 00000000..423dc1ee
--- /dev/null
+++ b/examples/hooks/redact.py
@@ -0,0 +1,34 @@
+"""Redact PII from the state before the model ever sees it.
+
+A start hook can rewrite `ctx.states`; the rewrite is what gets tokenized.
+
+ python examples/hooks/redact.py
+"""
+import re
+
+import laya
+
+EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
+PHONE = re.compile(r"\+?\d[\d ()-]{7,}\d")
+
+
+def scrub(value):
+ if isinstance(value, str):
+ return PHONE.sub("[phone]", EMAIL.sub("[email]", value))
+ if isinstance(value, dict):
+ return {k: scrub(v) for k, v in value.items()}
+ if isinstance(value, list):
+ return [scrub(v) for v in value]
+ return value
+
+
+def redact(ctx):
+ ctx.states = [scrub(state) for state in ctx.states]
+
+
+agent = laya.load("convaiinnovations/laya", on_predict_start=redact)
+result = agent.system_one(
+ "Email [email protected] or call +1 555 010 9999 about invoice 42.",
+ {"urgent": {"type": "noul", "instructions": "Is this urgent?"}},
+)
+print(result["answers"])
diff --git a/laya/__init__.py b/laya/__init__.py
index 3f99b89a..330016e8 100644
--- a/laya/__init__.py
+++ b/laya/__init__.py
@@ -1,6 +1,7 @@
"""Laya: Fast, non-autoregressive System 1 decision engine with calibrated probabilities."""
from .email import clean_email_body, email_state
+from .hooks import Hook, PredictContext, PredictHook
from .lang import analyse as detect_language
from .lang import detect_script, is_english
from .presets import (
@@ -87,5 +88,8 @@ __all__ = [
"LayaGuardrailError",
"LayaTriage",
"LayaEvaluator",
+ "PredictContext",
+ "PredictHook",
+ "Hook",
"__version__",
]
diff --git a/laya/agent.py b/laya/agent.py
index 99df622f..f5a21227 100644
--- a/laya/agent.py
+++ b/laya/agent.py
@@ -3,6 +3,7 @@ import json
import os
import tempfile
import threading
+import time
import warnings
from contextlib import nullcontext
from typing import Any, Dict, List, Optional, Union
@@ -24,6 +25,7 @@ from .common import (
render_options,
temp_bucket,
)
+from .hooks import HookRegistry, PredictContext, aggregate_usage, dispatch, normalise_hooks
def _fix_tokenizer_config(path: str):
@@ -177,9 +179,16 @@ def _amp_context(device, dtype):
return nullcontext()
-class Agent:
+class Agent(HookRegistry):
"""System 1 decision model runtime: fast, non-autoregressive, calibrated decisions."""
+ # Hooks are opt-in. `hooks`/`_hooks_mutex` defaults come from HookRegistry; the rest keep a
+ # hand-built instance (`Agent.__new__` in tests) working and make an unset hook a no-op.
+ hooks_raise = True
+ hooks_concurrent = True
+ _hooks_lock = None
+ model_id = None
+
def __init__(
self,
model_id_or_path: str = "convaiinnovations/laya",
@@ -188,6 +197,11 @@ class Agent:
subfolder: Optional[str] = None,
fast: bool = False,
compile: bool = False,
+ hooks=None,
+ on_predict_start=None,
+ on_predict_end=None,
+ hooks_raise: bool = True,
+ hooks_concurrent: bool = True,
):
"""Load a Laya checkpoint.
@@ -197,7 +211,18 @@ class Agent:
`subfolder` selects one checkpoint from a repo that bundles several, e.g.
`Agent("convaiinnovations/laya", subfolder="multilingual")`. Only that subfolder is
downloaded, so bundling does not cost every user the whole family.
+
+ `hooks` / `on_predict_start` / `on_predict_end` observe or shape every prediction; see
+ `laya.hooks`. `hooks_raise=False` warns and continues when a hook fails, and
+ `hooks_concurrent=False` serialises hooks that are not safe to run in parallel.
"""
+ self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
+ self.hooks_raise = bool(hooks_raise)
+ self.hooks_concurrent = bool(hooks_concurrent)
+ self._hooks_lock = threading.RLock() if not hooks_concurrent else None
+ self._hooks_mutex = threading.Lock()
+ self.model_id = model_id_or_path
+
from safetensors.torch import load_file
try:
from transformers.initialization import no_init_weights
@@ -478,10 +503,15 @@ class Agent:
q["labels"] = qdef["labels"]
return q
- def _encode_state(self, state: Union[str, dict, list], ids: List[str], internal: Dict[str, Dict]) -> List[Dict]:
- """Tokenize one state against every (already validated + normalized) question."""
- max_len = self.cfg.get("max_len", 512)
- head_max_len = self.cfg.get("head_max_len", 192)
+ def _encode_state(self, state: Union[str, dict, list], ids: List[str], internal: Dict[str, Dict],
+ max_len: Optional[int] = None, head_max_len: Optional[int] = None) -> List[Dict]:
+ """Tokenize one state against every (already validated + normalized) question.
+
+ `max_len` / `head_max_len` override the agent config for this call (a start hook may set
+ `ctx.max_len` / `ctx.head_max_len`).
+ """
+ max_len = self.cfg.get("max_len", 512) if max_len is None else max_len
+ head_max_len = self.cfg.get("head_max_len", 192) if head_max_len is None else head_max_len
# A chronological conversation list is serialized newest-last, so the default
# right-truncation (st[:room]) would silently drop the newest turn. Truncate
# from the left for lists so the most recent intent is preserved.
@@ -569,7 +599,11 @@ class Agent:
@torch.no_grad()
def predict_batch(self, states: List[Union[str, dict, list]], questions: Dict[str, Dict[str, Any]],
- batch_size: Optional[int] = None) -> List[Dict[str, Any]]:
+ batch_size: Optional[int] = None, hooks=None,
+ on_predict_start=None, on_predict_end=None,
+ hooks_raise: Optional[bool] = None,
+ max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None) -> List[Dict[str, Any]]:
"""Evaluate the same questions over many states, packing them into shared forward passes.
This is the throughput path. `system_one`/`predict` handle one state per forward pass; on a
@@ -583,56 +617,106 @@ class Agent:
questions: Question definitions, exactly as accepted by `system_one`.
batch_size: Optional cap on states per forward pass. `None` sends them all in one pass;
set it to bound peak memory when batching many or long states.
+ hooks, on_predict_start, on_predict_end: Per-call hooks, appended after any installed on
+ the Agent. `on_predict_start` may rewrite the state/questions or call
+ `ctx.skip(...)` to short-circuit inference; `on_predict_end` may rewrite the
+ results. See `laya.hooks`.
+ hooks_raise: Override the Agent's `hooks_raise` for this call.
+ max_len, head_max_len: Override the agent config for this call. A start hook may also
+ set `ctx.max_len` / `ctx.head_max_len` to shape the token budget.
Returns:
A list of per-state result dicts, each identical in shape to `system_one`'s output and
aligned with `states` by index.
"""
- if isinstance(states, (str, bytes, dict)):
- raise TypeError(
- "predict_batch expects a list of states; pass a single state to predict()/system_one()."
- )
- states = list(states)
- if not states:
- return []
-
- ids = list(questions.keys())
- # Empty questions: empty answers, zero usage, no tokenization or forward (matches system_one).
- if not ids:
- return [{"model": "laya-rl-agent", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}
- for _ in states]
-
- # Validate + normalize each question once (state-independent), preserving system_one's checks.
- for qid in ids:
- self._check_question(qid, questions[qid])
- internal = {qid: self._to_internal(questions[qid]) for qid in ids}
- chunk = batch_size if (batch_size and batch_size > 0) else len(states)
-
- results: List[Dict[str, Any]] = []
- for start in range(0, len(states), chunk):
- part = states[start:start + chunk]
- per_state_items = [self._encode_state(st, ids, internal) for st in part]
-
- b = collate_items(per_state_items, self.tok.pad_token_id)
- logits, act = self._forward(b)
- att = b["attention_mask"]
-
- row = 0
- for items in per_state_items:
- nrows = len(items)
- n_tokens = int(att[row:row + nrows].sum())
- answers = self._decode_answers(logits, act, items, ids, internal, row)
- results.append({
- "model": "laya-rl-agent",
- "answers": answers,
- "usage": {"input_tokens": n_tokens, "output_tokens": 0},
- })
- row += nrows
-
- return results
+ active = list(self.hooks) + normalise_hooks(hooks, on_predict_start, on_predict_end)
+ raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)
+ ctx = PredictContext(states=states, questions=questions, model=self.model_id, agent=self,
+ max_len=max_len, head_max_len=head_max_len)
+ try:
+ dispatch(active, "on_predict_start", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ states, questions = ctx.states, ctx.questions
+ if ctx.results is None:
+ # A start hook may have normalised a bare string/dict into a list; only the value
+ # that survives the hook is validated.
+ if isinstance(states, (str, bytes, dict)):
+ raise TypeError(
+ "predict_batch expects a list of states; pass a single state to predict()/system_one()."
+ )
+ states = list(states)
+ if not states:
+ ctx.results = []
+ else:
+ ids = list(questions.keys())
+ # Empty questions: empty answers, zero usage, no tokenization or forward.
+ if not ids:
+ ctx.results = [
+ {"model": "laya-rl-agent", "answers": {},
+ "usage": {"input_tokens": 0, "output_tokens": 0}} for _ in states
+ ]
+ else:
+ # Validate + normalize each question once (state-independent).
+ for qid in ids:
+ self._check_question(qid, questions[qid])
+ internal = {qid: self._to_internal(questions[qid]) for qid in ids}
+ chunk = batch_size if (batch_size and batch_size > 0) else len(states)
+
+ # Per-call token-budget overrides (a start hook may have set them).
+ overrides: Dict[str, int] = {}
+ if ctx.max_len is not None:
+ overrides["max_len"] = ctx.max_len
+ if ctx.head_max_len is not None:
+ overrides["head_max_len"] = ctx.head_max_len
+
+ results: List[Dict[str, Any]] = []
+ for start in range(0, len(states), chunk):
+ part = states[start:start + chunk]
+ per_state_items = [self._encode_state(st, ids, internal, **overrides) for st in part]
+
+ b = collate_items(per_state_items, self.tok.pad_token_id)
+ logits, act = self._forward(b)
+ att = b["attention_mask"]
+
+ row = 0
+ for items in per_state_items:
+ nrows = len(items)
+ n_tokens = int(att[row:row + nrows].sum())
+ answers = self._decode_answers(logits, act, items, ids, internal, row)
+ results.append({
+ "model": "laya-rl-agent",
+ "answers": answers,
+ "usage": {"input_tokens": n_tokens, "output_tokens": 0},
+ })
+ row += nrows
+ ctx.results = results
+ except BaseException as exc:
+ ctx.error = exc
+ try:
+ dispatch(active, "on_error", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ # A failing on_error hook must not hide the failure that triggered it.
+ exc.__context__ = hook_exc
+ raise
+ finally:
+ ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
+ if ctx.results is not None:
+ ctx.usage = aggregate_usage(ctx.results)
+ try:
+ dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ # End hooks run on the failure path too; do not let one mask the real error.
+ if ctx.error is not None:
+ ctx.error.__context__ = hook_exc
+ else:
+ raise
+ return ctx.results
@torch.no_grad()
- def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
+ def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise: Optional[bool] = None,
+ max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None) -> Dict[str, Any]:
"""Evaluate typed questions across state in a single, parallel forward pass.
Args:
@@ -655,7 +739,10 @@ class Agent:
To score many states at once, see `predict_batch`, which shares forward passes across them.
"""
- return self.predict_batch([state], questions)[0]
+ return self.predict_batch([state], questions, hooks=hooks,
+ on_predict_start=on_predict_start,
+ on_predict_end=on_predict_end, hooks_raise=hooks_raise,
+ max_len=max_len, head_max_len=head_max_len)[0]
def __enter__(self):
return self
@@ -681,7 +768,9 @@ RLAgent = Agent
def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str] = None,
- token: Optional[str] = None, subfolder: Optional[str] = None, fast: bool = False) -> Agent:
+ token: Optional[str] = None, subfolder: Optional[str] = None, fast: bool = False,
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise: bool = True, hooks_concurrent: bool = True) -> Agent:
"""Load a Laya agent.
`subfolder` picks one checkpoint out of a repo that bundles several:
@@ -689,5 +778,10 @@ def load(model_id_or_path: str = "convaiinnovations/laya", device: Optional[str]
laya.load("convaiinnovations/laya") # English (repo root)
laya.load("convaiinnovations/laya", subfolder="multilingual")
laya.load("convaiinnovations/laya", fast=True) # TileLang GPU fast path
+
+ `hooks` / `on_predict_start` / `on_predict_end` observe or shape every prediction; see
+ `laya.hooks`.
"""
- return Agent(model_id_or_path, device=device, token=token, subfolder=subfolder, fast=fast)
+ return Agent(model_id_or_path, device=device, token=token, subfolder=subfolder, fast=fast,
+ hooks=hooks, on_predict_start=on_predict_start, on_predict_end=on_predict_end,
+ hooks_raise=hooks_raise, hooks_concurrent=hooks_concurrent)
diff --git a/laya/hooks.py b/laya/hooks.py
new file mode 100644
index 00000000..96e3e3f0
--- /dev/null
+++ b/laya/hooks.py
@@ -0,0 +1,242 @@
+"""Opt-in prediction hooks: observe or shape every decision without forking.
+
+A hook is either a plain callable or an object implementing any subset of the lifecycle
+methods on `Hook`. Hooks are configured on `Agent` / `Router` and can be overridden per call.
+Everything here is pure Python: importing `laya` must not start pulling torch.
+"""
+from __future__ import annotations
+
+import threading
+import time
+import uuid
+import warnings
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from typing import Any, Callable, Dict, List, Optional, Protocol, Sequence, Union
+
+
+@dataclass(eq=False)
+class PredictContext:
+ """Mutable state passed to every hook for one call.
+
+ `states` / `questions` may be rewritten by `on_predict_start`; `results` may be rewritten
+ by `on_predict_end`. A start hook can call `skip()` to short-circuit inference with a
+ cached result.
+
+ Identity semantics (`eq=False`): two contexts are never equal, and a context is hashable
+ by identity, so a hook can keep one in a set without comparing agent/router internals.
+ """
+
+ states: List[Any]
+ questions: Dict[str, Any]
+ run_id: str = field(default_factory=lambda: uuid.uuid4().hex) # shared by every hook of one call
+ results: Optional[List[Dict[str, Any]]] = None
+ decision: Optional[Dict[str, Any]] = None # Router: the RouteDecision dict
+ model: Optional[str] = None # resolved checkpoint name
+ agent: Any = None
+ router: Any = None
+ max_len: Optional[int] = None # per-call overrides; None = agent config
+ head_max_len: Optional[int] = None
+ usage: Optional[Dict[str, int]] = None # aggregated input/output tokens
+ started_at: float = field(default_factory=time.perf_counter)
+ elapsed_ms: Optional[float] = None
+ error: Optional[BaseException] = None
+
+ def skip(self, results: List[Dict[str, Any]]) -> None:
+ """Set cached results from a start hook; inference is skipped, end hooks still run."""
+ self.results = results
+
+
+class Hook(Protocol):
+ """Optional lifecycle methods. Implement any subset; missing methods are skipped."""
+
+ def on_predict_start(self, ctx: PredictContext) -> None: ...
+ def on_predict_end(self, ctx: PredictContext) -> None: ...
+ def on_route(self, ctx: PredictContext) -> None: ...
+ def on_load(self, ctx: PredictContext) -> None: ...
+ def on_evict(self, ctx: PredictContext) -> None: ...
+ def on_error(self, ctx: PredictContext) -> None: ...
+
+
+PredictHook = Callable[[PredictContext], None]
+HookArg = Union[Hook, Sequence[Hook], None]
+PredictHookArg = Union[PredictHook, Sequence[PredictHook], None]
+
+HOOK_EVENTS = (
+ "on_predict_start",
+ "on_predict_end",
+ "on_route",
+ "on_load",
+ "on_evict",
+ "on_error",
+)
+
+
+class _StartAdapter:
+ """Wrap a plain `on_predict_start` callable as a `Hook`."""
+
+ __slots__ = ("fn",)
+
+ def __init__(self, fn: PredictHook):
+ if not callable(fn):
+ raise TypeError("on_predict_start must be callable, got %s" % type(fn).__name__)
+ self.fn = fn
+
+ def on_predict_start(self, ctx: PredictContext) -> None:
+ self.fn(ctx)
+
+
+class _EndAdapter:
+ """Wrap a plain `on_predict_end` callable as a `Hook`."""
+
+ __slots__ = ("fn",)
+
+ def __init__(self, fn: PredictHook):
+ if not callable(fn):
+ raise TypeError("on_predict_end must be callable, got %s" % type(fn).__name__)
+ self.fn = fn
+
+ def on_predict_end(self, ctx: PredictContext) -> None:
+ self.fn(ctx)
+
+
+def _as_sequence(value: Any) -> tuple:
+ if value is None:
+ return ()
+ if isinstance(value, (list, tuple)):
+ return tuple(value)
+ return (value,)
+
+
+def normalise_hooks(
+ hooks: HookArg = None,
+ on_predict_start: PredictHookArg = None,
+ on_predict_end: PredictHookArg = None,
+) -> List[Any]:
+ """Flatten `hooks` and the two convenience callables into one ordered hook list.
+
+ Installed hooks are normalised once at construction; per-call arguments are appended
+ after them, so per-call hooks always run last.
+ """
+ result: List[Any] = []
+ for hook in _as_sequence(hooks):
+ if isinstance(hook, type):
+ raise TypeError(
+ "hooks entries must be instances, not classes; got %s. Instantiate it first."
+ % hook.__name__
+ )
+ if not any(hasattr(hook, event) for event in HOOK_EVENTS):
+ raise TypeError(
+ "hooks entries must implement at least one of %s; got %s. Pass a plain "
+ "callable as on_predict_start=/on_predict_end= instead."
+ % (", ".join(HOOK_EVENTS), type(hook).__name__)
+ )
+ for event in HOOK_EVENTS:
+ method = getattr(hook, event, None)
+ if method is not None and not callable(method):
+ raise TypeError(
+ "hooks entry %s.%s must be callable, got %s"
+ % (type(hook).__name__, event, type(method).__name__)
+ )
+ result.append(hook)
+ for fn in _as_sequence(on_predict_start):
+ result.append(_StartAdapter(fn))
+ for fn in _as_sequence(on_predict_end):
+ result.append(_EndAdapter(fn))
+ return result
+
+
+class HookRegistry:
+ """Mixin giving a runtime-mutable hook list.
+
+ `Agent`, `Router` and `ONNXAgent` use it so hooks can be added, removed or scoped after
+ construction. Mutation is guarded by `_hooks_mutex`; a call reads a snapshot of the list,
+ so adding or removing a hook never disturbs a call in flight.
+ """
+
+ hooks = ()
+ _hooks_mutex = None
+
+ def add_hook(self, hook: HookArg) -> "HookRegistry":
+ """Install one hook or a sequence of them. Returns self for chaining."""
+ self._extend_hooks(normalise_hooks(hook))
+ return self
+
+ def remove_hook(self, hook: Any) -> bool:
+ """Remove a hook by identity. Returns True if it was installed."""
+ return self._remove_hooks(lambda installed: installed is hook) > 0
+
+ @contextmanager
+ def hooks_installed(self, *hooks: HookArg):
+ """Install hooks for the duration of the `with` block, then remove them.
+
+ with agent.hooks_installed(tracer):
+ agent.system_one(state, questions)
+ """
+ added = normalise_hooks(list(hooks))
+ self._extend_hooks(added)
+ try:
+ yield self
+ finally:
+ self._remove_hooks(lambda installed: any(installed is one for one in added))
+
+ def _extend_hooks(self, hooks: Sequence[Any]) -> None:
+ if not hooks:
+ return
+ with self._hooks_mutex_for_registry():
+ self.hooks = list(self.hooks) + list(hooks)
+
+ def _remove_hooks(self, predicate: Callable[[Any], bool]) -> int:
+ with self._hooks_mutex_for_registry():
+ current = list(self.hooks)
+ remaining = [hook for hook in current if not predicate(hook)]
+ self.hooks = remaining
+ return len(current) - len(remaining)
+
+ def _hooks_mutex_for_registry(self) -> threading.Lock:
+ lock = self._hooks_mutex
+ if lock is None:
+ # Only reachable for an instance built without __init__ (tests); real instances
+ # get their mutex in the constructor.
+ lock = self._hooks_mutex = threading.Lock()
+ return lock
+
+
+def aggregate_usage(results: Sequence[Dict[str, Any]]) -> Dict[str, int]:
+ """Sum the per-state usage blocks so a hook sees one total for the call."""
+ def total(key: str) -> int:
+ return sum(int((r.get("usage") or {}).get(key, 0) or 0) for r in results)
+ return {"input_tokens": total("input_tokens"), "output_tokens": total("output_tokens")}
+
+
+def dispatch(
+ hooks: Sequence[Any],
+ event: str,
+ ctx: PredictContext,
+ *,
+ raise_errors: bool = True,
+ lock: Any = None,
+) -> None:
+ """Call `event` on every hook that implements it.
+
+ `raise_errors=False` warns and continues, for hooks (telemetry) that must not fail a
+ request. `lock` serialises dispatch for hooks that are not safe to run concurrently.
+ """
+ for hook in hooks:
+ method = getattr(hook, event, None)
+ if method is None:
+ continue
+ try:
+ if lock is not None:
+ with lock:
+ method(ctx)
+ else:
+ method(ctx)
+ except Exception as exc: # noqa: BLE001 -- policy depends on raise_errors
+ if raise_errors:
+ raise
+ warnings.warn(
+ "laya: hook %s.%s failed: %s" % (type(hook).__name__, event, exc),
+ RuntimeWarning,
+ stacklevel=2,
+ )
diff --git a/laya/onnx_agent.py b/laya/onnx_agent.py
index 273454f1..8664bd25 100644
--- a/laya/onnx_agent.py
+++ b/laya/onnx_agent.py
@@ -1,10 +1,13 @@
import json
import os
+import threading
+import time
import warnings
from typing import Any, Dict, Optional, Union
import numpy as np
+from laya.hooks import HookRegistry, PredictContext, aggregate_usage, dispatch, normalise_hooks
from laya.common import (
QTYPES,
build_sequence,
@@ -18,14 +21,26 @@ from laya.common import (
)
-class ONNXAgent:
+class ONNXAgent(HookRegistry):
"""System 1 decision model runtime via ONNX: fast CPU-optimized decisions."""
+ # Opt-in hooks; defaults keep a hand-built instance working and make an unset hook a no-op.
+ # `hooks`/`_hooks_mutex` come from HookRegistry.
+ hooks_raise = True
+ hooks_concurrent = True
+ _hooks_lock = None
+ model_id = None
+
def __init__(
self,
model_id_or_path: str,
onnx_path: str = "laya.onnx",
subfolder: Optional[str] = None,
+ hooks=None,
+ on_predict_start=None,
+ on_predict_end=None,
+ hooks_raise: bool = True,
+ hooks_concurrent: bool = True,
):
"""Load a Laya agent backed by ONNX Runtime.
@@ -34,7 +49,17 @@ class ONNXAgent:
(used to load the tokenizer and config).
onnx_path: Path to the exported .onnx file.
subfolder: Optional subfolder if downloading from a repo bundle.
+ hooks, on_predict_start, on_predict_end: Opt-in prediction hooks; see `laya.hooks`.
+ hooks_raise: When False, a failing hook warns and inference continues.
+ hooks_concurrent: When False, hooks are serialised with a lock.
"""
+ self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
+ self.hooks_raise = bool(hooks_raise)
+ self.hooks_concurrent = bool(hooks_concurrent)
+ self._hooks_lock = threading.RLock() if not hooks_concurrent else None
+ self._hooks_mutex = threading.Lock()
+ self.model_id = model_id_or_path
+
import onnxruntime as ort
from transformers import AutoTokenizer
@@ -129,15 +154,55 @@ class ONNXAgent:
q["labels"] = qdef["labels"]
return q
- def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
+ def system_one(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
+ hooks=None, on_predict_start=None, on_predict_end=None,
+ hooks_raise: Optional[bool] = None,
+ max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None) -> Dict[str, Any]:
+ """Evaluate typed questions, running any opt-in hooks around the inference."""
+ active = list(self.hooks) + normalise_hooks(hooks, on_predict_start, on_predict_end)
+ raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)
+ ctx = PredictContext(states=[state], questions=questions, model=self.model_id, agent=self,
+ max_len=max_len, head_max_len=head_max_len)
+ try:
+ dispatch(active, "on_predict_start", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ if ctx.results is None:
+ overrides = {}
+ if ctx.max_len is not None:
+ overrides["max_len"] = ctx.max_len
+ if ctx.head_max_len is not None:
+ overrides["head_max_len"] = ctx.head_max_len
+ ctx.results = [self._infer(ctx.states[0], ctx.questions, **overrides)]
+ except BaseException as exc:
+ ctx.error = exc
+ try:
+ dispatch(active, "on_error", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ exc.__context__ = hook_exc
+ raise
+ finally:
+ ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
+ if ctx.results is not None:
+ ctx.usage = aggregate_usage(ctx.results)
+ try:
+ dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ if ctx.error is not None:
+ ctx.error.__context__ = hook_exc
+ else:
+ raise
+ return ctx.results[0]
+
+ def _infer(self, state: Union[str, dict, list], questions: Dict[str, Dict[str, Any]],
+ max_len: Optional[int] = None, head_max_len: Optional[int] = None) -> Dict[str, Any]:
from .agent import Agent as _Agent
ids = list(questions.keys())
for qid in ids:
_Agent._check_question(qid, questions[qid])
items = []
- max_len = self.cfg.get("max_len", 512)
- head_max_len = self.cfg.get("head_max_len", 192)
+ max_len = self.cfg.get("max_len", 512) if max_len is None else max_len
+ head_max_len = self.cfg.get("head_max_len", 192) if head_max_len is None else head_max_len
for qid in ids:
q = self._to_internal(questions[qid])
diff --git a/laya/router.py b/laya/router.py
index d756faf2..23cfc3a1 100644
--- a/laya/router.py
+++ b/laya/router.py
@@ -30,8 +30,10 @@ synthetic workflows and should not be a silent default.
import gc
import os
import threading
+import time
from typing import Any, Dict, List, Optional, Union
+from .hooks import HookRegistry, PredictContext, aggregate_usage, dispatch, normalise_hooks
from .lang import analyse
# The hub repo bundles all three checkpoints; only the requested subfolder is downloaded.
@@ -146,7 +148,7 @@ def _english_from_code(value: Any) -> Optional[bool]:
return primary in _ENGLISH_SUBTAGS
-class Router:
+class Router(HookRegistry):
"""Lazily loads Laya checkpoints and sends each request to the right one.
from laya import Router
@@ -173,8 +175,18 @@ class Router:
r = Router(preload=True) # all three resident, routing is free
r = Router(preload=True, device="cuda")
r.preload(["english", "multilingual"]) # or just the two you serve
+
+ Hooks are opt-in and run at the Router level: `on_route` sees the routing decision,
+ `on_load` / `on_evict` see model lifecycle, and `on_predict_start` / `on_predict_end`
+ wrap the whole route+infer call. See `laya.hooks`.
"""
+ # Opt-in defaults so a hand-built instance (`Router.__new__` in tests) works unset.
+ # `hooks`/`_hooks_mutex` come from HookRegistry.
+ hooks_raise = True
+ hooks_concurrent = True
+ _hooks_lock = None
+
def __init__(
self,
models: Optional[Dict[str, str]] = None,
@@ -186,7 +198,17 @@ class Router:
standalone_repos: bool = False,
preload: bool = False,
lang_guess: Optional[Any] = None,
+ hooks=None,
+ on_predict_start=None,
+ on_predict_end=None,
+ hooks_raise: bool = True,
+ hooks_concurrent: bool = True,
):
+ self.hooks = normalise_hooks(hooks, on_predict_start, on_predict_end)
+ self.hooks_raise = bool(hooks_raise)
+ self.hooks_concurrent = bool(hooks_concurrent)
+ self._hooks_lock = threading.RLock() if not hooks_concurrent else None
+ self._hooks_mutex = threading.Lock()
self.models = dict(STANDALONE_MODELS if standalone_repos else DEFAULT_MODELS)
if models:
self.models.update({normalise_name(k): v for k, v in models.items()})
@@ -226,8 +248,13 @@ class Router:
agent = Agent(repo, device=self.device, token=self.token, subfolder=sub)
self._agents[key] = agent
self._order.append(key)
- self._evict()
- return agent
+ evicted = self._evict_locked()
+ # Lifecycle hooks fire after the lock is released, so a hook can safely call the Router.
+ self._dispatch_lifecycle("on_evict", evicted)
+ dispatch(self.hooks, "on_load",
+ PredictContext(states=[], questions={}, model=key, agent=agent, router=self),
+ raise_errors=self.hooks_raise, lock=self._hooks_lock)
+ return agent
def _touch(self, key: str):
with self._lock:
@@ -235,30 +262,41 @@ class Router:
self._order.remove(key)
self._order.append(key)
- def _evict(self):
+ def _evict_locked(self) -> List[str]:
+ """Drop least-recently-used agents until `max_loaded` holds. Returns evicted names."""
+ evicted: List[str] = []
+ while len(self._order) > self.max_loaded:
+ victim = self._order.pop(0)
+ agent = self._agents.pop(victim, None)
+ if agent is not None:
+ evicted.append(victim)
+ del agent
+ if len(self._order) < len(self._agents): # keep the two views consistent
+ for k in list(self._agents):
+ if k not in self._order:
+ agent = self._agents.pop(k, None)
+ if agent is not None:
+ evicted.append(k)
+ del agent
+ if evicted:
+ gc.collect()
+ try:
+ import torch
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ except Exception:
+ pass
+ return evicted
+
+ def _evict(self) -> List[str]:
with self._lock:
- evicted = False
- while len(self._order) > self.max_loaded:
- victim = self._order.pop(0)
- agent = self._agents.pop(victim, None)
- if agent is not None:
- evicted = True
- del agent
- if len(self._order) < len(self._agents): # keep the two views consistent
- for k in list(self._agents):
- if k not in self._order:
- agent = self._agents.pop(k, None)
- if agent is not None:
- evicted = True
- del agent
- if evicted:
- gc.collect()
- try:
- import torch
- if torch.cuda.is_available():
- torch.cuda.empty_cache()
- except Exception:
- pass
+ return self._evict_locked()
+
+ def _dispatch_lifecycle(self, event: str, names: List[str]) -> None:
+ for name in names:
+ dispatch(self.hooks, event,
+ PredictContext(states=[], questions={}, model=name, router=self),
+ raise_errors=self.hooks_raise, lock=self._hooks_lock)
def attach(self, name: str, agent: Any):
"""Register an already-built Agent under `name` instead of loading a second copy.
@@ -285,15 +323,19 @@ class Router:
names = [normalise_name(n) for n in (list(self.models) if names is None else names)]
with self._lock:
self.max_loaded = max(self.max_loaded, len(set(names) | set(self._agents)))
- for n in names:
- if n not in self._agents: # an attached agent is already built
- self.load(n)
+ for n in names:
+ with self._lock:
+ already = n in self._agents # an attached agent is already built
+ if not already:
+ # load() dispatches on_load outside the lock; do not hold it across the call.
+ self.load(n)
return self
def unload(self, name: Optional[str] = None):
"""Free one model, or all of them."""
with self._lock:
if name is None:
+ freed = list(self._order)
self._agents.clear()
self._order.clear()
else:
@@ -301,6 +343,7 @@ class Router:
agent = self._agents.pop(key, None)
if key in self._order:
self._order.remove(key)
+ freed = [key] if agent is not None else []
del agent
gc.collect()
try:
@@ -309,6 +352,7 @@ class Router:
torch.cuda.empty_cache()
except Exception:
pass
+ self._dispatch_lifecycle("on_evict", freed)
@property
def loaded(self) -> List[str]:
@@ -337,6 +381,30 @@ class Router:
task: Optional[str] = None,
lang: Optional[str] = None,
lang_guess: Optional[Any] = None,
+ hooks=None,
+ hooks_raise: Optional[bool] = None,
+ ) -> RouteDecision:
+ """Decide which checkpoint to use, then let `on_route` hooks observe or replace it.
+
+ `ctx.decision` is the `RouteDecision`; a hook may replace it (for example to pin a
+ checkpoint) and the replacement is what gets returned and used. `hooks` are per-call
+ hooks, appended after any installed on the Router.
+ """
+ decision = self._route(state, questions, model=model, task=task, lang=lang, lang_guess=lang_guess)
+ raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)
+ active = list(self.hooks) + normalise_hooks(hooks)
+ ctx = PredictContext(states=[state], questions=questions or {}, decision=decision, router=self)
+ dispatch(active, "on_route", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ return ctx.decision
+
+ def _route(
+ self,
+ state: Union[str, dict, list, None],
+ questions: Optional[Dict[str, Any]] = None,
+ model: Optional[str] = None,
+ task: Optional[str] = None,
+ lang: Optional[str] = None,
+ lang_guess: Optional[Any] = None,
) -> RouteDecision:
"""Decide which checkpoint to use, without loading or running anything.
@@ -423,16 +491,68 @@ class Router:
task: Optional[str] = None,
lang: Optional[str] = None,
lang_guess: Optional[Any] = None,
+ hooks=None,
+ on_predict_start=None,
+ on_predict_end=None,
+ hooks_raise: Optional[bool] = None,
+ max_len: Optional[int] = None,
+ head_max_len: Optional[int] = None,
) -> Dict[str, Any]:
"""Route, then answer every question in one forward pass on the chosen checkpoint.
The result is the usual `system_one` payload plus a `routing` key recording the decision.
+ Router-level `on_predict_start` / `on_predict_end` hooks wrap the whole route+infer call
+ and see `ctx.decision`; see `laya.hooks`. `max_len` / `head_max_len` override the agent
+ token budget for this call (a start hook may set `ctx.max_len` / `ctx.head_max_len`).
"""
- decision = self.route(state, questions, model=model, task=task, lang=lang, lang_guess=lang_guess)
+ active = list(self.hooks) + normalise_hooks(hooks, on_predict_start, on_predict_end)
+ raise_errors = self.hooks_raise if hooks_raise is None else bool(hooks_raise)
+
+ # Per-call hooks apply to the whole call, including on_route inside route().
+ decision = self.route(state, questions, model=model, task=task, lang=lang,
+ lang_guess=lang_guess, hooks=hooks, hooks_raise=hooks_raise)
agent = self.load(decision["model"])
- result = agent.system_one(state, questions)
- result["routing"] = dict(decision)
- return result
+ ctx = PredictContext(states=[state], questions=questions, decision=dict(decision),
+ model=decision["model"], agent=agent, router=self,
+ max_len=max_len, head_max_len=head_max_len)
+ try:
+ dispatch(active, "on_predict_start", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ if ctx.results is None:
+ # Pass token-budget overrides only when set, so any Agent-like object that does
+ # not accept them still works on the default path.
+ overrides = {}
+ if ctx.max_len is not None:
+ overrides["max_len"] = ctx.max_len
+ if ctx.head_max_len is not None:
+ overrides["head_max_len"] = ctx.head_max_len
+ result = agent.system_one(ctx.states[0], ctx.questions, **overrides)
+ result["routing"] = dict(decision)
+ ctx.results = [result]
+ else:
+ # A cache hit short-circuits inference, but Router.predict still promises a
+ # `routing` key. Add it without overwriting a routing the cached payload has.
+ for result in ctx.results:
+ if isinstance(result, dict):
+ result.setdefault("routing", dict(decision))
+ except BaseException as exc:
+ ctx.error = exc
+ try:
+ dispatch(active, "on_error", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ exc.__context__ = hook_exc
+ raise
+ finally:
+ ctx.elapsed_ms = (time.perf_counter() - ctx.started_at) * 1000.0
+ if ctx.results is not None:
+ ctx.usage = aggregate_usage(ctx.results)
+ try:
+ dispatch(active, "on_predict_end", ctx, raise_errors=raise_errors, lock=self._hooks_lock)
+ except BaseException as hook_exc:
+ if ctx.error is not None:
+ ctx.error.__context__ = hook_exc
+ else:
+ raise
+ return ctx.results[0]
def __enter__(self):
return self
diff --git a/tests/test_hooks.py b/tests/test_hooks.py
new file mode 100644
index 00000000..c1f57fe0
--- /dev/null
+++ b/tests/test_hooks.py
@@ -0,0 +1,649 @@
+"""Prediction hooks: observe or shape a decision without a model or a download.
+
+The Agent path uses the same weight-free fake as `test_batch.py` (the real `predict_batch`
+with `_encode_state` / `_forward` / `_decode_answers` stubbed). The Router path patches
+`laya.agent.Agent` so a model build never touches the Hub.
+
+Run: python tests/test_hooks.py
+"""
+import os
+import sys
+import threading
+import time
+import warnings
+
+import numpy as np
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import laya.agent as _agent_mod # noqa: E402
+from laya.agent import Agent # noqa: E402
+from laya.hooks import normalise_hooks # noqa: E402
+from laya.router import RouteDecision, Router # noqa: E402
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s: got %r, want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s%s" % (name, ": " + detail if detail else ""))
+
+
+def check_raises(name, exc, fn):
+ try:
+ fn()
+ except exc:
+ PASS.append(name)
+ return
+ except BaseException as other: # noqa: BLE001
+ FAIL.append("%s: raised %r, want %r" % (name, other, exc))
+ return
+ FAIL.append("%s: did not raise %r" % (name, exc))
+
+
+NQ = 2
+QUESTIONS = {"a": {"type": "noul", "instructions": "?"}, "b": {"type": "noul", "instructions": "?"}}
+
+
+def make_fake():
+ """A real `predict_batch` with the three composed helpers stubbed out."""
+ fake = Agent.__new__(Agent)
+ fake.tok = type("Tok", (), {"pad_token_id": 0})()
+ fake._to_internal = staticmethod(Agent._to_internal).__func__
+ fake._encode_states = []
+ fake._forward_calls = []
+
+ def _encode_state(state, ids, internal):
+ fake._encode_states.append(state)
+ return [{"ids": [1, 2, 3], "markers": [0, 1], "qtype": 2} for _ in ids]
+
+ def _forward(b):
+ n = b["input_ids"].shape[0]
+ fake._forward_calls.append(n)
+ return np.zeros((n, 2), dtype=np.float32), np.full((n, 2), 0.5, dtype=np.float32)
+
+ def _decode_answers(logits, act, items, ids, internal, offset):
+ return {"_offset": offset}
+
+ fake._encode_state = _encode_state
+ fake._forward = _forward
+ fake._decode_answers = _decode_answers
+ return fake
+
+
+class FakeAgent:
+ def system_one(self, state, questions):
+ return {"model": "fake", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}
+
+
+# --------------------------------------------------------------- order and context
+events = []
+results_at_start = []
+results_at_end = []
+run_id_start = []
+run_id_end = []
+
+
+def start(ctx):
+ events.append(("start", ctx))
+ results_at_start.append(ctx.results)
+ run_id_start.append(ctx.run_id)
+
+
+def end(ctx):
+ events.append(("end", ctx))
+ results_at_end.append(ctx.results)
+ run_id_end.append(ctx.run_id)
+
+
+f = make_fake()
+res = f.predict_batch(["s0", "s1"], QUESTIONS, on_predict_start=start, on_predict_end=end)
+check("order/start then end", [e[0] for e in events], ["start", "end"])
+check("order/once each", len(events), 2)
+check("context/start sees states", events[0][1].states, ["s0", "s1"])
+check("context/start sees questions", events[0][1].questions, QUESTIONS)
+check("context/start has no results", results_at_start, [None])
+check_true("context/end results is the returned object", results_at_end[0] is res)
+check("context/end result count", len(results_at_end[0]), 2)
+check_true("context/elapsed_ms set", events[1][1].elapsed_ms is not None)
+check("context/usage aggregated", events[1][1].usage, {"input_tokens": 12, "output_tokens": 0})
+check_true("context/agent set", events[1][1].agent is f)
+check_true("context/run_id is a non-empty string",
+ isinstance(run_id_start[0], str) and bool(run_id_start[0]))
+check("context/run_id is shared by start and end", run_id_end[0], run_id_start[0])
+
+_rids = []
+_f2 = make_fake()
+_f2.predict_batch(["s0"], QUESTIONS, on_predict_end=lambda c: _rids.append(c.run_id))
+_f2.predict_batch(["s0"], QUESTIONS, on_predict_end=lambda c: _rids.append(c.run_id))
+check("context/run_id differs per call", len(set(_rids)), 2)
+
+
+# --------------------------------------------------------------- mutation
+def rewrite(ctx):
+ ctx.states = ["rewritten"]
+ ctx.questions = {"a": {"type": "noul", "instructions": "?"}}
+
+
+f = make_fake()
+f.predict_batch(["orig"], QUESTIONS, on_predict_start=rewrite)
+check("mutation/start rewrite is what gets encoded", f._encode_states, ["rewritten"])
+
+
+def replace_results(ctx):
+ ctx.results = [{"model": "replaced"}]
+
+
+f = make_fake()
+res = f.predict_batch(["s0"], QUESTIONS, on_predict_end=replace_results)
+check("mutation/end rewrite is returned", res, [{"model": "replaced"}])
+
+
+# --------------------------------------------------------------- caching short-circuit
+end_calls = {"n": 0}
+
+
+def cache_hit(ctx):
+ ctx.skip([{"model": "cached"}])
+
+
+def count_end(ctx):
+ end_calls["n"] += 1
+
+
+f = make_fake()
+res = f.predict_batch(["s0"], QUESTIONS, on_predict_start=cache_hit, on_predict_end=count_end)
+check("skip/returns cached results", res, [{"model": "cached"}])
+check("skip/no forward pass", f._forward_calls, [])
+check("skip/end still runs", end_calls["n"], 1)
+
+
+# --------------------------------------------------------------- installed vs per-call
+order = []
+f = make_fake()
+f.hooks = normalise_hooks(on_predict_start=lambda ctx: order.append("installed"))
+f.predict_batch(["s0"], QUESTIONS, on_predict_start=lambda ctx: order.append("percall"))
+check("installed/per-call additive and ordered", order, ["installed", "percall"])
+
+seq = []
+f = make_fake()
+f.predict_batch(["s0"], QUESTIONS,
+ on_predict_start=[lambda c: seq.append(1), lambda c: seq.append(2)])
+check("sequence/runs in order", seq, [1, 2])
+
+
+# --------------------------------------------------------------- system_one inherits
+seen = []
+f = make_fake()
+f.hooks = normalise_hooks(on_predict_start=lambda ctx: seen.append(ctx.states))
+out = f.system_one("state", QUESTIONS)
+check("system_one/hook fired", seen, [["state"]])
+check_true("system_one/returns a dict", isinstance(out, dict))
+
+
+# --------------------------------------------------------------- error policy
+def boom(ctx):
+ raise ValueError("hook boom")
+
+
+f = make_fake()
+check_raises("hooks_raise=True/propagates", ValueError,
+ lambda: f.predict_batch(["s0"], QUESTIONS, on_predict_start=boom))
+
+f = make_fake()
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ res = f.predict_batch(["s0"], QUESTIONS, on_predict_start=boom, hooks_raise=False)
+check("hooks_raise=False/continues", len(res), 1)
+check_true("hooks_raise=False/warns",
+ any(issubclass(w.category, RuntimeWarning) for w in caught))
+
+
+class ErrHook:
+ def __init__(self):
+ self.errors = []
+
+ def on_error(self, ctx):
+ self.errors.append(ctx.error)
+
+
+fail_seen = {}
+
+
+def end_on_fail(ctx):
+ fail_seen["error"] = ctx.error
+
+
+def bad_forward(b):
+ raise RuntimeError("infer boom")
+
+
+rec = ErrHook()
+f = make_fake()
+f._forward = bad_forward
+raised = None
+try:
+ f.predict_batch(["s0"], QUESTIONS, hooks=[rec], on_predict_end=end_on_fail)
+except RuntimeError as exc:
+ raised = exc
+check_true("on_error/propagates original", isinstance(raised, RuntimeError))
+check("on_error/fired once", len(rec.errors), 1)
+check("on_error/saw the error", str(rec.errors[0]), "infer boom")
+check("on_error/end still ran", str(fail_seen.get("error")), "infer boom")
+
+
+class BadErrHook:
+ def on_error(self, ctx):
+ raise ValueError("hook error")
+
+
+f = make_fake()
+f._forward = bad_forward
+raised = None
+try:
+ f.predict_batch(["s0"], QUESTIONS, hooks=[BadErrHook()])
+except BaseException as exc: # noqa: BLE001
+ raised = exc
+check_true("on_error/failing hook does not mask the original", isinstance(raised, RuntimeError))
+check_true("on_error/failing hook is chained",
+ isinstance(getattr(raised, "__context__", None), ValueError))
+
+
+# a start hook that raises must still run on_error and the end hooks
+class StartErrRec:
+ def __init__(self):
+ self.errors = []
+
+ def on_error(self, ctx):
+ self.errors.append(ctx.error)
+
+
+start_fail_seen = []
+
+
+def raising_start(ctx):
+ raise ValueError("start boom")
+
+
+def end_after_start_fail(ctx):
+ start_fail_seen.append(ctx.error)
+
+
+rec2 = StartErrRec()
+f = make_fake()
+raised = None
+try:
+ f.predict_batch(["s0"], QUESTIONS, hooks=[rec2],
+ on_predict_start=raising_start, on_predict_end=end_after_start_fail)
+except ValueError as exc:
+ raised = exc
+check_true("start failure/propagates", isinstance(raised, ValueError))
+check("start failure/on_error fired", len(rec2.errors), 1)
+check("start failure/end fired with the error", len(start_fail_seen), 1)
+check("start failure/end saw the error", str(start_fail_seen[0]), "start boom")
+
+
+# ctx.model carries the agent's model id
+f = make_fake()
+f.model_id = "convaiinnovations/laya"
+models = []
+f.predict_batch(["s0"], QUESTIONS, on_predict_end=lambda c: models.append(c.model))
+check("context/model is the agent model_id", models, ["convaiinnovations/laya"])
+
+
+# --------------------------------------------------------------- empty inputs
+empty = []
+f = make_fake()
+res = f.predict_batch([], QUESTIONS,
+ on_predict_start=lambda c: empty.append(("s", c.results)),
+ on_predict_end=lambda c: empty.append(("e", c.results)))
+check("empty states/returns []", res, [])
+check("empty states/start results None", empty[0][1], None)
+check("empty states/end results []", empty[1][1], [])
+
+empty = []
+f = make_fake()
+f.predict_batch(["s0"], {}, on_predict_end=lambda c: empty.append(c.results))
+check("empty questions/end sees empty results",
+ empty[0],
+ [{"model": "laya-rl-agent", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}])
+
+
+# --------------------------------------------------------------- input guards and rescue
+f = make_fake()
+check_raises("guard/bare string raises", TypeError, lambda: f.predict_batch("nope", QUESTIONS))
+f = make_fake()
+check_raises("guard/bare dict raises", TypeError, lambda: f.predict_batch({"body": "x"}, QUESTIONS))
+
+
+def normalise_state(ctx):
+ ctx.states = [ctx.states]
+
+
+f = make_fake()
+f.predict_batch("nope", QUESTIONS, on_predict_start=normalise_state)
+check("guard/a hook can normalise a bare string", f._encode_states, ["nope"])
+
+
+# --------------------------------------------------------------- hand-built instance
+f = make_fake()
+res = f.predict_batch(["s0", "s1", "s2"], QUESTIONS)
+check("defaults/unset hooks are a no-op", len(res), 3)
+check("defaults/offsets unchanged", [r["answers"]["_offset"] for r in res], [0, NQ, 2 * NQ])
+
+
+# --------------------------------------------------------------- dynamic registration
+class Tag:
+ def __init__(self, log, tag):
+ self.log = log
+ self.tag = tag
+
+ def on_predict_end(self, ctx):
+ self.log.append(self.tag)
+
+
+log = []
+f = make_fake()
+f.add_hook(Tag(log, "a"))
+f.predict_batch(["s0"], QUESTIONS)
+check("add_hook/fires", log, ["a"])
+check("remove_hook/returns True", f.remove_hook(f.hooks[-1]), True)
+log.clear()
+f.predict_batch(["s0"], QUESTIONS)
+check("remove_hook/no longer fires", log, [])
+check("remove_hook/unknown returns False", f.remove_hook(object()), False)
+
+log = []
+f = make_fake()
+check_true("add_hook/returns self", f.add_hook([Tag(log, "x"), Tag(log, "y")]) is f)
+f.predict_batch(["s0"], QUESTIONS)
+check("add_hook/sequence order", log, ["x", "y"])
+
+log = []
+f = make_fake()
+f.add_hook(Tag(log, "installed"))
+with f.hooks_installed(Tag(log, "temp")):
+ f.predict_batch(["s0"], QUESTIONS)
+f.predict_batch(["s0"], QUESTIONS)
+check("hooks_installed/only during the block", log, ["installed", "temp", "installed"])
+
+log = []
+r = Router()
+r.add_hook(Tag(log, "router"))
+r.attach("english", FakeAgent())
+r.predict("hello", QUESTIONS)
+check("router/add_hook fires", log, ["router"])
+
+
+# --------------------------------------------------------------- per-call token budget
+def make_len_fake():
+ fake = make_fake()
+ fake._seen = []
+
+ def _encode(state, ids, internal, max_len=None, head_max_len=None):
+ fake._seen.append((max_len, head_max_len))
+ return [{"ids": [1, 2, 3], "markers": [0, 1], "qtype": 2} for _ in ids]
+
+ fake._encode_state = _encode
+ return fake
+
+
+f = make_len_fake()
+f.predict_batch(["s"], QUESTIONS, max_len=128, head_max_len=64)
+check("budget/per-call kwargs reach _encode_state", f._seen, [(128, 64)])
+
+f = make_len_fake()
+f.predict_batch(["s"], QUESTIONS, on_predict_start=lambda c: setattr(c, "max_len", 200))
+check("budget/hook-set ctx reaches _encode_state", f._seen, [(200, None)])
+
+f = make_len_fake()
+f.predict_batch(["s"], QUESTIONS)
+check("budget/default passes no override", f._seen, [(None, None)])
+
+
+class LenFake:
+ def __init__(self):
+ self.seen = []
+
+ def system_one(self, state, questions, max_len=None, head_max_len=None):
+ self.seen.append((max_len, head_max_len))
+ return {"model": "x", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}
+
+
+lf = LenFake()
+r = Router()
+r.attach("english", lf)
+r.predict("hello", QUESTIONS, max_len=256, head_max_len=128)
+check("budget/router per-call reaches the agent", lf.seen, [(256, 128)])
+
+lf = LenFake()
+r = Router()
+r.attach("english", lf)
+r.predict("hello", QUESTIONS, on_predict_start=lambda c: setattr(c, "head_max_len", 96))
+check("budget/router hook-set reaches the agent", lf.seen, [(None, 96)])
+
+
+class StrictFake:
+ """An agent-like object that does not accept the override kwargs."""
+
+ def system_one(self, state, questions):
+ return {"model": "x", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}
+
+
+r = Router()
+r.attach("english", StrictFake())
+r.predict("hello", QUESTIONS)
+check("budget/default does not pass override kwargs", True, True)
+
+
+# --------------------------------------------------------------- context semantics
+from laya.hooks import PredictContext # noqa: E402
+
+_c1 = PredictContext(states=[], questions={})
+_c2 = PredictContext(states=[], questions={})
+check_true("context/hashable by identity", isinstance(hash(_c1), int))
+check("context/two contexts are never equal", _c1 == _c2, False)
+
+
+# --------------------------------------------------------------- validation
+class NotAHook:
+ pass
+
+
+check_raises("validation/hooks= rejects a class", TypeError, lambda: normalise_hooks(hooks=[NotAHook]))
+check_raises("validation/hooks= rejects a plain callable", TypeError,
+ lambda: normalise_hooks(hooks=[lambda ctx: None]))
+check_raises("validation/on_predict_start must be callable", TypeError,
+ lambda: normalise_hooks(on_predict_start=123))
+
+
+class BadMethod:
+ on_predict_start = 5
+
+
+check_raises("validation/hook method must be callable", TypeError,
+ lambda: normalise_hooks(hooks=[BadMethod()]))
+
+
+# --------------------------------------------------------------- Router
+class RouteHook:
+ def __init__(self):
+ self.decisions = []
+
+ def on_route(self, ctx):
+ self.decisions.append(dict(ctx.decision))
+ ctx.decision = RouteDecision(model="multilingual", repo="convaiinnovations/laya/multilingual",
+ reason="pinned by hook", detection=None, workflow=None)
+
+
+rh = RouteHook()
+r = Router(hooks=[rh])
+decision = r.route("hello", QUESTIONS)
+check("router/on_route fired", len(rh.decisions), 1)
+check("router/on_route saw the original", rh.decisions[0]["model"], "english")
+check("router/on_route can replace the decision", decision["model"], "multilingual")
+
+
+class LoadHook:
+ def __init__(self):
+ self.loads = []
+ self.evicts = []
+
+ def on_load(self, ctx):
+ self.loads.append(ctx.model)
+
+ def on_evict(self, ctx):
+ self.evicts.append(ctx.model)
+
+
+class BuiltAgent(FakeAgent):
+ def __init__(self, *args, **kwargs):
+ pass
+
+
+lh = LoadHook()
+real_agent = _agent_mod.Agent
+_agent_mod.Agent = BuiltAgent
+try:
+ r = Router(max_loaded=1, hooks=[lh])
+ r.load("english")
+ r.load("multilingual") # evicts english
+finally:
+ _agent_mod.Agent = real_agent
+check("router/on_load fired per build", lh.loads, ["english", "multilingual"])
+check("router/on_evict fired on eviction", lh.evicts, ["english"])
+
+
+predict_seen = {}
+r = Router(on_predict_start=lambda ctx: predict_seen.update(decision=dict(ctx.decision)),
+ on_predict_end=lambda ctx: predict_seen.update(results=ctx.results))
+r.attach("english", FakeAgent())
+out = r.predict("hello", QUESTIONS)
+check("router/predict start sees the decision", predict_seen["decision"]["model"], "english")
+check("router/predict end sees results", len(predict_seen["results"]), 1)
+check("router/predict keeps routing", out["routing"]["model"], "english")
+
+cached = [{"model": "cached", "answers": {}, "usage": {"input_tokens": 0, "output_tokens": 0}}]
+r = Router()
+r.attach("english", FakeAgent())
+skipped = r.predict("hello", QUESTIONS, on_predict_start=lambda c: c.skip(cached))
+check("router/skip returns the cached payload", skipped, cached[0])
+check("router/skip still adds routing", skipped.get("routing", {}).get("model"), "english")
+
+
+class PerCallRoute:
+ def __init__(self):
+ self.decisions = []
+
+ def on_route(self, ctx):
+ self.decisions.append(dict(ctx.decision))
+
+
+pcr = PerCallRoute()
+r = Router()
+r.attach("english", FakeAgent())
+r.predict("hello", QUESTIONS, hooks=[pcr])
+check("router/per-call hooks apply to on_route", len(pcr.decisions), 1)
+
+
+# --------------------------------------------------------------- hooks_concurrent storage
+r = Router()
+check("router/hooks_concurrent default True", r.hooks_concurrent, True)
+check_true("router/hooks_concurrent default has no lock", r._hooks_lock is None)
+r = Router(hooks_concurrent=False)
+check("router/hooks_concurrent=False stored", r.hooks_concurrent, False)
+check_true("router/hooks_concurrent=False installs a lock", r._hooks_lock is not None)
+
+
+# --------------------------------------------------------------- concurrency
+contexts = []
+
+
+def slow_start(ctx):
+ time.sleep(0.01)
+ contexts.append(id(ctx))
+
+
+f = make_fake()
+threads = [threading.Thread(target=lambda: f.predict_batch(["s0"], QUESTIONS,
+ on_predict_start=slow_start))
+ for _ in range(5)]
+for t in threads:
+ t.start()
+for t in threads:
+ t.join()
+check("concurrency/one independent context per call", len(set(contexts)), 5)
+
+
+class ConcHook:
+ def __init__(self):
+ self.active = 0
+ self.max_active = 0
+ self._lock = threading.Lock()
+
+ def on_predict_start(self, ctx):
+ with self._lock:
+ self.active += 1
+ self.max_active = max(self.max_active, self.active)
+ time.sleep(0.02)
+ with self._lock:
+ self.active -= 1
+
+
+conc = ConcHook()
+f = make_fake()
+f.hooks = [conc]
+f._hooks_lock = threading.RLock() # what hooks_concurrent=False installs
+threads = [threading.Thread(target=lambda: f.predict_batch(["s0"], QUESTIONS)) for _ in range(5)]
+for t in threads:
+ t.start()
+for t in threads:
+ t.join()
+check("concurrency/hooks_concurrent=False serialises", conc.max_active, 1)
+
+
+# --------------------------------------------------------------- ONNXAgent parity
+from laya.onnx_agent import ONNXAgent # noqa: E402
+
+o = ONNXAgent.__new__(ONNXAgent)
+o.model_id = "convaiinnovations/laya-onnx"
+o._infer = lambda state, questions: {"model": "onnx", "answers": {},
+ "usage": {"input_tokens": 0, "output_tokens": 0}}
+onnx_seen = []
+onnx_models = []
+onnx_out = o.system_one("s", QUESTIONS,
+ on_predict_start=lambda c: onnx_seen.append("start"),
+ on_predict_end=lambda c: (onnx_seen.append("end"), onnx_models.append(c.model)))
+check("onnx/hooks fire", onnx_seen, ["start", "end"])
+check("onnx/returns the inference result", onnx_out["model"], "onnx")
+check("onnx/context model is the agent model_id", onnx_models, ["convaiinnovations/laya-onnx"])
+
+o = ONNXAgent.__new__(ONNXAgent)
+
+
+def _never(*args):
+ raise AssertionError("inference should have been skipped")
+
+
+o._infer = _never
+check("onnx/skip short-circuits",
+ o.system_one("s", QUESTIONS, on_predict_start=lambda c: c.skip([{"model": "cached"}])),
+ {"model": "cached"})
+
+
+# --------------------------------------------------------------- report
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f_ in FAIL:
+ print(" FAIL " + f_)
+if not FAIL:
+ print("all hook tests passed")
+sys.exit(1 if FAIL else 0)
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
new file mode 100644
index 00000000..fd283504
--- /dev/null
+++ b/tests/test_hooks_api.py
@@ -0,0 +1,148 @@
+"""API-stability guard for prediction hooks.
+
+These tests pin the public hook surface (parameter names, kinds, defaults, context fields,
+lifecycle events, exports) so a change that would break callers fails here first. If a change
+is intentional, update this file in the same commit.
+
+Run: python tests/test_hooks_api.py
+"""
+import dataclasses
+import inspect
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import laya # noqa: E402
+from laya import Agent, PredictContext, PredictHook, Router, load # noqa: E402
+from laya.hooks import HOOK_EVENTS, Hook # noqa: E402
+from laya.onnx_agent import ONNXAgent # noqa: E402
+
+PASS, FAIL = [], []
+
+
+def check(name, got, want):
+ if got == want:
+ PASS.append(name)
+ else:
+ FAIL.append("%s: got %r, want %r" % (name, got, want))
+
+
+def check_true(name, cond, detail=""):
+ if cond:
+ PASS.append(name)
+ else:
+ FAIL.append("%s%s" % (name, ": " + detail if detail else ""))
+
+
+def sig(fn):
+ return inspect.signature(fn).parameters
+
+
+def check_param(name, fn, param, default, kind=None):
+ params = sig(fn)
+ if param not in params:
+ FAIL.append("%s/%s: missing parameter" % (name, param))
+ return
+ p = params[param]
+ check("%s/%s default" % (name, param), p.default, default)
+ if kind is not None:
+ check("%s/%s kind" % (name, param), p.kind, kind)
+
+
+HOOK_KEYS = {
+ "hooks": None,
+ "on_predict_start": None,
+ "on_predict_end": None,
+ "hooks_raise": True,
+ "hooks_concurrent": True,
+}
+
+# --------------------------------------------------------------- constructors
+for label, fn in (("Agent.__init__", Agent.__init__), ("load", load),
+ ("Router.__init__", Router.__init__), ("ONNXAgent.__init__", ONNXAgent.__init__)):
+ for param, default in HOOK_KEYS.items():
+ check_param(label, fn, param, default)
+
+# Router keeps lang_guess too
+check_param("Router.__init__", Router.__init__, "lang_guess", None)
+
+# --------------------------------------------------------------- predict surfaces
+for label, fn in (("Agent.predict_batch", Agent.predict_batch),
+ ("Agent.system_one", Agent.system_one),
+ ("Router.predict", Router.predict),
+ ("ONNXAgent.system_one", ONNXAgent.system_one)):
+ check_param(label, fn, "hooks", None)
+ check_param(label, fn, "on_predict_start", None)
+ check_param(label, fn, "on_predict_end", None)
+ check_param(label, fn, "hooks_raise", None)
+
+check_param("Agent.predict_batch", Agent.predict_batch, "batch_size", None)
+
+# per-call token-budget overrides
+for label, fn in (("Agent.predict_batch", Agent.predict_batch),
+ ("Agent.system_one", Agent.system_one),
+ ("Router.predict", Router.predict),
+ ("ONNXAgent.system_one", ONNXAgent.system_one)):
+ check_param(label, fn, "max_len", None)
+ check_param(label, fn, "head_max_len", None)
+
+# route() takes per-call hooks so a hook can pin a checkpoint for one call
+check_param("Router.route", Router.route, "hooks", None)
+check_param("Router.route", Router.route, "hooks_raise", None)
+
+# --------------------------------------------------------------- aliases
+check_true("Agent.predict is Agent.system_one", Agent.predict is Agent.system_one)
+check_true("Router.system_one is Router.predict", Router.system_one is Router.predict)
+check_true("ONNXAgent.predict is ONNXAgent.system_one", ONNXAgent.predict is ONNXAgent.system_one)
+
+# --------------------------------------------------------------- context
+FIELDS = ["states", "questions", "run_id", "results", "decision", "model", "agent", "router",
+ "max_len", "head_max_len", "usage", "started_at", "elapsed_ms", "error"]
+check("PredictContext fields", [f.name for f in dataclasses.fields(PredictContext)], FIELDS)
+check("PredictContext/states required", PredictContext.__dataclass_fields__["states"].default,
+ dataclasses.MISSING)
+check("PredictContext/questions required", PredictContext.__dataclass_fields__["questions"].default,
+ dataclasses.MISSING)
+for optional in ("results", "decision", "model", "agent", "router", "usage", "elapsed_ms", "error"):
+ check("PredictContext/%s default None" % optional,
+ PredictContext.__dataclass_fields__[optional].default, None)
+check_true("PredictContext/skip exists", callable(getattr(PredictContext, "skip", None)))
+check_true("PredictContext/run_id has a factory",
+ PredictContext.__dataclass_fields__["run_id"].default_factory is not dataclasses.MISSING)
+
+# --------------------------------------------------------------- hook protocol
+check("Hook lifecycle events", set(HOOK_EVENTS),
+ {"on_predict_start", "on_predict_end", "on_route", "on_load", "on_evict", "on_error"})
+for event in HOOK_EVENTS:
+ check_true("Hook/%s declared" % event, hasattr(Hook, event))
+check_true("PredictHook is callable-typed", callable(PredictHook))
+
+# --------------------------------------------------------------- exports
+for name in ("PredictContext", "PredictHook", "Hook"):
+ check_true("__all__/%s" % name, name in laya.__all__)
+ check_true("laya.%s exists" % name, hasattr(laya, name))
+
+# --------------------------------------------------------------- class defaults
+for label, cls in (("Agent", Agent), ("Router", Router), ("ONNXAgent", ONNXAgent)):
+ check("%s/hooks default" % label, cls.hooks, ())
+ check("%s/hooks_raise default" % label, cls.hooks_raise, True)
+ check("%s/hooks_concurrent default" % label, cls.hooks_concurrent, True)
+ check("%s/_hooks_lock default" % label, cls._hooks_lock, None)
+
+# Agent and ONNXAgent carry the checkpoint id for ctx.model; Router has no single model.
+for label, cls in (("Agent", Agent), ("ONNXAgent", ONNXAgent)):
+ check("%s/model_id default" % label, cls.model_id, None)
+
+# runtime registration surface
+for label, cls in (("Agent", Agent), ("Router", Router), ("ONNXAgent", ONNXAgent)):
+ for method in ("add_hook", "remove_hook", "hooks_installed"):
+ check_true("%s/%s exists" % (label, method), callable(getattr(cls, method, None)))
+
+
+print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
+for f in FAIL:
+ print(" FAIL", f)
+if not FAIL:
+ print("all hook API tests passed")
+sys.exit(1 if FAIL else 0)