Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge pull request #606 from hemanthrayuduu/fix/router-predict-batch-followups

Nandakishord3863bdade9c
diff --cc docs/hooks/lifecycle.md
index b4209515,692358a2..a4a11287
--- a/docs/hooks/lifecycle.md
+++ b/docs/hooks/lifecycle.md
@@@ -141,20 -140,26 +142,30 @@@ Key points
- Every start hook of a checkpoint's requests runs before any of their end hooks, because they
share forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicate
state within the same checkpoint group; it can across calls.
+ - For the same reason the requests end in reverse of the order they started, so a hook that sets
+ something in start and resets it in end (a `contextvars` value, an OpenTelemetry
+ `context.attach` / `detach`) unwinds to the value it found.
- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its own
request only: requests are grouped for the forward pass after their start hooks have run.
- Mutating a questions dict in place changes it for every request that shares that dict, and for
- the caller, as it would with `predict`.
+ Mutating a shared questions dict in place is different, and not what `predict` does: nothing of
+ the group is inferred until all of its start hooks have run, so the change reaches every request
+ that shares the dict, including those whose start hooks ran earlier, and the caller too. Assign
+ a new dict to `ctx.questions` instead.
+ - The checkpoint name is resolved before grouping, so an `on_route` hook that pins a request by
+ an alias (`"ml"`) shares that checkpoint's forward pass, and `ctx.model` is the resolved name.
+- A request may carry its own `max_len` / `head_max_len`, the per-request form of the token budget
+ `predict` takes as call arguments. A start hook that sets `ctx.max_len` overrides it, because the
+ hook runs after the context is built. Requests that ask for different budgets cannot share a
+ forward pass, so a batch that mixes budgets makes one `agent.predict_batch` call per budget.
- - Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook
+ - Every started request gets exactly one `on_predict_end`, even when another request's end hook
raises; the first such error is raised after all of them have run.
- - If the batch fails, a request that did not get a result is reported as failed (`on_error`, with
- `ctx.error` set to the exception that failed the batch), because the caller gets no result for
- it. Requests of checkpoint groups that already finished have ended with their results, as the
- earlier calls of `[router.predict(...) for ...]` would have.
+ - If a checkpoint group fails, every one of its started requests fails with the exception: each
+ gets `on_error` with `ctx.error` set to it, then `on_predict_end`. That includes a cache hit and
+ a request whose question group had already run, because the caller gets the exception and no
+ result for any of them, and it means `ctx.error` may be another request's failure (a start hook
+ that raises for one request fails its group). Requests of checkpoint groups that already
+ finished have ended with their results, as the earlier calls of `[router.predict(...) for ...]`
+ would have.
## Model lifecycle
diff --cc laya/router.py
index f673b5a9,a9f8fa6e..404f700b
--- a/laya/router.py
+++ b/laya/router.py
@@@ -1133,30 -860,21 +1138,27 @@@ class Router(HookRegistry)
for (i, ctx), result in zip(items, batch_results):
result["routing"] = dict(decisions[i])
ctx.results = [result]
+
+ # Usage is summed here, not while ending, so a malformed usage block (a cached
+ # payload's, say) fails the group like any other error instead of escaping from
+ # the end loop before a single end hook has run.
+ usages = [aggregate_usage(ctx.results) for ctx in started]
+ for ctx, usage in zip(started, usages):
+ ctx.usage = usage
except BaseException as exc:
- # Every started request is ended, so a hook that opens something in start (a
- # span, an in-flight count) always sees the matching end. A request that already
- # has its result keeps it; the rest failed with the batch.
- for ctx in started:
- if ctx.results is None:
- ctx.error = exc
- try:
- dispatch(active, "on_error", ctx, raise_errors=raise_errors,
- lock=self._hooks_lock, timeout=timeout)
- except BaseException as hook_exc:
- exc.__context__ = hook_exc
+ # The caller gets this exception and no result, so every started request of the
+ # group failed with it, including a cache hit or a request whose question group
+ # had already run. Each is still ended, so a hook that opens something in start
+ # (a span, an in-flight count) always sees the matching end.
+ for ctx in started:
+ if mc is not None and ctx.results:
+ flag_low_confidence(ctx.results, mc)
- try:
- self._end_contexts(active, started, raise_errors, timeout)
- except BaseException as hook_exc:
- exc.__context__ = hook_exc
+ self._end_contexts(active, started, raise_errors, timeout, error=exc)
raise
+ for ctx in started:
+ if mc is not None and ctx.results:
+ flag_low_confidence(ctx.results, mc)
self._end_contexts(active, started, raise_errors, timeout)
for i, ctx in zip(indices, started):
results[i] = ctx.results[0]