Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
Merge pull request #606 from hemanthrayuduu/fix/router-predict-batch-followups
Nandakishord3863bdade9c
diff --cc docs/hooks/lifecycle.mdindex b4209515,692358a2..a4a11287--- a/docs/hooks/lifecycle.md+++ b/docs/hooks/lifecycle.md@@@ -141,20 -140,26 +142,30 @@@ Key points- Every start hook of a checkpoint's requests runs before any of their end hooks, because theyshare forward passes. A cache that fills in `on_predict_end` therefore cannot serve a duplicatestate within the same checkpoint group; it can across calls.+ - For the same reason the requests end in reverse of the order they started, so a hook that sets+ something in start and resets it in end (a `contextvars` value, an OpenTelemetry+ `context.attach` / `detach`) unwinds to the value it found.- A start hook that replaces `ctx.states`, `ctx.questions` or the token budget changes its ownrequest only: requests are grouped for the forward pass after their start hooks have run.- Mutating a questions dict in place changes it for every request that shares that dict, and for- the caller, as it would with `predict`.+ Mutating a shared questions dict in place is different, and not what `predict` does: nothing of+ the group is inferred until all of its start hooks have run, so the change reaches every request+ that shares the dict, including those whose start hooks ran earlier, and the caller too. Assign+ a new dict to `ctx.questions` instead.+ - The checkpoint name is resolved before grouping, so an `on_route` hook that pins a request by+ an alias (`"ml"`) shares that checkpoint's forward pass, and `ctx.model` is the resolved name.+- A request may carry its own `max_len` / `head_max_len`, the per-request form of the token budget+ `predict` takes as call arguments. A start hook that sets `ctx.max_len` overrides it, because the+ hook runs after the context is built. Requests that ask for different budgets cannot share a+ forward pass, so a batch that mixes budgets makes one `agent.predict_batch` call per budget.- - Every started request gets exactly one `on_predict_end`, even when an earlier request's end hook+ - Every started request gets exactly one `on_predict_end`, even when another request's end hookraises; the first such error is raised after all of them have run.- - If the batch fails, a request that did not get a result is reported as failed (`on_error`, with- `ctx.error` set to the exception that failed the batch), because the caller gets no result for- it. Requests of checkpoint groups that already finished have ended with their results, as the- earlier calls of `[router.predict(...) for ...]` would have.+ - If a checkpoint group fails, every one of its started requests fails with the exception: each+ gets `on_error` with `ctx.error` set to it, then `on_predict_end`. That includes a cache hit and+ a request whose question group had already run, because the caller gets the exception and no+ result for any of them, and it means `ctx.error` may be another request's failure (a start hook+ that raises for one request fails its group). Requests of checkpoint groups that already+ finished have ended with their results, as the earlier calls of `[router.predict(...) for ...]`+ would have.## Model lifecyclediff --cc laya/router.pyindex f673b5a9,a9f8fa6e..404f700b--- a/laya/router.py+++ b/laya/router.py@@@ -1133,30 -860,21 +1138,27 @@@ class Router(HookRegistry)for (i, ctx), result in zip(items, batch_results):result["routing"] = dict(decisions[i])ctx.results = [result]++ # Usage is summed here, not while ending, so a malformed usage block (a cached+ # payload's, say) fails the group like any other error instead of escaping from+ # the end loop before a single end hook has run.+ usages = [aggregate_usage(ctx.results) for ctx in started]+ for ctx, usage in zip(started, usages):+ ctx.usage = usageexcept BaseException as exc:- # Every started request is ended, so a hook that opens something in start (a- # span, an in-flight count) always sees the matching end. A request that already- # has its result keeps it; the rest failed with the batch.- for ctx in started:- if ctx.results is None:- ctx.error = exc- try:- dispatch(active, "on_error", ctx, raise_errors=raise_errors,- lock=self._hooks_lock, timeout=timeout)- except BaseException as hook_exc:- exc.__context__ = hook_exc+ # The caller gets this exception and no result, so every started request of the+ # group failed with it, including a cache hit or a request whose question group+ # had already run. Each is still ended, so a hook that opens something in start+ # (a span, an in-flight count) always sees the matching end.+ for ctx in started:+ if mc is not None and ctx.results:+ flag_low_confidence(ctx.results, mc)- try:- self._end_contexts(active, started, raise_errors, timeout)- except BaseException as hook_exc:- exc.__context__ = hook_exc+ self._end_contexts(active, started, raise_errors, timeout, error=exc)raise+ for ctx in started:+ if mc is not None and ctx.results:+ flag_low_confidence(ctx.results, mc)self._end_contexts(active, started, raise_errors, timeout)for i, ctx in zip(indices, started):results[i] = ctx.results[0]