Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(java): Router.predictBatch, so a router is a BatchPredictor
Adeshina Lasisid76eacb25bf9
diff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex 43a1d72c..d01263dc 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -154,17 +154,17 @@ jobs:# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest# run. `--min-tests` is what stops an aborted suite from looking like a pass.#- # Measured on the merged tree: 1002 register here, 985 run, 17 abort on a checkpoint this+ # Measured on the merged tree: 1021 register here, 1003 run, 18 abort on a checkpoint this# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes- # `TestJvmVersionTest` run instead of skip, so it executes 986. Same suite, one more test,+ # `TestJvmVersionTest` run instead of skip, so it executes 1004. Same suite, one more test,# because the flag enables it. The step below is where the number is justified.- name: Tests that need no modelworking-directory: laya-javarun: ./gradlew --no-daemon test- # Measured on the merged tree, Corretto 17, no checkpoints: 1002 registered, 17 abort,- # 985 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in+ # Measured on the merged tree, Corretto 17, no checkpoints: 1021 registered, 18 abort,+ # 1003 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this# replaced. A margin only binds classes larger than itself, and three classes in this lane# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no@@ -179,7 +179,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 985 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 1003 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@ -335,7 +335,7 @@ jobs:# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice# and reported it as two JDKs. With `--require-passed` it exits 1.#- # 986, not the build lane's 985, and the one is the point: this lane runs one test MORE+ # 1004, not the build lane's 1003, and the one is the point: this lane runs one test MORE# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --# three classes here run exactly one test, so no positive margin can bind them.@@ -356,7 +356,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 986 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 1004 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.ymlindex deeba7e3..d41a89ee 100644--- a/.github/workflows/release-java.yml+++ b/.github/workflows/release-java.yml@@ -67,7 +67,7 @@ jobs:# that lane already covered.- name: Model-free testsrun: ./gradlew --no-daemon test- # 985 of the 1002 registered tests run without a checkpoint. This floor has been raised+ # 1003 of the 1021 registered tests run without a checkpoint. This floor has been raised# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor# that does not bind is not a gate: under 625 the whole confidence-gate family of 26# tests could have vanished from this lane with the gate still green.@@ -83,7 +83,7 @@ jobs:- name: The suite was not vacuousrun: >python3 scripts/check_test_results.py laya-java/build/test-results/test- --min-tests 985 --allow-aborted+ --min-tests 1003 --allow-aborted- name: Build the jarsrun: ./gradlew --no-daemon build -x testdiff --git a/laya-java/README.md b/laya-java/README.mdindex d5c3f01d..d8119f05 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -192,6 +192,23 @@ evidence of English. A deployment whose traffic is known to be English can setenglish, as the reference's alias table does: `load("default")` loads english whatever`defaultCheckpoint` is set to.+`predictBatch` routes many requests and answers them with one load per checkpoint used:++```java+List<Prediction> out = router.predictBatch(states, questions); // each state routed alone+List<Prediction> mixed = router.predictBatch(List.of(+ Router.Request.of(email, Presets.email()),+ Router.Request.of(ticket, Presets.triage())+ .options(Router.RouteOptions.none().model("multilingual"))+ .maxLen(512)));+```++Requests are routed first, grouped by checkpoint in order of first appearance, and within a+checkpoint those sharing a question schema and token budget share `Agent.predictBatch` calls.+Results come back in request order and equal per-request `predict`. `Router` is a+`BatchPredictor`, so `Decisions.decideBatch(router, states, schema)` works; to pin it, pass+`(s, q) -> router.predictBatch(s, q, options)` as the runner.+## Detection on its own`lang.LanguageDetection` is the router's evidence, usable without one:@@ -498,14 +515,25 @@ reaches `onError` and `onPredictEnd`, with no start event, and its `elapsedMs` cand the failed load. The agent's own pair stillfires for the forward pass, so a router-level hook and an agent-level hook see different spans.+`Router.predictBatch` dispatches one pair **per request**, as `predict` does, so a start hook can+rewrite or `skip` a request before it joins a shared forward pass. The order is the reference's:+every `onRoute`, then per checkpoint its `onEvict`/`onLoad`, its requests' starts in order and+their ends in reverse. If a checkpoint's inference fails, each of its started requests gets+`onError` and its end before the failure propagates. A routing failure reaches that request, and a+loading failure every request of that checkpoint, as `onError` and `onPredictEnd` with no start —+as on `predict`, where the reference's `predict_batch` reports neither. The agent's pair fires once per forward+pass, not per request. Unlike `predict`, the batch path honours `ctx.maxLen` / `ctx.headMaxLen`+from a router-level start hook, since a request's own budget is seeded there.+Hooks are dispatched outside the router's lock, so a hook may call back into the router —`loaded()`, `load`, `unload` — without deadlocking. Two consequences of that are yours to handle:`load()` can hand back an agent a hook unloaded while its `on_load` was being dispatched, and arouter-level `on_predict_start` that calls `router.predict` recurses until the stack runs out,because that inner call dispatches the same hook again.-`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured:-`Agent.predict` takes no budget arguments. An agent-level hook can still set them.+`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured by+`Router.predict`: `Agent.predict` takes no budget arguments. `predictBatch` honours them, and an+agent-level hook can set them on either path.```javarouter.hooks().addHook(new Hook() {@@ -518,11 +546,11 @@ router.hooks().addHook(new Hook() {| | ||---|---|-| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |+| a process-wide default hook sees **two** predict pairs per `Router.predict` (per request plus per forward pass on `predictBatch`) | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both || a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure || calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading || `concurrent(false)` with an `AsyncHook` whose callback dispatches a hook on the same registry | deadlocks with no deadline, and stalls until it with one. Serial dispatch holds a lock across the callback, and the callback is on another thread, so the inner dispatch cannot re-enter that lock. A callback that dispatches nothing — `loaded()`, a cached `load` — is fine |-| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |+| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored by `Router.predict`, which calls `Agent.predict` with no budget arguments; `predictBatch` honours the budgets. Both answer only the first state |`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlateon the model name, not on `runId`. That context holds the chosen checkpoint in `ctx.model()` and nodiff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.javaindex dc27c223..84511191 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java@@ -231,17 +231,30 @@ public final class Agent implements AutoCloseable, Predictor, BatchPredictor {public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,String language, int batchSize, boolean sortByLength,HookCall call) {+ return predictBatch(states, questions, language, batchSize, sortByLength, call, null,+ null);+ }++ /**+ * {@link #predictBatch(List, Map, String, int, boolean, HookCall)} with the token budgets+ * seeded on the context, so this agent's own start hooks can still replace them. For+ * {@link Router#predictBatch(List)}, which carries a per-request budget.+ */+ List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,+ String language, int batchSize, boolean sortByLength,+ HookCall call, Integer maxLen, Integer headMaxLen) {// Before any hook: a closed agent cannot answer, and dispatching a start hook that then// watched the call fail would report a prediction that was never going to happen.if (closed) {throw new IllegalStateException("this agent is closed; open a new one rather than reusing it");}- PredictContext ctx = new PredictContext(states, questions, modelName, this);+ PredictContext ctx = new PredictContext(states, questions, modelName, this, maxLen,+ headMaxLen);return Hooks.around(hooks.composeFor(call), ctx, hooks.policyFor(call),- (hooked, asked, maxLen, headMaxLen) ->- infer(hooked, asked, language, batchSize, sortByLength, maxLen,- headMaxLen));+ (hooked, asked, tokens, headTokens) ->+ infer(hooked, asked, language, batchSize, sortByLength, tokens,+ headTokens));}/**diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.javaindex 2f4c00d2..33291678 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java@@ -7,14 +7,13 @@ import java.util.Map;* Anything that can answer one question set over many states in a shared forward pass.** <p>Separate from {@link Predictor} rather than folded into it, because not every runtime- * batches. {@link Agent} does; {@link Router} does not — it selects a checkpoint per state, so- * there is no single graph call to share. The reference draws the same line at runtime, raising- * {@code TypeError} when a runner has no {@code predict_batch} rather than looping {@code decide}- * N times behind the caller's back; here the line is drawn by the type, so the same mistake does- * not compile.+ * batches. The reference draws the same line at runtime, raising {@code TypeError} when a runner+ * has no {@code predict_batch} rather than looping {@code decide} N times behind the caller's+ * back; here the line is drawn by the type, so the same mistake does not compile.*- * <p>{@link Agent#predictBatch(List, Map)} already had this signature, so this interface names an- * existing capability rather than adding one.+ * <p>{@link Agent} shares one graph call across the states. {@link Router} routes each state and+ * shares one call per checkpoint, so states that route apart are still answered together where+ * they can be.*/public interface BatchPredictor {diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.javaindex d9f98f29..12747086 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java@@ -67,12 +67,12 @@ import java.util.Set;* the one the reference's own {@code decide_batch} always takes.</li>* <li>{@code routing} is not reported. {@link Prediction} carries no routing block, so there is* nothing to pass through; {@link Router} reports its decision at the point it makes it.</li>- * <li>A <b>batched</b> decision cannot route per state. The reference's {@code decide_batch}- * special-cases a {@code route_batch}-capable runner and sends one request per state "so- * states may route to different checkpoints"; {@link Router} here implements- * {@link Predictor} and not {@link BatchPredictor}, so {@link #decideBatch} will not take- * one at all. Batching a router means deciding what a shared forward pass over two- * checkpoints is, which is a design question and not a port.</li>+ * <li>A batched decision on a {@link Router} takes no routing keywords. The reference's+ * {@code decide_batch} copies {@code model}, {@code task}, {@code lang},+ * {@code lang_guess} and the token budgets onto every request; here a router routes each+ * state on its own. A pin is a lambda runner,+ * {@code (states, asked) -> router.predictBatch(states, asked, options)}; a budget is a+ * list of {@link Router.Request}s.</li>* <li>An {@code enum} must be an <b>array</b>. The reference measures and iterates whatever* is there, so {@code "enum": "abc"} becomes a three-option choice over {@code "a"},* {@code "b"}, {@code "c"} — a schema nobody wrote, built from a typo — anddiff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.javaindex 587672b5..9302499f 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java@@ -12,8 +12,10 @@ import java.io.IOException;import java.io.UncheckedIOException;import java.nio.file.Path;import java.util.ArrayList;+import java.util.Arrays;import java.util.Collection;import java.util.Collections;+import java.util.IdentityHashMap;import java.util.LinkedHashMap;import java.util.LinkedHashSet;import java.util.List;@@ -46,7 +48,7 @@ import java.util.function.Supplier;* <p>This class decides; it does not load. {@link #route} runs no model and touches no disk, so it* is safe to call on every request and to test without a checkpoint.*/-public final class Router implements AutoCloseable, Predictor {+public final class Router implements AutoCloseable, Predictor, BatchPredictor {/** The hub repository that bundles all three checkpoints. */public static final String BUNDLE_REPO = "convaiinnovations/laya";@@ -151,6 +153,45 @@ public final class Router implements AutoCloseable, Predictor {}}+ /**+ * One request of a {@link #predictBatch(List)}: a state, its questions, and the routing and+ * token-budget overrides the reference reads off each request dict.+ *+ * @param options routing for this request alone; null means {@link RouteOptions#none()}+ * @param maxLen a token budget for this request alone, or null for the checkpoint's+ * @param headMaxLen a head budget for this request alone, or null for the checkpoint's+ */+ public record Request(Object state, Map<String, Question> questions, RouteOptions options,+ Integer maxLen, Integer headMaxLen) {++ /** @throws IllegalArgumentException for a null state or null questions */+ public Request {+ requireNonNull(state, "state");+ requireNonNull(questions, "questions");+ options = options == null ? RouteOptions.none() : options;+ }++ /** A request routed on its state and questions alone, at the checkpoint's budgets. */+ public static Request of(Object state, Map<String, Question> questions) {+ return new Request(state, questions, null, null, null);+ }++ /** This request with these routing options. */+ public Request options(RouteOptions value) {+ return new Request(state, questions, value, maxLen, headMaxLen);+ }++ /** This request with this token budget. */+ public Request maxLen(Integer value) {+ return new Request(state, questions, options, value, headMaxLen);+ }++ /** This request with this head budget. */+ public Request headMaxLen(Integer value) {+ return new Request(state, questions, options, maxLen, value);+ }+ }+/*** Builds the agent for a checkpoint.*@@ -1558,6 +1599,185 @@ public final class Router implements AutoCloseable, Predictor {return lang;}+ /**+ * Route every state and answer the shared questions, sharing forward passes per checkpoint.+ *+ * <p>{@link BatchPredictor}'s form, so {@link Decisions#decideBatch} takes a router. To pin+ * one, pass {@code (states, asked) -> router.predictBatch(states, asked, options)}.+ */+ @Override+ public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions) {+ return predictBatch(states, questions, RouteOptions.none());+ }++ /** {@link #predictBatch(List, Map)} with the same routing options on every state. */+ public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,+ RouteOptions options) {+ requireNonNull(states, "states");+ List<Request> requests = new ArrayList<>(states.size());+ for (Object state : states) {+ requests.add(new Request(state, questions, options, null, null));+ }+ return predictBatch(requests);+ }++ /** {@link #predictBatch(List, int, boolean)} with every state at once and no sorting. */+ public List<Prediction> predictBatch(List<Request> requests) {+ return predictBatch(requests, 0, false);+ }++ /**+ * Route each request and answer it, with one checkpoint load per checkpoint used.+ *+ * <p>Every request is routed first, then grouped by checkpoint in order of first appearance.+ * Within a checkpoint, requests that share a question schema, a token budget and (when the+ * checkpoint has per-language temperatures) a language share {@link Agent#predictBatch}+ * calls. Each request is answered exactly as {@link #predict(Object, Map, RouteOptions)}+ * would answer it.+ *+ * <p>Router hooks run per request, as {@link #predict} runs them, and in the reference's+ * order: every {@code on_route} first, then per checkpoint any {@code on_evict}/{@code on_load},+ * then that checkpoint's start events in request order, so a start hook can rewrite or skip a+ * request before it joins a forward pass, then its end events in reverse. If its inference+ * fails, every started request of that checkpoint gets the error and its end before the+ * failure propagates; checkpoints already answered have ended normally. A routing failure+ * reaches its own request, and a loading failure every request of that checkpoint, as+ * {@code on_error} and {@code on_predict_end} with no start, as in {@link #predict}.+ *+ * @param batchSize states per graph call, or 0 for each group at once+ * @param sortByLength forwarded to {@link Agent#predictBatch}; changes no answer+ * @return one prediction per request, in the order the requests were given+ */+ public List<Prediction> predictBatch(List<Request> requests, int batchSize,+ boolean sortByLength) {+ requireNonNull(requests, "requests");+ if (batchSize < 0) {+ throw new IllegalArgumentException("batchSize must be 0 or positive, got " + batchSize);+ }+ // Before any routing, as predict does: a malformed batch must not route, load or evict.+ for (int i = 0; i < requests.size(); i++) {+ if (requests.get(i) == null) {+ throw new IllegalArgumentException("request " + i + " is null");+ }+ }+ List<Hook> composed = hooks.composeFor(HookCall.none());+ Hooks.Policy policy = hooks.policyFor(HookCall.none());+ List<RouteDecision> decisions = new ArrayList<>(requests.size());+ for (Request request : requests) {+ long startedAt = System.nanoTime();+ try {+ decisions.add(route(request.state(), request.questions(), request.options()));+ } catch (RuntimeException | Error failure) {+ failedBeforeStart(composed, policy, request, null, failure, startedAt);+ throw failure;+ }+ }+ Map<Checkpoint, List<Integer>> groups = new LinkedHashMap<>();+ for (int i = 0; i < decisions.size(); i++) {+ groups.computeIfAbsent(decisions.get(i).model(), key -> new ArrayList<>()).add(i);+ }++ Prediction[] out = new Prediction[requests.size()];+ for (Map.Entry<Checkpoint, List<Integer>> group : groups.entrySet()) {+ Checkpoint checkpoint = group.getKey();+ // Loaded before the starts, as the reference loads, so elapsedMs excludes a cold load.+ long startedAt = System.nanoTime();+ Lease lease;+ try {+ lease = lease(checkpoint);+ } catch (RuntimeException | Error failure) {+ for (int i : group.getValue()) {+ failedBeforeStart(composed, policy, requests.get(i), checkpoint, failure,+ startedAt);+ }+ throw failure;+ }+ List<PredictContext> contexts = new ArrayList<>(group.getValue().size());+ Map<PredictContext, Integer> indexOf = new IdentityHashMap<>();+ try (Lease held = lease) {+ for (int i : group.getValue()) {+ Request request = requests.get(i);+ PredictContext ctx = new PredictContext(+ Collections.singletonList(request.state()), request.questions(),+ checkpoint.wireName(), this, request.maxLen(), request.headMaxLen());+ contexts.add(ctx);+ indexOf.put(ctx, i);+ }+ Hooks.aroundGroup(composed, contexts, policy, pending -> answer(held.agent(),+ pending, indexOf, requests, decisions, batchSize, sortByLength));+ }+ for (PredictContext ctx : contexts) {+ int i = indexOf.get(ctx);+ if (ctx.results() == null || ctx.results().isEmpty()) {+ // As in predict: only a hook can cause this, and inventing an answer would+ // report a decision the model never made.+ throw new IllegalStateException("a hook left no prediction for request " + i+ + "; ctx.skip(...) needs one result");+ }+ out[i] = ctx.results().get(0);+ }+ }+ // Not List.of: an end hook may leave a null result, which predict returns as it is.+ return Collections.unmodifiableList(Arrays.asList(out));+ }++ /** One request's {@link Hooks#failedBeforeStart}, skipped when no hook would see it. */+ private void failedBeforeStart(List<Hook> composed, Hooks.Policy policy, Request request,+ Checkpoint checkpoint, Throwable failure, long startedAt) {+ if (composed.isEmpty()) {+ return;+ }+ PredictContext failed = new PredictContext(Collections.singletonList(request.state()),+ request.questions(), checkpoint == null ? null : checkpoint.wireName(), this,+ request.maxLen(), request.headMaxLen());+ Hooks.failedBeforeStart(composed, failed, policy, failure, startedAt);+ }++ /** Answers one checkpoint's pending requests, one agent call per compatible group. */+ private static void answer(Agent agent, List<PredictContext> pending,+ Map<PredictContext, Integer> indexOf, List<Request> requests,+ List<RouteDecision> decisions, int batchSize, boolean sortByLength) {+ // The language only selects a temperature, so it splits a group only when the+ // checkpoint has per-language temperatures; otherwise it would cost forward passes.+ boolean tempered = !agent.config().langTemperatures().isEmpty();+ // Grouped on what the start hooks left, so a rewritten question set or budget applies to+ // its own request only. The schema is order-sensitive, as option order is positional.+ record Key(String schema, Integer maxLen, Integer headMaxLen, String language) {+ }+ Map<Key, List<PredictContext>> forwardPasses = new LinkedHashMap<>();+ for (PredictContext ctx : pending) {+ String language = null;+ if (tempered) {+ int i = indexOf.get(ctx);+ language = requests.get(i).options().lang();+ if (language == null && decisions.get(i).detection() != null) {+ language = decisions.get(i).detection().language();+ }+ }+ Key key = new Key(questionSchema(ctx.questions()), ctx.maxLen(), ctx.headMaxLen(),+ language);+ forwardPasses.computeIfAbsent(key, k -> new ArrayList<>()).add(ctx);+ }+ for (Map.Entry<Key, List<PredictContext>> pass : forwardPasses.entrySet()) {+ List<PredictContext> members = pass.getValue();+ List<Object> states = new ArrayList<>(members.size());+ for (PredictContext ctx : members) {+ states.add(ctx.states().get(0));+ }+ Key key = pass.getKey();+ List<Prediction> answered = agent.predictBatch(states, members.get(0).questions(),+ key.language(), batchSize, sortByLength, HookCall.none(), key.maxLen(),+ key.headMaxLen());+ if (answered.size() != members.size()) {+ throw new IllegalStateException("internal error: Agent.predictBatch returned "+ + answered.size() + " results for " + members.size() + " states");+ }+ for (int m = 0; m < members.size(); m++) {+ members.get(m).results(List.of(answered.get(m)));+ }+ }+ }+/*** Close the router, freeing every checkpoint it built.*diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.javaindex eb76b6e6..b914bc90 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java@@ -812,4 +812,88 @@ public final class Hooks {problem.addSuppressed(hookFailure);}}++ /**+ * {@link #around} for many requests that share forward passes: each keeps its own context and+ * its own start and end, while {@code inference} answers the ones still pending in one go.+ *+ * <p>The reference's {@code Router.predict_batch} sequence. Every context starts, in order,+ * before any inference; {@code inference} must assign {@code results} on each context it is+ * handed. Contexts then end in <b>reverse</b> of the order they started, so a hook that sets+ * something in start and resets it in end unwinds the last one first. On a failure every+ * started context fails with it, a skipped one included, and gets the error event before its+ * end; a hook throwing on a failed context is attached to the failure. Every context gets its+ * end even when another's end hook throws; the first such throw is raised afterwards.+ *+ * @param inference answers the contexts no start hook answered; not called when there are none+ */+ public static void aroundGroup(List<? extends Hook> hooks, List<PredictContext> contexts,+ Policy policy, Consumer<List<PredictContext>> inference) {+ List<PredictContext> started = new ArrayList<>(contexts.size());+ Throwable raised = null;+ try {+ for (PredictContext ctx : contexts) {+ started.add(ctx);+ dispatch(hooks, Event.PREDICT_START, ctx, policy);+ }+ List<PredictContext> pending = new ArrayList<>();+ for (PredictContext ctx : started) {+ if (ctx.results() != null) {+ continue;+ }+ if (ctx.states().isEmpty()) {+ ctx.results(List.of());+ } else {+ pending.add(ctx);+ }+ }+ if (!pending.isEmpty()) {+ inference.accept(List.copyOf(pending));+ }+ // Totalled before any end, so a malformed result fails the whole group rather than+ // escaping after some of its requests have already ended.+ List<Totals> totals = new ArrayList<>(started.size());+ for (PredictContext ctx : started) {+ totals.add(ctx.results() == null ? null : aggregateUsage(ctx.results()));+ }+ for (int i = 0; i < started.size(); i++) {+ started.get(i).usage(totals.get(i));+ }+ } catch (RuntimeException | Error problem) {+ raised = problem;+ }++ double now = System.nanoTime();+ for (PredictContext ctx : started) {+ ctx.elapsedMs((now - ctx.startedAt()) / 1_000_000.0);+ if (raised != null) {+ ctx.error(raised);+ }+ }+ Throwable firstEndFailure = null;+ for (int i = started.size() - 1; i >= 0; i--) {+ PredictContext ctx = started.get(i);+ List<Event> events = raised == null ? List.of(Event.PREDICT_END)+ : List.of(Event.ERROR, Event.PREDICT_END);+ for (Event event : events) {+ try {+ dispatch(hooks, event, ctx, policy);+ } catch (RuntimeException | Error hookFailure) {+ if (raised != null) {+ attach(raised, hookFailure);+ } else if (firstEndFailure == null) {+ firstEndFailure = hookFailure;+ }+ }+ }+ }++ Throwable failure = raised != null ? raised : firstEndFailure;+ if (failure instanceof RuntimeException problem) {+ throw problem;+ }+ if (failure instanceof Error problem) {+ throw problem;+ }+ }}diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.javanew file mode 100644index 00000000..e0f26086--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.java@@ -0,0 +1,143 @@+package com.convaiinnovations.laya;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertIterableEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;++import com.convaiinnovations.laya.Router.Checkpoint;+import com.convaiinnovations.laya.Router.Request;+import com.convaiinnovations.laya.Router.RouteOptions;+import com.convaiinnovations.laya.hooks.HookCall;+import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.nio.file.Paths;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.Test;++/**+ * {@link Router#predictBatch(List)} on real checkpoints equals one {@link Router#predict} per+ * request. Skipped unless the multilingual and typed-decisions checkpoints and both graphs are+ * configured; see {@link Fixtures}.+ */+final class RouterBatchParityTest {++ /** Batched rows are padded to their group's longest, so the logits may move in the last bits. */+ private static final double TOLERANCE = 1e-4;++ private static Path graph(String env) {+ String graph = System.getenv(env);+ Assumptions.assumeTrue(graph != null && !graph.isBlank(), "set " + env);+ Path path = Paths.get(graph);+ Path directory = Files.isDirectory(path) ? path : path.getParent();+ Assumptions.assumeTrue(directory != null && Files.isDirectory(directory),+ env + " does not name a graph: " + graph);+ return directory;+ }++ private static Path checkpoint(String name) {+ Path model = Fixtures.checkpoint(name);+ Assumptions.assumeTrue(model != null, Fixtures.missingCheckpoint(name));+ return model;+ }++ private static Map<String, Question> triage() {+ Map<String, Question> questions = new LinkedHashMap<>();+ questions.put("urgent", Question.noul("Does this need a human today?"));+ questions.put("topic", Question.choice("What is this about?", TinyCheckpoint.ordered(+ "billing", "charges, refunds, invoices", "access", "login or password",+ "other", "anything else")));+ return questions;+ }++ private static Map<String, Question> sentiment() {+ return Map.of("angry", Question.noul("Is the writer angry?"));+ }++ @Test+ @DisplayName("predictBatch equals per-request predict on real checkpoints")+ void batchEqualsPerRequest() throws IOException {+ Path multilingual = checkpoint("multilingual");+ Path typed = checkpoint("typed-decisions");+ Path multilingualGraph = graph(Fixtures.GRAPH_ENV);+ Path typedGraph = graph(Fixtures.TYPED_GRAPH_ENV);+ RouteOptions toMultilingual = RouteOptions.none().model("multilingual");+ RouteOptions toTyped = RouteOptions.none().model("typed-decisions");+ List<Request> requests = List.of(+ Request.of("मेरे कार्ड से दो बार पैसे कटे, कृपया जल्दी मदद करें", triage()),+ Request.of("I was charged twice for one order", triage()).options(toMultilingual),+ Request.of("No puedo entrar a mi cuenta y estoy harto", sentiment())+ .options(toMultilingual),+ Request.of("Please refund the duplicate charge on invoice 4411", triage())+ .options(toTyped),+ Request.of("我的账户被锁定了,无法登录", triage()),+ Request.of("The app logs me out every time, fix it now", sentiment())+ .options(toTyped),+ Request.of("I was charged twice for one order and nobody has answered my three "+ + "emails about it, so I am asking again", sentiment())+ .options(toMultilingual).maxLen(32));+ Router.AgentFactory agents = checkpointName -> switch (checkpointName) {+ case MULTILINGUAL -> Agent.open(multilingual, multilingualGraph);+ case TYPED_DECISIONS -> Agent.open(typed, typedGraph);+ default -> throw new IOException("no graph for " + checkpointName);+ };+ try (Router router = Router.builder().agents(agents).maxLoaded(2).build()) {+ List<Prediction> batched = router.predictBatch(requests);+ assertEquals(requests.size(), batched.size());+ for (int i = 0; i < requests.size(); i++) {+ Request request = requests.get(i);+ Prediction single;+ if (request.maxLen() == null) {+ single = router.predict(request.state(), request.questions(),+ request.options());+ } else {+ // Router.predict takes no budget, so the reference is the agent with one.+ Checkpoint target = router.route(request.state(), request.questions(),+ request.options()).model();+ try (Router.Lease lease = router.lease(target)) {+ single = lease.agent().predictBatch(List.of(request.state()),+ request.questions(), null, 0, false,+ HookCall.none().onStart(ctx -> ctx.maxLen(request.maxLen())))+ .get(0);+ }+ assertTrue(single.usage().stateTokensDropped() > 0,+ "the budget truncated: " + single.usage());+ }+ assertSame("request " + i, single, batched.get(i));+ }+ assertEquals(List.of(Checkpoint.MULTILINGUAL, Checkpoint.TYPED_DECISIONS),+ new ArrayList<>(router.loaded()).stream().sorted().toList());+ }+ }++ private static void assertSame(String at, Prediction want, Prediction got) {+ assertEquals(want.model(), got.model(), at + ".model");+ assertEquals(want.usage(), got.usage(), at + ".usage");+ assertIterableEquals(want.answers().keySet(), got.answers().keySet(), at + " ids");+ for (Map.Entry<String, Answer> entry : want.answers().entrySet()) {+ String id = at + "." + entry.getKey();+ Answer w = entry.getValue();+ Answer g = got.answers().get(entry.getKey());+ assertEquals(w.type(), g.type(), id + ".type");+ assertEquals(w.confidence(), g.confidence(), TOLERANCE, id + ".confidence");+ assertEquals(w.answerConfidence(), g.answerConfidence(), TOLERANCE,+ id + ".answer_confidence");+ assertEquals(w.actProbability(), g.actProbability(), TOLERANCE, id + ".act");+ if (w instanceof Answer.Choice wc && g instanceof Answer.Choice gc) {+ assertIterableEquals(wc.probabilities().keySet(), gc.probabilities().keySet());+ for (String option : wc.probabilities().keySet()) {+ assertEquals(wc.probabilities().get(option), gc.probabilities().get(option),+ TOLERANCE, id + ".p[" + option + "]");+ }+ assertEquals(wc.choice(), gc.choice(), id + ".choice");+ } else if (w instanceof Answer.Noul wn && g instanceof Answer.Noul gn) {+ assertEquals(wn.noul(), gn.noul(), TOLERANCE, id + ".noul");+ }+ }+ }+}diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.javanew file mode 100644index 00000000..e28d9367--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java@@ -0,0 +1,497 @@+package com.convaiinnovations.laya;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNotEquals;+import static org.junit.jupiter.api.Assertions.assertSame;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;++import com.convaiinnovations.laya.Router.Checkpoint;+import com.convaiinnovations.laya.Router.Request;+import com.convaiinnovations.laya.Router.RouteOptions;+import com.convaiinnovations.laya.hooks.Hook;+import com.convaiinnovations.laya.hooks.PredictContext;+import com.convaiinnovations.laya.infer.InferenceSession;+import com.convaiinnovations.laya.sequence.Collator;+import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.ArrayList;+import java.util.EnumMap;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++/**+ * {@link Router#predictBatch(List)} with a stub agent per checkpoint, each on its own recording+ * session, so which checkpoint answered and how many graph calls it took are observable.+ */+class RouterBatchTest {++ private static final RouteOptions ENGLISH = RouteOptions.none().model("english");+ private static final RouteOptions MULTILINGUAL = RouteOptions.none().model("multilingual");++ @TempDir+ Path root;++ private final Map<Checkpoint, TinyCheckpoint.RecordingSession> sessions =+ new EnumMap<>(Checkpoint.class);++ @BeforeEach+ void writeCheckpoint() throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ }++ /** A router whose every checkpoint is a stub agent on a session of its own. */+ private Router router() throws IOException {+ Router router = Router.builder().maxLoaded(3).build();+ for (Checkpoint checkpoint : Checkpoint.values()) {+ TinyCheckpoint.RecordingSession session = new TinyCheckpoint.RecordingSession();+ sessions.put(checkpoint, session);+ router.attach(checkpoint.wireName(), TinyCheckpoint.agent(root, session));+ }+ return router;+ }++ private int calls(Checkpoint checkpoint) {+ return sessions.get(checkpoint).batches.size();+ }++ private static Map<String, Question> urgent() {+ return Map.of("urgent", Question.noul("Needs a human."));+ }++ private static Map<String, Question> topic() {+ Map<String, Question> questions = new LinkedHashMap<>();+ questions.put("topic", Question.choice("Topic.", TinyCheckpoint.ordered(+ "billing", "money", "login", "access", "other", "anything else")));+ return questions;+ }++ private static List<Request> interleaved() {+ return List.of(+ Request.of("refund my card", urgent()).options(ENGLISH),+ Request.of("mi tarjeta", urgent()).options(MULTILINGUAL),+ Request.of("cannot log in", topic()).options(ENGLISH),+ Request.of("बिल गलत है", topic()).options(MULTILINGUAL),+ Request.of("charged twice", urgent()).options(ENGLISH),+ Request.of("doppelt belastet", urgent()).options(MULTILINGUAL));+ }++ @Test+ @DisplayName("results come back in request order and equal per-request predict")+ void orderPreservedAndEqualToPredict() throws IOException {+ List<Request> requests = interleaved();+ List<Prediction> batched;+ try (Router router = router()) {+ batched = router.predictBatch(requests);+ }+ assertEquals(requests.size(), batched.size());+ try (Router router = router()) {+ for (int i = 0; i < requests.size(); i++) {+ Request request = requests.get(i);+ Prediction single = router.predict(request.state(), request.questions(),+ request.options());+ assertEquals(single, batched.get(i), "request " + i);+ }+ }+ // Distinct per state, so a swap of two results could not pass the comparison above.+ assertNotEquals(batched.get(0), batched.get(4));+ }++ @Test+ @DisplayName("an interleaved workload costs one forward pass per checkpoint and schema")+ void oneCallPerCheckpoint() throws IOException {+ try (Router router = router()) {+ router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(MULTILINGUAL),+ Request.of("c", urgent()).options(ENGLISH),+ Request.of("d", urgent()).options(MULTILINGUAL),+ Request.of("e", urgent()).options(ENGLISH)));+ assertEquals(1, calls(Checkpoint.ENGLISH));+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));+ assertEquals(0, calls(Checkpoint.TYPED_DECISIONS));+ assertEquals(3, sessions.get(Checkpoint.ENGLISH).batches.get(0).rows());+ assertEquals(2, sessions.get(Checkpoint.MULTILINGUAL).batches.get(0).rows());+ }+ }++ @Test+ @DisplayName("requests with different schemas or budgets on one checkpoint split the pass")+ void schemaAndBudgetSplit() throws IOException {+ try (Router router = router()) {+ router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", topic()).options(ENGLISH),+ Request.of("c", urgent()).options(ENGLISH).maxLen(48),+ Request.of("d", urgent()).options(ENGLISH)));+ List<Collator.Batch> batches = sessions.get(Checkpoint.ENGLISH).batches;+ assertEquals(3, batches.size(), "urgent, topic, urgent at maxLen 48");+ assertEquals(List.of(2, 1, 1), batches.stream().map(Collator.Batch::rows).toList());+ }+ }++ @Test+ @DisplayName("a per-request budget reaches the agent")+ void budgetForwarded() throws IOException {+ String state = "x".repeat(200);+ try (Router router = router()) {+ Prediction full = router.predictBatch(List.of(+ Request.of(state, urgent()).options(ENGLISH))).get(0);+ Prediction capped = router.predictBatch(List.of(+ Request.of(state, urgent()).options(ENGLISH).maxLen(40))).get(0);+ assertTrue(capped.usage().inputTokens() < full.usage().inputTokens(),+ full.usage() + " vs " + capped.usage());+ }+ }++ @Test+ @DisplayName("model pins each request, overriding detection")+ void pinForwarded() throws IOException {+ try (Router router = router()) {+ // English text, which detection would send to the english checkpoint.+ router.predictBatch(List.of("my card was charged twice", "cannot log in"), urgent(),+ MULTILINGUAL);+ assertEquals(0, calls(Checkpoint.ENGLISH));+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));+ router.predictBatch(List.of(+ Request.of("my card was charged twice", urgent()).options(+ RouteOptions.none().model("typed")),+ Request.of("my card was charged twice", urgent())));+ assertEquals(1, calls(Checkpoint.TYPED_DECISIONS), "alias pin");+ assertEquals(1, calls(Checkpoint.ENGLISH), "the unpinned request routes itself");+ }+ }++ @Test+ @DisplayName("the BatchPredictor form routes each state on its own")+ void statesRouteApart() throws IOException {+ try (Router router = router()) {+ List<Prediction> out = router.predictBatch(+ List.of("my card was charged twice", "मेरा कार्ड दो बार चार्ज हुआ"), urgent());+ assertEquals(2, out.size());+ assertEquals(1, calls(Checkpoint.ENGLISH));+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));+ }+ }++ @Test+ @DisplayName("Decisions.decideBatch runs on a router and equals decide per state")+ void decideBatchOnRouter() throws IOException {+ Map<String, Object> schema = Map.of("type", "object", "properties",+ new LinkedHashMap<>(Map.of("urgent", Map.of("type", "boolean"))));+ List<String> states = List.of("my card was charged twice", "मेरा कार्ड दो बार चार्ज हुआ",+ "login broken");+ try (Router router = router()) {+ List<Decisions.Decision> batched = Decisions.decideBatch(router, states, schema);+ assertEquals(states.size(), batched.size());+ for (int i = 0; i < states.size(); i++) {+ assertEquals(Decisions.decide(router, states.get(i), schema).values(),+ batched.get(i).values(), "state " + i);+ }+ }+ try (Router router = router()) {+ BatchPredictor pinned = (s, asked) -> router.predictBatch(s, asked, MULTILINGUAL);+ Decisions.decideBatch(pinned, states, schema);+ assertEquals(0, calls(Checkpoint.ENGLISH));+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));+ }+ }++ @Test+ @DisplayName("an empty batch routes nothing and returns nothing")+ void empty() throws IOException {+ List<String> events = new ArrayList<>();+ try (Router router = router()) {+ router.hooks().addHook(new Hook() {+ @Override+ public void onRoute(PredictContext ctx) {+ events.add("route");+ }+ });+ assertEquals(List.of(), router.predictBatch(List.<Request>of()));+ assertEquals(List.of(), events);+ }+ }++ @Test+ @DisplayName("a null request, state or questions, or a negative batch size, is refused")+ void refusals() throws IOException {+ try (Router router = router()) {+ List<Request> withNull = new ArrayList<>();+ withNull.add(Request.of("a", urgent()));+ withNull.add(null);+ IllegalArgumentException missing = assertThrows(IllegalArgumentException.class,+ () -> router.predictBatch(withNull));+ assertTrue(missing.getMessage().contains("request 1"), missing.getMessage());+ assertThrows(IllegalArgumentException.class, () -> Request.of(null, urgent()));+ assertThrows(IllegalArgumentException.class, () -> Request.of("a", null));+ assertThrows(IllegalArgumentException.class,+ () -> router.predictBatch(List.of(Request.of("a", urgent())), -1, false));+ assertEquals(0, calls(Checkpoint.ENGLISH));+ }+ }++ /** Records each predict event with the state it carried. */+ private static final class Trace implements Hook {++ final List<String> events = new ArrayList<>();++ @Override+ public void onPredictStart(PredictContext ctx) {+ events.add("start:" + ctx.states().get(0));+ }++ @Override+ public void onError(PredictContext ctx) {+ events.add("error:" + ctx.states().get(0));+ }++ @Override+ public void onPredictEnd(PredictContext ctx) {+ events.add("end:" + ctx.states().get(0));+ }+ }++ @Test+ @DisplayName("router hooks run per request: starts in order, ends reversed, per checkpoint")+ void hooksPerRequest() throws IOException {+ Trace trace = new Trace();+ try (Router router = router()) {+ router.hooks().addHook(trace);+ router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(MULTILINGUAL),+ Request.of("c", urgent()).options(ENGLISH)));+ }+ assertEquals(List.of("start:a", "start:c", "end:c", "end:a", "start:b", "end:b"),+ trace.events);+ }++ @Test+ @DisplayName("a start hook's skip keeps its request out of the forward pass")+ void skipLeavesThePass() throws IOException {+ Prediction canned = new Prediction(Prediction.MODEL, Map.of(),+ new Usage(0, 0, 0, 0, false, List.of(), Map.of()));+ try (Router router = router()) {+ router.hooks().addHook(new Hook() {+ @Override+ public void onPredictStart(PredictContext ctx) {+ if ("cached".equals(ctx.states().get(0))) {+ ctx.skip(List.of(canned));+ }+ }+ });+ List<Prediction> out = router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("cached", urgent()).options(ENGLISH),+ Request.of("c", urgent()).options(ENGLISH)));+ assertSame(canned, out.get(1));+ assertEquals(1, calls(Checkpoint.ENGLISH));+ assertEquals(2, sessions.get(Checkpoint.ENGLISH).batches.get(0).rows());+ }+ }++ @Test+ @DisplayName("a start hook's budget applies to its own request only")+ void hookBudgetPerRequest() throws IOException {+ try (Router router = router()) {+ router.hooks().addHook(new Hook() {+ @Override+ public void onPredictStart(PredictContext ctx) {+ if ("b".equals(ctx.states().get(0))) {+ ctx.maxLen(48);+ }+ }+ });+ router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(ENGLISH),+ Request.of("c", urgent()).options(ENGLISH)));+ assertEquals(List.of(2, 1), sessions.get(Checkpoint.ENGLISH).batches.stream()+ .map(Collator.Batch::rows).toList());+ }+ }++ @Test+ @DisplayName("a failed checkpoint fails each of its requests, after earlier ones ended")+ void failureEndsEveryStartedRequest() throws IOException {+ Trace trace = new Trace();+ try (Router router = Router.builder().maxLoaded(3).build()) {+ router.attach("english", TinyCheckpoint.agent(root,+ new TinyCheckpoint.RecordingSession()));+ router.attach("multilingual", TinyCheckpoint.agent(root, new InferenceSession() {+ @Override+ public Output run(Collator.Batch batch) {+ throw new IllegalStateException("graph failed");+ }++ @Override+ public void close() {+ }+ }));+ router.hooks().addHook(trace);+ IllegalStateException failure = assertThrows(IllegalStateException.class,+ () -> router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(MULTILINGUAL),+ Request.of("c", urgent()).options(MULTILINGUAL))));+ assertEquals("graph failed", failure.getMessage());+ }+ assertEquals(List.of("start:a", "end:a", "start:b", "start:c",+ "error:c", "end:c", "error:b", "end:b"), trace.events);+ }++ @Test+ @DisplayName("a hook that leaves no result is refused, naming the request")+ void emptyResultRefused() throws IOException {+ try (Router router = router()) {+ router.hooks().addHook(new Hook() {+ @Override+ public void onPredictEnd(PredictContext ctx) {+ ctx.results(List.of());+ }+ });+ IllegalStateException refused = assertThrows(IllegalStateException.class,+ () -> router.predictBatch(List.of(Request.of("a", urgent()))));+ assertTrue(refused.getMessage().contains("request 0"), refused.getMessage());+ }+ }++ @Test+ @DisplayName("per-language temperatures split the pass by language and still equal predict")+ void languageSplitsOnlyWhenTempered(@TempDir Path tempered) throws IOException {+ TinyCheckpoint.write(tempered, 64, 32);+ Files.writeString(tempered.resolve("rl_agent_config.json"),+ "{\"max_len\": 64, \"head_max_len\": 32, \"temperature\": [1.0, 1.0, 1.0],"+ + " \"temperature_by_options\": {},"+ + " \"lang_temperatures\": {\"hi\": {\"temperature\": [3.0, 3.0, 3.0]}}}");+ List<Request> requests = List.of(+ Request.of("a", urgent()).options(MULTILINGUAL.lang("hi")),+ Request.of("b", urgent()).options(MULTILINGUAL.lang("en")),+ Request.of("c", urgent()).options(MULTILINGUAL.lang("hi")));+ TinyCheckpoint.RecordingSession session = new TinyCheckpoint.RecordingSession();+ try (Router router = Router.builder().build()) {+ router.attach("multilingual", TinyCheckpoint.agent(tempered, session));+ List<Prediction> batched = router.predictBatch(requests);+ assertEquals(2, session.batches.size(), "hi and en apart");+ for (int i = 0; i < requests.size(); i++) {+ Request request = requests.get(i);+ assertEquals(router.predict(request.state(), request.questions(),+ request.options()), batched.get(i), "request " + i);+ }+ assertNotEquals(router.predict("a", urgent(), MULTILINGUAL.lang("en")),+ batched.get(0), "the temperature took effect");+ }+ // Untempered, the same three share one pass.+ try (Router router = router()) {+ router.predictBatch(requests);+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));+ }+ }++ /** Lifecycle events by checkpoint, predict events by state. */+ private static final class Lifecycle implements Hook {++ final List<String> events = new ArrayList<>();++ @Override+ public void onRoute(PredictContext ctx) {+ events.add("route:" + ctx.model());+ }++ @Override+ public void onLoad(PredictContext ctx) {+ events.add("load:" + ctx.model());+ }++ @Override+ public void onEvict(PredictContext ctx) {+ events.add("evict:" + ctx.model());+ }++ @Override+ public void onPredictStart(PredictContext ctx) {+ events.add("start:" + ctx.states().get(0));+ }++ @Override+ public void onError(PredictContext ctx) {+ events.add("error:" + ctx.states().get(0) + "@" + ctx.model());+ }++ @Override+ public void onPredictEnd(PredictContext ctx) {+ events.add("end:" + ctx.states().get(0));+ }+ }++ /** Builds stub agents on demand, failing for the checkpoints named. */+ private Router.AgentFactory building(Checkpoint... failing) {+ List<Checkpoint> refused = List.of(failing);+ return checkpoint -> {+ if (refused.contains(checkpoint)) {+ throw new IOException("cannot build " + checkpoint.wireName());+ }+ return TinyCheckpoint.agent(root, new TinyCheckpoint.RecordingSession());+ };+ }++ @Test+ @DisplayName("every route, then per checkpoint evict/load before its starts, as the reference")+ void lifecycleOrder() throws IOException {+ Lifecycle trace = new Lifecycle();+ try (Router router = Router.builder().agents(building()).maxLoaded(1).build()) {+ router.hooks().addHook(trace);+ router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(MULTILINGUAL),+ Request.of("c", urgent()).options(ENGLISH)));+ assertEquals(List.of("route:english", "route:multilingual", "route:english",+ "load:english", "start:a", "start:c", "end:c", "end:a",+ "evict:english", "load:multilingual", "start:b", "end:b"), trace.events);+ }+ }++ @Test+ @DisplayName("a load failure reaches every request of its checkpoint, with no start")+ void loadFailureBeforeStart() throws IOException {+ Lifecycle trace = new Lifecycle();+ try (Router router = Router.builder().agents(building(Checkpoint.MULTILINGUAL))+ .build()) {+ router.hooks().addHook(trace);+ RuntimeException failure = assertThrows(RuntimeException.class,+ () -> router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("b", urgent()).options(MULTILINGUAL),+ Request.of("c", urgent()).options(MULTILINGUAL))));+ assertTrue(String.valueOf(failure.getCause()).contains("cannot build multilingual")+ || failure.getMessage().contains("cannot build multilingual"),+ failure.toString());+ assertEquals(List.of("route:english", "route:multilingual", "route:multilingual",+ "load:english", "start:a", "end:a",+ "error:b@multilingual", "end:b", "error:c@multilingual", "end:c"),+ trace.events);+ }+ }++ @Test+ @DisplayName("a routing failure reaches its own request before anything loads")+ void routeFailureBeforeStart() throws IOException {+ Lifecycle trace = new Lifecycle();+ try (Router router = Router.builder().agents(building()).build()) {+ router.hooks().addHook(trace);+ assertThrows(IllegalArgumentException.class, () -> router.predictBatch(List.of(+ Request.of("a", urgent()).options(ENGLISH),+ Request.of("bad", urgent()).options(RouteOptions.none().model("nope")),+ Request.of("c", urgent()).options(ENGLISH))));+ assertEquals(List.of(), router.loaded());+ assertEquals(List.of("route:english", "error:bad@null", "end:bad"), trace.events);+ }+ }+}