Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(java): Router.predictBatch, so a router is a BatchPredictor

Adeshina Lasisid76eacb25bf9
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 43a1d72c..d01263dc 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 1002 register here, 985 run, 17 abort on a checkpoint this
+ # Measured on the merged tree: 1021 register here, 1003 run, 18 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 986. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 1004. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 1002 registered, 17 abort,
- # 985 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 1021 registered, 18 abort,
+ # 1003 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 985 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1003 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 986, not the build lane's 985, and the one is the point: this lane runs one test MORE
+ # 1004, not the build lane's 1003, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 986 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1004 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index deeba7e3..d41a89ee 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 985 of the 1002 registered tests run without a checkpoint. This floor has been raised
+ # 1003 of the 1021 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 985 --allow-aborted
+ --min-tests 1003 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/README.md b/laya-java/README.md
index d5c3f01d..d8119f05 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -192,6 +192,23 @@ evidence of English. A deployment whose traffic is known to be English can set
english, as the reference's alias table does: `load("default")` loads english whatever
`defaultCheckpoint` is set to.
+`predictBatch` routes many requests and answers them with one load per checkpoint used:
+
+```java
+List<Prediction> out = router.predictBatch(states, questions); // each state routed alone
+List<Prediction> mixed = router.predictBatch(List.of(
+ Router.Request.of(email, Presets.email()),
+ Router.Request.of(ticket, Presets.triage())
+ .options(Router.RouteOptions.none().model("multilingual"))
+ .maxLen(512)));
+```
+
+Requests are routed first, grouped by checkpoint in order of first appearance, and within a
+checkpoint those sharing a question schema and token budget share `Agent.predictBatch` calls.
+Results come back in request order and equal per-request `predict`. `Router` is a
+`BatchPredictor`, so `Decisions.decideBatch(router, states, schema)` works; to pin it, pass
+`(s, q) -> router.predictBatch(s, q, options)` as the runner.
+
## Detection on its own
`lang.LanguageDetection` is the router's evidence, usable without one:
@@ -498,14 +515,25 @@ reaches `onError` and `onPredictEnd`, with no start event, and its `elapsedMs` c
and the failed load. The agent's own pair still
fires for the forward pass, so a router-level hook and an agent-level hook see different spans.
+`Router.predictBatch` dispatches one pair **per request**, as `predict` does, so a start hook can
+rewrite or `skip` a request before it joins a shared forward pass. The order is the reference's:
+every `onRoute`, then per checkpoint its `onEvict`/`onLoad`, its requests' starts in order and
+their ends in reverse. If a checkpoint's inference fails, each of its started requests gets
+`onError` and its end before the failure propagates. A routing failure reaches that request, and a
+loading failure every request of that checkpoint, as `onError` and `onPredictEnd` with no start —
+as on `predict`, where the reference's `predict_batch` reports neither. The agent's pair fires once per forward
+pass, not per request. Unlike `predict`, the batch path honours `ctx.maxLen` / `ctx.headMaxLen`
+from a router-level start hook, since a request's own budget is seeded there.
+
Hooks are dispatched outside the router's lock, so a hook may call back into the router —
`loaded()`, `load`, `unload` — without deadlocking. Two consequences of that are yours to handle:
`load()` can hand back an agent a hook unloaded while its `on_load` was being dispatched, and a
router-level `on_predict_start` that calls `router.predict` recurses until the stack runs out,
because that inner call dispatches the same hook again.
-`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured:
-`Agent.predict` takes no budget arguments. An agent-level hook can still set them.
+`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured by
+`Router.predict`: `Agent.predict` takes no budget arguments. `predictBatch` honours them, and an
+agent-level hook can set them on either path.
```java
router.hooks().addHook(new Hook() {
@@ -518,11 +546,11 @@ router.hooks().addHook(new Hook() {
| | |
|---|---|
-| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |
+| a process-wide default hook sees **two** predict pairs per `Router.predict` (per request plus per forward pass on `predictBatch`) | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |
| a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure |
| calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |
| `concurrent(false)` with an `AsyncHook` whose callback dispatches a hook on the same registry | deadlocks with no deadline, and stalls until it with one. Serial dispatch holds a lock across the callback, and the callback is on another thread, so the inner dispatch cannot re-enter that lock. A callback that dispatches nothing — `loaded()`, a cached `load` — is fine |
-| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |
+| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored by `Router.predict`, which calls `Agent.predict` with no budget arguments; `predictBatch` honours the budgets. Both answer only the first state |
`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlate
on the model name, not on `runId`. That context holds the chosen checkpoint in `ctx.model()` and no
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java
index dc27c223..84511191 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Agent.java
@@ -231,17 +231,30 @@ public final class Agent implements AutoCloseable, Predictor, BatchPredictor {
public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,
String language, int batchSize, boolean sortByLength,
HookCall call) {
+ return predictBatch(states, questions, language, batchSize, sortByLength, call, null,
+ null);
+ }
+
+ /**
+ * {@link #predictBatch(List, Map, String, int, boolean, HookCall)} with the token budgets
+ * seeded on the context, so this agent's own start hooks can still replace them. For
+ * {@link Router#predictBatch(List)}, which carries a per-request budget.
+ */
+ List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,
+ String language, int batchSize, boolean sortByLength,
+ HookCall call, Integer maxLen, Integer headMaxLen) {
// Before any hook: a closed agent cannot answer, and dispatching a start hook that then
// watched the call fail would report a prediction that was never going to happen.
if (closed) {
throw new IllegalStateException(
"this agent is closed; open a new one rather than reusing it");
}
- PredictContext ctx = new PredictContext(states, questions, modelName, this);
+ PredictContext ctx = new PredictContext(states, questions, modelName, this, maxLen,
+ headMaxLen);
return Hooks.around(hooks.composeFor(call), ctx, hooks.policyFor(call),
- (hooked, asked, maxLen, headMaxLen) ->
- infer(hooked, asked, language, batchSize, sortByLength, maxLen,
- headMaxLen));
+ (hooked, asked, tokens, headTokens) ->
+ infer(hooked, asked, language, batchSize, sortByLength, tokens,
+ headTokens));
}
/**
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java
index 2f4c00d2..33291678 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/BatchPredictor.java
@@ -7,14 +7,13 @@ import java.util.Map;
* Anything that can answer one question set over many states in a shared forward pass.
*
* <p>Separate from {@link Predictor} rather than folded into it, because not every runtime
- * batches. {@link Agent} does; {@link Router} does not — it selects a checkpoint per state, so
- * there is no single graph call to share. The reference draws the same line at runtime, raising
- * {@code TypeError} when a runner has no {@code predict_batch} rather than looping {@code decide}
- * N times behind the caller's back; here the line is drawn by the type, so the same mistake does
- * not compile.
+ * batches. The reference draws the same line at runtime, raising {@code TypeError} when a runner
+ * has no {@code predict_batch} rather than looping {@code decide} N times behind the caller's
+ * back; here the line is drawn by the type, so the same mistake does not compile.
*
- * <p>{@link Agent#predictBatch(List, Map)} already had this signature, so this interface names an
- * existing capability rather than adding one.
+ * <p>{@link Agent} shares one graph call across the states. {@link Router} routes each state and
+ * shares one call per checkpoint, so states that route apart are still answered together where
+ * they can be.
*/
public interface BatchPredictor {
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java
index d9f98f29..12747086 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Decisions.java
@@ -67,12 +67,12 @@ import java.util.Set;
* the one the reference's own {@code decide_batch} always takes.</li>
* <li>{@code routing} is not reported. {@link Prediction} carries no routing block, so there is
* nothing to pass through; {@link Router} reports its decision at the point it makes it.</li>
- * <li>A <b>batched</b> decision cannot route per state. The reference's {@code decide_batch}
- * special-cases a {@code route_batch}-capable runner and sends one request per state "so
- * states may route to different checkpoints"; {@link Router} here implements
- * {@link Predictor} and not {@link BatchPredictor}, so {@link #decideBatch} will not take
- * one at all. Batching a router means deciding what a shared forward pass over two
- * checkpoints is, which is a design question and not a port.</li>
+ * <li>A batched decision on a {@link Router} takes no routing keywords. The reference's
+ * {@code decide_batch} copies {@code model}, {@code task}, {@code lang},
+ * {@code lang_guess} and the token budgets onto every request; here a router routes each
+ * state on its own. A pin is a lambda runner,
+ * {@code (states, asked) -> router.predictBatch(states, asked, options)}; a budget is a
+ * list of {@link Router.Request}s.</li>
* <li>An {@code enum} must be an <b>array</b>. The reference measures and iterates whatever
* is there, so {@code "enum": "abc"} becomes a three-option choice over {@code "a"},
* {@code "b"}, {@code "c"} — a schema nobody wrote, built from a typo — and
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
index 587672b5..9302499f 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
@@ -12,8 +12,10 @@ import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Path;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.IdentityHashMap;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
@@ -46,7 +48,7 @@ import java.util.function.Supplier;
* <p>This class decides; it does not load. {@link #route} runs no model and touches no disk, so it
* is safe to call on every request and to test without a checkpoint.
*/
-public final class Router implements AutoCloseable, Predictor {
+public final class Router implements AutoCloseable, Predictor, BatchPredictor {
/** The hub repository that bundles all three checkpoints. */
public static final String BUNDLE_REPO = "convaiinnovations/laya";
@@ -151,6 +153,45 @@ public final class Router implements AutoCloseable, Predictor {
}
}
+ /**
+ * One request of a {@link #predictBatch(List)}: a state, its questions, and the routing and
+ * token-budget overrides the reference reads off each request dict.
+ *
+ * @param options routing for this request alone; null means {@link RouteOptions#none()}
+ * @param maxLen a token budget for this request alone, or null for the checkpoint's
+ * @param headMaxLen a head budget for this request alone, or null for the checkpoint's
+ */
+ public record Request(Object state, Map<String, Question> questions, RouteOptions options,
+ Integer maxLen, Integer headMaxLen) {
+
+ /** @throws IllegalArgumentException for a null state or null questions */
+ public Request {
+ requireNonNull(state, "state");
+ requireNonNull(questions, "questions");
+ options = options == null ? RouteOptions.none() : options;
+ }
+
+ /** A request routed on its state and questions alone, at the checkpoint's budgets. */
+ public static Request of(Object state, Map<String, Question> questions) {
+ return new Request(state, questions, null, null, null);
+ }
+
+ /** This request with these routing options. */
+ public Request options(RouteOptions value) {
+ return new Request(state, questions, value, maxLen, headMaxLen);
+ }
+
+ /** This request with this token budget. */
+ public Request maxLen(Integer value) {
+ return new Request(state, questions, options, value, headMaxLen);
+ }
+
+ /** This request with this head budget. */
+ public Request headMaxLen(Integer value) {
+ return new Request(state, questions, options, maxLen, value);
+ }
+ }
+
/**
* Builds the agent for a checkpoint.
*
@@ -1558,6 +1599,185 @@ public final class Router implements AutoCloseable, Predictor {
return lang;
}
+ /**
+ * Route every state and answer the shared questions, sharing forward passes per checkpoint.
+ *
+ * <p>{@link BatchPredictor}'s form, so {@link Decisions#decideBatch} takes a router. To pin
+ * one, pass {@code (states, asked) -> router.predictBatch(states, asked, options)}.
+ */
+ @Override
+ public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions) {
+ return predictBatch(states, questions, RouteOptions.none());
+ }
+
+ /** {@link #predictBatch(List, Map)} with the same routing options on every state. */
+ public List<Prediction> predictBatch(List<?> states, Map<String, Question> questions,
+ RouteOptions options) {
+ requireNonNull(states, "states");
+ List<Request> requests = new ArrayList<>(states.size());
+ for (Object state : states) {
+ requests.add(new Request(state, questions, options, null, null));
+ }
+ return predictBatch(requests);
+ }
+
+ /** {@link #predictBatch(List, int, boolean)} with every state at once and no sorting. */
+ public List<Prediction> predictBatch(List<Request> requests) {
+ return predictBatch(requests, 0, false);
+ }
+
+ /**
+ * Route each request and answer it, with one checkpoint load per checkpoint used.
+ *
+ * <p>Every request is routed first, then grouped by checkpoint in order of first appearance.
+ * Within a checkpoint, requests that share a question schema, a token budget and (when the
+ * checkpoint has per-language temperatures) a language share {@link Agent#predictBatch}
+ * calls. Each request is answered exactly as {@link #predict(Object, Map, RouteOptions)}
+ * would answer it.
+ *
+ * <p>Router hooks run per request, as {@link #predict} runs them, and in the reference's
+ * order: every {@code on_route} first, then per checkpoint any {@code on_evict}/{@code on_load},
+ * then that checkpoint's start events in request order, so a start hook can rewrite or skip a
+ * request before it joins a forward pass, then its end events in reverse. If its inference
+ * fails, every started request of that checkpoint gets the error and its end before the
+ * failure propagates; checkpoints already answered have ended normally. A routing failure
+ * reaches its own request, and a loading failure every request of that checkpoint, as
+ * {@code on_error} and {@code on_predict_end} with no start, as in {@link #predict}.
+ *
+ * @param batchSize states per graph call, or 0 for each group at once
+ * @param sortByLength forwarded to {@link Agent#predictBatch}; changes no answer
+ * @return one prediction per request, in the order the requests were given
+ */
+ public List<Prediction> predictBatch(List<Request> requests, int batchSize,
+ boolean sortByLength) {
+ requireNonNull(requests, "requests");
+ if (batchSize < 0) {
+ throw new IllegalArgumentException("batchSize must be 0 or positive, got " + batchSize);
+ }
+ // Before any routing, as predict does: a malformed batch must not route, load or evict.
+ for (int i = 0; i < requests.size(); i++) {
+ if (requests.get(i) == null) {
+ throw new IllegalArgumentException("request " + i + " is null");
+ }
+ }
+ List<Hook> composed = hooks.composeFor(HookCall.none());
+ Hooks.Policy policy = hooks.policyFor(HookCall.none());
+ List<RouteDecision> decisions = new ArrayList<>(requests.size());
+ for (Request request : requests) {
+ long startedAt = System.nanoTime();
+ try {
+ decisions.add(route(request.state(), request.questions(), request.options()));
+ } catch (RuntimeException | Error failure) {
+ failedBeforeStart(composed, policy, request, null, failure, startedAt);
+ throw failure;
+ }
+ }
+ Map<Checkpoint, List<Integer>> groups = new LinkedHashMap<>();
+ for (int i = 0; i < decisions.size(); i++) {
+ groups.computeIfAbsent(decisions.get(i).model(), key -> new ArrayList<>()).add(i);
+ }
+
+ Prediction[] out = new Prediction[requests.size()];
+ for (Map.Entry<Checkpoint, List<Integer>> group : groups.entrySet()) {
+ Checkpoint checkpoint = group.getKey();
+ // Loaded before the starts, as the reference loads, so elapsedMs excludes a cold load.
+ long startedAt = System.nanoTime();
+ Lease lease;
+ try {
+ lease = lease(checkpoint);
+ } catch (RuntimeException | Error failure) {
+ for (int i : group.getValue()) {
+ failedBeforeStart(composed, policy, requests.get(i), checkpoint, failure,
+ startedAt);
+ }
+ throw failure;
+ }
+ List<PredictContext> contexts = new ArrayList<>(group.getValue().size());
+ Map<PredictContext, Integer> indexOf = new IdentityHashMap<>();
+ try (Lease held = lease) {
+ for (int i : group.getValue()) {
+ Request request = requests.get(i);
+ PredictContext ctx = new PredictContext(
+ Collections.singletonList(request.state()), request.questions(),
+ checkpoint.wireName(), this, request.maxLen(), request.headMaxLen());
+ contexts.add(ctx);
+ indexOf.put(ctx, i);
+ }
+ Hooks.aroundGroup(composed, contexts, policy, pending -> answer(held.agent(),
+ pending, indexOf, requests, decisions, batchSize, sortByLength));
+ }
+ for (PredictContext ctx : contexts) {
+ int i = indexOf.get(ctx);
+ if (ctx.results() == null || ctx.results().isEmpty()) {
+ // As in predict: only a hook can cause this, and inventing an answer would
+ // report a decision the model never made.
+ throw new IllegalStateException("a hook left no prediction for request " + i
+ + "; ctx.skip(...) needs one result");
+ }
+ out[i] = ctx.results().get(0);
+ }
+ }
+ // Not List.of: an end hook may leave a null result, which predict returns as it is.
+ return Collections.unmodifiableList(Arrays.asList(out));
+ }
+
+ /** One request's {@link Hooks#failedBeforeStart}, skipped when no hook would see it. */
+ private void failedBeforeStart(List<Hook> composed, Hooks.Policy policy, Request request,
+ Checkpoint checkpoint, Throwable failure, long startedAt) {
+ if (composed.isEmpty()) {
+ return;
+ }
+ PredictContext failed = new PredictContext(Collections.singletonList(request.state()),
+ request.questions(), checkpoint == null ? null : checkpoint.wireName(), this,
+ request.maxLen(), request.headMaxLen());
+ Hooks.failedBeforeStart(composed, failed, policy, failure, startedAt);
+ }
+
+ /** Answers one checkpoint's pending requests, one agent call per compatible group. */
+ private static void answer(Agent agent, List<PredictContext> pending,
+ Map<PredictContext, Integer> indexOf, List<Request> requests,
+ List<RouteDecision> decisions, int batchSize, boolean sortByLength) {
+ // The language only selects a temperature, so it splits a group only when the
+ // checkpoint has per-language temperatures; otherwise it would cost forward passes.
+ boolean tempered = !agent.config().langTemperatures().isEmpty();
+ // Grouped on what the start hooks left, so a rewritten question set or budget applies to
+ // its own request only. The schema is order-sensitive, as option order is positional.
+ record Key(String schema, Integer maxLen, Integer headMaxLen, String language) {
+ }
+ Map<Key, List<PredictContext>> forwardPasses = new LinkedHashMap<>();
+ for (PredictContext ctx : pending) {
+ String language = null;
+ if (tempered) {
+ int i = indexOf.get(ctx);
+ language = requests.get(i).options().lang();
+ if (language == null && decisions.get(i).detection() != null) {
+ language = decisions.get(i).detection().language();
+ }
+ }
+ Key key = new Key(questionSchema(ctx.questions()), ctx.maxLen(), ctx.headMaxLen(),
+ language);
+ forwardPasses.computeIfAbsent(key, k -> new ArrayList<>()).add(ctx);
+ }
+ for (Map.Entry<Key, List<PredictContext>> pass : forwardPasses.entrySet()) {
+ List<PredictContext> members = pass.getValue();
+ List<Object> states = new ArrayList<>(members.size());
+ for (PredictContext ctx : members) {
+ states.add(ctx.states().get(0));
+ }
+ Key key = pass.getKey();
+ List<Prediction> answered = agent.predictBatch(states, members.get(0).questions(),
+ key.language(), batchSize, sortByLength, HookCall.none(), key.maxLen(),
+ key.headMaxLen());
+ if (answered.size() != members.size()) {
+ throw new IllegalStateException("internal error: Agent.predictBatch returned "
+ + answered.size() + " results for " + members.size() + " states");
+ }
+ for (int m = 0; m < members.size(); m++) {
+ members.get(m).results(List.of(answered.get(m)));
+ }
+ }
+ }
+
/**
* Close the router, freeing every checkpoint it built.
*
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
index eb76b6e6..b914bc90 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
@@ -812,4 +812,88 @@ public final class Hooks {
problem.addSuppressed(hookFailure);
}
}
+
+ /**
+ * {@link #around} for many requests that share forward passes: each keeps its own context and
+ * its own start and end, while {@code inference} answers the ones still pending in one go.
+ *
+ * <p>The reference's {@code Router.predict_batch} sequence. Every context starts, in order,
+ * before any inference; {@code inference} must assign {@code results} on each context it is
+ * handed. Contexts then end in <b>reverse</b> of the order they started, so a hook that sets
+ * something in start and resets it in end unwinds the last one first. On a failure every
+ * started context fails with it, a skipped one included, and gets the error event before its
+ * end; a hook throwing on a failed context is attached to the failure. Every context gets its
+ * end even when another's end hook throws; the first such throw is raised afterwards.
+ *
+ * @param inference answers the contexts no start hook answered; not called when there are none
+ */
+ public static void aroundGroup(List<? extends Hook> hooks, List<PredictContext> contexts,
+ Policy policy, Consumer<List<PredictContext>> inference) {
+ List<PredictContext> started = new ArrayList<>(contexts.size());
+ Throwable raised = null;
+ try {
+ for (PredictContext ctx : contexts) {
+ started.add(ctx);
+ dispatch(hooks, Event.PREDICT_START, ctx, policy);
+ }
+ List<PredictContext> pending = new ArrayList<>();
+ for (PredictContext ctx : started) {
+ if (ctx.results() != null) {
+ continue;
+ }
+ if (ctx.states().isEmpty()) {
+ ctx.results(List.of());
+ } else {
+ pending.add(ctx);
+ }
+ }
+ if (!pending.isEmpty()) {
+ inference.accept(List.copyOf(pending));
+ }
+ // Totalled before any end, so a malformed result fails the whole group rather than
+ // escaping after some of its requests have already ended.
+ List<Totals> totals = new ArrayList<>(started.size());
+ for (PredictContext ctx : started) {
+ totals.add(ctx.results() == null ? null : aggregateUsage(ctx.results()));
+ }
+ for (int i = 0; i < started.size(); i++) {
+ started.get(i).usage(totals.get(i));
+ }
+ } catch (RuntimeException | Error problem) {
+ raised = problem;
+ }
+
+ double now = System.nanoTime();
+ for (PredictContext ctx : started) {
+ ctx.elapsedMs((now - ctx.startedAt()) / 1_000_000.0);
+ if (raised != null) {
+ ctx.error(raised);
+ }
+ }
+ Throwable firstEndFailure = null;
+ for (int i = started.size() - 1; i >= 0; i--) {
+ PredictContext ctx = started.get(i);
+ List<Event> events = raised == null ? List.of(Event.PREDICT_END)
+ : List.of(Event.ERROR, Event.PREDICT_END);
+ for (Event event : events) {
+ try {
+ dispatch(hooks, event, ctx, policy);
+ } catch (RuntimeException | Error hookFailure) {
+ if (raised != null) {
+ attach(raised, hookFailure);
+ } else if (firstEndFailure == null) {
+ firstEndFailure = hookFailure;
+ }
+ }
+ }
+ }
+
+ Throwable failure = raised != null ? raised : firstEndFailure;
+ if (failure instanceof RuntimeException problem) {
+ throw problem;
+ }
+ if (failure instanceof Error problem) {
+ throw problem;
+ }
+ }
}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.java
new file mode 100644
index 00000000..e0f26086
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchParityTest.java
@@ -0,0 +1,143 @@
+package com.convaiinnovations.laya;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertIterableEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.convaiinnovations.laya.Router.Checkpoint;
+import com.convaiinnovations.laya.Router.Request;
+import com.convaiinnovations.laya.Router.RouteOptions;
+import com.convaiinnovations.laya.hooks.HookCall;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+
+/**
+ * {@link Router#predictBatch(List)} on real checkpoints equals one {@link Router#predict} per
+ * request. Skipped unless the multilingual and typed-decisions checkpoints and both graphs are
+ * configured; see {@link Fixtures}.
+ */
+final class RouterBatchParityTest {
+
+ /** Batched rows are padded to their group's longest, so the logits may move in the last bits. */
+ private static final double TOLERANCE = 1e-4;
+
+ private static Path graph(String env) {
+ String graph = System.getenv(env);
+ Assumptions.assumeTrue(graph != null && !graph.isBlank(), "set " + env);
+ Path path = Paths.get(graph);
+ Path directory = Files.isDirectory(path) ? path : path.getParent();
+ Assumptions.assumeTrue(directory != null && Files.isDirectory(directory),
+ env + " does not name a graph: " + graph);
+ return directory;
+ }
+
+ private static Path checkpoint(String name) {
+ Path model = Fixtures.checkpoint(name);
+ Assumptions.assumeTrue(model != null, Fixtures.missingCheckpoint(name));
+ return model;
+ }
+
+ private static Map<String, Question> triage() {
+ Map<String, Question> questions = new LinkedHashMap<>();
+ questions.put("urgent", Question.noul("Does this need a human today?"));
+ questions.put("topic", Question.choice("What is this about?", TinyCheckpoint.ordered(
+ "billing", "charges, refunds, invoices", "access", "login or password",
+ "other", "anything else")));
+ return questions;
+ }
+
+ private static Map<String, Question> sentiment() {
+ return Map.of("angry", Question.noul("Is the writer angry?"));
+ }
+
+ @Test
+ @DisplayName("predictBatch equals per-request predict on real checkpoints")
+ void batchEqualsPerRequest() throws IOException {
+ Path multilingual = checkpoint("multilingual");
+ Path typed = checkpoint("typed-decisions");
+ Path multilingualGraph = graph(Fixtures.GRAPH_ENV);
+ Path typedGraph = graph(Fixtures.TYPED_GRAPH_ENV);
+ RouteOptions toMultilingual = RouteOptions.none().model("multilingual");
+ RouteOptions toTyped = RouteOptions.none().model("typed-decisions");
+ List<Request> requests = List.of(
+ Request.of("मेरे कार्ड से दो बार पैसे कटे, कृपया जल्दी मदद करें", triage()),
+ Request.of("I was charged twice for one order", triage()).options(toMultilingual),
+ Request.of("No puedo entrar a mi cuenta y estoy harto", sentiment())
+ .options(toMultilingual),
+ Request.of("Please refund the duplicate charge on invoice 4411", triage())
+ .options(toTyped),
+ Request.of("我的账户被锁定了,无法登录", triage()),
+ Request.of("The app logs me out every time, fix it now", sentiment())
+ .options(toTyped),
+ Request.of("I was charged twice for one order and nobody has answered my three "
+ + "emails about it, so I am asking again", sentiment())
+ .options(toMultilingual).maxLen(32));
+ Router.AgentFactory agents = checkpointName -> switch (checkpointName) {
+ case MULTILINGUAL -> Agent.open(multilingual, multilingualGraph);
+ case TYPED_DECISIONS -> Agent.open(typed, typedGraph);
+ default -> throw new IOException("no graph for " + checkpointName);
+ };
+ try (Router router = Router.builder().agents(agents).maxLoaded(2).build()) {
+ List<Prediction> batched = router.predictBatch(requests);
+ assertEquals(requests.size(), batched.size());
+ for (int i = 0; i < requests.size(); i++) {
+ Request request = requests.get(i);
+ Prediction single;
+ if (request.maxLen() == null) {
+ single = router.predict(request.state(), request.questions(),
+ request.options());
+ } else {
+ // Router.predict takes no budget, so the reference is the agent with one.
+ Checkpoint target = router.route(request.state(), request.questions(),
+ request.options()).model();
+ try (Router.Lease lease = router.lease(target)) {
+ single = lease.agent().predictBatch(List.of(request.state()),
+ request.questions(), null, 0, false,
+ HookCall.none().onStart(ctx -> ctx.maxLen(request.maxLen())))
+ .get(0);
+ }
+ assertTrue(single.usage().stateTokensDropped() > 0,
+ "the budget truncated: " + single.usage());
+ }
+ assertSame("request " + i, single, batched.get(i));
+ }
+ assertEquals(List.of(Checkpoint.MULTILINGUAL, Checkpoint.TYPED_DECISIONS),
+ new ArrayList<>(router.loaded()).stream().sorted().toList());
+ }
+ }
+
+ private static void assertSame(String at, Prediction want, Prediction got) {
+ assertEquals(want.model(), got.model(), at + ".model");
+ assertEquals(want.usage(), got.usage(), at + ".usage");
+ assertIterableEquals(want.answers().keySet(), got.answers().keySet(), at + " ids");
+ for (Map.Entry<String, Answer> entry : want.answers().entrySet()) {
+ String id = at + "." + entry.getKey();
+ Answer w = entry.getValue();
+ Answer g = got.answers().get(entry.getKey());
+ assertEquals(w.type(), g.type(), id + ".type");
+ assertEquals(w.confidence(), g.confidence(), TOLERANCE, id + ".confidence");
+ assertEquals(w.answerConfidence(), g.answerConfidence(), TOLERANCE,
+ id + ".answer_confidence");
+ assertEquals(w.actProbability(), g.actProbability(), TOLERANCE, id + ".act");
+ if (w instanceof Answer.Choice wc && g instanceof Answer.Choice gc) {
+ assertIterableEquals(wc.probabilities().keySet(), gc.probabilities().keySet());
+ for (String option : wc.probabilities().keySet()) {
+ assertEquals(wc.probabilities().get(option), gc.probabilities().get(option),
+ TOLERANCE, id + ".p[" + option + "]");
+ }
+ assertEquals(wc.choice(), gc.choice(), id + ".choice");
+ } else if (w instanceof Answer.Noul wn && g instanceof Answer.Noul gn) {
+ assertEquals(wn.noul(), gn.noul(), TOLERANCE, id + ".noul");
+ }
+ }
+ }
+}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java
new file mode 100644
index 00000000..e28d9367
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java
@@ -0,0 +1,497 @@
+package com.convaiinnovations.laya;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.convaiinnovations.laya.Router.Checkpoint;
+import com.convaiinnovations.laya.Router.Request;
+import com.convaiinnovations.laya.Router.RouteOptions;
+import com.convaiinnovations.laya.hooks.Hook;
+import com.convaiinnovations.laya.hooks.PredictContext;
+import com.convaiinnovations.laya.infer.InferenceSession;
+import com.convaiinnovations.laya.sequence.Collator;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * {@link Router#predictBatch(List)} with a stub agent per checkpoint, each on its own recording
+ * session, so which checkpoint answered and how many graph calls it took are observable.
+ */
+class RouterBatchTest {
+
+ private static final RouteOptions ENGLISH = RouteOptions.none().model("english");
+ private static final RouteOptions MULTILINGUAL = RouteOptions.none().model("multilingual");
+
+ @TempDir
+ Path root;
+
+ private final Map<Checkpoint, TinyCheckpoint.RecordingSession> sessions =
+ new EnumMap<>(Checkpoint.class);
+
+ @BeforeEach
+ void writeCheckpoint() throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ }
+
+ /** A router whose every checkpoint is a stub agent on a session of its own. */
+ private Router router() throws IOException {
+ Router router = Router.builder().maxLoaded(3).build();
+ for (Checkpoint checkpoint : Checkpoint.values()) {
+ TinyCheckpoint.RecordingSession session = new TinyCheckpoint.RecordingSession();
+ sessions.put(checkpoint, session);
+ router.attach(checkpoint.wireName(), TinyCheckpoint.agent(root, session));
+ }
+ return router;
+ }
+
+ private int calls(Checkpoint checkpoint) {
+ return sessions.get(checkpoint).batches.size();
+ }
+
+ private static Map<String, Question> urgent() {
+ return Map.of("urgent", Question.noul("Needs a human."));
+ }
+
+ private static Map<String, Question> topic() {
+ Map<String, Question> questions = new LinkedHashMap<>();
+ questions.put("topic", Question.choice("Topic.", TinyCheckpoint.ordered(
+ "billing", "money", "login", "access", "other", "anything else")));
+ return questions;
+ }
+
+ private static List<Request> interleaved() {
+ return List.of(
+ Request.of("refund my card", urgent()).options(ENGLISH),
+ Request.of("mi tarjeta", urgent()).options(MULTILINGUAL),
+ Request.of("cannot log in", topic()).options(ENGLISH),
+ Request.of("बिल गलत है", topic()).options(MULTILINGUAL),
+ Request.of("charged twice", urgent()).options(ENGLISH),
+ Request.of("doppelt belastet", urgent()).options(MULTILINGUAL));
+ }
+
+ @Test
+ @DisplayName("results come back in request order and equal per-request predict")
+ void orderPreservedAndEqualToPredict() throws IOException {
+ List<Request> requests = interleaved();
+ List<Prediction> batched;
+ try (Router router = router()) {
+ batched = router.predictBatch(requests);
+ }
+ assertEquals(requests.size(), batched.size());
+ try (Router router = router()) {
+ for (int i = 0; i < requests.size(); i++) {
+ Request request = requests.get(i);
+ Prediction single = router.predict(request.state(), request.questions(),
+ request.options());
+ assertEquals(single, batched.get(i), "request " + i);
+ }
+ }
+ // Distinct per state, so a swap of two results could not pass the comparison above.
+ assertNotEquals(batched.get(0), batched.get(4));
+ }
+
+ @Test
+ @DisplayName("an interleaved workload costs one forward pass per checkpoint and schema")
+ void oneCallPerCheckpoint() throws IOException {
+ try (Router router = router()) {
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(MULTILINGUAL),
+ Request.of("c", urgent()).options(ENGLISH),
+ Request.of("d", urgent()).options(MULTILINGUAL),
+ Request.of("e", urgent()).options(ENGLISH)));
+ assertEquals(1, calls(Checkpoint.ENGLISH));
+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));
+ assertEquals(0, calls(Checkpoint.TYPED_DECISIONS));
+ assertEquals(3, sessions.get(Checkpoint.ENGLISH).batches.get(0).rows());
+ assertEquals(2, sessions.get(Checkpoint.MULTILINGUAL).batches.get(0).rows());
+ }
+ }
+
+ @Test
+ @DisplayName("requests with different schemas or budgets on one checkpoint split the pass")
+ void schemaAndBudgetSplit() throws IOException {
+ try (Router router = router()) {
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", topic()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH).maxLen(48),
+ Request.of("d", urgent()).options(ENGLISH)));
+ List<Collator.Batch> batches = sessions.get(Checkpoint.ENGLISH).batches;
+ assertEquals(3, batches.size(), "urgent, topic, urgent at maxLen 48");
+ assertEquals(List.of(2, 1, 1), batches.stream().map(Collator.Batch::rows).toList());
+ }
+ }
+
+ @Test
+ @DisplayName("a per-request budget reaches the agent")
+ void budgetForwarded() throws IOException {
+ String state = "x".repeat(200);
+ try (Router router = router()) {
+ Prediction full = router.predictBatch(List.of(
+ Request.of(state, urgent()).options(ENGLISH))).get(0);
+ Prediction capped = router.predictBatch(List.of(
+ Request.of(state, urgent()).options(ENGLISH).maxLen(40))).get(0);
+ assertTrue(capped.usage().inputTokens() < full.usage().inputTokens(),
+ full.usage() + " vs " + capped.usage());
+ }
+ }
+
+ @Test
+ @DisplayName("model pins each request, overriding detection")
+ void pinForwarded() throws IOException {
+ try (Router router = router()) {
+ // English text, which detection would send to the english checkpoint.
+ router.predictBatch(List.of("my card was charged twice", "cannot log in"), urgent(),
+ MULTILINGUAL);
+ assertEquals(0, calls(Checkpoint.ENGLISH));
+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));
+ router.predictBatch(List.of(
+ Request.of("my card was charged twice", urgent()).options(
+ RouteOptions.none().model("typed")),
+ Request.of("my card was charged twice", urgent())));
+ assertEquals(1, calls(Checkpoint.TYPED_DECISIONS), "alias pin");
+ assertEquals(1, calls(Checkpoint.ENGLISH), "the unpinned request routes itself");
+ }
+ }
+
+ @Test
+ @DisplayName("the BatchPredictor form routes each state on its own")
+ void statesRouteApart() throws IOException {
+ try (Router router = router()) {
+ List<Prediction> out = router.predictBatch(
+ List.of("my card was charged twice", "मेरा कार्ड दो बार चार्ज हुआ"), urgent());
+ assertEquals(2, out.size());
+ assertEquals(1, calls(Checkpoint.ENGLISH));
+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));
+ }
+ }
+
+ @Test
+ @DisplayName("Decisions.decideBatch runs on a router and equals decide per state")
+ void decideBatchOnRouter() throws IOException {
+ Map<String, Object> schema = Map.of("type", "object", "properties",
+ new LinkedHashMap<>(Map.of("urgent", Map.of("type", "boolean"))));
+ List<String> states = List.of("my card was charged twice", "मेरा कार्ड दो बार चार्ज हुआ",
+ "login broken");
+ try (Router router = router()) {
+ List<Decisions.Decision> batched = Decisions.decideBatch(router, states, schema);
+ assertEquals(states.size(), batched.size());
+ for (int i = 0; i < states.size(); i++) {
+ assertEquals(Decisions.decide(router, states.get(i), schema).values(),
+ batched.get(i).values(), "state " + i);
+ }
+ }
+ try (Router router = router()) {
+ BatchPredictor pinned = (s, asked) -> router.predictBatch(s, asked, MULTILINGUAL);
+ Decisions.decideBatch(pinned, states, schema);
+ assertEquals(0, calls(Checkpoint.ENGLISH));
+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));
+ }
+ }
+
+ @Test
+ @DisplayName("an empty batch routes nothing and returns nothing")
+ void empty() throws IOException {
+ List<String> events = new ArrayList<>();
+ try (Router router = router()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onRoute(PredictContext ctx) {
+ events.add("route");
+ }
+ });
+ assertEquals(List.of(), router.predictBatch(List.<Request>of()));
+ assertEquals(List.of(), events);
+ }
+ }
+
+ @Test
+ @DisplayName("a null request, state or questions, or a negative batch size, is refused")
+ void refusals() throws IOException {
+ try (Router router = router()) {
+ List<Request> withNull = new ArrayList<>();
+ withNull.add(Request.of("a", urgent()));
+ withNull.add(null);
+ IllegalArgumentException missing = assertThrows(IllegalArgumentException.class,
+ () -> router.predictBatch(withNull));
+ assertTrue(missing.getMessage().contains("request 1"), missing.getMessage());
+ assertThrows(IllegalArgumentException.class, () -> Request.of(null, urgent()));
+ assertThrows(IllegalArgumentException.class, () -> Request.of("a", null));
+ assertThrows(IllegalArgumentException.class,
+ () -> router.predictBatch(List.of(Request.of("a", urgent())), -1, false));
+ assertEquals(0, calls(Checkpoint.ENGLISH));
+ }
+ }
+
+ /** Records each predict event with the state it carried. */
+ private static final class Trace implements Hook {
+
+ final List<String> events = new ArrayList<>();
+
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ events.add("start:" + ctx.states().get(0));
+ }
+
+ @Override
+ public void onError(PredictContext ctx) {
+ events.add("error:" + ctx.states().get(0));
+ }
+
+ @Override
+ public void onPredictEnd(PredictContext ctx) {
+ events.add("end:" + ctx.states().get(0));
+ }
+ }
+
+ @Test
+ @DisplayName("router hooks run per request: starts in order, ends reversed, per checkpoint")
+ void hooksPerRequest() throws IOException {
+ Trace trace = new Trace();
+ try (Router router = router()) {
+ router.hooks().addHook(trace);
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(MULTILINGUAL),
+ Request.of("c", urgent()).options(ENGLISH)));
+ }
+ assertEquals(List.of("start:a", "start:c", "end:c", "end:a", "start:b", "end:b"),
+ trace.events);
+ }
+
+ @Test
+ @DisplayName("a start hook's skip keeps its request out of the forward pass")
+ void skipLeavesThePass() throws IOException {
+ Prediction canned = new Prediction(Prediction.MODEL, Map.of(),
+ new Usage(0, 0, 0, 0, false, List.of(), Map.of()));
+ try (Router router = router()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ if ("cached".equals(ctx.states().get(0))) {
+ ctx.skip(List.of(canned));
+ }
+ }
+ });
+ List<Prediction> out = router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("cached", urgent()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH)));
+ assertSame(canned, out.get(1));
+ assertEquals(1, calls(Checkpoint.ENGLISH));
+ assertEquals(2, sessions.get(Checkpoint.ENGLISH).batches.get(0).rows());
+ }
+ }
+
+ @Test
+ @DisplayName("a start hook's budget applies to its own request only")
+ void hookBudgetPerRequest() throws IOException {
+ try (Router router = router()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ if ("b".equals(ctx.states().get(0))) {
+ ctx.maxLen(48);
+ }
+ }
+ });
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH)));
+ assertEquals(List.of(2, 1), sessions.get(Checkpoint.ENGLISH).batches.stream()
+ .map(Collator.Batch::rows).toList());
+ }
+ }
+
+ @Test
+ @DisplayName("a failed checkpoint fails each of its requests, after earlier ones ended")
+ void failureEndsEveryStartedRequest() throws IOException {
+ Trace trace = new Trace();
+ try (Router router = Router.builder().maxLoaded(3).build()) {
+ router.attach("english", TinyCheckpoint.agent(root,
+ new TinyCheckpoint.RecordingSession()));
+ router.attach("multilingual", TinyCheckpoint.agent(root, new InferenceSession() {
+ @Override
+ public Output run(Collator.Batch batch) {
+ throw new IllegalStateException("graph failed");
+ }
+
+ @Override
+ public void close() {
+ }
+ }));
+ router.hooks().addHook(trace);
+ IllegalStateException failure = assertThrows(IllegalStateException.class,
+ () -> router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(MULTILINGUAL),
+ Request.of("c", urgent()).options(MULTILINGUAL))));
+ assertEquals("graph failed", failure.getMessage());
+ }
+ assertEquals(List.of("start:a", "end:a", "start:b", "start:c",
+ "error:c", "end:c", "error:b", "end:b"), trace.events);
+ }
+
+ @Test
+ @DisplayName("a hook that leaves no result is refused, naming the request")
+ void emptyResultRefused() throws IOException {
+ try (Router router = router()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictEnd(PredictContext ctx) {
+ ctx.results(List.of());
+ }
+ });
+ IllegalStateException refused = assertThrows(IllegalStateException.class,
+ () -> router.predictBatch(List.of(Request.of("a", urgent()))));
+ assertTrue(refused.getMessage().contains("request 0"), refused.getMessage());
+ }
+ }
+
+ @Test
+ @DisplayName("per-language temperatures split the pass by language and still equal predict")
+ void languageSplitsOnlyWhenTempered(@TempDir Path tempered) throws IOException {
+ TinyCheckpoint.write(tempered, 64, 32);
+ Files.writeString(tempered.resolve("rl_agent_config.json"),
+ "{\"max_len\": 64, \"head_max_len\": 32, \"temperature\": [1.0, 1.0, 1.0],"
+ + " \"temperature_by_options\": {},"
+ + " \"lang_temperatures\": {\"hi\": {\"temperature\": [3.0, 3.0, 3.0]}}}");
+ List<Request> requests = List.of(
+ Request.of("a", urgent()).options(MULTILINGUAL.lang("hi")),
+ Request.of("b", urgent()).options(MULTILINGUAL.lang("en")),
+ Request.of("c", urgent()).options(MULTILINGUAL.lang("hi")));
+ TinyCheckpoint.RecordingSession session = new TinyCheckpoint.RecordingSession();
+ try (Router router = Router.builder().build()) {
+ router.attach("multilingual", TinyCheckpoint.agent(tempered, session));
+ List<Prediction> batched = router.predictBatch(requests);
+ assertEquals(2, session.batches.size(), "hi and en apart");
+ for (int i = 0; i < requests.size(); i++) {
+ Request request = requests.get(i);
+ assertEquals(router.predict(request.state(), request.questions(),
+ request.options()), batched.get(i), "request " + i);
+ }
+ assertNotEquals(router.predict("a", urgent(), MULTILINGUAL.lang("en")),
+ batched.get(0), "the temperature took effect");
+ }
+ // Untempered, the same three share one pass.
+ try (Router router = router()) {
+ router.predictBatch(requests);
+ assertEquals(1, calls(Checkpoint.MULTILINGUAL));
+ }
+ }
+
+ /** Lifecycle events by checkpoint, predict events by state. */
+ private static final class Lifecycle implements Hook {
+
+ final List<String> events = new ArrayList<>();
+
+ @Override
+ public void onRoute(PredictContext ctx) {
+ events.add("route:" + ctx.model());
+ }
+
+ @Override
+ public void onLoad(PredictContext ctx) {
+ events.add("load:" + ctx.model());
+ }
+
+ @Override
+ public void onEvict(PredictContext ctx) {
+ events.add("evict:" + ctx.model());
+ }
+
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ events.add("start:" + ctx.states().get(0));
+ }
+
+ @Override
+ public void onError(PredictContext ctx) {
+ events.add("error:" + ctx.states().get(0) + "@" + ctx.model());
+ }
+
+ @Override
+ public void onPredictEnd(PredictContext ctx) {
+ events.add("end:" + ctx.states().get(0));
+ }
+ }
+
+ /** Builds stub agents on demand, failing for the checkpoints named. */
+ private Router.AgentFactory building(Checkpoint... failing) {
+ List<Checkpoint> refused = List.of(failing);
+ return checkpoint -> {
+ if (refused.contains(checkpoint)) {
+ throw new IOException("cannot build " + checkpoint.wireName());
+ }
+ return TinyCheckpoint.agent(root, new TinyCheckpoint.RecordingSession());
+ };
+ }
+
+ @Test
+ @DisplayName("every route, then per checkpoint evict/load before its starts, as the reference")
+ void lifecycleOrder() throws IOException {
+ Lifecycle trace = new Lifecycle();
+ try (Router router = Router.builder().agents(building()).maxLoaded(1).build()) {
+ router.hooks().addHook(trace);
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(MULTILINGUAL),
+ Request.of("c", urgent()).options(ENGLISH)));
+ assertEquals(List.of("route:english", "route:multilingual", "route:english",
+ "load:english", "start:a", "start:c", "end:c", "end:a",
+ "evict:english", "load:multilingual", "start:b", "end:b"), trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("a load failure reaches every request of its checkpoint, with no start")
+ void loadFailureBeforeStart() throws IOException {
+ Lifecycle trace = new Lifecycle();
+ try (Router router = Router.builder().agents(building(Checkpoint.MULTILINGUAL))
+ .build()) {
+ router.hooks().addHook(trace);
+ RuntimeException failure = assertThrows(RuntimeException.class,
+ () -> router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(MULTILINGUAL),
+ Request.of("c", urgent()).options(MULTILINGUAL))));
+ assertTrue(String.valueOf(failure.getCause()).contains("cannot build multilingual")
+ || failure.getMessage().contains("cannot build multilingual"),
+ failure.toString());
+ assertEquals(List.of("route:english", "route:multilingual", "route:multilingual",
+ "load:english", "start:a", "end:a",
+ "error:b@multilingual", "end:b", "error:c@multilingual", "end:c"),
+ trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("a routing failure reaches its own request before anything loads")
+ void routeFailureBeforeStart() throws IOException {
+ Lifecycle trace = new Lifecycle();
+ try (Router router = Router.builder().agents(building()).build()) {
+ router.hooks().addHook(trace);
+ assertThrows(IllegalArgumentException.class, () -> router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("bad", urgent()).options(RouteOptions.none().model("nope")),
+ Request.of("c", urgent()).options(ENGLISH))));
+ assertEquals(List.of(), router.loaded());
+ assertEquals(List.of("route:english", "error:bad@null", "end:bad"), trace.events);
+ }
+ }
+}