Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): refuse a malformed predict before it loads, and what else reviewing 743fc47 found

Adeshina Lasisi1d1b6564fec9
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index b777fd0d..30ed56c2 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 991 register here, 974 run, 17 abort on a checkpoint this
+ # Measured on the merged tree: 1000 register here, 983 run, 17 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 975. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 984. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 991 registered, 17 abort,
- # 974 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 1000 registered, 17 abort,
+ # 983 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 974 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 983 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 975, not the build lane's 974, and the one is the point: this lane runs one test MORE
+ # 984, not the build lane's 983, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 975 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 984 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index e90a85cc..9d0e571b 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 974 of the 991 registered tests run without a checkpoint. This floor has been raised
+ # 983 of the 1000 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 974 --allow-aborted
+ --min-tests 983 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/README.md b/laya-java/README.md
index e3645e9c..d5c3f01d 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -493,8 +493,9 @@ events no agent can:
`Router.predict` also dispatches one `onPredictStart`/`onPredictEnd` pair for the answer. As in the
reference, routing and loading run first, so the order is `onRoute`, any `onEvict`/`onLoad`, then
-the pair, and `elapsedMs` does not include a cold load. A routing or loading failure still reaches
-`onError` and `onPredictEnd`, with no start event. The agent's own pair still
+the pair, and on success `elapsedMs` does not include a cold load. A routing or loading failure
+reaches `onError` and `onPredictEnd`, with no start event, and its `elapsedMs` covers the routing
+and the failed load. The agent's own pair still
fires for the forward pass, so a router-level hook and an agent-level hook see different spans.
Hooks are dispatched outside the router's lock, so a hook may call back into the router —
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
index 6485ae55..7a9ef177 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
@@ -962,12 +962,19 @@ public final class Router implements AutoCloseable, Predictor {
}
}
+ /** {@code addSuppressed}, unless the hook rethrew {@code failure} itself, which cannot suppress itself. */
+ private static void attachHookFailure(Throwable failure, Throwable hookFailure) {
+ if (hookFailure != failure) {
+ failure.addSuppressed(hookFailure);
+ }
+ }
+
/** {@link #dispatchOutcome} on a failing path: a hook failure is attached, not substituted. */
private void dispatchOwed(Checkpoint checkpoint, Outcome outcome, Throwable failure) {
try {
dispatchOutcome(checkpoint, outcome);
} catch (RuntimeException | Error hookFailure) {
- failure.addSuppressed(hookFailure);
+ attachHookFailure(failure, hookFailure);
}
}
@@ -1364,7 +1371,7 @@ public final class Router implements AutoCloseable, Predictor {
try {
dispatchLifecycle(Hooks.Event.EVICT, evicted);
} catch (RuntimeException | Error hookFailure) {
- failure.addSuppressed(hookFailure);
+ attachHookFailure(failure, hookFailure);
}
}
throw failure;
@@ -1408,7 +1415,7 @@ public final class Router implements AutoCloseable, Predictor {
try {
dispatchLifecycle(Hooks.Event.EVICT, evicted);
} catch (RuntimeException | Error hookFailure) {
- failure.addSuppressed(hookFailure);
+ attachHookFailure(failure, hookFailure);
}
}
throw failure;
@@ -1484,6 +1491,14 @@ public final class Router implements AutoCloseable, Predictor {
*/
private Prediction predictRouted(Supplier<RouteDecision> decide, Object state,
Map<String, Question> questions, String lang) {
+ // Before routing, as the reference does: a malformed call must not cold-load a
+ // checkpoint, fire on_load or evict a resident one before it is refused.
+ if (state == null) {
+ throw new IllegalArgumentException("state must not be null");
+ }
+ if (questions == null) {
+ throw new IllegalArgumentException("questions must not be null");
+ }
List<Hook> composed = hooks.composeFor(HookCall.none());
if (composed.isEmpty()) {
RouteDecision decision = decide.get();
@@ -1492,6 +1507,7 @@ public final class Router implements AutoCloseable, Predictor {
}
}
Hooks.Policy policy = hooks.policyFor(HookCall.none());
+ long startedAt = System.nanoTime(); // a failure's elapsedMs covers route + load
RouteDecision decision = null;
Lease lease;
try {
@@ -1500,7 +1516,7 @@ public final class Router implements AutoCloseable, Predictor {
} catch (RuntimeException | Error failure) {
PredictContext failed = new PredictContext(Collections.singletonList(state), questions,
decision == null ? null : decision.model().wireName(), this);
- Hooks.failedBeforeStart(composed, failed, policy, failure);
+ Hooks.failedBeforeStart(composed, failed, policy, failure, startedAt);
throw failure;
}
// `Hooks.around` so ctx.skip, the error-before-end order, failure suppression and
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
index 0e1c3305..4eb31433 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
@@ -27,8 +27,9 @@ import java.util.function.Consumer;
* {@code hooks_timeout}: anything it writes to the context afterwards is refused. Unlike that
* case the executor's thread is reused, so the abandonment ends when the overrunning task
* returns, and a later callback of the same call on that thread is accepted; an abandonment
- * made by anything else on that thread stays in force. A callback that had not started by the
- * deadline never runs. It still occupies the executor's
+ * made by anything else on that thread stays in force. A callback that had not started when the
+ * caller stopped waiting -- on the deadline, or on an interrupt while it waited for one -- never
+ * runs. Without a deadline the wait ignores interrupts, so it always ends with the callback. It still occupies the executor's
* thread until it returns, so a single-threaded executor has none left for the next callback --
* size the executor for the deadline, or leave the deadline off.
*/
@@ -153,14 +154,7 @@ public final class AsyncHook implements Hook {
} catch (TimeoutException expired) {
// cancel(true) cannot interrupt a task already running on an executor, so the callback
// is cut off from the call instead: whatever it writes from here on is refused.
- synchronized (gate) {
- gaveUp[0] = true;
- Thread overrunning = runner.get();
- if (overrunning != null && !finished[0]) {
- ctx.abandon(overrunning);
- abandonedHere[0] = true;
- }
- }
+ giveUp(gate, gaveUp, finished, abandonedHere, runner, ctx);
running.cancel(true);
throw new IllegalStateException(
delegate + " did not finish " + event + " within " + timeout, expired);
@@ -168,11 +162,27 @@ public final class AsyncHook implements Hook {
// The flag is restored before unwinding, so a caller using interruption to cancel is
// not left thinking the interrupt was swallowed.
Thread.currentThread().interrupt();
+ // The call stops waiting here just as it does on a deadline, so the same cut-off.
+ giveUp(gate, gaveUp, finished, abandonedHere, runner, ctx);
running.cancel(true);
throw new IllegalStateException(delegate + " was interrupted in " + event, interrupted);
}
}
+ /** The call has stopped waiting: a callback not yet started never runs, a running one is cut off. */
+ private static void giveUp(Object gate, boolean[] gaveUp, boolean[] finished,
+ boolean[] abandonedHere, AtomicReference<Thread> runner,
+ PredictContext ctx) {
+ synchronized (gate) {
+ gaveUp[0] = true;
+ Thread overrunning = runner.get();
+ if (overrunning != null && !finished[0]) {
+ ctx.abandon(overrunning);
+ abandonedHere[0] = true;
+ }
+ }
+ }
+
@Override
public String toString() {
return "AsyncHook(" + delegate + (timeout == null ? "" : ", timeout=" + timeout) + ")";
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
index e0bde893..2f37a0dd 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hooks.java
@@ -1,5 +1,7 @@
package com.convaiinnovations.laya.hooks;
+import static java.util.Objects.requireNonNull;
+
import com.convaiinnovations.laya.Prediction;
import com.convaiinnovations.laya.Question;
import java.time.Duration;
@@ -567,6 +569,8 @@ public final class Hooks {
// flag: a caller up the stack that polls `Thread.interrupted()` to decide whether to
// shut down would otherwise be told it was never asked to.
Thread.currentThread().interrupt();
+ // The call stops waiting here, so the hook is cut off exactly as on a deadline.
+ ctx.abandon(runner);
throw new IllegalStateException("interrupted while waiting for hook "
+ hookName(hook) + "." + event.wireName(), interrupted);
}
@@ -741,7 +745,7 @@ public final class Hooks {
} catch (RuntimeException | Error hookFailure) {
// Attached, never substituted: the thing that broke the request is what the
// caller has to catch, and a failing observer must not be able to hide it.
- problem.addSuppressed(hookFailure);
+ attach(problem, hookFailure);
}
}
@@ -755,7 +759,7 @@ public final class Hooks {
if (raised == null) {
throw hookFailure;
}
- raised.addSuppressed(hookFailure);
+ attach(raised, hookFailure);
}
if (raised instanceof RuntimeException problem) {
@@ -770,20 +774,41 @@ public final class Hooks {
/**
* Reports a failure that happened before {@code on_predict_start} could fire -- routing or
* loading, which the reference runs inside its predict {@code try} -- as {@code on_error} and
- * then {@code on_predict_end}. A hook failure is attached to {@code problem}, never substituted.
+ * then {@code on_predict_end}, with {@code elapsedMs} measured from {@code startedAtNanos}.
+ * A hook failure is attached to {@code problem}, never substituted.
+ *
+ * <p>For a {@link com.convaiinnovations.laya.Predictor} that must do work before its start
+ * event, as {@code Router} does. {@code ctx} must be fresh: one that already carries results
+ * or an error belongs to a call that has been reported, and is refused.
*/
public static void failedBeforeStart(List<? extends Hook> hooks, PredictContext ctx,
- Policy policy, Throwable problem) {
+ Policy policy, Throwable problem, long startedAtNanos) {
+ requireNonNull(problem, "problem");
+ if (ctx.results() != null || ctx.error() != null) {
+ throw new IllegalStateException(
+ "failedBeforeStart needs a fresh context; this one already has an outcome");
+ }
ctx.error(problem);
try {
dispatch(hooks, Event.ERROR, ctx, policy);
} catch (RuntimeException | Error hookFailure) {
- problem.addSuppressed(hookFailure);
+ attach(problem, hookFailure);
}
- ctx.elapsedMs((System.nanoTime() - ctx.startedAt()) / 1_000_000.0);
+ ctx.elapsedMs((System.nanoTime() - startedAtNanos) / 1_000_000.0);
try {
dispatch(hooks, Event.PREDICT_END, ctx, policy);
} catch (RuntimeException | Error hookFailure) {
+ attach(problem, hookFailure);
+ }
+ }
+
+ /**
+ * {@code addSuppressed}, except when a hook rethrew the failure it was shown: a throwable
+ * cannot suppress itself, and trying replaces the real failure with an
+ * {@code IllegalArgumentException}.
+ */
+ private static void attach(Throwable problem, Throwable hookFailure) {
+ if (hookFailure != problem) {
problem.addSuppressed(hookFailure);
}
}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
index 3fa12769..e4e2c4d2 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
@@ -7,6 +7,7 @@ import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@@ -65,9 +66,7 @@ public final class PredictContext {
* <p>Empty for every call that does not use {@link Hooks.Policy#timeout}, and at most one
* entry per overrun. A set rather than a flag because a chain can overrun more than once.
*/
- // A count, not a set: a hook abandoned by Hooks can call an AsyncHook that abandons and
- // releases the same thread, and that release must not cancel the outer abandonment.
- private final Map<Thread, Integer> abandoned = new ConcurrentHashMap<>();
+ private final Set<Thread> abandoned = ConcurrentHashMap.newKeySet();
/**
* A context for one call.
@@ -310,20 +309,20 @@ public final class PredictContext {
* call stops being able to wait for it. See {@link #states(List)}.
*/
void abandon(Thread runner) {
- abandoned.merge(runner, 1, Integer::sum);
+ abandoned.add(runner);
}
/**
* Undoes one {@link #abandon} once the overrunning task has returned, so a pooled thread that
* runs this call's next callback is not refused for the previous one's overrun. Only the
- * caller that abandoned may release, and abandonments of one thread nest.
+ * caller that abandoned a thread may release it.
*/
void release(Thread runner) {
- abandoned.computeIfPresent(runner, (thread, count) -> count == 1 ? null : count - 1);
+ abandoned.remove(runner);
}
private void refuseIfAbandoned(String what) {
- if (!abandoned.isEmpty() && abandoned.containsKey(Thread.currentThread())) {
+ if (!abandoned.isEmpty() && abandoned.contains(Thread.currentThread())) {
throw new IllegalStateException(String.format(
"laya: hook thread %s exceeded its deadline and was abandoned; this call has "
+ "moved on, so ctx.%s() is refused rather than applied to it",
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
index 73ff48b1..030e94e6 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
@@ -236,6 +236,86 @@ class AsyncHookTest {
assertEquals(List.of("state"), ctx.states());
}
+ @Test
+ @DisplayName("an interrupted wait cuts the callback off, as a deadline does")
+ void anInterruptedWaitAbandonsTheCallback() throws Exception {
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+ try {
+ CountDownLatch started = new CountDownLatch(1);
+ CountDownLatch done = new CountDownLatch(1);
+ Hook async = AsyncHook.of(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext c) {
+ started.countDown();
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ try {
+ c.states(List.of("late"));
+ } catch (RuntimeException refused) {
+ // refused, which is the point
+ } finally {
+ done.countDown();
+ }
+ }
+ }, pool, Duration.ofSeconds(5)); // timed: an untimed join() ignores interrupts
+ assertLateWriteRefusedAfterInterrupt(ctx -> async.onPredictStart(ctx), started, done);
+ } finally {
+ pool.shutdownNow();
+ }
+ }
+
+ @Test
+ @DisplayName("an interrupted wait on a Hooks timeout thread cuts that hook off too")
+ void anInterruptedHooksWaitAbandonsTheHook() throws Exception {
+ CountDownLatch started = new CountDownLatch(1);
+ CountDownLatch done = new CountDownLatch(1);
+ Hook slow = new Hook() {
+ @Override
+ public void onPredictStart(PredictContext c) {
+ started.countDown();
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ try {
+ c.states(List.of("late"));
+ } catch (RuntimeException refused) {
+ // refused, which is the point
+ } finally {
+ done.countDown();
+ }
+ }
+ };
+ assertLateWriteRefusedAfterInterrupt(ctx -> Hooks.dispatch(List.of(slow),
+ Hooks.Event.PREDICT_START, ctx, Hooks.Policy.raising().timeout(Duration.ofSeconds(5))),
+ started, done);
+ }
+
+ private static void assertLateWriteRefusedAfterInterrupt(
+ java.util.function.Consumer<PredictContext> waitOn, CountDownLatch started,
+ CountDownLatch done) throws Exception {
+ PredictContext ctx = ctx();
+ AtomicReference<Throwable> caught = new AtomicReference<>();
+ Thread caller = new Thread(() -> {
+ try {
+ waitOn.accept(ctx);
+ } catch (RuntimeException e) {
+ caught.set(e);
+ }
+ });
+ caller.start();
+ assertTrue(started.await(3, TimeUnit.SECONDS));
+ caller.interrupt();
+ caller.join(3_000);
+ assertTrue(caught.get() instanceof IllegalStateException, String.valueOf(caught.get()));
+ assertTrue(done.await(3, TimeUnit.SECONDS));
+ assertEquals(List.of("state"), ctx.states(), "a write after the caller moved on landed");
+ }
+
@Test
@DisplayName("a non-positive timeout is refused rather than meaning 'never wait'")
void refusesANonPositiveTimeout() {
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
index 10f7603f..5d8fe542 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
@@ -2,6 +2,7 @@ package com.convaiinnovations.laya;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -300,8 +301,10 @@ class RouterHooksTest {
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> router.predict("hello", questions()));
assertEquals("no weights here", thrown.getMessage());
- List<String> kinds = trace.events.stream().map(e -> e.split(":")[0]).toList();
- assertEquals(List.of("route", "error", "end"), kinds, trace.events.toString());
+ // The load was attempted on a decided checkpoint, so error and end name it, as the
+ // reference's ctx.model does once route() has returned.
+ String model = trace.events.get(0).split(":")[1];
+ assertEquals(List.of("route:" + model, "error:" + model, "end:" + model), trace.events);
}
}
@@ -380,6 +383,143 @@ class RouterHooksTest {
}
}
+ /** A factory that keeps every session it hands out, so a test can see which were closed. */
+ private static Router.AgentFactory recording(Path root, List<TinyCheckpoint.RecordingSession> out) {
+ return checkpoint -> {
+ TinyCheckpoint.RecordingSession session = new TinyCheckpoint.RecordingSession();
+ out.add(session);
+ return TinyCheckpoint.agent(root, session);
+ };
+ }
+
+ @Test
+ @DisplayName("a hooked predict releases its lease, so unloading closes the agent")
+ void hookedPredictReleasesItsLease(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ List<TinyCheckpoint.RecordingSession> sessions = new ArrayList<>();
+ try (Router router = Router.builder().agents(recording(root, sessions)).build()) {
+ router.hooks().addHook(new Trace());
+ router.predict("hello", questions());
+ router.unloadAll();
+ assertEquals(1, sessions.size());
+ assertTrue(sessions.get(0).closed, "a held lease keeps a retired agent open forever");
+ }
+ }
+
+ @Test
+ @DisplayName("a hooked predict whose start hook throws still releases its lease")
+ void hookedPredictReleasesItsLeaseOnFailure(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ List<TinyCheckpoint.RecordingSession> sessions = new ArrayList<>();
+ try (Router router = Router.builder().agents(recording(root, sessions)).build()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ throw new IllegalStateException("start refused");
+ }
+ });
+ assertThrows(IllegalStateException.class, () -> router.predict("hello", questions()));
+ router.unloadAll();
+ assertTrue(sessions.get(0).closed, "a held lease keeps a retired agent open forever");
+ }
+ }
+
+ @Test
+ @DisplayName("unloadAll: every eviction is dispatched and a hook failure is attached, not swapped")
+ void unloadAllDoesNotMaskTheCloseFailure(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ Trace trace = new Trace();
+ try (Router router = Router.builder().agents(angryEnglish(root)).maxLoaded(2).build()) {
+ router.load("english");
+ router.load("multilingual");
+ router.hooks().addHook(trace); // first, so it sees the event that throws
+ router.hooks().addHook(throwsOnceOnEvict(new IllegalArgumentException("the sink refused it")));
+ IllegalStateException thrown =
+ assertThrows(IllegalStateException.class, router::unloadAll);
+ assertEquals("a native close can fail", thrown.getMessage());
+ assertEquals(1, thrown.getSuppressed().length);
+ assertEquals(List.of("evict:english", "evict:multilingual"), trace.events,
+ "a hook failure on the first eviction must not skip the second");
+ }
+ }
+
+ @Test
+ @DisplayName("null questions are refused before routing: nothing is built, loaded or evicted")
+ void nullQuestionsAreRefusedBeforeAnyLoad(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ StubAgents agents = new StubAgents(root);
+ Trace trace = new Trace();
+ try (Router router = Router.builder().agents(agents).build()) {
+ router.hooks().addHook(trace);
+ assertThrows(IllegalArgumentException.class, () -> router.predict("hello", null));
+ assertEquals(0, agents.builds);
+ assertEquals(List.of(), trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("a failure before start: elapsedMs covers the load, ctx.error is the failure")
+ void aFailureBeforeStartIsTimedAndCarriesTheError(@TempDir Path root) throws IOException {
+ IllegalStateException failure = new IllegalStateException("no weights here");
+ Router.AgentFactory slow = checkpoint -> {
+ try {
+ Thread.sleep(60);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ throw failure;
+ };
+ AtomicReference<Double> elapsed = new AtomicReference<>();
+ AtomicReference<Throwable> seen = new AtomicReference<>();
+ try (Router router = Router.builder().agents(slow).build()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictEnd(PredictContext ctx) {
+ elapsed.set(ctx.elapsedMs());
+ seen.set(ctx.error());
+ }
+ });
+ assertSame(failure, assertThrows(IllegalStateException.class,
+ () -> router.predict("hello", questions())));
+ assertTrue(elapsed.get() >= 50.0, "elapsedMs " + elapsed.get() + " missed the load");
+ assertSame(failure, seen.get());
+ }
+ }
+
+ @Test
+ @DisplayName("an on_error hook that rethrows ctx.error() leaves the caller the real failure")
+ void aRethrowingErrorHookDoesNotReplaceTheFailure(@TempDir Path root) throws IOException {
+ Router.AgentFactory broken = checkpoint -> {
+ throw new IllegalStateException("no weights here");
+ };
+ try (Router router = Router.builder().agents(broken).build()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onError(PredictContext ctx) {
+ throw (RuntimeException) ctx.error(); // log-and-rethrow
+ }
+ });
+ IllegalStateException thrown = assertThrows(IllegalStateException.class,
+ () -> router.predict("hello", questions()));
+ assertEquals("no weights here", thrown.getMessage());
+ }
+ }
+
+ @Test
+ @DisplayName("an Error while loading still reaches on_error and on_predict_end")
+ void anErrorBeforeStartIsReported(@TempDir Path root) throws IOException {
+ Trace trace = new Trace();
+ Router.AgentFactory native_ = checkpoint -> {
+ throw new UnsatisfiedLinkError("no onnxruntime here");
+ };
+ try (Router router = Router.builder().agents(native_).build()) {
+ router.hooks().addHook(trace);
+ assertThrows(UnsatisfiedLinkError.class, () -> router.predict("hello", questions()));
+ List<String> kinds = trace.events.stream().map(e -> e.split(":")[0]).toList();
+ assertEquals(List.of("route", "error", "end"), kinds, trace.events.toString());
+ }
+ }
+
/** A hook whose first on_evict throws {@code failure}; later ones (from close()) do not. */
private static Hook throwsOnceOnEvict(Throwable failure) {
AtomicBoolean once = new AtomicBoolean();