Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): router hooks fire in the reference's order, and three more review findings

Adeshina Lasisi1acb2e991902
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 5b65c99f..d1844b8b 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this
+ # Measured on the merged tree: 983 register here, 966 run, 17 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 967. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,
- # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 983 registered, 17 abort,
+ # 966 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 966 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE
+ # 967, not the build lane's 966, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 967 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index c9e6629f..4dd3d2fa 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 963 of the 980 registered tests run without a checkpoint. This floor has been raised
+ # 966 of the 983 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 963 --allow-aborted
+ --min-tests 966 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/README.md b/laya-java/README.md
index 34db248d..dfa9e513 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -488,15 +488,18 @@ events no agent can:
| event | when |
|---|---|
| `onRoute` | after a checkpoint is chosen, from every branch of `route` |
-| `onLoad` | after a checkpoint is built — not on a cache hit |
-| `onEvict` | after one is dropped, by `maxLoaded`, `unload` or `unloadAll` |
+| `onLoad` | after a checkpoint is built by `load`, `lease`, `predict` or `preload` — not on a cache hit |
+| `onEvict` | after one is dropped, by `maxLoaded`, `unload` or `unloadAll` — before the `onLoad` that caused it, as in the reference |
`Router.predict` also dispatches one `onPredictStart`/`onPredictEnd` pair for the whole
route-and-answer call, with `onLoad` and `onEvict` landing inside it. The agent's own pair still
fires for the forward pass, so a router-level hook and an agent-level hook see different spans.
Hooks are dispatched outside the router's lock, so a hook may call back into the router —
-including `loaded()` or another `predict` — without deadlocking.
+`loaded()`, `load`, `unload` — without deadlocking. Two consequences of that are yours to handle:
+`load()` can hand back an agent a hook unloaded while its `on_load` was being dispatched, and a
+router-level `on_predict_start` that calls `router.predict` recurses until the stack runs out,
+because that inner call dispatches the same hook again.
`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured:
`Agent.predict` takes no budget arguments. An agent-level hook can still set them.
@@ -515,11 +518,13 @@ router.hooks().addHook(new Hook() {
| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |
| a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure |
| calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |
-| `concurrent(false)` with an `AsyncHook` | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so re-entrancy cannot save it. Use one or the other |
+| `concurrent(false)` with an `AsyncHook` whose callback calls back into the router or agent | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so the inner dispatch cannot re-enter that lock. A callback that does not call back in is fine |
| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |
`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlate
-on the model name or `ctx.decision`, not on `runId`.
+on the model name, not on `runId`. That context holds the chosen checkpoint in `ctx.model()` and no
+states, questions or decision: unlike the reference, a hook here observes the route but cannot
+replace it.
`AsyncHook.of(hook, executor)` runs a hook on a particular thread — a framework request scope, an
actor, a UI loop. It waits for the callback, so an exception still reaches the hook policy and the
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
index b06bd335..c12aef5c 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
@@ -878,7 +878,15 @@ public final class Router implements AutoCloseable, Predictor {
public Lease lease(Checkpoint checkpoint) {
requireNonNull(checkpoint, "checkpoint");
Outcome outcome = new Outcome();
- Slot slot = acquire(checkpoint, outcome);
+ Slot slot;
+ try {
+ slot = acquire(checkpoint, outcome);
+ } catch (RuntimeException | Error failure) {
+ // A build that published and then failed to close the evicted agent still loaded and
+ // evicted, so both events are owed even though no lease is handed back.
+ dispatchOwed(checkpoint, outcome, failure);
+ throw failure;
+ }
// Before the Lease is handed over, so on_load precedes any use of the agent -- but the
// lease is this method's until the Lease exists, so a throwing hook must not strand it.
// `build` guards the same shape at the `release(built); throw` below.
@@ -940,13 +948,22 @@ public final class Router implements AutoCloseable, Predictor {
Hooks.dispatch(composed, event, ctx, hooks.policyFor(HookCall.none()));
}
- /** The load and evict events one acquire owes, in that order. */
+ /** The evict and load events one acquire owes, evictions first, as the reference fires them. */
private void dispatchOutcome(Checkpoint checkpoint, Outcome outcome) {
+ for (Checkpoint evicted : outcome.evicted) {
+ dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ }
if (outcome.built) {
dispatchLifecycle(Hooks.Event.LOAD, checkpoint);
}
- for (Checkpoint evicted : outcome.evicted) {
- dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ }
+
+ /** {@link #dispatchOutcome} on a failing path: a hook failure is attached, not substituted. */
+ private void dispatchOwed(Checkpoint checkpoint, Outcome outcome, Throwable failure) {
+ try {
+ dispatchOutcome(checkpoint, outcome);
+ } catch (RuntimeException | Error hookFailure) {
+ failure.addSuppressed(hookFailure);
}
}
@@ -1286,7 +1303,13 @@ public final class Router implements AutoCloseable, Predictor {
}
if (!already) {
Outcome preloaded = new Outcome();
- release(acquire(checkpoint, preloaded));
+ try {
+ release(acquire(checkpoint, preloaded));
+ } catch (RuntimeException | Error failure) {
+ // maxLoaded was raised above, so only a concurrent load can make this evict.
+ dispatchOwed(checkpoint, preloaded, failure);
+ throw failure;
+ }
dispatchOutcome(checkpoint, preloaded);
}
}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
index b5a2f0b6..0794d30d 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
@@ -23,10 +23,12 @@ import java.util.function.Consumer;
*
* <p>Except on a deadline. {@code CompletableFuture.cancel} does not interrupt a task already
* running on an executor, so an overrunning callback keeps going. It is {@link
- * PredictContext#abandon abandoned} instead, exactly as {@link Hooks} does for a hook that
- * overruns {@code hooks_timeout}: anything it writes to the context afterwards is refused. It still
- * occupies the executor's thread until it returns, so a single-threaded executor has none left for
- * the next callback -- size the executor for the deadline, or leave the deadline off.
+ * PredictContext#abandon abandoned} instead, as {@link Hooks} does for a hook that overruns
+ * {@code hooks_timeout}: anything it writes to the context afterwards is refused. Unlike that
+ * case the thread is pooled, so the abandonment ends when the overrunning task returns, and a
+ * later callback of the same call on that thread is accepted. It still occupies the executor's
+ * thread until it returns, so a single-threaded executor has none left for the next callback --
+ * size the executor for the deadline, or leave the deadline off.
*/
public final class AsyncHook implements Hook {
@@ -97,11 +99,23 @@ public final class AsyncHook implements Hook {
// The executor picks the thread, so the task reports it back: abandoning the context needs
// the thread that will be writing to it.
AtomicReference<Thread> runner = new AtomicReference<>();
+ // Guards abandon against the task finishing: abandoning a thread whose task has already
+ // returned would refuse whatever that pooled thread runs for this call next.
+ Object gate = new Object();
+ boolean[] finished = new boolean[1];
CompletableFuture<Void> running;
try {
running = CompletableFuture.runAsync(() -> {
- runner.set(Thread.currentThread());
- callback.accept(ctx);
+ Thread self = Thread.currentThread();
+ runner.set(self);
+ try {
+ callback.accept(ctx);
+ } finally {
+ synchronized (gate) {
+ finished[0] = true;
+ ctx.release(self);
+ }
+ }
}, executor);
} catch (RuntimeException rejected) {
// A saturated or shut-down executor. Thrown as-is so the hook policy decides, exactly
@@ -127,9 +141,11 @@ public final class AsyncHook implements Hook {
} catch (TimeoutException expired) {
// cancel(true) cannot interrupt a task already running on an executor, so the callback
// is cut off from the call instead: whatever it writes from here on is refused.
- Thread overrunning = runner.get();
- if (overrunning != null) {
- ctx.abandon(overrunning);
+ synchronized (gate) {
+ Thread overrunning = runner.get();
+ if (overrunning != null && !finished[0]) {
+ ctx.abandon(overrunning);
+ }
}
running.cancel(true);
throw new IllegalStateException(
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
index 68089030..dd595718 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
@@ -27,14 +27,9 @@ package com.convaiinnovations.laya.hooks;
* <p>{@link AsyncHook} runs a hook on an {@link java.util.concurrent.Executor} and waits for it,
* for a callback that must run on a particular thread. It is not a way to make a slow hook
* free: the prediction still waits, as the reference's does.
- * <p>Porting {@code AsyncHook} would mean choosing a future type for every caller and owning a
- * thread pool to await it on, to wrap a {@code join} the caller can write in one line and bound
- * however their own runtime wants. Half-porting it — accepting a {@code CompletionStage} and
- * blocking on it with no deadline — would be worse: {@link Hooks.Policy#timeout()} already
- * bounds a slow hook, and a second, invisible wait inside one would make a hook that never
- * returns look like a hook that never started.
*
- * <p>A hook runs on the calling thread, so it is as thread-safe as the call around it. Install
+ * <p>A hook runs on the calling thread, or an {@code AsyncHook}'s executor while the caller
+ * waits, so it is as thread-safe as the call around it. Install
* one that is not, and set {@link HookRegistry#concurrent(boolean)} to false to have dispatch
* serialise it.
*/
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
index a198c6fd..a7342944 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java
@@ -25,16 +25,17 @@ import java.util.concurrent.ConcurrentHashMap;
* sit in a set keyed by the call rather than by its contents, which is what a Java object does
* by default — so there is no {@code equals} or {@code hashCode} here, on purpose.
*
- * <p>Not thread-safe, and not meant to be: hooks run on the calling thread, in order. The one
- * exception is the thread a {@link Hooks.Policy#timeout} runs a hook on, which keeps running
+ * <p>Not thread-safe, and not meant to be: hooks run in order, on the calling thread or, for an
+ * {@link AsyncHook}, on its executor while the caller waits. The exception is a hook that
+ * overruns a {@link Hooks.Policy#timeout} or an {@code AsyncHook} deadline, which keeps running
* after its deadline expires while the call moves on without it — see {@link #states(List)} for
* what this class does about that and what it cannot do.
*
* <p>Two of the reference's fields are absent. {@code router} and {@code decision} carry a
- * {@code RouteDecision} to an {@code on_route} hook, and nothing in this port dispatches that
- * event yet — {@link com.convaiinnovations.laya.Router} does its own selection without hooks. A
- * field that is always null is not a port of a field, it is a promise the caller cannot tell
- * from a bug. What the reference splits across {@code agent} and {@code router} is one field
+ * {@code RouteDecision} to an {@code on_route} hook. {@link com.convaiinnovations.laya.Router}
+ * dispatches {@code on_route} with the chosen checkpoint in {@link #model()} and no states,
+ * questions or decision, so a hook here observes the route but cannot replace it, which the
+ * reference allows. What the reference splits across {@code agent} and {@code router} is one field
* here, {@link #predictor}, because both are a {@link Predictor}.
*/
public final class PredictContext {
@@ -311,6 +312,14 @@ public final class PredictContext {
abandoned.add(runner);
}
+ /**
+ * Undoes {@link #abandon} once the overrunning task has returned, so a pooled thread that
+ * runs this call's next callback is not refused for the previous one's overrun.
+ */
+ void release(Thread runner) {
+ abandoned.remove(runner);
+ }
+
private void refuseIfAbandoned(String what) {
if (!abandoned.isEmpty() && abandoned.contains(Thread.currentThread())) {
throw new IllegalStateException(String.format(
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
index df22a5c6..7edec9a8 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
@@ -147,6 +147,38 @@ class AsyncHookTest {
}
}
+ @Test
+ @DisplayName("after an overrun, the same call's next callback on that pooled thread is accepted")
+ void aPooledThreadIsReleasedWhenTheOverrunEnds() {
+ // One thread, so the later callback necessarily runs on the thread that overran. Before
+ // the release, that thread stayed abandoned for this context and the write was refused.
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+ try {
+ Hook slow = AsyncHook.of(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext c) {
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }, pool, Duration.ofMillis(50));
+ Hook later = AsyncHook.of(new Hook() {
+ @Override
+ public void onPredictEnd(PredictContext c) {
+ c.states(List.of("written by the next callback"));
+ }
+ }, pool);
+ PredictContext ctx = ctx();
+ assertThrows(IllegalStateException.class, () -> slow.onPredictStart(ctx));
+ later.onPredictEnd(ctx); // queued behind the overrun, then runs
+ assertEquals(List.of("written by the next callback"), ctx.states());
+ } finally {
+ pool.shutdownNow();
+ }
+ }
+
@Test
@DisplayName("a non-positive timeout is refused rather than meaning 'never wait'")
void refusesANonPositiveTimeout() {
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
index f6ea6211..79596efc 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
@@ -118,7 +118,8 @@ class RouterHooksTest {
router.load("english");
router.load("multilingual"); // maxLoaded 1, so english must go
assertTrue(trace.events.contains("evict:english"), trace.events.toString());
- assertEquals(List.of("load:english", "load:multilingual", "evict:english"),
+ // The reference's order: the victim's on_evict, then the build's on_load.
+ assertEquals(List.of("load:english", "evict:english", "load:multilingual"),
trace.events);
}
}
@@ -232,6 +233,55 @@ class RouterHooksTest {
}
}
+ /** English's session throws on close, so evicting it fails after multilingual has published. */
+ private static Router.AgentFactory angryEnglish(Path root) {
+ return checkpoint -> TinyCheckpoint.agent(root, checkpoint == Checkpoint.ENGLISH
+ ? new TinyCheckpoint.RecordingSession() {
+ @Override
+ public void close() {
+ super.close();
+ throw new IllegalStateException("a native close can fail");
+ }
+ }
+ : new TinyCheckpoint.RecordingSession());
+ }
+
+ @Test
+ @DisplayName("lease still dispatches on_evict and on_load when closing the victim throws")
+ void leaseDispatchesWhenTheEvictedCloseThrows(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ Trace trace = new Trace();
+ try (Router router = Router.builder().agents(angryEnglish(root)).maxLoaded(1).build()) {
+ router.load("english");
+ router.hooks().addHook(trace);
+ assertThrows(IllegalStateException.class, () -> router.lease("multilingual"));
+ assertEquals(List.of("evict:english", "load:multilingual"), trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("a hook failing on that owed dispatch is attached to the close failure, not swapped")
+ void anOwedHookFailureDoesNotMaskTheCloseFailure(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ try (Router router = Router.builder().agents(angryEnglish(root)).maxLoaded(1).build()) {
+ router.load("english");
+ AtomicBoolean once = new AtomicBoolean(); // only once: close() evicts again
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onEvict(PredictContext ctx) {
+ if (once.compareAndSet(false, true)) {
+ throw new IllegalArgumentException("the sink refused it");
+ }
+ }
+ });
+ IllegalStateException thrown =
+ assertThrows(IllegalStateException.class, () -> router.lease("multilingual"));
+ assertEquals("a native close can fail", thrown.getMessage());
+ assertEquals(1, thrown.getSuppressed().length);
+ assertEquals("the sink refused it", thrown.getSuppressed()[0].getMessage());
+ }
+ }
+
@Test
@DisplayName("a router with no hooks dispatches nothing and builds no context")
void noHooksNoDispatch(@TempDir Path root) throws IOException {