Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): close four holes an adversarial review found in the router hooks

Adeshina Lasisic9d7800e5dae
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index af747767..5b65c99f 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 963 register here, 946 run, 17 abort on a checkpoint this
+ # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 947. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 963 registered, 17 abort,
- # 946 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,
+ # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 946 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 947, not the build lane's 946, and the one is the point: this lane runs one test MORE
+ # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 947 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index 95f5205f..c9e6629f 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 946 of the 963 registered tests run without a checkpoint. This floor has been raised
+ # 963 of the 980 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 946 --allow-aborted
+ --min-tests 963 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/README.md b/laya-java/README.md
index 2f83f062..6787b608 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -506,6 +506,19 @@ router.hooks().addHook(new Hook() {
});
```
+### What a router hook cannot do
+
+| | |
+|---|---|
+| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |
+| a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure |
+| calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |
+| `concurrent(false)` with an `AsyncHook` | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so re-entrancy cannot save it. Use one or the other |
+| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |
+
+`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlate
+on the model name or `ctx.decision`, not on `runId`.
+
`AsyncHook.of(hook, executor)` runs a hook on a particular thread — a framework request scope, an
actor, a UI loop. It waits for the callback, so an exception still reaches the hook policy and the
mutable `PredictContext` is never read after the call has moved on. Pass a `Duration` to fail a
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
index 8f333cef..959e0415 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
@@ -382,6 +382,11 @@ public final class Router implements AutoCloseable, Predictor {
/** Serialises builds, so two cold loads do not hold two checkpoints in flight at once. */
private final ReentrantLock buildLock = new ReentrantLock();
private final Map<Checkpoint, Slot> slots = new LinkedHashMap<>();
+ /** This router's hooks: install, remove, and set the error, timeout and concurrency policy. */
+ public HookRegistry hooks() {
+ return hooks;
+ }
+
/** Least recently used first, which is the end eviction takes from. */
private final List<Checkpoint> order = new ArrayList<>();
private final Map<Checkpoint, InFlight> loading = new LinkedHashMap<>();
@@ -874,8 +879,15 @@ public final class Router implements AutoCloseable, Predictor {
requireNonNull(checkpoint, "checkpoint");
Outcome outcome = new Outcome();
Slot slot = acquire(checkpoint, outcome);
- // Before the Lease is handed over, so on_load precedes any use of the agent.
- dispatchOutcome(checkpoint, outcome);
+ // Before the Lease is handed over, so on_load precedes any use of the agent -- but the
+ // lease is this method's until the Lease exists, so a throwing hook must not strand it.
+ // `build` guards the same shape at the `release(built); throw` below.
+ try {
+ dispatchOutcome(checkpoint, outcome);
+ } catch (RuntimeException | Error failure) {
+ release(slot);
+ throw failure;
+ }
return new Lease(checkpoint, slot);
}
@@ -892,11 +904,16 @@ public final class Router implements AutoCloseable, Predictor {
public Agent load(String name) {
Checkpoint checkpoint = normaliseName(name);
Outcome outcome = new Outcome();
- Slot slot = acquire(checkpoint, outcome);
- release(slot);
- // After release: a hook calling back in must not find this call still holding a lease.
- dispatchOutcome(checkpoint, outcome);
- return slot.agent;
+ try {
+ Slot slot = acquire(checkpoint, outcome);
+ release(slot);
+ return slot.agent;
+ } finally {
+ // `finally`: a build that published and then threw while closing the evicted agent
+ // still loaded and still evicted, so both events are owed. After `release`, so a hook
+ // calling back in does not find this call holding a lease.
+ dispatchOutcome(checkpoint, outcome);
+ }
}
/**
@@ -1268,7 +1285,9 @@ public final class Router implements AutoCloseable, Predictor {
lock.unlock();
}
if (!already) {
- release(acquire(checkpoint));
+ Outcome preloaded = new Outcome();
+ release(acquire(checkpoint, preloaded));
+ dispatchOutcome(checkpoint, preloaded);
}
}
return this;
@@ -1305,9 +1324,14 @@ public final class Router implements AutoCloseable, Predictor {
lock.unlock();
}
if (inflight == null) {
- closeAll(toClose);
- for (Checkpoint evicted : freed) {
- dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ // `finally`: the eviction happened whether or not closing the agent threw, so the
+ // event is owed either way. `closeAll` rethrows the first failure.
+ try {
+ closeAll(toClose);
+ } finally {
+ for (Checkpoint evicted : freed) {
+ dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ }
}
return freed;
}
@@ -1336,9 +1360,12 @@ public final class Router implements AutoCloseable, Predictor {
lock.unlock();
}
if (inflights.isEmpty()) {
- closeAll(toClose);
- for (Checkpoint evicted : freed) {
- dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ try {
+ closeAll(toClose);
+ } finally {
+ for (Checkpoint evicted : freed) {
+ dispatchLifecycle(Hooks.Event.EVICT, evicted);
+ }
}
return freed;
}
@@ -1366,11 +1393,6 @@ public final class Router implements AutoCloseable, Predictor {
* ceiling in place. That is the reference's behaviour and is kept deliberately; a caller who
* needs the original ceiling back should build a new router.
*/
- /** This router's hooks: install, remove, and set the error, timeout and concurrency policy. */
- public HookRegistry hooks() {
- return hooks;
- }
-
public int maxLoaded() {
lock.lock();
try {
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
index b27f8b63..b5a2f0b6 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java
@@ -7,6 +7,8 @@ import java.util.concurrent.CompletionException;
import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
/**
* Runs another hook's callbacks on an {@link Executor}, and waits for each one.
@@ -16,9 +18,15 @@ import java.util.concurrent.TimeoutException;
* coroutine is driven to completion on a background loop ({@code laya/hooks.run_coroutine_sync}), so
* the prediction still waits for it.
*
- * <p>Waiting is what makes it safe. {@link PredictContext} is mutable and shared by one call, so a
- * callback still reading it after the call moved on would see another call's state; a thrown
- * exception would also be lost rather than reaching the hook policy.
+ * <p>Waiting is what keeps a thrown exception reaching the hook policy, and keeps the mutable
+ * {@link PredictContext} from being read after the call moved on.
+ *
+ * <p>Except on a deadline. {@code CompletableFuture.cancel} does not interrupt a task already
+ * running on an executor, so an overrunning callback keeps going. It is {@link
+ * PredictContext#abandon abandoned} instead, exactly as {@link Hooks} does for a hook that
+ * overruns {@code hooks_timeout}: anything it writes to the context afterwards is refused. It still
+ * occupies the executor's thread until it returns, so a single-threaded executor has none left for
+ * the next callback -- size the executor for the deadline, or leave the deadline off.
*/
public final class AsyncHook implements Hook {
@@ -57,38 +65,44 @@ public final class AsyncHook implements Hook {
@Override
public void onPredictStart(PredictContext ctx) {
- await("on_predict_start", () -> delegate.onPredictStart(ctx));
+ await("on_predict_start", ctx, delegate::onPredictStart);
}
@Override
public void onPredictEnd(PredictContext ctx) {
- await("on_predict_end", () -> delegate.onPredictEnd(ctx));
+ await("on_predict_end", ctx, delegate::onPredictEnd);
}
@Override
public void onRoute(PredictContext ctx) {
- await("on_route", () -> delegate.onRoute(ctx));
+ await("on_route", ctx, delegate::onRoute);
}
@Override
public void onLoad(PredictContext ctx) {
- await("on_load", () -> delegate.onLoad(ctx));
+ await("on_load", ctx, delegate::onLoad);
}
@Override
public void onEvict(PredictContext ctx) {
- await("on_evict", () -> delegate.onEvict(ctx));
+ await("on_evict", ctx, delegate::onEvict);
}
@Override
public void onError(PredictContext ctx) {
- await("on_error", () -> delegate.onError(ctx));
+ await("on_error", ctx, delegate::onError);
}
- private void await(String event, Runnable callback) {
+ private void await(String event, PredictContext ctx, Consumer<PredictContext> callback) {
+ // The executor picks the thread, so the task reports it back: abandoning the context needs
+ // the thread that will be writing to it.
+ AtomicReference<Thread> runner = new AtomicReference<>();
CompletableFuture<Void> running;
try {
- running = CompletableFuture.runAsync(callback, executor);
+ running = CompletableFuture.runAsync(() -> {
+ runner.set(Thread.currentThread());
+ callback.accept(ctx);
+ }, executor);
} catch (RuntimeException rejected) {
// A saturated or shut-down executor. Thrown as-is so the hook policy decides, exactly
// as it would for a synchronous hook that threw.
@@ -111,6 +125,12 @@ public final class AsyncHook implements Hook {
}
throw new IllegalStateException(delegate + " failed in " + event, cause);
} catch (TimeoutException expired) {
+ // cancel(true) cannot interrupt a task already running on an executor, so the callback
+ // is cut off from the call instead: whatever it writes from here on is refused.
+ Thread overrunning = runner.get();
+ if (overrunning != null) {
+ ctx.abandon(overrunning);
+ }
running.cancel(true);
throw new IllegalStateException(
delegate + " did not finish " + event + " within " + timeout, expired);
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
index 377ac8b1..68089030 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java
@@ -24,19 +24,9 @@ package com.convaiinnovations.laya.hooks;
* calling a default no-op and the reference finding no attribute to call both leave the context
* untouched and nothing in the log.
*
- * <p>Nothing here mirrors the reference's {@code AsyncHook}, deliberately. That class exists to
- * solve a problem the JVM does not have: in Python an {@code async def} hook returns a coroutine
- * that only an event loop can finish, so a synchronous caller needs
- * {@code run_coroutine_sync} — and a background loop thread when it is already inside one — to
- * get a value back. A JVM method call is already synchronous, so a hook that wants to do
- * asynchronous work composes it and blocks on the result itself:
- *
- * <pre>{@code
- * @Override public void onPredictEnd(PredictContext ctx) {
- * ship(ctx.results()).toCompletableFuture().join(); // or orTimeout(...), or ignore it
- * }
- * }</pre>
- *
+ * <p>{@link AsyncHook} runs a hook on an {@link java.util.concurrent.Executor} and waits for it,
+ * for a callback that must run on a particular thread. It is not a way to make a slow hook
+ * free: the prediction still waits, as the reference's does.
* <p>Porting {@code AsyncHook} would mean choosing a future type for every caller and owning a
* thread pool to await it on, to wrap a {@code join} the caller can write in one line and bound
* however their own runtime wants. Half-porting it — accepting a {@code CompletionStage} and
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
index b599c04c..df22a5c6 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java
@@ -16,6 +16,7 @@ import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
@@ -113,6 +114,39 @@ class AsyncHookTest {
}
}
+ @Test
+ @DisplayName("an overrunning callback is abandoned, so its late writes are refused")
+ void anOverrunningCallbackIsAbandoned() throws Exception {
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+ try {
+ CountDownLatch wrote = new CountDownLatch(1);
+ Hook async = AsyncHook.of(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext c) {
+ try {
+ Thread.sleep(400); // overruns the 50ms deadline below
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ try {
+ c.states(List.of("written after the call moved on"));
+ } catch (RuntimeException refused) {
+ wrote.countDown(); // refused, which is the point
+ }
+ }
+ }, pool, Duration.ofMillis(50));
+ PredictContext ctx = ctx();
+ assertThrows(IllegalStateException.class, () -> async.onPredictStart(ctx));
+ // cancel(true) cannot interrupt a task already on an executor, so it keeps running.
+ // ctx.abandon is what stops it writing to a call that has finished.
+ assertTrue(wrote.await(3, TimeUnit.SECONDS), "the late write was never attempted");
+ assertEquals(List.of("state"), ctx.states(),
+ "an abandoned callback's write reached the context");
+ } finally {
+ pool.shutdownNow();
+ }
+ }
+
@Test
@DisplayName("a non-positive timeout is refused rather than meaning 'never wait'")
void refusesANonPositiveTimeout() {
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
index f7f79757..f6ea6211 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java
@@ -2,6 +2,7 @@ package com.convaiinnovations.laya;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.convaiinnovations.laya.Router.Checkpoint;
@@ -12,6 +13,7 @@ import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
@@ -146,29 +148,87 @@ class RouterHooksTest {
trace.events.toString());
assertEquals(1, trace.events.stream().filter(e -> e.startsWith("end:")).count(),
trace.events.toString());
- // on_load lands inside the pair, which is the order the reference dispatches in.
- int start = trace.events.indexOf("start:english");
- int load = trace.events.indexOf("load:english");
- int end = trace.events.indexOf("end:english");
- assertTrue(start >= 0 && load > start && end > load, trace.events.toString());
+ // on_route first, then on_load inside the predict pair -- the reference's order.
+ // Derived from the trace, not hard-coded: which checkpoint answers is the default's
+ // business, and this test is about the sequence.
+ List<String> kinds = trace.events.stream().map(e -> e.split(":")[0]).toList();
+ assertEquals(List.of("route", "start", "load", "end"), kinds,
+ trace.events.toString());
}
}
@Test
- @DisplayName("a hook that calls back into the router does not deadlock")
- void aReentrantHookDoesNotDeadlock(@TempDir Path root) throws IOException {
+ @DisplayName("on_load is dispatched with no router lock held, as seen from a SECOND thread")
+ void dispatchHoldsNoRouterLock(@TempDir Path root) throws Exception {
TinyCheckpoint.write(root, 64, 32);
- List<Integer> seen = new ArrayList<>();
+ // A second thread, not the dispatching one: `lock` is a ReentrantLock, so a callback on the
+ // dispatching thread re-enters it successfully whether or not it is held. Only another
+ // thread can tell the difference.
+ AtomicBoolean reached = new AtomicBoolean();
try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(2).build()) {
- // Dispatching under the router lock would park this call on itself for ever.
router.hooks().addHook(new Hook() {
@Override
public void onLoad(PredictContext ctx) {
- seen.add(router.loaded().size());
+ Thread other = new Thread(() -> {
+ router.loaded();
+ reached.set(true);
+ }, "router-lock-probe");
+ other.start();
+ try {
+ other.join(5_000);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
}
});
router.load("english");
- assertEquals(List.of(1), seen);
+ assertTrue(reached.get(),
+ "a second thread could not reach the router during on_load, so the dispatch "
+ + "held the lock");
+ }
+ }
+
+ @Test
+ @DisplayName("unloadAll dispatches on_evict for every checkpoint, which is what close() uses")
+ void unloadAllDispatches(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ Trace trace = new Trace();
+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(3).build()) {
+ router.load("english");
+ router.load("multilingual");
+ router.hooks().addHook(trace);
+ assertEquals(2, router.unloadAll().size());
+ assertEquals(List.of("evict:english", "evict:multilingual"), trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("preload dispatches on_load for what it built")
+ void preloadDispatches(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ Trace trace = new Trace();
+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(3).build()) {
+ router.hooks().addHook(trace);
+ router.preload(List.of("english", "multilingual"));
+ assertEquals(List.of("load:english", "load:multilingual"), trace.events);
+ }
+ }
+
+ @Test
+ @DisplayName("a throwing on_load hook does not strand the lease that lease() took")
+ void aThrowingHookDoesNotLeakTheLease(@TempDir Path root) throws IOException {
+ TinyCheckpoint.write(root, 64, 32);
+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(2).build()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onLoad(PredictContext ctx) {
+ throw new IllegalStateException("the sink refused it");
+ }
+ });
+ assertThrows(IllegalStateException.class, () -> router.lease("english"));
+ // A stranded lease leaves the slot retired-but-never-closed: unload would report it
+ // freed while the session stayed open.
+ assertEquals(List.of(Checkpoint.ENGLISH), router.unload("english"));
}
}