Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(java): close four holes an adversarial review found in the router hooks
Adeshina Lasisic9d7800e5dae
diff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex af747767..5b65c99f 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -154,17 +154,17 @@ jobs:# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest# run. `--min-tests` is what stops an aborted suite from looking like a pass.#- # Measured on the merged tree: 963 register here, 946 run, 17 abort on a checkpoint this+ # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes- # `TestJvmVersionTest` run instead of skip, so it executes 947. Same suite, one more test,+ # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,# because the flag enables it. The step below is where the number is justified.- name: Tests that need no modelworking-directory: laya-javarun: ./gradlew --no-daemon test- # Measured on the merged tree, Corretto 17, no checkpoints: 963 registered, 17 abort,- # 946 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in+ # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,+ # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this# replaced. A margin only binds classes larger than itself, and three classes in this lane# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no@@ -179,7 +179,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 946 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@ -335,7 +335,7 @@ jobs:# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice# and reported it as two JDKs. With `--require-passed` it exits 1.#- # 947, not the build lane's 946, and the one is the point: this lane runs one test MORE+ # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --# three classes here run exactly one test, so no positive margin can bind them.@@ -356,7 +356,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 947 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.ymlindex 95f5205f..c9e6629f 100644--- a/.github/workflows/release-java.yml+++ b/.github/workflows/release-java.yml@@ -67,7 +67,7 @@ jobs:# that lane already covered.- name: Model-free testsrun: ./gradlew --no-daemon test- # 946 of the 963 registered tests run without a checkpoint. This floor has been raised+ # 963 of the 980 registered tests run without a checkpoint. This floor has been raised# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor# that does not bind is not a gate: under 625 the whole confidence-gate family of 26# tests could have vanished from this lane with the gate still green.@@ -83,7 +83,7 @@ jobs:- name: The suite was not vacuousrun: >python3 scripts/check_test_results.py laya-java/build/test-results/test- --min-tests 946 --allow-aborted+ --min-tests 963 --allow-aborted- name: Build the jarsrun: ./gradlew --no-daemon build -x testdiff --git a/laya-java/README.md b/laya-java/README.mdindex 2f83f062..6787b608 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -506,6 +506,19 @@ router.hooks().addHook(new Hook() {});```+### What a router hook cannot do++| | |+|---|---|+| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |+| a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure |+| calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |+| `concurrent(false)` with an `AsyncHook` | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so re-entrancy cannot save it. Use one or the other |+| `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |++`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlate+on the model name or `ctx.decision`, not on `runId`.+`AsyncHook.of(hook, executor)` runs a hook on a particular thread — a framework request scope, anactor, a UI loop. It waits for the callback, so an exception still reaches the hook policy and themutable `PredictContext` is never read after the call has moved on. Pass a `Duration` to fail adiff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.javaindex 8f333cef..959e0415 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java@@ -382,6 +382,11 @@ public final class Router implements AutoCloseable, Predictor {/** Serialises builds, so two cold loads do not hold two checkpoints in flight at once. */private final ReentrantLock buildLock = new ReentrantLock();private final Map<Checkpoint, Slot> slots = new LinkedHashMap<>();+ /** This router's hooks: install, remove, and set the error, timeout and concurrency policy. */+ public HookRegistry hooks() {+ return hooks;+ }+/** Least recently used first, which is the end eviction takes from. */private final List<Checkpoint> order = new ArrayList<>();private final Map<Checkpoint, InFlight> loading = new LinkedHashMap<>();@@ -874,8 +879,15 @@ public final class Router implements AutoCloseable, Predictor {requireNonNull(checkpoint, "checkpoint");Outcome outcome = new Outcome();Slot slot = acquire(checkpoint, outcome);- // Before the Lease is handed over, so on_load precedes any use of the agent.- dispatchOutcome(checkpoint, outcome);+ // Before the Lease is handed over, so on_load precedes any use of the agent -- but the+ // lease is this method's until the Lease exists, so a throwing hook must not strand it.+ // `build` guards the same shape at the `release(built); throw` below.+ try {+ dispatchOutcome(checkpoint, outcome);+ } catch (RuntimeException | Error failure) {+ release(slot);+ throw failure;+ }return new Lease(checkpoint, slot);}@@ -892,11 +904,16 @@ public final class Router implements AutoCloseable, Predictor {public Agent load(String name) {Checkpoint checkpoint = normaliseName(name);Outcome outcome = new Outcome();- Slot slot = acquire(checkpoint, outcome);- release(slot);- // After release: a hook calling back in must not find this call still holding a lease.- dispatchOutcome(checkpoint, outcome);- return slot.agent;+ try {+ Slot slot = acquire(checkpoint, outcome);+ release(slot);+ return slot.agent;+ } finally {+ // `finally`: a build that published and then threw while closing the evicted agent+ // still loaded and still evicted, so both events are owed. After `release`, so a hook+ // calling back in does not find this call holding a lease.+ dispatchOutcome(checkpoint, outcome);+ }}/**@@ -1268,7 +1285,9 @@ public final class Router implements AutoCloseable, Predictor {lock.unlock();}if (!already) {- release(acquire(checkpoint));+ Outcome preloaded = new Outcome();+ release(acquire(checkpoint, preloaded));+ dispatchOutcome(checkpoint, preloaded);}}return this;@@ -1305,9 +1324,14 @@ public final class Router implements AutoCloseable, Predictor {lock.unlock();}if (inflight == null) {- closeAll(toClose);- for (Checkpoint evicted : freed) {- dispatchLifecycle(Hooks.Event.EVICT, evicted);+ // `finally`: the eviction happened whether or not closing the agent threw, so the+ // event is owed either way. `closeAll` rethrows the first failure.+ try {+ closeAll(toClose);+ } finally {+ for (Checkpoint evicted : freed) {+ dispatchLifecycle(Hooks.Event.EVICT, evicted);+ }}return freed;}@@ -1336,9 +1360,12 @@ public final class Router implements AutoCloseable, Predictor {lock.unlock();}if (inflights.isEmpty()) {- closeAll(toClose);- for (Checkpoint evicted : freed) {- dispatchLifecycle(Hooks.Event.EVICT, evicted);+ try {+ closeAll(toClose);+ } finally {+ for (Checkpoint evicted : freed) {+ dispatchLifecycle(Hooks.Event.EVICT, evicted);+ }}return freed;}@@ -1366,11 +1393,6 @@ public final class Router implements AutoCloseable, Predictor {* ceiling in place. That is the reference's behaviour and is kept deliberately; a caller who* needs the original ceiling back should build a new router.*/- /** This router's hooks: install, remove, and set the error, timeout and concurrency policy. */- public HookRegistry hooks() {- return hooks;- }-public int maxLoaded() {lock.lock();try {diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.javaindex b27f8b63..b5a2f0b6 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java@@ -7,6 +7,8 @@ import java.util.concurrent.CompletionException;import java.util.concurrent.Executor;import java.util.concurrent.TimeUnit;import java.util.concurrent.TimeoutException;+import java.util.concurrent.atomic.AtomicReference;+import java.util.function.Consumer;/*** Runs another hook's callbacks on an {@link Executor}, and waits for each one.@@ -16,9 +18,15 @@ import java.util.concurrent.TimeoutException;* coroutine is driven to completion on a background loop ({@code laya/hooks.run_coroutine_sync}), so* the prediction still waits for it.*- * <p>Waiting is what makes it safe. {@link PredictContext} is mutable and shared by one call, so a- * callback still reading it after the call moved on would see another call's state; a thrown- * exception would also be lost rather than reaching the hook policy.+ * <p>Waiting is what keeps a thrown exception reaching the hook policy, and keeps the mutable+ * {@link PredictContext} from being read after the call moved on.+ *+ * <p>Except on a deadline. {@code CompletableFuture.cancel} does not interrupt a task already+ * running on an executor, so an overrunning callback keeps going. It is {@link+ * PredictContext#abandon abandoned} instead, exactly as {@link Hooks} does for a hook that+ * overruns {@code hooks_timeout}: anything it writes to the context afterwards is refused. It still+ * occupies the executor's thread until it returns, so a single-threaded executor has none left for+ * the next callback -- size the executor for the deadline, or leave the deadline off.*/public final class AsyncHook implements Hook {@@ -57,38 +65,44 @@ public final class AsyncHook implements Hook {@Overridepublic void onPredictStart(PredictContext ctx) {- await("on_predict_start", () -> delegate.onPredictStart(ctx));+ await("on_predict_start", ctx, delegate::onPredictStart);}@Overridepublic void onPredictEnd(PredictContext ctx) {- await("on_predict_end", () -> delegate.onPredictEnd(ctx));+ await("on_predict_end", ctx, delegate::onPredictEnd);}@Overridepublic void onRoute(PredictContext ctx) {- await("on_route", () -> delegate.onRoute(ctx));+ await("on_route", ctx, delegate::onRoute);}@Overridepublic void onLoad(PredictContext ctx) {- await("on_load", () -> delegate.onLoad(ctx));+ await("on_load", ctx, delegate::onLoad);}@Overridepublic void onEvict(PredictContext ctx) {- await("on_evict", () -> delegate.onEvict(ctx));+ await("on_evict", ctx, delegate::onEvict);}@Overridepublic void onError(PredictContext ctx) {- await("on_error", () -> delegate.onError(ctx));+ await("on_error", ctx, delegate::onError);}- private void await(String event, Runnable callback) {+ private void await(String event, PredictContext ctx, Consumer<PredictContext> callback) {+ // The executor picks the thread, so the task reports it back: abandoning the context needs+ // the thread that will be writing to it.+ AtomicReference<Thread> runner = new AtomicReference<>();CompletableFuture<Void> running;try {- running = CompletableFuture.runAsync(callback, executor);+ running = CompletableFuture.runAsync(() -> {+ runner.set(Thread.currentThread());+ callback.accept(ctx);+ }, executor);} catch (RuntimeException rejected) {// A saturated or shut-down executor. Thrown as-is so the hook policy decides, exactly// as it would for a synchronous hook that threw.@@ -111,6 +125,12 @@ public final class AsyncHook implements Hook {}throw new IllegalStateException(delegate + " failed in " + event, cause);} catch (TimeoutException expired) {+ // cancel(true) cannot interrupt a task already running on an executor, so the callback+ // is cut off from the call instead: whatever it writes from here on is refused.+ Thread overrunning = runner.get();+ if (overrunning != null) {+ ctx.abandon(overrunning);+ }running.cancel(true);throw new IllegalStateException(delegate + " did not finish " + event + " within " + timeout, expired);diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.javaindex 377ac8b1..68089030 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java@@ -24,19 +24,9 @@ package com.convaiinnovations.laya.hooks;* calling a default no-op and the reference finding no attribute to call both leave the context* untouched and nothing in the log.*- * <p>Nothing here mirrors the reference's {@code AsyncHook}, deliberately. That class exists to- * solve a problem the JVM does not have: in Python an {@code async def} hook returns a coroutine- * that only an event loop can finish, so a synchronous caller needs- * {@code run_coroutine_sync} — and a background loop thread when it is already inside one — to- * get a value back. A JVM method call is already synchronous, so a hook that wants to do- * asynchronous work composes it and blocks on the result itself:- *- * <pre>{@code- * @Override public void onPredictEnd(PredictContext ctx) {- * ship(ctx.results()).toCompletableFuture().join(); // or orTimeout(...), or ignore it- * }- * }</pre>- *+ * <p>{@link AsyncHook} runs a hook on an {@link java.util.concurrent.Executor} and waits for it,+ * for a callback that must run on a particular thread. It is not a way to make a slow hook+ * free: the prediction still waits, as the reference's does.* <p>Porting {@code AsyncHook} would mean choosing a future type for every caller and owning a* thread pool to await it on, to wrap a {@code join} the caller can write in one line and bound* however their own runtime wants. Half-porting it — accepting a {@code CompletionStage} anddiff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.javaindex b599c04c..df22a5c6 100644--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java@@ -16,6 +16,7 @@ import java.util.Map;import java.util.concurrent.CountDownLatch;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;+import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicReference;import org.junit.jupiter.api.DisplayName;import org.junit.jupiter.api.Test;@@ -113,6 +114,39 @@ class AsyncHookTest {}}+ @Test+ @DisplayName("an overrunning callback is abandoned, so its late writes are refused")+ void anOverrunningCallbackIsAbandoned() throws Exception {+ ExecutorService pool = Executors.newSingleThreadExecutor();+ try {+ CountDownLatch wrote = new CountDownLatch(1);+ Hook async = AsyncHook.of(new Hook() {+ @Override+ public void onPredictStart(PredictContext c) {+ try {+ Thread.sleep(400); // overruns the 50ms deadline below+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }+ try {+ c.states(List.of("written after the call moved on"));+ } catch (RuntimeException refused) {+ wrote.countDown(); // refused, which is the point+ }+ }+ }, pool, Duration.ofMillis(50));+ PredictContext ctx = ctx();+ assertThrows(IllegalStateException.class, () -> async.onPredictStart(ctx));+ // cancel(true) cannot interrupt a task already on an executor, so it keeps running.+ // ctx.abandon is what stops it writing to a call that has finished.+ assertTrue(wrote.await(3, TimeUnit.SECONDS), "the late write was never attempted");+ assertEquals(List.of("state"), ctx.states(),+ "an abandoned callback's write reached the context");+ } finally {+ pool.shutdownNow();+ }+ }+@Test@DisplayName("a non-positive timeout is refused rather than meaning 'never wait'")void refusesANonPositiveTimeout() {diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.javaindex f7f79757..f6ea6211 100644--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java@@ -2,6 +2,7 @@ package com.convaiinnovations.laya;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;import com.convaiinnovations.laya.Router.Checkpoint;@@ -12,6 +13,7 @@ import java.nio.file.Path;import java.util.ArrayList;import java.util.List;import java.util.Map;+import java.util.concurrent.atomic.AtomicBoolean;import org.junit.jupiter.api.DisplayName;import org.junit.jupiter.api.Test;import org.junit.jupiter.api.io.TempDir;@@ -146,29 +148,87 @@ class RouterHooksTest {trace.events.toString());assertEquals(1, trace.events.stream().filter(e -> e.startsWith("end:")).count(),trace.events.toString());- // on_load lands inside the pair, which is the order the reference dispatches in.- int start = trace.events.indexOf("start:english");- int load = trace.events.indexOf("load:english");- int end = trace.events.indexOf("end:english");- assertTrue(start >= 0 && load > start && end > load, trace.events.toString());+ // on_route first, then on_load inside the predict pair -- the reference's order.+ // Derived from the trace, not hard-coded: which checkpoint answers is the default's+ // business, and this test is about the sequence.+ List<String> kinds = trace.events.stream().map(e -> e.split(":")[0]).toList();+ assertEquals(List.of("route", "start", "load", "end"), kinds,+ trace.events.toString());}}@Test- @DisplayName("a hook that calls back into the router does not deadlock")- void aReentrantHookDoesNotDeadlock(@TempDir Path root) throws IOException {+ @DisplayName("on_load is dispatched with no router lock held, as seen from a SECOND thread")+ void dispatchHoldsNoRouterLock(@TempDir Path root) throws Exception {TinyCheckpoint.write(root, 64, 32);- List<Integer> seen = new ArrayList<>();+ // A second thread, not the dispatching one: `lock` is a ReentrantLock, so a callback on the+ // dispatching thread re-enters it successfully whether or not it is held. Only another+ // thread can tell the difference.+ AtomicBoolean reached = new AtomicBoolean();try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(2).build()) {- // Dispatching under the router lock would park this call on itself for ever.router.hooks().addHook(new Hook() {@Overridepublic void onLoad(PredictContext ctx) {- seen.add(router.loaded().size());+ Thread other = new Thread(() -> {+ router.loaded();+ reached.set(true);+ }, "router-lock-probe");+ other.start();+ try {+ other.join(5_000);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }}});router.load("english");- assertEquals(List.of(1), seen);+ assertTrue(reached.get(),+ "a second thread could not reach the router during on_load, so the dispatch "+ + "held the lock");+ }+ }++ @Test+ @DisplayName("unloadAll dispatches on_evict for every checkpoint, which is what close() uses")+ void unloadAllDispatches(@TempDir Path root) throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ Trace trace = new Trace();+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(3).build()) {+ router.load("english");+ router.load("multilingual");+ router.hooks().addHook(trace);+ assertEquals(2, router.unloadAll().size());+ assertEquals(List.of("evict:english", "evict:multilingual"), trace.events);+ }+ }++ @Test+ @DisplayName("preload dispatches on_load for what it built")+ void preloadDispatches(@TempDir Path root) throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ Trace trace = new Trace();+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(3).build()) {+ router.hooks().addHook(trace);+ router.preload(List.of("english", "multilingual"));+ assertEquals(List.of("load:english", "load:multilingual"), trace.events);+ }+ }++ @Test+ @DisplayName("a throwing on_load hook does not strand the lease that lease() took")+ void aThrowingHookDoesNotLeakTheLease(@TempDir Path root) throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ try (Router router = Router.builder().agents(new StubAgents(root)).maxLoaded(2).build()) {+ router.hooks().addHook(new Hook() {+ @Override+ public void onLoad(PredictContext ctx) {+ throw new IllegalStateException("the sink refused it");+ }+ });+ assertThrows(IllegalStateException.class, () -> router.lease("english"));+ // A stranded lease leaves the slot retired-but-never-closed: unload would report it+ // freed while the session stayed open.+ assertEquals(List.of(Checkpoint.ENGLISH), router.unload("english"));}}