Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(java): router hooks fire in the reference's order, and three more review findings
Adeshina Lasisi1acb2e991902
diff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex 5b65c99f..d1844b8b 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -154,17 +154,17 @@ jobs:# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest# run. `--min-tests` is what stops an aborted suite from looking like a pass.#- # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this+ # Measured on the merged tree: 983 register here, 966 run, 17 abort on a checkpoint this# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes- # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,+ # `TestJvmVersionTest` run instead of skip, so it executes 967. Same suite, one more test,# because the flag enables it. The step below is where the number is justified.- name: Tests that need no modelworking-directory: laya-javarun: ./gradlew --no-daemon test- # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,- # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in+ # Measured on the merged tree, Corretto 17, no checkpoints: 983 registered, 17 abort,+ # 966 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this# replaced. A margin only binds classes larger than itself, and three classes in this lane# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no@@ -179,7 +179,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 966 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@ -335,7 +335,7 @@ jobs:# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice# and reported it as two JDKs. With `--require-passed` it exits 1.#- # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE+ # 967, not the build lane's 966, and the one is the point: this lane runs one test MORE# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --# three classes here run exactly one test, so no positive margin can bind them.@@ -356,7 +356,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 967 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.ymlindex c9e6629f..4dd3d2fa 100644--- a/.github/workflows/release-java.yml+++ b/.github/workflows/release-java.yml@@ -67,7 +67,7 @@ jobs:# that lane already covered.- name: Model-free testsrun: ./gradlew --no-daemon test- # 963 of the 980 registered tests run without a checkpoint. This floor has been raised+ # 966 of the 983 registered tests run without a checkpoint. This floor has been raised# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor# that does not bind is not a gate: under 625 the whole confidence-gate family of 26# tests could have vanished from this lane with the gate still green.@@ -83,7 +83,7 @@ jobs:- name: The suite was not vacuousrun: >python3 scripts/check_test_results.py laya-java/build/test-results/test- --min-tests 963 --allow-aborted+ --min-tests 966 --allow-aborted- name: Build the jarsrun: ./gradlew --no-daemon build -x testdiff --git a/laya-java/README.md b/laya-java/README.mdindex 34db248d..dfa9e513 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -488,15 +488,18 @@ events no agent can:| event | when ||---|---|| `onRoute` | after a checkpoint is chosen, from every branch of `route` |-| `onLoad` | after a checkpoint is built — not on a cache hit |-| `onEvict` | after one is dropped, by `maxLoaded`, `unload` or `unloadAll` |+| `onLoad` | after a checkpoint is built by `load`, `lease`, `predict` or `preload` — not on a cache hit |+| `onEvict` | after one is dropped, by `maxLoaded`, `unload` or `unloadAll` — before the `onLoad` that caused it, as in the reference |`Router.predict` also dispatches one `onPredictStart`/`onPredictEnd` pair for the wholeroute-and-answer call, with `onLoad` and `onEvict` landing inside it. The agent's own pair stillfires for the forward pass, so a router-level hook and an agent-level hook see different spans.Hooks are dispatched outside the router's lock, so a hook may call back into the router —-including `loaded()` or another `predict` — without deadlocking.+`loaded()`, `load`, `unload` — without deadlocking. Two consequences of that are yours to handle:+`load()` can hand back an agent a hook unloaded while its `on_load` was being dispatched, and a+router-level `on_predict_start` that calls `router.predict` recurses until the stack runs out,+because that inner call dispatches the same hook again.`ctx.maxLen` / `ctx.headMaxLen` set by a router-level start hook are not honoured:`Agent.predict` takes no budget arguments. An agent-level hook can still set them.@@ -515,11 +518,13 @@ router.hooks().addHook(new Hook() {| a process-wide default hook sees **two** predict pairs per `Router.predict` | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both || a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure || calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |-| `concurrent(false)` with an `AsyncHook` | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so re-entrancy cannot save it. Use one or the other |+| `concurrent(false)` with an `AsyncHook` whose callback calls back into the router or agent | deadlocks. Serial dispatch holds a lock across the callback, and the callback is on another thread, so the inner dispatch cannot re-enter that lock. A callback that does not call back in is fine || `ctx.maxLen` / `ctx.headMaxLen`, and states past the first | ignored on the router path. `Agent.predict` takes no budget arguments, and the router answers one state |`on_route` carries its own context, not the predict pair's, so `runId` differs between them. Correlate-on the model name or `ctx.decision`, not on `runId`.+on the model name, not on `runId`. That context holds the chosen checkpoint in `ctx.model()` and no+states, questions or decision: unlike the reference, a hook here observes the route but cannot+replace it.`AsyncHook.of(hook, executor)` runs a hook on a particular thread — a framework request scope, anactor, a UI loop. It waits for the callback, so an exception still reaches the hook policy and thediff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.javaindex b06bd335..c12aef5c 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java@@ -878,7 +878,15 @@ public final class Router implements AutoCloseable, Predictor {public Lease lease(Checkpoint checkpoint) {requireNonNull(checkpoint, "checkpoint");Outcome outcome = new Outcome();- Slot slot = acquire(checkpoint, outcome);+ Slot slot;+ try {+ slot = acquire(checkpoint, outcome);+ } catch (RuntimeException | Error failure) {+ // A build that published and then failed to close the evicted agent still loaded and+ // evicted, so both events are owed even though no lease is handed back.+ dispatchOwed(checkpoint, outcome, failure);+ throw failure;+ }// Before the Lease is handed over, so on_load precedes any use of the agent -- but the// lease is this method's until the Lease exists, so a throwing hook must not strand it.// `build` guards the same shape at the `release(built); throw` below.@@ -940,13 +948,22 @@ public final class Router implements AutoCloseable, Predictor {Hooks.dispatch(composed, event, ctx, hooks.policyFor(HookCall.none()));}- /** The load and evict events one acquire owes, in that order. */+ /** The evict and load events one acquire owes, evictions first, as the reference fires them. */private void dispatchOutcome(Checkpoint checkpoint, Outcome outcome) {+ for (Checkpoint evicted : outcome.evicted) {+ dispatchLifecycle(Hooks.Event.EVICT, evicted);+ }if (outcome.built) {dispatchLifecycle(Hooks.Event.LOAD, checkpoint);}- for (Checkpoint evicted : outcome.evicted) {- dispatchLifecycle(Hooks.Event.EVICT, evicted);+ }++ /** {@link #dispatchOutcome} on a failing path: a hook failure is attached, not substituted. */+ private void dispatchOwed(Checkpoint checkpoint, Outcome outcome, Throwable failure) {+ try {+ dispatchOutcome(checkpoint, outcome);+ } catch (RuntimeException | Error hookFailure) {+ failure.addSuppressed(hookFailure);}}@@ -1286,7 +1303,13 @@ public final class Router implements AutoCloseable, Predictor {}if (!already) {Outcome preloaded = new Outcome();- release(acquire(checkpoint, preloaded));+ try {+ release(acquire(checkpoint, preloaded));+ } catch (RuntimeException | Error failure) {+ // maxLoaded was raised above, so only a concurrent load can make this evict.+ dispatchOwed(checkpoint, preloaded, failure);+ throw failure;+ }dispatchOutcome(checkpoint, preloaded);}}diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.javaindex b5a2f0b6..0794d30d 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/AsyncHook.java@@ -23,10 +23,12 @@ import java.util.function.Consumer;** <p>Except on a deadline. {@code CompletableFuture.cancel} does not interrupt a task already* running on an executor, so an overrunning callback keeps going. It is {@link- * PredictContext#abandon abandoned} instead, exactly as {@link Hooks} does for a hook that- * overruns {@code hooks_timeout}: anything it writes to the context afterwards is refused. It still- * occupies the executor's thread until it returns, so a single-threaded executor has none left for- * the next callback -- size the executor for the deadline, or leave the deadline off.+ * PredictContext#abandon abandoned} instead, as {@link Hooks} does for a hook that overruns+ * {@code hooks_timeout}: anything it writes to the context afterwards is refused. Unlike that+ * case the thread is pooled, so the abandonment ends when the overrunning task returns, and a+ * later callback of the same call on that thread is accepted. It still occupies the executor's+ * thread until it returns, so a single-threaded executor has none left for the next callback --+ * size the executor for the deadline, or leave the deadline off.*/public final class AsyncHook implements Hook {@@ -97,11 +99,23 @@ public final class AsyncHook implements Hook {// The executor picks the thread, so the task reports it back: abandoning the context needs// the thread that will be writing to it.AtomicReference<Thread> runner = new AtomicReference<>();+ // Guards abandon against the task finishing: abandoning a thread whose task has already+ // returned would refuse whatever that pooled thread runs for this call next.+ Object gate = new Object();+ boolean[] finished = new boolean[1];CompletableFuture<Void> running;try {running = CompletableFuture.runAsync(() -> {- runner.set(Thread.currentThread());- callback.accept(ctx);+ Thread self = Thread.currentThread();+ runner.set(self);+ try {+ callback.accept(ctx);+ } finally {+ synchronized (gate) {+ finished[0] = true;+ ctx.release(self);+ }+ }}, executor);} catch (RuntimeException rejected) {// A saturated or shut-down executor. Thrown as-is so the hook policy decides, exactly@@ -127,9 +141,11 @@ public final class AsyncHook implements Hook {} catch (TimeoutException expired) {// cancel(true) cannot interrupt a task already running on an executor, so the callback// is cut off from the call instead: whatever it writes from here on is refused.- Thread overrunning = runner.get();- if (overrunning != null) {- ctx.abandon(overrunning);+ synchronized (gate) {+ Thread overrunning = runner.get();+ if (overrunning != null && !finished[0]) {+ ctx.abandon(overrunning);+ }}running.cancel(true);throw new IllegalStateException(diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.javaindex 68089030..dd595718 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/Hook.java@@ -27,14 +27,9 @@ package com.convaiinnovations.laya.hooks;* <p>{@link AsyncHook} runs a hook on an {@link java.util.concurrent.Executor} and waits for it,* for a callback that must run on a particular thread. It is not a way to make a slow hook* free: the prediction still waits, as the reference's does.- * <p>Porting {@code AsyncHook} would mean choosing a future type for every caller and owning a- * thread pool to await it on, to wrap a {@code join} the caller can write in one line and bound- * however their own runtime wants. Half-porting it — accepting a {@code CompletionStage} and- * blocking on it with no deadline — would be worse: {@link Hooks.Policy#timeout()} already- * bounds a slow hook, and a second, invisible wait inside one would make a hook that never- * returns look like a hook that never started.*- * <p>A hook runs on the calling thread, so it is as thread-safe as the call around it. Install+ * <p>A hook runs on the calling thread, or an {@code AsyncHook}'s executor while the caller+ * waits, so it is as thread-safe as the call around it. Install* one that is not, and set {@link HookRegistry#concurrent(boolean)} to false to have dispatch* serialise it.*/diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.javaindex a198c6fd..a7342944 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/hooks/PredictContext.java@@ -25,16 +25,17 @@ import java.util.concurrent.ConcurrentHashMap;* sit in a set keyed by the call rather than by its contents, which is what a Java object does* by default — so there is no {@code equals} or {@code hashCode} here, on purpose.*- * <p>Not thread-safe, and not meant to be: hooks run on the calling thread, in order. The one- * exception is the thread a {@link Hooks.Policy#timeout} runs a hook on, which keeps running+ * <p>Not thread-safe, and not meant to be: hooks run in order, on the calling thread or, for an+ * {@link AsyncHook}, on its executor while the caller waits. The exception is a hook that+ * overruns a {@link Hooks.Policy#timeout} or an {@code AsyncHook} deadline, which keeps running* after its deadline expires while the call moves on without it — see {@link #states(List)} for* what this class does about that and what it cannot do.** <p>Two of the reference's fields are absent. {@code router} and {@code decision} carry a- * {@code RouteDecision} to an {@code on_route} hook, and nothing in this port dispatches that- * event yet — {@link com.convaiinnovations.laya.Router} does its own selection without hooks. A- * field that is always null is not a port of a field, it is a promise the caller cannot tell- * from a bug. What the reference splits across {@code agent} and {@code router} is one field+ * {@code RouteDecision} to an {@code on_route} hook. {@link com.convaiinnovations.laya.Router}+ * dispatches {@code on_route} with the chosen checkpoint in {@link #model()} and no states,+ * questions or decision, so a hook here observes the route but cannot replace it, which the+ * reference allows. What the reference splits across {@code agent} and {@code router} is one field* here, {@link #predictor}, because both are a {@link Predictor}.*/public final class PredictContext {@@ -311,6 +312,14 @@ public final class PredictContext {abandoned.add(runner);}+ /**+ * Undoes {@link #abandon} once the overrunning task has returned, so a pooled thread that+ * runs this call's next callback is not refused for the previous one's overrun.+ */+ void release(Thread runner) {+ abandoned.remove(runner);+ }+private void refuseIfAbandoned(String what) {if (!abandoned.isEmpty() && abandoned.contains(Thread.currentThread())) {throw new IllegalStateException(String.format(diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.javaindex df22a5c6..7edec9a8 100644--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/AsyncHookTest.java@@ -147,6 +147,38 @@ class AsyncHookTest {}}+ @Test+ @DisplayName("after an overrun, the same call's next callback on that pooled thread is accepted")+ void aPooledThreadIsReleasedWhenTheOverrunEnds() {+ // One thread, so the later callback necessarily runs on the thread that overran. Before+ // the release, that thread stayed abandoned for this context and the write was refused.+ ExecutorService pool = Executors.newSingleThreadExecutor();+ try {+ Hook slow = AsyncHook.of(new Hook() {+ @Override+ public void onPredictStart(PredictContext c) {+ try {+ Thread.sleep(300);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }+ }+ }, pool, Duration.ofMillis(50));+ Hook later = AsyncHook.of(new Hook() {+ @Override+ public void onPredictEnd(PredictContext c) {+ c.states(List.of("written by the next callback"));+ }+ }, pool);+ PredictContext ctx = ctx();+ assertThrows(IllegalStateException.class, () -> slow.onPredictStart(ctx));+ later.onPredictEnd(ctx); // queued behind the overrun, then runs+ assertEquals(List.of("written by the next callback"), ctx.states());+ } finally {+ pool.shutdownNow();+ }+ }+@Test@DisplayName("a non-positive timeout is refused rather than meaning 'never wait'")void refusesANonPositiveTimeout() {diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.javaindex f6ea6211..79596efc 100644--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterHooksTest.java@@ -118,7 +118,8 @@ class RouterHooksTest {router.load("english");router.load("multilingual"); // maxLoaded 1, so english must goassertTrue(trace.events.contains("evict:english"), trace.events.toString());- assertEquals(List.of("load:english", "load:multilingual", "evict:english"),+ // The reference's order: the victim's on_evict, then the build's on_load.+ assertEquals(List.of("load:english", "evict:english", "load:multilingual"),trace.events);}}@@ -232,6 +233,55 @@ class RouterHooksTest {}}+ /** English's session throws on close, so evicting it fails after multilingual has published. */+ private static Router.AgentFactory angryEnglish(Path root) {+ return checkpoint -> TinyCheckpoint.agent(root, checkpoint == Checkpoint.ENGLISH+ ? new TinyCheckpoint.RecordingSession() {+ @Override+ public void close() {+ super.close();+ throw new IllegalStateException("a native close can fail");+ }+ }+ : new TinyCheckpoint.RecordingSession());+ }++ @Test+ @DisplayName("lease still dispatches on_evict and on_load when closing the victim throws")+ void leaseDispatchesWhenTheEvictedCloseThrows(@TempDir Path root) throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ Trace trace = new Trace();+ try (Router router = Router.builder().agents(angryEnglish(root)).maxLoaded(1).build()) {+ router.load("english");+ router.hooks().addHook(trace);+ assertThrows(IllegalStateException.class, () -> router.lease("multilingual"));+ assertEquals(List.of("evict:english", "load:multilingual"), trace.events);+ }+ }++ @Test+ @DisplayName("a hook failing on that owed dispatch is attached to the close failure, not swapped")+ void anOwedHookFailureDoesNotMaskTheCloseFailure(@TempDir Path root) throws IOException {+ TinyCheckpoint.write(root, 64, 32);+ try (Router router = Router.builder().agents(angryEnglish(root)).maxLoaded(1).build()) {+ router.load("english");+ AtomicBoolean once = new AtomicBoolean(); // only once: close() evicts again+ router.hooks().addHook(new Hook() {+ @Override+ public void onEvict(PredictContext ctx) {+ if (once.compareAndSet(false, true)) {+ throw new IllegalArgumentException("the sink refused it");+ }+ }+ });+ IllegalStateException thrown =+ assertThrows(IllegalStateException.class, () -> router.lease("multilingual"));+ assertEquals("a native close can fail", thrown.getMessage());+ assertEquals(1, thrown.getSuppressed().length);+ assertEquals("the sink refused it", thrown.getSuppressed()[0].getMessage());+ }+ }+@Test@DisplayName("a router with no hooks dispatches nothing and builds no context")void noHooksNoDispatch(@TempDir Path root) throws IOException {