Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): run the router's forward pass without default hooks, and pin what review found

Adeshina Lasisi7a7916811ef9
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index d01263dc..c85eb5c2 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 1021 register here, 1003 run, 18 abort on a checkpoint this
+ # Measured on the merged tree: 1025 register here, 1007 run, 18 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 1004. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 1008. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 1021 registered, 18 abort,
- # 1003 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 1025 registered, 18 abort,
+ # 1007 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 1003 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1007 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 1004, not the build lane's 1003, and the one is the point: this lane runs one test MORE
+ # 1008, not the build lane's 1007, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 1004 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1008 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index d41a89ee..239728e7 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 1003 of the 1021 registered tests run without a checkpoint. This floor has been raised
+ # 1007 of the 1025 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 1003 --allow-aborted
+ --min-tests 1007 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/README.md b/laya-java/README.md
index d8119f05..fa35a088 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -205,7 +205,8 @@ List<Prediction> mixed = router.predictBatch(List.of(
Requests are routed first, grouped by checkpoint in order of first appearance, and within a
checkpoint those sharing a question schema and token budget share `Agent.predictBatch` calls.
-Results come back in request order and equal per-request `predict`. `Router` is a
+Results come back in request order and match per-request `predict` within 1e-4 (padding in a
+shared forward pass moves the last bits). `Router` is a
`BatchPredictor`, so `Decisions.decideBatch(router, states, schema)` works; to pin it, pass
`(s, q) -> router.predictBatch(s, q, options)` as the runner.
@@ -514,6 +515,9 @@ the pair, and on success `elapsedMs` does not include a cold load. A routing or
reaches `onError` and `onPredictEnd`, with no start event, and its `elapsedMs` covers the routing
and the failed load. The agent's own pair still
fires for the forward pass, so a router-level hook and an agent-level hook see different spans.
+Process-wide defaults (`Hooks.setDefaultHooks`) see only the router's pair: the forward pass runs
+without them, as the reference runs it under `_SKIP_DEFAULTS`, so they fire once per request on
+`predict` and `predictBatch` alike.
`Router.predictBatch` dispatches one pair **per request**, as `predict` does, so a start hook can
rewrite or `skip` a request before it joins a shared forward pass. The order is the reference's:
@@ -521,8 +525,8 @@ every `onRoute`, then per checkpoint its `onEvict`/`onLoad`, its requests' start
their ends in reverse. If a checkpoint's inference fails, each of its started requests gets
`onError` and its end before the failure propagates. A routing failure reaches that request, and a
loading failure every request of that checkpoint, as `onError` and `onPredictEnd` with no start —
-as on `predict`, where the reference's `predict_batch` reports neither. The agent's pair fires once per forward
-pass, not per request. Unlike `predict`, the batch path honours `ctx.maxLen` / `ctx.headMaxLen`
+as on `predict`, where the reference's `predict_batch` reports neither. The agent's installed hooks fire once
+per forward pass, not per request. Unlike `predict`, the batch path honours `ctx.maxLen` / `ctx.headMaxLen`
from a router-level start hook, since a request's own budget is seeded there.
Hooks are dispatched outside the router's lock, so a hook may call back into the router —
@@ -546,7 +550,6 @@ router.hooks().addHook(new Hook() {
| | |
|---|---|
-| a process-wide default hook sees **two** predict pairs per `Router.predict` (per request plus per forward pass on `predictBatch`) | the router's and the agent's. A metric that sums `usage()` on `on_predict_end` double-counts. Install on `router.hooks()` or `agent.hooks()`, not `Hooks.setDefaultHooks`, unless you want both |
| a throwing `on_evict` hook fails `Router.close()` | and the remaining evictions are not dispatched. In try-with-resources that masks the body's own failure |
| calling back into the router from `on_evict` **during `close()`** | throws `this Router is closed`: `close()` marks the router closed before unloading |
| `concurrent(false)` with an `AsyncHook` whose callback dispatches a hook on the same registry | deadlocks with no deadline, and stalls until it with one. Serial dispatch holds a lock across the callback, and the callback is on another thread, so the inner dispatch cannot re-enter that lock. A callback that dispatches nothing — `loaded()`, a cached `load` — is fine |
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
index 9302499f..d50abf2c 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Router.java
@@ -1575,8 +1575,8 @@ public final class Router implements AutoCloseable, Predictor, BatchPredictor {
PredictContext ctx = new PredictContext(Collections.singletonList(state), questions,
decision.model().wireName(), this);
answered = Hooks.around(composed, ctx, policy,
- (states, asked, maxLen, headMaxLen) ->
- List.of(held.agent().predict(states.get(0), asked, resolved)));
+ (states, asked, maxLen, headMaxLen) -> withoutDefaultHooks(() ->
+ List.of(held.agent().predict(states.get(0), asked, resolved))));
}
if (answered == null || answered.isEmpty()) {
// Only a hook can cause this, by skipping with no result. Inventing one would report
@@ -1587,6 +1587,17 @@ public final class Router implements AutoCloseable, Predictor, BatchPredictor {
return answered.get(0);
}
+ /**
+ * The agent's forward pass without the process-wide defaults, as the reference runs it under
+ * {@code _SKIP_DEFAULTS}: a default hook already sees this request's router-level pair.
+ */
+ @SuppressWarnings("try")
+ private static <T> T withoutDefaultHooks(Supplier<T> pass) {
+ try (Hooks.DefaultsScope ignored = Hooks.withoutDefaultHooks()) {
+ return pass.get();
+ }
+ }
+
/**
* The language the checkpoint is told: the caller's if they gave one, else whatever routing
* detected -- the reference's rule, and it matters, because the multilingual checkpoint takes
@@ -1632,8 +1643,8 @@ public final class Router implements AutoCloseable, Predictor, BatchPredictor {
* <p>Every request is routed first, then grouped by checkpoint in order of first appearance.
* Within a checkpoint, requests that share a question schema, a token budget and (when the
* checkpoint has per-language temperatures) a language share {@link Agent#predictBatch}
- * calls. Each request is answered exactly as {@link #predict(Object, Map, RouteOptions)}
- * would answer it.
+ * calls. Each answer matches {@link #predict(Object, Map, RouteOptions)} within padding noise
+ * (1e-4 measured on real checkpoints).
*
* <p>Router hooks run per request, as {@link #predict} runs them, and in the reference's
* order: every {@code on_route} first, then per checkpoint any {@code on_evict}/{@code on_load},
@@ -1765,9 +1776,9 @@ public final class Router implements AutoCloseable, Predictor, BatchPredictor {
states.add(ctx.states().get(0));
}
Key key = pass.getKey();
- List<Prediction> answered = agent.predictBatch(states, members.get(0).questions(),
- key.language(), batchSize, sortByLength, HookCall.none(), key.maxLen(),
- key.headMaxLen());
+ List<Prediction> answered = withoutDefaultHooks(() -> agent.predictBatch(states,
+ members.get(0).questions(), key.language(), batchSize, sortByLength,
+ HookCall.none(), key.maxLen(), key.headMaxLen()));
if (answered.size() != members.size()) {
throw new IllegalStateException("internal error: Agent.predictBatch returned "
+ answered.size() + " results for " + members.size() + " states");
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java
index e28d9367..54db30b5 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/RouterBatchTest.java
@@ -10,6 +10,7 @@ import com.convaiinnovations.laya.Router.Checkpoint;
import com.convaiinnovations.laya.Router.Request;
import com.convaiinnovations.laya.Router.RouteOptions;
import com.convaiinnovations.laya.hooks.Hook;
+import com.convaiinnovations.laya.hooks.Hooks;
import com.convaiinnovations.laya.hooks.PredictContext;
import com.convaiinnovations.laya.infer.InferenceSession;
import com.convaiinnovations.laya.sequence.Collator;
@@ -494,4 +495,100 @@ class RouterBatchTest {
assertEquals(List.of("route:english", "error:bad@null", "end:bad"), trace.events);
}
}
+
+ @Test
+ @DisplayName("a start hook's replacement questions are the ones its request is asked")
+ void hookQuestionsHonoured() throws IOException {
+ try (Router router = router()) {
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ if ("b".equals(ctx.states().get(0))) {
+ ctx.questions(topic());
+ }
+ }
+ });
+ List<Prediction> out = router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH)));
+ assertEquals(List.of("urgent"), List.copyOf(out.get(0).answers().keySet()));
+ assertEquals(List.of("topic"), List.copyOf(out.get(1).answers().keySet()));
+ assertEquals(List.of("urgent"), List.copyOf(out.get(2).answers().keySet()));
+ assertEquals(2, calls(Checkpoint.ENGLISH), "the rewritten request has its own pass");
+ }
+ }
+
+ @Test
+ @DisplayName("a throwing start hook fails every started request once, itself included")
+ void startHookFailure() throws IOException {
+ Lifecycle trace = new Lifecycle();
+ try (Router router = router()) {
+ router.hooks().addHook(trace);
+ router.hooks().addHook(new Hook() {
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ if ("b".equals(ctx.states().get(0))) {
+ throw new IllegalStateException("start refused");
+ }
+ }
+ });
+ IllegalStateException failure = assertThrows(IllegalStateException.class,
+ () -> router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH))));
+ assertEquals("start refused", failure.getMessage());
+ assertEquals(List.of("route:english", "route:english", "route:english",
+ "start:a", "start:b", "error:b@english", "end:b", "error:a@english", "end:a"),
+ trace.events);
+ assertEquals(0, calls(Checkpoint.ENGLISH), "nothing reached the model");
+ }
+ }
+
+ /** Counts start events, for the process-wide defaults. */
+ private static final class Starts implements Hook {
+
+ int count;
+
+ @Override
+ public void onPredictStart(PredictContext ctx) {
+ count++;
+ }
+ }
+
+ @Test
+ @DisplayName("a process-wide default hook starts once per request on predictBatch")
+ void defaultsOncePerRequestBatch() throws IOException {
+ Starts starts = new Starts();
+ Hooks.setDefaultHooks(List.of(starts));
+ try (Router router = router()) {
+ router.predictBatch(List.of(
+ Request.of("a", urgent()).options(ENGLISH),
+ Request.of("b", urgent()).options(ENGLISH),
+ Request.of("c", urgent()).options(ENGLISH)));
+ assertEquals(3, starts.count, "the forward pass runs without the defaults");
+ assertEquals(1, calls(Checkpoint.ENGLISH));
+ } finally {
+ Hooks.clearDefaultHooks();
+ }
+ }
+
+ @Test
+ @DisplayName("a process-wide default hook starts once on Router.predict")
+ void defaultsOncePerPredict() throws IOException {
+ Starts starts = new Starts();
+ Starts installed = new Starts();
+ Hooks.setDefaultHooks(List.of(starts));
+ try (Router router = Router.builder().build()) {
+ Agent agent = TinyCheckpoint.agent(root, new TinyCheckpoint.RecordingSession());
+ agent.hooks().addHook(installed);
+ router.attach("english", agent);
+ router.predict("a", urgent(), ENGLISH);
+ assertEquals(1, starts.count, "the forward pass runs without the defaults");
+ assertEquals(1, installed.count, "the agent's own hooks still run");
+ } finally {
+ Hooks.clearDefaultHooks();
+ }
+ }
}