Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(java): the typed-decisions identity check now gates the forward pass

Adeshina Lasisi6b4cac54b41f
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 7b5a117d..43fb881f 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -153,6 +153,12 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
+ #
+ # Measured: 703 register here, 686 run, 17 abort on a checkpoint this lane does not have.
+ # Exact, not padded -- it stood at 680 long enough for the real figure to pass it by, which
+ # is how a floor stops being one. ONE below the JDK-matrix lane's floor on purpose: that
+ # lane passes `-PtestJavaVersion`, which is what makes `TestJvmVersionTest` run instead of
+ # skip, so it executes 687. Same suite, one more test, because the flag enables it.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
@@ -160,7 +166,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 686 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -315,11 +321,14 @@ jobs:
# checkpoint-backed factories abort here, would forgive it. Measured: with the flag
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
+ #
+ # 687, one above the `build` lane's 686: `-PtestJavaVersion` is what makes
+ # `TestJvmVersionTest` run rather than skip, so this lane executes exactly one test more.
- name: Model-free suite was not vacuous, and ran on the right JVM
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 687 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
@@ -409,27 +418,33 @@ jobs:
# holding both would export two graphs and run two exports serially -- the 40
# minutes this matrix was split to avoid, paid twice in one cell instead of once
# in each of two that run in parallel.
- # * The export cache is keyed on `graph-model`, so each checkpoint's graph is
- # invalidated by its own revision bump. A merged cell would key two graphs under
- # one entry and re-export both whenever either moved.
+ # * The export cache is keyed per checkpoint (`graph-model` is in the key), so a
+ # restore brings back ONE graph and a miss re-exports one. A merged cell would key
+ # two graphs under one entry and re-export both on any miss, including the 1.7 GB
+ # it already had. Not "each checkpoint's graph is invalidated by its own revision
+ # bump" -- there is no such thing: `prepare_checkpoint.py` pins a single
+ # `HF_REVISION` for every checkpoint, and that file is inside this key's
+ # `hashFiles(...)`, so a bump invalidates every cell at once. What stops a restored
+ # pre-bump cache from being reused is the `.laya-revision` stamp `stamped()` reads,
+ # not the key.
# * These two goldens fail for unrelated reasons. A merged cell reports one status
# for both, and the first thing a reader of a red parity lane needs to know is
# which checkpoint moved.
#
# Cost, measured locally with the Hub blobs already cached: 846 MB of checkpoint, a
# 39 s ONNX export producing 3.6 MB of graph beside 1.7 GB of external weights, 103 s
- # to record the family and 109 s for the 13 tests. On a cold runner the download is
+ # to record the family and 140-180 s for the 14 tests. On a cold runner the download is
# the open variable and is why this carries the same 75-minute ceiling as `predict`.
- name: typed-decisions
checkpoints: typed-decisions
graph-model: typed-decisions
families: --only typed_decisions.json
tests: --tests com.convaiinnovations.laya.TypedDecisionsParityTest
- # Measured locally: 13 -- 8 single-state cases, 3 batch configs, the config identity
+ # Measured locally: 14 -- 9 single-state cases, 3 batch configs, the config identity
# check and the workflow-signature check. Exact rather than padded down, because the
# two non-factory tests are what catch a wrong checkpoint/graph pairing and a margin
# of 2 would let both be dropped with this floor still green.
- min-tests: 13
+ min-tests: 14
timeout: 75
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index 13de5510..71d06961 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -66,18 +66,39 @@ see [Installing](README.md#installing).
### Testing
-- 769 tests, 626 of which need no model at all.
+- **703 tests register with no model present and 686 of them run**; the other 17 abort by
+ assumption on a checkpoint the machine does not have. Measured with every `LAYA_*` variable
+ unset. (The figure here used to read 626, which was 60 low — CI's own `--min-tests` floor had
+ already drifted 54 above it.)
- `typed-decisions` is covered end to end by `TypedDecisionsParityTest` against
- `fixtures/typed_decisions.json`: **13 tests** with the checkpoint present — eight single-state
+ `fixtures/typed_decisions.json`: **14 tests** with the checkpoint present — nine single-state
cases spanning all four fine-tuned workflow signatures, three batch configurations, and two
- checks on the weights themselves. Four of the 13 register without a checkpoint and one of those
+ checks on the weights themselves. Four of the 14 register without a checkpoint and one of those
passes there, since the workflow-signature check reads only the fixture.
-- That second pair of checks is not ceremony. `typed-decisions` is fine-tuned from `english` and
- ships its `tokenizer.json` byte for byte and the same encoder, so opening the english
- checkpoint against the typed graph raises nothing and **seven of the eight** single-state cases
- still pass. The eighth is `over-budget`, which only truncates differently because `max_len` is
- 512 there and 1024 here; the config check is what names the problem, and it runs before any
- probability is compared.
+- That second pair of checks is not ceremony, and it was not load-bearing either until now.
+ `typed-decisions` is fine-tuned from `english` and ships its `tokenizer.json` byte for byte and
+ the same encoder, so opening the english checkpoint against the typed graph raises nothing and
+ answers. Measured with the english checkpoint symlinked in as `typed-decisions/`: **11 of the
+ 13 tests passed** under that pairing — every single-state case but `over-budget`, and all three
+ batch configurations, so the batch factory contributed nothing to the defence. The config check
+ did NOT run "before any probability is compared": JUnit 5 orders nothing by default and the
+ class shares one agent, so it ran ninth of thirteen, after all eight probability comparisons.
+ It is asserted on the shared agent as it is opened now, so a mispairing fails every case that
+ touches the weights and both factories fail to register at all — measured, the same pairing now
+ yields 4 tests, 3 failed, and the cell's exact `min-tests` floor fails on the count as well.
+ `@Order` puts the named check first in the report to match.
+- What actually separates the two checkpoints is narrower than that section used to imply. Their
+ `temperature_by_options` tables are **identical** — every key, every digit, `choice:11+` at
+ 0.10058280825614929 included — so the bucket keyset assertion, the per-bucket loop, the raw
+ value and the `TEMP_MIN` clamp all pass unchanged under the wrong checkpoint. Only `max_len`
+ (1024 against 512), `head_max_len` (256 against 192) and the fitted `temperature` triple
+ discriminate. The triple never reached a forward pass, because every recorded question landed
+ in a bucket both files ship, so `unbucketed-score` was added: a six-level score question, which
+ `temp_bucket` maps to `score:6-10`, a bucket neither checkpoint ships. The lookup misses and the
+ fitted `temperature[QTYPE_SCORE]` is what scales it — 1.0374 here against 1.2514 on english.
+- The golden records the `.laya-revision` stamp the graph was exported under, so a `HF_REVISION`
+ bump cannot re-record this family in silence: `gen_fixtures.py --check` is what notices.
+ `predict.json` still records none — regenerating it needs the multilingual graph.
- Every expectation is **generated from the Python package**, never written by hand:
`laya-java/fixtures/*.json`, regenerated by `scripts/gen_fixtures.py` and gated by `--check` in
CI so a change to `laya/` that moves a recorded contract shows up here rather than as drift.
diff --git a/laya-java/MODELS.md b/laya-java/MODELS.md
index 6905e492..a29a6fe2 100644
--- a/laya-java/MODELS.md
+++ b/laya-java/MODELS.md
@@ -57,7 +57,7 @@ an explicit `task` or `autoTaskDetection` — it is fine-tuned on four specific
wrong silent default.
Its forward pass **is** recorded, in `fixtures/typed_decisions.json`, and asserted by
-`TypedDecisionsParityTest` in the `jvm parity (typed-decisions)` cell: eight single-state cases
+`TypedDecisionsParityTest` in the `jvm parity (typed-decisions)` cell: nine single-state cases
covering all four workflow signatures, three batch configurations, and the two checks that the
weights under test are the right ones. Regenerate it with:
@@ -70,18 +70,33 @@ LAYA_TYPED_ONNX_GRAPH=laya-java/.work/onnx/typed-decisions/laya.onnx \
`LAYA_TYPED_ONNX_GRAPH`, not `LAYA_ONNX_GRAPH`, and the two must never be crossed. A graph
carries the weights and nothing in it names the checkpoint it was traced from, so a wrong pairing
answers instead of failing — `typed-decisions` is fine-tuned **from** `english` and ships its
-`tokenizer.json` byte for byte and its ModernBERT-large encoder. Measured: the english checkpoint
-opened against the typed-decisions graph raises nothing and returns
-`{refund: 0.9077, replace: 0.0923}`, the same digits as the correct pairing, and `model` reads
-`laya-rl-agent-onnx` either way. Measured with the english checkpoint symlinked in as
-`typed-decisions/`: **seven of the eight** single-state cases still pass, and the eighth,
-`over-budget`, fails only because `max_len` is 512 there and 1024 here so the state truncates at
-a different point. What names the problem is the `config` section of the golden — `max_len` 1024
-against 512, `head_max_len` 256 against 192, base temperatures `[1.0148, 1.0374, 1.0575]` against
-`[1.6369, 1.2514, 1.9834]` — which the test asserts before it compares a probability. Only the
-multilingual checkpoint fails loudly against that graph, and for an unrelated reason: a 235292
+`tokenizer.json` byte for byte and its ModernBERT-large encoder. Measured with the english
+checkpoint symlinked in as `typed-decisions/`: opening it against the typed-decisions graph raises
+nothing and reproduces the recorded probabilities inside the golden's tolerance — `off-workflow`
+answers `{billing: 0.8967, other: 0.1033}`, which is what the committed fixture records for the
+right pairing — and `model` reads `laya-rl-agent-onnx` either way. **11 of the 13 tests passed**
+under that pairing: every single-state case but `over-budget`, which truncates differently only
+because `max_len` is 512 there and 1024 here, plus all three batch configurations.
+
+What names the problem is the `config` section of the golden, and only three fields of it do:
+`max_len` 1024 against 512, `head_max_len` 256 against 192, and the fitted base temperatures
+`[1.0148, 1.0374, 1.0575]` against `[1.6369, 1.2514, 1.9834]`. The `temperature_by_options` tables
+are **identical** between the two checkpoints — every key and every digit, `choice:11+` at 0.1006
+included — so the bucket assertions and the `TEMP_MIN` clamp assertion catch a port bug and
+nothing about a wrong pairing. That config check is asserted on the shared agent as it is opened,
+not as one test among thirteen: it used to run ninth, after every probability comparison, because
+JUnit 5 orders nothing by default. Measured now, the same pairing registers 4 tests and fails 3,
+and the cell's exact `min-tests: 14` floor fails on the count as a second signal. The
+`unbucketed-score` case is there so the fitted triple has power of its own: a six-level score
+question lands in `score:6-10`, a bucket neither checkpoint ships, so the lookup misses and
+`temperature[QTYPE_SCORE]` — 1.0374 here, 1.2514 on english — is what scales those logits. Only
+the multilingual checkpoint fails loudly against that graph, and for an unrelated reason: a 235292
token id against a 50368-row embedding, which ONNX Runtime rejects at the `node_embedding` Gather.
+The golden also records the `.laya-revision` stamp the graph carries, so a `HF_REVISION` bump
+shows up as `gen_fixtures.py --check` drift instead of a silent re-record. `predict.json` records
+no revision; it would have to be re-recorded against the multilingual graph to gain one.
+
Of the four workflows, only `invoice_processing`'s trained question schema is in this repository
(`examples/32_typed_decisions_workflow.py`). The other three are known by their question-id
signatures alone, from `laya.router._TYPED_DECISION_WORKFLOWS`, so the golden carries those ids
diff --git a/laya-java/build.gradle.kts b/laya-java/build.gradle.kts
index e0e7697d..3d3932d9 100644
--- a/laya-java/build.gradle.kts
+++ b/laya-java/build.gradle.kts
@@ -93,7 +93,13 @@ subprojects {
// are supposed to abort by assumption. Treating blank as unset is what makes "this cell
// has no graph" and "this machine has no graph" the same case, which is what the tests
// are written against.
- listOf("LAYA_CHECKPOINTS", "LAYA_ONNX_GRAPH", "LAYA_PREDICT_GOLDEN").forEach { name ->
+ //
+ // EVERY variable the tests read belongs in this list. `LAYA_TYPED_ONNX_GRAPH` was left
+ // out when it was added and worked anyway, because the Test JVM inherits the ambient
+ // environment -- which is exactly the quiet inheritance this block exists to replace,
+ // and it is the one of the four that two of the three parity cells set to `''`.
+ listOf("LAYA_CHECKPOINTS", "LAYA_ONNX_GRAPH", "LAYA_TYPED_ONNX_GRAPH",
+ "LAYA_PREDICT_GOLDEN").forEach { name ->
System.getenv(name)?.takeIf { it.isNotBlank() }?.let { environment(name, it) }
}
}
diff --git a/laya-java/fixtures/typed_decisions.json b/laya-java/fixtures/typed_decisions.json
index 6e8a202d..c5369adc 100644
--- a/laya-java/fixtures/typed_decisions.json
+++ b/laya-java/fixtures/typed_decisions.json
@@ -1,6 +1,7 @@
{
"checkpoint": "typed-decisions",
"graph": "laya.onnx",
+ "revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
"config": {
"model_name": "laya-typed-decisions",
"encoder": "answerdotai/ModernBERT-large",
@@ -1106,6 +1107,61 @@
"truncated": false,
"truncated_questions": []
}
+ },
+ "unbucketed-score": {
+ "state": "Invoice INV-2026-8842 for 300,020 USD is three days overdue and the vendor has put the next casting run on hold until it clears.",
+ "lang": null,
+ "workflow": null,
+ "questions": {
+ "payment_priority": {
+ "type": "score",
+ "instructions": "How high a priority is settling this invoice this week?",
+ "criteria": [
+ "No priority; it can wait a quarter.",
+ "Low: settle within the month.",
+ "Normal: settle on the agreed terms.",
+ "Elevated: settle this week.",
+ "High: the vendor has threatened to stop shipping.",
+ "Critical: the line stops without it."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "payment_priority": {
+ "type": "score",
+ "score": 3.0829,
+ "legend": {
+ "0": "No priority; it can wait a quarter.",
+ "1": "Low: settle within the month.",
+ "2": "Normal: settle on the agreed terms.",
+ "3": "Elevated: settle this week.",
+ "4": "High: the vendor has threatened to stop shipping.",
+ "5": "Critical: the line stops without it."
+ },
+ "probabilities": {
+ "0": 0.0086,
+ "1": 0.107,
+ "2": 0.1279,
+ "3": 0.3645,
+ "4": 0.3335,
+ "5": 0.0585
+ },
+ "confidence": 0.1946,
+ "answer_confidence": 0.3645,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 128,
+ "output_tokens": 0,
+ "state_tokens": 36,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
}
},
"batch": [
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
index c9334bdd..226e07d4 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
@@ -36,11 +36,14 @@ public final class Fixtures {
*
* <p>A second variable rather than reusing the first, because a graph carries the weights and
* nothing in it says which checkpoint traced it -- so a wrong pairing answers rather than
- * failing. Measured: the english checkpoint opened against the typed-decisions graph raises
- * nothing and returns {@code {refund: 0.9077, replace: 0.0923}}, digit for digit what the
- * typed-decisions checkpoint returns against its own graph, and {@link Prediction#model()}
- * reads {@code laya-rl-agent-onnx} either way. One variable holding one directory would make
- * that pairing the default whenever both suites run in the same JVM.
+ * failing. Measured, with the english checkpoint symlinked in as {@code typed-decisions/}:
+ * opening it against the typed-decisions graph raises nothing and reproduces the recorded
+ * probabilities inside the golden's tolerance -- {@code off-workflow} answers
+ * {@code {billing: 0.8967, other: 0.1033}}, which is what
+ * {@code fixtures/typed_decisions.json} records for the right pairing -- and
+ * {@link Prediction#model()} reads {@code laya-rl-agent-onnx} either way. One variable
+ * holding one directory would make that pairing the default whenever both suites run in the
+ * same JVM.
*/
public static final String TYPED_GRAPH_ENV = "LAYA_TYPED_ONNX_GRAPH";
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java
index 2b8c9e92..4706592b 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java
@@ -20,8 +20,11 @@ import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.MethodOrderer;
+import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestFactory;
+import org.junit.jupiter.api.TestMethodOrder;
/**
* End to end on the {@code typed-decisions} checkpoint, which nothing in this port used to run.
@@ -49,8 +52,16 @@ import org.junit.jupiter.api.TestFactory;
* probabilities are the only thing that separates the two implementations.
* <li>{@code max_len} 1024 against the english checkpoint's 512. The {@code over-budget} case
* drops 1211 state tokens at that budget and marks all five questions truncated.
+ * <li>A question whose option count lands in a bucket NEITHER checkpoint ships.
+ * {@code unbucketed-score} is a six-level score question, so {@code temp_bucket} returns
+ * {@code score:6-10}, the lookup misses, and the fitted {@code temperature[QTYPE_SCORE]}
+ * is the scale applied: 1.0374 here against 1.2514 on english. Every other recorded
+ * question resolves to a bucket both checkpoints ship -- their
+ * {@code temperature_by_options} tables are identical, every key and every digit -- so
+ * without this case the fitted triple was asserted and never applied.
* </ul>
*/
+@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
final class TypedDecisionsParityTest {
/** The checkpoint this golden was recorded from; also the directory it lives in. */
@@ -59,32 +70,58 @@ final class TypedDecisionsParityTest {
/** Shared for the class: the graph is 1.7 GB of weights and opening it twice buys nothing. */
private static Agent shared;
+ /**
+ * The identity failure, if the checkpoint opened is not the one this golden was recorded from.
+ *
+ * <p>Recorded rather than thrown once, because {@link #agent} rethrows it to EVERY consumer.
+ * Caching it is what keeps that from costing a second 1.7 GB open per test.
+ */
+ private static AssertionError mispaired;
+
@AfterAll
static void closeAgent() {
if (shared != null) {
shared.close();
shared = null;
}
+ mispaired = null;
}
/**
- * The loaded config is the typed-decisions one, before any probability is compared.
+ * The loaded config is the typed-decisions one.
*
* <p>Not ceremony. This checkpoint is fine-tuned FROM the english one and ships its
* {@code tokenizer.json} byte for byte and its {@code answerdotai/ModernBERT-large} encoder,
- * so the english checkpoint opened against this graph raises nothing and answers
- * {@code {refund: 0.9077, replace: 0.0923}} -- the same digits as the right pairing, because
- * the weights are the graph's. {@link Prediction#model()} does not separate them either: the
- * ONNX path reports {@code laya-rl-agent-onnx} whichever checkpoint it opened. The budgets
- * and the temperatures do separate them, so they are what is asserted.
+ * so the english checkpoint opened against this graph raises nothing and reproduces the
+ * recorded probabilities -- {@code off-workflow}'s {@code {billing: 0.8967, other: 0.1033}}
+ * among them, digit for digit, because the weights are the graph's.
+ * {@link Prediction#model()} does not separate them either: the ONNX path reports
+ * {@code laya-rl-agent-onnx} whichever checkpoint it opened. What separates them is the
+ * budgets and the fitted temperatures, so those are what is asserted. NOT
+ * {@code config.model_name}, which the golden records and which would name the checkpoint
+ * outright: {@link AgentConfig} does not parse that key, so there is nothing to compare it
+ * against without adding a field to it.
+ *
+ * <p>This test is only the NAMED place that failure is reported. The assertion itself runs
+ * inside {@link #agent}, on the shared agent as it is opened, so it gates every case in this
+ * class instead of being one more test among them -- which is what it was: JUnit 5 applies no
+ * ordering of its own, and measured against the english checkpoint symlinked in as
+ * {@code typed-decisions} this check ran NINTH of thirteen, after all eight probability
+ * comparisons. {@link Order} pins it first as well, so the report reads in the order the
+ * gating actually happens.
*/
@Test
+ @Order(1)
@DisplayName("the agent under test is the typed-decisions checkpoint, not the english one")
- @SuppressWarnings("unchecked")
void configIsTheTypedCheckpoint() throws IOException {
Map<String, Object> golden = requireGolden();
+ assertTypedCheckpoint(golden, agent(golden).config());
+ }
+
+ /** The identity comparison itself, so {@link #agent} can apply it to every consumer. */
+ @SuppressWarnings("unchecked")
+ private static void assertTypedCheckpoint(Map<String, Object> golden, AgentConfig config) {
Map<String, Object> want = (Map<String, Object>) golden.get("config");
- AgentConfig config = agent(golden).config();
assertEquals(want.get("encoder"), config.encoder(), "encoder");
assertEquals(((Number) want.get("max_len")).intValue(), config.maxLen(), "max_len");
assertEquals(((Number) want.get("head_max_len")).intValue(), config.headMaxLen(),
@@ -109,6 +146,14 @@ final class TypedDecisionsParityTest {
// above. 0.1006 in `rl_agent_config.json`, 0.5 in use: a port that read the file value
// straight through would sharpen the many-options distribution by ~5x, and a lone 0.5 in
// the golden could not be told from a bucket that was fitted at 0.5.
+ //
+ // This catches a PORT bug, and nothing else in this method does less. It has no power
+ // over the wrong-checkpoint pairing: measured, the english and typed-decisions
+ // `temperature_by_options` tables are identical, every key and every digit, 0.1006
+ // included, so the keyset assert, the per-bucket loop, the raw-value assert and this
+ // clamp all pass unchanged under the wrong checkpoint. What discriminates above is
+ // `max_len` (1024 vs 512), `head_max_len` (256 vs 192) and the fitted `temperature`
+ // triple -- those three and nothing more.
Map<String, Object> raw = (Map<String, Object>) want.get("temperature_by_options_raw");
assertEquals(0.10058280825614929, ((Number) raw.get("choice:11+")).doubleValue(),
"the checkpoint still ships the sharpening bucket this case depends on");
@@ -124,6 +169,7 @@ final class TypedDecisionsParityTest {
* returns, which reads as coverage of a path that no longer exists.
*/
@Test
+ @Order(2)
@DisplayName("each case's question ids still match the workflow it was recorded under")
@SuppressWarnings("unchecked")
void workflowsStillMatch() {
@@ -151,6 +197,7 @@ final class TypedDecisionsParityTest {
}
@TestFactory
+ @Order(3)
@DisplayName("single-state typed-decisions predictions match Python")
@SuppressWarnings("unchecked")
List<DynamicTest> singleMatches() throws IOException {
@@ -169,14 +216,15 @@ final class TypedDecisionsParityTest {
assertUsage((Map<String, Object>) c.get("usage"), got.usage());
}));
}
- // Four workflows plus the three cases that are here for the checkpoint's own budgets and
+ // Four workflows plus the four cases that are here for the checkpoint's own budgets and
// temperature buckets. A factory whose fixture key emptied used to generate zero
// assertions and still exit 0, which is what this floor is for.
- assertTrue(tests.size() >= 8, "expected the golden's cases, got " + tests.size());
+ assertTrue(tests.size() >= 9, "expected the golden's cases, got " + tests.size());
return tests;
}
@TestFactory
+ @Order(4)
@DisplayName("batched typed-decisions predictions match Python at every batch size")
@SuppressWarnings("unchecked")
List<DynamicTest> batchMatches() throws IOException {
@@ -197,6 +245,10 @@ final class TypedDecisionsParityTest {
assertEquals(want.size(), got.size(), "one prediction per state");
for (int i = 0; i < want.size(); i++) {
Map<String, Object> w = (Map<String, Object>) want.get(i);
+ // Recorded on every batch result by the generator and, until now,
+ // compared only on the single-state path -- a field the fixture
+ // carries that nothing read.
+ assertEquals(w.get("model"), got.get(i).model(), "model");
// The caller's order, whatever order the batcher grouped them in.
assertAnswers((Map<String, Object>) w.get("answers"),
got.get(i).answers());
@@ -216,10 +268,33 @@ final class TypedDecisionsParityTest {
return golden;
}
- /** The shared agent, opened on first use so the assumptions still skip cleanly. */
+ /**
+ * The shared agent, opened on first use so the assumptions still skip cleanly.
+ *
+ * <p>Where the identity check lives, so it gates every consumer rather than racing them.
+ * JUnit 5 orders nothing by default and this class shares one agent, so whichever test ran
+ * first opened the weights and the config comparison was just another test in the list --
+ * measured ninth of thirteen, after all eight probability comparisons had already passed
+ * against the wrong checkpoint. Asserted here, a mispairing fails every case that touches
+ * the weights, with the reason named. The failure is cached because rethrowing it is cheap
+ * and reopening 1.7 GB of graph per test is not; {@code shared} is set either way so the
+ * open happens once.
+ */
private static synchronized Agent agent(Map<String, Object> golden) throws IOException {
if (shared == null) {
- shared = openAgent(golden);
+ // Deliberately NOT guarded by a separate `opened` flag: `openAgent` aborts by
+ // assumption when the checkpoint or the graph is missing, and that abort has to be
+ // rethrown to every caller, not swallowed into a null `shared`.
+ Agent agent = openAgent(golden);
+ try {
+ assertTypedCheckpoint(golden, agent.config());
+ } catch (AssertionError failure) {
+ mispaired = failure;
+ }
+ shared = agent;
+ }
+ if (mispaired != null) {
+ throw mispaired;
}
return shared;
}
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index 26f66aa2..087ac0c2 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -1822,20 +1822,23 @@ def predict_golden():
# `laya-typed-decisions` is fine-tuned from the ENGLISH checkpoint, which is what makes a
# fixture family for it worth its weight. It ships the same `tokenizer.json` byte for byte
# (diffed: identical) and the same `answerdotai/ModernBERT-large` encoder, and nothing in a
-# `.onnx` file says which checkpoint traced it. Measured: opening the ENGLISH checkpoint against
-# the typed graph raises nothing and answers {refund: 0.9077, replace: 0.0923} on a two-option
-# question -- the same digits, to the last one, as the typed checkpoint against its own graph,
-# because the weights are entirely the graph's and the two configs agree on the `choice:2`
-# temperature. `model` does not separate them either: the ONNX path reports
-# `laya-rl-agent-onnx` whichever checkpoint it opened. Only the multilingual checkpoint fails
-# loudly, and for an unrelated reason -- a 235292 token id against a 50368-row embedding, which
-# ONNX Runtime rejects at the `node_embedding` Gather.
+# `.onnx` file says which checkpoint traced it. Measured, with the english checkpoint symlinked
+# in as `typed-decisions/`: opening it against the typed graph raises nothing and reproduces the
+# probabilities recorded below inside the Java test's tolerance -- `off-workflow` answers
+# {billing: 0.8967, other: 0.1033}, which is what this file records for the right pairing, because
+# the weights are entirely the graph's and the two configs agree on the `choice:2` temperature.
+# `model` does not separate them either: the ONNX path reports `laya-rl-agent-onnx` whichever
+# checkpoint it opened. Only the multilingual checkpoint fails loudly, and for an unrelated
+# reason -- a 235292 token id against a 50368-row embedding, which ONNX Runtime rejects at the
+# `node_embedding` Gather.
#
-# So the discriminator has to come from `rl_agent_config.json`, which is where the two do differ:
-# `max_len` 1024 against english's 512, `head_max_len` 256 against 192, `model_name`
-# `laya-typed-decisions` against `rl-agent`, and base temperatures [1.0148, 1.0374, 1.0575]
-# against [1.6369, 1.2514, 1.9834]. Those are recorded below as `config`, and the Java test
-# asserts them before it compares a single probability.
+# So the discriminator has to come from `rl_agent_config.json`, and from a narrower part of it
+# than it looks: `max_len` 1024 against english's 512, `head_max_len` 256 against 192,
+# `model_name` `laya-typed-decisions` against `rl-agent`, and base temperatures
+# [1.0148, 1.0374, 1.0575] against [1.6369, 1.2514, 1.9834]. NOT `temperature_by_options`, which
+# the two ship identically, every key and every digit, `choice:11+` at 0.1006 included. Those are
+# recorded below as `config`, and the Java test asserts them on the shared agent as it opens it,
+# so a wrong pairing fails every case rather than being caught by one test among fourteen.
#
# The question ids are read from `laya.router` rather than retyped: an exact id-set match is the
# entire signature `match_typed_decisions_workflow` tests, so a signature change has to move this
@@ -2093,6 +2096,29 @@ TYPED_MANY_OPTION_QUESTIONS = {
TYPED_OVER_BUDGET_STATE = ("line item number one, unit price 300.02 against a purchase order at "
"214.30, delivery accepted with exceptions. ") * 90
+# A score question with SIX levels, which `common.temp_bucket` maps to `score:6-10` -- a bucket
+# NEITHER this checkpoint nor the english one it is fine-tuned from ships. Their
+# `temperature_by_options` tables are identical, every key and every digit, `choice:11+` at
+# 0.10058280825614929 included, so the bucket lookup and the clamp cannot tell the two apart. The
+# miss is what makes the fitted `temperature[QTYPE_SCORE]` the scale actually applied here: 1.0374
+# on this checkpoint against 1.2514 on english. Without this case every recorded question landed in
+# a bucket both files ship, so the fitted triple was asserted in `config` and never once reached a
+# forward pass, and `over-budget` was the only case whose NUMBERS could tell the two apart.
+TYPED_UNBUCKETED_SCORE_QUESTIONS = {
+ "payment_priority": {
+ "type": "score",
+ "instructions": "How high a priority is settling this invoice this week?",
+ "criteria": [
+ "No priority; it can wait a quarter.",
+ "Low: settle within the month.",
+ "Normal: settle on the agreed terms.",
+ "Elevated: settle this week.",
+ "High: the vendor has threatened to stop shipping.",
+ "Critical: the line stops without it.",
+ ],
+ },
+}
+
# A question set whose ids are NOT a workflow signature. The checkpoint still answers it -- being
# fine-tuned on four schemas does not make it refuse a fifth -- and recording that is what stops
# a port from special-casing the signatures somewhere in the forward pass, where they have no
@@ -2153,9 +2179,37 @@ TYPED_CASES = [
TYPED_INVOICE_QUESTIONS),
("off-workflow", "We were billed twice for March and want a refund today.", "en", None,
TYPED_OFF_WORKFLOW_QUESTIONS),
+ ("unbucketed-score", "Invoice INV-2026-8842 for 300,020 USD is three days overdue and the "
+ "vendor has put the next casting run on hold until it clears.", None,
+ None, TYPED_UNBUCKETED_SCORE_QUESTIONS),
]
+def _graph_revision(graph):
+ """The `.laya-revision` stamp `prepare_checkpoint.py` wrote beside `graph`, or None.
+
+ `graph` is recorded as a basename, which is the same string at every revision, so nothing in
+ this file used to say WHICH revision traced the weights it was recorded from. The stamp is the
+ only thing on disk that does. With it recorded, a `HF_REVISION` bump is caught by `--check`:
+ the regenerated `revision` differs from the committed one and the parity cell goes red naming
+ the bump, rather than a golden being re-recorded from new weights in silence.
+ """
+ # Imported rather than re-spelled: the stamp name and its layout belong to the script that
+ # writes them, and a second copy here would be a constant that can drift from the only code
+ # that produces the file it names.
+ if HERE not in sys.path:
+ sys.path.insert(0, HERE)
+ from prepare_checkpoint import STAMP
+
+ marker = (os.path.join(graph, STAMP) if os.path.isdir(graph) else graph + STAMP)
+ try:
+ with open(marker, "r", encoding="utf-8") as handle:
+ recorded = handle.read().strip().split()
+ except OSError:
+ return None
+ return recorded[0] if recorded else None
+
+
def typed_decisions_golden():
"""`ONNXAgent.predict` and `predict_batch` on the `typed-decisions` checkpoint.
@@ -2165,8 +2219,8 @@ def typed_decisions_golden():
would re-record `predict.json` from the wrong weights, and both sides of that comparison come
from the same mistake, so it passes. `LAYA_TYPED_ONNX_GRAPH` keeps the two unable to collide.
- Records `skipped` when the graph is absent, so `unverifiable` leaves the committed file alone
- rather than replacing it with a marker.
+ Records `skipped` when the graph OR the checkpoint is absent, so `unverifiable` leaves the
+ committed file alone rather than replacing it with a marker.
Platform-sensitive in the last reported digit, as `predict_golden` is; the Java test's
tolerance is the gate, not byte equality of this file.
@@ -2179,6 +2233,14 @@ def typed_decisions_golden():
if not graph or not os.path.exists(graph):
return {"skipped": "set LAYA_TYPED_ONNX_GRAPH to a laya.onnx exported from the "
"typed-decisions checkpoint to record this family"}
+ # The graph is not enough on its own: `ONNXAgent` reads the budgets, the fitted temperatures
+ # and the tokenizer out of the checkpoint directory. Checked here because without it "graph
+ # present, checkpoint absent" was the one mode of this family that raised a bare
+ # `FileNotFoundError` from inside `laya/onnx_agent.py` instead of writing the marker every
+ # sibling builder writes -- loud, but naming a path rather than the command to run.
+ if not os.path.isfile(os.path.join(rig, model, "rl_agent_config.json")):
+ return {"skipped": "no rl_agent_config.json for %s under %s: run "
+ "prepare_checkpoint.py --checkpoint %s" % (model, rig, model)}
from laya.onnx_agent import ONNXAgent
from laya.router import _TYPED_DECISION_WORKFLOWS
@@ -2215,10 +2277,12 @@ def typed_decisions_golden():
return {
"checkpoint": model,
"graph": os.path.basename(graph),
+ "revision": _graph_revision(graph),
# What tells this checkpoint from the english one it was fine-tuned from, which ships the
- # identical tokenizer and the identical encoder. Asserted by the Java test before any
- # probability is compared, because a mismatch here means the numbers below were produced
- # by the wrong weights and comparing them is worse than not running.
+ # identical tokenizer and the identical encoder. Asserted on the shared agent as it is
+ # opened, so it gates every case below rather than being one more test among them: a
+ # mismatch here means those numbers were produced by the wrong weights and comparing them
+ # is worse than not running.
"config": {
"model_name": cfg["model_name"],
"encoder": cfg["encoder"],