Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
merge: typed-decisions, structured decide, and hooks into the routing branch
Adeshina Lasisi28cb344c927a
diff --cc .github/workflows/java.ymlindex b8a8ac0c,43fb881f,5a30cc56,73712f8b..f9d8ebd1--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@@@@ -156,11 -162,11 -156,16 -156,22 +162,23 @@@@@ jobs- name: Tests that need no modelworking-directory: laya-javarun: ./gradlew --no-daemon test- # Measured locally on JDK 17 with no checkpoints: 798 registered, 14 abort, 784 run. The- # floor keeps the 5-test margin the 685/680 pair had, which is what makes it bind the- # class this bump is for: DecisionsTest is 99 of those 784, so dropping it leaves 685 and- # the floor exits 1. Measured both ways -- 784 passes at 779, and with DecisionsTest.java- # moved out of the tree the suite is still green and the floor still exits 1.- # Measured locally on Corretto 17: 831 registered, 817 RUN -- the floor counts what ran,- # so the 14 checkpoint-backed cases this lane aborts are not in it. The margin is 4, the- # same shape as the 812/798 it replaces, and for the reason the `tokenizer` parity cell- # gives: a floor below the total minus the SMALLEST class does not bind that class.- # `AgentHooksTest` is 8 tests, so a floor more than 8 below the total would let the whole- # of it be deleted with this step still green. Verified by actually shrinking the suite:- # with `AgentHooksTest` dropped 809 run and this exits 1; with `HooksTest` and its stubs- # dropped 695 run and it exits 1.++++ # Measured on the merged tree, Corretto 17, no checkpoints: 934 registered, 17 abort,++++ # 917 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in++++ # it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this++++ # replaced. A margin only binds classes larger than itself, and three classes in this lane++++ # run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no++++ # positive margin can bind them -- any slack at all lets a whole class be deleted with this++++ # step still green. Four branches each raised this floor against their own base (779, 686,++++ # 813); every one of those would have left over a hundred tests of slack here, which is why++++ # the number is re-derived on the merge rather than reconciled from the four.+++ #+++ # This floor and the jdk-matrix one differ by one ON PURPOSE -- see that lane for why a+++ # shared number let a whole five-test class be deleted there.- name: Model-free suite was not vacuousif: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 686 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 779 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 813 --allow-aborted++++ laya-java/laya-java/build/test-results/test --min-tests 917 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@@@@ -315,11 -321,14 -320,11 -326,22 +333,22 @@@@@# checkpoint-backed factories abort here, would forgive it. Measured: with the flag# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice# and reported it as two JDKs. With `--require-passed` it exits 1.+ + #- # 687, one above the `build` lane's 686: `-PtestJavaVersion` is what makes- # `TestJvmVersionTest` run rather than skip, so this lane executes exactly one test more.+++ # 814, not the build lane's 813, and the one is the point. This lane runs one test MORE+++ # than that one -- `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only enabled+++ # here -- so 818 run against a shared floor of 813 would be a margin of 5, and the+++ # smallest class this lane could lose whole is `PreTokenizerTablesTest` at exactly 5.+++ # That is not hypothetical. Measured on the previous shared floor of 794, with the+++ # smallest `hooks.json` section emptied and no per-section guard: the build lane ran 793+++ # and exited 1 by exactly one test, and this lane ran 794 -- exactly ON the floor, so+++ # `check_test_results` exited 0 and the vacuity was invisible in the one lane that had the+++ # margin to hide it. The emptiness guard inside `HooksTest.section` is the real fix for+++ # that case; this keeps the arithmetic from being what saved it.- name: Model-free suite was not vacuous, and ran on the right JVMif: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 680 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 687 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 779 --allow-aborted- laya-java/laya-java/build/test-results/test --min-tests 814 --allow-aborted++++ laya-java/laya-java/build/test-results/test --min-tests 918 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}diff --cc laya-java/CHANGELOG.mdindex ef9dba65,71d06961,8290e73e,1616f90d..b0d6dad1--- a/laya-java/CHANGELOG.md+++ b/laya-java/CHANGELOG.md@@@@@ -66,7 -66,39 -82,12 -119,7 +135,42 @@@@@ see [Installing](README.md#installing)### Testing- -- 769 tests, 626 of which need no model at all.- - **703 tests register with no model present and 686 of them run**; the other 17 abort by- assumption on a checkpoint the machine does not have. Measured with every `LAYA_*` variable- unset. (The figure here used to read 626, which was 60 low — CI's own `--min-tests` floor had- already drifted 54 above it.)- - 1,195 tests **run** with both shipped checkpoints present (1,198 registered, 3 aborted), and- 784 **run** with no checkpoints at all (798 registered, 14 aborted). Both counts are- tests that actually executed, not tests that were registered: a `@TestFactory` that aborts by- assumption still appears in the report, so the two kinds of number are not comparable and- mixing them overstates whichever one is the smaller. Measured on JDK 17; the aborted ones need- a checkpoint, and the end-to-end golden additionally needs a traced graph.++++- 1,328 tests **run** with both shipped checkpoints present (1,334 registered,++++ 6 aborted), and 917 **run** with no checkpoints at all++++ (934 registered, 17 aborted). Both counts are tests that actually++++ executed, not tests that were registered: a `@TestFactory` that aborts by assumption still++++ appears in the report, so the two kinds of number are not comparable and mixing them overstates++++ whichever one is the smaller. Measured on JDK 17; the aborted ones need a checkpoint, and the++++ end-to-end golden additionally needs a traced graph.+ ++- `typed-decisions` is covered end to end by `TypedDecisionsParityTest` against+ ++ `fixtures/typed_decisions.json`: **14 tests** with the checkpoint present — nine single-state+ ++ cases spanning all four fine-tuned workflow signatures, three batch configurations, and two+ ++ checks on the weights themselves. Four of the 14 register without a checkpoint and one of those+ ++ passes there, since the workflow-signature check reads only the fixture.+ ++- That second pair of checks is not ceremony, and it was not load-bearing either until now.+ ++ `typed-decisions` is fine-tuned from `english` and ships its `tokenizer.json` byte for byte and+ ++ the same encoder, so opening the english checkpoint against the typed graph raises nothing and+ ++ answers. Measured with the english checkpoint symlinked in as `typed-decisions/`: **11 of the+ ++ 13 tests passed** under that pairing — every single-state case but `over-budget`, and all three+ ++ batch configurations, so the batch factory contributed nothing to the defence. The config check+ ++ did NOT run "before any probability is compared": JUnit 5 orders nothing by default and the+ ++ class shares one agent, so it ran ninth of thirteen, after all eight probability comparisons.+ ++ It is asserted on the shared agent as it is opened now, so a mispairing fails every case that+ ++ touches the weights and both factories fail to register at all — measured, the same pairing now+ ++ yields 4 tests, 3 failed, and the cell's exact `min-tests` floor fails on the count as well.+ ++ `@Order` puts the named check first in the report to match.+ ++- What actually separates the two checkpoints is narrower than that section used to imply. Their+ ++ `temperature_by_options` tables are **identical** — every key, every digit, `choice:11+` at+ ++ 0.10058280825614929 included — so the bucket keyset assertion, the per-bucket loop, the raw+ ++ value and the `TEMP_MIN` clamp all pass unchanged under the wrong checkpoint. Only `max_len`+ ++ (1024 against 512), `head_max_len` (256 against 192) and the fitted `temperature` triple+ ++ discriminate. The triple never reached a forward pass, because every recorded question landed+ ++ in a bucket both files ship, so `unbucketed-score` was added: a six-level score question, which+ ++ `temp_bucket` maps to `score:6-10`, a bucket neither checkpoint ships. The lookup misses and the+ ++ fitted `temperature[QTYPE_SCORE]` is what scales it — 1.0374 here against 1.2514 on english.+ ++- The golden records the `.laya-revision` stamp the graph was exported under, so a `HF_REVISION`+ ++ bump cannot re-record this family in silence: `gen_fixtures.py --check` is what notices.+ ++ `predict.json` still records none — regenerating it needs the multilingual graph.- Every expectation is **generated from the Python package**, never written by hand:`laya-java/fixtures/*.json`, regenerated by `scripts/gen_fixtures.py` and gated by `--check` inCI so a change to `laya/` that moves a recorded contract shows up here rather than as drift.@@@@@ -87,11 -120,12 -108,11 -152,28 +204,32 @@@@@### Known limitations--- - Not implemented: hooks, `predictLong`, structured `decide`, the `laya-java-client` HTTP module,--- Android.-- Not implemented: structured `decide`, the `laya-java-client` HTTP module, Android.++++- Not implemented: the `laya-java-client` HTTP module, Android.+++ - `AsyncHook` is not ported, and will not be. It exists in the reference to finish a coroutine+++ from synchronous code; a JVM method call is already synchronous, so a hook that wants+++ asynchronous work composes it and blocks on it in one line. Porting the wrapper would mean+++ choosing a future type for every caller and owning a thread pool to await on.+++ - Hooks run on `predict` and `predictBatch` only. `Router` does not dispatch them — so+++ `PredictContext` has no `router`/`decision` pair, rather than two fields that are always null+++ — and `predictLong` deliberately does not either. The reference's `predict_long` survives+++ hooks only by way of a start probe, a post-chain budget check and two separate "a hook+++ answered the document" paths, none of which this port has; without them a hook that adds one+++ option silently re-truncates every window, and a hook that answers the call leaves `windows`+++ claiming a scan that never ran. Running hooks there without that machinery would be worse+++ than not running them.+++ - `Hooks.withoutDefaultHooks` is a `ThreadLocal` where the reference's `_SKIP_DEFAULTS` is a+++ `contextvars.ContextVar`. A `ContextVar` is copied into an asyncio task; a `ThreadLocal` is+++ not inherited by a thread started inside the scope. Nothing here dispatches hooks off the+++ calling thread, so there is nowhere for it to bite today.+++ - A timed-out hook keeps running, on both runtimes: neither Java nor Python can interrupt a+++ thread that will not cooperate, so the deadline bounds the request and not the process.+ - `typed-decisions` is routable — `Router` knows it and resolves its aliases — but has no recorded+ end-to-end fixtures. The routing decision is tested; a forward pass against that checkpoint is+ not.+ ++- Only `invoice_processing`'s trained question schema is in this repository, so the other three+ ++ typed-decisions workflows are recorded by their question-id signature with instruction text+ ++ written for the fixture. The ids are what routing reads and the text is recorded verbatim, so+ ++ the forward pass is real; the trained wording is not claimed to be.- The softmax is computed in `double` where Python computes it in float32, as `laya-ts` and`laya-dotnet` also do. Over 200,000 random logit rows the four-decimal probabilities differ on88 of them. Rounding, by contrast, is exact: `BigDecimal` at scale 4 with `HALF_EVEN` isdiff --cc laya-java/README.mdindex 3337f767,3337f767,00e5a1c2,39cfd853..da0c7429--- a/laya-java/README.md+++ b/laya-java/README.md@@@@@ -13,12 -13,12 -13,13 -13,14 +13,15 @@@@@ four questions about a document cost onImplemented: tokenizer, sequence builder, config, ONNX inference (fused or split graph), answerdecoding, `predict`, `predictBatch`, usage and truncation reporting, script and language detection(`lang.LanguageDetection`), the question presets (`Presets`), the checkpoint `Router` with its-- load-and-evict lifecycle, the embedding `Shortlist` with its LRU cache, and the email cleaner-- and state builder (`LayaEmail`).- load-and-evict lifecycle, the embedding `Shortlist` with its LRU cache, the email cleaner- and state builder (`LayaEmail`), the abstention gate (`ConfidenceGate`), scanning a state longer- than the context window (`predictLong`), and schema-driven decisions (`Decisions.decide` /- `decideBatch`).-- Not implemented yet: hooks, the `laya-java-client` HTTP module, Android. **Not published to+++ load-and-evict lifecycle, the embedding `Shortlist` with its LRU cache, the email cleaner and-state builder (`LayaEmail`), the long-state scan (`predictLong`), the abstention gate-(`ConfidenceGate`), and the prediction hooks (`hooks`).++++state builder (`LayaEmail`), the abstention gate (`ConfidenceGate`), scanning a state longer than++++the context window (`predictLong`), schema-driven decisions (`Decisions.decide` / `decideBatch`),++++and the prediction hooks (`hooks`).+-- Not implemented yet: hooks, `predictLong`, structured `decide`, the-- `laya-java-client` HTTP module, Android. **Not published to Maven Central** — see-- [Installing](#installing).-Not implemented yet: structured `decide`, the `laya-java-client` HTTP module, Android. Hooks run-on `predict` and `predictBatch`; `Router` and `predictLong` do not dispatch them, and there is no-`AsyncHook` — see [Watching and shaping a call](#watching-and-shaping-a-call). **Not published to++++Not implemented yet: the `laya-java-client` HTTP module, Android. Hooks run on `predict` and++++`predictBatch`; `Router` and `predictLong` do not dispatch them, and there is no `AsyncHook` — see++++[Watching and shaping a call](#watching-and-shaping-a-call). **Not published to++ Maven Central** — see [Installing](#installing).## Where to go nextdiff --cc laya-java/build.gradle.ktsindex e0e7697d,3d3932d9,cf9946a5,57f05211..4bd3b790--- a/laya-java/build.gradle.kts+++ b/laya-java/build.gradle.kts@@@@@ -93,8 -93,14 -93,18 -102,8 +102,25 @@@@@ subprojects// are supposed to abort by assumption. Treating blank as unset is what makes "this cell// has no graph" and "this machine has no graph" the same case, which is what the tests// are written against.- - listOf("LAYA_CHECKPOINTS", "LAYA_ONNX_GRAPH", "LAYA_PREDICT_GOLDEN").forEach { name ->- - System.getenv(name)?.takeIf { it.isNotBlank() }?.let { environment(name, it) }+ + //+ ++ // EVERY variable the tests read belongs in this list. `LAYA_TYPED_ONNX_GRAPH` was left+ ++ // out when it was added and worked anyway, because the Test JVM inherits the ambient+ ++ // environment -- which is exactly the quiet inheritance this block exists to replace,+ ++ // and it is the one of the four that two of the three parity cells set to `''`.++++ //++ + // Declared as an INPUT as well as set. `environment(...)` alone is not a task input, so++ + // toggling one of these left `test` UP-TO-DATE and handed back the PREVIOUS lane's++ + // results -- a with-checkpoints run reporting the model-free run's counts, which is a++ + // measurement of the wrong thing that looks exactly like a measurement of the right one.++ + // CI never saw it (fresh runners), and every local count taken without `cleanTest` could++ + // silently have been the other lane's. The input makes the lane part of the task's- // identity, so the toggle re-runs the suite on its own.- listOf("LAYA_CHECKPOINTS", "LAYA_ONNX_GRAPH", "LAYA_PREDICT_GOLDEN").forEach { name ->++++ // identity, so the toggle re-runs the suite on its own. Both halves of this block were++++ // found by separate reviews: the missing variable, and the missing input declaration.+ ++ listOf("LAYA_CHECKPOINTS", "LAYA_ONNX_GRAPH", "LAYA_TYPED_ONNX_GRAPH",+ ++ "LAYA_PREDICT_GOLDEN").forEach { name ->- System.getenv(name)?.takeIf { it.isNotBlank() }?.let { environment(name, it) }++ + val value = System.getenv(name)?.takeIf { it.isNotBlank() }++ + inputs.property("env.$name", value).optional(true)++ + value?.let { environment(name, it) }}}diff --cc laya-java/scripts/gen_fixtures.pyindex 1b335cb9,087ac0c2,740c7613,6e92c5fa..f7a1856f--- a/laya-java/scripts/gen_fixtures.py+++ b/laya-java/scripts/gen_fixtures.py@@@@@ -3069,28 -3556,28 -3069,668 -3069,745 +3556,1380 @@@@@ def confidence_gate()" the three states and the no-op" % (gated, len(GATE_STATES)))return {"states": list(GATE_STATES), "cases": cases, "refusals": refusals}-# ----------------------------------------------------------------------------# laya.hooks: composition order, what a hook sees, and what it can change.-# ---------------------------------------------------------------------------++ +#: Mirrored from `laya.structured` so the case lists below can be built at import time, which++ +#: keeping this script's `laya` imports inside the builders requires. `structured()` asserts them++ +#: against the reference, so a change there fails the generator rather than silently recording a++ +#: boundary case that is no longer on the boundary.++ +STRUCTURED_MAX_PROPERTIES = 32++ +STRUCTURED_MAX_OPTIONS = 32++ +STRUCTURED_MAX_SCORE_LEVELS = 10++ +++ +++ +def _sd_answer(kind, **kw):++ + """One answer dict in the shape `predict` returns, with only the fields `_project` reads.++ +++ + `action` and `legend` are carried because the Java side's typed `Answer` records require++ + them, not because the projection looks at either.++ + """++ + answer = {"type": kind,++ + "confidence": kw.get("confidence", 0.6),++ + "answer_confidence": kw.get("answer_confidence", 0.7),++ + "action": {"act_probability": kw.get("act", 0.2)}}++ + if kind == "choice":++ + answer["choice"] = kw["choice"]++ + answer["probabilities"] = kw.get("probabilities", {})++ + elif kind == "score":++ + answer["score"] = kw["score"]++ + answer["probabilities"] = kw.get("probabilities", {})++ + answer["legend"] = kw.get("legend", {})++ + else:++ + answer["noul"] = kw["noul"]++ + if kw.get("low_confidence"):++ + answer["low_confidence"] = True++ + return answer++ +++ +++ +def _sd_enum(values, **extra):++ + prop = {"enum": list(values)}++ + prop.update(extra)++ + return prop++ +++ +++ +#: `schema -> questions`. Each case is the schema and nothing else: what is recorded is what++ +#: `questions_from_json_schema` makes of it, including the generated instruction wording and the++ +#: criteria ORDER, which is positional in laya and therefore part of the question.++ +STRUCTURED_SCHEMAS = [++ + ("enum-is-a-choice",++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support", "sales"])}}),++ + ("enum-of-numbers-keeps-its-values",++ + {"type": "object", "properties": {"code": _sd_enum([10, 20, 30])}}),++ + ("enum-with-a-null-branch",++ + {"type": "object", "properties": {"tier": _sd_enum(["gold", None])}}),++ + ("const-is-a-one-option-choice",++ + {"type": "object", "properties": {"kind": {"const": "refund"}}}),++ + # `{"const": None}` is a KEY THAT IS PRESENT WITH A NULL VALUE, not an absent key. A port++ + # testing `prop.get("const") is not None` drops this field into the "unsupported schema"++ + # branch instead of asking about it.++ + ("const-null-is-still-a-const",++ + {"type": "object", "properties": {"kind": {"const": None}}}),++ + ("boolean-is-a-noul",++ + {"type": "object", "properties": {"needs_human": {"type": "boolean"}}}),++ + # An enum whose values are ALL bools is a noul, not a two-option choice: the labels would be++ + # "True"/"False" and the head would be asked a choice question about a boolean.++ + ("enum-of-booleans-is-a-noul",++ + {"type": "object", "properties": {"flag": _sd_enum([True, False])}}),++ + ("integer-is-a-score",++ + {"type": "object", "properties": {"urgency": {"type": "integer",++ + "minimum": 1, "maximum": 5}}}),++ + ("number-with-integer-bounds-is-a-score",++ + {"type": "object", "properties": {"n": {"type": "number", "minimum": 0, "maximum": 2}}}),++ + ("score-from-zero",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 0, "maximum": 3}}}),++ + ("score-with-a-negative-minimum",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": -2, "maximum": 2}}}),++ + ("score-of-one-level",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 7, "maximum": 7}}}),++ + # An EMPTY description means "generate one", matching the reference's `description or ...`.++ + # A port returning it unchanged asks a question with no instructions at all -- and a port++ + # that returned the generated text for a NON-empty one would be caught by the case below,++ + # not by this one. Measured: a mutant dropping the emptiness test survived.++ + ("an-empty-description-generates-the-instruction",++ + {"type": "object", "properties": {++ + "dept": _sd_enum(["billing", "support"], description=""),++ + "ok": {"type": "boolean", "description": ""},++ + "n": {"type": "integer", "minimum": 1, "maximum": 3, "description": ""}}}),++ + ("description-replaces-the-generated-instruction",++ + {"type": "object", "properties": {++ + "dept": _sd_enum(["billing", "support"], description="Which desk owns this?"),++ + "ok": {"type": "boolean", "description": "Is the customer satisfied?"},++ + "n": {"type": "integer", "minimum": 1, "maximum": 3,++ + "description": "How loud is the complaint?"}}}),++ + # pydantic v2 renders `Optional[X]` as a two-branch anyOf with no top-level type.++ + ("anyof-optional-is-unwrapped",++ + {"type": "object", "properties": {++ + "dept": {"anyOf": [_sd_enum(["billing", "support"]), {"type": "null"}],++ + "description": "pick a desk"}}}),++ + ("oneof-optional-is-unwrapped",++ + {"type": "object", "properties": {++ + "n": {"oneOf": [{"type": "integer", "minimum": 1, "maximum": 4}, {"type": "null"}]}}}),++ + # `anyOf or oneOf`, so a FALSY `anyOf` falls through to `oneOf` -- and in Python falsy is++ + # not "absent": an empty list, an empty object, an empty string and a zero are all falsy and++ + # all fall through. A port spelling the test as a null check, or as "not a list", sends the++ + # empty-list case to `oneOf` by accident and the empty-OBJECT case there too while refusing++ + # a truthy non-array that should have reached `got 0`. Measured: a mutant that dropped the++ + # emptiness test survived the whole suite, because nothing asked.++ + ("falsy-anyof-falls-through-to-oneof",++ + {"type": "object", "properties": {++ + "a": {"anyOf": [], "oneOf": [{"type": "boolean"}, {"type": "null"}]},++ + "b": {"anyOf": {}, "oneOf": [_sd_enum(["x", "y"]), {"type": "null"}]},++ + "c": {"anyOf": "", "oneOf": [{"type": "integer", "minimum": 2, "maximum": 4},++ + {"type": "null"}]},++ + "d": {"anyOf": 0, "oneOf": [{"type": "boolean"}, {"type": "null"}]},++ + "e": {"anyOf": False, "oneOf": [{"type": "boolean"}, {"type": "null"}]},++ + "f": {"anyOf": None, "oneOf": [{"type": "boolean"}, {"type": "null"}]}}}),++ + # The outer description is carried onto the branch only when the branch has none.++ + ("anyof-branch-keeps-its-own-description",++ + {"type": "object", "properties": {++ + "dept": {"anyOf": [_sd_enum(["a", "b"], description="the branch's"), {"type": "null"}],++ + "description": "the outer one"}}}),++ + ("nullable-type-list-is-unwrapped",++ + {"type": "object", "properties": {"dept": {"type": ["string", "null"],++ + "enum": ["billing", "support"]}}}),++ + ("nullable-integer-type-list",++ + {"type": "object", "properties": {"n": {"type": ["integer", "null"],++ + "minimum": 0, "maximum": 3}}}),++ + # pydantic v1 wraps a described `$ref` in a one-item allOf; the outer keys win.++ + ("allof-of-one-is-unwrapped-outer-keys-on-top",++ + {"type": "object",++ + "properties": {"dept": {"allOf": [{"$ref": "#/$defs/Desk"}], "description": "the field's"}},++ + "$defs": {"Desk": _sd_enum(["billing", "support"], description="An enumeration.")}}),++ + # "Outer keys on top" is only observable when the one-item `allOf` member and the outer++ + # object SHARE a key. With a bare `$ref` member they never do, so a port that let the member++ + # win passed that case -- measured, by a mutant that swapped the two and survived.++ + ("allof-of-one-inline-outer-keys-win",++ + {"type": "object", "properties": {++ + "dept": {"allOf": [_sd_enum(["a", "b"], description="the branch's")],++ + "description": "the field's"}}}),++ + # The same rule for `$ref`: the property's own keys sit on top of the definition's, so a++ + # sibling `maximum` narrows the definition's range rather than being ignored.++ + ("ref-sibling-keys-beat-the-definition",++ + {"type": "object", "properties": {"n": {"$ref": "#/$defs/Scale", "maximum": 1}},++ + "$defs": {"Scale": {"type": "integer", "minimum": 0, "maximum": 3}}}),++ + # The definition's own description is DROPPED: pydantic fills it from the enum's docstring,++ + # which describes the type rather than asking about this field.++ + ("ref-into-defs-drops-the-definition-description",++ + {"type": "object", "properties": {"dept": {"$ref": "#/$defs/Desk"}},++ + "$defs": {"Desk": _sd_enum(["billing", "support"], description="An enumeration.")}}),++ + ("ref-into-definitions-draft-07",++ + {"type": "object", "properties": {"dept": {"$ref": "#/definitions/Desk"}},++ + "definitions": {"Desk": _sd_enum([1, 2, 3])}}),++ + ("ref-with-a-sibling-description",++ + {"type": "object",++ + "properties": {"dept": {"$ref": "#/$defs/Desk", "description": "the field's"}},++ + "$defs": {"Desk": _sd_enum(["billing", "support"])}}),++ + # A `$ref` the definition itself carries is an alias, followed by the caller's loop.++ + ("ref-alias-chain",++ + {"type": "object", "properties": {"dept": {"$ref": "#/$defs/Alias"}},++ + "$defs": {"Alias": {"$ref": "#/$defs/Desk"}, "Desk": _sd_enum(["billing", "support"])}}),++ + ("top-level-type-may-be-absent",++ + {"properties": {"ok": {"type": "boolean"}}}),++ + # Property order is the question order, and the criteria order is the option order.++ + ("property-order-is-preserved",++ + {"type": "object", "properties": {++ + "zebra": {"type": "boolean"}, "alpha": _sd_enum(["z", "a", "m"]),++ + "middle": {"type": "integer", "minimum": 0, "maximum": 1}}}),++ + ("max-options-boundary",++ + {"type": "object", "properties": {"c": _sd_enum([str(i) for i in range(STRUCTURED_MAX_OPTIONS)])}}),++ + ("max-score-levels-boundary",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 1,++ + "maximum": STRUCTURED_MAX_SCORE_LEVELS}}}),++ + ("max-properties-boundary",++ + {"type": "object",++ + "properties": {"p%02d" % i: {"type": "boolean"} for i in range(STRUCTURED_MAX_PROPERTIES)}}),++ +]++ +++ +#: `answers -> schema values`. The schema is carried with each case because the projection is a++ +#: function of BOTH: a choice maps back through its options and a score adds its `minimum`.++ +STRUCTURED_PROJECTIONS = [++ + # The schema VALUE, not the label the model answered with. A port returning the label is++ + # indistinguishable here for a string enum and wrong for every other value type.++ + ("choice-returns-the-value-not-the-label",++ + {"type": "object", "properties": {"code": _sd_enum([10, 20, 30])}},++ + {"code": _sd_answer("choice", choice="20",++ + probabilities={"10": 0.1, "20": 0.8, "30": 0.1})}),++ + ("choice-value-and-label-coincide-for-a-string-enum",++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"])}},++ + {"dept": _sd_answer("choice", choice="support",++ + probabilities={"billing": 0.3, "support": 0.7})}),++ + ("const-projects-to-the-const-value",++ + {"type": "object", "properties": {"kind": {"const": 7}}},++ + {"kind": _sd_answer("choice", choice="7", probabilities={"7": 1.0})}),++ + ("choice-null-value",++ + {"type": "object", "properties": {"tier": _sd_enum(["gold", None])}},++ + {"tier": _sd_answer("choice", choice="null",++ + probabilities={"gold": 0.2, "null": 0.8})}),++ + # A label no option carries comes back as the label itself rather than raising -- the model++ + # is constrained to the options, so this is a contract for a hand-built answer.++ + ("choice-unknown-label-falls-back-to-the-label",++ + {"type": "object", "properties": {"code": _sd_enum([10, 20])}},++ + {"code": _sd_answer("choice", choice="99", probabilities={"10": 0.5, "20": 0.5})}),++ + # argmax over the probabilities, plus `minimum`. `score` itself is ignored when they exist,++ + # and it is set here to a level that disagrees so a port reading it is caught.++ + ("score-is-argmax-plus-minimum",++ + {"type": "object", "properties": {"urgency": {"type": "integer",++ + "minimum": 1, "maximum": 5}}},++ + {"urgency": _sd_answer("score", score=0.9,++ + probabilities={"0": 0.1, "1": 0.1, "2": 0.6, "3": 0.1, "4": 0.1})}),++ + ("score-argmax-with-a-zero-minimum",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 0, "maximum": 3}}},++ + {"n": _sd_answer("score", score=0.0,++ + probabilities={"0": 0.1, "1": 0.2, "2": 0.1, "3": 0.6})}),++ + ("score-argmax-with-a-negative-minimum",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": -2, "maximum": 2}}},++ + {"n": _sd_answer("score", score=4.0,++ + probabilities={"0": 0.6, "1": 0.1, "2": 0.1, "3": 0.1, "4": 0.1})}),++ + # A tie takes the FIRST level, because `max` over the index range keeps the earliest.++ + ("score-argmax-tie-takes-the-first-level",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 1, "maximum": 3}}},++ + {"n": _sd_answer("score", score=2.0, probabilities={"0": 0.4, "1": 0.4, "2": 0.2})}),++ + # With no probabilities the fallback rounds `score`, which is already a 0-based LEVEL INDEX++ + # -- so `minimum` is added once and never subtracted first.++ + ("score-without-probabilities-rounds",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 3, "maximum": 7}}},++ + {"n": _sd_answer("score", score=2.4, probabilities={})}),++ + # Python's `round` is half to EVEN. `Math.round` is half up and gives 3 here.++ + ("score-without-probabilities-rounds-half-to-even",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": -2, "maximum": 2}}},++ + {"n": _sd_answer("score", score=2.5, probabilities={})}),++ + ("score-without-probabilities-rounds-half-to-even-odd-case",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": -2, "maximum": 2}}},++ + {"n": _sd_answer("score", score=1.5, probabilities={})}),++ + # Exactly 0.5 is TRUE: the comparison is `>=`, so a port spelling it `>` flips this one case++ + # and no other.++ + ("noul-exactly-half-is-true",++ + {"type": "object", "properties": {"ok": {"type": "boolean"}}},++ + {"ok": _sd_answer("noul", noul=0.5)}),++ + ("noul-just-below-half-is-false",++ + {"type": "object", "properties": {"ok": {"type": "boolean"}}},++ + {"ok": _sd_answer("noul", noul=0.49999999999999994)}),++ + ("noul-true",++ + {"type": "object", "properties": {"ok": {"type": "boolean"}}},++ + {"ok": _sd_answer("noul", noul=0.93)}),++ + # A flagged answer is NULL, whatever it said. Null, not absent: the field was asked.++ + ("low-confidence-choice-is-null",++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"])}},++ + {"dept": _sd_answer("choice", choice="support", low_confidence=True,++ + probabilities={"billing": 0.3, "support": 0.7})}),++ + ("low-confidence-score-is-null",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 1, "maximum": 3}}},++ + {"n": _sd_answer("score", score=1.0, low_confidence=True,++ + probabilities={"0": 0.2, "1": 0.7, "2": 0.1})}),++ + ("low-confidence-noul-is-null",++ + {"type": "object", "properties": {"ok": {"type": "boolean"}}},++ + {"ok": _sd_answer("noul", noul=0.93, low_confidence=True)}),++ + # An ABSENT answer is absent from the values. Different from a flagged one, which is null:++ + # "the model was not asked" and "the model was not believed" are not the same outcome.++ + ("an-unanswered-field-is-omitted-not-null",++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"]),++ + "ok": {"type": "boolean"}}},++ + {"ok": _sd_answer("noul", noul=0.8)}),++ + ("every-kind-at-once",++ + {"type": "object", "properties": {++ + "dept": _sd_enum(["billing", "support"]),++ + "urgency": {"type": "integer", "minimum": 1, "maximum": 5},++ + "needs_human": {"type": "boolean"}}},++ + {"dept": _sd_answer("choice", choice="billing",++ + probabilities={"billing": 0.9, "support": 0.1}),++ + "urgency": _sd_answer("score", score=3.0,++ + probabilities={"0": 0.0, "1": 0.1, "2": 0.1, "3": 0.7, "4": 0.1}),++ + "needs_human": _sd_answer("noul", noul=0.2)}),++ +]++ +++ +#: Every schema the mapping refuses, and the message a caller reads. The message is the contract++ +#: here: it names the path, which is the only thing that makes a 32-field schema debuggable.++ +STRUCTURED_REFUSALS = [++ + ("free-string", {"type": "object", "properties": {"note": {"type": "string"}}}),++ + ("array", {"type": "object",++ + "properties": {"tags": {"type": "array", "items": {"type": "string"}}}}),++ + ("nested-object", {"type": "object",++ + "properties": {"inner": {"type": "object",++ + "properties": {"a": {"type": "boolean"}}}}}),++ + ("union-of-two-real-types",++ + {"type": "object", "properties": {"x": {"anyOf": [{"type": "string"},++ + {"type": "integer"}]}}}),++ + ("union-of-null-only",++ + {"type": "object", "properties": {"x": {"anyOf": [{"type": "null"}]}}}),++ + ("oneof-of-two-real-types",++ + {"type": "object", "properties": {"x": {"oneOf": [{"type": "boolean"},++ + {"type": "integer"}]}}}),++ + ("type-list-with-two-non-null-types",++ + {"type": "object", "properties": {"x": {"type": ["string", "integer"]}}}),++ + ("top-level-is-an-array", {"type": "array", "items": {"type": "string"}}),++ + ("top-level-has-no-properties", {"type": "object"}),++ + # Both halves of the top-level test, separately. Every other case here is missing++ + # `properties` as well as having the wrong `type`, so a port that dropped the type check++ + # entirely still refused all of them -- measured, by a mutant that did exactly that.++ + ("top-level-type-is-wrong-but-properties-is-there",++ + {"type": "array", "properties": {"ok": {"type": "boolean"}}}),++ + ("top-level-is-a-string-schema", {"type": "string"}),++ + ("empty-properties", {"type": "object", "properties": {}}),++ + ("properties-is-not-an-object", {"type": "object", "properties": []}),++ + ("too-many-properties",++ + {"type": "object",++ + "properties": {"p%02d" % i: {"type": "boolean"} for i in range(STRUCTURED_MAX_PROPERTIES + 1)}}),++ + ("too-many-options",++ + {"type": "object",++ + "properties": {"c": _sd_enum([str(i) for i in range(STRUCTURED_MAX_OPTIONS + 1)])}}),++ + ("empty-enum", {"type": "object", "properties": {"c": _sd_enum([])}}),++ + ("duplicate-choice-labels", {"type": "object", "properties": {"c": _sd_enum(["1", 1])}}),++ + ("numeric-without-bounds", {"type": "object", "properties": {"n": {"type": "integer"}}}),++ + ("numeric-with-float-bounds",++ + {"type": "object", "properties": {"n": {"type": "number",++ + "minimum": 0.5, "maximum": 1.5}}}),++ + ("maximum-below-minimum",++ + {"type": "object", "properties": {"n": {"type": "integer",++ + "minimum": 5, "maximum": 1}}}),++ + ("too-many-score-levels",++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 0,++ + "maximum": STRUCTURED_MAX_SCORE_LEVELS}}}),++ + ("empty-property-schema", {"type": "object", "properties": {"c": {}}}),++ + ("property-is-not-an-object", {"type": "object", "properties": {"c": "nope"}}),++ + ("unresolvable-ref",++ + {"type": "object", "properties": {"c": {"$ref": "#/$defs/Missing"}}, "$defs": {}}),++ + ("remote-ref",++ + {"type": "object", "properties": {"c": {"$ref": "https://example.test/Desk"}}}),++ + ("recursive-ref",++ + {"type": "object", "properties": {"c": {"$ref": "#/$defs/Node"}},++ + "$defs": {"Node": {"$ref": "#/$defs/Node"}}}),++ + # A TRUTHY non-array `anyOf` does NOT fall through to `oneOf`: it is iterated, yields no++ + # mapping branch, and lands here. The other side of `falsy-anyof-falls-through-to-oneof`,++ + # and the case a port testing "is it a list" gets wrong in the opposite direction -- it++ + # falls through and, with nothing to fall through to, refuses with "unsupported schema"++ + # instead of this.++ + ("truthy-non-array-anyof-is-not-a-fallthrough",++ + {"type": "object", "properties": {"x": {"anyOf": "abc"}}}),++ + ("truthy-non-array-anyof-with-a-oneof-to-shadow",++ + {"type": "object", "properties": {"x": {"anyOf": {"a": 1},++ + "oneOf": [{"type": "boolean"},++ + {"type": "null"}]}}}),++ + # The one case of the reference's top-level `isinstance(schema, dict)` check that is++ + # reachable on a side where the argument is typed: a list, a string or a number cannot be++ + # passed at all, and null can.++ + ("top-level-is-null", None),++ +]++ +++ +++ +def _structured_runner(results):++ + """A runner that answers from a script, so `decide` is exercised without a checkpoint.++ +++ + It accepts and records `**kwargs`, which is how the reference's `min_confidence` forwarding++ + stays visible: `decide` passes it down to `predict` and `decide_batch` does not.++ + """++ + import copy++ +++ + class _Scripted:++ + def __init__(self):++ + self.predict_kwargs = []++ + self.batch_kwargs = []++ +++ + def predict(self, state, questions, **kwargs):++ + self.predict_kwargs.append(sorted(kwargs))++ + return copy.deepcopy(results[0])++ +++ + def predict_batch(self, states, questions, **kwargs):++ + self.batch_kwargs.append(sorted(kwargs))++ + return [copy.deepcopy(results[i]) for i in range(len(states))]++ +++ + return _Scripted()++ +++ +++ +def _structured_result(answers):++ + return {"model": "laya-rl-agent-onnx", "answers": answers,++ + "usage": {"input_tokens": 12, "output_tokens": 0, "state_tokens": 9,++ + "state_tokens_dropped": 0, "truncated": False,++ + "truncated_questions": []}}++ +++ +++ +#: End-to-end `decide` / `decide_batch`: the answers a runner returns, and the decision that++ +#: comes back. `min_confidence` is the wiring these pin -- the gate decides which fields are++ +#: nulled, and the port must read the gate rather than re-implementing the rule.++ +STRUCTURED_CALLS = [++ + ("decide-ungated", False,++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"]),++ + "ok": {"type": "boolean"}}}, None,++ + [{"dept": _sd_answer("choice", choice="support", answer_confidence=0.9,++ + probabilities={"billing": 0.3, "support": 0.7}),++ + "ok": _sd_answer("noul", noul=0.9, answer_confidence=0.55)}]),++ + # One field below the bar and one above it, in the same decision: the gate is per answer,++ + # so a port that nulls all or nothing passes neither half of this.++ + ("decide-gated-nulls-only-the-field-below-the-bar", False,++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"]),++ + "ok": {"type": "boolean"}}}, 0.6,++ + [{"dept": _sd_answer("choice", choice="support", answer_confidence=0.9,++ + probabilities={"billing": 0.3, "support": 0.7}),++ + "ok": _sd_answer("noul", noul=0.9, answer_confidence=0.55)}]),++ + ("decide-gated-nulls-everything", False,++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"]),++ + "ok": {"type": "boolean"}}}, 0.95,++ + [{"dept": _sd_answer("choice", choice="support", answer_confidence=0.9,++ + probabilities={"billing": 0.3, "support": 0.7}),++ + "ok": _sd_answer("noul", noul=0.9, answer_confidence=0.55)}]),++ + # A scalar 0.0 IS a gate: states are reported and nothing is nulled.++ + ("decide-gated-at-zero-nulls-nothing", False,++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"])}}, 0.0,++ + [{"dept": _sd_answer("choice", choice="billing", answer_confidence=0.1,++ + probabilities={"billing": 0.6, "support": 0.4})}]),++ + # A noul reports the two-sided distribution it implies, ROUNDED to four decimals as every++ + # other number in an answer is. 0.123456 is chosen so the rounding is visible: unrounded the++ + # sides are 0.876544 and 0.123456, which differ from the recorded 0.8765 and 0.1235 by 4.4e-5++ + # -- a gap a tolerance can see, where 0.9 would hide it at 1e-17.++ + ("decide-rounds-the-nouls-two-sided-distribution", False,++ + {"type": "object", "properties": {"ok": {"type": "boolean"}}}, None,++ + [{"ok": _sd_answer("noul", noul=0.123456, answer_confidence=0.88)}]),++ + ("decide-batch-keeps-input-order", True,++ + {"type": "object", "properties": {"n": {"type": "integer", "minimum": 1, "maximum": 3}}},++ + None,++ + [{"n": _sd_answer("score", score=0.0, probabilities={"0": 0.8, "1": 0.1, "2": 0.1})},++ + {"n": _sd_answer("score", score=1.0, probabilities={"0": 0.1, "1": 0.8, "2": 0.1})},++ + {"n": _sd_answer("score", score=2.0, probabilities={"0": 0.1, "1": 0.1, "2": 0.8})}]),++ + ("decide-batch-gates-each-state-on-its-own", True,++ + {"type": "object", "properties": {"dept": _sd_enum(["billing", "support"]),++ + "ok": {"type": "boolean"}}}, 0.5,++ + [{"dept": _sd_answer("choice", choice="support", answer_confidence=0.9,++ + probabilities={"billing": 0.3, "support": 0.7}),++ + "ok": _sd_answer("noul", noul=0.9, answer_confidence=0.55)},++ + {"dept": _sd_answer("choice", choice="billing", answer_confidence=0.4,++ + probabilities={"billing": 0.6, "support": 0.4}),++ + "ok": _sd_answer("noul", noul=0.2, answer_confidence=0.55)}]),++ +]++ +++ +++ +def structured():++ + """`laya.structured`: a JSON schema as questions, and the answers projected back onto it.++ +++ + Four things a port gets wrong by default, and each has cases chosen so that getting it wrong++ + changes something here:++ +++ + * a `choice` projects to the schema VALUE, not the label the model answered with. For a++ + string enum the two coincide, so `enum: [10, 20, 30]` is what tells them apart.++ + * a `score` is `minimum + argmax(probabilities)`. `score` itself is the probability-weighted++ + LEVEL INDEX, so the fallback for an answer with no probabilities rounds it and adds++ + `minimum` once -- subtracting `minimum` first and adding it back cancels out and silently++ + drops it. Python's `round` is half to EVEN, which `Math.round` is not.++ + * a `noul` is `>= 0.5`, so exactly 0.5 is TRUE and a port spelling it `>` differs on that++ + one value and no other.++ + * a low-confidence answer projects to NULL, and an UNANSWERED field is absent instead.++ + "not believed" and "not asked" are different outcomes and a port that conflates them++ + reports a decision nobody made.++ +++ + The pydantic half of the module is deliberately NOT recorded: there is no pydantic on the++ + JVM, and `_schema_of` is the only thing between a model and the JSON-schema path this++ + records.++ + """++ + import copy++ +++ + from laya import structured as reference++ + from laya.structured import (SchemaError, answers_to_json, decide, decide_batch,++ + plan_from_json_schema, questions_from_json_schema)++ +++ + # The mirrored limits above are only safe if they are still the reference's.++ + limits = {"max_properties": reference.MAX_PROPERTIES,++ + "max_options": reference.MAX_OPTIONS,++ + "max_score_levels": reference.MAX_SCORE_LEVELS}++ + mirrored = {"max_properties": STRUCTURED_MAX_PROPERTIES,++ + "max_options": STRUCTURED_MAX_OPTIONS,++ + "max_score_levels": STRUCTURED_MAX_SCORE_LEVELS}++ + if limits != mirrored:++ + raise AssertionError(++ + "the mirrored limits %r are no longer laya.structured's %r, so every boundary case"++ + " below is recorded off the boundary" % (mirrored, limits))++ +++ + schemas = []++ + for cid, schema in STRUCTURED_SCHEMAS:++ + plan = plan_from_json_schema(schema)++ + schemas.append({++ + "case": cid,++ + "schema": schema,++ + "questions": questions_from_json_schema(schema),++ + # The plan as well as the questions: the OPTION VALUES and the score `minimum` never++ + # reach a question, and they are exactly what the projection needs. A fixture that++ + # recorded only the questions would leave both unpinned until a projection case++ + # happened to cover them.++ + "plan": [{"name": f.name, "kind": f.kind,++ + "options": [[label, value] for label, value in f.options],++ + "minimum": f.minimum} for f in plan],++ + })++ +++ + projections = []++ + for cid, schema, answers in STRUCTURED_PROJECTIONS:++ + projections.append({"case": cid, "schema": schema, "answers": answers,++ + "values": answers_to_json(answers, schema)})++ +++ + refusals = []++ + for cid, schema in STRUCTURED_REFUSALS:++ + try:++ + questions_from_json_schema(schema)++ + except SchemaError as problem:++ + refusals.append({"case": cid, "schema": schema, "error": str(problem)})++ + else:++ + raise AssertionError("%s was supposed to be refused and was not" % cid)++ +++ + # `decide` itself: exactly one of schema/questions, and a batch over a real sequence.++ + arguments = []++ + for cid, call in (++ + ("decide-with-both", lambda: decide(None, "s", schema={"type": "object"},++ + questions={})),++ + ("decide-with-neither", lambda: decide(None, "s")),++ + ("decide-batch-with-both",++ + lambda: decide_batch(None, ["s"], schema={"type": "object"}, questions={})),++ + ("decide-batch-with-neither", lambda: decide_batch(None, ["s"])),++ + ("decide-batch-over-a-string",++ + lambda: decide_batch(None, "not a list", schema={"type": "object"})),++ + ("decide-batch-without-a-batching-runner",++ + lambda: decide_batch(object(), ["s"],++ + schema={"type": "object",++ + "properties": {"ok": {"type": "boolean"}}})),++ + ):++ + try:++ + call()++ + except (ValueError, TypeError) as problem:++ + arguments.append({"case": cid, "error": str(problem)})++ + else:++ + raise AssertionError("%s was supposed to be refused and was not" % cid)++ +++ + calls = []++ + for cid, batch, schema, threshold, scripted in STRUCTURED_CALLS:++ + runner = _structured_runner([_structured_result(a) for a in scripted])++ + states = ["state %d" % i for i in range(len(scripted))]++ + if batch:++ + decisions = decide_batch(runner, states, schema=schema, return_details=True,++ + min_confidence=threshold)++ + else:++ + decisions = [decide(runner, states[0], schema=schema, return_details=True,++ + min_confidence=threshold)]++ + calls.append({++ + "case": cid,++ + "batch": batch,++ + "schema": schema,++ + "min_confidence": threshold,++ + # The answers the runner HANDED BACK, before the gate wrote anything into them --++ + # which is what a port has to feed its own runner stub.++ + "predictions": copy.deepcopy(scripted),++ + "decisions": [{"values": d.values,++ + "confidence": d.confidence,++ + "answer_confidence": d.answer_confidence,++ + "probabilities": d.probabilities} for d in decisions],++ + # `decide` forwards `min_confidence` to `predict`; `decide_batch` applies the gate++ + # itself and forwards nothing. Recorded because a port whose runner has no such++ + # keyword must take the second path for both, and this is what says so.++ + "forwarded": sorted(set(sum(runner.predict_kwargs + runner.batch_kwargs, []))),++ + })++ +++ + # Coverage. Each of these is a branch of the mapping, and a case list that stops reaching one++ + # leaves that branch asserted by nothing -- which is how a fixture family quietly stops being++ + # a gate.++ + kinds = {f["kind"] for case in schemas for f in case["plan"]}++ + missing_kinds = {"choice", "score", "noul"} - kinds++ + if missing_kinds:++ + raise AssertionError("no schema case produces a %s question"++ + % ", ".join(sorted(missing_kinds)))++ + required_messages = [++ + "a free string cannot be a fixed option set",++ + "arrays are not supported",++ + "nested objects are not supported",++ + "only 'Optional[...]' unions (one non-null branch) are supported",++ + "'type' has multiple non-null types",++ + "expected a JSON schema object",++ + "the top level must be an object with 'properties'",++ + "'properties' must be a non-empty object",++ + "exceeds MAX_PROPERTIES=%d" % STRUCTURED_MAX_PROPERTIES,++ + "exceeds MAX_OPTIONS=%d" % STRUCTURED_MAX_OPTIONS,++ + "exceeds MAX_SCORE_LEVELS=%d" % STRUCTURED_MAX_SCORE_LEVELS,++ + "'enum' must not be empty",++ + "duplicate choice labels",++ + "needs integer 'minimum' and 'maximum'",++ + "is below 'minimum'",++ + "unsupported schema",++ + "property must be an object",++ + "is recursive",++ + "does not resolve to an entry",++ + ]++ + recorded = [r["error"] for r in refusals]++ + unreached = [m for m in required_messages if not any(m in e for e in recorded)]++ + if unreached:++ + raise AssertionError("no refusal case reaches: %s" % "; ".join(unreached))++ +++ + projected = [v for case in projections for v in case["values"].values()]++ + omitted = sum(1 for case in projections++ + if set(case["schema"]["properties"]) - set(case["values"]))++ + # A null from a FLAG and a null that is the schema's own value are different branches.++ + nulled = sum(1 for case in projections for name, value in case["values"].items()++ + if value is None and case["answers"][name].get("low_confidence"))++ + null_valued = sum(1 for case in projections for name, value in case["values"].items()++ + if value is None and not case["answers"][name].get("low_confidence"))++ + booleans = sum(1 for v in projected if isinstance(v, bool))++ + integers = sum(1 for v in projected if isinstance(v, int) and not isinstance(v, bool))++ + relabelled = sum(1 for case in projections++ + for name, value in case["values"].items()++ + if isinstance(value, int) and not isinstance(value, bool)++ + and case["answers"][name]["type"] == "choice")++ + no_probability_scores = sum(1 for case in projections++ + for answer in case["answers"].values()++ + if answer["type"] == "score" and not answer["probabilities"])++ + shifted = sum(1 for case in projections++ + for name in case["values"]++ + if case["answers"][name]["type"] == "score"++ + and case["schema"]["properties"][name].get("minimum") not in (0, None))++ + half = sum(1 for case in projections++ + for answer in case["answers"].values()++ + if answer.get("noul") == 0.5)++ + shortfalls = [name for name, count, floor in (++ + ("a null from a low-confidence answer", nulled, 3),++ + ("an omitted field", omitted, 1),++ + ("a schema whose own value is null", null_valued, 1),++ + ("a boolean", booleans, 4),++ + ("an integer level", integers, 7),++ + ("a choice whose VALUE is not its label", relabelled, 2),++ + ("a score with no probabilities", no_probability_scores, 3),++ + ("a score whose minimum is not zero", shifted, 4),++ + ("a noul of exactly 0.5", half, 1),++ + ) if count < floor]++ + if shortfalls:++ + raise AssertionError("the projection cases no longer reach: %s" % "; ".join(shortfalls))++ +++ + gated_nulls = sum(1 for case in calls for d in case["decisions"]++ + for v in d["values"].values() if v is None)++ + batched = sum(1 for case in calls if case["batch"])++ + if gated_nulls < 3 or batched < 2 or len(arguments) < 4:++ + raise AssertionError(++ + "the end-to-end cases no longer pin the wiring: %d gated null(s), %d batch case(s),"++ + " %d argument refusal(s)" % (gated_nulls, batched, len(arguments)))++ +++ + return {++ + "limits": limits,++ + "schemas": schemas,++ + "projections": projections,++ + "refusals": refusals,++ + "arguments": arguments,++ + "calls": calls,++ + }++ +++ ++def _decoder_shape(tokenizer_json):+ """The decoder section's type chain, flattened, so the fixture says what it was recorded on."""+ import io+ import json++ document = json.load(io.open(tokenizer_json, encoding="utf-8"))+ node = document.get("decoder")+ if node is None:+ return None++ def shape(n):+ kind = n.get("type")+ if kind == "Sequence":+ return {"type": "Sequence",+ "decoders": [shape(step) for step in n.get("decoders", [])]}+ return {"type": kind}++ return shape(node)++++ #: Results carry a usage block only, so a case's provenance is readable from one number:+++ #: inference returns 110+, a start hook's `ctx.skip()` returns 10+, an end hook's rewrite 510+.+++ def _hook_result(i):+++ """One result dict in the shape `predict_batch` returns, with a usage block to aggregate."""+++ return {"model": "laya-rl-agent-onnx", "answers": {},+++ "usage": {"input_tokens": 10 + i, "output_tokens": i, "state_tokens": 7,+++ "state_tokens_dropped": 0, "truncated": False,+++ "truncated_questions": []}}+++++++++ def _hook_questions(ids):+++ """A question mapping with those ids: one noul each, which is all these cases need asked."""+++ return {qid: {"noul": "is this %s" % qid} for qid in ids}+++++++++ def _apply_hook_behaviour(spec, ctx, name):+++ """The one behaviour a stub hook performs, named by a string the port reconstructs.++++++ A behaviour is data rather than a closure so the Java side can build the same chain from the+++ fixture. Anything a hook is allowed to do to a call is one of these.+++ """+++ if spec is None or spec == "record":+++ return+++ verb, _, argument = spec.partition(":")+++ if verb == "raise":+++ raise ValueError("boom from %s" % name)+++ if verb == "skip":+++ ctx.skip([_hook_result(i) for i in range(int(argument))])+++ return+++ if verb == "states":+++ ctx.states = argument.split(",") if argument else []+++ return+++ if verb == "questions":+++ ctx.questions = _hook_questions(argument.split(",") if argument else [])+++ return+++ if verb == "results":+++ ctx.results = [_hook_result(500 + i) for i in range(int(argument))]+++ return+++ if verb == "max_len":+++ ctx.max_len = int(argument)+++ return+++ if verb == "head_max_len":+++ ctx.head_max_len = int(argument)+++ return+++ if verb == "sleep":+++ import time as _time+++ _time.sleep(float(argument))+++ return+++ raise AssertionError("unknown hook behaviour %r" % (spec,))+++++++++ def _stub_hook(log, name, events, behaviour=None, observe=False):+++ """A hook implementing exactly `events`, appending to `log` and then doing `behaviour`.++++++ The class is called `StubHook` whatever the instance is named, because `dispatch`'s+++ `raise_errors=False` warning quotes `type(hook).__name__` and that text is part of what this+++ family pins -- so the port's stub has to be able to produce the same string.+++ """+++ def make(event):+++ def method(self, ctx):+++ if observe:+++ log.append(_observation(name, event, ctx))+++ else:+++ log.append("%s.%s" % (name, event))+++ spec = behaviour.get(event) if isinstance(behaviour, dict) else behaviour+++ _apply_hook_behaviour(spec, ctx, name)+++ method.__name__ = event+++ method.__qualname__ = "StubHook.%s" % event+++ return method++++++ namespace = {event: make(event) for event in events}+++ namespace["stub_name"] = name+++ return type("StubHook", (object,), namespace)()+++++++++ def _observation(name, event, ctx):+++ """Everything one hook can read off the context, as the fixture records it."""+++ states = ctx.states+++ if isinstance(states, (list, tuple)):+++ states = list(states)+++ questions = ctx.questions+++ return {+++ "call": "%s.%s" % (name, event),+++ "states": states,+++ "questions": list(questions) if isinstance(questions, dict) else questions,+++ "model": ctx.model,+++ "max_len": ctx.max_len,+++ "head_max_len": ctx.head_max_len,+++ # `None` is not `[]`: "inference has not run" and "inference returned nothing" are+++ # different states of the call, and a start hook distinguishes them to decide whether an+++ # earlier hook has already answered.+++ "results": None if ctx.results is None else [r["usage"]["input_tokens"]+++ for r in ctx.results],+++ "usage": ctx.usage,+++ "error": None if ctx.error is None else str(ctx.error),+++ # Popped before the fixture is written -- it is a fresh uuid4 per call, so recording+++ # the value would make the file differ on every run. What IS recorded is that one call's+++ # hooks all saw the SAME one, and how long it is.+++ "run_id": ctx.run_id,+++ "elapsed_ms_set": ctx.elapsed_ms is not None,+++ }+++++++++ #: (case, default hook names, installed names, per-call `hooks=` names, `on_predict_start=`+++ #: names, `on_predict_end=` names, whether the call runs under `_SKIP_DEFAULTS`).+++ HOOK_COMPOSITION_CASES = [+++ ("nothing-installed", [], [], [], [], [], False),+++ ("defaults-only", ["d1", "d2"], [], [], [], [], False),+++ ("installed-only", [], ["i1", "i2"], [], [], [], False),+++ ("per-call-only", [], [], ["p1", "p2"], [], [], False),+++ # THE order contract: defaults, then installed, then per-call.+++ ("defaults-then-installed-then-per-call", ["d1"], ["i1"], ["p1"], [], [], False),+++ # The two convenience callables are appended AFTER the per-call hook objects, starts before+++ # ends, so a `on_predict_end=` callable never runs before a `hooks=` entry's end method.+++ ("callables-come-after-hook-objects", ["d1"], ["i1"], ["p1"], ["s1"], ["e1"], False),+++ ("several-start-callables-keep-their-order", [], [], [], ["s1", "s2", "s3"], [], False),+++ ("start-callables-before-end-callables", [], [], [], ["s1"], ["e1"], False),+++ # Defaults are read at CALL time, and a scope can switch them off without unsetting them.+++ ("skip-defaults-drops-only-the-defaults", ["d1", "d2"], ["i1"], ["p1"], [], [], True),+++ ("skip-defaults-with-nothing-else", ["d1"], [], [], [], [], True),+++ ]+++++++++ #: (case, chain of (name, implemented events, behaviour), event dispatched, raise_errors).+++ HOOK_DISPATCH_CASES = [+++ ("every-hook-in-order", [("a", ["on_predict_start"], None), ("b", ["on_predict_start"], None),+++ ("c", ["on_predict_start"], None)], "on_predict_start", True),+++ # A hook implementing a different subset is SKIPPED for this event, not an error.+++ ("a-hook-that-does-not-implement-the-event-is-skipped",+++ [("a", ["on_predict_start"], None), ("b", ["on_predict_end"], None),+++ ("c", ["on_predict_start"], None)], "on_predict_start", True),+++ ("no-hook-implements-the-event",+++ [("a", ["on_predict_end"], None)], "on_route", True),+++ # Under `raise_errors` the chain STOPS at the first failure; the hooks after it never run.+++ ("raising-stops-the-chain",+++ [("a", ["on_predict_start"], None), ("b", ["on_predict_start"], "raise"),+++ ("c", ["on_predict_start"], None)], "on_predict_start", True),+++ # With it off the failure is a warning and the rest of the chain still runs, which is what a+++ # telemetry hook needs: it must not be able to fail the request.+++ ("raise-errors-false-warns-and-continues",+++ [("a", ["on_predict_start"], None), ("b", ["on_predict_start"], "raise"),+++ ("c", ["on_predict_start"], None)], "on_predict_start", False),+++ ("raise-errors-false-with-every-hook-failing",+++ [("a", ["on_predict_start"], "raise"), ("b", ["on_predict_start"], "raise")],+++ "on_predict_start", False),+++ # `ctx.skip()` does NOT short-circuit the chain. It assigns `ctx.results`, and it is+++ # `predict_batch` that reads it afterwards -- so every later start hook still runs and can+++ # see, and overwrite, what the earlier one answered.+++ ("skip-does-not-stop-the-chain",+++ [("a", ["on_predict_start"], "skip:1"), ("b", ["on_predict_start"], None)],+++ "on_predict_start", True),+++ ("the-other-four-events-dispatch-the-same-way",+++ [("a", ["on_route", "on_load", "on_evict", "on_error"], None),+++ ("b", ["on_route"], None)], "on_route", True),+++ ("on-error-dispatches-over-the-same-chain",+++ [("a", ["on_error"], None), ("b", ["on_error"], "raise")], "on_error", False),+++ ("on-predict-end-dispatches-over-the-same-chain",+++ [("a", ["on_predict_end"], None), ("b", ["on_predict_end"], None)], "on_predict_end", True),+++ ]+++++++++ #: (case, state count, result count handed to `ctx.skip`).+++ HOOK_SKIP_CASES = [+++ ("one-result-per-state", 3, 3),+++ ("one-result-for-the-whole-call", 3, 1),+++ ("one-state-one-result", 1, 1),+++ # Two results for one state is neither shape, and it is the shape that silently dropped rows+++ # the caller was about to zip against.+++ ("two-results-for-one-state", 1, 2),+++ ("too-few-results", 3, 2),+++ ("no-results-at-all", 3, 0),+++ ("too-many-results", 2, 5),+++ # An empty call: zero is both "one per state" and "none", so it is accepted.+++ ("no-states-no-results", 0, 0),+++ ("no-states-one-result", 0, 1),+++ ("no-states-two-results", 0, 2),+++ ]+++++++++ #: (case, per-result usage blocks).+++ HOOK_USAGE_CASES = [+++ ("one-result", [{"input_tokens": 11, "output_tokens": 0}]),+++ ("several-results-sum", [{"input_tokens": 11, "output_tokens": 1},+++ {"input_tokens": 22, "output_tokens": 2},+++ {"input_tokens": 33, "output_tokens": 3}]),+++ ("no-results-at-all", []),+++ ("zeroes", [{"input_tokens": 0, "output_tokens": 0}]),+++ # The reference coerces, because a result dict is a plain dict a hook may have rewritten.+++ ("a-missing-usage-block", [{}]),+++ ("a-null-usage-block", [{"__usage_is_none__": True}]),+++ ("a-missing-key", [{"input_tokens": 7}]),+++ ("a-null-value", [{"input_tokens": 7, "output_tokens": None}]),+++ ("a-float-is-truncated", [{"input_tokens": 7.9, "output_tokens": 0}]),+++ ]+++++++++ #: Values offered to `validate_timeout`.+++ #:+++ #: The whole seconds -- 1, 2, 10, 60 -- and 1e-07 are here for the RENDERING, not the validation:+++ #: `_call_hook` reports an overrun with `%g`, which prints `1` where a naive port prints `1.0`+++ #: and `1e-07` where it prints `1.0E-7`. A whole number of seconds is the common case for a+++ #: deadline, so it is the common case for that line, and `laya-java` compares its own formatter+++ #: against the `seconds` field below rather than against a string somebody typed.+++ HOOK_TIMEOUT_VALUES = [None, 1, 2, 10, 60, 1.5, 0.05, 1e-07, "0.5", 0, 0.0, -1, -0.5,+++ float("nan"), float("inf"), float("-inf")]+++++++++ #: (case, scripted registry operations). `enter`/`exit` bracket a `hooks_installed` block.+++ HOOK_REGISTRY_CASES = [+++ ("add-keeps-insertion-order", ["add:a", "add:b", "add:c"]),+++ ("add-a-sequence-at-once", ["add:[a,b]", "add:c"]),+++ # The same instance twice is two installations, and removing by identity takes both.+++ ("the-same-hook-twice", ["add:a", "add:a"]),+++ ("remove-by-identity", ["add:a", "add:b", "remove:a"]),+++ ("remove-takes-every-copy", ["add:a", "add:a", "add:b", "remove:a"]),+++ ("remove-something-never-installed", ["add:a", "remove:b"]),+++ ("a-block-installs-and-restores", ["enter:1:[t]", "exit:1"]),+++ # The rule `_remove_instances` exists for: the application had already installed `a`, so the+++ # block's exit must take ONE copy -- its own, the most recent -- and leave the other.+++ ("a-block-over-an-already-installed-hook", ["add:a", "enter:1:[a]", "exit:1"]),+++ # Two overlapping blocks. Restoring a snapshot would make the first exit remove the second+++ # block's hook; taking one occurrence each does not.+++ ("overlapping-blocks", ["enter:1:[x]", "enter:2:[y]", "exit:2", "exit:1"]),+++ ("overlapping-blocks-exited-outermost-first",+++ ["enter:1:[x]", "enter:2:[y]", "exit:1", "exit:2"]),+++ # A hook added with add_hook INSIDE a block is not in any snapshot and must survive the exit.+++ ("add-inside-a-block-survives-the-exit", ["enter:1:[t]", "add:d", "exit:1"]),+++ ("a-block-with-several-arguments", ["enter:1:[x],[y,z]", "exit:1"]),+++ ("a-block-with-nothing-in-it", ["enter:1:", "exit:1"]),+++ ]+++++++++ #: (case, default hooks, installed hooks, per-call hooks, states, question ids, hooks_raise,+++ #: whether the stubbed inference itself raises). Each hook is `(name, {event: behaviour})` and+++ #: implements EVERY event, so the log says exactly which events fired and in which order --+++ #: which event a hook implements is the dispatch section's subject, not this one's.+++ HOOK_PREDICT_CASES = [+++ ("no-hooks", [], [], [], ["s0"], ["q"], True, False),+++ # Question ids that are NOT in alphabetical order, deliberately: a choice's options are+++ # positional and a question map is iterated, so a port that re-sorts the mapping on the way+++ # into the context asks a different question than the caller wrote -- and with ids like+++ # "q", "r" it would still compare clean.+++ ("a-start-hook-sees-the-call", [], [], [("p", {})], ["s0", "s1"], ["urgent", "intent"],+++ True, False),+++ # Order end to end: a default, an installed and a per-call hook all fire, in that order, for+++ # both events -- and `compose_hooks` reads the defaults at CALL time, not at construction.+++ ("default-installed-and-per-call-all-fire", [("d", {})], [("i", {})], [("p", {})], ["s0"],+++ ["q"], True, False),+++ # A start hook rewrites the call, and it is the REWRITTEN value inference is handed.+++ ("a-start-hook-rewrites-the-states", [], [], [("p", {"on_predict_start": "states:x,y,z"})],+++ ["s0"], ["q"], True, False),+++ # Two ids, out of alphabetical order, so a port that re-sorts the rewritten mapping on the+++ # way back onto the context differs here rather than passing on a one-question rewrite.+++ ("a-start-hook-rewrites-the-questions", [], [],+++ [("p", {"on_predict_start": "questions:urgent,intent"})], ["s0"], ["q", "r"], True, False),+++ ("a-start-hook-sets-the-token-budget", [], [], [("p", {"on_predict_start": "max_len:128"})],+++ ["s0"], ["q"], True, False),+++ ("a-start-hook-sets-the-head-budget", [], [],+++ [("p", {"on_predict_start": "head_max_len:64"})], ["s0"], ["q"], True, False),+++ # `ctx.skip()` short-circuits inference: the stub is never called and the hook's own payload+++ # is what comes back.+++ ("skip-answers-the-call", [], [], [("p", {"on_predict_start": "skip:1"})], ["s0"], ["q"],+++ True, False),+++ ("skip-one-per-state", [], [], [("p", {"on_predict_start": "skip:2"})], ["s0", "s1"], ["q"],+++ True, False),+++ ("skip-one-for-the-whole-call", [], [], [("p", {"on_predict_start": "skip:1"})],+++ ["s0", "s1"], ["q"], True, False),+++ ("skip-with-a-count-that-fits-nothing", [], [], [("p", {"on_predict_start": "skip:3"})],+++ ["s0", "s1"], ["q"], True, False),+++ # A later start hook can overwrite what an earlier one answered, because `ctx.skip` assigns+++ # and dispatch does not stop.+++ ("the-last-skip-wins", [], [], [("p", {"on_predict_start": "skip:1"}),+++ ("p2", {"on_predict_start": "skip:2"})],+++ ["s0", "s1"], ["q"], True, False),+++ # An end hook may replace the whole result list, and that is the return value.+++ ("an-end-hook-rewrites-the-results", [], [], [("p", {"on_predict_end": "results:1"})],+++ ["s0"], ["q"], True, False),+++ # A throwing hook, under each policy and at each end of the call.+++ ("a-raising-start-hook-fails-the-call", [], [], [("p", {"on_predict_start": "raise"}),+++ ("p2", {})], ["s0"], ["q"], True, False),+++ ("a-raising-start-hook-is-only-a-warning", [], [], [("p", {"on_predict_start": "raise"}),+++ ("p2", {})], ["s0"], ["q"], False, False),+++ ("a-raising-end-hook-fails-the-call", [], [], [("p", {"on_predict_end": "raise"})], ["s0"],+++ ["q"], True, False),+++ ("a-raising-end-hook-is-only-a-warning", [], [], [("p", {"on_predict_end": "raise"})],+++ ["s0"], ["q"], False, False),+++ # Inference itself fails: on_error runs, then on_predict_end runs ANYWAY, with the error on+++ # the context and `ctx.results` still None.+++ ("inference-fails", [], [], [("p", {})], ["s0"], ["q"], True, True),+++ # A failing on_error hook must not replace the failure that triggered it.+++ ("inference-fails-with-a-raising-error-hook", [], [], [("p", {"on_error": "raise"})], ["s0"],+++ ["q"], True, True),+++ # A failing end hook on the FAILURE path must not mask the real error either.+++ ("inference-fails-with-a-raising-end-hook", [], [], [("p", {"on_predict_end": "raise"})],+++ ["s0"], ["q"], True, True),+++ # The empty call: no inference, an empty result list, and the end hook still sees it.+++ ("no-states", [], [], [("p", {})], [], ["q"], True, False),+++ ]+++++++++ #: The lifecycle event names, in the reference's own order, asserted against `HOOK_EVENTS` rather+++ #: than imported: the ORDER is what a port's enum has to agree with, and importing it would make+++ #: a reordering invisible here.+++ HOOK_EVENT_NAMES = ["on_predict_start", "on_predict_end", "on_route", "on_load", "on_evict",+++ "on_error"]+++++++++ def _hook_name_of(entry):+++ """The label the fixture gives one entry of a composed hook list."""+++ kind = type(entry).__name__+++ if kind in ("_StartAdapter", "_EndAdapter"):+++ return ("start:" if kind == "_StartAdapter" else "end:") + entry.fn.laya_stub_name+++ return getattr(entry, "stub_name", kind)+++++++++ def _named_callable(name):+++ """A plain `on_predict_start=` / `on_predict_end=` callable, labelled for the fixture."""+++ def fn(ctx):+++ return None+++ fn.laya_stub_name = name+++ return fn+++++++++ def hook_dispatch():+++ """`laya.hooks`: what runs, in what order, and what each hook is allowed to change.++++++ Hooks are the one place a caller's own code runs inside a prediction, so every observable+++ here is a contract a port has to reproduce exactly. The sections below are chosen so a port+++ that gets any of them wrong differs somewhere:++++++ * ORDER. `compose_hooks` is defaults, then installed, then per-call, and `dispatch` calls+++ them in that order. A port that composes the other way round still runs every hook, so+++ nothing fails -- except that the tracer a caller installed to see what the per-call hook+++ did now runs first and sees nothing.+++ * `ctx.skip()` assigns `ctx.results` and does NOT stop the chain. The short-circuit is+++ `predict_batch` reading `ctx.results` afterwards, so a later start hook still runs and+++ can overwrite the answer. A port that returns early from dispatch loses that.+++ * `hooks_raise`. True fails the request at the first throwing hook and the rest of the+++ chain never runs; False warns, with a message quoting the hook's class and event, and+++ continues. Both ends of the call, and the failure path, are recorded.+++ * what a hook SEES. The states, the questions, the model, the token-budget overrides, the+++ run id shared across one call's hooks, and -- at the end -- the aggregated usage, the+++ elapsed time and the error. A port that builds the context late, or rebuilds it per+++ event, breaks the identity those depend on.+++ * `ctx.skip()`'s count check. One result for the whole call or one per state, nothing else,+++ refused INSIDE the hook under the caller's own `hooks_raise` rather than downstream --+++ where it had been an `IndexError` the server mapped to a 500, and a short list+++ `predict_batch` returned while the caller zipped it against their states.++++++ Inference is stubbed, so this family needs no checkpoint and no graph: `ONNXAgent.predict_batch`+++ is driven unbound over a stub that carries only what it touches. That is also what makes the+++ "skip answers the call" cases legible -- whether the model ran is recorded, not inferred.+++ """+++ import threading+++ import warnings++++++ from laya.hooks import (+++ HOOK_EVENTS, HookRegistry, PredictContext, _as_sequence, add_default_hook,+++ aggregate_usage, clear_default_hooks, compose_hooks, default_hooks, dispatch,+++ normalise_hooks, set_default_hooks, validate_timeout,+++ )+++ from laya.hooks import _SKIP_DEFAULTS++++++ assert list(HOOK_EVENTS) == HOOK_EVENT_NAMES, (+++ "the lifecycle events moved: %r" % (HOOK_EVENTS,))++++++ # -- composition order ---------------------------------------------------+++ composition = []+++ for (cid, defaults, installed, per_call, starts, ends, skip_defaults) in \+++ HOOK_COMPOSITION_CASES:+++ log = []+++ clear_default_hooks()+++ set_default_hooks([_stub_hook(log, n, HOOK_EVENT_NAMES) for n in defaults])+++ token = _SKIP_DEFAULTS.set(True) if skip_defaults else None+++ try:+++ active = compose_hooks(+++ [_stub_hook(log, n, HOOK_EVENT_NAMES) for n in installed],+++ [_stub_hook(log, n, HOOK_EVENT_NAMES) for n in per_call],+++ [_named_callable(n) for n in starts],+++ [_named_callable(n) for n in ends])+++ finally:+++ if token is not None:+++ _SKIP_DEFAULTS.reset(token)+++ clear_default_hooks()+++ composition.append({+++ "case": cid, "defaults": defaults, "installed": installed, "hooks": per_call,+++ "on_predict_start": starts, "on_predict_end": ends,+++ "skip_defaults": skip_defaults,+++ "order": [_hook_name_of(entry) for entry in active],+++ })++++++ # `default_hooks()` hands back a COPY: a caller that mutates it must not be editing the+++ # process-wide list, which is the difference between a read and a handle.+++ clear_default_hooks()+++ probe = _stub_hook([], "probe", HOOK_EVENT_NAMES)+++ set_default_hooks([probe])+++ defaults_api = {"after_set": [_hook_name_of(h) for h in default_hooks()]}+++ borrowed = default_hooks()+++ borrowed.append(_stub_hook([], "intruder", HOOK_EVENT_NAMES))+++ defaults_api["mutating_the_returned_list_changes_nothing"] = [+++ _hook_name_of(h) for h in default_hooks()]+++ add_default_hook(_stub_hook([], "added", HOOK_EVENT_NAMES))+++ defaults_api["after_add"] = [_hook_name_of(h) for h in default_hooks()]+++ clear_default_hooks()+++ defaults_api["after_clear"] = [_hook_name_of(h) for h in default_hooks()]++++++ # -- what normalise_hooks refuses ---------------------------------------+++ class _NotAHook(object):+++ pass++++++ class _NotCallable(object):+++ on_predict_start = "not a function"++++++ refusals = []+++ for label, argument in (("a class rather than an instance", _NotAHook),+++ ("an object implementing no event", _NotAHook()),+++ ("an event attribute that is not callable", _NotCallable()),+++ ("a plain object", object())):+++ try:+++ normalise_hooks(argument)+++ refusals.append({"input": label, "error": None})+++ except TypeError as problem:+++ refusals.append({"input": label, "error": str(problem)})+++ for label, argument in (("a non-callable on_predict_start", 7),+++ ("a non-callable on_predict_end", "nope")):+++ try:+++ if "start" in label:+++ normalise_hooks(None, argument, None)+++ else:+++ normalise_hooks(None, None, argument)+++ refusals.append({"input": label, "error": None})+++ except TypeError as problem:+++ refusals.append({"input": label, "error": str(problem)})++++++ # `_as_sequence`: one hook, a list of them, or nothing, flattened the same way.+++ as_sequence = [+++ {"input": "none", "length": len(_as_sequence(None))},+++ {"input": "one", "length": len(_as_sequence(probe))},+++ {"input": "list-of-two", "length": len(_as_sequence([probe, probe]))},+++ {"input": "tuple-of-three", "length": len(_as_sequence((probe, probe, probe)))},+++ {"input": "empty-list", "length": len(_as_sequence([]))},+++ ]++++++ # -- dispatch ------------------------------------------------------------+++ dispatched = []+++ for cid, chain, event, raise_errors in HOOK_DISPATCH_CASES:+++ log = []+++ hooks = [_stub_hook(log, name, events, behaviour) for name, events, behaviour in chain]+++ ctx = PredictContext(states=["s0"], questions=_hook_questions(["q"]))+++ raised = None+++ with warnings.catch_warnings(record=True) as caught:+++ warnings.simplefilter("always")+++ try:+++ dispatch(hooks, event, ctx, raise_errors=raise_errors)+++ except BaseException as problem:+++ raised = {"type": type(problem).__name__, "message": str(problem)}+++ dispatched.append({+++ "case": cid, "event": event, "raise_errors": raise_errors,+++ "chain": [{"name": name, "events": events, "behaviour": behaviour}+++ for name, events, behaviour in chain],+++ "calls": log, "raised": raised,+++ "warnings": [str(w.message) for w in caught],+++ "results": None if ctx.results is None else len(ctx.results),+++ })++++++ # -- ctx.skip() ----------------------------------------------------------+++ skips = []+++ for cid, n_states, n_results in HOOK_SKIP_CASES:+++ ctx = PredictContext(states=["s%d" % i for i in range(n_states)],+++ questions=_hook_questions(["q"]))+++ try:+++ ctx.skip([_hook_result(i) for i in range(n_results)])+++ skips.append({"case": cid, "states": n_states, "results": n_results,+++ "accepted": True, "error": None,+++ "assigned": [r["usage"]["input_tokens"] for r in ctx.results]})+++ except ValueError as problem:+++ skips.append({"case": cid, "states": n_states, "results": n_results,+++ "accepted": False, "error": str(problem),+++ "assigned": None})++++++ # -- aggregate_usage -----------------------------------------------------+++ usages = []+++ for cid, blocks in HOOK_USAGE_CASES:+++ results = []+++ for block in blocks:+++ if block.get("__usage_is_none__"):+++ results.append({"usage": None})+++ elif block:+++ results.append({"usage": dict(block)})+++ else:+++ results.append({})+++ usages.append({"case": cid, "results": blocks, "total": aggregate_usage(results)})++++++ # -- validate_timeout ----------------------------------------------------+++ timeouts = []+++ for value in HOOK_TIMEOUT_VALUES:+++ entry = {"input": repr(value)}+++ try:+++ accepted = validate_timeout(value)+++ entry["accepted"] = True+++ entry["value"] = accepted+++ # Exactly the rendering `_call_hook` puts in its TimeoutError, which the port has to+++ # reproduce character for character -- see that function and `HOOK_TIMEOUT_VALUES`.+++ entry["seconds"] = None if accepted is None else "%g" % accepted+++ entry["error"] = None+++ except (ValueError, OverflowError) as problem:+++ entry["accepted"] = False+++ entry["value"] = None+++ entry["seconds"] = None+++ entry["error"] = str(problem)+++ timeouts.append(entry)++++++ # -- HookRegistry --------------------------------------------------------+++ registries = []+++ for cid, script in HOOK_REGISTRY_CASES:+++ class _Registry(HookRegistry):+++ def __init__(self):+++ self.hooks = ()+++ self._hooks_mutex = threading.Lock()++++++ registry = _Registry()+++ pool = {}++++++ def named(label):+++ if label not in pool:+++ pool[label] = _stub_hook([], label, HOOK_EVENT_NAMES)+++ return pool[label]++++++ blocks = {}+++ steps = []+++ for op in script:+++ verb, _, rest = op.partition(":")+++ returned = None+++ if verb == "add":+++ if rest.startswith("["):+++ registry.add_hook([named(x) for x in rest[1:-1].split(",") if x])+++ else:+++ registry.add_hook(named(rest))+++ elif verb == "remove":+++ returned = registry.remove_hook(named(rest))+++ elif verb == "enter":+++ block, _, arguments = rest.partition(":")+++ groups = []+++ for group in ([] if not arguments else arguments.split("],[")):+++ labels = [x for x in group.strip("[]").split(",") if x]+++ groups.append([named(x) for x in labels])+++ manager = registry.hooks_installed(*groups)+++ blocks[block] = manager+++ manager.__enter__()+++ elif verb == "exit":+++ blocks.pop(rest).__exit__(None, None, None)+++ else:+++ raise AssertionError("unknown registry op %r" % (op,))+++ steps.append({"op": op, "returned": returned,+++ "hooks": [_hook_name_of(h) for h in registry.hooks]})+++ for leftover in list(blocks.values()):+++ leftover.__exit__(None, None, None)+++ registries.append({"case": cid, "script": list(script), "steps": steps,+++ "final": [_hook_name_of(h) for h in registry.hooks]})++++++ predicted = _hook_predict_cases()++++++ # The branches have to be REACHED. Every case below compares clean against a port that never+++ # runs a hook at all, so a case list that stopped constructing a short-circuit, a refusal or+++ # a failure would still be green -- which is the one outcome this family must not produce.+++ reached = {+++ "three-tier order": sum(1 for c in composition+++ if c["defaults"] and c["installed"] and c["hooks"]+++ and c["order"][:len(c["defaults"])] == c["defaults"]),+++ "defaults skipped": sum(1 for c in composition if c["skip_defaults"] and c["defaults"]+++ and not set(c["defaults"]) & set(c["order"])),+++ "dispatch stopped early": sum(1 for c in dispatched if c["raised"]),+++ "dispatch warned and continued": sum(1 for c in dispatched+++ if c["warnings"] and not c["raised"]),+++ "dispatch skipped a hook": sum(1 for c in dispatched+++ if len(c["calls"]) < len(c["chain"])+++ and not c["raised"]),+++ "skip refused": sum(1 for c in skips if not c["accepted"]),+++ "skip accepted": sum(1 for c in skips if c["accepted"]),+++ "timeout refused": sum(1 for t in timeouts if not t["accepted"]),+++ "block left a hook behind": sum(1 for r in registries if r["final"]),+++ "block restored": sum(1 for r in registries if not r["final"]),+++ "inference short-circuited": sum(1 for c in predicted if c["inference"] is None+++ and c["raised"] is None),+++ "inference saw a rewrite": sum(1 for c in predicted if c["inference"]+++ and (c["inference"]["states"] != c["states"]+++ or c["inference"]["questions"] != c["questions"]+++ or c["inference"]["overrides"])),+++ "call failed": sum(1 for c in predicted if c["raised"]),+++ "call warned": sum(1 for c in predicted if c["warnings"]),+++ "on_error fired": sum(1 for c in predicted+++ if any(e["call"].endswith(".on_error") for e in c["calls"])),+++ "a hook failure was attached, not substituted": sum(+++ 1 for c in predicted if c["raised"] and c["raised"]["context"]),+++ "an end hook saw the aggregated usage": sum(+++ 1 for c in predicted+++ if any(e["call"].endswith(".on_predict_end") and e["usage"] for e in c["calls"])),+++ }+++ thin = {name: count for name, count in reached.items() if count < 1}+++ if thin or reached["skip refused"] < 4 or reached["inference short-circuited"] < 3:+++ raise AssertionError(+++ "the hook cases do not reach every branch this family exists to pin: %s"+++ % ", ".join("%s=%d" % pair for pair in sorted(reached.items())))++++++ return {+++ "events": HOOK_EVENT_NAMES,+++ "composition": composition,+++ "default_hooks_api": defaults_api,+++ "normalise_refusals": refusals,+++ "as_sequence": as_sequence,+++ "dispatch": dispatched,+++ "skip": skips,+++ "usage": usages,+++ "timeouts": timeouts,+++ "registry": registries,+++ "predict_batch": predicted,+++ }+++++++++ def _hook_predict_cases():+++ """`ONNXAgent.predict_batch` driven over a stubbed inference, one case per contract.++++++ Unbound over a stub that carries only what the method touches -- the hook list, the three+++ policy fields, the model id and `_infer_batch` -- so this needs no checkpoint and no graph,+++ and so the fixture can record whether the model was reached at all. That last part is the+++ point of the `skip` cases: "inference did not run" is the observable, and inferring it from+++ the payload would not distinguish a short-circuit from a model that happened to agree.+++ """+++ import warnings++++++ from laya.hooks import clear_default_hooks, set_default_hooks++++++ from laya.onnx_agent import ONNXAgent++++++ cases = []+++ for (cid, defaults, installed, per_call, states, qids, hooks_raise,+++ inference_raises) in HOOK_PREDICT_CASES:+++ log = []+++ seen = {}++++++ class Stub(object):+++ """Everything `predict_batch` touches on `self`, and nothing else."""++++++ model_id = "stub-checkpoint"+++ hooks_concurrent = True+++ _hooks_lock = None++++++ def __init__(self):+++ self.hooks = [_stub_hook(log, n, HOOK_EVENT_NAMES, b, observe=True)+++ for n, b in installed]+++ self.hooks_raise = hooks_raise+++ self.hooks_timeout = None++++++ def _infer_batch(self, inner_states, inner_questions, lang=None, batch_size=None,+++ sort_by_length=False, **overrides):+++ seen["states"] = list(inner_states)+++ seen["questions"] = list(inner_questions)+++ seen["overrides"] = dict(overrides)+++ if inference_raises:+++ raise RuntimeError("the graph refused this batch")+++ return [_hook_result(100 + i) for i in range(len(inner_states))]++++++ clear_default_hooks()+++ if defaults:+++ set_default_hooks([_stub_hook(log, n, HOOK_EVENT_NAMES, b, observe=True)+++ for n, b in defaults])+++ questions = _hook_questions(qids)+++ raised = None+++ results = None+++ try:+++ with warnings.catch_warnings(record=True) as caught:+++ warnings.simplefilter("always")+++ results = ONNXAgent.predict_batch(+++ Stub(), list(states), questions,+++ hooks=[_stub_hook(log, n, HOOK_EVENT_NAMES, b, observe=True)+++ for n, b in per_call])+++ except BaseException as problem:+++ raised = {"type": type(problem).__name__, "message": str(problem),+++ # The reference attaches a failing end/error hook to the real failure+++ # rather than letting it replace it, so the original is still what the+++ # caller catches and the hook's is reachable underneath.+++ "context": None if problem.__context__ is None+++ else type(problem.__context__).__name__}+++ finally:+++ clear_default_hooks()++++++ run_ids = {entry["run_id"] for entry in log}+++ for entry in log:+++ del entry["run_id"]+++ cases.append({+++ "case": cid,+++ "defaults": [{"name": n, "behaviour": b} for n, b in defaults],+++ "installed": [{"name": n, "behaviour": b} for n, b in installed],+++ "hooks": [{"name": n, "behaviour": b} for n, b in per_call],+++ "states": list(states), "questions": qids, "hooks_raise": hooks_raise,+++ "inference_raises": inference_raises,+++ # What the model was handed, or null when a hook answered before it ran.+++ "inference": None if "states" not in seen else {+++ "states": seen["states"], "questions": seen["questions"],+++ "overrides": seen["overrides"]},+++ "calls": log,+++ "result": None if results is None else [r["usage"]["input_tokens"] for r in results],+++ "raised": raised,+++ "warnings": [str(w.message) for w in caught],+++ # One context per call, shared by every hook of it: that is what lets a tracer+++ # correlate a start with its end without threading state of its own.+++ "one_run_id_per_call": len(run_ids) <= 1,+++ "run_id_hex_length": len(next(iter(run_ids))) if run_ids else None,+++ })+++ return cases++++++-def _decoder_shape(tokenizer_json):- """The decoder section's type chain, flattened, so the fixture says what it was recorded on."""- import io- import json-- document = json.load(io.open(tokenizer_json, encoding="utf-8"))- node = document.get("decoder")- if node is None:- return None-- def shape(n):- kind = n.get("type")- if kind == "Sequence":- return {"type": "Sequence",- "decoders": [shape(step) for step in n.get("decoders", [])]}- return {"type": kind}-- return shape(node)--FAMILIES = {++ + "structured.json": structured,"confidence_gate.json": confidence_gate,- "hooks.json": hook_dispatch,"predict_long.json": predict_long_scan,"window_plan.json": window_plan,"lang_tables.json": lang_tables,@@@@@ -3105,6 -3592,7 -3745,6 -3822,6 +4944,8 @@@@@"python_json.json": python_json,"email.json": email_clean,"predict.json": predict_golden,+ ++ "typed_decisions.json": typed_decisions_golden,++++ "hooks.json": hook_dispatch,}